lang

package
v0.0.9 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 12, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

Documentation

Overview

Package lang runs the native tests of the implementations (spec 09.9).

Each language has a recipe. A recipe copies the files of one implementation to a new temporary directory, writes the build files that the language needs, and runs the test command there. The recipe does not change the content tree.

The tests run code from the content repository. Run them only in a sandbox, such as a CI runner or a container. The CLI asks for -unsafe-local outside CI.

Index

Constants

View Source
const (
	// ArgTest is the file of the test block.
	ArgTest = "{test}"
	// ArgSources is the files of the block that have the extension of Main, in sorted order.
	ArgSources = "{sources}"
)

The placeholders in the steps of a test runner. WithFiles replaces them with the names of the files of the block.

View Source
const JUnitVersion = "1.13.4"

JUnitVersion is the version of the JUnit console launcher for Java and Kotlin.

View Source
const MaxDiffFailures = 3

MaxDiffFailures is the number of failures after which the differential run of one implementation stops. One failure is enough to keep the implementation at I1 (rule I03).

View Source
const ShrinkBudget = 400

ShrinkBudget is the largest number of runs that the shrinker makes for one failure.

Variables

View Source
var (
	// HelloTimeout is the longest time that an adapter can take to start. The JVM and .NET start slowly.
	HelloTimeout = 60 * time.Second
	// CaseTimeout is the longest time of one shared case.
	CaseTimeout = 10 * time.Second
)
View Source
var BuildTimeout = 2 * time.Minute

BuildTimeout is the longest time of one build step of a program, such as javac or cargo build. The timeout of a block applies to the last step only, which runs the program.

View Source
var DepsSupport = map[string]Deps{
	"rust": {Lock: "Cargo.lock", With: []string{"Cargo.toml"}, Dockerfile: rustDockerfile, Program: rustDeps, Test: rustTestDeps, Prepare: rustPrepare},
	"java": {Lock: "pom.xml", Dockerfile: javaDockerfile, Program: javaDeps, Test: javaDeps, Prepare: javaPrepare},
	"cpp":  {Lock: "vcpkg.json", Optional: []string{"vcpkg-configuration.json"}, Dockerfile: cppDockerfile, Program: cppDeps, Test: cppTestDeps},

	"csharp": {Lock: "*.csproj", Test: csharpTestDeps, Prepare: csharpPrepare, Missing: nugetMissing},
}

DepsSupport are the languages that can run a block with deps=. A block of another language with deps= is skipped with the reason deps.

View Source
var DepsTimeout = 20 * time.Minute

DepsTimeout is the longest time that the packages of one lock file can take to build, in an image or on this machine.

View Source
var Programs = map[string]Program{
	"python":     {ID: "python", Main: "main.py", Tools: []string{"python3"}, Steps: [][]string{{"python3", "main.py"}}, Version: []string{"python3", "--version"}},
	"javascript": {ID: "javascript", Main: "main.js", Tools: []string{"node"}, Steps: [][]string{{"node", "main.js"}}, Version: []string{"node", "--version"}},
	"typescript": {ID: "typescript", Main: "main.ts", Tools: []string{"node"}, Steps: [][]string{{"node", "main.ts"}}, Version: []string{"node", "--version"}},
	"go":         {ID: "go", Main: "main.go", Tools: []string{"go"}, Steps: [][]string{{"go", "build", "-o", "main", "main.go"}, {"./main"}}, Version: []string{"go", "version"}},
	"rust":       {ID: "rust", Main: "main.rs", Tools: []string{"rustc"}, Steps: [][]string{{"rustc", "--edition", "2024", "-o", "main", "main.rs"}, {"./main"}}, Version: []string{"rustc", "--version"}},
	"c":          {ID: "c", Main: "main.c", Tools: []string{"cc"}, Steps: [][]string{{"cc", "-std=c23", "-Wall", "-o", "main", "main.c", "-lm"}, {"./main"}}, Version: []string{"cc", "--version"}},
	"cpp":        {ID: "cpp", Main: "main.cpp", Tools: []string{"c++"}, Steps: [][]string{{"c++", "-std=c++23", "-Wall", "-o", "main", "main.cpp"}, {"./main"}}, Version: []string{"c++", "--version"}},
	"java":       {ID: "java", Main: "Main.java", Tools: []string{"javac", "java"}, Steps: [][]string{{"javac", "-d", "classes", "Main.java"}, {"java", "-cp", "classes", "Main"}}, Version: []string{"javac", "-version"}},
	"kotlin":     {ID: "kotlin", Main: "main.kt", Tools: []string{"kotlinc", "java"}, Steps: [][]string{{"kotlinc", "-nowarn", "main.kt", "-include-runtime", "-d", "main.jar"}, {"java", "-jar", "main.jar"}}, Version: []string{"kotlinc", "-version"}},
	"csharp":     {ID: "csharp", Main: "main.cs", Tools: []string{"dotnet"}, Steps: [][]string{{"dotnet", "build", "main.cs", "-p:PublishAot=false", "-o", "out", "-v", "q", "-nologo", "-clp:NoSummary;ErrorsOnly"}, {"dotnet", "out/main.dll"}}, Version: []string{"dotnet", "--version"}},
	"bash":       {ID: "bash", Main: "main.sh", Tools: []string{"bash"}, Steps: [][]string{{"bash", "main.sh"}}, Version: []string{"bash", "--version"}},
	"powershell": {ID: "powershell", Main: "main.ps1", Tools: []string{"pwsh"}, Steps: [][]string{{"pwsh", "-NoProfile", "-NonInteractive", "-File", "main.ps1"}}, Version: []string{"pwsh", "--version"}},
	"sql":        {ID: "sql", Main: "main.sql", Tools: []string{"sqlite3"}, Steps: [][]string{{"sqlite3", "-bail", ":memory:", ".read main.sql"}}, Version: []string{"sqlite3", "--version"}},
}

Programs are the languages that prog run can run. Regex and HTML have no runner, so their blocks are norun. The test blocks use TestPrograms.

View Source
var Recipes = map[string]Recipe{
	"python":     {ID: "python", Tools: []string{"python3"}, Version: []string{"python3", "--version"}, Run: python},
	"go":         {ID: "go", Tools: []string{"go"}, Version: []string{"go", "version"}, Run: golang},
	"rust":       {ID: "rust", Tools: []string{"cargo"}, Version: []string{"rustc", "--version"}, Run: rust},
	"c":          {ID: "c", Tools: []string{"cc"}, Version: []string{"cc", "--version"}, Run: clang},
	"cpp":        {ID: "cpp", Tools: []string{"c++"}, Version: []string{"c++", "--version"}, Run: cpp},
	"javascript": {ID: "javascript", Tools: []string{"node"}, Version: []string{"node", "--version"}, Run: javascript},
	"typescript": {ID: "typescript", Tools: []string{"node"}, Version: []string{"node", "--version"}, Run: typescript},
	"java":       {ID: "java", Tools: []string{"javac", "java"}, Version: []string{"javac", "-version"}, Run: java},
	"kotlin":     {ID: "kotlin", Tools: []string{"kotlinc", "java"}, Version: []string{"kotlinc", "-version"}, Run: kotlin},
	"csharp":     {ID: "csharp", Tools: []string{"dotnet"}, Version: []string{"dotnet", "--version"}, Run: csharp},
	"bash":       {ID: "bash", Tools: []string{"bash", "bats"}, Version: []string{"bash", "--version"}, Run: bash},
	"powershell": {ID: "powershell", Tools: []string{"pwsh"}, Version: []string{"pwsh", "-NoProfile", "-Version"}, Run: powershell},
}

Recipes are the tier 1 languages (spec 08.2).

View Source
var TestPrograms = map[string]Program{

	"python":     {ID: "python", Main: "test_main.py", Tools: []string{"python3"}, Steps: [][]string{{"python3", "-m", "unittest", "-v", ArgTest}}, Version: []string{"python3", "--version"}, Test: true, Report: "stderr"},
	"javascript": {ID: "javascript", Main: "main.test.js", Tools: []string{"node"}, Steps: [][]string{{"node", "--test", "--test-reporter=spec", ArgTest}}, Version: []string{"node", "--version"}, Test: true},

	"typescript": {ID: "typescript", Main: "main.test.ts", Tools: []string{"node"}, Steps: [][]string{{"node", "--test", "--test-reporter=spec", ArgTest}}, Version: []string{"node", "--version"}, Test: true},

	"go":     {ID: "go", Main: "main_test.go", Tools: []string{"go"}, Steps: [][]string{{"go", "mod", "init", "example"}, {"go", "test", "-v", "-count=1", "."}}, Version: []string{"go", "version"}, Test: true, Setup: 1},
	"rust":   {ID: "rust", Main: "main.rs", Tools: []string{"rustc"}, Steps: [][]string{{"rustc", "--edition", "2024", "--test", "-o", "main_test.exe", ArgTest}, {"./main_test.exe", "--test-threads=1"}}, Version: []string{"rustc", "--version"}, Test: true},
	"java":   {ID: "java", Main: "MainTest.java", Tools: []string{"javac", "java"}, Steps: javaTestSteps(imageJUnit, ":"), Version: []string{"javac", "-version"}, Test: true},
	"csharp": {ID: "csharp", Main: "MainTests.cs", Tools: []string{"dotnet"}, Version: []string{"dotnet", "--version"}, Test: true},
	"cpp":    {ID: "cpp", Main: "main_test.cpp", Tools: []string{"c++"}, Version: []string{"c++", "--version"}, Test: true},
}

TestPrograms are the test runners of the test blocks (spec 07.11). A test block is a test file, not a program with a main. The last step runs the tests, and its exit status is 0 when every test passes. The steps can hold ArgTest and ArgSources. C# and C++ have no steps here, because their test frameworks are third-party packages: a block of these languages runs only with deps=, and the steps come from DepsSupport.

View Source
var Timeout = 3 * time.Minute

Timeout is the longest time that the tests of one implementation can take.

Functions

func CheckerWrap

func CheckerWrap(box *Sandbox, k *contract.Contract) func([]string) []string

CheckerWrap returns the wrap that runs the custom checker of the contract in the Python image of the sandbox, with the node directory as a read-only mount. A nil sandbox gives a nil wrap, so the checker runs on the host.

func DepsDockerfile added in v0.0.3

func DepsDockerfile(d Deps, base string) string

DepsDockerfile gives the Dockerfile of the image with the packages, on top of the image base.

func DepsMissing added in v0.0.3

func DepsMissing(lang string, got RunResult) bool

DepsMissing tells if a failed build of a block with deps= failed because the packages are not there. The block is then a skip with the reason deps, not a failure.

func DepsTag added in v0.0.3

func DepsTag(lang, base, dockerfile string, files map[string][]byte) string

DepsTag gives the tag of the image with the packages. It is a hash of the image of the language, the Dockerfile and the files of the build context. The same inputs give the same tag, so a second run finds the image and does not build it again.

func LoadImages

func LoadImages(path string) (map[string]string, error)

LoadImages reads the image manifest: a map from a language to an image reference. Each reference must be pinned by digest, so that a tag that moves cannot change a result.

func Regression

func Regression(k *contract.Contract, f DiffFailure, found string, day time.Time) contract.Case

Regression gives the regression case of a shrunk failure for cases/regress.jsonl (spec 09.11): the name regress-<date>-<short hash>, the node and language that it broke, and the oracle output as expect.

func WriteScript

func WriteScript(path string, cmds [][]string) error

WriteScript writes a POSIX shell script that runs the commands in order, so that a runner image can run a tree without prog.

Types

type Adapter

type Adapter struct {
	Dir   string
	Build [][]string
	Run   []string
	// Types are the types of T that the adapter has.
	Types []string
}

Adapter is the generated adapter of one implementation for one problem (spec 09.2).

type Deps added in v0.0.3

type Deps struct {
	// Lock is the name of the lock file, such as Cargo.lock, or a pattern of names, such as *.csproj.
	Lock string
	// With are the other files next to the lock file that the build needs, such as Cargo.toml.
	With []string
	// Optional are files next to the lock file that the build uses when they are there.
	Optional []string
	// Dockerfile is the text of the image with the packages. %[1]s is the image of the language. The build context holds the lock file and the files of With and Optional. It is "" when the image of the language has the packages already, so the block runs in that image.
	Dockerfile string
	// Program changes the program of a block so that it uses the packages. files are the lock file and the files of With and Optional, by name. dir is the directory of the packages that Prepare made on this machine. It is "" in the image. Program is nil when the language runs only the test blocks with deps=.
	Program func(p Program, files map[string][]byte, dir string) (Program, error)
	// Test is Program for the test runner of a test block (spec 07.11). It is nil when the language cannot run a test block with deps=.
	Test func(p Program, files map[string][]byte, dir string) (Program, error)
	// Missing matches the output of a failed build when the packages are not there, such as a restore error of NuGet. The block is then a skip with the reason deps.
	Missing *regexp.Regexp
	// Prepare gets the packages into dir on this machine, with no network. dir holds the lock file and the files of With and Optional. Prepare is nil when the language has no local support.
	Prepare func(ctx context.Context, dir string) error
}

Deps is how a language gets the third-party packages of a lock file (spec 07.7). The image of the language does not have them. In a sandbox, the runner builds an image from the image of the language with the packages in it. The image build is the only step with network, and the block runs in the new image with no network. On this machine, the runner gets the packages with the local toolchain and no network, or the block is skipped.

func DepsFor added in v0.0.3

func DepsFor(lang string, sandbox, test bool) (Deps, error)

DepsFor returns how the language gets its packages. test tells if the block is a test block. It returns a DepsSkip when the language has no support for deps= in the mode of the block, or no support on this machine when sandbox is false.

type DepsSkip added in v0.0.3

type DepsSkip struct{ Lang, Why string }

DepsSkip is the error of a block with deps= that cannot get its packages here. The block is skipped with the reason deps. It is not a failure.

func (DepsSkip) Error added in v0.0.3

func (e DepsSkip) Error() string

type DiffFailure

type DiffFailure struct {
	Case   contract.Case
	Reason string
	Got    json.RawMessage
	// Shrunk is the smallest failing case that the shrinker found, with the oracle output as expect, and ShrunkReason its reason. Calls is the number of runs that the shrinker made.
	Shrunk       contract.Case
	ShrunkReason string
	ShrunkGot    json.RawMessage
	Calls        int
}

DiffFailure is one generated case that an implementation fails, with the shrunk case.

type DiffResult

type DiffResult struct {
	Cases, Passed, Skipped int
	Failures               []DiffFailure
	Took                   time.Duration
}

DiffResult is the result of the differential run of one implementation.

func Diff

func Diff(ss *Session, in *Inputs) (DiffResult, error)

Diff runs each generated case on the implementation of the session, and shrinks each failure (spec 09.5, 09.11).

type Inputs

type Inputs struct {
	K      *contract.Contract
	G      *contract.Generated
	Gen    *contract.Generator
	Oracle *contract.Oracle
	Cases  []contract.Case
}

Inputs are the generated cases of a problem, with the oracle output as expect, and what is needed to check a new input: the generators for pre, and the oracle.

func MakeInputs

func MakeInputs(k *contract.Contract, g *contract.Generated, gen *contract.Generator, oracle *contract.Oracle, plan []contract.GenRequest) (*Inputs, error)

MakeInputs runs the plan with the generators of the problem, and runs the oracle on each input (spec 09.4, 09.5). The caller closes gen and oracle.

type Mount

type Mount struct {
	Dir      string
	ReadOnly bool
}

Mount is a host directory that the container sees at the same path.

type Program added in v0.0.2

type Program struct {
	ID    string
	Main  string
	Tools []string
	Steps [][]string
	// Version prints the version of the toolchain, for the record of the run.
	Version []string
	// Image is the image that runs the steps in a sandbox. It is empty for the image of the language. A block with deps= runs in an image that has the packages of its lock file (spec 07.7).
	Image string
	// Seed is a directory on this machine. RunProgram copies its files into the work directory before the files of the block. A block with deps= gets the packages of its lock file this way when it runs on this machine.
	Seed string
	// Files are more files that RunProgram writes into the work directory after the seed and before the files of the block, such as the .csproj of a C# test block.
	Files map[string][]byte
	// Test is true for the test runner of a test block (spec 07.11). A block with deps= then uses the Test function of DepsSupport.
	Test bool
	// Report is the stream of the report of a test runner: "" for stdout, or stderr. It is the default stream of the output block of a test block.
	Report string
	// Setup is the number of first steps that only prepare the directory, such as go mod init. The flags of a block go to the step after them.
	Setup int
}

Program is how one language runs one example block (spec 07.4). The block is the file Main. The steps run in order in a new directory; the last step runs the program, and the steps before it build it.

func TestProgramFor added in v0.0.3

func TestProgramFor(lang string, deps bool) (Program, error)

TestProgramFor gives the test runner of a language. deps tells if the block has deps=. It returns a TestSkip when prog run has no test runner for the language, or when the runner needs deps= and the block has none.

func (Program) WithFiles added in v0.0.3

func (p Program) WithFiles(files map[string]string, test string) Program

WithFiles replaces ArgTest with test, the file of the test block, and ArgSources with the files that have the extension of Main.

func (Program) WithTool added in v0.0.2

func (p Program) WithTool(from, to string) Program

WithTool gives a copy of the program that uses the command to in place of the tool from, such as python3.15 in place of python3.

type Recipe

type Recipe struct {
	ID string
	// Tools are the commands that the recipe needs. If one is missing, the language is skipped.
	Tools []string
	// Version prints the version of the toolchain.
	Version []string
	// Run prepares dir, which holds a copy of the files, and returns the commands to run in it.
	Run func(r *Runner, dir string, n *node.Node, im node.Impl) ([][]string, error)
}

Recipe runs the tests of one language.

func (Recipe) Available

func (rc Recipe) Available() error

Available tells if the tools of a recipe are installed.

func (Recipe) ToolVersion

func (rc Recipe) ToolVersion() string

ToolVersion returns the first line of the version output of the toolchain.

type RunResult added in v0.0.2

type RunResult struct {
	Stdout, Stderr string
	Exit           int
	// Phase is "build" when a build step failed, and "run" otherwise.
	Phase    string
	TimedOut bool
	Ms       int64
}

RunResult is what one run of a block gives.

type Runner

type Runner struct {
	// Cache holds the downloaded jars. The default is the user cache directory.
	Cache string
	// Log gets one line for each implementation.
	Log func(format string, args ...any)
	// Keep keeps the temporary directories, for debugging.
	Keep bool
	// Box runs the commands in containers. A nil Box runs them on this machine.
	Box *Sandbox
	// TempDir holds the build trees. The default is the temporary directory of the system. A sandbox needs a directory that the container engine can mount.
	TempDir string
	// CaseTimeout is the longest time of one case. The default is the constant CaseTimeout. The mutation run uses a shorter time, because a mutant that loops forever is common.
	CaseTimeout time.Duration
	// Generate is set when the runner only writes a build tree (prog gen). The tree runs later in a runner image, so the tools of this machine are not checked.
	Generate bool
}

Runner runs tests and keeps the results.

func (*Runner) Available

func (r *Runner) Available(rc Recipe) error

Available tells if the language can run with this runner. In a sandbox, the image replaces the tools of this machine.

func (*Runner) Gen

func (r *Runner) Gen(dir string, n *node.Node, im node.Impl) (Tree, error)

Gen writes the build tree of one implementation to dir: a copy of the files with the tests, and the build files of the language. It never writes to the content tree.

func (*Runner) GenAdapter

func (r *Runner) GenAdapter(dir string, n *node.Node, im node.Impl, k *contract.Contract) (Adapter, error)

GenAdapter writes the adapter of the contract k for one implementation to dir. It does not build it. A language without generics gets the int variant only. `prog test` also drops T = string when the entry does not build with it.

func (*Runner) Open

func (r *Runner) Open(n *node.Node, im node.Impl, k *contract.Contract) (*Session, error)

Open builds the adapter of the implementation for the contract in a new temporary directory, and starts the custom checker when the contract has one. Close removes the directory.

func (*Runner) ProgramReady added in v0.0.2

func (r *Runner) ProgramReady(p Program) error

ProgramReady tells if the runner can run the blocks of the language. In a sandbox, the image of the language replaces the tools of this machine.

func (*Runner) ResolveTest added in v0.0.3

func (r *Runner) ResolveTest(p Program) (Program, error)

ResolveTest puts the paths of this machine into a test runner, such as the path of the JUnit jar. In a sandbox, the paths of the image stay. It downloads the JUnit jar once when it is not in the cache.

func (*Runner) RunProgram added in v0.0.2

func (r *Runner) RunProgram(p Program, files map[string]string, stdin string, args []string, timeout time.Duration) (RunResult, error)

RunProgram writes the files to a new directory and runs the steps of the language. files maps a file name to its content; the block itself is under the Main name of the language. The output has the paths of the directory removed.

func (*Runner) Shared

func (r *Runner) Shared(n *node.Node, im node.Impl, k *contract.Contract, cases []contract.Case) status.Shared

Shared runs the shared cases of the contract k on one implementation (spec 09.2). It builds the adapter in a new temporary directory, starts it once and sends the cases in order. A crash fails the case, and the next case gets a new adapter.

func (*Runner) Test

func (r *Runner) Test(n *node.Node, im node.Impl) status.Test

Test runs the native tests of one implementation.

func (*Runner) ToolVersion

func (r *Runner) ToolVersion(rc Recipe) string

ToolVersion returns the version of the toolchain that runs the tests of the language.

func (*Runner) WithDeps added in v0.0.3

func (r *Runner) WithDeps(p Program, lock string) (Program, error)

WithDeps changes the program of a block with deps= so that it uses the packages of the lock file (spec 07.7). In a sandbox, it builds the image with the packages, or reuses it. On this machine, it gets the packages with no network into the cache directory. It returns a DepsSkip when the packages cannot be had here.

type Sandbox

type Sandbox struct {
	// Engine is the container command, such as docker or podman.
	Engine string
	// Images maps each language to its image, pinned by digest.
	Images map[string]string
	// Memory is the memory limit of one container, such as 2g.
	Memory string
	// CPUs is the CPU limit of one container, such as 2.
	CPUs string
	// Pids is the process limit of one container.
	Pids int
	// contains filtered or unexported fields
}

Sandbox runs the builds, the tests and the adapters in containers (spec 08.7). Each command gets a new container with no network, a read-only root, a memory limit, a CPU limit and a process limit. The build tree is the only writable mount.

func NewSandbox

func NewSandbox(engine string, images map[string]string) *Sandbox

NewSandbox returns a sandbox with the default limits.

func (*Sandbox) Argv

func (s *Sandbox) Argv(lang string, interactive bool, mounts []Mount, cmd []string) []string

Argv returns the command that runs cmd in a new container for the language. The first mount is the work directory. A mount keeps its host path in the container, so that the paths in an error message are the same as in host mode and Tree.Rewrite works on them. HOME points into the work directory, so that the caches of one implementation stay between its commands and go away with the tree.

func (*Sandbox) ArgvImage added in v0.0.3

func (s *Sandbox) ArgvImage(image string, interactive bool, mounts []Mount, cmd []string) []string

ArgvImage is Argv with an image that is not in the manifest, such as the image with the packages of a lock file (spec 07.7). The container has the same limits and no network.

func (*Sandbox) Derive added in v0.0.3

func (s *Sandbox) Derive(lang string, d Deps, files map[string][]byte) (string, error)

Derive builds the image of the language with the packages of a lock file, or reuses it (spec 07.7). The tag comes from DepsTag. The build is the only step with network.

func (*Sandbox) Ready

func (s *Sandbox) Ready(lang string) error

Ready pulls the image of the language once. It returns a Skip when the manifest has no image for the language.

func (*Sandbox) Reap

func (s *Sandbox) Reap(work ...string)

Reap removes the containers that are still there, such as the container of an adapter that was killed after a timeout. Killing the engine client does not stop its container. With work directories, Reap removes only the containers that have one of them as the work directory, so that a run that ends does not stop the runs that go on at the same time. With no argument, Reap removes all the containers of the sandbox.

func (*Sandbox) Version

func (s *Sandbox) Version(lang string, cmd []string) string

Version returns the first line of the version output of the toolchain in the image.

type Session

type Session struct {

	// Types are the types of T that the adapter supports. SkipReason says why a type of the contract is not in Types, and SkipExpected tells if the skip is expected (a language without generics).
	Types        []string
	SkipReason   string
	SkipExpected bool
	// contains filtered or unexported fields
}

Session is the built adapter of one implementation for one contract, with the custom checker of the contract. It runs one case at a time, so a caller can run many cases, as the differential run and the shrinker do (spec 09.5, 09.11).

func (*Session) Close

func (ss *Session) Close()

Close stops the adapter and the checker, and removes the directory unless the runner keeps it.

func (*Session) Run

func (ss *Session) Run(c contract.Case) (json.RawMessage, string, error)

Run runs one case. It gives the result of the call, and the reason why the case fails, which is empty when the case passes. A crash, a timeout or a reply that is not valid fails the case, and the next case gets a new adapter. The error is for a problem of the run itself, such as a checker that does not answer.

func (*Session) Supports

func (ss *Session) Supports(c contract.Case) bool

Supports tells if the adapter supports the type of T of the case.

type Skip

type Skip struct{ Lang, Tool string }

Skip is the error of a language whose tools are missing.

func (Skip) Error

func (s Skip) Error() string

type TestSkip added in v0.0.3

type TestSkip struct{ Lang, Reason, Why string }

TestSkip is the error of a test block that prog run cannot run. Reason is runner when the language has no test runner here, or deps when the test framework is a package and the block has no deps=. It is a skip, not a failure.

func (TestSkip) Error added in v0.0.3

func (e TestSkip) Error() string

type Tree

type Tree struct {
	Dir string
	// Test are the commands that build the implementation and run its native tests. They run in Dir.
	Test [][]string
	// Sources maps each copied file, relative to Dir, to its path in the content repository.
	Sources map[string]string
}

Tree is the generated build tree of one implementation (spec 08.5).

func (Tree) Rewrite

func (t Tree) Rewrite(s string) string

Rewrite changes the paths of the tree in a compiler error or a test failure to the paths in the content repository, so that the author sees nodes/.../merge_sort.rs:14 (spec 08.5).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL