Documentation
¶
Overview ¶
Package lang runs the native tests of the implementations (spec 09.9).
Each language has a recipe. A recipe copies the files of one implementation to a new temporary directory, writes the build files that the language needs, and runs the test command there. The recipe does not change the content tree.
The tests run code from the content repository. Run them only in a sandbox, such as a CI runner or a container. The CLI asks for -unsafe-local outside CI.
Index ¶
- Constants
- Variables
- func CheckerWrap(box *Sandbox, k *contract.Contract) func([]string) []string
- func LoadImages(path string) (map[string]string, error)
- func Regression(k *contract.Contract, f DiffFailure, found string, day time.Time) contract.Case
- func WriteScript(path string, cmds [][]string) error
- type Adapter
- type DiffFailure
- type DiffResult
- type Inputs
- type Mount
- type Program
- type Recipe
- type RunResult
- type Runner
- func (r *Runner) Available(rc Recipe) error
- func (r *Runner) Gen(dir string, n *node.Node, im node.Impl) (Tree, error)
- func (r *Runner) GenAdapter(dir string, n *node.Node, im node.Impl, k *contract.Contract) (Adapter, error)
- func (r *Runner) Open(n *node.Node, im node.Impl, k *contract.Contract) (*Session, error)
- func (r *Runner) ProgramReady(p Program) error
- func (r *Runner) RunProgram(p Program, files map[string]string, stdin string, args []string, ...) (RunResult, error)
- func (r *Runner) Shared(n *node.Node, im node.Impl, k *contract.Contract, cases []contract.Case) status.Shared
- func (r *Runner) Test(n *node.Node, im node.Impl) status.Test
- func (r *Runner) ToolVersion(rc Recipe) string
- type Sandbox
- type Session
- type Skip
- type Tree
Constants ¶
const JUnitVersion = "1.13.4"
JUnitVersion is the version of the JUnit console launcher for Java and Kotlin.
const MaxDiffFailures = 3
MaxDiffFailures is the number of failures after which the differential run of one implementation stops. One failure is enough to keep the implementation at I1 (rule I03).
const ShrinkBudget = 400
ShrinkBudget is the largest number of runs that the shrinker makes for one failure.
Variables ¶
var ( // HelloTimeout is the longest time that an adapter can take to start. The JVM and .NET start slowly. HelloTimeout = 60 * time.Second // CaseTimeout is the longest time of one shared case. CaseTimeout = 10 * time.Second )
var Programs = map[string]Program{ "python": {ID: "python", Main: "main.py", Tools: []string{"python3"}, Steps: [][]string{{"python3", "main.py"}}, Version: []string{"python3", "--version"}}, "javascript": {ID: "javascript", Main: "main.js", Tools: []string{"node"}, Steps: [][]string{{"node", "main.js"}}, Version: []string{"node", "--version"}}, "typescript": {ID: "typescript", Main: "main.ts", Tools: []string{"node"}, Steps: [][]string{{"node", "main.ts"}}, Version: []string{"node", "--version"}}, "go": {ID: "go", Main: "main.go", Tools: []string{"go"}, Steps: [][]string{{"go", "build", "-o", "main", "main.go"}, {"./main"}}, Version: []string{"go", "version"}}, "rust": {ID: "rust", Main: "main.rs", Tools: []string{"rustc"}, Steps: [][]string{{"rustc", "--edition", "2024", "-o", "main", "main.rs"}, {"./main"}}, Version: []string{"rustc", "--version"}}, "c": {ID: "c", Main: "main.c", Tools: []string{"cc"}, Steps: [][]string{{"cc", "-std=c23", "-Wall", "-o", "main", "main.c", "-lm"}, {"./main"}}, Version: []string{"cc", "--version"}}, "cpp": {ID: "cpp", Main: "main.cpp", Tools: []string{"c++"}, Steps: [][]string{{"c++", "-std=c++23", "-Wall", "-o", "main", "main.cpp"}, {"./main"}}, Version: []string{"c++", "--version"}}, "java": {ID: "java", Main: "Main.java", Tools: []string{"javac", "java"}, Steps: [][]string{{"javac", "-d", "classes", "Main.java"}, {"java", "-cp", "classes", "Main"}}, Version: []string{"javac", "-version"}}, "kotlin": {ID: "kotlin", Main: "main.kt", Tools: []string{"kotlinc", "java"}, Steps: [][]string{{"kotlinc", "-nowarn", "main.kt", "-include-runtime", "-d", "main.jar"}, {"java", "-jar", "main.jar"}}, Version: []string{"kotlinc", "-version"}}, "csharp": {ID: "csharp", Main: "main.cs", Tools: []string{"dotnet"}, Steps: [][]string{{"dotnet", "build", "main.cs", "-o", "out", "-v", "q", "-nologo", "-clp:NoSummary;ErrorsOnly"}, {"dotnet", "out/main.dll"}}, Version: []string{"dotnet", "--version"}}, "bash": {ID: "bash", Main: "main.sh", Tools: []string{"bash"}, Steps: [][]string{{"bash", "main.sh"}}, Version: []string{"bash", "--version"}}, "powershell": {ID: "powershell", Main: "main.ps1", Tools: []string{"pwsh"}, Steps: [][]string{{"pwsh", "-NoProfile", "-NonInteractive", "-File", "main.ps1"}}, Version: []string{"pwsh", "--version"}}, "sql": {ID: "sql", Main: "main.sql", Tools: []string{"sqlite3"}, Steps: [][]string{{"sqlite3", "-bail", ":memory:", ".read main.sql"}}, Version: []string{"sqlite3", "--version"}}, }
Programs are the languages that prog run can run. Regex and HTML have no runner, so their blocks are norun.
var Recipes = map[string]Recipe{ "python": {ID: "python", Tools: []string{"python3"}, Version: []string{"python3", "--version"}, Run: python}, "go": {ID: "go", Tools: []string{"go"}, Version: []string{"go", "version"}, Run: golang}, "rust": {ID: "rust", Tools: []string{"cargo"}, Version: []string{"rustc", "--version"}, Run: rust}, "c": {ID: "c", Tools: []string{"cc"}, Version: []string{"cc", "--version"}, Run: clang}, "cpp": {ID: "cpp", Tools: []string{"c++"}, Version: []string{"c++", "--version"}, Run: cpp}, "javascript": {ID: "javascript", Tools: []string{"node"}, Version: []string{"node", "--version"}, Run: javascript}, "typescript": {ID: "typescript", Tools: []string{"node"}, Version: []string{"node", "--version"}, Run: typescript}, "java": {ID: "java", Tools: []string{"javac", "java"}, Version: []string{"javac", "-version"}, Run: java}, "kotlin": {ID: "kotlin", Tools: []string{"kotlinc", "java"}, Version: []string{"kotlinc", "-version"}, Run: kotlin}, "csharp": {ID: "csharp", Tools: []string{"dotnet"}, Version: []string{"dotnet", "--version"}, Run: csharp}, "bash": {ID: "bash", Tools: []string{"bash", "bats"}, Version: []string{"bash", "--version"}, Run: bash}, "powershell": {ID: "powershell", Tools: []string{"pwsh"}, Version: []string{"pwsh", "-NoProfile", "-Version"}, Run: powershell}, }
Recipes are the tier 1 languages (spec 08.2).
var Timeout = 3 * time.Minute
Timeout is the longest time that the tests of one implementation can take.
Functions ¶
func CheckerWrap ¶
CheckerWrap returns the wrap that runs the custom checker of the contract in the Python image of the sandbox, with the node directory as a read-only mount. A nil sandbox gives a nil wrap, so the checker runs on the host.
func LoadImages ¶
LoadImages reads the image manifest: a map from a language to an image reference. Each reference must be pinned by digest, so that a tag that moves cannot change a result.
func Regression ¶
Regression gives the regression case of a shrunk failure for cases/regress.jsonl (spec 09.11): the name regress-<date>-<short hash>, the node and language that it broke, and the oracle output as expect.
func WriteScript ¶
WriteScript writes a POSIX shell script that runs the commands in order, so that a runner image can run a tree without prog.
Types ¶
type Adapter ¶
type Adapter struct {
Dir string
Build [][]string
Run []string
// Types are the types of T that the adapter has.
Types []string
}
Adapter is the generated adapter of one implementation for one problem (spec 09.2).
type DiffFailure ¶
type DiffFailure struct {
Case contract.Case
Reason string
Got json.RawMessage
// Shrunk is the smallest failing case that the shrinker found, with the oracle output as expect, and ShrunkReason its reason. Calls is the number of runs that the shrinker made.
Shrunk contract.Case
ShrunkReason string
ShrunkGot json.RawMessage
Calls int
}
DiffFailure is one generated case that an implementation fails, with the shrunk case.
type DiffResult ¶
type DiffResult struct {
Cases, Passed, Skipped int
Failures []DiffFailure
Took time.Duration
}
DiffResult is the result of the differential run of one implementation.
type Inputs ¶
type Inputs struct {
K *contract.Contract
G *contract.Generated
Gen *contract.Generator
Oracle *contract.Oracle
Cases []contract.Case
}
Inputs are the generated cases of a problem, with the oracle output as expect, and what is needed to check a new input: the generators for pre, and the oracle.
func MakeInputs ¶
func MakeInputs(k *contract.Contract, g *contract.Generated, gen *contract.Generator, oracle *contract.Oracle, plan []contract.GenRequest) (*Inputs, error)
MakeInputs runs the plan with the generators of the problem, and runs the oracle on each input (spec 09.4, 09.5). The caller closes gen and oracle.
type Program ¶ added in v0.0.2
type Program struct {
ID string
Main string
Tools []string
Steps [][]string
// Version prints the version of the toolchain, for the record of the run.
Version []string
}
Program is how one language runs one example block (spec 07.4). The block is the file Main. The steps run in order in a new directory; the last step runs the program, and the steps before it build it.
type Recipe ¶
type Recipe struct {
ID string
// Tools are the commands that the recipe needs. If one is missing, the language is skipped.
Tools []string
// Version prints the version of the toolchain.
Version []string
// Run prepares dir, which holds a copy of the files, and returns the commands to run in it.
Run func(r *Runner, dir string, n *node.Node, im node.Impl) ([][]string, error)
}
Recipe runs the tests of one language.
func (Recipe) ToolVersion ¶
ToolVersion returns the first line of the version output of the toolchain.
type RunResult ¶ added in v0.0.2
type RunResult struct {
Stdout, Stderr string
Exit int
// Phase is "build" when a build step failed, and "run" otherwise.
Phase string
TimedOut bool
Ms int64
}
RunResult is what one run of a block gives.
type Runner ¶
type Runner struct {
// Cache holds the downloaded jars. The default is the user cache directory.
Cache string
// Log gets one line for each implementation.
Log func(format string, args ...any)
// Keep keeps the temporary directories, for debugging.
Keep bool
// Box runs the commands in containers. A nil Box runs them on this machine.
Box *Sandbox
// TempDir holds the build trees. The default is the temporary directory of the system. A sandbox needs a directory that the container engine can mount.
TempDir string
// CaseTimeout is the longest time of one case. The default is the constant CaseTimeout. The mutation run uses a shorter time, because a mutant that loops forever is common.
CaseTimeout time.Duration
// Generate is set when the runner only writes a build tree (prog gen). The tree runs later in a runner image, so the tools of this machine are not checked.
Generate bool
}
Runner runs tests and keeps the results.
func (*Runner) Available ¶
Available tells if the language can run with this runner. In a sandbox, the image replaces the tools of this machine.
func (*Runner) Gen ¶
Gen writes the build tree of one implementation to dir: a copy of the files with the tests, and the build files of the language. It never writes to the content tree.
func (*Runner) GenAdapter ¶
func (r *Runner) GenAdapter(dir string, n *node.Node, im node.Impl, k *contract.Contract) (Adapter, error)
GenAdapter writes the adapter of the contract k for one implementation to dir. It does not build it. A language without generics gets the int variant only. `prog test` also drops T = string when the entry does not build with it.
func (*Runner) Open ¶
Open builds the adapter of the implementation for the contract in a new temporary directory, and starts the custom checker when the contract has one. Close removes the directory.
func (*Runner) ProgramReady ¶ added in v0.0.2
ProgramReady tells if the runner can run the blocks of the language. In a sandbox, the image of the language replaces the tools of this machine.
func (*Runner) RunProgram ¶ added in v0.0.2
func (r *Runner) RunProgram(p Program, files map[string]string, stdin string, args []string, timeout time.Duration) (RunResult, error)
RunProgram writes the files to a new directory and runs the steps of the language. files maps a file name to its content; the block itself is under the Main name of the language. The output has the paths of the directory removed.
func (*Runner) Shared ¶
func (r *Runner) Shared(n *node.Node, im node.Impl, k *contract.Contract, cases []contract.Case) status.Shared
Shared runs the shared cases of the contract k on one implementation (spec 09.2). It builds the adapter in a new temporary directory, starts it once and sends the cases in order. A crash fails the case, and the next case gets a new adapter.
func (*Runner) ToolVersion ¶
ToolVersion returns the version of the toolchain that runs the tests of the language.
type Sandbox ¶
type Sandbox struct {
// Engine is the container command, such as docker or podman.
Engine string
// Images maps each language to its image, pinned by digest.
Images map[string]string
// Memory is the memory limit of one container, such as 2g.
Memory string
// CPUs is the CPU limit of one container, such as 2.
CPUs string
// Pids is the process limit of one container.
Pids int
// contains filtered or unexported fields
}
Sandbox runs the builds, the tests and the adapters in containers (spec 08.7). Each command gets a new container with no network, a read-only root, a memory limit, a CPU limit and a process limit. The build tree is the only writable mount.
func NewSandbox ¶
NewSandbox returns a sandbox with the default limits.
func (*Sandbox) Argv ¶
Argv returns the command that runs cmd in a new container for the language. The first mount is the work directory. A mount keeps its host path in the container, so that the paths in an error message are the same as in host mode and Tree.Rewrite works on them. HOME points into the work directory, so that the caches of one implementation stay between its commands and go away with the tree.
func (*Sandbox) Ready ¶
Ready pulls the image of the language once. It returns a Skip when the manifest has no image for the language.
type Session ¶
type Session struct {
// Types are the types of T that the adapter supports. SkipReason says why a type of the contract is not in Types, and SkipExpected tells if the skip is expected (a language without generics).
Types []string
SkipReason string
SkipExpected bool
// contains filtered or unexported fields
}
Session is the built adapter of one implementation for one contract, with the custom checker of the contract. It runs one case at a time, so a caller can run many cases, as the differential run and the shrinker do (spec 09.5, 09.11).
func (*Session) Close ¶
func (ss *Session) Close()
Close stops the adapter and the checker, and removes the directory unless the runner keeps it.
func (*Session) Run ¶
Run runs one case. It gives the result of the call, and the reason why the case fails, which is empty when the case passes. A crash, a timeout or a reply that is not valid fails the case, and the next case gets a new adapter. The error is for a problem of the run itself, such as a checker that does not answer.
type Skip ¶
type Skip struct{ Lang, Tool string }
Skip is the error of a language whose tools are missing.
type Tree ¶
type Tree struct {
Dir string
// Test are the commands that build the implementation and run its native tests. They run in Dir.
Test [][]string
// Sources maps each copied file, relative to Dir, to its path in the content repository.
Sources map[string]string
}
Tree is the generated build tree of one implementation (spec 08.5).