Documentation
¶
Overview ¶
Package lang runs the native tests of the implementations (spec 09.9).
Each language has a recipe. A recipe copies the files of one implementation to a new temporary directory, writes the build files that the language needs, and runs the test command there. The recipe does not change the content tree.
The tests run code from the content repository. Run them only in a sandbox, such as a CI runner or a container. The CLI asks for -unsafe-local outside CI.
Index ¶
- Constants
- Variables
- func CheckerWrap(box *Sandbox, k *contract.Contract) func([]string) []string
- func LoadImages(path string) (map[string]string, error)
- func Regression(k *contract.Contract, f DiffFailure, found string, day time.Time) contract.Case
- func WriteScript(path string, cmds [][]string) error
- type Adapter
- type DiffFailure
- type DiffResult
- type Inputs
- type Mount
- type Recipe
- type Runner
- func (r *Runner) Available(rc Recipe) error
- func (r *Runner) Gen(dir string, n *node.Node, im node.Impl) (Tree, error)
- func (r *Runner) GenAdapter(dir string, n *node.Node, im node.Impl, k *contract.Contract) (Adapter, error)
- func (r *Runner) Open(n *node.Node, im node.Impl, k *contract.Contract) (*Session, error)
- func (r *Runner) Shared(n *node.Node, im node.Impl, k *contract.Contract, cases []contract.Case) status.Shared
- func (r *Runner) Test(n *node.Node, im node.Impl) status.Test
- func (r *Runner) ToolVersion(rc Recipe) string
- type Sandbox
- type Session
- type Skip
- type Tree
Constants ¶
const JUnitVersion = "1.13.4"
JUnitVersion is the version of the JUnit console launcher for Java and Kotlin.
const MaxDiffFailures = 3
MaxDiffFailures is the number of failures after which the differential run of one implementation stops. One failure is enough to keep the implementation at I1 (rule I03).
const ShrinkBudget = 400
ShrinkBudget is the largest number of runs that the shrinker makes for one failure.
Variables ¶
var ( // HelloTimeout is the longest time that an adapter can take to start. The JVM and .NET start slowly. HelloTimeout = 60 * time.Second // CaseTimeout is the longest time of one shared case. CaseTimeout = 10 * time.Second )
var Recipes = map[string]Recipe{ "python": {ID: "python", Tools: []string{"python3"}, Version: []string{"python3", "--version"}, Run: python}, "go": {ID: "go", Tools: []string{"go"}, Version: []string{"go", "version"}, Run: golang}, "rust": {ID: "rust", Tools: []string{"cargo"}, Version: []string{"rustc", "--version"}, Run: rust}, "c": {ID: "c", Tools: []string{"cc"}, Version: []string{"cc", "--version"}, Run: clang}, "cpp": {ID: "cpp", Tools: []string{"c++"}, Version: []string{"c++", "--version"}, Run: cpp}, "javascript": {ID: "javascript", Tools: []string{"node"}, Version: []string{"node", "--version"}, Run: javascript}, "typescript": {ID: "typescript", Tools: []string{"node"}, Version: []string{"node", "--version"}, Run: typescript}, "java": {ID: "java", Tools: []string{"javac", "java"}, Version: []string{"javac", "-version"}, Run: java}, "kotlin": {ID: "kotlin", Tools: []string{"kotlinc", "java"}, Version: []string{"kotlinc", "-version"}, Run: kotlin}, "csharp": {ID: "csharp", Tools: []string{"dotnet"}, Version: []string{"dotnet", "--version"}, Run: csharp}, }
Recipes are the tier 1 languages (spec 08.2).
var Timeout = 3 * time.Minute
Timeout is the longest time that the tests of one implementation can take.
Functions ¶
func CheckerWrap ¶ added in v0.1.8
CheckerWrap returns the wrap that runs the custom checker of the contract in the Python image of the sandbox, with the node directory as a read-only mount. A nil sandbox gives a nil wrap, so the checker runs on the host.
func LoadImages ¶ added in v0.1.4
LoadImages reads the image manifest: a map from a language to an image reference. Each reference must be pinned by digest, so that a tag that moves cannot change a result.
func Regression ¶ added in v0.1.11
Regression gives the regression case of a shrunk failure for cases/regress.jsonl (spec 09.11): the name regress-<date>-<short hash>, the node and language that it broke, and the oracle output as expect.
func WriteScript ¶ added in v0.1.3
WriteScript writes a POSIX shell script that runs the commands in order, so that a runner image can run a tree without algo.
Types ¶
type Adapter ¶ added in v0.1.3
type Adapter struct {
Dir string
Build [][]string
Run []string
// Types are the types of T that the adapter has.
Types []string
}
Adapter is the generated adapter of one implementation for one problem (spec 09.2).
type DiffFailure ¶ added in v0.1.11
type DiffFailure struct {
Case contract.Case
Reason string
Got json.RawMessage
// Shrunk is the smallest failing case that the shrinker found, with the oracle output as expect, and ShrunkReason its reason. Calls is the number of runs that the shrinker made.
Shrunk contract.Case
ShrunkReason string
ShrunkGot json.RawMessage
Calls int
}
DiffFailure is one generated case that an implementation fails, with the shrunk case.
type DiffResult ¶ added in v0.1.11
type DiffResult struct {
Cases, Passed, Skipped int
Failures []DiffFailure
Took time.Duration
}
DiffResult is the result of the differential run of one implementation.
type Inputs ¶ added in v0.1.11
type Inputs struct {
K *contract.Contract
G *contract.Generated
Gen *contract.Generator
Oracle *contract.Oracle
Cases []contract.Case
}
Inputs are the generated cases of a problem, with the oracle output as expect, and what is needed to check a new input: the generators for pre, and the oracle.
func MakeInputs ¶ added in v0.1.11
func MakeInputs(k *contract.Contract, g *contract.Generated, gen *contract.Generator, oracle *contract.Oracle, plan []contract.GenRequest) (*Inputs, error)
MakeInputs runs the plan with the generators of the problem, and runs the oracle on each input (spec 09.4, 09.5). The caller closes gen and oracle.
type Recipe ¶
type Recipe struct {
ID string
// Tools are the commands that the recipe needs. If one is missing, the language is skipped.
Tools []string
// Version prints the version of the toolchain.
Version []string
// Run prepares dir, which holds a copy of the files, and returns the commands to run in it.
Run func(r *Runner, dir string, n *node.Node, im node.Impl) ([][]string, error)
}
Recipe runs the tests of one language.
func (Recipe) ToolVersion ¶
ToolVersion returns the first line of the version output of the toolchain.
type Runner ¶
type Runner struct {
// Cache holds the downloaded jars. The default is the user cache directory.
Cache string
// Log gets one line for each implementation.
Log func(format string, args ...any)
// Keep keeps the temporary directories, for debugging.
Keep bool
// Box runs the commands in containers. A nil Box runs them on this machine.
Box *Sandbox
// TempDir holds the build trees. The default is the temporary directory of the system. A sandbox needs a directory that the container engine can mount.
TempDir string
// CaseTimeout is the longest time of one case. The default is the constant CaseTimeout. The mutation run uses a shorter time, because a mutant that loops forever is common.
CaseTimeout time.Duration
// Generate is set when the runner only writes a build tree (algo gen). The tree runs later in a runner image, so the tools of this machine are not checked.
Generate bool
}
Runner runs tests and keeps the results.
func (*Runner) Available ¶ added in v0.1.4
Available tells if the language can run with this runner. In a sandbox, the image replaces the tools of this machine.
func (*Runner) Gen ¶ added in v0.1.3
Gen writes the build tree of one implementation to dir: a copy of the files with the tests, and the build files of the language. It never writes to the content tree.
func (*Runner) GenAdapter ¶ added in v0.1.3
func (r *Runner) GenAdapter(dir string, n *node.Node, im node.Impl, k *contract.Contract) (Adapter, error)
GenAdapter writes the adapter of the contract k for one implementation to dir. It does not build it. A language without generics gets the int variant only. `algo test` also drops T = string when the entry does not build with it.
func (*Runner) Open ¶ added in v0.1.11
Open builds the adapter of the implementation for the contract in a new temporary directory, and starts the custom checker when the contract has one. Close removes the directory.
func (*Runner) Shared ¶ added in v0.1.2
func (r *Runner) Shared(n *node.Node, im node.Impl, k *contract.Contract, cases []contract.Case) status.Shared
Shared runs the shared cases of the contract k on one implementation (spec 09.2). It builds the adapter in a new temporary directory, starts it once and sends the cases in order. A crash fails the case, and the next case gets a new adapter.
func (*Runner) ToolVersion ¶ added in v0.1.4
ToolVersion returns the version of the toolchain that runs the tests of the language.
type Sandbox ¶ added in v0.1.4
type Sandbox struct {
// Engine is the container command, such as docker or podman.
Engine string
// Images maps each language to its image, pinned by digest.
Images map[string]string
// Memory is the memory limit of one container, such as 2g.
Memory string
// CPUs is the CPU limit of one container, such as 2.
CPUs string
// Pids is the process limit of one container.
Pids int
// contains filtered or unexported fields
}
Sandbox runs the builds, the tests and the adapters in containers (spec 08.7). Each command gets a new container with no network, a read-only root, a memory limit, a CPU limit and a process limit. The build tree is the only writable mount.
func NewSandbox ¶ added in v0.1.4
NewSandbox returns a sandbox with the default limits.
func (*Sandbox) Argv ¶ added in v0.1.4
Argv returns the command that runs cmd in a new container for the language. The first mount is the work directory. A mount keeps its host path in the container, so that the paths in an error message are the same as in host mode and Tree.Rewrite works on them. HOME points into the work directory, so that the caches of one implementation stay between its commands and go away with the tree.
func (*Sandbox) Ready ¶ added in v0.1.4
Ready pulls the image of the language once. It returns a Skip when the manifest has no image for the language.
type Session ¶ added in v0.1.11
type Session struct {
// Types are the types of T that the adapter supports. SkipReason says why a type of the contract is not in Types, and SkipExpected tells if the skip is expected (a language without generics).
Types []string
SkipReason string
SkipExpected bool
// contains filtered or unexported fields
}
Session is the built adapter of one implementation for one contract, with the custom checker of the contract. It runs one case at a time, so a caller can run many cases, as the differential run and the shrinker do (spec 09.5, 09.11).
func (*Session) Close ¶ added in v0.1.11
func (ss *Session) Close()
Close stops the adapter and the checker, and removes the directory unless the runner keeps it.
func (*Session) Run ¶ added in v0.1.11
Run runs one case. It gives the result of the call, and the reason why the case fails, which is empty when the case passes. A crash, a timeout or a reply that is not valid fails the case, and the next case gets a new adapter. The error is for a problem of the run itself, such as a checker that does not answer.
type Skip ¶
type Skip struct{ Lang, Tool string }
Skip is the error of a language whose tools are missing.
type Tree ¶ added in v0.1.3
type Tree struct {
Dir string
// Test are the commands that build the implementation and run its native tests. They run in Dir.
Test [][]string
// Sources maps each copied file, relative to Dir, to its path in the content repository.
Sources map[string]string
}
Tree is the generated build tree of one implementation (spec 08.5).