lang

package
v0.1.12 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 3, 2026 License: Apache-2.0 Imports: 25 Imported by: 0

Documentation

Overview

Package lang runs the native tests of the implementations (spec 09.9).

Each language has a recipe. A recipe copies the files of one implementation to a new temporary directory, writes the build files that the language needs, and runs the test command there. The recipe does not change the content tree.

The tests run code from the content repository. Run them only in a sandbox, such as a CI runner or a container. The CLI asks for -unsafe-local outside CI.

Index

Constants

View Source
const JUnitVersion = "1.13.4"

JUnitVersion is the version of the JUnit console launcher for Java and Kotlin.

View Source
const MaxDiffFailures = 3

MaxDiffFailures is the number of failures after which the differential run of one implementation stops. One failure is enough to keep the implementation at I1 (rule I03).

View Source
const ShrinkBudget = 400

ShrinkBudget is the largest number of runs that the shrinker makes for one failure.

Variables

View Source
var (
	// HelloTimeout is the longest time that an adapter can take to start. The JVM and .NET start slowly.
	HelloTimeout = 60 * time.Second
	// CaseTimeout is the longest time of one shared case.
	CaseTimeout = 10 * time.Second
)
View Source
var Recipes = map[string]Recipe{
	"python":     {ID: "python", Tools: []string{"python3"}, Version: []string{"python3", "--version"}, Run: python},
	"go":         {ID: "go", Tools: []string{"go"}, Version: []string{"go", "version"}, Run: golang},
	"rust":       {ID: "rust", Tools: []string{"cargo"}, Version: []string{"rustc", "--version"}, Run: rust},
	"c":          {ID: "c", Tools: []string{"cc"}, Version: []string{"cc", "--version"}, Run: clang},
	"cpp":        {ID: "cpp", Tools: []string{"c++"}, Version: []string{"c++", "--version"}, Run: cpp},
	"javascript": {ID: "javascript", Tools: []string{"node"}, Version: []string{"node", "--version"}, Run: javascript},
	"typescript": {ID: "typescript", Tools: []string{"node"}, Version: []string{"node", "--version"}, Run: typescript},
	"java":       {ID: "java", Tools: []string{"javac", "java"}, Version: []string{"javac", "-version"}, Run: java},
	"kotlin":     {ID: "kotlin", Tools: []string{"kotlinc", "java"}, Version: []string{"kotlinc", "-version"}, Run: kotlin},
	"csharp":     {ID: "csharp", Tools: []string{"dotnet"}, Version: []string{"dotnet", "--version"}, Run: csharp},
}

Recipes are the tier 1 languages (spec 08.2).

View Source
var Timeout = 3 * time.Minute

Timeout is the longest time that the tests of one implementation can take.

Functions

func CheckerWrap added in v0.1.8

func CheckerWrap(box *Sandbox, k *contract.Contract) func([]string) []string

CheckerWrap returns the wrap that runs the custom checker of the contract in the Python image of the sandbox, with the node directory as a read-only mount. A nil sandbox gives a nil wrap, so the checker runs on the host.

func LoadImages added in v0.1.4

func LoadImages(path string) (map[string]string, error)

LoadImages reads the image manifest: a map from a language to an image reference. Each reference must be pinned by digest, so that a tag that moves cannot change a result.

func Regression added in v0.1.11

func Regression(k *contract.Contract, f DiffFailure, found string, day time.Time) contract.Case

Regression gives the regression case of a shrunk failure for cases/regress.jsonl (spec 09.11): the name regress-<date>-<short hash>, the node and language that it broke, and the oracle output as expect.

func WriteScript added in v0.1.3

func WriteScript(path string, cmds [][]string) error

WriteScript writes a POSIX shell script that runs the commands in order, so that a runner image can run a tree without algo.

Types

type Adapter added in v0.1.3

type Adapter struct {
	Dir   string
	Build [][]string
	Run   []string
	// Types are the types of T that the adapter has.
	Types []string
}

Adapter is the generated adapter of one implementation for one problem (spec 09.2).

type DiffFailure added in v0.1.11

type DiffFailure struct {
	Case   contract.Case
	Reason string
	Got    json.RawMessage
	// Shrunk is the smallest failing case that the shrinker found, with the oracle output as expect, and ShrunkReason its reason. Calls is the number of runs that the shrinker made.
	Shrunk       contract.Case
	ShrunkReason string
	ShrunkGot    json.RawMessage
	Calls        int
}

DiffFailure is one generated case that an implementation fails, with the shrunk case.

type DiffResult added in v0.1.11

type DiffResult struct {
	Cases, Passed, Skipped int
	Failures               []DiffFailure
	Took                   time.Duration
}

DiffResult is the result of the differential run of one implementation.

func Diff added in v0.1.11

func Diff(ss *Session, in *Inputs) (DiffResult, error)

Diff runs each generated case on the implementation of the session, and shrinks each failure (spec 09.5, 09.11).

type Inputs added in v0.1.11

type Inputs struct {
	K      *contract.Contract
	G      *contract.Generated
	Gen    *contract.Generator
	Oracle *contract.Oracle
	Cases  []contract.Case
}

Inputs are the generated cases of a problem, with the oracle output as expect, and what is needed to check a new input: the generators for pre, and the oracle.

func MakeInputs added in v0.1.11

func MakeInputs(k *contract.Contract, g *contract.Generated, gen *contract.Generator, oracle *contract.Oracle, plan []contract.GenRequest) (*Inputs, error)

MakeInputs runs the plan with the generators of the problem, and runs the oracle on each input (spec 09.4, 09.5). The caller closes gen and oracle.

type Mount added in v0.1.4

type Mount struct {
	Dir      string
	ReadOnly bool
}

Mount is a host directory that the container sees at the same path.

type Recipe

type Recipe struct {
	ID string
	// Tools are the commands that the recipe needs. If one is missing, the language is skipped.
	Tools []string
	// Version prints the version of the toolchain.
	Version []string
	// Run prepares dir, which holds a copy of the files, and returns the commands to run in it.
	Run func(r *Runner, dir string, n *node.Node, im node.Impl) ([][]string, error)
}

Recipe runs the tests of one language.

func (Recipe) Available

func (rc Recipe) Available() error

Available tells if the tools of a recipe are installed.

func (Recipe) ToolVersion

func (rc Recipe) ToolVersion() string

ToolVersion returns the first line of the version output of the toolchain.

type Runner

type Runner struct {
	// Cache holds the downloaded jars. The default is the user cache directory.
	Cache string
	// Log gets one line for each implementation.
	Log func(format string, args ...any)
	// Keep keeps the temporary directories, for debugging.
	Keep bool
	// Box runs the commands in containers. A nil Box runs them on this machine.
	Box *Sandbox
	// TempDir holds the build trees. The default is the temporary directory of the system. A sandbox needs a directory that the container engine can mount.
	TempDir string
	// CaseTimeout is the longest time of one case. The default is the constant CaseTimeout. The mutation run uses a shorter time, because a mutant that loops forever is common.
	CaseTimeout time.Duration
}

Runner runs tests and keeps the results.

func (*Runner) Available added in v0.1.4

func (r *Runner) Available(rc Recipe) error

Available tells if the language can run with this runner. In a sandbox, the image replaces the tools of this machine.

func (*Runner) Gen added in v0.1.3

func (r *Runner) Gen(dir string, n *node.Node, im node.Impl) (Tree, error)

Gen writes the build tree of one implementation to dir: a copy of the files with the tests, and the build files of the language. It never writes to the content tree.

func (*Runner) GenAdapter added in v0.1.3

func (r *Runner) GenAdapter(dir string, n *node.Node, im node.Impl, k *contract.Contract) (Adapter, error)

GenAdapter writes the adapter of the contract k for one implementation to dir. It does not build it. A language without generics gets the int variant only. `algo test` also drops T = string when the entry does not build with it.

func (*Runner) Open added in v0.1.11

func (r *Runner) Open(n *node.Node, im node.Impl, k *contract.Contract) (*Session, error)

Open builds the adapter of the implementation for the contract in a new temporary directory, and starts the custom checker when the contract has one. Close removes the directory.

func (*Runner) Shared added in v0.1.2

func (r *Runner) Shared(n *node.Node, im node.Impl, k *contract.Contract, cases []contract.Case) status.Shared

Shared runs the shared cases of the contract k on one implementation (spec 09.2). It builds the adapter in a new temporary directory, starts it once and sends the cases in order. A crash fails the case, and the next case gets a new adapter.

func (*Runner) Test

func (r *Runner) Test(n *node.Node, im node.Impl) status.Test

Test runs the native tests of one implementation.

func (*Runner) ToolVersion added in v0.1.4

func (r *Runner) ToolVersion(rc Recipe) string

ToolVersion returns the version of the toolchain that runs the tests of the language.

type Sandbox added in v0.1.4

type Sandbox struct {
	// Engine is the container command, such as docker or podman.
	Engine string
	// Images maps each language to its image, pinned by digest.
	Images map[string]string
	// Memory is the memory limit of one container, such as 2g.
	Memory string
	// CPUs is the CPU limit of one container, such as 2.
	CPUs string
	// Pids is the process limit of one container.
	Pids int
	// contains filtered or unexported fields
}

Sandbox runs the builds, the tests and the adapters in containers (spec 08.7). Each command gets a new container with no network, a read-only root, a memory limit, a CPU limit and a process limit. The build tree is the only writable mount.

func NewSandbox added in v0.1.4

func NewSandbox(engine string, images map[string]string) *Sandbox

NewSandbox returns a sandbox with the default limits.

func (*Sandbox) Argv added in v0.1.4

func (s *Sandbox) Argv(lang string, interactive bool, mounts []Mount, cmd []string) []string

Argv returns the command that runs cmd in a new container for the language. The first mount is the work directory. A mount keeps its host path in the container, so that the paths in an error message are the same as in host mode and Tree.Rewrite works on them. HOME points into the work directory, so that the caches of one implementation stay between its commands and go away with the tree.

func (*Sandbox) Ready added in v0.1.4

func (s *Sandbox) Ready(lang string) error

Ready pulls the image of the language once. It returns a Skip when the manifest has no image for the language.

func (*Sandbox) Reap added in v0.1.4

func (s *Sandbox) Reap()

Reap removes the containers that are still there, such as the container of an adapter that was killed after a timeout. Killing the engine client does not stop its container.

func (*Sandbox) Version added in v0.1.4

func (s *Sandbox) Version(lang string, cmd []string) string

Version returns the first line of the version output of the toolchain in the image.

type Session added in v0.1.11

type Session struct {

	// Types are the types of T that the adapter supports. SkipReason says why a type of the contract is not in Types, and SkipExpected tells if the skip is expected (a language without generics).
	Types        []string
	SkipReason   string
	SkipExpected bool
	// contains filtered or unexported fields
}

Session is the built adapter of one implementation for one contract, with the custom checker of the contract. It runs one case at a time, so a caller can run many cases, as the differential run and the shrinker do (spec 09.5, 09.11).

func (*Session) Close added in v0.1.11

func (ss *Session) Close()

Close stops the adapter and the checker, and removes the directory unless the runner keeps it.

func (*Session) Run added in v0.1.11

func (ss *Session) Run(c contract.Case) (json.RawMessage, string, error)

Run runs one case. It gives the result of the call, and the reason why the case fails, which is empty when the case passes. A crash, a timeout or a reply that is not valid fails the case, and the next case gets a new adapter. The error is for a problem of the run itself, such as a checker that does not answer.

func (*Session) Supports added in v0.1.11

func (ss *Session) Supports(c contract.Case) bool

Supports tells if the adapter supports the type of T of the case.

type Skip

type Skip struct{ Lang, Tool string }

Skip is the error of a language whose tools are missing.

func (Skip) Error

func (s Skip) Error() string

type Tree added in v0.1.3

type Tree struct {
	Dir string
	// Test are the commands that build the implementation and run its native tests. They run in Dir.
	Test [][]string
	// Sources maps each copied file, relative to Dir, to its path in the content repository.
	Sources map[string]string
}

Tree is the generated build tree of one implementation (spec 08.5).

func (Tree) Rewrite added in v0.1.3

func (t Tree) Rewrite(s string) string

Rewrite changes the paths of the tree in a compiler error or a test failure to the paths in the content repository, so that the author sees nodes/.../merge_sort.rs:14 (spec 08.5).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL