Documentation
¶
Overview ¶
Package contract reads the contract and the shared cases of a problem node, and decides whether an output is right (spec 09.1 to 09.3).
Index ¶
- Variables
- func Exists(dir string) bool
- func Load(dir string) (*Contract, []Case, error)
- func SafeInts(t Type, raw json.RawMessage) (json.RawMessage, error)
- func Short(s string) string
- type Case
- type Checker
- type Contract
- func (k *Contract) Agree(o *Oracle, c Case, custom *Custom) (reason string, output json.RawMessage, err error)
- func (k *Contract) CheckArgs(c Case) error
- func (k *Contract) CheckCase(c Case) error
- func (k *Contract) LoadGenerated() (*Generated, error)
- func (k *Contract) OracleArgs(args map[string]json.RawMessage, t string) (map[string]json.RawMessage, error)
- func (k *Contract) OracleFile() string
- func (k *Contract) OracleFiles() []string
- func (k *Contract) OutName() string
- func (k *Contract) Output() int
- func (k *Contract) OutputType(t string) Type
- func (k *Contract) SafeCase(c Case) (Case, error)
- func (k *Contract) StartCustom(wrap func(argv []string) []string) (*Custom, error)
- func (k *Contract) StartGenerator(wrap func(argv []string) []string) (*Generator, error)
- func (k *Contract) StartOracle(wrap func(argv []string) []string) (*Oracle, error)
- func (k *Contract) StartOracleFile(file string, wrap func(argv []string) []string) (*Oracle, error)
- func (k *Contract) Types() []string
- func (k *Contract) Validate() error
- func (k *Contract) Verdict(c Case, raw json.RawMessage, custom *Custom) (string, error)
- type Custom
- type GenRequest
- type GenSpec
- type Generated
- type Generator
- type Oracle
- type Param
- type Return
- type SizeClass
- type Type
Constants ¶
This section is empty.
Variables ¶
var CaseFiles = []string{"basic", "edge", "regress"}
CaseFiles are the case files of a problem, in the order that the runner uses (spec 09.2).
var Checkers = []string{"exact", "unordered", "multiset", "float", "any_of", "custom"}
Checkers are the checker kinds of spec 09.3.
var ClassOrder = []string{"tiny", "small", "medium", "large"}
ClassOrder is the order of the size classes, from the smallest.
var ErrNoOracle = errors.New("the problem has no reference.py or brute.py")
ErrNoOracle is the error of StartOracle for a problem with no Python oracle.
Functions ¶
func SafeInts ¶ added in v0.1.11
func SafeInts(t Type, raw json.RawMessage) (json.RawMessage, error)
SafeInts writes each integer of the value outside the safe range of JSON readers as a string, as a case file does (spec 09.2). t is the type of the value.
Types ¶
type Case ¶
type Case struct {
Name string `json:"name"`
Class []string `json:"class,omitempty"`
T string `json:"T,omitempty"`
Args map[string]json.RawMessage `json:"args"`
Expect json.RawMessage `json:"expect,omitempty"`
ExpectAny []json.RawMessage `json:"expect_any,omitempty"`
// Flaky marks a case that gave two results on the same input (rule S09). It counts as a failure, and Issue names the open issue.
Flaky bool `json:"flaky,omitempty"`
Issue string `json:"issue,omitempty"`
// Found names the node and the language that a regression case broke, such as merge-sort/go (spec 09.11).
Found string `json:"found,omitempty"`
// File is the case file without .jsonl, such as basic. Line is the line in it.
File string `json:"-"`
Line int `json:"-"`
}
Case is one line of a case file.
func Shrink ¶ added in v0.1.11
Shrink makes a failing case smaller (spec 09.11). It tries to remove parts of each list, then items one at a time, then to make each integer nearer to 0 and each string shorter, and keeps each change for which fails still gives true. It stops after budget calls of fails. It gives the smallest case that it found and the number of calls.
fails must give false for an input that does not meet the contract, so the shrinker never keeps an input that the problem does not allow.
type Checker ¶
type Checker struct {
Kind string `yaml:"kind" json:"kind"`
Abs float64 `yaml:"abs,omitempty" json:"abs,omitempty"`
Rel float64 `yaml:"rel,omitempty" json:"rel,omitempty"`
}
Checker is the checker kind of a contract. In YAML it is a word, such as exact, or a mapping with the tolerances of the float kind.
type Contract ¶
type Contract struct {
Problem string `yaml:"problem" json:"problem"`
Entry string `yaml:"entry" json:"entry"`
Params []Param `yaml:"params" json:"params"`
Returns Return `yaml:"returns" json:"returns"`
Generic map[string]string `yaml:"generic,omitempty" json:"generic,omitempty"`
Bounds map[string]any `yaml:"bounds,omitempty" json:"bounds,omitempty"`
Checker Checker `yaml:"checker" json:"checker"`
Oracle map[string]any `yaml:"oracle,omitempty" json:"oracle,omitempty"`
EdgeClasses []string `yaml:"edge_classes,omitempty" json:"edge_classes,omitempty"`
Metamorphic []string `yaml:"metamorphic,omitempty" json:"metamorphic,omitempty"`
Reviewed map[string]string `yaml:"reviewed,omitempty" json:"reviewed,omitempty"`
// Dir is the directory of the problem node.
Dir string `yaml:"-" json:"-"`
}
Contract is contract.yaml of a problem node.
func (*Contract) Agree ¶ added in v0.1.10
func (k *Contract) Agree(o *Oracle, c Case, custom *Custom) (reason string, output json.RawMessage, err error)
Agree checks the oracle on one case (rule S02). It runs the oracle on the arguments, and gives the verdict of the checker on the oracle output: against expect when the case has it, and for a custom checker with the oracle output as expected. The reason is empty when the oracle agrees.
func (*Contract) CheckArgs ¶ added in v0.1.11
CheckArgs checks the type of T of a case and that it gives each parameter a value of its type. It does not look at expect, so it also checks a generated input.
func (*Contract) CheckCase ¶
CheckCase checks that a case gives each parameter a value of its type, and that its expect has the output type.
func (*Contract) LoadGenerated ¶ added in v0.1.11
LoadGenerated reads cases/generated.yaml of the problem. A problem without the file gives nil.
func (*Contract) OracleArgs ¶ added in v0.1.10
func (k *Contract) OracleArgs(args map[string]json.RawMessage, t string) (map[string]json.RawMessage, error)
OracleArgs gives the arguments of a case in the form that the oracle reads. Python reads integers of any size, so an integer that the case writes as a string (spec 09.2) becomes a JSON number. t is the type of T.
func (*Contract) OracleFile ¶ added in v0.1.10
OracleFile gives the Python oracle of the problem: the file that the field oracle.reference or oracle.brute names, or else reference.py or brute.py when it exists. It gives "" when the problem has no Python oracle.
func (*Contract) OracleFiles ¶ added in v0.1.11
OracleFiles gives each Python oracle of the problem that exists, in order of preference: the reference, then the brute force.
func (*Contract) OutName ¶ added in v0.1.10
OutName is the name of the inout parameter whose value is the output, or "" when the return value is the output.
func (*Contract) Output ¶
Output is the index of the inout parameter whose value is the output, or -1 when the return value is the output.
func (*Contract) OutputType ¶
OutputType is the type of the output for a case whose T is t.
func (*Contract) SafeCase ¶ added in v0.1.11
SafeCase writes the integers of the arguments and of expect of the case as a case file does.
func (*Contract) StartCustom ¶
StartCustom starts the checker of the contract. A non-nil wrap changes the command, so that the checker can run in a sandbox.
func (*Contract) StartGenerator ¶ added in v0.1.11
StartGenerator starts the generator process of the problem. A non-nil wrap changes the command, so that the generators can run in a sandbox.
func (*Contract) StartOracle ¶ added in v0.1.10
StartOracle starts the Python oracle of the problem. A non-nil wrap changes the command, so that the oracle can run in a sandbox.
func (*Contract) StartOracleFile ¶ added in v0.1.11
StartOracleFile starts one Python oracle of the problem, from its file relative to the problem directory.
func (*Contract) Types ¶
Types are the types that a generic contract is tested with (spec 08.3.4). A contract that is not generic has only int.
type Custom ¶
type Custom struct {
// contains filtered or unexported fields
}
Custom runs checker.py of a problem in one Python process (spec 09.3).
type GenRequest ¶ added in v0.1.11
type GenRequest struct {
Gen string `json:"gen"`
File string `json:"file"`
Params map[string]any `json:"params,omitempty"`
Class string `json:"class"`
N int `json:"n"`
Seed int `json:"seed"`
Index int `json:"index"`
T string `json:"T"`
// All asks for every input of the size, from the function exhaustive of the generator, for an exhaustive class.
All bool `json:"all,omitempty"`
}
GenRequest is one call of a generator: the generator with one combination of its parameters, a size, a seed, an index and the type of T.
func (GenRequest) CaseName ¶ added in v0.1.11
func (q GenRequest) CaseName(i int) string
CaseName is the name of the i-th case of the request, such as small-uniform-k=3-n42-s1-7.
type GenSpec ¶ added in v0.1.11
type GenSpec struct {
Name string `yaml:"name"`
File string `yaml:"file"`
Params map[string][]any `yaml:"params,omitempty"`
Adversarial bool `yaml:"adversarial,omitempty"`
Note string `yaml:"note,omitempty"`
}
GenSpec is one generator of generated.yaml. Params maps each parameter of the generator to the list of its values; the generator runs with each combination.
type Generated ¶ added in v0.1.11
type Generated struct {
// Size is the size parameter that the classes measure, such as len(xs).
Size string `yaml:"size"`
Classes map[string]SizeClass `yaml:"classes"`
Generators []GenSpec `yaml:"generators"`
// Pre is the file with the function pre(args), which gives None when the arguments meet the preconditions of the contract and a reason when they do not. The shrinker uses it, so that it keeps only inputs that the problem allows. It is optional.
Pre string `yaml:"pre,omitempty"`
Seeds struct {
PR []int `yaml:"pr"`
Nightly string `yaml:"nightly,omitempty"`
} `yaml:"seeds"`
}
Generated is cases/generated.yaml of a problem (spec 09.4). It registers the generators and the size classes of the random inputs.
func (*Generated) Plan ¶ added in v0.1.11
func (g *Generated) Plan(k *Contract, classes []string, seeds []int) []GenRequest
Plan gives the generator calls of a run for the classes, with the seeds. Each class splits its count over the generators and their parameter combinations in turn, and draws each size from its range with a random source made from the seed, so the plan is the same on every machine. An exhaustive class asks each generator for every input of each size of the range. The types of T alternate over the types of the contract. An adversarial generator runs only at the small size, as spec 09.5 says for the PR run.
type Generator ¶ added in v0.1.11
type Generator struct {
// contains filtered or unexported fields
}
Generator runs the generators of a problem in one Python process.
func (*Generator) Close ¶ added in v0.1.11
func (g *Generator) Close()
Close stops the generator process.
type Oracle ¶ added in v0.1.10
type Oracle struct {
// File is the oracle file, relative to the problem directory.
File string
// contains filtered or unexported fields
}
Oracle runs the Python oracle of a problem in one process (spec 09.5).
func (*Oracle) Run ¶ added in v0.1.10
func (o *Oracle) Run(args map[string]json.RawMessage, out string) (json.RawMessage, error)
Run gives the output of the oracle for the arguments, as JSON. out is the name of the inout parameter whose value is the output, or "" when the return value is the output.
type Param ¶
type Param struct {
Name string `yaml:"name" json:"name"`
Type string `yaml:"type" json:"type"`
// Mode is "in" or "inout". An inout parameter is changed by the call, and its value after the call is the output when the entry returns unit.
Mode string `yaml:"mode,omitempty" json:"mode,omitempty"`
Pre string `yaml:"pre,omitempty" json:"pre,omitempty"`
}
Param is one parameter of the entry.
type Return ¶
type Return struct {
Type string `yaml:"type" json:"type"`
Post string `yaml:"post,omitempty" json:"post,omitempty"`
}
Return is the return value of the entry.
type SizeClass ¶ added in v0.1.11
type SizeClass struct {
Range [2]int `yaml:"range"`
Count int `yaml:"count"`
Exhaustive bool `yaml:"exhaustive,omitempty"`
}
SizeClass is one size class: tiny, small, medium or large.
type Type ¶
Type is a type of the neutral type language (spec 08.3.3), such as list<int> or option<index>.
func Parse ¶
Parse reads a type. A single capital letter, such as T, is a type variable of a generic contract.
func (Type) Check ¶
Check tells if the decoded JSON value v has the type t. It also checks the kind of each value, so a float where the type is int fails.