contract

package
v0.0.14 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 12, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Overview

Package contract reads the contract and the shared cases of a problem node, and decides whether an output is right (spec 09.1 to 09.3).

Index

Constants

This section is empty.

Variables

View Source
var CaseFiles = []string{"basic", "edge", "regress"}

CaseFiles are the case files of a problem, in the order that the runner uses (spec 09.2).

View Source
var Checkers = []string{"exact", "unordered", "multiset", "float", "any_of", "custom"}

Checkers are the checker kinds of spec 09.3.

View Source
var ClassOrder = []string{"tiny", "small", "medium", "large"}

ClassOrder is the order of the size classes, from the smallest.

View Source
var ErrNoOracle = errors.New("the problem has no reference.py or brute.py")

ErrNoOracle is the error of StartOracle for a problem with no Python oracle.

Functions

func Exists

func Exists(dir string) bool

Exists tells if dir has contract.yaml.

func Load

func Load(dir string) (*Contract, []Case, error)

Load reads contract.yaml and the case files of the problem node in dir.

func SafeInts

func SafeInts(t Type, raw json.RawMessage) (json.RawMessage, error)

SafeInts writes each integer of the value outside the safe range of JSON readers as a string, as a case file does (spec 09.2). t is the type of the value.

func Short

func Short(s string) string

Short cuts a long JSON text for a report.

Types

type Case

type Case struct {
	Name      string                     `json:"name"`
	Class     []string                   `json:"class,omitempty"`
	T         string                     `json:"T,omitempty"`
	Args      map[string]json.RawMessage `json:"args"`
	Expect    json.RawMessage            `json:"expect,omitempty"`
	ExpectAny []json.RawMessage          `json:"expect_any,omitempty"`
	// Flaky marks a case that gave two results on the same input (rule S09). It counts as a failure, and Issue names the open issue.
	Flaky bool   `json:"flaky,omitempty"`
	Issue string `json:"issue,omitempty"`
	// Found names the node and the language that a regression case broke, such as merge-sort/go (spec 09.11).
	Found string `json:"found,omitempty"`

	// File is the case file without .jsonl, such as basic. Line is the line in it.
	File string `json:"-"`
	Line int    `json:"-"`
}

Case is one line of a case file.

func Shrink

func Shrink(c Case, fails func(Case) (bool, error), budget int) (Case, int, error)

Shrink makes a failing case smaller (spec 09.11). It tries to remove parts of each list, then items one at a time, then to make each integer nearer to 0 and each string shorter, and keeps each change for which fails still gives true. It stops after budget calls of fails. It gives the smallest case that it found and the number of calls.

fails must give false for an input that does not meet the contract, so the shrinker never keeps an input that the problem does not allow.

func (Case) ID

func (c Case) ID() string

ID is the id of the case in the protocol and in reports: the file and the name, such as basic/duplicates.

func (Case) Type

func (c Case) Type() string

Type is the type of T for the case: int unless the case says otherwise.

type Checker

type Checker struct {
	Kind string  `yaml:"kind" json:"kind"`
	Abs  float64 `yaml:"abs,omitempty" json:"abs,omitempty"`
	Rel  float64 `yaml:"rel,omitempty" json:"rel,omitempty"`
}

Checker is the checker kind of a contract. In YAML it is a word, such as exact, or a mapping with the tolerances of the float kind.

func (*Checker) UnmarshalYAML

func (c *Checker) UnmarshalYAML(n *yaml.Node) error

UnmarshalYAML accepts a word or a mapping.

type Contract

type Contract struct {
	Problem     string            `yaml:"problem" json:"problem"`
	Entry       string            `yaml:"entry" json:"entry"`
	Params      []Param           `yaml:"params" json:"params"`
	Returns     Return            `yaml:"returns" json:"returns"`
	Generic     map[string]string `yaml:"generic,omitempty" json:"generic,omitempty"`
	Bounds      map[string]any    `yaml:"bounds,omitempty" json:"bounds,omitempty"`
	Checker     Checker           `yaml:"checker" json:"checker"`
	Oracle      map[string]any    `yaml:"oracle,omitempty" json:"oracle,omitempty"`
	EdgeClasses []string          `yaml:"edge_classes,omitempty" json:"edge_classes,omitempty"`
	Metamorphic []string          `yaml:"metamorphic,omitempty" json:"metamorphic,omitempty"`
	Reviewed    map[string]string `yaml:"reviewed,omitempty" json:"reviewed,omitempty"`

	// Dir is the directory of the problem node.
	Dir string `yaml:"-" json:"-"`
}

Contract is contract.yaml of a problem node.

func (*Contract) Agree

func (k *Contract) Agree(o *Oracle, c Case, custom *Custom) (reason string, output json.RawMessage, err error)

Agree checks the oracle on one case (rule S02). It runs the oracle on the arguments, and gives the verdict of the checker on the oracle output: against expect when the case has it, and for a custom checker with the oracle output as expected. The reason is empty when the oracle agrees.

func (*Contract) CheckArgs

func (k *Contract) CheckArgs(c Case) error

CheckArgs checks the type of T of a case and that it gives each parameter a value of its type. It does not look at expect, so it also checks a generated input.

func (*Contract) CheckCase

func (k *Contract) CheckCase(c Case) error

CheckCase checks that a case gives each parameter a value of its type, and that its expect has the output type.

func (*Contract) LoadGenerated

func (k *Contract) LoadGenerated() (*Generated, error)

LoadGenerated reads cases/generated.yaml of the problem. A problem without the file gives nil.

func (*Contract) OracleArgs

func (k *Contract) OracleArgs(args map[string]json.RawMessage, t string) (map[string]json.RawMessage, error)

OracleArgs gives the arguments of a case in the form that the oracle reads. Python reads integers of any size, so an integer that the case writes as a string (spec 09.2) becomes a JSON number. t is the type of T.

func (*Contract) OracleFile

func (k *Contract) OracleFile() string

OracleFile gives the Python oracle of the problem: the file that the field oracle.reference or oracle.brute names, or else reference.py or brute.py when it exists. It gives "" when the problem has no Python oracle.

func (*Contract) OracleFiles

func (k *Contract) OracleFiles() []string

OracleFiles gives each Python oracle of the problem that exists, in order of preference: the reference, then the brute force.

func (*Contract) OutName

func (k *Contract) OutName() string

OutName is the name of the inout parameter whose value is the output, or "" when the return value is the output.

func (*Contract) Output

func (k *Contract) Output() int

Output is the index of the inout parameter whose value is the output, or -1 when the return value is the output.

func (*Contract) OutputType

func (k *Contract) OutputType(t string) Type

OutputType is the type of the output for a case whose T is t.

func (*Contract) SafeCase

func (k *Contract) SafeCase(c Case) (Case, error)

SafeCase writes the integers of the arguments and of expect of the case as a case file does.

func (*Contract) StartCustom

func (k *Contract) StartCustom(wrap func(argv []string) []string) (*Custom, error)

StartCustom starts the checker of the contract. A non-nil wrap changes the command, so that the checker can run in a sandbox.

func (*Contract) StartGenerator

func (k *Contract) StartGenerator(wrap func(argv []string) []string) (*Generator, error)

StartGenerator starts the generator process of the problem. A non-nil wrap changes the command, so that the generators can run in a sandbox.

func (*Contract) StartOracle

func (k *Contract) StartOracle(wrap func(argv []string) []string) (*Oracle, error)

StartOracle starts the Python oracle of the problem. A non-nil wrap changes the command, so that the oracle can run in a sandbox.

func (*Contract) StartOracleFile

func (k *Contract) StartOracleFile(file string, wrap func(argv []string) []string) (*Oracle, error)

StartOracleFile starts one Python oracle of the problem, from its file relative to the problem directory.

func (*Contract) Types

func (k *Contract) Types() []string

Types are the types that a generic contract is tested with (spec 08.3.4). A contract that is not generic has only int.

func (*Contract) Validate

func (k *Contract) Validate() error

Validate checks the parts of the contract that the runner needs.

func (*Contract) Verdict

func (k *Contract) Verdict(c Case, raw json.RawMessage, custom *Custom) (string, error)

Verdict tells if the output of an adapter is right for a case. It returns an empty reason when the output is right. The checker of the contract decides (spec 09.3). Before it, the output must have the output type, so an int result that is a float fails.

type Custom

type Custom struct {
	// contains filtered or unexported fields
}

Custom runs checker.py of a problem in one Python process (spec 09.3).

func (*Custom) Check

func (c *Custom) Check(cs Case, output, expected any) (string, error)

Check returns the reason of the checker, or an empty reason when the checker accepts the output.

func (*Custom) Close

func (c *Custom) Close()

Close stops the checker.

type GenRequest

type GenRequest struct {
	Gen    string         `json:"gen"`
	File   string         `json:"file"`
	Params map[string]any `json:"params,omitempty"`
	Class  string         `json:"class"`
	N      int            `json:"n"`
	Seed   int            `json:"seed"`
	Index  int            `json:"index"`
	T      string         `json:"T"`
	// All asks for every input of the size, from the function exhaustive of the generator, for an exhaustive class.
	All bool `json:"all,omitempty"`
}

GenRequest is one call of a generator: the generator with one combination of its parameters, a size, a seed, an index and the type of T.

func (GenRequest) CaseName

func (q GenRequest) CaseName(i int) string

CaseName is the name of the i-th case of the request, such as small-uniform-k=3-n42-s1-7.

type GenSpec

type GenSpec struct {
	Name        string           `yaml:"name"`
	File        string           `yaml:"file"`
	Params      map[string][]any `yaml:"params,omitempty"`
	Adversarial bool             `yaml:"adversarial,omitempty"`
	Note        string           `yaml:"note,omitempty"`
}

GenSpec is one generator of generated.yaml. Params maps each parameter of the generator to the list of its values; the generator runs with each combination.

type Generated

type Generated struct {
	// Size is the size parameter that the classes measure, such as len(xs).
	Size       string               `yaml:"size"`
	Classes    map[string]SizeClass `yaml:"classes"`
	Generators []GenSpec            `yaml:"generators"`
	// Pre is the file with the function pre(args), which gives None when the arguments meet the preconditions of the contract and a reason when they do not. The shrinker uses it, so that it keeps only inputs that the problem allows. It is optional.
	Pre   string `yaml:"pre,omitempty"`
	Seeds struct {
		PR      []int  `yaml:"pr"`
		Nightly string `yaml:"nightly,omitempty"`
	} `yaml:"seeds"`
}

Generated is cases/generated.yaml of a problem (spec 09.4). It registers the generators and the size classes of the random inputs.

func (*Generated) Plan

func (g *Generated) Plan(k *Contract, classes []string, seeds []int) []GenRequest

Plan gives the generator calls of a run for the classes, with the seeds. Each class splits its count over the generators and their parameter combinations in turn, and draws each size from its range with a random source made from the seed, so the plan is the same on every machine. An exhaustive class asks each generator for every input of each size of the range. The types of T alternate over the types of the contract. An adversarial generator runs only at the small size, as spec 09.5 says for the PR run.

type Generator

type Generator struct {
	// contains filtered or unexported fields
}

Generator runs the generators of a problem in one Python process.

func (*Generator) Close

func (g *Generator) Close()

Close stops the generator process.

func (*Generator) Make

func (g *Generator) Make(q GenRequest) ([]Case, error)

Make runs one request and gives the cases that it makes, without expect. Each case is checked against the contract, so a bad input is a generator bug (rule S05).

func (*Generator) Pre

func (g *Generator) Pre(file string, args map[string]json.RawMessage) (string, error)

Pre checks the arguments with the function pre of the file, and gives the reason why they do not meet the preconditions, or "" when they do.

type Oracle

type Oracle struct {
	// File is the oracle file, relative to the problem directory.
	File string
	// contains filtered or unexported fields
}

Oracle runs the Python oracle of a problem in one process (spec 09.5).

func (*Oracle) Close

func (o *Oracle) Close()

Close stops the oracle.

func (*Oracle) Run

func (o *Oracle) Run(args map[string]json.RawMessage, out string) (json.RawMessage, error)

Run gives the output of the oracle for the arguments, as JSON. out is the name of the inout parameter whose value is the output, or "" when the return value is the output.

type Param

type Param struct {
	Name string `yaml:"name" json:"name"`
	Type string `yaml:"type" json:"type"`
	// Mode is "in" or "inout". An inout parameter is changed by the call, and its value after the call is the output when the entry returns unit.
	Mode string `yaml:"mode,omitempty" json:"mode,omitempty"`
	Pre  string `yaml:"pre,omitempty" json:"pre,omitempty"`
}

Param is one parameter of the entry.

type Return

type Return struct {
	Type string `yaml:"type" json:"type"`
	Post string `yaml:"post,omitempty" json:"post,omitempty"`
}

Return is the return value of the entry.

type SizeClass

type SizeClass struct {
	Range      [2]int `yaml:"range"`
	Count      int    `yaml:"count"`
	Exhaustive bool   `yaml:"exhaustive,omitempty"`
}

SizeClass is one size class: tiny, small, medium or large.

type Type

type Type struct {
	Name string
	Args []Type
}

Type is a type of the neutral type language (spec 08.3.3), such as list<int> or option<index>.

func Parse

func Parse(s string) (Type, error)

Parse reads a type. A single capital letter, such as T, is a type variable of a generic contract.

func (Type) Check

func (t Type) Check(v any) error

Check tells if the decoded JSON value v has the type t. It also checks the kind of each value, so a float where the type is int fails.

func (Type) HasVar

func (t Type) HasVar() bool

HasVar tells if the type has a type variable in it.

func (Type) IsVar

func (t Type) IsVar() bool

IsVar tells if the type is a type variable, such as T.

func (Type) String

func (t Type) String() string

func (Type) Subst

func (t Type) Subst(v, by string) Type

Subst replaces the type variable v with the type named by.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL