Help us shape the future of Go! Take the 2026 Go Developer Survey

evaluation

package
v0.16.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 27, 2026 License: MIT Imports: 21 Imported by: 0

Documentation

Overview

Package evaluation compares explicit inference configurations against bounded, deterministic task checks. Checks establish only the supplied expectations; they do not establish general model quality or authorize automatic routing.

Index

Constants

View Source 
const MaxCountTextBytes = 4 << 20

MaxCountTextBytes bounds text accepted by the deterministic count checks and the largest count bound in a check. Counting uses constant additional space.

View Source 
const Schema = 1

Variables

This section is empty.

Functions

func SupportedCheckKinds added in v0.13.0 

func SupportedCheckKinds() []string

SupportedCheckKinds returns the deterministic kinds accepted by ValidateChecks in stable order. The returned slice is independent of subsequent calls.

func Validate 

func Validate(s Suite) error

Validate checks every case/configuration plan before any runner is created.

func ValidateChecks added in v0.6.0 

func ValidateChecks(checks []Check) error

ValidateChecks validates one bounded set of the deterministic predicates supported by CheckOutput, without constructing an inference configuration.

Types

type Candidate 

type Candidate struct {
	ConfigurationID string   `json:"configuration_id"`
	CasesPassed     int      `json:"cases_passed"`
	CasesTotal      int      `json:"cases_total"`
	Eligible        bool     `json:"eligible"`
	GeneratedTokens int64    `json:"generated_tokens"`
	EstimatedUSD    *float64 `json:"estimated_usd,omitempty"`
	FirstRunSamples int      `json:"first_run_samples"`
	ReuseSamples    int      `json:"reuse_samples"`
	MeanFirstRunMS  float64  `json:"mean_first_run_ms"`
	MeanReuseMS     float64  `json:"mean_reuse_ms"`
}

type Case 

type Case struct {
	ID           string                  `json:"id"`
	Complexity   string                  `json:"complexity"`
	Instructions string                  `json:"instructions"`
	Input        string                  `json:"input"`
	Context      []inference.ContextItem `json:"context,omitempty"`
	Checks       []Check                 `json:"checks"`
	Fixtures     map[string]Fixture      `json:"fixture_by_configuration,omitempty"`
}

type CaseResult 

type CaseResult struct {
	CaseID          string       `json:"case_id"`
	Complexity      string       `json:"complexity"`
	ConfigurationID string       `json:"configuration_id"`
	Fingerprint     string       `json:"fingerprint,omitempty"`
	First           Measurement  `json:"first"`
	Reuse           *Measurement `json:"reuse,omitempty"`
	Eligible        bool         `json:"eligible"`
}

type Check 

type Check struct {
	ID         string          `json:"id"`
	Kind       string          `json:"kind"`
	Pointer    string          `json:"pointer,omitempty"`
	Expected   json.RawMessage `json:"expected,omitempty"`
	Text       string          `json:"text,omitempty"`
	IgnoreCase bool            `json:"ignore_case,omitempty"`
	Min        *int            `json:"min,omitempty"`
	Max        *int            `json:"max,omitempty"`
}

Check supports JSON predicates, literal substrings, and bounded text counts. Substring and count checks do not establish factual support, tone, suitability, or the correctness of arbitrary generated code. Count bounds are inclusive; pointers distinguish an explicit zero from an omitted bound.

type CheckResult 

type CheckResult struct {
	ID       string `json:"id"`
	Kind     string `json:"kind"`
	Passed   bool   `json:"passed"`
	Observed *int   `json:"observed,omitempty"`
	Status   string `json:"status,omitempty"`
}

func CheckOutput 

func CheckOutput(text string, checks []Check) ([]CheckResult, bool)

CheckOutput checks a response without retaining the response in the report. JSON checks reject duplicate keys, trailing data, and excessive nesting. Count checks reject invalid UTF-8 or text exceeding MaxCountTextBytes. Words are maximal spans separated by Unicode whitespace; characters are Unicode code points, including whitespace. Paragraphs are groups of nonblank LF/CRLF lines separated by one or more whitespace-only lines. Leading/trailing blank lines do not add paragraphs. These counts are not linguistic word segmentation or displayed grapheme counts.

func NotEvaluated added in v0.13.0 

func NotEvaluated(checks []Check) []CheckResult

NotEvaluated records the declared checks when no complete answer is available. It does not assert an observed zero or a failed test of an absent answer.

type Configuration 

type Configuration struct {
	ID            string         `json:"id"`
	Spec          inference.Spec `json:"spec"`
	MaxFirstRunMS int64          `json:"max_first_run_ms,omitempty"`
	MaxReuseMS    int64          `json:"max_reuse_ms,omitempty"`
}

Configuration.Spec supplies model, provider and explicit resource limits. Task identity and prompt material come exclusively from the suite and case.

type Fixture 

type Fixture struct {
	Response   inference.Response `json:"response"`
	FirstRunMS int64              `json:"first_run_ms"`
	ReuseMS    int64              `json:"reuse_ms"`
}

Fixture is supplied public test data. Its usage and durations are simulated, never a benchmark of the model named by a configuration.

type Measurement 

type Measurement struct {
	ElapsedMS       float64                `json:"elapsed_ms"`
	TimingSource    string                 `json:"timing_source"`
	FreshGeneration bool                   `json:"fresh_generation"`
	Reused          bool                   `json:"reused"`
	Complete        bool                   `json:"complete"`
	Durable         bool                   `json:"durable"`
	QualityPassed   bool                   `json:"quality_passed"`
	Checks          []CheckResult          `json:"checks"`
	ThisRunUsage    inference.Usage        `json:"this_run_usage"`
	AvoidedUsage    inference.Usage        `json:"avoided_usage"`
	CostUSD         *float64               `json:"cost_usd,omitempty"`
	AvoidedCostUSD  *float64               `json:"avoided_cost_usd,omitempty"`
	OutputSHA256    string                 `json:"output_sha256,omitempty"`
	ErrorCategory   string                 `json:"error_category,omitempty"`
	Inference       *inference.Measurement `json:"inference,omitempty"`
}

type Report 

type Report struct {
	Schema        int          `json:"schema"`
	SuiteID       string       `json:"suite_id"`
	Revision      string       `json:"revision"`
	SuiteSHA256   string       `json:"suite_sha256"`
	Mode          string       `json:"mode"`
	Objective     string       `json:"objective"`
	Results       []CaseResult `json:"results"`
	Candidates    []Candidate  `json:"candidates"`
	Selection     *Selection   `json:"selection,omitempty"`
	SelectionNote string       `json:"selection_note"`
	Limitations   []string     `json:"limitations"`
}

func Evaluate 

func Evaluate(ctx context.Context, suite Suite, factory RunnerFactory) (Report, error)

Evaluate runs each explicit configuration once per case, then measures one reuse of that exact successful snapshot. It never retries, changes models, or invokes a judge. Callers choose runners and isolated broker scopes; an already-reused first result cannot qualify as a fresh-generation baseline.

func RunFixtures 

func RunFixtures(ctx context.Context, suite Suite) (Report, error)

RunFixtures uses supplied answers through the real inference Runner and a fresh in-memory CAS broker. No provider/network call or real-time delay is performed. Every duration/usage value is explicitly marked simulated.

type Runner 

type Runner interface {
	Run(context.Context, inference.Spec) (inference.Result, error)
}

type RunnerFactory 

type RunnerFactory func(Case, Configuration) (Runner, error)

type Selection 

type Selection struct {
	ConfigurationID string `json:"configuration_id"`
	Metric          string `json:"metric"`
	Scope           string `json:"scope"`
}

type Suite 

type Suite struct {
	Schema         int             `json:"schema"`
	ID             string          `json:"id"`
	Revision       string          `json:"revision"`
	Objective      string          `json:"objective,omitempty"` // latency, generated_tokens, or priced_cost
	Cases          []Case          `json:"cases"`
	Configurations []Configuration `json:"configurations"`
}

func DefaultSuite 

func DefaultSuite() Suite

DefaultSuite returns an independently decoded public six-case fixture suite. Embedded data is validated by tests and never used as generated evidence.

func Load 

func Load(reader io.Reader) (Suite, error)

Load parses at most 4MiB of strict suite JSON. Unknown fields and duplicate object keys are rejected so misspelled limits cannot silently disappear.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.