Documentation
¶
Overview ¶
Package evaluation compares explicit inference configurations against bounded, deterministic task checks. Checks establish only the supplied expectations; they do not establish general model quality or authorize automatic routing.
Index ¶
Constants ¶
const Schema = 1
Variables ¶
This section is empty.
Functions ¶
func ValidateChecks ¶ added in v0.6.0
ValidateChecks validates one bounded set of the deterministic predicates supported by CheckOutput, without constructing an inference configuration.
Types ¶
type Candidate ¶
type Candidate struct {
ConfigurationID string `json:"configuration_id"`
CasesPassed int `json:"cases_passed"`
CasesTotal int `json:"cases_total"`
Eligible bool `json:"eligible"`
GeneratedTokens int64 `json:"generated_tokens"`
EstimatedUSD *float64 `json:"estimated_usd,omitempty"`
FirstRunSamples int `json:"first_run_samples"`
ReuseSamples int `json:"reuse_samples"`
MeanFirstRunMS float64 `json:"mean_first_run_ms"`
MeanReuseMS float64 `json:"mean_reuse_ms"`
}
type Case ¶
type Case struct {
ID string `json:"id"`
Complexity string `json:"complexity"`
Instructions string `json:"instructions"`
Input string `json:"input"`
Context []inference.ContextItem `json:"context,omitempty"`
Checks []Check `json:"checks"`
Fixtures map[string]Fixture `json:"fixture_by_configuration,omitempty"`
}
type CaseResult ¶
type CaseResult struct {
CaseID string `json:"case_id"`
Complexity string `json:"complexity"`
ConfigurationID string `json:"configuration_id"`
Fingerprint string `json:"fingerprint,omitempty"`
First Measurement `json:"first"`
Reuse *Measurement `json:"reuse,omitempty"`
Eligible bool `json:"eligible"`
}
type Check ¶
type Check struct {
ID string `json:"id"`
Kind string `json:"kind"`
Pointer string `json:"pointer,omitempty"`
Expected json.RawMessage `json:"expected,omitempty"`
Text string `json:"text,omitempty"`
IgnoreCase bool `json:"ignore_case,omitempty"`
}
Check supports json_valid, json_equals (at an RFC 6901 pointer), contains and not_contains. Substring checks are literal; they do not understand negation, paraphrase, factual support, or the correctness of arbitrary generated code.
type CheckResult ¶
type CheckResult struct {
ID string `json:"id"`
Kind string `json:"kind"`
Passed bool `json:"passed"`
}
func CheckOutput ¶
func CheckOutput(text string, checks []Check) ([]CheckResult, bool)
CheckOutput checks a response without retaining the response in the report. JSON checks reject duplicate keys, trailing data, and excessive nesting.
type Configuration ¶
type Configuration struct {
ID string `json:"id"`
Spec inference.Spec `json:"spec"`
MaxFirstRunMS int64 `json:"max_first_run_ms,omitempty"`
MaxReuseMS int64 `json:"max_reuse_ms,omitempty"`
}
Configuration.Spec supplies model, provider and explicit resource limits. Task identity and prompt material come exclusively from the suite and case.
type Fixture ¶
type Fixture struct {
Response inference.Response `json:"response"`
FirstRunMS int64 `json:"first_run_ms"`
ReuseMS int64 `json:"reuse_ms"`
}
Fixture is supplied public test data. Its usage and durations are simulated, never a benchmark of the model named by a configuration.
type Measurement ¶
type Measurement struct {
ElapsedMS float64 `json:"elapsed_ms"`
TimingSource string `json:"timing_source"`
FreshGeneration bool `json:"fresh_generation"`
Reused bool `json:"reused"`
Complete bool `json:"complete"`
Durable bool `json:"durable"`
QualityPassed bool `json:"quality_passed"`
Checks []CheckResult `json:"checks"`
ThisRunUsage inference.Usage `json:"this_run_usage"`
AvoidedUsage inference.Usage `json:"avoided_usage"`
CostUSD *float64 `json:"cost_usd,omitempty"`
AvoidedCostUSD *float64 `json:"avoided_cost_usd,omitempty"`
OutputSHA256 string `json:"output_sha256,omitempty"`
ErrorCategory string `json:"error_category,omitempty"`
Inference *inference.Measurement `json:"inference,omitempty"`
}
type Report ¶
type Report struct {
Schema int `json:"schema"`
SuiteID string `json:"suite_id"`
Revision string `json:"revision"`
SuiteSHA256 string `json:"suite_sha256"`
Mode string `json:"mode"`
Objective string `json:"objective"`
Results []CaseResult `json:"results"`
Candidates []Candidate `json:"candidates"`
Selection *Selection `json:"selection,omitempty"`
SelectionNote string `json:"selection_note"`
Limitations []string `json:"limitations"`
}
func Evaluate ¶
Evaluate runs each explicit configuration once per case, then measures one reuse of that exact successful snapshot. It never retries, changes models, or invokes a judge. Callers choose runners and isolated broker scopes; an already-reused first result cannot qualify as a fresh-generation baseline.
func RunFixtures ¶
RunFixtures uses supplied answers through the real inference Runner and a fresh in-memory CAS broker. No provider/network call or real-time delay is performed. Every duration/usage value is explicitly marked simulated.
type RunnerFactory ¶
type RunnerFactory func(Case, Configuration) (Runner, error)
type Suite ¶
type Suite struct {
Schema int `json:"schema"`
ID string `json:"id"`
Revision string `json:"revision"`
Objective string `json:"objective,omitempty"` // latency, generated_tokens, or priced_cost
Cases []Case `json:"cases"`
Configurations []Configuration `json:"configurations"`
}
func DefaultSuite ¶
func DefaultSuite() Suite
DefaultSuite returns an independently decoded public six-case fixture suite. Embedded data is validated by tests and never used as generated evidence.
Source Files