Documentation
¶
Overview ¶
Package eval is a statistical evaluation harness for agents. It answers "does the model decide well?" the only way a stochastic model can be answered: run labeled cases multiple times and report a pass-rate distribution with confidence intervals, not a single verdict.
Read the boundary carefully, it is the point of this package. eval measures the MODEL's judgment, statistically and best-effort; it is NOT a proof and a pass rate is not a guarantee. The provable parts of the system are separate: the governed policy's convergence and invariant enforcement (the govern package and the gsm proof) bound what the model can do for all inputs, and the audit package proves what it did. eval quantifies the model; governance contains it; audit records it.
Rigor: because a raw pass rate over few runs is misleading (4/5 is not "80%", it is 80% with a wide interval), every rate carries a 95% Wilson confidence interval, and the harness evaluates the agent's TRAJECTORY (which tools it called, how many steps it took), not only the final message, using the durable journal. Evaluate with Runs well above 1, pin the model version, and set temperature 0 for the most reproducible baseline (still not perfectly deterministic).
Index ¶
- Constants
- Variables
- func HashCases(cases []Case) string
- func RequiredRuns(baselineRate, minDetectableDrop, alpha, power float64) (int, error)
- type Case
- type CaseReport
- type CompareOption
- type Comparison
- type Metric
- func CalledTool(name string) Metric
- func Contains(substr string) Metric
- func Custom(name string, fn func(ctx context.Context, c Case, out RunOutput) (bool, error)) Metric
- func GovernanceHeld(name string, compliant func(ctx context.Context, out RunOutput) (bool, error)) Metric
- func Judge(name string, m agent.Model, rubric string) Metric
- func Matches(re *regexp.Regexp) Metric
- func MaxSteps(n int) Metric
- func NoError() Metric
- func ToolOrder(names ...string) Metric
- type MetricComparison
- type MetricDirection
- type MetricStat
- type Options
- type Provenance
- type Report
- type RunFunc
- type RunOutput
- type Trajectory
Constants ¶
const ReportFormat = "bide.eval.report.v2"
ReportFormat is the format tag Run stamps on every Report (Report.Format). It names the JSON layout of a Report; Compare refuses a Report carrying any other tag with ErrFormat, so a report written by a different layout is never read as if its fields meant the same thing.
Variables ¶
var ErrFormat = fmt.Errorf("eval: unsupported report format: %w", agent.ErrProtocol)
ErrFormat is a Report whose Format is not ReportFormat: one from another layout, or a zero Report that no Run produced. It wraps agent.ErrProtocol.
var ErrInconclusive = errors.New("eval: inconclusive comparison")
ErrInconclusive is Gate's verdict when a metric could not be compared, or nothing was. It is a verdict, not a fault, so it wraps no agent error category.
var ErrRegression = errors.New("eval: significant regression")
ErrRegression is Gate's verdict when a metric regressed significantly. It is a verdict, not a fault, so it wraps no agent error category.
Functions ¶
func HashCases ¶
HashCases returns a stable lowercase-hex sha256 over the case set, in order. Each case contributes its Name, Input, and the JSON encoding of its Want, so the hash changes if any case's identity, input, or expectation changes but stays stable across runs of the same set. Use it to confirm two Reports scored the same cases before comparing them. A Want JSON cannot encode (a NaN or infinite float, a channel, a func) contributes its Go type and %#v form instead, marked apart from JSON; for a func or channel that form is an address, so it is stable only within one process.
func RequiredRuns ¶
RequiredRuns returns the per-arm sample size needed to detect a drop of minDetectableDrop from baselineRate in a two-proportion comparison at the given alpha (two-sided) and power, using the standard normal-approximation formula. It rounds up. Use it to size an evaluation before running it: a drop smaller than what RequiredRuns can resolve at your Runs will read as noise.
baselineRate must be in [0, 1] and alpha and power in (0, 1); anything else (NaN included) returns an error and no count. A drop of zero or less returns 0 (nothing to detect), and a drop past the baseline is sized as a drop to 0. A drop so small that the runs it needs exceed int returns math.MaxInt. When power is so low that any sample reaches it, the answer is 1.
Types ¶
type Case ¶
Case is one evaluation input plus optional expectations that metrics interpret. Tags are arbitrary labels (for example "hard", "kyc", "refund") used to stratify the Report: every tag a case carries gets its own aggregate pass rate, so a hard subset can be reported apart from an easy one.
type CaseReport ¶
type CaseReport struct {
Name string `json:"name"`
// Hash identifies the case: HashCases of this case alone, so it changes with the case's Name,
// Input or Want. Compare uses it to find the cases two reports share.
Hash string `json:"hash"`
Metrics map[string]MetricStat `json:"metrics"`
}
CaseReport holds one case's per-metric stats over its RunsPerCase executions.
type CompareOption ¶ added in v0.8.0
type CompareOption interface {
// contains filtered or unexported methods
}
CompareOption configures Compare.
func WithCaseSetMismatchAllowed ¶ added in v0.8.0
func WithCaseSetMismatchAllowed() CompareOption
WithCaseSetMismatchAllowed lets Compare compare two reports whose case sets differ (Provenance.CaseSetHash), over only the cases both ran: a case is shared when a CaseReport of each report has the same Hash, so a case whose input or expectation changed is not shared. Each metric's stats are then recomputed from the shared cases, the comparison of each metric says so in its Note, and with no shared case every metric is inconclusive.
Without it, reports over different case sets make every metric inconclusive: a rate that moved may only mean the cases changed.
func WithUnscoredTolerance ¶ added in v0.8.0
func WithUnscoredTolerance(frac float64) CompareOption
WithUnscoredTolerance sets the largest share of a metric's runs, in each report, that may be unscored for Compare to still compare it: frac is Unscored/(Scored+Unscored) and must be in [0, 1]. The default is 0: any unscored run makes the metric inconclusive. A metric with no scored run in either report is inconclusive whatever the tolerance.
Raising it trades rigor for availability. The pass rate is over scored runs, so it is unbiased only if the runs left unscored are like the rest; a grader that fails more on some outputs than others breaks that.
type Comparison ¶
type Comparison struct {
Metrics []MetricComparison `json:"metrics"`
}
Comparison is the full set of per-metric comparisons between two Reports, over the metrics present in either. It is a distribution-aware regression check, not a single pass/fail: it separates a real shift from sampling noise using a significance test and a multiple-comparison correction. Gate reduces it to the pass/fail a CI gate needs.
func Compare ¶
func Compare(old, new Report, opts ...CompareOption) (Comparison, error)
Compare pairs each metric in either report's Overall stats and tests whether its pass rate changed. For each metric both reports scored (within the unscored tolerance) it computes a two-sided p-value over the scored runs (a two-proportion z-test in general, or Fisher's exact test when any expected cell count is below 5), then applies a Benjamini-Hochberg correction across those metrics to control the false discovery rate. A metric is flagged Significant when its adjusted p-value is below 0.05, and Direction names the sign of the change. Every other metric, including one present in only one report, is DirectionInconclusive.
The reports must have run the same case set: the same non-empty Provenance.CaseSetHash. If they did not, every metric is inconclusive, with a Note naming both hashes, unless WithCaseSetMismatchAllowed compares them over their shared cases.
Both reports must carry Format ReportFormat, as every Report that Run returns does; otherwise Compare returns an error wrapping ErrFormat and no comparison. A report of another layout (or a zero Report) decoded into this type would otherwise compare as if its metrics were absent. A bad option is an error wrapping agent.ErrConfig.
func (Comparison) Gate ¶ added in v0.8.0
func (c Comparison) Gate() error
Gate is the comparison as a CI gate: nil only when at least one metric was compared and every metric is DirectionFlat or DirectionImprovement. Otherwise it returns an error naming each offending metric, wrapping ErrRegression for a significant regression and ErrInconclusive for a metric that was not compared (or for an empty comparison), so an unmeasured metric never passes.
func (Comparison) String ¶
func (c Comparison) String() string
String renders a comparison as a readable table: rate delta, raw and adjusted p-values, and the significance direction per metric, or why a metric is inconclusive. Metrics keep the ascending order Compare produced.
type Metric ¶
type Metric struct {
Name string
Fn func(ctx context.Context, c Case, out RunOutput) (bool, error)
// contains filtered or unexported fields
}
Metric scores a single run of a case. Fn returns (true, nil) for a pass and (false, nil) for a fail. A non-nil error means the metric could not score the run (its judge model was unreachable, the trajectory could not be read): the run is unscored for this metric, which is neither a pass nor a fail, and the boolean is ignored. A Report counts unscored runs apart and computes pass rates over scored runs only; Compare treats a metric with unscored runs as inconclusive unless a tolerance allows them (WithUnscoredTolerance).
Return an error only when the metric itself could not do its job, never for a property of the agent's output: an output that makes a grader misbehave is a fail, or it would drop out of the pass rate instead of lowering it.
Name keys the metric in a Report, so it must be non-empty and distinct among the metrics of one Run, and Fn must be non-nil; Run returns an error wrapping agent.ErrConfig otherwise.
func CalledTool ¶
CalledTool passes iff the run called a tool with this name at least once. A run whose trajectory could not be read (RunOutput.TraceErr) is unscored.
func Custom ¶
Custom builds a metric from a name and a predicate with Metric.Fn's contract: an error leaves the run unscored. The predicate may inspect out.Trace. A nil fn makes Run return an error wrapping agent.ErrConfig.
func GovernanceHeld ¶
func GovernanceHeld(name string, compliant func(ctx context.Context, out RunOutput) (bool, error)) Metric
GovernanceHeld passes iff compliant(ctx, out) returns true, and leaves the run unscored when it returns an error (the governance log it reads could not be read, for example). The caller supplies compliant to inspect the run's governed outcome (for example the final message or the trajectory's recorded decision) and report whether the containment layer kept it compliant.
The point is the two-number reporting pattern it enables. Run a raw judgment metric (did the model decide correctly?) alongside GovernanceHeld (did the governed outcome stay within policy?) and report both, for example "model correct 91%, governance held 100%". The gap between the two quantifies the value of containment: it is the fraction of runs where the model erred but the governance layer still caught it, so the outcome remained compliant despite the model's mistake. This metric stays lean and does not couple to the govern package; the predicate is the only seam. It receives the evaluation's context, so a predicate that reads a store or a governance log can honour cancellation. A nil compliant makes Run return an error wrapping agent.ErrConfig.
func Judge ¶
Judge is an LLM-as-judge metric: it asks a model whether the output satisfies rubric, passing iff the judge's reply, less surrounding whitespace, is exactly PASS. Any other reply fails, including one that only starts with PASS, and so does a run that ended in an error. A judge call that fails (the judge provider is down, rate limited, or ctx ended) leaves the run unscored, since the output was never graded. A reply that is not PASS is a fail and never unscored: the output being graded may have steered the judge, and it must not steer itself out of the pass rate. It is itself stochastic (a model grading a model), so treat it as a signal, run it with Runs well above 1, and never read it as a verdict.
The rubric and the grading instructions are the judge's system message. The case input and the output graded, which the operator does not control, are the user message: one JSON object, so no line break or text in them can pass for the rubric, the instructions, or the end of the data.
A nil m makes Run return an error wrapping agent.ErrConfig.
func Matches ¶
Matches passes iff the run did not error and the final message text matches re. Its name is "matches:" followed by re's source text. The caller compiles re, so a bad pattern is the caller's regexp.Compile error; a nil re makes Run return an error wrapping agent.ErrConfig.
type MetricComparison ¶
type MetricComparison struct {
Metric string `json:"metric"`
OldRate float64 `json:"old_rate"`
NewRate float64 `json:"new_rate"`
RateDelta float64 `json:"rate_delta"`
OldPasses int `json:"old_passes"`
OldScored int `json:"old_scored"`
OldUnscored int `json:"old_unscored"`
NewPasses int `json:"new_passes"`
NewScored int `json:"new_scored"`
NewUnscored int `json:"new_unscored"`
PValue float64 `json:"p_value"`
AdjustedP float64 `json:"adjusted_p"`
Significant bool `json:"significant"`
Direction MetricDirection `json:"direction"`
Note string `json:"note,omitempty"`
}
MetricComparison is one metric's before/after comparison between two Reports. RateDelta is the signed change in pass rate (new minus old), each rate over that report's scored runs. PValue is the raw two-sided significance of the change; AdjustedP is that p-value after a Benjamini-Hochberg correction across the metrics compared in the same Comparison. Significant is true when AdjustedP < 0.05. Direction reads the sign of a significant change (DirectionRegression for a drop, DirectionImprovement for a rise) or DirectionFlat when the change is not significant.
A metric Compare could not compare is DirectionInconclusive, with Note saying why: the reports ran different case sets (unless WithCaseSetMismatchAllowed), it is missing from one report, one report has no scored run of it, or its unscored share in either report is above the tolerance (WithUnscoredTolerance). A metric compared over only the cases both reports share has a Note saying so. No test is run for it: Significant is false, RateDelta is 0, PValue and AdjustedP are 1, and it is left out of the Benjamini-Hochberg family.
type MetricDirection ¶ added in v0.8.0
type MetricDirection string
MetricDirection is the reading of one metric's change between two Reports. It is a closed set: DirectionRegression, DirectionImprovement, DirectionFlat and DirectionInconclusive.
const ( // DirectionRegression is a significant drop in pass rate. DirectionRegression MetricDirection = "regression" // DirectionImprovement is a significant rise in pass rate. DirectionImprovement MetricDirection = "improvement" // DirectionFlat is a change that is not significant (or no change). DirectionFlat MetricDirection = "flat" // DirectionInconclusive is a metric that was not compared (MetricComparison.Note says why). It // is never a pass: Gate reports it. DirectionInconclusive MetricDirection = "inconclusive" )
type MetricStat ¶
type MetricStat struct {
Passes int `json:"passes"`
Scored int `json:"scored"`
Unscored int `json:"unscored"`
Rate float64 `json:"rate"`
CILow float64 `json:"ci_low"`
CIHigh float64 `json:"ci_high"`
}
MetricStat is a metric's pass count over some runs, with a 95% Wilson score confidence interval on the true pass rate. Scored is the runs the metric scored (passed or failed) and Unscored the runs it could not score (its Fn returned an error); Scored+Unscored is every run counted. Rate, CILow and CIHigh are over the scored runs only, so an unscored run moves neither the rate nor the interval: with no scored runs the rate is 0 and the interval is [0, 1], no evidence. The interval is wide for few scored runs, by design: it stops a lucky 4/5 from reading as a solid 80%.
type Options ¶
type Options struct {
Runs int
Concurrency int
// Provenance records the run conditions (model, temperature, seed, timestamp) to stamp onto the
// Report. It is optional; Run always overwrites CaseSetHash with a hash of the cases scored,
// regardless of what the caller supplies here.
Provenance Provenance
}
Options configures a run. Runs is the executions per case (default 1); use many more to sample a stochastic model and get a meaningful confidence interval. Concurrency caps in-flight executions (default 8, friendly to provider rate limits).
type Provenance ¶
type Provenance struct {
ModelID string `json:"model_id"`
Temperature float64 `json:"temperature"`
Seed int64 `json:"seed"`
Timestamp time.Time `json:"timestamp"`
CaseSetHash string `json:"case_set_hash"`
}
Provenance records the conditions under which an evaluation ran, so a Report can be reproduced and audited: which model version, at what temperature and seed, when, and a hash of the exact case set scored. Pin ModelID and set Temperature to 0 for the most reproducible baseline. CaseSetHash is filled by Run from the cases regardless of what the caller supplies.
type Report ¶
type Report struct {
Format string `json:"format"`
RunsPerCase int `json:"runs_per_case"`
TotalRuns int `json:"total_runs"`
Cases []CaseReport `json:"cases"`
Overall map[string]MetricStat `json:"overall"`
ByTag map[string]map[string]MetricStat `json:"by_tag,omitempty"`
LatencyP50 time.Duration `json:"latency_p50"`
LatencyP95 time.Duration `json:"latency_p95"`
Provenance Provenance `json:"provenance"`
}
Report is the statistical result of an evaluation: per-case and aggregate pass rates with confidence intervals, plus latency percentiles. JSON-marshalable. A distribution, not a verdict. ByTag stratifies the aggregate by case tag: tag -> metric -> stat, over the runs of every case carrying that tag. Provenance records the conditions under which the run happened (with a hash of the case set) for reproducibility and audit. Format is ReportFormat.
func Run ¶
func Run(ctx context.Context, run RunFunc, cases []Case, metrics []Metric, opts Options) (Report, error)
Run executes each case Runs times through run, scores every execution with each metric, and returns a Report of pass rates (with Wilson confidence intervals) and latency percentiles. Executions run concurrently up to Options.Concurrency. Because the model is stochastic, the report is a distribution over runs, not a single pass/fail.
A metric whose Fn returns an error leaves that run unscored for that metric (MetricStat.Unscored) instead of failing it; see Metric.
If ctx is cancelled before every execution has finished, Run starts no further executions and returns ctx's error with an empty Report: executions cut short by the cancellation would score as failures, and the ones that did finish are not a representative sample.
Run returns an error wrapping agent.ErrConfig, before executing anything, when run is nil or a metric is misconfigured: a constructor reported a bad argument (Matches(nil), for example), its Fn is nil, its Name is empty, or two metrics share a Name (a Report keys metrics by name).
type RunFunc ¶
RunFunc executes one case input and returns its output. Wrap an agent with AgentRunner, or supply any function (useful for tests and non-agent baselines).
func AgentRunner ¶
AgentRunner wraps an Agent as a RunFunc, giving each run a unique runID so evaluation runs are independent, durable, and auditable. The ID is the prefix, a random nonce drawn once per AgentRunner, and a counter: an eval never reuses an earlier eval's runs, which would replay their recorded answers instead of sampling the model, even over a store kept between evals. It reads the run's journal from store (the same journal the Agent was built with) to populate the Trajectory for trajectory metrics; if that read fails, the output's TraceErr carries the failure and the trajectory metrics fail the run. A nil a or store is an error wrapping agent.ErrConfig.
type RunOutput ¶
type RunOutput struct {
Final agent.Message
Err error
RunID string
Trace Trajectory
TraceErr error
}
RunOutput is the result of executing one case: the final message, any error, the runID, and the trajectory (populated by AgentRunner; empty for a bare RunFunc).
TraceErr is set when the trajectory could not be read (AgentRunner sets it when the run's journal read fails). The run itself may have succeeded, so Err stays the run's own error; the trajectory metrics (CalledTool, MaxSteps, ToolOrder) leave a run whose TraceErr is set unscored rather than score an empty trajectory, and a custom metric that reads Trace should do the same.
type Trajectory ¶
Trajectory is the agent's behavior on one run, reconstructed from the durable journal: how many model turns it took and which tools it called, in order, plus the raw records for custom metrics.
func TrajectoryFrom ¶
func TrajectoryFrom(records []agent.Record) Trajectory
TrajectoryFrom reconstructs a Trajectory from a run's journal records.
Source Files