Documentation
¶
Overview ¶
Package eval is a statistical evaluation harness for agents. It answers "does the model decide well?" the only way a stochastic model can be answered: run labeled cases multiple times and report a pass-rate distribution with confidence intervals, not a single verdict.
Read the boundary carefully, it is the point of this package. eval measures the MODEL's judgment, statistically and best-effort; it is NOT a proof and a pass rate is not a guarantee. The provable parts of the system are separate: the governed policy's convergence and invariant enforcement (the govern package and the gsm proof) bound what the model can do for all inputs, and the audit package proves what it did. eval quantifies the model; governance contains it; audit records it.
Rigor: because a raw pass rate over few runs is misleading (4/5 is not "80%", it is 80% with a wide interval), every rate carries a 95% Wilson confidence interval, and the harness evaluates the agent's TRAJECTORY (which tools it called, how many steps it took), not only the final message, using the durable journal. Evaluate with Runs well above 1, pin the model version, and set temperature 0 for the most reproducible baseline (still not perfectly deterministic).
Index ¶
- func HashCases(cases []Case) string
- func RequiredRuns(baselineRate, minDetectableDrop, alpha, power float64) int
- type Case
- type CaseReport
- type Comparison
- type Metric
- func CalledTool(name string) Metric
- func Contains(substr string) Metric
- func Custom(name string, fn func(ctx context.Context, c Case, out RunOutput) bool) Metric
- func GovernanceHeld(name string, compliant func(out RunOutput) bool) Metric
- func Judge(name string, m agent.Model, rubric string) Metric
- func Matches(pattern string) Metric
- func MaxSteps(n int) Metric
- func NoError() Metric
- func ToolOrder(names ...string) Metric
- type MetricComparison
- type MetricStat
- type Options
- type Provenance
- type Report
- type RunFunc
- type RunOutput
- type Trajectory
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func HashCases ¶
HashCases returns a stable lowercase-hex sha256 over the case set, in order. Each case contributes its Name, Input, and the JSON encoding of its Want, so the hash changes if any case's identity, input, or expectation changes but stays stable across runs of the same set. Use it to confirm two Reports scored the same cases before comparing them.
func RequiredRuns ¶
RequiredRuns returns the per-arm sample size needed to detect a drop of minDetectableDrop from baselineRate in a two-proportion comparison at the given alpha (two-sided) and power, using the standard normal-approximation formula. It rounds up. Use it to size an evaluation before running it: a drop smaller than what RequiredRuns can resolve at your Runs will read as noise.
Types ¶
type Case ¶
Case is one evaluation input plus optional expectations that metrics interpret. Tags are arbitrary labels (for example "hard", "kyc", "refund") used to stratify the Report: every tag a case carries gets its own aggregate pass rate, so a hard subset can be reported apart from an easy one.
type CaseReport ¶
type CaseReport struct {
Name string `json:"name"`
Metrics map[string]MetricStat `json:"metrics"`
}
CaseReport holds one case's per-metric stats over Runs executions.
type Comparison ¶
type Comparison struct {
Metrics []MetricComparison `json:"metrics"`
}
Comparison is the full set of per-metric comparisons between two Reports, over the metrics present in both. It is a distribution-aware regression check, not a single pass/fail: it separates a real shift from sampling noise using a significance test and a multiple-comparison correction.
func Compare ¶
func Compare(old, new Report) Comparison
Compare pairs each metric present in BOTH reports' Overall stats and tests whether its pass rate changed. For each metric it computes a two-sided p-value (a two-proportion z-test in general, or Fisher's exact test when any expected cell count is below 5), then applies a Benjamini-Hochberg correction across all compared metrics to control the false discovery rate. A metric is flagged Significant when its adjusted p-value is below 0.05, and Direction names the sign of the change.
func (Comparison) String ¶
func (c Comparison) String() string
String renders a comparison as a readable table: rate delta, raw and adjusted p-values, and the significance direction per metric. Metrics keep the ascending order Compare produced.
type Metric ¶
Metric scores a single run of a case as pass (true) or fail (false).
func CalledTool ¶
CalledTool passes iff the run called a tool with this name at least once.
func Custom ¶
Custom builds a metric from a name and a predicate; the predicate may inspect out.Trace.
func GovernanceHeld ¶
GovernanceHeld passes iff compliant(out) is true. The caller supplies compliant to inspect the run's governed outcome (for example the final message or the trajectory's recorded decision) and report whether the containment layer kept it compliant.
The point is the two-number reporting pattern it enables. Run a raw judgment metric (did the model decide correctly?) alongside GovernanceHeld (did the governed outcome stay within policy?) and report both, for example "model correct 91%, governance held 100%". The gap between the two quantifies the value of containment: it is the fraction of runs where the model erred but the governance layer still caught it, so the outcome remained compliant despite the model's mistake. This metric stays lean and does not couple to the govern package; the predicate is the only seam.
func Judge ¶
Judge is an LLM-as-judge metric: it asks a model whether the output satisfies rubric, passing iff the judge answers PASS. It is itself stochastic (a model grading a model), so treat it as a signal, run it with Runs well above 1, and never read it as a verdict.
type MetricComparison ¶
type MetricComparison struct {
Metric string `json:"metric"`
OldRate float64 `json:"old_rate"`
NewRate float64 `json:"new_rate"`
RateDelta float64 `json:"rate_delta"`
OldPasses int `json:"old_passes"`
OldRuns int `json:"old_runs"`
NewPasses int `json:"new_passes"`
NewRuns int `json:"new_runs"`
PValue float64 `json:"p_value"`
AdjustedP float64 `json:"adjusted_p"`
Significant bool `json:"significant"`
Direction string `json:"direction"`
}
MetricComparison is one metric's before/after comparison between two Reports. RateDelta is the signed change in pass rate (new minus old). PValue is the raw two-sided significance of the change; AdjustedP is that p-value after a Benjamini-Hochberg correction across all metrics compared in the same Comparison. Significant is true when AdjustedP < 0.05. Direction reads the sign of a significant change ("regression" for a drop, "improvement" for a rise) or "flat" when the change is not significant.
type MetricStat ¶
type MetricStat struct {
Passes int `json:"passes"`
Runs int `json:"runs"`
Rate float64 `json:"rate"`
CILow float64 `json:"ci_low"`
CIHigh float64 `json:"ci_high"`
}
MetricStat is a metric's pass count over some runs, with a 95% Wilson score confidence interval on the true pass rate. The interval is wide for small Runs, by design: it stops a lucky 4/5 from reading as a solid 80%.
type Options ¶
type Options struct {
Runs int
Concurrency int
// Provenance records the run conditions (model, temperature, seed, timestamp) to stamp onto the
// Report. It is optional; Run always overwrites CaseSetHash with a hash of the cases scored,
// regardless of what the caller supplies here.
Provenance Provenance
}
Options configures a run. Runs is the executions per case (default 1); use many more to sample a stochastic model and get a meaningful confidence interval. Concurrency caps in-flight executions (default 8, friendly to provider rate limits).
type Provenance ¶
type Provenance struct {
ModelID string `json:"model_id"`
Temperature float64 `json:"temperature"`
Seed int64 `json:"seed"`
Timestamp time.Time `json:"timestamp"`
CaseSetHash string `json:"case_set_hash"`
}
Provenance records the conditions under which an evaluation ran, so a Report can be reproduced and audited: which model version, at what temperature and seed, when, and a hash of the exact case set scored. Pin ModelID and set Temperature to 0 for the most reproducible baseline. CaseSetHash is filled by Run from the cases regardless of what the caller supplies.
type Report ¶
type Report struct {
RunsPerCase int `json:"runs_per_case"`
TotalRuns int `json:"total_runs"`
Cases []CaseReport `json:"cases"`
Overall map[string]MetricStat `json:"overall"`
ByTag map[string]map[string]MetricStat `json:"by_tag,omitempty"`
LatencyP50 time.Duration `json:"latency_p50"`
LatencyP95 time.Duration `json:"latency_p95"`
Provenance Provenance `json:"provenance"`
}
Report is the statistical result of an evaluation: per-case and aggregate pass rates with confidence intervals, plus latency percentiles. JSON-marshalable. A distribution, not a verdict. ByTag stratifies the aggregate by case tag: tag -> metric -> stat, over the runs of every case carrying that tag. Provenance records the conditions under which the run happened (with a hash of the case set) for reproducibility and audit.
func Run ¶
Run executes each case Runs times through run, scores every execution with each metric, and returns a Report of pass rates (with Wilson confidence intervals) and latency percentiles. Executions run concurrently up to Options.Concurrency. Because the model is stochastic, the report is a distribution over runs, not a single pass/fail.
type RunFunc ¶
RunFunc executes one case input and returns its output. Wrap an agent with AgentRunner, or supply any function (useful for tests and non-agent baselines).
func AgentRunner ¶
AgentRunner wraps an Agent as a RunFunc, giving each run a unique runID (prefix plus a counter) so evaluation runs are independent, durable, and auditable. It reads the run's journal from store (the same Durable the Agent was built with) to populate the Trajectory for trajectory metrics.
type RunOutput ¶
type RunOutput struct {
Final agent.Message
Err error
RunID string
Trace Trajectory
}
RunOutput is the result of executing one case: the final message, any error, the runID, and the trajectory (populated by AgentRunner; empty for a bare RunFunc).
type Trajectory ¶
Trajectory is the agent's behavior on one run, reconstructed from the durable journal: how many model turns it took and which tools it called, in order, plus the raw records for custom metrics.
func TrajectoryFrom ¶
func TrajectoryFrom(records []agent.Record) Trajectory
TrajectoryFrom reconstructs a Trajectory from a run's journal records.
Source Files