Documentation
¶
Overview ¶
Package baseline diffs two eval reports and produces a structured Diff + threshold-driven exit-code decision suitable for CI gating.
Inputs may be a single report.json or a directory containing one (the CLI accepts both — most users point at tests/eval/reports/<run-id>/). The schema_version of both inputs is checked; mismatched versions hard-fail rather than silently misalign fields.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Breaches ¶
type Breaches struct {
ScoreDrop bool `json:"score_drop"`
LatencyP95Drop bool `json:"latency_p95_drop"`
NewFailures bool `json:"new_failures"`
CustomScoreDrop bool `json:"custom_score_drop"`
}
Breaches signals which thresholds were crossed; populated by Decide.
type CaseDelta ¶
type CaseDelta struct {
CaseID string `json:"case_id"`
PassBefore bool `json:"pass_before"`
PassAfter bool `json:"pass_after"`
ScoreDelta float64 `json:"score_delta"`
LatencyP50Ms int `json:"latency_p50_ms_after,omitempty"`
LatencyP95Ms int `json:"latency_p95_ms_after,omitempty"`
LatencyP50Diff int `json:"latency_p50_ms_delta"`
LatencyP95Diff int `json:"latency_p95_ms_delta"`
TokensDelta int `json:"tokens_delta"`
// CustomScoreDeltas holds candidate-minus-baseline for every name
// present in either side's CustomScores (the union of both key sets).
// A name present on only one side is treated as zero on the missing
// side — introducing a new custom score reads as a full gain from 0,
// and a custom score that disappears reads as a full drop to 0. This
// mirrors how TokensDelta/latency already treat an absent diagnostic
// as zero rather than skipping the case.
CustomScoreDeltas map[string]float64 `json:"custom_score_deltas,omitempty"`
// Notes are short human-readable explanations of state transitions —
// "regression" (pass→fail), "recovery" (fail→pass), "" otherwise.
Note string `json:"note,omitempty"`
}
CaseDelta records how a single case moved between runs.
type Diff ¶
type Diff struct {
SchemaVersion string `json:"schema_version"`
GeneratedAt time.Time `json:"generated_at"`
AgainstRunID string `json:"against_run_id,omitempty"`
FreshRunID string `json:"fresh_run_id,omitempty"`
Summary DiffSummary `json:"summary"`
Cases []*CaseDelta `json:"cases"`
// Threshold breaches surface as bools so `nexus eval baseline` can
// translate them into exit codes without re-walking the structure.
Breached Breaches `json:"breaches"`
NewCases []string `json:"new_cases,omitempty"`
GoneCases []string `json:"gone_cases,omitempty"`
}
Diff is the per-run summary of how Fresh compares to Against.
func Compute ¶
Compute builds a Diff from two reports. against is the baseline; fresh is the new run. Stable ordering: cases are sorted by ID for snapshot stability.
func (*Diff) Decide ¶
func (d *Diff) Decide(t Thresholds) bool
Decide applies thresholds. Returns true when CI should fail.
type DiffSummary ¶
type DiffSummary struct {
AgainstTotal int `json:"against_total"`
FreshTotal int `json:"fresh_total"`
AgainstPassed int `json:"against_passed"`
FreshPassed int `json:"fresh_passed"`
PassRateDelta float64 `json:"pass_rate_delta"`
}
DiffSummary aggregates pass/fail movement across the union of cases.
type Thresholds ¶
type Thresholds struct {
// FailOnScoreDrop is the absolute pass-rate drop that triggers a failure
// exit (e.g. 0.05 = "5 percentage-point drop in pass-rate fails CI").
// Zero or negative disables the gate.
FailOnScoreDrop float64
// FailOnLatencyP95Drop is the relative latency p95 increase ratio that
// triggers a failure exit (0.20 = "20% slower fails CI"). Negative
// values disable the gate.
FailOnLatencyP95Drop float64
// FailOnCustomScoreDrop is keyed by the same names used in
// runner.Result.CustomScores / report.CaseEntry.CustomScores. Each
// name's value is the absolute drop (baseline - candidate) that
// triggers a failure exit for that name; a name absent from this map
// is not gated at all. Zero or negative for a given name disables its
// gate. Each configured name is evaluated independently — any one
// name's breach, in any one case, fails the run.
FailOnCustomScoreDrop map[string]float64
}
Thresholds encodes the eval.baseline config block.