baseline

package
v0.31.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 31, 2026 License: MIT Imports: 7 Imported by: 0

Documentation

Overview

Package baseline diffs two eval reports and produces a structured Diff + threshold-driven exit-code decision suitable for CI gating.

Inputs may be a single report.json or a directory containing one (the CLI accepts both — most users point at tests/eval/reports/<run-id>/). The schema_version of both inputs is checked; mismatched versions hard-fail rather than silently misalign fields.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func LoadReport

func LoadReport(path string) (*report.Report, error)

LoadReport reads a report.json from path. path may be a file or a directory; for a dir we look for "report.json" inside.

Types

type Breaches

type Breaches struct {
	ScoreDrop       bool `json:"score_drop"`
	LatencyP95Drop  bool `json:"latency_p95_drop"`
	NewFailures     bool `json:"new_failures"`
	CustomScoreDrop bool `json:"custom_score_drop"`
}

Breaches signals which thresholds were crossed; populated by Decide.

type CaseDelta

type CaseDelta struct {
	CaseID         string  `json:"case_id"`
	PassBefore     bool    `json:"pass_before"`
	PassAfter      bool    `json:"pass_after"`
	ScoreDelta     float64 `json:"score_delta"`
	LatencyP50Ms   int     `json:"latency_p50_ms_after,omitempty"`
	LatencyP95Ms   int     `json:"latency_p95_ms_after,omitempty"`
	LatencyP50Diff int     `json:"latency_p50_ms_delta"`
	LatencyP95Diff int     `json:"latency_p95_ms_delta"`
	TokensDelta    int     `json:"tokens_delta"`
	// CustomScoreDeltas holds candidate-minus-baseline for every name
	// present in either side's CustomScores (the union of both key sets).
	// A name present on only one side is treated as zero on the missing
	// side — introducing a new custom score reads as a full gain from 0,
	// and a custom score that disappears reads as a full drop to 0. This
	// mirrors how TokensDelta/latency already treat an absent diagnostic
	// as zero rather than skipping the case.
	CustomScoreDeltas map[string]float64 `json:"custom_score_deltas,omitempty"`
	// Notes are short human-readable explanations of state transitions —
	// "regression" (pass→fail), "recovery" (fail→pass), "" otherwise.
	Note string `json:"note,omitempty"`
}

CaseDelta records how a single case moved between runs.

type Diff

type Diff struct {
	SchemaVersion string       `json:"schema_version"`
	GeneratedAt   time.Time    `json:"generated_at"`
	AgainstRunID  string       `json:"against_run_id,omitempty"`
	FreshRunID    string       `json:"fresh_run_id,omitempty"`
	Summary       DiffSummary  `json:"summary"`
	Cases         []*CaseDelta `json:"cases"`
	// Threshold breaches surface as bools so `nexus eval baseline` can
	// translate them into exit codes without re-walking the structure.
	Breached  Breaches `json:"breaches"`
	NewCases  []string `json:"new_cases,omitempty"`
	GoneCases []string `json:"gone_cases,omitempty"`
}

Diff is the per-run summary of how Fresh compares to Against.

func Compute

func Compute(against, fresh *report.Report) (*Diff, error)

Compute builds a Diff from two reports. against is the baseline; fresh is the new run. Stable ordering: cases are sorted by ID for snapshot stability.

func (*Diff) Decide

func (d *Diff) Decide(t Thresholds) bool

Decide applies thresholds. Returns true when CI should fail.

type DiffSummary

type DiffSummary struct {
	AgainstTotal  int     `json:"against_total"`
	FreshTotal    int     `json:"fresh_total"`
	AgainstPassed int     `json:"against_passed"`
	FreshPassed   int     `json:"fresh_passed"`
	PassRateDelta float64 `json:"pass_rate_delta"`
}

DiffSummary aggregates pass/fail movement across the union of cases.

type Thresholds

type Thresholds struct {
	// FailOnScoreDrop is the absolute pass-rate drop that triggers a failure
	// exit (e.g. 0.05 = "5 percentage-point drop in pass-rate fails CI").
	// Zero or negative disables the gate.
	FailOnScoreDrop float64
	// FailOnLatencyP95Drop is the relative latency p95 increase ratio that
	// triggers a failure exit (0.20 = "20% slower fails CI"). Negative
	// values disable the gate.
	FailOnLatencyP95Drop float64
	// FailOnCustomScoreDrop is keyed by the same names used in
	// runner.Result.CustomScores / report.CaseEntry.CustomScores. Each
	// name's value is the absolute drop (baseline - candidate) that
	// triggers a failure exit for that name; a name absent from this map
	// is not gated at all. Zero or negative for a given name disables its
	// gate. Each configured name is evaluated independently — any one
	// name's breach, in any one case, fails the run.
	FailOnCustomScoreDrop map[string]float64
}

Thresholds encodes the eval.baseline config block.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL