compare

type CaseClass ¶

type CaseClass string

CaseClass is the classification of one compared case. It is a closed set.

const (
	// CaseAdded: the case exists in the candidate but not the baseline.
	CaseAdded CaseClass = "added"
	// CaseRemoved: the case exists in the baseline but not the candidate.
	CaseRemoved CaseClass = "removed"
	// CaseIncompatible: the case exists in both but the evaluator revision
	// changed, so the two are not comparable and distributions are not compared.
	CaseIncompatible CaseClass = "incompatible"
	// CaseErrored: the candidate case has an error-status assessment — the
	// evaluator failed to reach a verdict. Surfaced distinctly from a regression.
	CaseErrored CaseClass = "errored"
	// CaseUnverified: the candidate case has an unverified assessment (and no
	// error). Unknown is never a pass.
	CaseUnverified CaseClass = "unverified"
	// CaseFailed: the candidate case has a failing verdict (and no error or
	// unverified). A standing quality failure.
	CaseFailed CaseClass = "failed"
	// CaseChanged: the case passes in the candidate but its outcome or score
	// distribution differs from the baseline.
	CaseChanged CaseClass = "changed"
	// CaseUnchanged: the case passes in both with an equivalent distribution.
	CaseUnchanged CaseClass = "unchanged"
)

type CaseComparison ¶

type CaseComparison struct {
	Key               CaseKey
	Class             CaseClass
	Compatible        bool
	BaselineRevision  eval.Revision
	CandidateRevision eval.Revision
	Baseline          []TrialResult
	Candidate         []TrialResult
	Distributions     []MeasurementDelta
}

CaseComparison is the diff of one case. Baseline and Candidate hold the retained per-trial results; Distributions is non-empty only for a compatible case that carries measurements.

type CaseKey ¶

type CaseKey struct {
	ScenarioID string
	Evaluator  eval.Name
}

CaseKey identifies a compared case: a scenario and an evaluator name. The evaluator revision is a compatibility attribute, not part of the key, so a revision bump is reported as an incompatible case rather than as an unrelated add/remove pair.

type Comparison ¶

type Comparison struct {
	Cases []CaseComparison
}

Comparison is the full baseline-vs-candidate diff: one entry per case, in a canonical order.

func Compare(baseline, candidate eval.Report) (Comparison, error)

type ComparisonSide ¶

type ComparisonSide string

ComparisonSide names which of Compare's two input reports a failure concerns. It is a closed set of package constants, always safe to render — it carries no report content.

const (
	// SideBaseline is the baseline report passed to Compare.
	SideBaseline ComparisonSide = "baseline"
	// SideCandidate is the candidate report passed to Compare.
	SideCandidate ComparisonSide = "candidate"
)

type Distribution ¶

type Distribution struct {
	Count int
	Mean  float64
	Min   float64
	Max   float64
}

Distribution summarises a measurement across a case's trials.

type EvaluatorRevisionDriftError ¶

type EvaluatorRevisionDriftError struct{}

EvaluatorRevisionDriftError reports that a SINGLE report carried the same evaluator name under two different revisions across its samples. Within one report a name must identify exactly one revision; comparison keys a case by evaluator name, so a name mapped to two revisions cannot be gathered into one case without silently absorbing one revision as a trial of the other. Comparison rejects it fail-closed rather than corrupt the case. This is distinct from a legitimate cross-report revision change (baseline E@v1 vs candidate E@v2, each internally consistent), which surfaces as an incompatible case, not this error. No name or revision is embedded — both are report-supplied.

func (e *EvaluatorRevisionDriftError) Error() string

type InvalidReportError ¶

type InvalidReportError struct {
	// Side is the input report that failed validation.
	Side ComparisonSide
	// Cause is the report's own typed validation error.
	Cause error
}

InvalidReportError reports that one of Compare's input reports failed its own Report.Validate boundary check. Compare validates BOTH inputs before indexing so a hand-built or non-decoded report cannot bypass the report-level invariants (unique sample identities, unique evaluator names, consistent revisions) and silently merge distinct assessments into one case. Side names which input failed (baseline or candidate); the wrapped Cause is the report's own typed validation error, available via Unwrap so a caller can classify the underlying reason with errors.As. No report content is embedded: Side is a closed constant and the cause is itself content-free.

func (e *InvalidReportError) Error() string

func (e *InvalidReportError) Unwrap() error

type MeasurementDelta ¶

type MeasurementDelta struct {
	Name          eval.Name
	Unit          eval.Unit
	BaselineUnit  eval.Unit
	CandidateUnit eval.Unit
	UnitMismatch  bool
	Baseline      Distribution
	Candidate     Distribution
}

MeasurementDelta pairs a measurement's baseline and candidate distributions. It is populated only for compatible cases.

A measurement name may appear on both sides carrying DIFFERENT units (for example latency in seconds on the baseline and a bare count on the candidate). Numbers measured in different units are not comparable, so the two units are tracked separately and never collapsed: BaselineUnit and CandidateUnit expose each side's unit, and UnitMismatch flags the incompatibility. Unit is the agreed unit when both sides match (and the baseline's when they do not), retained for the common compatible case.

type NonFiniteMeasurementError ¶

type NonFiniteMeasurementError struct{}

NonFiniteMeasurementError reports that a report carried a measurement whose value was NaN or ±Inf. A well-formed report never contains one (the runner validates measurements at its boundary); comparison rejects it fail-closed rather than propagate a poisoned mean or min/max. No value is embedded — it is not finite and not safe to render as a number.

func (e *NonFiniteMeasurementError) Error() string

type TrialResult ¶

type TrialResult struct {
	TrialIndex   int
	Status       eval.AssessmentStatus
	Measurements []eval.Measurement
}

TrialResult is one evaluator assessment on one trial, retained so per-trial variance is visible. Measurements are the safe (name/value/unit) triples.