comparison

package
v1.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 25, 2026 License: MIT Imports: 23 Imported by: 0

README

Comparing captured evaluations

Use CompareRecordedSets when you store each trial in its own evaluation job:

result, err := comparison.CompareRecordedSets(ctx, comparison.RecordedSetComparisonRequest{
    Store:        store,
    BaselineIDs:  baselineJobIDs,
    CandidateIDs: candidateJobIDs,
    Seed:         42,
    MinCases:     20,
})

Both arms must use the same frozen plan and contain the same case and repetition identities. You can batch either arm differently or pass job IDs in any order. The comparison reads the persisted jobs, checks their authenticated scope and request digests, and rejects repeated jobs or overlapping case/repetition pairs within an arm. A job cannot appear in both arms.

The result keeps each job ID and request digest in BaselineJobs and CandidateJobs. The older single-job fields are populated only when that arm contains one job. CompareRecorded remains available for two batched jobs and uses the same validation and statistical calculation.

Repetitions are averaged within each case. The bootstrap resamples distinct cases, so splitting a batch into more jobs cannot increase the sample size. The seed reproduces the 2,000 bootstrap samples and the 95 percent interval. Missing pairs are rejected. Pending or failed jobs produce an inconclusive report without an estimate, and a high mean score does not clear failed required criteria. Unknown costs stay unavailable.

You still need to check the selected job IDs against your experiment's frozen roster. Equal subsets on both arms do not prove that every planned case ran. The comparison does not execute targets or approve production activation.

Documentation

Overview

Package comparison enables side-by-side evaluation of multiple models or configurations against the same suite.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func FormatTable 

func FormatTable(w io.Writer, report *CompareReport)

FormatTable writes a terminal comparison table.

Types

type CompareConfig 

type CompareConfig struct {
	SuiteID id.SuiteID
	Targets []target.Target
	Scorers []scorer.Scorer
	Tags    []string
}

CompareConfig configures a multi-model comparison run.

type CompareReport 

type CompareReport struct {
	SuiteID   id.SuiteID    `json:"suite_id"`
	Models    []ModelResult `json:"models"`
	Duration  time.Duration `json:"duration"`
	CreatedAt time.Time     `json:"created_at"`
}

CompareReport holds the comparison results across all models.

func Compare 

func Compare(ctx context.Context, eng *engine.Engine, cfg *CompareConfig) (*CompareReport, error)

Compare runs the same suite against multiple targets and produces a comparison report. It runs inside a "compare targets" span, so each target's "evaluate run" span lands in one trace.

type ConfidenceInterval 

type ConfidenceInterval struct {
	Lower float64 `json:"lower"`
	Upper float64 `json:"upper"`
}

ConfidenceInterval is a 95 percent paired bootstrap interval over distinct cases.

type CriterionFailure 

type CriterionFailure struct {
	CaseID      id.CaseID `json:"case_id"`
	Repetition  int       `json:"repetition"`
	CriterionID string    `json:"criterion_id"`
	Reason      string    `json:"reason"`
}

CriterionFailure retains a candidate's failed required criterion.

type DiffReport 

type DiffReport struct {
	Deltas          []ScoreDelta       `json:"deltas"`
	DimensionDeltas map[string]float64 `json:"dimension_deltas,omitempty"`
}

DiffReport holds the deltas between a current run and a baseline.

func Diff 

func Diff(current, baseline *evalrun.ResultStats) *DiffReport

Diff computes score deltas between current run stats and baseline stats.

type ModelResult 

type ModelResult struct {
	ModelName string            `json:"model_name"`
	RunResult *engine.RunResult `json:"run_result"`
}

ModelResult holds the eval result for a single model in a comparison.

type RecordedComparison 

type RecordedComparison struct {
	Plans                  []RecordedPlanRef   `json:"plans,omitempty"`
	BaselineJobs           []RecordedJobRef    `json:"baseline_jobs"`
	CandidateJobs          []RecordedJobRef    `json:"candidate_jobs"`
	BaselineID             id.ID               `json:"baseline_id"`
	CandidateID            id.ID               `json:"candidate_id"`
	BaselineDigest         string              `json:"baseline_digest"`
	CandidateDigest        string              `json:"candidate_digest"`
	PlanDigest             string              `json:"plan_digest"`
	Seed                   uint64              `json:"seed"`
	Method                 string              `json:"method"`
	BootstrapSamples       int                 `json:"bootstrap_samples"`
	CaseCount              int                 `json:"case_count"`
	PairCount              int                 `json:"pair_count"`
	Pairs                  []RecordedPair      `json:"pairs"`
	MeanDelta              *float64            `json:"mean_delta,omitempty"`
	Interval               *ConfidenceInterval `json:"interval,omitempty"`
	DimensionDeltas        map[string]float64  `json:"dimension_deltas"`
	UnavailableDimensions  []string            `json:"unavailable_dimensions,omitempty"`
	RequiredCriteriaPassed bool                `json:"required_criteria_passed"`
	RequiredFailures       []CriterionFailure  `json:"required_failures,omitempty"`
	CostComparable         bool                `json:"cost_comparable"`
	Currency               string              `json:"currency,omitempty"`
	MeanCostDelta          *float64            `json:"mean_cost_delta,omitempty"`
	MeanLatencyDeltaMs     *float64            `json:"mean_latency_delta_ms,omitempty"`
	Conclusive             bool                `json:"conclusive"`
	IncompletenessReasons  []string            `json:"incompleteness_reasons,omitempty"`
}

RecordedComparison reports evidence without deciding production activation. Plans preserves every frozen identity. PlanDigest is empty for multiple plans.

func CompareRecorded 

func CompareRecorded(ctx context.Context, req RecordedComparisonRequest) (*RecordedComparison, error)

CompareRecorded checks exact plan/case/repetition identities before comparing. It averages repetitions within each case, then resamples 2,000 sets of cases. The nearest-rank 2.5 and 97.5 percentiles form the interval. No target is called.

func CompareRecordedGroups 

func CompareRecordedGroups(ctx context.Context, req RecordedGroupComparisonRequest) (*RecordedComparison, error)

CompareRecordedGroups preserves each plan and job identity while applying the same paired case bootstrap as CompareRecordedSets. A case cannot appear in two groups, even with different repetitions. The caller owns the complete roster.

func CompareRecordedSets 

func CompareRecordedSets(ctx context.Context, req RecordedSetComparisonRequest) (*RecordedComparison, error)

CompareRecordedSets compares disjoint persisted job sets from one frozen plan. Cases and repetitions must match across arms. Job order and batch size do not change the estimate. The caller remains responsible for selecting its full roster.

type RecordedComparisonGroup 

type RecordedComparisonGroup struct {
	Plan         RecordedPlanRef `json:"plan"`
	BaselineIDs  []id.ID         `json:"baseline_ids"`
	CandidateIDs []id.ID         `json:"candidate_ids"`
}

RecordedComparisonGroup selects paired job sets under one exact frozen plan.

type RecordedComparisonRequest 

type RecordedComparisonRequest struct {
	Store       RecordedStore
	BaselineID  id.ID
	CandidateID id.ID
	Seed        uint64
	MinCases    int // Zero uses 20 distinct cases. Repetitions never count as cases.
}

RecordedComparisonRequest selects persisted evaluations and a reproducible seed.

type RecordedGroupComparisonRequest 

type RecordedGroupComparisonRequest struct {
	Store    RecordedStore
	Groups   []RecordedComparisonGroup
	Seed     uint64
	MinCases int // Zero requires 20 distinct cases.
}

RecordedGroupComparisonRequest compares disjoint case sets across frozen plans. Every plan appears once. MinCases counts distinct cases across all groups.

type RecordedJobRef 

type RecordedJobRef struct {
	ID            id.ID  `json:"id"`
	RequestDigest string `json:"request_digest"`
}

RecordedJobRef identifies an authoritative job and its immutable input digest.

type RecordedPair 

type RecordedPair struct {
	CaseID     id.CaseID `json:"case_id"`
	Repetition int       `json:"repetition"`
	Baseline   float64   `json:"baseline"`
	Candidate  float64   `json:"candidate"`
	Delta      float64   `json:"delta"`
}

RecordedPair preserves the identities and scores behind one paired difference.

type RecordedPlanRef 

type RecordedPlanRef struct {
	ID     id.ID  `json:"id"`
	Digest string `json:"digest"`
}

RecordedPlanRef preserves a frozen plan's storage identity and content digest.

type RecordedSetComparisonRequest 

type RecordedSetComparisonRequest struct {
	Store        RecordedStore
	BaselineIDs  []id.ID
	CandidateIDs []id.ID
	Seed         uint64
	MinCases     int // Zero uses 20 distinct cases, regardless of job or repetition count.
}

RecordedSetComparisonRequest selects explicit job sets for each comparison arm. Every job must belong to the authenticated scope and the same frozen plan.

type RecordedStore 

type RecordedStore interface {
	GetRecordedJob(context.Context, id.ID) (*recorded.Job, error)
	GetEvaluationPlan(context.Context, id.ID) (*evalplan.Plan, error)
}

RecordedStore provides immutable evidence without any target execution API.

type ScoreDelta 

type ScoreDelta struct {
	Metric   string  `json:"metric"`
	Current  float64 `json:"current"`
	Baseline float64 `json:"baseline"`
	Delta    float64 `json:"delta"`
}

ScoreDelta records the change in a metric between current and baseline.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.