Documentation
¶
Overview ¶
Package baseline defines evaluation baselines for regression detection. A baseline captures the results of a known-good evaluation run against which future runs can be compared.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Baseline ¶
type Baseline struct {
ID id.BaselineID `json:"id" bun:"id,pk"`
SuiteID id.SuiteID `json:"suite_id" bun:"suite_id,notnull"`
RunID id.EvalRunID `json:"run_id" bun:"run_id,notnull"`
Name string `json:"name" bun:"name,notnull"`
Results []Result `json:"results" bun:"results,type:jsonb"`
PassRate float64 `json:"pass_rate" bun:"pass_rate,notnull,default:0"`
AvgScore float64 `json:"avg_score" bun:"avg_score,notnull,default:0"`
DimensionScores map[string]float64 `json:"dimension_scores,omitempty" bun:"dimension_scores,type:jsonb,default:'{}'"`
IsCurrent bool `json:"is_current" bun:"is_current,notnull,default:false"`
CreatedAt time.Time `json:"created_at" bun:"created_at,notnull,default:current_timestamp"`
}
Baseline captures the results of a known-good evaluation run.
type RegressedCase ¶
type RegressedCase struct {
CaseID string `json:"case_id"`
CaseName string `json:"case_name"`
OldScore float64 `json:"old_score"`
NewScore float64 `json:"new_score"`
Delta float64 `json:"delta"`
}
RegressedCase identifies a single case that regressed from baseline.
type RegressionResult ¶
type RegressionResult struct {
HasRegression bool `json:"has_regression"`
Threshold float64 `json:"threshold"`
PassRateDelta float64 `json:"pass_rate_delta"`
AvgScoreDelta float64 `json:"avg_score_delta"`
DimensionDeltas map[string]float64 `json:"dimension_deltas,omitempty"`
RegressedCases []RegressedCase `json:"regressed_cases,omitempty"`
// MissingCases are baseline cases with no result in the run. They do not
// set HasRegression: the suite changed, which is a different question.
MissingCases []CaseRef `json:"missing_cases,omitempty"`
// NewCases are run results with no baseline case to compare against.
NewCases []CaseRef `json:"new_cases,omitempty"`
// MissingDimensions are baseline dimensions the run did not measure.
// They set HasRegression, the conservative answer for CI, and are kept
// out of DimensionDeltas so nobody reads "not measured" as a large drop.
MissingDimensions []string `json:"missing_dimensions,omitempty"`
// contains filtered or unexported fields
}
RegressionResult holds the outcome of a regression detection check.
func DetectRegression ¶
func DetectRegression(stats *evalrun.ResultStats, results []*evalrun.Result, b *Baseline, threshold float64) *RegressionResult
DetectRegression compares a run's results against a baseline. A metric regresses when it falls more than threshold below the baseline; one absolute threshold covers pass rate, average score, each dimension and each case, and lower is always worse.
func (*RegressionResult) WorstDelta ¶
func (rr *RegressionResult) WorstDelta() float64
WorstDelta is the most negative delta found: pass rate, average score, any dimension or any regressed case. An unmeasured dimension counts as losing its whole baseline score, the most it could have fallen, so the value always agrees with HasRegression. It is what RegressionDetected hooks receive.
type Result ¶
type Result struct {
CaseID id.CaseID `json:"case_id"`
CaseName string `json:"case_name"`
Score float64 `json:"score"`
Status string `json:"status"`
DimensionScores map[string]float64 `json:"dimension_scores,omitempty"`
}
Result stores per-case baseline data for comparison.
type Store ¶
type Store interface {
SaveBaseline(ctx context.Context, b *Baseline) error
GetBaseline(ctx context.Context, baselineID id.BaselineID) (*Baseline, error)
GetLatestBaseline(ctx context.Context, suiteID id.SuiteID) (*Baseline, error)
ListBaselines(ctx context.Context, suiteID id.SuiteID) ([]*Baseline, error)
DeleteBaseline(ctx context.Context, baselineID id.BaselineID) error
}
Store defines persistence operations for baselines.
Source Files