Documentation
¶
Overview ¶
Package compare diffs two report.Result values (typically JSON from `vegaload run -out`) so a later run can be checked against a baseline without a baseline registry or history store — just two files.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type CheckDiff ¶ added in v0.6.0
type CheckDiff struct {
Name string `json:"name"`
Status string `json:"status"` // "both", "new" (only in the candidate) or "removed" (only in the baseline)
BaselineRate float64 `json:"baseline_rate"`
CandidateRate float64 `json:"candidate_rate"`
BaselineRuns int64 `json:"baseline_runs"`
CandidateRuns int64 `json:"candidate_runs"`
Delta float64 `json:"delta"` // candidate rate - baseline rate
Regressed bool `json:"regressed"`
}
CheckDiff is one named check compared across the two runs. A check that only one run made is reported with the status "new" or "removed" and is never a regression: the scenario changed, so there is nothing to compare.
type Metric ¶
type Metric struct {
Name string `json:"name"`
Baseline float64 `json:"baseline"`
Candidate float64 `json:"candidate"`
Delta float64 `json:"delta"` // candidate - baseline
Unit string `json:"unit"` // "ratio", "ns", "rps", "count"
Regressed bool `json:"regressed"`
}
Metric is one compared field: baseline vs candidate, with a delta and whether that field alone counts as a regression under Options.
type Options ¶
type Options struct {
// ErrorRateDelta is absolute slack on error rate (0.01 = one
// percentage point). Candidate may exceed baseline by this much
// without counting as a regression.
ErrorRateDelta float64
// P95Ratio is the maximum allowed candidate/baseline p95 ratio.
// 1.0 means any p95 increase is a regression; 1.2 allows 20% headroom.
// Values below 1.0 are treated as 1.0.
P95Ratio float64
// CheckRateDelta is absolute slack on each named check's pass rate
// (0.01 = one percentage point). A check whose pass rate falls by more
// than this is a regression. Zero means any drop is.
CheckRateDelta float64
// SkipChecks leaves checks out of the verdict. The run's baseline gate
// (-baseline) sets it: that gate is about p95 and the error rate.
SkipChecks bool
}
Options tunes what counts as a regression. Zero values mean strict compare: any increase in error rate or p95 latency is a regression.
func GateOptions ¶ added in v0.4.0
GateOptions turns "no more than pct percent worse than the baseline" into Compare's options. p95 may rise by pct percent. The error rate may rise by pct percent of the baseline's error rate, so a baseline with no errors allows none: use a -threshold on error_rate for an absolute limit.
type Result ¶
type Result struct {
BaselinePath string `json:"baseline_path,omitempty"`
CandidatePath string `json:"candidate_path,omitempty"`
Metrics []Metric `json:"metrics"`
Checks []CheckDiff `json:"checks,omitempty"`
Regressed bool `json:"regressed"`
Notes []string `json:"notes"`
}
Result is the structured outcome of Compare — the same shape printed as text or JSON by `vegaload compare`, and returned by the MCP compare_reports tool.