compare

package
v0.8.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 12, 2026 License: Apache-2.0 Imports: 3 Imported by: 0

Documentation

Overview

Package compare diffs two report.Result values (typically JSON from `vegaload run -out`) so a later run can be checked against a baseline without a baseline registry or history store — just two files.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Gate added in v0.4.0

func Gate(baseline, candidate *report.Result, path string, pct float64) *report.BaselineResult

Gate judges a finished run against a baseline report (FR-CLI-14). It uses Compare, so `vegaload run -baseline` and `vegaload compare` agree. A run that completed no requests fails the gate: with nothing measured, "no worse than the baseline" would pass trivially.

Types

type CheckDiff added in v0.6.0

type CheckDiff struct {
	Name          string  `json:"name"`
	Status        string  `json:"status"` // "both", "new" (only in the candidate) or "removed" (only in the baseline)
	BaselineRate  float64 `json:"baseline_rate"`
	CandidateRate float64 `json:"candidate_rate"`
	BaselineRuns  int64   `json:"baseline_runs"`
	CandidateRuns int64   `json:"candidate_runs"`
	Delta         float64 `json:"delta"` // candidate rate - baseline rate
	Regressed     bool    `json:"regressed"`
}

CheckDiff is one named check compared across the two runs. A check that only one run made is reported with the status "new" or "removed" and is never a regression: the scenario changed, so there is nothing to compare.

type Metric

type Metric struct {
	Name      string  `json:"name"`
	Baseline  float64 `json:"baseline"`
	Candidate float64 `json:"candidate"`
	Delta     float64 `json:"delta"` // candidate - baseline
	Unit      string  `json:"unit"`  // "ratio", "ns", "rps", "count"
	Regressed bool    `json:"regressed"`
}

Metric is one compared field: baseline vs candidate, with a delta and whether that field alone counts as a regression under Options.

type Options

type Options struct {
	// ErrorRateDelta is absolute slack on error rate (0.01 = one
	// percentage point). Candidate may exceed baseline by this much
	// without counting as a regression.
	ErrorRateDelta float64
	// P95Ratio is the maximum allowed candidate/baseline p95 ratio.
	// 1.0 means any p95 increase is a regression; 1.2 allows 20% headroom.
	// Values below 1.0 are treated as 1.0.
	P95Ratio float64
	// CheckRateDelta is absolute slack on each named check's pass rate
	// (0.01 = one percentage point). A check whose pass rate falls by more
	// than this is a regression. Zero means any drop is.
	CheckRateDelta float64
	// SkipChecks leaves checks out of the verdict. The run's baseline gate
	// (-baseline) sets it: that gate is about p95 and the error rate.
	SkipChecks bool
}

Options tunes what counts as a regression. Zero values mean strict compare: any increase in error rate or p95 latency is a regression.

func GateOptions added in v0.4.0

func GateOptions(baseline *report.Result, pct float64) Options

GateOptions turns "no more than pct percent worse than the baseline" into Compare's options. p95 may rise by pct percent. The error rate may rise by pct percent of the baseline's error rate, so a baseline with no errors allows none: use a -threshold on error_rate for an absolute limit.

type Result

type Result struct {
	BaselinePath  string      `json:"baseline_path,omitempty"`
	CandidatePath string      `json:"candidate_path,omitempty"`
	Metrics       []Metric    `json:"metrics"`
	Checks        []CheckDiff `json:"checks,omitempty"`
	Regressed     bool        `json:"regressed"`
	Notes         []string    `json:"notes"`
}

Result is the structured outcome of Compare — the same shape printed as text or JSON by `vegaload compare`, and returned by the MCP compare_reports tool.

func Compare

func Compare(baseline, candidate *report.Result, opts Options) Result

Compare diffs candidate against baseline. Paths are optional labels filled in by the CLI; the comparison itself is purely numeric.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL