Documentation
¶
Overview ¶
Package bench holds the pure-logic measurement core for the codegraph benchmark harness (PERF-01, INDX-06). Peak RSS is always measured externally, via the OS-level rusage of a child process the caller spawned (exec.Cmd.ProcessState.SysUsage()) — never via in-process Go runtime memory statistics: an externally observed child-process rusage figure is the only peak-RSS number that stays comparable across runs and machines, because an in-process Go runtime statistic measures the harness rather than the subject (D-05). This package has no network or crypto surface and does not shell out itself; callers own the exec.Cmd. Measurement helpers never panic — they return (T, error) so a single bad run fails its CI step loudly instead of crashing the whole gate.
Index ¶
Constants ¶
const DefaultRSSTolerance = 0.15
DefaultRSSTolerance is the maximum allowed relative growth in peak RSS before the gate fails: 15% larger than the committed baseline. This is a D-05 starting point, deliberately tune-able by callers that need a different budget.
const DefaultThroughputTolerance = 0.10
DefaultThroughputTolerance is the maximum allowed relative regression in indexing throughput (files/sec) before the gate fails: 10% slower than the committed baseline. This is a D-05 starting point, deliberately tune-able by callers that need a different budget.
Variables ¶
This section is empty.
Functions ¶
func CheckRegression ¶
CheckRegression compares a candidate run's Metrics (current) against a committed baseline and fails the gate (PERF-02) when any of:
- baseline and current were measured on different platforms (GOOS/GOARCH), which makes every numeric comparison below meaningless — see the platform-mismatch check for why, or
- indexing throughput regresses beyond DefaultThroughputTolerance, or
- peak RSS grows beyond DefaultRSSTolerance relative to the baseline, or
- peak RSS exceeds ceilingBytes, an absolute bounded-memory budget (INDX-06) enforced independently of the relative RSS delta — a baseline that itself already used a lot of memory must not let further growth hide behind a large denominator.
CheckRegression NEVER mutates baseline or current, and it never panics: a degenerate baseline (zero or negative FilesPerSec) returns a plain error instead of dividing by zero. A degenerate CURRENT reading (zero or negative FilesPerSec or PeakRSSBytes) is refused the same way, rather than being read as "no regression" or misattributed as a real one — backlog 999.4 recorded a current.PeakRSSBytes of 0 silently passing both the relative RSS check and the absolute INDX-06 ceiling. Re-blessing the baseline (updating baseline.json to accept a new normal) is a separate, explicit action — an operator-invoked `-rebless` flag on the runner (Plan 08-07) — and is never a side effect of running this check. That separation is D-05's point: an accidental auto-rewrite here would silently defeat the gate.
func PeakRSSBytes ¶
func PeakRSSBytes(state *os.ProcessState) (int64, error)
PeakRSSBytes returns the peak resident set size, in bytes, of the process described by state. state must come from a completed exec.Cmd (i.e. after Wait() returns) so ProcessState.SysUsage() is populated. The raw ru_maxrss value is normalized to bytes based on runtime.GOOS (see normalizeMaxrss) since its unit differs by OS.
Types ¶
type Metrics ¶
type Metrics struct {
Subject string `json:"subject"` // provenance label, e.g. "go" — not part of a measurement's identity
Repo string `json:"repo"`
GOOS string `json:"goos"`
GOARCH string `json:"goarch"`
// Runner records the CI runner identity this Metrics was measured on
// (e.g. a Namespace runs-on profile label such as
// "namespace-profile-linux-amd64-4x8"), captured verbatim from the
// workflow that ran it. It sits alongside GOOS/GOARCH because a
// runner-class change (e.g. GitHub-hosted ubuntu-latest ->
// Namespace) can hold GOOS/GOARCH constant while still changing the
// measurement environment — namespace-profile-linux-amd64-4x8 IS
// linux/amd64, so the GOOS/GOARCH guard alone is structurally blind
// to that migration. Empty means "recorded before this field
// existed" (e.g. any baseline.json predating this change), not an
// error — measurement must not require it, since it is only
// meaningful in CI. Runner does NOT yet participate in
// CheckRegression's comparison; that lands separately, once a
// Runner-labelled baseline exists to compare against.
Runner string `json:"runner"`
// ScratchFS records which filesystem class the regression corpus +
// Pebble store were materialized on for THIS measurement: "tmpfs" or
// "disk" (see tools/bench/runner's scratchFSTmpfs/scratchFSDisk).
// Same philosophy as Runner: the storage frame is exactly as
// load-bearing to a throughput comparison as the runner class is —
// PERF-02's synthetic corpus is ~18MB of raw content but ~770MB of
// actual on-disk footprint once small-file block overhead is
// accounted for (120k files at ~4KB/block minimum each), and at that
// file count the workload is IOPS/metadata-bound, not
// bandwidth-bound. Comparing a tmpfs-recorded baseline against a
// disk-recorded gate run (or vice versa) would be exactly the kind
// of measurement-frame mismatch that produced the original
// fictitious 10.6% "regression" — a different location for the same
// failure class. Empty means "recorded before this field existed"
// or "the caller supplied an explicit -scratch-dir and its
// filesystem class wasn't characterized" — not an error. ScratchFS
// does NOT yet participate in CheckRegression's comparison; wiring
// that in, like Runner's, is a deliberately separate, later change.
ScratchFS string `json:"scratch_fs"`
// MedianOfTrials records how many INDEPENDENT measurement sessions
// these numbers are the median of — where a session is a full
// corpus-materialize + init + measure cycle, not a repeat of the
// measured command inside one session (that inner repetition is
// D-05's fixed median-of-5 and is not represented here).
//
// This is provenance, deliberately not a gate. Unlike GOOS/GOARCH, a
// differing trial count is not a category error: median-of-3 and
// median-of-5 estimate the same population median, just with
// different precision, so refusing to compare them would be enforcing
// a rule that isn't a validity rule. It is recorded and printed so
// that a human reviewing a candidate baseline can see the procedure
// that produced it — the thing nobody could see when a darwin
// baseline was gating a linux runner.
MedianOfTrials int `json:"median_of_trials"`
FilesPerSec float64 `json:"files_per_sec"`
BytesPerSec float64 `json:"bytes_per_sec"`
QueryLatencyMedianMS float64 `json:"query_latency_median_ms"`
PeakRSSBytes int64 `json:"peak_rss_bytes"`
ColdStartMS float64 `json:"cold_start_ms"`
}
Metrics is a plain json-tagged data holder for one benchmark run's results — no methods with side effects, no network, no shelling out (mirrors internal/version.VersionInfo's discipline). Consumed by the committed-baseline regression gate and the absolute-numbers publisher (bench.yml's publish job).