bench

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 13, 2026 License: MIT Imports: 4 Imported by: 0

Documentation

Overview

Package bench holds the pure-logic measurement core for the codegraph benchmark harness (PERF-01, INDX-06). Peak RSS is always measured externally, via the OS-level rusage of a child process the caller spawned (exec.Cmd.ProcessState.SysUsage()) — never via in-process Go runtime memory statistics, which cannot be compared fairly against the TS Node process (D-05). This package has no network or crypto surface and does not shell out itself; callers own the exec.Cmd. Measurement helpers never panic — they return (T, error) so a single bad run fails its CI step loudly instead of crashing the whole gate.

Index

Constants

View Source
const DefaultRSSTolerance = 0.15

DefaultRSSTolerance is the maximum allowed relative growth in peak RSS before the gate fails: 15% larger than the committed baseline. This is a D-05 starting point, deliberately tune-able by callers that need a different budget.

View Source
const DefaultThroughputTolerance = 0.10

DefaultThroughputTolerance is the maximum allowed relative regression in indexing throughput (files/sec) before the gate fails: 10% slower than the committed baseline. This is a D-05 starting point, deliberately tune-able by callers that need a different budget.

Variables

This section is empty.

Functions

func CheckRegression

func CheckRegression(baseline, current Metrics, ceilingBytes int64) error

CheckRegression compares a candidate run's Metrics (current) against a committed baseline and fails the gate (PERF-02) when any of:

  • baseline and current were measured on different platforms (GOOS/GOARCH), which makes every numeric comparison below meaningless — see the platform-mismatch check for why, or
  • indexing throughput regresses beyond DefaultThroughputTolerance, or
  • peak RSS grows beyond DefaultRSSTolerance relative to the baseline, or
  • peak RSS exceeds ceilingBytes, an absolute bounded-memory budget (INDX-06) enforced independently of the relative RSS delta — a baseline that itself already used a lot of memory must not let further growth hide behind a large denominator.

CheckRegression NEVER mutates baseline or current, and it never panics: a degenerate baseline (zero or negative FilesPerSec) returns a plain error instead of dividing by zero. Re-blessing the baseline (updating baseline.json to accept a new normal) is a separate, explicit action — an operator-invoked `-rebless` flag on the runner (Plan 08-07) — and is never a side effect of running this check. That separation is D-05's point: an accidental auto-rewrite here would silently defeat the gate.

func PeakRSSBytes

func PeakRSSBytes(state *os.ProcessState) (int64, error)

PeakRSSBytes returns the peak resident set size, in bytes, of the process described by state. state must come from a completed exec.Cmd (i.e. after Wait() returns) so ProcessState.SysUsage() is populated. The raw ru_maxrss value is normalized to bytes based on runtime.GOOS (see normalizeMaxrss) since its unit differs by OS.

Types

type Metrics

type Metrics struct {
	Subject string `json:"subject"` // e.g. "go" or "ts"
	Repo    string `json:"repo"`
	GOOS    string `json:"goos"`
	GOARCH  string `json:"goarch"`

	// Runner records the CI runner identity this Metrics was measured on
	// (e.g. a Namespace runs-on profile label such as
	// "namespace-profile-linux-amd64-4x8"), captured verbatim from the
	// workflow that ran it. It sits alongside GOOS/GOARCH because a
	// runner-class change (e.g. GitHub-hosted ubuntu-latest ->
	// Namespace) can hold GOOS/GOARCH constant while still changing the
	// measurement environment — namespace-profile-linux-amd64-4x8 IS
	// linux/amd64, so the GOOS/GOARCH guard alone is structurally blind
	// to that migration. Empty means "recorded before this field
	// existed" (e.g. any baseline.json predating this change), not an
	// error — measurement must not require it, since it is only
	// meaningful in CI. Runner does NOT yet participate in
	// CheckRegression's comparison; that lands separately, once a
	// Runner-labelled baseline exists to compare against.
	Runner string `json:"runner"`

	// ScratchFS records which filesystem class the regression corpus +
	// Pebble store were materialized on for THIS measurement: "tmpfs" or
	// "disk" (see tools/bench/runner's scratchFSTmpfs/scratchFSDisk).
	// Same philosophy as Runner: the storage frame is exactly as
	// load-bearing to a throughput comparison as the runner class is —
	// PERF-02's synthetic corpus is ~18MB of raw content but ~770MB of
	// actual on-disk footprint once small-file block overhead is
	// accounted for (120k files at ~4KB/block minimum each), and at that
	// file count the workload is IOPS/metadata-bound, not
	// bandwidth-bound. Comparing a tmpfs-recorded baseline against a
	// disk-recorded gate run (or vice versa) would be exactly the kind
	// of measurement-frame mismatch that produced the original
	// fictitious 10.6% "regression" — a different location for the same
	// failure class. Empty means "recorded before this field existed"
	// or "the caller supplied an explicit -scratch-dir and its
	// filesystem class wasn't characterized" — not an error. ScratchFS
	// does NOT yet participate in CheckRegression's comparison; wiring
	// that in, like Runner's, is a deliberately separate, later change.
	ScratchFS string `json:"scratch_fs"`

	// MedianOfTrials records how many INDEPENDENT measurement sessions
	// these numbers are the median of — where a session is a full
	// corpus-materialize + init + measure cycle, not a repeat of the
	// measured command inside one session (that inner repetition is
	// D-05's fixed median-of-5 and is not represented here).
	//
	// This is provenance, deliberately not a gate. Unlike GOOS/GOARCH, a
	// differing trial count is not a category error: median-of-3 and
	// median-of-5 estimate the same population median, just with
	// different precision, so refusing to compare them would be enforcing
	// a rule that isn't a validity rule. It is recorded and printed so
	// that a human reviewing a candidate baseline can see the procedure
	// that produced it — the thing nobody could see when a darwin
	// baseline was gating a linux runner.
	MedianOfTrials int `json:"median_of_trials"`

	FilesPerSec          float64 `json:"files_per_sec"`
	BytesPerSec          float64 `json:"bytes_per_sec"`
	QueryLatencyMedianMS float64 `json:"query_latency_median_ms"`
	PeakRSSBytes         int64   `json:"peak_rss_bytes"`
	ColdStartMS          float64 `json:"cold_start_ms"`
}

Metrics is a plain json-tagged data holder for one benchmark run's results — no methods with side effects, no network, no shelling out (mirrors internal/version.VersionInfo's discipline). Consumed by the committed-baseline regression gate (Plan 08-06) and the head-to-head Go-vs-TS runner (Plan 08-07).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL