Documentation
¶
Overview ¶
Package kpi is the stdlib-only KPI-capture support for the offline performance scenario harness (perf-tracking.md "The benchmark harness"). It owns the per-scenario metric shape (ScenarioResult), the allocation/RSS/wall-clock capture bracket (Capture), the linux /proc-based RSS sampler, and a settle-then-count goroutine probe.
LAYERING: this package imports ONLY the standard library. It must NOT import engine/... or internal/... — the engine-shaped KPIs (tokens, cache-hit-rate) are passed IN by the scenario caller (perf/scenarios), which is the only side that knows about session.Usage. That keeps kpi a leaf the engine never depends on and a scenario can freely consume.
Index ¶
Constants ¶
const SchemaVersion = 1
SchemaVersion is the on-disk JSON schema version for a ScenarioResult. Bump it when the field set changes shape so a trend-store ingester can branch.
Variables ¶
This section is empty.
Functions ¶
func GoroutineDelta ¶
GoroutineDelta returns the leaked-goroutine count: the settled end count minus a baseline the caller captured (also via GoroutinesAfterSettle) BEFORE the measured region, clamped at 0. So 0 = no leak; a positive value is the number of goroutines the scenario started and did not join. Clamping at 0 keeps a transient that the baseline happened to catch (a sampler/GC goroutine that has since exited) from reporting a spurious negative. This is the meaningful KPI for the delegation/team/background leak class — the raw process-wide count carries the test runner's own goroutines as noise.
func GoroutinesAfterSettle ¶
GoroutinesAfterSettle returns the live goroutine count after a settle delay, so transient run-teardown goroutines (a draining emitter, a parked sampler, the GC) are not miscounted as a leak. It is the tracked-number promotion of goleak's boolean pass/fail (perf-tracking.md "The benchmark harness"): the end-of-run goroutine count is a gate-hard KPI for the delegation/team/background leak class.
It sleeps d (clamped to a small floor), runs a GC to release goroutines parked on finalizers/timers, then reads runtime.NumGoroutine. The reading includes the caller's own goroutine and the test runner's, so the harness compares it against a same-process baseline rather than treating it as an absolute — see GoroutineDelta, which is what scenarios actually store.
func WriteJSON ¶
func WriteJSON(path string, results []ScenarioResult) error
WriteJSON marshals the accumulated scenario results to path as a pretty JSON array. The caller decides the path (typically $MECATL_PERF_JSON); an empty slice still writes a valid (empty) array so a downstream ingester never chokes on a missing file vs an empty one.
Types ¶
type Capture ¶
type Capture struct {
// contains filtered or unexported fields
}
Capture brackets a measured region and reports its allocation, RSS, and wall-clock deltas. Use it as:
c := kpi.NewCapture() c.Begin() ... run the scenario ... m := c.End()
Begin runs a GC and snapshots the allocation counters so the captured allocs exclude prior setup garbage; End runs a final GC and reads the deltas. The RSS sampler runs between Begin and End. The numbers are process-wide (Go's MemStats and /proc are per-process), so a Capture must wrap a single scenario run with no concurrent benchmark in the same process — the harness runs scenarios sequentially, which holds.
type Metrics ¶
type Metrics struct {
// Allocs is the number of heap allocations during the measured region.
Allocs uint64
// Bytes is the cumulative bytes allocated during the measured region.
Bytes uint64
// RSSPeak is the highest sampled resident set size, in bytes (0 off-linux).
RSSPeak uint64
// RSSFinal is the resident set size at End, in bytes (0 off-linux).
RSSFinal uint64
// WallNs is the elapsed wall-clock time of the measured region, in ns.
WallNs int64
}
Metrics is the result of one Capture bracket.
type RSSSampler ¶
type RSSSampler struct {
// contains filtered or unexported fields
}
RSSSampler polls the process resident-set size on a fixed interval and tracks the peak and the most-recent (final) sample. It is platform-pluggable through readRSS (a direct /proc/self/status read on linux, 0 elsewhere — see rss_linux.go / rss_other.go), so the harness takes no new dependency.
Start launches a sampling goroutine; Stop signals it, waits for it to drain, takes one final sample, and returns (peak, final). A sampler is single-use.
func NewRSSSampler ¶
func NewRSSSampler(interval time.Duration) *RSSSampler
NewRSSSampler returns a sampler that polls every interval. An interval <= 0 is clamped to 10ms.
func (*RSSSampler) Start ¶
func (s *RSSSampler) Start()
Start begins sampling in the background. It records an immediate first sample so a region shorter than one interval still yields a peak.
func (*RSSSampler) Stop ¶
func (s *RSSSampler) Stop() (peak, final uint64)
Stop ends sampling, takes a final reading, and returns (peak, final) in bytes. On a non-linux build both are 0.
type ScenarioResult ¶
type ScenarioResult struct {
SchemaVersion int `json:"schema_version"`
Name string `json:"name"`
// Sample is the per-scenario-name ordinal (0,1,2…) of this row within one
// process run. Under `go test -count=N` a scenario emits N rows with identical
// Name+GitSHA; Sample is the only thing that distinguishes them, letting the CI
// ingester group by (Name, GitSHA) and aggregate the N samples.
Sample int `json:"sample"`
GitSHA string `json:"git_sha"`
Timestamp string `json:"timestamp"`
Iterations int `json:"iterations"`
AllocsPerOp uint64 `json:"allocs_per_op"`
BytesPerOp uint64 `json:"bytes_per_op"`
// GoroutinesEnd is a DELTA: live goroutines at end-of-run MINUS a baseline
// captured before the measured region, clamped at 0. So 0 = the scenario leaked
// no goroutines; a positive number is the leak count (the team-fanout /
// background-subagents leak class). It is NOT the raw process-wide
// runtime.NumGoroutine (which would carry the test runner's own goroutines as
// noise). See kpi.GoroutineDelta.
GoroutinesEnd int `json:"goroutines_end"`
TokensInput int64 `json:"tokens_input"`
TokensOutput int64 `json:"tokens_output"`
TokensCacheRead int64 `json:"tokens_cache_read"`
CacheHitRate float64 `json:"cache_hit_rate"`
RSSPeakBytes uint64 `json:"rss_peak_bytes"`
RSSFinalBytes uint64 `json:"rss_final_bytes"`
WallClockNs int64 `json:"wall_clock_ns"`
}
ScenarioResult is one offline scenario's captured KPIs, serialised to the trend-store JSON. The deterministic, low-noise fields (allocs/op, bytes/op, goroutines, tokens, cache-hit-rate) are the gate-hard signals; the RSS and wall-clock fields are advisory (machine-dependent). Token fields are zero for scenarios with no model usage (e.g. the TUI scrollback render bench).
Field normalization — the contract the CI ingester reads (group rows by (Name, GitSHA), then aggregate the same-named samples):
- PER-OP (divided by the benchmark's b.N): AllocsPerOp, BytesPerOp. These are the amortised cost of ONE scenario iteration and are the deterministic, gate-hard signals.
- PER-RUN (the LAST iteration's cumulative figure, NOT divided): TokensInput, TokensOutput, TokensCacheRead, CacheHitRate. They are read off the final session's (or team outcome's) accumulated session.Usage — a single iteration's whole-run total, which is constant across iterations because the script is fixed. CacheHitRate == 0 is honest for scenarios with no cache-read in their script (e.g. compaction_cycle) — see perf-tracking.md.
- WHOLE-BRACKET (captured ONCE across all b.N iterations, NOT divided): RSSPeakBytes, RSSFinalBytes, WallClockNs. They describe the whole measured region, so they scale with b.N and are advisory (machine-dependent); a trend gate should compare them ratio-wise on the same runner, not absolutely.
- DELTA: GoroutinesEnd is end-minus-baseline (see its field comment) — 0 means no goroutine leak, clamped at 0.
- DISCRIMINATOR: Sample is the per-(Name) ordinal within ONE process run (0,1,2…), so -count=N emits N groupable rows per scenario rather than N indistinguishable ones.
Source Files