Documentation
¶
Overview ¶
Package benchmark implements the first-class benchmark record type (P-01KYJMVX2Q): entries compare implementations against each other on a FRAME SYSTEM, keyed uniquely by canonicalized name+frame, with bounded per-key version history (default 1 — the latest is what the codebase cares about; benchmarks.history raises it).
The store is a keyed last-writer-wins map serialized as ndjson (.spectackle/bench.ndjson per context, union-merged like the journal), NOT an append-only log: a benchmark is re-measured constantly and its identity is its key, never its position.
Index ¶
- Variables
- func CanonicalFrame(frame map[string]string) map[string]string
- func CanonicalKey(name string, frame map[string]string) (string, error)
- func SameContent(a, b Record) bool
- type Impl
- type Metric
- type Record
- type Store
- func (st *Store) ByID(id string) (Record, bool)
- func (st *Store) Head(key string) (Record, bool)
- func (st *Store) Keys() []string
- func (st *Store) Put(rec Record, depth int) (stored Record, prev *Record, changed bool, err error)
- func (st *Store) Remove(key string) int
- func (st *Store) Save(path string) error
- func (st *Store) Versions(key string) []Record
Constants ¶
This section is empty.
Variables ¶
var RequiredDims = []string{"os", "arch", "cpu", "ram", "gpu"}
RequiredDims are the frame keys every record must carry (user requirement: os/arch/cpu/ram/gpu at minimum). Values may be real host facts, or the sentinels: "none" (hardware genuinely absent — no GPU) and "any" (dimension irrelevant to this benchmark — machine-independent measurements share one key across hosts, ADR-01KYJMWE1N).
Functions ¶
func CanonicalFrame ¶
CanonicalFrame returns the folded copy of a frame, the form Records store — key derivation and storage must agree byte-for-byte.
func CanonicalKey ¶
CanonicalKey derives the unique key from a benchmark name and its frame: folded name, then the folded dims as k=v joined sorted by key with "|". Deterministic across insertion order; every RequiredDim must be present (sentinels are legal values); forbidden separator characters refuse.
func SameContent ¶
SameContent reports whether two records measure identically — the idempotent-replay check: key, metrics, impls and values equal (ID, Ver, T, Ag, Tool, Note excluded — provenance may differ on a re-put).
Types ¶
type Impl ¶
type Impl struct {
Label string `json:"l"` // "go", "python", "cuda-v2"
Src string `json:"src,omitempty"` // provenance: commit, package, variant note
Res map[string]float64 `json:"res"` // metric name -> value
}
Impl is one implementation's measured results within a record.
type Metric ¶
type Metric struct {
Name string `json:"n"` // compare join key; folded [a-z0-9._/-]+
// Unit is free-form ("ns/op", "ops/s", "B", "token/s") and is
// byte-compared on cmp — NEVER converted or summed across units.
Unit string `json:"u"`
// Dir is the comparability direction: "+" higher is better, "-" lower
// is better, "~" diagnostic (no better/worse verdicts).
Dir string `json:"d"`
// Noise is the absolute jitter floor: deltas within ±Noise render "~"
// (tie) instead of a direction verdict. 0 = every delta counts.
Noise float64 `json:"nz,omitempty"`
}
Metric declares one measured quantity, shared by every Impl in a record.
type Record ¶
type Record struct {
ID string `json:"id"` // "M-" + record ID, minted fresh per version
Name string `json:"name"` // verbatim benchmark name (folded only inside Key)
Key string `json:"key"` // CanonicalKey(Name, Frame) — stored AND verified at load
Ver int `json:"ver"` // 1-based, monotonic per Key
Frame map[string]string `json:"frame"`
Metrics []Metric `json:"metrics"` // declaration order = render order
Impls []Impl `json:"impls"` // order = presentation order
T time.Time `json:"t"`
Ag string `json:"ag,omitempty"` // producing agent
Tool string `json:"tool,omitempty"` // producing harness/fixture
Note string `json:"note,omitempty"`
Dir string `json:"-"` // context dir, backfilled at load
}
Record is one retained version of one benchmark entry.
type Store ¶
type Store struct {
// Quarantined lines: content whose stored key failed verification,
// whose JSON did not parse, whose ID duplicates a live record with
// DIFFERENT content, or which lost a (key, ver) collision while
// measuring something different from the winner. Never silently
// dropped — they are reported and preserved byte-identically on
// rewrite (union merges and hand edits must not lose data to a strict
// reader; B-01KYJTASR5EKW).
Quarantine []string
// contains filtered or unexported fields
}
Store is the loaded bench.ndjson of one context: records grouped by key, each key's retained versions sorted ascending by Ver.
func Load ¶
Load reads a bench.ndjson. A missing file is an empty store. Duplicate IDs (union-merge artifacts) dedup to one; a (key, ver) collision between DIFFERENT records resolves deterministically: newer T wins, ties break by ID — both clones converge on the same winner.
func (*Store) Put ¶
Put inserts rec: Ver is ASSIGNED here (head+1, or 1 for a new key). Identical content to the head is an idempotent replay — nothing is written and the head returns with changed=false. The superseded head (if any) returns so the caller can journal its raw values (ADR-01KYJMWEWQ). History trims to depth newest versions.
Source Files