Documentation
¶
Overview ¶
Package metadata collects the reproducibility metadata every full benchmark run must capture (issue #141 "Reproducibility metadata"): enough about the host, toolchain, and run parameters that a published table can be reproduced. "Never publish a table without enough metadata to reproduce it."
Index ¶
- Constants
- Variables
- func CheckHostClassEnv() error
- func ParseHostClass(s string) (string, error)
- func SiblingPath(dataFile string) string
- func StampUnitFile(path, group, unit string, prov Provenance) error
- func ValidGroup(name string) bool
- func WriteJSON(w io.Writer, m Metadata) error
- type Metadata
- func (m Metadata) AnyUnitDirty(group string, units []string) bool
- func (m Metadata) NonDedicatedUnits(group string, units []string) []string
- func (m Metadata) Provenance() Provenance
- func (m Metadata) RecordsUnits() bool
- func (m Metadata) RunParams() RunParams
- func (m Metadata) UnitProvenance(group, unit string) Provenance
- func (m Metadata) UnitRunParams(group, unit string) (RunParams, bool)
- func (m Metadata) UnitsProvenance(group string, units []string) (map[string]Provenance, bool)
- func (m Metadata) WithLegacyProtocols() Metadata
- func (m Metadata) WithProvenance(prov Provenance) Metadata
- func (m Metadata) WithRunParams(p RunParams) Metadata
- type Options
- type Provenance
- type RunParams
- type Runner
Constants ¶
const ( GroupMicrobench = "microbench" GroupCRUD = "crud" GroupFootprint = "footprint" )
The measurement groups a results/latest snapshot holds. They look like one artifact but are produced by three different targets at wildly different costs — `make benchmark-micro` is minutes of pure Go, `benchmark-crud-all` is hours of containers and k6, `benchmark-footprint` is minutes of Docker — so in practice they are measured at different commits, on different days, and potentially on different hosts and toolchains. One snapshot-wide git_commit cannot describe all three without misattributing at least two of them, which is how the published framework-tax table came to advertise numbers from before the PERF-1..4 optimizations (issue #266).
const ( HostClassDedicated = "dedicated" HostClassDeveloper = "developer" )
The host classes an operator may declare. Leaving HostClassEnv unset is the third valid choice and records nothing; any other value is refused by ParseHostClass.
const HostClassEnv = "BENCHMARK_HOST_CLASS"
HostClassEnv is the environment variable through which the operator declares what kind of host a run measured on. Collect records it; the report treats only HostClassDedicated as the canonical, publishable kind (issue #291).
It is a declaration, not a measurement, on purpose: "dedicated benchmark hardware" cannot be read from /proc. A heuristic (load average, core count, throttling) would both miss contended hosts and flag quiet ones, and a wrong banner is worse than none. So the safe default is the absence of a claim: a unit that does not declare HostClassDedicated is reported as not measured on dedicated hardware, and silence is never read as canonicity.
const SchemaVersion = 1
SchemaVersion is bumped when the Metadata shape changes incompatibly.
Variables ¶
var HostClasses = []string{"", HostClassDedicated, HostClassDeveloper}
HostClasses is every accepted HostClassEnv value, "" (unset) included. The Makefile and the orchestration scripts check the same set before measuring, from benchmarks/scripts/host-class.sh; a test keeps the two equal.
var KnownGroups = []string{GroupMicrobench, GroupCRUD, GroupFootprint}
KnownGroups is every valid group name. Producers validate against it so a typo'd group writes a phantom entry no reader ever looks at, instead of silently leaving the table it meant to stamp on the stale fallback.
Functions ¶
func CheckHostClassEnv ¶ added in v0.7.0
func CheckHostClassEnv() error
CheckHostClassEnv validates the process's HostClassEnv. Every producer calls it before writing anything: the declaration only matters once the report renders, and a typo found there would cost the whole run. Where a producer only records what a script already measured (footprint, microbench), the Makefile target and the script check first (host-class.sh).
func ParseHostClass ¶ added in v0.7.0
ParseHostClass validates a HostClassEnv declaration against HostClasses. The match is exact, so `Dedicated` or a typo is an error rather than a silently non-dedicated run.
func SiblingPath ¶ added in v0.2.0
SiblingPath returns the metadata.json that belongs beside a data file, so a producer given `-out .../footprint.json` writes provenance to the snapshot it is contributing to without a second path flag to keep in sync.
func StampUnitFile ¶ added in v0.2.0
func StampUnitFile(path, group, unit string, prov Provenance) error
StampUnitFile records one unit's provenance into the metadata.json at path, preserving everything already there.
It is the single read-modify-write used by every producer that writes rows: scripts/microbench, scripts/footprint and scripts/run-crud each call it for the unit they just measured. Keeping one implementation is what makes the "a producer stamps only what it measured" invariant enforceable rather than a convention three call sites are trusted to follow.
It reads through ReadFile, so it fails closed on a snapshot it cannot read.
func ValidGroup ¶ added in v0.2.0
ValidGroup reports whether name is one of KnownGroups.
Types ¶
type Metadata ¶
type Metadata struct {
SchemaVersion int `json:"schema_version"`
Timestamp string `json:"timestamp"`
GitCommit string `json:"git_commit"`
// GitDirty is a pointer, not a bool, so the metadata can honestly say "I
// did not determine this" (null) when `git status` failed or git was
// missing — a plain bool's zero value (false = clean) would claim a
// clean tree on the exact runs where the check could not run.
GitDirty *bool `json:"git_dirty"`
OS string `json:"os"`
Kernel string `json:"kernel"`
Arch string `json:"arch"`
CPUModel string `json:"cpu_model"`
LogicalCPUs int `json:"logical_cpus"`
RAMBytes int64 `json:"ram_bytes"`
HostClass string `json:"host_class"`
GoVersion string `json:"go_version"`
DockerVersion string `json:"docker_version"`
DockerComposeVersion string `json:"docker_compose_version"`
PostgresVersion string `json:"postgres_version"`
FrameworkVersions map[string]string `json:"framework_versions"`
RuntimeVersions map[string]string `json:"runtime_versions"`
BenchmarkTool string `json:"benchmark_tool"`
// ResourceLimits is the single scalar the standalone/host-only path records
// (an intended budget). For a multi-app run the authoritative field is
// ResourceLimitsByFramework: the *applied* verdict verified per app (from
// inspect-limits), preserved by merge so a partial/not-applied on one app is
// never overwritten by an enforced on the next. PostgresResourceLimits is the
// database container's verified verdict.
ResourceLimits string `json:"resource_limits"`
ResourceLimitsByFramework map[string]string `json:"resource_limits_by_framework"`
PostgresResourceLimits string `json:"postgres_resource_limits"`
DurationSeconds float64 `json:"duration_seconds"`
WarmupSeconds float64 `json:"warmup_seconds"`
Concurrency []int `json:"concurrency"`
Trials int `json:"trials"`
// Groups maps a group name (Group*) to the provenance of each UNIT it
// measured. A unit is that group's merge key — the thing a single run can
// replace on its own:
//
// microbench -> stack namespace (microbench.MergeStack replaces it whole)
// crud -> framework (run-crud replaces one app's rows)
// footprint -> framework:variant (footprint.Merge keys on both; see
// Footprint.ProvenanceUnit)
//
// Provenance is per unit, not per group, because every one of those files is
// merged row-wise and subset runs are supported (`APPS="gin-gorm gombit"`).
// A group-wide stamp would let a run that replaced ONE row relabel the whole
// table: six rows measured at A, `APPS=gombit make benchmark-footprint` at B,
// and five untouched rows would be captioned B. Keying provenance to the same
// unit the data merges on makes that state unrepresentable rather than merely
// detectable.
//
// A unit with no entry has no recorded provenance. Only when no unit at all
// is recorded (a pre-Groups snapshot) do readers use the top-level block
// instead; see UnitProvenance for why that fallback cannot be per unit.
// Empty ({}) rather than null so "nothing recorded" is a collected fact, not
// a dropped field.
Groups map[string]map[string]Provenance `json:"groups"`
}
Metadata is the machine-readable run environment (written to results/latest/metadata.json). Discovered fields come from the host and toolchain; the rest are run parameters the orchestrator supplies via Options because they are choices, not facts about the machine.
Which field answers "when was this measured?" ¶
Groups does, per measurement UNIT. It is the authoritative provenance: each unit records the commit, host and toolchain ITS OWN rows were produced at, and the report captions each table from the units that table publishes.
The top-level discovered fields (Timestamp, GitCommit, GitDirty, the host block, GoVersion) describe whichever collection last rewrote the whole record: every run-crud invocation (including an `APPS=` subset of one app) and every `make benchmark-metadata`. They are NOT a summary of the newest unit, and stamping a unit through StampUnitFile does not advance them. Two consequences worth stating plainly:
- The top-level block is rewritable by producers that did not measure most of the rows beside it, so it can never stand in for a unit's provenance once units are being recorded. After `make benchmark-micro` the top-level GitCommit is older than the microbench units'; after `APPS=gombit make benchmark-crud-all` it is newer than five of the six CRUD rows. Neither is a caption for any table.
- Ask Groups, never the top level, when you want a table's provenance. UnitProvenance and UnitsProvenance do this for you. They fall back to the top level only for a snapshot that records no unit at all — one written before Groups existed, where one run produced everything — and report every other unrecorded unit as unrecorded (see UnitProvenance).
The shape is additive, so SchemaVersion stays 1 and older readers keep parsing: they see the flat fields they always saw, plus a `groups` object they can ignore.
func Collect ¶
Collect gathers the metadata. Discovery is best-effort: a missing tool (docker on a CI runner without it, git outside a checkout) leaves its field empty rather than failing the whole collection — a benchmark run should still record everything it *can*.
func Merge ¶ added in v0.2.0
Merge folds an incoming whole-snapshot collection into what earlier producers already wrote to metadata.json. A snapshot is written by several producers in turn — one run-crud invocation per app — so the last writer must not erase what the earlier ones recorded: the version maps and the per-framework applied-limit verdicts are unioned, a Postgres verdict this run did not re-check is preserved, and every group's provenance survives.
A non-empty PostgresResourceLimits — including an explicit "unknown …" from a check that looked but could not classify — is authoritative for this run and overwrites, so a stale enforced/partial never sticks across a re-run whose check failed.
func ReadFile ¶ added in v0.2.0
ReadFile returns the metadata.json at path. It is the one way every producer reads an existing snapshot before adding to it.
A missing file is not an error: the first producer to run in a fresh OUT_DIR starts the record, so it returns an empty record at the current SchemaVersion. A file that exists but cannot be read or parsed IS an error — silently replacing a corrupt snapshot would discard whatever hours-long runs produced it, along with every unit's provenance.
Producers call it before writing any rows, not only when stamping afterwards: otherwise a corrupt metadata.json is discovered after the rows are already merged, leaving them on disk beside the previous run's provenance for their unit.
func ReadJSON ¶ added in v0.2.0
ReadJSON decodes a metadata.json. It is the inverse of WriteJSON, used by the producers that merge into an existing snapshot rather than replacing it.
A CRUD unit recorded before protocols were per unit is read with the top-level protocol filed on it (Metadata.WithLegacyProtocols), so every producer sees the same answer the report does, and records it before rewriting the top level.
func StampUnit ¶ added in v0.2.0
func StampUnit(meta Metadata, group, unit string, prov Provenance) Metadata
StampUnit records prov as one unit's provenance in meta and changes nothing else — not the top-level block, not the shared run parameters, not another unit's entry, not even a sibling unit of the same group.
That last restraint is the whole point. These data files merge row-wise and subset runs are supported, so a producer that re-measured one app must record exactly one app's provenance. Stamping the group instead would let it relabel rows it never touched (issue #266, review round 2).
func (Metadata) AnyUnitDirty ¶ added in v0.2.0
AnyUnitDirty reports whether any of the given units of group was measured on a dirty working tree, judged through UnitProvenance so a snapshot that records no unit is judged by its top-level block. Unknown dirtiness (nil) is not dirt.
Callers pass the units they publish, not every unit on file: a snapshot's data files also hold rows no table renders (the `gombit-ablation` ladder in microbench.json, an embedded footprint variant), and a dirty diagnostic run of those says nothing about the numbers a reader is looking at.
func (Metadata) NonDedicatedUnits ¶ added in v0.7.0
NonDedicatedUnits returns the given units of group that were not declared to be measured on dedicated hardware, judged through UnitProvenance like AnyUnitDirty and passed the published units for the same reason. Unlike unknown dirtiness, an undeclared host class counts: the guard exists so a snapshot cannot read as canonical by saying nothing, and a unit with no recorded provenance at all is no more canonical than one declared otherwise.
func (Metadata) Provenance ¶ added in v0.2.0
func (m Metadata) Provenance() Provenance
Provenance returns the top-level commit/host/toolchain block as a Provenance value — what the collection that wrote this record measured.
func (Metadata) RecordsUnits ¶ added in v0.2.0
RecordsUnits reports whether any unit of any group has recorded provenance, i.e. whether this snapshot is past the point where its top-level block could describe every row.
func (Metadata) RunParams ¶ added in v0.2.1
RunParams returns the run parameters recorded at the top level.
func (Metadata) UnitProvenance ¶ added in v0.2.0
func (m Metadata) UnitProvenance(group, unit string) Provenance
UnitProvenance returns the provenance to attribute one unit's data to, or an Empty Provenance when none was recorded for it.
The top-level block is used only for a snapshot that records no unit at all. Such a snapshot predates per-unit stamping, one run produced the whole file, and its top-level block is exactly every unit's provenance.
The fallback is deliberately NOT per unit. Once any unit is recorded, the top-level block no longer belongs to the unrecorded ones: run-crud and collect-host-info rewrite it on every invocation, including an `APPS=` subset that measured one app. Borrowing it would caption five untouched CRUD rows — and a footprint table nobody re-measured — with that one app's commit and host (issue #266). An unrecorded unit is reported as unrecorded, which is true in every order the producers can run in.
func (Metadata) UnitRunParams ¶ added in v0.7.0
UnitRunParams returns the protocol one unit's rows were measured under, and false when none was recorded for it. It follows UnitProvenance's rule: the top-level parameters stand in only for a snapshot that records no unit at all, where one run produced every row. Once any unit is recorded, the top level is rewritten by producers that did not measure most rows (run-crud for another unit, `make benchmark-metadata` for none), so an unrecorded unit is reported as unrecorded.
func (Metadata) UnitsProvenance ¶ added in v0.2.0
UnitsProvenance returns each unit's provenance and whether the units are comparable with one another. Callers rendering a table pass the units that table actually publishes: comparable means one honest caption covers every row, and anything else means the rows were not measured together and must be captioned individually.
func (Metadata) WithLegacyProtocols ¶ added in v0.7.0
WithLegacyProtocols returns m with the top-level run parameters filed on every CRUD unit that has no protocol of its own.
Before protocols were per unit (#377), the top-level parameters were the only protocol a CRUD unit had, and the README described every CRUD row with them. That is not a guarantee they ran under it: per-unit provenance (#292) predates the run-crud merge guard (#371), and collect-host-info was never guarded. It is the best record there is, and the one already published, so it is kept rather than dropped: ReadJSON applies this on every read and the report on every render, so producers and the report agree on it, and a producer records it per unit before it rewrites the top level. Only CRUD: the top-level protocol never described microbench or footprint rows.
When the top level records no parameters, there is nothing to attribute, and the unit gets an explicit empty protocol instead: "not recorded" is then a fact on file, and parameters a later `make benchmark-metadata` writes at the top level are never read as what this unit ran (#377 review).
func (Metadata) WithProvenance ¶ added in v0.2.0
func (m Metadata) WithProvenance(prov Provenance) Metadata
WithProvenance returns m with its top-level commit/host/toolchain block replaced by prov, leaving the run parameters and Groups untouched. It is the inverse of Provenance.
func (Metadata) WithRunParams ¶ added in v0.7.0
WithRunParams returns m with its top-level run parameters replaced by p. It is the inverse of RunParams.
type Options ¶
type Options struct {
Now func() time.Time
Run Runner
// Getenv reads HostClassEnv; nil uses os.Getenv. Collect has no error to
// return, so an invalid value is recorded as no declaration (not
// dedicated); producers refuse it up front with CheckHostClassEnv.
Getenv func(string) string
PostgresVersion string
FrameworkVersions map[string]string
RuntimeVersions map[string]string
BenchmarkTool string
ResourceLimits string
ResourceLimitsByFramework map[string]string
PostgresResourceLimits string
DurationSeconds float64
WarmupSeconds float64
Concurrency []int
Trials int
}
Options carries the injectable seams (Now, Run) and the run-parameter fields the orchestrator knows but the host does not.
type Provenance ¶ added in v0.2.0
type Provenance struct {
Timestamp string `json:"timestamp"`
GitCommit string `json:"git_commit"`
// GitDirty is a pointer for the same reason as Metadata.GitDirty: null is
// an honest "could not determine", which a plain bool would report clean.
GitDirty *bool `json:"git_dirty"`
OS string `json:"os"`
Kernel string `json:"kernel"`
Arch string `json:"arch"`
CPUModel string `json:"cpu_model"`
LogicalCPUs int `json:"logical_cpus"`
RAMBytes int64 `json:"ram_bytes"`
// HostClass is the operator's HostClassEnv declaration at collection time;
// empty means none was made.
HostClass string `json:"host_class"`
GoVersion string `json:"go_version"`
// Protocol is the sweep protocol and load generator this unit's rows were
// measured under, recorded on the same boundary as the rows (#377). Only CRUD
// units have one; a group with no protocol (microbench, footprint) and a unit
// stamped before protocols were per unit leave it nil, which the JSON omits.
// A pointer keeps Provenance a comparable value and makes "not recorded"
// distinct from a recorded zero. An empty object ({}) is an explicit "not
// recorded" filed by WithLegacyProtocols, so a top level written later is
// never mistaken for what this unit ran.
Protocol *RunParams `json:"protocol,omitempty"`
}
Provenance is what one measurement group can answer about itself: which source state ran, when, on which machine, under which toolchain.
The Go toolchain is recorded here, not only at the top level, because it moves the numbers on its own: the framework-tax ladder's net/http rung — whose code is stdlib plus the harness, untouched by any framework change — reported 18 allocs/op on one toolchain and 21 on another with no source change between them. Rows are only comparable against rows produced by the same toolchain, so each group has to say which one it used.
func (Provenance) ComparableTo ¶ added in v0.2.0
func (p Provenance) ComparableTo(other Provenance) bool
ComparableTo reports whether two units' numbers may be read against each other: same source state, same machine, same toolchain.
Timestamp is deliberately excluded. One run measures its units in sequence — `make benchmark-micro` is four `go test` processes and finishes each stack a minute or two apart — so comparing timestamps would flag every ordinary run as "not measured together". A warning that fires on every run is worth nothing when the real case arrives. What makes rows comparable is the commit, the host and the toolchain; the clock is provenance to report, not a difference to act on.
HostClass is excluded too. It is the operator's declaration about the machine, not a fact that moves the numbers; the machine itself is compared field by field. Units whose declarations differ are judged by the report's host-class banner, and a caption claiming they "mix source states" would be false when commit, host and toolchain all agree.
GitDirty is a pointer, so it is compared by what it points at: two separately collected records of the same clean tree are the same state, not different ones because they hold different addresses.
Protocol is excluded for the same reason as HostClass: the caption this answers is "same source, machine and toolchain". Which protocol each unit ran is reported separately, per unit, by the methodology block and the reduced-snapshot banner.
func (Provenance) Dedicated ¶ added in v0.7.0
func (p Provenance) Dedicated() bool
Dedicated reports whether the unit was declared to be measured on dedicated benchmark hardware. An empty or unrecognised class is not dedicated.
func (Provenance) Empty ¶ added in v0.2.0
func (p Provenance) Empty() bool
Empty reports whether nothing at all was recorded — a pre-run placeholder, which a caller renders as "unknown" rather than as a row of dashes.
type RunParams ¶ added in v0.2.1
type RunParams struct {
Concurrency []int `json:"concurrency,omitempty"`
Trials int `json:"trials,omitempty"`
DurationSeconds float64 `json:"duration_seconds,omitempty"`
WarmupSeconds float64 `json:"warmup_seconds,omitempty"`
BenchmarkTool string `json:"benchmark_tool,omitempty"`
}
RunParams is the sweep protocol and load generator a CRUD run measured under.
The authoritative record is per unit, in each CRUD unit's Provenance.Protocol, because rows are per (framework, benchmark) and two workloads, or two apps, may legitimately run under different pins (#377). The report describes and judges each table by the protocols of the units it renders (UnitRunParams).
The top-level fields (Metadata.Concurrency and friends) are still written for older readers. Like the top-level provenance block they describe whichever collection last rewrote the record, run-crud or `make benchmark-metadata`, and no reader may use them to describe a unit once any unit is recorded.
Every field is omitempty so the explicit "not recorded" marker serializes as {}. A recorded protocol always states trials, duration, concurrency and tool (run-crud validates them), so only a zero warm-up is ever omitted, and it reads back as the same zero.
func (RunParams) Equal ¶ added in v0.7.0
Equal reports whether two protocols are the same sweep with the same load generator. Concurrency compares as an ordered list, the way it is printed.
func (RunParams) Recorded ¶ added in v0.2.1
Recorded reports whether the record states any run parameter at all. Every writer of these fields (run-crud, collect-host-info) writes all five together, so presence is judged for the record, not per field: a record that states anything states a value for each field, zero included.
Source Files