Documentation
¶
Overview ¶
Package metadata collects the reproducibility metadata every full benchmark run must capture (issue #141 "Reproducibility metadata"): enough about the host, toolchain, and run parameters that a published table can be reproduced. "Never publish a table without enough metadata to reproduce it."
Index ¶
- Constants
- Variables
- func SiblingPath(dataFile string) string
- func StampUnitFile(path, group, unit string, prov Provenance) error
- func ValidGroup(name string) bool
- func WriteJSON(w io.Writer, m Metadata) error
- type Metadata
- func (m Metadata) AnyUnitDirty(group string, units []string) bool
- func (m Metadata) Provenance() Provenance
- func (m Metadata) RecordsUnits() bool
- func (m Metadata) RunParams() RunParams
- func (m Metadata) UnitProvenance(group, unit string) Provenance
- func (m Metadata) UnitsProvenance(group string, units []string) (map[string]Provenance, bool)
- func (m Metadata) WithProvenance(prov Provenance) Metadata
- type Options
- type Provenance
- type RunParams
- type Runner
Constants ¶
const ( GroupMicrobench = "microbench" GroupCRUD = "crud" GroupFootprint = "footprint" )
The measurement groups a results/latest snapshot holds. They look like one artifact but are produced by three different targets at wildly different costs — `make benchmark-micro` is minutes of pure Go, `benchmark-crud-all` is hours of containers and k6, `benchmark-footprint` is minutes of Docker — so in practice they are measured at different commits, on different days, and potentially on different hosts and toolchains. One snapshot-wide git_commit cannot describe all three without misattributing at least two of them, which is how the published framework-tax table came to advertise numbers from before the PERF-1..4 optimizations (issue #266).
const SchemaVersion = 1
SchemaVersion is bumped when the Metadata shape changes incompatibly.
Variables ¶
var KnownGroups = []string{GroupMicrobench, GroupCRUD, GroupFootprint}
KnownGroups is every valid group name. Producers validate against it so a typo'd group writes a phantom entry no reader ever looks at, instead of silently leaving the table it meant to stamp on the stale fallback.
Functions ¶
func SiblingPath ¶ added in v0.2.0
SiblingPath returns the metadata.json that belongs beside a data file, so a producer given `-out .../footprint.json` writes provenance to the snapshot it is contributing to without a second path flag to keep in sync.
func StampUnitFile ¶ added in v0.2.0
func StampUnitFile(path, group, unit string, prov Provenance) error
StampUnitFile records one unit's provenance into the metadata.json at path, preserving everything already there.
It is the single read-modify-write used by every producer that writes rows: scripts/microbench, scripts/footprint and scripts/run-crud each call it for the unit they just measured. Keeping one implementation is what makes the "a producer stamps only what it measured" invariant enforceable rather than a convention three call sites are trusted to follow.
It reads through ReadFile, so it fails closed on a snapshot it cannot read.
func ValidGroup ¶ added in v0.2.0
ValidGroup reports whether name is one of KnownGroups.
Types ¶
type Metadata ¶
type Metadata struct {
SchemaVersion int `json:"schema_version"`
Timestamp string `json:"timestamp"`
GitCommit string `json:"git_commit"`
// GitDirty is a pointer, not a bool, so the metadata can honestly say "I
// did not determine this" (null) when `git status` failed or git was
// missing — a plain bool's zero value (false = clean) would claim a
// clean tree on the exact runs where the check could not run.
GitDirty *bool `json:"git_dirty"`
OS string `json:"os"`
Kernel string `json:"kernel"`
Arch string `json:"arch"`
CPUModel string `json:"cpu_model"`
LogicalCPUs int `json:"logical_cpus"`
RAMBytes int64 `json:"ram_bytes"`
GoVersion string `json:"go_version"`
DockerVersion string `json:"docker_version"`
DockerComposeVersion string `json:"docker_compose_version"`
PostgresVersion string `json:"postgres_version"`
FrameworkVersions map[string]string `json:"framework_versions"`
RuntimeVersions map[string]string `json:"runtime_versions"`
BenchmarkTool string `json:"benchmark_tool"`
// ResourceLimits is the single scalar the standalone/host-only path records
// (an intended budget). For a multi-app run the authoritative field is
// ResourceLimitsByFramework: the *applied* verdict verified per app (from
// inspect-limits), preserved by merge so a partial/not-applied on one app is
// never overwritten by an enforced on the next. PostgresResourceLimits is the
// database container's verified verdict.
ResourceLimits string `json:"resource_limits"`
ResourceLimitsByFramework map[string]string `json:"resource_limits_by_framework"`
PostgresResourceLimits string `json:"postgres_resource_limits"`
DurationSeconds float64 `json:"duration_seconds"`
WarmupSeconds float64 `json:"warmup_seconds"`
Concurrency []int `json:"concurrency"`
Trials int `json:"trials"`
// Groups maps a group name (Group*) to the provenance of each UNIT it
// measured. A unit is that group's merge key — the thing a single run can
// replace on its own:
//
// microbench -> stack namespace (microbench.MergeStack replaces it whole)
// crud -> framework (run-crud replaces one app's rows)
// footprint -> framework:variant (footprint.Merge keys on both; see
// Footprint.ProvenanceUnit)
//
// Provenance is per unit, not per group, because every one of those files is
// merged row-wise and subset runs are supported (`APPS="gin-gorm gombit"`).
// A group-wide stamp would let a run that replaced ONE row relabel the whole
// table: six rows measured at A, `APPS=gombit make benchmark-footprint` at B,
// and five untouched rows would be captioned B. Keying provenance to the same
// unit the data merges on makes that state unrepresentable rather than merely
// detectable.
//
// A unit with no entry has no recorded provenance. Only when no unit at all
// is recorded (a pre-Groups snapshot) do readers use the top-level block
// instead; see UnitProvenance for why that fallback cannot be per unit.
// Empty ({}) rather than null so "nothing recorded" is a collected fact, not
// a dropped field.
Groups map[string]map[string]Provenance `json:"groups"`
}
Metadata is the machine-readable run environment (written to results/latest/metadata.json). Discovered fields come from the host and toolchain; the rest are run parameters the orchestrator supplies via Options because they are choices, not facts about the machine.
Which field answers "when was this measured?" ¶
Groups does, per measurement UNIT. It is the authoritative provenance: each unit records the commit, host and toolchain ITS OWN rows were produced at, and the report captions each table from the units that table publishes.
The top-level discovered fields (Timestamp, GitCommit, GitDirty, the host block, GoVersion) describe whichever collection last rewrote the whole record: every run-crud invocation (including an `APPS=` subset of one app) and every `make benchmark-metadata`. They are NOT a summary of the newest unit, and stamping a unit through StampUnitFile does not advance them. Two consequences worth stating plainly:
- The top-level block is rewritable by producers that did not measure most of the rows beside it, so it can never stand in for a unit's provenance once units are being recorded. After `make benchmark-micro` the top-level GitCommit is older than the microbench units'; after `APPS=gombit make benchmark-crud-all` it is newer than five of the six CRUD rows. Neither is a caption for any table.
- Ask Groups, never the top level, when you want a table's provenance. UnitProvenance and UnitsProvenance do this for you. They fall back to the top level only for a snapshot that records no unit at all — one written before Groups existed, where one run produced everything — and report every other unrecorded unit as unrecorded (see UnitProvenance).
The shape is additive, so SchemaVersion stays 1 and older readers keep parsing: they see the flat fields they always saw, plus a `groups` object they can ignore.
func Collect ¶
Collect gathers the metadata. Discovery is best-effort: a missing tool (docker on a CI runner without it, git outside a checkout) leaves its field empty rather than failing the whole collection — a benchmark run should still record everything it *can*.
func Merge ¶ added in v0.2.0
Merge folds an incoming whole-snapshot collection into what earlier producers already wrote to metadata.json. A snapshot is written by several producers in turn — one run-crud invocation per app — so the last writer must not erase what the earlier ones recorded: the version maps and the per-framework applied-limit verdicts are unioned, a Postgres verdict this run did not re-check is preserved, and every group's provenance survives.
A non-empty PostgresResourceLimits — including an explicit "unknown …" from a check that looked but could not classify — is authoritative for this run and overwrites, so a stale enforced/partial never sticks across a re-run whose check failed.
func ReadFile ¶ added in v0.2.0
ReadFile returns the metadata.json at path. It is the one way every producer reads an existing snapshot before adding to it.
A missing file is not an error: the first producer to run in a fresh OUT_DIR starts the record, so it returns an empty record at the current SchemaVersion. A file that exists but cannot be read or parsed IS an error — silently replacing a corrupt snapshot would discard whatever hours-long runs produced it, along with every unit's provenance.
Producers call it before writing any rows, not only when stamping afterwards: otherwise a corrupt metadata.json is discovered after the rows are already merged, leaving them on disk beside the previous run's provenance for their unit.
func ReadJSON ¶ added in v0.2.0
ReadJSON decodes a metadata.json. It is the inverse of WriteJSON, used by the producers that merge into an existing snapshot rather than replacing it.
func StampUnit ¶ added in v0.2.0
func StampUnit(meta Metadata, group, unit string, prov Provenance) Metadata
StampUnit records prov as one unit's provenance in meta and changes nothing else — not the top-level block, not the shared run parameters, not another unit's entry, not even a sibling unit of the same group.
That last restraint is the whole point. These data files merge row-wise and subset runs are supported, so a producer that re-measured one app must record exactly one app's provenance. Stamping the group instead would let it relabel rows it never touched (issue #266, review round 2).
func (Metadata) AnyUnitDirty ¶ added in v0.2.0
AnyUnitDirty reports whether any of the given units of group was measured on a dirty working tree, judged through UnitProvenance so a snapshot that records no unit is judged by its top-level block. Unknown dirtiness (nil) is not dirt.
Callers pass the units they publish, not every unit on file: a snapshot's data files also hold rows no table renders (the `gombit-ablation` ladder in microbench.json, an embedded footprint variant), and a dirty diagnostic run of those says nothing about the numbers a reader is looking at.
func (Metadata) Provenance ¶ added in v0.2.0
func (m Metadata) Provenance() Provenance
Provenance returns the top-level commit/host/toolchain block as a Provenance value — what the collection that wrote this record measured.
func (Metadata) RecordsUnits ¶ added in v0.2.0
RecordsUnits reports whether any unit of any group has recorded provenance, i.e. whether this snapshot is past the point where its top-level block could describe every row.
func (Metadata) RunParams ¶ added in v0.2.1
RunParams returns the run parameters recorded at the top level.
func (Metadata) UnitProvenance ¶ added in v0.2.0
func (m Metadata) UnitProvenance(group, unit string) Provenance
UnitProvenance returns the provenance to attribute one unit's data to, or an Empty Provenance when none was recorded for it.
The top-level block is used only for a snapshot that records no unit at all. Such a snapshot predates per-unit stamping, one run produced the whole file, and its top-level block is exactly every unit's provenance.
The fallback is deliberately NOT per unit. Once any unit is recorded, the top-level block no longer belongs to the unrecorded ones: run-crud and collect-host-info rewrite it on every invocation, including an `APPS=` subset that measured one app. Borrowing it would caption five untouched CRUD rows — and a footprint table nobody re-measured — with that one app's commit and host (issue #266). An unrecorded unit is reported as unrecorded, which is true in every order the producers can run in.
func (Metadata) UnitsProvenance ¶ added in v0.2.0
UnitsProvenance returns each unit's provenance and whether the units are comparable with one another. Callers rendering a table pass the units that table actually publishes: comparable means one honest caption covers every row, and anything else means the rows were not measured together and must be captioned individually.
func (Metadata) WithProvenance ¶ added in v0.2.0
func (m Metadata) WithProvenance(prov Provenance) Metadata
WithProvenance returns m with its top-level commit/host/toolchain block replaced by prov, leaving the run parameters and Groups untouched. It is the inverse of Provenance.
type Options ¶
type Options struct {
Now func() time.Time
Run Runner
PostgresVersion string
FrameworkVersions map[string]string
RuntimeVersions map[string]string
BenchmarkTool string
ResourceLimits string
ResourceLimitsByFramework map[string]string
PostgresResourceLimits string
DurationSeconds float64
WarmupSeconds float64
Concurrency []int
Trials int
}
Options carries the injectable seams (Now, Run) and the run-parameter fields the orchestrator knows but the host does not.
type Provenance ¶ added in v0.2.0
type Provenance struct {
Timestamp string `json:"timestamp"`
GitCommit string `json:"git_commit"`
// GitDirty is a pointer for the same reason as Metadata.GitDirty: null is
// an honest "could not determine", which a plain bool would report clean.
GitDirty *bool `json:"git_dirty"`
OS string `json:"os"`
Kernel string `json:"kernel"`
Arch string `json:"arch"`
CPUModel string `json:"cpu_model"`
LogicalCPUs int `json:"logical_cpus"`
RAMBytes int64 `json:"ram_bytes"`
GoVersion string `json:"go_version"`
}
Provenance is what one measurement group can answer about itself: which source state ran, when, on which machine, under which toolchain.
The Go toolchain is recorded here, not only at the top level, because it moves the numbers on its own: the framework-tax ladder's net/http rung — whose code is stdlib plus the harness, untouched by any framework change — reported 18 allocs/op on one toolchain and 21 on another with no source change between them. Rows are only comparable against rows produced by the same toolchain, so each group has to say which one it used.
func (Provenance) ComparableTo ¶ added in v0.2.0
func (p Provenance) ComparableTo(other Provenance) bool
ComparableTo reports whether two units' numbers may be read against each other: same source state, same machine, same toolchain.
Timestamp is deliberately excluded. One run measures its units in sequence — `make benchmark-micro` is four `go test` processes and finishes each stack a minute or two apart — so comparing timestamps would flag every ordinary run as "not measured together". A warning that fires on every run is worth nothing when the real case arrives. What makes rows comparable is the commit, the host and the toolchain; the clock is provenance to report, not a difference to act on.
GitDirty is a pointer, so it is compared by what it points at: two separately collected records of the same clean tree are the same state, not different ones because they hold different addresses.
func (Provenance) Empty ¶ added in v0.2.0
func (p Provenance) Empty() bool
Empty reports whether nothing at all was recorded — a pre-run placeholder, which a caller renders as "unknown" rather than as a row of dashes.
type RunParams ¶ added in v0.2.1
type RunParams struct {
Concurrency []int
Trials int
DurationSeconds float64
WarmupSeconds float64
BenchmarkTool string
}
RunParams is the sweep protocol and load generator a snapshot records ONCE, at the top level, for every row in it.
Unlike the version maps and the per-framework limit verdicts, Merge cannot union these: metadata.json holds exactly one protocol, and the report prints exactly one "Protocol:" / "Load generator:" line (report.writeMethodology) and judges one "reduced development snapshot" banner from it (report.reducedFrom). Last-writer-wins is therefore only honest while every row in the file was measured under the incoming parameters.
That is what makes them different in kind from provenance. Provenance became per unit (Groups) because a run measures one unit and must not caption the others (issue #266). These cannot follow the same boundary without the report growing a per-unit protocol block, so run-crud holds the invariant from the producer side instead: it refuses to merge a run whose parameters differ from the recorded ones while rows it does not replace remain (#361, review round 1).
ConflictsWith is directional (see there): an incoming zero is a value, never a wildcard, because Merge writes it (#361, review round 2).
That covers run-crud only. `make benchmark-metadata` (collect-host-info) also writes these fields, replaces them wholesale, and measures nothing, so it can still describe existing rows under other parameters; it is not guarded.
func (RunParams) ConflictsWith ¶ added in v0.2.1
ConflictsWith describes every parameter the incoming run would change in a recorded snapshot, as "<what> <recorded> -> <incoming>" in a fixed order so an error message is stable. Concurrency compares as an ordered list, the way the report prints it. No conflict returns nil.
It is directional because Merge is: the incoming values are written whole, so every incoming field is a value, and a zero there (a zero-second warm-up) is compared like any other rather than read as "unstated". Callers must hand it complete incoming parameters; run-crud validates them before measuring.
Only the recorded side can be absent, and only as a whole: a record that states nothing (a fresh OUT_DIR, a snapshot written before these fields were recorded) has no protocol to misdescribe, so nothing conflicts with it. Once it states anything, all five fields are compared exactly. A partial record, such as `make benchmark-metadata` given only -trials, therefore conflicts on its zeros; that fails closed rather than guessing what those rows ran.
func (RunParams) Recorded ¶ added in v0.2.1
Recorded reports whether the record states any run parameter at all. Every writer of these fields (run-crud, collect-host-info) writes all five together, so presence is judged for the record, not per field: a record that states anything states a value for each field, zero included.
Source Files