Documentation
¶
Overview ¶
Package summary turns the per-trial rows a benchmark run records (benchmarks/internal/result) into per-(framework, benchmark, concurrency) aggregates with trial variance, and renders the Markdown report from them. Markdown is generated from this structured summary, never hand-authored (issue #141 §9); the coefficient-of-variation flag surfaces runs whose trials disagree enough to distrust (issue §7 "coefficient-of-variation flag at >5%").
Index ¶
Constants ¶
const DefaultCoVThreshold = 0.05
DefaultCoVThreshold is the >5% trial-variance flag from issue #141 §7.
Variables ¶
This section is empty.
Functions ¶
func WriteMarkdown ¶
WriteMarkdown renders the human report from the groups — one table per benchmark, one row per (framework, concurrency). The headline numbers are the **median** across trials (issue #141's published location statistic, robust to one noisy trial); `rps CoV` is the coefficient of variation of throughput (on the mean) and ⚠ marks high-variance rows. It leads with the methodology caveats the numbers must be read with (closed-loop coordinated omission, same-host topology) so a copied table can't be read as more than it is.
Groups must be benchmark-contiguous (as Summarize returns them): the one-table-per-benchmark layout is a walk over that order, so an unsorted or framework-primary slice would split a workload across several tables.
Types ¶
type Group ¶
type Group struct {
Framework string `json:"framework"`
FrameworkVersion string `json:"framework_version"`
Runtime string `json:"runtime"`
Benchmark string `json:"benchmark"`
Concurrency int `json:"concurrency"`
Trials int `json:"trials"`
RequestsPerSecond Stats `json:"requests_per_second"`
LatencyP50 Stats `json:"latency_p50_ms"`
LatencyP95 Stats `json:"latency_p95_ms"`
LatencyP99 Stats `json:"latency_p99_ms"`
Errors int64 `json:"errors"`
CoVThreshold float64 `json:"cov_threshold"`
// HighVariance is true when the throughput CoV exceeds CoVThreshold — the
// group's trials disagree enough that the mean should be read with care.
HighVariance bool `json:"high_variance"`
}
Group aggregates every trial that shares a (Framework, Benchmark, Concurrency) key. RequestsPerSecond variance drives the CoV flag; latency percentiles are averaged across trials. Errors is the total observed (any non-zero is worth surfacing even though the runner rejects error trials — a summary of previously-collected data may still contain them).
func Summarize ¶
Summarize groups the results and computes per-group stats. covThreshold is the fraction (e.g. 0.05) above which a group's throughput CoV sets HighVariance. Groups are returned benchmark-primary — deterministic order (benchmark, framework, concurrency) — so every framework for one workload is contiguous, which is the order WriteMarkdown's one-table-per-benchmark walk requires and the comparison the report exists to show.
type Stats ¶
type Stats struct {
Median float64 `json:"median"`
Mean float64 `json:"mean"`
StdDev float64 `json:"stddev"`
CoV float64 `json:"cov"`
}
Stats is the median, mean, sample standard deviation, and coefficient of variation (stddev/mean) of one metric across a group's trials. Median is the issue's published location statistic (#141: "report at minimum the median result across trials") — it is the reported headline, so one noisy trial can't become the number someone copies; Mean/StdDev feed CoV, the stability measure, not the headline.
Source Files