report

package
v1.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 7, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Overview

Package report turns run metrics into a verdict, per-journey and per-step statistics, a timeline and exportable documents.

Index

Constants

View Source
const (
	CmpRegression   = "regression"
	CmpImprovement  = "improvement"
	CmpNoChange     = "no-change"
	CmpInconclusive = "inconclusive"
)

Comparison verdicts.

View Source
const (
	VerdictPass           = "pass"
	VerdictFail           = "fail"
	VerdictGeneratorLimit = "generator-limited"
	VerdictNoTargets      = "no-targets"
)

Verdicts.

Variables

This section is empty.

Functions

func AllPass

func AllPass(cs []Check) bool

AllPass reports whether every check passed.

func Bytes

func Bytes(n uint64) string

Bytes formats a byte count.

func CLS

func CLS(f float64) string

CLS formats a cumulative layout shift score.

func MetricValue

func MetricValue(f float64) string

MetricValue formats a target metric value compactly: 0.0123, 12.3, 4.56k, 120M.

func Ms

func Ms(s float64) string

Ms formats seconds as milliseconds (or seconds above 10s).

func Observe

func Observe(p *scenario.Program, scope, metric string, total *metrics.Snapshot, dur float64) float64

Observe computes one metric for a scope ("http", a journey, or "journey/step"). It returns NaN when the scope has no data.

func Pct

func Pct(f float64) string

Pct formats a fraction as a percentage.

func Unit

func Unit(mode string) string

Unit returns the load unit for the plan mode.

func VerdictLabel

func VerdictLabel(v string) string

VerdictLabel is a human label for a verdict.

Types

type Breakpoint

type Breakpoint struct {
	// Found is false when every level passed.
	Found bool `json:"found"`
	// LastPass is the highest planned load that met every target.
	LastPass float64 `json:"lastPass"`
	// FirstFail is the lowest planned load that missed a target.
	FirstFail float64  `json:"firstFail,omitempty"`
	Unit      string   `json:"unit"`
	FailedOn  []string `json:"failedOn,omitempty"`
	// Refined lists the confirmation holds that narrowed LastPass and
	// FirstFail after the first failing step; the breakpoint lies
	// between the two.
	Refined []RefineStep `json:"refined,omitempty"`
}

Breakpoint is the outcome of a breakpoint search.

type BrowserStat

type BrowserStat struct {
	TTFB *PhaseStat `json:"ttfb,omitempty"`
	FCP  *PhaseStat `json:"fcp,omitempty"`
	LCP  *PhaseStat `json:"lcp,omitempty"`
	CLS  *PhaseStat `json:"cls,omitempty"`
	INP  *PhaseStat `json:"inp,omitempty"`
	Load *PhaseStat `json:"load,omitempty"`
}

BrowserStat holds a browser step's page timings and Web Vitals: mean and p95 in seconds, except CLS, which is unitless. Page loads have TTFB, FCP, LCP, CLS and Load; clicks and key presses have INP.

type Check

type Check struct {
	Source   string  `json:"source"`
	Scope    string  `json:"scope"`
	Metric   string  `json:"metric"`
	Op       string  `json:"op"`
	Target   float64 `json:"target"`
	Observed float64 `json:"observed"`
	Pass     bool    `json:"pass"`
	// Display strings in the metric's unit.
	TargetText   string `json:"targetText"`
	ObservedText string `json:"observedText"`
}

Check is the outcome of one target.

func Evaluate

func Evaluate(p *scenario.Program, ths []scenario.Threshold, total *metrics.Snapshot, dur float64) []Check

Evaluate checks each threshold against merged metrics covering dur seconds.

type Claim

type Claim struct {
	Text  string   `json:"text"`
	Label string   `json:"label"` // measured or suspected
	Refs  []string `json:"refs"`  // fact ids, see Facts
}

Claim is one statement in a narrative.

type Comparison

type Comparison struct {
	A          Side          `json:"a"`
	B          Side          `json:"b"`
	Comparable bool          `json:"comparable"`
	Problems   []string      `json:"problems,omitempty"`
	Metrics    []MetricDelta `json:"metrics"`
	// Steps compares each request step's p95 and error rate. They are
	// informational: with many steps some differ by chance, so they do
	// not change the verdict.
	Steps      []StepDelta `json:"steps,omitempty"`
	Verdict    string      `json:"verdict"` // regression, improvement, no-change, inconclusive
	Confidence float64     `json:"confidence"`
}

Comparison judges whether version B differs from version A, using repeated runs of each. A single run is noisy, so a change is only called a regression or improvement when the bootstrap confidence interval of the difference excludes zero and the change is larger than the noise floor measured from the repeats themselves.

func Compare

func Compare(a, b []*Report, labelA, labelB string) *Comparison

Compare compares runs of version A with runs of version B.

func (*Comparison) WriteMarkdown

func (c *Comparison) WriteMarkdown(w io.Writer)

WriteMarkdown writes the comparison for a pull request comment.

func (*Comparison) WriteText

func (c *Comparison) WriteText(w io.Writer)

WriteText writes the comparison as a terminal table.

type CurvePoint

type CurvePoint struct {
	// Offered is the planned load: users, or iterations per second.
	Offered float64 `json:"offered"`
	// Throughput is completed iterations per second.
	Throughput float64 `json:"throughput"`
	RPS        float64 `json:"rps"`
	P50        float64 `json:"p50"`
	P95        float64 `json:"p95"`
	P99        float64 `json:"p99"`
	ErrorRate  float64 `json:"errorRate"`
	Seconds    int     `json:"seconds"`
}

CurvePoint is one load level of a run whose load changed over time.

type ErrorRow

type ErrorRow struct {
	Journey string `json:"journey"`
	Step    string `json:"step"`
	Error   string `json:"error"`
	Count   uint64 `json:"count"`
	// Examples are up to three of these failures, with what was sent and
	// received (credentials and secrets redacted).
	Examples []metrics.ErrorExample `json:"examples,omitempty"`
}

ErrorRow counts one kind of failure at one step.

type Fact

type Fact struct {
	ID    string `json:"id"`
	Text  string `json:"text"`
	Where string `json:"where"` // the report section that shows it
}

Fact is one figure from a report, with a stable id a narrative cites.

type FaultEvent

type FaultEvent struct {
	Label  string  `json:"label"`
	Kind   string  `json:"kind"` // proxy, container or deployment
	Target string  `json:"target"`
	Start  float64 `json:"start"`
	End    float64 `json:"end"`
	// Error says why the fault could not be applied or reverted.
	Error string `json:"error,omitempty"`
}

FaultEvent is a fault injected during the run. Start and End are seconds since the run started, like Point.T.

type FirstEventStat

type FirstEventStat struct {
	Mean float64 `json:"mean"`
	P50  float64 `json:"p50"`
	P95  float64 `json:"p95"`
	P99  float64 `json:"p99"`
}

FirstEventStat is a time-to-first-event summary in seconds.

type Input

type Input struct {
	RunID      string
	Program    *scenario.Program
	Plan       *scenario.Plan
	Target     string
	Started    time.Time
	Ended      time.Time
	StopReason string
	PeakVUs    int
	Workers    int
	Interval   time.Duration
	Snapshots  []*metrics.Snapshot // already merged per interval, any order
	Phases     map[int]*[metrics.NumPhases]*metrics.Histogram
	Breakpoint *Breakpoint
}

Input is everything needed to build a report.

type Journey

type Journey struct {
	Name  string `json:"name"`
	Stats Stats  `json:"stats"`
	Steps []Step `json:"steps"`
}

Journey is per-journey output.

type Knee

type Knee struct {
	Found bool `json:"found"`
	// At is the last level that still scaled; Next is the first that did not.
	At     CurvePoint  `json:"at"`
	Next   *CurvePoint `json:"next,omitempty"`
	Unit   string      `json:"unit"`
	Reason string      `json:"reason,omitempty"`
}

Knee marks where adding load stops adding throughput.

type LoadInfo

type LoadInfo struct {
	Shape    string  `json:"shape,omitempty"`
	Mode     string  `json:"mode"`
	Executor string  `json:"executor"`
	Peak     float64 `json:"peak"`
	PeakVUs  int     `json:"peakVUs"`
	Workers  int     `json:"workers"`
}

LoadInfo summarises the plan.

type MetricDelta

type MetricDelta struct {
	Name string `json:"name"`
	// HigherIsBetter is true for throughput-like metrics.
	HigherIsBetter bool      `json:"higherIsBetter"`
	A              []float64 `json:"a"`
	B              []float64 `json:"b"`
	MeanA          float64   `json:"meanA"`
	MeanB          float64   `json:"meanB"`
	// Change is (meanB - meanA) / meanA.
	Change float64 `json:"change"`
	// CILow and CIHigh bound the relative change.
	CILow  float64 `json:"ciLow"`
	CIHigh float64 `json:"ciHigh"`
	// NoiseFloor is the relative spread between repeats of one version.
	NoiseFloor float64 `json:"noiseFloor"`
	Verdict    string  `json:"verdict"`
}

MetricDelta compares one metric.

func (MetricDelta) MarshalJSON

func (d MetricDelta) MarshalJSON() ([]byte, error)

MarshalJSON writes an infinite change (A was zero and B was not) as null, which JSON cannot otherwise represent.

type MetricPoint

type MetricPoint struct {
	// T is seconds since the run started, like Point.T.
	T     float64 `json:"t"`
	Value float64 `json:"value"`
}

MetricPoint is one value of a target metric.

type Narrative

type Narrative struct {
	Summary string  `json:"summary"`
	Claims  []Claim `json:"claims"`
	// Model is the model that wrote it, for the record.
	Model string `json:"model,omitempty"`
	// Facts are the report figures the claims cite.
	Facts []Fact `json:"facts,omitempty"`
}

Narrative is a written summary of a report. Every claim cites the report facts it rests on and says whether those facts show it (measured) or only suggest it (suspected).

func (*Narrative) Check

func (n *Narrative) Check(facts []Fact) int

Check drops claims that cite unknown facts or carry an unknown label, and claims labelled measured that cite nothing. It returns how many claims were dropped.

func (*Narrative) KeepCited

func (n *Narrative) KeepCited(facts []Fact)

KeepCited sets Facts to the facts the claims cite, in report order.

type PhaseStat

type PhaseStat struct {
	Mean float64 `json:"mean"`
	P95  float64 `json:"p95"`
}

PhaseStat is a timing phase summary in seconds.

type Point

type Point struct {
	T          float64 `json:"t"` // seconds since start
	RPS        float64 `json:"rps"`
	ErrorRate  float64 `json:"errorRate"`
	P50        float64 `json:"p50"`
	P95        float64 `json:"p95"`
	P99        float64 `json:"p99"`
	VUs        int     `json:"vus"`
	Planned    float64 `json:"planned"`
	Dropped    uint64  `json:"dropped"`
	Iterations uint64  `json:"iterations"`
	SchedLag99 float64 `json:"schedLagP99"`
}

Point is one timeline interval.

type Recovery

type Recovery struct {
	// Recovered is false when the run ended before the target was back.
	Recovered bool `json:"recovered"`
	// Seconds from load returning to normal to the target holding its
	// baseline again.
	Seconds float64 `json:"seconds,omitempty"`
	// NormalAt is when load returned to normal, in seconds since start.
	NormalAt float64 `json:"normalAt"`
	// Baseline is the target before the overload: median p95 (seconds)
	// and error rate over the normal-load hold.
	BaselineP95       float64 `json:"baselineP95"`
	BaselineErrorRate float64 `json:"baselineErrorRate"`
}

Recovery is how long the target took to return to normal after an overload: set for the spike and recovery shapes, which hold normal load, push past it, then return to it.

type RefineStep

type RefineStep struct {
	Level    float64  `json:"level"`
	Pass     bool     `json:"pass"`
	FailedOn []string `json:"failedOn,omitempty"`
}

RefineStep is one confirmation hold of a breakpoint search.

type Report

type Report struct {
	FormatVersion int       `json:"formatVersion"`
	Stampede      string    `json:"stampede"`
	RunID         string    `json:"runId,omitempty"`
	Scenario      string    `json:"scenario"`
	Target        string    `json:"target"`
	Started       time.Time `json:"started"`
	Ended         time.Time `json:"ended"`
	// Duration is the measured load phase in seconds.
	Duration   float64     `json:"duration"`
	StopReason string      `json:"stopReason"`
	Load       LoadInfo    `json:"load"`
	Verdict    string      `json:"verdict"`
	Thresholds []Check     `json:"thresholds"`
	Overall    Stats       `json:"overall"`
	Journeys   []Journey   `json:"journeys"`
	Errors     []ErrorRow  `json:"errors"`
	Timeline   []Point     `json:"timeline"`
	Breakpoint *Breakpoint `json:"breakpoint,omitempty"`
	// Curve and Knee are set for runs whose load changed over time.
	Curve []CurvePoint `json:"curve,omitempty"`
	Knee  *Knee        `json:"knee,omitempty"`
	// Recovery is set for spike and recovery shapes.
	Recovery *Recovery `json:"recovery,omitempty"`
	Notes    []string  `json:"notes,omitempty"`
	// Workers describes each worker of a distributed run; an in-process
	// run lists its machine only when it was saturated.
	Workers []WorkerRow `json:"workers,omitempty"`
	// Faults are the faults a stampede agent injected during the run.
	Faults []FaultEvent `json:"faults,omitempty"`
	// Narrative is an optional AI-written summary that cites the figures.
	Narrative *Narrative `json:"narrative,omitempty"`
	// TargetMetrics are the target's own metrics over the run, queried
	// from Prometheus after it ended (observe.prometheus).
	TargetMetrics []TargetMetric `json:"targetMetrics,omitempty"`
}

Report is the complete result of a run. It is the JSON export format.

func Build

func Build(in Input) *Report

Build computes a report.

func ReadJSON

func ReadJSON(rd io.Reader) (*Report, error)

ReadJSON loads a report written by WriteJSON.

func (*Report) Facts

func (r *Report) Facts() []Fact

Facts lists the figures a narrative may cite. Ids are stable for a given report, so a claim's citations can be checked and linked.

func (r *Report) SetTraceLinks(tmpl string)

SetTraceLinks fills TraceURL for every slow request from a template containing {traceId}. An empty template clears the links.

func (*Report) SlowestOverall

func (r *Report) SlowestOverall(n int) []SlowRow

SlowestOverall returns up to n of the run's slowest requests across all steps, slowest first.

func (*Report) WriteCSV

func (r *Report) WriteCSV(w io.Writer) error

WriteCSV writes the per-step table. Journey totals have an empty step and the run's totals have an empty journey and step.

func (*Report) WriteHTML

func (r *Report) WriteHTML(w io.Writer) error

WriteHTML writes a self-contained HTML report with inline SVG charts and no external requests, so it can be archived or attached to CI runs.

func (*Report) WriteJSON

func (r *Report) WriteJSON(w io.Writer) error

WriteJSON writes the report as indented JSON.

func (*Report) WriteJUnit

func (r *Report) WriteJUnit(w io.Writer) error

WriteJUnit writes one test case per target, for CI systems.

func (*Report) WriteMarkdown

func (r *Report) WriteMarkdown(w io.Writer)

WriteMarkdown writes a summary suitable for a pull request comment.

func (*Report) WriteNarrativeText

func (r *Report) WriteNarrativeText(w io.Writer)

WriteNarrativeText writes the narrative for the terminal.

func (*Report) WriteText

func (r *Report) WriteText(w io.Writer)

WriteText writes the terminal summary as plain text.

func (*Report) WriteTextColor added in v1.2.0

func (r *Report) WriteTextColor(w io.Writer)

WriteTextColor writes the terminal summary with colours, for a terminal.

func (*Report) WriteTimelineCSV

func (r *Report) WriteTimelineCSV(w io.Writer) error

WriteTimelineCSV writes one row per second of the run.

type Side

type Side struct {
	Label string   `json:"label"`
	Runs  []string `json:"runs"`
}

Side describes one version's runs.

type SlowRequest

type SlowRequest struct {
	// Latency is in seconds, measured from the intended send time.
	Latency float64 `json:"latency"`
	// At is when the request was sent; T is the same in seconds since
	// the run started.
	At time.Time `json:"at"`
	T  float64   `json:"t"`
	// TraceID is the W3C trace ID the request carried in traceparent.
	TraceID string `json:"traceId,omitempty"`
	// TraceURL links to the trace when a trace link template is set.
	TraceURL string `json:"traceUrl,omitempty"`
	Status   int    `json:"status,omitempty"`
	Error    string `json:"error,omitempty"`
}

SlowRequest is one of a step's slowest requests.

type SlowRow

type SlowRow struct {
	Journey string
	Step    string
	SlowRequest
}

SlowRow is a slow request with the step it belongs to.

type Span

type Span struct {
	From float64 `json:"from"`
	To   float64 `json:"to"`
}

Span is a time window in seconds since the start; To is exclusive.

type Stats

type Stats struct {
	Requests     uint64              `json:"requests"`
	Failed       uint64              `json:"failed"`
	ErrorRate    float64             `json:"errorRate"`
	RPS          float64             `json:"rps"`
	Latency      metrics.Percentiles `json:"latency"`
	Service      metrics.Percentiles `json:"service"`
	BytesIn      uint64              `json:"bytesIn"`
	BytesOut     uint64              `json:"bytesOut"`
	ChecksPassed uint64              `json:"checksPassed"`
	ChecksFailed uint64              `json:"checksFailed"`
	Status       map[int]uint64      `json:"status,omitempty"`
	// Iteration-level figures (overall and per journey).
	Iterations       uint64              `json:"iterations,omitempty"`
	IterationsFailed uint64              `json:"iterationsFailed,omitempty"`
	IterationTime    metrics.Percentiles `json:"iterationTime,omitzero"`
	Dropped          uint64              `json:"dropped,omitempty"`
}

Stats summarises a set of requests.

type Step

type Step struct {
	ID    int    `json:"id"`
	Name  string `json:"name"`
	Stats Stats  `json:"stats"`
	// Phases holds mean and p95 seconds for dns, connect, tls, wait and download.
	Phases map[string]PhaseStat `json:"phases"`
	// Protocols counts requests by negotiated protocol (HTTP/1.1, HTTP/2.0).
	Protocols map[string]uint64 `json:"protocols,omitempty"`
	// Stream is set for streaming steps (server-sent events, gRPC server
	// streaming).
	Stream *StreamStat `json:"stream,omitempty"`
	// Slowest lists the step's slowest requests, slowest first.
	Slowest []SlowRequest `json:"slowest,omitempty"`
	// Browser is set for browser page loads and interactions.
	Browser *BrowserStat `json:"browser,omitempty"`
}

Step is per-request-step output.

type StepDelta

type StepDelta struct {
	Journey string        `json:"journey"`
	Step    string        `json:"step"`
	Metrics []MetricDelta `json:"metrics"`
}

StepDelta compares one request step across the runs of both versions.

type StreamStat

type StreamStat struct {
	// Streams counts steps that received at least one event.
	Streams uint64 `json:"streams"`
	Events  uint64 `json:"events"`
	// EventsPerSec is the rate of the events after the first, measured
	// from each stream's first event to its end.
	EventsPerSec float64 `json:"eventsPerSec"`
	// FirstEvent is the time from the start of the step to its first
	// event, in seconds.
	FirstEvent FirstEventStat `json:"firstEvent"`
}

StreamStat summarises a streaming step. For an LLM API, FirstEvent is the time to first token and EventsPerSec is tokens per second.

type TargetMetric

type TargetMetric struct {
	Name   string        `json:"name"`
	Query  string        `json:"query"`
	Points []MetricPoint `json:"points"`
	// Error explains why points are missing or incomplete.
	Error string `json:"error,omitempty"`
}

TargetMetric is one Prometheus query evaluated over the run.

func (TargetMetric) Range

func (m TargetMetric) Range() (lo, hi, last float64, ok bool)

Range returns the minimum, maximum and last value; ok is false when there are no points.

type WorkerRow

type WorkerRow struct {
	ID         string  `json:"id"`
	Name       string  `json:"name"`
	Region     string  `json:"region,omitempty"`
	ShareLo    float64 `json:"shareLo"`
	ShareHi    float64 `json:"shareHi"`
	State      string  `json:"state"`
	StopReason string  `json:"stopReason,omitempty"`
	Error      string  `json:"error,omitempty"`
	PeakVUs    int     `json:"peakVUs"`
	Requests   uint64  `json:"requests"`
	// Saturated lists the windows in which the worker reported itself
	// saturated (CPU, scheduling lag, GC pauses, file descriptors or
	// dropped iterations); latency measured there may reflect the
	// generator rather than the target.
	Saturated         []Span   `json:"saturated,omitempty"`
	SaturationReasons []string `json:"saturationReasons,omitempty"`
	// Lost is the window from the worker's loss to the end of the run, or
	// until another worker took over its share: its share of the load was
	// not generated in that window.
	Lost *Span `json:"lost,omitempty"`
	// Replaces names the lost worker whose share this one took over.
	Replaces string `json:"replaces,omitempty"`
	// ClockOffset is the worker's measured clock offset in seconds.
	ClockOffset float64 `json:"clockOffset"`
}

WorkerRow is one worker's part in a distributed run. Times are seconds since the run started, like the timeline.

func (WorkerRow) SharePct

func (w WorkerRow) SharePct() float64

SharePct is the worker's share of the load in percent.

Directories

Path Synopsis
Package pdf prints a report to PDF with headless Chrome: the same page as the HTML report, in its light theme.
Package pdf prints a report to PDF with headless Chrome: the same page as the HTML report, in its light theme.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL