Documentation
¶
Overview ¶
Package report turns run metrics into a verdict, per-journey and per-step statistics, a timeline and exportable documents.
Index ¶
- Constants
- func AllPass(cs []Check) bool
- func Bytes(n uint64) string
- func CLS(f float64) string
- func MetricValue(f float64) string
- func Ms(s float64) string
- func Observe(p *scenario.Program, scope, metric string, total *metrics.Snapshot, ...) float64
- func Pct(f float64) string
- func Unit(mode string) string
- func VerdictLabel(v string) string
- type Breakpoint
- type BrowserStat
- type Check
- type Claim
- type Comparison
- type CurvePoint
- type ErrorRow
- type Fact
- type FaultEvent
- type FirstEventStat
- type Input
- type Journey
- type Knee
- type LoadInfo
- type MetricDelta
- type MetricPoint
- type Narrative
- type PhaseStat
- type Point
- type Recovery
- type RefineStep
- type Report
- func (r *Report) Facts() []Fact
- func (r *Report) SetTraceLinks(tmpl string)
- func (r *Report) SlowestOverall(n int) []SlowRow
- func (r *Report) WriteCSV(w io.Writer) error
- func (r *Report) WriteHTML(w io.Writer) error
- func (r *Report) WriteJSON(w io.Writer) error
- func (r *Report) WriteJUnit(w io.Writer) error
- func (r *Report) WriteMarkdown(w io.Writer)
- func (r *Report) WriteNarrativeText(w io.Writer)
- func (r *Report) WriteText(w io.Writer)
- func (r *Report) WriteTextColor(w io.Writer)
- func (r *Report) WriteTimelineCSV(w io.Writer) error
- type Side
- type SlowRequest
- type SlowRow
- type Span
- type Stats
- type Step
- type StepDelta
- type StreamStat
- type TargetMetric
- type WorkerRow
Constants ¶
const ( CmpRegression = "regression" CmpImprovement = "improvement" CmpNoChange = "no-change" CmpInconclusive = "inconclusive" )
Comparison verdicts.
const ( VerdictPass = "pass" VerdictFail = "fail" VerdictGeneratorLimit = "generator-limited" VerdictNoTargets = "no-targets" )
Verdicts.
Variables ¶
This section is empty.
Functions ¶
func MetricValue ¶
MetricValue formats a target metric value compactly: 0.0123, 12.3, 4.56k, 120M.
Types ¶
type Breakpoint ¶
type Breakpoint struct {
// Found is false when every level passed.
Found bool `json:"found"`
// LastPass is the highest planned load that met every target.
LastPass float64 `json:"lastPass"`
// FirstFail is the lowest planned load that missed a target.
FirstFail float64 `json:"firstFail,omitempty"`
Unit string `json:"unit"`
FailedOn []string `json:"failedOn,omitempty"`
// Refined lists the confirmation holds that narrowed LastPass and
// FirstFail after the first failing step; the breakpoint lies
// between the two.
Refined []RefineStep `json:"refined,omitempty"`
}
Breakpoint is the outcome of a breakpoint search.
type BrowserStat ¶
type BrowserStat struct {
TTFB *PhaseStat `json:"ttfb,omitempty"`
FCP *PhaseStat `json:"fcp,omitempty"`
LCP *PhaseStat `json:"lcp,omitempty"`
CLS *PhaseStat `json:"cls,omitempty"`
INP *PhaseStat `json:"inp,omitempty"`
Load *PhaseStat `json:"load,omitempty"`
}
BrowserStat holds a browser step's page timings and Web Vitals: mean and p95 in seconds, except CLS, which is unitless. Page loads have TTFB, FCP, LCP, CLS and Load; clicks and key presses have INP.
type Check ¶
type Check struct {
Source string `json:"source"`
Scope string `json:"scope"`
Metric string `json:"metric"`
Op string `json:"op"`
Target float64 `json:"target"`
Observed float64 `json:"observed"`
Pass bool `json:"pass"`
// Display strings in the metric's unit.
TargetText string `json:"targetText"`
ObservedText string `json:"observedText"`
}
Check is the outcome of one target.
type Claim ¶
type Claim struct {
Text string `json:"text"`
Label string `json:"label"` // measured or suspected
Refs []string `json:"refs"` // fact ids, see Facts
}
Claim is one statement in a narrative.
type Comparison ¶
type Comparison struct {
A Side `json:"a"`
B Side `json:"b"`
Comparable bool `json:"comparable"`
Problems []string `json:"problems,omitempty"`
Metrics []MetricDelta `json:"metrics"`
// Steps compares each request step's p95 and error rate. They are
// informational: with many steps some differ by chance, so they do
// not change the verdict.
Steps []StepDelta `json:"steps,omitempty"`
Verdict string `json:"verdict"` // regression, improvement, no-change, inconclusive
Confidence float64 `json:"confidence"`
}
Comparison judges whether version B differs from version A, using repeated runs of each. A single run is noisy, so a change is only called a regression or improvement when the bootstrap confidence interval of the difference excludes zero and the change is larger than the noise floor measured from the repeats themselves.
func Compare ¶
func Compare(a, b []*Report, labelA, labelB string) *Comparison
Compare compares runs of version A with runs of version B.
func (*Comparison) WriteMarkdown ¶
func (c *Comparison) WriteMarkdown(w io.Writer)
WriteMarkdown writes the comparison for a pull request comment.
func (*Comparison) WriteText ¶
func (c *Comparison) WriteText(w io.Writer)
WriteText writes the comparison as a terminal table.
type CurvePoint ¶
type CurvePoint struct {
// Offered is the planned load: users, or iterations per second.
Offered float64 `json:"offered"`
// Throughput is completed iterations per second.
Throughput float64 `json:"throughput"`
RPS float64 `json:"rps"`
P50 float64 `json:"p50"`
P95 float64 `json:"p95"`
P99 float64 `json:"p99"`
ErrorRate float64 `json:"errorRate"`
Seconds int `json:"seconds"`
}
CurvePoint is one load level of a run whose load changed over time.
type ErrorRow ¶
type ErrorRow struct {
Journey string `json:"journey"`
Step string `json:"step"`
Error string `json:"error"`
Count uint64 `json:"count"`
// Examples are up to three of these failures, with what was sent and
// received (credentials and secrets redacted).
Examples []metrics.ErrorExample `json:"examples,omitempty"`
}
ErrorRow counts one kind of failure at one step.
type Fact ¶
type Fact struct {
ID string `json:"id"`
Text string `json:"text"`
Where string `json:"where"` // the report section that shows it
}
Fact is one figure from a report, with a stable id a narrative cites.
type FaultEvent ¶
type FaultEvent struct {
Label string `json:"label"`
Kind string `json:"kind"` // proxy, container or deployment
Target string `json:"target"`
Start float64 `json:"start"`
End float64 `json:"end"`
// Error says why the fault could not be applied or reverted.
Error string `json:"error,omitempty"`
}
FaultEvent is a fault injected during the run. Start and End are seconds since the run started, like Point.T.
type FirstEventStat ¶
type FirstEventStat struct {
Mean float64 `json:"mean"`
P50 float64 `json:"p50"`
P95 float64 `json:"p95"`
P99 float64 `json:"p99"`
}
FirstEventStat is a time-to-first-event summary in seconds.
type Input ¶
type Input struct {
RunID string
Program *scenario.Program
Plan *scenario.Plan
Target string
Started time.Time
Ended time.Time
StopReason string
PeakVUs int
Workers int
Interval time.Duration
Snapshots []*metrics.Snapshot // already merged per interval, any order
Phases map[int]*[metrics.NumPhases]*metrics.Histogram
Breakpoint *Breakpoint
}
Input is everything needed to build a report.
type Journey ¶
type Journey struct {
Name string `json:"name"`
Stats Stats `json:"stats"`
Steps []Step `json:"steps"`
}
Journey is per-journey output.
type Knee ¶
type Knee struct {
Found bool `json:"found"`
// At is the last level that still scaled; Next is the first that did not.
At CurvePoint `json:"at"`
Next *CurvePoint `json:"next,omitempty"`
Unit string `json:"unit"`
Reason string `json:"reason,omitempty"`
}
Knee marks where adding load stops adding throughput.
type LoadInfo ¶
type LoadInfo struct {
Shape string `json:"shape,omitempty"`
Mode string `json:"mode"`
Executor string `json:"executor"`
Peak float64 `json:"peak"`
PeakVUs int `json:"peakVUs"`
Workers int `json:"workers"`
}
LoadInfo summarises the plan.
type MetricDelta ¶
type MetricDelta struct {
Name string `json:"name"`
// HigherIsBetter is true for throughput-like metrics.
HigherIsBetter bool `json:"higherIsBetter"`
A []float64 `json:"a"`
B []float64 `json:"b"`
MeanA float64 `json:"meanA"`
MeanB float64 `json:"meanB"`
// Change is (meanB - meanA) / meanA.
Change float64 `json:"change"`
// CILow and CIHigh bound the relative change.
CILow float64 `json:"ciLow"`
CIHigh float64 `json:"ciHigh"`
// NoiseFloor is the relative spread between repeats of one version.
NoiseFloor float64 `json:"noiseFloor"`
Verdict string `json:"verdict"`
}
MetricDelta compares one metric.
func (MetricDelta) MarshalJSON ¶
func (d MetricDelta) MarshalJSON() ([]byte, error)
MarshalJSON writes an infinite change (A was zero and B was not) as null, which JSON cannot otherwise represent.
type MetricPoint ¶
type MetricPoint struct {
// T is seconds since the run started, like Point.T.
T float64 `json:"t"`
Value float64 `json:"value"`
}
MetricPoint is one value of a target metric.
type Narrative ¶
type Narrative struct {
Summary string `json:"summary"`
Claims []Claim `json:"claims"`
// Model is the model that wrote it, for the record.
Model string `json:"model,omitempty"`
// Facts are the report figures the claims cite.
Facts []Fact `json:"facts,omitempty"`
}
Narrative is a written summary of a report. Every claim cites the report facts it rests on and says whether those facts show it (measured) or only suggest it (suspected).
type Point ¶
type Point struct {
T float64 `json:"t"` // seconds since start
RPS float64 `json:"rps"`
ErrorRate float64 `json:"errorRate"`
P50 float64 `json:"p50"`
P95 float64 `json:"p95"`
P99 float64 `json:"p99"`
VUs int `json:"vus"`
Planned float64 `json:"planned"`
Dropped uint64 `json:"dropped"`
Iterations uint64 `json:"iterations"`
SchedLag99 float64 `json:"schedLagP99"`
}
Point is one timeline interval.
type Recovery ¶
type Recovery struct {
// Recovered is false when the run ended before the target was back.
Recovered bool `json:"recovered"`
// Seconds from load returning to normal to the target holding its
// baseline again.
Seconds float64 `json:"seconds,omitempty"`
// NormalAt is when load returned to normal, in seconds since start.
NormalAt float64 `json:"normalAt"`
// Baseline is the target before the overload: median p95 (seconds)
// and error rate over the normal-load hold.
BaselineP95 float64 `json:"baselineP95"`
BaselineErrorRate float64 `json:"baselineErrorRate"`
}
Recovery is how long the target took to return to normal after an overload: set for the spike and recovery shapes, which hold normal load, push past it, then return to it.
type RefineStep ¶
type RefineStep struct {
Level float64 `json:"level"`
Pass bool `json:"pass"`
FailedOn []string `json:"failedOn,omitempty"`
}
RefineStep is one confirmation hold of a breakpoint search.
type Report ¶
type Report struct {
FormatVersion int `json:"formatVersion"`
Stampede string `json:"stampede"`
RunID string `json:"runId,omitempty"`
Scenario string `json:"scenario"`
Target string `json:"target"`
Started time.Time `json:"started"`
Ended time.Time `json:"ended"`
// Duration is the measured load phase in seconds.
Duration float64 `json:"duration"`
StopReason string `json:"stopReason"`
Load LoadInfo `json:"load"`
Verdict string `json:"verdict"`
Thresholds []Check `json:"thresholds"`
Overall Stats `json:"overall"`
Journeys []Journey `json:"journeys"`
Errors []ErrorRow `json:"errors"`
Timeline []Point `json:"timeline"`
Breakpoint *Breakpoint `json:"breakpoint,omitempty"`
// Curve and Knee are set for runs whose load changed over time.
Curve []CurvePoint `json:"curve,omitempty"`
Knee *Knee `json:"knee,omitempty"`
// Recovery is set for spike and recovery shapes.
Recovery *Recovery `json:"recovery,omitempty"`
Notes []string `json:"notes,omitempty"`
// Workers describes each worker of a distributed run; an in-process
// run lists its machine only when it was saturated.
Workers []WorkerRow `json:"workers,omitempty"`
// Faults are the faults a stampede agent injected during the run.
Faults []FaultEvent `json:"faults,omitempty"`
// Narrative is an optional AI-written summary that cites the figures.
Narrative *Narrative `json:"narrative,omitempty"`
// TargetMetrics are the target's own metrics over the run, queried
// from Prometheus after it ended (observe.prometheus).
TargetMetrics []TargetMetric `json:"targetMetrics,omitempty"`
}
Report is the complete result of a run. It is the JSON export format.
func (*Report) Facts ¶
Facts lists the figures a narrative may cite. Ids are stable for a given report, so a claim's citations can be checked and linked.
func (*Report) SetTraceLinks ¶
SetTraceLinks fills TraceURL for every slow request from a template containing {traceId}. An empty template clears the links.
func (*Report) SlowestOverall ¶
SlowestOverall returns up to n of the run's slowest requests across all steps, slowest first.
func (*Report) WriteCSV ¶
WriteCSV writes the per-step table. Journey totals have an empty step and the run's totals have an empty journey and step.
func (*Report) WriteHTML ¶
WriteHTML writes a self-contained HTML report with inline SVG charts and no external requests, so it can be archived or attached to CI runs.
func (*Report) WriteJUnit ¶
WriteJUnit writes one test case per target, for CI systems.
func (*Report) WriteMarkdown ¶
WriteMarkdown writes a summary suitable for a pull request comment.
func (*Report) WriteNarrativeText ¶
WriteNarrativeText writes the narrative for the terminal.
func (*Report) WriteTextColor ¶ added in v1.2.0
WriteTextColor writes the terminal summary with colours, for a terminal.
type SlowRequest ¶
type SlowRequest struct {
// Latency is in seconds, measured from the intended send time.
Latency float64 `json:"latency"`
// At is when the request was sent; T is the same in seconds since
// the run started.
At time.Time `json:"at"`
T float64 `json:"t"`
// TraceID is the W3C trace ID the request carried in traceparent.
TraceID string `json:"traceId,omitempty"`
// TraceURL links to the trace when a trace link template is set.
TraceURL string `json:"traceUrl,omitempty"`
Status int `json:"status,omitempty"`
Error string `json:"error,omitempty"`
}
SlowRequest is one of a step's slowest requests.
type SlowRow ¶
type SlowRow struct {
Journey string
Step string
SlowRequest
}
SlowRow is a slow request with the step it belongs to.
type Stats ¶
type Stats struct {
Requests uint64 `json:"requests"`
Failed uint64 `json:"failed"`
ErrorRate float64 `json:"errorRate"`
RPS float64 `json:"rps"`
Latency metrics.Percentiles `json:"latency"`
Service metrics.Percentiles `json:"service"`
BytesIn uint64 `json:"bytesIn"`
BytesOut uint64 `json:"bytesOut"`
ChecksPassed uint64 `json:"checksPassed"`
ChecksFailed uint64 `json:"checksFailed"`
Status map[int]uint64 `json:"status,omitempty"`
// Iteration-level figures (overall and per journey).
Iterations uint64 `json:"iterations,omitempty"`
IterationsFailed uint64 `json:"iterationsFailed,omitempty"`
IterationTime metrics.Percentiles `json:"iterationTime,omitzero"`
Dropped uint64 `json:"dropped,omitempty"`
}
Stats summarises a set of requests.
type Step ¶
type Step struct {
ID int `json:"id"`
Name string `json:"name"`
Stats Stats `json:"stats"`
// Phases holds mean and p95 seconds for dns, connect, tls, wait and download.
Phases map[string]PhaseStat `json:"phases"`
// Protocols counts requests by negotiated protocol (HTTP/1.1, HTTP/2.0).
Protocols map[string]uint64 `json:"protocols,omitempty"`
// Stream is set for streaming steps (server-sent events, gRPC server
// streaming).
Stream *StreamStat `json:"stream,omitempty"`
// Slowest lists the step's slowest requests, slowest first.
Slowest []SlowRequest `json:"slowest,omitempty"`
// Browser is set for browser page loads and interactions.
Browser *BrowserStat `json:"browser,omitempty"`
}
Step is per-request-step output.
type StepDelta ¶
type StepDelta struct {
Journey string `json:"journey"`
Step string `json:"step"`
Metrics []MetricDelta `json:"metrics"`
}
StepDelta compares one request step across the runs of both versions.
type StreamStat ¶
type StreamStat struct {
// Streams counts steps that received at least one event.
Streams uint64 `json:"streams"`
Events uint64 `json:"events"`
// EventsPerSec is the rate of the events after the first, measured
// from each stream's first event to its end.
EventsPerSec float64 `json:"eventsPerSec"`
// FirstEvent is the time from the start of the step to its first
// event, in seconds.
FirstEvent FirstEventStat `json:"firstEvent"`
}
StreamStat summarises a streaming step. For an LLM API, FirstEvent is the time to first token and EventsPerSec is tokens per second.
type TargetMetric ¶
type TargetMetric struct {
Name string `json:"name"`
Query string `json:"query"`
Points []MetricPoint `json:"points"`
// Error explains why points are missing or incomplete.
Error string `json:"error,omitempty"`
}
TargetMetric is one Prometheus query evaluated over the run.
func (TargetMetric) Range ¶
func (m TargetMetric) Range() (lo, hi, last float64, ok bool)
Range returns the minimum, maximum and last value; ok is false when there are no points.
type WorkerRow ¶
type WorkerRow struct {
ID string `json:"id"`
Name string `json:"name"`
Region string `json:"region,omitempty"`
State string `json:"state"`
StopReason string `json:"stopReason,omitempty"`
Error string `json:"error,omitempty"`
PeakVUs int `json:"peakVUs"`
Requests uint64 `json:"requests"`
// Saturated lists the windows in which the worker reported itself
// saturated (CPU, scheduling lag, GC pauses, file descriptors or
// dropped iterations); latency measured there may reflect the
// generator rather than the target.
Saturated []Span `json:"saturated,omitempty"`
SaturationReasons []string `json:"saturationReasons,omitempty"`
// Lost is the window from the worker's loss to the end of the run, or
// until another worker took over its share: its share of the load was
// not generated in that window.
Lost *Span `json:"lost,omitempty"`
// Replaces names the lost worker whose share this one took over.
Replaces string `json:"replaces,omitempty"`
// ClockOffset is the worker's measured clock offset in seconds.
ClockOffset float64 `json:"clockOffset"`
}
WorkerRow is one worker's part in a distributed run. Times are seconds since the run started, like the timeline.