Documentation
¶
Overview ¶
Package report implements FR-RPT-01 through 04 and FR-CLI-05: turning a run's recorded iterations into the structured data a self-contained HTML report, a JSON summary, and (eventually) the MCP get_results tool all read from. It implements engine.Recorder, exactly the seam internal/engine's own doc comment on Recorder sets aside for it, and knows nothing about protocol drivers, scripting, or the CLI. A protocol call arrives as a name, the same way a step does.
Index ¶
- Constants
- func AppendStepSummary(path string, res *Result) error
- func FormatByteCount(n int64) string
- func JUnitXML(res *Result) ([]byte, error)
- func MarkdownSummary(res *Result) string
- func ShowProtocols(res *Result) bool
- func WriteHTML(path string, res *Result) error
- func WriteJSON(path string, res *Result) error
- func WriteJUnit(path string, res *Result) error
- type AbortInfo
- type BaselineMetric
- type BaselineResult
- type BucketStat
- type CheckResult
- type Collector
- func (c *Collector) Begin(start time.Time)
- func (c *Collector) Counts() (total, failed int64)
- func (c *Collector) End(start time.Time)
- func (c *Collector) Finish(executor string, elapsed time.Duration) *Result
- func (c *Collector) LiveBuckets() *LiveBuckets
- func (c *Collector) Record(r engine.IterationResult)
- func (c *Collector) RecordCall(protocol, op string, d time.Duration, failed bool, sent, received int64)
- func (c *Collector) RecordCheck(name string, passed bool)
- func (c *Collector) RecordMeasure(protocol, name string, d time.Duration)
- func (c *Collector) RecordStep(name string, d time.Duration, failed bool)
- func (c *Collector) Start() time.Time
- type DataFile
- type Latency
- type LiveBuckets
- type MeasureResult
- type OperationResult
- type Point
- type ProtocolResult
- type Result
- type StepResult
- type ThresholdResult
Constants ¶
const LiveMaxHold = 10 * time.Second
LiveMaxHold is the longest a bucket is held back for calls that started in it (or earlier) and have not ended: this long past the end of its second. After that it is handed out with what it has, so that one hung call cannot freeze a live view. The final Result.TimeSeries stays authoritative.
const MaxCheckNames = 100
MaxCheckNames bounds how many distinct check names a run keeps. Further names are counted together under OtherChecksName.
const MaxStepNames = 100
MaxStepNames bounds how many distinct step names a run keeps. Further names are counted together under OtherStepsName.
const OtherChecksName = "(other checks)"
OtherChecksName is the bucket for check names past MaxCheckNames.
const OtherStepsName = "(other steps)"
OtherStepsName is the bucket for step names past MaxStepNames.
Variables ¶
This section is empty.
Functions ¶
func AppendStepSummary ¶ added in v0.4.0
AppendStepSummary appends MarkdownSummary(res) to the file at path. A step summary file is shared by every step of a job, so it is appended to, never replaced.
func FormatByteCount ¶ added in v0.9.0
FormatByteCount renders a byte count as B, KiB, MiB, GiB, TiB, or PiB.
func MarkdownSummary ¶ added in v0.4.0
MarkdownSummary renders res as a short Markdown page: the key numbers and the verdicts (FR-RPT-05). `vegaload run` appends it to the file GitHub Actions names in GITHUB_STEP_SUMMARY, so it shows on the job's page. The verdicts are the words PASS and FAIL, not colour or symbols.
func ShowProtocols ¶ added in v0.9.0
ShowProtocols reports whether the text, Markdown, and HTML views should print the protocol breakdown. JSON always keeps the data. A run that only called http, http1, or http2 keeps the layout it had before this breakdown existed.
func WriteHTML ¶
WriteHTML implements FR-RPT-01: one self-contained HTML file for res, with no server and nothing fetched at render time — every number is inlined, and the two charts are hand-drawn inline SVG (see the artifact-diagramming conventions this follows: currentColor strokes so the report reads in both light and dark browser themes, a viewBox sized to the content, labelled axes instead of a legend). The file opens correctly from disk with no network access at all, which is what "self-contained" means for FR-RPT-03's offline-renderable guarantee.
func WriteJSON ¶
WriteJSON implements FR-CLI-05's structured JSON output mode for a finished run's result: res marshalled as indented JSON to path.
func WriteJUnit ¶ added in v0.4.0
WriteJUnit writes res as JUnit XML to path.
Types ¶
type AbortInfo ¶ added in v0.6.0
type AbortInfo struct {
// Threshold is the name of the threshold that stopped the run.
Threshold string `json:"threshold"`
// Observed is what the run measured when it was stopped.
Observed string `json:"observed"`
// At is how far into the run it was stopped.
At time.Duration `json:"at_ns"`
}
AbortInfo says why and when a run was stopped early (FR-CLI-13).
type BaselineMetric ¶ added in v0.4.0
type BaselineMetric struct {
Name string `json:"name"`
Baseline float64 `json:"baseline"`
Candidate float64 `json:"candidate"`
Unit string `json:"unit"`
Regressed bool `json:"regressed"`
}
BaselineMetric is one gated metric: its baseline and this run's value, in the unit given ("ns" for p95, "ratio" for the error rate).
type BaselineResult ¶ added in v0.4.0
type BaselineResult struct {
Path string `json:"path"`
// MaxRegressionPercent is how much worse than the baseline each gated
// metric may be before the gate fails.
MaxRegressionPercent float64 `json:"max_regression_percent"`
Passed bool `json:"passed"`
Metrics []BaselineMetric `json:"metrics"`
Notes []string `json:"notes,omitempty"`
}
BaselineResult is the verdict of a run's baseline gate (FR-CLI-14). It only covers the two gated metrics, p95 latency and error rate; the full comparison is what `vegaload compare` prints.
type BucketStat ¶ added in v0.9.1
type BucketStat struct {
Offset time.Duration // start of the bucket
Requests int64
Failed int64
RPS float64
ErrorRate float64
P50 time.Duration
P95 time.Duration
P99 time.Duration
Total int64 // calls in this and every earlier bucket, as emitted
TotalFailed int64
// Capped is true when the bucket was handed out while a call that started
// in it, or in an earlier second, was still running, because the hold hit
// its limit (LiveMaxHold). The bucket then lacks that call; the final
// Result.TimeSeries has it.
Capped bool
}
BucketStat is what one one-second bucket of a run holds: the calls that started in that second. Requests, Failed, RPS and ErrorRate are the same numbers as the matching Point of Result.TimeSeries, worked out the same way. The latency percentiles use the same nearest-rank rule as Finish.
type CheckResult ¶ added in v0.4.0
type CheckResult struct {
Name string `json:"name"`
Passes int64 `json:"passes"`
Fails int64 `json:"fails"`
}
CheckResult is one named check's totals across the whole run.
type Collector ¶
type Collector struct {
// contains filtered or unexported fields
}
Collector implements engine.Recorder, accumulating every iteration of a run so Finish can compute the percentiles and time-bucketed series a Result needs. It is safe for concurrent use — Run calls Record from every VU goroutine at once.
func NewCollector ¶
func NewCollector() *Collector
NewCollector returns a Collector ready to record a run that is starting now.
func (*Collector) Begin ¶ added in v0.9.1
Begin and End implement engine.IterationTracker. They keep the start of each running iteration, only while a live cursor exists (see LiveBuckets), so the cursor can hold a bucket back until the calls that started in it have ended.
func (*Collector) Counts ¶
Counts returns the total and failed iteration counts recorded so far. Safe to call concurrently with Record, including while the run it belongs to is still in progress.
func (*Collector) End ¶ added in v0.9.1
End undoes the Begin made with the same start. The engine calls it after the iteration was recorded.
func (*Collector) Finish ¶
Finish computes a Result from every sample recorded so far. Call it once, after the executor has returned; executor identifies the load shape (ex.Name()) and elapsed is the run's measured wall-clock time, both of which the caller already has from runScenario.
func (*Collector) LiveBuckets ¶ added in v0.9.1
func (c *Collector) LiveBuckets() *LiveBuckets
LiveBuckets returns a cursor over the collector's buckets, from the first. Use one cursor per run, from one goroutine. Creating it makes the collector keep the start of every running iteration from then on, so create it before the executor starts.
func (*Collector) Record ¶
func (c *Collector) Record(r engine.IterationResult)
Record implements engine.Recorder.
func (*Collector) RecordCall ¶ added in v0.9.0
func (c *Collector) RecordCall(protocol, op string, d time.Duration, failed bool, sent, received int64)
RecordCall counts one finished protocol call. A nil collector counts nothing. protocol and op are redacted before they are stored.
func (*Collector) RecordCheck ¶ added in v0.4.0
RecordCheck counts one evaluation of the named check (FR-CLI-12). It implements netapi.CheckRecorder and is safe for concurrent use.
func (*Collector) RecordMeasure ¶ added in v0.9.0
RecordMeasure counts one phase time inside a protocol. A measure is not a call and does not change that protocol's totals.
func (*Collector) RecordStep ¶ added in v0.6.0
RecordStep counts one run of the named step (FR-CLI-18). It implements netapi.StepRecorder and is safe for concurrent use.
type DataFile ¶ added in v0.9.0
DataFile is one data file a protocol run read. Name is the file's base name. Rows is how many rows it has. A cell is not stored.
type Latency ¶
type Latency struct {
Min time.Duration `json:"min_ns"`
Mean time.Duration `json:"mean_ns"`
Max time.Duration `json:"max_ns"`
P50 time.Duration `json:"p50_ns"`
P90 time.Duration `json:"p90_ns"`
P95 time.Duration `json:"p95_ns"`
P99 time.Duration `json:"p99_ns"`
}
Latency holds the percentiles and summary stats FR-RPT-01 asks for.
type LiveBuckets ¶ added in v0.9.1
type LiveBuckets struct {
// contains filtered or unexported fields
}
LiveBuckets hands out the one-second buckets of a run that is still going, in order, each exactly once. It reads the collector's samples without copying them all: a bucket's calls are found by scanning back from the end of the slice, which holds the newest.
A call is recorded when it ends, but it belongs to the second it started in. A bucket is therefore handed out only when its second has closed, a grace period has passed (for a call that is just starting or just ending), and no call that started in that second or an earlier one is still running (the collector learns this from the engine through Begin and End). A call that runs longer than LiveMaxHold past the end of its second no longer holds the bucket: it is handed out without that call, marked Capped, and the final Result.TimeSeries, which has the call, is authoritative.
The final series ends at the last second a recorded call started in. A bucket past that second is therefore not handed out while the run goes on (a quiet second is held until a later one has a call, or the run ends), so the points never cover a second the series does not. Rest hands out what is left. A Capped bucket is the one exception: it goes out as it is.
func (*LiveBuckets) Next ¶ added in v0.9.1
func (l *LiveBuckets) Next(now, grace time.Duration) (BucketStat, bool)
Next returns the next bucket once it is due: now is the time since the collector started, and the bucket is due when now has passed the end of its second by grace and no call that started in it or earlier is running. A hold ends at the latest maxHold past the end of the second, and the bucket is then Capped. ok is false when it is not due yet.
func (*LiveBuckets) Rest ¶ added in v0.9.1
func (l *LiveBuckets) Rest() []BucketStat
Rest returns every bucket not handed out yet, up to the last second a call started in. Call it once the run has ended, before Finish's result is used, so that the points and the final time series cover the same seconds.
func (*LiveBuckets) SetMaxHold ¶ added in v0.9.1
func (l *LiveBuckets) SetMaxHold(d time.Duration)
SetMaxHold changes the hold limit (LiveMaxHold by default). It is for tests.
type MeasureResult ¶ added in v0.9.0
type MeasureResult struct {
Name string `json:"name"`
Count int64 `json:"count"`
Latency Latency `json:"latency"`
}
MeasureResult is a phase time inside a protocol, such as ftp upload_time. A measure is not a call: it does not change the protocol's totals.
type OperationResult ¶ added in v0.9.0
type OperationResult struct {
Name string `json:"name"`
Total int64 `json:"total"`
Failed int64 `json:"failed"`
ErrorRate float64 `json:"error_rate"`
Latency Latency `json:"latency"`
BytesSent int64 `json:"bytes_sent"`
BytesReceived int64 `json:"bytes_received"`
}
OperationResult is one operation inside a protocol, such as ftp roundtrip.
type Point ¶
type Point struct {
Offset time.Duration `json:"offset_ns"`
Requests int64 `json:"requests"`
Failed int64 `json:"failed"`
RPS float64 `json:"rps"`
ErrorRate float64 `json:"error_rate"`
}
Point is one bucket of Result.TimeSeries.
type ProtocolResult ¶ added in v0.9.0
type ProtocolResult struct {
Name string `json:"name"`
Total int64 `json:"total"`
Failed int64 `json:"failed"`
ErrorRate float64 `json:"error_rate"`
Latency Latency `json:"latency"`
BytesSent int64 `json:"bytes_sent"`
BytesReceived int64 `json:"bytes_received"`
BytesPerSec float64 `json:"bytes_per_second"`
Operations []OperationResult `json:"operations"`
Measures []MeasureResult `json:"measures,omitempty"`
}
ProtocolResult is one protocol's totals across the whole run. Latency is computed from every call, not from the operation percentiles. BytesPerSec is (sent + received) divided by the run's elapsed time.
func (ProtocolResult) Operation ¶ added in v0.9.0
func (p ProtocolResult) Operation(name string) (OperationResult, bool)
Operation returns the operation with the given name.
type Result ¶
type Result struct {
Executor string `json:"executor"`
StartedAt time.Time `json:"started_at"`
Elapsed time.Duration `json:"elapsed_ns"`
Total int64 `json:"total"`
Failed int64 `json:"failed"`
ErrorRate float64 `json:"error_rate"`
Latency Latency `json:"latency"`
// TimeSeries is RPS and error rate bucketed by one-second intervals
// from the run's start (FR-RPT-01's "RPS over time" and "error rate
// over time"). It stays small regardless of request volume — one
// point per second of run duration, not one per request — which is
// what lets the HTML report embed it directly even for a
// 1,000,000-request run (NFR-03).
TimeSeries []Point `json:"time_series"`
// Thresholds is the verdict of each pass/fail threshold the run was
// given (FR-CLI-11), in the order they were given. It is omitted when
// the run had none, so reports from before thresholds existed, and
// runs that do not use them, look exactly as before. ThresholdsPassed
// is true only when every threshold passed.
Thresholds []ThresholdResult `json:"thresholds,omitempty"`
ThresholdsPassed *bool `json:"thresholds_passed,omitempty"`
// Aborted is set when the run was stopped early because a threshold
// was broken (-abort-on-breach, FR-CLI-13). It is omitted for a run
// that ran to its end.
Aborted *AbortInfo `json:"aborted,omitempty"`
// Baseline is the verdict of the baseline gate (FR-CLI-14): this run
// compared with a baseline report the user supplied. It is omitted
// when the run had no baseline.
Baseline *BaselineResult `json:"baseline,omitempty"`
// Steps is the latency and error rate of each named step the scenario
// ran with step() (FR-CLI-18), in the order each name was first seen.
// It is omitted when the scenario used no steps, so reports from
// before steps existed look exactly as before. A step that throws is a
// failed step, and it also fails its iteration.
Steps []StepResult `json:"steps,omitempty"`
// Protocols is the latency, error rate, and byte totals of each
// protocol the run called, in the order each name was first seen.
// It is omitted when the run recorded no call, so a report written
// before this field existed, and a run that makes no recorded call,
// look exactly as before.
Protocols []ProtocolResult `json:"protocols,omitempty"`
// Checks is the pass and fail count of each named check the scenario
// made with check() (FR-CLI-12), in the order each name was first
// seen. It is omitted when the scenario made no checks. A failed
// check does not fail an iteration or count as a failed request: it
// is its own measure, which a threshold can judge through the
// check_rate metric.
Checks []CheckResult `json:"checks,omitempty"`
// DataFiles is the name and row count of each data file a protocol
// run read. It is omitted when the run had no -data, so that JSON
// stays the same. A cell is never recorded.
DataFiles []DataFile `json:"data_files,omitempty"`
}
Result is a run's structured outcome: everything the HTML report renders, and the same shape FR-RPT-04 says the MCP get_results tool (Phase 2) must return, so an agent host can render its own view from exactly the same data a human reads in the HTML report.
func (*Result) CheckRate ¶ added in v0.4.0
CheckRate is the share of all checks that passed, from 0 to 1. The second result is false when the run made no checks, since a rate of nothing is not a rate of 100%.
type StepResult ¶ added in v0.6.0
type StepResult struct {
Name string `json:"name"`
Total int64 `json:"total"`
Failed int64 `json:"failed"`
ErrorRate float64 `json:"error_rate"`
Latency Latency `json:"latency"`
}
StepResult is one named step's totals across the whole run: how many times it ran, how many of those failed, and its latency.
type ThresholdResult ¶ added in v0.4.0
type ThresholdResult struct {
Name string `json:"name"`
Metric string `json:"metric"`
// Step is the step the threshold judges (FR-CLI-18). It is empty for a
// threshold on the whole run.
Step string `json:"step,omitempty"`
// Protocol and Op are the protocol, and optionally one operation, the
// threshold judges. They are empty for a threshold on the whole run.
Protocol string `json:"protocol,omitempty"`
Op string `json:"op,omitempty"`
Operator string `json:"operator"`
Value string `json:"value"`
Observed string `json:"observed"`
Passed bool `json:"passed"`
}
ThresholdResult is one threshold's verdict: what was asked (a name, a metric, an operator and a value) and what the run observed. The shape is deliberately plain — a name, a metric, an operator, a value — so anything that reads a saved report can read it without knowing how VegaLoad evaluated it. Value and Observed are strings so durations ("300ms") and plain numbers share one field.