Documentation
¶
Overview ¶
Package threshold implements FR-CLI-11: pass/fail thresholds on a run's own results, so a run can gate a CI pipeline.
A threshold is a name, a metric, an operator and a value, for example
fast-api: p95 < 300ms
It is judged once, against the finished run's numbers — the latency percentiles, error rate, request rate and counts VegaLoad itself measured. It never looks outside the run, and it produces a verdict (pass or fail) and nothing else.
A threshold can judge one named step of a scenario (FR-CLI-18) instead of the whole run, by adding a selector after the metric:
login-fast: p95{step="login"} < 300ms
Index ¶
Constants ¶
const MinLiveSamples = 20
MinLiveSamples is how many requests (or checks, or runs of a step) a statistic needs before a live judgement counts. A p95 of ten requests says very little.
Variables ¶
This section is empty.
Functions ¶
func AllPassed ¶
func AllPassed(rs []report.ThresholdResult) bool
AllPassed reports whether every result passed. It is true for an empty list, since a run with no thresholds has nothing to breach.
func Evaluate ¶
func Evaluate(ts []Threshold, res *report.Result) []report.ThresholdResult
Evaluate judges every threshold against a finished run's result, in order. A run that completed no requests fails every threshold: with nothing measured, "p95 < 300ms" would otherwise pass trivially.
func MetricNames ¶
func MetricNames() []string
MetricNames lists the metrics a threshold can use, for help text and error messages.
Types ¶
type LiveKind ¶ added in v0.6.0
type LiveKind int
LiveKind says how far a threshold can be judged while a run is still in progress (FR-CLI-13, -abort-on-breach).
const ( // LiveNone thresholds are judged only on the finished run. An example // is rps, which is an average over the whole run, or "total >= 1000", // which cannot be broken until the run is over. LiveNone LiveKind = iota // LiveSticky thresholds cannot recover once broken. The number of // failed requests only goes up, so "failed < 5" is broken for good the // moment the fifth one fails. The same holds for "max" and "total" // with an upper limit, and for "min" with a lower limit. LiveSticky // LiveSustained thresholds are statistics, such as p95 or error_rate. // They move up and down, so a breach early in the run may go away. A // run is stopped on one only when it stays broken (see the watcher in // cmd/vegaload). LiveSustained )
type Threshold ¶
type Threshold struct {
Name string
Metric string
// Step is the named step this threshold judges (FR-CLI-18). It is empty
// for a threshold on the whole run.
Step string
Operator string
// Value is the canonical text of the limit: a duration such as
// "300ms", or a plain number such as "0.01".
Value string
// contains filtered or unexported fields
}
Threshold is one parsed pass/fail threshold.
func LoadFile ¶
LoadFile reads thresholds from a JSON file (the -thresholds flag).
Three shapes are accepted, so a file can come straight from another command's output:
[{"name": "fast", "metric": "p95", "operator": "<", "value": "300ms"}]
[{"metric": "p95", "step": "login", "operator": "<", "value": "300ms"}]
{"thresholds": [ ...same list... ]}
{"latency_p95": "300ms", "error_rate": 0.015}
The last is what `vegaload diagnose -output json` suggests, under a "suggested_thresholds" key, which is also accepted. The suggested values are ceilings, so they become "<=" thresholds.
func NotLive ¶ added in v0.6.0
NotLive returns the thresholds that cannot be judged during a run, so the caller can say they wait for the end.
func Parse ¶
Parse parses one threshold expression: an optional "name:" prefix, then metric, operator and value, e.g. "p95 < 300ms" or "fast-api: error_rate <= 1%".
func (Threshold) Expression ¶
Expression returns the threshold as text that Parse accepts again, including its name, e.g. "fast-api: p95 < 300ms". It is what other commands print as a ready-to-paste -threshold flag.
func (Threshold) LiveBreach ¶ added in v0.6.0
LiveBreach judges t on a snapshot of a run in progress. It reports breached only when there is enough evidence: a step that has not run yet, or too few samples for a statistic, is not a breach. Those are settled on the finished run, as before.