Documentation
¶
Overview ¶
Package report turns comparison results into output for people (text, and markdown for a pull-request comment) and for machines (JSON, JUnit XML for CI test reporters), with drifts grouped by signature and explained.
Index ¶
- Constants
- Variables
- func Printable(s string) string
- func ValidFormat(f string) bool
- type Group
- type Location
- type Percentiles
- type Report
- func (r *Report) Incomplete() bool
- func (r *Report) JSON(w io.Writer) error
- func (r *Report) JUnit(w io.Writer, maxDrifts int) error
- func (r *Report) Locate(find func(id string) *Location)
- func (r *Report) Markdown(maxDrifts int) string
- func (r *Report) NoBaselineNote() string
- func (r *Report) NothingNote() string
- func (r *Report) PrivateNote() string
- func (r *Report) ResultNotes() string
- func (r *Report) Save(path string) error
- func (r *Report) Text(w io.Writer, maxDrifts int)
- func (r *Report) TimingsNote() string
- func (r *Report) UnusedNote() string
- func (r *Report) Write(w io.Writer, format string, maxDrifts int) error
- type Result
- type Slower
- type StatusChange
- type StatusGroup
- type Timings
Constants ¶
const ( Pass = "pass" Drift = "drift" Skipped = "skipped" Observed = "observed" // measured, never compared )
Outcomes of one request.
const ( SlowerFactor = 2 SlowerMS = 200 )
A request is much slower when it takes at least twice as long and at least 200 ms more: the ratio alone flags noise on fast requests (3 → 7 ms), the difference alone the jitter of slow ones (2.0 → 2.3 s); both together are a change a user of the frontend notices.
const DefaultDrifts = -1
DefaultDrifts is the maxDrifts of compare, run and report without --max-drifts: up to 20 drifts per request, and a request whose status changed leads with the change, keeps what says where it went (location, content type, side effects) and summarizes the rest in one line. Any other maxDrifts lists the drifts as they are.
const MarkdownLimit = 65536
MarkdownLimit is the size of a GitHub comment (65,536 characters). The markdown report stays below it, counting bytes, which are never fewer than characters.
const NoteNoBaseline = "no baseline"
NoteNoBaseline is the note of a request the baseline has no response for: a new request (accept takes it into the baseline), or a baseline made for another catalog.
const NotePrivateMissing = "private body not on this machine"
NotePrivateMissing is the note of a request skipped because its private body is not on the machine that compared: a private body never enters the repository, and it is never compared by its ref alone.
const Version = 1
Variables ¶
var Formats = []string{"text", "json", "junit", "markdown"}
Formats are the output formats of compare and report.
Functions ¶
func Printable ¶
Printable escapes the characters of response-derived text that a terminal or a CI log would act on instead of showing: control characters (a line break would start a line of its own, where GitHub reads "::warning::" as a workflow command; ESC starts terminal sequences), line and paragraph separators and the marks that reorder text (bidi).
Types ¶
type Group ¶
type Group struct {
N int `json:"n"`
Signature string `json:"signature"`
Requests int `json:"requests"`
Drifts int `json:"drifts"`
Example compare.Drift `json:"example"`
Explanation string `json:"explanation"`
Routes []Location `json:"routes,omitempty"`
RouteCount int `json:"route_count,omitempty"`
}
Group is a set of drifts with the same signature: one cause. Explanation says in plain words what changed; Routes are the routes it touches (at most 20, of RouteCount), with their action and source file.
type Location ¶
type Location struct {
Route string `json:"route"`
Action string `json:"action,omitempty"`
File string `json:"file,omitempty"`
}
Location is a route a group of drifts touches, from the inventory: the route, the action that handles it and the action's source file.
type Percentiles ¶
Percentiles of the response times of one side (nearest rank).
type Report ¶
type Report struct {
Version int `json:"version"`
Tag string `json:"tag"`
Against string `json:"against"`
Compared int `json:"compared"`
Passed int `json:"passed"`
Drifted int `json:"drifted"`
Skipped int `json:"skipped"`
Observed int `json:"observed"`
Results []Result `json:"results"`
Groups []Group `json:"groups"`
// The status changes shared by requests, the most common first: one
// cause, whatever drifts follow from it.
StatusChanges []StatusGroup `json:"status_changes,omitempty"`
// What review, accept and report need to find the compared sets again.
Baseline string `json:"baseline,omitempty"` // the baseline compared against, when it is one: its name
Level string `json:"level,omitempty"`
Mode string `json:"mode,omitempty"`
Created string `json:"created,omitempty"`
// What accept checks to know the comparison still describes the compared
// sets (manifest of the baseline, records of the capture), and the
// requests it has accepted from it since.
BaselineDigest string `json:"baseline_digest,omitempty"`
CaptureDigest string `json:"capture_digest,omitempty"`
Accepted []string `json:"accepted,omitempty"`
// Requests skipped because their private body is not on this machine;
// the result is incomplete unless they were skipped on purpose
// (--skip-private).
PrivateMissing []string `json:"private_missing,omitempty"`
SkipPrivate bool `json:"skip_private,omitempty"`
// Requests skipped because the baseline has no response for them.
NoBaseline []string `json:"no_baseline,omitempty"`
// Allowlist rules that applied to no change in this comparison: a stale
// rule (a renamed value, a request that is gone) stays unnoticed
// otherwise. A warning, never a failure.
UnusedRules []allowlist.Unused `json:"unused_rules,omitempty"`
// Response times of the two sides, as information (CompareTimings).
Timings *Timings `json:"timings,omitempty"`
// Origins name the services whose two sides were captured from
// different origins, each taken as the other ("http://localhost:8080 ≙
// http://localhost:8081").
Origins []string `json:"origins,omitempty"`
}
Report is the result of one comparison.
func Build ¶
Build counts results and groups their drifts. Groups are ordered by the number of requests they touch, then by the number of drifts, then by signature.
func (*Report) Incomplete ¶
Incomplete reports whether the comparison checked less than it should: private requests could not be compared and that was not accepted with --skip-private, or nothing at all was compared (--only matched no request, the baseline has none of the requests, every request is private or observed): a green result would mean nothing was checked. compare and run then exit 3.
func (*Report) JUnit ¶
JUnit writes the report as JUnit XML, with up to maxDrifts drifts per failed request.
func (*Report) Locate ¶
Locate attaches to every group the routes its drifts touch. find returns the location of a request, or nil when no route of the inventory matches it. Routes are ordered by the number of drifts they get, then by name.
func (*Report) Markdown ¶
Markdown writes the report as a pull-request comment: a one-line verdict, a summary table, the explained groups, then every drifted request with up to maxDrifts drifts in a collapsed <details>. What does not fit under MarkdownLimit is left out, and the comment says so.
func (*Report) NoBaselineNote ¶
NoBaselineNote lists the requests the baseline has no response for; empty when there are none.
func (*Report) NothingNote ¶
NothingNote says why nothing was compared; empty when something was.
func (*Report) PrivateNote ¶
PrivateNote says what happened to the private requests that could not be compared here; empty when there are none.
func (*Report) ResultNotes ¶
ResultNotes lists the notes of the requests (Result.Notes), up to ten; empty when there are none.
func (*Report) Text ¶
Text writes a readable report: every drifted request with up to maxDrifts drifts, then the groups and a summary line.
func (*Report) TimingsNote ¶
TimingsNote is one line on the response times, with up to ten much slower requests; empty when they were not compared.
func (*Report) UnusedNote ¶
UnusedNote warns of the allowlist rules that applied to no change, up to ten of them; empty when every rule did.
type Result ¶
type Result struct {
ID string `json:"id"`
Outcome string `json:"outcome"`
Level string `json:"level,omitempty"`
Note string `json:"note,omitempty"`
Drifts []compare.Drift `json:"drifts,omitempty"`
// StatusChange is how the status changed, when it did. Collapsed is
// how many of the drifts follow from it (the body, headers other than
// location and content type): text, markdown and JUnit summarize them
// in one line by default; Drifts has them all.
StatusChange *StatusChange `json:"status_change,omitempty"`
Collapsed int `json:"collapsed,omitempty"`
// Notes say how the comparison went where that is not a drift: an
// array the allowlist pairs by key whose items were paired by position.
Notes []string `json:"notes,omitempty"`
}
Result is the outcome for one request.
type Slower ¶
type Slower struct {
ID string `json:"id"`
Baseline int64 `json:"baseline_ms"`
Current int64 `json:"current_ms"`
}
Slower is a request that became much slower.
type StatusChange ¶
type StatusChange struct {
From string `json:"from"`
To string `json:"to"`
Location string `json:"location,omitempty"`
}
StatusChange is how the status of a request changed; Location is where a redirect now points, when its location header drifted.
type StatusGroup ¶
type StatusGroup struct {
StatusChange
Requests []string `json:"requests"`
}
StatusGroup is a status change shared by requests: the same cause, read once.
type Timings ¶
type Timings struct {
Requests int `json:"requests"` // compared requests timed on both sides
Baseline Percentiles `json:"baseline"`
Current Percentiles `json:"current"`
Slower []Slower `json:"slower,omitempty"` // the most added time first
}
Timings compares the response times of the two sides, in milliseconds per request as the capture measured them. It is information only, never part of the verdict: one measurement per request on a shared machine is too noisy for a gate.