Documentation
¶
Overview ¶
Package bench is "ypy bench": the performance harness of the framework. It opens the benchmark page (bench/pages) in Chrome through the DevTools protocol, runs the scenarios in fresh pages, summarizes the samples and stores them as JSON in bench/results; it also compares and shows stored results and profiles scenarios with the DevTools Profiler.
Index ¶
- func Command() cli.Command
- func Profile(ctx context.Context, o ProfileOptions) error
- func Show(w io.Writer, r *Result, md bool)
- func Smoke(ctx context.Context, o RunOptions) (skipped string, err error)
- func WriteJSON(path string, v any) error
- type Checkpoint
- type CompareOptions
- type Comparison
- type Config
- type Environment
- type Metric
- type Ordered
- type Outcome
- type ProfileOptions
- type ProfileSummary
- type Ranked
- type Result
- type RunOptions
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Profile ¶
func Profile(ctx context.Context, o ProfileOptions) error
Profile profiles the scenarios through the DevTools Profiler domain. The build defaults to dist/ypy.js, the unminified one, so the profile shows function names (private methods appear as #name).
func Smoke ¶
func Smoke(ctx context.Context, o RunOptions) (skipped string, err error)
Smoke is the "bench-smoke" gate: a quick run of the scenarios with deterministic metrics against the most recent result in bench/results, with threshold 0. skipped says why the gate could not run (no Chrome, no reference); err is a regression or a failed run.
Types ¶
type Checkpoint ¶
type Checkpoint struct {
At any `json:"at"`
HeapUsed float64 `json:"heapUsed"`
Nodes int64 `json:"nodes"`
Listeners int64 `json:"listeners"`
Documents int64 `json:"documents"`
UAM *float64 `json:"uam"`
}
Checkpoint is a memory reading during the session scenario. At is a request count or "end".
type CompareOptions ¶
type CompareOptions struct {
// Threshold is the difference in percent above which a metric counts as
// worse (or better).
Threshold float64
// Only keeps the metrics whose id matches; nil keeps all.
Only *regexp.Regexp
// Terse leaves out the lists of metrics only one of the results has.
Terse bool
}
CompareOptions tune a comparison.
type Comparison ¶
Comparison is the outcome of comparing two results.
func Compare ¶
func Compare(w io.Writer, a, b *Result, o CompareOptions) Comparison
Compare prints, for every metric both results have, the median of each, the difference in percent and a flag. A is the reference, B the build under review. Every metric is "lower is better" unless the result says otherwise:
REGRESSION B is more than the threshold worse than A faster B is more than the threshold better than A (blank) inside the threshold
A trailing "~" means the noise between the rounds of either run (the interquartile range of the round medians, stored in the result) is at least as large as the difference: the flag is not trustworthy, run again with more rounds. The p95 difference is shown too, since a median can hide a worse tail.
type Config ¶
type Config struct {
Rounds int `json:"rounds"`
Scale float64 `json:"scale"`
Quick bool `json:"quick"`
Filter *string `json:"filter"`
}
Config is how a result was produced.
type Environment ¶
type Environment struct {
Chrome string `json:"chrome"`
// Node is only present in results recorded by the earlier harness.
Node string `json:"node,omitempty"`
Go string `json:"go,omitempty"`
OS string `json:"os"`
CPU string `json:"cpu"`
CrossOriginIsolated bool `json:"crossOriginIsolated"`
}
Environment is the machine and the browser of a result.
type Metric ¶
type Metric struct {
Unit string `json:"unit"`
LowerIsBetter *bool `json:"lowerIsBetter,omitempty"`
N int `json:"n"`
Median float64 `json:"median"`
P95 float64 `json:"p95"`
Min float64 `json:"min"`
Mean float64 `json:"mean"`
CV float64 `json:"cv"`
RoundMedians []float64 `json:"roundMedians"`
NoisePct float64 `json:"noisePct"`
}
Metric is the summary of one measured quantity. Every metric is "lower is better"; the unit says what it counts (ms, us, KB, B, count).
type Ordered ¶
type Ordered[V any] struct { // contains filtered or unexported fields }
Ordered is a JSON object that keeps the order of its keys: results list their metrics in the order the scenarios produced them.
func (Ordered[V]) MarshalJSON ¶
MarshalJSON implements json.Marshaler.
func (*Ordered[V]) UnmarshalJSON ¶
UnmarshalJSON implements json.Unmarshaler.
type Outcome ¶
type Outcome struct {
Result *Result
// Path of the JSON file; empty when nothing was written (--list).
Path string
// Failed is true when a scenario reported an error; the result is written
// all the same, without that scenario.
Failed bool
}
Outcome of a run.
type ProfileOptions ¶
type ProfileOptions struct {
RunOptions
Scenarios []string
Seconds float64
Top int
// Raw, when set, receives the .cpuprofile files (loadable in the DevTools
// Performance panel).
Raw string
}
ProfileOptions configure a CPU profile run. Every path is absolute.
type ProfileSummary ¶
type ProfileSummary struct {
TotalMs float64 `json:"totalMs"`
HarnessMs float64 `json:"harnessMs"`
IdleMs float64 `json:"idleMs"`
Self []Ranked `json:"self"`
Inclusive []Ranked `json:"inclusive"`
Scenario string `json:"scenario"`
SHA string `json:"sha"`
Build string `json:"build"`
Runs int `json:"runs"`
}
ProfileSummary is the JSON a profile run writes as profile-<scenario>-<sha>.json.
type Result ¶
type Result struct {
Schema int `json:"schema"`
Date string `json:"date"`
Time string `json:"time,omitempty"`
SHA string `json:"sha"`
Build string `json:"build"`
Config Config `json:"config"`
Environment Environment `json:"environment"`
DurationSeconds int `json:"durationSeconds"`
Skipped Ordered[string] `json:"skipped"`
Notes Ordered[string] `json:"notes"`
MemoryCheckpoints Ordered[[]Checkpoint] `json:"memoryCheckpoints"`
Metrics Ordered[Metric] `json:"metrics"`
}
Result is the JSON file a run writes to bench/results.
func LatestReference ¶
LatestReference finds the reference result in dir: the most recent full run (not a quick or filtered one, not a profile). Results sort by date, then by the time they record, then by modification time.
type RunOptions ¶
type RunOptions struct {
// Root is the repository root: it holds bench/pages and dist/.
Root string
// Build is the file to measure; empty means $YPY_BUILD, then dist/ypy.min.js.
Build string
// Ref, when set, builds that commit in a temporary directory and measures it.
Ref string
// Out is the directory of the JSON result.
Out string
// Label is added to the file name: <date>-<sha>-<label>.json.
Label string
// Filter keeps the scenarios whose name matches (the size entries always run).
Filter string
// Quick is one round with fewer samples: a smoke run, not a result.
Quick bool
// Rounds is the number of fresh pages per scenario, pooled (default 5).
Rounds int
// UAM also takes performance.measureUserAgentSpecificMemory() before and
// after the session (about 18 s per call).
UAM bool
// Port of the local server; 0 picks a free one.
Port int
ChromePath string
// List prints the scenario names and stops.
List bool
Out2 io.Writer // table and progress
Err io.Writer
IsTTY bool
Env func(string) string
}
RunOptions configure a measurement run. Every path is absolute.