bench

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 15, 2026 License: MIT Imports: 35 Imported by: 0

Documentation

Overview

Package bench is "ypy bench": the performance harness of the framework. It opens the benchmark page (bench/pages) in Chrome through the DevTools protocol, runs the scenarios in fresh pages, summarizes the samples and stores them as JSON in bench/results; it also compares and shows stored results and profiles scenarios with the DevTools Profiler.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Command

func Command() cli.Command

Command is the "bench" subcommand.

func Profile

func Profile(ctx context.Context, o ProfileOptions) error

Profile profiles the scenarios through the DevTools Profiler domain. The build defaults to dist/ypy.js, the unminified one, so the profile shows function names (private methods appear as #name).

func Show

func Show(w io.Writer, r *Result, md bool)

Show prints a stored result as a table; md makes it a Markdown table.

func Smoke

func Smoke(ctx context.Context, o RunOptions) (skipped string, err error)

Smoke is the "bench-smoke" gate: a quick run of the scenarios with deterministic metrics against the most recent result in bench/results, with threshold 0. skipped says why the gate could not run (no Chrome, no reference); err is a regression or a failed run.

func WriteJSON

func WriteJSON(path string, v any) error

WriteJSON writes v as JSON indented by two spaces, like the earlier harness.

Types

type Checkpoint

type Checkpoint struct {
	At        any      `json:"at"`
	HeapUsed  float64  `json:"heapUsed"`
	Nodes     int64    `json:"nodes"`
	Listeners int64    `json:"listeners"`
	Documents int64    `json:"documents"`
	UAM       *float64 `json:"uam"`
}

Checkpoint is a memory reading during the session scenario. At is a request count or "end".

type CompareOptions

type CompareOptions struct {
	// Threshold is the difference in percent above which a metric counts as
	// worse (or better).
	Threshold float64
	// Only keeps the metrics whose id matches; nil keeps all.
	Only *regexp.Regexp
	// Terse leaves out the lists of metrics only one of the results has.
	Terse bool
}

CompareOptions tune a comparison.

type Comparison

type Comparison struct {
	Regressions int
	Faster      int
	Compared    int
}

Comparison is the outcome of comparing two results.

func Compare

func Compare(w io.Writer, a, b *Result, o CompareOptions) Comparison

Compare prints, for every metric both results have, the median of each, the difference in percent and a flag. A is the reference, B the build under review. Every metric is "lower is better" unless the result says otherwise:

REGRESSION  B is more than the threshold worse than A
faster      B is more than the threshold better than A
(blank)     inside the threshold

A trailing "~" means the noise between the rounds of either run (the interquartile range of the round medians, stored in the result) is at least as large as the difference: the flag is not trustworthy, run again with more rounds. The p95 difference is shown too, since a median can hide a worse tail.

type Config

type Config struct {
	Rounds int     `json:"rounds"`
	Scale  float64 `json:"scale"`
	Quick  bool    `json:"quick"`
	Filter *string `json:"filter"`
}

Config is how a result was produced.

type Environment

type Environment struct {
	Chrome string `json:"chrome"`
	// Node is only present in results recorded by the earlier harness.
	Node                string `json:"node,omitempty"`
	Go                  string `json:"go,omitempty"`
	OS                  string `json:"os"`
	CPU                 string `json:"cpu"`
	CrossOriginIsolated bool   `json:"crossOriginIsolated"`
}

Environment is the machine and the browser of a result.

type Metric

type Metric struct {
	Unit          string    `json:"unit"`
	LowerIsBetter *bool     `json:"lowerIsBetter,omitempty"`
	N             int       `json:"n"`
	Median        float64   `json:"median"`
	P95           float64   `json:"p95"`
	Min           float64   `json:"min"`
	Mean          float64   `json:"mean"`
	CV            float64   `json:"cv"`
	RoundMedians  []float64 `json:"roundMedians"`
	NoisePct      float64   `json:"noisePct"`
}

Metric is the summary of one measured quantity. Every metric is "lower is better"; the unit says what it counts (ms, us, KB, B, count).

func (Metric) Lower

func (m Metric) Lower() bool

Lower reports whether a smaller value is better (the default).

type Ordered

type Ordered[V any] struct {
	// contains filtered or unexported fields
}

Ordered is a JSON object that keeps the order of its keys: results list their metrics in the order the scenarios produced them.

func (*Ordered[V]) Get

func (o *Ordered[V]) Get(key string) (V, bool)

Get returns the value of a key.

func (*Ordered[V]) Keys

func (o *Ordered[V]) Keys() []string

Keys returns the keys in order.

func (*Ordered[V]) Len

func (o *Ordered[V]) Len() int

Len is the number of keys.

func (Ordered[V]) MarshalJSON

func (o Ordered[V]) MarshalJSON() ([]byte, error)

MarshalJSON implements json.Marshaler.

func (*Ordered[V]) Set

func (o *Ordered[V]) Set(key string, v V)

Set adds or replaces a key; a new key goes last.

func (*Ordered[V]) UnmarshalJSON

func (o *Ordered[V]) UnmarshalJSON(data []byte) error

UnmarshalJSON implements json.Unmarshaler.

type Outcome

type Outcome struct {
	Result *Result
	// Path of the JSON file; empty when nothing was written (--list).
	Path string
	// Failed is true when a scenario reported an error; the result is written
	// all the same, without that scenario.
	Failed bool
}

Outcome of a run.

func Run

func Run(ctx context.Context, o RunOptions) (*Outcome, error)

Run measures the build and writes the result. It returns the outcome, or an error: *browsers.Unavailable when Chrome is missing, any other for what could not run at all.

type ProfileOptions

type ProfileOptions struct {
	RunOptions
	Scenarios []string
	Seconds   float64
	Top       int
	// Raw, when set, receives the .cpuprofile files (loadable in the DevTools
	// Performance panel).
	Raw string
}

ProfileOptions configure a CPU profile run. Every path is absolute.

type ProfileSummary

type ProfileSummary struct {
	TotalMs   float64  `json:"totalMs"`
	HarnessMs float64  `json:"harnessMs"`
	IdleMs    float64  `json:"idleMs"`
	Self      []Ranked `json:"self"`
	Inclusive []Ranked `json:"inclusive"`
	Scenario  string   `json:"scenario"`
	SHA       string   `json:"sha"`
	Build     string   `json:"build"`
	Runs      int      `json:"runs"`
}

ProfileSummary is the JSON a profile run writes as profile-<scenario>-<sha>.json.

type Ranked

type Ranked struct {
	Name string  `json:"name"`
	Ms   float64 `json:"ms"`
	Pct  float64 `json:"pct"`
}

Ranked is one function of a profile summary.

type Result

type Result struct {
	Schema            int                   `json:"schema"`
	Date              string                `json:"date"`
	Time              string                `json:"time,omitempty"`
	SHA               string                `json:"sha"`
	Build             string                `json:"build"`
	Config            Config                `json:"config"`
	Environment       Environment           `json:"environment"`
	DurationSeconds   int                   `json:"durationSeconds"`
	Skipped           Ordered[string]       `json:"skipped"`
	Notes             Ordered[string]       `json:"notes"`
	MemoryCheckpoints Ordered[[]Checkpoint] `json:"memoryCheckpoints"`
	Metrics           Ordered[Metric]       `json:"metrics"`
}

Result is the JSON file a run writes to bench/results.

func LatestReference

func LatestReference(dir string) (*Result, string, error)

LatestReference finds the reference result in dir: the most recent full run (not a quick or filtered one, not a profile). Results sort by date, then by the time they record, then by modification time.

func ReadResult

func ReadResult(path string) (*Result, error)

ReadResult reads a result file.

type RunOptions

type RunOptions struct {
	// Root is the repository root: it holds bench/pages and dist/.
	Root string
	// Build is the file to measure; empty means $YPY_BUILD, then dist/ypy.min.js.
	Build string
	// Ref, when set, builds that commit in a temporary directory and measures it.
	Ref string
	// Out is the directory of the JSON result.
	Out string
	// Label is added to the file name: <date>-<sha>-<label>.json.
	Label string
	// Filter keeps the scenarios whose name matches (the size entries always run).
	Filter string
	// Quick is one round with fewer samples: a smoke run, not a result.
	Quick bool
	// Rounds is the number of fresh pages per scenario, pooled (default 5).
	Rounds int
	// UAM also takes performance.measureUserAgentSpecificMemory() before and
	// after the session (about 18 s per call).
	UAM bool
	// Port of the local server; 0 picks a free one.
	Port       int
	ChromePath string
	// List prints the scenario names and stops.
	List bool

	Out2  io.Writer // table and progress
	Err   io.Writer
	IsTTY bool
	Env   func(string) string
}

RunOptions configure a measurement run. Every path is absolute.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL