benchmark

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 11, 2026 License: AGPL-3.0 Imports: 17 Imported by: 0

Documentation

Overview

Package benchmark provides performance benchmarking for evaluable components.

Overview

The benchmark package enables systematic performance measurement of Trace's algorithms and components. It integrates with the eval framework to provide standardized benchmarking with statistical rigor.

Architecture 

┌─────────────────────────────────────────────────────────────────────────┐
│                         Benchmark Framework                              │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  ┌──────────────┐      ┌──────────────┐      ┌──────────────┐          │
│  │   Runner     │──────│   Metrics    │──────│   Reporter   │          │
│  │              │      │   Collector  │      │              │          │
│  │ • Warmup     │      │ • Latency    │      │ • JSON       │          │
│  │ • Iterations │      │ • Throughput │      │ • Console    │          │
│  │ • Cooldown   │      │ • Memory     │      │ • Prometheus │          │
│  └──────────────┘      └──────────────┘      └──────────────┘          │
│         │                     │                     │                   │
│         └─────────────────────┴─────────────────────┘                   │
│                               │                                         │
│                               ▼                                         │
│                    ┌──────────────────────┐                            │
│                    │    BenchmarkResult    │                            │
│                    │                       │                            │
│                    │ • Name                │                            │
│                    │ • Iterations          │                            │
│                    │ • Duration            │                            │
│                    │ • LatencyStats        │                            │
│                    │ • MemoryStats         │                            │
│                    │ • ThroughputStats     │                            │
│                    └──────────────────────┘                            │
│                                                                          │
└─────────────────────────────────────────────────────────────────────────┘

Usage

Basic benchmarking: 

runner := benchmark.NewRunner(registry)
result, err := runner.Run(ctx, "cdcl",
    benchmark.WithIterations(1000),
    benchmark.WithWarmup(100),
)
if err != nil {
    log.Fatal(err)
}
fmt.Printf("P99 latency: %v\n", result.Latency.P99)

With custom input generator: 

result, err := runner.Run(ctx, "cdcl",
    benchmark.WithInputGenerator(func() any {
        return generateRandomClauses(100)
    }),
)

Comparing algorithms: 

comparison, err := runner.Compare(ctx,
    []string{"cdcl_v1", "cdcl_v2"},
    benchmark.WithIterations(10000),
)
if comparison.Winner != "" {
    fmt.Printf("%s is %.2fx faster\n", comparison.Winner, comparison.Speedup)
}

Statistical Rigor

The benchmark package provides:

  • Percentile calculations (P50, P90, P95, P99, P999)
  • Standard deviation and variance
  • Confidence intervals
  • Statistical significance tests for comparisons
  • Outlier detection and removal

Thread Safety

All types in this package are safe for concurrent use unless documented otherwise.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrNoSamples indicates that no samples were collected.
	ErrNoSamples = errors.New("no samples collected")

	// ErrInvalidConfig indicates an invalid benchmark configuration.
	ErrInvalidConfig = errors.New("invalid benchmark configuration")

	// ErrBenchmarkFailed indicates that the benchmark failed to run.
	ErrBenchmarkFailed = errors.New("benchmark failed")
)

Functions

func CalculateCohensD 

func CalculateCohensD(samples1, samples2 []time.Duration) float64

CalculateCohensD calculates Cohen's d effect size between two sample sets.

Description: 

Cohen's d measures the standardized difference between two means.
Uses the pooled standard deviation for the denominator.
Positive d indicates samples1 > samples2, negative indicates samples1 < samples2.

Inputs:

  • samples1: First sample set. Must not be empty.
  • samples2: Second sample set. Must not be empty.

Outputs:

  • float64: Cohen's d value. Returns 0 if either sample is empty or if pooled standard deviation is 0.

Thread Safety: This function is stateless and safe for concurrent use.

Example: 

fast := []time.Duration{10*time.Millisecond, 11*time.Millisecond}
slow := []time.Duration{100*time.Millisecond, 101*time.Millisecond}
d := CalculateCohensD(fast, slow)  // Large negative d (fast is faster)

func ConfidenceInterval 

func ConfidenceInterval(samples []time.Duration, confidenceLevel float64) (lower, upper time.Duration)

ConfidenceInterval calculates the confidence interval for the mean.

Description: 

Calculates a symmetric confidence interval around the sample mean
using the standard error. For small samples (n < 30), uses t-distribution
critical values; for large samples, uses z-scores (normal distribution).

Inputs:

  • samples: Sample set. Must have at least 2 samples for meaningful interval.
  • confidenceLevel: Confidence level (e.g., 0.95 for 95%). Supported: 0.90, 0.95, 0.99.

Outputs:

  • lower: Lower bound of the interval.
  • upper: Upper bound of the interval.

Thread Safety: This function is stateless and safe for concurrent use.

Example: 

lower, upper := ConfidenceInterval(samples, 0.95)
fmt.Printf("95%% CI: [%v, %v]\n", lower, upper)

func RemoveOutliers 

func RemoveOutliers(samples []time.Duration, threshold float64) []time.Duration

RemoveOutliers removes outliers using the IQR method.

Description: 

Uses the Interquartile Range (IQR) method to identify and remove outliers.
Values outside [Q1 - threshold*IQR, Q3 + threshold*IQR] are removed.
If removing outliers would remove more than half the samples, returns
the original samples unchanged.

Inputs:

  • samples: Duration samples. Small samples (< 4) are returned unchanged.
  • threshold: IQR multiplier (typically 1.5 for mild outliers, 3.0 for extreme).

Outputs:

  • []time.Duration: Samples with outliers removed.

Thread Safety: This function is stateless and safe for concurrent use.

Example: 

samples := []time.Duration{10*time.Millisecond, 11*time.Millisecond, 1000*time.Millisecond}
filtered := RemoveOutliers(samples, 1.5)
// 1000ms outlier removed

func WelchTTest 

func WelchTTest(samples1, samples2 []time.Duration) (tStatistic float64, pValue float64)

WelchTTest performs Welch's t-test for two sample sets.

Description: 

Welch's t-test is used when the two samples have unequal variances
and/or unequal sample sizes. It does not assume equal population
variances, making it more robust than Student's t-test.

Inputs:

  • samples1: First sample set. Must have at least 2 samples.
  • samples2: Second sample set. Must have at least 2 samples.

Outputs:

  • tStatistic: The t-statistic. Negative if samples1 < samples2.
  • pValue: Approximate two-tailed p-value. Returns 1 if samples are insufficient or have zero variance.

Thread Safety: This function is stateless and safe for concurrent use.

Example: 

t, p := WelchTTest(fastSamples, slowSamples)
if p < 0.05 {
    fmt.Println("Significant difference at 95% confidence")
}

Limitations:

  • Uses normal approximation for p-value when df >= 30.
  • For smaller df, uses a rough approximation. For precise p-values, use a statistical library with t-distribution tables.

Types

type ComparisonResult 

type ComparisonResult struct {
	// Results holds the individual benchmark results keyed by name.
	Results map[string]*Result

	// Winner is the name of the fastest component (if statistically significant).
	// Empty if no clear winner.
	Winner string

	// Speedup is the ratio of slowest to fastest (e.g., 2.0 means 2x faster).
	Speedup float64

	// Significant indicates whether the difference is statistically significant.
	Significant bool

	// PValue is the p-value from the statistical test.
	PValue float64

	// ConfidenceLevel is the confidence level used (e.g., 0.95).
	ConfidenceLevel float64

	// EffectSize is Cohen's d effect size.
	EffectSize float64

	// EffectSizeCategory categorizes the effect size.
	EffectSizeCategory EffectSizeCategory

	// Ranking is the components ranked from fastest to slowest.
	Ranking []string
}

ComparisonResult holds the results of comparing two or more benchmarks.

Description: 

ComparisonResult provides statistical comparison between multiple
components, determining if performance differences are significant.
Uses Welch's t-test for significance and Cohen's d for effect size.

Thread Safety: Safe for concurrent read access after creation.

type Config 

type Config struct {
	// Iterations is the number of benchmark iterations to run.
	// Default: 1000
	Iterations int

	// Warmup is the number of warmup iterations before measurement.
	// Default: 100
	Warmup int

	// Cooldown is the duration to wait between warmup and measurement.
	// Default: 100ms
	Cooldown time.Duration

	// Timeout is the maximum time for the entire benchmark.
	// Default: 5 minutes
	Timeout time.Duration

	// IterationTimeout is the maximum time for a single iteration.
	// Default: 10 seconds
	IterationTimeout time.Duration

	// CollectMemory enables memory statistics collection.
	// Default: true
	CollectMemory bool

	// RemoveOutliers removes statistical outliers from results.
	// Default: true
	RemoveOutliers bool

	// OutlierThreshold is the IQR multiplier for outlier detection.
	// Default: 1.5
	OutlierThreshold float64

	// Parallelism is the number of concurrent benchmark goroutines.
	// Default: 1 (sequential)
	Parallelism int

	// InputGenerator generates input for each iteration.
	// If nil, uses the component's default generator.
	InputGenerator func() any
}

Config holds benchmark configuration.

Description: 

Config controls all aspects of a benchmark run including iteration counts,
timeouts, memory collection, and outlier handling. Use DefaultConfig() to
get sensible defaults, then override specific fields as needed.

Thread Safety: Safe for concurrent read access after initialization.

func DefaultConfig 

func DefaultConfig() *Config

DefaultConfig returns a configuration with default values.

Description: 

Creates a new Config with sensible defaults for most benchmarking
scenarios. Values can be overridden after creation.

Outputs:

  • *Config: Configuration with default values. Never nil.

Example: 

config := DefaultConfig()
config.Iterations = 10000  // Override defaults
config.Parallelism = 4

func (*Config) Validate 

func (c *Config) Validate() error

Validate checks that the configuration is valid.

Description: 

Validates all configuration fields to ensure they have acceptable
values. Must be called before using the configuration.

Outputs:

  • error: Non-nil if configuration is invalid, with message indicating which field failed validation.

Example: 

config := DefaultConfig()
config.Iterations = -1  // Invalid
if err := config.Validate(); err != nil {
    log.Fatal(err)  // "iterations must be positive"
}

type ConsoleReporter 

type ConsoleReporter struct {
	// contains filtered or unexported fields
}

ConsoleReporter outputs benchmark results to the console in human-readable format.

Description: 

ConsoleReporter formats benchmark results as human-readable text with
tables and statistics. Supports verbose mode for detailed memory stats.

Thread Safety: Safe for concurrent use if the underlying Writer is safe.

func NewConsoleReporter 

func NewConsoleReporter(out io.Writer, verbose bool) *ConsoleReporter

NewConsoleReporter creates a new console reporter.

Description: 

Creates a reporter that outputs human-readable benchmark results
to the specified writer. Verbose mode includes memory statistics.

Inputs:

  • out: Writer to output to. Must not be nil.
  • verbose: Whether to include detailed statistics.

Outputs:

  • *ConsoleReporter: The new reporter. Never nil.

Example: 

reporter := benchmark.NewConsoleReporter(os.Stdout, true)
reporter.Report(result)

Assumptions:

  • The writer is ready to accept output and will not be nil.

func (*ConsoleReporter) Report 

func (r *ConsoleReporter) Report(result *Result) error

Report writes a benchmark result to the console.

Description: 

Formats and writes the benchmark result including iterations,
latency statistics, percentiles, throughput, and optionally memory stats.

Inputs:

  • result: The benchmark result to report. Must not be nil.

Outputs:

  • error: Non-nil if writing failed.

Thread Safety: Safe for concurrent use if the underlying Writer is safe.

func (*ConsoleReporter) ReportAll 

func (r *ConsoleReporter) ReportAll(results []*Result) error

ReportAll writes multiple results to the console.

Description: 

Writes each result sequentially using Report.

Inputs:

  • results: The benchmark results to report.

Outputs:

  • error: Non-nil if writing any result failed.

Thread Safety: Safe for concurrent use if the underlying Writer is safe.

func (*ConsoleReporter) ReportComparison 

func (r *ConsoleReporter) ReportComparison(comparison *ComparisonResult) error

ReportComparison writes a comparison result to the console.

Description: 

Formats and writes the comparison result including a table of all
results, statistical analysis, and winner determination.

Inputs:

  • comparison: The comparison result to report. Must not be nil.

Outputs:

  • error: Non-nil if writing failed.

Thread Safety: Safe for concurrent use if the underlying Writer is safe.

type EffectSizeCategory 

type EffectSizeCategory int

EffectSizeCategory categorizes effect sizes using Cohen's conventions.

Description: 

Effect sizes help interpret the practical significance of differences.
Cohen's d thresholds: negligible (<0.2), small (0.2-0.5), medium (0.5-0.8),
large (≥0.8).

const (
	// EffectNegligible indicates Cohen's d < 0.2
	EffectNegligible EffectSizeCategory = iota
	// EffectSmall indicates Cohen's d between 0.2 and 0.5
	EffectSmall
	// EffectMedium indicates Cohen's d between 0.5 and 0.8
	EffectMedium
	// EffectLarge indicates Cohen's d >= 0.8
	EffectLarge
)

func CategorizeEffectSize 

func CategorizeEffectSize(d float64) EffectSizeCategory

CategorizeEffectSize returns the category for a given Cohen's d value.

Description: 

Categorizes the effect size using Cohen's conventional thresholds.
Uses the absolute value of d, so direction doesn't affect category.

Inputs:

  • d: Cohen's d value (can be positive or negative).

Outputs:

  • EffectSizeCategory: The corresponding category.

Example: 

cat := CategorizeEffectSize(0.3)  // EffectSmall
cat := CategorizeEffectSize(-0.9) // EffectLarge (uses absolute value)

func (EffectSizeCategory) String 

func (e EffectSizeCategory) String() string

String returns the string representation of the effect size category.

Outputs:

  • string: Human-readable category name.

type JSONReporter 

type JSONReporter struct {
	// contains filtered or unexported fields
}

JSONReporter outputs benchmark results as JSON.

Description: 

JSONReporter formats benchmark results as JSON for machine consumption.
Supports pretty-printing for human readability.

Thread Safety: Safe for concurrent use if the underlying Writer is safe.

func NewJSONReporter 

func NewJSONReporter(out io.Writer, pretty bool) *JSONReporter

NewJSONReporter creates a new JSON reporter.

Description: 

Creates a reporter that outputs benchmark results as JSON
to the specified writer. Pretty mode adds indentation.

Inputs:

  • out: Writer to output to. Must not be nil.
  • pretty: Whether to pretty-print the JSON with indentation.

Outputs:

  • *JSONReporter: The new reporter. Never nil.

Example: 

reporter := benchmark.NewJSONReporter(file, true)
reporter.Report(result)

Assumptions:

  • The writer is ready to accept output and will not be nil.

func (*JSONReporter) Report 

func (r *JSONReporter) Report(result *Result) error

Report writes a benchmark result as JSON.

Description: 

Converts the result to JSON-serializable form and writes it.

Inputs:

  • result: The benchmark result to report. Must not be nil.

Outputs:

  • error: Non-nil if conversion or writing failed.

Thread Safety: Safe for concurrent use if the underlying Writer is safe.

func (*JSONReporter) ReportAll 

func (r *JSONReporter) ReportAll(results []*Result) error

ReportAll writes multiple results as a JSON array.

Description: 

Converts all results to JSON-serializable form and writes as an array.

Inputs:

  • results: The benchmark results to report.

Outputs:

  • error: Non-nil if conversion or writing failed.

Thread Safety: Safe for concurrent use if the underlying Writer is safe.

func (*JSONReporter) ReportComparison 

func (r *JSONReporter) ReportComparison(comparison *ComparisonResult) error

ReportComparison writes a comparison result as JSON.

Description: 

Converts the comparison to JSON-serializable form and writes it.

Inputs:

  • comparison: The comparison result to report. Must not be nil.

Outputs:

  • error: Non-nil if conversion or writing failed.

Thread Safety: Safe for concurrent use if the underlying Writer is safe.

type LatencyStats 

type LatencyStats struct {
	// Min is the minimum latency observed.
	Min time.Duration

	// Max is the maximum latency observed.
	Max time.Duration

	// Mean is the arithmetic mean of latencies.
	Mean time.Duration

	// Median is the 50th percentile (P50).
	Median time.Duration

	// StdDev is the standard deviation.
	StdDev time.Duration

	// Variance is the variance (StdDev^2).
	Variance float64

	// P50 is the 50th percentile.
	P50 time.Duration

	// P90 is the 90th percentile.
	P90 time.Duration

	// P95 is the 95th percentile.
	P95 time.Duration

	// P99 is the 99th percentile.
	P99 time.Duration

	// P999 is the 99.9th percentile.
	P999 time.Duration
}

LatencyStats holds latency percentile statistics.

Description: 

LatencyStats provides comprehensive latency analysis including
min/max, mean/median, standard deviation, and percentiles.
All percentiles are calculated using linear interpolation.

Thread Safety: Safe for concurrent read access after creation.

func CalculateLatencyStats 

func CalculateLatencyStats(samples []time.Duration) (LatencyStats, error)

CalculateLatencyStats computes latency statistics from samples.

Description: 

Computes comprehensive statistics including min, max, mean, median,
standard deviation, variance, and percentiles (P50, P90, P95, P99, P999).
Percentiles are calculated using linear interpolation.

Inputs:

  • samples: Duration samples. Must not be empty.

Outputs:

  • LatencyStats: Computed statistics with all fields populated.
  • error: ErrNoSamples if samples is empty.

Thread Safety: This function is stateless and safe for concurrent use.

Example: 

samples := []time.Duration{10*time.Millisecond, 20*time.Millisecond, 30*time.Millisecond}
stats, err := CalculateLatencyStats(samples)
if err != nil {
    return err
}
fmt.Printf("P99: %v\n", stats.P99)

Assumptions:

  • All sample values are non-negative durations.

type MemoryStats 

type MemoryStats struct {
	// AllocBytes is the total bytes allocated per iteration (mean).
	AllocBytes uint64

	// AllocObjects is the total objects allocated per iteration (mean).
	AllocObjects uint64

	// HeapAllocBefore is the heap allocation before the benchmark.
	HeapAllocBefore uint64

	// HeapAllocAfter is the heap allocation after the benchmark.
	HeapAllocAfter uint64

	// HeapAllocDelta is the change in heap allocation.
	HeapAllocDelta int64

	// GCPauses is the number of GC pauses during the benchmark.
	GCPauses uint32

	// GCPauseTotal is the total GC pause time.
	GCPauseTotal time.Duration
}

MemoryStats holds memory usage statistics.

Description: 

MemoryStats captures heap allocation changes and GC behavior
during the benchmark run. Useful for identifying memory leaks
and understanding allocation patterns.

Thread Safety: Safe for concurrent read access after creation.

type Reporter 

type Reporter interface {
	// Report writes a single benchmark result to the output.
	//
	// Inputs:
	//   - result: The benchmark result to report. Must not be nil.
	//
	// Outputs:
	//   - error: Non-nil if writing failed.
	Report(result *Result) error

	// ReportComparison writes a comparison result to the output.
	//
	// Inputs:
	//   - comparison: The comparison result to report. Must not be nil.
	//
	// Outputs:
	//   - error: Non-nil if writing failed.
	ReportComparison(comparison *ComparisonResult) error

	// ReportAll writes multiple results to the output.
	//
	// Inputs:
	//   - results: The benchmark results to report.
	//
	// Outputs:
	//   - error: Non-nil if writing failed.
	ReportAll(results []*Result) error
}

Reporter formats and outputs benchmark results.

Description: 

Reporter provides a standardized interface for outputting benchmark
results in various formats. Implementations include console (human-readable)
and JSON (machine-readable) formats.

Thread Safety: Implementations must be safe for concurrent use.

type Result 

type Result struct {
	// Name is the name of the benchmarked component.
	Name string

	// Iterations is the number of iterations run.
	Iterations int

	// TotalDuration is the total time for all iterations.
	TotalDuration time.Duration

	// Latency holds latency statistics.
	Latency LatencyStats

	// Memory holds memory statistics (if collected).
	Memory *MemoryStats

	// Throughput holds throughput statistics.
	Throughput ThroughputStats

	// Errors is the number of iterations that resulted in errors.
	Errors int

	// ErrorRate is Errors / Iterations.
	ErrorRate float64

	// Timestamp is when the benchmark was run (Unix milliseconds UTC).
	Timestamp int64

	// Config is the configuration used for this benchmark.
	Config *Config

	// RawSamples holds the raw latency samples (before outlier removal).
	RawSamples []time.Duration

	// Samples holds the latency samples used for statistics.
	Samples []time.Duration
}

Result holds the results of a benchmark run.

Description: 

Result contains comprehensive statistics from a benchmark execution
including latency percentiles, throughput, memory usage, and error rates.
The raw samples are preserved for custom analysis.

Thread Safety: Safe for concurrent read access after creation.

type RunOption 

type RunOption func(*Config)

RunOption configures a benchmark run.

Description: 

RunOption functions modify the benchmark Config. They are applied
in order, so later options override earlier ones.

func WithCooldown 

func WithCooldown(d time.Duration) RunOption

WithCooldown sets the cooldown duration between warmup and measurement.

Description: 

Cooldown allows the system to settle after warmup before measurement
begins. Useful for letting GC complete and caches stabilize.

Inputs:

  • d: Cooldown duration. Must be non-negative; negative values are ignored.

Example: 

runner.Run(ctx, "algo", benchmark.WithCooldown(500*time.Millisecond))

func WithInputGenerator 

func WithInputGenerator(gen func() any) RunOption

WithInputGenerator sets a custom input generator.

Description: 

Provides a function that generates input for each benchmark iteration.
If not set, the component's property generator is used, or nil input.

Inputs:

  • gen: Function that generates input. Can return any value.

Example: 

runner.Run(ctx, "algo", benchmark.WithInputGenerator(func() any {
    return generateRandomInput(100)
}))

func WithIterationTimeout 

func WithIterationTimeout(d time.Duration) RunOption

WithIterationTimeout sets the per-iteration timeout.

Description: 

Sets the maximum time for a single iteration. Iterations exceeding
this timeout are counted as errors.

Inputs:

  • d: Per-iteration timeout. Must be positive; non-positive values are ignored.

Example: 

runner.Run(ctx, "algo", benchmark.WithIterationTimeout(5*time.Second))

func WithIterations 

func WithIterations(n int) RunOption

WithIterations sets the number of benchmark iterations.

Description: 

Configures the number of measured iterations to run. More iterations
provide more accurate statistics but take longer.

Inputs:

  • n: Number of iterations. Must be positive; non-positive values are ignored.

Example: 

runner.Run(ctx, "algo", benchmark.WithIterations(10000))

func WithMemoryCollection 

func WithMemoryCollection(enabled bool) RunOption

WithMemoryCollection enables or disables memory statistics collection.

Description: 

When enabled, the runner collects heap allocation statistics before
and after the benchmark. Disabling can slightly reduce overhead.

Inputs:

  • enabled: Whether to collect memory statistics.

Example: 

runner.Run(ctx, "algo", benchmark.WithMemoryCollection(false))

func WithOutlierRemoval 

func WithOutlierRemoval(enabled bool) RunOption

WithOutlierRemoval enables or disables outlier removal.

Description: 

When enabled, statistical outliers are removed before computing
latency statistics. Uses the IQR method with the configured threshold.

Inputs:

  • enabled: Whether to remove outliers.

Example: 

runner.Run(ctx, "algo", benchmark.WithOutlierRemoval(false))

func WithOutlierThreshold 

func WithOutlierThreshold(threshold float64) RunOption

WithOutlierThreshold sets the IQR threshold for outlier detection.

Description: 

Values outside [Q1 - threshold*IQR, Q3 + threshold*IQR] are considered
outliers. Common values: 1.5 (mild), 3.0 (extreme).

Inputs:

  • threshold: IQR multiplier. Must be positive; non-positive values are ignored.

Example: 

runner.Run(ctx, "algo", benchmark.WithOutlierThreshold(3.0))

func WithParallelism 

func WithParallelism(n int) RunOption

WithParallelism sets the number of concurrent benchmark goroutines.

Description: 

Runs multiple benchmark iterations concurrently. Useful for testing
thread safety and measuring contention. Note that parallel execution
may increase variance.

Inputs:

  • n: Number of concurrent goroutines. Must be positive; non-positive values are ignored.

Example: 

runner.Run(ctx, "algo", benchmark.WithParallelism(runtime.NumCPU()))

func WithTimeout 

func WithTimeout(d time.Duration) RunOption

WithTimeout sets the total benchmark timeout.

Description: 

Sets the maximum time for the entire benchmark run including warmup,
cooldown, and all iterations. The benchmark will stop early if exceeded.

Inputs:

  • d: Total timeout. Must be positive; non-positive values are ignored.

Example: 

runner.Run(ctx, "algo", benchmark.WithTimeout(10*time.Minute))

func WithWarmup 

func WithWarmup(n int) RunOption

WithWarmup sets the number of warmup iterations.

Description: 

Warmup iterations run before measurement to allow JIT, caches, and
other runtime optimizations to stabilize. Results are discarded.

Inputs:

  • n: Number of warmup iterations. Must be non-negative; negative values are ignored.

Example: 

runner.Run(ctx, "algo", benchmark.WithWarmup(1000))

type Runner 

type Runner struct {
	// contains filtered or unexported fields
}

Runner executes benchmarks against evaluable components.

Description: 

Runner provides the core benchmarking functionality, executing
iterations against registered components, collecting statistics,
and enabling comparisons between implementations.

Thread Safety: Safe for concurrent use.

func NewRunner 

func NewRunner(registry *eval.Registry) *Runner

NewRunner creates a new benchmark runner.

Description: 

Creates a runner that benchmarks components from the given registry.
The runner uses slog.Default() for logging; use SetLogger to override.

Inputs:

  • registry: The registry of evaluable components. Must not be nil.

Outputs:

  • *Runner: The new runner. Never nil.

Example: 

registry := eval.NewRegistry()
registry.MustRegister(myAlgorithm)
runner := benchmark.NewRunner(registry)

Assumptions:

  • Registry is initialized and will not be nil.

func (*Runner) Compare 

func (r *Runner) Compare(ctx context.Context, names []string, opts ...RunOption) (*ComparisonResult, error)

Compare runs benchmarks for multiple components and compares results.

Description: 

Runs benchmarks for each component with the same configuration,
then performs statistical comparison to determine if there's a
significant difference. Uses Welch's t-test and Cohen's d.

Inputs:

  • ctx: Context for cancellation and timeout. Must not be nil.
  • names: Names of components to compare. Must have at least 2.
  • opts: Optional configuration options (applied to all benchmarks).

Outputs:

  • *ComparisonResult: The comparison results with statistical analysis.
  • error: Non-nil if comparison could not be performed. Wraps underlying error.

Thread Safety: Safe for concurrent use.

Example: 

comparison, err := runner.Compare(ctx,
    []string{"cdcl_v1", "cdcl_v2"},
    benchmark.WithIterations(10000),
)
if err != nil {
    return fmt.Errorf("comparing algorithms: %w", err)
}
if comparison.Winner != "" {
    fmt.Printf("%s is %.2fx faster (p=%.4f)\n",
        comparison.Winner, comparison.Speedup, comparison.PValue)
}

Assumptions:

  • All named components are registered and comparable.

func (*Runner) Run 

func (r *Runner) Run(ctx context.Context, name string, opts ...RunOption) (*Result, error)

Run executes a benchmark for a single component.

Description: 

Executes a complete benchmark run including warmup, cooldown, and
measured iterations. Computes comprehensive statistics from the results.

Inputs:

  • ctx: Context for cancellation and timeout. Must not be nil.
  • name: The name of the component to benchmark. Must be registered.
  • opts: Optional configuration options.

Outputs:

  • *Result: The benchmark results with statistics. Never nil on success.
  • error: Non-nil if benchmark could not be run. Wraps underlying error.

Thread Safety: Safe for concurrent use.

Example: 

result, err := runner.Run(ctx, "cdcl",
    benchmark.WithIterations(10000),
    benchmark.WithWarmup(1000),
)
if err != nil {
    return fmt.Errorf("benchmark cdcl: %w", err)
}
fmt.Printf("P99: %v, Ops/sec: %.2f\n", result.Latency.P99, result.Throughput.OpsPerSecond)

Limitations:

  • Uses component's HealthCheck as the benchmarked operation.
  • Memory statistics may be affected by concurrent operations.

Assumptions:

  • Component is properly registered and its HealthCheck is meaningful.

func (*Runner) RunAll 

func (r *Runner) RunAll(ctx context.Context, opts ...RunOption) ([]*Result, error)

RunAll runs benchmarks for all registered components.

Description: 

Runs benchmarks for every component in the registry. Components
that fail are logged and skipped; the runner continues with others.

Inputs:

  • ctx: Context for cancellation and timeout. Must not be nil.
  • opts: Optional configuration options (applied to all benchmarks).

Outputs:

  • []*Result: Results for all successfully benchmarked components.
  • error: Non-nil only if benchmarking could not be started (nil context).

Thread Safety: Safe for concurrent use.

Example: 

results, err := runner.RunAll(ctx, benchmark.WithIterations(1000))
if err != nil {
    return fmt.Errorf("running all benchmarks: %w", err)
}
for _, result := range results {
    fmt.Printf("%s: P99=%v\n", result.Name, result.Latency.P99)
}

func (*Runner) SetLogger 

func (r *Runner) SetLogger(logger *slog.Logger)

SetLogger sets the logger for the runner.

Description: 

Replaces the runner's logger. Nil values are ignored.

Inputs:

  • logger: The logger to use. If nil, the current logger is retained.

Thread Safety: Safe for concurrent use.

type ThroughputStats 

type ThroughputStats struct {
	// OpsPerSecond is the number of operations per second.
	OpsPerSecond float64

	// BytesPerSecond is the throughput in bytes per second (if applicable).
	BytesPerSecond float64

	// ItemsPerSecond is the throughput in items per second (if applicable).
	ItemsPerSecond float64
}

ThroughputStats holds throughput statistics.

Description: 

ThroughputStats measures the rate of operations, optionally
including byte and item throughput if the benchmark provides
this information.

Thread Safety: Safe for concurrent read access after creation.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.