bench

package
v0.12.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 12, 2026 License: Apache-2.0 Imports: 30 Imported by: 0

Documentation

Overview

Package bench is the funcd benchmark & sustainability harness (ADR-0040): it embeds the platform, drives the public data plane, and samples process memory to answer whether the platform is sustainable on the target box (RAM-bound, scale-to-zero). It compares the memory and file substrate (blob + JetStream bus); the store is memory in both.

Load generation + latency percentiles use fortio, RSS sampling uses gopsutil — both test-only libraries confined to this package (ADR-0051).

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func WriteFootprintReport

func WriteFootprintReport(dir string, r FootprintReport) (mdPath, jsonPath string, err error)

WriteFootprintReport writes the containerd footprint lane (ADR-0052) to dir as footprint-report.{md,json}, returning their paths. Unlike report.md (process RSS, ADR-0040), this measures the container cgroup — the honest absolute footprint over the production path.

func WritePoolReport

func WritePoolReport(dir string, reports []Report) (mdPath, jsonPath string, err error)

WritePoolReport writes the worker-pool comparison (ADR-0044) to dir as pool-report.{md,json}, returning their paths. Returns ("", "", nil) — no error, no files — when no report carries pooled numbers (pooled measurement skipped or failed). Separate from the substrate report so the density-vs-latency trade reads on its own.

func WritePyPoolReport

func WritePyPoolReport(dir string, c PyPoolComparison) (mdPath, jsonPath string, err error)

WritePyPoolReport writes the Python pool comparison (ADR-0050) to dir as py-pool-report.{md,json}.

func WriteReport

func WriteReport(dir string, reports []Report) (mdPath, jsonPath string, err error)

WriteReport writes the bench reports to dir as both machine-readable JSON (for a future CI gate) and a human-readable markdown sustainability report, returning their paths.

Types

type Backend

type Backend string

Backend selects the durable substrate compared by the bench. The store is memory in both (the file store = badger (ADR-0065), out of scope, ADR-0040); only the blob + JetStream bus move.

const (
	BackendMemory Backend = "memory" // gocloud mem:// blob + NATS MemoryStorage (InMemory())
	BackendFile   Backend = "file"   // gocloud file:// blob + NATS FileStorage
)

type Config

type Config struct {
	Backends    []Backend     // which substrates to run (default: memory, file)
	Concurrency int           // load workers
	Duration    time.Duration // load phase per backend
	Density     int           // extra warm functions deployed for the density sweep
	MemBudgetMB int           // function-memory budget for the verdict (e.g. 16384)
	TargetFns   int           // agents to size for (e.g. 100)
}

Config parameterizes a bench run.

type ContainerdConfig

type ContainerdConfig struct {
	Socket      string // containerd socket to dial (ADR-0054 Manager-provided)
	Namespace   string // isolated containerd namespace for the lane's throwaway containers (ADR-0055 dedicated-box); "" ⇒ "funcd-bench"
	Snapshotter string // overlayfs
	CNIBinDir   string // /opt/cni/bin
	CNIConfDir  string // funcd-written conflist dir
	SubnetCIDR  string // lateral bridge subnet
	ImagePrefix string // "funcd/runtime-"; image = prefix + runtime + ":latest"

	// ImageOverride maps a runtime to the registry ref the lane runs instead (ctrmanager.Config.ImageFor).
	ImageOverride map[string]string

	// ShimPath is the in-container shim cmdline token (the curated image's entrypoint —
	// "/opt/funcd/shim.mjs", ADR-0032). The container processes are host-visible PIDs, so
	// shimRSSMB(ShimPath) sums the SAME containers' RSS — a same-population cgroup-vs-RSS ratio.
	ShimPath string

	Density     int           // extra warm functions for the marginal-cgroup sweep
	MemBudgetMB int           // function-memory budget for the verdict
	TargetFns   int           // agents to size for
	Concurrency int           // load workers
	Duration    time.Duration // load phase
}

ContainerdConfig parameterizes the containerd footprint lane (ADR-0052). It mirrors the composition root's containerd wiring (cmd/funcd executionOptions) plus the bench's own measurement inputs. The lane boots funcd over the real containerd/crun production path and measures each function container's cgroup-v2 memory.current — the true footprint.

type FootprintReport

type FootprintReport struct {
	Skipped    bool   `json:"skipped"`
	SkipReason string `json:"skipReason,omitempty"`

	PerFunctionCgroupMB float64 `json:"perFunctionCgroupMB"` // marginal cgroup memory.current / fn (the true footprint)
	PerFunctionRSSMB    float64 `json:"perFunctionRSSMB"`    // marginal RSS of the SAME container processes (the baseline)
	CgroupOverRSS       float64 `json:"cgroupOverRSS"`       // container cgroup ÷ RSS — how much process RSS under-counts
	PlatformBaselineMB  float64 `json:"platformBaselineMB"`  // funcd control-plane RSS
	MaxDensity          int     `json:"maxDensity"`          // functions within MemBudgetMB, from the cgroup number (honest)
	FitsTarget          bool    `json:"fitsTarget"`          // TargetFns within MemBudgetMB? — NOT optimistic (cgroup-measured)
	RPS                 float64 `json:"rps"`
	Latency             Latency `json:"latency"`
}

FootprintReport is the containerd lane's result (ADR-0052). Skipped is true (with a reason and a nil error from RunContainerd) on any host that lacks the production path — the lane never fails the run. When not skipped, every memory number is the container cgroup memory.current, the honest production footprint.

func RunContainerd

func RunContainerd(ctx context.Context, cfg ContainerdConfig) (FootprintReport, error)

RunContainerd boots funcd over the real containerd/crun production path (ADR-0032 driver), deploys functions that run in real crun containers, and measures each container's cgroup-v2 memory.current — the honest production footprint (ADR-0052). It runs only on a Linux host (root) with a reachable containerd; on any missing precondition it returns FootprintReport{Skipped:true,…} with a nil error, never failing the run.

type Latency

type Latency struct {
	P50  time.Duration `json:"p50"`
	P90  time.Duration `json:"p90"`
	P99  time.Duration `json:"p99"`
	P999 time.Duration `json:"p999"`
	Max  time.Duration `json:"max"`
}

Latency is the tail-latency distribution of a load phase.

type PoolComparison

type PoolComparison struct {
	K         int     `json:"k"`
	Substrate Backend `json:"substrate"`
	// density (stable, primary)
	PerFunctionMB float64 `json:"perFunctionMB"`       // marginal RSS of one more separate shim
	PooledMBPerFn float64 `json:"pooledMBPerFunction"` // pool RSS / K
	DensityGain   float64 `json:"densityGain"`         // perFunctionMB / pooledMBPerFn
	// throughput (indicative) — both measured DIRECTLY on the shim port (no data plane)
	SingleDirectRPS    float64       `json:"singleDirectRPS"`    // one single-tenant shim, direct — HIGH VARIANCE
	SingleDirectP99    time.Duration `json:"singleDirectP99"`    //
	PooledRPS          float64       `json:"pooledRPS"`          // one handler in the pool, direct (stable)
	PooledP99          time.Duration `json:"pooledP99"`          //
	ThroughputRetained float64       `json:"throughputRetained"` // pooledRPS / singleDirectRPS — indicative only
}

PoolComparison is the head-to-head the separate pool report renders (ADR-0044). Two axes, NOT of equal reliability:

  • density (the primary, stable result) — K handlers in ONE worker_threads pool (pool RSS / K) vs K separate single-tenant shim processes (the marginal RSS of one more).
  • throughput/latency (indicative only) — the same handler hit DIRECTLY on each shim's own port (no gateway/activator/data plane). The pooled figure is stable run-to-run; the single-tenant baseline is ONE short-lived single-process shim and swings widely (≈15–65k req/s on a dev box) — single-threaded throughput is far more sensitive to JIT/core/GC timing than the pool's long-lived worker threads. So the pair shows throughput is COMPARABLE within that noise, not a precise ratio, and it is NOT a clean "worker_threads hop" isolation (the pool also gains a 2-core HTTP/handler pipeline). Read density as the decisive number.

type PyPoolComparison

type PyPoolComparison struct {
	K             int           `json:"k"`
	Python        string        `json:"python"`
	SoloMBPerFn   float64       `json:"soloMBPerFunction"`   // one solo shim's RSS = a function's marginal cost
	PooledMBPerFn float64       `json:"pooledMBPerFunction"` // pool host RSS / K
	DensityGain   float64       `json:"densityGain"`         // soloMBPerFn / pooledMBPerFn
	SoloRPS       float64       `json:"soloRPS"`
	SoloP99       time.Duration `json:"soloP99"`
	PooledRPS     float64       `json:"pooledRPS"`
	PooledP99     time.Duration `json:"pooledP99"`
}

PyPoolComparison is the Python solo-vs-pooled head-to-head (ADR-0050) — the Python analog of the Node WritePoolReport comparison. K trivial handlers in one subinterpreter pool host (pool.py) vs one solo shim (shim.py), both hit DIRECTLY on their own port (no data plane), so the difference isolates the pool host. Process RSS, dev machine (ADR-0040 caveats apply).

func MeasurePythonPool

func MeasurePythonPool(ctx context.Context, python, soloEntry, poolHostEntry string, k, concurrency int, duration time.Duration) (PyPoolComparison, error)

MeasurePythonPool runs the Python solo-vs-pooled comparison (ADR-0050). It launches the subinterpreter pool host (poolEntry) with k identical handlers, samples its RSS (→ MB/function, the density), and drives load at one handler; then — for a fair baseline — it launches ONE solo shim (shimEntry) hosting the same handler and drives load on the SAME direct path. The two run sequentially so neither contends for CPU. The python interpreter must be ≥3.14 (the pool host).

type Report

type Report struct {
	Backend             Backend       `json:"backend"`
	RPS                 float64       `json:"rps"`
	Latency             Latency       `json:"latency"`
	PlatformBaselineMB  float64       `json:"platformBaselineMB"`
	PerWorkerMB         float64       `json:"perWorkerMB"`
	IdleMB              float64       `json:"idleMB"`              // shim RSS while the only fn is scaled-to-zero (≈0 ⇒ reclaim works)
	ColdStart           time.Duration `json:"coldStart"`           // wake → first response (timed only after 0 replicas)
	PerFunctionMB       float64       `json:"perFunctionMB"`       // marginal slope: (RSS_at_K − perWorker) / K
	MaxDensity          int           `json:"maxDensity"`          // functions within MemBudgetMB
	FitsTarget          bool          `json:"fitsTarget"`          // TargetFns within MemBudgetMB? (optimistic — process RSS < cgroup)
	PooledPerFunctionMB float64       `json:"pooledPerFunctionMB"` // same K handlers in ONE worker_threads pool / K (ADR-0044); 0 if not measured
	PooledRPS           float64       `json:"pooledRPS"`           // throughput at one handler IN the pool, hit DIRECTLY (the worker_threads path); 0 if not measured
	PooledLatency       Latency       `json:"pooledLatency"`       // tail latency at one pooled handler, direct — the structured-clone cost
	SingleDirectRPS     float64       `json:"singleDirectRPS"`     // throughput of ONE single-tenant shim on the SAME direct path — the fair baseline for the pool
	SingleDirectLatency Latency       `json:"singleDirectLatency"` // tail latency of the single-tenant shim, same direct path
	PooledK             int           `json:"pooledK"`             // pool size the pooled numbers were measured at
}

Report is one substrate's sustainability numbers.

func Run

func Run(ctx context.Context, shimPath, poolShimPath string, cfg Config) ([]Report, error)

Run boots funcd once per Backend, drives the scenarios, and returns a Report per substrate. shimPath is the single-tenant node shim; poolShimPath is the pooled worker_threads shim (ADR-0044) — if non-empty, Run also measures pooled density for comparison. Node-gated.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL