scale

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 2, 2026 License: MIT Imports: 11 Imported by: 0

Documentation

Overview

Package scale is the timed, resource-instrumented sibling of the bench package.

bench measures the deterministic, box-independent facts of a partition: bytes per URL, seen-set bits per URL, the achieved false-positive rate, rebalance bytes. Those numbers are exact and need no clock. scale measures the other half, the numbers a clock and a machine produce: wall and CPU time per stage, peak resident memory and heap, allocations per URL, disk bytes and fsync, and the throughput each stage sustains. It drives the real engine entry points (the same frontier.Seed and engine.Run the CLI calls), so the numbers are the engine's own numbers, not a reimplementation's.

Every timed number this package emits is only as trustworthy as the box it ran on, so a Result carries its provenance (box label, commit, corpus) and the harness refuses to stamp a run "measured" without a box label and a real corpus. The honesty rule from Spec 2071 doc 14 and Spec scale doc 00 holds here unchanged.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func HeldHeap 

func HeldHeap(ref any) uint64

HeldHeap measures the live heap an object holds after a forced GC, the resident footprint that survives between operations rather than the transient peak a one-shot encode spikes to. The seed stage uses it to separate the held frontier (the number the residency budget caps) from the checkpoint-encode high-water the peak RSS captures. It forces a GC so only reachable bytes remain, then reads the heap in-use. ref keeps the measured object alive across the GC so it is not collected before the read.

Types

type CPUTime 

type CPUTime struct {
	UserSeconds float64 `json:"user_seconds"`
	SysSeconds  float64 `json:"sys_seconds"`
}

CPUTime is the user and system CPU a stage burned, read from getrusage deltas. CPU-seconds, not wall, is the throughput metric of record (doc 14 section 2.3): wall moves with machine load, user CPU does not.

type CountingWriter 

type CountingWriter struct {
	W io.Writer
	N uint64
}

CountingWriter tallies the bytes written through it, the disk-accounting wrapper the harness wraps a checkpoint writer in so the write path's byte cost is the engine's real output, not an estimate.

func (*CountingWriter) Write 

func (c *CountingWriter) Write(p []byte) (int, error)

type DiskSummary 

type DiskSummary struct {
	BytesWritten uint64  `json:"bytes_written"`
	BytesRead    uint64  `json:"bytes_read"`
	OutputBytes  uint64  `json:"output_bytes"`
	FsyncCount   uint64  `json:"fsync_count"`
	FsyncP50Ms   float64 `json:"fsync_p50_ms"`
	FsyncP99Ms   float64 `json:"fsync_p99_ms"`
}

DiskSummary is the byte and durability side of a stage: bytes written and read through the harness's counting wrappers, the fsync count, and the output file size. fsync latency is captured by the SyncTimer when the stage flushes.

type IOSummary 

type IOSummary struct {
	MajorFaults uint64 `json:"major_faults"`
	MinorFaults uint64 `json:"minor_faults"`
	BlockIn     uint64 `json:"block_in"`
	BlockOut    uint64 `json:"block_out"`
}

IOSummary is the kernel-charged IO side of a stage, read as a getrusage delta. MajorFaults is the load-bearing metric for the file-backed engine: a major fault is a page the mmap read pulled off the disk, so its per-stage delta is that stage's real read-IO, the same figure /usr/bin/time labels "Major (requiring I/O) page faults". MinorFaults are served from the page cache with no disk touch, so the major/minor ratio reads how well the working set fits cache. BlockIn/BlockOut are the process's charged block-layer operations; on Linux they read zero for page-cache and mmap traffic (the block accounting is charged at fault time, not to the process), so MajorFaults, not BlockIn, is the mmap read signal to trust.

type LatencySummary 

type LatencySummary struct {
	Op            string  `json:"op"` // the measured operation, e.g. PutURL, Seen, NextBatch
	Samples       uint64  `json:"samples"`
	P50Ns         uint64  `json:"p50_ns"`
	P90Ns         uint64  `json:"p90_ns"`
	P99Ns         uint64  `json:"p99_ns"`
	MaxNs         uint64  `json:"max_ns"`
	EngineOpsPerS float64 `json:"engine_ops_per_second"`
}

LatencySummary is the structured form of a per-op latency histogram: the percentile edges and the max, in nanoseconds, plus the sample count and the engine-only rate (count over summed observed durations, isolating the measured op from the corpus parse the stage wall also includes). The command already renders these into the stage Notes; this carries the same figures machine-readably into the ledger JSON so a regression check can read p99 without parsing prose. Percentiles are the log2-bucket upper edge, so they are order-of-magnitude exact, which is what a hundreds-of-nanoseconds per-op latency needs.

type MemSummary 

type MemSummary struct {
	PeakRSSBytes    uint64  `json:"peak_rss_bytes"`
	PeakHeapInUse   uint64  `json:"peak_heap_inuse_bytes"`
	HeldHeapInUse   uint64  `json:"held_heap_inuse_bytes,omitempty"`
	TotalAllocBytes uint64  `json:"total_alloc_bytes"`
	Mallocs         uint64  `json:"mallocs"`
	NumGC           uint32  `json:"num_gc"`
	GCPauseTotalNs  uint64  `json:"gc_pause_total_ns"`
	GCCPUFraction   float64 `json:"gc_cpu_fraction"`
}

MemSummary is the memory side of a stage: the peak resident set the OS charged the process (getrusage max-rss, the real ceiling), the peak Go heap in use, the bytes and objects allocated over the stage, and the GC work those allocations drove. PeakRSS is the metric of record for the memory ceiling because it includes stacks, the runtime, and mmap, not just the heap.

type Provenance 

type Provenance struct {
	Box    string `json:"box"`    // the machine label, the box of record
	Commit string `json:"commit"` // the meguri commit the run was built from
	Corpus string `json:"corpus"` // the pinned corpus path or name
	GOOS   string `json:"goos"`
	GOARCH string `json:"goarch"`
	NumCPU int    `json:"num_cpu"`
}

Provenance is the stamp every measured number carries. A number without all of these is not admissible to the spec tables (scale doc 10).

type RSSSplit 

type RSSSplit struct {
	VMRSSBytes uint64 `json:"vm_rss_bytes"`
	AnonBytes  uint64 `json:"rss_anon_bytes"`
	FileBytes  uint64 `json:"rss_file_bytes"`
	ShmemBytes uint64 `json:"rss_shmem_bytes"`
	Available  bool   `json:"available"`
}

RSSSplit is the resident-set breakdown the file-backed engine is measured against (spec 2073 doc 08). The whole residency claim of the mmap design is that the multi-gigabyte .meguri file maps as reclaimable file-backed page cache, not anonymous heap, so the OOM budget is the anonymous part alone. Capturing only the total VmRSS reads a healthy file-backed run as a budget blowout, because the mapped file counts toward VmRSS but is evictable under pressure. AnonBytes is the number that must stay under the box budget; FileBytes is the reclaimable page cache the mapped base occupies and is expected to be large.

func ReadRSSSplit 

func ReadRSSSplit() RSSSplit

ReadRSSSplit reads the anon/file resident split from /proc/self/status. The fleet boxes of record are Linux, so this is the path the doc 08 residency numbers run through: RssAnon is the heap-and-stack term the box budget caps, RssFile is the mapped .meguri page cache the kernel reclaims first. The fields are reported in kibibytes, so each is scaled to bytes.

type Result 

type Result struct {
	Profile    string        `json:"profile"`
	Provenance Provenance    `json:"provenance"`
	StartedAt  string        `json:"started_at"`
	Stages     []StageResult `json:"stages"`
	PprofDir   string        `json:"pprof_dir,omitempty"`
}

Result aggregates a whole scale run: its provenance, the profile it ran, and one StageResult per pipeline stage. It is the immutable ledger entry (scale doc 10) and marshals straight to the JSON the regression ledger consumes.

func (Result) WriteHuman 

func (r Result) WriteHuman(w io.Writer)

WriteHuman prints a readable per-stage summary, the form an operator reads at the terminal. The honesty discipline shows in the header: a run without a box says so.

func (Result) WriteJSON 

func (r Result) WriteJSON(w io.Writer) error

WriteJSON marshals the result as indented JSON to w, the machine-readable ledger entry. The same bytes go to scale-results/ and feed the doc 14 As-Built table.

type Sampler 

type Sampler struct {
	// contains filtered or unexported fields
}

Sampler watches the Go heap on a ticker so a stage can report its peak heap in use, not just the heap at the end. It runs in its own goroutine between Start and Stop. Peak RSS comes from getrusage (the OS view, in measure.go), not from here; this catches the in-flight heap high-water mark a single end-of-stage read misses.

func NewSampler 

func NewSampler(interval time.Duration) *Sampler

NewSampler builds a heap sampler that reads every interval.

func (*Sampler) Start 

func (s *Sampler) Start()

Start begins sampling in a background goroutine.

func (*Sampler) Stop 

func (s *Sampler) Stop() uint64

Stop ends sampling and returns the peak heap-in-use it observed.

type StageResult 

type StageResult struct {
	Stage            string          `json:"stage"`
	URLs             int             `json:"urls"`
	WallSeconds      float64         `json:"wall_seconds"`
	CPU              CPUTime         `json:"cpu"`
	Mem              MemSummary      `json:"mem"`
	Disk             DiskSummary     `json:"disk"`
	IO               IOSummary       `json:"io,omitzero"`         // kernel-charged faults and block ops, the read-IO term
	URLsPerSecond    float64         `json:"urls_per_second"`     // count / wall, paired with CPU below
	URLsPerCPUSec    float64         `json:"urls_per_cpu_second"` // count / user CPU, the load-stable rate
	AllocBytesPerURL float64         `json:"alloc_bytes_per_url"`
	RSS              RSSSplit        `json:"rss_split,omitzero"` // anon/file resident split, the doc 08 residency term
	Latency          *LatencySummary `json:"latency,omitempty"`  // per-op histogram where the stage has a hot op
	Notes            string          `json:"notes,omitempty"`
}

StageResult is one measured pipeline stage: what it processed, how long it took in wall and CPU, the memory it cost, the disk and kernel IO it touched, the per-op latency where the stage has a hot op, and the throughput that implies. Derived per-URL numbers (alloc/url, bytes/url) are computed from the counts and the URL total so a reader does not recompute them. Network has no field: the scale path is offline (the seed is a local .seed and no stage fetches), so there is no network IO to measure and an empty field would only imply otherwise.

func StageResultFromIngest 

func StageResultFromIngest(urls int, fn func() (written uint64, err error)) (StageResult, error)

StageResultFromIngest measures an ingest-type stage: fn drives the durable store path with a resident budget, building and writing one record per URL so the cold bulk spills to the log rather than staying resident. urls is the input URL count, the denominator for ingest throughput and the bytes-on-disk ratio.

func StageResultFromInspect 

func StageResultFromInspect(urls int, fn func() (read uint64, err error)) (StageResult, error)

StageResultFromInspect measures an inspect-type stage: fn reads a .meguri checkpoint off disk and decodes its columns, returning the bytes it read. urls is the URL count the decode reconstructed, the denominator for the decode throughput. The bytes are accounted as read, not written, so this is the one stage that fills the disk read side of the ledger.

func StageResultFromLive 

func StageResultFromLive(urls int, fn func() (bytes uint64, err error)) (StageResult, error)

StageResultFromLive measures a live-engine stage: fn drives the file-backed engine (a bulk load that writes one .meguri, or a dedup/lookup pass over the mapped file), returning the file bytes it produced or touched. urls is the URL count fn processed. The caller stamps the anon/file RSS split onto the result, the residency term the mmap design is judged on (spec 2073 doc 08).

func StageResultFromRun 

func StageResultFromRun(urls int, fn func() (written uint64, err error)) (StageResult, error)

StageResultFromRun measures a run-type stage: fn drives the engine drain loop and urls is the resident URL count it drained.

func StageResultFromSeed 

func StageResultFromSeed(urls int, fn func() (written uint64, err error)) (StageResult, error)

StageResultFromSeed measures a seed-type stage: fn builds the frontier and returns the checkpoint bytes it wrote, and urls is the input URL count, the denominator for the intake throughput and alloc-per-URL numbers.

func WithURLs 

func WithURLs(res StageResult, urls int) StageResult

WithURLs restamps a stage's URL denominator and recomputes the per-URL and throughput fields from it. It is for a stage that does not know its work count until fn has run (an inspect decode learns the URL count only after decoding), so it measures with urls=0 and restamps the real count here. Wall, CPU, and allocation totals are unchanged; only the denominated ratios are recomputed.

type SyncTimer 

type SyncTimer struct {
	// contains filtered or unexported fields
}

SyncTimer records fsync latencies so a stage can report fsync count and the p50 and p99 of the flush. The device fsync floor is a property of the disk, measured on the box of record, not a number the engine chooses; this only times the calls the engine actually makes.

func (*SyncTimer) Summary 

func (s *SyncTimer) Summary() (count uint64, p50ms, p99ms float64)

Summary returns the fsync count and the p50 and p99 latency in milliseconds.

func (*SyncTimer) Time 

func (s *SyncTimer) Time(sync func() error) error

Time runs sync and records how long it took.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.