memusage

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 11, 2026 License: BSD-3-Clause Imports: 16 Imported by: 0

Documentation

Overview

Package memusage implements the core computation behind the /debug/memusage HTTP endpoint: it pairs heap-dump-derived reachability with heap-native pprof label recovery so callers can ask "how much heap is reachable from goroutines carrying these standard runtime/pprof labels?".

The package is split between pure computation (validation, label matching, object-set arithmetic, ComputeFromAnalysis) and a small production helper that captures a real heap dump, parses it, builds the graph, and recovers labels before delegating to ComputeFromAnalysis. Tests exercise the pure half with synthetic snapshotgraph.Analysis inputs so they do not need to write real heap dumps.

Index

Constants

View Source
const (
	DefaultMaxLabels           = 32
	DefaultMaxLabelKeyBytes    = 1024
	DefaultMaxLabelValueBytes  = 4096
	DefaultMaxRequestBodyBytes = 1 << 20 // 1 MiB
)

Variables

This section is empty.

Functions

func Handler

func Handler(compute ComputeFunc, hopts HandlerOptions) http.Handler

Handler returns an http.Handler that serves POST /debug/memusage by delegating heavy lifting to compute. The handler enforces a single in-flight request at a time and returns 429 if a second arrives.

HTTP semantics:

POST -> JSON success or JSON error
other methods -> 405 with Allow: POST

On structural errors (bad JSON, validation failures) the body is a JSON ErrorResponse with HTTP 400. Unsupported-runtime, string-missing, and structural label-recovery diagnostics map to HTTP 422.

func IntersectCountBytes

func IntersectCountBytes(g *snapshotgraph.Graph, a, b map[snapshotgraph.ObjectID]struct{}) (int, uint64)

IntersectCountBytes returns the size of a∩b and the sum of shallow sizes of objects in that intersection.

func LabelsMatch

func LabelsMatch(have, want map[string]string) bool

LabelsMatch reports whether the goroutine's recovered labels (have) contain every key/value pair in the selector (want). Extra labels on the goroutine are allowed.

func ObjectSetBytes

func ObjectSetBytes(g *snapshotgraph.Graph, set map[snapshotgraph.ObjectID]struct{}) uint64

ObjectSetBytes sums the shallow size of the objects in set, skipping IDs that are out of range (defensive; a normal Build never produces such IDs).

Types

type CaptureFailedError

type CaptureFailedError struct{ Cause error }

CaptureFailedError is returned when the heap dump could not be written. The handler maps it to HTTP 500 with code "capture_failed".

func (*CaptureFailedError) Error

func (e *CaptureFailedError) Error() string

func (*CaptureFailedError) Unwrap

func (e *CaptureFailedError) Unwrap() error

type ComputeFunc

type ComputeFunc func(ctx context.Context, req Request) (*Response, error)

ComputeFunc is the request-side surface used by the handler. Production wires it to (*Computer).Compute; tests can pass a fake.

type Computer

type Computer struct {
	Capturer  HeapDumpCapturer
	Recoverer LabelRecoverer
	Opts      Options
}

Computer captures, parses, and analyzes a heap dump to answer one /debug/memusage request. It is a value with default-zero fields wired to production implementations.

The process memory reader is opened at the start of each Compute call and closed before it returns; on Linux this re-reads /proc/self/maps per request, so mappings added after startup (dlopen, plugin.Open) are visible. Sources by platform:

  • Linux: /proc/self/mem.
  • FreeBSD: /proc/self/mem when procfs is mounted; otherwise the on-disk ELF executable (correct only for non-PIE binaries).
  • Darwin: Mach-O segments of the running executable with ASLR slide correction.
  • Windows: PE sections of the running executable with ASLR slide correction.

func NewComputer

func NewComputer(opts Options) *Computer

NewComputer returns a Computer wired to the runtime implementations.

func (*Computer) Close

func (c *Computer) Close() error

Close is a no-op retained for backward compatibility: Compute opens and closes its process memory reader per request and the Computer holds no other resources. Safe to call multiple times.

func (*Computer) Compute

func (c *Computer) Compute(ctx context.Context, req Request) (*Response, error)

Compute runs the full /debug/memusage pipeline (steps 2-6 are the shared analyse side implemented by AnalyzeDump):

  1. Capture a heap dump to a temp file.
  2. Parse it with ParseLazyContents so object content bytes are not retained in the Go heap; instead a ContentResolver fetches them from the dump file on demand during label recovery.
  3. Open an in-process address-space reader (Linux, macOS, FreeBSD, Windows; gated by Opts.DisableProcessMemoryReader) so the heap-label decoder can recover ordinary runtime/pprof string literals that live outside heap object contents. On FreeBSD the reader requires procfs mounted at /proc or a non-PIE binary; if neither condition holds, literal labels surface as string_missing.
  4. Resolve the runtime layout via runtimelayout.Lookup and recover pprof labels via the configured LabelRecoverer.
  5. If the runtime layout is unsupported, return UnsupportedRuntimeError before paying the graph-build cost.
  6. Build the object graph (structural; no reachability) and hand off to ComputeFromAnalysis, which performs the single reachability pass from matched roots.

ctx.Err() is checked between stages so a cancelled client does not pay for the parse/build work that follows WriteHeapDump (WriteHeapDump itself is stop-the-world and cannot be interrupted).

type DefaultLabelRecoverer

type DefaultLabelRecoverer struct{}

DefaultLabelRecoverer recovers labels via internal/heaplabels using the runtime layout chosen by internal/runtimelayout. When the runtime layout is unsupported, every goroutine is reported with StatusUnsupportedRuntime so the compute layer can short-circuit before the expensive graph build.

func (DefaultLabelRecoverer) Recover

Recover implements LabelRecoverer.

type Diagnostics

type Diagnostics struct {
	GoVersion string
	GOARCH    string

	// UnsupportedRuntime is true when the heap-native label decoder
	// could not locate runtime.g.labels for this dump, e.g. because the
	// Go version/GOARCH is not in the verified layout table.
	UnsupportedRuntime bool

	// StringMissingCount is the number of goroutines whose label decoding
	// failed because the string bytes (key or value) were not preserved
	// in heap dump object contents. Common for literal pprof.Labels.
	StringMissingCount int

	// FailedGoroutines is the number of goroutines whose label decode
	// failed for any reason (excluding StatusDecoded / StatusNoLabels /
	// StatusUnsupportedRuntime).
	FailedGoroutines int

	// StringMissingGIDs and FailedGIDs identify which goroutines failed
	// label decoding (FailedGIDs is a superset that includes the
	// string-missing goroutines). ComputeFromAnalysis uses them to ignore
	// failures on goroutines that are excluded from matching anyway
	// (system/background goroutines under the default options): an
	// excluded goroutine cannot change the match set, so its decode
	// failure must not fail the request. When both GID sets are nil but
	// the counts are non-zero (hand-constructed Diagnostics), every
	// counted failure is conservatively treated as match-eligible.
	StringMissingGIDs map[uint64]struct{}
	FailedGIDs        map[uint64]struct{}

	Warnings []string
}

Diagnostics summarizes label-recovery state needed by ComputeFromAnalysis to decide whether to return a success response or an error.

func DiagnosticsFromHeapLabels

func DiagnosticsFromHeapLabels(snap *heapsnapshot.HeapSnapshot, res heaplabels.Result) Diagnostics

DiagnosticsFromHeapLabels converts a heaplabels.Result into Diagnostics. It is the bridge between the heap-native label decoder and the compute layer.

type ErrorResponse

type ErrorResponse struct {
	Error     string   `json:"error"`
	Code      string   `json:"code"`
	GoVersion string   `json:"go_version,omitempty"`
	GOARCH    string   `json:"goarch,omitempty"`
	Warnings  []string `json:"warnings"`
}

ErrorResponse is the failure body for /debug/memusage. HTTP status is chosen by the handler, not this body.

func ErrorResponseFor added in v0.2.0

func ErrorResponseFor(err error) (int, *ErrorResponse)

ErrorResponseFor maps a compute/analyse error to the HTTP status code and JSON error body served by the /debug/memusage endpoint. The CLI uses the same mapping so error codes (unsupported_runtime, string_missing, label_recovery_failed, capture_failed, parse_failed, validation codes) are identical in both modes.

type HandlerOptions

type HandlerOptions struct {
	Opts                Options
	MaxRequestBodyBytes int64

	// Semaphore is the single-flight gate (capacity-1 channel). When nil
	// the handler creates a private one. Supplying a shared channel lets
	// multiple heap-dumping endpoints (e.g. /debug/memusage and
	// /debug/memusage/bundle) serialize against each other, since each
	// triggers a stop-the-world WriteHeapDump.
	Semaphore chan struct{}
}

HandlerOptions configures Handler.

type HeapDumpCapturer

type HeapDumpCapturer interface {
	CaptureHeapDump(ctx context.Context, gcBefore bool) (path string, cleanup func(), err error)
}

HeapDumpCapturer captures the calling process's heap into a file that the parser can stream from. Production uses RuntimeHeapDumpCapturer; tests can supply a fake that writes a prerecorded dump.

type LabelRecoverer

type LabelRecoverer interface {
	Recover(snap *heapsnapshot.HeapSnapshot, mem *heaplabels.Memory, extra addrspace.Reader) (heaplabels.Result, error)
}

LabelRecoverer is the interface used by Compute to recover heap-native pprof labels from a parsed snapshot. The default implementation is DefaultLabelRecoverer, which delegates to internal/heaplabels.

mem is an optional pre-built heap-memory source. When non-nil, the recoverer delegates structural reads to it (used by the lazy parse path so structural reads go through a heapdump.ContentResolver instead of materialized object content bytes). Pass nil to let the recoverer build an eager Memory from snap.Objects[i].Contents.

extra is an optional addrspace.Reader (typically an addrspace.ProcessReader on Linux/Darwin for /debug/memusage) consulted when label string bytes are not present in heap object contents. nil disables the fallback.

type LabelRecoveryFailedError

type LabelRecoveryFailedError struct {
	GoVersion        string
	GOARCH           string
	FailedGoroutines int
	Warnings         []string
}

LabelRecoveryFailedError is returned by ComputeFromAnalysis when heap-native label decode failed for reasons other than missing string bytes (e.g. g_object_missing, labels_object_missing, malformed map). The handler turns this into HTTP 422 with code "label_recovery_failed".

func (*LabelRecoveryFailedError) Error

func (e *LabelRecoveryFailedError) Error() string

type Options

type Options struct {
	// GCBeforeHeapDump triggers runtime.GC() immediately before
	// debug.WriteHeapDump in the production capture path. The pure
	// compute path ignores it.
	GCBeforeHeapDump bool

	// IncludeSystemGoroutines lets system/background goroutines (g0, GC
	// workers, finalizer goroutine, etc.) participate in label matching
	// like ordinary user goroutines. Default false.
	IncludeSystemGoroutines bool

	// DisableProcessMemoryReader turns off the in-process address-space
	// reader the handler opens on Linux, macOS, FreeBSD, and Windows so
	// the heap-label decoder can recover ordinary runtime/pprof string
	// literals that live outside heap object contents. When true, label
	// decoding sees heap object contents only, which makes literal-allocated
	// labels fail with string_missing. Useful in tests and in environments
	// where process memory access is undesirable.
	DisableProcessMemoryReader bool

	// Resource limits. Zero values fall back to the Default* constants.
	MaxLabels          int
	MaxLabelKeyBytes   int
	MaxLabelValueBytes int
}

Options tune the behavior of Compute and ComputeFromAnalysis.

type ParseFailedError

type ParseFailedError struct{ Cause error }

ParseFailedError is returned when the heap dump could not be parsed. The handler maps it to HTTP 500 with code "parse_failed".

func (*ParseFailedError) Error

func (e *ParseFailedError) Error() string

func (*ParseFailedError) Unwrap

func (e *ParseFailedError) Unwrap() error

type Request

type Request struct {
	Labels map[string]string `json:"labels"`
}

Request is the JSON body POSTed to /debug/memusage.

type Response

type Response struct {
	Labels map[string]string `json:"labels"`

	MatchedGoroutines int `json:"matched_goroutines"`

	ReachableObjects int    `json:"reachable_objects"`
	ReachableBytes   uint64 `json:"reachable_bytes"`

	GlobalOverlapObjects int    `json:"global_overlap_objects"`
	GlobalOverlapBytes   uint64 `json:"global_overlap_bytes"`

	SystemOverlapObjects int    `json:"system_overlap_objects"`
	SystemOverlapBytes   uint64 `json:"system_overlap_bytes"`
}

Response is the success body for /debug/memusage.

func AnalyzeDump added in v0.2.0

func AnalyzeDump(
	ctx context.Context,
	r io.Reader,
	ra io.ReaderAt,
	extra addrspace.Reader,
	extraWarnings []string,
	req Request,
	opts Options,
) (*Response, error)

AnalyzeDump runs the analyse side of the /debug/memusage pipeline against an already-captured heap dump: parse, heap-native label recovery, graph build, and the single reachability pass from matched roots. It is the shared core behind both the in-process endpoint ((*Computer).Compute) and the external analyser (bundle + CLI), so error codes and diagnostics are identical in both modes.

r and ra must view the same dump bytes: r is streamed by the parser while ra serves lazy object-content reads, and both must remain valid until AnalyzeDump returns (an *os.File satisfies both).

extra is the optional reader for label string bytes that live outside heap object contents (.rodata literals). In-process this is an addrspace.ProcessReader; offline it is a reader over the bundle's saved read-only segments. nil disables the fallback, in which case literal labels may fail with StringMissingError — identical to the DisableProcessMemoryReader path.

extraWarnings are appended to the response diagnostics (e.g. the "process memory reader unavailable..." phrasing when no extra reader could be provided).

func ComputeFromAnalysis

func ComputeFromAnalysis(
	req Request,
	analysis *snapshotgraph.Analysis,
	labelsByGID map[uint64]map[string]string,
	diag Diagnostics,
	opts Options,
) (*Response, error)

ComputeFromAnalysis is the pure core of /debug/memusage: it takes a structural object graph (built by snapshotgraph.Build, without the optional ComputeReachability pass), a precomputed labelsByGID map, and label diagnostics, and returns the response payload.

Reachability is traversed on demand: one BFS over the union of matched-goroutine roots, plus at most one BFS each for the global and system-goroutine overlap denominators (skipped entirely when nothing matches). This is the key reason Build no longer eagerly walks every goroutine: for a selector matching S of N goroutines we now pay O(reach(S)) instead of O(reach(N)).

A label-decode failure makes the match set non-authoritative only when the failed goroutine could have participated in matching: an undecodable eligible goroutine might also carry the requested labels, so a partial or zero match count is not returned as 200. Failures on goroutines that are excluded from matching (system/background under the default options) cannot change the match set and are ignored. Two distinct 422 codes:

  • eligible string-missing failures → StringMissingError (string bytes unavailable)
  • other eligible failures → LabelRecoveryFailedError

Validation runs first so direct callers (e.g. unit tests, future CLI adapters) cannot bypass the same checks the HTTP handler applies. Errors are reported via concrete types the handler can translate into HTTP status codes:

*ValidationError           -> 400
*UnsupportedRuntimeError   -> 422
*StringMissingError        -> 422
*LabelRecoveryFailedError  -> 422
other                      -> 500

type RuntimeHeapDumpCapturer

type RuntimeHeapDumpCapturer struct{}

RuntimeHeapDumpCapturer is the production implementation. It writes debug.WriteHeapDump output into a freshly created temp file and seeks back to offset 0 so the caller can parse directly.

func (RuntimeHeapDumpCapturer) CaptureHeapDump

func (RuntimeHeapDumpCapturer) CaptureHeapDump(ctx context.Context, gcBefore bool) (string, func(), error)

CaptureHeapDump implements HeapDumpCapturer.

type StringMissingError

type StringMissingError struct {
	GoVersion string
	GOARCH    string
	Warnings  []string
}

StringMissingError is returned by ComputeFromAnalysis when label decode failed because string bytes (key or value) were unavailable in the heap dump. The handler turns this into HTTP 422 with code "string_missing".

func (*StringMissingError) Error

func (e *StringMissingError) Error() string

type UnsupportedRuntimeError

type UnsupportedRuntimeError struct {
	GoVersion string
	GOARCH    string
}

UnsupportedRuntimeError is returned by ComputeFromAnalysis when label recovery is structurally impossible on the current runtime layout. The handler turns this into HTTP 422 with code "unsupported_runtime".

func (*UnsupportedRuntimeError) Error

func (e *UnsupportedRuntimeError) Error() string

type ValidationError

type ValidationError struct {
	Code string
	Msg  string
}

ValidationError describes a structural problem with the request body that should map to a 400 response.

func NewValidationError

func NewValidationError(code, msg string) *ValidationError

NewValidationError is a small helper so callers can construct one in a single line.

func ValidateRequest

func ValidateRequest(req *Request, opts Options) *ValidationError

ValidateRequest enforces request-body constraints: non-empty labels map, non-empty keys, and per-option limits on label count and key/value length.

func (*ValidationError) Error

func (e *ValidationError) Error() string

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL