Documentation
¶
Overview ¶
Package memusage implements the core computation behind the /debug/memusage HTTP endpoint: it pairs heap-dump-derived reachability with heap-native pprof label recovery so callers can ask "how much heap is reachable from goroutines carrying these standard runtime/pprof labels?".
The package is split between pure computation (validation, label matching, object-set arithmetic, ComputeFromAnalysis) and a small production helper that captures a real heap dump, parses it, builds the graph, and recovers labels before delegating to ComputeFromAnalysis. Tests exercise the pure half with synthetic snapshotgraph.Analysis inputs so they do not need to write real heap dumps.
Index ¶
- Constants
- func Handler(compute ComputeFunc, hopts HandlerOptions) http.Handler
- func IntersectCountBytes(g *snapshotgraph.Graph, a, b map[snapshotgraph.ObjectID]struct{}) (int, uint64)
- func LabelsMatch(have, want map[string]string) bool
- func ObjectSetBytes(g *snapshotgraph.Graph, set map[snapshotgraph.ObjectID]struct{}) uint64
- type CaptureFailedError
- type ComputeFunc
- type Computer
- type DefaultLabelRecoverer
- type Diagnostics
- type ErrorResponse
- type HandlerOptions
- type HeapDumpCapturer
- type LabelRecoverer
- type LabelRecoveryFailedError
- type Options
- type ParseFailedError
- type Request
- type Response
- type RuntimeHeapDumpCapturer
- type StringMissingError
- type UnsupportedRuntimeError
- type ValidationError
Constants ¶
const ( DefaultMaxLabels = 32 DefaultMaxLabelKeyBytes = 1024 DefaultMaxLabelValueBytes = 4096 DefaultMaxRequestBodyBytes = 1 << 20 // 1 MiB )
Variables ¶
This section is empty.
Functions ¶
func Handler ¶
func Handler(compute ComputeFunc, hopts HandlerOptions) http.Handler
Handler returns an http.Handler that serves POST /debug/memusage by delegating heavy lifting to compute. The handler enforces a single in-flight request at a time and returns 429 if a second arrives.
HTTP semantics:
POST -> JSON success or JSON error other methods -> 405 with Allow: POST
On structural errors (bad JSON, validation failures) the body is a JSON ErrorResponse with HTTP 400. Unsupported-runtime, string-missing, and structural label-recovery diagnostics map to HTTP 422.
func IntersectCountBytes ¶
func IntersectCountBytes(g *snapshotgraph.Graph, a, b map[snapshotgraph.ObjectID]struct{}) (int, uint64)
IntersectCountBytes returns the size of a∩b and the sum of shallow sizes of objects in that intersection.
func LabelsMatch ¶
LabelsMatch reports whether the goroutine's recovered labels (have) contain every key/value pair in the selector (want). Extra labels on the goroutine are allowed.
func ObjectSetBytes ¶
func ObjectSetBytes(g *snapshotgraph.Graph, set map[snapshotgraph.ObjectID]struct{}) uint64
ObjectSetBytes sums the shallow size of the objects in set, skipping IDs that are out of range (defensive; a normal Build never produces such IDs).
Types ¶
type CaptureFailedError ¶
type CaptureFailedError struct{ Cause error }
CaptureFailedError is returned when the heap dump could not be written. The handler maps it to HTTP 500 with code "capture_failed".
func (*CaptureFailedError) Error ¶
func (e *CaptureFailedError) Error() string
func (*CaptureFailedError) Unwrap ¶
func (e *CaptureFailedError) Unwrap() error
type ComputeFunc ¶
ComputeFunc is the request-side surface used by the handler. Production wires it to (*Computer).Compute; tests can pass a fake.
type Computer ¶
type Computer struct {
Capturer HeapDumpCapturer
Recoverer LabelRecoverer
Opts Options
}
Computer captures, parses, and analyzes a heap dump to answer one /debug/memusage request. It is a value with default-zero fields wired to production implementations.
The process memory reader is opened at the start of each Compute call and closed before it returns; on Linux this re-reads /proc/self/maps per request, so mappings added after startup (dlopen, plugin.Open) are visible. Sources by platform:
- Linux: /proc/self/mem.
- FreeBSD: /proc/self/mem when procfs is mounted; otherwise the on-disk ELF executable (correct only for non-PIE binaries).
- Darwin: Mach-O segments of the running executable with ASLR slide correction.
- Windows: PE sections of the running executable with ASLR slide correction.
func NewComputer ¶
NewComputer returns a Computer wired to the runtime implementations.
func (*Computer) Close ¶
Close is a no-op retained for backward compatibility: Compute opens and closes its process memory reader per request and the Computer holds no other resources. Safe to call multiple times.
func (*Computer) Compute ¶
Compute runs the full /debug/memusage pipeline (steps 2-6 are the shared analyse side implemented by AnalyzeDump):
- Capture a heap dump to a temp file.
- Parse it with ParseLazyContents so object content bytes are not retained in the Go heap; instead a ContentResolver fetches them from the dump file on demand during label recovery.
- Open an in-process address-space reader (Linux, macOS, FreeBSD, Windows; gated by Opts.DisableProcessMemoryReader) so the heap-label decoder can recover ordinary runtime/pprof string literals that live outside heap object contents. On FreeBSD the reader requires procfs mounted at /proc or a non-PIE binary; if neither condition holds, literal labels surface as string_missing.
- Resolve the runtime layout via runtimelayout.Lookup and recover pprof labels via the configured LabelRecoverer.
- If the runtime layout is unsupported, return UnsupportedRuntimeError before paying the graph-build cost.
- Build the object graph (structural; no reachability) and hand off to ComputeFromAnalysis, which performs the single reachability pass from matched roots.
ctx.Err() is checked between stages so a cancelled client does not pay for the parse/build work that follows WriteHeapDump (WriteHeapDump itself is stop-the-world and cannot be interrupted).
type DefaultLabelRecoverer ¶
type DefaultLabelRecoverer struct{}
DefaultLabelRecoverer recovers labels via internal/heaplabels using the runtime layout chosen by internal/runtimelayout. When the runtime layout is unsupported, every goroutine is reported with StatusUnsupportedRuntime so the compute layer can short-circuit before the expensive graph build.
func (DefaultLabelRecoverer) Recover ¶
func (r DefaultLabelRecoverer) Recover(snap *heapsnapshot.HeapSnapshot, mem *heaplabels.Memory, extra addrspace.Reader) (heaplabels.Result, error)
Recover implements LabelRecoverer.
type Diagnostics ¶
type Diagnostics struct {
GoVersion string
GOARCH string
// UnsupportedRuntime is true when the heap-native label decoder
// could not locate runtime.g.labels for this dump, e.g. because the
// Go version/GOARCH is not in the verified layout table.
UnsupportedRuntime bool
// StringMissingCount is the number of goroutines whose label decoding
// failed because the string bytes (key or value) were not preserved
// in heap dump object contents. Common for literal pprof.Labels.
StringMissingCount int
// FailedGoroutines is the number of goroutines whose label decode
// failed for any reason (excluding StatusDecoded / StatusNoLabels /
// StatusUnsupportedRuntime).
FailedGoroutines int
// StringMissingGIDs and FailedGIDs identify which goroutines failed
// label decoding (FailedGIDs is a superset that includes the
// string-missing goroutines). ComputeFromAnalysis uses them to ignore
// failures on goroutines that are excluded from matching anyway
// (system/background goroutines under the default options): an
// excluded goroutine cannot change the match set, so its decode
// failure must not fail the request. When both GID sets are nil but
// the counts are non-zero (hand-constructed Diagnostics), every
// counted failure is conservatively treated as match-eligible.
StringMissingGIDs map[uint64]struct{}
FailedGIDs map[uint64]struct{}
Warnings []string
}
Diagnostics summarizes label-recovery state needed by ComputeFromAnalysis to decide whether to return a success response or an error.
func DiagnosticsFromHeapLabels ¶
func DiagnosticsFromHeapLabels(snap *heapsnapshot.HeapSnapshot, res heaplabels.Result) Diagnostics
DiagnosticsFromHeapLabels converts a heaplabels.Result into Diagnostics. It is the bridge between the heap-native label decoder and the compute layer.
type ErrorResponse ¶
type ErrorResponse struct {
Error string `json:"error"`
Code string `json:"code"`
GoVersion string `json:"go_version,omitempty"`
GOARCH string `json:"goarch,omitempty"`
Warnings []string `json:"warnings"`
}
ErrorResponse is the failure body for /debug/memusage. HTTP status is chosen by the handler, not this body.
func ErrorResponseFor ¶ added in v0.2.0
func ErrorResponseFor(err error) (int, *ErrorResponse)
ErrorResponseFor maps a compute/analyse error to the HTTP status code and JSON error body served by the /debug/memusage endpoint. The CLI uses the same mapping so error codes (unsupported_runtime, string_missing, label_recovery_failed, capture_failed, parse_failed, validation codes) are identical in both modes.
type HandlerOptions ¶
type HandlerOptions struct {
Opts Options
MaxRequestBodyBytes int64
// Semaphore is the single-flight gate (capacity-1 channel). When nil
// the handler creates a private one. Supplying a shared channel lets
// multiple heap-dumping endpoints (e.g. /debug/memusage and
// /debug/memusage/bundle) serialize against each other, since each
// triggers a stop-the-world WriteHeapDump.
Semaphore chan struct{}
}
HandlerOptions configures Handler.
type HeapDumpCapturer ¶
type HeapDumpCapturer interface {
CaptureHeapDump(ctx context.Context, gcBefore bool) (path string, cleanup func(), err error)
}
HeapDumpCapturer captures the calling process's heap into a file that the parser can stream from. Production uses RuntimeHeapDumpCapturer; tests can supply a fake that writes a prerecorded dump.
type LabelRecoverer ¶
type LabelRecoverer interface {
Recover(snap *heapsnapshot.HeapSnapshot, mem *heaplabels.Memory, extra addrspace.Reader) (heaplabels.Result, error)
}
LabelRecoverer is the interface used by Compute to recover heap-native pprof labels from a parsed snapshot. The default implementation is DefaultLabelRecoverer, which delegates to internal/heaplabels.
mem is an optional pre-built heap-memory source. When non-nil, the recoverer delegates structural reads to it (used by the lazy parse path so structural reads go through a heapdump.ContentResolver instead of materialized object content bytes). Pass nil to let the recoverer build an eager Memory from snap.Objects[i].Contents.
extra is an optional addrspace.Reader (typically an addrspace.ProcessReader on Linux/Darwin for /debug/memusage) consulted when label string bytes are not present in heap object contents. nil disables the fallback.
type LabelRecoveryFailedError ¶
type LabelRecoveryFailedError struct {
GoVersion string
GOARCH string
FailedGoroutines int
Warnings []string
}
LabelRecoveryFailedError is returned by ComputeFromAnalysis when heap-native label decode failed for reasons other than missing string bytes (e.g. g_object_missing, labels_object_missing, malformed map). The handler turns this into HTTP 422 with code "label_recovery_failed".
func (*LabelRecoveryFailedError) Error ¶
func (e *LabelRecoveryFailedError) Error() string
type Options ¶
type Options struct {
// GCBeforeHeapDump triggers runtime.GC() immediately before
// debug.WriteHeapDump in the production capture path. The pure
// compute path ignores it.
GCBeforeHeapDump bool
// IncludeSystemGoroutines lets system/background goroutines (g0, GC
// workers, finalizer goroutine, etc.) participate in label matching
// like ordinary user goroutines. Default false.
IncludeSystemGoroutines bool
// DisableProcessMemoryReader turns off the in-process address-space
// reader the handler opens on Linux, macOS, FreeBSD, and Windows so
// the heap-label decoder can recover ordinary runtime/pprof string
// literals that live outside heap object contents. When true, label
// decoding sees heap object contents only, which makes literal-allocated
// labels fail with string_missing. Useful in tests and in environments
// where process memory access is undesirable.
DisableProcessMemoryReader bool
// Resource limits. Zero values fall back to the Default* constants.
MaxLabels int
MaxLabelKeyBytes int
MaxLabelValueBytes int
}
Options tune the behavior of Compute and ComputeFromAnalysis.
type ParseFailedError ¶
type ParseFailedError struct{ Cause error }
ParseFailedError is returned when the heap dump could not be parsed. The handler maps it to HTTP 500 with code "parse_failed".
func (*ParseFailedError) Error ¶
func (e *ParseFailedError) Error() string
func (*ParseFailedError) Unwrap ¶
func (e *ParseFailedError) Unwrap() error
type Response ¶
type Response struct {
Labels map[string]string `json:"labels"`
MatchedGoroutines int `json:"matched_goroutines"`
ReachableObjects int `json:"reachable_objects"`
ReachableBytes uint64 `json:"reachable_bytes"`
GlobalOverlapObjects int `json:"global_overlap_objects"`
GlobalOverlapBytes uint64 `json:"global_overlap_bytes"`
SystemOverlapObjects int `json:"system_overlap_objects"`
SystemOverlapBytes uint64 `json:"system_overlap_bytes"`
}
Response is the success body for /debug/memusage.
func AnalyzeDump ¶ added in v0.2.0
func AnalyzeDump( ctx context.Context, r io.Reader, ra io.ReaderAt, extra addrspace.Reader, extraWarnings []string, req Request, opts Options, ) (*Response, error)
AnalyzeDump runs the analyse side of the /debug/memusage pipeline against an already-captured heap dump: parse, heap-native label recovery, graph build, and the single reachability pass from matched roots. It is the shared core behind both the in-process endpoint ((*Computer).Compute) and the external analyser (bundle + CLI), so error codes and diagnostics are identical in both modes.
r and ra must view the same dump bytes: r is streamed by the parser while ra serves lazy object-content reads, and both must remain valid until AnalyzeDump returns (an *os.File satisfies both).
extra is the optional reader for label string bytes that live outside heap object contents (.rodata literals). In-process this is an addrspace.ProcessReader; offline it is a reader over the bundle's saved read-only segments. nil disables the fallback, in which case literal labels may fail with StringMissingError — identical to the DisableProcessMemoryReader path.
extraWarnings are appended to the response diagnostics (e.g. the "process memory reader unavailable..." phrasing when no extra reader could be provided).
func ComputeFromAnalysis ¶
func ComputeFromAnalysis( req Request, analysis *snapshotgraph.Analysis, labelsByGID map[uint64]map[string]string, diag Diagnostics, opts Options, ) (*Response, error)
ComputeFromAnalysis is the pure core of /debug/memusage: it takes a structural object graph (built by snapshotgraph.Build, without the optional ComputeReachability pass), a precomputed labelsByGID map, and label diagnostics, and returns the response payload.
Reachability is traversed on demand: one BFS over the union of matched-goroutine roots, plus at most one BFS each for the global and system-goroutine overlap denominators (skipped entirely when nothing matches). This is the key reason Build no longer eagerly walks every goroutine: for a selector matching S of N goroutines we now pay O(reach(S)) instead of O(reach(N)).
A label-decode failure makes the match set non-authoritative only when the failed goroutine could have participated in matching: an undecodable eligible goroutine might also carry the requested labels, so a partial or zero match count is not returned as 200. Failures on goroutines that are excluded from matching (system/background under the default options) cannot change the match set and are ignored. Two distinct 422 codes:
- eligible string-missing failures → StringMissingError (string bytes unavailable)
- other eligible failures → LabelRecoveryFailedError
Validation runs first so direct callers (e.g. unit tests, future CLI adapters) cannot bypass the same checks the HTTP handler applies. Errors are reported via concrete types the handler can translate into HTTP status codes:
*ValidationError -> 400 *UnsupportedRuntimeError -> 422 *StringMissingError -> 422 *LabelRecoveryFailedError -> 422 other -> 500
type RuntimeHeapDumpCapturer ¶
type RuntimeHeapDumpCapturer struct{}
RuntimeHeapDumpCapturer is the production implementation. It writes debug.WriteHeapDump output into a freshly created temp file and seeks back to offset 0 so the caller can parse directly.
func (RuntimeHeapDumpCapturer) CaptureHeapDump ¶
func (RuntimeHeapDumpCapturer) CaptureHeapDump(ctx context.Context, gcBefore bool) (string, func(), error)
CaptureHeapDump implements HeapDumpCapturer.
type StringMissingError ¶
StringMissingError is returned by ComputeFromAnalysis when label decode failed because string bytes (key or value) were unavailable in the heap dump. The handler turns this into HTTP 422 with code "string_missing".
func (*StringMissingError) Error ¶
func (e *StringMissingError) Error() string
type UnsupportedRuntimeError ¶
UnsupportedRuntimeError is returned by ComputeFromAnalysis when label recovery is structurally impossible on the current runtime layout. The handler turns this into HTTP 422 with code "unsupported_runtime".
func (*UnsupportedRuntimeError) Error ¶
func (e *UnsupportedRuntimeError) Error() string
type ValidationError ¶
ValidationError describes a structural problem with the request body that should map to a 400 response.
func NewValidationError ¶
func NewValidationError(code, msg string) *ValidationError
NewValidationError is a small helper so callers can construct one in a single line.
func ValidateRequest ¶
func ValidateRequest(req *Request, opts Options) *ValidationError
ValidateRequest enforces request-body constraints: non-empty labels map, non-empty keys, and per-option limits on label count and key/value length.
func (*ValidationError) Error ¶
func (e *ValidationError) Error() string