bubblepprof

bubblepprof

bubblepprof is a Go heap profiler that reports memory usage per pprof bubble — a group of goroutines that share the same runtime/pprof labels.

It works in two modes:

• In-process: POST /debug/memusage captures and analyses a heap dump inside the profiled process and returns JSON.
• External analyser (gops/pprof style): GET /debug/memusage/bundle streams a cheap capture bundle from the target; the bubblepprof CLI runs the same analysis out of process, so the target pays only for the stop-the-world dump, not for parsing or graph traversal.

What it does

The standard Go profiling tools (net/http/pprof, runtime/pprof) answer process-level questions: total heap size, allocation hotspots, goroutine count. They do not answer per-workload questions like "how much heap is my tenant=acme checkout job holding right now?"

bubblepprof answers that question via a single endpoint:

POST /debug/memusage

It captures a heap dump, recovers pprof labels from goroutine runtime state, and returns the shallow size of all heap objects reachable from goroutines whose labels match your selector.

How it differs from net/http/pprof

Install

go get github.com/NuperSu/bubblepprof@latest
go mod tidy

The module path is github.com/NuperSu/bubblepprof; the public API lives in github.com/NuperSu/bubblepprof/pkg/bubblepprof. There is nothing to install beyond go get — no codegen, no cgo, no extra runtime dependency.

For the external analyser CLI:

go install github.com/NuperSu/bubblepprof/cmd/bubblepprof@latest

Registering the endpoints

import "github.com/NuperSu/bubblepprof/pkg/bubblepprof"

mux := http.NewServeMux()
bubblepprof.Register(mux) // mounts /debug/memusage and /debug/memusage/bundle

Register makes the two endpoints share one single-flight gate (each triggers a stop-the-world heap dump, so only one runs at a time). To mount just one of them, use RegisterMemUsage(mux) or RegisterBundle(mux).

For custom options:

h := bubblepprof.MemUsageHandlerWithOptions(bubblepprof.MemUsageOptions{
    DisableGCBeforeHeapDump: false, // default: GC before dump
})
mux.Handle(bubblepprof.MemUsagePath, h)

Labeling work

Use the standard runtime/pprof API. No bubblepprof wrapper is required.

import (
    "context"
    "runtime/pprof"
)

pprof.Do(ctx, pprof.Labels("job", "42", "tenant", "acme"), func(ctx context.Context) {
    runJob(ctx)
})

For long-lived goroutines, stamp labels at goroutine start:

go func() {
    ctx := pprof.WithLabels(parent, pprof.Labels("role", "worker", "tenant", t))
    pprof.SetGoroutineLabels(ctx)
    runWorker(ctx)
}()

Querying memory usage

# How much heap do tenant=acme goroutines hold?
curl -s -X POST http://127.0.0.1:6060/debug/memusage \
  -H 'Content-Type: application/json' \
  -d '{"labels":{"tenant":"acme"}}' | jq .

Multiple labels narrow the match (AND semantics):

curl -s -X POST http://127.0.0.1:6060/debug/memusage \
  -H 'Content-Type: application/json' \
  -d '{"labels":{"tenant":"acme","tier":"enterprise"}}' | jq .

Response fields

Success (200 OK):

{
  "labels": {"tenant": "acme"},
  "matched_goroutines": 3,
  "reachable_objects": 18420,
  "reachable_bytes": 73400320,
  "global_overlap_objects": 12,
  "global_overlap_bytes": 49152,
  "system_overlap_objects": 3,
  "system_overlap_bytes": 12288
}

Diagnostic fields (go_version, goarch, warnings) appear on error responses only, not on successful measurements.

Error (422 Unprocessable Entity):

{
  "error": "heap-native pprof label recovery is unsupported for this Go runtime (go1.25 arm64)",
  "code": "unsupported_runtime",
  "go_version": "go1.25",
  "goarch": "arm64",
  "warnings": []
}

The code field distinguishes failure modes:

External analyser mode

In-process analysis allocates roughly the workload's live heap again (transiently) inside the profiled process. When that is unacceptable — large heaps, tight memory limits — move the analysis out of process: the target serves a bundle and the CLI does the rest.

# One step: fetch a bundle from the target and analyse it
bubblepprof memusage http://127.0.0.1:6060 -labels tenant=acme

# Or two steps: save the bundle, analyse it later (or elsewhere)
bubblepprof fetch http://127.0.0.1:6060 -o app.tar
bubblepprof memusage app.tar -labels tenant=acme,tier=enterprise

# Use -label repeatedly when a label value contains a comma
bubblepprof memusage app.tar -label tenant=acme -label note=a,b

The output is the same JSON (and the same error codes) as POST /debug/memusage.

A bundle is a plain tar stream containing:

• heap.dump — runtime/debug.WriteHeapDump output (reachability + label structures);
• rodata/* — a snapshot of the target's read-only memory segments, so ordinary string-literal labels like pprof.Labels("job", "42") can be recovered out of process (their bytes live in .rodata, not the heap);
• meta.json — format version, capture time, producer, runtime info.

When the rodata snapshot is unavailable on the target platform, the bundle remains analysable and literal labels surface as string_missing — exactly like the in-process endpoint with DisableProcessMemoryReader.

On the target, only debug.WriteHeapDump (stop-the-world) plus file streaming runs; no parsing, no graph build. The endpoint accepts ?gc=0|1 to override the pre-dump GC per request.

Running the demo

go run ./examples/order_pipeline

Then in another terminal:

# Query memory held by the atlas-bikes tenant aggregator
curl -s -X POST http://127.0.0.1:6060/debug/memusage \
  -H 'Content-Type: application/json' \
  -d '{"labels":{"tenant":"atlas-bikes"}}' | jq .

# Query memory held by notification workers
curl -s -X POST http://127.0.0.1:6060/debug/memusage \
  -H 'Content-Type: application/json' \
  -d '{"labels":{"component":"async","role":"notification_worker"}}' | jq .

# A missing label returns zero matches, not an error
curl -s -X POST http://127.0.0.1:6060/debug/memusage \
  -H 'Content-Type: application/json' \
  -d '{"labels":{"tenant":"nonexistent"}}' | jq .

The example also exposes /debug/pprof and /stats:

curl http://127.0.0.1:6060/stats
go tool pprof http://127.0.0.1:6060/debug/pprof/heap

Load-test example (profiler_load)

profiler_load is a stress test that sustains a large number of labeled goroutines and a configurable resident heap, giving /debug/memusage meaningful bytes to attribute to each bubble.

# Default: 768 workers, 160 MiB pinned resident heap
go run ./examples/profiler_load

# Heavier load: 1024 workers, 500 MiB pinned heap, 2-minute run
go run ./examples/profiler_load -mem-mb 500 -workers 1024 -duration 2m

Six worker types are distributed round-robin — each carries a role label, a pool label (compute, memory, or pipeline), and a shard label:

# Heap reachable from all compute workers (cpu-hash + sorter)
curl -s -X POST http://127.0.0.1:6060/debug/memusage \
  -H 'Content-Type: application/json' \
  -d '{"labels":{"pool":"compute"}}' | jq .

# Heap reachable from allocator workers (high churn pool)
curl -s -X POST http://127.0.0.1:6060/debug/memusage \
  -H 'Content-Type: application/json' \
  -d '{"labels":{"role":"allocator"}}' | jq .

# Heap reachable from heap-scan workers (resident heap visible from matched roots)
curl -s -X POST http://127.0.0.1:6060/debug/memusage \
  -H 'Content-Type: application/json' \
  -d '{"labels":{"role":"heap-scan"}}' | jq .

# Pipeline workers (channel producers + mutex consumers)
curl -s -X POST http://127.0.0.1:6060/debug/memusage \
  -H 'Content-Type: application/json' \
  -d '{"labels":{"pool":"pipeline"}}' | jq .

The process prints goroutine count, heap stats, and ops/sec every two seconds. Use it to stress-test label recovery and reachability performance on realistic heap sizes.

Multi-tenant example (log_ingest)

log_ingest models a multi-tenant in-memory log ingester (Loki / Cortex / Mimir flavor), the canonical place where "how much heap is tenant X holding right now?" is a real operational question. It is the example that best shows what bubblepprof is for.

# Default: ~736 MiB live (24 ingesters * 24 MiB private + 160 MiB shared dictionary)
go run ./examples/log_ingest

# Heavier:
go run ./examples/log_ingest -tenants 8 -shards 4 -dict-mb 256

One long-lived ingester goroutine per (tenant, stream, shard) privately owns a ring of in-memory log chunks held only on its own stack — no chunk is shared between workloads. Every ingester also references a single process-wide interned-label dictionary, which is the only shared memory and the sole source of global_overlap. Labels form a six-dimension hierarchy you can drill through:

service = log-ingester
tenant  = atlas-bikes | globex | initech | umbrella-corp | ...
stream  = app | nginx | kernel | audit | ...
region  = us-east | eu-west | ap-south
tier    = enterprise | standard
shard   = 0..shards-1

The key insight: private heap = reachable_bytes − global_overlap_bytes. As you add labels the match narrows, reachable_bytes shrinks with it, and global_overlap_bytes (the shared dictionary) stays roughly constant:

# Everything tenant=atlas-bikes holds, across all its streams and shards:
curl -s -XPOST 127.0.0.1:6060/debug/memusage -d '{"labels":{"tenant":"atlas-bikes"}}' | jq .

# Same question answered out of process via the external analyser:
bubblepprof memusage http://127.0.0.1:6060 -labels tenant=atlas-bikes

# Narrow to one stream, then one shard — reachable falls, global_overlap holds steady:
curl -s -XPOST 127.0.0.1:6060/debug/memusage -d '{"labels":{"tenant":"atlas-bikes","stream":"app"}}' | jq .
curl -s -XPOST 127.0.0.1:6060/debug/memusage -d '{"labels":{"tenant":"atlas-bikes","stream":"app","shard":"0"}}' | jq .

# Cross-cutting views that don't follow the tenant axis at all:
curl -s -XPOST 127.0.0.1:6060/debug/memusage -d '{"labels":{"region":"eu-west"}}'  | jq .
curl -s -XPOST 127.0.0.1:6060/debug/memusage -d '{"labels":{"tier":"enterprise"}}' | jq .

# Everything: global_overlap ~= dictionary size, reachable ~= all chunks + dictionary:
curl -s -XPOST 127.0.0.1:6060/debug/memusage -d '{"labels":{"service":"log-ingester"}}' | jq .

Example progression (small run, -dict-mb 32 -ring 8 -chunk-kb 256), reading matched / reachable / global_overlap / private in MiB:

{tenant:atlas-bikes}              -> 4 / 41 / 33 /  8
{tenant:atlas-bikes,stream:app}   -> 2 / 37 / 33 /  4
{...,stream:app,shard:0}          -> 1 / 35 / 33 /  2
{tier:enterprise}                 -> 8 / 49 / 33 / 16
{service:log-ingester}            -> all / 57 / 33 / 24

The example sets no finalizers on chunks (a finalizer would make an object a global root and pollute per-tenant attribution) — which is exactly what keeps reachable − global_overlap equal to the private heap. Interface data-word reachability is fully preserved for current Go versions (1.24+): the runtime's GC bitmap already emits interface data words as ordinary pointer slots, so no concrete-type restriction is needed on the data path.

Security and performance

/debug/memusage is equivalent in sensitivity to /debug/pprof/heap. Every call:

• stops all goroutines (runtime/debug.WriteHeapDump),
• writes a full heap dump to a temporary file,
• reads the file back, and
• deletes the temporary file.

Latency is proportional to live heap size. Concurrent callers receive 429 Too Many Requests.

/debug/memusage/bundle hands out the full heap dump and read-only program memory of the process — the same data exposure as /debug/pprof heap and goroutine dumps combined.

Protect both endpoints with the same controls you apply to /debug/pprof. Do not expose them to untrusted callers.

Current limitations

See docs/limitations.md for a complete list. Key points:

• Heap-native label recovery is verified for go1.24.*–go1.26.* on Linux, macOS, Windows, and FreeBSD (amd64, arm64, arm, 386). Experimental tip (go1.27-devel) support is tested in CI but not required. Other Go versions return unsupported_runtime.
• Ordinary string literal labels are recovered via the in-process reader on Linux, macOS, FreeBSD, and Windows. On FreeBSD specifically, recovery requires either procfs mounted at /proc (the reader uses /proc/self/mem) or a non-PIE binary (the reader falls back to the on-disk ELF; PIE would shift the runtime addresses and break that fallback). On other platforms — or on FreeBSD when neither condition holds — literal labels return string_missing.
• Sizes are shallow (the object itself, not transitive) and counts are BFS-reachable from the matched goroutine roots, not total process heap.
• Global and system overlap is reported separately; it is not subtracted automatically.

Development tools

# Probe runtime.g.labels offset; prints a pasteable runtimelayout.TableEntry
go run ./cmd/labeloffsetprobe

# Run all tests
go test ./...

# Vet
go vet ./...

License

BSD 3-Clause. See LICENSE.