bubblepprof

bubblepprof

bubblepprof is an in-process Go heap profiler that reports memory usage per pprof bubble — a group of goroutines that share the same runtime/pprof labels.

What it does

The standard Go profiling tools (net/http/pprof, runtime/pprof) answer process-level questions: total heap size, allocation hotspots, goroutine count. They do not answer per-workload questions like "how much heap is my tenant=acme checkout job holding right now?"

bubblepprof answers that question via a single endpoint:

POST /debug/memusage

It captures a heap dump, recovers pprof labels from goroutine runtime state, and returns the shallow size of all heap objects reachable from goroutines whose labels match your selector.

How it differs from net/http/pprof

Install

go get github.com/NuperSu/bubblepprof@latest
go mod tidy

The module path is github.com/NuperSu/bubblepprof; the public API lives in github.com/NuperSu/bubblepprof/pkg/bubblepprof. There is nothing to install beyond go get — no codegen, no cgo, no extra runtime dependency.

Registering the endpoint

import "github.com/NuperSu/bubblepprof/pkg/bubblepprof"

mux := http.NewServeMux()
bubblepprof.RegisterMemUsage(mux) // mounts at /debug/memusage

For custom options:

h := bubblepprof.MemUsageHandlerWithOptions(bubblepprof.MemUsageOptions{
    DisableGCBeforeHeapDump: false, // default: GC before dump
})
mux.Handle(bubblepprof.MemUsagePath, h)

Labeling work

Use the standard runtime/pprof API. No bubblepprof wrapper is required.

import (
    "context"
    "runtime/pprof"
)

pprof.Do(ctx, pprof.Labels("job", "42", "tenant", "acme"), func(ctx context.Context) {
    runJob(ctx)
})

For long-lived goroutines, stamp labels at goroutine start:

go func() {
    ctx := pprof.WithLabels(parent, pprof.Labels("role", "worker", "tenant", t))
    pprof.SetGoroutineLabels(ctx)
    runWorker(ctx)
}()

Querying memory usage

# How much heap do tenant=acme goroutines hold?
curl -s -X POST http://127.0.0.1:6060/debug/memusage \
  -H 'Content-Type: application/json' \
  -d '{"labels":{"tenant":"acme"}}' | jq .

Multiple labels narrow the match (AND semantics):

curl -s -X POST http://127.0.0.1:6060/debug/memusage \
  -H 'Content-Type: application/json' \
  -d '{"labels":{"tenant":"acme","tier":"enterprise"}}' | jq .

Response fields

Success (200 OK):

{
  "labels": {"tenant": "acme"},
  "matched_goroutines": 3,
  "reachable_objects": 18420,
  "reachable_bytes": 73400320,
  "global_overlap_objects": 12,
  "global_overlap_bytes": 49152,
  "system_overlap_objects": 3,
  "system_overlap_bytes": 12288
}

Diagnostic fields (go_version, goarch, warnings) appear on error responses only, not on successful measurements.

Error (422 Unprocessable Entity):

{
  "error": "heap-native pprof label recovery is unsupported for this Go runtime (go1.25 arm64)",
  "code": "unsupported_runtime",
  "go_version": "go1.25",
  "goarch": "arm64",
  "warnings": []
}

The code field distinguishes failure modes:

Running the demo

go run ./examples/order_pipeline

Then in another terminal:

# Query memory held by the atlas-bikes tenant aggregator
curl -s -X POST http://127.0.0.1:6060/debug/memusage \
  -H 'Content-Type: application/json' \
  -d '{"labels":{"tenant":"atlas-bikes"}}' | jq .

# Query memory held by notification workers
curl -s -X POST http://127.0.0.1:6060/debug/memusage \
  -H 'Content-Type: application/json' \
  -d '{"labels":{"component":"async","role":"notification_worker"}}' | jq .

# A missing label returns zero matches, not an error
curl -s -X POST http://127.0.0.1:6060/debug/memusage \
  -H 'Content-Type: application/json' \
  -d '{"labels":{"tenant":"nonexistent"}}' | jq .

The example also exposes /debug/pprof and /stats:

curl http://127.0.0.1:6060/stats
go tool pprof http://127.0.0.1:6060/debug/pprof/heap

Load-test example (profiler_load)

profiler_load is a stress test that sustains a large number of labeled goroutines and a configurable resident heap, giving /debug/memusage meaningful bytes to attribute to each bubble.

# Default: 768 workers, 160 MiB pinned resident heap
go run ./examples/profiler_load

# Heavier load: 1024 workers, 500 MiB pinned heap, 2-minute run
go run ./examples/profiler_load -mem-mb 500 -workers 1024 -duration 2m

Six worker types are distributed round-robin — each carries a role label, a pool label (compute, memory, or pipeline), and a shard label:

# Heap reachable from all compute workers (cpu-hash + sorter)
curl -s -X POST http://127.0.0.1:6060/debug/memusage \
  -H 'Content-Type: application/json' \
  -d '{"labels":{"pool":"compute"}}' | jq .

# Heap reachable from allocator workers (high churn pool)
curl -s -X POST http://127.0.0.1:6060/debug/memusage \
  -H 'Content-Type: application/json' \
  -d '{"labels":{"role":"allocator"}}' | jq .

# Heap reachable from heap-scan workers (resident heap visible from matched roots)
curl -s -X POST http://127.0.0.1:6060/debug/memusage \
  -H 'Content-Type: application/json' \
  -d '{"labels":{"role":"heap-scan"}}' | jq .

# Pipeline workers (channel producers + mutex consumers)
curl -s -X POST http://127.0.0.1:6060/debug/memusage \
  -H 'Content-Type: application/json' \
  -d '{"labels":{"pool":"pipeline"}}' | jq .

The process prints goroutine count, heap stats, and ops/sec every two seconds. Use it to stress-test label recovery and reachability performance on realistic heap sizes.

Multi-tenant example (log_ingest)

log_ingest models a multi-tenant in-memory log ingester (Loki / Cortex / Mimir flavor), the canonical place where "how much heap is tenant X holding right now?" is a real operational question. It is the example that best shows what bubblepprof is for.

# Default: ~736 MiB live (24 ingesters * 24 MiB private + 160 MiB shared dictionary)
go run ./examples/log_ingest

# Heavier:
go run ./examples/log_ingest -tenants 8 -shards 4 -dict-mb 256

One long-lived ingester goroutine per (tenant, stream, shard) privately owns a ring of in-memory log chunks held only on its own stack — no chunk is shared between workloads. Every ingester also references a single process-wide interned-label dictionary, which is the only shared memory and the sole source of global_overlap. Labels form a six-dimension hierarchy you can drill through:

service = log-ingester
tenant  = atlas-bikes | globex | initech | umbrella-corp | ...
stream  = app | nginx | kernel | audit | ...
region  = us-east | eu-west | ap-south
tier    = enterprise | standard
shard   = 0..shards-1

The key insight: private heap = reachable_bytes − global_overlap_bytes. As you add labels the match narrows, reachable_bytes shrinks with it, and global_overlap_bytes (the shared dictionary) stays roughly constant:

# Everything tenant=atlas-bikes holds, across all its streams and shards:
curl -s -XPOST 127.0.0.1:6060/debug/memusage -d '{"labels":{"tenant":"atlas-bikes"}}' | jq .

# Narrow to one stream, then one shard — reachable falls, global_overlap holds steady:
curl -s -XPOST 127.0.0.1:6060/debug/memusage -d '{"labels":{"tenant":"atlas-bikes","stream":"app"}}' | jq .
curl -s -XPOST 127.0.0.1:6060/debug/memusage -d '{"labels":{"tenant":"atlas-bikes","stream":"app","shard":"0"}}' | jq .

# Cross-cutting views that don't follow the tenant axis at all:
curl -s -XPOST 127.0.0.1:6060/debug/memusage -d '{"labels":{"region":"eu-west"}}'  | jq .
curl -s -XPOST 127.0.0.1:6060/debug/memusage -d '{"labels":{"tier":"enterprise"}}' | jq .

# Everything: global_overlap ~= dictionary size, reachable ~= all chunks + dictionary:
curl -s -XPOST 127.0.0.1:6060/debug/memusage -d '{"labels":{"service":"log-ingester"}}' | jq .

Example progression (small run, -dict-mb 32 -ring 8 -chunk-kb 256), reading matched / reachable / global_overlap / private in MiB:

{tenant:atlas-bikes}              -> 4 / 41 / 33 /  8
{tenant:atlas-bikes,stream:app}   -> 2 / 37 / 33 /  4
{...,stream:app,shard:0}          -> 1 / 35 / 33 /  2
{tier:enterprise}                 -> 8 / 49 / 33 / 16
{service:log-ingester}            -> all / 57 / 33 / 24

The example sets no finalizers on chunks (a finalizer would make an object a global root and pollute per-tenant attribution) — which is exactly what keeps reachable − global_overlap equal to the private heap. Interface data-word reachability is fully preserved for current Go versions (1.24+): the runtime's GC bitmap already emits interface data words as ordinary pointer slots, so no concrete-type restriction is needed on the data path.

Security and performance

/debug/memusage is equivalent in sensitivity to /debug/pprof/heap. Every call:

• stops all goroutines (runtime/debug.WriteHeapDump),
• writes a full heap dump to a temporary file,
• reads the file back, and
• deletes the temporary file.

Latency is proportional to live heap size. Concurrent callers receive 429 Too Many Requests.

Protect the endpoint with the same controls you apply to /debug/pprof. Do not expose it to untrusted callers.

Current limitations

See docs/limitations.md for a complete list. Key points:

• Heap-native label recovery is verified for go1.24.*–go1.26.* on Linux, macOS, Windows, and FreeBSD (amd64, arm64, arm, 386). Experimental tip (go1.27-devel) support is tested in CI but not required. Other Go versions return unsupported_runtime.
• Ordinary string literal labels are recovered via the in-process reader on Linux, macOS, FreeBSD, and Windows. On FreeBSD specifically, recovery requires either procfs mounted at /proc (the reader uses /proc/self/mem) or a non-PIE binary (the reader falls back to the on-disk ELF; PIE would shift the runtime addresses and break that fallback). On other platforms — or on FreeBSD when neither condition holds — literal labels return string_missing.
• Sizes are shallow (the object itself, not transitive) and counts are BFS-reachable from the matched goroutine roots, not total process heap.
• Global and system overlap is reported separately; it is not subtracted automatically.

Development tools

# Probe runtime.g.labels offset; prints a pasteable runtimelayout.TableEntry
go run ./cmd/labeloffsetprobe

# Run all tests
go test ./...

# Vet
go vet ./...

License

BSD 3-Clause. See LICENSE.