wallflower

wallflower

Finds the metrics nobody ever asks to dance.

wallflower audits what your Prometheus or Mimir ingests against what your Grafana dashboards and alerting rules actually use, ranks the waste by cardinality and cost, generates the drop rules to reclaim it, and can run in CI so the waste never creeps back. It is read-only: it never applies anything to a running system.

If a metric is emitted in the forest and no dashboard is around to query it, does it make a sound? It does. It sounds like your invoice.

What it looks like

The HTML report (--output html), one self-contained file with no server and no external assets, rendered from the bundled demo stack:

And the same audit in the terminal, against the same sandbox where only 18 of 42 metrics are consumed by anything (output verbatim from make demo-audit, middle of the table elided):

wallflower audit  source http://localhost:9009/prometheus  lookback 1h  generated 2026-07-29 09:07 UTC

  42 metrics ingested, 18 used, 24 unused (57.1%)
  3569 active series total, 31 reclaimable by dropping unused metrics
  label cleanup could reclaim up to ~3498 more (see DANCING TOO HARD)
  estimated savings: $0.25/month

NEVER ASKED TO DANCE  (24 metrics unused)
  METRIC                                  SERIES  EST COST/MO  NOTE
  app_config_reloads_total                3       $0.02
  app_gc_pause_seconds                    3       $0.02
  checkout_ab_test_assignments_total      2       $0.02
  checkout_legacy_coupon_hits_total       1       $0.01
  ...
  search_zero_results_total               1       $0.01        only feeds a recording rule nobody consumes

DANCING TOO HARD  (2 high-cardinality metrics)
  METRIC                             SERIES  OFFENDER LABEL  VALUES  LOOKS LIKE      CONSUMERS
  checkout_request_duration_seconds  2000    session_id      2000    per-request ID  dashboard Checkout
  payments_attempts_total            1500    user_id         1500    per-request ID  dashboard Payments

DANCES ALONE  (1 recording rule with unconsumed output)
  RULE                         GROUP   EXPR
  search:zero_result_ratio:5m  search  rate(search_zero_results_total[5m]) / sum(rate(search_querie...
  full expressions are in --output json

caveat: usage is derived from dashboards and rules; a metric queried only ad hoc
will appear unused here. Check before dropping.

Then turn the audit into rules you can review and apply:

$ wallflower rules --from audit.json --format prometheus
# Generated by wallflower on 2026-07-29 (lookback 1h, source http://localhost:9009/prometheus).
# Usage was derived from dashboards and rules, so a metric queried only
# ad hoc looks unused and may appear below. Review before applying.
# Add under scrape_configs[].metric_relabel_configs:
- source_labels: [__name__]
  regex: app_config_reloads_total|app_gc_pause_seconds|app_open_fds|...
  action: drop
# These recording rules produce output nothing consumes. Drop rules cannot
# remove rule output; delete the rules themselves instead:
#   search:zero_result_ratio:5m (group search)

Install

go install github.com/mateusgetulio/wallflower@latest

Or grab a binary from releases.

Usage

The quickest start is the interactive setup, which asks for your connection details, verifies each one with a real request, and writes wallflower.yaml for the other commands to read:

wallflower init
wallflower audit

Flags always win over the file, and environment variables (WALLFLOWER_PROMETHEUS_URL, WALLFLOWER_GRAFANA_URL, WALLFLOWER_TENANT, GRAFANA_TOKEN) sit in between. The wizard offers to keep the Grafana token out of the file and in GRAFANA_TOKEN instead; if you do store it, the file is written chmod 600.

Or pass everything explicitly:

wallflower audit \
  --prometheus-url http://prometheus:9090 \
  --grafana-url https://grafana.example.com \
  --grafana-token "$GRAFANA_SA_TOKEN"

The Grafana token needs a service account with the Viewer role; wallflower only calls read endpoints (dashboard search, dashboards, provisioned alert rules). Basic auth also works by putting userinfo in the URL, as the demo does.

For Mimir, point at the Prometheus-compatible prefix and set the tenant:

wallflower audit \
  --prometheus-url http://mimir:9009/prometheus \
  --tenant my-tenant \
  --grafana-url https://grafana.example.com \
  --grafana-token "$GRAFANA_SA_TOKEN"

Useful flags:

Generate rules (--format prometheus or --format otelcol):

wallflower audit ... --output json > audit.json
wallflower rules --from audit.json --format otelcol

The OTel Collector output pairs a filter processor (drops unused metrics) with a transform processor (deletes offender labels from metrics you keep).

Run it as a CI gate

The gate fails the build when metric waste crosses a limit:

wallflower gate --prometheus-url ... --grafana-url ... --max-unused-pct 30

Most teams cannot start from zero, so gate on regressions instead. Commit a baseline once:

wallflower gate --prometheus-url ... --baseline baseline.json --write-baseline
git add baseline.json

and in CI only new waste fails:

wallflower gate --prometheus-url ... --baseline baseline.json

A new unused metric, a growing unused share, or a tracked metric growing past --max-series fails the gate; everything already in the baseline passes. Ratchet the baseline down as you clean up.

Exit codes: 0 means the gate passed, 1 means a gate violation, 2 means an operational error (unreachable Prometheus or Grafana, bad flags), so CI can tell "you regressed" from "the check itself broke".

Add --report report.html and the gate writes the HTML report when it fails; upload it as a CI artifact and the reviewer sees the evidence without rerunning anything (--report-always writes it on pass too).

How "unused" is decided

wallflower collects every expression from:

• Grafana dashboard panels and template variables (via the Grafana API, including label_values(...) queries)
• Grafana-managed alert rules (when the provisioning API is available)
• Prometheus/Mimir alerting and recording rules (via the rules API)

Each expression is parsed with Prometheus's own PromQL parser and the metric selectors are extracted from the AST, so rate(x[5m]), {__name__=~"payments_.*"}, subqueries and Grafana template variables all resolve correctly. Expressions that are not PromQL (for example SQL panels) fall back to conservative identifier matching and are counted in the report.

Recording rules get one level of transitivity: a rule whose output nothing consumes is reported under "dances alone", and a metric whose only consumer is such a rule is flagged as transitively unused.

The honest caveat: wallflower sees dashboards and rules, not ad hoc queries. A metric someone greps in Explore once a month will look unused. Treat the report as a shortlist backed by evidence, not as a kill list. Query-log analysis is the top roadmap item.

Design notes with the architecture and the reasoning behind these decisions are in SPEC.md.

Comparison

wallflower is not the first tool in this space. Here is where it stands:

If you just want a raw list of unused metrics and already run Mimir, mimirtool analyze may be enough. wallflower exists for the step after the list: what does it cost, who caused it, and how do you keep it from coming back.

The demo stack

The repo ships a deterministic sandbox that doubles as the e2e test fixture:

make demo         # Mimir + Grafana + a seeded metrics generator (docker compose)
make demo-audit   # run wallflower against it
make demo-report  # render the HTML report (the screenshot above)
make demo-down

The seeder pushes 3,567 series across 40 scraped-style metrics: three fake services, planted session_id and user_id cardinality bombs, a recording rule nobody consumes, and 22 metrics no dashboard references. The Mimir ruler adds 2 recording-rule outputs, which is how the audit sees 42 metric names and 3,569 series; the 24 unused it reports are the 22 planted ones plus the unconsumed rule output plus its input, which only that rule reads.

Grafana is provisioned from files in the repo, so the ground truth of what is used is fixed and diffable. The demo is also the e2e fixture: CI brings this stack up and asserts that the audit finds exactly the planted set, nothing more and nothing less.

Grafana comes up at http://localhost:3000 (admin/admin); override ports with WALLFLOWER_GRAFANA_PORT / WALLFLOWER_MIMIR_PORT.

Why this exists

Metrics accrete. Someone instruments a service in 2022, the dashboard gets deleted in 2023, and the series keep flowing forever, because asking "can we drop this?" in Slack returns silence. Meanwhile every active series costs RAM on your own Prometheus or money on a per-series vendor bill, and one deploy that adds a user_id label can double either overnight. wallflower replaces the silence with evidence: no dashboard, no alert, no rule touched this metric in the window, and here is the rule that stops ingesting it.

Grafana Cloud sells a very good managed version of this idea (Adaptive Metrics). wallflower is the small, self-hosted, audit-first take for people who run their own stack.

Roadmap

• Query-log analysis, so ad hoc usage counts (removes the big caveat)
• Deeper recording-rule transitivity chains
• Per-team attribution for chargeback

License

Apache-2.0