observability

Overview ¶

Package observability — Auth metric bag (Plan 04-06 GREEN per MET-15 / GAP-6).

Owns one family per ADR §2.3 + CONTEXT GAP-6:

nornicdb_auth_attempts_total{result, protocol}

Closed enums (CONTEXT D-05e):

result   ∈ AllowedAuthResults    = {success, failure, denied}
protocol ∈ AllowedAuthProtocols  = {bolt, http, grpc}

Cardinality ceiling = |result| × |protocol| = 3 × 3 = 9 (RESEARCH §Q11). No tenant flag — auth is per-process global, not per-database (CONTEXT MET-21 omits `database` for auth; surfacing tenant identity on an unauthenticated counter would itself be a leak).

Forbidden-label discipline (Phase 3 D-03 / registration.go ForbiddenLabels): `user`, `user_id`, `email`, `ip` are all in ForbiddenLabels and the registration helper panics on attempt — defense in depth against PII surfacing through label values. The closed `result` enum prevents callers from passing a raw error message as the result label.

MET-25 hot path: callers cache `Attempts.WithLabelValues(result, protocol)` in a local var or struct field; the per-attempt observation pays only an atomic add. The cardinality ceiling is so low (9) that we do not even expose a Bind helper — direct WithLabelValues at the call site is fine.

D-02d leaf-package boundary: pkg/observability never imports pkg/auth. The bag is constructed at cmd/nornicdb startup and injected into:

• pkg/bolt/server.SetAuthMetrics (Plan 04-02 added the call site)
• pkg/auth.Authenticator.SetMetrics (HTTP path; this plan wires it)
• gRPC interceptor (TBD by integration site)

Package observability — Bolt metric bag (Plan 04-02 GREEN).

Owns six families per MET-07 + ADR §2.3:

nornicdb_bolt_connections_active
nornicdb_bolt_connections_total{result}
nornicdb_bolt_session_duration_seconds
nornicdb_bolt_messages_total{op, result}
nornicdb_bolt_message_duration_seconds{op}
nornicdb_bolt_packstream_decode_errors_total{reason}

Closed enums (CONTEXT D-11 / D-11a / D-11c):

result   ∈ AllowedBoltResults     = {success, error, timeout}
op       ∈ AllowedBoltOps         = {hello, run, pull, begin, commit,
                                     discard, reset, goodbye, route,
                                     ack_failure}
reason   ∈ AllowedPackstreamReasons =
                                     {truncated, invalid_marker,
                                      wrong_type, oversize}

Forbidden-label discipline (Phase 3 D-03 / registration.go ForbiddenLabels): `op`, `result`, and `reason` are NOT in the forbidden list — closed-enum string literals at the call site enforce cardinality. The reason enum is the highest-risk axis (free-form `err.Error()` would be a cardinality bomb); CONTEXT D-11c locks it to four values, classified at the packstream decode boundary via `reasonFromError(err)`.

PULL chunks NOT separately observed (CONTEXT D-11b). Chunk timing rolls up into the parent PULL `message_duration_seconds`. Aligns with Phase 8 TRC-13 ("PULL chunks do not span") — same rationale: per-chunk observation taxes the streaming hot path with no SRE alerting benefit.

No `database` label on Bolt subsystem: sessions/messages cross databases via USE clauses; per-DB instrumentation lives at the Cypher subsystem. CONTEXT D-08 bool not threaded through this constructor.

Package observability — Cache + Runtime metric bag (Plan 04-01 GREEN).

Owns six families per MET-16 + ADR §2.3:

nornicdb_cache_hits_total{cache}
nornicdb_cache_misses_total{cache}
nornicdb_cache_size_bytes{cache}
nornicdb_cache_evictions_total{cache, reason}
nornicdb_process_uptime_seconds       (GaugeFunc; time.Since(start).Seconds())
nornicdb_build_info                   (GaugeFunc; constant 1 + const labels)

Closed enums (CONTEXT D-12, D-12b):

cache  ∈ AllowedCacheNames        = {query_result, schema, label, node_lookup}
reason ∈ AllowedEvictionReasons   = {lru, ttl, capacity, manual}

RISK-4 carry-forward: if a downstream plan determines a cache name has no actual increment site, prune the slice here AND amend ADR §2.3. Today we keep all four to match the planning contract; the catalog_cache_test.go AssertCardinalityCeiling(4) passes whether 3 or 4 emit (helper has no lower bound — RESEARCH RISK-7 / Plan 04-07 ships the lower-bound test).

Forbidden-label discipline (Phase 3 D-03 / registration.go ForbiddenLabels): `cache` and `reason` are NOT in the forbidden list — closed-enum string literals at the call site enforce cardinality. Subsystem callers MUST use only AllowedCacheNames / AllowedEvictionReasons values.

Pitfall 1 / RESEARCH RISK-8 mitigation: every GaugeFunc body wraps a defer-recover that returns 0 on panic, preventing a buggy callback from poisoning the entire /metrics scrape (Pitfall 1: shutdown ordering vs. live scrape window — listener drains LAST per OBS-08).

Leaf-package boundary (Phase 1 D-01a / boundary_test.go) preserved: imports limited to stdlib + prometheus/client_golang + pkg/buildinfo. pkg/buildinfo is a leaf utility (it embeds VERSION + carries Commit ldflag vars), not a forbidden subsystem package.

Package observability — Cypher metric bag (Plan 04-03 GREEN).

Owns eleven families per MET-08 + ADR §2.3:

nornicdb_cypher_queries_total{op_type[, database]}
nornicdb_cypher_query_duration_seconds{op_type[, database]}
nornicdb_cypher_planner_duration_seconds{op_type}
nornicdb_cypher_planner_cache_hits_total
nornicdb_cypher_planner_cache_misses_total
nornicdb_cypher_planner_cache_size
nornicdb_cypher_rows_returned{op_type}
nornicdb_cypher_active_transactions
nornicdb_cypher_transaction_conflicts_total{[database]}
nornicdb_cypher_slow_queries_total{[database]}
nornicdb_cypher_slow_query_threshold_seconds   (GaugeFunc; D-15b)

Closed enums (CONTEXT D-04 corrected per RISK-1; D-04b parse_error sixth value):

op_type ∈ AllowedCypherOpTypes = {read, write, schema, admin, fabric, parse_error}

RISK-1 corrected classifier surface — the actual normal-path classifier reads `*QueryInfo` from QueryAnalyzer (see pkg/cypher/op_type.go), NOT a non-existent `plan.Root.Op` field. The closed enum above is the contract; `pkg/cypher/op_type.go::classifyOpType` is the producer; this bag is the consumer. Three observation sites (admin-dispatch, parse-error site, normal-path-after-Analyze) cover all six values.

Forbidden-label discipline (Phase 3 D-03 / registration.go ForbiddenLabels): `query`, `user`, `user_id`, `ip`, `uuid`, `email` would all be cardinality bombs for Cypher and panic at registration. `op_type` is bounded by the closed enum above; `database` is bounded by the per-tenant-flag axis (D-08); callers MUST NEVER pass raw query text or query_hash as a Prometheus label.

D-12a planner cache locality: planner_cache_hits_total / planner_cache_misses_total / planner_cache_size live HERE under the cypher subsystem, not under the cross-cutting cache subsystem. Planner cache is Cypher-specific. The cross-cutting cache bag (catalog_cache.go) carries a separate `cache_hits_total{cache="query_result"}` mirror that the Cypher query-result cache emits in parallel — both layers see the metric.

MET-25 hot path: subsystems pre-bind via `BindQueryDuration("read", db)` and cache the resulting BoundLatencyObserver in struct fields at constructor time (see pkg/cypher/executor.go). The chokepoint Observe() call funnels through observeWithExemplar — zero exemplar overhead under Phase 1's NeverSample default; lights up automatically at Phase 6's sampler flip with no second-pass migration.

MET-26 / D-15b slow_query_threshold_seconds: registered directly on reg via prometheus.NewGaugeFunc(slowQueryThresholdFn) — NO struct field. The callback wraps a defer-recover that returns 0 on panic per RESEARCH RISK-8 / Pitfall 1. The value reflects cfg.Logging.SlowQueryThreshold (or whatever the injected callback reads) on every scrape — config reload is automatic; no event wiring required.

D-08 forward-compat: NewCypherMetrics(reg, tenantLabelsEnabled bool, slowQueryThresholdFn func() float64) decides label-set shape ONCE at construction. When false (Phase 4 default outside K8s), the `database` label is OMITTED from the labelnames slice — not set to empty string. Phase 5's K8s autodetect flips the bool's default value with no re-registration.

Package observability — Embeddings metric bag (Plan 04-05 GREEN).

Owns six families per MET-12 + ADR §2.3, plus the D-09 FFI panic counter:

nornicdb_embed_queue_depth                                (GaugeFunc; D-15b)
nornicdb_embed_processed_total{provider, model, result, mode}
nornicdb_embed_duration_seconds{provider, model, mode}    (long-tail; MET-05)
nornicdb_embed_cache_hits_total
nornicdb_embed_cache_misses_total
nornicdb_embed_worker_running                             (Set(0)/Set(1) at lifecycle)
nornicdb_embed_ffi_panics_total{mode}                     (D-09 — pkg/embed FFI recovery)

Closed enums (CONTEXT D-06a):

mode     ∈ AllowedEmbedBackends  = {gpu, cpu, cuda, metal, vulkan}
result   ∈ AllowedEmbedResults   = {success, failure, cached}
provider ∈ AllowedEmbedProviders = {ollama, openai, local, other}
model    is open-ish (bounded ceiling 250 per RESEARCH §Q11)

Forbidden-label discipline (Phase 3 D-03 / registration.go ForbiddenLabels): `embedding_text` is the keystone — embedding *content* MUST NEVER be a label value. The forbidden-label panic at registration prevents anyone from labeling by raw input text. The other ForbiddenLabels (path, query, user, user_id, ip, uuid, trace_id, span_id, email) are not reachable from this subsystem.

MET-25 hot path: per-call observation uses pre-bound observers cached at the call site (see Plan 04-05 wiring + BenchmarkObserve_HotEmbed). The bag's Duration.Bind(provider, model, mode) returns a BoundEmbeddingLatencyObserver that is value-typed and concurrent-safe; callers store it in struct fields per MET-25.

D-15b GaugeFunc panic safety: the queue_depth callback wraps defer-recover that returns 0 on probe panic — concurrent shutdown of the embed queue cannot poison /metrics. Same pattern as Plan 04-04 MVCC GaugeFuncs.

D-15b distinction (worker_running): binary lifecycle state uses `Set(0)` / `Set(1)` at the lifecycle start/stop hook sites, NOT a GaugeFunc — the lifecycle event boundary is the right observation chokepoint.

D-02d leaf-package boundary: pkg/observability never imports pkg/embed or pkg/nornicdb. EmbedProbe is declared HERE; the embedder (or the embed-queue accessor adapter) satisfies it via a thin shim at the cmd/nornicdb wiring site.

D-08 forward-compat: Embed metrics are NOT tenant-tagged. provider, model, and mode are global (not per-database). Per CONTEXT MET-21 omission of embed/runtime/cache from the tenant axis. NewEmbedMetrics therefore omits the tenantLabelsEnabled bool parameter.

D-09 FFI panic counter: ffi_panics_total{mode} is the only counter where pkg/embed observably crashed during a CGo call — incremented from the deferred-recover wrapper in pkg/embed/ffi_recover.go (Plan 04-05-03). Closed mode enum prevents arbitrary panic-class labeling.

Package observability — HTTP metric bag (Plan 04-02 GREEN).

Owns five families per MET-06 + ADR §2.3:

nornicdb_http_requests_total{method, path_template, status_class[, database]}
nornicdb_http_request_duration_seconds{method, path_template, status_class[, database]}
nornicdb_http_in_flight_requests
nornicdb_http_request_body_bytes{method, path_template}
nornicdb_http_response_body_bytes{method, path_template}

Closed enums:

status_class ∈ AllowedStatusClasses = {1xx, 2xx, 3xx, 4xx, 5xx}
method       — bounded by HTTP method enum (GET, POST, PUT, DELETE,
                PATCH, HEAD, OPTIONS, CONNECT) ≈ 8 entries
path_template — open shape but bounded by route table (~15 templates;
                see RESEARCH §Q11 ceiling=1000 for headroom)

Forbidden-label discipline (Phase 3 D-03 / registration.go ForbiddenLabels): `path` is in ForbiddenLabels; passing the raw URL would panic at registration. The label name `path_template` is NOT forbidden — but values MUST come from `r.Pattern` (Go 1.22+ stdlib ServeMux), NEVER from `r.URL.Path`. The instrumentedMux wrapper in pkg/server/server.go is the SOLE call site (D-03 single chokepoint).

Tenant-flag forward-compat (CONTEXT D-08): `NewHTTPMetrics(reg, tenantLabelsEnabled bool)` decides label-set shape ONCE at construction. When false (Phase 4 default outside K8s), the `database` label is OMITTED from the labelnames slice — not set to empty string. When true (Phase 5 K8s autodetect default), `database` is included. Subsystems use the bool-agnostic `BindRequestDuration` / `BindRequests` helpers; they thread the database arg through but the helper drops it when the bag was constructed with tenantLabelsEnabled=false.

Package observability — Knowledge-policy metric bag.

Owns ten families (see docs/plans/knowledge-policy-observability-plan.md):

nornicdb_knowledge_policy_scored_total{entity_kind,result[, database]}
nornicdb_knowledge_policy_decay_score{entity_kind[, database]}       (sampled 1/32)
nornicdb_knowledge_policy_suppressions_total{entity_kind,reason[, database]}
nornicdb_knowledge_policy_access_flush_batch_rows                     (no tenant label)
nornicdb_knowledge_policy_access_flush_duration_seconds               (no tenant label)
nornicdb_knowledge_policy_access_flush_buffer_fullness                (GaugeFunc; D-15b)
nornicdb_knowledge_policy_on_access_mutations_total{result[, database]}
nornicdb_knowledge_policy_deindex_enqueued_total{entity_kind[, database]}
nornicdb_knowledge_policy_read_filter_dropped_total{entity_kind[, database]}
nornicdb_knowledge_policy_reconcile_total{trigger[, database]}

Closed enums:

entity_kind ∈ AllowedKnowledgePolicyEntityKinds = {node, edge, property}
result      ∈ AllowedKnowledgePolicyScoreResults = {visible, suppressed, no_decay}
reason      ∈ AllowedKnowledgePolicySuppressReasons = {below_threshold, score_floor,
                                                        on_access, explicit_flag, rule_cap}
on_access_result ∈ AllowedKnowledgePolicyOnAccessResults = {applied, skipped_no_policy, error}
trigger     ∈ AllowedKnowledgePolicyReconcileTriggers = {schema_change, startup, manual}

Forbidden-label discipline: the graph-schema node `label` axis and user property keys are DELIBERATELY excluded — user DDL can create unbounded values. Use exemplar trace attributes or structured logs for per-label detail, NEVER Prometheus labels.

Hot-path discipline (MET-25): `Scorer.score()` runs per node returned from any Cypher match when decay is enabled. The bag pre-binds the per- entity_kind counters and histogram at construction; the caller caches the Bound* observers in a struct field and calls Observe/Inc without a WithLabelValues lookup on the hot path.

Sampling: DecayScore is sampled 1/32 via ObserveDecayScoreSampled so a 1M-row scan pays only ~31k histogram Observe calls instead of 1M. Counter fires (ScoredTotal, SuppressionsTotal) are cheap enough to fire every time.

D-08 forward-compat: NewKnowledgePolicyMetrics(reg, tenantLabelsEnabled, bufferFullnessFn) decides label-set shape ONCE at construction. When tenantLabelsEnabled is false, the `database` label is OMITTED from all labelnames. Flush-level metrics (batch_rows, duration, buffer_fullness) are intentionally NOT tenant-scoped — the AccessFlusher is cross-namespace.

Package observability — MVCC metric bag (Plan 04-04 GREEN).

Owns four families per MET-11 + ADR §2.3:

nornicdb_mvcc_pressure_band{[database], band}
nornicdb_mvcc_pinned_bytes               (GaugeFunc; D-15b)
nornicdb_mvcc_oldest_reader_age_seconds  (GaugeFunc; D-15b)
nornicdb_mvcc_active_readers             (GaugeFunc; D-15b)

Closed enums (CONTEXT D-14):

band ∈ AllowedMVCCBands = {normal, warn, high, critical}

D-14 thresholds (constants below):

ratio < 0.50           → normal
0.50 ≤ ratio < 0.75    → warn
0.75 ≤ ratio < 0.90    → high
ratio ≥ 0.90           → critical

PressureBand is set indicator-style: the active band gauge holds value 1 while the other three bands hold value 0 for the same database. The indicator pattern lets dashboards do `max by (database) (... == 1)` to pick the current band; alert rules can fire on `nornicdb_mvcc_pressure_band{band="critical"} == 1`.

Forbidden-label discipline (Phase 3 D-03 / registration.go ForbiddenLabels): `query`, `user`, `user_id`, `ip`, `uuid`, `email` are never used here; `band` is bounded by AllowedMVCCBands; `database` is bounded by the per-tenant-flag axis (D-08).

RISK-2 fixed (Plan 04-04-01): the three GaugeFunc callbacks read the MVCCProbe accessors (PinnedBytes / OldestReaderAgeSeconds / ActiveReaders) which now exist on *BadgerEngine. Each callback is wrapped in a defer-recover that returns 0 on panic per RESEARCH RISK-8 / Pitfall 1 — concurrent shutdown cannot poison the /metrics scrape.

D-02d leaf-package boundary: pkg/observability never imports pkg/storage. MVCCProbe is declared HERE; pkg/storage's *BadgerEngine satisfies it via the accessors in pkg/storage/badger_mvcc.go.

D-08 forward-compat: NewMVCCMetrics(reg, tenantLabelsEnabled, probe) decides whether the `database` label is included in pressure_band ONCE at construction. The three live-read gauges have no labels (single process-wide value) so they are tenant-flag-independent.

Package observability — minimal type stubs for catalog bags whose GREEN implementations land in Plans 04-03..04-06 (Cypher, Storage, MVCC, Embed, Search, Replication, Auth).

These stubs exist to keep pkg/observability compiling while the per-bag RED tests sit in their `t.Skip("RED: pending Plan 04-NN")` state (Plan 04-01 Wave-0 RED-first cadence). Each stub:

• Declares the struct fields the RED tests reference (e.g. `Bytes`, `IndexRebuild`, `LagBytes`, `AuthAttempts`, `FFIPanicTotal`).
• Provides a constructor that PANICS at runtime — the RED test's leading `t.Skip` ensures the constructor is never called from a running test, so the panic is purely a load-bearing guard against accidental production usage before the GREEN bag lands.

When Plan 04-NN ships its real `NewXxxMetrics` constructor + bag, that plan REPLACES the stub here with the production bag (split into a per-subsystem catalog_<sub>.go file per CONTEXT D-02a).

Plan ownership map:

NewCypherMetrics        — Plan 04-03 (SHIPPED — see catalog_cypher.go)
NewStorageMetrics       — Plan 04-04 (SHIPPED — see catalog_storage.go)
NewMVCCMetrics          — Plan 04-04 (SHIPPED — see catalog_mvcc.go;
                          RISK-2 PinnedBytes accessor lives on
                          *BadgerEngine in pkg/storage/badger_mvcc.go)
NewEmbedMetrics         — Plan 04-05 (SHIPPED — see catalog_embed.go)
NewSearchMetrics        — Plan 04-05 (SHIPPED — see catalog_search.go)
NewReplicationMetrics   — Plan 04-06 (with RISK-3 PeerConfig.ID + GAP-1
                          last_contact_seconds + per-mode cardinality)
NewAuthMetrics          — Plan 04-06 (GAP-6 / MET-15)

Plan 04-02 (this plan) DELIVERS NewHTTPMetrics + NewBoltMetrics — those live in catalog_http.go and catalog_bolt.go respectively (NOT in this stubs file).

As of Plan 04-06 all bags have shipped — the stub bodies are empty and this file remains as a historical pointer to the per-subsystem catalog files (one entry per <plan ownership map> row).

Package observability — Replication metric bag (Plan 04-06 GREEN per MET-14 + GAP-1 + RISK-3 corrected).

Owns ten families per ADR §2.3 + CONTEXT §domain:

nornicdb_replication_role                          (Gauge int enum)
nornicdb_replication_term                          (Gauge int)
nornicdb_replication_commit_index                  (Gauge int)
nornicdb_replication_apply_index                   (Gauge int)
nornicdb_replication_lag_bytes{peer}               (GaugeVec)
nornicdb_replication_lag_entries{peer}             (GaugeVec)
nornicdb_replication_apply_duration_seconds        (LatencyHistogram)
nornicdb_replication_rtt_seconds{peer}             (HistogramVec)
nornicdb_replication_leader_changes_total          (Counter)
nornicdb_replication_last_contact_seconds{peer}    (GaugeVec — GAP-1)

**RISK-3 FIX (RESEARCH §RISK-3 verified):** pkg/replication/config.PeerConfig has fields {ID string; Addr string} — NOT `Name`. CONTEXT D-05 wording is obsolete. PeerLabel(p) returns `p.ID` if non-empty, falling back to `p.Addr`. NEVER reads from `conn.RemoteAddr()` — runtime sockets would explode cardinality.

**D-05a mode-aware ceiling:**

ha_standby:   8  (single peer + churn slack)
raft:         16 (5-node typical quorum + churn slack)
multi_region: 64 (3-region × ~5-peer + churn slack)

Stored in modeCeiling and accessible via Mode() / Ceiling() so the AssertCardinalityCeiling test parameterizes by mode.

**D-08a tenant-flag accepted but IGNORED at registration:** Replication is per-cluster, not per-database. `tenantLabelsEnabled` is part of the constructor signature so callers pass cfg uniformly across all subsystem bags, but the database label is never added to any replication family. Documented inline; verified by TestReplicationMetrics_*.

**D-15a per-event observation (zero polling):** pkg/replication.RaftReplicator + HAStandbyReplicator + MultiRegionReplicator call metrics.Role.Set(RoleEnum("leader")) at the SAME sites that emit the existing role-transition log line. No background goroutine polls state. Pre-bound observers cached in struct fields per MET-25.

**D-05b stale-peer GC (lifecycle.Component) lives in pkg/replication/peer_metrics_gc.go.** It calls DeleteLabelValues on the per-peer GaugeVecs after a configurable staleness threshold. Replicator rebinds on reconnect (Pitfall 3 mitigation per RESEARCH §Q7).

**D-02d leaf-package boundary:** pkg/observability never imports pkg/replication. The replication code imports pkg/observability and satisfies its peer-label inputs with PeerConfig (whose two-field shape {ID, Addr} is documented here for the test suite).

Package observability — Search metric bag (Plan 04-05 GREEN).

Owns four families per MET-13 + ADR §2.3:

nornicdb_search_requests_total{[database,] mode, result}
nornicdb_search_duration_seconds{[database,] mode, stage}
nornicdb_search_candidates                          (RowCountHistogram)
nornicdb_search_index_size_bytes{kind}              (GaugeFunc; D-15b)

Closed enums (CONTEXT MET-13 / D-15b):

mode   ∈ AllowedSearchModes        = {vector, bm25, hybrid}
result ∈ AllowedSearchResults      = {success, no_results, error}
stage  ∈ AllowedSearchStages       = {embed, index, fuse}
kind   ∈ AllowedSearchIndexKinds   = {hnsw, bm25}

stage is the keystone: per-stage observation lets SREs see whether embed (LLM call), index (vector / BM25 lookup), or fuse (RRF + rerank) dominates the latency budget. Closed enum at internal call sites prevents arbitrary stage-name labeling.

Forbidden-label discipline (Phase 3 D-03 / registration.go ForbiddenLabels): `query` is the keystone — raw query text MUST NEVER be a label value. The registration.go panic catches anyone passing `query` as a label.

MET-25 hot path: per-stage observation uses pre-bound observers cached at the call site. The bag's BindDuration helper returns a BoundLatencyObserver that callers store in struct fields per MET-25.

D-15b GaugeFunc panic safety: the index_size_bytes callback per kind wraps defer-recover that returns 0 on probe panic — same pattern as Plan 04-04 storage.bytes / mvcc.pinned_bytes.

D-02d leaf-package boundary: pkg/observability never imports pkg/search. SearchProbe is declared HERE; pkg/search.Service satisfies it via thin adapter or a setter that AttachesMetrics.

D-08 forward-compat: NewSearchMetrics(reg, tenantLabelsEnabled, probe) decides whether the `database` label is included in requests_total + duration_seconds ONCE at construction. candidates and index_size_bytes are global — bytes per index kind, candidates per request distribution.

Package observability — Storage metric bag (Plan 04-04 GREEN).

Owns eight families per MET-10 + ADR §2.3:

nornicdb_storage_nodes_total                              (GaugeFunc; D-15b)
nornicdb_storage_edges_total                              (GaugeFunc; D-15b)
nornicdb_storage_bytes{kind}
nornicdb_storage_op_duration_seconds{op}
nornicdb_storage_compactions_total{level, result}
nornicdb_storage_compaction_duration_seconds{level}
nornicdb_storage_wal_lag_bytes
nornicdb_storage_index_rebuild_total{[database,] index, result}

Closed enums (CONTEXT D-07 / D-13c):

kind   ∈ AllowedStorageBytesKinds = {nodes, edges, index, wal, search}
op     ∈ AllowedStorageOps        = {get, put, delete, scan}
index  ∈ AllowedStorageIndexes    = {label, edge_between, temporal,
                                     embedding, user_created}
result ∈ AllowedStorageResults    = {success, failure, aborted}

D-13c the `index` enum is the keystone of T-04-02 mitigation: arbitrary user-created index names NEVER become label values. The pure mapping function `pkg/storage.classifyIndexName(internal string) string` buckets any unknown name to `user_created`. RESEARCH §Q11 cardinality ceiling.

Forbidden-label discipline (Phase 3 D-03 / registration.go ForbiddenLabels): `path`, `query`, `user`, `user_id`, `ip`, `uuid`, `embedding_text`, `trace_id`, `span_id`, `email` would all be cardinality bombs and panic at registration. Storage hot paths never label by raw key bytes or full paths.

D-07 bytes{kind} is populated by a 30s lifecycle.Component sweep that calls `badger.DB.EstimateSize(prefix)` for each prefix. Sweep lives in pkg/storage/bytes_metrics.go (Plan 04-04-04). RESEARCH §Q3.

MET-25 hot path: per-op observation uses pre-bound BoundLatencyObserver cached in BadgerEngine struct fields at constructor time. The bag's OpDuration.Bind("get") is hoisted out of the request loop. CompactionDuration is bound at compaction-event hook attachment.

RISK-6 wal_lag_bytes semantics: best-effort heuristic estimate of WAL backlog (vlog size minus LSM size from `badger.DB.Size()`). Alerting SHOULD use 5-minute trends, not single scrapes. The metric is documented here and in the bytes_metrics.go sweeper that populates it.

D-08 forward-compat: NewStorageMetrics(reg, tenantLabelsEnabled, probe) decides whether the `database` label is included in IndexRebuildTotal ONCE at construction. The plan deliberately does NOT add database to op_duration (D-08a) — per-database op latency lives at the Cypher subsystem (Plan 04-03).

Package observability is NornicDB's telemetry seam: metrics (Prometheus + OTel), traces (OTLP), and the building blocks for /metrics, /livez, /readyz, /version, and opt-in pprof endpoints.

Architecture ¶

observability is a leaf package. It depends on stdlib, OpenTelemetry SDK, prometheus/client_golang, and pkg/lifecycle (one-way; pkg/lifecycle has zero observability deps). Business packages (cypher, storage, bolt, server) MUST NOT import this package directly — they receive instrumentation via accessors passed through dependency injection from cmd/nornicdb/main.go.

Wiring ¶

Construct a *Provider via:

prov, err := observability.New(ctx, cfg.Observability, observability.ServiceInfo{
    Name:    "nornicdb",
    Version: buildinfo.Version(),
    NodeID:  cfg.NodeID,
})

New always succeeds at the Provider-construction level: a misconfigured or unreachable OTLP collector is reported via WARN log and a noop tracer provider is installed (OBS-11). Process startup never fails because of observability init failure.

Init order (OBS-03) ¶

New constructs in this order, mandated by the ADR-0001 §2.5 contract:

1. logger (deferred to Phase 2 slog migration; Phase 1 uses stdlib log).
2. resource attributes (semconv.ServiceName/Version/InstanceID).
3. meter + tracer providers (real SDK or noop fallback).
4. registries (the Prometheus registry and the OTel→Prom bridge).
5. middleware + listeners (Plan 03 — listener.go, health.go, pprof.go).

Endpoint precedence (OBS-12) ¶

OTLP endpoint resolution honors:

OTEL_EXPORTER_OTLP_ENDPOINT > YAML tracing.endpoint > built-in default

See TracingConfig.OTLPEndpoint().

service.instance.id resolution (OBS-10) ¶

resolveInstanceID resolves through this chain:

cfg.NodeID → POD_NAME env → os.Hostname() → "standalone"

The resolved value and its source are logged once at startup.

Compliance boundary ¶

pkg/audit (Phase-2 untouched) is NOT migrated to OTel logs. The compliance audit trail keeps its existing retention/signing/serialization. This package only owns observability — operator-facing telemetry — not compliance.

Test isolation ¶

Each test owns a fresh *prometheus.Registry, an in-memory span exporter, and a discard logger. Never touch prometheus.DefaultRegisterer or slog.SetDefault in tests. The NewTestEnv helper that codifies this ships in Plan 03.

Package observability — typed exemplar-wrapper layer for histogram observation (Phase 3, MET-24 + MET-25).

Design (CONTEXT D-02):

• Wrappers (LatencyHistogram etc.) own the MET-24 exemplar-emission chokepoint. Subsystems use the wrapper for normal Observe paths; the raw *prometheus.HistogramVec from metrics.go remains accessible via Vec() for cardinality-ceiling assertions and edge cases.
• Bind(lvs ...) returns a Bound*Observer that subsystems cache as a struct field at construction time — eliminates per-Observe WithLabelValues lookup (MET-25 hot-path discipline).
• The IsValid()→ExemplarObserver chokepoint is a SINGLE function (observeWithExemplar) per AGENTS.md §7 DRY. Every wrapper Observe and every Bound Observe funnel through it.

Forward-compat (CONTEXT D-02a + Phase 2 D-05 precedent):

Phase 1 ships sdktrace.NeverSample() ⇒ every emitted SpanContext has
IsSampled()=false ⇒ chokepoint short-circuits ⇒ ZERO exemplar-path
allocations today (also true on the truly-empty noop SpanContext where
IsValid()=false). Phase 6 flips the sampler in ONE place (provider.go);
ALL Phase-3-wrapped histograms emit exemplars automatically — no
second-pass migration of the ~60 Phase 4 metric families. Same posture
as Phase 2's mandatory_fields trace-id resolution.

Allocation budget (CONTEXT D-02b, falsified by BenchmarkObserve_Hot):

hot_no_span sub-bench:  ≤ 2 allocs/op (target: 0 — chokepoint short-circuits)
hot_with_span sub-bench: ≤ 4 allocs/op (1 for Labels map, 2 for *.String() calls)
If hot_with_span exceeds 4, D-02b1 escalation triggers:
  sync.Pool[*prometheus.Labels] — captured here so future plans don't re-derive.

Counter/gauge exemplars deferred (CONTEXT D-02e). Phase 3 ships histograms only.

Package observability — Phase 5 K8s autodetect.

k8sProbe inspects env + filesystem signals (KUBERNETES_SERVICE_HOST and /var/run/secrets/kubernetes.io/serviceaccount/token presence) to decide whether tenant labels should default ON. Conservative AND-signal logic per CONTEXT.md D-02. Pure-ish function: only stdlib os reads.

Phase 5 / Plan 05-03 fills the Detect + ResolveTenantLabels bodies.

Package observability — knowledge-policy metrics global ref.

The Badger read-path filter (pkg/storage/badger_decay_filter.go) fires on EVERY node returned from a Cypher MATCH when decay is enabled. Threading a *KnowledgePolicyMetrics handle through every storage iterator signature would widen CreateNode / GetNode / iterateNodesVisibleAtInTxn and their many callers. Instead we use the same atomic.Pointer bridge the BSP self- metrics pattern (bsp_self_metrics.go) already established for injecting observability into a deep-nested subsystem without inverting ownership.

Semantics:

• Set once at Provider init (from cmd/nornicdb/main.go, after NewKnowledgePolicyMetrics returns).
• Overwritten on each New() so per-test TestEnv isolation still works.
• Nil safe: GetKnowledgePolicyMetrics returns nil before Set is called or when metrics are disabled; all IncScored / IncSuppression / etc. helpers short-circuit on nil receiver.

Package observability — Phase 5 legacy translation layer.

RenderLegacy walks the unified pkg/observability registry and emits the 12 metric families that customer scrapers expect on :7474/metrics. Pure function: input *prometheus.Registry + time.Time, output []byte in Prometheus exposition format v0.0.4.

Phase 5 / Plan 05-02 fills the legacyMappings table function fields and the RenderLegacy body; Wave-0 (05-01) declared the public-API surface.

Package observability — typed metric constructors that enforce naming, bucket, and label discipline at registration time (Phase 3, MET-01..MET-05).

Design (CONTEXT D-01):

• Bucket choice is encoded in the constructor function name; subsystem authors cannot pass arbitrary Buckets. Single source of truth (D-01).
• Namespace="nornicdb" is injected by the helper; caller never sets it (D-01b/D-01c — final name is nornicdb_<subsystem>_<name>).
• Subsystem is enforced against the closed allowedSubsystems list (registration.go).
• Forbidden labels (cardinality bombs + PII) panic at registration (registration.go::validateLabels — D-03a).

Dual-access pattern (CONTEXT D-02 + <specifics>):

• These constructors return native *prometheus.HistogramVec / *prometheus.CounterVec / *prometheus.GaugeVec so subsystems can pre-bind via WithLabelValues for MET-25 hot-path discipline.
• The typed wrapper structs in exemplar.go (Plan 03-03) wrap the same *HistogramVec to centralize the MET-24 exemplar-emission chokepoint.
• Wrappers do NOT hide the raw types — they're parallel returns. Phase 4 subsystems receive both: the raw *Vec for cardinality assertions and edge cases, the wrapper for normal Observe() paths.

Pitfall 8 / MustRegister precedent (analog: registry.go:28-29): validation failure at construction is a programming bug; panic IS the desired startup behavior — surfaces the bug before any traffic reaches the binary.

Package observability — registration-time validation primitives for the metrics-helper layer (Phase 3, MET-01..MET-05). Each helper in metrics.go calls these primitives BEFORE invoking reg.MustRegister; a violation is a programming bug per Phase 1 Pitfall 8 / MustRegister precedent — panic IS the desired startup behavior.

Doc analog: pkg/observability/registry.go:28-29 (MustRegister panic-as-feature).