metrics

type BodyLimitRejectionKey ¶

type BodyLimitRejectionKey struct {
	Cell  string
	Route string
}

BodyLimitRejectionKey identifies one body-limit rejection metric series.

type CellLabel ¶

type CellLabel struct {
	// contains filtered or unexported fields
}

CellLabel is the sealed, validated cell dimension for a metric series. It is the only type the Collector / GRPCCollector record methods accept for the `cell` label.

Sealed construction (Hard, same shape as kernel/outbox.Entry): the single field is unexported and the sole exported constructor is ResolveCellLabel, so constructing a non-zero CellLabel outside this package is a compile error. Passing a raw, unvalidated cell-id string to a collector is therefore unexpressible — every cell label that reaches a collector has been checked against the assembly's closed cell-id set, and an out-of-set or absent cell degrades to RuntimeCellSentinel rather than polluting the platform SLO series.

This is the M12b (#1093) runtime defense-in-depth, deliberately LAYERED with — not contradicting — the M12a build-time closed-set gate (composition.Builder.Build):

• M12a hard-rejects an out-of-set cell id at BUILD time: a misconfigured assembly must fail fast before serving ("越界 cellID 硬拒、不降级").
• M12b degrades an out-of-set cell id to the sentinel at the RUNTIME metric write point: a live request must not panic over a metric label, so the one label degrades rather than crashing or polluting the series ("→ _runtime").

"不降级" governs the build path; "→ sentinel" governs the runtime label path. See .claude/rules/gocell/observability.md "HTTP Metrics cell Label".

func ResolveCellLabel(ctx context.Context, valid map[string]struct{}) CellLabel

func (c CellLabel) String() string

type Collector ¶

type Collector interface {
	// RecordRequest records a completed HTTP request with the given labels.
	// route is the route pattern (e.g. "/api/v1/users/{id}"), not the actual
	// request path. Using route patterns prevents metric cardinality explosion.
	//
	// cell is the coarse owner dimension for the request, carried as the sealed
	// [CellLabel] whose only constructor is [ResolveCellLabel]. The collector
	// never infers it from assembly, config, route path, tenant, slice, or
	// contract metadata — ResolveCellLabel resolves it from request context and
	// the assembly closed set, degrading framework-owned paths
	// (healthz/readyz/metrics, unmatched 404s, listeners with no business
	// RouteGroup, out-of-set cells) to RuntimeCellSentinel ("_runtime").
	RecordRequest(ctx context.Context, cell CellLabel, method, route string, status int, durationSeconds float64)

	// RecordBodyLimitRejection increments the body-limit rejection counter for
	// the given cell and route. It is called only on the Content-Length
	// fast-path reject (r.ContentLength > maxBytes before the request body is
	// read).
	//
	// Streaming overruns (MaxBytesReader kick-in, after the body has started
	// being read) are handler-dependent: the handler receives
	// *http.MaxBytesError when it reads the body, and if it propagates the
	// error through pkg/httputil.WriteError, the framework maps it to a 413
	// response captured in http_requests_total{status="413"} via RecordRequest.
	// Framework-generated handlers follow this path; custom handlers that do not
	// call httputil.WriteError will not produce a 413 in http_requests_total.
	//
	// cell follows the same semantics as RecordRequest: the sealed [CellLabel]
	// from [ResolveCellLabel] (validated cell id or RuntimeCellSentinel).
	// route is the low-cardinality route pattern from RouteFor.
	RecordBodyLimitRejection(ctx context.Context, cell CellLabel, route string)
}

Collector records HTTP request metrics.

func NewProviderCollector(p kernelmetrics.Provider, cfg ProviderCollectorConfig) (Collector, error)

type ConfigEventCollector ¶

type ConfigEventCollector interface {
	RecordEventProcess(ctx context.Context, cellID, sliceID string, reason ConfigEventProcessReason)
	RecordEventSettlement(ctx context.Context, cellID, sliceID, disposition string, result outbox.SettlementResult)
}

ConfigEventCollector records config event process and settlement metrics.

func NewProviderConfigEventCollector(p kernelmetrics.Provider) (ConfigEventCollector, error)

type ConfigEventProcessReason ¶

type ConfigEventProcessReason string

ConfigEventProcessReason is the low-cardinality handler/process taxonomy for config event consumers. It is intentionally separate from broker settlement.

const (
	ConfigEventProcessReasonAck            ConfigEventProcessReason = "ack"
	ConfigEventProcessReasonStale          ConfigEventProcessReason = "stale"
	ConfigEventProcessReasonPermanentError ConfigEventProcessReason = "permanent_error"
	// ConfigEventProcessReasonNoTenant is recorded when tenant.FromContext returns
	// an error (no tenant in consumer context). The event is Ack-ed as
	// retrying cannot supply a tenant that was absent at delivery time.
	ConfigEventProcessReasonNoTenant ConfigEventProcessReason = "no_tenant"
	// ConfigEventProcessReasonTransient is recorded when the config getter
	// returns a transient (non-permanent, non-404) error so the entry is
	// Requeued for retry.
	ConfigEventProcessReasonTransient ConfigEventProcessReason = "transient"
)

type ConfigStaleCipherCollector ¶

type ConfigStaleCipherCollector interface {
	RecordStaleCipher(ctx context.Context)
}

ConfigStaleCipherCollector records reads of config values that are encrypted with a non-current key version (M3 stale-key observability).

func NewProviderConfigStaleCipherCollector(p kernelmetrics.Provider) (ConfigStaleCipherCollector, error)

type EventRouterCollector ¶

type EventRouterCollector struct {
	// contains filtered or unexported fields
}

EventRouterCollector registers event-router lifecycle metrics:

• event_router_subscriptions_active{cell} (Gauge): active subscription count
• event_router_setup_errors_total{cell,topic,reason} (Counter): setup-phase failures (Phase 1-3)
• event_router_ready_wait_seconds{cell} (Histogram): time waited for Ready signal
• event_router_runtime_errors_total{cell,topic,reason} (Counter): runtime-phase faults (Phase 4)

Histogram buckets for ready_wait_seconds are: 0.001, 0.01, 0.1, 0.5, 1, 5, 30 seconds. Ready usually completes in <100ms; bootstrap timeout is 30s, so the upper bound covers the full expected range.

PR #593 review fix-up (P2#3): the setup vs runtime metrics partition the router lifecycle by phase, NOT by error variety. event_router_setup_errors_total owns Phase 1–3 failures; event_router_runtime_errors_total owns Phase 4 faults. The reason label set is therefore disjoint across the two metrics:

• event_router_setup_errors_total reasons: "setup_error": Subscriber.Setup failed (Phase 1) "panic": Subscribe goroutine panicked (Phase 2) "ready_timeout": Ready not signaled within timeout (Phase 3) "subscribe_failure": SubscribeEntry failed before Running() (Phase 3)
• event_router_runtime_errors_total reasons: "runtime_fault": any fault detected in Phase 4 (after Running())

ref: Watermill router metrics middleware — subscription lifecycle counters and gauges matching the router_messages_processed_total / router_handler_active pattern.

func NewEventRouterCollector(p kernelmetrics.Provider) (*EventRouterCollector, error)

func (c *EventRouterCollector) DecSubscriptionActive(ctx context.Context, cellID string)

func (c *EventRouterCollector) IncSubscriptionActive(ctx context.Context, cellID string)

func (c *EventRouterCollector) ObserveReadyWait(ctx context.Context, cellID string, d time.Duration)

func (c *EventRouterCollector) RecordRuntimeError(ctx context.Context, cellID, topic string, reason eventrouter.RuntimeErrorReason)

func (c *EventRouterCollector) RecordSetupError(ctx context.Context, cellID, topic, reason string)

type EventbusCacheCollector ¶

type EventbusCacheCollector interface {
	RecordTombstoneEvicted(ctx context.Context, cellID, sliceID string)
}

EventbusCacheCollector records eventbus subscriber-cache lifecycle metrics.

func NewProviderEventbusCacheCollector(p kernelmetrics.Provider) (EventbusCacheCollector, error)

type GRPCCollector ¶

type GRPCCollector interface {
	RecordRPC(ctx context.Context, cell CellLabel, method, code string, durationSeconds float64)
}

GRPCCollector records gRPC unary server request metrics. It mirrors the HTTP Collector but emits a distinct metric family (grpc_server_*) with a gRPC-shaped label set: method (the full RPC method, e.g. "/pkg.Service/Method"), code (the gRPC status code name, e.g. "OK" / "Internal"), and cell (the coarse owner dimension, RuntimeCellSentinel until cell attribution is wired for gRPC).

The method is named RecordRPC (not RecordRequest like the HTTP Collector) because the gRPC signature is intentionally distinct — code (a gRPC status code name) replaces the HTTP status int + route; the two interfaces are not unified.

cell is the sealed CellLabel from ResolveCellLabel; the collector never infers it. gRPC cell attribution is not yet wired, so the interceptor passes ResolveCellLabel(ctx, nil), which resolves to RuntimeCellSentinel ("_runtime") until #1383 threads a real closed set.

func NewGRPCProviderCollector(p kernelmetrics.Provider, cfg ProviderCollectorConfig) (GRPCCollector, error)

type GRPCRequestKey ¶

type GRPCRequestKey struct {
	Cell   string
	Method string
	Code   string
}

GRPCRequestKey identifies one gRPC request metric series. Method holds the full RPC method ("/pkg.Service/Method"); its cardinality is bounded because gRPC methods are a registration-time enumerated set (unlike an HTTP path, which is collapsed to a route template to bound cardinality — gRPC needs no such templating).

type IdempotencyCollector ¶

type IdempotencyCollector struct {
	// contains filtered or unexported fields
}

IdempotencyCollector records idempotency_requests_total{cell,state} — one increment per terminal HTTP idempotency decision made by the idempotency middleware. It implements idempotency.MetricsObserver.

Label semantics:

• {cell}: coarse owner dimension read from ctx via kernel/ctxkeys.CellID (written by the router root CellAttribution middleware). Falls back to RuntimeCellSentinel ("_runtime") for framework paths or requests that do not match any registered RouteGroup. The resolution is identical to the one used by the HTTP metrics middleware for http_requests_total{cell}.

• {state}: terminal idempotency decision, one of the six idempotency.RequestState constants. The value set is frozen by archtest IDEMPOTENCY-REQUESTS-STATE-LABEL-VALUES-FROZEN-01 (A1 freezes the constant set; A2 forbids inline literals or RequestState("x") conversions from reaching the label, requiring every value to flow from one of the declared constants). The string-typed-enum funnel mirrors executor.TickResult / DriveResult / LeaderSkipReason and reconcile resultLabel.

State relationship: StateAcquired ⊇ StateOversize. An acquired request whose response body exceeds the buffer limit emits both StateAcquired and StateOversize; the ratio oversize/acquired is the uncacheable-response rate.

Cardinality: 6 state values × bounded cell set = tiny worst-case series count, well within the metrics provider cap of 2000. No per-request or per-key dimension is exposed (keys are SHA-256 hashed for logs only).

Caller contract: to disable metric emission, pass nil to the middleware's idempotency.WithMetrics option — it silently skips a nil/typed-nil observer, so the middleware runs unmetered. Never pass a nil *IdempotencyCollector to ObserveRequest directly — it will dereference a nil CounterVec and panic.

func NewIdempotencyCollector(p kernelmetrics.Provider) (*IdempotencyCollector, error)

func (c *IdempotencyCollector) ObserveRequest(ctx context.Context, state idemhttp.RequestState)

type InMemoryCollector ¶

type InMemoryCollector struct {
	// contains filtered or unexported fields
}

InMemoryCollector is a simple in-memory metrics collector for development and testing. It records request counts, cumulative duration, and body-limit rejection counts.

func NewInMemoryCollector() *InMemoryCollector

func (c *InMemoryCollector) Handler() http.Handler

{"data": {"requests": [...], "bodyLimitRejections": [...]}}

func (c *InMemoryCollector) RecordBodyLimitRejection(_ context.Context, cell CellLabel, route string)

func (c *InMemoryCollector) RecordRequest(_ context.Context, cell CellLabel, method, route string, status int, durationSeconds float64)

func (c *InMemoryCollector) Snapshot() Snapshot

type InMemoryGRPCCollector ¶

type InMemoryGRPCCollector struct {
	// contains filtered or unexported fields
}

InMemoryGRPCCollector is a simple in-memory GRPCCollector for development and testing. It records request counts and cumulative duration per label set.

func NewInMemoryGRPCCollector() *InMemoryGRPCCollector

func (c *InMemoryGRPCCollector) Count(cellID, method, code string) int64

func (c *InMemoryGRPCCollector) RecordRPC(_ context.Context, cell CellLabel, method, code string, durationSeconds float64)

type NoopConfigEventCollector ¶

type NoopConfigEventCollector struct{}

NoopConfigEventCollector drops config event observations.

func (NoopConfigEventCollector) RecordEventProcess(_ context.Context, _, _ string, _ ConfigEventProcessReason)

func (NoopConfigEventCollector) RecordEventSettlement(_ context.Context, _, _, _ string, _ outbox.SettlementResult)

type NoopConfigStaleCipherCollector ¶

type NoopConfigStaleCipherCollector struct{}

NoopConfigStaleCipherCollector drops stale-cipher observations.

func (NoopConfigStaleCipherCollector) RecordStaleCipher(_ context.Context)

type NoopEventbusCacheCollector ¶

type NoopEventbusCacheCollector struct{}

NoopEventbusCacheCollector drops eventbus cache observations.

func (NoopEventbusCacheCollector) RecordTombstoneEvicted(_ context.Context, _, _ string)

type OutboxPendingDepthCollector ¶

type OutboxPendingDepthCollector struct {
	// contains filtered or unexported fields
}

OutboxPendingDepthCollector registers outbox_pending_depth{cell} scoped to the owner cell supplied at construction. One collector per relay.

Metric registered:

• outbox_pending_depth{cell}: current ELIGIBLE pending outbox entry depth — rows with status=pending AND (next_retry_at IS NULL OR next_retry_at <= now()). Rows still in retry backoff are EXCLUDED, matching the ClaimPending eligibility predicate (Store.CountPending). Set on each Relay reclaim tick.

Sampling cadence is bound to Relay.ReclaimInterval (defaults to runtime/outbox.DefaultRelayReclaimInterval = 30s), not Prometheus scrape interval. SREs should treat this Gauge as "eligible depth at last reclaim tick", not real-time. A growing retry backlog will not inflate this value — sustained retry backlog must be diagnosed via the relay-side outcome counter outbox_relayed_total{outcome="dead"|"lost"} or the reclaim-budget readyz probe (outbox_consumer_rejected_total is a subscriber-side DLX counter, not a relay-side retry indicator).

PR #593 review fix-up (P2#5) aligned the help text and dashboards with the eligibility semantics introduced when Store.CountPending narrowed to claimable rows.

ref: Watermill router metrics middleware — gauge mirroring router queue depth.

func NewOutboxPendingDepthCollector(p kernelmetrics.Provider, cellID string) (*OutboxPendingDepthCollector, error)

func (c *OutboxPendingDepthCollector) ObservePendingDepth(ctx context.Context, n int64)

type OutboxRejectCollector ¶

type OutboxRejectCollector struct {
	// contains filtered or unexported fields
}

OutboxRejectCollector registers outbox_consumer_rejected_total{cell,topic,reason}.

This collector is multi-cell-shared: the cell label flows from the ObserveReject call-site argument, NOT from construction time. There is therefore no cellID constructor argument. One instance covers all cells.

Metric registered:

• outbox_consumer_rejected_total{cell,topic,reason}: total terminal Reject dispositions observed by ConsumerBase. consumerGroup is excluded from the label set to keep time-series cardinality bounded; one cell may have multiple consumer groups sharing the same counter family.

ref: Watermill router metrics middleware — per-handler counters mirroring the router's router_messages_processed_total pattern.

func NewOutboxRejectCollector(p kernelmetrics.Provider) (*OutboxRejectCollector, error)

func (c *OutboxRejectCollector) ObserveReject(ctx context.Context, cellID, topic, _, reason string)

type ProviderCollectorConfig ¶

type ProviderCollectorConfig struct {
	// DurationBuckets overrides DefaultDurationBuckets; zero value uses defaults.
	DurationBuckets []float64
}

ProviderCollectorConfig configures NewProviderCollector.

Cell identity is not part of this config — it is derived per request by router-owned root attribution and supplied through RecordRequest's cellID argument. See D1 HTTP-METRICS-LABEL-REALIGN.

type RequestKey ¶

type RequestKey struct {
	Cell   string
	Method string
	Route  string
	Status int
}

RequestKey identifies one low-cardinality HTTP request metric series.

type SagaCollector ¶

type SagaCollector struct {
	// contains filtered or unexported fields
}

SagaCollector registers the six saga metrics scoped to the owner cell supplied at construction. One collector per Coordinator (= per-cell). It satisfies executor.Observer, the saga-wide observability sink, so the same collector receives both Executor-emitted (step-level) and Coordinator-emitted (tick / drive / leader-skip) events.

Step-level metrics (Executor-emitted):

• saga_step_outcome_total{cell,definition_id,outcome}: total Execute calls terminated, labeled by the 5 Outcome variants (succeeded / failed / expired / canceled / lease_lost). step_name is intentionally NOT a label to keep the {step × outcome} Cartesian product bounded — outcome-level dashboards are aggregated across steps.
• saga_step_retry_total{cell,definition_id,step_name}: total retry attempts emitted between Execute runAttempt invocations (attempts ≥ 2). step_name IS a label here because retry-budget tuning is per-step.
• saga_heartbeat_failed_total{cell,reason}: total heartbeat tick failures, labeled by reason (infra_error / stale_lease). definition_id and step_name are excluded — heartbeat failure is an infra-level signal, not a per-step business metric.

Coordinator-level metrics (Coordinator-emitted):

• saga_tick_total{cell,result}: total ClaimPending cycles, labeled by result (claimed / empty / error). Loop liveness + claim activity; definition_id is not available pre-claim.
• saga_drive_total{cell,definition_id,result}: total driveOne completions, labeled by result (ok / error). Per-instance forward-progress throughput.
• saga_leader_elect_skip_total{cell,definition_id,reason}: total leader-elect skips, labeled by reason (contended / ctx_canceled / backend_error). reason="contended" sustained with drive{result="ok"}≈0 signals an instance stuck skipping; reason="backend_error" is the distlock lock-acquire failure rate. Replaces log-scraping the Debug-level skip path (#1109).

Cardinality discipline: definition_id × step_name is the worst case; all label dimensions are static enumeration sets (registered at compile time). result/reason add ≤3 values each over a bounded definition_id set. Expected upper bound: ≤ 300 active series per cell; metrics provider cap=2000 is the runtime tripwire.

ref: temporalio/sdk-go internal_task_handlers.go — server-emitted activity outcome / heartbeat-failure metric envelope. ref: itimofeev/go-saga (no native metrics — pattern adapted from outbox).

func NewSagaCollector(p kernelmetrics.Provider, cellID string) (*SagaCollector, error)

func (c *SagaCollector) ObserveDrive(ctx context.Context, definitionID string, result executor.DriveResult)

func (c *SagaCollector) ObserveHeartbeatFailure(
	ctx context.Context,
	_ idutil.SafeID,
	_ idutil.SafeID,
	reason executor.HeartbeatFailureReason,
)

func (c *SagaCollector) ObserveLeaderSkip(ctx context.Context, definitionID string, reason executor.LeaderSkipReason)

func (c *SagaCollector) ObserveOutcome(
	ctx context.Context,
	_ idutil.SafeID,
	_ idutil.SafeID,
	definitionID, _ string,
	outcome executor.Outcome,
	_ int,
)

func (c *SagaCollector) ObserveRetry(
	ctx context.Context,
	_ idutil.SafeID,
	_ idutil.SafeID,
	definitionID, stepName string,
)

func (c *SagaCollector) ObserveTick(ctx context.Context, result executor.TickResult)

type SessionCacheCollector ¶

type SessionCacheCollector struct {
	// contains filtered or unexported fields
}

SessionCacheCollector registers the four session-cache metrics scoped to the owner cell supplied at construction. One collector per CachingSessionStore (= per-cell; the session cache is always owned by exactly one cell — accesscore today). It is the Prometheus sink for the read-through Redis session cache (adapters/redis.CachingSessionStore), distinguishing cache-hit-rate erosion from partial Redis faults that previously were only visible in slog (#794).

• session_cache_hits_total{cell}: clean cache hits (a well-formed, valid entry was served from Redis without touching the inner store).
• session_cache_misses_total{cell}: cache misses (no entry in Redis; the inner store was consulted and the result lazily populated).
• session_cache_errors_total{cell}: read-path cache-access errors (Redis GET/SET failure, corrupt JSON, or a schema-invalid entry). All are fail-safe — they degrade to a miss and never propagate to the caller — so errors ⊆ misses holds — but their rate distinguishes "cache cold" from "Redis degraded". A single Get may emit both a miss and an error (e.g. a corrupt entry → evict-and-fall-through): the two series are orthogonal events, not mutually exclusive.
• session_cache_revoke_del_errors_total{cell}: post-commit single-session revoke cache-DEL failures (#796). A DISTINCT series from session_cache_errors_total — it is NOT a read-path miss (it fires on the committing goroutine after a durable revoke, outside any Get), so folding it into errors_total would break the read-path errors ⊆ misses math (every other error has a paired miss; a revoke DEL failure does not). A non-zero rate is a security-relevant signal: the logged-out session's invalidation degraded from near-zero to "expires at TTL" (stale ≤ TTL, the accepted backstop — see adapters/redis.CachingSessionStore godoc).

Cardinality: the sole label dimension is `cell`, a registration-time enumerated value (the owning cell ID), so total series = number of cells with a session cache enabled (1 today).

The cell label is baked at construction (mirrors SagaCollector) rather than resolved per-record via metrics.ResolveCellLabel: the session cache is a per-cell adapter primitive, not request-scoped framework middleware, so its owner is fixed at wiring time.

ref: prometheus/client_golang prometheus/counter.go (CounterVec). ref: runtime/observability/metrics/saga.go (per-cell collector pattern).

func NewSessionCacheCollector(p kernelmetrics.Provider, cellID string) (*SessionCacheCollector, error)

func (c *SessionCacheCollector) RecordError(ctx context.Context)

func (c *SessionCacheCollector) RecordHit(ctx context.Context)

func (c *SessionCacheCollector) RecordMiss(ctx context.Context)

func (c *SessionCacheCollector) RecordRevokeDelError(ctx context.Context)

type ShutdownCollector ¶

type ShutdownCollector struct {
	// contains filtered or unexported fields
}

ShutdownCollector groups the three observability signals emitted during phase10OrchestrateShutdown. A nil *ShutdownCollector is safe to call — all methods are nil-receiver guards. This enables the "disabled without provider" contract without nil checks at every call site.

Design decision (plan option B): ShutdownCollector is created once in New() and stored on Bootstrap so metric instruments are registered against the Provider at construction time, matching the "register at start-up" pattern used by relay_collector.go and the kernel hook dispatcher.

func NewShutdownCollector(p kernelmetrics.Provider) (*ShutdownCollector, error)

func (m *ShutdownCollector) CountOutcome(ctx context.Context, outcome string)

func (m *ShutdownCollector) ObservePhaseDuration(ctx context.Context, phase string, d time.Duration)

func (m *ShutdownCollector) RecordPhaseEntry(ctx context.Context, phase string)

type Snapshot ¶

type Snapshot struct {
	RequestCounts       map[RequestKey]int64
	DurationSumsMs      map[RequestKey]int64
	BodyLimitRejections map[BodyLimitRejectionKey]int64
}

Snapshot is a point-in-time view of recorded metrics.