Documentation
¶
Overview ¶
Package metrics defines the storage server's Prometheus metrics and registers them with the k8s.io/component-base legacyregistry, the same registry the generic apiserver's built-in "/metrics" endpoint already serves (see k8s.io/apiserver/pkg/server, EnableMetrics).
This currently covers Phase 0 of the storage-locking investigation (docs/features/storage-lock-pool-metrics.md): lock-hold and connection-pool-wait durations, so that contention which used to be visible only via Debug-level log lines becomes a real, always-on, percentile-queryable metric.
Index ¶
- Constants
- Variables
- func IncConsolidationDivergence(shape string)
- func IncConsolidationFrozenReclaimed(what string, n int)
- func IncConsolidationFrozenRefusals()
- func IncConsolidationHealFailed(reason string)
- func IncSingleWriterCommit(kind, priority, outcome string)
- func IncSingleWriterConflictRetry(kind string)
- func IncSingleWriterDirtyConnection()
- func IncSingleWriterDroppedConnection()
- func ObserveLockWait(kind, outcome string, d time.Duration)
- func ObservePoolWait(kind, outcome string, d time.Duration)
- func ObserveSingleWriterQueueWait(kind, priority string, d time.Duration)
- func SetSingleWriterQueueDepth(priority string, depth int)
Constants ¶
const ( OutcomeAcquired = "acquired" OutcomeTimeout = "timeout" )
Outcome label values for LockWaitDuration / PoolWaitDuration: whether the acquisition succeeded or hit its timeout.
const ( PriorityHigh = "high" PriorityLow = "low" )
Priority label values for the single-writer priority queue metrics: which lane a commitJob traveled through.
const ( CommitOutcomeCommitted = "committed" CommitOutcomeConflict = "conflict" CommitOutcomeError = "error" // CommitOutcomePanic: the commit panicked past every guard and the shard // goroutine's recover converted it to an error. Must stay zero. CommitOutcomePanic = "panic" )
Outcome label values for SingleWriterCommitTotal: how a commit attempt concluded.
const ( // FrozenReclaimedRow / FrozenReclaimedObject: what the frozen gate reclaimed. FrozenReclaimedRow = "row" FrozenReclaimedObject = "object" // DivergencePayloadAhead: the payload says Completed/Full, the metadata row // does not (a process crash or a failed COMMIT between the payload rename // and the row's commit) -- healed by re-persisting the payload as-is. // DivergenceMetadataAhead: the metadata row says Completed/Full, the payload // does not (a lost payload rename after a power loss) -- observed only. DivergencePayloadAhead = "payload_ahead" DivergenceMetadataAhead = "metadata_ahead" // HealFailed* : which step of the divergence heal failed. HealFailedLockTimeout = "lock_timeout" HealFailedBegin = "begin" HealFailedRead = "read" HealFailedSave = "save" HealFailedCommit = "commit" )
Label values for the consolidation counters below.
Variables ¶
var ( // LockWaitDuration observes how long a caller waited to acquire the // per-key in-process lock (pkg/utils.MapMutex), labeled by resource // "kind" (see file.resourceFromKey) and "outcome" (acquired/timeout). LockWaitDuration = metrics.NewHistogramVec( &metrics.HistogramOpts{ Subsystem: "storage", Name: "lock_wait_duration_seconds", Help: "Time spent waiting to acquire the per-key in-process lock, by resource kind and outcome.", Buckets: waitBuckets, StabilityLevel: metrics.ALPHA, }, []string{"kind", "outcome"}, ) // PoolWaitDuration observes how long a caller waited to acquire a SQLite // connection from the pool (sqlitemigration.Pool.Take), labeled by // resource "kind" and "outcome" (acquired/timeout). PoolWaitDuration = metrics.NewHistogramVec( &metrics.HistogramOpts{ Subsystem: "storage", Name: "pool_wait_duration_seconds", Help: "Time spent waiting to acquire a SQLite connection from the pool, by resource kind and outcome.", Buckets: waitBuckets, StabilityLevel: metrics.ALPHA, }, []string{"kind", "outcome"}, ) // SingleWriterQueueWaitDuration observes how long a commitJob sat in the // single writer's high/low priority channel before the writer goroutine // (singleWriter.run) picked it up and started processing it. This is the // new contention point the single-writer design introduces by // centralizing every write through one goroutine, labeled by resource // "kind" and queue "priority" (high/low). SingleWriterQueueWaitDuration = metrics.NewHistogramVec( &metrics.HistogramOpts{ Subsystem: "storage", Name: "single_writer_queue_wait_duration_seconds", Help: "Time a commit job spent waiting in the single writer's priority queue before being picked up, by resource kind and priority.", Buckets: waitBuckets, StabilityLevel: metrics.ALPHA, }, []string{"kind", "priority"}, ) // SingleWriterCommitTotal counts single-writer commit attempts, labeled by // resource "kind", queue "priority" (high/low), and "outcome" // (committed/conflict/error). This shows how often optimistic commits // succeed vs. hit a resourceVersion conflict vs. fail for another reason, // and whether high- vs low-priority jobs are serviced as expected. SingleWriterCommitTotal = metrics.NewCounterVec( &metrics.CounterOpts{ Subsystem: "storage", Name: "single_writer_commit_total", Help: "Count of single-writer commit attempts, by resource kind, priority, and outcome (committed/conflict/error/panic).", StabilityLevel: metrics.ALPHA, }, []string{"kind", "priority", "outcome"}, ) // SingleWriterDirtyConnectionTotal counts pool connections a shard commit // was about to return with an open transaction/savepoint or a stepped, // unreset statement (a panic in the commit path skipped the release). // The connection is rolled back and reset before reuse. Must stay zero. SingleWriterDirtyConnectionTotal = metrics.NewCounter( &metrics.CounterOpts{ Subsystem: "storage", Name: "single_writer_dirty_connection_total", Help: "Count of pool connections a single-writer commit found with an open transaction or unreset statement on release; rolled back before reuse.", StabilityLevel: metrics.ALPHA, }, ) // SingleWriterDroppedConnectionTotal counts dirty connections that could // not be rolled back and were dropped instead of returned, shrinking the // pool by one permanently. Must stay zero. SingleWriterDroppedConnectionTotal = metrics.NewCounter( &metrics.CounterOpts{ Subsystem: "storage", Name: "single_writer_dropped_connection_total", Help: "Count of dirty pool connections a single-writer commit could not roll back and dropped; each one shrinks the pool permanently.", StabilityLevel: metrics.ALPHA, }, ) // SingleWriterConflictRetryTotal counts how many times // guaranteedUpdateSingleWriter's retry loop redoes the prepare phase // because a commit was rejected with errWriteConflict, labeled by // resource "kind". This directly measures retry-storm risk under real // contention. SingleWriterConflictRetryTotal = metrics.NewCounterVec( &metrics.CounterOpts{ Subsystem: "storage", Name: "single_writer_conflict_retry_total", Help: "Count of GuaranteedUpdate prepare-phase retries caused by a resourceVersion conflict at commit time, by resource kind.", StabilityLevel: metrics.ALPHA, }, []string{"kind"}, ) // SingleWriterQueueDepth gauges the current number of commitJobs waiting // in each single-writer priority lane, updated on enqueue and dequeue. // This shows whether a backlog is building up under real load, labeled // by queue "priority" (high/low). SingleWriterQueueDepth = metrics.NewGaugeVec( &metrics.GaugeOpts{ Subsystem: "storage", Name: "single_writer_queue_depth", Help: "Current number of commit jobs waiting in the single writer's priority queue, by priority.", StabilityLevel: metrics.ALPHA, }, []string{"priority"}, ) // ConsolidationFrozenReclaimedTotal counts the time_series rows and TS // objects consolidation's frozen gate reclaimed unmerged because the base // ContainerProfile was already Completed/Full when the pass read it, // labeled by "what" (row/object). Expected low and non-zero on // multi-replica workloads (each late series is reclaimed once); rising for // one key on many consecutive ticks while divergence heals stay at zero // means a writer other than consolidation keeps producing rows for a // completed profile. ConsolidationFrozenReclaimedTotal = metrics.NewCounterVec( &metrics.CounterOpts{ Subsystem: "storage", Name: "consolidation_frozen_reclaimed_total", Help: "Count of time_series rows and TS objects reclaimed unmerged by consolidation because the base profile was already Completed/Full, by what (row/object).", StabilityLevel: metrics.ALPHA, }, []string{"what"}, ) // ConsolidationFrozenRefusalsTotal counts consolidation saves refused // because the persisted base ContainerProfile was Completed/Full at write // time (under the per-key lock) although it was not when the pass read it. // Expected zero: a non-zero value means a concurrent writer completed the // base between the pass's read and its write. ConsolidationFrozenRefusalsTotal = metrics.NewCounter( &metrics.CounterOpts{ Subsystem: "storage", Name: "consolidation_frozen_refusals_total", Help: "Count of consolidation saves refused because the persisted base profile became Completed/Full between the pass's read and its write.", StabilityLevel: metrics.ALPHA, }, ) // ConsolidationDivergenceTotal counts payload/metadata divergences // consolidation observed on a base ContainerProfile, by "shape" // (payload_ahead: healed; metadata_ahead: observed only). Expected zero in // steady state; payload_ahead after no pod restart means a COMMIT failed. ConsolidationDivergenceTotal = metrics.NewCounterVec( &metrics.CounterOpts{ Subsystem: "storage", Name: "consolidation_divergence_total", Help: "Count of payload/metadata divergences observed on a base profile by consolidation, by shape (payload_ahead healed, metadata_ahead observed).", StabilityLevel: metrics.ALPHA, }, []string{"shape"}, ) // ConsolidationHealFailedTotal counts failed divergence heals by the step // that failed (lock_timeout/begin/read/save/commit). A failing heal errors the // tick before the frozen gate runs, so ConsolidationFrozenReclaimedTotal // does not move; this series is what makes a wedged heal visible. ConsolidationHealFailedTotal = metrics.NewCounterVec( &metrics.CounterOpts{ Subsystem: "storage", Name: "consolidation_heal_failed_total", Help: "Count of failed payload/metadata divergence heals, by the step that failed (lock_timeout/begin/read/save/commit).", StabilityLevel: metrics.ALPHA, }, []string{"reason"}, ) )
Functions ¶
func IncConsolidationDivergence ¶ added in v0.0.334
func IncConsolidationDivergence(shape string)
IncConsolidationDivergence records one observed payload/metadata divergence of the given shape (DivergencePayloadAhead / DivergenceMetadataAhead).
func IncConsolidationFrozenReclaimed ¶ added in v0.0.334
IncConsolidationFrozenReclaimed adds n to the frozen gate's reclaim count for what (FrozenReclaimedRow / FrozenReclaimedObject).
func IncConsolidationFrozenRefusals ¶ added in v0.0.334
func IncConsolidationFrozenRefusals()
IncConsolidationFrozenRefusals records one consolidation save refused on a persisted Completed/Full base profile.
func IncConsolidationHealFailed ¶ added in v0.0.334
func IncConsolidationHealFailed(reason string)
IncConsolidationHealFailed records one failed divergence heal for the given reason (HealFailedLockTimeout / HealFailedBegin / HealFailedRead / HealFailedSave / HealFailedCommit).
func IncSingleWriterCommit ¶
func IncSingleWriterCommit(kind, priority, outcome string)
IncSingleWriterCommit records one single-writer commit attempt for the given resource kind, priority (PriorityHigh / PriorityLow), and outcome (CommitOutcomeCommitted / CommitOutcomeConflict / CommitOutcomeError / CommitOutcomePanic).
func IncSingleWriterConflictRetry ¶
func IncSingleWriterConflictRetry(kind string)
IncSingleWriterConflictRetry records one GuaranteedUpdate prepare-phase retry caused by a resourceVersion conflict at commit time, for the given resource kind.
func IncSingleWriterDirtyConnection ¶ added in v0.0.334
func IncSingleWriterDirtyConnection()
IncSingleWriterDirtyConnection records one pool connection found dirty on release from a single-writer commit.
func IncSingleWriterDroppedConnection ¶ added in v0.0.334
func IncSingleWriterDroppedConnection()
IncSingleWriterDroppedConnection records one dirty pool connection that could not be rolled back and was dropped instead of returned.
func ObserveLockWait ¶
ObserveLockWait records a lock-hold-wait observation for the given resource kind and outcome (OutcomeAcquired / OutcomeTimeout).
func ObservePoolWait ¶
ObservePoolWait records a connection-pool-wait observation for the given resource kind and outcome (OutcomeAcquired / OutcomeTimeout).
func ObserveSingleWriterQueueWait ¶
ObserveSingleWriterQueueWait records how long a commitJob waited in the single writer's priority queue for the given resource kind and priority (PriorityHigh / PriorityLow) before being picked up by the writer goroutine.
func SetSingleWriterQueueDepth ¶
SetSingleWriterQueueDepth sets the current number of commit jobs waiting in the given priority lane (PriorityHigh / PriorityLow).
Types ¶
This section is empty.