metrics

package
v0.0.334 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 8, 2026 License: Apache-2.0 Imports: 3 Imported by: 0

Documentation

Overview

Package metrics defines the storage server's Prometheus metrics and registers them with the k8s.io/component-base legacyregistry, the same registry the generic apiserver's built-in "/metrics" endpoint already serves (see k8s.io/apiserver/pkg/server, EnableMetrics).

This currently covers Phase 0 of the storage-locking investigation (docs/features/storage-lock-pool-metrics.md): lock-hold and connection-pool-wait durations, so that contention which used to be visible only via Debug-level log lines becomes a real, always-on, percentile-queryable metric.

Index

Constants

View Source
const (
	OutcomeAcquired = "acquired"
	OutcomeTimeout  = "timeout"
)

Outcome label values for LockWaitDuration / PoolWaitDuration: whether the acquisition succeeded or hit its timeout.

View Source
const (
	PriorityHigh = "high"
	PriorityLow  = "low"
)

Priority label values for the single-writer priority queue metrics: which lane a commitJob traveled through.

View Source
const (
	CommitOutcomeCommitted = "committed"
	CommitOutcomeConflict  = "conflict"
	CommitOutcomeError     = "error"
	// CommitOutcomePanic: the commit panicked past every guard and the shard
	// goroutine's recover converted it to an error. Must stay zero.
	CommitOutcomePanic = "panic"
)

Outcome label values for SingleWriterCommitTotal: how a commit attempt concluded.

View Source
const (
	// FrozenReclaimedRow / FrozenReclaimedObject: what the frozen gate reclaimed.
	FrozenReclaimedRow    = "row"
	FrozenReclaimedObject = "object"

	// DivergencePayloadAhead: the payload says Completed/Full, the metadata row
	// does not (a process crash or a failed COMMIT between the payload rename
	// and the row's commit) -- healed by re-persisting the payload as-is.
	// DivergenceMetadataAhead: the metadata row says Completed/Full, the payload
	// does not (a lost payload rename after a power loss) -- observed only.
	DivergencePayloadAhead  = "payload_ahead"
	DivergenceMetadataAhead = "metadata_ahead"

	// HealFailed* : which step of the divergence heal failed.
	HealFailedLockTimeout = "lock_timeout"
	HealFailedBegin       = "begin"
	HealFailedRead        = "read"
	HealFailedSave        = "save"
	HealFailedCommit      = "commit"
)

Label values for the consolidation counters below.

Variables

View Source
var (
	// LockWaitDuration observes how long a caller waited to acquire the
	// per-key in-process lock (pkg/utils.MapMutex), labeled by resource
	// "kind" (see file.resourceFromKey) and "outcome" (acquired/timeout).
	LockWaitDuration = metrics.NewHistogramVec(
		&metrics.HistogramOpts{
			Subsystem:      "storage",
			Name:           "lock_wait_duration_seconds",
			Help:           "Time spent waiting to acquire the per-key in-process lock, by resource kind and outcome.",
			Buckets:        waitBuckets,
			StabilityLevel: metrics.ALPHA,
		},
		[]string{"kind", "outcome"},
	)

	// PoolWaitDuration observes how long a caller waited to acquire a SQLite
	// connection from the pool (sqlitemigration.Pool.Take), labeled by
	// resource "kind" and "outcome" (acquired/timeout).
	PoolWaitDuration = metrics.NewHistogramVec(
		&metrics.HistogramOpts{
			Subsystem:      "storage",
			Name:           "pool_wait_duration_seconds",
			Help:           "Time spent waiting to acquire a SQLite connection from the pool, by resource kind and outcome.",
			Buckets:        waitBuckets,
			StabilityLevel: metrics.ALPHA,
		},
		[]string{"kind", "outcome"},
	)

	// SingleWriterQueueWaitDuration observes how long a commitJob sat in the
	// single writer's high/low priority channel before the writer goroutine
	// (singleWriter.run) picked it up and started processing it. This is the
	// new contention point the single-writer design introduces by
	// centralizing every write through one goroutine, labeled by resource
	// "kind" and queue "priority" (high/low).
	SingleWriterQueueWaitDuration = metrics.NewHistogramVec(
		&metrics.HistogramOpts{
			Subsystem:      "storage",
			Name:           "single_writer_queue_wait_duration_seconds",
			Help:           "Time a commit job spent waiting in the single writer's priority queue before being picked up, by resource kind and priority.",
			Buckets:        waitBuckets,
			StabilityLevel: metrics.ALPHA,
		},
		[]string{"kind", "priority"},
	)

	// SingleWriterCommitTotal counts single-writer commit attempts, labeled by
	// resource "kind", queue "priority" (high/low), and "outcome"
	// (committed/conflict/error). This shows how often optimistic commits
	// succeed vs. hit a resourceVersion conflict vs. fail for another reason,
	// and whether high- vs low-priority jobs are serviced as expected.
	SingleWriterCommitTotal = metrics.NewCounterVec(
		&metrics.CounterOpts{
			Subsystem:      "storage",
			Name:           "single_writer_commit_total",
			Help:           "Count of single-writer commit attempts, by resource kind, priority, and outcome (committed/conflict/error/panic).",
			StabilityLevel: metrics.ALPHA,
		},
		[]string{"kind", "priority", "outcome"},
	)

	// SingleWriterDirtyConnectionTotal counts pool connections a shard commit
	// was about to return with an open transaction/savepoint or a stepped,
	// unreset statement (a panic in the commit path skipped the release).
	// The connection is rolled back and reset before reuse. Must stay zero.
	SingleWriterDirtyConnectionTotal = metrics.NewCounter(
		&metrics.CounterOpts{
			Subsystem:      "storage",
			Name:           "single_writer_dirty_connection_total",
			Help:           "Count of pool connections a single-writer commit found with an open transaction or unreset statement on release; rolled back before reuse.",
			StabilityLevel: metrics.ALPHA,
		},
	)

	// SingleWriterDroppedConnectionTotal counts dirty connections that could
	// not be rolled back and were dropped instead of returned, shrinking the
	// pool by one permanently. Must stay zero.
	SingleWriterDroppedConnectionTotal = metrics.NewCounter(
		&metrics.CounterOpts{
			Subsystem:      "storage",
			Name:           "single_writer_dropped_connection_total",
			Help:           "Count of dirty pool connections a single-writer commit could not roll back and dropped; each one shrinks the pool permanently.",
			StabilityLevel: metrics.ALPHA,
		},
	)

	// SingleWriterConflictRetryTotal counts how many times
	// guaranteedUpdateSingleWriter's retry loop redoes the prepare phase
	// because a commit was rejected with errWriteConflict, labeled by
	// resource "kind". This directly measures retry-storm risk under real
	// contention.
	SingleWriterConflictRetryTotal = metrics.NewCounterVec(
		&metrics.CounterOpts{
			Subsystem:      "storage",
			Name:           "single_writer_conflict_retry_total",
			Help:           "Count of GuaranteedUpdate prepare-phase retries caused by a resourceVersion conflict at commit time, by resource kind.",
			StabilityLevel: metrics.ALPHA,
		},
		[]string{"kind"},
	)

	// SingleWriterQueueDepth gauges the current number of commitJobs waiting
	// in each single-writer priority lane, updated on enqueue and dequeue.
	// This shows whether a backlog is building up under real load, labeled
	// by queue "priority" (high/low).
	SingleWriterQueueDepth = metrics.NewGaugeVec(
		&metrics.GaugeOpts{
			Subsystem:      "storage",
			Name:           "single_writer_queue_depth",
			Help:           "Current number of commit jobs waiting in the single writer's priority queue, by priority.",
			StabilityLevel: metrics.ALPHA,
		},
		[]string{"priority"},
	)

	// ConsolidationFrozenReclaimedTotal counts the time_series rows and TS
	// objects consolidation's frozen gate reclaimed unmerged because the base
	// ContainerProfile was already Completed/Full when the pass read it,
	// labeled by "what" (row/object). Expected low and non-zero on
	// multi-replica workloads (each late series is reclaimed once); rising for
	// one key on many consecutive ticks while divergence heals stay at zero
	// means a writer other than consolidation keeps producing rows for a
	// completed profile.
	ConsolidationFrozenReclaimedTotal = metrics.NewCounterVec(
		&metrics.CounterOpts{
			Subsystem:      "storage",
			Name:           "consolidation_frozen_reclaimed_total",
			Help:           "Count of time_series rows and TS objects reclaimed unmerged by consolidation because the base profile was already Completed/Full, by what (row/object).",
			StabilityLevel: metrics.ALPHA,
		},
		[]string{"what"},
	)

	// ConsolidationFrozenRefusalsTotal counts consolidation saves refused
	// because the persisted base ContainerProfile was Completed/Full at write
	// time (under the per-key lock) although it was not when the pass read it.
	// Expected zero: a non-zero value means a concurrent writer completed the
	// base between the pass's read and its write.
	ConsolidationFrozenRefusalsTotal = metrics.NewCounter(
		&metrics.CounterOpts{
			Subsystem:      "storage",
			Name:           "consolidation_frozen_refusals_total",
			Help:           "Count of consolidation saves refused because the persisted base profile became Completed/Full between the pass's read and its write.",
			StabilityLevel: metrics.ALPHA,
		},
	)

	// ConsolidationDivergenceTotal counts payload/metadata divergences
	// consolidation observed on a base ContainerProfile, by "shape"
	// (payload_ahead: healed; metadata_ahead: observed only). Expected zero in
	// steady state; payload_ahead after no pod restart means a COMMIT failed.
	ConsolidationDivergenceTotal = metrics.NewCounterVec(
		&metrics.CounterOpts{
			Subsystem:      "storage",
			Name:           "consolidation_divergence_total",
			Help:           "Count of payload/metadata divergences observed on a base profile by consolidation, by shape (payload_ahead healed, metadata_ahead observed).",
			StabilityLevel: metrics.ALPHA,
		},
		[]string{"shape"},
	)

	// ConsolidationHealFailedTotal counts failed divergence heals by the step
	// that failed (lock_timeout/begin/read/save/commit). A failing heal errors the
	// tick before the frozen gate runs, so ConsolidationFrozenReclaimedTotal
	// does not move; this series is what makes a wedged heal visible.
	ConsolidationHealFailedTotal = metrics.NewCounterVec(
		&metrics.CounterOpts{
			Subsystem:      "storage",
			Name:           "consolidation_heal_failed_total",
			Help:           "Count of failed payload/metadata divergence heals, by the step that failed (lock_timeout/begin/read/save/commit).",
			StabilityLevel: metrics.ALPHA,
		},
		[]string{"reason"},
	)
)

Functions

func IncConsolidationDivergence added in v0.0.334

func IncConsolidationDivergence(shape string)

IncConsolidationDivergence records one observed payload/metadata divergence of the given shape (DivergencePayloadAhead / DivergenceMetadataAhead).

func IncConsolidationFrozenReclaimed added in v0.0.334

func IncConsolidationFrozenReclaimed(what string, n int)

IncConsolidationFrozenReclaimed adds n to the frozen gate's reclaim count for what (FrozenReclaimedRow / FrozenReclaimedObject).

func IncConsolidationFrozenRefusals added in v0.0.334

func IncConsolidationFrozenRefusals()

IncConsolidationFrozenRefusals records one consolidation save refused on a persisted Completed/Full base profile.

func IncConsolidationHealFailed added in v0.0.334

func IncConsolidationHealFailed(reason string)

IncConsolidationHealFailed records one failed divergence heal for the given reason (HealFailedLockTimeout / HealFailedBegin / HealFailedRead / HealFailedSave / HealFailedCommit).

func IncSingleWriterCommit

func IncSingleWriterCommit(kind, priority, outcome string)

IncSingleWriterCommit records one single-writer commit attempt for the given resource kind, priority (PriorityHigh / PriorityLow), and outcome (CommitOutcomeCommitted / CommitOutcomeConflict / CommitOutcomeError / CommitOutcomePanic).

func IncSingleWriterConflictRetry

func IncSingleWriterConflictRetry(kind string)

IncSingleWriterConflictRetry records one GuaranteedUpdate prepare-phase retry caused by a resourceVersion conflict at commit time, for the given resource kind.

func IncSingleWriterDirtyConnection added in v0.0.334

func IncSingleWriterDirtyConnection()

IncSingleWriterDirtyConnection records one pool connection found dirty on release from a single-writer commit.

func IncSingleWriterDroppedConnection added in v0.0.334

func IncSingleWriterDroppedConnection()

IncSingleWriterDroppedConnection records one dirty pool connection that could not be rolled back and was dropped instead of returned.

func ObserveLockWait

func ObserveLockWait(kind, outcome string, d time.Duration)

ObserveLockWait records a lock-hold-wait observation for the given resource kind and outcome (OutcomeAcquired / OutcomeTimeout).

func ObservePoolWait

func ObservePoolWait(kind, outcome string, d time.Duration)

ObservePoolWait records a connection-pool-wait observation for the given resource kind and outcome (OutcomeAcquired / OutcomeTimeout).

func ObserveSingleWriterQueueWait

func ObserveSingleWriterQueueWait(kind, priority string, d time.Duration)

ObserveSingleWriterQueueWait records how long a commitJob waited in the single writer's priority queue for the given resource kind and priority (PriorityHigh / PriorityLow) before being picked up by the writer goroutine.

func SetSingleWriterQueueDepth

func SetSingleWriterQueueDepth(priority string, depth int)

SetSingleWriterQueueDepth sets the current number of commit jobs waiting in the given priority lane (PriorityHigh / PriorityLow).

Types

This section is empty.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL