metrics

package
v0.2.0-alpha.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 5, 2026 License: Apache-2.0 Imports: 9 Imported by: 0

README

pkg/controller/metrics

Domain metrics for the HAProxy Template Ingress Controller. Two things live here:

  • A Metrics struct that owns every controller-defined Prometheus metric (instance-based prometheus.Registry, not the global default).
  • A Component event adapter that subscribes to controller events and updates metrics accordingly.

User-facing queries, alerting rules, and dashboard templates live in docs/controller/docs/operations/monitoring.md. This README is the authoritative developer reference for which metrics exist and which component owns them.

Complete Metric Catalogue

All names are listed exactly as exported. metrics.go contains the authoritative list; the TestMetrics_AllMetricsRegistered assertion in metrics_test.go covers a representative subset (~12 of 32) — extend that slice when you add or rename a metric.

Reconciliation pipeline
Metric Type Labels What it tracks
haptic_reconciliation_total counter — Reconciliation cycles triggered
haptic_reconciliation_errors_total counter — Reconciliations that failed
haptic_reconciliation_duration_seconds histogram — End-to-end reconciliation wall-clock
haptic_reconciliation_queue_wait_seconds histogram — Time between ReconciliationTriggeredEvent and the pipeline actually picking it up (debounce + queue depth)
Deployment
Metric Type Labels What it tracks
haptic_deployment_total counter — Deployments dispatched to at least one HAProxy endpoint
haptic_deployment_errors_total counter — Deployments that failed
haptic_deployment_duration_seconds histogram — Deployment duration, aggregated across all parallel endpoint calls
haptic_haproxy_reloads_total counter — HAProxy reloads triggered by deployments. A reload forks the HAProxy process; reload rate (vs runtime-API updates) is the key capacity/SLO signal
haptic_dataplane_api_operations_total counter — DataPlane API operations issued across deployments (structural changes applied to HAProxy)
Runtime-eligible fast path
Metric Type Labels What it tracks
haptic_runtime_fast_path_fires_total counter — Fast-path attempts (one per HAProxy pod per reconcile)
haptic_runtime_fast_path_applies_total counter — Attempts that applied ≥1 runtime-eligible server update
haptic_runtime_fast_path_failures_total counter — Attempts that errored (best-effort; the scheduled deploy converges)
haptic_runtime_fast_path_server_updates_total counter — Runtime-eligible server updates applied via the fast path

applies_total flat at 0 while fires_total climbs means the fast path runs but finds no runtime-eligible change to apply (the deploy is keeping pods current) — the signal that separates a healthy-but-idle fast path from a broken one.

Config validation
Metric Type Labels What it tracks
haptic_validation_total counter — Controller-side validations (haproxy -c + parser)
haptic_validation_errors_total counter — Controller-side validation failures
Watched resources
Metric Type Labels What it tracks
haptic_resource_count gauge type Current size of each watched-resource store (including haproxy-pods)
HAProxy pod discovery
Metric Type Labels What it tracks
haptic_haproxy_pods_rejected_total counter reason HAProxy pods refused admission by the discovery component. Persistent non-zero growth typically means the controller cannot talk to the deployed HAProxy pods (e.g. bundled HAProxy major.minor differs from the chart's haproxyVersion).
Event bus
Metric Type Labels What it tracks
haptic_event_subscribers gauge — Live subscribers on the EventBus. Drops during normal ops usually indicate a crash
haptic_events_published_total counter — Total publishes
haptic_events_dropped_total counter — Publishes where the subscriber's channel was full
haptic_events_dropped_critical_total counter — Drops where the buffered event was marked critical
haptic_events_dropped_by_subscriber_total counter subscriber, event_type Drops attributed to each subscriber/event-type pair (the second label lets dashboards split by which event type the subscriber couldn't keep up with)
haptic_events_dropped_observability_total gauge — Drops to the observability subscribers (commentator, debug buffer); expected to be low but non-zero on bursts
Webhook
Metric Type Labels What it tracks
haptic_webhook_requests_total counter gvk, result Admission requests by kind and allow/deny/error
haptic_webhook_request_duration_seconds histogram gvk Per-request wall-clock
haptic_webhook_validation_total counter gvk, result Validation-only tally (no timing) — handy for ratios
Leader election
Metric Type Labels What it tracks
haptic_leader_election_is_leader gauge — 1 if this replica holds the lease, else 0
haptic_leader_election_transitions_total counter — Leadership changes observed
haptic_leader_election_time_as_leader_seconds_total counter — Cumulative seconds spent as leader
Dataplane parser cache
Metric Type Labels What it tracks
haptic_parser_cache_hits_total counter — client-native parser result cache hits
haptic_parser_cache_misses_total counter — client-native parser result cache misses
Build info
Metric Type Labels What it tracks
haptic_build_info gauge (always 1) version, haproxy_version, go_version Static metadata for dashboards

Component

Component subscribes to reconciliation, deployment, validation, resource, event-bus, webhook, leader-election, and parser events. Metric updates happen inside the internal handleEvent dispatcher — there's no direct caller path into the Metrics struct from other components; they emit events and this component records them.

import (
    "gitlab.com/haproxy-haptic/haptic/pkg/controller/metrics"
    pkgmetrics "gitlab.com/haproxy-haptic/haptic/pkg/metrics"

    "github.com/prometheus/client_golang/prometheus"
)

registry := prometheus.NewRegistry()          // instance-based, swapped per controller iteration
m := metrics.NewMetrics(registry)              // single arg — Registerer only
m.SetBuildInfo(version, haproxyVersion, goVersion) // populate the haptic_build_info gauge

comp := metrics.New(m, bus)                    // (*Metrics, *EventBus) — subscribes during construction
go comp.Start(ctx)                             // runs the event loop

// Serve on the port pkg/metrics exposes
pkgmetrics.NewServer(":9090", registry).Start(ctx)

The Metrics struct is intentionally safe to use stand-alone (CLI validation, tests) without the Component — call the typed update methods directly when you already have a value.

Why an instance-based registry?

The controller's reinitialisation loop creates a fresh EventBus on every config change. Using the global default registry would leak prior-iteration collectors into the new one and produce duplicate-registration panics. NewMetrics always takes a caller-provided registry, which the controller swaps per iteration; Prometheus sees a clean slate with the same metric names each time.

Resource-count tracking

ResourceCount is a gauge with a type label. The component seeds it from IndexSynchronizedEvent (absolute counts per resource type) and then applies deltas from ResourceIndexUpdatedEvent (created − deleted), skipping events marked IsInitialSync. The running totals live in a map[string]int on the component struct, so gauge values match reality across churn without re-listing from the API server.

Dropping or Renaming a Metric

  • The TestMetrics_AllMetricsRegistered assertion covers a representative subset of the exported names. Update that slice when you add, rename, or remove one — and ideally extend it so every name is guarded.
  • Dashboards and alert rules in docs/controller/docs/operations/monitoring.md reference names too; keep that file in sync or link the dashboard PR to the metric PR.

Testing

go test ./pkg/controller/metrics/...          # unit tests
go test ./pkg/controller/metrics/... -race    # race detector

component_test.go asserts that publishing each event type produces exactly the expected metric update — no surprise side-effects, no accidental double counting.

See Also

License

Apache-2.0 — see root LICENSE.

Documentation

Index

Constants

View Source 
const ComponentName = "metrics"

ComponentName is the unique identifier for the metrics component.

Variables

This section is empty.

Functions

This section is empty.

Types

type Component 

type Component struct {
	// contains filtered or unexported fields
}

Component is an event-driven metrics collector.

Subscribes to controller events and updates metrics via the Metrics struct. This is an event adapter that bridges domain events to Prometheus metrics.

IMPORTANT: Instance-based, created fresh per application iteration. When the iteration ends (context cancelled), the component stops and the metrics it was updating become eligible for garbage collection.

func New 

func New(metrics *Metrics, eventBus *busevents.EventBus) *Component

New creates a new metrics component that listens to events.

Parameters:

  • metrics: The Metrics instance to update (created with metrics.NewMetrics)
  • eventBus: The EventBus to subscribe to for events

Usage: 

registry := prometheus.NewRegistry()
metrics := metrics.NewMetrics(registry)
component := metrics.New(metrics, eventBus)
go component.Start(ctx)
eventBus.Start()

func (*Component) Metrics 

func (c *Component) Metrics() *Metrics

Metrics returns the underlying Metrics instance for direct access.

This allows other components (like webhook) to record metrics directly without going through the event bus.

func (*Component) Start 

func (c *Component) Start(ctx context.Context) error

Start begins the metrics event processing loop.

This method blocks until the context is cancelled. It also periodically updates the observability drop metric from EventBus stats.

type Metrics 

type Metrics struct {
	// Reconciliation metrics
	ReconciliationDuration prometheus.Histogram
	ReconciliationTotal    prometheus.Counter
	ReconciliationErrors   prometheus.Counter

	// Deployment metrics
	DeploymentDuration prometheus.Histogram
	DeploymentTotal    prometheus.Counter
	DeploymentErrors   prometheus.Counter

	// HAProxy reload + DataPlane API operation counters, populated from
	// DeploymentCompletedEvent. Reloads are the canonical capacity/SLO signal: a
	// reload momentarily forks the HAProxy process, so a high reload rate (vs
	// runtime-API updates) is what to capacity-plan and alert on. The data is
	// already carried on the event; these surface it as cumulative counters.
	HAProxyReloadsTotal         prometheus.Counter
	DataplaneAPIOperationsTotal prometheus.Counter

	// Runtime-eligible fast-path metrics. Fires counts every fast-path attempt
	// (one per pod per reconcile); Applies counts the subset that actually
	// applied >=1 runtime-eligible server update. Applies stuck at 0 while
	// Fires climbs means the fast path runs but the render diff never carries a
	// runtime-eligible change.
	RuntimeFastPathFires         prometheus.Counter
	RuntimeFastPathApplies       prometheus.Counter
	RuntimeFastPathFailures      prometheus.Counter
	RuntimeFastPathServerUpdates prometheus.Counter

	// Validation metrics
	ValidationTotal  prometheus.Counter
	ValidationErrors prometheus.Counter

	// Resource metrics
	ResourceCount *prometheus.GaugeVec

	// Event metrics
	EventSubscribers           prometheus.Gauge
	EventsPublished            prometheus.Counter
	EventsDropped              prometheus.Counter     // Total drops (backwards compatible)
	EventsDroppedCritical      prometheus.Counter     // Drops from critical subscribers (alert-worthy)
	EventsDroppedBySubscriber  *prometheus.CounterVec // Drops by subscriber and event type
	EventsDroppedObservability prometheus.Gauge       // Drops from observability subscribers (polled, expected)

	// Queue wait metrics - time events spend waiting in channels before processing
	ReconciliationQueueWait prometheus.Histogram

	// Webhook metrics
	WebhookRequestsTotal   *prometheus.CounterVec
	WebhookRequestDuration prometheus.Histogram
	WebhookValidationTotal *prometheus.CounterVec

	// Leader election metrics
	LeaderElectionIsLeader            prometheus.Gauge
	LeaderElectionTransitionsTotal    prometheus.Counter
	LeaderElectionTimeAsLeaderSeconds prometheus.Counter

	// Parser cache metrics. Registered as CounterFuncs that report the
	// parser package's cumulative hit/miss counters directly on each scrape,
	// so there is no readback or delta tracking to keep them in sync.
	ParserCacheHits   prometheus.CounterFunc
	ParserCacheMisses prometheus.CounterFunc

	// Discovery metrics — surfaces rejection of HAProxy pods that the
	// controller refuses to talk to (most commonly version-incompatible).
	// Without this counter, a misconfigured cluster (e.g., controller image
	// bundling HAProxy 3.3 while chart deploys 3.2) presents only as
	// "deployment.skipped" in the pipeline status, masking a real fault.
	HAProxyPodsRejectedTotal *prometheus.CounterVec

	// ConfigRejectedTotal counts HAProxyTemplateConfig loads refused by the
	// config-validation gate, labelled by the validator that rejected it
	// (basic / template / jsonpath / validationtests), or "coordinator" when the
	// scatter-gather itself failed — a validator timed out or didn't respond —
	// rather than a specific validator refusing the config. Persistent growth
	// means the controller is repeatedly refusing a new config and continuing to
	// serve the last-good one — the operator's config never took effect.
	ConfigRejectedTotal *prometheus.CounterVec

	// Build info metric
	BuildInfo *prometheus.GaugeVec
}

Metrics holds all controller-specific Prometheus metrics.

IMPORTANT: Create one instance per application iteration. When the iteration ends (e.g., on config reload), metrics are garbage collected. This prevents stale state from surviving across reinitialization cycles.

func NewMetrics 

func NewMetrics(registry prometheus.Registerer) *Metrics

New creates all controller metrics and registers them with the provided registry.

IMPORTANT: Pass an instance-based registry (prometheus.NewRegistry()), NOT prometheus.DefaultRegisterer. Metrics are scoped to the registry's lifetime. When the registry is garbage collected (iteration ends), metrics are freed.

This is critical for supporting application reinitialization on configuration changes without leaking metrics or accumulating stale state.

Example: 

registry := prometheus.NewRegistry()  // Create per iteration
metrics := metrics.NewMetrics(registry)  // Metrics tied to iteration
// ... use metrics ...
// When iteration ends, both registry and metrics are GC'd

func (*Metrics) AddTimeAsLeader 

func (m *Metrics) AddTimeAsLeader(seconds float64)

AddTimeAsLeader adds time spent as leader to the cumulative counter.

Parameters:

  • seconds: Time spent as leader in seconds

func (*Metrics) RecordConfigRejected 

func (m *Metrics) RecordConfigRejected(validator string)

RecordConfigRejected increments the config-rejection counter for the given validator (the one whose check failed).

func (*Metrics) RecordDeployment 

func (m *Metrics) RecordDeployment(durationSeconds float64, success bool)

RecordDeployment records a deployment attempt.

Parameters:

  • durationSeconds: Time spent deploying (use time.Since(start).Seconds())
  • success: Whether the deployment completed successfully

func (*Metrics) RecordDeploymentOperations 

func (m *Metrics) RecordDeploymentOperations(reloads, apiOperations int)

RecordDeploymentOperations records the HAProxy reload count and DataPlane API operation count from a completed deployment. Both are cumulative; reloads is the headline capacity/SLO signal (see the metric help text). Zero values are skipped so a no-op deployment doesn't perturb the counters.

func (*Metrics) RecordEvent 

func (m *Metrics) RecordEvent()

RecordEvent records an event publication. Call this for every event published to the EventBus.

func (*Metrics) RecordEventDrop 

func (m *Metrics) RecordEventDrop(subscriberName, eventType string)

RecordEventDrop records an event drop due to full subscriber buffer. This increments both aggregate counters and per-subscriber counters. Call this from the drop callback registered with EventBus.SetDropCallback().

Parameters:

  • subscriberName: The name of the subscriber that dropped the event
  • eventType: The event type that was dropped

func (*Metrics) RecordHAProxyPodRejected 

func (m *Metrics) RecordHAProxyPodRejected(reason string)

RecordHAProxyPodRejected increments the rejection counter for a stable reason label (e.g. "version_mismatch_older", "version_check_failed"). Discovery publishes HAProxyPodRejectedEvent; the metrics component translates each event into a counter increment.

func (*Metrics) RecordLeadershipTransition 

func (m *Metrics) RecordLeadershipTransition()

RecordLeadershipTransition records a leadership state change. Call this whenever leadership is gained or lost.

func (*Metrics) RecordQueueWait 

func (m *Metrics) RecordQueueWait(seconds float64)

RecordQueueWait records how long a reconciliation event waited in the coordinator queue before processing started.

func (*Metrics) RecordReconciliation 

func (m *Metrics) RecordReconciliation(durationSeconds float64, success bool)

RecordReconciliation records a completed reconciliation cycle.

Parameters:

  • durationSeconds: Time spent in reconciliation (use time.Since(start).Seconds())
  • success: Whether the reconciliation completed successfully

func (*Metrics) RecordRuntimeFastPath 

func (m *Metrics) RecordRuntimeFastPath(serverUpdates int, failed bool)

RecordRuntimeFastPath records one runtime-eligible fast-path apply attempt: serverUpdates is how many server updates it applied (0 = fired but nothing to do), failed reports whether it errored.

func (*Metrics) RecordValidation 

func (m *Metrics) RecordValidation(success bool)

RecordValidation records a validation attempt.

Parameters:

  • success: Whether the validation passed

func (*Metrics) RecordWebhookRequest 

func (m *Metrics) RecordWebhookRequest(gvk, result string, durationSeconds float64)

RecordWebhookRequest records a webhook admission request.

Parameters:

  • gvk: The GVK of the resource being validated (e.g., "v1.ConfigMap")
  • result: The result of the request ("allowed", "denied", or "error")
  • durationSeconds: Time spent processing the request

func (*Metrics) RecordWebhookValidation 

func (m *Metrics) RecordWebhookValidation(gvk, result string)

RecordWebhookValidation records a webhook validation result.

Parameters:

  • gvk: The GVK of the resource being validated
  • result: The validation result ("allowed", "denied", or "error")

func (*Metrics) SetBuildInfo 

func (m *Metrics) SetBuildInfo(version, haproxyVersion, goVersion string)

SetBuildInfo sets the build info metric with version labels. Call once at startup with the version information for this binary.

Parameters:

  • version: The controller version (e.g., "0.1.0-alpha.10")
  • haproxyVersion: The HAProxy major.minor version (e.g., "3.2")
  • goVersion: The Go runtime version (e.g., "go1.26.1")

func (*Metrics) SetEventSubscribers 

func (m *Metrics) SetEventSubscribers(count int)

SetEventSubscribers sets the number of active event subscribers.

Parameters:

  • count: The current number of event subscribers

func (*Metrics) SetIsLeader 

func (m *Metrics) SetIsLeader(isLeader bool)

SetIsLeader sets whether this replica is the leader.

Parameters:

  • isLeader: true if this replica is the leader, false otherwise

func (*Metrics) SetObservabilityDrops 

func (m *Metrics) SetObservabilityDrops(count uint64)

SetObservabilityDrops sets the observability drop gauge from EventBus statistics. This should be called periodically since observability drops don't trigger callbacks.

func (*Metrics) SetResourceCount 

func (m *Metrics) SetResourceCount(resourceType string, count int)

SetResourceCount sets the count for a specific resource type.

Parameters:

  • resourceType: The type of resource (e.g., "ingresses", "services")
  • count: The current number of resources of this type

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.