metrics

package
v0.0.0-...-fe86bdc Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 5, 2026 License: Apache-2.0 Imports: 14 Imported by: 0

Documentation

Overview

Package metrics holds Nebula's Prometheus instrumentation.

Everything here registers into controller-runtime's registry, so it is served on the manager's existing --metrics-bind-address endpoint alongside the standard controller/workqueue metrics. Importing this package is enough wiring: every metric self-registers (see the init in each file) and every entry point is a helper the callers already on those paths invoke.

The cost counter is the one exception: its label names depend on --cost-labels, so main must call InitCost once after flags are parsed. See InitCost for why it cannot self-register.

Nothing here reads or writes the API. Cost also accrues onto a NodeClaim status field, so the loop that advances it stays in internal/controller and calls in here with each closed window.

The instrumented surface is the path a Pod takes from admission to a running external instance, one file per leg, plus what that instance costs while it runs: 

placement.go   the Pod is gated -> a candidate is chosen -> the gate is removed
provision.go   the provider is called -> the instance reports Running
cost.go        dollars accrued, one billing window at a time
attribution.go whose dollars those are, from the operator's chosen Pod labels

Those are the legs whose cost and failure modes are otherwise invisible: placement can silently leave a Pod gated forever, provisioning runs against a third party, takes minutes, bills money, and fails for reasons Pod status flattens away, and the instance it produces keeps charging whether or not anything is using it. Everything else is covered elsewhere and not duplicated — reconcile counts, queue depth and API latency by controller-runtime's collectors, "how many Pods are gated right now?" by kube-state-metrics.

The two legs deliberately share one label set (labels.go) so a placement and the provisioning attempt it led to carry identical label values and can be joined in PromQL without label surgery. See docs/metrics.md for the operator-facing view.

Index

Constants

View Source 
const (
	DeferNoPool         = "no_pool"
	DeferInvalidRequest = "invalid_request"
	DeferAllBlocked     = "all_blocked"
	DeferNoCandidate    = "no_candidate"
	DeferStaleClaim     = "stale_claim"
)

Reason values for the placement deferral label: why one reconcile ended without placing the Pod. Closed set, and each value points at a DIFFERENT owner — which is the whole reason for splitting them:

  • no_pool / invalid_request: the request is wrong. Nobody is retrying their way out of these; a human must edit the Pod (or the workload that generates it).
  • all_blocked: the request is fine and a candidate exists, but failover is holding it off. Self-clearing, and the Pod is already requeued for the moment it frees.
  • no_candidate: the pool cannot serve this request at all today. Self-clearing only if an operator adds a provider or a provider registers.
  • stale_claim: a NodeClaim from a prior same-named Pod has not been reaped yet. Self-clearing in seconds; a sustained rate means the NodeClaim backstop is stuck.
View Source 
const (
	SkipProviderUnregistered   = "provider_unregistered"
	SkipCapacityUnsupported    = "capacity_type_unsupported"
	SkipAcceleratorUnsupported = "accelerator_unsupported"
	SkipEgressUnsupported      = "egress_policy_unsupported"
	SkipBlocked                = "blocked"
)

Reason values for the candidate skip label: why the placement walk passed over one (tier, provider, region) candidate. Only "blocked" clears on its own.

View Source 
const (
	ResultSuccess = "success"
	ResultFailure = "failure"
)

Result values for the result label. A provisioning attempt either returned an instance id or an error; there is no third outcome.

View Source 
const (
	ReasonCapacity    = "capacity"
	ReasonQuota       = "quota"
	ReasonAuth        = "auth"
	ReasonUnsupported = "unsupported_accelerator"
	ReasonTimeout     = "timeout"
	// ReasonOther: the failure carried no sentinel, so its category is unavailable — either
	// the adapter returned a raw API error without wrapping it, or the provider never told
	// us what it decided at all (a transport failure, a 503, an unparseable response). A
	// sustained rate here is a to-do rather than an incident: wrap the condition in the
	// adapter (see docs/add-a-provider.md) and the failure moves onto its real category.
	// Which of the two it was is in the vnode-handler error log.
	ReasonOther = "other"
)

Reason values for the failure label. This is a deliberately COARSE, closed set: it is a metric label, so it must stay bounded no matter what text a provider API returns. The fine-grained detail stays where it is already available (the Pod's Failed status message and the vnode-handler error log); this exists to answer "are we losing capacity, or are our credentials broken?" at a glance.

Variables

View Source 
var (
	// PlacementDecisions counts Pods actually placed, by the candidate they landed on.
	// It carries the same labels as the provisioning metrics on purpose: the two are
	// joinable without label surgery, so "placed on Spot but never provisioned" is one
	// query rather than a correlation exercise. The tier breakdown is the money
	// question — a fleet quietly sliding from Spot to OnDemand is a cost regression
	// with no error anywhere.
	PlacementDecisions = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "nebula_placement_decisions_total",
		Help: "Pods placed, by provider, region, capacity type, accelerator type and accelerator count.",
	}, candidateLabels)

	// PlacementWaitDuration measures from Pod creation to the gate being removed: the
	// user-visible queue time BEFORE provisioning starts, which
	// nebula_instance_ready_duration_seconds then continues from. Together they cover
	// the whole path from `kubectl apply` to a Running instance.
	//
	// Unlike the ready duration, this one has no restart gap: the start timestamp is
	// the Pod's own creationTimestamp, so a placement that happens after a manager
	// restart still reports the true total wait. Buckets span three orders of
	// magnitude because the honest range does: an unblocked Pod is placed in
	// milliseconds, while one waiting out a failover block waits the blocklist TTL.
	PlacementWaitDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
		Name: "nebula_placement_wait_duration_seconds",
		Help: "Time from Pod creation to placement (scheduling gate removal), by provider, region, " +
			"capacity type, accelerator type and accelerator count.",
		Buckets: []float64{0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300, 600, 1800},
	}, candidateLabels)

	// PlacementDeferrals counts reconciles that ended without placing the Pod, by
	// reason.
	//
	// It counts DEFERRALS, not Pods: a gated Pod is reconciled again on every requeue
	// and resync, so one Pod stuck for an hour contributes many increments. That makes
	// the rate a measure of placement pressure, not a population count — for "how many
	// Pods are stuck right now", read the SchedulingGated Pod count from
	// kube-state-metrics and use this series to explain WHY.
	PlacementDeferrals = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "nebula_placement_deferrals_total",
		Help: "Reconciles that ended without placing a Pod, by pool and reason " +
			"(no_pool, invalid_request, all_blocked, no_candidate, stale_claim).",
	}, []string{"pool", "reason"})

	// CandidateSkips counts individual (tier, provider, region) candidates passed over
	// during the placement walk. This is the only view into failover actually working:
	// when every Pod lands on OnDemand, a rate on {capacity_type="Spot",
	// reason="blocked"} is the explanation, and one on
	// reason="capacity_type_unsupported" says the pool is misconfigured instead.
	//
	// Cardinality is bounded by pool configuration (providers x tiers x regions x four
	// reasons), not by Pod count.
	CandidateSkips = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "nebula_placement_candidate_skips_total",
		Help: "Placement candidates skipped, by provider, capacity type, region and reason.",
	}, []string{"provider", "capacity_type", "region", "reason"})
)
View Source 
var (
	// ProvisionAttempts counts provisioning attempts by outcome. Rate of the
	// result="failure" series over the total is the provisioning error rate; the
	// per-region/accelerator breakdown is what tells you WHERE it is failing.
	ProvisionAttempts = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "nebula_provision_attempts_total",
		Help: "Total external instance provisioning attempts, by provider, region, capacity type, " +
			"accelerator type, accelerator count and outcome.",
	}, withExtra("result"))

	// ProvisionFailures breaks failures down by coarse cause. It deliberately
	// overlaps ProvisionAttempts{result="failure"} rather than adding a reason label
	// there: reason is only meaningful on failure, and carrying it on the attempts
	// counter would multiply the success series by a label that is constant for
	// them.
	ProvisionFailures = prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "nebula_provision_failures_total",
		Help: "Failed provisioning attempts by coarse cause " +
			"(capacity, quota, auth, unsupported_accelerator, timeout, other).",
	}, withExtra("reason"))

	// ProvisionDuration measures the provider's Provision call alone — not the
	// wait for the instance to become usable. What lands here is provider-specific
	// and worth knowing per provider: AWS sweeps a region's availability zones
	// inside the call (so a capacity shortage shows up as latency HERE), Modal
	// builds the image inside it (so a cache miss does).
	//
	// The buckets run past the largest Capabilities.ProvisionTimeout (Modal's 5
	// minutes), rather than stopping at it: a ceiling AT the deadline would bury every
	// slow build in +Inf, while this way the 300s bucket is where a call killed by its
	// own deadline lands and anything above it is overshoot.
	ProvisionDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
		Name: "nebula_provision_duration_seconds",
		Help: "Latency of the provider's Provision call, by provider, region, capacity type, " +
			"accelerator type, accelerator count and outcome.",
		Buckets: []float64{0.5, 1, 2.5, 5, 10, 20, 30, 45, 60, 90, 120, 180, 300, 420, 600},
	}, withExtra("result"))

	// InstanceReadyDuration measures the whole user-visible wait: from the moment
	// CreatePod starts provisioning to the first poll tick that reports the instance
	// Running. It therefore includes the Provision call, any provider-side queueing
	// for capacity, image pull, GPU attach, container boot, and up to one poll
	// interval of detection lag — which is the honest number, because that is what a
	// user waits.
	//
	// Only observed ONCE per Pod, on the first transition to Running, and never for
	// an instance re-adopted after a restart (the original start time is gone, and a
	// duration measured from re-adoption would understate it wildly). Buckets run to
	// 30min because a queueing provider on a large GPU shape genuinely takes that
	// long.
	InstanceReadyDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
		Name: "nebula_instance_ready_duration_seconds",
		Help: "Time from the start of provisioning to the instance first reporting Running, " +
			"by provider, region, capacity type, accelerator type and accelerator count.",
		Buckets: []float64{5, 10, 20, 30, 45, 60, 90, 120, 180, 300, 600, 900, 1800},
	}, candidateLabels)
)
View Source 
var CostTotal = newCostTotal(nil)

CostTotal accumulates spend one CLOSED WINDOW at a time, which is what makes it billable: increase(...[w]) over any window is a pure function of that window, so a consumer replaying an old window re-derives the same dollars and can upsert them idempotently. That holds only for a series scraped before its first charge, which is what TouchSeries is for.

Deliberately carries no claim identity. A per-claim series would churn — one per instance ever created, retained until the process exits — and worse, a claim that lived and died between two window boundaries could not be billed from it at all: differencing a cumulative per-claim series needs a sample at each boundary, and a short-lived instance has neither. Aggregating first does not help, since a sum over a changing claim set is not monotonic. Per-claim spend lives in NodeClaim.status.estimatedCostUSD (the EST_COST column) instead.

Functions

func ConfigureCostForTest 

func ConfigureCostForTest(podKeys []string)

ConfigureCostForTest points the cost counter at a set of Pod label keys, for tests in other packages that exercise attribution. Not for production use: InitCost is the real entry point.

It exists because InitCost can only ever succeed ONCE per process — a registry remembers a metric name's label dimensions for the life of the process — so a test binary cannot try two label sets through it. This skips registration; testutil collects from the collector directly.

Keys go through the real ParseCostLabels, so a test cannot configure a shape the flag could not, and an unparseable key panics rather than silently configuring nothing.

Pass nil to restore the unconfigured default, and do so in a Cleanup: the counter is a package variable, so a test that leaves it swapped changes what every later test measures.

func CostLabelKeys 

func CostLabelKeys() []string

CostLabelKeys is the Pod label keys attribution reads, in emit order. Exported so the controller that stamps the values onto a claim reads the SAME list the metric emits, rather than keeping a second copy that could disagree with it.

POD KEYS, not the metric's label names — the two differ whenever a key is qualified. Stamping a claim needs the former; building the counter needs metricNames.

func FailureReason 

func FailureReason(err error) string

FailureReason maps a Provision error onto the closed reason set.

It matches on the shared sentinels in pkg/provider rather than on message text, which is what keeps the label bounded: an adapter that wraps its API errors with ErrNoCapacity/ErrQuota/ErrAuth gets an accurate reason, and one that does not falls through to "other" instead of inventing a new series per distinct provider message. This is intentionally NOT provider.ClassifyError: that answers a different question (what to blocklist, and how widely), and its "unrecognized errors are scoped like capacity" default would be an outright lie as a metric — it would report unknown failures as capacity shortfalls.

func InitCost 

func InitCost(podKeys []string) error

InitCost fixes the attribution dimension the cost counter carries, from --cost-labels, and registers it. MUST be called once from main after flags are parsed and before the manager starts — cost is the one metric here that is not scraped until it is.

Two facts force that. A CounterVec's label names are set at CONSTRUCTION, while the flag is not known until main runs; and a Prometheus registry remembers a metric NAME's label dimensions for the life of the process even across Unregister, so registering a placeholder first would permanently forbid the real shape. Nothing is at risk in the gap: no window can be recorded before the manager starts.

Not safe against concurrent recording.

func ObservePlacement 

func ObservePlacement(l Labels, waited time.Duration)

ObservePlacement records one Pod placed onto the candidate l describes, having waited waited since it was created.

func ObserveProvision 

func ObserveProvision(l Labels, d time.Duration, err error)

ObserveProvision records one completed provisioning attempt: its outcome, its latency, and — when it failed — the coarse cause. It is the single call the virtual kubelet makes on both the success and failure paths, so the attempt and failure counters cannot drift out of step.

err nil means success. d is the duration of the Provision call itself.

func ObserveReady 

func ObserveReady(l Labels, d time.Duration)

ObserveReady records the end-to-end wait for an instance to reach Running.

func ParseCostLabels 

func ParseCostLabels(spec string) ([]string, error)

ParseCostLabels turns the --cost-labels value ("example.com/org-id,team_id") into the attribution dimension, in the order given. Empty input means no attribution, which is the default.

Each entry is a POD LABEL KEY, qualified or not; the metric label it emits under is derived from it, so a key is configured as the Pod carries it and queried as PromQL can express it. It fails rather than skipping a bad entry: silently dropping one would be discovered at invoicing time, when every series has already been recorded as "none".

Four rejections, all at startup: a key Kubernetes would not accept, one whose derived name Prometheus would not, the same key twice, and two keys folding to the SAME name. That last one is a corner case now that the prefix is kept ("org-id" and "org.id" still meet), but it would merge two tenants' spend into one series, so it fails rather than being tolerated. A derived name that shadows a dimension the counter already carries ("provider", "phase") gets through here and fails at InitCost, where the registry rejects a duplicate label name; still at startup, just with a less pointed message.

Setting this at all is what makes cost the only metric here whose CARDINALITY is not bounded by configuration: the values come from Pod labels. Nothing caps them — noteSeries only warns — so pick keys whose value set the cluster's admission policy actually constrains.

func RecordCandidateSkip 

func RecordCandidateSkip(prov string, tier nebulav1alpha1.CapacityType, region, reason string)

RecordCandidateSkip records one candidate the placement walk passed over. region is empty for the skips decided before the walk reaches the region axis (an unregistered provider, an unservable tier, a missing accelerator), which is honest: those rule out every region at once.

func RecordDeferral 

func RecordDeferral(pool, reason string)

RecordDeferral records one reconcile that placed nothing.

pool MUST be the name of a NodePool that actually exists, or "" for the deferral where it does not (DeferNoPool). The pool a Pod asks for is a Pod LABEL — user controlled and unbounded — so filing the unresolved string here would let a mislabeled workload mint a new time series per typo. Once the pool has been resolved to a real object, its name is bounded by cluster resources and safe.

func RecordWindow 

func RecordWindow(l Labels, phase string, attribution map[string]string, usd float64)

RecordWindow books the dollars one claim ran up over a single accrual window, attributed with the labels stamped on the claim (NodeClaimStatus.CostLabels).

Call it only AFTER the window has been persisted: a counter has no idempotency key, so a window added twice is charged twice. Nothing is lost by waiting, because a failed write leaves the anchor in place and the next tick re-derives the same window.

func TouchSeries 

func TouchSeries(l Labels, attribution map[string]string, phases ...string)

TouchSeries publishes one claim's label set as a zero-valued series under each of phases, so the first charge booked there has an earlier sample to be differenced against.

increase() recovers a RISE between two samples, so a series whose very first sample already holds money reads as no rise at all: those dollars are in the counter's absolute value but not in any increase()/rate() query, which is what a billing consumer runs. Sharing series across claims usually hides that — but attribution makes them tenant-scoped, and a tenant whose whole usage is one short job would be billed nothing.

This does not contradict RecordWindow's refusal of zeros. A zero WINDOW is a measurement claiming something cost nothing; a zero COUNTER only says nothing has been charged here yet, and publishing it is the ordinary way to make rate() work over label values not known until runtime.

Series are PROCESS-local, so the baseline has to be republished by whatever process is doing the charging, on every pass rather than once at startup — see controller.seedClaimBaseline, the one caller. A scrape still has to land between the baseline and the first charge, which is what keeps this a mitigation rather than a fix; see docs/metrics.md.

Types

type Labels 

type Labels struct {
	Provider     string
	Region       string
	CapacityType string
	// Accelerator is the accelerator TYPE alone (e.g. "H100"), and AcceleratorCount how
	// many were requested. They are kept apart so both aggregations work — see
	// candidateLabels. Empty/zero for a CPU-only Pod.
	Accelerator      string
	AcceleratorCount int32
}

Labels identifies the candidate one placement or provisioning attempt was made against. The zero value is valid: every field normalizes to "none".

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.