hubmetrics

package
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 13, 2026 License: MIT Imports: 8 Imported by: 0

Documentation

Overview

Package hubmetrics is the process-level metrics registry for a long-lived cloop hub.

It is deliberately not pkg/metrics. That package models one *run*: a Metrics value is created by the orchestrator, keyed to a (provider, model) pair, and serialised to .cloop/metrics.json when the run ends. A hub is the opposite shape — one process, many tenants, many runs, no end — so the two cannot share a type. What a hub needs is a registry that outlives every individual thing it measures.

The package is stdlib-only, and must stay that way. Every enterprise subsystem imports it (pkg/executor, pkg/secretbroker, pkg/sessionstore, pkg/apitoken, pkg/writeback, pkg/mergequeue, pkg/gitproxy, pkg/egressbroker), several of which sit near the bottom of the import graph. A single dependency on a cloop package here would make an import cycle out of the next subsystem that wants a counter.

Cardinality is a safety property, not a tuning knob

A metrics registry in a multi-tenant process is an unbounded-memory hazard wearing a friendly name. Every distinct label combination is a series that lives until the process exits, so labelling by task ID, project path, or identity subject converts tenant activity directly into hub memory — and on a hosted hub that is a denial-of-service any signed-in user can trigger by creating projects in a loop. The rule is therefore absolute: label values come from closed enumerations that exist in the source (executor kinds, isolation levels, placement constraints, denial reasons), never from user or tenant input.

Because "never" is a property of code review and code review is fallible, the registry enforces a ceiling underneath it. Each metric admits at most MaxSeriesPerMetric distinct label combinations; past that, samples fold into a single overflow series and cloop_metrics_series_dropped_total names the metric that overflowed. That turns the failure mode of a mistake from "the hub runs out of memory in a week" into "one metric goes flat and a counter says which one" — degraded observability instead of an outage.

Index

Constants

View Source
const (
	// Task failure reasons.
	FailStart     = "start"
	FailRun       = "run"
	FailCancelled = "cancelled"

	// Lease lifecycle events.
	LeaseIssued  = "issued"
	LeaseRenewed = "renewed"
	LeaseRevoked = "revoked"
	LeaseExpired = "expired"

	// Session termination reasons.
	SessionIdleEvicted     = "idle_evicted"
	SessionAbsoluteExpired = "absolute_expired"
	SessionAdminRevoked    = "admin_revoked"
	SessionSelfLogout      = "self_logout"

	// Generic binary results.
	ResultSuccess = "success"
	ResultFailure = "failure"
	ResultAllowed = "allowed"
	ResultDenied  = "denied"

	// Placement results.
	PlacementPlaced = "placed"
	PlacementFailed = "failed"

	// Write-back results.
	WriteBackAccepted    = "accepted"
	WriteBackRejected    = "rejected"
	WriteBackUnavailable = "unavailable"

	// Merge outcomes.
	MergeClean        = "clean"
	MergeAutoResolved = "auto_resolved"
	MergeUnresolved   = "unresolved"
	MergeError        = "error"

	// KEK rotation phases.
	RotationTotal     = "total"
	RotationRewrapped = "rewrapped"
	RotationSkipped   = "skipped"
	RotationFailed    = "failed"

	// Egress directions.
	EgressUp   = "up"
	EgressDown = "down"
)

Bounded reason vocabularies used by call sites whose own packages have no enumeration to borrow. Declaring them here rather than as string literals at the call site is what keeps the label a closed set: a typo becomes a compile error instead of a new series.

View Source
const DefaultMaxSeriesPerMetric = 128

DefaultMaxSeriesPerMetric bounds distinct label combinations per metric.

Sized against the real enumerations rather than picked round: the widest label schema in the catalog is placement failures by constraint, where pkg/executor declares 24 constraints. 128 leaves an order of magnitude of headroom for enumerations that grow, while still capping the registry's footprint at a few hundred kilobytes for the whole catalog.

Variables

View Source
var (
	TaskStarts = Default.MustRegister(Definition{
		Name:   "cloop_executor_task_starts_total",
		Help:   "Workloads dispatched to an executor, by driver kind and isolation level.",
		Type:   TypeCounter,
		Labels: []string{"executor_kind", "isolation"},
	})

	TaskCompletions = Default.MustRegister(Definition{
		Name:   "cloop_executor_task_completions_total",
		Help:   "Workloads that ran to completion and exited zero.",
		Type:   TypeCounter,
		Labels: []string{"executor_kind", "isolation"},
	})

	TaskFailures = Default.MustRegister(Definition{
		Name:   "cloop_executor_task_failures_total",
		Help:   "Workloads that did not complete successfully. reason is one of start (the executor refused to launch it), run (it launched and failed), cancelled (the control plane withdrew it).",
		Type:   TypeCounter,
		Labels: []string{"executor_kind", "isolation", "reason"},
	})

	// Buckets span a second to two hours because that is the real range: a
	// `cloop suggest` subcommand returns in seconds, an agent task on a
	// remote sandbox runs for an hour. Linear buckets would put every
	// interesting value in one bin at one end or the other.
	TaskDuration = Default.MustRegister(Definition{
		Name:    "cloop_executor_task_duration_seconds",
		Help:    "Wall-clock duration of workloads that reached a terminal state.",
		Type:    TypeHistogram,
		Labels:  []string{"executor_kind", "isolation"},
		Buckets: []float64{1, 5, 15, 60, 300, 900, 1800, 3600, 7200},
	})
)

Executor and task lifecycle.

executor_kind is pkg/executor.Kind* (4 values); isolation is pkg/executor.Isolation (4 values). Neither carries an executor ID: a hub fronting a fleet of edge devices would turn that into one series per device per metric, which is exactly the unbounded-by-tenant-behaviour case the ceiling exists to catch.

View Source
var (
	ExecutorsByState = Default.MustRegister(Definition{
		Name:   "cloop_executors",
		Help:   "Registered executors by driver kind and scheduling state (ready, degraded, unreachable, cordoned, draining).",
		Type:   TypeGauge,
		Labels: []string{"kind", "state"},
	})

	// The oldest heartbeat per kind rather than one gauge per executor.
	// The alert that matters is "some executor of this kind has gone
	// quiet", and a max answers it in one series instead of one per device
	// — which for an edge fleet is the difference between a bounded metric
	// and a per-device one.
	ExecutorHeartbeatAgeMax = Default.MustRegister(Definition{
		Name:   "cloop_executor_heartbeat_age_seconds_max",
		Help:   "Age of the least-recently-successful liveness probe across executors of this kind. Rises without bound while a node is unreachable.",
		Type:   TypeGauge,
		Labels: []string{"kind"},
	})

	Placements = Default.MustRegister(Definition{
		Name:   "cloop_executor_placements_total",
		Help:   "Placement decisions, by result (placed or failed).",
		Type:   TypeCounter,
		Labels: []string{"result"},
	})

	// constraint is pkg/executor.Constraint (24 values), the headline
	// reason Select reports. The per-candidate Detail string is not a label:
	// it interpolates executor IDs and byte counts.
	PlacementFailures = Default.MustRegister(Definition{
		Name:   "cloop_executor_placement_failures_total",
		Help:   "Placement attempts that found no eligible executor, by the constraint that rejected the most candidates.",
		Type:   TypeCounter,
		Labels: []string{"constraint"},
	})
)

Executor fleet health. Collector-backed: the truth is the registry and the supervisor's health store, and a shadow copy maintained at mutation time would be a second source of truth with drift.

View Source
var (
	LeaseEvents = Default.MustRegister(Definition{
		Name:   "cloop_secret_lease_events_total",
		Help:   "Lease lifecycle transitions by credential kind and event (issued, renewed, revoked, expired).",
		Type:   TypeCounter,
		Labels: []string{"kind", "event"},
	})

	LeasesLive = Default.MustRegister(Definition{
		Name:   "cloop_secret_leases_live",
		Help:   "Leases the broker currently considers valid, by credential kind.",
		Type:   TypeGauge,
		Labels: []string{"kind"},
	})
)

Secret broker leases. kind is pkg/secretbroker.Kind* (7 values).

View Source
var (
	UnsealFailures = Default.MustRegister(Definition{
		Name:   "cloop_secret_unseal_failures_total",
		Help:   "Envelope opens that failed, by reason (key_unknown, key_retired, key_unavailable, seal_failed). Any non-zero rate means secrets are unreadable.",
		Type:   TypeCounter,
		Labels: []string{"reason"},
	})

	KEKRotationRecords = Default.MustRegister(Definition{
		Name:   "cloop_secret_kek_rotation_records",
		Help:   "Progress of the current or most recent key-encryption-key rotation, by phase (total, rewrapped, skipped, failed).",
		Type:   TypeGauge,
		Labels: []string{"phase"},
	})

	KEKRotationActive = Default.MustRegister(Definition{
		Name: "cloop_secret_kek_rotation_active",
		Help: "1 while a key-encryption-key rotation is in progress, 0 otherwise. A rotation that stays at 1 across scrapes has stalled with records still wrapped under the old key.",
		Type: TypeGauge,
	})
)

Sealing key health. An unseal failure is the signal that the hub can no longer read its own secrets — a wrong passphrase, a retired key still referenced, or ciphertext that failed authentication. All three are operator-visible emergencies and none of them is self-healing.

View Source
var (
	SessionsCreated = Default.MustRegister(Definition{
		Name: "cloop_sessions_created_total",
		Help: "Browser sessions established after a successful OIDC exchange.",
		Type: TypeCounter,
	})

	SessionsTerminated = Default.MustRegister(Definition{
		Name:   "cloop_sessions_terminated_total",
		Help:   "Sessions ended, by reason (idle_evicted, absolute_expired, admin_revoked, self_logout).",
		Type:   TypeCounter,
		Labels: []string{"reason"},
	})

	SessionsLive = Default.MustRegister(Definition{
		Name: "cloop_sessions_live",
		Help: "Sessions currently valid: neither past their absolute expiry nor idle beyond the timeout.",
		Type: TypeGauge,
	})
)

Sessions. Deliberately unlabelled by subject: the count of sessions is operational, the roster of who holds them is in the audit trail behind audit.read, and putting subjects in a scrape would copy that roster into every monitoring system that touches it.

View Source
var (
	TokenAuth = Default.MustRegister(Definition{
		Name:   "cloop_apitoken_auth_total",
		Help:   "API token verification attempts, by result (success or failure).",
		Type:   TypeCounter,
		Labels: []string{"result"},
	})

	TokenAuthFailures = Default.MustRegister(Definition{
		Name:   "cloop_apitoken_auth_failures_total",
		Help:   "API token verifications that failed, by reason (malformed, not_found, bad_secret, revoked, expired, no_roles). A bad_secret spike against many token IDs is credential stuffing.",
		Type:   TypeCounter,
		Labels: []string{"reason"},
	})
)

API token authentication. The failure reasons are pkg/apitoken's sentinel errors, which is what makes them a closed set.

View Source
var (
	WriteBackBundles = Default.MustRegister(Definition{
		Name:   "cloop_writeback_bundles_total",
		Help:   "Write-back bundles offered by executors, by result (accepted, rejected, unavailable).",
		Type:   TypeCounter,
		Labels: []string{"result"},
	})

	WriteBackRejections = Default.MustRegister(Definition{
		Name:   "cloop_writeback_rejections_total",
		Help:   "Bundles refused on their content, by reason code. Reasons are executor.WriteBackReason*; the prose refusal is in the audit trail, not here, because it interpolates branch names and commit SHAs.",
		Type:   TypeCounter,
		Labels: []string{"reason"},
	})
)

Result write-back. The distinction the labels carry is the one the code makes: a rejection is the hub refusing what a sandbox returned, an outage is the hub being unable to find out. They page differently — the first is a misbehaving or compromised sandbox, the second is broken infrastructure.

View Source
var (
	MergeQueueDepth = Default.MustRegister(Definition{
		Name: "cloop_mergequeue_depth",
		Help: "Merges waiting to be applied. The queue is serial, so sustained depth is the integration path falling behind task completion.",
		Type: TypeGauge,
	})

	MergeOutcomes = Default.MustRegister(Definition{
		Name:   "cloop_mergequeue_merges_total",
		Help:   "Completed merge attempts, by outcome (clean, auto_resolved, unresolved, error).",
		Type:   TypeCounter,
		Labels: []string{"outcome"},
	})
)

Merge queue.

View Source
var (
	GitProxyPushes = Default.MustRegister(Definition{
		Name:   "cloop_gitproxy_pushes_total",
		Help:   "Pushes evaluated by the interception proxy, by result (allowed or denied).",
		Type:   TypeCounter,
		Labels: []string{"result"},
	})

	GitProxyDenials = Default.MustRegister(Definition{
		Name:   "cloop_gitproxy_push_denials_total",
		Help:   "Pushes refused, by reason (ref_not_allowed, delete_denied, create_denied, update_denied, no_write, too_many_commands, push_cert). A sandbox repeatedly hitting ref_not_allowed is probing the allowlist.",
		Type:   TypeCounter,
		Labels: []string{"reason"},
	})
)

Git interception proxy. This is the boundary a sandbox pushes through, so its denial counter is a security signal and not only an operational one.

View Source
var (
	EgressRequests = Default.MustRegister(Definition{
		Name:   "cloop_egress_requests_total",
		Help:   "Requests through the egress broker, by result (allowed or denied).",
		Type:   TypeCounter,
		Labels: []string{"result"},
	})

	EgressDenials = Default.MustRegister(Definition{
		Name:   "cloop_egress_denials_total",
		Help:   "Egress requests refused, by reason (no_grant, revoked, expired, host_not_allowed, port_not_allowed, method_not_allowed, destination_blocked, quota_exhausted).",
		Type:   TypeCounter,
		Labels: []string{"reason"},
	})

	EgressBytes = Default.MustRegister(Definition{
		Name:   "cloop_egress_bytes_total",
		Help:   "Bytes proxied on behalf of sandboxes, by direction (up or down).",
		Type:   TypeCounter,
		Labels: []string{"direction"},
	})

	EgressSessionsLive = Default.MustRegister(Definition{
		Name: "cloop_egress_sessions_live",
		Help: "Egress leases currently redeemable.",
		Type: TypeGauge,
	})
)

Egress broker: the hub's Internet connection, leased to sandboxes.

View Source
var (
	QuotaEnforcementEnabled = Default.MustRegister(Definition{
		Name: "cloop_quota_enforcement_enabled",
		Help: "Whether a per-identity quota policy is in force.",
		Type: TypeGauge,
	})

	QuotaLimit = Default.MustRegister(Definition{
		Name:      "cloop_quota_limit",
		Help:      "Configured ceiling per identity and resource.",
		Type:      TypeGauge,
		Labels:    []string{"identity", "resource"},
		MaxSeries: quotaIdentityCeiling,
	})

	QuotaUsage = Default.MustRegister(Definition{
		Name:      "cloop_quota_usage",
		Help:      "Live consumption per identity and resource.",
		Type:      TypeGauge,
		Labels:    []string{"identity", "resource"},
		MaxSeries: quotaIdentityCeiling,
	})

	QuotaDenials = Default.MustRegister(Definition{
		Name:   "cloop_quota_denials_total",
		Help:   "Admission refusals since this hub started, by resource.",
		Type:   TypeCounter,
		Labels: []string{"resource"},
	})

	QuotaIdentities = Default.MustRegister(Definition{
		Name: "cloop_quota_identities",
		Help: "Identities the hub is currently accounting.",
		Type: TypeGauge,
	})

	ProjectsRegistered = Default.MustRegister(Definition{
		Name: "cloop_projects_registered",
		Help: "Projects in the hub registry.",
		Type: TypeGauge,
	})
)
View Source
var Default = New()

Default is the process-wide registry. A hub has exactly one.

A package-level registry is the shape every metrics library converges on for the same reason: the alternative is threading a *Registry through constructors in a dozen packages that otherwise have no reason to know metrics exist, and the first constructor that cannot take a new parameter gets a global anyway. The cost — that instrumentation is not injectable — is bounded here because the registry is the only global and tests can build their own with New.

Functions

func RegisterCollectors

func RegisterCollectors(fns map[string]Collector)

RegisterCollectors installs scrape-time collectors on the default registry, at most once per process.

Types

type Collector

type Collector func()

Collector recomputes derived gauges immediately before a scrape.

Some of what an operator needs is not a running tally but a reading of state that lives elsewhere: how many executors are in each health state, how deep the merge queue is, how many leases are live. Counting those at mutation time would mean every subsystem maintaining a shadow copy of its own state and keeping it consistent — a second source of truth, with the drift that implies. A collector reads the real one at scrape time instead.

type Definition

type Definition struct {
	Name   string
	Help   string
	Type   MetricType
	Labels []string
	// Buckets are the histogram upper bounds in ascending order, excluding
	// +Inf which is implied. Ignored for counters and gauges.
	Buckets []float64
	// MaxSeries overrides Registry.MaxSeriesPerMetric for this metric. Zero
	// takes the registry default.
	//
	// Raising it is only defensible for a collector-backed gauge, because
	// those Reset before every scrape: their cardinality is the number of
	// objects alive at scrape time rather than the number ever observed, so
	// churn cannot accumulate series the way it does on a counter. A raised
	// ceiling on a counter is just a slower memory leak.
	MaxSeries int
}

Definition declares a metric family.

type Metric

type Metric struct {
	// contains filtered or unexported fields
}

Metric is one metric family: a name, a type, a fixed label schema, and the series observed for it so far.

func (*Metric) Add

func (m *Metric) Add(delta float64, labelValues ...string)

Add adds delta to the counter or gauge identified by labelValues.

A nil Metric is a no-op so a subsystem can hold an optional metric without guarding every call site, and a negative delta on a counter is dropped rather than applied: a counter that can go down breaks rate() in ways that surface as impossible graphs long after the bug that caused them.

func (*Metric) Inc

func (m *Metric) Inc(labelValues ...string)

Inc adds one to the counter or gauge identified by labelValues.

func (*Metric) Observe

func (m *Metric) Observe(value float64, labelValues ...string)

Observe records one sample in the histogram identified by labelValues.

func (*Metric) Reset

func (m *Metric) Reset()

Reset drops every series in the metric.

Collector-backed gauges call this before repopulating, so a series whose underlying object is gone (a deregistered executor, a closed session) stops being exported rather than freezing at its last value. A frozen gauge is worse than a missing one: it reads as a live measurement and will hold an alert open forever.

func (*Metric) SeriesCount

func (m *Metric) SeriesCount() int

SeriesCount reports how many distinct label combinations a metric holds, including the overflow series. Used by tests that assert the ceiling holds.

func (*Metric) Set

func (m *Metric) Set(value float64, labelValues ...string)

Set assigns an absolute value to the gauge identified by labelValues.

type MetricType

type MetricType string

MetricType is the Prometheus exposition type of a metric family.

const (
	TypeCounter   MetricType = "counter"
	TypeGauge     MetricType = "gauge"
	TypeHistogram MetricType = "histogram"
)

type Registry

type Registry struct {
	// MaxSeriesPerMetric is the per-metric cardinality ceiling. Read
	// without synchronisation, so set it before the registry is shared.
	MaxSeriesPerMetric int
	// contains filtered or unexported fields
}

Registry holds every metric family in the process and renders them in the Prometheus text exposition format. The zero value is not usable; call New.

func New

func New() *Registry

New returns an empty registry.

func (*Registry) Gather

func (r *Registry) Gather() string

Gather renders the registry to a string.

func (*Registry) MustRegister

func (r *Registry) MustRegister(def Definition) *Metric

MustRegister declares a metric family and returns it, panicking on a malformed definition or a duplicate name.

Panicking is right here and would be wrong almost anywhere else in cloop: the catalog is built at package initialisation from string literals, so a panic can only fire on a programming error that every single startup would hit. Returning an error would mean every call site handling a failure that cannot occur at runtime.

func (*Registry) Register

func (r *Registry) Register(def Definition) (*Metric, error)

Register declares a metric family.

func (*Registry) RegisterCollector

func (r *Registry) RegisterCollector(name string, fn Collector)

RegisterCollector adds a function run before each scrape. name appears in the panic-recovery path so a broken collector is identifiable.

func (*Registry) TotalSeries

func (r *Registry) TotalSeries() int

TotalSeries reports the registry's whole footprint in series.

func (*Registry) WriteTo

func (r *Registry) WriteTo(w io.Writer) (int64, error)

WriteTo renders the registry in the Prometheus text exposition format (version 0.0.4), running collectors first.

Families are emitted in registration order and series within a family in sorted label order, so a scrape is byte-stable when nothing changed. That makes diffs between scrapes meaningful and keeps the golden-file tests from depending on Go's map iteration order.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL