Documentation
¶
Overview ¶
Package hubmetrics is the process-level metrics registry for a long-lived cloop hub.
It is deliberately not pkg/metrics. That package models one *run*: a Metrics value is created by the orchestrator, keyed to a (provider, model) pair, and serialised to .cloop/metrics.json when the run ends. A hub is the opposite shape — one process, many tenants, many runs, no end — so the two cannot share a type. What a hub needs is a registry that outlives every individual thing it measures.
The package is stdlib-only, and must stay that way. Every enterprise subsystem imports it (pkg/executor, pkg/secretbroker, pkg/sessionstore, pkg/apitoken, pkg/writeback, pkg/mergequeue, pkg/gitproxy, pkg/egressbroker), several of which sit near the bottom of the import graph. A single dependency on a cloop package here would make an import cycle out of the next subsystem that wants a counter.
Cardinality is a safety property, not a tuning knob ¶
A metrics registry in a multi-tenant process is an unbounded-memory hazard wearing a friendly name. Every distinct label combination is a series that lives until the process exits, so labelling by task ID, project path, or identity subject converts tenant activity directly into hub memory — and on a hosted hub that is a denial-of-service any signed-in user can trigger by creating projects in a loop. The rule is therefore absolute: label values come from closed enumerations that exist in the source (executor kinds, isolation levels, placement constraints, denial reasons), never from user or tenant input.
Because "never" is a property of code review and code review is fallible, the registry enforces a ceiling underneath it. Each metric admits at most MaxSeriesPerMetric distinct label combinations; past that, samples fold into a single overflow series and cloop_metrics_series_dropped_total names the metric that overflowed. That turns the failure mode of a mistake from "the hub runs out of memory in a week" into "one metric goes flat and a counter says which one" — degraded observability instead of an outage.
Index ¶
- Constants
- Variables
- func RegisterCollectors(fns map[string]Collector)
- type Collector
- type Definition
- type Metric
- type MetricType
- type Registry
- func (r *Registry) Gather() string
- func (r *Registry) MustRegister(def Definition) *Metric
- func (r *Registry) Register(def Definition) (*Metric, error)
- func (r *Registry) RegisterCollector(name string, fn Collector)
- func (r *Registry) TotalSeries() int
- func (r *Registry) WriteTo(w io.Writer) (int64, error)
Constants ¶
const ( // Task failure reasons. FailStart = "start" FailRun = "run" FailCancelled = "cancelled" // Lease lifecycle events. LeaseIssued = "issued" LeaseRenewed = "renewed" LeaseRevoked = "revoked" LeaseExpired = "expired" // Session termination reasons. SessionIdleEvicted = "idle_evicted" SessionAbsoluteExpired = "absolute_expired" SessionAdminRevoked = "admin_revoked" SessionSelfLogout = "self_logout" // Generic binary results. ResultSuccess = "success" ResultFailure = "failure" ResultAllowed = "allowed" ResultDenied = "denied" // Placement results. PlacementPlaced = "placed" PlacementFailed = "failed" // Write-back results. WriteBackAccepted = "accepted" WriteBackRejected = "rejected" // Merge outcomes. MergeClean = "clean" MergeAutoResolved = "auto_resolved" MergeUnresolved = "unresolved" MergeError = "error" // KEK rotation phases. RotationTotal = "total" RotationRewrapped = "rewrapped" RotationSkipped = "skipped" RotationFailed = "failed" // Egress directions. EgressUp = "up" EgressDown = "down" )
Bounded reason vocabularies used by call sites whose own packages have no enumeration to borrow. Declaring them here rather than as string literals at the call site is what keeps the label a closed set: a typo becomes a compile error instead of a new series.
const DefaultMaxSeriesPerMetric = 128
DefaultMaxSeriesPerMetric bounds distinct label combinations per metric.
Sized against the real enumerations rather than picked round: the widest label schema in the catalog is placement failures by constraint, where pkg/executor declares 24 constraints. 128 leaves an order of magnitude of headroom for enumerations that grow, while still capping the registry's footprint at a few hundred kilobytes for the whole catalog.
Variables ¶
var ( TaskStarts = Default.MustRegister(Definition{ Name: "cloop_executor_task_starts_total", Help: "Workloads dispatched to an executor, by driver kind and isolation level.", Type: TypeCounter, Labels: []string{"executor_kind", "isolation"}, }) TaskCompletions = Default.MustRegister(Definition{ Name: "cloop_executor_task_completions_total", Help: "Workloads that ran to completion and exited zero.", Type: TypeCounter, Labels: []string{"executor_kind", "isolation"}, }) TaskFailures = Default.MustRegister(Definition{ Name: "cloop_executor_task_failures_total", Help: "Workloads that did not complete successfully. reason is one of start (the executor refused to launch it), run (it launched and failed), cancelled (the control plane withdrew it).", Type: TypeCounter, Labels: []string{"executor_kind", "isolation", "reason"}, }) // Buckets span a second to two hours because that is the real range: a // `cloop suggest` subcommand returns in seconds, an agent task on a // remote sandbox runs for an hour. Linear buckets would put every // interesting value in one bin at one end or the other. TaskDuration = Default.MustRegister(Definition{ Name: "cloop_executor_task_duration_seconds", Help: "Wall-clock duration of workloads that reached a terminal state.", Type: TypeHistogram, Labels: []string{"executor_kind", "isolation"}, Buckets: []float64{1, 5, 15, 60, 300, 900, 1800, 3600, 7200}, }) )
Executor and task lifecycle.
executor_kind is pkg/executor.Kind* (4 values); isolation is pkg/executor.Isolation (4 values). Neither carries an executor ID: a hub fronting a fleet of edge devices would turn that into one series per device per metric, which is exactly the unbounded-by-tenant-behaviour case the ceiling exists to catch.
var ( ExecutorsByState = Default.MustRegister(Definition{ Name: "cloop_executors", Help: "Registered executors by driver kind and scheduling state (ready, degraded, unreachable, cordoned, draining).", Type: TypeGauge, Labels: []string{"kind", "state"}, }) // The oldest heartbeat per kind rather than one gauge per executor. // The alert that matters is "some executor of this kind has gone // quiet", and a max answers it in one series instead of one per device // — which for an edge fleet is the difference between a bounded metric // and a per-device one. ExecutorHeartbeatAgeMax = Default.MustRegister(Definition{ Name: "cloop_executor_heartbeat_age_seconds_max", Help: "Age of the least-recently-successful liveness probe across executors of this kind. Rises without bound while a node is unreachable.", Type: TypeGauge, Labels: []string{"kind"}, }) Placements = Default.MustRegister(Definition{ Name: "cloop_executor_placements_total", Help: "Placement decisions, by result (placed or failed).", Type: TypeCounter, Labels: []string{"result"}, }) // constraint is pkg/executor.Constraint (24 values), the headline // reason Select reports. The per-candidate Detail string is not a label: // it interpolates executor IDs and byte counts. PlacementFailures = Default.MustRegister(Definition{ Name: "cloop_executor_placement_failures_total", Help: "Placement attempts that found no eligible executor, by the constraint that rejected the most candidates.", Type: TypeCounter, Labels: []string{"constraint"}, }) )
Executor fleet health. Collector-backed: the truth is the registry and the supervisor's health store, and a shadow copy maintained at mutation time would be a second source of truth with drift.
var ( LeaseEvents = Default.MustRegister(Definition{ Name: "cloop_secret_lease_events_total", Help: "Lease lifecycle transitions by credential kind and event (issued, renewed, revoked, expired).", Type: TypeCounter, Labels: []string{"kind", "event"}, }) LeasesLive = Default.MustRegister(Definition{ Name: "cloop_secret_leases_live", Help: "Leases the broker currently considers valid, by credential kind.", Type: TypeGauge, Labels: []string{"kind"}, }) )
Secret broker leases. kind is pkg/secretbroker.Kind* (7 values).
var ( UnsealFailures = Default.MustRegister(Definition{ Name: "cloop_secret_unseal_failures_total", Help: "Envelope opens that failed, by reason (key_unknown, key_retired, key_unavailable, seal_failed). Any non-zero rate means secrets are unreadable.", Type: TypeCounter, Labels: []string{"reason"}, }) KEKRotationRecords = Default.MustRegister(Definition{ Name: "cloop_secret_kek_rotation_records", Help: "Progress of the current or most recent key-encryption-key rotation, by phase (total, rewrapped, skipped, failed).", Type: TypeGauge, Labels: []string{"phase"}, }) KEKRotationActive = Default.MustRegister(Definition{ Name: "cloop_secret_kek_rotation_active", Help: "1 while a key-encryption-key rotation is in progress, 0 otherwise. A rotation that stays at 1 across scrapes has stalled with records still wrapped under the old key.", Type: TypeGauge, }) )
Sealing key health. An unseal failure is the signal that the hub can no longer read its own secrets — a wrong passphrase, a retired key still referenced, or ciphertext that failed authentication. All three are operator-visible emergencies and none of them is self-healing.
var ( SessionsCreated = Default.MustRegister(Definition{ Name: "cloop_sessions_created_total", Help: "Browser sessions established after a successful OIDC exchange.", Type: TypeCounter, }) SessionsTerminated = Default.MustRegister(Definition{ Name: "cloop_sessions_terminated_total", Help: "Sessions ended, by reason (idle_evicted, absolute_expired, admin_revoked, self_logout).", Type: TypeCounter, Labels: []string{"reason"}, }) SessionsLive = Default.MustRegister(Definition{ Name: "cloop_sessions_live", Help: "Sessions currently valid: neither past their absolute expiry nor idle beyond the timeout.", Type: TypeGauge, }) )
Sessions. Deliberately unlabelled by subject: the count of sessions is operational, the roster of who holds them is in the audit trail behind audit.read, and putting subjects in a scrape would copy that roster into every monitoring system that touches it.
var ( OIDCLogins = Default.MustRegister(Definition{ Name: "cloop_oidc_login_total", Help: "Sign-in attempts that reached a verdict, by outcome (success, discovery_failed, idp_error, invalid_request, invalid_state, exchange_failed, token_invalid, session_error, state_error). A redirect to the provider is not counted until it comes back.", Type: TypeCounter, Labels: []string{"outcome"}, }) // Unlabelled deliberately. The reason for a discovery failure is in the // hub's log and in `cloop hub doctor`, both of which can afford prose; // what a scrape needs is the single number an alert fires on, because // any non-zero rate means nobody new can sign in. OIDCDiscoveryFailures = Default.MustRegister(Definition{ Name: "cloop_oidc_discovery_failures_total", Help: "Failures to resolve the OIDC issuer (discovery or JWKS). Any non-zero rate means no new sign-in can complete; run `cloop hub doctor` for which of the two failed and why.", Type: TypeCounter, }) )
OIDC sign-in. The credential path a human uses, which until Task 20247 was the only one with no metric at all while API tokens had two.
The outcomes are oidcauth.Login* — a closed set declared in Go source, which is what makes them admissible as a label. They are worth separating rather than folding into success/failure because they name different problems with different owners: discovery_failed is the hub's config, exchange_failed is the client registration at the provider, invalid_state at any volume is somebody replaying callbacks, and idp_error is the provider's own decision.
var ( TokenAuth = Default.MustRegister(Definition{ Name: "cloop_apitoken_auth_total", Help: "API token verification attempts, by result (success or failure).", Type: TypeCounter, Labels: []string{"result"}, }) TokenAuthFailures = Default.MustRegister(Definition{ Name: "cloop_apitoken_auth_failures_total", Help: "API token verifications that failed, by reason (malformed, not_found, bad_secret, revoked, expired, no_roles). A bad_secret spike against many token IDs is credential stuffing.", Type: TypeCounter, Labels: []string{"reason"}, }) )
API token authentication. The failure reasons are pkg/apitoken's sentinel errors, which is what makes them a closed set.
var ( WriteBackBundles = Default.MustRegister(Definition{ Name: "cloop_writeback_bundles_total", Help: "Write-back bundles offered by executors, by result (accepted, rejected, unavailable).", Type: TypeCounter, Labels: []string{"result"}, }) WriteBackRejections = Default.MustRegister(Definition{ Name: "cloop_writeback_rejections_total", Help: "Bundles refused on their content, by reason code. Reasons are executor.WriteBackReason*; the prose refusal is in the audit trail, not here, because it interpolates branch names and commit SHAs.", Type: TypeCounter, Labels: []string{"reason"}, }) )
Result write-back. The distinction the labels carry is the one the code makes: a rejection is the hub refusing what a sandbox returned, an outage is the hub being unable to find out. They page differently — the first is a misbehaving or compromised sandbox, the second is broken infrastructure.
var ( MergeQueueDepth = Default.MustRegister(Definition{ Name: "cloop_mergequeue_depth", Help: "Merges waiting to be applied. The queue is serial, so sustained depth is the integration path falling behind task completion.", Type: TypeGauge, }) MergeOutcomes = Default.MustRegister(Definition{ Name: "cloop_mergequeue_merges_total", Help: "Completed merge attempts, by outcome (clean, auto_resolved, unresolved, error).", Type: TypeCounter, Labels: []string{"outcome"}, }) )
Merge queue.
var ( GitProxyPushes = Default.MustRegister(Definition{ Name: "cloop_gitproxy_pushes_total", Help: "Pushes evaluated by the interception proxy, by result (allowed or denied).", Type: TypeCounter, Labels: []string{"result"}, }) GitProxyDenials = Default.MustRegister(Definition{ Name: "cloop_gitproxy_push_denials_total", Help: "Pushes refused, by reason (ref_not_allowed, delete_denied, create_denied, update_denied, no_write, too_many_commands, push_cert). A sandbox repeatedly hitting ref_not_allowed is probing the allowlist.", Type: TypeCounter, Labels: []string{"reason"}, }) )
Git interception proxy. This is the boundary a sandbox pushes through, so its denial counter is a security signal and not only an operational one.
var ( KubeGuardRequests = Default.MustRegister(Definition{ Name: "cloop_kubeguard_requests_total", Help: "Kubernetes API requests evaluated by the access monitor, by result (allowed or denied).", Type: TypeCounter, Labels: []string{"result"}, }) KubeGuardDenials = Default.MustRegister(Definition{ Name: "cloop_kubeguard_denials_total", Help: "Kubernetes API requests refused, by reason (verb_not_allowed, namespace_not_allowed, cluster_scope, resource_not_allowed, dangerous_subresource, non_resource_path, protocol_upgrade, body_too_large, unauthenticated). Sustained verb_not_allowed is a workload that expects write access it was not granted; dangerous_subresource is an attempt to exec into a pod.", Type: TypeCounter, Labels: []string{"reason"}, }) )
Kubernetes access monitor. Same shape and same reasoning as the git proxy above: this is the boundary a sandbox's kubectl passes through, so the denial counter says whether the boundary is being tested.
var ( EgressRequests = Default.MustRegister(Definition{ Name: "cloop_egress_requests_total", Help: "Requests through the egress broker, by result (allowed or denied).", Type: TypeCounter, Labels: []string{"result"}, }) EgressDenials = Default.MustRegister(Definition{ Name: "cloop_egress_denials_total", Help: "Egress requests refused, by reason (no_grant, revoked, expired, host_not_allowed, port_not_allowed, method_not_allowed, destination_blocked, quota_exhausted).", Type: TypeCounter, Labels: []string{"reason"}, }) EgressBytes = Default.MustRegister(Definition{ Name: "cloop_egress_bytes_total", Help: "Bytes proxied on behalf of sandboxes, by direction (up or down).", Type: TypeCounter, Labels: []string{"direction"}, }) EgressSessionsLive = Default.MustRegister(Definition{ Name: "cloop_egress_sessions_live", Help: "Egress leases currently redeemable.", Type: TypeGauge, }) )
Egress broker: the hub's Internet connection, leased to sandboxes.
var ( QuotaEnforcementEnabled = Default.MustRegister(Definition{ Name: "cloop_quota_enforcement_enabled", Help: "Whether a per-identity quota policy is in force.", Type: TypeGauge, }) QuotaLimit = Default.MustRegister(Definition{ Name: "cloop_quota_limit", Help: "Configured ceiling per identity and resource.", Type: TypeGauge, Labels: []string{"identity", "resource"}, MaxSeries: quotaIdentityCeiling, }) QuotaUsage = Default.MustRegister(Definition{ Name: "cloop_quota_usage", Help: "Live consumption per identity and resource.", Type: TypeGauge, Labels: []string{"identity", "resource"}, MaxSeries: quotaIdentityCeiling, }) QuotaDenials = Default.MustRegister(Definition{ Name: "cloop_quota_denials_total", Help: "Admission refusals since this hub started, by resource.", Type: TypeCounter, Labels: []string{"resource"}, }) QuotaIdentities = Default.MustRegister(Definition{ Name: "cloop_quota_identities", Help: "Identities the hub is currently accounting.", Type: TypeGauge, }) ProjectsRegistered = Default.MustRegister(Definition{ Name: "cloop_projects_registered", Help: "Projects in the hub registry.", Type: TypeGauge, }) )
var Default = New()
Default is the process-wide registry. A hub has exactly one.
A package-level registry is the shape every metrics library converges on for the same reason: the alternative is threading a *Registry through constructors in a dozen packages that otherwise have no reason to know metrics exist, and the first constructor that cannot take a new parameter gets a global anyway. The cost — that instrumentation is not injectable — is bounded here because the registry is the only global and tests can build their own with New.
Functions ¶
func RegisterCollectors ¶
RegisterCollectors installs scrape-time collectors on the default registry, at most once per process.
Types ¶
type Collector ¶
type Collector func()
Collector recomputes derived gauges immediately before a scrape.
Some of what an operator needs is not a running tally but a reading of state that lives elsewhere: how many executors are in each health state, how deep the merge queue is, how many leases are live. Counting those at mutation time would mean every subsystem maintaining a shadow copy of its own state and keeping it consistent — a second source of truth, with the drift that implies. A collector reads the real one at scrape time instead.
type Definition ¶
type Definition struct {
Name string
Help string
Type MetricType
Labels []string
// Buckets are the histogram upper bounds in ascending order, excluding
// +Inf which is implied. Ignored for counters and gauges.
Buckets []float64
// MaxSeries overrides Registry.MaxSeriesPerMetric for this metric. Zero
// takes the registry default.
//
// Raising it is only defensible for a collector-backed gauge, because
// those Reset before every scrape: their cardinality is the number of
// objects alive at scrape time rather than the number ever observed, so
// churn cannot accumulate series the way it does on a counter. A raised
// ceiling on a counter is just a slower memory leak.
MaxSeries int
}
Definition declares a metric family.
type Metric ¶
type Metric struct {
// contains filtered or unexported fields
}
Metric is one metric family: a name, a type, a fixed label schema, and the series observed for it so far.
func (*Metric) Add ¶
Add adds delta to the counter or gauge identified by labelValues.
A nil Metric is a no-op so a subsystem can hold an optional metric without guarding every call site, and a negative delta on a counter is dropped rather than applied: a counter that can go down breaks rate() in ways that surface as impossible graphs long after the bug that caused them.
func (*Metric) Reset ¶
func (m *Metric) Reset()
Reset drops every series in the metric.
Collector-backed gauges call this before repopulating, so a series whose underlying object is gone (a deregistered executor, a closed session) stops being exported rather than freezing at its last value. A frozen gauge is worse than a missing one: it reads as a live measurement and will hold an alert open forever.
func (*Metric) SeriesCount ¶
SeriesCount reports how many distinct label combinations a metric holds, including the overflow series. Used by tests that assert the ceiling holds.
type MetricType ¶
type MetricType string
MetricType is the Prometheus exposition type of a metric family.
const ( TypeCounter MetricType = "counter" TypeGauge MetricType = "gauge" TypeHistogram MetricType = "histogram" )
type Registry ¶
type Registry struct {
// MaxSeriesPerMetric is the per-metric cardinality ceiling. Read
// without synchronisation, so set it before the registry is shared.
MaxSeriesPerMetric int
// contains filtered or unexported fields
}
Registry holds every metric family in the process and renders them in the Prometheus text exposition format. The zero value is not usable; call New.
func (*Registry) MustRegister ¶
func (r *Registry) MustRegister(def Definition) *Metric
MustRegister declares a metric family and returns it, panicking on a malformed definition or a duplicate name.
Panicking is right here and would be wrong almost anywhere else in cloop: the catalog is built at package initialisation from string literals, so a panic can only fire on a programming error that every single startup would hit. Returning an error would mean every call site handling a failure that cannot occur at runtime.
func (*Registry) Register ¶
func (r *Registry) Register(def Definition) (*Metric, error)
Register declares a metric family.
func (*Registry) RegisterCollector ¶
RegisterCollector adds a function run before each scrape. name appears in the panic-recovery path so a broken collector is identifiable.
func (*Registry) TotalSeries ¶
TotalSeries reports the registry's whole footprint in series.
func (*Registry) WriteTo ¶
WriteTo renders the registry in the Prometheus text exposition format (version 0.0.4), running collectors first.
Families are emitted in registration order and series within a family in sorted label order, so a scrape is byte-stable when nothing changed. That makes diffs between scrapes meaningful and keeps the golden-file tests from depending on Go's map iteration order.