hubmetrics

package
v0.0.7 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 27, 2026 License: MIT Imports: 8 Imported by: 0

Documentation

Overview

Package hubmetrics is the process-level metrics registry for a long-lived cloop hub.

It is deliberately not pkg/metrics. That package models one *run*: a Metrics value is created by the orchestrator, keyed to a (provider, model) pair, and serialised to .cloop/metrics.json when the run ends. A hub is the opposite shape — one process, many tenants, many runs, no end — so the two cannot share a type. What a hub needs is a registry that outlives every individual thing it measures.

The package is stdlib-only, and must stay that way. Every enterprise subsystem imports it (pkg/executor, pkg/secretbroker, pkg/sessionstore, pkg/apitoken, pkg/writeback, pkg/mergequeue, pkg/gitproxy, pkg/egressbroker), several of which sit near the bottom of the import graph. A single dependency on a cloop package here would make an import cycle out of the next subsystem that wants a counter.

Cardinality is a safety property, not a tuning knob

A metrics registry in a multi-tenant process is an unbounded-memory hazard wearing a friendly name. Every distinct label combination is a series that lives until the process exits, so labelling by task ID, project path, or identity subject converts tenant activity directly into hub memory — and on a hosted hub that is a denial-of-service any signed-in user can trigger by creating projects in a loop. The rule is therefore absolute: label values come from closed enumerations that exist in the source (executor kinds, isolation levels, placement constraints, denial reasons), never from user or tenant input.

Because "never" is a property of code review and code review is fallible, the registry enforces a ceiling underneath it. Each metric admits at most MaxSeriesPerMetric distinct label combinations; past that, samples fold into a single overflow series and cloop_metrics_series_dropped_total names the metric that overflowed. That turns the failure mode of a mistake from "the hub runs out of memory in a week" into "one metric goes flat and a counter says which one" — degraded observability instead of an outage.

Index

Constants

View Source
const (
	IsolationProbeVerified     = "verified"
	IsolationProbeContradicted = "contradicted"
	IsolationProbeUnverified   = "unverified"
	IsolationProbeFailed       = "failed"
)

IsolationProbes status values.

View Source
const (
	ModelRelayAllowed  = "allowed"
	ModelRelayRefused  = "refused"
	ModelRelayEnded    = "ended"
	ModelRelayRejected = "rejected"

	ModelRelayInput      = "input"
	ModelRelayOutput     = "output"
	ModelRelayCacheRead  = "cache_read"
	ModelRelayCacheWrite = "cache_write"

	ModelRelayUnreported   = "unreported"
	ModelRelayOverreported = "overreported"
)

Model relay label values.

View Source
const (
	RestoreKindGit      = "git"
	RestoreKindKube     = "kube"
	RestoreKindEgress   = "egress"
	RestoreKindAppToken = "app_token"
	RestoreKindCI       = "ci"
	RestoreKindModel    = "model"

	RestoreRestored = "restored"
	RestoreRefused  = "refused"
	RestoreExpired  = "expired"
	RestoreLost     = "lost"
	RestoreFailed   = "failed"
)

SessionRestores label values.

View Source
const (
	ApprovalNotifyDelivered  = "delivered"
	ApprovalNotifyFailed     = "failed"
	ApprovalNotifyDropped    = "dropped"
	ApprovalNotifySuperseded = "superseded"
)

What became of one approval notice at one webhook.

View Source
const (
	AuditForwardDelivered = "delivered"
	AuditForwardTruncated = "truncated"
	AuditForwardFailed    = "failed"
	AuditForwardLost      = "lost"
)

What became of audit rows the forwarder handled.

View Source
const (
	// The role of the database handle an audit append went through
	// (statedb.Role).
	AuditChainControlPlane = "control-plane"
	AuditChainProject      = "project"
	AuditChainUnclassified = "unclassified"

	// Why an audit append failed.
	AuditFailLocked   = "locked"
	AuditFailReadOnly = "readonly"
	AuditFailFull     = "full"
	AuditFailIO       = "io"
	AuditFailClosed   = "closed"
	AuditFailOther    = "other"

	// What became of one checkpoint record.
	AuditCheckpointSigned   = "signed"
	AuditCheckpointUnsigned = "unsigned"
	AuditCheckpointFailed   = "failed"
)

Audit metric label values.

View Source
const (
	// Task failure reasons. disk_limit is a run its executor stopped because
	// the workspace grew past its disk limit (Task 20405).
	FailStart     = "start"
	FailRun       = "run"
	FailCancelled = "cancelled"
	FailDiskLimit = "disk_limit"

	// Lease lifecycle events.
	LeaseIssued  = "issued"
	LeaseRenewed = "renewed"
	LeaseRevoked = "revoked"
	LeaseExpired = "expired"

	// Session termination reasons. The last two are the endings neither the
	// user nor an operator chose: the identity provider refusing to renew the
	// grant, and the per-identity session quota making room for a new one.
	SessionIdleEvicted     = "idle_evicted"
	SessionAbsoluteExpired = "absolute_expired"
	SessionAdminRevoked    = "admin_revoked"
	SessionSelfLogout      = "self_logout"
	SessionIdPRevoked      = "idp_revoked"
	SessionQuotaEvicted    = "quota_evicted"

	// API token verification failures: one per pkg/apitoken sentinel, and
	// store_error for a token the hub could not check at all.
	TokenMalformed  = "malformed"
	TokenNotFound   = "not_found"
	TokenBadSecret  = "bad_secret"
	TokenRevoked    = "revoked"
	TokenExpired    = "expired"
	TokenNoRoles    = "no_roles"
	TokenStoreError = "store_error"

	// Egress refusals: one per pkg/egressbroker denial sentinel. expired
	// covers both an expired grant and an expired session.
	EgressNoGrant            = "no_grant"
	EgressRevoked            = "revoked"
	EgressExpired            = "expired"
	EgressHostNotAllowed     = "host_not_allowed"
	EgressPortNotAllowed     = "port_not_allowed"
	EgressMethodNotAllowed   = "method_not_allowed"
	EgressDestinationBlocked = "destination_blocked"
	EgressQuotaExhausted     = "quota_exhausted"

	// Generic binary results.
	ResultSuccess = "success"
	ResultFailure = "failure"
	ResultAllowed = "allowed"
	ResultDenied  = "denied"

	// Placement results.
	PlacementPlaced = "placed"
	PlacementFailed = "failed"

	// Write-back results.
	WriteBackAccepted    = "accepted"
	WriteBackRejected    = "rejected"
	WriteBackUnavailable = "unavailable"

	// Write-back frame refusals (Task 20399).
	WriteBackRefusedLate           = "late"
	WriteBackRefusedNotRequested   = "not_requested"
	WriteBackRefusedOverCap        = "over_cap"
	WriteBackRefusedExecutorBudget = "executor_budget"
	WriteBackRefusedProcessBudget  = "process_budget"

	// Merge outcomes.
	MergeClean        = "clean"
	MergeAutoResolved = "auto_resolved"
	MergeUnresolved   = "unresolved"
	MergeError        = "error"

	// Hub CA rotation: the proxies a rotation ends sessions on.
	HubCAProxyGit   = "git_proxy"
	HubCAProxyKube  = "kube_guard"
	HubCAProxyRelay = "model_relay"

	// KEK rotation phases.
	RotationTotal     = "total"
	RotationRewrapped = "rewrapped"
	RotationSkipped   = "skipped"
	RotationFailed    = "failed"

	// Egress directions.
	EgressUp   = "up"
	EgressDown = "down"
)

Bounded reason vocabularies used by call sites whose own packages have no enumeration to borrow. Declaring them here rather than as string literals at the call site is what keeps the label a closed set: a typo becomes a compile error instead of a new series.

View Source
const DefaultMaxSeriesPerMetric = 128

DefaultMaxSeriesPerMetric bounds distinct label combinations per metric.

Sized against the real enumerations rather than picked round: the widest label schema in the catalog is placement failures by constraint, where pkg/executor declares 24 constraints. 128 leaves an order of magnitude of headroom for enumerations that grow, while still capping the registry's footprint at a few hundred kilobytes for the whole catalog.

View Source
const DiskVolumesMax = 32

DiskVolumesMax caps the volumes cloop_hub_disk_free_bytes reports. A hub writes to a handful of filesystems; one registering projects on hundreds is past what a per-volume series is good for.

Variables

View Source
var (
	TaskStarts = Default.MustRegister(Definition{
		Name:   "cloop_executor_task_starts_total",
		Help:   "Runs dispatched to an executor, by driver kind and isolation level, counted once the executor is chosen whether or not it then launches the run.",
		Type:   TypeCounter,
		Labels: []string{"executor_kind", "isolation"},
	})

	TaskCompletions = Default.MustRegister(Definition{
		Name:   "cloop_executor_task_completions_total",
		Help:   "Dispatched runs whose workload ran to completion and exited zero.",
		Type:   TypeCounter,
		Labels: []string{"executor_kind", "isolation"},
	})

	TaskFailures = Default.MustRegister(Definition{
		Name:   "cloop_executor_task_failures_total",
		Help:   "Dispatched runs that did not complete successfully. reason is one of start (it never began: the executor refused it or could not launch it), run (it launched and failed, or ended in a way its executor could not account for), cancelled (the control plane stopped it), disk_limit (its executor stopped it because its workspace grew past its disk limit).",
		Type:   TypeCounter,
		Labels: []string{"executor_kind", "isolation", "reason"},
	})

	// Buckets span a second to two hours because that is the real range: a
	// run with nothing left to do, or one that crashes on start, is over in
	// seconds; an agent task on a remote sandbox runs for an hour. Linear
	// buckets would put every interesting value in one bin at one end or the
	// other.
	TaskDuration = Default.MustRegister(Definition{
		Name:    "cloop_executor_task_duration_seconds",
		Help:    "Wall-clock duration of dispatched runs that launched and reached a terminal state, as the executor timed them.",
		Type:    TypeHistogram,
		Labels:  []string{"executor_kind", "isolation"},
		Buckets: []float64{1, 5, 15, 60, 300, 900, 1800, 3600, 7200},
	})
)

Executor and task lifecycle: the harness runs the hub dispatches to an executor and settles when they end (pkg/ui startWorkloadAs and runEnded).

executor_kind is pkg/executor.Kind* (5 values); isolation is pkg/executor.Isolation (4 values). Neither carries an executor ID: a hub fronting a fleet of edge devices would turn that into one series per device per metric, which is exactly the unbounded-by-tenant-behaviour case the ceiling exists to catch.

Every start is counted once more when the run is settled — as a completion, or as a failure with one of the Fail* reasons — so starts minus completions minus failures is the number of runs in flight.

View Source
var (
	ExecutorsByState = Default.MustRegister(Definition{
		Name:   "cloop_executors",
		Help:   "Registered executors by driver kind and scheduling state (ready, degraded, unreachable, cordoned, draining). Exported by the cluster leader only.",
		Type:   TypeGauge,
		Labels: []string{"kind", "state"},
	})

	// The oldest heartbeat per kind rather than one gauge per executor.
	// The alert that matters is "some executor of this kind has gone
	// quiet", and a max answers it in one series instead of one per device
	// — which for an edge fleet is the difference between a bounded metric
	// and a per-device one.
	ExecutorHeartbeatAgeMax = Default.MustRegister(Definition{
		Name:   "cloop_executor_heartbeat_age_seconds_max",
		Help:   "Age of the least-recently-successful liveness probe across executors of this kind. Rises without bound while a node is unreachable. Exported by the cluster leader only.",
		Type:   TypeGauge,
		Labels: []string{"kind"},
	})

	Placements = Default.MustRegister(Definition{
		Name:   "cloop_executor_placements_total",
		Help:   "Placement decisions, by result (placed or failed).",
		Type:   TypeCounter,
		Labels: []string{"result"},
	})

	// constraint is pkg/executor.Constraint (24 values), the headline
	// reason Select reports. The per-candidate Detail string is not a label:
	// it interpolates executor IDs and byte counts.
	PlacementFailures = Default.MustRegister(Definition{
		Name:   "cloop_executor_placement_failures_total",
		Help:   "Placement attempts that found no eligible executor, by the constraint that rejected the most candidates.",
		Type:   TypeCounter,
		Labels: []string{"constraint"},
	})

	// point is executor.ProbePoint* (5 values); status is IsolationProbe*
	// (4 values). Task 20439.
	IsolationProbes = Default.MustRegister(Definition{
		Name:   "cloop_executor_isolation_probes_total",
		Help:   "Isolation probes run under an executor's runtime, by where they were run from and how the runtime's claim stood after: verified, contradicted, unverified, or failed when the probe could not run.",
		Type:   TypeCounter,
		Labels: []string{"point", "status"},
	})
)

Executor fleet health. Collector-backed: the truth is the registry and the supervisor's health store, and a shadow copy maintained at mutation time would be a second source of truth with drift. The health store is the shared database, so a hub cluster exports both from its leader only.

View Source
var (
	LeaseEvents = Default.MustRegister(Definition{
		Name:   "cloop_secret_lease_events_total",
		Help:   "Lease lifecycle transitions by credential kind and event (issued, renewed, revoked, expired).",
		Type:   TypeCounter,
		Labels: []string{"kind", "event"},
	})

	// Per process, not per cluster: a lease's material lives with the member
	// that materialised it, so each member reports the leases it holds and
	// the sum is the cluster's.
	LeasesLive = Default.MustRegister(Definition{
		Name:   "cloop_secret_leases_live",
		Help:   "Leases this hub process issued, still holds and has not seen expire, by credential kind. Each cluster member reports its own.",
		Type:   TypeGauge,
		Labels: []string{"kind"},
	})
)

Secret broker leases. kind is pkg/secretbroker.Kind* (9 values).

View Source
var (
	UnsealFailures = Default.MustRegister(Definition{
		Name:   "cloop_secret_unseal_failures_total",
		Help:   "Envelope opens that failed, by reason (key_unknown, key_retired, key_unavailable, kms_unavailable, seal_failed). Any non-zero rate means secrets are unreadable.",
		Type:   TypeCounter,
		Labels: []string{"reason"},
	})

	// Calls to the external key-management service the sealing keys are
	// wrapped by (Task 20433). backend is config.KMSProvider* (2 values),
	// operation pkg/kms Op* (4) and result pkg/kms Result* (6). Per process:
	// each member counts the calls it made, and the sum is the cluster's.
	KMSRequests = Default.MustRegister(Definition{
		Name:   "cloop_kms_requests_total",
		Help:   "Calls this hub process made to the key-management service its sealing keys are wrapped by, by backend (kmsv2, transit), operation (status, encrypt, decrypt, login) and result (ok, unavailable, timeout, denied, invalid, error).",
		Type:   TypeCounter,
		Labels: []string{"backend", "operation", "result"},
	})

	// Buckets from 5 ms, a plugin answering from the same Pod, to 10 s, past
	// any configured timeout's default: a call slower than that has failed.
	KMSRequestDuration = Default.MustRegister(Definition{
		Name:    "cloop_kms_request_duration_seconds",
		Help:    "Latency of calls to the key-management service, by backend and operation, whatever their result.",
		Type:    TypeHistogram,
		Labels:  []string{"backend", "operation"},
		Buckets: []float64{0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10},
	})

	// Both read the rotation history `cloop hub key rotate` writes to the
	// shared database, so a hub cluster exports them from its leader only.
	KEKRotationRecords = Default.MustRegister(Definition{
		Name:   "cloop_secret_kek_rotation_records",
		Help:   "Progress of the current or most recent key-encryption-key rotation, by phase (total, rewrapped, skipped, failed). Exported by the cluster leader only.",
		Type:   TypeGauge,
		Labels: []string{"phase"},
	})

	KEKRotationActive = Default.MustRegister(Definition{
		Name: "cloop_secret_kek_rotation_active",
		Help: "1 while a key-encryption-key rotation is in progress, 0 otherwise. A rotation that stays at 1 across scrapes has stalled with records still wrapped under the old key. Exported by the cluster leader only.",
		Type: TypeGauge,
	})
)

Sealing key health. An unseal failure is the signal that the hub can no longer read its own secrets — a wrong passphrase, a retired key still referenced, or ciphertext that failed authentication. All three are operator-visible emergencies and none of them is self-healing.

View Source
var (
	HubCALeavesIssued = Default.MustRegister(Definition{
		Name:   "cloop_hub_ca_leaves_issued_total",
		Help:   "Credential monitor certificates this hub process issued from the hub CA, by monitor (git_proxy, kube_guard) and reason (first, renewal, names, activation, unreadable).",
		Type:   TypeCounter,
		Labels: []string{"monitor", "reason"},
	})

	HubCATrustAdoptions = Default.MustRegister(Definition{
		Name: "cloop_hub_ca_trust_adoptions_total",
		Help: "Times this hub process started handing sandboxes a different set of hub CAs, or signing with a different one: once per rotation step it saw.",
		Type: TypeCounter,
	})

	HubCASessionsEnded = Default.MustRegister(Definition{
		Name:   "cloop_hub_ca_sessions_ended_total",
		Help:   "Proxy sessions this hub process ended because an emergency rotation named their runs, by proxy (git_proxy, kube_guard, model_relay).",
		Type:   TypeCounter,
		Labels: []string{"proxy"},
	})

	HubCANotAfter = Default.MustRegister(Definition{
		Name:   "cloop_hub_ca_not_after_timestamp_seconds",
		Help:   "When each hub CA expires, as a Unix time, by slot (active, next, previous). Alert well ahead of the active one's: replacing it is a staged rotation. Exported by the cluster leader only.",
		Type:   TypeGauge,
		Labels: []string{"slot"},
	})

	HubCARotationPhase = Default.MustRegister(Definition{
		Name:   "cloop_hub_ca_rotation_phase",
		Help:   "1 for the phase the hub CA is in (none, steady, staged, activated), 0 for the others. Exported by the cluster leader only.",
		Type:   TypeGauge,
		Labels: []string{"phase"},
	})

	HubCARotationStepAge = Default.MustRegister(Definition{
		Name: "cloop_hub_ca_rotation_step_age_seconds",
		Help: "Seconds since the rotation under way took its last step (staged, or activated with the previous CA not yet retired); 0 when none is under way. Exported by the cluster leader only.",
		Type: TypeGauge,
	})

	HubCARunsBlocking = Default.MustRegister(Definition{
		Name: "cloop_hub_ca_runs_blocking",
		Help: "Running runs the rotation's next step would break: while staged, those not given the staged CA; once activated, those given only the previous one. Exported by the cluster leader only.",
		Type: TypeGauge,
	})

	HubCAMembersBehind = Default.MustRegister(Definition{
		Name: "cloop_hub_ca_members_behind",
		Help: "Live hub members that have not adopted the rotation's current step, or whose build predates rotation. Exported by the cluster leader only.",
		Type: TypeGauge,
	})
)

The hub's own certificate authority and its rotation (Task 20438). The labels are pkg/hubca's closed sets: monitor is hubca.Monitor (git_proxy, kube_guard), reason a hubca.Reason* (first, renewal, names, activation, unreadable), slot a hubca.Slot (active, next, previous), phase a hubca.Phase (none, steady, staged, activated), proxy the three a rotation ends sessions on (HubCAProxy*). The three counters are each member's own; the gauges read the shared database and are exported by the cluster leader only.

View Source
var (
	SessionsCreated = Default.MustRegister(Definition{
		Name: "cloop_sessions_created_total",
		Help: "Browser sessions established after a successful OIDC exchange.",
		Type: TypeCounter,
	})

	// Counted where the session store's delete is the arbiter, so a session
	// two requests both found expired, or a janitor pass racing a sign-out,
	// ends once in this counter as it does in the audit trail.
	SessionsTerminated = Default.MustRegister(Definition{
		Name:   "cloop_sessions_terminated_total",
		Help:   "Sessions ended, by reason (idle_evicted, absolute_expired, admin_revoked, self_logout, idp_revoked, quota_evicted).",
		Type:   TypeCounter,
		Labels: []string{"reason"},
	})

	SessionsLive = Default.MustRegister(Definition{
		Name: "cloop_sessions_live",
		Help: "Sessions currently valid: neither past their absolute expiry nor idle beyond the timeout. Exported by the cluster leader only.",
		Type: TypeGauge,
	})
)

Sessions. Deliberately unlabelled by subject: the count of sessions is operational, the roster of who holds them is in the audit trail behind audit.read, and putting subjects in a scrape would copy that roster into every monitoring system that touches it.

View Source
var (
	OIDCLogins = Default.MustRegister(Definition{
		Name:   "cloop_oidc_login_total",
		Help:   "Sign-in attempts that reached a verdict, by outcome (success, discovery_failed, idp_error, invalid_request, invalid_state, exchange_failed, token_invalid, session_error, state_error). A redirect to the provider is not counted until it comes back.",
		Type:   TypeCounter,
		Labels: []string{"outcome"},
	})

	// Unlabelled deliberately. The reason for a discovery failure is in the
	// hub's log and in `cloop hub doctor`, both of which can afford prose;
	// what a scrape needs is the single number an alert fires on, because
	// any non-zero rate means nobody new can sign in.
	OIDCDiscoveryFailures = Default.MustRegister(Definition{
		Name: "cloop_oidc_discovery_failures_total",
		Help: "Failures to resolve the OIDC issuer (discovery or JWKS). Any non-zero rate means no new sign-in can complete; run `cloop hub doctor` for which of the two failed and why.",
		Type: TypeCounter,
	})

	// Silent claim renewals (Task 20359): the prompt=none round trip a
	// signed-in dashboard makes from a hidden frame when the hub holds no
	// refresh token to re-assert the session's claims with. The outcomes are
	// oidcauth.Renew*, a closed set.
	//
	// Separate from OIDCLogins rather than another outcome on it: a renewal
	// runs every few minutes per open tab, so folding it in would swamp the
	// sign-in ratio an operator reads as "can people sign in". The ratio worth
	// watching here is interaction_required against ok — a hub where it climbs
	// is one whose users are being sent back to the provider mid-session, most
	// often because their browser blocks the provider's cookies inside a frame.
	OIDCRenewals = Default.MustRegister(Definition{
		Name:   "cloop_oidc_renewal_total",
		Help:   "Silent claim renewals that reached a verdict, by outcome (ok, interaction_required, no_session, not_enabled, idp_error, discovery_failed, state_error, invalid_state, exchange_failed, token_invalid, subject_mismatch, store_error). A rising interaction_required rate means users are being sent back to the provider mid-session.",
		Type:   TypeCounter,
		Labels: []string{"outcome"},
	})
)

OIDC sign-in. The credential path a human uses, which until Task 20247 was the only one with no metric at all while API tokens had two.

The outcomes are oidcauth.Login* — a closed set declared in Go source, which is what makes them admissible as a label. They are worth separating rather than folding into success/failure because they name different problems with different owners: discovery_failed is the hub's config, exchange_failed is the client registration at the provider, invalid_state at any volume is somebody replaying callbacks, and idp_error is the provider's own decision.

View Source
var (
	TokenAuth = Default.MustRegister(Definition{
		Name:   "cloop_apitoken_auth_total",
		Help:   "API token verification attempts, by result (success or failure).",
		Type:   TypeCounter,
		Labels: []string{"result"},
	})

	TokenAuthFailures = Default.MustRegister(Definition{
		Name:   "cloop_apitoken_auth_failures_total",
		Help:   "API token verifications that failed, by reason (malformed, not_found, bad_secret, revoked, expired, no_roles, store_error). A bad_secret spike against many token IDs is credential stuffing; store_error is the hub, not the caller.",
		Type:   TypeCounter,
		Labels: []string{"reason"},
	})
)

API token authentication. The failure reasons are pkg/apitoken's sentinel errors, which is what makes them a closed set, plus store_error for a token the hub could not check because its own store failed.

View Source
var (
	WriteBackBundles = Default.MustRegister(Definition{
		Name:   "cloop_writeback_bundles_total",
		Help:   "Write-back bundles offered by executors, by result (accepted, rejected, unavailable).",
		Type:   TypeCounter,
		Labels: []string{"result"},
	})

	WriteBackRejections = Default.MustRegister(Definition{
		Name:   "cloop_writeback_rejections_total",
		Help:   "Bundles refused on their content, by reason code. Reasons are executor.WriteBackReason*; the prose refusal is in the audit trail, not here, because it interpolates branch names and commit SHAs.",
		Type:   TypeCounter,
		Labels: []string{"reason"},
	})
)

Result write-back. The distinction the labels carry is the one the code makes: a rejection is the hub refusing what a sandbox returned, an outage is the hub being unable to find out. They page differently — the first is a misbehaving or compromised sandbox, the second is broken infrastructure.

View Source
var (
	WriteBackFrameRefusals = Default.MustRegister(Definition{
		Name:   "cloop_writeback_frame_refusals_total",
		Help:   "Result frames from remote executors the hub refused before keeping their bytes, by reason (late: after the handle's final status or its result; not_requested: for a handle dispatched with no such write-back or no project seed; over_cap: past the bundle cap the handle's spec asked for; executor_budget, process_budget: holding it would cross executors.remote.max_pinned_writeback_bytes or max_pinned_writeback_total_bytes). An honest agent sends none of the first three.",
		Type:   TypeCounter,
		Labels: []string{"reason"},
	})

	WriteBackPinnedBytes = Default.MustRegister(Definition{
		Name: "cloop_writeback_pinned_bytes",
		Help: "Returned work this hub process holds in memory waiting to be collected: bundles remote executors are sending or have sent, and seeded runs' project-state documents. Bounded by executors.remote.max_pinned_writeback_total_bytes. Each cluster member reports its own.",
		Type: TypeGauge,
	})

	WriteBackPinnedBytesMax = Default.MustRegister(Definition{
		Name: "cloop_writeback_pinned_bytes_max",
		Help: "The most returned work any single remote executor has this hub process hold, against its executors.remote.max_pinned_writeback_bytes budget. Each cluster member reports its own.",
		Type: TypeGauge,
	})
)

The transport half of write-back (Task 20399): what a remote executor's returned work holds in hub memory before anyone collects it, and the frames refused before their bytes were kept. reason is one of the WriteBackRefused* constants below.

Both gauges are per process — a member holds what reached it — and neither is labelled by executor: the catalog's rule, and a fleet of edge devices is exactly the unbounded set it exists for. The largest single executor is what an alert on one device approaching its budget needs, and a max answers it in one series.

View Source
var (
	MergeQueueDepth = Default.MustRegister(Definition{
		Name: "cloop_mergequeue_depth",
		Help: "Merges waiting to be applied. The queue is serial, so sustained depth is the integration path falling behind task completion.",
		Type: TypeGauge,
	})

	MergeOutcomes = Default.MustRegister(Definition{
		Name:   "cloop_mergequeue_merges_total",
		Help:   "Completed merge attempts, by outcome (clean, auto_resolved, unresolved, error).",
		Type:   TypeCounter,
		Labels: []string{"outcome"},
	})
)

Merge queue.

View Source
var (
	GitProxyPushes = Default.MustRegister(Definition{
		Name:   "cloop_gitproxy_pushes_total",
		Help:   "Pushes evaluated by the interception proxy, by result (allowed or denied).",
		Type:   TypeCounter,
		Labels: []string{"result"},
	})

	GitProxyDenials = Default.MustRegister(Definition{
		Name:   "cloop_gitproxy_push_denials_total",
		Help:   "Pushes refused, by reason (ref_not_allowed, delete_denied, create_denied, update_denied, no_write, too_many_commands, push_cert, and from the push scan secret_in_push, scan_limit, unreadable_pack). A sandbox repeatedly hitting ref_not_allowed is probing the allowlist; any secret_in_push is a run that tried to publish a credential it was given.",
		Type:   TypeCounter,
		Labels: []string{"reason"},
	})

	// The push scan (Task 20436): every push the policy allows is inflated
	// and scanned for the run's credentials on its way to the forge.
	GitProxyPushScans = Default.MustRegister(Definition{
		Name:   "cloop_gitproxy_push_scans_total",
		Help:   "Pushes the policy allowed, by what their scan found (clean, secret, limit, unreadable, incomplete — the sandbox or the forge went away mid-scan — or unscanned, with executors.git_proxy.push_scan: off).",
		Type:   TypeCounter,
		Labels: []string{"result"},
	})

	GitProxyPushScanBytes = Default.MustRegister(Definition{
		Name: "cloop_gitproxy_push_scanned_bytes_total",
		Help: "Bytes the push scan inflated and scanned. Its rate against cloop_gitproxy_push_scan_seconds says what the scan costs the hub.",
		Type: TypeCounter,
	})

	// Buckets from 10 ms, a push of a few small objects, to the 300 s default
	// of executors.git_proxy.push_scan_seconds.
	GitProxyPushScanSeconds = Default.MustRegister(Definition{
		Name:    "cloop_gitproxy_push_scan_seconds",
		Help:    "Time one push's scan spent inflating and matching, not waiting on the sandbox or the forge. Pushes nearing executors.git_proxy.push_scan_seconds are about to be refused for it.",
		Type:    TypeHistogram,
		Buckets: []float64{0.01, 0.05, 0.25, 1, 5, 15, 60, 300},
	})

	// Counted rather than audited: see Proxy.reject.
	GitProxyAnonymous = Default.MustRegister(Definition{
		Name: "cloop_gitproxy_anonymous_requests_total",
		Help: "Requests to the git proxy that presented no credential and were refused before reaching a session. Most are git's own authentication challenge, answered with 401 before it retries with the session credential; a rate far above the proxy's request rate is something other than git reaching the port.",
		Type: TypeCounter,
	})
)

Git interception proxy. This is the boundary a sandbox pushes through, so its denial counter is a security signal and not only an operational one.

View Source
var (
	KubeGuardRequests = Default.MustRegister(Definition{
		Name:   "cloop_kubeguard_requests_total",
		Help:   "Kubernetes API requests evaluated by the access monitor, by result (allowed or denied).",
		Type:   TypeCounter,
		Labels: []string{"result"},
	})

	KubeGuardDenials = Default.MustRegister(Definition{
		Name:   "cloop_kubeguard_denials_total",
		Help:   "Kubernetes API requests refused, by reason (verb_not_allowed, namespace_not_allowed, cluster_scope, resource_not_allowed, dangerous_subresource, non_resource_path, protocol_upgrade, body_too_large, unauthenticated, and for writes that would mint a credential outside the monitor: service_account_token_secret, workload_service_account, host_access, rbac_binding, rbac_role, certificate_signing_request, unreadable_body). Sustained verb_not_allowed is a workload that expects write access it was not granted; dangerous_subresource is an attempt to exec into a pod; any of the credential reasons is a workload reaching for a credential of its own.",
		Type:   TypeCounter,
		Labels: []string{"reason"},
	})
)

Kubernetes access monitor. Same shape and same reasoning as the git proxy above: this is the boundary a sandbox's kubectl passes through, so the denial counter says whether the boundary is being tested.

View Source
var (
	EgressRequests = Default.MustRegister(Definition{
		Name:   "cloop_egress_requests_total",
		Help:   "Requests through the egress proxy that reached a policy verdict, by result (allowed or denied). A request that failed for want of DNS or a route is neither.",
		Type:   TypeCounter,
		Labels: []string{"result"},
	})

	// Every refusal the broker makes, which is more than the denied requests:
	// no_grant, revoked and an expired grant refuse a session before any
	// request exists, and quota_exhausted and expired also cut transfers
	// already under way.
	EgressDenials = Default.MustRegister(Definition{
		Name:   "cloop_egress_denials_total",
		Help:   "Egress refusals, by reason (no_grant, revoked, expired, host_not_allowed, port_not_allowed, method_not_allowed, destination_blocked, quota_exhausted).",
		Type:   TypeCounter,
		Labels: []string{"reason"},
	})

	EgressBytes = Default.MustRegister(Definition{
		Name:   "cloop_egress_bytes_total",
		Help:   "Bytes proxied on behalf of sandboxes, by direction (up or down).",
		Type:   TypeCounter,
		Labels: []string{"direction"},
	})

	EgressSessionsLive = Default.MustRegister(Definition{
		Name: "cloop_egress_sessions_live",
		Help: "Egress proxy sessions this process has issued and not yet closed. An expired session is closed within the proxy's reap interval.",
		Type: TypeGauge,
	})
)

Egress broker: the hub's Internet connection, leased to sandboxes. Recorded by pkg/egressbroker in whichever process runs the broker and its proxy, so all four are per process; a session lives in the broker that issued it.

View Source
var (
	FirewallDenials = Default.MustRegister(Definition{
		Name:   "cloop_firewall_denials_total",
		Help:   "Connections a sandbox's IP firewall refused, as its packet log reported them, by executor kind and the deny path that refused them (default, deny, meta, block, xlat, unitemised). The log is rate-limited per deny path, so a flood counts as at least the rate.",
		Type:   TypeCounter,
		Labels: []string{"executor_kind", "rule"},
	})

	FirewallDenialReportsDropped = Default.MustRegister(Definition{
		Name:   "cloop_firewall_denial_reports_dropped_total",
		Help:   "Reports of sandbox firewall refusals the hub did not record, by reason: late (after the run's final status), no_sink (no hub in this process), rate (a device past its per-run report rate), malformed (failed validation), unknown_run (a handle no run here owns), queue_full (the recorder was behind).",
		Type:   TypeCounter,
		Labels: []string{"reason"},
	})
)

Sandbox firewall refusals (Task 20437): what a sandbox's IP firewall refused, as its packet log reported it — read by the hub for its own container executor and by each device's agent for its sandboxes. Recorded by pkg/ui as each batch is recorded on its run, so both are per process.

executor_kind is pkg/executor.Kind*; rule is the deny path's kind (pkg/netfilter tags: default, deny, meta, block, xlat) or unitemised for the refusals a run's capped aggregate counted without itemising. reason is pkg/ui's closed list: late, no_sink, rate, malformed, unknown_run, queue_full.

View Source
var (
	ModelRelayRequests = Default.MustRegister(Definition{
		Name:   "cloop_model_relay_requests_total",
		Help:   "Model calls that reached the relay, by outcome: allowed (relayed to the API), refused (a policy or budget refusal, never relayed), ended (presented a session that has ended, told why), rejected (no usable session token at all).",
		Type:   TypeCounter,
		Labels: []string{"outcome"},
	})

	ModelRelayRefusals = Default.MustRegister(Definition{
		Name:   "cloop_model_relay_refusals_total",
		Help:   "Relayed model calls refused before they reached the API, by reason (budget_exhausted, model_not_allowed, path_not_allowed, method_not_allowed, body_too_large, body_malformed). Sustained budget_exhausted is a run working past its payer's daily budget; model_not_allowed a harness asking for a model the project's allowlist leaves out.",
		Type:   TypeCounter,
		Labels: []string{"reason"},
	})

	ModelRelayTokens = Default.MustRegister(Definition{
		Name:   "cloop_model_relay_tokens_total",
		Help:   "Tokens the API reported spending on relayed calls, by kind (input, output, cache_read, cache_write). What the relay charges each run's payer.",
		Type:   TypeCounter,
		Labels: []string{"kind"},
	})

	ModelRelayCostUSD = Default.MustRegister(Definition{
		Name: "cloop_model_relay_cost_usd_total",
		Help: "Estimated USD of relayed calls, priced from the API's counts with cache reads and writes at their own rates.",
		Type: TypeCounter,
	})

	ModelRelaySessionsLive = Default.MustRegister(Definition{
		Name: "cloop_model_relay_sessions_live",
		Help: "Model relay sessions this process serves.",
		Type: TypeGauge,
	})

	ModelRelayUsageGap = Default.MustRegister(Definition{
		Name:   "cloop_model_relay_usage_gap_tokens_total",
		Help:   "Tokens by which relayed runs' own reports differed from the relay's count at run end, by direction: unreported (the relay counted more than the run reported) or overreported. A claudecode run reports none, so its whole spend is unreported.",
		Type:   TypeCounter,
		Labels: []string{"direction"},
	})
)

The model relay for sandboxed runs (Task 20435): a run's Claude credential kept on the hub, its calls relayed and metered. Recorded by pkg/ui in the process holding each session, so every family is per process; the identity paying is never a label — what each identity spent is the quota gauges'.

View Source
var (
	QuotaEnforcementEnabled = Default.MustRegister(Definition{
		Name: "cloop_quota_enforcement_enabled",
		Help: "Whether a per-identity quota policy is in force. Exported by the cluster leader only.",
		Type: TypeGauge,
	})

	QuotaLimit = Default.MustRegister(Definition{
		Name:      "cloop_quota_limit",
		Help:      "Configured ceiling per identity and resource. Exported by the cluster leader only.",
		Type:      TypeGauge,
		Labels:    []string{"identity", "resource"},
		MaxSeries: quotaIdentityCeiling,
	})

	QuotaUsage = Default.MustRegister(Definition{
		Name:      "cloop_quota_usage",
		Help:      "Live consumption per identity and resource. Exported by the cluster leader only.",
		Type:      TypeGauge,
		Labels:    []string{"identity", "resource"},
		MaxSeries: quotaIdentityCeiling,
	})

	QuotaDenials = Default.MustRegister(Definition{
		Name:   "cloop_quota_denials_total",
		Help:   "Admission refusals since this hub started, by resource.",
		Type:   TypeCounter,
		Labels: []string{"resource"},
	})

	QuotaIdentities = Default.MustRegister(Definition{
		Name: "cloop_quota_identities",
		Help: "Identities the hub is currently accounting. Exported by the cluster leader only.",
		Type: TypeGauge,
	})

	ProjectsRegistered = Default.MustRegister(Definition{
		Name: "cloop_projects_registered",
		Help: "Projects in the hub registry. Exported by the cluster leader only.",
		Type: TypeGauge,
	})
)
View Source
var (
	AuditAppendFailures = Default.MustRegister(Definition{
		Name:   "cloop_audit_append_failures_total",
		Help:   "Audit events that could not be appended to a hash chain, by the role of the database handle and the failure class. Every one is later recorded in that chain as part of an audit.gap row, or reported on stderr if the process exits first.",
		Type:   TypeCounter,
		Labels: []string{"chain", "reason"},
	})

	AuditGapEvents = Default.MustRegister(Definition{
		Name:   "cloop_audit_gap_events_total",
		Help:   "Lost audit events that have been written into their chain as part of an audit.gap row, by the role of the database handle. Trails cloop_audit_append_failures_total by what is still unrecorded.",
		Type:   TypeCounter,
		Labels: []string{"chain"},
	})

	AuditCheckpoints = Default.MustRegister(Definition{
		Name:   "cloop_audit_checkpoints_total",
		Help:   "Audit head checkpoints the cluster leader wrote off the database, by outcome: signed under CLOOP_SECRET_KEY, unsigned because the hub has no key, or failed to reach the configured file.",
		Type:   TypeCounter,
		Labels: []string{"outcome"},
	})
)

The audit trail's own losses and its off-database checkpoints (Task 20404).

An append that fails leaves nothing in a hash chain, so statedb counts it on the way out of its append path, where no caller can skip the counting, and later writes an audit.gap row saying what was lost. The two counters are the two halves of that: events lost, and events since recorded as lost. Their difference is what this process still owes the trail. Counters, so each cluster member counts its own and sum() is the cluster's.

`chain` is the role of the database handle (AuditChain*) and `reason` the failure class (AuditFail*): both closed sets, and neither names a database, because a project path in a label is a series per project.

View Source
var (
	AuditForwardRows = Default.MustRegister(Definition{
		Name:   "cloop_audit_forward_rows_total",
		Help:   "Audit rows the forwarder handled, by sink and outcome: delivered (the sink acknowledged them), truncated (acknowledged with the payload left out to fit the sink's message size), failed (a send that failed; the rows are sent again), lost (rows the sink can no longer receive: pruned before it acknowledged them, or missing from the chain).",
		Type:   TypeCounter,
		Labels: []string{"sink", "outcome"},
	})

	AuditForwardLag = Default.MustRegister(Definition{
		Name:   "cloop_audit_forward_lag_seconds",
		Help:   "Age of the oldest audit row the sink has not acknowledged, across every chain the hub serves; 0 when it is caught up. Exported by the cluster leader only.",
		Type:   TypeGauge,
		Labels: []string{"sink"},
	})

	AuditForwardBacklog = Default.MustRegister(Definition{
		Name:   "cloop_audit_forward_backlog_rows",
		Help:   "Audit rows written but not yet acknowledged by the sink, across every chain the hub serves. The backlog is read from the database, never held in memory. Exported by the cluster leader only.",
		Type:   TypeGauge,
		Labels: []string{"sink"},
	})
)

Audit forwarding (Task 20426): the leader ships every chain's rows to the sinks in audit.forward, advancing a cursor per sink and chain only once the sink acknowledges.

`sink` is the one label in the catalog whose values are not declared in Go source, and it is closed anyway: the values are the names of the configured sinks, which the hub's own configuration declares — at most config.AuditForwardMaxSinks of them, each a short lower-case token — not anything a tenant or a request can choose. The counter can accumulate a series for a sink renamed or removed since the process started, which is bounded by how often an operator edits the file; the two gauges are collector-backed and Reset every scrape, so they name only the sinks in force. `outcome` is AuditForward*.

The gauges describe the shared database — the same rows, cursors and backlog whichever member reads them — so only the leader exports them.

View Source
var ApprovalNotifications = Default.MustRegister(Definition{
	Name:   "cloop_approval_notifications_total",
	Help:   "Approval notices handled per webhook, by event and outcome: delivered (the webhook acknowledged it), failed (one send that failed; the notice is retried), dropped (given up on after its retries, refused for good, or a day old), superseded (the request or token was settled before the notice went out, so it was not sent).",
	Type:   TypeCounter,
	Labels: []string{"sink", "event", "outcome"},
})

Approval notifications (Task 20427): the leader posts a notice to every webhook in notifications.webhooks when a grant request is filed, twelve hours from lapsing, decided, withdrawn or lapsed, and when an API token starts waiting for an administrator.

`sink` is the webhook's name, closed for the reason audit forwarding's is: the hub's own configuration declares the values, at most config.NotificationsMaxWebhooks of them, each a short lower-case token. `event` is config.NotifyEvent* (7 values), or `test` for the message Settings sends to check a webhook, and `outcome` ApprovalNotify*. Counters only, so every member may count what it sent: only the leader sends notices (a test goes from whichever member served it), and the sum over a cluster is the cluster's total.

View Source
var (
	CSPViolations = Default.MustRegister(Definition{
		Name:   "cloop_csp_violations_total",
		Help:   "Content-Security-Policy violations browsers reported, by effective directive (script-src-attr, script-src-elem, script-src, style-src-attr, style-src-elem, style-src, img-src, connect-src, frame-src, frame-ancestors, base-uri, form-action, object-src, default-src, worker-src, font-src, media-src, manifest-src, other) and by document (dashboard, glasses, sign-in, other). script-src-attr on the dashboard is an inline handler a conversion missed, or markup somebody injected: a button that does nothing, or an attack the policy stopped.",
		Type:   TypeCounter,
		Labels: []string{"directive", "document"},
	})
)

What a browser reports the hub's Content-Security-Policy blocked (Task 20408), from the same-origin report endpoint. directive is the effective directive, and document which of the hub's documents the page was — both mapped onto the closed sets in pkg/ui's csp.go, since a report is whatever its sender wrote and an unrecognised value counts as "other". Never labelled by the blocked URL or the source file, which a sender chooses.

View Source
var (
	CrossOriginRefusals = Default.MustRegister(Definition{
		Name:   "cloop_cross_origin_refusals_total",
		Help:   "Requests refused because a page on another origin made them, by reason (cross_site, same_site, unknown_fetch_site, foreign_origin, media_type, unknown_host). Any sustained rate is a page somebody opened trying to drive the hub, or a proxy that hides the hub's own origin from it.",
		Type:   TypeCounter,
		Labels: []string{"reason"},
	})
)

Requests a page on another origin made (Task 20394): a state-changing request or a WebSocket handshake the hub refused because the browser said it came from elsewhere, a form body a JSON API does not take, or — on a hub with no sign-in — a Host name DNS rebinding would use. The reasons are sameorigin.Reason's refusals, a closed set.

Never labelled by origin or host: those are chosen by whoever sends the request, so every one would be a new series. The origin is in the audit trail (request.origin_refused, request.host_refused), which can afford it.

View Source
var Default = New()

Default is the process-wide registry. A hub has exactly one.

A package-level registry is the shape every metrics library converges on for the same reason: the alternative is threading a *Registry through constructors in a dozen packages that otherwise have no reason to know metrics exist, and the first constructor that cannot take a new parameter gets a global anyway. The cost — that instrumentation is not injectable — is bounded here because the registry is the only global and tests can build their own with New.

View Source
var (
	DiskFreeBytes = Default.MustRegister(Definition{
		Name:      "cloop_hub_disk_free_bytes",
		Help:      "Free space this hub process could still write on each volume it writes to, by mount point: its own state volume and its projects'. Runs pause below orchestrator.min_free_disk_mb. Each cluster member reports its own.",
		Type:      TypeGauge,
		Labels:    []string{"volume"},
		MaxSeries: DiskVolumesMax,
	})
)

Disk (Task 20381). Per process, not per cluster: each member reports the volumes it writes to — the one holding its control-plane .cloop and those of the projects registered with it, deduplicated by device — so a member whose disk fills shows up as itself. Several members on one volume each report it; aggregate with max() or min() by volume, not sum().

`volume` is the mount point, a closed set at scrape time: the collector resets the family before every run, so its cardinality is the number of filesystems the hub writes to now, capped by DiskVolumesMax.

View Source
var (
	SessionRestores = Default.MustRegister(Definition{
		Name:   "cloop_proxy_session_restores_total",
		Help:   "Sessions and GitHub App token slots a hub process adopting a run, or receiving a CI job's call, tried to bring back, by kind (git, kube, egress, app_token, ci, model) and outcome (restored; refused — its grant or rule was revoked, expired or narrowed while no process held it; expired — it lapsed first; lost — another process restored it first; failed — its credential could not be re-derived for a reason that may pass, and it is retried).",
		Type:   TypeCounter,
		Labels: []string{"kind", "outcome"},
	})
)

What a hub process adopting a run brings back of what the run's lease feeds (Task 20383): its git proxy, Kubernetes monitor and egress sessions, and its GitHub App token slots — and, since Task 20390, the CI relay sessions a hub process restores for the job call that presents one. Per process, like the leases themselves: the member that restores counts its restores.

View Source
var (
	StateDBWALBytes = Default.MustRegister(Definition{
		Name: "cloop_statedb_wal_bytes",
		Help: "Size of the control-plane database's write-ahead log (.cloop/state.db-wal) in bytes. Read-write connections trim it to 64 MiB when they restart it and the leader truncates it while it is larger, so one that stays above that is held open by a reader or written only by builds that predate the limit. Exported by the cluster leader only.",
		Type: TypeGauge,
	})
)

The control-plane database (Task 20392). Its write-ahead log takes every write before a checkpoint copies it into the database, and SQLite never shrinks the file on its own: read-write connections trim it to 64 MiB when they start it over, and the leader's janitor truncates it after each retention pass and every minute while it is larger. One file in the shared database's directory, so exported by the cluster leader only.

Functions

func RegisterCollectors

func RegisterCollectors(specs ...CollectorSpec)

RegisterCollectors installs scrape-time collectors on the default registry, at most once per process and in the order given. Calls after the first are no-ops, so the hub registers every collector it has in one call.

Types

type Collector

type Collector func()

Collector recomputes derived gauges immediately before a scrape.

Some of what an operator needs is not a running tally but a reading of state that lives elsewhere: how many executors are in each health state, how deep the merge queue is, how many leases are live. Counting those at mutation time would mean every subsystem maintaining a shadow copy of its own state and keeping it consistent — a second source of truth, with the drift that implies. A collector reads the real one at scrape time instead.

type CollectorSpec added in v0.0.5

type CollectorSpec struct {
	Name     string
	Families []*Metric
	Collect  Collector
}

CollectorSpec is one scrape-time collector for the default registry and the gauges it owns (see Registry.RegisterCollector, which Resets them before every run).

Families is also how TestEveryFamilyIsRecorded knows a gauge is populated: a family named here counts as recorded even where no Set call names it.

type Definition

type Definition struct {
	Name   string
	Help   string
	Type   MetricType
	Labels []string
	// Buckets are the histogram upper bounds in ascending order, excluding
	// +Inf which is implied. Ignored for counters and gauges.
	Buckets []float64
	// MaxSeries overrides Registry.MaxSeriesPerMetric for this metric. Zero
	// takes the registry default.
	//
	// Raising it is only defensible for a collector-backed gauge, because
	// those Reset before every scrape: their cardinality is the number of
	// objects alive at scrape time rather than the number ever observed, so
	// churn cannot accumulate series the way it does on a counter. A raised
	// ceiling on a counter is just a slower memory leak.
	MaxSeries int
}

Definition declares a metric family.

type Metric

type Metric struct {
	// contains filtered or unexported fields
}

Metric is one metric family: a name, a type, a fixed label schema, and the series observed for it so far.

func (*Metric) Add

func (m *Metric) Add(delta float64, labelValues ...string)

Add adds delta to the counter or gauge identified by labelValues.

A nil Metric is a no-op so a subsystem can hold an optional metric without guarding every call site, and a negative delta on a counter is dropped rather than applied: a counter that can go down breaks rate() in ways that surface as impossible graphs long after the bug that caused them.

func (*Metric) Inc

func (m *Metric) Inc(labelValues ...string)

Inc adds one to the counter or gauge identified by labelValues.

func (*Metric) Observe

func (m *Metric) Observe(value float64, labelValues ...string)

Observe records one sample in the histogram identified by labelValues.

func (*Metric) Reset

func (m *Metric) Reset()

Reset drops every series in the metric.

Collector-backed gauges call this before repopulating, so a series whose underlying object is gone (a deregistered executor, a closed session) stops being exported rather than freezing at its last value. A frozen gauge is worse than a missing one: it reads as a live measurement and will hold an alert open forever.

func (*Metric) SeriesCount

func (m *Metric) SeriesCount() int

SeriesCount reports how many distinct label combinations a metric holds, including the overflow series. Used by tests that assert the ceiling holds.

func (*Metric) Set

func (m *Metric) Set(value float64, labelValues ...string)

Set assigns an absolute value to the gauge identified by labelValues.

type MetricType

type MetricType string

MetricType is the Prometheus exposition type of a metric family.

const (
	TypeCounter   MetricType = "counter"
	TypeGauge     MetricType = "gauge"
	TypeHistogram MetricType = "histogram"
)

type Registry

type Registry struct {
	// MaxSeriesPerMetric is the per-metric cardinality ceiling. Read
	// without synchronisation, so set it before the registry is shared.
	MaxSeriesPerMetric int
	// contains filtered or unexported fields
}

Registry holds every metric family in the process and renders them in the Prometheus text exposition format. The zero value is not usable; call New.

func New

func New() *Registry

New returns an empty registry.

func (*Registry) CollectorFamilies added in v0.0.5

func (r *Registry) CollectorFamilies() map[string]string

CollectorFamilies reports which collector owns each collector-backed family, keyed by family name.

func (*Registry) Gather

func (r *Registry) Gather() string

Gather renders the registry to a string.

func (*Registry) MustRegister

func (r *Registry) MustRegister(def Definition) *Metric

MustRegister declares a metric family and returns it, panicking on a malformed definition or a duplicate name.

Panicking is right here and would be wrong almost anywhere else in cloop: the catalog is built at package initialisation from string literals, so a panic can only fire on a programming error that every single startup would hit. Returning an error would mean every call site handling a failure that cannot occur at runtime.

func (*Registry) Register

func (r *Registry) Register(def Definition) (*Metric, error)

Register declares a metric family.

func (*Registry) RegisterCollector

func (r *Registry) RegisterCollector(name string, fn Collector, families ...*Metric)

RegisterCollector adds a function run before each scrape. name appears in the panic-recovery path so a broken collector is identifiable.

families are the gauges fn populates, and the collector owns them: each is Reset immediately before fn runs. That is what keeps a collector-backed gauge honest without every collector having to remember it — a series whose object is gone stops being exported instead of freezing at its last value, and a collector that decides to export nothing this scrape (a cluster member that is not the leader, declining a gauge the leader reports) exports nothing rather than whatever it wrote last time it led.

Only a gauge can be owned. A counter Reset every scrape is one that rate() reads as a spike that never happened, so owning one panics — like MustRegister, this is called at startup with families named in source, and the mistake would fire on every start.

func (*Registry) TotalSeries

func (r *Registry) TotalSeries() int

TotalSeries reports the registry's whole footprint in series.

func (*Registry) WriteTo

func (r *Registry) WriteTo(w io.Writer) (int64, error)

WriteTo renders the registry in the Prometheus text exposition format (version 0.0.4), running collectors first.

Families are emitted in registration order and series within a family in sorted label order, so a scrape is byte-stable when nothing changed. That makes diffs between scrapes meaningful and keeps the golden-file tests from depending on Go's map iteration order.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL