metrics

package
v1.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 13, 2026 License: GPL-2.0, GPL-3.0 Imports: 5 Imported by: 0

Documentation

Overview

Package metrics defines knell's Prometheus metrics and the registry that serves them. Metrics are package-level singletons registered once at init (registration only panics on programmer error: a duplicate or invalid name); the exposition prefix is "knell_".

The registry and the collectors are unexported: the package's edge is Handler plus the knell-semantic recording functions below (InitBeat, RecordBeat, SetBeatFresh, RecordOutage, RecordOutageRecordDropped, RecordHTTP, RecordPreRouteRefusal and the three RecordNotification* functions). Callers therefore cannot register, rename or delete a series, nor write a raw label position, which is what keeps the metric names, label sets and cold-start zero samples the quorum and delivery alerts read in one place.

Label-cardinality contract (read this before adding a caller)

Label CARDINALITY is not structurally contained, and cannot be: Prometheus creates the labelled child on first use, so InitBeat, RecordBeat, SetBeatFresh, RecordOutage and RecordOutageRecordDropped mint a per-beat series for whatever id they are handed and this package has nothing to reject it with. A series, once minted, is permanent for the process lifetime — in knell and in every observer scraping it.

Keeping the beat label bounded is therefore the CALLER's contract, enforced by exactly three things upstream of here:

  • internal/config validates every id against [A-Za-z0-9][A-Za-z0-9_-]{0,63} and caps a fleet at 64 beats;
  • internal/watch passes only ids from that configured set (New's beat map is the whole domain of every call it makes);
  • internal/webapi does not constrain the id's CONTENT: it hands the raw /beat/{id} path segment straight to watch.Beat and answers 404 from the BeatUnknown outcome that call returns. So the w.beats[id] lookup inside watch.Beat is the only check on the id itself, and a segment reaches it only from a caller that passed beatHandler's BEAT_TOKEN gate (internal/watch/watch_metrics_test.go pins that an unconfigured id mints no series).

The only PRODUCTION caller of the five id-taking functions is internal/watch; test code in this module may also call them, with fixed literal ids only. If you are adding any other production call, you have just inherited that contract: pass only an id internal/config validated and capped, or route through internal/watch, or validate the id yourself before the call. A raw id from a request path, a filename, a log line or a remote payload makes Mimir's label set grow without bound, and the growth is not recoverable by fixing the caller afterwards.

RecordHTTP is one of the two recording functions reached from internal/webapi in PRODUCTION (RecordPreRouteRefusal below is the other), and it is deliberately not id-taking: it carries no beat label, so it is outside the contract above and adding it did not widen the set of packages allowed to mint a per-beat series. It has its own bounded-label contract instead, and the same reasoning drives it — its method and path labels must be bounded by the ROUTE TABLE and a closed method set, never by the request line, because /beat/{id} is served to unauthenticated callers (webhttp.Logging wraps the whole mux, so its metric hook fires before beatHandler's BEAT_TOKEN gate; only SecurityHeaders, NoStore and the failed-auth throttle sit outside it, which is why the throttle's 429 reaches this counter not at all — see preRouteRefusals). The derivation lives in webhttp (RouteMetricLabels): internal/webapi.New passes this function straight to webhttp.WithRecordRouteMetric, so the library computes the pair and knell has none of its own to get wrong.

RecordPreRouteRefusal is reached from internal/webapi too, and its label is narrowed by its argument TYPE and closed by the caller contract below: Refusal is not string, so no request-derived value reaches the label without an explicit Refusal(...) conversion written at the call site. The set is still not closed to the compiler — that conversion, and any untyped literal, remain legal — so the rule is the same as Kind's: pass one of the constants declared in this file, never a converted runtime string. That matters more here than anywhere else in this package, because every one of its call sites sits on an unauthenticated path.

Runtime enforcement inside this package is deliberately NOT the answer: it would add state plus an init-order dependency on config to defend against a caller that does not exist. This comment is the guard instead — a reviewer's (or an agent's) tripwire, not the compiler's.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Handler added in v1.1.0

func Handler() http.Handler

Handler serves the Prometheus exposition: every registered metric plus process metrics. It is the only exported view of the registry, so a consumer can scrape the metrics but cannot mint, delete or re-register a series.

func InitBeat added in v1.1.0

func InitBeat(id string, deadline time.Duration, start time.Time)

InitBeat declares a configured beat at process start with start as its last-seen baseline (the boot-armed clock every first deadline counts from), and pre-mints its per-beat counters at zero for the same reason mintNotificationKinds pre-mints the notification counters — increase() needs an earlier sample to diff against. deadline is published as beat_deadline_seconds, the one per-beat series that is configuration rather than state: this is the only place it is written, and it is what lets an operator compute when a beat goes overdue (last-seen plus the deadline) and see whether the observers one quorum rule aggregates agree on it. The beat's freshness verdict is NOT published here: the caller publishes it through SetBeatFresh, the single door for that gauge. Only configured ids reach here, which is what keeps the beat label bounded (see the package doc's label-cardinality contract).

func RecordBeat added in v1.1.0

func RecordBeat(id string, now time.Time)

RecordBeat records an accepted ping for id observed at now: the ping is counted and its new last-seen timestamp published. The resulting freshness verdict goes through SetBeatFresh, so this never decides freshness itself. The caller publishes under its own lock so concurrent pings cannot write the gauges out of state order. id is a label: see the package doc's label-cardinality contract.

func RecordHTTP added in v1.1.0

func RecordHTTP(method, path string, status int, d time.Duration)

RecordHTTP records one served HTTP request: its latency, and one increment on the method/path/status combination. method and path must be bounded by the route table and a closed method vocabulary, not taken off the request line — see the package doc's RecordHTTP contract. Its only production caller is webhttp's WithRecordRouteMetric hook, wired in internal/webapi.New: the library derives the pair (webhttp.RouteMetricLabels) and calls this function with it. Unlike the beat counters this series is not pre-minted: an HTTP counter has no known-in-advance status set, so a refusal series (401 after a token rotation, 404 for a misspelled id) is BORN with the first refused ping. increase() cannot see that birth — a series whose first sample is already 1 has no earlier sample to diff against, so a lone refusal never satisfies increase(...) > 0. Alert on the ABSOLUTE value instead (knell_http_requests_total{status="401"} > 0, which simply does not fire while the series is absent). That expression latches: a counter never returns to zero, so it keeps firing until knell restarts - read it as "this has happened at least once on this process", not as a windowed rate like the README's increase(...[15m]) rules. increase() is only meaningful on a status that is already being served.

func RecordNotificationDropped added in v1.1.0

func RecordNotificationDropped(kind Kind)

RecordNotificationDropped counts one notification message of kind that will never be delivered: a fire-once recovered notice whose send failed and left no record behind. Nothing retries it. A discarded outage RECORD is not a message and goes to RecordOutageRecordDropped instead.

func RecordNotificationFailed added in v1.1.0

func RecordNotificationFailed(kind Kind)

RecordNotificationFailed counts one notification message of kind whose delivery was attempted and failed after retries. Its record survives, so the send retries: this is never the counter for something that was dropped unsent.

func RecordNotificationSent added in v1.1.0

func RecordNotificationSent(kind Kind)

RecordNotificationSent counts one delivered notification message of kind.

func RecordOutage added in v1.1.0

func RecordOutage(id string)

RecordOutage counts one detected outage for id, at detection time and independent of whether its notice is ever delivered. It counts OUTAGES rather than messages. id is a label: see the package doc's label-cardinality contract.

func RecordOutageRecordDropped added in v1.1.0

func RecordOutageRecordDropped(id string)

RecordOutageRecordDropped counts one ended-outage RECORD for id discarded for good because the per-beat queue was full: that record was the outage's last trace, so no notice for it will ever arrive. It counts RECORDS, not messages, which is why it is not a notification counter (see outageRecordsDropped). The ongoing-outage overflow, which loses nothing, must not come here. id is a label: see the package doc's label-cardinality contract.

func RecordPreRouteRefusal added in v1.1.0

func RecordPreRouteRefusal(reason Refusal)

RecordPreRouteRefusal counts one request refused before the mux routed it, under the reason knell refused it. The label is narrowed by the Refusal type and no caller converts a request-derived value into it (see the package doc). Its callers are internal/webapi's pre-route guards: the non-canonical /beat path 404 and the ALLOWED_HOSTS 403, which collapse onto http_requests_total's "unmatched" path label beside scanner traffic, and the failed-auth throttle's 429, which is answered outside webhttp.Logging and is therefore on no other series at all. Read it while investigating a missing beat; it is deliberately not an alert source (see preRouteRefusals).

func SetBeatFresh added in v1.1.0

func SetBeatFresh(id string, fresh bool)

SetBeatFresh publishes the quorum gauge for id: fresh when the last ping is within the beat's deadline, overdue otherwise. The freshness boundary itself belongs to the watch state machine; this is only the exposition of its verdict. id is a label: see the package doc's label-cardinality contract.

Types

type Kind added in v1.1.0

type Kind string

Kind distinguishes notification label values from runtime strings. String variables require an explicit conversion before reaching the recording functions. It is not a closed set: untyped literals remain assignable, so callers must use the constants below and keep notificationKinds in sync.

const (
	KindMissing   Kind = "missing"
	KindRecovered Kind = "recovered"
	KindHistory   Kind = "history"
)

The notification kinds are the legal values of kindLabel on the sent, failed and dropped notification counters; dashboards and the KnellNotifyFailing alert key on them. They count MESSAGES, so KindHistory moves by one for a notice covering any number of ended outages; beatOutages counts outages.

type Refusal added in v1.1.0

type Refusal string

Refusal distinguishes pre-route refusal reason values from runtime strings, the same way Kind does for the notification counters. The label it feeds is NARROWED by this type, not closed by it: no plain string reaches the label without an explicit Refusal(...) conversion written at the call site, and that conversion stays legal. It is not a closed set to the compiler — untyped literals remain assignable — so callers must use the constants and keep refusalReasons in sync.

const (
	// RefusalNonCanonicalBeatPath is a /beat spelling net/http would rewrite
	// before routing (repeated slashes, a dot segment), refused 404 by
	// internal/webapi's canonicalBeatPath. It means a sender is pinging a
	// malformed URL.
	RefusalNonCanonicalBeatPath Refusal = "non_canonical_beat_path"
	// RefusalHostNotAllowed is a request whose Host is not in ALLOWED_HOSTS,
	// refused 403 by the webhttp host policy. It means either a DNS-rebinding
	// attempt or a hostname the deployment forgot to list.
	RefusalHostNotAllowed Refusal = "host_not_allowed"
	// RefusalAuthThrottled is a beat request refused 429 by the failed-auth
	// throttle because the shared bucket was empty. It means failed
	// authentication is arriving faster than the budget allows — a rotated
	// token on a whole sender fleet, or a guessing run.
	RefusalAuthThrottled Refusal = "auth_throttled"
)

The refusal reasons are the legal values of reasonLabel on the pre-route refusal counter. Each names the CAUSE rather than the status code knell answers, because the status is what an operator already sees on http_requests_total and the cause is what this counter exists to add.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL