metrics

package
v0.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 1, 2026 License: Apache-2.0 Imports: 4 Imported by: 0

Documentation

Overview

Package metrics owns the Prometheus registry and the /metrics HTTP handler. Consumers declare their own metrics (typically in their package's metrics.go), register them via Register, and use prometheus.Counter/Gauge/Histogram normally.

Conventions

Every metric must set Namespace to metrics.Namespace ("relay") and pick a Subsystem matching the package name (e.g. "eventlog", "auth", "usage").

Naming:

  • Counters end in _total (prometheus convention; client_golang enforces this)
  • Gauges describe a current value (e.g. queue_depth)
  • Histograms end in _seconds or _bytes

Labels: use sparingly. High-cardinality labels (e.g. per-request IDs) explode memory. Prefer separate counters over a label with O(N) values.

Index

Constants

View Source
const Namespace = "relay"

Namespace is the shared Prometheus namespace for every Relay metric. Concrete metrics SHOULD set this as their Namespace and pick a Subsystem matching their package (e.g. "eventlog", "auth", "usage").

Variables

View Source
var (
	// RecordsLost counts background records dropped because a bounded
	// emitter queue was full (usage rows, payload captures). The async
	// contract mandates this counter: a drop you can't see is a billing
	// or audit hole. `kind` is the emitter: "usage" | "payload".
	RecordsLost = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Namespace: Namespace,
			Name:      "records_lost_total",
			Help:      "Background records dropped due to a full emitter queue, by kind (usage/payload).",
		},
		[]string{"kind"},
	)

	// ProviderKeysDown counts the times a pooled provider key was placed
	// into cooldown by a failure — the leading indicator of heading
	// toward "no healthy keys in pool". `reason` is the failure class
	// (auth/rate_limit/server_error/network/local_rl).
	//
	// A counter, not a gauge: breaker state lives in shared kv, so a
	// per-pod gauge of "keys down right now" would be inconsistent across
	// the fleet. Trip counts sum cleanly. A faithful global gauge is
	// deferred (would need a kv scan — not lightweight).
	ProviderKeysDown = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Namespace: Namespace,
			Name:      "provider_keys_down_total",
			Help:      "Times a pooled provider key was put into cooldown by a failure, by reason.",
		},
		[]string{"reason"},
	)
)

Health signals: the two "am I about to have a problem" metrics — silent data loss (Q3) and provider keys dying (Q4). Both are one-liner emitters the owning packages call at the relevant moment; nothing here reaches back into them.

View Source
var (
	// RequestsTotal answers "how much traffic, how many errors" (Q1/Q2).
	// `status` is a bounded class (2xx/4xx/5xx), never the raw code.
	RequestsTotal = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Namespace: Namespace,
			Name:      "requests_total",
			Help:      "Total requests handled, by source runner and status class (2xx/4xx/5xx).",
		},
		[]string{"source", "status"},
	)

	// RequestSeconds is total end-to-end time (Q2 — the "whose time is
	// it" numerator). Wide buckets span cached/error responses through
	// long generation runs.
	RequestSeconds = prometheus.NewHistogramVec(
		prometheus.HistogramOpts{
			Namespace: Namespace,
			Name:      "request_seconds",
			Help:      "End-to-end request latency (handler entry to response closed).",
			Buckets: []float64{
				0.001, 0.005, 0.01, 0.05, 0.1,
				0.25, 0.5, 1, 2, 5, 10, 30, 60, 120,
			},
		},
		[]string{"source"},
	)

	// OverheadSeconds is THE wedge metric (Q1): Relay's own time, total
	// minus the upstream call. The performance contract puts a hard SLO
	// here (p99 < 10ms live). Tight sub-second buckets: 100µs → 500ms.
	OverheadSeconds = prometheus.NewHistogramVec(
		prometheus.HistogramOpts{
			Namespace: Namespace,
			Name:      "overhead_seconds",
			Help:      "Relay overhead = total handler time minus the upstream call. Observed only when upstream was reached.",
			Buckets:   []float64{0.0001, 0.0002, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.1, 0.25, 0.5},
		},
		[]string{"source"},
	)

	// InflightRequests answers "how many streams are open right now /
	// where does concurrency plateau" (Q5). A streamed request counts
	// until its body closes (post-flight fires at Close).
	InflightRequests = prometheus.NewGaugeVec(
		prometheus.GaugeOpts{
			Namespace: Namespace,
			Name:      "inflight_requests",
			Help:      "Currently-open requests, by source runner. Streams count until body close.",
		},
		[]string{"source"},
	)

	// AdmissionSeconds is Timing.Start → upstream handoff: auth +
	// rate-limit reserve + key selection — the Redis-contention proxy
	// (Q5). Skipped when the request never reached upstream. Same
	// sub-second bucket style as OverheadSeconds: 100µs → 250ms.
	AdmissionSeconds = prometheus.NewHistogramVec(
		prometheus.HistogramOpts{
			Namespace: Namespace,
			Name:      "admission_seconds",
			Help:      "Time from request accept to upstream handoff (auth + rate-limit reserve + key selection). Observed only when upstream was reached.",
			Buckets:   []float64{0.0001, 0.0002, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.1, 0.25},
		},
		[]string{"source"},
	)

	// PostFlightSeconds is the duration of one full post-flight fan-out
	// (Registry.Finalize): is post-flight keeping up with stream closes,
	// or building a goroutine backlog (Q5)?
	PostFlightSeconds = prometheus.NewHistogram(
		prometheus.HistogramOpts{
			Namespace: Namespace,
			Name:      "post_flight_seconds",
			Help:      "Duration of the post-flight observer fan-out (hooks + collectors) per request.",
			Buckets:   []float64{0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5},
		},
	)
)

The request-flow metrics (Rate / Errors / Duration) plus the Relay-vs- upstream split.

Labelled by `source` (the runner: pipeline/proxy/ws/batch) — the lowest-cardinality dimension already on the lifecycle Context. Wire `shape` is deliberately NOT a label: it isn't carried on the Context and plumbing it would touch the inference handler. Add it the day the question "openai vs anthropic traffic split" is actually asked.

Functions

func Handler

func Handler() http.Handler

Handler returns the http.Handler serving /metrics in Prometheus text format.

func InflightDec added in v0.3.0

func InflightDec(source string)

InflightDec marks a request finalized. Callers must pair with a prior InflightInc for the same request or the gauge goes negative.

func InflightInc added in v0.3.0

func InflightInc(source string)

InflightInc marks a request admitted. Hot path: O(1), allocation-free.

func ProviderKeyDown

func ProviderKeyDown(reason string)

ProviderKeyDown is the one-liner keypool calls when a key transitions into cooldown.

func RecordAdmission added in v0.3.0

func RecordAdmission(source string, admission time.Duration)

RecordAdmission is the one-liner the post-flight metrics observer calls when the request reached upstream (Timing.Upstream.Start was stamped).

func RecordLost

func RecordLost(kind string)

RecordLost is the one-liner an emitter calls at its drop site.

func RecordPostFlight added in v0.3.0

func RecordPostFlight(d time.Duration)

RecordPostFlight is the Finalize-duration observer wired onto lifecycle.Registry at boot (the Registry stays metrics-free).

func RecordServed

func RecordServed(source string, status int, total, upstream time.Duration)

RecordServed is the one-liner the post-flight metrics observer calls once per finalized request. total is the full handler time; upstream is the upstream-call duration (zero when upstream wasn't reached, in which case the overhead split is not observed — it'd be meaningless).

func Register

func Register(collectors ...prometheus.Collector)

Register adds collectors to the shared registry. Panics on duplicate registration (consistent with prometheus.MustRegister).

func RegisterQueueDepth added in v0.3.0

func RegisterQueueDepth(kind string, fn func() float64)

RegisterQueueDepth exposes a bounded emitter's queue depth as relay_emit_queue_depth{kind=...} — the leading signal for the drops RecordsLost counts after the fact. Same kinds as records_lost_total. fn is sampled at scrape time (chan len is concurrency-safe). Call once per kind at boot, where the emitter is constructed.

func SafeLabel

func SafeLabel(s string) string

SafeLabel returns "unknown" if s is empty, preventing empty-string Prometheus labels.

func StatusClass

func StatusClass(code int) string

StatusClass returns a bounded status label ("2xx", "4xx", "5xx", etc.) for an HTTP status code. Using a class avoids high-cardinality label explosion.

Types

This section is empty.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL