Documentation
¶
Overview ¶
Package kvmetrics samples a self-hosted server's KV-cache statistics from its Prometheus endpoint (docs/design.md §8): prefix-cache queries and hits, the hit rate, and KV-cache utilization.
It is opt-in, and it provides facts, not policy: the library reads the server, and each product decides when to sample (mast at session start and end, core-agent per /stats). A library embed with no metrics endpoint never builds one.
What it reports is FLEET-LEVEL and TIME-CORRELATED. A server's counters cover every request it served in the window, from every client, so a Window says what the server did while a session ran, not what the session did. It is never attributed to a request or a specialist, and never enters usage.Detail or a cost figure.
Every count is a pointer, as in usage.Detail (AGENTS.md rule 11): a metric the server does not expose is nil, "not reported", never 0.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrNoMetrics = errors.New("kvmetrics: this server exposes no KV-cache statistics")
ErrNoMetrics is returned for a server that exposes no KV-cache statistics (Ollama: profile.MetricsNone).
var ErrStopped = errors.New("kvmetrics: Stop called twice")
ErrStopped is returned by a second Stop.
Functions ¶
This section is empty.
Types ¶
type Options ¶
type Options struct {
// URL is the Prometheus endpoint. FromProfile takes it from the
// profile.
URL string
// Format names the server's metric vocabulary. FromProfile takes it
// from the profile.
Format profile.MetricsFormat
// Model, when set, reads only series labelled with that model
// (vLLM's and SGLang's model_name). Empty sums every model the server
// serves. A model the endpoint has no series for is an error, not an
// empty reading.
Model string
// HTTPClient sends the scrape, as given: FromProfile attaches no
// credential to a caller's client. Nil means a client with a
// 10-second timeout.
HTTPClient *http.Client
// Now replaces time.Now, for tests.
Now func() time.Time
}
Options configures a Sampler.
type Reading ¶
type Reading struct {
// At is when the snapshot was taken.
At time.Time
// Queries and Hits are the server's cumulative prefix-cache counters,
// in tokens, summed across series: tokens looked up in the cache, and
// tokens found there. Nil when the server does not count them (an
// SGLang build without the token counters, llama.cpp).
Queries *float64
Hits *float64
// HitRate is the hit rate the server computes itself (SGLang's
// cache_hit_rate gauge), averaged across its series, for servers
// without hit/query counters. Instantaneous, not a window's rate.
HitRate *float64
// KVUsage is the fraction of the KV cache in use, 0–1: the largest
// across engines, since a hot engine is the one that matters.
KVUsage *float64
// contains filtered or unexported fields
}
Reading is one snapshot of a server's metrics, summed over its engines (and over its models, unless Options.Model narrows it).
type Run ¶
type Run struct {
// contains filtered or unexported fields
}
Run is a window in progress, from Start to Stop. It keeps a running fold of its readings, not the readings, so its memory does not grow with the session.
func (*Run) PollErr ¶
PollErr is the most recent failed poll, or nil. Polls that fail are skipped; this is for a product that wants to log them.
func (*Run) Stop ¶
Stop ends the polling, takes the closing reading, and returns the window. If the closing scrape fails, Stop still returns the window up to the last successful reading, with EndFailed set, together with the scrape's error: the window is usable whenever it is non-zero. A second Stop returns ErrStopped without scraping.
type Sampler ¶
type Sampler struct {
// contains filtered or unexported fields
}
Sampler reads one server's metrics endpoint. Safe for concurrent use.
func FromProfile ¶
FromProfile builds a Sampler for a resolved profile's metrics_url and metrics format. opts supplies the rest (Model, HTTPClient, Now); its URL and Format are ignored.
The profile's credential is attached only when metrics_url has the same origin (scheme, host and port) as the profile's base_url, and a Google ADC token only over https: a metrics endpoint on another host gets no token meant for the API. It is sent the way package auth sends it to the API (BearerTransport: "Authorization: Bearer", for an API key too). An endpoint that needs a different credential takes a caller's opts.HTTPClient, to which nothing is added.
func New ¶
New builds a Sampler. It fails for profile.MetricsNone (ErrNoMetrics) and for an unknown or empty Format: a sampler that silently reported nothing would look like a server with no cache activity.
func (*Sampler) Start ¶
Start takes the opening reading and, when interval > 0, polls the endpoint every interval until Stop, so the window's peak KV utilization and any counter reset between the endpoints are seen. A failed poll is skipped, not fatal: one missed scrape should not void the window. The polling goroutine ends when Stop is called or ctx is done.
Start/Stop rather than two Snapshot calls because a peak needs the readings between the two points, and only a running poller has them. A product that schedules its own scrapes can call Snapshot and Between instead.
type Window ¶
type Window struct {
// Fleet is always true: this is the whole server's activity in the
// window, from every client, not one session's. The field is here so
// the fact travels with the numbers into a log or a report.
Fleet bool
// Start and End are the first and last readings' times.
Start, End time.Time
// Queries and Hits are the counters' increase over the window, in
// tokens, computed per series and summed. Nil when fewer than two
// readings carried the counter.
Queries *float64
Hits *float64
// HitRate is Hits/Queries over the window, when both are reported and
// Queries > 0. Nil otherwise: it is never filled from a gauge.
HitRate *float64
// HitRateGauge is the server's own hit-rate gauge at the last reading
// that had one (Reading.HitRate): instantaneous, for servers with no
// counters. Kept apart from HitRate so a consumer cannot mistake one
// for the other.
HitRateGauge *float64
// PeakKVUsage is the highest KV utilization seen at Start, End, or
// any poll between them. Without polling it is the higher of the two
// endpoints, which can miss a spike between them.
PeakKVUsage *float64
// Readings is how many snapshots the window was computed from.
Readings int
// Reset reports that some series' counter went backwards between two
// readings: that engine restarted. Its increase is then summed per
// segment, the way Prometheus's increase() does, and undercounts what
// it served between its last reading before the restart and the
// restart itself.
Reset bool
// EndFailed reports that the closing scrape failed, so the window
// ends at the last successful poll (End is that poll's time).
EndFailed bool
}
Window is what a server's KV cache did between two readings.