stats

pkg/stats

Per-item resource statistics: 15 minutes at 5 s in memory, 7 days of minutes on disk.

Format and limits

One fixed-size ring file per item, <hash of the item>.ring. A query never waits for an item's writer: it reads the ring file without the writer's lock. It does take the store read lock for the lookup, so it can wait behind the creation of an item (two fsyncs under the store lock), Forget and Sweep, which hold that lock.

• Layout: an 8192-byte header area (two header copies, format version 1, item name, series keys), then columns of Minutes (10 080) cells. Column 0 holds the minute stamps (4 bytes each); every series adds three columns (avg, min, max; float32). A cell holds no value when its stamp does not match the minute.
• Size is a pure function of the series count: 8192 + 40320 x (1 + 3 x series) bytes. It never changes while the ring wraps.
• Bounds: 48 series per item (a typical container with 24 series is 2 951 552 bytes, about 2.8 MiB; the maximum is 5 854 592 bytes, 5.6 MiB), series key at most 63 bytes, item name at most 256 bytes, 256 live items plus at most 256 forgotten items.
• Worst case for the directory: (256 live + 256 forgotten) x 5.6 MiB = 2.8 GiB. Realistic case: 41 items of 24 series, about 115 MiB.
• File handles: each ring keeps one open handle for the life of the store (up to 512 with the forgotten items); the service's LimitNOFILE (or ulimit) must allow it.
• Memory: 180 live slots per series, no growth with time.
• Modes: the directory is created 0750 and every file the package writes (*.ring, alerts.json, their .tmp and .bad) 0640, since they hold container, VM and interface names. Open narrows an existing directory and existing *.ring files to these modes (best effort, an error is ignored); NewEvaluator does the same for its directory and an existing alerts.json. Missing parent directories are created 0755. A symlink is never followed for this (the directory or a file that is a symlink keeps its target's mode). A *.ring.bad or alerts.json.bad made before this version keeps its old mode; the 0750 directory covers it. Windows ignores modes.
• *.ring.bad (a quarantined file) is kept once per item; alerts.json only holds items that differ from defaults or are active, and is pruned with the store.

Series keys and partial errors

A ring never frees a series slot. Once an item has seen 48 distinct keys, every new key is rejected for the life of the item, so key suffixes MUST be stable: a MAC address or an index, never a name that changes at each start (libvirt vnetN, host veth names).

Record returns an error wrapping ErrSeriesCap or ErrBadKey when some keys of a sample were rejected; the other keys were stored. This is PARTIAL, not a failure: do not retry and do not drop the sample. stats.IsPartial(err) is true exactly for such errors (and false if a real failure, such as a write error, is joined to them).

Other behaviour: negative values are stored as given (services clamp what must not be negative); item names must be valid UTF-8 (ErrBadItem otherwise, from Record and SetRules); Sweep also removes live items silent for 7 days, and Prune then drops their rules.

How to wire

st, _ := stats.Open(dir, stats.SystemClock{})
ev, _ := stats.NewEvaluator(st, dir, emit) // emit(Event) should not block; it may call the Evaluator
st.SetOnMinute(ev.OnMinute)                // the only wiring; services never call OnMinute
// at start, per item that has history:   ev.Rebuild(item)
// every 5 s, per item:                   st.Record(item, now, sample)
// item stops being measured:             ev.Resolve(item)
// item removed:                          ev.Resolve(item); st.Forget(item, now)
// hourly:                                st.Sweep(now); ev.Prune()

Notes:

• Use one sampling goroutine per item. Record calls for one item from two goroutines are safe for the data, but the OnMinute hooks they trigger can run out of order.
• A Record that crosses a minute boundary commits the previous minute at the cost of two fsyncs per item; see Cost before sampling many items from one goroutine.
• Prune drops the rules and state of every item no longer in the store, including an item that has rules but never recorded.
• Events are emitted in the order of the state changes that made them, across items, outside every lock. Each state change queues its events; the call that finds no one emitting becomes the drainer and emits the queue until it is empty, so one goroutine (a Record, a route's SetRules) may emit events made by other goroutines. A slow emit delays the later events only, never a state change, a read or another call, which queues and returns. emit may call back into the Evaluator (Resolve, SetRules, Rebuild): its events are queued and emitted after the current emit returns. If emit panics, the panic reaches the drainer's caller, the event it was given counts as emitted, the rest go back to the front of the queue and the next call that changes or evaluates state (SetRules, Rebuild, Resolve, OnMinute; not Prune, which changes state but does not drain) emits them.
• Resolve, SetRules, Rebuild and Record/OnMinute can return BEFORE their own events have been emitted: when another goroutine is draining the queue, or after a panic in emit until a later call drains the leftovers. There is no flush at shutdown: a caller that does ev.Resolve(item); st.Forget(item) and then exits can lose the resolved event. A caller that needs its events delivered waits for them itself (for example in its emit).
• Close commits the pending minute of every item before it closes the rings: the samples of the minute in progress are written as a partial minute (avg, min and max of the samples held, alert series included) and the OnMinute hook is called for it, outside the store lock, after the hooks of Record calls in flight. A restart within the next minute therefore loses no minute and an alert run keeps counting; a downtime that covers one or more whole minutes is still a gap and restarts the run. An item with no pending sample writes nothing. No hook runs after Close returns. A hook (so emit) must NOT call Close, nor wait for a goroutine that is in Close: Close waits for the hooks in flight, so it deadlocks. Close costs about two fsyncs per item with a pending minute, one item after the other (like a rollover tick). A restart inside the SAME wall minute: the next commit of that minute overwrites its slot with the samples taken after the restart only, replacing the partial written at Close; the evaluator judged that minute on the partial and ignores the rewrite (same minute as its Last), so that minute's history can differ from what the evaluator saw.
• Evaluator.LastError() returns the error of the last write of alerts.json (nil once a later write succeeds); the service can log or expose it.

Extensions

Beyond the base contract: SetOnMinute (the single hook), Evaluator.Resolve, Evaluator.Prune, Evaluator.LastError, SystemClock and FakeClock, Store.Quarantined and Store.Skipped, ErrRingSkipped, the exported errors (ErrUnknownItem, ErrBadRange, ErrBadItem, ErrBadKey, ErrSeriesCap, ErrTooManyItems, ErrOutOfOrder, ErrClosed, *RuleError), the exported constants (Minutes, LiveSlots, MaxSeries, MaxItems, MaxForgotten, MaxKeyLen, MaxItemLen, FormatVersion, MaxFileSize) and SplitKey, IsPartial. At Open, a ring file that cannot be read for a reason other than corruption (read-only, dangling symlink, symlink to a directory, I/O error) is left in place and listed by Skipped(); Open carries on with the other items, and a Record of a skipped item returns ErrRingSkipped (new exported error, a full failure for that item whose text names the file; never partial) instead of replacing the file, so a temporary EACCES/EIO cannot destroy its history; the next Open that can read the file picks it up. A corrupt file that could not be moved to *.bad is also listed by Skipped() but is replaced by the next Record. A quarantined *.ring.bad gets the time of the quarantine as its mtime, so it is kept 7 days from then. Query results: series of chart alert are never returned, a series with no data in the window is omitted, live points have Min = Max = Avg, floats are stored as float32.

Added in v0.4.29:

• Store.Unreadable(item) is true exactly when a Record of item returns ErrRingSkipped (its ring file was left in place at Open as unreadable). It is false for a healthy or unknown item and for a corrupt file Open could not move aside, which Record replaces. A service needs no copy of the ring file naming rule.

• File modes 0750 (directory) and 0640 (files), existing ones narrowed at Open: see Format and limits.

• Emit order: events are queued and emitted in the order of the state changes, by one drainer at a time, and emit may re-enter; see the notes of How to wire.

Added in v0.4.30:

• Close commits the pending (partial) minute and calls the hook for it, then closes; before, that minute was dropped and a restart across a minute boundary reset alert runs. See the notes of How to wire.

Measured cost

50 items, 24 series each, one Record per item every 5 s (one tick in twelve commits a minute for all 50: the heavy tick). Host: AMD Ryzen 9 5900X, 24 threads, NVMe SSD (Windows) and the ext4 virtual disk of a WSL2 distribution (Linux), measured 2026-10-20 with go test ./pkg/stats/ -run xxx -bench 'RecordTick|MinuteValue' -benchmem -count 3. cpu-percent-of-core is the mean tick time over 5 s (wall time, so it includes the wait for fsync).

With the Evaluator wired through SetOnMinute (20 of the 50 items above their alert thresholds, so their state changes at every minute), measured after minuteValue was made to read only the series it needs: Windows no fsync 4.1 ms mean tick, about 1 100 allocs, OnMinute 0.4 ms mean; Linux no fsync 3.0 ms; with fsync 18-40 ms mean and the same worst ticks as without the Evaluator (up to about 2.7 s). Evaluator.minuteValue reads only the requested series: 0.027 ms and 14 allocs on Windows (0.004 ms on Linux), twice per item per minute. The Evaluator holds its lock through marshal and the write of alerts.json (0.4-0.9 ms on average, a few ms worst).

TestRecordTickBudget runs one simulated hour. Without fsync it asserts the mean tick (< 5 ms, < 25 ms with the Evaluator) and fewer than 5 000 allocations per tick; every variant has a 15 s worst-tick hang guard, and the worst tick and the fsync numbers are logged, not asserted. The fsync worst tick is dominated by the heavy tick (50 commits, several fsyncs each) and scales with the disk's fsync latency; committing minutes from a per-store goroutine is the upgrade if a real disk needs it; it is not done.

Cost

Headline numbers (50 items, 24 series, 5 s ticks): without fsync a tick costs 2.4 ms on Windows and 0.7 ms on Linux (0.05 % and 0.013 % of a core), about 460 allocations; with the Evaluator wired, 4 ms on Windows and 3 ms on Linux, about 1 100 allocations. With fsync the mean is 21-35 ms (Windows NVMe) and 38-42 ms (WSL2 virtual disk); the worst tick depends on the host, from about 1 s up to about 3 s.

A Record that crosses a minute boundary commits the previous minute, which costs two fsyncs per item. A service that samples its 50 items in one goroutine can therefore see one tick delayed by seconds, at the minute boundary, on a slow disk. Samples keep their own timestamps, so nothing is lost; the service should either sample each item from its own goroutine or accept the jitter. TestRecordTickBudget is strict on the mean and the allocation count without fsync, and only guards against a hang (15 s) on the worst tick; with fsync it logs the numbers.