Documentation
¶
Overview ¶
Package loadstudy pulls real recorded load metrics and application events out of ClickHouse so the ADR-0150 detectors can be measured against something nobody synthesised.
Every accuracy figure in github.com/stergiotis/boxer/public/analytics/timeseries/adscore and its siblings comes from fixtures this repository generated. That is a real gap: a generator and a detector written by the same hand can agree with each other about a signal that does not exist. This package closes it with data that was recorded for other reasons entirely.
What it reads, and what it does not ¶
Load metrics come from ClickHouse's own `system.asynchronous_metric_log`, sampled at 1 Hz: CPU split into user, system and iowait, run-queue depth, resident memory, block IO in both directions, inbound network. Per-device families are summed across devices, so the channel set is portable across machines whose disks and interfaces are named differently.
They do *not* come from github.com/stergiotis/boxer/public/observability/sysmetrics, which would be the natural source. That scraper published to NATS and persisted nothing, so its data did not exist to analyse — the gap this package's fallback was chosen around, and the demand ADR-0184 cites for closing it.
That gap is now closeable: `sysmetricsd --tee` writes the plane into `boxer.facts` through github.com/stergiotis/boxer/public/keelson/runtime/sysmtee. Migrating these channels onto it is a separate decision and has not been made — the study's published results were computed against `system.asynchronous_metric_log`, and swapping the source silently would make old and new runs incomparable.
An earlier version of this comment added that `boxer.facts` "carries no numeric payload at all — it is an event log". That was wrong about the schema even when written: the table has the full `u8`…`i64Array` set, `u32Set`/`u64Set`, and `f32Array`/`f64Array` under an encoding hint chosen for slowly-changing series. It was not quite right about the contents either — github.com/stergiotis/boxer/public/gov/capmapfacts has been writing an f64 there, a normalized compression distance, since before any of this. What was true, and only this, is that nothing wrote *load metrics*; the tee is what changes that.
Events come from `boxer.facts`: application lifecycle, run starts and stops. Heartbeats are excluded, because they fire on a timer rather than on anything happening.
The caveat that governs how any result may be read ¶
**Events are not anomaly labels.** An application starting is normal behaviour. Scoring a detector against event times measures whether it fires when the workload composition changed — useful, and not the same as accuracy. Two consequences follow and neither is optional:
- A detection with no event near it is not necessarily a false positive. The precision term is therefore pessimistic by an unknown amount.
- Any figure produced here is meaningful only *relative to* the one-liner baselines in adscore, run over the same series. If a moving-average residual correlates with events just as well, then the correlation is trivial and says nothing about the detector.
EventLabels widens each event across a tolerance, because a start changes what the machine does over the following seconds rather than in the bin the log line landed in. That tolerance is the study's most arguable parameter.
Gaps ¶
The 1 Hz source is not gap-free: ClickHouse is not always running. Bins with no sample are forward-filled and counted in Series.Gaps. A large count means part of the series is invented, and a report that does not say so is misleading.
Running it ¶
The study itself is an integration test in this package, carrying `//go:build integration` because it needs a live server. It skips when CLICKHOUSE_ENDPOINT is unset. See doc/adr/0150-timeseries-subsequence-anomaly-detection.md for what it was built to decide.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var DefaultChannels = []Channel{
{Name: "cpu_user", Metric: "OSUserTimeNormalized"},
{Name: "cpu_system", Metric: "OSSystemTimeNormalized"},
{Name: "cpu_iowait", Metric: "OSIOWaitTimeNormalized"},
{Name: "load1", Metric: "LoadAverage1"},
{Name: "mem_resident", Metric: "MemoryResident"},
{Name: "block_read", Prefix: "BlockReadBytes_", MaxPlausible: bytesPerSecondCeiling},
{Name: "block_write", Prefix: "BlockWriteBytes_", MaxPlausible: bytesPerSecondCeiling},
{Name: "net_rx", Prefix: "NetworkReceiveBytes_", MaxPlausible: bytesPerSecondCeiling},
}
DefaultChannels is the load surface this study reads: CPU split three ways, run-queue depth, resident memory, block IO in both directions, and inbound network.
var DefaultEventKinds = []string{"app-lifecycle", "runtime-run", "started", "stopped"}
DefaultEventKinds are the fact symbols that mark a change in what the machine was being asked to do. Heartbeats are deliberately excluded: they fire on a timer rather than on anything happening.
var PackageProps = packageprops.Props{ Kind: packageprops.KindIntegrationTest, }
PackageProps records this package's curated properties (ADR-0080).
The WASM fields are deliberately left unset. This package speaks HTTP to a ClickHouse server, so no wasm target is a plausible destination for it, and asserting Blocked without having run the survey would be a claim rather than a verdict. The zero value asserts nothing, which is the honest state.
Functions ¶
func EventLabels ¶
EventLabels turns the bins that held an event into a label vector, widened by tolerance bins on each side.
The widening is not slack. An app start changes what the machine is doing over the seconds that follow, not in the bin the log line landed in, so a detector that fires shortly after is agreeing with the event rather than missing it. The tolerance is a judgement about that lag and is the study's most arguable parameter — vary it and see.
func LabelledFraction ¶
LabelledFraction returns the share of bins the labels cover, which is the prevalence any precision figure has to be read against.
Types ¶
type Channel ¶
type Channel struct {
Name string
// Metric selects one metric exactly; Prefix sums every metric starting with
// it, across devices, before binning. Exactly one is set.
Metric string
Prefix string
// MaxPlausible discards samples at or above it, zero meaning no bound.
//
// ClickHouse derives the per-device byte counters as deltas, and a delta
// underflows when an interface disappears — this host produced a
// NetworkReceiveBytes reading of 2^64 minus a hundred thousand. Such a value
// is not a large measurement, it is a wrapped one, and a detector handed it
// will report the wrap as the most anomalous event in the series. The bound
// belongs far above any real hardware and far below the wrap.
MaxPlausible float64
}
Channel names the series this study extracts. Each is either a single ClickHouse asynchronous metric or a sum over a family of per-device ones, which is what keeps the set portable across hosts: device and interface names differ per machine, the aggregate does not.
type Client ¶
Client is a minimal ClickHouse HTTP query surface — enough to pull a series out of the system logs and no more. The heavier clients in github.com/stergiotis/boxer/public/db/clickhouse carry a dependency chain this package has no use for.
func NewClientFromEnv ¶
func NewClientFromEnv() (inst *Client)
NewClientFromEnv builds a client from the CLICKHOUSE_* registry entries. The endpoint falls back to CLICKHOUSE_URL, then to localhost.
type Series ¶
type Series struct {
// Start is the timestamp of bin 0; Step is the bin width.
Start time.Time
Step time.Duration
// Names indexes Values: Values[i] is the series for Names[i].
Names []string
Values [][]float64
// EventRate counts events per bin — a channel in its own right, and the one
// place the cause side of the correlation is visible.
EventRate []float64
// EventBins marks bins that contained at least one event.
EventBins []bool
// Gaps counts grid bins no sample landed in, worst channel. They are
// forward-filled, so a large count means the series is partly fabricated and
// the study should say so rather than quietly average over it.
Gaps int32
// ChannelGaps is the same count per channel, aligned with Names. Metric
// families are not sampled on a common schedule, so a grid that is gap-free
// for CPU can still be sparse for block IO.
ChannelGaps []int32
// Rejected counts samples dropped by a channel's plausibility bound.
Rejected int32
}
Series is the extracted study data: a regular time grid, one value slice per channel, the binned event count, and the bins an event fell into.
func ExtractE ¶
ExtractE pulls the study series out of ClickHouse.
Metric values are averaged within each bin. Per-device families are summed across devices at each source timestamp *before* averaging, so a machine with three disks and one with one produce comparable numbers.
func (*Series) SortedNames ¶
SortedNames returns the channel names in a stable order, so a report reads the same way twice.
type Span ¶
Span is a stretch of the metric grid with no missing bin, together with how many events fell inside it.
Studying spans rather than a fixed window is not tidiness. This host records intermittently, so a fixed window is mostly forward-filled — invented — and a detector run over invented data reports on the invention.
func FindSpansE ¶
func FindSpansE(ctx context.Context, client *Client, lookback time.Duration, step int32, minBins int32, minEvents int32, kinds []string) (spans []Span, err error)
FindSpansE returns the gap-free spans of the metric grid over the lookback, longest first, keeping those with at least minBins bins and minEvents events.
type Spec ¶
type Spec struct {
// From and To bound the study window. To of zero means now.
From time.Time
To time.Time
// StepSeconds is the bin width the irregular 1 Hz samples are averaged onto.
StepSeconds int32
Channels []Channel
// EventKinds are the boxer.facts symbol values that count as events. Empty
// accepts [DefaultEventKinds].
EventKinds []string
}
Spec parameterizes an extraction.
Source Files