monitor

Overview ¶

Command monitor is the soak battery's continuous invariant checker.

It is the answer to "a soak that only reports it did not crash is worthless". The workload (test/soak/dags) generates load; this program decides, every sample, whether the control plane is still behaving, and writes the evidence down as it goes rather than at the end, because the interesting failure is the one that kills the harness.

It follows the test/load convention: a small `package main` driving the REAL control-plane code (storage.SchedulerStore) against a REAL Postgres. Unlike the load experiments it does not seed anything and it never writes: every query it issues is a SELECT, and the one product call it makes, SchedulerStore.ActiveRuns, is the read half of a scheduler tick and mutates nothing. A monitor that could perturb the system it measures would be worse than no monitor.

Outputs, all under --out, all written continuously:

samples.jsonl     one JSON object per sample (machine-readable)
violations.jsonl  one JSON object per invariant violation, with evidence
summary.md        a human summary, rewritten atomically every sample
verdict.json      the final verdict (also written every sample, so a killed
                  harness still leaves a current one behind)

Exit codes: 0 = every invariant held; 1 = at least one violation; 2 = the monitor itself could not run (bad flags, unreachable database).

Usage (see test/soak/README.md for the full runbook):

go run ./test/soak/monitor \
  --db "$SOAK_DATABASE_URL" --api http://127.0.0.1:18700 \
  --out ./soak-out --duration 30m