halo-server

halo-server

Serves the Halos API and console. Optionally runs the automated experiment loop (--controller) and the gateway kill switch.

Core flags

HA on Postgres (ADR-0021)

halo-server migrate --database-url-file F                       # apply migrations, print the schema version
halo-server import --from-data-dir DIR --database-url-file F    # copy a STOPPED server's data dir into an empty database, then verify counts and the audit chain

--killswitch-key-file and --controller accept --database-url-file in place of --data-dir. /readyz also fails while the database is unreachable, and --metrics-listen adds halo_leader, halo_leader_changes_total, halo_db_errors_total and halo_db_open_connections. Integration tests run against a real Postgres when HALO_TEST_PG_DSN is set.

Kill switch

Instant rollback on the traffic plane, with no PR merge needed. Killed experiments are treated as not running by halo-proxy and halo-kong: control routing and no shadow mirroring.

The key needs the token; startup fails otherwise, and fails if the key is set without --data-dir. The token alone serves only GET /api/v1/gateway/posture (below). Without a key the server is honest about it: no kill list is served, kill and unkill return 501, GET /api/v1/capabilities reports killSwitch: false, and the in-process controller reports rollbacks as not enforced ("merge the pause PR urgently") instead of recording a kill nobody reads. The gateway endpoint returns {"payload": base64(JSON {version, experiments, issuedAt}), "signature": base64(ed25519("halo-killswitch-v1\n" + payload))}.

Gateways reject a list if:

• the signature is bad,
• it is older than 10 minutes or more than 1 minute in the future, or
• it is not strictly newer than the list they already hold.

On a fetch failure a gateway keeps its last-known list. A gateway that has never fetched a list kills nothing.

Admin endpoints (session plus admin role, same-origin, audit-logged as experiment.kill or experiment.unkill):

• POST /api/v1/experiments/{name}/kill with body {"reason": "..."} (required, 500 characters max; HTTP 422 without one)
• POST /api/v1/experiments/{name}/unkill with body {"reason": "..."} (optional)
• GET /api/v1/killswitch lists active kills with who, when and why.

A kill does not expire. Pausing or concluding the experiment in policy does not lift it. Unkill it explicitly before you restart the experiment.

Audit

Privileged actions append to the hash-chained audit.jsonl in --data-dir, fsynced before the action is acknowledged. They fail closed: kill, unkill, device enrollment, session and device revocation and experiment-status PRs return HTTP 500 if the append fails. A kill whose audit append fails stays applied (fail safe), a failed unkill is reverted, and a revocation stays in force. audit.jsonl has a single writer: do not point halo controller run at a data dir this server is using.

With --audit-config, rotation (maxBytes, maxAgeSeconds) renames the active audit.jsonl to audit-<first seq, 20 digits>.jsonl and starts a new one; the hash chain runs on across segments. halo audit verify --data-dir DIR checks the whole chain. export.sinks forwards every entry, at least once and in order, to a Splunk HEC endpoint, an HTTPS webhook (signed like the controller's) or RFC 5424 syslog over TLS, each with a cursor in <data-dir>/audit-forward/. Export never blocks or fails an action. See the SIEM export guide.

Controller (--controller)

On every tick the controller evaluates each status: running experiment against ClickHouse evidence and then:

• rollback: kills the experiment at once if the deciding evidence is gateway-sourced and a kill key is configured, opens a PR that sets status: paused with the evidence in the body, and notifies. Rollbacks decided on CLI telemetry or eval (LLM-judge) evidence are not auto-killed (killOutcome: not_gateway_evidence); without a kill key the notification says to merge urgently.
• promote: opens a PR that concludes the experiment and notifies. Before merging, a human adds the rollout change (ring release or model route).
• expired: opens a PR that concludes the experiment and notifies.

The controller never merges a PR. It takes each action at most once per experiment run; the record lives in controller-state.jsonl. When the experiment is next seen not running, that record resets. Errors are retried at 15s, 30s and so on, up to the tick interval. A running experiment that is already killed is held, not evaluated; notifications tell humans to unkill before resuming. Notifications are tracked per channel and a failed channel is retried on later ticks without re-posting to the others.

Webhook URLs are secrets. They are read from files and never logged.

For CI or cron, halo controller run --once --policy-dir . --clickhouse URL --data-dir DIR runs the same loop once. Its PRs come from the CI checkout. Its kills reach the gateways only when DIR is halo-server's --data-dir (a shared volume). Run only one controller per data dir. halo controller run also appends its kills to audit.jsonl, so it must not run against a data dir a live halo-server is writing; use --controller there. Pass --killswitch-served to halo controller run only when this server (with --killswitch-key-file) serves that data dir; otherwise its messages say the kill is only recorded.

Device posture

GET /api/v1/gateway/posture?subject=<id> (gateway token) answers {"compliant": bool, "reason": "..."} for the posture gate of rings with posture: warn|enforce. Compliant means the subject has at least one enrolled device and every enrolled device (not revoked, not expired) reported within --posture-max-age (default 45m), with no drift, on its ring's current verified release or one of that release's experiment channels (checked only when the portal config names a registry). See ADR-0011.