Overview ¶

crashguard.go — M1 panic-recovery (supportability framework).

Design (adversarially synthesized): a single top-level middleware CANNOT catch panics in detached relay/geo/alert/SOCKS5 goroutines (Go's recover() only unwinds its own goroutine) and MUST NOT try to write an HTTP status after a CONNECT hijack. So recovery is split into structural planes plus a per- goroutine tap at every detached-goroutine birth site, all funneling into ONE re-panic-proof, rate-limited, redaction-aware sink (recordCrash).

• PROXY plane = record-only guard (proxyCrashGuard): never touches the ResponseWriter, so the happy path is byte-identical and zero-alloc and a hijacked tunnel is never corrupted. Re-panics http.ErrAbortHandler so net/http keeps its intentional silent-abort protocol.
• ADMIN plane = withAdminPanicRecovery: the admin chain never hijacks, so a clean 500 is valid when nothing was committed. trackedRW implements Unwrap()+Flush() so it is NOT interface-masking (ResponseController + SSE keep working).
• Detached go-sites = recoverGoroutine / the relay defer.

The sink logs only crash id/component/correlation (never the raw panic value — sanitizeLog scrubs control chars, it does NOT mask a secret), keeps the bounded text only in an in-memory lastCrash reachable exclusively via a redacting accessor, and is throttled per component so a request-triggered panic cannot flood the SIEM or evict the shared audit ring. Timeline note: M1 builds no timeline store; the crashRecord is shaped as the enumerated "crash" event so the M3 timeline consumes it with no schema change (ADR-0012).

D1.6b: --list-backups one-shot CLI.

The Maintenance Agent's GET /v1/backups endpoint shells out to the cli container (via `docker compose --profile cli run --rm cli --list-backups --backup-dir /backup`) and parses the JSON array this command emits to stdout. Operators can also invoke it manually for debugging — same output format either way.

Encryption detection uses the D1.4 magic prefix `CVRTBK01` at byte 0 (see backup_encrypt.go: backupEncMagic). We peek 8 bytes; entries whose first 8 bytes match are flagged encrypted=true. Anything else is reported as encrypted=false; we do NOT attempt deeper validation (gzip-magic, tar header, etc.) because the agent only needs an operator-facing "encrypted-or-not" hint here, and partial reads are cheap.

Symlinks, directories, and unreadable entries are skipped silently rather than failing the whole listing — the human (or agent) can act on whatever IS readable. Errors at the directory level (e.g. non-existent --backup-dir) DO fail the command.

Culvert — Enterprise-grade open source HTTP/HTTPS proxy https://github.com/KidCarmi/Claude-Test

Release Management API — P1.6d-0 (foundation; NO GUI in this slice).

Read-only catalog/current endpoints plus the async dispatch + resume control endpoints, all backed by the P1.6c DispatchService. The handlers own only HTTP concerns (auth, decode, the async 202 split, and a bounded per-agent status store); planning/execution/verify-by-digest/audit/alert all live in the service. Agent endpoints are resolved by an injected resolver (NOT a config route — config is a later slice), so this slice adds no /api/releases/config, no GUI panel, no auto-update, no scheduling.

Dispatch is asynchronous: the handler starts DispatchService.Dispatch in the background and returns 202 the instant the op_id + resume context are durably recorded (the service's onApplied observer). A planner/preflight refusal that happens BEFORE apply returns a synchronous 4xx/503 instead. The watch then runs to terminal in the background and updates the status store, which the GET status endpoint surfaces.

Release Catalog Verified Auto-Seed — P1.7.

At Control-Plane startup, if CULVERT_RELEASE_CATALOG_URL is set AND signature verification is in enforce mode, fetch the official signed catalog from that URL, verify it (signature + freshness + rollback), and atomically install it into <dataDir>/release_catalog/ so a clean install reaches available:true without manual file placement. Anything wrong ⇒ leave the existing on-disk catalog untouched and continue (fail closed). NO unsigned auto-download.

Verification lives HERE, in the Go binary (it holds the trust roots) — never in the installer/shell. The installer only forwards the env var.

Scope (roadmap/D1.6d-P1.7-catalog-autoseed-plan.md): boot-time seed only. NO background ticker/polling, NO GUI, NO mirror/air-gap, NO agent changes. The provider + this function are reusable by a later refresh slice.

Release Catalog Runtime — Slice 1 (Catalog Load + Resolve).

A pure, in-process, UNSIGNED, local-source, read-only release catalog. It parses the release index + referenced manifests, validates them fail-closed, verifies each manifest's content hash against its RAW bytes, and builds the forward (channel → release) and reverse (pinned ref → release) indexes the rest of the release layer depends on.

Scope (roadmap/D1.6d-P1.2-release-catalog-runtime-plan.md): Catalog Load + Resolve only. NO signatures, NO network/refresh, NO GUI, NO agent calls, NO upgrade dispatch, NO rollback-candidate computation, NO CP→DP propagation.

Control-Plane-only: nothing here is wired into the proxy/data-plane path by this slice; it returns a DIGEST (`repo@sha256:…`, the agent's native currency) and has no release/channel awareness on the agent side.

Integrity ≠ authenticity: the manifest_sha256 check (§4.9) is a content hash, NOT a signature. It catches corruption/drift but is worthless against a tampering attacker, and the INDEX itself has no integrity protection until a later slice signs it. Do not treat a loaded catalog as authenticated.

Release Catalog Distribution — P1.5 Slice d (air-gap BundleCatalogProvider).

BundleCatalogProvider is a TRANSPORT that stages the CATALOG portion of a signed offline release bundle (a tar / tar.gz archive carried into an air-gapped site) into a fresh temp dir, which the caller then hands to LoadVerifiedCatalog (the P1.3 trust boundary) exactly like any other source. Verification is identical to online — only the transport differs (plan §8).

Unlike the HTTP provider, the bundle is COMPLETE before any parse, so no two-phase gate is needed (plan §5.1: local/bundle providers stage-then-verify unchanged). The provider therefore does NO trust work at all — it only moves bytes, under a hostile-archive discipline:

• any absolute / traversal ("..") / backslash / NUL entry name rejects the whole bundle (never silently skipped);
• any symlink or hardlink entry rejects the whole bundle;
• catalog entries must be regular files within per-file and total bounds;
• non-catalog entries (the future image blobs, plan §8) are skipped WITHOUT reading their content;
• index.json is required; index.json.sig and index.json.sigstore are staged when present — the enforce/permissive decision for a missing signature belongs to the P1.3 mode at verify time, not to the transport.

Scope (roadmap/D1.6d-P1.5-catalog-distribution-plan.md — Slice d): catalog extraction + staging + cleanup only. NO image docker-load (the bundle slice that loads images MUST bind loaded-image-digest == authenticated list_digest, P1.3 §7 — recorded there, not here), NO outer bundle signature, NO GUI/API, NO dispatch, NO agent changes, NO HTTP changes.

Release Catalog Freshness & Rollback Protection — Phase 1.

Authenticity (release_catalog_verify.go) proves a catalog is GENUINE; it does NOT prove it is the LATEST. This file adds the two missing trust dimensions a signed update channel needs (TUF "timestamp"/"snapshot" roles, in spirit):

• Freshness: a signed catalog declares expires_at; a CP past that instant (plus a small clock-skew tolerance) refuses it, so a captured-and-replayed stale-but-validly-signed catalog cannot silently pin users to an old, vulnerable release forever.
• Rollback / freeze protection: each catalog carries a monotonic catalog_version; the CP persists the highest version it has ever accepted and refuses anything below it, so an attacker who can serve an OLD signed catalog cannot downgrade the deployment.

Both checks are ENFORCE-mode only and are composed ON TOP of signature verification — they never widen trust, only narrow it. They are deliberately kept out of the structural loader (release_catalog.go) and the signature gate (release_catalog_verify.go) so those layers stay pure/offline; freshness needs a clock and rollback needs persisted state, which only the holder owns.

Release Catalog Distribution — P1.5 Slice a (CatalogHolder + atomic publish).

The CatalogHolder owns the live, atomically-swappable verified *Catalog that the rest of the Control Plane reads. It is the swap component P1.2 deferred ("reload = construct a new *Catalog; the refresh slice will own swap semantics; not here"): this slice loads + verifies from a single local directory, publishes atomically, exposes an explicit no-catalog state, and a manual reload that keeps the current catalog on any failure.

Scope (roadmap/D1.6d-P1.5-catalog-distribution-plan.md — Slice a): holder + atomic publish + local-dir reload only. NO goroutine, NO HTTP provider, NO cache persistence, NO staleness, NO GUI/API, NO agent/dispatch/air-gap. The only path to a published catalog is LoadVerifiedCatalog (the P1.3 trust boundary) — there is deliberately no "publish raw *Catalog" entry point.

Release Catalog Distribution — P1.5 Slice c (HTTP CatalogProvider).

HTTPCatalogProvider is a TRANSPORT that stages a catalog candidate (index.json + index.json.sig + referenced manifests) from an HTTP(S) origin into a fresh temp dir, which the caller then hands to LoadVerifiedCatalog (the P1.3 trust boundary) exactly like a local-dir source.

The §5.1 contract is the heart of this slice: because manifests cannot be fetched without first reading their refs out of the index, the provider runs a TWO-PHASE verify — it verifies the index signature over the RAW index bytes BEFORE parsing the index to enumerate manifest fetches. A forged/unsigned index (in enforce mode) therefore triggers ZERO manifest requests. The final LoadVerifiedCatalog over the staged dir re-verifies everything (defense in depth, incl. the manifest_sha256 hash check that authenticates each fetched manifest).

Scope (roadmap/D1.6d-P1.5-catalog-distribution-plan.md — Slice c): the HTTP transport + two-phase verify + staging + cleanup + timeout + minimal retry/backoff + conditional (ETag/If-Modified-Since) fetch. NO GUI, NO dispatch, NO agent changes, NO air-gap bundle, NO CP→DP propagation. The provider is transport-only apart from the §5.1 index-verify gate.

Release Catalog Distribution — P1.5 Slice b (Refresher + last-good cache + staleness).

The Refresher orchestrates stage → verify → publish on top of the P1.5a CatalogHolder, adds a last-good on-disk cache (restart durability), single- flight, refresh metadata, and staleness. It still uses a LOCAL directory as the source — the HTTP provider and air-gap bundle are later slices.

Scope (roadmap/D1.6d-P1.5-catalog-distribution-plan.md — Slice b): refresher + cache + staleness + single-flight + an OPTIONAL (default-off) interval ticker. NO HTTP provider, NO air-gap bundle, NO GUI/API, NO agent/dispatch, NO metrics/alert wiring. Every publish still goes through LoadVerifiedCatalog (the P1.3 trust boundary); the cache is re-verified on load.

Release Catalog Runtime — Slice 1: query surface (Resolve / Lookup / Current / List). All methods are pure and I/O-free after LoadCatalog.

Release Catalog Keyless Trust — P2b-1 (Sigstore-identity verifier).

Adds a SECOND catalog-signature scheme that coexists with the ed25519 scheme (release_catalog_verify.go): a cosign "keyless" bundle (Fulcio cert + signature + Rekor inclusion proof) over the RAW index.json bytes, verified offline against a baked Sigstore TUF trusted root plus a PINNED workflow identity (issuer + SAN regex). Because the index binds every manifest by manifest_sha256, one signature over one document authenticates the whole catalog — same kernel as ed25519.

Trust model: NO private key exists anywhere (keyless). Trust is "this exact GitHub Actions workflow, on a release tag, signed via Fulcio". Verification is fully offline: the bundle carries its own Rekor inclusion proof + integrated timestamp, and the trusted root is baked, so nothing reaches sigstore.dev at runtime (air-gap safe).

Scope (roadmap/D1.6d-P2b-sigstore-identity-trust-plan.md): P2b-1 shipped the verifier, the .sigstore sidecar source, scheme selection, and wiring resolution. As of P2b-2a the official Sigstore public-good trusted root is BAKED into the embed below, so the scheme is ACTIVE by default (an operator can override or deactivate it via CULVERT_RELEASE_SIGSTORE_TRUSTED_ROOT). NO release-side signing yet — CI keyless signing + the end-to-end/image-sig gates are P2b-2b. Verifying identity (cert chain + SAN + issuer) happens BEFORE any other bundle content is trusted.

Release Catalog Authenticity — P1.3 Slice 1 (index signature verification).

Adds a Control-Plane-side authenticity gate on TOP of the P1.2 catalog runtime (release_catalog.go): an ed25519 detached signature over the RAW index.json bytes, verified against a baked/operator TrustStore BEFORE the index is parsed or any manifest_sha256 entry is trusted (plan §5/§5.0). Signing the index transitively authenticates every manifest (the index binds each by sha256), so one signature over one document is the whole authenticity kernel.

Scope (roadmap/D1.6d-P1.3-catalog-authenticity-plan.md — Slice 1): TrustStore + envelope parse + LoadVerifiedCatalog + the three enforcement modes. NO GUI, NO agent/dispatch, NO air-gap bundle, NO network/refresh, NO CP→DP propagation, NO metrics wiring. The verifier is release-agnostic and verifies fully offline.

Release Dispatch — P1.6 Slice a (CP-side dispatch kernel, PURE PLANNING).

The Dispatcher turns an operator target (release_id or channel) into a DispatchPlan: it resolves the release from a PINNED immutable catalog snapshot (P1.5 holder), reconciles the catalog repo against the deployment's proxy_repo (repo equality / one explicit air-gap rewrite, design §4), derives Current/already-current from the agent's running_image.repo_digests (P1.1, design §3/§5), and builds the EXISTING upgrades.apply request object (design §6) — WITHOUT sending it.

It is a PURE, deterministic planner: no I/O, no randomness, no agent contact. The idempotency key is an INPUT (higher orchestration owns op identity); the catalog snapshot is read exactly once at plan start. The agent receives only an image_ref + existing apply flags — no release/channel/version/catalog data crosses to it, and it stays release-agnostic.

Scope (roadmap/D1.6d-P1.6-release-dispatch-plan.md — Slice a): planning + the request object + tests. NO agent POST, NO upgrades.check, NO tags, NO tag updater, NO new agent endpoint, NO agent changes, NO GUI, NO auto-update, NO rollback-candidate computation, NO legacy-updater changes.

Release Dispatch — P1.6 Slice b (CP-side execution wrapper).

DispatchExecutor takes a FROZEN DispatchPlan (P1.6a), generates the CP idempotency key, POSTs the EXISTING upgrades.apply, polls the EXISTING agent op to a terminal state, re-reads running_image.repo_digests, and classifies a terminal DispatchTerminal by VERIFYING THE RUNNING DIGEST itself (never the agent's self-report). It is single-flight per agent and emits audit events via a hook.

The agent is untouched and stays release-agnostic: only image_ref + existing apply flags cross the wire (no upgrades.check, no tags, no fallback). The transport is behind the AgentClient seam so the orchestration is fully testable with a fake; httpAgentClient is the concrete adapter over the existing /v1 endpoints.

Scope (roadmap/D1.6d-P1.6-release-dispatch-plan.md — Slice b): execution + verify-by-digest + terminal classification + audit structs/hooks. NO GUI, NO auto-update, NO rollback-candidate computation, NO legacy-updater change, NO new agent endpoint.

Release Dispatch — P1.6 Slice c (CP-side orchestration wiring).

DispatchService is the CP-side wrapper that ties the PURE planner (P1.6a) to the execution wrapper (P1.6b/c-0) and the rest of the Control Plane: the audit ring (release.dispatch + release.dispatch.outcome), the alert webhook hook, and the real HTTP transport to the EXISTING agent /v1 surface.

It owns an AGENT-KEYED single-flight registry: exactly one DispatchExecutor per agent identity, so the executor's per-instance single-flight becomes a per-AGENT guarantee regardless of how many Dispatch calls arrive. A second dispatch to an agent already mid-op is rejected (errDispatchInFlight) and audited, never queued or duplicated.

The agent stays release-agnostic: only image_ref + existing apply flags cross the wire (no upgrades.check, no tags, no fallback). Verify-by-digest remains the only success gate, and the CP idempotency key is generated ONCE per dispatch op and threaded through the plan so the executor honors it (P1.6c-0).

Scope (roadmap/D1.6d-P1.6-release-dispatch-plan.md — Slice c): service wrapper, agent registry, Resume/re-poll, audit + alert wiring, real transport. NO GUI, NO agent change, NO new agent endpoint, NO legacy-updater work, NO auto-update, NO rollback-candidate logic.

Release Catalog Generator — P2a.

Deterministically generates a release catalog (index.json + per-release manifests) from release metadata, so an official catalog is produced by CI from the digest the workflow actually pushed — never hand-edited. The output is the EXACT on-disk shape the P1.2 loader (release_catalog.go) consumes, and the P2a CI gate (release_gen_test.go) round-trips the generated bundle back through LoadVerifiedCatalog so the generator can never drift from runtime verification.

Determinism is load-bearing: the loader authenticates each manifest by hashing its RAW bytes, so generation MUST be byte-stable — compact json.Marshal (no indent; encoding/json already emits map keys in sorted order), RFC3339-UTC timestamps supplied by the caller, and releases emitted in sorted release_id order. TestGenerateReleaseCatalog_Deterministic is the merge gate.

Scope (roadmap/D1.6d-P2-release-pipeline-signing-plan.md — P2a): generation + validation only. NO signing trust decision (the CI gate signs with an EPHEMERAL key for round-trip proof only), NO Control-Plane trust-model change (that is P2b), NO network.

Release Catalog Spec Construction — M0 / E1 + E2 (correctness core).

buildReleaseSpec assembles a *deterministic* releaseCatalogSpec from immutable inputs (a release version, the pushed image digest, and the tagged commit's date) so that re-running the same release produces byte-identical catalog bytes. It is PURE: no wall-clock, no git, no network. All timestamp and version derivation that used to live in CI shell (`date -u`, `git tag | wc -l`) moves here where it is unit-tested.

The two load-bearing correctness properties (M0 design v2):

1. Deterministic timestamps — generated_at/created_at come from the tagged commit's committer date (a fixed input per tag), normalized to UTC "Z". expires_at = generated_at + ExpiryDays. Re-running a tag yields identical bytes; the loader authenticates each manifest by hashing its RAW bytes (release_catalog.go), so a one-byte timestamp drift would break the digest binding.

2. Race-free monotonic version — catalog_version is a pure function of the release semver (catalogVersionFromSemver), NOT a live count of git tags. This makes allocation idempotent (re-running any tag, before or after other tags exist, yields the same version), monotonic (a later GA release encodes higher), and collision-free (distinct semver ⇒ distinct integer) — with no shared mutable source to race on. Re-signing the same release trivially reuses its version (same semver), so "re-sign does not allocate" holds by construction.

buildReleaseSpec validates only its DERIVATION inputs (required fields, a strict GA semver for the version encoding, timestamp parseability, the expiry/dead-on- arrival guards); all OUTPUT-shape/semantic validation (repo format, digest, channel-known, created_at) stays in generateReleaseCatalog (release_gen.go), which remains the sole shape validator. The strict semver core here is deliberately no looser than the generator's, so this file never accepts a version the generator would reject.

Release Management startup wiring (P1.6d-0.1).

Constructs and publishes the Release Management backend (catalog provider + DispatchService + releaseManager) so the /api/releases* routes are actually usable instead of reporting "not configured". It is deliberately MINIMAL and NON-FATAL: any failure leaves globalReleaseMgr nil and the routes report a clear 503 — never a panic.

Scope: empty catalog holder (a later refresh slice populates it), the default proxy_repo, an optional empty repo_rewrite, and the single CP-LOCAL maintenance agent (key "local"), reached over its unix socket by default or an http(s) URL via CULVERT_MAINT_AGENT_URL. NO mutable config route, NO GUI.