conformance/

Conformance fixtures

Fixtures are generated by TypeScript extractors against the pinned checkout in .upstream/ (make upstream); committed goldens are never edited by hand. make fixtures regenerates every registered family, and make fixtures-check generates into a temporary directory and diffs it with the committed tree. The extractors run with Node 22 or newer; the Makefile installs only the pinned dev-time tools they need.

There are two tiers. The render-golden families — every F12* directory plus the WP450 replay/preview/UI-demo files — are Orb-owned snapshots of Orb's own renderer (DECISIONS.md P6): they are excluded from upstream extraction and from the fixtures-check diff, guarded by their Go comparison tests, and regenerated only by make fixtures-tui (ORB_UPDATE_F12=1), which rewrites presentation values (frames, highlight output, theme colors, layout) in place while behavior-shaped values in the same files stay frozen upstream captures. Their manifest.json files keep the provenance of the original capture. Every other family is an upstream-extracted pi-parity gate.

A fixture change is accepted only when the upstream behavior changed, a case was deliberately added, or (render goldens only) an Orb presentation change was deliberately regenerated. A failing Go comparison is fixed in Go, never by weakening or hand-editing the golden.

Adding a family

Add one extractor under conformance/extract/, call it from generate.ts, emit a stable manifest.json plus deterministic data under conformance/fixtures/<family>/, and add the Go consumer. Extractors omit timestamps and other ambient data; IDs, clocks, temp paths and HOME are fixed or canonicalized at extraction time so goldens stay machine-independent.

Families

• F1 — message serialization, partial JSON and TypeBox-style schemas.
• F2 — provider request bytes, recorded stream/error traces and auth for every API shape: OpenAI, Anthropic, Google and Vertex, Mistral Conversations, Azure OpenAI Responses, Bedrock (through the pinned AWS serializer), pi-messages, Codex Responses (SSE and WebSocket framing), Copilot and xAI subscription metadata. Auth storage cases are byte-exact in both directions, including the OAuth callback pages and a live cross-runtime proper-lockfile contention check.
• F3 — upstream's real agent loop and faux provider with fixed clocks and IDs, one exact JSON.stringify(AgentEvent) per JSONL line; F3-session runs upstream's runPrintMode over a real AgentSession (multiple prompts, queues, retry, compaction, error and abort).
• F4 — the edit-diff module: normalization, matching, replacement, display diff and unified patch.
• F5 — truncation helpers and the streaming output accumulator; spilled bytes compare as base64 so invalid UTF-8 stays observable.
• F6 — session tree, branch persistence, fork wire, listing and HTML export (complete hash, embedded payload and a DOM projection).
• F7 — a strict-LF bidirectional RPC transcript replayed through the Go mode and a built orb --mode rpc. make upstream-rpc-tests also runs upstream's RPC test files unmodified against the Go binary through upstream's RpcClient.
• F8 — skills and prompt templates: argument substitution, invocation formatting, and extension-command, input-hook, skill-command and template resolution order.
• F9 — system-prompt assembly, including progressive skill disclosure; Orb's identity substitutions are the exact ledgered ones.
• F10 — compaction boundaries and token accounting.
• F11 — the Go-native extension runner (ordered dispatch, middleware, errors, trust, input), built-in tools and product wiring.
• F12 — Orb-owned TUI render snapshots.
• F13-dynamic-workflows — @quintinshaw/pi-dynamic-workflows@3.5.1 driven through thirteen scenarios against real pinned createAgentSession sessions with the faux provider. The plugin is npm ci-installed from an embedded integrity-pinned lockfile (the one network-dependent step, served from the npm cache when offline). Files under reference-tui/ are reference-only upstream observations; the Go consumer enforces that marking.
• WP250 — model pattern resolution, fuzzy --list-models tables, models.json composition and model-store persistence. The large models.dev input lives under ai/models/testdata/.
• WP360 — pi packages and project trust: parseGitUrl over the full URL-form battery, DefaultPackageManager.resolve() over declarative fixture trees, settings persistence and the trust manager.
• WP370Runtime — AgentSessionRuntime lifecycle over the session manager.
• WP440 / WP440Read — image format handling and the read tool's image results.
• WP450 — Orb-owned replay, preview and UI-demo render snapshots of the interactive components, plus the startup session picker and its search; recorded status-line events and callbacks stay frozen upstream captures.