conformance/

Conformance fixtures

Fixtures are generated by TypeScript extractors against the exact checkout in .upstream/; committed goldens are never edited by hand. make fixtures regenerates every registered family, while make fixtures-check generates into a temporary directory and compares it with the committed tree.

To add a family, add one extractor under conformance/extract/, call it from generate.ts, emit a stable manifest.json plus deterministic data under conformance/fixtures/<family>/, and add the corresponding Go consumer. Extractors must omit timestamps and other ambient data; values such as IDs or clocks must be fixed or canonicalized at extraction time.

F11-native covers the Go-native extension runner described by the historical WP-350 spec. Its extractor executes the pinned upstream runner directly and records ordered dispatch, middleware, error, trust, and input semantics; the broader F11 matrix for unmodified JavaScript example extensions remains a Sprint 3 expansion gate.

The extractor runs from the .upstream/ working directory with Node 22 or newer. The Makefile installs only the pinned dev-time tools needed by registered extractors; F1 uses tsx@4.22.1, upstream's partial-json@0.1.7, and upstream's typebox@1.1.38, F2 also uses upstream's openai@6.26.0, @anthropic-ai/sdk@0.91.1, @google/genai@1.52.0, @mistralai/mistralai@2.2.6, and the pinned AWS Bedrock/Smithy transports, and F4 uses upstream's diff@8.0.4 plus cross-spawn@7.0.6, which its path helper imports. F3 drives upstream's real agent loop and faux provider with fixed clocks and IDs, emitting one exact JSON.stringify(AgentEvent) record per JSONL line; because the legacy compatibility entrypoint eagerly imports gitignored provider catalogs, its extractor creates the pinned offline catalog in a temporary copy and restores .upstream/ after the run. F4 calls the pinned edit-diff module directly and records its pure normalization, matching, replacement, display-diff, and unified-patch results; filesystem permission errors, mutation-queue scheduling, and legacy argument preparation remain ordinary tests at their respective layers, while full TUI redraw conformance belongs to Sprint 2. F5 drives both upstream truncation helpers and the streaming output accumulator; temp-file names are canonicalized to a presence bit, while spilled raw bytes are compared as base64 so invalid UTF-8 remains observable. F12 calls the pinned TUI package's public primitive components and snapshots their rendered lines; editor, markdown, and full-screen composite cases land in their scheduled Sprint 2 chunks. A fixture change is accepted only when the cited upstream behavior changed or a new case was deliberately added; a failing Go comparison is fixed in Go, never by weakening or hand-editing the golden.

WP-221 extends F2 with the Google provider, byte-exact Gemini requests, recorded stream/error traces, signed-thought replay, SDK-header precedence, hook-added generation fields, UTF-16 identifiers, slash-qualified models, nested Gemini 3 image results, root-routed safety settings, raw JSON error chunks with JavaScript number coercion, SDK-normalized HTTP errors, post-hook Mldev config/content transforms including $schema compatibility and empty arrays, and the pinned SDK's multiline-event quirk.

WP-231 extends F2 with byte-exact Mistral Conversations and Azure OpenAI Responses requests plus recorded stream/error traces, including Mistral SDK serialization and the pinned OpenAI SDK's proxy-query replacement quirk. WP-232 adds Bedrock request and event-stream fixtures through the pinned AWS serializer, plus generated provider login and credential-chain evidence. WP-260 adds the generic pi-messages request and serialized-event stream, including transport-only signature, redaction, tool metadata, and server-error member ordering.

WP-241 extends F2 with pinned Codex Responses request bytes, recorded SSE events, WebSocket cache framing, and OpenAI Codex, GitHub Copilot, and xAI subscription-provider metadata. Its Go consumers replay every generated case, including empty-session cache keys, nullable thinking mappings, cached-context deltas, transport fallback, Copilot token metadata, and provider-specific OAuth credential serialization.

WP-250 adds an auxiliary upstream-derived fixture family for model pattern resolution, fuzzy --list-models tables, provider-isolated models.json composition including fractional numeric limits, ordered model-store persistence, and the exact minimal/full documentation examples. The large models.dev input is committed separately under ai/models/testdata/ so catalog generation remains deterministic without duplicating it in the conformance tree.

WP-211 extends F2 with byte-exact TS/Go auth storage and legacy-migration cases, the upstream OAuth callback pages, reciprocal TS reads, and a live cross-runtime proper-lockfile contention check.

WP-320 extends F6 with upstream-generated tree, branch-persistence, fork-wire, session-listing, and HTML-export evidence. HTML extraction fixes the dark theme explicitly, compares the complete output hash and embedded payload, and records a DOM projection with its tolerated differences; the local Markdown divergence uses a separate Go golden because upstream has no Markdown exporter.

Sprint 1 adds F6Harness, generated from the pinned harness session, message, repository, and Node execution-environment sources. It covers exact JSONL rehydration and append bytes, memory and JSONL repositories, fork and context transforms, resource filesystem quirks, shell execution, and the live adapter shared by the SDK and coding-agent session manager.

WP-330 adds F3-session by invoking pinned upstream's actual runPrintMode over a real AgentSession and faux provider. Its six byte-exact JSONL traces cover multiple prompts, steering/follow-up queues, retry, manual short-session compaction, assistant error, and assistant abort; only the session-header timestamp is canonicalized to the fixed extractor clock. The Go consumer validates LF framing through EOF, rejects unknown or reordered header fields before canonicalizing CLI-only ID/timestamp/cwd nondeterminism, and separately covers piped stdin, resume/fork headers, metadata routing, serializer drain, and signal teardown.

The sync driver regenerates every registered family into a temporary tree at the candidate upstream revision, overlays that tree onto a temporary source copy, and runs go test -race ./.... A red sync therefore reports exact fixture deltas and failing tests without changing committed goldens; make sync-bump installs the generated tree and updates UPSTREAM.lock only after the same candidate passes.

WP-331 adds F7, a strict-LF bidirectional RPC transcript generated by the pinned TypeScript RPC mode and replayed without canonicalization through both the Go mode and an actual conformance-built pigo --mode rpc executable. An untagged production-binary conversation separately drives a local streamed completion. F7 covers framing and V8 parse errors, state/model/thinking/queue controls, prompt events, message and session projections, tree and cursor queries, bash persistence, stats, naming, and error envelopes. make upstream-rpc-tests builds the Go CLI, connects upstream's real RpcClient through the adapter, and runs all six pinned upstream RPC files unmodified against a local Anthropic-compatible SSE endpoint. The 17 process cases in rpc.test.ts exercise the Go binary; the remaining ten cases retain upstream's TypeScript client and framing assertions.

WP-340 adds F8 for skills and prompt templates. The extractor loads the same temporary skill and prompt trees through both upstream resource layers, records argument substitution and invocation formatting, and drives the real pinned AgentSession to prove extension-command, input-hook, skill-command, and template resolution order. F9 also carries an upstream-generated progressive disclosure block, including XML escaping and hidden-skill exclusion.

WP-360 adds the WP360 family for pi packages and project trust. Its extractor drives the pinned parseGitUrl over the full URL-form battery (including the hosted-git-info quirks: the eaten git: prefix on git:// URLs, WHATWG slash collapsing, user/repo github shorthand, and /tree/<ref> committish extraction), the real DefaultPackageManager.resolve() over declarative fixture trees (manifest globs and overrides, convention directories, package filters with !/+/- patterns, autoload:false deltas, global/project dedupe, top-level settings entries, trusted and untrusted auto-discovery with .agents/skills ancestors, installed npm/git package layouts, offline skip, and symlink dedupe), addSourceToSettings/removeSourceFromSettings persistence (identity matching across SSH/HTTPS forms, scope-relative local paths, ref replacement preserving filters), and the trust manager (getProjectTrustOptions labels and updates, store inheritance with null deletion, the exact sorted trust.json bytes, and hasTrustRequiringProjectResources layouts). HOME is pinned inside each case root and paths are canonicalized to <fixture> so the goldens stay machine-independent; the Go runner materializes the same trees and compares normalized JSON. It uses upstream's glob@13.0.6, ignore@7.0.5, minimatch@10.2.5, semver@7.8.0, hosted-git-info@9.0.3, and proper-lockfile@4.1.2.

WP-610's sync driver regenerates every registered family into a temporary tree at the candidate upstream revision, overlays that tree onto a temporary source copy, and runs go test -race ./.... A red sync therefore reports exact fixture deltas and failing tests without changing committed goldens; make sync-bump installs the generated tree and updates UPSTREAM.lock only after the same candidate passes.

WP-450 adds the WP450 side-by-side replay family without expanding F12 ahead of WP-460. The extractor drives upstream's real interactive components through its @xterm/headless@5.5.0 VirtualTerminal at two widths, records every settled visible frame, and executes the documented status-line, widget-placement, custom-header, and custom-footer examples. Terminal control streams, temporary paths, trailing cell padding, and loader clocks are normalized exactly as recorded in the manifest; the divergence ledger is empty, so the Go consumer must match every normalized frame. The upstream tool-update-diff and tool-result-diff frames are intentionally identical: the edit renderer retains the settled preview and suppresses a duplicate result diff when the bytes match.

WP450-session-selector drives the pinned startup session picker and its search module directly. Its normalized frames cover loading progress, current/all scope, threaded/recent/fuzzy ordering, fuzzy/quoted/regex queries, named and path toggles, delete confirmation and cancellation, removal, and select/cancel callbacks.