wireoracle

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 7, 2026 License: MIT Imports: 13 Imported by: 0

Documentation

Overview

Package wireoracle is the phase's wire-level regression oracle (D-01, VRFY-01): it spawns the REAL codegraph binary via os/exec, drives a scripted JSON-RPC session over real stdio, and returns the raw captured bytes. It never imports any github.com/mark3labs/mcp-go package and never decodes a line into an SDK type — the entire point is proving wire behavior independent of whichever SDK happens to be serving it.

This is a normal Go package under test/, never testdata/ (GOLDEN-01's own lesson: `go list ./...` silently skips any directory literally named testdata).

Index

Constants

View Source
const ExpectedScenarioCount = 28

Concurrency ordering constraint (discovered this session, verified by hand against the real binary via repeated captures): mark3labs v0.56.0's stdio transport (server/stdio.go processMessage) dispatches every "tools/call" message onto a worker-pool queue and returns immediately WITHOUT waiting for it — every OTHER method (initialize, tools/list, and any unrecognized method) is handled synchronously inline before the next stdin line is even read. Two consequences that matter for byte-reproducible capture:

  1. A synchronous request queued AFTER an async tools/call can complete and be WRITTEN TO STDOUT BEFORE that earlier tools/call's response — observed directly: a scenario sending tools/call, tools/call, unknown-method in that order printed the unknown-method response first, and the two tools/call responses in a racy, run-to-run-varying relative order (confirmed non-deterministic across 5 repeated runs of an otherwise-identical request script).
  2. Two tools/call requests in the SAME scenario race each other in the worker pool with no ordering guarantee between them.

Every scenario below therefore carries AT MOST ONE tools/call request, and when present it is always the LAST request in Requests — exactly mirroring handshake-explore's own proven shape (initialize, tools/list, tools/call). This is a load-bearing invariant for any future scenario added to this list (plan 05, plan 07): violating it reintroduces intermittent CI flakes in TestFrozenTranscriptsMatch's byte comparison, not a real regression in the server under test.

Scenarios returns the oracle's full scripted scenario list. Phase 1 scripts exactly one scenario, handshake-explore, proving the oracle architecture end-to-end; plan 04 adds the 16 scenarios that bring the suite to exactly 17 — the D-05 full coverage bar approved at plan 04's Task 1 blocking checkpoint (full-bar, no additional scenarios); plan 05 (this plan) adds the six-era Legacy handshake baseline below, bringing the suite to exactly 23 — the six-era selection approved at plan 05's own Task 1 blocking checkpoint — this same file, same function, no phase-conditional branch (must_haves). ExpectedScenarioCount is the exact size Scenarios() must return (D-07), declared immediately beside Scenarios() itself so the single source of truth for "how many scenarios exist" lives next to the function that produces them. Changing this value is a deliberate act that must land in the same commit as a matching change to Scenarios() and to the frozen transcripts directory — TestScenarioCountIsExact (oracle_test.go) compares len(Scenarios()) against this constant with EXACT equality, never a lower bound, because a lower bound cannot detect a scenario silently disappearing. TestTranscriptSetMatchesScenarioSet separately enforces the transcripts-directory half via two-way set equality. Value: 1 tracer scenario (plan 01) + 16 scenarios (plan 04's full D-05 coverage bar, the "full-bar" blocking-checkpoint selection: 4 tools/list variants, 7 tools/call, 4 error shapes, 1 statelessness edge) + 6 scenarios (plan 05's "six-era" blocking-checkpoint selection: the multi-era Legacy handshake baseline) + 1 scenario (phase 3 plan 01's tracer, modern-discover-explore: the Modern 2026-07-28 server/discover + sessionless tools/call proof) + 2 scenarios (phase 3 plan 03's SPEC-02 proof: modern-meta-invalid-params and modern-meta-unsupported-version, freezing the -32602 and -32022 halves of per-request `_meta` validation) + 1 scenario (phase 3 plan 04's SPEC-05 proof: index-appears-mid-session, a real `codegraph init` run mid-session against the same connected server, driving an empty tools/list to a one-tool tools/list on the same connection) + 1 scenario (phase 5 plan 01's SPEC-09 proof: modern-listen-catalog-change, an opted-in Modern subscriptions/listen stream observing notifications/tools/list_changed on the same live connection after a real mid-session `codegraph init`) = 28. A shrinking count is the failure mode this constant exists to catch.

Variables

View Source
var Rules = []Rule{
	{
		Name:        "repoDir",
		Placeholder: "<REPO>",

		ExpectFires: false,
		Why:         "handshake-explore's tools/call result carries no path/file/root/repoPath JSON field; retained for later scenarios that do (e.g. codegraph_node file-mode reads)",
	},
	{
		Name:        "serverVersion",
		Placeholder: "<VERSION>",

		ExpectFires: true,
		Why:         "",
	},
	{
		Name:        "timestamp",
		Placeholder: "<TS>",

		ExpectFires: false,
		Why:         "no response in the handshake-explore scenario carries a timestamp/time/ts field; retained for a future scenario that does",
	},
}

Rules is the complete, documented normalization allowlist (D-04). Normalization is named-field placeholder substitution only: every rule below is anchored on the JSON field name that owns the value, never on the value alone — a bare value-replacement rule would silently erase the same string appearing in a field that must stay byte-verbatim. Bytes not matched by any rule here are compared byte-for-byte; the transcript is never decoded and re-encoded through encoding/json (a round trip erases field presence/absence and key ordering, which is the only place an omitempty-on-bare-bool regression is visible).

Functions

func Normalize

func Normalize(raw []byte, subs Substitutions) []byte

Normalize applies NormalizeWithLedger and discards the ledger.

func NormalizeWithLedger

func NormalizeWithLedger(raw []byte, subs Substitutions) ([]byte, map[string]int)

NormalizeWithLedger applies every rule in Rules, in order, to raw and returns the normalized bytes alongside a per-rule hit count — the mechanism TestNormalizeRuleLedgerIsHonest checks against each rule's declared ExpectFires.

func TranscriptPath

func TranscriptPath(name string) string

TranscriptPath returns the path to name's frozen golden transcript, relative to this package's own directory.

Types

type Anchor

type Anchor struct {
	// Scenario is the name of the Scenarios() entry this anchor checks.
	Scenario string
	// Name describes what this anchor asserts, for -v test output and
	// failure messages.
	Name string
	// Assert decodes only the one field this anchor names out of the raw
	// captured stdout — never by unmarshalling into an SDK type — and
	// fails the test if it does not hold.
	Assert func(t *testing.T, stdout []byte)
}

Anchor is one hand-authored, spec-pinned assertion, run against a named scenario's freshly captured stdout independently of the frozen transcript (an anchor read from the golden file would be circular — TestSpecAnchorsHold re-captures rather than reading testdata/wireoracle/transcripts/*.golden).

func Anchors

func Anchors() []Anchor

Anchors returns the hand-authored spec anchor set (D-02) that needs nothing beyond a plain int comparison: the two JSON-RPC error codes.

The THIRD spec anchor this plan's Task 3 also requires — handshake-explore's initialize result.protocolVersion == internal/mcp.ProtocolVersion — is deliberately NOT modeled as an Anchor value here. internal/mcp is package mcp, and that package's OTHER files (server.go, tools.go) import github.com/mark3labs/mcp-go; Go resolves imports at the package level, so even a reference to the single SDK-free symbol internal/mcp.ProtocolVersion would transitively pull the SDK under test into this package's own dependency graph — breaking this same task's own acceptance criterion ("go list -deps ... test/wireoracle contains no line under the MCP SDK's module path", VRFY-01). assertProtocolVersionAnchor therefore stays exactly where the tracer (plan 01) put it — oracle_test.go, a _test.go file, whose imports are invisible to `go list -deps` on the plain (non -test) package — and TestSpecAnchorsHold calls it directly for handshake-explore rather than through this Anchors() slice.

No anchor exists for the six-era Legacy baseline's "legacy-unsupported-2026-07-28" scenario (plan 05's D-06 multi-era baseline), and this omission is STILL correct today, unlike the paragraph this one replaces: that scenario is a classic `initialize` (no `_meta` at all) driven against a client offering "2026-07-28" as `params.protocolVersion`, and today's go-sdk@v1.7.0 server silently coerces that unrecognized value to its own latest supported revision rather than returning an error [VERIFIED: 01-RESEARCH.md Pitfall 1, reconfirmed unchanged post-migration by 02-RESEARCH.md] — asserting an error code there would assert a behavior that never fires on that scenario's own wire shape. It remains captured-and-frozen as a SUCCESS; do not "fix" this omission by adding an anchor to it.

This is a DIFFERENT scenario, and a DIFFERENT outcome, from modern-meta-unsupported-version below (phase 3 plan 03): that scenario sends a well-formed Modern `_meta` object whose io.modelcontextprotocol/protocolVersion is modernUnsupportedVersion ("2099-01-01", chosen because it sorts lexically AFTER "2026-07-28" — see its doc comment in scenarios.go), which DOES return an error (`-32022`) under go-sdk@v1.7.0, and IS anchored, below. A prior version of this comment described no unsupported-version anchor existing at all — that was true only for mark3labs v0.56.0's classic-initialize path and is now retracted for the Modern `_meta` path; do not re-merge the two scenarios' outcomes into one paragraph, they are structurally distinct (see modernUnsupportedVersion's own doc comment for the full trace of why offering a lexically-SMALLER version instead would land back on legacy-unsupported-2026-07-28's silent-coercion territory by a totally different mechanism — a `-32601` availability-gate rejection, not a version-negotiation coercion — and must still never gain an anchor either).

The framing invariant (every stdout line is jsonrpc:"2.0", every request id has exactly one response line) is also checked outside this slice, in TestSpecAnchorsHold, across every scenario in Scenarios() — it is not "the one field a scenario's own captured line carries," it is a cross-cutting structural check that applies uniformly to every scenario.

modern-discover-explore's discover cache-control check (SPEC-04) exists alongside the frozen transcript for a reason distinct from redundancy: the transcript proves the captured bytes did not change, while the anchor proves the specific spec-pinned property still holds against a FRESH capture, so a wholesale transcript regeneration (which replaces the golden file's bytes wholesale, D-06) cannot launder a regression in this property past the byte-comparison test. The same reasoning applies to the two `_meta`-failure anchors below.

The four SPEC-09 anchors below (phase 5 plan 01) exist for the identical reason. assertToolsListChangedCapability is registered TWICE — once against handshake-explore's Legacy `initialize` response, once against modern-discover-explore's Modern `server/discover` response — because SPEC-09 criterion 1 says the server advertises `tools.listChanged: true`, not that one negotiation path does; a regression that only broke ONE path would be invisible to an anchor checking only the other. assertSubscriptionAckEcho and assertToolsListChangedNotification exist for the same "fresh capture, not frozen bytes" reason as the cache-control anchor, plus a second one specific to this plan: they are the ONLY mechanism proving the D-02 acknowledgment-echo discriminator and the T-05-02 content-free property stay true going forward — a byte-for-byte transcript match alone cannot distinguish "this property holds" from "this property happened to hold in the one transcript that was frozen."

NOTE (05-01-PLAN Task 2 deviation, Rule 1): the plan's own action text says "Register five Anchor entries" but then describes exactly four — the acknowledgment echo, the notification delivery, and the capability check registered twice (once per negotiation path). Four Anchor entries is what those four descriptions actually produce; a fifth was not separately specified anywhere in the plan (must_haves, behavior, or action), and inventing one would violate this plan's own calibration note ("one wire proof plus its assertions... do not inflate"). Recorded here and in 05-01-SUMMARY.md rather than silently reconciled.

type Rule

type Rule struct {
	Name        string
	Placeholder string
	ExpectFires bool
	Why         string
}

Rule documents one normalization rule in the fixed allowlist below. Name keys the per-capture hit ledger NormalizeWithLedger returns; ExpectFires records whether the tracer capture actually observed this rule fire, so a rule that silently stops matching is distinguishable from a genuine byte diff (TestNormalizeRuleLedgerIsHonest enforces this). A rule with ExpectFires: false must carry a non-empty Why explaining why it is retained anyway.

type Scenario

type Scenario struct {
	// Name identifies the scenario and its frozen transcript file
	// (testdata/wireoracle/transcripts/<Name>.golden).
	Name string
	// Env holds extra "KEY=VALUE" entries appended to the subprocess's
	// environment, alongside the always-on CODEGRAPH_NO_WATCH=1.
	Env []string
	// Index, when true, runs `codegraph init <workDir>` to completion
	// before starting `serve --mcp`.
	Index bool
	// Requests is the ordered list of JSON-RPC requests written to the
	// subprocess's stdin, one per line.
	Requests []map[string]any
	// ExpectTools is the tool count this scenario's session is expected
	// to advertise via VRFY-03's stderr session line.
	ExpectTools int
	// NoInitialize, when true, means this scenario's session never sends an
	// "initialize" request at all — request id 1 is some other method
	// entirely (used by edge-call-before-initialize, the statelessness
	// edge, RESEARCH Pitfall 2: mark3labs v0.56.0 never gates
	// tools/list/tools/call on Initialized()). Two invariants that only
	// make sense downstream of a real initialize handshake are skipped for
	// a NoInitialize scenario — via this field, not a scenario-name
	// special case (01-04-PLAN Task 2): the VRFY-03 stderr session line
	// (its AddAfterInitialize hook never fires with no initialize call to
	// hook) and the D-02 protocolVersion spec anchor (there is no
	// initialize result.protocolVersion field to assert against).
	NoInitialize bool
	// EraScenario, when true, marks this as one of the six-era Legacy
	// handshake baseline scenarios (01-05-PLAN Task 1 checkpoint:
	// six-era). Every OTHER scenario in this package offers the same
	// literal protocol version as internal/mcp.ProtocolVersion
	// (handshakeExploreProtocolVersion), so oracle_test.go's default D-02
	// spec-anchor comparison hardcodes that one shared literal; an era
	// scenario deliberately offers a DIFFERENT revision, so it carries its
	// own expected offered/negotiated pair instead via
	// EraOfferedVersion/EraNegotiatedVersion.
	EraScenario bool
	// EraOfferedVersion is the literal protocolVersion this era
	// scenario's initialize request offers — "" for legacy-omitted-version,
	// whose params carry no protocolVersion key at all (a plain Go string
	// field defaults to "" whether the key is omitted or sent empty; this
	// package always sends the true omitted-key wire shape, never an
	// empty-string literal — see initializeRequestOmittingVersion).
	EraOfferedVersion string
	// EraNegotiatedVersion is what mark3labs v0.56.0 is expected to
	// negotiate for EraOfferedVersion: itself, for the four supported
	// revisions; the server's own latest, for the unsupported revision
	// (silent coercion); or the server's older backwards-compat default,
	// for the omitted-version case. Three structurally distinct outcomes,
	// never assumed equal to EraOfferedVersion except for the four
	// supported revisions (RESEARCH Pitfall 1).
	EraNegotiatedVersion string
	// InitAfterRequest is the 1-based index into Requests after whose
	// RESPONSE HAS BEEN OBSERVED the harness runs `binPath init workDir`
	// against the already-running server's working directory — SPEC-05's
	// index-appears-mid-session proof (03-04-PLAN.md Task 2). Zero (the
	// field's Go zero value) means "never run init mid-session," so every
	// scenario that predates this field is completely unaffected.
	//
	// "Response has been observed" is load-bearing, not a nicety: writing
	// every request up front and running init partway through would be
	// non-deterministic — the later requests would already sit in the
	// subprocess's stdin pipe, and a pre-init tools/list could be
	// serviced by the server before or after init completes depending on
	// scheduling, flipping the frozen bytes run to run. Capture instead
	// blocks, after writing the InitAfterRequest'th request, until THAT
	// request's own response id has been read from stdout (bounded by
	// Capture's existing runCtx deadline) before running init and before
	// writing any further request. Determinism comes from observing the
	// response, never from a sleep.
	InitAfterRequest int
	// AwaitAfterRequest maps a 1-based index into Requests to a JSON-RPC
	// "method" value. After writing that request, Capture blocks until a
	// frame carrying that method has been observed on stdout — before
	// writing the next request, or, when the index names the LAST request,
	// before closing stdin. A nil map (every scenario predating this field)
	// waits for nothing and takes the identical code path it took before
	// this field existed.
	//
	// Two entries are load-bearing for two DIFFERENT reasons (05-01-PLAN,
	// SPEC-09), and both matter for a subscriptions/listen scenario:
	//
	//  1. An entry on the LAST request is what makes a long-lived
	//     notification stream's transcript deterministic rather than
	//     quietly weaker. go-sdk's changeAndNotify debounces a list-changed
	//     notification through a 10ms time.AfterFunc; if Capture closed
	//     stdin immediately after the mutating request instead of waiting
	//     for the notification to actually arrive, process shutdown would
	//     race that timer and could drop the notification from the
	//     transcript entirely — not a loud failure, a silently thinner one.
	//  2. An entry on a request whose response the harness does NOT wait
	//     for via InitAfterRequest exists because go-sdk calls
	//     jsonrpc2.Async for every call except "initialize" — so the
	//     acknowledgment frame a request like subscriptions/listen produces
	//     races the response to whatever request is written after it.
	//     Waiting for the acknowledgment's own method before writing the
	//     next request is what keeps the captured line order deterministic.
	AwaitAfterRequest map[int]string
	// NoResponseRequests holds 1-based indices into Requests whose JSON-RPC
	// id never carries a response line on this transport, at all, when the
	// session ends at stdin EOF.
	//
	// MEASURED (05-01-PLAN, 5/5 isolated runs against a freshly built
	// binary): a subscriptions/listen request's own id-bearing
	// SubscriptionsListenResult is sent ONLY when the server tears the
	// subscription down gracefully — go-sdk's own doc comment on that type
	// says so verbatim — and an abrupt stdio close (stdin EOF) is not that
	// path. This is go-sdk's own documented design, not a codegraph defect
	// and not something to work around: a scenario with a live
	// subscriptions/listen stream must name that request's index here, or
	// Capture's completion condition (and assertFramingInvariant's
	// exactly-one check) would wait forever for a response frame that is
	// never coming.
	//
	// Capture's own completion condition and assertFramingInvariant
	// (oracle_test.go) both read the ONE seam this field feeds —
	// Scenario.expectedResponseIDs() — so the two can never drift apart on
	// what a scenario owes, mirroring the one-seam rule Phase 3 applied to
	// registerTools/unregisterTools.
	NoResponseRequests []int
}

Scenario describes one scripted MCP session the oracle drives against a real spawned binary.

func ScenarioByName

func ScenarioByName(name string) (Scenario, bool)

ScenarioByName returns the named scenario from Scenarios(), or ok=false if no scenario has that name.

func Scenarios

func Scenarios() []Scenario

type Substitutions

type Substitutions struct {
	// RepoDir is the capture-time absolute path of the copied fixture
	// tree (Transcript.RepoDir) — the value the repoDir rule replaces.
	RepoDir string
}

Substitutions carries the capture-time values normalization rules match against — today, only the fixture's absolute repo directory.

type Transcript

type Transcript struct {
	// Stdout is every captured stdout line, newline-joined, RAW (not yet
	// normalized).
	Stdout []byte
	// Stderr is the subprocess's complete captured stderr output.
	Stderr string
	// RepoDir is the workDir the fixture was copied into and the
	// subprocess ran against — callers use it to build Substitutions
	// without recomputing the path.
	RepoDir string
}

Transcript is one capture's raw output.

func Capture

func Capture(ctx context.Context, binPath, fixtureSrc, workDir string, sc Scenario) (Transcript, error)

Capture copies fixtureSrc into workDir, optionally runs `binPath init workDir` (sc.Index), spawns `binPath serve --mcp` with cmd.Dir=workDir, writes sc.Requests to its stdin one at a time (one JSON-marshaled line per request, then closes stdin so the server sees EOF and exits on its own in the common case), and collects every stdout line until one response per request id has been seen, the scanner reaches EOF, or a 30-second deadline fires.

When sc.InitAfterRequest is non-zero, Capture additionally blocks after writing that request until its own response has been observed on stdout, then runs `binPath init workDir` to completion against the already-running server's working directory before writing any further request — see Scenario.InitAfterRequest's doc comment for why this ordering (not a sleep) is what makes the resulting transcript deterministic.

Capture takes no *testing.T and therefore owns its own lifecycle explicitly: it never relies on t.Cleanup, because its second caller (test/wireoracle/cmd/wireoracle) has no *testing.T at all. On every error path and on the deadline path the subprocess is killed; cmd.Wait() always runs before Capture returns, via a deferred call registered immediately after cmd.Start() succeeds, so no subprocess and no stderr-copy goroutine ever outlives this call.

Every captured byte is treated as untrusted display data: Capture never executes or shell-interpolates anything it reads back from the subprocess.

Directories

Path Synopsis
cmd
wireoracle command
Command wireoracle is the human-redirect capture entrypoint (D-03): it spawns one scenario against a caller-supplied binary and fixture, prints ONLY the normalized transcript to stdout, and prints the per-rule hit ledger to stderr.
Command wireoracle is the human-redirect capture entrypoint (D-03): it spawns one scenario against a caller-supplied binary and fixture, prints ONLY the normalized transcript to stdout, and prints the per-rule hit ledger to stderr.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL