Documentation
¶
Overview ¶
Package a2aconform is the shared A2A conformance vector suite: a declarative, transport-neutral corpus that every mapping producing A2A output in this repository must satisfy.
Why this package exists ¶
Two independent mappings turn Nexus activity into A2A frames, and they share only the wire types in pkg/a2a — no mapping logic at all:
- plugins/io/a2a maps the ENGINE BUS (agent.turn.start, tool.result, hitl.requested, ...) onto A2A.
- the session broker maps the BROKER IO ENVELOPE (the flat ioMessage union forwarded over the dial-back WebSocket) onto A2A.
Nothing in the type system makes the two agree. Without a shared corpus they drift silently: one completes a task where the other interrupts it, one puts artifacts before the terminal status and the other after, one closes a stream that the other parks. A client that works against one Nexus deployment then fails against the other, and the failure surfaces as a partner bug report rather than as a test.
So the vectors here describe expected A2A OUTPUT ONLY. A vector never mentions a bus event and never mentions an ioMessage — it names an abstract step ("the agent produced final text", "the agent asked the human a question") and states the exact A2A frame sequence that step must produce. Each mapping supplies a Driver that realizes the abstract steps in its own vocabulary; the runner does the comparing. A vector that can only be satisfied by knowing which mapping produced the frames is a broken vector.
THE RULE ¶
ANY NEW A2A BEHAVIOR MUST ADD A VECTOR HERE BEFORE IT IS IMPLEMENTED IN A SECOND MAPPING.
Building a second mapping first and back-filling vectors afterwards is exactly the failure this package exists to prevent: the back-filled vector is written from the second mapping's observed output, so it encodes the drift instead of catching it. Add the vector, watch it fail, then implement.
Corollary: if a mapping cannot satisfy a vector, do not weaken the vector. Either the mapping has a bug, or the vector encodes a wrong expectation and the fix belongs in the vector WITH a rationale saying why the old expectation was wrong. Silently relaxing an expectation to make a run green removes the only thing keeping the two mappings honest.
Why the corpus lives here and not with either mapping ¶
It cannot live in plugins/io/a2a: that is one of the two mappings under test, and a shared oracle owned by one participant is not shared. It cannot live in cmd/nexus-broker either: that is a main package, so nothing can import it, and it is the other participant besides. pkg/a2a is the one thing both mappings already depend on, which makes a sub-package of it the only home that both can reach without either depending on the other. The layering mirrors pkg/agui/aguiclient.
This package must never import a mapping. It depends on pkg/a2a, the standard library and nothing else.
Corpus format ¶
The vectors are JSON documents under vectors/, embedded at build time. They are data rather than Go code on purpose: adding a vector is a data-only change, the corpus is trivially readable by a reviewer who does not read Go, and a JSON document structurally cannot reference either mapping's types. Decoding is strict (unknown fields are an error), so a typo in a vector is a build-time failure rather than an expectation that silently never runs.
What a vector asserts ¶
Vectors pin the CANONICAL A2A frame stream — the frames an ordinary client sees. Nexus extension telemetry (see a2a.NexusExtensionURI) is deliberately out of scope: it is delivered only to clients that opted in with A2A-Extensions, so it is not part of the output a conforming A2A client is entitled to, and pinning it here would make the corpus mapping-specific. A driver must attach as a NON-extension observer.
Beyond an exact frame-by-frame comparison, the runner independently replays the observed frames through a2a.SSEWriter, so every vector also asserts the stream contract of specification section 11.7: the stream opens with a Task, task and context identity stay stable, every status transition is legal, and nothing is written after a terminal frame.
Deviations this corpus deliberately pins ¶
The effort made four decisions that read as spec deviations at a glance. They are encoded here precisely so that a second implementer cannot "fix" them into divergence:
- STATUS AND ARTIFACT FRAMES INTERLEAVE after the opening frame. A strict reading of section 11.7 ("all status updates, then all artifact updates") is not implementable, because the COMPLETED frame that closes a stream necessarily follows the artifacts whose production it completes. Vectors assert Expect.StatusFollowsArtifact to hold this open.
- STREAMS PARK, THEY DO NOT CLOSE, on a non-terminal INPUT_REQUIRED. Closing on a non-terminal state is indistinguishable client-side from a dropped connection. Vector hitl-parks-stream-open pins the stream staying open, and pins a2a.SSEWriter.Interrupted reporting the parked condition.
- ARTIFACTS PRECEDE THE TERMINAL STATUS FRAME. Section 11.7 closes a stream the moment a terminal state is reported, so an artifact emitted after COMPLETED would be refused and the client would see a finished task with no output. Vectors assert Expect.ArtifactsPrecedeTerminal.
- THE TASK COMPLETES AT THE END OF THE AGENT TURN, not at the raw terminal model response. Nexus publishes output and runs its output gates between the two, so completing at the model response would publish the model's draft rather than what Nexus decided to say. Vectors encode this with an assert_active step placed between the agent_text step and the output/ turn_end steps.
Index ¶
- Constants
- func Check(v Vector, obs Observation) []error
- func Run(t *testing.T, d Driver)
- func RunVector(t *testing.T, d Driver, v Vector)
- type Driver
- type Env
- type Expect
- type Feature
- type FrameExpect
- type Observation
- type PartExpect
- type Policy
- type Session
- type Step
- type StepChoice
- type StepFile
- type StepKind
- type Vector
Constants ¶
const ( FrameTask = "task" FrameStatus = "status" FrameArtifact = "artifact" FrameMessage = "message" )
Frame kinds a FrameExpect may name.
Variables ¶
This section is empty.
Functions ¶
func Check ¶
func Check(v Vector, obs Observation) []error
Check compares one observation against one vector and returns every way they disagree.
It is a PURE function of the vector and the observation, with no *testing.T and no driver, for two reasons. It lets a test feed it a deliberately-wrong observation and assert that the disagreement is caught, which is the only way to know this harness is not vacuously green. And it keeps the whole oracle in one place, so a second mapping is judged by exactly the same code as the first rather than by a second copy of the rules that can drift.
func Run ¶
Run replays the whole corpus against one mapping.
Each vector is a subtest named for its id, so a failure names the behaviour that drifted rather than a line number. Vectors the driver has not declared the features for are skipped and reported together at the end: a mapping's conformance claim is "these vectors passed AND these were not exercised", and hiding the second half would let a mapping claim conformance it does not have.
func RunVector ¶
RunVector replays one vector against one mapping and reports every disagreement.
Every difference is reported rather than only the first: a mapping that drifted usually drifted in more than one place, and fixing them one test run at a time is how a small divergence becomes a long afternoon.
Types ¶
type Driver ¶
type Driver interface {
// Name identifies the mapping in failure messages.
Name() string
// Features declares what this mapping can express. Vectors requiring
// anything absent are skipped, and the skips are reported.
Features() []Feature
// Begin prepares one vector's scenario. It must NOT start the task: the
// vector's first step does that, so the opening frame is produced by the
// mapping rather than by the harness.
Begin(t *testing.T, v Vector, env Env) (Session, error)
}
Driver is one mapping under test. It is implemented in the mapping's own package, where its internals are reachable; this package never imports a mapping.
type Env ¶
type Env struct {
// FileDir is the directory the runner materializes StepFile paths into. A
// mapping that publishes file artifacts must resolve reported paths against
// it. Fresh per vector.
FileDir string
// Policy is the artifact bound this vector runs under. Nil fields mean the
// mapping's own default.
Policy Policy
}
Env is what the runner hands a driver for one vector.
type Expect ¶
type Expect struct {
// Frames is the exact frame sequence, compared position by position. Exact
// comparison is the point: two mappings replaying one scenario must produce
// the SAME stream, not merely compatible ones.
Frames []FrameExpect `json:"frames"`
// FinalState is the task's state once the scenario has been replayed, as the
// mapping's own task snapshot reports it.
FinalState a2a.TaskState `json:"finalState"`
// StreamClosed is whether the frame sequence closes the stream under
// specification section 11.7's terminal-close rule. It is always asserted:
// false is the meaningful expectation for a parked task.
StreamClosed bool `json:"streamClosed"`
// StreamInterrupted is whether the stream ends parked at an interrupted
// state. Always asserted, for the same reason.
StreamInterrupted bool `json:"streamInterrupted"`
// ArtifactsPrecedeTerminal requires every artifact frame to arrive before
// the terminal status frame. False means "not asserted by this vector".
ArtifactsPrecedeTerminal bool `json:"artifactsPrecedeTerminal,omitempty"`
// StatusFollowsArtifact requires at least one status frame AFTER an artifact
// frame, which is what pins the interleaving deviation open. False means
// "not asserted by this vector".
StatusFollowsArtifact bool `json:"statusFollowsArtifact,omitempty"`
}
Expect is the A2A output a vector's scenario must produce.
type Feature ¶
type Feature string
Feature is a capability a mapping either has or does not. It exists so a mapping that genuinely cannot express part of the model — the broker's IO envelope carries no tool results, so it can produce no tool-result artifacts — declares that ONCE, visibly, instead of each vector growing a special case.
A missing feature skips a vector and the skip is reported. It is not an escape hatch: a mapping that CAN do something must declare the feature and pass the vectors, and a mapping that later gains a capability must add the feature and satisfy them.
const ( // FeatureTurn is the baseline: a message drives a turn that produces text. // Every mapping has it. FeatureTurn Feature = "turn" // FeatureFailure is rendering an unrecoverable error as TASK_STATE_FAILED. FeatureFailure Feature = "failure" // FeatureCancel is client-initiated cancellation to TASK_STATE_CANCELED. FeatureCancel Feature = "cancel" // FeatureHITL is parking at TASK_STATE_INPUT_REQUIRED and resuming. FeatureHITL Feature = "hitl" // FeatureToolArtifacts is publishing tool results as artifacts. FeatureToolArtifacts Feature = "tool_artifacts" // FeatureFileArtifacts is publishing files a turn wrote as artifacts, // including the oversized-file degradation. FeatureFileArtifacts Feature = "file_artifacts" // FeatureArtifactBudget is the per-task artifact ceiling and its suppression // notice. FeatureArtifactBudget Feature = "artifact_budget" )
The capability vocabulary.
type FrameExpect ¶
type FrameExpect struct {
// Kind is one of task, status, artifact, message.
Kind string `json:"kind"`
// Note is a human comment. The runner ignores it.
Note string `json:"note,omitempty"`
// State is the task state a task or status frame reports.
State a2a.TaskState `json:"state,omitempty"`
// MessageRole, when set, requires a status message with that role.
MessageRole a2a.Role `json:"messageRole,omitempty"`
// MessageContains are substrings the status message's text must contain.
MessageContains []string `json:"messageContains,omitempty"`
// MessageMetadata are key/value pairs the status message's metadata must
// carry.
MessageMetadata map[string]any `json:"messageMetadata,omitempty"`
// NoMessage requires the status to carry no message at all.
NoMessage bool `json:"noMessage,omitempty"`
// ArtifactID is the artifact's exact id.
ArtifactID string `json:"artifactId,omitempty"`
// ArtifactName is the artifact's human-readable label.
ArtifactName string `json:"artifactName,omitempty"`
// LastChunk and Append pin the chunking flags when a vector cares.
LastChunk *bool `json:"lastChunk,omitempty"`
Append *bool `json:"append,omitempty"`
// Parts is the artifact's part list, compared position by position.
Parts []PartExpect `json:"parts,omitempty"`
// Metadata are key/value pairs the artifact metadata must carry. Comparison
// is by JSON value, so 4096 matches whether the mapping produced an int, an
// int64 or a float64.
Metadata map[string]any `json:"metadata,omitempty"`
// MetadataAbsent are keys the artifact metadata must NOT carry.
MetadataAbsent []string `json:"metadataAbsent,omitempty"`
}
FrameExpect is one expected A2A stream frame.
String fields and Contains entries may use the placeholders {taskId} and {contextId}, since both are minted by the mapping at runtime and cannot be written into a static corpus.
type Observation ¶
type Observation struct {
// Frames is the canonical frame stream, in order.
Frames []a2a.StreamResponse
// Task is the mapping's final task snapshot.
Task a2a.Task
// Marks records, per step index, how many frames had been observed when the
// runner reached that step. Only assertion steps have an entry.
Marks map[int]int
}
Observation is everything the runner collected from one replay. It is split out from the checking so Check is a pure function that a test can feed a deliberately-wrong observation to — which is how this harness proves it is not vacuous.
type PartExpect ¶
type PartExpect struct {
// Kind is the part's content arm: text, raw, url or data.
Kind a2a.PartKind `json:"kind"`
// MediaType is the part's declared media type.
MediaType string `json:"mediaType,omitempty"`
// Filename is the part's declared filename.
Filename string `json:"filename,omitempty"`
// Contains are substrings the text content must contain.
Contains []string `json:"contains,omitempty"`
// Equals is the exact text content.
Equals *string `json:"equals,omitempty"`
// JSONEquals is the exact structured content, compared as JSON values.
JSONEquals json.RawMessage `json:"jsonEquals,omitempty"`
// Bytes is the exact length of a raw part's content.
Bytes *int `json:"bytes,omitempty"`
}
PartExpect is one expected Part of an artifact.
type Policy ¶
type Policy struct {
// MaxFileBytes caps the inline content of one file artifact. A file over it
// must degrade to a metadata note rather than being dropped or inlined.
MaxFileBytes *int `json:"maxFileBytes,omitempty"`
// MaxToolOutputBytes caps one tool-result artifact's text. Zero disables it.
MaxToolOutputBytes *int `json:"maxToolOutputBytes,omitempty"`
// MaxTaskBytes is one task's total artifact budget, excluding its final
// response artifact. Zero disables it.
MaxTaskBytes *int `json:"maxTaskBytes,omitempty"`
}
Policy is the artifact bound a vector runs under. Nil fields keep the mapping's own default, so a vector states only the knob it is about.
type Session ¶
type Session interface {
// Apply realizes one abstract step. Assertion steps never reach it.
Apply(step Step) error
// Frames returns every canonical frame produced so far, in order.
Frames() []a2a.StreamResponse
// Task returns the mapping's own current task snapshot, which is what a
// non-streaming client would be answered with.
Task() a2a.Task
// Close releases whatever Begin acquired. It must be safe to call on a task
// that never reached a terminal state.
Close()
}
Session is one vector's replay against one mapping.
Frames must return the CANONICAL frame stream a non-extension A2A client would see, opening Task snapshot included as the first element. Apply must be synchronous: when it returns, every frame the step caused must already be visible to Frames.
type Step ¶
type Step struct {
Kind StepKind `json:"kind"`
// Note is a human comment. The runner ignores it.
Note string `json:"note,omitempty"`
// Text carries the message text (message, answer), the assistant text
// (agent_text, output) or the reasoning text (thinking).
Text string `json:"text,omitempty"`
// ContextID is the A2A context the creating message names.
ContextID string `json:"contextId,omitempty"`
// MessageID is the client's message id.
MessageID string `json:"messageId,omitempty"`
// TurnID correlates turn_start with turn_end.
TurnID string `json:"turnId,omitempty"`
// CallID and Tool identify a tool call and its result.
CallID string `json:"callId,omitempty"`
Tool string `json:"tool,omitempty"`
// Arguments are the tool call's arguments.
Arguments map[string]any `json:"arguments,omitempty"`
// Output is the tool's text output.
Output string `json:"output,omitempty"`
// OutputRepeat renders Output repeated n times, so a vector can express a
// large payload without embedding one.
OutputRepeat int `json:"outputRepeat,omitempty"`
// Structured is the tool's typed result.
Structured map[string]any `json:"structured,omitempty"`
// Error is the tool's failure text.
Error string `json:"error,omitempty"`
// File is a file the tool wrote. The runner materializes it under
// Env.FileDir before the step is applied, so a driver only has to report it.
File *StepFile `json:"file,omitempty"`
// RequestID correlates ask_user with the answer that resolves it.
RequestID string `json:"requestId,omitempty"`
// Prompt is the question text.
Prompt string `json:"prompt,omitempty"`
// Choices are the options a multiple-choice question offers.
Choices []StepChoice `json:"choices,omitempty"`
// Reason is the failure or cancellation reason.
Reason string `json:"reason,omitempty"`
}
Step is one entry in a vector's scenario. Only the fields relevant to Kind are populated; it is a flat union for the same reason the broker's IO envelope is.
type StepChoice ¶
StepChoice is one option of a multiple-choice question.
type StepFile ¶
type StepFile struct {
// Path is relative to Env.FileDir.
Path string `json:"path"`
// Bytes is the file's size. The content is filler; no vector asserts it.
Bytes int `json:"bytes"`
}
StepFile is a file the scenario says a tool wrote.
type StepKind ¶
type StepKind string
StepKind names one abstract thing that happens during a turn.
The vocabulary is deliberately at the level of what the AGENT DID, not at the level of any transport's payloads: every kind below has an obvious realization both from engine bus events and from the broker's flat IO envelope.
const ( // StepMessage is the client sending the message that creates the task. StepMessage StepKind = "message" // StepTurnStart is the agent turn beginning. StepTurnStart StepKind = "turn_start" // StepThinking is one reasoning step. StepThinking StepKind = "thinking" // StepToolCall is the agent deciding to invoke a tool. StepToolCall StepKind = "tool_call" // StepToolResult is a tool reporting its outcome, optionally naming a file // it wrote. StepToolResult StepKind = "tool_result" // StepAgentText is the model producing the turn's final text. It is NOT the // end of the task: see StepOutput and StepTurnEnd. StepAgentText StepKind = "agent_text" // StepOutput is the text the agent actually published, after whatever // output filtering the host applies. StepOutput StepKind = "output" // StepAskUser is the agent asking the human a question. StepAskUser StepKind = "ask_user" // StepAnswer is the client answering that question, resuming the task. StepAnswer StepKind = "answer" // StepTurnEnd is the agent turn ending, which is what completes the task. StepTurnEnd StepKind = "turn_end" // StepFailure is an unrecoverable error ending the turn. StepFailure StepKind = "failure" // StepCancel is the client canceling the task. StepCancel StepKind = "cancel" // StepAssertActive asserts that no terminal frame has been produced yet. It // is evaluated by the runner and never reaches a driver. StepAssertActive StepKind = "assert_active" // StepAssertParked asserts the task is currently interrupted at // INPUT_REQUIRED and the stream is still open. StepAssertParked StepKind = "assert_parked" )
The step vocabulary.
type Vector ¶
type Vector struct {
// ID names the vector and is the subtest name. Unique across the corpus.
ID string `json:"id"`
// Title is a one-line human summary.
Title string `json:"title"`
// Rationale says why the expectation is what it is. It is the field a future
// implementer reads before deciding a vector is "wrong", so it is required.
Rationale string `json:"rationale"`
// Features are the capabilities a mapping must have to replay this vector. A
// driver declaring fewer skips it, visibly.
Features []Feature `json:"features"`
// Policy is the artifact policy the mapping must apply for this vector. It
// is transport-neutral: it describes the A2A artifact bound, not how any
// mapping is configured. Nil means the mapping's defaults.
Policy *Policy `json:"policy,omitempty"`
// Steps are the abstract scenario, applied in order.
Steps []Step `json:"steps"`
// Expect is the A2A output the scenario must produce.
Expect Expect `json:"expect"`
}
Vector is one conformance case: an abstract scenario plus the exact A2A output any mapping replaying that scenario must produce.