conform

type JUnitFailure ¶

type JUnitFailure struct {
	XMLName xml.Name `xml:"failure"`
	Message string   `xml:"message,attr"`
	Type    string   `xml:"type,attr"`
	Body    string   `xml:",chardata"`
}

JUnitFailure holds the failure metadata. Message is the short summary shown in CI dashboards; Type categorises the failure ("Failure" vs "Error" — we always emit "Failure" because conformance is binary). The chardata body carries the detail (stack-trace equivalent).

type JUnitTestCase ¶

type JUnitTestCase struct {
	XMLName   xml.Name      `xml:"testcase"`
	ClassName string        `xml:"classname,attr"`
	Name      string        `xml:"name,attr"`
	Time      string        `xml:"time,attr"`
	Failure   *JUnitFailure `xml:"failure,omitempty"`
}

JUnitTestCase represents a single <testcase> element. ClassName is the suite name (we use "mcp-tui.conform"), Name is the scenario identifier. Time is elapsed-seconds in fixed-point form. Failure is non-nil when the scenario failed; SystemOut carries any captured diagnostic output (skipped for now to keep payloads compact).

type JUnitTestSuite ¶

type JUnitTestSuite struct {
	XMLName  xml.Name        `xml:"testsuite"`
	Name     string          `xml:"name,attr"`
	Tests    int             `xml:"tests,attr"`
	Failures int             `xml:"failures,attr"`
	Errors   int             `xml:"errors,attr"`
	Time     string          `xml:"time,attr"` // seconds, fixed-point — Surefire format
	Cases    []JUnitTestCase `xml:"testcase"`
}

JUnitTestSuite is the top-level <testsuite> element of a JUnit-XML report.

We follow the de-facto Maven Surefire schema (the most widely consumed shape): a testsuite element with attributes for counts and elapsed time, containing testcase elements. Each testcase has classname (the suite), name (the scenario), and time. A failing testcase contains a <failure> element with the failure message and stack/text body.

The struct tags use `xml:"...,attr"` for attributes and `xml:",chardata"` for element bodies. Field order in the struct dictates emission order because encoding/xml marshals fields top-to-bottom.

func BuildJUnitReport(suiteName string, results []ScenarioResult) JUnitTestSuite

type Runner ¶

type Runner struct {
	// contains filtered or unexported fields
}

Runner executes scenarios against a Target. It owns a single mcp.Service that is connected lazily on first protocol-level scenario and reused for the remainder of the run, so a full conform pass against an stdio target pays the spawn-and-handshake cost exactly once.

Verify probes never share Runner.svc — they drive their own per-probe connections so a hung verify doesn't stall the whole suite.

Runner is not safe for concurrent use; the conform suite runs scenarios sequentially by design (so failures are easy to read in a CI log).

func NewRunner(target Target) *Runner

func (r *Runner) Close()

func (r *Runner) Run(ctx context.Context, name string) ScenarioResult

type ScenarioResult ¶

type ScenarioResult struct {
	Name    string        `json:"name"`
	Pass    bool          `json:"pass"`
	Skipped bool          `json:"skipped,omitempty"`
	Error   string        `json:"error,omitempty"`
	Detail  string        `json:"detail,omitempty"`
	Elapsed time.Duration `json:"elapsed"`
}

ScenarioResult is the typed outcome of a single conformance scenario.

Name    — scenario identifier (matches the --scenario flag value)
Pass    — true if the scenario satisfied its acceptance criteria
Skipped — true if the scenario was skipped because the target lacks
          the required capability (counts as Pass for exit-code purposes)
Error   — short, single-line summary shown in the text report
Detail  — multi-line diagnostic body (goes into JUnit <failure>)
Elapsed — wall time spent running the scenario

type Target ¶

type Target struct {
	// URL is the streamable-HTTP endpoint for HTTP-class targets.
	URL string

	// Command + Args are the stdio command for stdio targets.
	Command string
	Args    []string

	// SamplingStub, when non-empty, is installed via
	// sampling.NewTextStubHandler before connect so the sampling scenario
	// (which calls a tool that triggers sampling/createMessage) gets a
	// canned reply.
	SamplingStub string

	// ElicitStub, when non-empty, is installed via
	// elicitation.NewJSONStubHandler before connect so the elicitation
	// scenario gets a canned reply.
	ElicitStub string

	// SamplingTriggerTool overrides the default "sampleLLM" tool name used
	// by the sampling scenario. The scenario calls this tool to provoke a
	// server-initiated sampling/createMessage request.
	SamplingTriggerTool string

	// ElicitTriggerTool overrides the default "startElicitation" tool name
	// used by the elicitation scenario.
	ElicitTriggerTool string

	// CompletionPromptName is the prompt name (or resource template URI when
	// CompletionRefIsResource=true) used to drive completion/complete. When
	// empty the scenario falls back to the first prompt with arguments
	// returned by prompts/list.
	CompletionPromptName    string
	CompletionRefIsResource bool
	CompletionArgumentName  string
	CompletionArgumentValue string
}

Target describes what the conformance suite drives. URL/Command/Args have the same meaning as verify.Target — when both are set, HTTP-class scenarios use URL and stdio-class scenarios use the command (mirroring `verify`).

SamplingStub and ElicitStub are non-empty when the user wants the suite to auto-reply to server-initiated sampling/elicitation requests; the conform runner installs the appropriate stub handler on the connected service before the scenario fires its trigger tool.