Documentation
¶
Overview ¶
Package conformance hosts the bundled frontend × backend conformance matrix (spec tasks 12.x). Tests drive official reference clients against frontend handlers, the core executor, and official backend connectors backed by reference backend emulators or protocol-faithful error stubs.
The generic cell selector (DeploymentSpec/Deploy) and the TestCellSelect_SmokeScaffold_* tests are Phase 7 SMOKE SCAFFOLDING: they prove the reusable harness composes any authoritative cell with contract-fake origins and injectable failure modes. They are NOT Phase 8 compatibility evidence — Phase 8 replaces the contract-fake origins with the independent OpenResponses refbackend/refclient emulators and certifies every cell with official-wire scenarios (tasks 8.1–8.5).
Package conformance drives end-to-end matrix tests. Upstream error-shape checks use NewUpstream400Server for minimal JSON bodies that match each provider family’s typical 400 invalid_request patterns (aligned with internal/refbackend error shapes where those emulators return structured errors). Success paths use internal/refbackend via NewSuccessRefBackend.
Index ¶
- Constants
- Variables
- func AllBundledProtocolIDs() []string
- func BackendFor(tb testing.TB, backendID, upstreamBaseURL string, httpClient *http.Client) execbackend.Backend
- func BackendForDualCredential(tb testing.TB, backendID, upstreamBaseURL string, httpClient *http.Client) execbackend.Backend
- func BundledBackendIDs() []string
- func BundledFrontendIDs() []string
- func ConnectorColumnRouteSelector(backendID string) string
- func DefaultModel(backendID string) string
- func GeminiConformanceBaseURL(proxyOrigin string) string
- func GenAITestCtx() context.Context
- func HarnessBackendIDs() []string
- func HarnessFrontendIDs() []string
- func MountFrontend(mux *http.ServeMux, frontendID string, exec *runtime.Executor, ...) error
- func NewSuccessRefBackend(tb testing.TB, backendID string, onRequestBody func([]byte)) *httptest.Server
- func NewTestExecutor(tb testing.TB, backendID, upstreamBaseURL string, httpClient *http.Client) *runtime.Executor
- func NewTestExecutorDualCredential(tb testing.TB, backendID, upstreamBaseURL string, httpClient *http.Client) *runtime.Executor
- func NewToolCallRepairRefBackend(tb testing.TB, backendID string) *httptest.Server
- func NewToolRefBackend(tb testing.TB, backendID string, onBody func([]byte)) *httptest.Server
- func NewUpstream400Server(tb testing.TB, backendID string) *httptest.Server
- func RouteSelector(backendID, model string) string
- func WantRepairedToolArgsJSON(backendID string) string
- func WantSafeTailPendingArgsJSON(backendID string) string
- type Candidate
- type ClientEntrypoint
- type ClientTransport
- type Deployment
- func Deploy(tb testing.TB, spec DeploymentSpec) *Deployment
- func DeployConnectorColumn(tb testing.TB, backendID string, transport ClientTransport) *Deployment
- func DeployConnectorColumnFor(tb testing.TB, frontend, backendID string, transport ClientTransport) *Deployment
- func DeployConnectorColumnWithFail(tb testing.TB, frontend, backendID string, transport ClientTransport, ...) *Deployment
- func DeployConnectorColumnWithOrigin(tb testing.TB, frontend, backendID string, transport ClientTransport, ...) *Deployment
- func (d *Deployment) Backend(backendID string) execbackend.Backend
- func (d *Deployment) BaseURL() string
- func (d *Deployment) CandidateOrigin(i int) *Origin
- func (d *Deployment) Close() error
- func (d *Deployment) FrontendAddr() string
- func (d *Deployment) OriginFor(backendID string) *Origin
- func (d *Deployment) RawFrontendPost(ctx context.Context, path, rawBody string) (int, error)
- func (d *Deployment) RequestCount(backendID string) int
- func (d *Deployment) RoundTripModel(ctx context.Context, model, prompt string) error
- func (d *Deployment) SendRawCompact(ctx context.Context, rawBody string) error
- func (d *Deployment) SendRawCreate(ctx context.Context, rawBody string) error
- func (d *Deployment) SendRawWSTurn(ctx context.Context, rawTurn string) error
- type DeploymentSpec
- type Origin
- type OriginFailMode
- type RoundTripResult
- type SentinelCase
Constants ¶
const ( FrontendOpenAIResponses = "openai-responses" FrontendOpenAILegacy = "openai-legacy" FrontendAnthropic = "anthropic" FrontendGemini = "gemini" FrontendOpenResponses = "openresponses" )
Authoritative harness frontend identities. The set is a superset of the locked baseline matrix (BundledFrontendIDs) and adds the OpenResponses frontend.
const ( BackendOpenAIResponses = "openai-responses" BackendOpenAILegacy = "openai-legacy" BackendAnthropic = "anthropic" BackendGemini = "gemini" BackendBedrock = "bedrock" BackendACP = "acp" BackendOpenRouter = "openrouter" BackendNVIDIA = "nvidia" BackendOpenResponses = "openresponses" BackendCompatibleOpenAI = standardplugins.CustomOpenAIResponsesCompatibleID )
Authoritative harness backend identities. openrouter and nvidia are provider connector columns that are not constructible in the base essential bundle; the generic selector fails them closed and the OpenResponses row (Task 8.3) proves their actual route through the real connector executables (DeployConnectorColumnFor / connector_host.go) without promoting the connectors to essential status.
const ConnectorColumnModel = "gpt-4o-mini"
ConnectorColumnModel is the canonical model used by connector-column deployments (OpenRouter/NVIDIA). The value is shared with the route-selector the frontend default routes to.
const (
HarnessFakeText = harnessFakeText
)
harnessFakeText is the deterministic assistant text emitted by harness contract-fake reference-provider origins. HarnessFakeText is the exported form for integration packages.
Variables ¶
var ExpectedMigrationGoldenJSON = []string{
"python_lip_anthropic_messages_nonstream.json",
"python_lip_openai_responses_http_nonstream.json",
"python_lip_openai_responses_http_streaming.json",
}
ExpectedMigrationGoldenJSON lists migration parity JSON under testdata/migration/ (Req. 15.13). Keep in sync with docs/release-gates.md.
var ParityProtocolEvidence = map[string][]string{
"openai-responses": {"parity_openai_test.go"},
"openai-legacy": {"parity_openai_test.go"},
"anthropic": {"parity_anthropic_test.go"},
"gemini": {"parity_gemini_test.go"},
"bedrock": {"parity_bedrock_test.go"},
"acp": {"connector_host_test.go"},
"openresponses": {"sentinel_test.go"},
"openrouter": {"sentinel_test.go"},
"nvidia": {"sentinel_test.go"},
}
ParityProtocolEvidence maps every bundled frontend/backend protocol id to at least one parity suite source file in this package (llm-api-parity P5.1). Shared families may list the same file for multiple ids. OpenResponses evidence spans the explicit frontend row and backend column conformance suites; the OpenRouter/NVIDIA compatibility identities reuse the configured provider-mode row proof.
var ParitySuiteGoFiles = []string{
"parity_openai_test.go",
"parity_anthropic_test.go",
"parity_gemini_test.go",
"parity_bedrock_test.go",
"connector_host_test.go",
}
ParitySuiteGoFiles lists parity suite sources (.kiro/specs/archive/llm-api-parity/tasks.md Phase 5). Keep in sync with testdata/migration/README.md and docs/release-gates.md.
Functions ¶
func AllBundledProtocolIDs ¶
func AllBundledProtocolIDs() []string
AllBundledProtocolIDs returns the sorted union of bundled frontend and backend protocol ids.
func BackendFor ¶
func BackendFor(tb testing.TB, backendID, upstreamBaseURL string, httpClient *http.Client) execbackend.Backend
BackendFor returns the bundled execbackend.Backend for upstreamBaseURL (httptest origin or /v1 layout per plugin).
func BackendForDualCredential ¶
func BackendForDualCredential(tb testing.TB, backendID, upstreamBaseURL string, httpClient *http.Client) execbackend.Backend
BackendForDualCredential is like BackendFor but supplies a second synthetic key for hosted providers that support credential pools. Reference backends accept any non-empty key material used by conformance clients (sk-test, synthetic Anthropic key, fake Gemini key).
func BundledBackendIDs ¶
func BundledBackendIDs() []string
func BundledFrontendIDs ¶
func BundledFrontendIDs() []string
Protocol identities support independent parity discovery. They are not paired into a mandatory frontend-by-backend completeness list.
func ConnectorColumnRouteSelector ¶
ConnectorColumnRouteSelector returns the route selector a connector-column deployment resolves to (backendID:ConnectorColumnModel).
func DefaultModel ¶
DefaultModel returns the model name wired into routing.AttemptCandidate for a bundled backend ID.
func GeminiConformanceBaseURL ¶
GeminiConformanceBaseURL is the genai client base URL for conformance tests against the bundled Gemini frontend (mounted under /v1beta/ and /v1beta1/).
func GenAITestCtx ¶
GenAITestCtx is a shared background context for genai client construction in tests.
func HarnessBackendIDs ¶
func HarnessBackendIDs() []string
HarnessBackendIDs returns the deterministic authoritative harness backend list (the 6 essential backends, the OpenResponses backend, and the provider-connector columns openrouter/nvidia).
func HarnessFrontendIDs ¶
func HarnessFrontendIDs() []string
HarnessFrontendIDs returns the deterministic authoritative harness frontend list (5 bundled real frontends including OpenResponses).
func MountFrontend ¶
func MountFrontend(mux *http.ServeMux, frontendID string, exec *runtime.Executor, routeSelector string) error
MountFrontend registers the bundled frontend handler on mux for conformance tests.
func NewSuccessRefBackend ¶
func NewSuccessRefBackend(tb testing.TB, backendID string, onRequestBody func([]byte)) *httptest.Server
NewSuccessRefBackend returns a reference backend whose streaming and non-streaming paths both surface parityText as assistant text (ACP keeps the stock emulator which answers "ok"). Optional onRequestBody observes the raw upstream HTTP body after route/auth checks.
func NewTestExecutor ¶
func NewTestExecutor(tb testing.TB, backendID, upstreamBaseURL string, httpClient *http.Client) *runtime.Executor
NewTestExecutor wires a single backend against refBackendURL (or error injection URL) for conformance cells.
func NewTestExecutorDualCredential ¶
func NewTestExecutorDualCredential(tb testing.TB, backendID, upstreamBaseURL string, httpClient *http.Client) *runtime.Executor
NewTestExecutorDualCredential wires hosted OpenAI, Anthropic, and Gemini backends with two ordered API keys so credential pools are populated. Bedrock and ACP use the same construction as NewTestExecutor (no multi-key pool in this harness).
func NewToolCallRepairRefBackend ¶
NewToolCallRepairRefBackend wires a reference backend that emits intentionally terminal-comma tool-call argument JSON so canonical safe-tail repair must rewrite. Safe-tail engine/runtime and local-dogfood suites cover the schema-determined pending-value class; protocol adapters remain unaware of both repair classes.
func NewToolRefBackend ¶
NewToolRefBackend wires a reference backend emulator that completes a single tool call and includes usage fields where the wire shape supports it (Task 12.2).
func NewUpstream400Server ¶
NewUpstream400Server returns an httptest server that answers like a provider HTTP 400 for the given bundled backend wire family (Task 12.1 protocol-valid upstream error shape).
func RouteSelector ¶
RouteSelector builds a core routing selector primary for a single-backend executor.
func WantRepairedToolArgsJSON ¶
WantRepairedToolArgsJSON is the closed JSON expected after syntax repair for NewToolCallRepairRefBackend streams (gemini is skipped; see helper).
Types ¶
type Candidate ¶
type Candidate struct {
// Backend is an authoritative backend ID.
Backend string
// ProfileID selects a validated profile when Backend is a compatible family.
ProfileID string
// OriginFail selects a deterministic failure mode for the candidate origin.
OriginFail OriginFailMode
// ProviderOrigin injects an external reference-provider origin base URL.
ProviderOrigin string
}
Candidate is one additional real backend candidate in the deployment's failover chain, with its own injectable contract-fake origin.
type ClientEntrypoint ¶
type ClientEntrypoint interface {
// RoundTrip drives one create/turn and returns the deterministic result.
RoundTrip(ctx context.Context, prompt string) (RoundTripResult, error)
// Close releases any client-held resources (e.g. WebSocket connections).
Close() error
}
ClientEntrypoint is the configurable client seam of the deployment harness. The OpenResponses family uses the harness raw wire clients (JSON/SSE/compact/ WebSocket); existing families delegate to the repository independent reference clients. The Phase 8 independent OpenResponses refclient can implement this contract and drive the same deployment.
type ClientTransport ¶
type ClientTransport string
ClientTransport selects the client entrypoint used to drive one deployment.
const ( TransportJSON ClientTransport = "json" TransportSSE ClientTransport = "sse" TransportCompact ClientTransport = "compact" TransportWebSocket ClientTransport = "websocket" )
type Deployment ¶
type Deployment struct {
Spec DeploymentSpec
// Exec is the wired real core executor.
Exec *runtime.Executor
// RouteSelector is the primary route selector the frontend default routes to.
RouteSelector string
// Mux is the composed frontend mux.
Mux *http.ServeMux
// Server is the full-path frontend origin.
Server *httptest.Server
// Client is the transport-selected client entrypoint.
Client ClientEntrypoint
// Clock is the injected virtual clock (nil when not injected).
Clock testkitopenresponses.VirtualClock
// contains filtered or unexported fields
}
Deployment is one deterministic full-path deployment: a configurable client entrypoint in front of a real frontend handler, the real core executor, real backend adapter(s), and injectable reference-provider origins.
func Deploy ¶
func Deploy(tb testing.TB, spec DeploymentSpec) *Deployment
Deploy composes one full-path deployment from a single generic cell selector. It returns nil for invalid or not-yet-constructible cells without starting any server, origin, or port.
func DeployConnectorColumn ¶
func DeployConnectorColumn(tb testing.TB, backendID string, transport ClientTransport) *Deployment
DeployConnectorColumn deploys the OpenResponses frontend over the actual connectors/openrouter or connectors/nvidia executable for the connector-column evidence identity. The connector process is configured with a fresh origin that emulates an OpenAI-compatible provider endpoint; the observing origin counts and redacts every request so tests can prove the actual connector route reaches the provider endpoint. backendID must be one of BackendOpenRouter / BackendNVIDIA.
func DeployConnectorColumnFor ¶
func DeployConnectorColumnFor(tb testing.TB, frontend, backendID string, transport ClientTransport) *Deployment
DeployConnectorColumnFor deploys an arbitrary bundled frontend over the actual OpenRouter/NVIDIA connector executable for the connector-column evidence identities. frontend must be a bundled frontend ID; only frontends that produce an OpenAI Responses operation (openresponses, openai-responses) can round-trip the Responses-wire connector; other frontend operations fail closed before any upstream request.
func DeployConnectorColumnWithFail ¶
func DeployConnectorColumnWithFail(tb testing.TB, frontend, backendID string, transport ClientTransport, fail OriginFailMode) *Deployment
DeployConnectorColumnWithFail deploys a connector-column cell whose observing origin injects a deterministic failure mode (e.g. OriginFailUnauthorized).
func DeployConnectorColumnWithOrigin ¶
func DeployConnectorColumnWithOrigin(tb testing.TB, frontend, backendID string, transport ClientTransport, originHandler http.Handler) *Deployment
DeployConnectorColumnWithOrigin deploys a connector-column cell with a custom observing-origin responder (nil keeps the default connectorColumnOrigin). Custom origins let evidence assert on the real request headers/body the connector process sends.
func (*Deployment) Backend ¶
func (d *Deployment) Backend(backendID string) execbackend.Backend
Backend returns the constructed backend for a backend slot. It exposes the host-built connector backend so evidence can drive connector-specific surfaces (for example dynamic model inventory) that only the real connector exposes.
func (*Deployment) BaseURL ¶
func (d *Deployment) BaseURL() string
BaseURL returns the full-path frontend origin base URL.
func (*Deployment) CandidateOrigin ¶
func (d *Deployment) CandidateOrigin(i int) *Origin
CandidateOrigin returns the i-th candidate origin.
func (*Deployment) Close ¶
func (d *Deployment) Close() error
Close releases every owned resource deterministically: the frontend listener, all reference-provider origins, and the frontend generation context. It is idempotent.
func (*Deployment) FrontendAddr ¶
func (d *Deployment) FrontendAddr() string
FrontendAddr returns the frontend origin listen address (host:port).
func (*Deployment) OriginFor ¶
func (d *Deployment) OriginFor(backendID string) *Origin
OriginFor returns the primary contract-fake origin for backendID.
func (*Deployment) RawFrontendPost ¶
RawFrontendPost posts rawBody to an arbitrary frontend path and returns the HTTP status without fataling, so fail-closed cells can be asserted cleanly.
func (*Deployment) RequestCount ¶
func (d *Deployment) RequestCount(backendID string) int
RequestCount returns the number of upstream requests observed by the primary contract-fake origin for backendID.
func (*Deployment) RoundTripModel ¶
func (d *Deployment) RoundTripModel(ctx context.Context, model, prompt string) error
RoundTripModel performs one JSON create round trip with an explicit model, for pre-network rejection and unroutable-model proofs.
func (*Deployment) SendRawCompact ¶
func (d *Deployment) SendRawCompact(ctx context.Context, rawBody string) error
SendRawCompact posts an arbitrary JSON compact body to the OpenResponses frontend and returns an error when the frontend does not accept it.
func (*Deployment) SendRawCreate ¶
func (d *Deployment) SendRawCreate(ctx context.Context, rawBody string) error
SendRawCreate posts an arbitrary JSON create body to the OpenResponses frontend and returns an error when the frontend does not accept it.
func (*Deployment) SendRawWSTurn ¶
func (d *Deployment) SendRawWSTurn(ctx context.Context, rawTurn string) error
SendRawWSTurn dials one WebSocket connection, sends an arbitrary turn frame, and returns an error when the session rejects the turn (classified error).
type DeploymentSpec ¶
type DeploymentSpec struct {
// Frontend is an authoritative frontend ID (HarnessFrontendIDs).
Frontend string
// Backend is an authoritative backend ID (HarnessBackendIDs).
Backend string
// Model is the canonical model used by the client; empty uses the harness
// default for Backend.
Model string
// Transport selects the client entrypoint (json/sse/compact/websocket).
// Empty defaults to json.
Transport ClientTransport
// ProviderOrigin injects an external reference-provider origin base URL for
// the primary backend (existing reference families, Phase 8 refbackend).
// Empty deploys a harness contract-fake origin.
ProviderOrigin string
// ProfileID selects a validated provider profile for a compatible family.
ProfileID string
// ProviderClient is the HTTP client used by the real backend to reach the
// origin; empty uses the origin loopback client.
ProviderClient *http.Client
// OriginFail selects a deterministic contract-fake origin failure mode for
// the primary backend.
OriginFail OriginFailMode
// Candidates appends additional real backend candidates to the failover
// chain, each with an independent injectable origin.
Candidates []Candidate
// Clock injects a virtual clock into the harness origins.
Clock testkitopenresponses.VirtualClock
// ArtifactLimit bounds the redacted request-capture artifact list per origin.
ArtifactLimit int
// OriginHandler injects a custom reference-provider origin responder for the
// primary backend. When set, it replaces the harness contract-fake family
// responder while the observing proxy still counts, captures, and redacts
// every request. nil keeps the default family responder. This is the seam
// Task 7.4 adversarial origins use (event injection, native ID/native
// opaque evidence, abrupt mid-stream death).
OriginHandler http.Handler
// ContinuationMaxChainDepth overrides the OpenResponses frontend
// continuation max_chain_depth when greater than zero, so amplification
// proofs can exercise a short chain instead of the production default of 64.
ContinuationMaxChainDepth int
}
DeploymentSpec is the generic cell selector: one spec resolves the entire deployment — client entrypoint, real frontend, core executor, real backend(s), and injectable reference-provider origin(s) — with no bespoke pairwise wiring.
This is the Phase 7 SMOKE SCAFFOLDING selector. It proves the reusable harness can compose any authoritative cell with contract-fake origins and injectable failure modes; it is NOT Phase 8 compatibility evidence. Phase 8 injects the independent OpenResponses refbackend/refclient emulators and certifies each cell with official-wire scenarios (tasks 8.1–8.5).
func (DeploymentSpec) Validate ¶
func (s DeploymentSpec) Validate() error
Validate returns a non-nil error for cells the generic selector must not deploy: unknown/empty identities, unknown transports, or provider-connector backends that are not constructible in the base harness.
type Origin ¶
type Origin struct {
// contains filtered or unexported fields
}
Origin is the injectable reference-provider origin behind one real backend in a deployment. Base smoke deploys contract-fake origins (counter, bounded redacted capture, failure modes, virtual clock); existing reference families and the Phase 8 independent OpenResponses refbackend can be injected through DeploymentSpec.ProviderOrigin and are observed through the same counter and redacted-capture surface via a transparent observing proxy.
func (*Origin) Capture ¶
func (o *Origin) Capture() []testkitopenresponses.RequestObservation
Capture returns the bounded, redacted request-capture artifacts.
func (*Origin) Clock ¶
func (o *Origin) Clock() testkitopenresponses.VirtualClock
Clock returns the virtual clock injected into the origin.
type OriginFailMode ¶
type OriginFailMode string
OriginFailMode selects a deterministic contract-fake origin failure used for credential/failure injection through the deployment.
const ( OriginFailNone OriginFailMode = "" OriginFailServerError OriginFailMode = "server_error" OriginFailMalformed OriginFailMode = "malformed" )
type RoundTripResult ¶
type RoundTripResult struct {
// Text is the assembled assistant text.
Text string
// ResponseID is the proxy response id.
ResponseID string
// Status is the terminal response status ("completed", "failed", ...).
Status string
// Object is the response resource discriminator ("response" or
// "response.compaction").
Object string
// Events is the ordered client-visible event type trajectory.
Events []string
}
RoundTripResult is the deterministic client-visible outcome of one harness round trip over any client transport.
type SentinelCase ¶
type SentinelCase struct {
ID string
Frontend string
Backend string
Transport ClientTransport
Negative bool
ProfileID string
Protects string
}
SentinelCase is an explicit real-stack composition boundary. It is not a provider inventory and must not be generated from frontend/backend lists.
func BoundedSentinelCases ¶
func BoundedSentinelCases() []SentinelCase
BoundedSentinelCases returns a defensive copy of the reviewed sentinel policy.