conformance

package
v0.1.0-rc.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 1, 2026 License: Apache-2.0 Imports: 64 Imported by: 0

Documentation

Overview

Package conformance hosts the bundled frontend × backend conformance matrix (spec tasks 12.x). Tests drive official reference clients against frontend handlers, the core executor, and official backend connectors backed by reference backend emulators or protocol-faithful error stubs.

The generic cell selector (DeploymentSpec/Deploy) and the TestCellSelect_SmokeScaffold_* tests are Phase 7 SMOKE SCAFFOLDING: they prove the reusable harness composes any authoritative cell with contract-fake origins and injectable failure modes. They are NOT Phase 8 compatibility evidence — Phase 8 replaces the contract-fake origins with the independent OpenResponses refbackend/refclient emulators and certifies every cell with official-wire scenarios (tasks 8.1–8.5).

Package conformance drives end-to-end matrix tests. Upstream error-shape checks use NewUpstream400Server for minimal JSON bodies that match each provider family’s typical 400 invalid_request patterns (aligned with internal/refbackend error shapes where those emulators return structured errors). Success paths use internal/refbackend via NewSuccessRefBackend.

Index

Constants

View Source
const (
	FrontendOpenAIResponses = "openai-responses"
	FrontendOpenAILegacy    = "openai-legacy"
	FrontendAnthropic       = "anthropic"
	FrontendGemini          = "gemini"
	FrontendOpenResponses   = "openresponses"
)

Authoritative harness frontend identities. The set is a superset of the locked baseline matrix (BundledFrontendIDs) and adds the OpenResponses frontend.

View Source
const (
	BackendOpenAIResponses  = "openai-responses"
	BackendOpenAILegacy     = "openai-legacy"
	BackendAnthropic        = "anthropic"
	BackendGemini           = "gemini"
	BackendBedrock          = "bedrock"
	BackendACP              = "acp"
	BackendOpenRouter       = "openrouter"
	BackendNVIDIA           = "nvidia"
	BackendOpenResponses    = "openresponses"
	BackendCompatibleOpenAI = standardplugins.CustomOpenAIResponsesCompatibleID
)

Authoritative harness backend identities. openrouter and nvidia are provider connector columns that are not constructible in the base essential bundle; the generic selector fails them closed and the OpenResponses row (Task 8.3) proves their actual route through the real connector executables (DeployConnectorColumnFor / connector_host.go) without promoting the connectors to essential status.

View Source
const ConnectorColumnModel = "gpt-4o-mini"

ConnectorColumnModel is the canonical model used by connector-column deployments (OpenRouter/NVIDIA). The value is shared with the route-selector the frontend default routes to.

View Source
const (
	HarnessFakeText = harnessFakeText
)

harnessFakeText is the deterministic assistant text emitted by harness contract-fake reference-provider origins. HarnessFakeText is the exported form for integration packages.

Variables

View Source
var ExpectedMigrationGoldenJSON = []string{
	"python_lip_anthropic_messages_nonstream.json",
	"python_lip_openai_responses_http_nonstream.json",
	"python_lip_openai_responses_http_streaming.json",
}

ExpectedMigrationGoldenJSON lists migration parity JSON under testdata/migration/ (Req. 15.13). Keep in sync with docs/release-gates.md.

View Source
var ParityProtocolEvidence = map[string][]string{
	"openai-responses": {"parity_openai_test.go"},
	"openai-legacy":    {"parity_openai_test.go"},
	"anthropic":        {"parity_anthropic_test.go"},
	"gemini":           {"parity_gemini_test.go"},
	"bedrock":          {"parity_bedrock_test.go"},
	"acp":              {"connector_host_test.go"},
	"openresponses":    {"sentinel_test.go"},
	"openrouter":       {"sentinel_test.go"},
	"nvidia":           {"sentinel_test.go"},
}

ParityProtocolEvidence maps every bundled frontend/backend protocol id to at least one parity suite source file in this package (llm-api-parity P5.1). Shared families may list the same file for multiple ids. OpenResponses evidence spans the explicit frontend row and backend column conformance suites; the OpenRouter/NVIDIA compatibility identities reuse the configured provider-mode row proof.

View Source
var ParitySuiteGoFiles = []string{
	"parity_openai_test.go",
	"parity_anthropic_test.go",
	"parity_gemini_test.go",
	"parity_bedrock_test.go",
	"connector_host_test.go",
}

ParitySuiteGoFiles lists parity suite sources (.kiro/specs/archive/llm-api-parity/tasks.md Phase 5). Keep in sync with testdata/migration/README.md and docs/release-gates.md.

Functions

func AllBundledProtocolIDs

func AllBundledProtocolIDs() []string

AllBundledProtocolIDs returns the sorted union of bundled frontend and backend protocol ids.

func BackendFor

func BackendFor(tb testing.TB, backendID, upstreamBaseURL string, httpClient *http.Client) execbackend.Backend

BackendFor returns the bundled execbackend.Backend for upstreamBaseURL (httptest origin or /v1 layout per plugin).

func BackendForDualCredential

func BackendForDualCredential(tb testing.TB, backendID, upstreamBaseURL string, httpClient *http.Client) execbackend.Backend

BackendForDualCredential is like BackendFor but supplies a second synthetic key for hosted providers that support credential pools. Reference backends accept any non-empty key material used by conformance clients (sk-test, synthetic Anthropic key, fake Gemini key).

func BundledBackendIDs

func BundledBackendIDs() []string

func BundledFrontendIDs

func BundledFrontendIDs() []string

Protocol identities support independent parity discovery. They are not paired into a mandatory frontend-by-backend completeness list.

func ConnectorColumnRouteSelector

func ConnectorColumnRouteSelector(backendID string) string

ConnectorColumnRouteSelector returns the route selector a connector-column deployment resolves to (backendID:ConnectorColumnModel).

func DefaultModel

func DefaultModel(backendID string) string

DefaultModel returns the model name wired into routing.AttemptCandidate for a bundled backend ID.

func GeminiConformanceBaseURL

func GeminiConformanceBaseURL(proxyOrigin string) string

GeminiConformanceBaseURL is the genai client base URL for conformance tests against the bundled Gemini frontend (mounted under /v1beta/ and /v1beta1/).

func GenAITestCtx

func GenAITestCtx() context.Context

GenAITestCtx is a shared background context for genai client construction in tests.

func HarnessBackendIDs

func HarnessBackendIDs() []string

HarnessBackendIDs returns the deterministic authoritative harness backend list (the 6 essential backends, the OpenResponses backend, and the provider-connector columns openrouter/nvidia).

func HarnessFrontendIDs

func HarnessFrontendIDs() []string

HarnessFrontendIDs returns the deterministic authoritative harness frontend list (5 bundled real frontends including OpenResponses).

func MountFrontend

func MountFrontend(mux *http.ServeMux, frontendID string, exec *runtime.Executor, routeSelector string) error

MountFrontend registers the bundled frontend handler on mux for conformance tests.

func NewSuccessRefBackend

func NewSuccessRefBackend(tb testing.TB, backendID string, onRequestBody func([]byte)) *httptest.Server

NewSuccessRefBackend returns a reference backend whose streaming and non-streaming paths both surface parityText as assistant text (ACP keeps the stock emulator which answers "ok"). Optional onRequestBody observes the raw upstream HTTP body after route/auth checks.

func NewTestExecutor

func NewTestExecutor(tb testing.TB, backendID, upstreamBaseURL string, httpClient *http.Client) *runtime.Executor

NewTestExecutor wires a single backend against refBackendURL (or error injection URL) for conformance cells.

func NewTestExecutorDualCredential

func NewTestExecutorDualCredential(tb testing.TB, backendID, upstreamBaseURL string, httpClient *http.Client) *runtime.Executor

NewTestExecutorDualCredential wires hosted OpenAI, Anthropic, and Gemini backends with two ordered API keys so credential pools are populated. Bedrock and ACP use the same construction as NewTestExecutor (no multi-key pool in this harness).

func NewToolCallRepairRefBackend

func NewToolCallRepairRefBackend(tb testing.TB, backendID string) *httptest.Server

NewToolCallRepairRefBackend wires a reference backend that emits intentionally terminal-comma tool-call argument JSON so canonical safe-tail repair must rewrite. Safe-tail engine/runtime and local-dogfood suites cover the schema-determined pending-value class; protocol adapters remain unaware of both repair classes.

func NewToolRefBackend

func NewToolRefBackend(tb testing.TB, backendID string, onBody func([]byte)) *httptest.Server

NewToolRefBackend wires a reference backend emulator that completes a single tool call and includes usage fields where the wire shape supports it (Task 12.2).

func NewUpstream400Server

func NewUpstream400Server(tb testing.TB, backendID string) *httptest.Server

NewUpstream400Server returns an httptest server that answers like a provider HTTP 400 for the given bundled backend wire family (Task 12.1 protocol-valid upstream error shape).

func RouteSelector

func RouteSelector(backendID, model string) string

RouteSelector builds a core routing selector primary for a single-backend executor.

func WantRepairedToolArgsJSON

func WantRepairedToolArgsJSON(backendID string) string

WantRepairedToolArgsJSON is the closed JSON expected after syntax repair for NewToolCallRepairRefBackend streams (gemini is skipped; see helper).

func WantSafeTailPendingArgsJSON

func WantSafeTailPendingArgsJSON(backendID string) string

Types

type Candidate

type Candidate struct {
	// Backend is an authoritative backend ID.
	Backend string
	// ProfileID selects a validated profile when Backend is a compatible family.
	ProfileID string
	// OriginFail selects a deterministic failure mode for the candidate origin.
	OriginFail OriginFailMode
	// ProviderOrigin injects an external reference-provider origin base URL.
	ProviderOrigin string
}

Candidate is one additional real backend candidate in the deployment's failover chain, with its own injectable contract-fake origin.

type ClientEntrypoint

type ClientEntrypoint interface {
	// RoundTrip drives one create/turn and returns the deterministic result.
	RoundTrip(ctx context.Context, prompt string) (RoundTripResult, error)
	// Close releases any client-held resources (e.g. WebSocket connections).
	Close() error
}

ClientEntrypoint is the configurable client seam of the deployment harness. The OpenResponses family uses the harness raw wire clients (JSON/SSE/compact/ WebSocket); existing families delegate to the repository independent reference clients. The Phase 8 independent OpenResponses refclient can implement this contract and drive the same deployment.

type ClientTransport

type ClientTransport string

ClientTransport selects the client entrypoint used to drive one deployment.

const (
	TransportJSON      ClientTransport = "json"
	TransportSSE       ClientTransport = "sse"
	TransportCompact   ClientTransport = "compact"
	TransportWebSocket ClientTransport = "websocket"
)

type Deployment

type Deployment struct {
	Spec DeploymentSpec
	// Exec is the wired real core executor.
	Exec *runtime.Executor
	// RouteSelector is the primary route selector the frontend default routes to.
	RouteSelector string
	// Mux is the composed frontend mux.
	Mux *http.ServeMux
	// Server is the full-path frontend origin.
	Server *httptest.Server
	// Client is the transport-selected client entrypoint.
	Client ClientEntrypoint
	// Clock is the injected virtual clock (nil when not injected).
	Clock testkitopenresponses.VirtualClock
	// contains filtered or unexported fields
}

Deployment is one deterministic full-path deployment: a configurable client entrypoint in front of a real frontend handler, the real core executor, real backend adapter(s), and injectable reference-provider origins.

func Deploy

func Deploy(tb testing.TB, spec DeploymentSpec) *Deployment

Deploy composes one full-path deployment from a single generic cell selector. It returns nil for invalid or not-yet-constructible cells without starting any server, origin, or port.

func DeployConnectorColumn

func DeployConnectorColumn(tb testing.TB, backendID string, transport ClientTransport) *Deployment

DeployConnectorColumn deploys the OpenResponses frontend over the actual connectors/openrouter or connectors/nvidia executable for the connector-column evidence identity. The connector process is configured with a fresh origin that emulates an OpenAI-compatible provider endpoint; the observing origin counts and redacts every request so tests can prove the actual connector route reaches the provider endpoint. backendID must be one of BackendOpenRouter / BackendNVIDIA.

func DeployConnectorColumnFor

func DeployConnectorColumnFor(tb testing.TB, frontend, backendID string, transport ClientTransport) *Deployment

DeployConnectorColumnFor deploys an arbitrary bundled frontend over the actual OpenRouter/NVIDIA connector executable for the connector-column evidence identities. frontend must be a bundled frontend ID; only frontends that produce an OpenAI Responses operation (openresponses, openai-responses) can round-trip the Responses-wire connector; other frontend operations fail closed before any upstream request.

func DeployConnectorColumnWithFail

func DeployConnectorColumnWithFail(tb testing.TB, frontend, backendID string, transport ClientTransport, fail OriginFailMode) *Deployment

DeployConnectorColumnWithFail deploys a connector-column cell whose observing origin injects a deterministic failure mode (e.g. OriginFailUnauthorized).

func DeployConnectorColumnWithOrigin

func DeployConnectorColumnWithOrigin(tb testing.TB, frontend, backendID string, transport ClientTransport, originHandler http.Handler) *Deployment

DeployConnectorColumnWithOrigin deploys a connector-column cell with a custom observing-origin responder (nil keeps the default connectorColumnOrigin). Custom origins let evidence assert on the real request headers/body the connector process sends.

func (*Deployment) Backend

func (d *Deployment) Backend(backendID string) execbackend.Backend

Backend returns the constructed backend for a backend slot. It exposes the host-built connector backend so evidence can drive connector-specific surfaces (for example dynamic model inventory) that only the real connector exposes.

func (*Deployment) BaseURL

func (d *Deployment) BaseURL() string

BaseURL returns the full-path frontend origin base URL.

func (*Deployment) CandidateOrigin

func (d *Deployment) CandidateOrigin(i int) *Origin

CandidateOrigin returns the i-th candidate origin.

func (*Deployment) Close

func (d *Deployment) Close() error

Close releases every owned resource deterministically: the frontend listener, all reference-provider origins, and the frontend generation context. It is idempotent.

func (*Deployment) FrontendAddr

func (d *Deployment) FrontendAddr() string

FrontendAddr returns the frontend origin listen address (host:port).

func (*Deployment) OriginFor

func (d *Deployment) OriginFor(backendID string) *Origin

OriginFor returns the primary contract-fake origin for backendID.

func (*Deployment) RawFrontendPost

func (d *Deployment) RawFrontendPost(ctx context.Context, path, rawBody string) (int, error)

RawFrontendPost posts rawBody to an arbitrary frontend path and returns the HTTP status without fataling, so fail-closed cells can be asserted cleanly.

func (*Deployment) RequestCount

func (d *Deployment) RequestCount(backendID string) int

RequestCount returns the number of upstream requests observed by the primary contract-fake origin for backendID.

func (*Deployment) RoundTripModel

func (d *Deployment) RoundTripModel(ctx context.Context, model, prompt string) error

RoundTripModel performs one JSON create round trip with an explicit model, for pre-network rejection and unroutable-model proofs.

func (*Deployment) SendRawCompact

func (d *Deployment) SendRawCompact(ctx context.Context, rawBody string) error

SendRawCompact posts an arbitrary JSON compact body to the OpenResponses frontend and returns an error when the frontend does not accept it.

func (*Deployment) SendRawCreate

func (d *Deployment) SendRawCreate(ctx context.Context, rawBody string) error

SendRawCreate posts an arbitrary JSON create body to the OpenResponses frontend and returns an error when the frontend does not accept it.

func (*Deployment) SendRawWSTurn

func (d *Deployment) SendRawWSTurn(ctx context.Context, rawTurn string) error

SendRawWSTurn dials one WebSocket connection, sends an arbitrary turn frame, and returns an error when the session rejects the turn (classified error).

type DeploymentSpec

type DeploymentSpec struct {
	// Frontend is an authoritative frontend ID (HarnessFrontendIDs).
	Frontend string
	// Backend is an authoritative backend ID (HarnessBackendIDs).
	Backend string
	// Model is the canonical model used by the client; empty uses the harness
	// default for Backend.
	Model string
	// Transport selects the client entrypoint (json/sse/compact/websocket).
	// Empty defaults to json.
	Transport ClientTransport
	// ProviderOrigin injects an external reference-provider origin base URL for
	// the primary backend (existing reference families, Phase 8 refbackend).
	// Empty deploys a harness contract-fake origin.
	ProviderOrigin string
	// ProfileID selects a validated provider profile for a compatible family.
	ProfileID string
	// ProviderClient is the HTTP client used by the real backend to reach the
	// origin; empty uses the origin loopback client.
	ProviderClient *http.Client
	// OriginFail selects a deterministic contract-fake origin failure mode for
	// the primary backend.
	OriginFail OriginFailMode
	// Candidates appends additional real backend candidates to the failover
	// chain, each with an independent injectable origin.
	Candidates []Candidate
	// Clock injects a virtual clock into the harness origins.
	Clock testkitopenresponses.VirtualClock
	// ArtifactLimit bounds the redacted request-capture artifact list per origin.
	ArtifactLimit int
	// OriginHandler injects a custom reference-provider origin responder for the
	// primary backend. When set, it replaces the harness contract-fake family
	// responder while the observing proxy still counts, captures, and redacts
	// every request. nil keeps the default family responder. This is the seam
	// Task 7.4 adversarial origins use (event injection, native ID/native
	// opaque evidence, abrupt mid-stream death).
	OriginHandler http.Handler
	// ContinuationMaxChainDepth overrides the OpenResponses frontend
	// continuation max_chain_depth when greater than zero, so amplification
	// proofs can exercise a short chain instead of the production default of 64.
	ContinuationMaxChainDepth int
}

DeploymentSpec is the generic cell selector: one spec resolves the entire deployment — client entrypoint, real frontend, core executor, real backend(s), and injectable reference-provider origin(s) — with no bespoke pairwise wiring.

This is the Phase 7 SMOKE SCAFFOLDING selector. It proves the reusable harness can compose any authoritative cell with contract-fake origins and injectable failure modes; it is NOT Phase 8 compatibility evidence. Phase 8 injects the independent OpenResponses refbackend/refclient emulators and certifies each cell with official-wire scenarios (tasks 8.1–8.5).

func (DeploymentSpec) Validate

func (s DeploymentSpec) Validate() error

Validate returns a non-nil error for cells the generic selector must not deploy: unknown/empty identities, unknown transports, or provider-connector backends that are not constructible in the base harness.

type Origin

type Origin struct {
	// contains filtered or unexported fields
}

Origin is the injectable reference-provider origin behind one real backend in a deployment. Base smoke deploys contract-fake origins (counter, bounded redacted capture, failure modes, virtual clock); existing reference families and the Phase 8 independent OpenResponses refbackend can be injected through DeploymentSpec.ProviderOrigin and are observed through the same counter and redacted-capture surface via a transparent observing proxy.

func (*Origin) Addr

func (o *Origin) Addr() string

Addr returns the origin listen address (host:port) for leak assertions.

func (*Origin) Capture

Capture returns the bounded, redacted request-capture artifacts.

func (*Origin) Client

func (o *Origin) Client() *http.Client

Client returns a loopback HTTP client bound to the origin server.

func (*Origin) Clock

Clock returns the virtual clock injected into the origin.

func (*Origin) Close

func (o *Origin) Close() error

Close deterministically shuts the origin down. It is idempotent.

func (*Origin) Count

func (o *Origin) Count() int

Count returns the number of requests the origin received. Injected external origins are observed through the proxy, so the count includes their requests.

func (*Origin) URL

func (o *Origin) URL() string

URL returns the origin base URL the real backend is wired to.

type OriginFailMode

type OriginFailMode string

OriginFailMode selects a deterministic contract-fake origin failure used for credential/failure injection through the deployment.

const (
	OriginFailNone         OriginFailMode = ""
	OriginFailUnauthorized OriginFailMode = "unauthorized"
	OriginFailServerError  OriginFailMode = "server_error"
	OriginFailMalformed    OriginFailMode = "malformed"
)

type RoundTripResult

type RoundTripResult struct {
	// Text is the assembled assistant text.
	Text string
	// ResponseID is the proxy response id.
	ResponseID string
	// Status is the terminal response status ("completed", "failed", ...).
	Status string
	// Object is the response resource discriminator ("response" or
	// "response.compaction").
	Object string
	// Events is the ordered client-visible event type trajectory.
	Events []string
}

RoundTripResult is the deterministic client-visible outcome of one harness round trip over any client transport.

type SentinelCase

type SentinelCase struct {
	ID        string
	Frontend  string
	Backend   string
	Transport ClientTransport
	Negative  bool
	ProfileID string
	Protects  string
}

SentinelCase is an explicit real-stack composition boundary. It is not a provider inventory and must not be generated from frontend/backend lists.

func BoundedSentinelCases

func BoundedSentinelCases() []SentinelCase

BoundedSentinelCases returns a defensive copy of the reviewed sentinel policy.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL