Documentation
¶
Overview ¶
Package conformance is the provider adapter contract as an executable table. Every first-party adapter (weft/openai, weft/anthropic, weft/google) runs Run in its own tests: offline against recorded fixtures and live behind the `live` build tag. When Run is green for an adapter, the adapter honours the contract — the suite *is* the documented adapter contract (ADR 0013).
The package lives in the root module and imports no vendor SDK: it drives the model through weft's public API only, so it can also run against wefttest models (proving the harness itself).
Index ¶
- func ErrorServer(t *testing.T, status int, header http.Header, body string) *httptest.Server
- func FixtureServer(t *testing.T, file string) *httptest.Server
- func NoRequestServer(t *testing.T) *httptest.Server
- func RecordingFixtureServer(t *testing.T, file string) (*httptest.Server, func() []string)
- func Run(t *testing.T, caps Caps, newModel func(t *testing.T, name string) weft.Model)
- func SilentServer(t *testing.T) *httptest.Server
- func SlowServer(t *testing.T, drip, done string, interval time.Duration, count int) *httptest.Server
- func StallServer(t *testing.T, firstChunk string) *httptest.Server
- type Caps
- type RecordingModel
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ErrorServer ¶ added in v0.3.3
ErrorServer answers every request with status, the given headers, and body — a provider refusing the call (a 429 with a retry-after, say), for the provider_error case. The body bytes are vendor-specific (each SDK parses its own error envelope); the behaviour is not.
func FixtureServer ¶
FixtureServer serves the recorded responses in file, in order, one per HTTP request — the offline backbone of an adapter's conformance run. The file holds one or more responses separated by a line of the form
=== response 2 ===
(everything before the first separator is response 1). Bodies are served verbatim as text/event-stream, the wire format of every first-party adapter's streaming endpoint. A request beyond the last recorded response fails with 500 and a message naming the file, so an under-recorded script is loud, never silently green.
func NoRequestServer ¶
NoRequestServer fails the test if any request reaches it — the backbone of the kill_switch case, which promises that a denied call makes no request at all, not merely that it returns the sentinel.
func RecordingFixtureServer ¶ added in v0.3.0
RecordingFixtureServer is FixtureServer plus a record of every request body, in request order — for cases that must assert on what the adapter sent (tool_choice_forcing), not only on what came back. The returned func is safe to call after the case's requests.
func Run ¶
Run executes every conformance case as a subtest named after its fixture key. newModel is called once per case (cases share no state) and receives the case name so it can pick its fixture recording or apply case-specific construction — max_tokens, notably, must build the adapter with its token-limit option.
func SilentServer ¶ added in v0.3.3
SilentServer accepts the connection and never answers — not even response headers — until the client goes away: the pre-headers stall of the idle_timeout_before_headers case, which the per-chunk idle timer must cover as it covers a mid-stream one.
func SlowServer ¶ added in v0.2.0
func SlowServer(t *testing.T, drip, done string, interval time.Duration, count int) *httptest.Server
SlowServer drips drip every interval, count times, then serves done and ends the stream. The chunk gaps sit below an adapter's IdleTimeout while the stream's total exceeds it — a slow but actively streaming response the per-chunk idle timer must let through, the flip side of StallServer's true stall.
func StallServer ¶
StallServer writes firstChunk (one complete SSE event, flushed), then holds the response open until the request context ends — a stream that starts but never finishes, for the cancel_mid_stream and idle_timeout cases. The chunk bytes are vendor-specific; the behaviour is not.
Types ¶
type Caps ¶
type Caps struct {
// Reasoning: the adapter emits ModelReasoningDelta and accepts a
// ReasoningPart (with signature) back.
Reasoning bool
// Files: the adapter accepts a FilePart image input.
Files bool
// Sequential: the adapter forwards SequentialTools and the provider
// honours the hint (emits at most one call per step).
Sequential bool
// ToolArgDeltas: the adapter surfaces ModelToolCallDelta progress
// while a provider streams a tool call's argument fragments.
// Google's calls arrive whole, so it legitimately emits none — the
// case is declared, not assumed.
ToolArgDeltas bool
// ToolChoice: the adapter forwards ModelRequest.ToolChoice and the
// provider honours the constraint. The offline case asserts on the
// request bytes too (tool_choice / toolConfig must be in what the
// adapter sent), which needs a RecordingFixtureServer wiring.
ToolChoice bool
// Usage: the provider reports non-zero token usage. False for
// compatible servers that never send usage — the adapter must not
// fake numbers, so the caller declares the gap instead.
Usage bool
// ErrorHeaders: the SDK's error type carries the HTTP response,
// so mw.RetryAfter can read a provider's retry-after header off a
// failed call. False where the SDK keeps only the status code
// (genai's APIError) — mw.Retry then falls back to backoff on that
// provider's 429s, a declared gap, not a silent one. HTTPStatus is
// asserted for every adapter regardless.
ErrorHeaders bool
// Live: the model is reached over the network. The suite relaxes
// determinism (a live model may make a different but valid choice;
// those cases t.Skip rather than fail) and skips fixture-only cases
// (idle_timeout, provider_error).
Live bool
}
Caps declares what the adapter under test supports, so the suite asserts the right behaviour instead of skipping silently.
type RecordingModel ¶ added in v0.3.0
RecordingModel wraps a model with the request bodies a RecordingFixtureServer captured, so the tool_choice_forcing case can assert on the bytes the adapter sent. It exists for that wiring — a pass-through for Stream and Info, nothing else; it is not a general proxy.
func (RecordingModel) Info ¶ added in v0.3.0
func (m RecordingModel) Info() weft.ModelInfo
Info forwards the wrapped model's identity.
func (RecordingModel) RecordedBodies ¶ added in v0.3.0
func (m RecordingModel) RecordedBodies() []string
RecordedBodies exposes the captured request bodies, in request order.
func (RecordingModel) Stream ¶ added in v0.3.0
func (m RecordingModel) Stream(ctx context.Context, req weft.ModelRequest) iter.Seq2[weft.ModelEvent, error]
Stream implements weft.Model by forwarding.
Source Files