Documentation
¶
Overview ¶
Package conformance is a reusable test suite that a chat provider module runs against its own client to prove it behaves like every other provider.
The point is that `ChatClient` promises providers are interchangeable, and nothing tested that promise. Each provider interpreted the contract its own way — `Ask` returning raw text on one provider and failing on another, `RequestTimeout` bounding every HTTP provider but not the subprocess one, a stream leaking on the documented cancellation path — and none of it failed anything until someone swapped providers. This suite is what makes a divergence fail CI in the module that introduced it.
A provider module writes one test:
func TestConformance(t *testing.T) {
conformance.Run(t, conformance.Suite{
Provider: chat.ProviderOpenAI,
NewClient: func(t *testing.T, cfg chat.Config, b conformance.Behaviour) (chat.ChatClient, conformance.Control) {
stub := newStubServer(b) // an httptest server shaped by b
cfg.HTTPClient = stub.Client()
return newOpenAIClient(t, cfg), stub
},
})
}
Why the module supplies the backend ¶
The suite asserts behaviour identical across providers, but the wire format producing it is not: a hanging backend is an httptest server that never writes for one provider and a sleeping subprocess for another. So the suite says what the backend should *do* (Behaviour) and asks what it *saw* (Control), and the module bridges the two.
Capability gating ¶
Cases needing an optional contract — chat.StreamingChatClient, chat.CachingChatClient, chat.PersistentChatClient, chat.StatelessCapable — run only when the client the factory returns satisfies it, the same structural test chat.New already applies. A provider that gains an interface starts running its cases without anyone remembering to switch them on.
The same applies to observation: a Control that cannot report whether a request offered tools simply does not implement ToolObserver, and the case needing it skips with a line naming what was missed rather than passing vacuously.
It uses the standard library testing package and nothing else — deliberately no testify — so a provider module that runs it takes on no assertion-library dependency it would otherwise avoid.
Specified by https://gitlab.com/phpboyscout/go/chat/-/wikis/specs/0011-chat-provider-conformance.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Run ¶
Run executes the whole suite against s, one named subtest per contract, so a failing provider sees exactly which one it breaks.
Every provider has the Ask contract, timeout honouring and error-chain preservation checked. A chat.StreamingChatClient additionally has its stream torn down on a callback error — the leak the suite exists for. Cases depending on an observation the Control cannot make skip with a line naming the gap, so a run never reports coverage it did not have.
Types ¶
type Behaviour ¶
type Behaviour struct {
// Text is the assistant text the backend returns for a normal completion.
// For a schema-bound call the stub sends it as the structured payload.
Text string
// Hang makes the backend block until the request context is cancelled,
// without ever responding. It is how the suite proves a client honours its
// own timeout rather than waiting on the backend forever.
Hang bool
// FailStatus, when non-zero, makes the backend fail the request with this
// HTTP status, so the module's registered status extractor is exercised. A
// provider with no HTTP transport ignores it and the case skips.
FailStatus int
// ToolCall, when non-empty, makes the backend request this tool on the
// first step and return Text on the second — the minimum ReAct exchange.
ToolCall string
// ToolInput is the raw JSON argument object sent with ToolCall. Empty means
// an empty object.
ToolInput string
}
Behaviour tells the module's stub backend how to respond. The zero value is a backend that returns empty text successfully.
type Control ¶
type Control interface {
// Requests returns every request the backend received, oldest first. An
// empty slice means the client never called out.
Requests(t *testing.T) []Request
}
Control stands in for the backend the client under test talks to. It is how the suite asks what actually arrived, rather than inferring it from the client's own return values — which is exactly what a diverging provider gets wrong.
type Request ¶
type Request struct {
// Turns is how many conversation turns the request carried.
Turns int
}
Request is one normalised observation of what the backend received. The fields are provider-independent concepts even though the wire formats carrying them are not.
type StreamObserver ¶
type StreamObserver interface {
// StreamClosed reports whether the streamed response body was closed.
StreamClosed(t *testing.T) bool
}
StreamObserver is the optional half of Control for a stub that can detect its streamed response body being closed.
Without it the stream-lifecycle case can prove the callback's error reaches the caller but not that the connection was released, which is the half that actually leaks. An httptest-based stub implements it by wrapping the response body; a stub that cannot omits the method and the assertion skips.
type Suite ¶
type Suite struct {
// Provider names the client under test. It is used in failure messages and
// to build the snapshots the persistence cases restore.
Provider chat.Provider
// NewClient builds a client for cfg whose backend behaves as b describes,
// together with the [Control] over that same backend.
//
// It is called once per case, so each case gets a clean conversation. A
// factory should register its own teardown with t.Cleanup rather than
// expecting the suite to tear anything down.
NewClient func(t *testing.T, cfg chat.Config, b Behaviour) (chat.ChatClient, Control)
}
Suite describes the provider under test and the factory that builds it.
type ToolObserver ¶
type ToolObserver interface {
// ToolsOffered reports whether the most recent request advertised any tool.
ToolsOffered(t *testing.T) bool
}
ToolObserver is the optional half of Control for a stub that can report whether a request advertised tool definitions.
A Control that cannot see this omits the method and the tools-in-Ask case skips. Implement it wherever the wire format makes it visible — every HTTP provider can, since tools are a field in the request body.
Source Files