Documentation
¶
Overview ¶
Package modeltest provides reusable contract tests for implementations of Core model interfaces. Provider modules use these suites and transport fixtures from their tests; production code should not import this package.
A provider that writes its own tests proves only that it works, not that it works the same way as its siblings. These suites exist so that agreement is the default: request immutability, delta validity, terminal-error position, cancellation identity, and teardown are asserted identically for every provider, and a provider that quietly diverges fails here rather than in a caller that swapped one vendor for another.
The suites are transport-honest. They drive a real SDK against a local server instead of a hand-written fake, because the failures worth catching — a wrong path, a mis-encoded option, a stream that never closes — live in the SDK layer that a fake would replace.
Example ¶
package main
import (
"fmt"
"iter"
"github.com/Tangerg/scope/core/modeltest"
)
func main() {
sequence := iter.Seq2[string, error](func(yield func(string, error) bool) {
yield("first", nil)
yield("second", nil)
})
values, err := modeltest.Collect(sequence)
fmt.Println(values, err)
}
Output: [first second] <nil>
Index ¶
- func AnthropicSSEServer(events []AnthropicEvent) *httptest.Server
- func BinaryServer(status int, contentType string, body []byte, ...) *httptest.Server
- func Collect[T any](seq iter.Seq2[T, error]) ([]T, error)
- func CollectN[T any](sequence iter.Seq2[T, error], count int) ([]T, error)
- func JSONServer(status int, body string, inspections ...func(request *http.Request)) *httptest.Server
- func LookupEnv(name string) (string, bool)
- func MuxServer(routes ...Route) *httptest.Server
- func OpenAISSEServer(chunks []string) *httptest.Server
- func RequireEnv(t *testing.T, name string) string
- func RequireKey(t *testing.T, provider string) string
- func RunEmbeddingContract(t *testing.T, contract EmbeddingContract)
- func RunIntegrationEmbedding(t *testing.T, probe IntegrationEmbeddingProbe)
- func RunIntegrationRerank(t *testing.T, probe IntegrationRerankProbe)
- func RunRerankContract(t *testing.T, contract RerankContract)
- func WithTimeout(t *testing.T, duration time.Duration) (context.Context, context.CancelFunc)
- type AnthropicEvent
- type CallBehaviorCase
- type ChatBehaviorSuite
- type ChatSuite
- type EmbeddingContract
- type IntegrationEmbeddingProbe
- type IntegrationRerankProbe
- type Lifecycle
- type PollCounter
- type RerankContract
- type Route
- type StreamBehaviorCase
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func AnthropicSSEServer ¶
func AnthropicSSEServer(events []AnthropicEvent) *httptest.Server
AnthropicSSEServer returns an httptest.Server that streams `events` as Anthropic-shaped SSE:
event: message_start\ndata: {...}\n\n
event: content_block_delta\ndata: {...}\n\n
...
event: message_stop\ndata: {...}\n\n
Anthropic uses named events rather than a single sentinel; the caller is responsible for providing the right sequence.
func BinaryServer ¶
func BinaryServer(status int, contentType string, body []byte, inspections ...func(request *http.Request)) *httptest.Server
BinaryServer serves the modalities whose success path is bytes rather than JSON — synthesized speech and generated images — where forcing the payload through a string fixture would corrupt it.
func Collect ¶
Collect drains an iter.Seq2[T, error] iterator into a slice. The iteration stops on the first non-nil error, which is returned along with whatever was yielded so far.
This is the canonical helper for streaming-test assertions: spin up a mock SSE server, call model.Stream(ctx, req), Collect the result, then assert on the slice + final error.
func CollectN ¶
CollectN drains at most n items from the iterator. Use this for cancellation tests — break early to verify the iterator's stop function tears down the upstream connection cleanly.
func JSONServer ¶
func JSONServer(status int, body string, inspections ...func(request *http.Request)) *httptest.Server
JSONServer runs every inspection before writing the response so a test can assert on the request the adapter actually sent without racing the client's return. Inspections receive the live request, so a body must be read there or not at all.
func LookupEnv ¶
LookupEnv reads an optional setting whose absence is a valid configuration rather than a reason to skip, so the caller decides what a missing value means instead of losing the test to an unconditional skip.
func MuxServer ¶
MuxServer exists because the asynchronous modalities submit work on one path and poll for it on another, so their adapters cannot be exercised by a single-handler fixture. Routes are matched in order, which lets a caller put specific paths first and keep a catch-all last.
func OpenAISSEServer ¶
OpenAISSEServer returns an httptest.Server that streams `chunks` as OpenAI-shaped Server-Sent Events:
data: <chunk-1>\n\n data: <chunk-2>\n\n ... data: [DONE]\n\n
Each chunk should be a JSON-encoded `ChatCompletionChunk` body. The server is registered with t.Cleanup so callers don't have to defer Close().
Used by every OpenAI-compatible vendor (openai / azureopenai / deepseek / moonshot / openrouter / xai / groq / together / fireworks / perplexity / alibaba / zhipu / minimax / ...).
func RequireEnv ¶
RequireEnv carries the same skip semantics as RequireKey for the settings that are not credentials: an endpoint, region, or deployment name a provider cannot infer. It is separate because those names are vendor-specific and cannot be derived from the provider alone.
func RequireKey ¶
RequireKey skips rather than fails when a provider credential is absent, so the same suite is runnable both in CI without secrets and locally with them. Deriving the variable name from the provider keeps credentials discoverable without a per-provider convention to look up.
func RunEmbeddingContract ¶
func RunEmbeddingContract(t *testing.T, contract EmbeddingContract)
RunEmbeddingContract sends two inputs rather than one because a provider that drops or collapses a batch still satisfies a single-input test. Pairing the output count with the requested path also catches an adapter that reaches the wrong endpoint yet happens to decode a plausible response.
func RunIntegrationEmbedding ¶
func RunIntegrationEmbedding(t *testing.T, probe IntegrationEmbeddingProbe)
RunIntegrationEmbedding repeats the mock assertions against the live service, because a canned body cannot reject an adapter whose auth header, path, or request encoding the real vendor would refuse.
func RunIntegrationRerank ¶ added in v0.12.0
func RunIntegrationRerank(t *testing.T, probe IntegrationRerankProbe)
RunIntegrationRerank leans on ValidateFor rather than on an expected ordering, because a live model may legitimately rank two documents differently between calls while still owing index-addressed, in-range results for the query it was given.
func RunRerankContract ¶ added in v0.12.0
func RunRerankContract(t *testing.T, contract RerankContract)
RunRerankContract accepts the cap under either top_n or top_k because reranking vendors disagree on the name. What matters is that the portable Options.TopK reaches the wire at all: an adapter that drops it returns a plausible ranking over every document and would otherwise look correct.
func WithTimeout ¶
WithTimeout bounds one integration call. A provider that hangs would otherwise consume the whole package deadline and report the timeout against an unrelated test; deriving from t.Context keeps the bound tied to this test so cancellation still propagates when the test fails first.
Types ¶
type AnthropicEvent ¶
AnthropicEvent is a single named SSE event for Anthropic's multi-event-type streaming protocol.
type CallBehaviorCase ¶
CallBehaviorCase supplies an in-flight Call and its provider lifecycle.
type ChatBehaviorSuite ¶
type ChatBehaviorSuite struct {
Request func(t *testing.T) *chat.Request
CallCancellation func(t *testing.T) CallBehaviorCase
StreamCancellation func(t *testing.T) StreamBehaviorCase
EarlyStop func(t *testing.T) StreamBehaviorCase
FirstError func(t *testing.T) chat.Streamer
}
ChatBehaviorSuite exercises lifecycle and terminal-error behavior against a provider's real SDK transport. Each factory must return fresh state.
func (ChatBehaviorSuite) Run ¶
func (c ChatBehaviorSuite) Run(t *testing.T)
Run executes the shared Call/Stream behavior contract.
type ChatSuite ¶
type ChatSuite struct {
New func(t *testing.T) (chat.Model, chat.Streamer)
Request func(t *testing.T) *chat.Request
AssertCall func(t *testing.T, response *chat.Response)
AssertStream func(t *testing.T, deltas []*chat.ResponseDelta)
AssertAggregated func(t *testing.T, response *chat.Response)
}
ChatSuite describes one provider's happy-path Model and Streamer contract. New and Request are called independently for each subtest so provider state and request mutation cannot leak between Call and Stream.
type EmbeddingContract ¶
type EmbeddingContract struct {
// ModelID is the model id passed into the embedding request.
ModelID string
// Response is the canned JSON body — must encode 2 outputs so the
// contract can validate batching.
Response string
// ExpectedPath is the URL path the SDK should hit (e.g. "/embeddings"
// or "/embedding/text"). Empty means skip the path assertion.
ExpectedPath string
// Build returns the model wired against the mock server.
Build func(t *testing.T, baseURL string) embedding.Model
}
EmbeddingContract drives the mock-test contract for any embedding vendor. The `Response` field is the canned JSON body the mock server returns — it should encode a response with 2 embeddings (matching the 2-input request the contract sends).
type IntegrationEmbeddingProbe ¶
type IntegrationEmbeddingProbe struct {
Provider string
Build func(t *testing.T, key string) embedding.Model
}
IntegrationEmbeddingProbe is the standard real-API embedding smoke probe: Call returns 2 outputs with non-empty embeddings.
type IntegrationRerankProbe ¶ added in v0.12.0
type IntegrationRerankProbe struct {
Provider string
Build func(t *testing.T, key string) rerank.Model
}
IntegrationRerankProbe is the standard real-API rerank smoke probe.
type Lifecycle ¶
type Lifecycle struct {
Started <-chan struct{}
Stopped <-chan struct{}
}
Lifecycle observes one in-flight provider request. Started closes after the mock has sent any initial stream event; Stopped closes when the request context is released and the handler exits.
func NewBlockingServer ¶
func NewBlockingServer(t *testing.T, writeInitial func(http.ResponseWriter)) (*httptest.Server, Lifecycle)
NewBlockingServer holds a request open until its context is released, which is what makes cancellation and early-stop assertions deterministic: the returned Lifecycle proves the request was genuinely in flight before the test cancels, and proves the handler exited afterward. A sleeping fixture could only guess at both, and would turn a provider that leaks its connection into a flake instead of a failure.
type PollCounter ¶
type PollCounter struct {
// contains filtered or unexported fields
}
PollCounter holds a goroutine-safe attempt counter. Polling vendors typically need to return "in-progress" for the first N polls then "completed" — bind a PollCounter to the GET handler to drive that.
func (*PollCounter) Inc ¶
func (p *PollCounter) Inc() int32
func (*PollCounter) N ¶
func (p *PollCounter) N() int32
type RerankContract ¶ added in v0.12.0
type RerankContract struct {
// ModelID is the model id the adapter is expected to put on the wire, so a
// silently substituted default is caught rather than accepted.
ModelID string
// Response is the canned JSON body. It must decode to a result set that is
// valid for a three-document query capped at two, because the contract
// checks the response against the request it actually sent.
Response string
// ExpectedPath is the URL path the SDK should reach.
ExpectedPath string
// Build returns the model wired against the mock server.
Build func(t *testing.T, baseURL string) rerank.Model
}
RerankContract drives the mock transport contract for a reranking provider.
type Route ¶
type Route struct {
Method string
Contains string
Handle http.HandlerFunc
}
Route names an HTTP method + path-substring pair plus its handler. The Contains field is matched against r.URL.Path with strings.Contains, so "/transcript" matches both "/v2/transcript" (the POST) and "/v2/transcript/job-1" (the GET poll). When Contains is empty the route matches every path — useful as a fallback.
type StreamBehaviorCase ¶
StreamBehaviorCase supplies an in-flight Stream and its provider lifecycle.
Source Files