protocoltest

package
v0.260806.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 6, 2026 License: MPL-2.0 Imports: 49 Imported by: 0

Documentation

Overview

Package protocoltest provides a framework for end-to-end validation of the model gateway's protocol transformation layer.

Architecture

  1. VirtualServer — a mock HTTP provider that speaks OpenAI, Anthropic, and Google response formats. Conceptually a "virtual model" for testing.

  2. TestEnv — wires a real gateway Server (with transform pipeline) to a VirtualServer, configures routing rules, and provides SendAs() for round-trip testing.

  3. Matrix — executes the full cross-product of sources × targets × scenarios × streaming modes.

This package is the test-only framework, while internal/virtualmodel is the production Gin server. They share the vmodel.GenericRegistry primitive for scenario / model storage — Scenario implements vmodel.VirtualModel — but their HTTP handlers remain separate by design: virtualserver/handler.go operates on structured request/response shapes, while protocoltest scenarios serve pre-rendered byte / SSE-line payloads that exercise the gateway transform pipeline at the wire-format level.

Usage 

env := protocoltest.NewTestEnv(t)
defer env.Close()
env.SetupRoute(protocol.TypeAnthropicV1, protocol.TypeOpenAIChat, protocoltest.TextScenario())
result := env.SendAs(t, protocol.TypeAnthropicV1, protocol.TypeOpenAIChat, protocoltest.TextScenario(), false)
assert.Equal(t, "assistant", result.Role)

Index

Constants

View Source 
const (
	FormatOpenAIChat      = scenario.FormatOpenAIChat
	FormatOpenAIResponses = scenario.FormatOpenAIResponses
	FormatAnthropic       = scenario.FormatAnthropic
	FormatGoogle          = scenario.FormatGoogle
)
View Source 
const (
	DuoProviderChat      = "tb1-openai-chat"
	DuoProviderResponses = "tb1-openai-responses"
	DuoProviderAnthropic = "tb1-anthropic"
)

tb2 provider UUIDs wired by seedDuoGateway; duo_routing.go's scenario services reference the same IDs, so they live in one place.

View Source 
const (
	DuoSlowOpenAIModel    = "duo-slow-gpt"
	DuoSlowAnthropicModel = "duo-slow-claude"
)

Slow-stream vmodel IDs registered on tb1 for the backpressure routes. Unlike the builtin virtual-gpt-4/virtual-claude-3 (tiny instant response), these stream a configurably large response over a configurable duration.

View Source 
const (
	FailMockPreContent429  = "virtual-fail-429"
	FailMockPreContent500  = "virtual-fail-500"
	FailMockMidStreamCut   = "virtual-fail-midstream-close"
	FailMockMidStreamErr   = "virtual-fail-midstream-event"
	FailMockMidStreamClean = "virtual-fail-midstream-cleaneof"
)

Built-in failing mock IDs registered by vmodel.SharedDefaultMocks. Tests pass one of these as the primary tier to drive a deterministic failure without writing handler scaffolding.

View Source 
const (
	EndpointChat      = benchmark.EndpointChat
	EndpointResponses = benchmark.EndpointResponses
	EndpointAnthropic = benchmark.EndpointAnthropic
	EndpointGoogle    = benchmark.EndpointGoogle
)
View Source 
const DuoDefaultMaxSlopeKB = 32.0

DuoDefaultMaxSlopeKB is the default per-instance retention-slope failure threshold in KB/request — the single number shared by the CLI (`harness duo --max-slope-kb`) and the Go regression test, so tuning it is a one-place change. The #1255 leak measured 823 KB/request on the gateway instance; healthy builds measure ~0.5. 32 leaves generous headroom against GC noise while still catching any per-request pin of a request-body-sized buffer by orders of magnitude.

View Source 
const VirtualMockAnswerMarker = "Paris"

VirtualMockAnswerMarker is a substring guaranteed to appear in the default virtual upstream's answer (the shared TextScenario's fixed reply, "The capital of France is Paris."). Agent-CLI callers use it to verify that the gateway round trip's content actually reached the CLI's output — a zero exit code alone can mask a CLI that printed an error and exited cleanly.

Variables

View Source 
var (
	AssertContentEquals        = check.AssertContentEquals
	AssertContentContains      = check.AssertContentContains
	AssertContentNonEmpty      = check.AssertContentNonEmpty
	AssertRoleEquals           = check.AssertRoleEquals
	AssertFinishReason         = check.AssertFinishReason
	AssertFinishReasonOneOf    = check.AssertFinishReasonOneOf
	AssertHasToolCalls         = check.AssertHasToolCalls
	AssertToolCallName         = check.AssertToolCallName
	AssertToolCallArgs         = check.AssertToolCallArgs
	AssertHasThinking          = check.AssertHasThinking
	AssertNoThinking           = check.AssertNoThinking
	AssertUsageNonZero         = check.AssertUsageNonZero
	AssertHTTPStatus           = check.AssertHTTPStatus
	AssertStreamEventCount     = check.AssertStreamEventCount
	AssertStreamError          = check.AssertStreamError
	AssertStreamNotCompleted   = check.AssertStreamNotCompleted
	AssertHTTPStatusAtLeast    = check.AssertHTTPStatusAtLeast
	AssertErrorMessageContains = check.AssertErrorMessageContains
	AssertModelContains        = check.AssertModelContains
	AssertStreamEventsContain  = check.AssertStreamEventsContain
	AssertFinishReasonNonEmpty = check.AssertFinishReasonNonEmpty
	AssertUsagePropagated      = check.AssertUsagePropagated
)
View Source 
var (
	AllScenarios       = scenario.AllScenarios
	AllErrorScenarios  = scenario.AllErrorScenarios
	GetErrorSpec       = scenario.GetErrorSpec
	BuildErrorFromSpec = scenario.BuildErrorFromSpec

	TextScenario                = scenario.TextScenario
	ToolUseScenario             = scenario.ToolUseScenario
	ToolResultScenario          = scenario.ToolResultScenario
	ThinkingScenario            = scenario.ThinkingScenario
	MultiTurnScenario           = scenario.MultiTurnScenario
	StreamingTextScenario       = scenario.StreamingTextScenario
	StreamingToolUseScenario    = scenario.StreamingToolUseScenario
	IncompleteScenario          = scenario.IncompleteScenario
	ErrorScenario               = scenario.ErrorScenario
	Error500Scenario            = scenario.Error500Scenario
	ErrorAuth401Scenario        = scenario.ErrorAuth401Scenario
	ErrorMidStreamCloseScenario = scenario.ErrorMidStreamCloseScenario
)
View Source 
var DuoDefaultRoute = DuoRoute{Name: "beta-chat", Beta: true, Target: "chat"}

DuoDefaultRoute is the memory-phase default: the Claude Code hot path (Anthropic beta client → OpenAI Chat provider) where #1255 was reported.

View Source 
var DuoServiceIdentities = []string{"a", "b", "c", "d", "e", "f"}

DuoServiceIdentities is the pool of service-identity vmodels registered on tb1 for the routing scenarios. Each identity answers with a distinct, recognizable marker (see DuoServiceMarker), so which service a request was routed to is readable directly from the response body — a wire-level assertion that needs no cooperation from the gateway under test.

Functions

func AgentSourceAPIType added in v0.260716.1 

func AgentSourceAPIType(at AgentType) protocol.APIType

AgentSourceAPIType returns the protocol-matrix source type matching the agent CLI's wire format: codex speaks the OpenAI Responses API, claude and opencode post Anthropic messages. This is the key consumers use to look up an agent's runs in the shared known-defect registry (KnownDefectReason).

func BuildConversationBody added in v0.260716.1 

func BuildConversationBody(route DuoRoute, totalBytes int, streaming bool) []byte

BuildConversationBody builds a Claude-Code-shaped Anthropic request of approximately totalBytes for the given route: alternating user/assistant text messages, so the gateway parses and converts a realistically large agentic context. The shape is valid for both the v1 and beta surfaces.

func BuiltinRequestModel added in v0.260716.1 

func BuiltinRequestModel(at AgentType) string

BuiltinRequestModel returns the fixed RequestModel the agent's built-in rule matches — what the agent CLI actually sends ("" for unknown agents).

func BuiltinRuleRef added in v0.260716.1 

func BuiltinRuleRef(at AgentType) (uuid, requestModel string, err error)

BuiltinRuleRef returns the agent's built-in rule UUID and its fixed RequestModel — the one copy of the mapping shared by every setup path and the CLI reporting layer.

func DuoServiceMarker added in v0.260716.1 

func DuoServiceMarker(identity string) string

DuoServiceMarker returns the response-content marker a service-identity vmodel answers with.

func DuoServiceModel added in v0.260716.1 

func DuoServiceModel(identity string) string

DuoServiceModel returns the tb1 vmodel ID for a service identity.

func KnownDefectReason added in v0.260716.1 

func KnownDefectReason(source protocol.APIType, scenarioName string) (string, bool)

KnownDefectReason reports whether a (source protocol, scenario) combination is in the known-defect registry, and why. Consumers outside the matrix (e.g. `harness replay`) use this to skip runs that exercise a documented gateway bug instead of keeping their own copy of the list.

func MaybeRunDuoServe added in v0.260716.1 

func MaybeRunDuoServe()

MaybeRunDuoServe runs a duo child instance and exits the process when the duo spec is present in the environment; otherwise it returns immediately. Call it first thing in main() (cli/harness) and TestMain (duo_test.go) so the parent can re-execute the same binary as a server.

func ResolveAPIStyle 

func ResolveAPIStyle(entry RealModelEntry) (string, error)

ResolveAPIStyle returns the effective api_style for an entry. Returns an error if api_style is empty or contains an invalid value. Valid values are: "openai", "anthropic", "google".

func ResolveAPIType 

func ResolveAPIType(entry RealModelEntry) (string, error)

ResolveAPIType returns the effective api_type for an entry. If the entry specifies one, it is validated and returned. If empty, returns a default based on api_style:

  • "anthropic" → "anthropic_v1"
  • "openai" → "openai_chat"
  • "google" → "google"

Types

type AgentTestEnv 

type AgentTestEnv struct {
	// contains filtered or unexported fields
}

AgentTestEnv provides an isolated test environment for Agent testing It includes: - A temporary config directory - A gateway server with virtual provider - Routing rules configured for the Agent - A virtual server that captures requests for validation

func NewAgentTestEnv 

func NewAgentTestEnv(AgentType AgentType) (*AgentTestEnv, error)

NewAgentTestEnv creates a new Agent test environment The environment is isolated with a temporary config directory and must be cleaned up with Close() when done

func (*AgentTestEnv) AppConfig 

func (env *AgentTestEnv) AppConfig() *serverconfig.Config

AppConfig returns the application configuration

func (*AgentTestEnv) BaseURL 

func (env *AgentTestEnv) BaseURL() string

BaseURL returns the base URL of the gateway server

func (*AgentTestEnv) Close 

func (env *AgentTestEnv) Close(preserve bool) error

Close cleans up the test environment If preserve is true, the config directory is kept for inspection

func (*AgentTestEnv) ConfigDir 

func (env *AgentTestEnv) ConfigDir() string

ConfigDir returns the temporary config directory path

func (*AgentTestEnv) ModelToken 

func (env *AgentTestEnv) ModelToken() string

ModelToken returns the model token for requests

func (*AgentTestEnv) ReplayFixture 

func (env *AgentTestEnv) ReplayFixture(agentType AgentType, body []byte, streaming bool) (*RoundTripResult, error)

ReplayFixture sends a raw request body to the agent's gateway endpoint and parses the response into a RoundTripResult ready for Scenario assertions.

The endpoint path, auth header, and response API style are all derived from agentType. `streaming` selects SSE vs JSON response parsing — it must match the fixture's own "stream" flag.

func (*AgentTestEnv) SetupAgent 

func (env *AgentTestEnv) SetupAgent(AgentType AgentType, providerName string, modelName string) error

SetupAgent configures the environment for a specific Agent type This creates the necessary provider and routing rules

func (*AgentTestEnv) SetupRealAgent 

func (env *AgentTestEnv) SetupRealAgent(AgentType AgentType, providerName string, modelName string, apiBase string, apiKey string, apiStyle string) error

SetupRealAgent configures the environment to route through a real upstream provider. Unlike SetupAgent, it does not use the virtual server — the provider points at the real apiBase with the real apiKey. apiStyle must be "openai" or "anthropic". apiType is optional and specifies the target API type (e.g., "anthropic_v1", "openai_chat"). If empty, a default is chosen based on apiStyle.

func (*AgentTestEnv) SetupVModelAgent 

func (env *AgentTestEnv) SetupVModelAgent(AgentType AgentType, vmodelID string) error

SetupVModelAgent configures the environment so the agent's built-in rule routes to a seeded builtin virtual-model provider.

Unlike SetupAgent (external VirtualServer mock) and SetupRealAgent (real upstream), this exercises the in-process vmodel dispatch path: 

gateway → built-in-<agent> rule → vmodel builtin provider
        → provider.IsVirtual() short-circuit → in-process vmodel handler

The builtin vmodel providers are seeded into the provider store by server.NewServer, so no provider is added here — only the rule is repointed.

vmodelID must be a model registered in the vmodel registry for the agent's protocol (e.g. "virtual-claude-3", "echo-model" for Anthropic-style agents).

func (*AgentTestEnv) SetupVirtualAgentScenario 

func (env *AgentTestEnv) SetupVirtualAgentScenario(agentType AgentType, scenario Scenario) error

SetupVirtualAgentScenario wires the agent's built-in rule to the in-process VirtualServer and registers `scenario`'s mock responses so the VirtualServer serves them deterministically.

This is the "virtual" replay upstream: because the response is fully controlled by the scenario's MockResponses, the caller can run the scenario's content-level Assertions against the round-trip result.

The rule's upstream model encodes the scenario name as "virtual-model-<scenario>" so the VirtualServer's scenario detection resolves the right mock.

func (*AgentTestEnv) VirtualServerURL 

func (env *AgentTestEnv) VirtualServerURL() string

VirtualServerURL returns the URL of the virtual server

type AgentTestResult 

type AgentTestResult struct {
	// Name is the test name
	Name string

	// Agent is the Agent type being tested
	Agent AgentType

	// Scenario is the test scenario (e.g., "text", "streaming", "tool_use")
	Scenario string

	// Passed indicates whether the test passed
	Passed bool

	// Skipped indicates whether the test was skipped
	Skipped bool

	// SkipReason explains why the test was skipped
	SkipReason string

	// Errors contains any assertion errors
	Errors []AssertionError

	// Duration is how long the test took
	Duration int64 // milliseconds

	// HTTPStatus is the HTTP status code received
	HTTPStatus int

	// RequestHeaders contains the request headers sent to the virtual server
	RequestHeaders http.Header

	// RequestBody contains the request body sent to the virtual server
	RequestBody []byte

	// ResponseBody contains the raw response body
	ResponseBody []byte
}

AgentTestResult represents the result of a single Agent test

type AgentType 

type AgentType string

AgentType represents the type of agent Agent to test 

const (
	AgentTypeClaudeCode AgentType = "claude"
	AgentTypeCodex      AgentType = "codex"
	AgentTypeOpenCode   AgentType = "opencode"
)

func (AgentType) Scenario 

func (pt AgentType) Scenario() typ.RuleScenario

Scenario returns the corresponding RuleScenario for this Agent

func (AgentType) String 

func (pt AgentType) String() string

String returns the string representation of AgentType

type Assertion 

type Assertion = check.Assertion

Assertion is a named check applied to a RoundTripResult.

func AnthropicStreamShape added in v0.260716.1 

func AnthropicStreamShape() Assertion

AnthropicStreamShape pins the canonical Anthropic SSE frame sequence a client-facing /v1/messages stream must carry, independent of content. It is the shared event-shape vocabulary of the duo functional phase and replay's streaming runs for anthropic-style agents.

func ResponsesStreamShape added in v0.260716.1 

func ResponsesStreamShape() Assertion

ResponsesStreamShape pins the OpenAI Responses SSE lifecycle frames a client-facing /v1/responses stream must carry.

func StreamShapeForAgent added in v0.260716.1 

func StreamShapeForAgent(at AgentType) Assertion

StreamShapeForAgent returns the event-shape assertion matching the stream format the given agent's CLI consumes.

type AssertionError 

type AssertionError struct {
	Assertion string // assertion name
	Error     string // error message
	Context   string // additional context (truncated body, etc.)
}

AssertionError represents a single assertion failure.

type CapturedRequest 

type CapturedRequest = benchmark.CapturedRequest

CapturedRequest is the request the gateway forwarded to a provider endpoint. Aliased to the benchmark foundation.

type Client added in v0.260611.1 

type Client interface {
	// Name identifies the driver: "http", "gosdk", "python", "node".
	Name() string
	// Supports reports whether the driver can speak the given source protocol.
	Supports(source protocol.APIType) bool
	// Send issues one request and returns the normalized result. Gateway-side
	// API errors (4xx/5xx) must be reported in the result (HTTPStatus/RawBody),
	// not as a returned error; a non-nil error means the driver itself failed.
	Send(env *TestEnv, spec SendSpec) (*RoundTripResult, error)
}

Client drives a single request through the gateway and returns the normalized RoundTripResult the assertion layer consumes. Implementations range from the raw in-process HTTP client (default) to official SDKs and external subprocess drivers (python/node), so the same matrix exercises the gateway through progressively more realistic client stacks.

func NewAISDKClient added in v0.260611.1 

func NewAISDKClient(driverDir string) Client

NewAISDKClient returns a driver backed by tests/clients/aisdk/driver.mjs (AI SDK by Vercel: ai + @ai-sdk/anthropic + @ai-sdk/openai). driverDir is the tests/clients root.

func NewGoSDKClient added in v0.260611.1 

func NewGoSDKClient() Client

NewGoSDKClient returns a client driver backed by the official Go SDKs.

func NewHTTPClient added in v0.260611.1 

func NewHTTPClient() Client

NewHTTPClient returns the default raw-HTTP client driver.

func NewNodeClient added in v0.260611.1 

func NewNodeClient(driverDir string) Client

NewNodeClient returns a driver backed by tests/clients/node/driver.mjs (real @anthropic-ai/sdk + openai Node SDKs). driverDir is the tests/clients root.

func NewPythonClient added in v0.260611.1 

func NewPythonClient(driverDir string) Client

NewPythonClient returns a driver backed by tests/clients/python/driver.py (real anthropic + openai Python SDKs). driverDir is the tests/clients root.

func NewSubprocessClient added in v0.260611.1 

func NewSubprocessClient(name string, argv ...string) Client

NewSubprocessClient returns a client driver that shells out to argv for each request, speaking the JSON-over-stdin/stdout driver contract.

type DuoCheck added in v0.260716.1 

type DuoCheck struct {
	Route  string `json:"route"`
	Name   string `json:"name"`
	Pass   bool   `json:"pass"`
	Detail string `json:"detail,omitempty"`
}

DuoCheck is one functional verification result.

type DuoEnv added in v0.260716.1 

type DuoEnv struct {
	TB1 *DuoInstance // upstream: serves /virtual vmodel endpoints
	TB2 *DuoInstance // gateway under test: converts + proxies to tb1
	// contains filtered or unexported fields
}

DuoEnv holds the two running child instances and the wiring between them.

func NewDuoEnv added in v0.260716.1 

func NewDuoEnv(cfg DuoEnvConfig) (*DuoEnv, error)

NewDuoEnv boots tb1 (vmodel upstream) and tb2 (gateway under test) as two full server processes and wires one tb2 rule per route in allDuoRoutesWithSlow to tb1's virtual endpoints. Callers must Close() the returned env.

func (*DuoEnv) Close added in v0.260716.1 

func (env *DuoEnv) Close()

Close terminates both instances and removes their config dirs.

func (*DuoEnv) DrainStreaming added in v0.260716.1 

func (env *DuoEnv) DrainStreaming(route DuoRoute, body []byte, readDelay time.Duration) (int, error)

DrainStreaming drives one streaming request over the route and fully drains the SSE body, returning the number of `event:` lines seen. A non-zero readDelay reads the body slowly (see slowReader).

func (*DuoEnv) RunFunctionalChecks added in v0.260716.1 

func (env *DuoEnv) RunFunctionalChecks(route DuoRoute, bodyBytes int) []DuoCheck

RunFunctionalChecks verifies protocol correctness of one conversion route with a bodyBytes-sized conversation: streaming SSE shape, assembled content, usage propagation, and the non-streaming response body.

func (*DuoEnv) RunMemoryPhase added in v0.260716.1 

func (env *DuoEnv) RunMemoryPhase(cfg DuoMemoryConfig) (*DuoMemoryReport, error)

RunMemoryPhase measures allocation churn, post-GC retention slope, and concurrent-burst peak heap on one conversion route — separately for tb1 and tb2. A near-zero slope means no per-request leak on that instance (reference numbers live with duo_test.go's threshold).

func (*DuoEnv) RunRoutingScenario added in v0.260716.1 

func (env *DuoEnv) RunRoutingScenario(sc *DuoRoutingScenario) []DuoCheck

RunRoutingScenario seeds the scenario's rule and drives its request program, returning one DuoCheck per assertion.

type DuoEnvConfig added in v0.260716.1 

type DuoEnvConfig struct {
	// Upstream configures tb1, the vmodel upstream.
	Upstream DuoUpstreamConfig
	// Gateway configures tb2, the gateway under test.
	Gateway DuoGatewayConfig
	// ChildLog, when non-nil, receives both children's stdout/stderr live
	// (in addition to the per-instance tail buffer used for diagnostics).
	ChildLog io.Writer
	// BootTimeout caps how long each instance may take to become healthy
	// (default 90s — first boot may attempt a provider-template fetch).
	BootTimeout time.Duration
}

DuoEnvConfig parameterizes NewDuoEnv. Role-specific knobs live in the Upstream / Gateway sections — mirroring the child-side spec's role split — so an instance-specific parameter has a structural home instead of a name-prefixed field; top-level fields apply to the environment as a whole.

type DuoGatewayConfig added in v0.260723.1 

type DuoGatewayConfig struct {
	// HTTPTimeouts overrides tb2's real http.Server timeouts — the server's
	// own packaged type (server.WithHTTPTimeouts), all four deadlines
	// configurable; zero fields keep Start()'s defaults. Lets a test arm a
	// short WriteTimeout and prove ClearServerIOTimeouts still lets a slower
	// stream complete under the full two-process production stack (#1384) —
	// not just the isolated middleware unit test.
	HTTPTimeouts server.HTTPTimeouts
}

DuoGatewayConfig is the tb2 section of DuoEnvConfig.

type DuoInstance added in v0.260716.1 

type DuoInstance struct {
	Name       string
	ConfigDir  string
	Port       int
	BaseURL    string
	UserToken  string
	ModelToken string
	// contains filtered or unexported fields
}

DuoInstance is one child tingly-box process.

func (*DuoInstance) MemStats added in v0.260716.1 

func (inst *DuoInstance) MemStats(gc bool) (*debugmodule.MemStatsResponse, error)

MemStats fetches a runtime memory snapshot from the instance's debug endpoint. With gc=true the instance forces a full GC first, so HeapAllocBytes is its post-GC retained set; the endpoint throttles forced GCs, so a throttled sample is retried until the GC actually ran.

func (*DuoInstance) OutputTail added in v0.260716.1 

func (inst *DuoInstance) OutputTail() string

OutputTail returns the last captured child stdout/stderr output.

func (*DuoInstance) WriteHeapProfile added in v0.260716.1 

func (inst *DuoInstance) WriteHeapProfile(dir, name string) (string, error)

WriteHeapProfile fetches a post-GC pprof heap profile from the instance and writes it under dir, returning the file path. The endpoint throttles profile serialization, so a 429 is retried past the throttle window.

type DuoInstanceMemory added in v0.260716.1 

type DuoInstanceMemory struct {
	Instance           string  `json:"instance"` // "tb1" (vmodel upstream) or "tb2" (gateway)
	BaselineHeapMB     float64 `json:"baseline_heap_mb"`
	AfterBatch1MB      float64 `json:"after_batch1_delta_mb"`
	AfterBatch2MB      float64 `json:"after_batch2_delta_mb"`
	SlopeKBPerRequest  float64 `json:"retention_slope_kb_per_request"`
	ChurnMBPerRequest  float64 `json:"alloc_churn_mb_per_request"`
	PeakHeapMB         float64 `json:"concurrent_peak_heap_mb"`
	PostBurstDeltaMB   float64 `json:"post_burst_delta_mb"`
	BaselineGoroutines int     `json:"baseline_goroutines"`
	FinalGoroutines    int     `json:"final_goroutines"`
	BaselineProfile    string  `json:"baseline_profile,omitempty"`
	FinalProfile       string  `json:"final_profile,omitempty"`
}

DuoInstanceMemory is the memory outcome for ONE instance.

type DuoMemoryConfig added in v0.260716.1 

type DuoMemoryConfig struct {
	Route     *DuoRoute // conversion route to drive (default DuoDefaultRoute)
	BodyBytes int       // conversation size per request (default 2MB)
	Warmup    int       // warmup requests before the baseline (default 3)
	Batch     int       // requests per sequential batch, two batches are run (default 15)
	Workers   int       // concurrent workers in the burst phase (default 4)
	PerWorker int       // requests per worker in the burst phase (default 5)
	// ReadDelay throttles client-side SSE consumption (see slowReader),
	// building real TCP backpressure against tb2. 0 = read at full speed.
	ReadDelay  time.Duration
	ProfileDir string // write pprof heap profiles here ("" = skip)
	Progress   func(format string, args ...any)
}

DuoMemoryConfig parameterizes RunMemoryPhase.

type DuoMemoryReport added in v0.260716.1 

type DuoMemoryReport struct {
	Route             string            `json:"route"`
	BodyBytes         int               `json:"body_bytes"`
	Batch             int               `json:"batch_requests"` // two sequential batches of this size are run
	ReadDelayMS       int               `json:"read_delay_ms"`  // client-side slow-reader pause (0 = full speed)
	ConcurrentWorkers int               `json:"concurrent_workers"`
	ConcurrentTotal   int               `json:"concurrent_requests"`
	TB1               DuoInstanceMemory `json:"tb1"`
	TB2               DuoInstanceMemory `json:"tb2"`
}

DuoMemoryReport is the outcome of RunMemoryPhase across both instances.

func (*DuoMemoryReport) Instances added in v0.260716.1 

func (r *DuoMemoryReport) Instances() []*DuoInstanceMemory

Instances returns both per-instance results, tb1 first.

func (*DuoMemoryReport) MaxSlopeKB added in v0.260716.1 

func (r *DuoMemoryReport) MaxSlopeKB() float64

MaxSlopeKB returns the larger of the two instances' retention slopes.

type DuoRoute added in v0.260716.1 

type DuoRoute struct {
	// Name identifies the route in flags, check names, and reports,
	// e.g. "beta-chat", "v1-responses", "beta-chat-slow".
	Name string
	// Beta selects the Anthropic beta source surface (?beta=true) over v1.
	Beta bool
	// Target is the provider protocol tb2 converts to: "chat", "responses",
	// or "anthropic" (passthrough).
	Target string
	// Slow selects the backpressure variant: tb1 answers with the slow/large
	// duo stream vmodel instead of the tiny instant builtin.
	Slow bool
}

DuoRoute is one anthropic-source conversion route through tb2.

func AllDuoRoutes added in v0.260716.1 

func AllDuoRoutes() []DuoRoute

AllDuoRoutes lists every fast anthropic-source route the production vmodel endpoint can back: {v1, beta} × {anthropic, openai chat, openai responses}.

func FindDuoRoute added in v0.260716.1 

func FindDuoRoute(name string) (DuoRoute, bool)

FindDuoRoute resolves a route by name, including "-slow" variants.

func (DuoRoute) RequestModel added in v0.260716.1 

func (r DuoRoute) RequestModel() string

RequestModel returns the tb2 request model wired for this route.

func (DuoRoute) SlowVariant added in v0.260716.1 

func (r DuoRoute) SlowVariant() DuoRoute

SlowVariant returns the backpressure variant of the route.

type DuoRoutingBody added in v0.260716.1 

type DuoRoutingBody struct {
	// SizeKB pads the conversation with filler user text (drives the token
	// position: tokens ≈ SizeKB*1024/4).
	SizeKB int `yaml:"size_kb,omitempty" json:"size_kb,omitempty"`
	// UserText is the final user message ("duo routing probe" if empty).
	UserText string `yaml:"user_text,omitempty" json:"user_text,omitempty"`
	// System sets the system prompt (e.g. Claude Code fingerprints).
	System string `yaml:"system,omitempty" json:"system,omitempty"`
	// Thinking enables the thinking parameter.
	Thinking bool `yaml:"thinking,omitempty" json:"thinking,omitempty"`
}

DuoRoutingBody describes the request shape (built as an Anthropic messages request; valid for both v1 and beta surfaces).

type DuoRoutingExpect added in v0.260716.1 

type DuoRoutingExpect struct {
	// Svc asserts which service identity answered (wire level).
	Svc string `yaml:"svc,omitempty" json:"svc,omitempty"`
	// Outcome asserts the smart-routing trace outcome ("matched",
	// "no_match", ...).
	Outcome string `yaml:"outcome,omitempty" json:"outcome,omitempty"`
	// Matched asserts the matched partition's description.
	Matched string `yaml:"matched,omitempty" json:"matched,omitempty"`
	// Source asserts the production routing source response header
	// (smart_routing, affinity, or load_balancer).
	Source string `yaml:"source,omitempty" json:"source,omitempty"`
	// SelectedModel asserts the service selected before dispatch/failover. Use
	// Svc for the independent final wire responder assertion.
	SelectedModel string `yaml:"selected_model,omitempty" json:"selected_model,omitempty"`
	// Stages asserts the exact cumulative ServiceSelector path.
	Stages []string `yaml:"stages,omitempty" json:"stages,omitempty"`
}

DuoRoutingExpect is the per-request expectation; empty fields are skipped.

type DuoRoutingRequest added in v0.260716.1 

type DuoRoutingRequest struct {
	Name string `yaml:"name" json:"name"`
	// Beta selects the Anthropic beta surface (?beta=true).
	Beta bool `yaml:"beta,omitempty" json:"beta,omitempty"`
	// Session sets X-Tingly-Session-ID (affinity identity); "" = none.
	Session string           `yaml:"session,omitempty" json:"session,omitempty"`
	Body    DuoRoutingBody   `yaml:"body" json:"body"`
	Expect  DuoRoutingExpect `yaml:"expect" json:"expect"`
}

DuoRoutingRequest is one request in a scenario's program.

type DuoRoutingRule added in v0.260716.1 

type DuoRoutingRule struct {
	// Scenario is the gateway scenario: "anthropic" (default) or
	// "claude_code" (required for the agent.claude_code position).
	Scenario string `yaml:"scenario,omitempty" json:"scenario,omitempty"`
	// AffinitySecs enables session affinity (Flags.SessionAffinity).
	AffinitySecs int `yaml:"affinity_secs,omitempty" json:"affinity_secs,omitempty"`
	// LBTactic selects the terminal load-balancing strategy. Empty means the
	// harness default (random).
	LBTactic string `yaml:"lb_tactic,omitempty" json:"lb_tactic,omitempty"`
	// WithinTierTactic selects among services in the same tier. Empty means
	// random. It is only meaningful when LBTactic is tier.
	WithinTierTactic string `yaml:"within_tier_tactic,omitempty" json:"within_tier_tactic,omitempty"`
	// Services is the base pool the LB falls back to when no partition
	// matches.
	Services []DuoRoutingService `yaml:"services" json:"services"`
	// Smart lists the partitions, evaluated in order.
	Smart []DuoSmartPartition `yaml:"smart" json:"smart"`
}

DuoRoutingRule is the rule shape under test.

type DuoRoutingScenario added in v0.260716.1 

type DuoRoutingScenario struct {
	Name        string              `yaml:"name" json:"name"`
	Description string              `yaml:"description,omitempty" json:"description,omitempty"`
	Rule        DuoRoutingRule      `yaml:"rule" json:"rule"`
	Requests    []DuoRoutingRequest `yaml:"requests" json:"requests"`
}

DuoRoutingScenario is one rule shape plus a request program against it.

func BuiltinRoutingScenarios added in v0.260716.1 

func BuiltinRoutingScenarios() []*DuoRoutingScenario

BuiltinRoutingScenarios returns the self-checking scenario catalog. Time-dependent scenarios are constructed relative to the current wall clock (the smart-routing clock is not injectable across processes).

func FindRoutingScenarios added in v0.260716.1 

func FindRoutingScenarios(names []string) ([]*DuoRoutingScenario, error)

FindRoutingScenarios resolves scenario names ("all" or empty = every built-in).

func LoadRoutingScenarios added in v0.260716.1 

func LoadRoutingScenarios(path string) ([]*DuoRoutingScenario, error)

LoadRoutingScenarios reads user-defined scenarios from a YAML file: 

scenarios:
  - name: my-rule
    rule:
      services: [{svc: b}]
      smart:
        - description: big
          ops: [{position: token, operation: ge, value: "50000"}]
          services: [{svc: a}]
    requests:
      - name: big
        body: {size_kb: 256}
        expect: {svc: a, outcome: matched, matched: big}

func (*DuoRoutingScenario) RequestModel added in v0.260716.1 

func (sc *DuoRoutingScenario) RequestModel() string

RequestModel returns the tb2 request model the scenario's rule binds.

type DuoRoutingService added in v0.260716.1 

type DuoRoutingService struct {
	// Svc is the service identity ("a".."f"); the tb1 vmodel is
	// DuoServiceModel(Svc) and its response carries DuoServiceMarker(Svc).
	Svc string `yaml:"svc" json:"svc"`
	// Target is the provider protocol: "chat" (default), "responses", or
	// "anthropic".
	Target string `yaml:"target,omitempty" json:"target,omitempty"`
	// Model overrides the service-identity model derived from Svc. It is used
	// for production mock models such as virtual-fail-429 in pipeline setup
	// requests. Normal wire-identity services should use Svc.
	Model string `yaml:"model,omitempty" json:"model,omitempty"`
	// Tier is the service priority for the tier tactic (lower is preferred).
	Tier int `yaml:"tier,omitempty" json:"tier,omitempty"`
	// Weight controls selection within tactics that support weighting. Zero
	// keeps the harness default of 1.
	Weight int `yaml:"weight,omitempty" json:"weight,omitempty"`
}

DuoRoutingService names one upstream candidate: a service identity from tb1's pool (DuoServiceIdentities) reached through one of tb2's provider protocols.

type DuoSmartOpSpec added in v0.260716.1 

type DuoSmartOpSpec struct {
	Position  string `yaml:"position" json:"position"`
	Operation string `yaml:"operation" json:"operation"`
	Value     string `yaml:"value,omitempty" json:"value,omitempty"`
}

DuoSmartOpSpec is one smart-routing condition in scenario form; it maps 1:1 onto smartrouting.SmartOp.

type DuoSmartPartition added in v0.260716.1 

type DuoSmartPartition struct {
	Description string              `yaml:"description" json:"description"`
	Ops         []DuoSmartOpSpec    `yaml:"ops" json:"ops"`
	Services    []DuoRoutingService `yaml:"services" json:"services"`
}

DuoSmartPartition is one smart-routing rule: AND-ed ops selecting a service subset. First matching partition wins.

type DuoStreamShape added in v0.260723.1 

type DuoStreamShape struct {
	SizeKB int           `json:"size_kb,omitempty"`
	Delay  time.Duration `json:"delay,omitempty"`
}

DuoStreamShape parameterizes tb1's slow backpressure vmodels: an approximately SizeKB-sized response whose Delay is applied once as TTFT by the virtualserver handler and spread again across chunks by the mock's stream loop, so a request's wall time is roughly 2×Delay.

type DuoUpstreamConfig added in v0.260723.1 

type DuoUpstreamConfig struct {
	// Stream shapes tb1's slow backpressure vmodels (see DuoStreamShape).
	// Defaults: 256 KB over ~1 s of wall time (Delay 500ms, applied twice).
	Stream DuoStreamShape
}

DuoUpstreamConfig is the tb1 section of DuoEnvConfig.

type EndpointKind added in v0.260611.1 

type EndpointKind = benchmark.EndpointKind

EndpointKind identifies which provider-native endpoint a request hit. Aliased to the benchmark foundation so observers share one vocabulary.

type FailoverRoute 

type FailoverRoute struct {
	ModelName         string
	PrimaryCallCount  *atomic.Int64
	FallbackCallCount *atomic.Int64 // nil when fallback is env.virtual
}

FailoverRoute is the handle returned by SetupFailoverRoute / SetupBothFailingRoute. ModelName is the gateway-facing model (pass to SendWithModel). PrimaryCallCount tracks how often the primary tier was hit. FallbackCallCount is non-nil only when both tiers run through vmodel mock servers (i.e. SetupBothFailingRoute); otherwise it is nil and tests must rely on content discrimination for fallback assertions.

type IdempotentCase added in v0.260611.1 

type IdempotentCase struct {
	Name     string           // human-readable label
	Source   protocol.APIType // A: the client-facing protocol
	Mid      protocol.APIType // B: the intermediate protocol the chain passes through
	Baseline protocol.APIType // A': passthrough target for the baseline (same API style as A)
}

IdempotentCase describes a round-trip idempotency check. It compares the client-visible result of two request paths that should be observationally identical: 

baseline:   Client(A) ──[A→A passthrough]──────────────→ R0
round-trip: Client(A) ──[A→B]──[B→A]─────────────────────→ R1

The round-trip chains two real conversions: the first hop converts A→B and forwards to the gateway itself, which re-enters through B's inbound route and converts B→A before hitting the mock provider. If either conversion drops information, R0 and R1 diverge. 

g(f(A)) == A   where f = A→B, g = B→A

func DefaultIdempotentCases added in v0.260611.1 

func DefaultIdempotentCases() []IdempotentCase

DefaultIdempotentCases returns the canonical round-trip idempotency cases: every pair among the three first-class protocols (Anthropic, OpenAI Chat, OpenAI Responses), in both directions. Chat and Responses are treated as distinct protocols — the chain head sets OpenAIEndpointMode so a Responses intermediate genuinely re-enters /responses, not /chat/completions.

Baseline is the source's same-style passthrough target:

  • anthropic_v1 → anthropic_beta (Anthropic passthrough)
  • openai_chat → openai_chat
  • openai_responses → openai_responses

Google is omitted: the harness's virtual-provider plumbing does not yet support Google as a target, so it cannot serve as an intermediate hop.

type Matrix 

type Matrix struct {
	Pairs      []ProtocolPair
	Scenarios  []Scenario
	Streaming  []bool
	RecordDir  string // Optional directory for recording requests/responses
	BatchCount int    // Number of times to run each test
	MCPEnabled bool   // Enable MCP feature flag in test env
	Client     Client // Client driver (nil = raw HTTP default)
}

Matrix defines the set of (source, target) pairs, scenarios, and streaming modes to validate.

func DefaultMatrix 

func DefaultMatrix() *Matrix

DefaultMatrix returns the full validation matrix covering every supported (source, target) pair, all built-in scenarios, and both streaming modes.

func (*Matrix) DefaultChains added in v0.260611.1 

func (m *Matrix) DefaultChains() []TransitiveChain

DefaultChains builds transitive chains by composing pairs from the matrix where the first pair's target matches the second pair's source. Self-loops (A→B→A where first == reverse of second) are included — they test round-trip fidelity.

func (*Matrix) ExecuteAll 

func (m *Matrix) ExecuteAll() []TestResult

ExecuteAll runs all matrix combinations and returns structured results. This is a pure function that can be called from both tests and CLI. It does not use testing.T, making it suitable for standalone execution.

One TestEnv is created per scenario and reused for all of its combinations (see executePerScenario).

func (*Matrix) ExecuteAllCacheControls added in v0.260801.1 

func (m *Matrix) ExecuteAllCacheControls() []TestResult

ExecuteAllCacheControls runs single-hop and ABA prompt-cache request checks. Each result validates both the positive cache case and the negative no-cache case. Name formats:

  • cache_controls/single/A→B/{stream|nonstream}
  • cache_controls/aba/A→B→A/{stream|nonstream}

func (*Matrix) ExecuteAllContentShapes added in v0.260801.1 

func (m *Matrix) ExecuteAllContentShapes() []TestResult

ExecuteAllContentShapes runs the request-content-shape regression suite without requiring testing.T. It is the CLI-compatible counterpart of TestContentShapes, returning []TestResult. Name format: "content_shapes/<case name>".

func (*Matrix) ExecuteAllFlags added in v0.260611.1 

func (m *Matrix) ExecuteAllFlags() []TestResult

ExecuteAllFlags runs the rule-flag behavior suite without requiring testing.T. It is the CLI-compatible counterpart of TestRuleFlags, returning []TestResult. Name format: "flags/<flag key>".

func (*Matrix) ExecuteAllIdempotent added in v0.260611.1 

func (m *Matrix) ExecuteAllIdempotent() []TestResult

ExecuteAllIdempotent runs round-trip idempotency tests without requiring testing.T. It is the CLI-compatible counterpart of RunIdempotent, returning []TestResult. For each scenario × case × mode it wires the baseline and round-trip routes in one gateway, drives both requests, and records whether the client-visible results are semantically equivalent (g(f(A)) == A).

Name format: "scenario/<case>/mode" (e.g. "text/openai_chat_via_anthropic/stream").

func (*Matrix) ExecuteAllTransitive added in v0.260611.1 

func (m *Matrix) ExecuteAllTransitive() []TestResult

ExecuteAllTransitive runs two-hop chain tests without requiring testing.T. It is the CLI-compatible counterpart of RunTransitive, returning []TestResult. Name format: "scenario/A→B→C/mode".

func (*Matrix) OnlyScenarios 

func (m *Matrix) OnlyScenarios(names ...string) *Matrix

OnlyScenarios returns a copy of the Matrix filtered to only the named scenarios.

func (*Matrix) OnlySources 

func (m *Matrix) OnlySources(sources ...string) *Matrix

OnlySources returns a copy of the Matrix filtered to pairs whose source matches one of the given protocols.

func (*Matrix) OnlyStreaming 

func (m *Matrix) OnlyStreaming(streaming bool) *Matrix

OnlyStreaming returns a copy of the Matrix filtered to only streaming or non-streaming tests. If streaming is true, only streaming tests are included. If false, only non-streaming tests.

func (*Matrix) OnlyTargets 

func (m *Matrix) OnlyTargets(targets ...string) *Matrix

OnlyTargets returns a copy of the Matrix filtered to pairs whose target matches one of the given protocols.

func (*Matrix) Run 

func (m *Matrix) Run(t *testing.T)

Run executes all matrix combinations as subtests under t. Each combination runs the same executeTest implementation the CLI path uses — the testing.T layer only provisions an env per subtest and reports the TestResult, so the skip logic and assertion loop exist exactly once.

func (*Matrix) RunFull added in v0.260611.1 

func (m *Matrix) RunFull(t *testing.T)

RunFull executes both single-hop and two-hop tests under t, organized as two named sub-sections:

  • "single_hop": every (source→target) pair × scenario × streaming mode
  • "two_hop": every (A→B→C) transitive chain × scenario × streaming mode

Run each section independently with -run TestFoo/single_hop or /two_hop.

func (*Matrix) RunIdempotent added in v0.260611.1 

func (m *Matrix) RunIdempotent(t *testing.T)

RunIdempotent executes round-trip idempotency tests for all cases × scenarios as subtests under t. Each case runs the same executeIdempotentCase implementation the CLI path uses; the testing.T layer (runPerScenario) only provisions one env per scenario and reports the TestResult.

Error / truncation scenarios are not round-trippable: the inner gateway wraps upstream errors and a mid-stream cut surfaces as an error on one hop but partial content on another, so the two paths legitimately diverge. SkipTransitive marks exactly these.

func (*Matrix) RunTransitive added in v0.260611.1 

func (m *Matrix) RunTransitive(t *testing.T)

RunTransitive executes two-hop transitive tests for all chains × scenarios as subtests under t. For each chain A→B→C:

  1. Send as A with target B → get result1 (in A's response format)
  2. Send as B with target C → get result2 (in B's response format)
  3. Assert result1 and result2 are semantically equivalent

This catches information loss that single-hop tests miss: if A→B drops a field that B→C needs, the results will diverge.

Each chain runs the same executeTransitiveChain implementation the CLI path uses; the testing.T layer (runPerScenario) only provisions one env per scenario and reports the TestResult.

func (*Matrix) WithBatchCount 

func (m *Matrix) WithBatchCount(count int) *Matrix

WithBatchCount returns a copy of the Matrix with the batch count set.

func (*Matrix) WithClient added in v0.260611.1 

func (m *Matrix) WithClient(c Client) *Matrix

WithClient returns a copy of the Matrix that drives requests through the given client driver (official SDKs, subprocess drivers) instead of the default raw HTTP client.

func (*Matrix) WithMCPEnabled 

func (m *Matrix) WithMCPEnabled() *Matrix

WithMCPEnabled returns a copy of the Matrix with the MCP feature flag enabled.

func (*Matrix) WithRecordDir 

func (m *Matrix) WithRecordDir(recordDir string) *Matrix

WithRecordDir returns a copy of the Matrix with the record directory set. If recordDir is empty, recording is disabled.

type MockResponseBuilder 

type MockResponseBuilder = scenario.MockResponseBuilder

MockResponseBuilder defines how a virtual server responds for one format.

type ParsedResponse 

type ParsedResponse = vmodeltest.ParsedResponse

ParsedResponse is the result of a request sent to a virtual server.

type ProtocolPair 

type ProtocolPair struct {
	Source protocol.APIType
	Target protocol.APIType
}

ProtocolPair is one (source → target) conversion path to validate. The matrix is built from an explicit list of pairs rather than the Cartesian product of all sources × all targets: many cells of that product map to the same dispatch path (e.g. target=anthropic_v1 and target=anthropic_beta both pick APIStyleAnthropic; OpenAI Chat vs. Responses targets are picked by ResolveOpenAIEndpoint, not the matrix) and listing pairs keeps the matrix in lock-step with the actual dispatch graph documented in internal/protocol/README.md.

func DefaultPairs 

func DefaultPairs() []ProtocolPair

DefaultPairs is the canonical list of (source → target) conversion paths the matrix exercises. Adding a new dispatch path means appending to this list.

Notes:

  • target=anthropic_v1 is intentionally absent. The harness picks providers by APIStyle and both Anthropic types map to the same style, so anthropic_beta as the target already exercises both Anthropic V1 passthrough (when source is V1) and the Beta conversions (when source is non-Anthropic). See internal/protocol/README.md.
  • Anthropic↔Anthropic cross-version (v1↔beta) is rejected by the transform layer and not represented here.
  • Google targets and the google→google passthrough are not yet supported by the harness's virtual provider plumbing.

func (ProtocolPair) String 

func (p ProtocolPair) String() string

String returns "source|target" for use as a map key or label.

type ProviderConfig 

type ProviderConfig struct {
	Name     string   `yaml:"name"`
	BaseURL  string   `yaml:"baseurl"`
	APIKey   string   `yaml:"apikey"`
	APIStyle string   `yaml:"api_style"` // required: "openai" | "anthropic" | "google"
	APIType  string   `yaml:"api_type"`  // optional: "openai_chat" | "openai_responses" | "anthropic_v1" | "anthropic_beta" | "google"
	Models   []string `yaml:"models"`    // list of model names to test
	Prompt   string   `yaml:"prompt"`    // optional per-provider prompt override; empty -> agent default
	// Disabled, when explicitly false, skips this provider entirely. Nil (unset)
	// or true means enabled — so omitting the field stays backward-compatible.
	Disabled *bool `yaml:"enable"`
}

ProviderConfig is one provider entry in the config YAML file. A provider can have multiple models under it.

type ProvidersConfig 

type ProvidersConfig struct {
	Providers []ProviderConfig `yaml:"providers"`
	// Env is an optional shared variable table. Values defined here are resolved
	// first when expanding ${VAR}/$VAR references in provider fields, falling
	// back to the process environment. Env values may themselves be references
	// (e.g. OPENAI_KEY: "${MY_SECRET}"). This lets one key be shared across
	// providers without scattering it across the shell, and keeps the file
	// self-contained.
	Env map[string]string `yaml:"env"`
	// Prompt is an optional top-level default prompt applied to every provider
	// that doesn't set its own `prompt`. Env-expanded like provider fields.
	// A provider-level `prompt` overrides this; a CLI prompt overrides both.
	Prompt string `yaml:"prompt"`
}

ProvidersConfig is the top-level structure of the config YAML file.

type RealModelEntry 

type RealModelEntry struct {
	Name     string // generated entry name: "provider" or "provider-model"
	Provider string // original provider name
	BaseURL  string
	APIKey   string
	Model    string
	APIStyle string
	APIType  string
	Prompt   string // per-provider prompt override (already env-expanded); empty -> agent default
}

RealModelEntry is an expanded entry for testing. Each (provider, model) pair becomes one entry.

func ExpandProvidersConfig 

func ExpandProvidersConfig(cfg *ProvidersConfig) []RealModelEntry

ExpandProvidersConfig expands a ProvidersConfig into individual test entries. Each provider's models array is expanded into separate entries.

func LoadProvidersConfig 

func LoadProvidersConfig(path string) ([]RealModelEntry, error)

LoadProvidersConfig reads and parses a providers config YAML file. Returns the expanded list of test entries.

type RealModelsConfig 

type RealModelsConfig struct {
	Models []RealModelEntry
}

RealModelsConfig is the legacy format kept for backward compatibility. Deprecated: Use ProvidersConfig instead.

func LoadRealModelsConfig 

func LoadRealModelsConfig(path string) (*RealModelsConfig, error)

LoadRealModelsConfig is an alias for LoadProvidersConfig for backward compatibility. Deprecated: Use LoadProvidersConfig instead.

type ResponseFormat 

type ResponseFormat = scenario.ResponseFormat

ResponseFormat selects which provider format a MockResponseBuilder serves.

type RoundTripResult 

type RoundTripResult = check.RoundTripResult

RoundTripResult is the protocol-neutral view of one gateway round trip.

type Scenario 

type Scenario = scenario.Scenario

Scenario is a named mock-provider fixture; implements vmodel.VirtualModel.

type SendSpec added in v0.260611.1 

type SendSpec struct {
	Source       protocol.APIType
	Target       protocol.APIType
	ScenarioName string
	RequestModel string
	Streaming    bool
	GatewayURL   string // real HTTP base URL of the gateway, e.g. http://127.0.0.1:PORT
	APIKey       string // gateway model token
}

SendSpec carries everything a client driver needs to issue one request through the gateway. Target and ScenarioName are response metadata only; the request itself varies only by (Source, RequestModel, Streaming).

type TestEnv 

type TestEnv struct {
	// contains filtered or unexported fields
}

TestEnv wires a real gateway Server (with the full transform pipeline) to a VirtualServer (mock provider). It manages config, routing rules, and provides SendAs() for full round-trip testing.

**Routing Architecture**: 

Client Request → Gateway (/tingly/{scenario}/v1/...)
              → Protocol Transform
              → Provider Request (virtual-server-url/v1/...)
              → VirtualServer (mock provider response)

The gateway handles /tingly/{scenario}/v1/... routes, transforms the request to provider format, and forwards to the virtual server which speaks provider native APIs (/v1/chat/completions, /v1/messages, etc.).

func NewTestEnv 

func NewTestEnv(t *testing.T, opts ...TestEnvOption) *TestEnv

NewTestEnv creates a TestEnv with a fresh gateway config and a new VirtualServer, cleaned up via t.Cleanup. It is the testing.T wrapper over NewTestEnvForCLI — one construction path for both entry points.

func NewTestEnvForCLI 

func NewTestEnvForCLI(opts ...TestEnvOption) (*TestEnv, error)

NewTestEnvForCLI creates a TestEnv without a testing.T. Resources must be cleaned up via an explicit Close() call.

func (*TestEnv) Close 

func (env *TestEnv) Close()

Close shuts down the gateway and virtual servers, releases the config's database handles, and removes the config directory. Closing the stores matters: e2e suites create hundreds of envs in one process, and an unclosed SQLite handle per env exhausts the fd limit. Safe to call more than once (t.Cleanup plus an explicit defer).

func (*TestEnv) GatewayURL added in v0.260611.1 

func (env *TestEnv) GatewayURL() string

GatewayURL returns the base URL of the real gateway HTTP server, for client drivers that speak real HTTP (SDKs, subprocess drivers).

func (*TestEnv) ModelToken added in v0.260611.1 

func (env *TestEnv) ModelToken() string

ModelToken returns the gateway model token client drivers authenticate with.

func (*TestEnv) SendAs 

func (env *TestEnv) SendAs(t *testing.T, source, target protocol.APIType, s Scenario, streaming bool) *RoundTripResult

SendAs sends a request to the gateway as the given source protocol, using the request model configured by SetupRoute, and returns the parsed result.

Streaming requests use the real httptest.Server (env.gatewayServer) because httptest.ResponseRecorder does not support Gin's streaming/SSE machinery. Non-streaming requests use the recorder for simplicity.

func (*TestEnv) SendAsCLI 

func (env *TestEnv) SendAsCLI(source, target protocol.APIType, s Scenario, streaming bool) (*RoundTripResult, error)

SendAsCLI sends a request to the gateway as the given source protocol, using the request model configured by SetupRoute, and returns the parsed result. This version is for CLI use and returns errors instead of calling t.Fatalf.

func (*TestEnv) SendWithModel 

func (env *TestEnv) SendWithModel(t *testing.T, source protocol.APIType, modelName string, streaming bool) *RoundTripResult

SendWithModel sends a request using an explicit model name (bypassing the SetupRoute route map). Used for failover tests where the rule's request model doesn't follow SetupRoute's naming convention.

func (*TestEnv) SetupBothFailingRoute 

func (env *TestEnv) SetupBothFailingRoute(
	t *testing.T,
	source, target protocol.APIType,
	failModel string,
) FailoverRoute

SetupBothFailingRoute wires a two-tier rule where BOTH tiers trip the same pre-content injection. Used for the all-tiers-fail test: client must see a non-200 once the orchestrator exhausts its budget.

func (*TestEnv) SetupCodexAssemblyRoute added in v0.260716.1 

func (env *TestEnv) SetupCodexAssemblyRoute(source protocol.APIType, s Scenario)

SetupCodexAssemblyRoute wires a route to a provider flagged as Codex via OAuthDetail.Issuer rather than a literal APIBase match, so it can point at the VirtualServer instead of the real chatgpt.com host. Codex only speaks the streaming Responses API, so a non-streaming request against it is routed by dispatchOpenAIResponses through the assembly path — the "nonstream client / stream upstream / assemble" cell of the {v1,beta} × {nonstream,stream,assemble} matrix (protocol_cross.go) that was unreachable before provider.IsCodexProvider() decoupled the routing check from the literal dial target. (The mux also needs /codex/responses registered — see vmodel/benchmark/scenario_responder.go — since that's the path Codex's RoundTripper rewrites /v1/responses to.)

func (*TestEnv) SetupCrossStyleFailoverRoute added in v0.260625.1 

func (env *TestEnv) SetupCrossStyleFailoverRoute(
	t *testing.T,
	source protocol.APIType,
	primaryStyle protocol.APIStyle,
	fallbackTarget protocol.APIType,
	successScenario Scenario,
	primaryFailModel string,
) FailoverRoute

SetupCrossStyleFailoverRoute wires a two-tier rule whose tiers use DIFFERENT API styles: the primary (primaryStyle, a vmodel error server) trips a pre-content failure, and the fallback (fallbackTarget's style, served by env.virtual) succeeds. The gateway receives one `source` request; the orchestrator must re-transform it into primaryStyle's wire format for the first attempt and, after failover, into the fallback's wire format for the second — the core guarantee of the lifted failover. env.virtual captures the fallback's request so the test can assert the re-transformed wire shape.

func (*TestEnv) SetupFailoverRoute 

func (env *TestEnv) SetupFailoverRoute(
	t *testing.T,
	source, target protocol.APIType,
	successScenario Scenario,
	primaryFailModel string,
) FailoverRoute

SetupFailoverRoute wires a two-tier rule using vmodel's pre-registered error mocks for the primary tier. Both tiers run inside httptest servers; the primary always trips the named injection (SharedDefaultMocks IDs above), the fallback serves successScenario via env.virtual.

The orchestrator dispatches under TacticTier; the primary (Tier 0) is tried first; pre-content failures are retryable; mid-stream failures commit the gate (no retry). This is the single helper that covers all failover-test shapes: 429/500 pre-content, mid-stream close, mid-stream event.

func (*TestEnv) SetupRoute 

func (env *TestEnv) SetupRoute(source, target protocol.APIType, s Scenario)

SetupRoute configures a gateway rule that routes source protocol requests to the virtual server acting as a target protocol provider.

The virtual server is pre-registered with the scenario's mock responses. If the route has already been set up, this is a no-op (idempotent).

**Routing Flow**: 1. Client sends request to gateway: POST /tingly/{scenario}/v1/chat/completions 2. Gateway transforms request to provider format based on source protocol 3. Gateway forwards to provider: POST {virtualURL}/v1/chat/completions 4. VirtualServer (provider mock) returns pre-configured scenario response

The provider's APIBase includes the /v1 suffix for OpenAI-style providers to match actual provider API structure.

func (*TestEnv) SetupRouteWithFlags added in v0.260611.1 

func (env *TestEnv) SetupRouteWithFlags(source, target protocol.APIType, s Scenario, flags typ.RuleFlags) string

SetupRouteWithFlags wires a route exactly like SetupRoute but stamps rule.Flags onto the gateway rule, so a request routed through it exercises the real flag-resolution and transform pipeline. Returns the request model to send to.

func (*TestEnv) SetupVModelFailoverRoute added in v0.260625.1 

func (env *TestEnv) SetupVModelFailoverRoute(
	t *testing.T,
	source protocol.APIType,
	primaryStyle, fallbackStyle protocol.APIStyle,
	primaryFailModel, fallbackModel string,
) FailoverRoute

SetupVModelFailoverRoute wires a two-tier rule where BOTH tiers are in-process virtual-model providers (AuthType = vmodel, #1249), so failover traverses the vmodel ClientPool path rather than httptest upstreams. The primary trips the named failing model (e.g. FailMockPreContent500) in primaryStyle; the fallback serves fallbackModel (e.g. "echo-model") in fallbackStyle. Set primaryStyle ≠ fallbackStyle to exercise cross-style failover through the vmodel clients.

PrimaryCallCount is nil (in-process providers have no httptest counter); assert on the client result instead.

func (*TestEnv) UpstreamEndpointHits added in v0.260625.1 

func (env *TestEnv) UpstreamEndpointHits(kind EndpointKind) int

UpstreamEndpointHits exposes env.virtual's endpoint-hit counter to out-of-package (_test) callers — e.g. to assert a cross-style failover reached the fallback on the expected provider-native endpoint.

func (*TestEnv) UpstreamLastRequest added in v0.260625.1 

func (env *TestEnv) UpstreamLastRequest(kind EndpointKind) *CapturedRequest

UpstreamLastRequest exposes the last request env.virtual captured on an endpoint, so tests can assert the re-transformed upstream wire shape.

func (*TestEnv) VirtualCallCount 

func (env *TestEnv) VirtualCallCount() int

VirtualCallCount returns the number of requests received by the virtual server.

func (*TestEnv) VirtualURL 

func (env *TestEnv) VirtualURL() string

VirtualURL returns the URL of the underlying virtual server.

type TestEnvOption 

type TestEnvOption func(*testEnvConfig)

TestEnvOption is a functional option for configuring TestEnv.

func NewTestEnvOptionWithClient added in v0.260611.1 

func NewTestEnvOptionWithClient(c Client) TestEnvOption

NewTestEnvOptionWithClient creates an option to set the client driver used for sending requests through the gateway. Defaults to the raw HTTP client.

func NewTestEnvOptionWithMCP 

func NewTestEnvOptionWithMCP() TestEnvOption

NewTestEnvOptionWithMCP creates an option to enable the MCP feature flag.

func NewTestEnvOptionWithRecordDir 

func NewTestEnvOptionWithRecordDir(dir string) TestEnvOption

NewTestEnvOptionWithRecordDir creates an option to set the record directory. If empty, recording is disabled.

type TestResult 

type TestResult struct {
	// Test identification
	Name      string // Full test name: "scenario/source/target/mode"
	Scenario  string // Scenario name: "text", "tool_use", etc.
	Source    protocol.APIType
	Target    protocol.APIType
	Streaming bool

	// Test outcome
	Passed     bool   // true if all assertions passed
	Skipped    bool   // true if test was skipped
	SkipReason string // reason for skipping

	// Error details
	Errors   []AssertionError // list of assertion failures
	Duration time.Duration    // test execution time

	// Batch statistics (populated when BatchCount > 1)
	BatchCount  int           // number of times the test was executed
	BatchPassed int           // number of executions that passed
	BatchMinDur time.Duration // minimum duration across executions
	BatchAvgDur time.Duration // average duration across executions
	BatchMaxDur time.Duration // maximum duration across executions
	BatchErrors []string      // unique error messages from failed executions

	// Response details (for debugging/verbose output)
	HTTPStatus int              // HTTP status code
	Response   *RoundTripResult // full round-trip result (from first or last execution)
}

TestResult represents the outcome of a single matrix test combination. This is returned by Matrix.ExecuteAll() for CLI and other non-testing contexts.

type TokenUsage 

type TokenUsage = check.TokenUsage

TokenUsage holds token counts extracted from a provider response.

type ToolCallResult 

type ToolCallResult = check.ToolCallResult

ToolCallResult holds a single tool/function call extracted from a response.

type TransitiveChain added in v0.260611.1 

type TransitiveChain struct {
	First  ProtocolPair // A→B
	Second ProtocolPair // B→C (Second.Source == First.Target)
}

TransitiveChain represents a two-hop conversion path: A→B then B→C. Both hops go through the full gateway pipeline. The test verifies that the semantic content (text, role, tool calls) is preserved across both conversions.

func (TransitiveChain) String added in v0.260611.1 

func (c TransitiveChain) String() string

String returns a human-readable label like "anthropic_v1→openai_chat→anthropic_beta".

func (TransitiveChain) TestName added in v0.260611.1 

func (c TransitiveChain) TestName(scenario string, streaming bool) string

TestName builds a TestResult name for this chain in a given scenario/mode.

type VirtualClient 

type VirtualClient struct {
	*vmodeltest.Client
	// contains filtered or unexported fields
}

VirtualClient sends provider-native HTTP requests for testing. It embeds vmodeltest.Client for model-parameterized methods and adds scenario-based methods that auto-register on a bound VirtualServer.

func NewVirtualClient 

func NewVirtualClient(baseURL string) *VirtualClient

NewVirtualClient creates a client pointing at baseURL.

func (*VirtualClient) SendAnthropicV1 

func (vc *VirtualClient) SendAnthropicV1(t *testing.T, s Scenario, streaming bool) *ParsedResponse

SendAnthropicV1 sends a request to the Anthropic Messages endpoint.

func (*VirtualClient) SendGoogle 

func (vc *VirtualClient) SendGoogle(t *testing.T, s Scenario, streaming bool) *ParsedResponse

SendGoogle sends a request to the Google GenerateContent endpoint.

func (*VirtualClient) SendOpenAIChat 

func (vc *VirtualClient) SendOpenAIChat(t *testing.T, s Scenario, streaming bool) *ParsedResponse

SendOpenAIChat sends a request to the OpenAI Chat Completions endpoint.

func (*VirtualClient) SendOpenAIResponses 

func (vc *VirtualClient) SendOpenAIResponses(t *testing.T, s Scenario, streaming bool) *ParsedResponse

SendOpenAIResponses sends a request to the OpenAI Responses API endpoint.

func (*VirtualClient) WithServer 

func (vc *VirtualClient) WithServer(vs *VirtualServer) *VirtualClient

WithServer binds the client to a VirtualServer.

type VirtualServer 

type VirtualServer struct {
	// contains filtered or unexported fields
}

VirtualServer is a mock provider server speaking OpenAI, Anthropic, and Google response formats, returning pre-configured scenario responses.

As of the benchmark unification it is a thin wrapper over benchmark.Server (scenario responder): the scenario-serving handlers, request capture, and endpoint-hit counting all live in vmodel/benchmark now. This type keeps the protocoltest-facing API (Client(), RegisterScenario, EndpointHits, LastRequest, …) stable. See .design/vmodel-benchmark.md.

**Provider Routes**: This server handles provider-native routes (/v1/chat/completions, /v1/messages, /v1beta/models/...), NOT gateway routes (/tingly/{scenario}/v1/...). The gateway transforms requests to provider format before forwarding here.

func NewVirtualServer 

func NewVirtualServer(t *testing.T) *VirtualServer

NewVirtualServer creates a new VirtualServer and registers cleanup with t.

func NewVirtualServerForCLI 

func NewVirtualServerForCLI() *VirtualServer

NewVirtualServerForCLI creates a new VirtualServer for CLI use (without testing.T). The caller must call Close() to clean up resources.

func (*VirtualServer) CallCount 

func (vs *VirtualServer) CallCount() int

CallCount returns the total number of requests received.

func (*VirtualServer) Client 

func (vs *VirtualServer) Client() *VirtualClient

Client returns a VirtualClient pre-pointed at this VirtualServer and bound to it.

func (*VirtualServer) Close 

func (vs *VirtualServer) Close()

Close shuts down the virtual server.

func (*VirtualServer) EndpointHits added in v0.260611.1 

func (vs *VirtualServer) EndpointHits(kind EndpointKind) int

EndpointHits returns how many requests hit a specific provider endpoint. Lets tests assert that, e.g., target=openai_responses actually forwarded to /v1/responses rather than silently falling back to /v1/chat/completions.

func (*VirtualServer) LastRequest added in v0.260611.1 

func (vs *VirtualServer) LastRequest(kind EndpointKind) *CapturedRequest

LastRequest returns the most recent request the gateway forwarded to the given provider endpoint, or nil if that endpoint was never hit.

func (*VirtualServer) RegisterScenario 

func (vs *VirtualServer) RegisterScenario(s Scenario)

RegisterScenario registers a scenario so the virtual server can serve its mock responses. A prior scenario with the same name is replaced.

func (*VirtualServer) URL 

func (vs *VirtualServer) URL() string

URL returns the base URL of the virtual server.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.