protocol_validate

package
v0.260414.2000 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 14, 2026 License: MPL-2.0 Imports: 19 Imported by: 0

Documentation

Overview

Package protocoltest provides a framework for end-to-end validation of the model gateway's protocol transformation layer.

Architecture

  1. server_validate.VirtualServer — a mock HTTP provider that speaks OpenAI, Anthropic, and Google response formats. Conceptually a "virtual model" for testing.

  2. TestEnv — wires a real gateway Server (with transform pipeline) to a VirtualServer, configures routing rules, and provides SendAs() for round-trip testing.

  3. Matrix — executes the full cross-product of sources × targets × scenarios × streaming modes.

Note: The existing internal/virtualmodel package is a production Gin server. This package (protocoltest) is the test-only framework. Future integration with virtualmodel is planned once both stabilize.

Usage 

env := protocoltest.NewTestEnv(t)
defer env.Close()
env.SetupRoute(protocol.TypeAnthropicV1, protocol.TypeOpenAIChat, protocoltest.TextScenario())
result := env.SendAs(t, protocol.TypeAnthropicV1, protocoltest.TextScenario(), false)
assert.Equal(t, "assistant", result.Role)

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Assertion 

type Assertion struct {
	Name  string
	Check func(r *RoundTripResult) error
}

Assertion is a named check applied to a RoundTripResult.

func AssertContentContains 

func AssertContentContains(substring string) Assertion

AssertContentContains returns an Assertion that the response content contains substring.

func AssertContentEquals 

func AssertContentEquals(expected string) Assertion

AssertContentEquals returns an Assertion that the response content equals expected.

func AssertFinishReason 

func AssertFinishReason(expected string) Assertion

AssertFinishReason returns an Assertion that the finish_reason equals expected.

func AssertHTTPStatus 

func AssertHTTPStatus(expected int) Assertion

AssertHTTPStatus returns an Assertion that the HTTP status code equals expected.

func AssertHasThinking 

func AssertHasThinking() Assertion

AssertHasThinking returns an Assertion that thinking content is non-empty.

func AssertHasToolCalls 

func AssertHasToolCalls(count int) Assertion

AssertHasToolCalls returns an Assertion that exactly count tool calls are present.

func AssertModelContains 

func AssertModelContains(substring string) Assertion

AssertModelContains returns an Assertion that the model name contains substring.

func AssertNoThinking 

func AssertNoThinking() Assertion

AssertNoThinking returns an Assertion that no thinking content is present.

func AssertRoleEquals 

func AssertRoleEquals(expected string) Assertion

AssertRoleEquals returns an Assertion that the response role equals expected.

func AssertStreamEventCount 

func AssertStreamEventCount(min int) Assertion

AssertStreamEventCount returns an Assertion that at least min SSE events were received.

func AssertToolCallArgs 

func AssertToolCallArgs(index int, key, value string) Assertion

AssertToolCallArgs returns an Assertion that tool call at index has key=value in its JSON args.

func AssertToolCallName 

func AssertToolCallName(index int, name string) Assertion

AssertToolCallName returns an Assertion that the tool call at index has the given name.

func AssertUsageNonZero 

func AssertUsageNonZero() Assertion

AssertUsageNonZero returns an Assertion that at least one token count > 0.

type AssertionError 

type AssertionError struct {
	Assertion string // assertion name
	Error     string // error message
	Context   string // additional context (truncated body, etc.)
}

AssertionError represents a single assertion failure.

type CapturedRequest 

type CapturedRequest struct {
	// Headers contains the request headers
	Headers http.Header

	// Body contains the request body
	Body []byte

	// Method is the HTTP method
	Method string

	// Path is the request path
	Path string
}

CapturedRequest represents a request captured by the virtual server

type Matrix 

type Matrix struct {
	Sources   []protocol.APIType
	Targets   []protocol.APIType
	Scenarios []Scenario
	Streaming []bool
}

Matrix defines the cross-product of source protocols, target protocols, scenarios, and streaming modes to validate.

func DefaultMatrix 

func DefaultMatrix() *Matrix

DefaultMatrix returns the full validation matrix covering all supported protocol combinations, all built-in scenarios, and both streaming modes.

func (*Matrix) ExecuteAll 

func (m *Matrix) ExecuteAll() []TestResult

ExecuteAll runs all matrix combinations and returns structured results. This is a pure function that can be called from both tests and CLI. It does not use testing.T, making it suitable for standalone execution.

Optimization: Reuses TestEnv per scenario to reduce server startup overhead. Each scenario creates one TestEnv that is reused for all its combinations.

func (*Matrix) OnlyScenarios 

func (m *Matrix) OnlyScenarios(names ...string) *Matrix

OnlyScenarios returns a copy of the Matrix filtered to only the named scenarios.

func (*Matrix) Run 

func (m *Matrix) Run(t *testing.T)

Run executes all matrix combinations as subtests under t. Each combination runs in its own TestEnv so state is isolated.

type MockResponseBuilder 

type MockResponseBuilder = server_validate.MockResponseBuilder

MockResponseBuilder re-exports server_validate.MockResponseBuilder for convenience.

type ProfileScenario 

type ProfileScenario struct {
	// Name is the scenario name
	Name string

	// Description describes what the scenario tests
	Description string

	// MockResponses are the mock responses keyed by API style
	MockResponses map[server_validate.APIStyle]server_validate.MockResponseBuilder
}

ProfileScenario defines a test scenario for profile testing Unlike matrix scenarios which test protocol transformations, profile scenarios test client implementations (OAuth, path rewriting, etc.)

func GetScenarioByName 

func GetScenarioByName(name string) (ProfileScenario, bool)

GetScenarioByName returns a profile scenario by name

func ProfileScenarios 

func ProfileScenarios() []ProfileScenario

ProfileScenarios returns all built-in profile test scenarios

func ProfileStreamingTextScenario 

func ProfileStreamingTextScenario() ProfileScenario

ProfileStreamingTextScenario returns a scenario for streaming text response

func ProfileTextScenario 

func ProfileTextScenario() ProfileScenario

ProfileTextScenario returns a scenario for basic text request/response

func ProfileToolUseScenario 

func ProfileToolUseScenario() ProfileScenario

ProfileToolUseScenario returns a scenario for tool use request/response

type ProfileTestEnv 

type ProfileTestEnv struct {
	// contains filtered or unexported fields
}

ProfileTestEnv provides an isolated test environment for profile testing It includes: - A temporary config directory - A gateway server with virtual provider - Routing rules configured for the profile - A virtual server that captures requests for validation

func NewProfileTestEnv 

func NewProfileTestEnv(profileType ProfileType) (*ProfileTestEnv, error)

NewProfileTestEnv creates a new profile test environment The environment is isolated with a temporary config directory and must be cleaned up with Close() when done

func (*ProfileTestEnv) AppConfig 

func (env *ProfileTestEnv) AppConfig() *serverconfig.Config

AppConfig returns the application configuration

func (*ProfileTestEnv) BaseURL 

func (env *ProfileTestEnv) BaseURL() string

BaseURL returns the base URL of the gateway server

func (*ProfileTestEnv) Close 

func (env *ProfileTestEnv) Close(preserve bool) error

Close cleans up the test environment If preserve is true, the config directory is kept for inspection

func (*ProfileTestEnv) ConfigDir 

func (env *ProfileTestEnv) ConfigDir() string

ConfigDir returns the temporary config directory path

func (*ProfileTestEnv) ModelToken 

func (env *ProfileTestEnv) ModelToken() string

ModelToken returns the model token for requests

func (*ProfileTestEnv) SetupProfile 

func (env *ProfileTestEnv) SetupProfile(profileType ProfileType, providerName string, modelName string) error

SetupProfile configures the environment for a specific profile type This creates the necessary provider and routing rules

func (*ProfileTestEnv) VirtualServerURL 

func (env *ProfileTestEnv) VirtualServerURL() string

VirtualServerURL returns the URL of the virtual server

type ProfileTestResult 

type ProfileTestResult struct {
	// Name is the test name
	Name string

	// Profile is the profile type being tested
	Profile ProfileType

	// Scenario is the test scenario (e.g., "text", "streaming", "tool_use")
	Scenario string

	// Passed indicates whether the test passed
	Passed bool

	// Skipped indicates whether the test was skipped
	Skipped bool

	// SkipReason explains why the test was skipped
	SkipReason string

	// Errors contains any assertion errors
	Errors []AssertionError

	// Duration is how long the test took
	Duration int64 // milliseconds

	// HTTPStatus is the HTTP status code received
	HTTPStatus int

	// RequestHeaders contains the request headers sent to the virtual server
	RequestHeaders http.Header

	// RequestBody contains the request body sent to the virtual server
	RequestBody []byte

	// ResponseBody contains the raw response body
	ResponseBody []byte
}

ProfileTestResult represents the result of a single profile test

type ProfileType 

type ProfileType string

ProfileType represents the type of agent profile to test 

const (
	ProfileTypeClaudeCode ProfileType = "claude"
	ProfileTypeCodex      ProfileType = "codex"
	ProfileTypeOpenCode   ProfileType = "opencode"
)

func (ProfileType) Scenario 

func (pt ProfileType) Scenario() typ.RuleScenario

Scenario returns the corresponding RuleScenario for this profile

func (ProfileType) String 

func (pt ProfileType) String() string

String returns the string representation of ProfileType

type RoundTripResult 

type RoundTripResult struct {
	// Source and target context
	SourceProtocol protocol.APIType
	TargetProtocol protocol.APIType
	ScenarioName   string
	IsStreaming    bool

	// HTTP layer
	HTTPStatus   int
	RawBody      []byte
	StreamEvents []string // raw SSE event lines (streaming only)

	// Extracted semantics (populated by the framework after parsing)
	Content         string
	Role            string
	Model           string
	FinishReason    string
	ToolCalls       []ToolCallResult
	ThinkingContent string
	Usage           *TokenUsage
}

RoundTripResult captures the full result of a round-trip request through the gateway.

type Scenario 

type Scenario struct {
	Name        string
	Description string
	Tags        []string

	// MockResponses keyed by provider APIStyle ("openai", "anthropic", "google").
	MockResponses map[server_validate.APIStyle]MockResponseBuilder

	// Assertions run after every round-trip for this scenario.
	Assertions []Assertion
}

Scenario is a named test scenario describing:

  • What the mock provider should return (MockResponses per APIStyle)
  • What assertions to run on the round-trip result

func AllScenarios 

func AllScenarios() []Scenario

AllScenarios returns the full set of built-in validation scenarios.

func ErrorScenario 

func ErrorScenario() Scenario

ErrorScenario tests that provider error responses are forwarded to the client.

func MultiTurnScenario 

func MultiTurnScenario() Scenario

MultiTurnScenario tests a conversation with system prompt + 2 turns of history.

func StreamingTextScenario 

func StreamingTextScenario() Scenario

StreamingTextScenario tests SSE streaming for a plain text response.

func StreamingToolUseScenario 

func StreamingToolUseScenario() Scenario

StreamingToolUseScenario tests SSE streaming for a tool call response.

func TextScenario 

func TextScenario() Scenario

TextScenario is the baseline: a single user message and a plain text reply.

func ThinkingScenario 

func ThinkingScenario() Scenario

ThinkingScenario tests that extended thinking blocks are present in Anthropic responses.

func ToolResultScenario 

func ToolResultScenario() Scenario

ToolResultScenario tests a multi-turn conversation including a tool result message.

func ToolUseScenario 

func ToolUseScenario() Scenario

ToolUseScenario exercises a single tool/function call.

type TestEnv 

type TestEnv struct {
	// contains filtered or unexported fields
}

TestEnv wires a real gateway Server (with the full transform pipeline) to a VirtualServer. It manages config, routing rules, and provides SendAs() for full round-trip testing.

func NewTestEnv 

func NewTestEnv(t *testing.T) *TestEnv

NewTestEnv creates a TestEnv with a fresh gateway config and a new VirtualServer. All resources are cleaned up via t.Cleanup.

func NewTestEnvForCLI 

func NewTestEnvForCLI() (*TestEnv, error)

NewTestEnvForCLI creates a TestEnv for CLI use (without testing.T). Resources must be cleaned up via explicit Close() call.

func (*TestEnv) Close 

func (env *TestEnv) Close()

Close cleans up resources. For testing mode, it's a no-op (resources are cleaned up via t.Cleanup). For CLI mode, it closes the servers and removes the config directory.

func (*TestEnv) SendAs 

func (env *TestEnv) SendAs(t *testing.T, source protocol.APIType, s Scenario, streaming bool) *RoundTripResult

SendAs sends a request to the gateway as the given source protocol, using the request model configured by SetupRoute, and returns the parsed result.

Streaming requests use the real httptest.Server (env.gatewayServer) because httptest.ResponseRecorder does not support Gin's streaming/SSE machinery. Non-streaming requests use the recorder for simplicity.

func (*TestEnv) SendAsCLI 

func (env *TestEnv) SendAsCLI(source protocol.APIType, s Scenario, streaming bool) (*RoundTripResult, error)

SendAsCLI sends a request to the gateway as the given source protocol, using the request model configured by SetupRoute, and returns the parsed result. This version is for CLI use and returns errors instead of calling t.Fatalf.

Streaming requests use the real httptest.Server (env.gatewayServer) because httptest.ResponseRecorder does not support Gin's streaming/SSE machinery. Non-streaming requests use the recorder for simplicity.

func (*TestEnv) SetupRoute 

func (env *TestEnv) SetupRoute(source, target protocol.APIType, s Scenario)

SetupRoute configures a gateway rule that routes source protocol requests to the virtual server acting as a target protocol provider.

The virtual server is pre-registered with the scenario's mock responses.

func (*TestEnv) VirtualCallCount 

func (env *TestEnv) VirtualCallCount() int

VirtualCallCount returns the number of requests received by the virtual server.

func (*TestEnv) VirtualURL 

func (env *TestEnv) VirtualURL() string

VirtualURL returns the URL of the underlying virtual server.

type TestResult 

type TestResult struct {
	// Test identification
	Name      string // Full test name: "scenario/source/target/mode"
	Scenario  string // Scenario name: "text", "tool_use", etc.
	Source    protocol.APIType
	Target    protocol.APIType
	Streaming bool

	// Test outcome
	Passed     bool   // true if all assertions passed
	Skipped    bool   // true if test was skipped
	SkipReason string // reason for skipping

	// Error details
	Errors   []AssertionError // list of assertion failures
	Duration time.Duration    // test execution time

	// Response details (for debugging/verbose output)
	HTTPStatus int              // HTTP status code
	Response   *RoundTripResult // full round-trip result
}

TestResult represents the outcome of a single matrix test combination. This is returned by Matrix.ExecuteAll() for CLI and other non-testing contexts.

type TokenUsage 

type TokenUsage struct {
	InputTokens  int
	OutputTokens int
}

TokenUsage holds token counts extracted from a provider response.

type ToolCallResult 

type ToolCallResult struct {
	ID        string
	Name      string
	Arguments string // raw JSON string
}

ToolCallResult holds a single tool/function call extracted from a response.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.