Documentation
¶
Overview ¶
Package llmtest provides utilities for testing LLM implementations.
Inspired by Go's testing/fstest package, llmtest offers a simple, backend-independent way to verify that LLM implementations conform to the expected interfaces and behaviors.
Design Philosophy ¶
Following the principles of testing/fstest:
- Minimal API surface - one main function (TestLLM)
- Automatic capability discovery - no configuration required
- Comprehensive by default - tests all detected capabilities
- Interface testing - works with any llms.Model implementation
- Simple usage pattern - just pass the model to test
Usage ¶
Testing an LLM implementation is straightforward:
func TestMyLLM(t *testing.T) {
llm, err := mylllm.New()
if err != nil {
t.Fatal(err)
}
llmtest.TestLLM(t, llm)
}
Automatic Capability Discovery ¶
The package automatically detects and tests supported capabilities:
- Basic operations (Call, GenerateContent)
- Streaming (unless the call site passes WithoutStreaming)
- Tool/Function calling (probed with a test tool; a door that takes the tools must answer with a call unless the call site passes WithoutToolCalls)
- A door that needs particular options to answer at all, such as the output budget a thinking model spends before its first word, declares them with WithCallOptions; every checked request carries them
- Token counting (if usage information provided)
- Context caching (if implemented)
Mock Implementation ¶
A MockLLM is provided for testing without making actual API calls. It answers in prose and never calls a tool, so it declares WithoutToolCalls:
mock := &llmtest.MockLLM{
GenerateResponse: &llms.ContentResponse{
Choices: []*llms.ContentChoice{{Content: "Hello"}},
},
}
llmtest.TestLLM(t, mock, llmtest.WithoutToolCalls())
Parallel Testing ¶
All tests run in parallel by default for better performance:
- Core tests (Call, GenerateContent) run concurrently
- Capability tests run in parallel when detected
- Safe for concurrent execution with independent contexts
Provider Coverage ¶
The package is used to test all LangChain Go providers: anthropic, bedrock, fake, googleai, huggingface, mistral, ollama, openai, and more.
Package llmtest provides support for testing LLM implementations.
Following the design of testing/fstest, this package provides a simple TestLLM function that verifies an LLM implementation behaves correctly.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func TestLLM ¶
TestLLM tests an LLM implementation. It performs basic operations and checks that the model behaves correctly. It automatically discovers and tests capabilities by probing the model.
If TestLLM finds any misbehaviors, it reports them via t.Error/t.Fatal.
Typical usage inside a test:
func TestLLM(t *testing.T) {
llm, err := mylllm.New(...)
if err != nil {
t.Fatal(err)
}
llmtest.TestLLM(t, llm)
}
func TestLLMWithOptions ¶
TestLLMWithOptions tests an LLM with specific test options.
func ValidateLLM ¶
ValidateLLM checks if a model satisfies basic requirements without running tests. It returns an error describing what's wrong, or nil if the model is valid.
Types ¶
type MockLLM ¶
type MockLLM struct {
// Response to return from Call
CallResponse string
CallError error
// Response to return from GenerateContent
GenerateResponse *llms.ContentResponse
GenerateError error
CallCount int
GenerateCount int
LastPrompt string
LastMessages []llms.MessageContent
// contains filtered or unexported fields
}
MockLLM provides a simple mock implementation for testing.
func (*MockLLM) Call ¶
func (m *MockLLM) Call(ctx context.Context, prompt string, options ...llms.CallOption) (string, error)
Call implements llms.Model
func (*MockLLM) GenerateContent ¶
func (m *MockLLM) GenerateContent(ctx context.Context, messages []llms.MessageContent, options ...llms.CallOption) (*llms.ContentResponse, error)
GenerateContent implements llms.Model
type Option ¶
type Option func(*TestOptions)
Option declares, at the call site, what the door under test cannot do or what it needs to be called with.
func WithCallOptions ¶
func WithCallOptions(opts ...llms.CallOption) Option
WithCallOptions declares options the door needs on every request to answer at all, such as the output budget a thinking model spends on thoughts before its first word. The suite appends them after its own, so they override its budgets and other defaults, while the Streaming test still installs its text collector last and the tool probe stays a pure capability probe. Every answer is held to the same checks as before.
func WithoutStreaming ¶
func WithoutStreaming() Option
WithoutStreaming declares a door that does not deliver a streaming callback, so the suite must not hold it to that contract.
func WithoutToolCalls ¶
func WithoutToolCalls() Option
WithoutToolCalls declares a door that takes tools but never answers with a call, so the suite must not hold it to that contract.
type TestOptions ¶
type TestOptions struct {
// Timeout for each test operation
Timeout time.Duration
// Skip specific test categories
SkipCall bool
SkipGenerateContent bool
SkipStreaming bool
SkipToolCalls bool
// Custom test prompts
TestPrompt string
TestMessages []llms.MessageContent
// For providers that need special options
CallOptions []llms.CallOption
}
TestOptions configures test execution.
Source Files