Documentation
¶
Overview ¶
Package llmtest provides support for testing implementations of llms.Model, in the spirit of testing/fstest.
TestModel ¶
TestModel checks that a model satisfies the llms.Model contract. It is not coupled to *testing.T: it returns an error describing every violation found, so it can run anywhere:
func TestConformance(t *testing.T) {
model, err := myprovider.New()
if err != nil {
t.Fatal(err)
}
if err := llmtest.TestModel(context.Background(), model, "streaming", "tools"); err != nil {
t.Fatal(err)
}
}
The variadic list names the capabilities the model must demonstrate, as fstest.TestFS's expected files do; unnamed capabilities are not exercised. Against a live provider TestModel performs network calls; record them (for example with httprr) to run offline.
The llms/fake package provides a canned-response model that conforms to the baseline contract, playing the role fstest.MapFS plays for io/fs.
TestLLM ¶
TestLLM is an older *testing.T-based harness that probes capabilities by issuing live requests. New tests should prefer TestModel.
Package llmtest provides support for testing LLM implementations.
Following the design of testing/fstest, this package provides a simple TestLLM function that verifies an LLM implementation behaves correctly.
Index ¶
- func TestLLM(t *testing.T, model llms.Model)
- func TestLLMWithOptions(t *testing.T, model llms.Model, opts TestOptions, expected ...string)
- func TestModel(ctx context.Context, model llms.Model, capabilities ...string) error
- func ValidateLLM(model llms.Model) error
- type MockLLM
- func (m *MockLLM) Call(ctx context.Context, prompt string, options ...llms.CallOption) (string, error)
- func (m *MockLLM) GenerateContent(ctx context.Context, messages []llms.MessageContent, ...) (*llms.ContentResponse, error)
- func (m *MockLLM) GenerateContentStream(ctx context.Context, messages []llms.MessageContent, ...) (<-chan llms.ContentResponse, error)
- type TestOptions
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func TestLLM ¶
TestLLM tests an LLM implementation. It performs basic operations and checks that the model behaves correctly. It automatically discovers and tests capabilities by probing the model.
If TestLLM finds any misbehaviors, it reports them via t.Error/t.Fatal.
Typical usage inside a test:
func TestLLM(t *testing.T) {
llm, err := mylllm.New(...)
if err != nil {
t.Fatal(err)
}
llmtest.TestLLM(t, llm)
}
func TestLLMWithOptions ¶
TestLLMWithOptions tests an LLM with specific test options.
func TestModel ¶ added in v0.1.15
TestModel tests a Model implementation.
It calls model.GenerateContent with a series of small requests and checks that the responses satisfy the llms.Model contract: a non-nil response with at least one non-nil choice carrying text content, tool calls, or typed parts, and honored context cancellation.
The capabilities list names behaviors the model must demonstrate. TestModel fails if a named capability misbehaves, and does not exercise capabilities that are not named. Recognized capabilities:
- "multiturn": a human/assistant/human conversation generates.
- "streaming": llms.WithStreamingFunc receives at least one chunk, and an error returned from the callback aborts the request.
- "tools": a declared tool is invoked with valid JSON arguments, and the tool loop round-trips: the response's assistant message replays with a tool result and generation succeeds.
- "usage": token usage is reported in GenerationInfo under a recognized token-count key.
Unrecognized capability names are reported as errors.
If TestModel finds any misbehaviors, it returns an error reporting all of them; the message is a multi-line report.
Against a live provider TestModel performs network calls; record them (for example with httprr) to run it offline.
Example ¶
package main
import (
"context"
"fmt"
"github.com/tmc/langchaingo/llms/fake"
"github.com/tmc/langchaingo/testing/llmtest"
)
func main() {
model := fake.NewFakeLLM([]string{"OK", "Grace"})
if err := llmtest.TestModel(context.Background(), model, "multiturn"); err != nil {
fmt.Println(err)
}
fmt.Println("conforms")
}
Output: conforms
func ValidateLLM ¶
ValidateLLM checks if a model satisfies basic requirements without running tests. It returns an error describing what's wrong, or nil if the model is valid.
Types ¶
type MockLLM ¶
type MockLLM struct {
// Response to return from Call
CallResponse string
CallError error
// Response to return from GenerateContent
GenerateResponse *llms.ContentResponse
GenerateError error
// Track calls for verification
CallCount int
GenerateCount int
LastPrompt string
LastMessages []llms.MessageContent
}
MockLLM provides a simple mock implementation for testing.
func (*MockLLM) Call ¶
func (m *MockLLM) Call(ctx context.Context, prompt string, options ...llms.CallOption) (string, error)
Call implements llms.Model
func (*MockLLM) GenerateContent ¶
func (m *MockLLM) GenerateContent(ctx context.Context, messages []llms.MessageContent, options ...llms.CallOption) (*llms.ContentResponse, error)
GenerateContent implements llms.Model
func (*MockLLM) GenerateContentStream ¶
func (m *MockLLM) GenerateContentStream(ctx context.Context, messages []llms.MessageContent, options ...llms.CallOption) (<-chan llms.ContentResponse, error)
GenerateContentStream implements streaming
type TestOptions ¶
type TestOptions struct {
// Timeout for each test operation
Timeout time.Duration
// Skip specific test categories
SkipCall bool
SkipGenerateContent bool
SkipStreaming bool
// Custom test prompts
TestPrompt string
TestMessages []llms.MessageContent
// For providers that need special options
CallOptions []llms.CallOption
}
TestOptions configures test execution.
Source Files