Documentation
¶
Overview ¶
Package providers implements multi-LLM provider support with unified interfaces.
This package provides a common abstraction for predict-based LLM providers including OpenAI, Anthropic Claude, and Google Gemini. It handles:
- Predict completion requests with streaming support
- Tool/function calling with provider-specific formats
- Cost tracking and token usage calculation
- Rate limiting and error handling
All providers implement the Provider interface for basic predict, and ToolSupport interface for function calling capabilities.
Index ¶
- func CheckHTTPError(resp *http.Response) error
- func HasAudioSupport(p Provider) bool
- func HasImageSupport(p Provider) bool
- func HasVideoSupport(p Provider) bool
- func IsFormatSupported(p Provider, contentType string, mimeType string) bool
- func IsValidationAbort(err error) bool
- func LoadFileAsBase64(filePath string) (string, error)
- func RegisterProviderFactory(providerType string, factory ProviderFactory)
- func SetErrorResponse(predictResp *PredictionResponse, respBody []byte, start time.Time)
- func StringPtr(s string) *string
- func SupportsMultimodal(p Provider) bool
- func UnmarshalJSON(respBody []byte, v interface{}, predictResp *PredictionResponse, ...) error
- func ValidateMultimodalMessage(p Provider, msg types.Message) error
- func ValidateMultimodalRequest(p MultimodalSupport, req PredictionRequest) error
- type AudioStreamingCapabilities
- type BaseProvider
- type ExecutionResult
- type ImageDetail
- type MultimodalCapabilities
- type MultimodalSupport
- type MultimodalToolSupport
- type PredictionRequest
- type PredictionResponse
- type Pricing
- type Provider
- type ProviderDefaults
- type ProviderFactory
- type ProviderSpec
- type Registry
- type SSEScanner
- type StreamChunk
- type StreamEvent
- type StreamInputRequest
- type StreamInputSession
- type StreamInputSupport
- type StreamObserver
- type StreamingCapabilities
- type ToolDescriptor
- type ToolResult
- type ToolSupport
- type UnsupportedContentError
- type UnsupportedProviderError
- type ValidationAbortError
- type VideoResolution
- type VideoStreamingCapabilities
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CheckHTTPError ¶ added in v1.1.0
CheckHTTPError checks if HTTP response is an error and returns formatted error with body
func HasAudioSupport ¶ added in v1.1.0
HasAudioSupport checks if a provider supports audio inputs
func HasImageSupport ¶ added in v1.1.0
HasImageSupport checks if a provider supports image inputs
func HasVideoSupport ¶ added in v1.1.0
HasVideoSupport checks if a provider supports video inputs
func IsFormatSupported ¶ added in v1.1.0
IsFormatSupported checks if a provider supports a specific media format (MIME type)
func IsValidationAbort ¶
IsValidationAbort checks if an error is a validation abort
func LoadFileAsBase64 ¶ added in v1.1.0
LoadFileAsBase64 reads a file and returns its content as a base64-encoded string. It supports home directory expansion (~/path) and is used by multimodal providers to load image files.
func RegisterProviderFactory ¶ added in v1.1.0
func RegisterProviderFactory(providerType string, factory ProviderFactory)
RegisterProviderFactory registers a factory function for a provider type
func SetErrorResponse ¶ added in v1.1.0
func SetErrorResponse(predictResp *PredictionResponse, respBody []byte, start time.Time)
SetErrorResponse sets latency and raw body on error responses
func StringPtr ¶ added in v1.1.0
StringPtr is a helper function that returns a pointer to a string. This is commonly used across provider implementations for optional fields.
func SupportsMultimodal ¶ added in v1.1.0
SupportsMultimodal checks if a provider implements multimodal support
func UnmarshalJSON ¶ added in v1.1.0
func UnmarshalJSON(respBody []byte, v interface{}, predictResp *PredictionResponse, start time.Time) error
UnmarshalJSON unmarshals JSON with error recovery that sets latency and raw response
func ValidateMultimodalMessage ¶ added in v1.1.0
ValidateMultimodalMessage checks if a message's multimodal content is supported by the provider
func ValidateMultimodalRequest ¶ added in v1.1.0
func ValidateMultimodalRequest(p MultimodalSupport, req PredictionRequest) error
ValidateMultimodalRequest validates all messages in a predict request for multimodal compatibility This is a helper function to reduce duplication across provider implementations
Types ¶
type AudioStreamingCapabilities ¶ added in v1.1.0
type AudioStreamingCapabilities struct {
// SupportedEncodings lists supported audio encodings
// Common values: "pcm", "opus", "mp3", "aac"
SupportedEncodings []string `json:"supported_encodings"`
// SupportedSampleRates lists supported sample rates in Hz
// Common values: 8000, 16000, 24000, 44100, 48000
SupportedSampleRates []int `json:"supported_sample_rates"`
// SupportedChannels lists supported channel counts
// Common values: 1 (mono), 2 (stereo)
SupportedChannels []int `json:"supported_channels"`
// SupportedBitDepths lists supported bit depths
// Common values: 16, 24, 32
SupportedBitDepths []int `json:"supported_bit_depths,omitempty"`
// PreferredEncoding is the recommended encoding for best quality/latency
PreferredEncoding string `json:"preferred_encoding"`
// PreferredSampleRate is the recommended sample rate
PreferredSampleRate int `json:"preferred_sample_rate"`
}
AudioStreamingCapabilities describes audio streaming support.
type BaseProvider ¶ added in v1.1.0
type BaseProvider struct {
// contains filtered or unexported fields
}
BaseProvider provides common functionality shared across all provider implementations. It should be embedded in concrete provider structs to avoid code duplication.
func NewBaseProvider ¶ added in v1.1.0
func NewBaseProvider(id string, includeRawOutput bool, client *http.Client) BaseProvider
NewBaseProvider creates a new BaseProvider with common fields
func NewBaseProviderWithAPIKey ¶ added in v1.1.0
func NewBaseProviderWithAPIKey(id string, includeRawOutput bool, primaryKey, fallbackKey string) (BaseProvider, string)
NewBaseProviderWithAPIKey creates a BaseProvider and retrieves API key from environment It tries the primary key first, then falls back to the secondary key if primary is empty.
func (*BaseProvider) Close ¶ added in v1.1.0
func (b *BaseProvider) Close() error
Close closes the HTTP client's idle connections
func (*BaseProvider) GetHTTPClient ¶ added in v1.1.0
func (b *BaseProvider) GetHTTPClient() *http.Client
GetHTTPClient returns the underlying HTTP client for provider-specific use
func (*BaseProvider) ID ¶ added in v1.1.0
func (b *BaseProvider) ID() string
ID returns the provider ID
func (*BaseProvider) ShouldIncludeRawOutput ¶ added in v1.1.0
func (b *BaseProvider) ShouldIncludeRawOutput() bool
ShouldIncludeRawOutput returns whether to include raw API responses in output
func (*BaseProvider) SupportsStreaming ¶ added in v1.1.0
func (b *BaseProvider) SupportsStreaming() bool
SupportsStreaming returns true by default (can be overridden by providers that don't support streaming)
type ExecutionResult ¶
type ExecutionResult interface{}
Forward declare ExecutionResult to avoid circular import
type ImageDetail ¶ added in v1.1.0
type ImageDetail string
ImageDetail specifies the level of detail for image processing
const ( ImageDetailLow ImageDetail = "low" // Faster, less detailed analysis ImageDetailHigh ImageDetail = "high" // Slower, more detailed analysis ImageDetailAuto ImageDetail = "auto" // Provider chooses automatically )
type MultimodalCapabilities ¶ added in v1.1.0
type MultimodalCapabilities struct {
SupportsImages bool // Provider can process image inputs
SupportsAudio bool // Provider can process audio inputs
SupportsVideo bool // Provider can process video inputs
ImageFormats []string // Supported image MIME types (e.g., "image/jpeg", "image/png")
AudioFormats []string // Supported audio MIME types (e.g., "audio/mpeg", "audio/wav")
VideoFormats []string // Supported video MIME types (e.g., "video/mp4")
MaxImageSizeMB int // Maximum image size in megabytes (0 = unlimited/unknown)
MaxAudioSizeMB int // Maximum audio size in megabytes (0 = unlimited/unknown)
MaxVideoSizeMB int // Maximum video size in megabytes (0 = unlimited/unknown)
}
MultimodalCapabilities describes what types of multimodal content a provider supports
type MultimodalSupport ¶ added in v1.1.0
type MultimodalSupport interface {
Provider // Extends the base Provider interface
// GetMultimodalCapabilities returns what types of multimodal content this provider supports
GetMultimodalCapabilities() MultimodalCapabilities
// PredictMultimodal performs a predict request with multimodal message content
// Messages in the request can contain Parts with images, audio, or video
PredictMultimodal(ctx context.Context, req PredictionRequest) (PredictionResponse, error)
// PredictMultimodalStream performs a streaming predict request with multimodal content
PredictMultimodalStream(ctx context.Context, req PredictionRequest) (<-chan StreamChunk, error)
}
MultimodalSupport interface for providers that support multimodal inputs
func GetMultimodalProvider ¶ added in v1.1.0
func GetMultimodalProvider(p Provider) MultimodalSupport
GetMultimodalProvider safely casts a provider to MultimodalSupport Returns nil if the provider doesn't support multimodal
type MultimodalToolSupport ¶ added in v1.1.0
type MultimodalToolSupport interface {
MultimodalSupport // Extends multimodal support
ToolSupport // Extends tool support
// PredictMultimodalWithTools performs a predict request with both multimodal content and tools
PredictMultimodalWithTools(ctx context.Context, req PredictionRequest, tools interface{}, toolChoice string) (PredictionResponse, []types.MessageToolCall, error)
}
MultimodalToolSupport interface for providers that support both multimodal and tools
type PredictionRequest ¶ added in v1.1.0
type PredictionRequest struct {
System string `json:"system"`
Messages []types.Message `json:"messages"`
Temperature float32 `json:"temperature"`
TopP float32 `json:"top_p"`
MaxTokens int `json:"max_tokens"`
Seed *int `json:"seed,omitempty"`
Metadata map[string]interface{} `json:"metadata,omitempty"` // Optional metadata for provider-specific context
}
PredictionRequest represents a request to a predict provider
type PredictionResponse ¶ added in v1.1.0
type PredictionResponse struct {
Content string `json:"content"`
Parts []types.ContentPart `json:"parts,omitempty"` // Multimodal content parts (text, image, audio, video)
CostInfo *types.CostInfo `json:"cost_info,omitempty"` // Cost breakdown for this response (includes token counts)
Latency time.Duration `json:"latency"`
Raw []byte `json:"raw,omitempty"`
RawRequest interface{} `json:"raw_request,omitempty"` // Raw API request (for debugging)
ToolCalls []types.MessageToolCall `json:"tool_calls,omitempty"` // Tools called in this response
}
PredictionResponse represents a response from a predict provider
type Provider ¶
type Provider interface {
ID() string
Predict(ctx context.Context, req PredictionRequest) (PredictionResponse, error)
// Streaming support
PredictStream(ctx context.Context, req PredictionRequest) (<-chan StreamChunk, error)
SupportsStreaming() bool
ShouldIncludeRawOutput() bool
Close() error // Close cleans up provider resources (e.g., HTTP connections)
// CalculateCost calculates cost breakdown for given token counts
CalculateCost(inputTokens, outputTokens, cachedTokens int) types.CostInfo
}
Provider interface defines the contract for predict providers
func CreateProviderFromSpec ¶
func CreateProviderFromSpec(spec ProviderSpec) (Provider, error)
CreateProviderFromSpec creates a provider implementation from a spec. Returns an error if the provider type is unsupported.
type ProviderDefaults ¶
ProviderDefaults holds default parameters for providers
type ProviderFactory ¶ added in v1.1.0
type ProviderFactory func(spec ProviderSpec) (Provider, error)
ProviderFactory is a function that creates a provider from a spec
type ProviderSpec ¶
type ProviderSpec struct {
ID string
Type string
Model string
BaseURL string
Defaults ProviderDefaults
IncludeRawOutput bool
AdditionalConfig map[string]interface{} // Flexible key-value pairs for provider-specific configuration
}
ProviderSpec holds the configuration needed to create a provider instance
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry manages available providers
func (*Registry) Close ¶
Close closes all registered providers and cleans up their resources. Returns the first error encountered, if any.
type SSEScanner ¶
type SSEScanner struct {
// contains filtered or unexported fields
}
SSEScanner scans Server-Sent Events (SSE) streams
func NewSSEScanner ¶
func NewSSEScanner(r io.Reader) *SSEScanner
NewSSEScanner creates a new SSE scanner
type StreamChunk ¶
type StreamChunk struct {
// Content is the accumulated content so far
Content string `json:"content"`
// Delta is the new content in this chunk
Delta string `json:"delta"`
// MediaDelta contains new media content in this chunk (audio, video, images)
// Uses the same MediaContent type as non-streaming messages for API consistency.
MediaDelta *types.MediaContent `json:"media_delta,omitempty"`
// TokenCount is the total number of tokens so far
TokenCount int `json:"token_count"`
// DeltaTokens is the number of tokens in this delta
DeltaTokens int `json:"delta_tokens"`
// ToolCalls contains accumulated tool calls (for assistant messages that invoke tools)
ToolCalls []types.MessageToolCall `json:"tool_calls,omitempty"`
// FinishReason is nil until stream is complete
// Values: "stop", "length", "content_filter", "tool_calls", "error", "validation_failed", "cancelled"
FinishReason *string `json:"finish_reason,omitempty"`
// Error is set if an error occurred during streaming
Error error `json:"error,omitempty"`
// Metadata contains provider-specific metadata
Metadata map[string]interface{} `json:"metadata,omitempty"`
// FinalResult contains the complete execution result (only set in the final chunk)
FinalResult ExecutionResult `json:"final_result,omitempty"`
// CostInfo contains cost breakdown (only present in final chunk when FinishReason != nil)
CostInfo *types.CostInfo `json:"cost_info,omitempty"`
}
StreamChunk represents a batch of tokens with metadata
type StreamEvent ¶
type StreamEvent struct {
// Type is the event type: "chunk", "complete", "error"
Type string `json:"type"`
// Chunk contains the stream chunk data
Chunk *StreamChunk `json:"chunk,omitempty"`
// Error is set for error events
Error error `json:"error,omitempty"`
// Timestamp is when the event occurred
Timestamp time.Time `json:"timestamp"`
}
StreamEvent is sent to observers for monitoring
type StreamInputRequest ¶ added in v1.1.0
type StreamInputRequest struct {
// Config specifies the media streaming configuration
Config types.StreamingMediaConfig `json:"config"`
// SystemMsg is the system message for the conversation
SystemMsg string `json:"system_msg,omitempty"`
// InitialText is optional text to send before streaming starts
// This can be used to provide initial context or instructions
InitialText string `json:"initial_text,omitempty"`
// Temperature controls randomness in responses (0.0 to 2.0)
Temperature float32 `json:"temperature,omitempty"`
// MaxTokens limits the response length
MaxTokens int `json:"max_tokens,omitempty"`
// Metadata contains provider-specific configuration
Metadata map[string]interface{} `json:"metadata,omitempty"`
}
StreamInputRequest configures a new streaming input session.
func (*StreamInputRequest) Validate ¶ added in v1.1.0
func (r *StreamInputRequest) Validate() error
Validate checks if the StreamInputRequest is valid
type StreamInputSession ¶ added in v1.1.0
type StreamInputSession interface {
// SendChunk sends a media chunk to the provider.
// Returns an error if the chunk cannot be sent or the session is closed.
// This method is safe to call from multiple goroutines.
SendChunk(ctx context.Context, chunk *types.MediaChunk) error
// SendText sends a text message to the provider during the streaming session.
// This is useful for sending text prompts or instructions during audio streaming.
SendText(ctx context.Context, text string) error
// Response returns a receive-only channel for streaming responses.
// The channel is closed when the session ends or encounters an error.
// Consumers should read from this channel in a separate goroutine.
Response() <-chan StreamChunk
// Close ends the streaming session and releases resources.
// After calling Close, SendChunk and SendText will return errors.
// The Response channel will be closed.
// Close is safe to call multiple times.
Close() error
// Error returns any error that occurred during the session.
// Returns nil if no error has occurred.
Error() error
// Done returns a channel that's closed when the session ends.
// This is useful for select statements to detect session completion.
Done() <-chan struct{}
}
StreamInputSession manages a bidirectional streaming session with a provider. The session allows sending media chunks (e.g., audio from a microphone) and receiving streaming responses from the LLM.
Example usage:
session, err := provider.CreateStreamSession(ctx, StreamInputRequest{
Config: types.StreamingMediaConfig{
Type: types.ContentTypeAudio,
ChunkSize: 8192,
SampleRate: 16000,
Encoding: "pcm",
Channels: 1,
},
SystemMsg: "You are a helpful voice assistant",
})
if err != nil {
return err
}
defer session.Close()
// Send audio chunks in a goroutine
go func() {
for chunk := range micInput {
if err := session.SendChunk(ctx, chunk); err != nil {
log.Printf("send error: %v", err)
break
}
}
}()
// Receive responses
for chunk := range session.Response() {
if chunk.Error != nil {
log.Printf("response error: %v", chunk.Error)
break
}
fmt.Print(chunk.Delta)
}
type StreamInputSupport ¶ added in v1.1.0
type StreamInputSupport interface {
Provider // Extends the base Provider interface
// CreateStreamSession creates a new bidirectional streaming session.
// The session remains active until Close() is called or an error occurs.
// Returns an error if the provider doesn't support the requested media type.
CreateStreamSession(ctx context.Context, req *StreamInputRequest) (StreamInputSession, error)
// SupportsStreamInput returns the media types supported for streaming input.
// Common values: types.ContentTypeAudio, types.ContentTypeVideo
SupportsStreamInput() []string
// GetStreamingCapabilities returns detailed information about streaming support.
// This includes supported codecs, sample rates, and other constraints.
GetStreamingCapabilities() StreamingCapabilities
}
StreamInputSupport extends the Provider interface for bidirectional streaming. Providers that implement this interface can handle streaming media input (e.g., real-time audio) and provide streaming responses.
type StreamObserver ¶
type StreamObserver interface {
OnChunk(chunk StreamChunk)
OnComplete(totalTokens int, duration time.Duration)
OnError(err error)
}
StreamObserver receives stream events for monitoring
type StreamingCapabilities ¶ added in v1.1.0
type StreamingCapabilities struct {
// SupportedMediaTypes lists the media types that can be streamed
// Values: types.ContentTypeAudio, types.ContentTypeVideo
SupportedMediaTypes []string `json:"supported_media_types"`
// Audio capabilities
Audio *AudioStreamingCapabilities `json:"audio,omitempty"`
// Video capabilities
Video *VideoStreamingCapabilities `json:"video,omitempty"`
// BidirectionalSupport indicates if the provider supports full bidirectional streaming
BidirectionalSupport bool `json:"bidirectional_support"`
// MaxSessionDuration is the maximum duration for a streaming session (in seconds)
// Zero means no limit
MaxSessionDuration int `json:"max_session_duration,omitempty"`
// MinChunkSize is the minimum chunk size in bytes
MinChunkSize int `json:"min_chunk_size,omitempty"`
// MaxChunkSize is the maximum chunk size in bytes
MaxChunkSize int `json:"max_chunk_size,omitempty"`
}
StreamingCapabilities describes what streaming features a provider supports.
type ToolDescriptor ¶
type ToolDescriptor struct {
Name string `json:"name"`
Description string `json:"description"`
InputSchema json.RawMessage `json:"input_schema"`
OutputSchema json.RawMessage `json:"output_schema"`
}
ToolDescriptor represents a tool that can be used by providers
type ToolResult ¶
type ToolResult = types.MessageToolResult
ToolResult represents the result of a tool execution This is an alias to types.MessageToolResult for provider-specific context
type ToolSupport ¶
type ToolSupport interface {
Provider // Extends the base Provider interface
// BuildTooling converts tool descriptors to provider-native format
BuildTooling(descriptors []*ToolDescriptor) (interface{}, error)
// PredictWithTools performs a predict request with tool support
PredictWithTools(ctx context.Context, req PredictionRequest, tools interface{}, toolChoice string) (PredictionResponse, []types.MessageToolCall, error)
}
ToolSupport interface for providers that support tool/function calling
type UnsupportedContentError ¶ added in v1.1.0
type UnsupportedContentError struct {
Provider string // Provider ID
ContentType string // "image", "audio", "video", or "multimodal"
Message string // Human-readable error message
PartIndex int // Index of the unsupported content part (if applicable)
MIMEType string // Specific MIME type that's unsupported (if applicable)
}
UnsupportedContentError is returned when a provider doesn't support certain content types
func (*UnsupportedContentError) Error ¶ added in v1.1.0
func (e *UnsupportedContentError) Error() string
type UnsupportedProviderError ¶
type UnsupportedProviderError struct {
ProviderType string
}
UnsupportedProviderError is returned when a provider type is not recognized
func (*UnsupportedProviderError) Error ¶
func (e *UnsupportedProviderError) Error() string
Error returns the error message for this unsupported provider error.
type ValidationAbortError ¶
type ValidationAbortError struct {
Reason string
Chunk StreamChunk
}
ValidationAbortError is returned when a streaming validator aborts a stream
func (*ValidationAbortError) Error ¶
func (e *ValidationAbortError) Error() string
Error returns the error message for this validation abort error.
type VideoResolution ¶ added in v1.1.0
VideoResolution represents a video resolution.
func (VideoResolution) String ¶ added in v1.1.0
func (r VideoResolution) String() string
String returns a string representation of the resolution (e.g., "1920x1080")
type VideoStreamingCapabilities ¶ added in v1.1.0
type VideoStreamingCapabilities struct {
// SupportedEncodings lists supported video encodings
// Common values: "h264", "vp8", "vp9", "av1"
SupportedEncodings []string `json:"supported_encodings"`
// SupportedResolutions lists supported resolutions (width x height)
SupportedResolutions []VideoResolution `json:"supported_resolutions"`
// SupportedFrameRates lists supported frame rates
// Common values: 15, 24, 30, 60
SupportedFrameRates []int `json:"supported_frame_rates"`
// PreferredEncoding is the recommended encoding
PreferredEncoding string `json:"preferred_encoding"`
// PreferredResolution is the recommended resolution
PreferredResolution VideoResolution `json:"preferred_resolution"`
// PreferredFrameRate is the recommended frame rate
PreferredFrameRate int `json:"preferred_frame_rate"`
}
VideoStreamingCapabilities describes video streaming support.