providers

package
v1.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 17, 2025 License: Apache-2.0 Imports: 13 Imported by: 10

Documentation

Overview

Package providers implements multi-LLM provider support with unified interfaces.

This package provides a common abstraction for predict-based LLM providers including OpenAI, Anthropic Claude, and Google Gemini. It handles:

  • Predict completion requests with streaming support
  • Tool/function calling with provider-specific formats
  • Cost tracking and token usage calculation
  • Rate limiting and error handling

All providers implement the Provider interface for basic predict, and ToolSupport interface for function calling capabilities.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CheckHTTPError added in v1.1.0

func CheckHTTPError(resp *http.Response) error

CheckHTTPError checks if HTTP response is an error and returns formatted error with body

func HasAudioSupport added in v1.1.0

func HasAudioSupport(p Provider) bool

HasAudioSupport checks if a provider supports audio inputs

func HasImageSupport added in v1.1.0

func HasImageSupport(p Provider) bool

HasImageSupport checks if a provider supports image inputs

func HasVideoSupport added in v1.1.0

func HasVideoSupport(p Provider) bool

HasVideoSupport checks if a provider supports video inputs

func IsFormatSupported added in v1.1.0

func IsFormatSupported(p Provider, contentType string, mimeType string) bool

IsFormatSupported checks if a provider supports a specific media format (MIME type)

func IsValidationAbort

func IsValidationAbort(err error) bool

IsValidationAbort checks if an error is a validation abort

func LoadFileAsBase64 added in v1.1.0

func LoadFileAsBase64(filePath string) (string, error)

LoadFileAsBase64 reads a file and returns its content as a base64-encoded string. It supports home directory expansion (~/path) and is used by multimodal providers to load image files.

func RegisterProviderFactory added in v1.1.0

func RegisterProviderFactory(providerType string, factory ProviderFactory)

RegisterProviderFactory registers a factory function for a provider type

func SetErrorResponse added in v1.1.0

func SetErrorResponse(predictResp *PredictionResponse, respBody []byte, start time.Time)

SetErrorResponse sets latency and raw body on error responses

func StringPtr added in v1.1.0

func StringPtr(s string) *string

StringPtr is a helper function that returns a pointer to a string. This is commonly used across provider implementations for optional fields.

func SupportsMultimodal added in v1.1.0

func SupportsMultimodal(p Provider) bool

SupportsMultimodal checks if a provider implements multimodal support

func UnmarshalJSON added in v1.1.0

func UnmarshalJSON(respBody []byte, v interface{}, predictResp *PredictionResponse, start time.Time) error

UnmarshalJSON unmarshals JSON with error recovery that sets latency and raw response

func ValidateMultimodalMessage added in v1.1.0

func ValidateMultimodalMessage(p Provider, msg types.Message) error

ValidateMultimodalMessage checks if a message's multimodal content is supported by the provider

func ValidateMultimodalRequest added in v1.1.0

func ValidateMultimodalRequest(p MultimodalSupport, req PredictionRequest) error

ValidateMultimodalRequest validates all messages in a predict request for multimodal compatibility This is a helper function to reduce duplication across provider implementations

Types

type AudioStreamingCapabilities added in v1.1.0

type AudioStreamingCapabilities struct {
	// SupportedEncodings lists supported audio encodings
	// Common values: "pcm", "opus", "mp3", "aac"
	SupportedEncodings []string `json:"supported_encodings"`

	// SupportedSampleRates lists supported sample rates in Hz
	// Common values: 8000, 16000, 24000, 44100, 48000
	SupportedSampleRates []int `json:"supported_sample_rates"`

	// SupportedChannels lists supported channel counts
	// Common values: 1 (mono), 2 (stereo)
	SupportedChannels []int `json:"supported_channels"`

	// SupportedBitDepths lists supported bit depths
	// Common values: 16, 24, 32
	SupportedBitDepths []int `json:"supported_bit_depths,omitempty"`

	// PreferredEncoding is the recommended encoding for best quality/latency
	PreferredEncoding string `json:"preferred_encoding"`

	// PreferredSampleRate is the recommended sample rate
	PreferredSampleRate int `json:"preferred_sample_rate"`
}

AudioStreamingCapabilities describes audio streaming support.

type BaseProvider added in v1.1.0

type BaseProvider struct {
	// contains filtered or unexported fields
}

BaseProvider provides common functionality shared across all provider implementations. It should be embedded in concrete provider structs to avoid code duplication.

func NewBaseProvider added in v1.1.0

func NewBaseProvider(id string, includeRawOutput bool, client *http.Client) BaseProvider

NewBaseProvider creates a new BaseProvider with common fields

func NewBaseProviderWithAPIKey added in v1.1.0

func NewBaseProviderWithAPIKey(id string, includeRawOutput bool, primaryKey, fallbackKey string) (BaseProvider, string)

NewBaseProviderWithAPIKey creates a BaseProvider and retrieves API key from environment It tries the primary key first, then falls back to the secondary key if primary is empty.

func (*BaseProvider) Close added in v1.1.0

func (b *BaseProvider) Close() error

Close closes the HTTP client's idle connections

func (*BaseProvider) GetHTTPClient added in v1.1.0

func (b *BaseProvider) GetHTTPClient() *http.Client

GetHTTPClient returns the underlying HTTP client for provider-specific use

func (*BaseProvider) ID added in v1.1.0

func (b *BaseProvider) ID() string

ID returns the provider ID

func (*BaseProvider) ShouldIncludeRawOutput added in v1.1.0

func (b *BaseProvider) ShouldIncludeRawOutput() bool

ShouldIncludeRawOutput returns whether to include raw API responses in output

func (*BaseProvider) SupportsStreaming added in v1.1.0

func (b *BaseProvider) SupportsStreaming() bool

SupportsStreaming returns true by default (can be overridden by providers that don't support streaming)

type ExecutionResult

type ExecutionResult interface{}

Forward declare ExecutionResult to avoid circular import

type ImageDetail added in v1.1.0

type ImageDetail string

ImageDetail specifies the level of detail for image processing

const (
	ImageDetailLow  ImageDetail = "low"  // Faster, less detailed analysis
	ImageDetailHigh ImageDetail = "high" // Slower, more detailed analysis
	ImageDetailAuto ImageDetail = "auto" // Provider chooses automatically
)

type MultimodalCapabilities added in v1.1.0

type MultimodalCapabilities struct {
	SupportsImages bool     // Provider can process image inputs
	SupportsAudio  bool     // Provider can process audio inputs
	SupportsVideo  bool     // Provider can process video inputs
	ImageFormats   []string // Supported image MIME types (e.g., "image/jpeg", "image/png")
	AudioFormats   []string // Supported audio MIME types (e.g., "audio/mpeg", "audio/wav")
	VideoFormats   []string // Supported video MIME types (e.g., "video/mp4")
	MaxImageSizeMB int      // Maximum image size in megabytes (0 = unlimited/unknown)
	MaxAudioSizeMB int      // Maximum audio size in megabytes (0 = unlimited/unknown)
	MaxVideoSizeMB int      // Maximum video size in megabytes (0 = unlimited/unknown)
}

MultimodalCapabilities describes what types of multimodal content a provider supports

type MultimodalSupport added in v1.1.0

type MultimodalSupport interface {
	Provider // Extends the base Provider interface

	// GetMultimodalCapabilities returns what types of multimodal content this provider supports
	GetMultimodalCapabilities() MultimodalCapabilities

	// PredictMultimodal performs a predict request with multimodal message content
	// Messages in the request can contain Parts with images, audio, or video
	PredictMultimodal(ctx context.Context, req PredictionRequest) (PredictionResponse, error)

	// PredictMultimodalStream performs a streaming predict request with multimodal content
	PredictMultimodalStream(ctx context.Context, req PredictionRequest) (<-chan StreamChunk, error)
}

MultimodalSupport interface for providers that support multimodal inputs

func GetMultimodalProvider added in v1.1.0

func GetMultimodalProvider(p Provider) MultimodalSupport

GetMultimodalProvider safely casts a provider to MultimodalSupport Returns nil if the provider doesn't support multimodal

type MultimodalToolSupport added in v1.1.0

type MultimodalToolSupport interface {
	MultimodalSupport // Extends multimodal support
	ToolSupport       // Extends tool support

	// PredictMultimodalWithTools performs a predict request with both multimodal content and tools
	PredictMultimodalWithTools(ctx context.Context, req PredictionRequest, tools interface{}, toolChoice string) (PredictionResponse, []types.MessageToolCall, error)
}

MultimodalToolSupport interface for providers that support both multimodal and tools

type PredictionRequest added in v1.1.0

type PredictionRequest struct {
	System      string                 `json:"system"`
	Messages    []types.Message        `json:"messages"`
	Temperature float32                `json:"temperature"`
	TopP        float32                `json:"top_p"`
	MaxTokens   int                    `json:"max_tokens"`
	Seed        *int                   `json:"seed,omitempty"`
	Metadata    map[string]interface{} `json:"metadata,omitempty"` // Optional metadata for provider-specific context
}

PredictionRequest represents a request to a predict provider

type PredictionResponse added in v1.1.0

type PredictionResponse struct {
	Content    string                  `json:"content"`
	Parts      []types.ContentPart     `json:"parts,omitempty"`     // Multimodal content parts (text, image, audio, video)
	CostInfo   *types.CostInfo         `json:"cost_info,omitempty"` // Cost breakdown for this response (includes token counts)
	Latency    time.Duration           `json:"latency"`
	Raw        []byte                  `json:"raw,omitempty"`
	RawRequest interface{}             `json:"raw_request,omitempty"` // Raw API request (for debugging)
	ToolCalls  []types.MessageToolCall `json:"tool_calls,omitempty"`  // Tools called in this response
}

PredictionResponse represents a response from a predict provider

type Pricing

type Pricing struct {
	InputCostPer1K  float64
	OutputCostPer1K float64
}

Pricing defines cost per 1K tokens for input and output

type Provider

type Provider interface {
	ID() string
	Predict(ctx context.Context, req PredictionRequest) (PredictionResponse, error)

	// Streaming support
	PredictStream(ctx context.Context, req PredictionRequest) (<-chan StreamChunk, error)
	SupportsStreaming() bool

	ShouldIncludeRawOutput() bool
	Close() error // Close cleans up provider resources (e.g., HTTP connections)

	// CalculateCost calculates cost breakdown for given token counts
	CalculateCost(inputTokens, outputTokens, cachedTokens int) types.CostInfo
}

Provider interface defines the contract for predict providers

func CreateProviderFromSpec

func CreateProviderFromSpec(spec ProviderSpec) (Provider, error)

CreateProviderFromSpec creates a provider implementation from a spec. Returns an error if the provider type is unsupported.

type ProviderDefaults

type ProviderDefaults struct {
	Temperature float32
	TopP        float32
	MaxTokens   int
	Pricing     Pricing
}

ProviderDefaults holds default parameters for providers

type ProviderFactory added in v1.1.0

type ProviderFactory func(spec ProviderSpec) (Provider, error)

ProviderFactory is a function that creates a provider from a spec

type ProviderSpec

type ProviderSpec struct {
	ID               string
	Type             string
	Model            string
	BaseURL          string
	Defaults         ProviderDefaults
	IncludeRawOutput bool
	AdditionalConfig map[string]interface{} // Flexible key-value pairs for provider-specific configuration
}

ProviderSpec holds the configuration needed to create a provider instance

type Registry

type Registry struct {
	// contains filtered or unexported fields
}

Registry manages available providers

func NewRegistry

func NewRegistry() *Registry

NewRegistry creates a new provider registry

func (*Registry) Close

func (r *Registry) Close() error

Close closes all registered providers and cleans up their resources. Returns the first error encountered, if any.

func (*Registry) Get

func (r *Registry) Get(id string) (Provider, bool)

Get retrieves a provider by ID, returning the provider and a boolean indicating if it was found.

func (*Registry) List

func (r *Registry) List() []string

List returns all registered provider IDs

func (*Registry) Register

func (r *Registry) Register(provider Provider)

Register adds a provider to the registry using its ID as the key.

type SSEScanner

type SSEScanner struct {
	// contains filtered or unexported fields
}

SSEScanner scans Server-Sent Events (SSE) streams

func NewSSEScanner

func NewSSEScanner(r io.Reader) *SSEScanner

NewSSEScanner creates a new SSE scanner

func (*SSEScanner) Data

func (s *SSEScanner) Data() string

Data returns the current event data

func (*SSEScanner) Err

func (s *SSEScanner) Err() error

Err returns any scanning error

func (*SSEScanner) Scan

func (s *SSEScanner) Scan() bool

Scan advances to the next SSE event

type StreamChunk

type StreamChunk struct {
	// Content is the accumulated content so far
	Content string `json:"content"`

	// Delta is the new content in this chunk
	Delta string `json:"delta"`

	// MediaDelta contains new media content in this chunk (audio, video, images)
	// Uses the same MediaContent type as non-streaming messages for API consistency.
	MediaDelta *types.MediaContent `json:"media_delta,omitempty"`

	// TokenCount is the total number of tokens so far
	TokenCount int `json:"token_count"`

	// DeltaTokens is the number of tokens in this delta
	DeltaTokens int `json:"delta_tokens"`

	// ToolCalls contains accumulated tool calls (for assistant messages that invoke tools)
	ToolCalls []types.MessageToolCall `json:"tool_calls,omitempty"`

	// FinishReason is nil until stream is complete
	// Values: "stop", "length", "content_filter", "tool_calls", "error", "validation_failed", "cancelled"
	FinishReason *string `json:"finish_reason,omitempty"`

	// Error is set if an error occurred during streaming
	Error error `json:"error,omitempty"`

	// Metadata contains provider-specific metadata
	Metadata map[string]interface{} `json:"metadata,omitempty"`

	// FinalResult contains the complete execution result (only set in the final chunk)
	FinalResult ExecutionResult `json:"final_result,omitempty"`

	// CostInfo contains cost breakdown (only present in final chunk when FinishReason != nil)
	CostInfo *types.CostInfo `json:"cost_info,omitempty"`
}

StreamChunk represents a batch of tokens with metadata

type StreamEvent

type StreamEvent struct {
	// Type is the event type: "chunk", "complete", "error"
	Type string `json:"type"`

	// Chunk contains the stream chunk data
	Chunk *StreamChunk `json:"chunk,omitempty"`

	// Error is set for error events
	Error error `json:"error,omitempty"`

	// Timestamp is when the event occurred
	Timestamp time.Time `json:"timestamp"`
}

StreamEvent is sent to observers for monitoring

type StreamInputRequest added in v1.1.0

type StreamInputRequest struct {
	// Config specifies the media streaming configuration
	Config types.StreamingMediaConfig `json:"config"`

	// SystemMsg is the system message for the conversation
	SystemMsg string `json:"system_msg,omitempty"`

	// InitialText is optional text to send before streaming starts
	// This can be used to provide initial context or instructions
	InitialText string `json:"initial_text,omitempty"`

	// Temperature controls randomness in responses (0.0 to 2.0)
	Temperature float32 `json:"temperature,omitempty"`

	// MaxTokens limits the response length
	MaxTokens int `json:"max_tokens,omitempty"`

	// Metadata contains provider-specific configuration
	Metadata map[string]interface{} `json:"metadata,omitempty"`
}

StreamInputRequest configures a new streaming input session.

func (*StreamInputRequest) Validate added in v1.1.0

func (r *StreamInputRequest) Validate() error

Validate checks if the StreamInputRequest is valid

type StreamInputSession added in v1.1.0

type StreamInputSession interface {
	// SendChunk sends a media chunk to the provider.
	// Returns an error if the chunk cannot be sent or the session is closed.
	// This method is safe to call from multiple goroutines.
	SendChunk(ctx context.Context, chunk *types.MediaChunk) error

	// SendText sends a text message to the provider during the streaming session.
	// This is useful for sending text prompts or instructions during audio streaming.
	SendText(ctx context.Context, text string) error

	// Response returns a receive-only channel for streaming responses.
	// The channel is closed when the session ends or encounters an error.
	// Consumers should read from this channel in a separate goroutine.
	Response() <-chan StreamChunk

	// Close ends the streaming session and releases resources.
	// After calling Close, SendChunk and SendText will return errors.
	// The Response channel will be closed.
	// Close is safe to call multiple times.
	Close() error

	// Error returns any error that occurred during the session.
	// Returns nil if no error has occurred.
	Error() error

	// Done returns a channel that's closed when the session ends.
	// This is useful for select statements to detect session completion.
	Done() <-chan struct{}
}

StreamInputSession manages a bidirectional streaming session with a provider. The session allows sending media chunks (e.g., audio from a microphone) and receiving streaming responses from the LLM.

Example usage:

session, err := provider.CreateStreamSession(ctx, StreamInputRequest{
    Config: types.StreamingMediaConfig{
        Type:       types.ContentTypeAudio,
        ChunkSize:  8192,
        SampleRate: 16000,
        Encoding:   "pcm",
        Channels:   1,
    },
    SystemMsg: "You are a helpful voice assistant",
})
if err != nil {
    return err
}
defer session.Close()

// Send audio chunks in a goroutine
go func() {
    for chunk := range micInput {
        if err := session.SendChunk(ctx, chunk); err != nil {
            log.Printf("send error: %v", err)
            break
        }
    }
}()

// Receive responses
for chunk := range session.Response() {
    if chunk.Error != nil {
        log.Printf("response error: %v", chunk.Error)
        break
    }
    fmt.Print(chunk.Delta)
}

type StreamInputSupport added in v1.1.0

type StreamInputSupport interface {
	Provider // Extends the base Provider interface

	// CreateStreamSession creates a new bidirectional streaming session.
	// The session remains active until Close() is called or an error occurs.
	// Returns an error if the provider doesn't support the requested media type.
	CreateStreamSession(ctx context.Context, req *StreamInputRequest) (StreamInputSession, error)

	// SupportsStreamInput returns the media types supported for streaming input.
	// Common values: types.ContentTypeAudio, types.ContentTypeVideo
	SupportsStreamInput() []string

	// GetStreamingCapabilities returns detailed information about streaming support.
	// This includes supported codecs, sample rates, and other constraints.
	GetStreamingCapabilities() StreamingCapabilities
}

StreamInputSupport extends the Provider interface for bidirectional streaming. Providers that implement this interface can handle streaming media input (e.g., real-time audio) and provide streaming responses.

type StreamObserver

type StreamObserver interface {
	OnChunk(chunk StreamChunk)
	OnComplete(totalTokens int, duration time.Duration)
	OnError(err error)
}

StreamObserver receives stream events for monitoring

type StreamingCapabilities added in v1.1.0

type StreamingCapabilities struct {
	// SupportedMediaTypes lists the media types that can be streamed
	// Values: types.ContentTypeAudio, types.ContentTypeVideo
	SupportedMediaTypes []string `json:"supported_media_types"`

	// Audio capabilities
	Audio *AudioStreamingCapabilities `json:"audio,omitempty"`

	// Video capabilities
	Video *VideoStreamingCapabilities `json:"video,omitempty"`

	// BidirectionalSupport indicates if the provider supports full bidirectional streaming
	BidirectionalSupport bool `json:"bidirectional_support"`

	// MaxSessionDuration is the maximum duration for a streaming session (in seconds)
	// Zero means no limit
	MaxSessionDuration int `json:"max_session_duration,omitempty"`

	// MinChunkSize is the minimum chunk size in bytes
	MinChunkSize int `json:"min_chunk_size,omitempty"`

	// MaxChunkSize is the maximum chunk size in bytes
	MaxChunkSize int `json:"max_chunk_size,omitempty"`
}

StreamingCapabilities describes what streaming features a provider supports.

type ToolDescriptor

type ToolDescriptor struct {
	Name         string          `json:"name"`
	Description  string          `json:"description"`
	InputSchema  json.RawMessage `json:"input_schema"`
	OutputSchema json.RawMessage `json:"output_schema"`
}

ToolDescriptor represents a tool that can be used by providers

type ToolResult

type ToolResult = types.MessageToolResult

ToolResult represents the result of a tool execution This is an alias to types.MessageToolResult for provider-specific context

type ToolSupport

type ToolSupport interface {
	Provider // Extends the base Provider interface

	// BuildTooling converts tool descriptors to provider-native format
	BuildTooling(descriptors []*ToolDescriptor) (interface{}, error)

	// PredictWithTools performs a predict request with tool support
	PredictWithTools(ctx context.Context, req PredictionRequest, tools interface{}, toolChoice string) (PredictionResponse, []types.MessageToolCall, error)
}

ToolSupport interface for providers that support tool/function calling

type UnsupportedContentError added in v1.1.0

type UnsupportedContentError struct {
	Provider    string // Provider ID
	ContentType string // "image", "audio", "video", or "multimodal"
	Message     string // Human-readable error message
	PartIndex   int    // Index of the unsupported content part (if applicable)
	MIMEType    string // Specific MIME type that's unsupported (if applicable)
}

UnsupportedContentError is returned when a provider doesn't support certain content types

func (*UnsupportedContentError) Error added in v1.1.0

func (e *UnsupportedContentError) Error() string

type UnsupportedProviderError

type UnsupportedProviderError struct {
	ProviderType string
}

UnsupportedProviderError is returned when a provider type is not recognized

func (*UnsupportedProviderError) Error

func (e *UnsupportedProviderError) Error() string

Error returns the error message for this unsupported provider error.

type ValidationAbortError

type ValidationAbortError struct {
	Reason string
	Chunk  StreamChunk
}

ValidationAbortError is returned when a streaming validator aborts a stream

func (*ValidationAbortError) Error

func (e *ValidationAbortError) Error() string

Error returns the error message for this validation abort error.

type VideoResolution added in v1.1.0

type VideoResolution struct {
	Width  int `json:"width"`
	Height int `json:"height"`
}

VideoResolution represents a video resolution.

func (VideoResolution) String added in v1.1.0

func (r VideoResolution) String() string

String returns a string representation of the resolution (e.g., "1920x1080")

type VideoStreamingCapabilities added in v1.1.0

type VideoStreamingCapabilities struct {
	// SupportedEncodings lists supported video encodings
	// Common values: "h264", "vp8", "vp9", "av1"
	SupportedEncodings []string `json:"supported_encodings"`

	// SupportedResolutions lists supported resolutions (width x height)
	SupportedResolutions []VideoResolution `json:"supported_resolutions"`

	// SupportedFrameRates lists supported frame rates
	// Common values: 15, 24, 30, 60
	SupportedFrameRates []int `json:"supported_frame_rates"`

	// PreferredEncoding is the recommended encoding
	PreferredEncoding string `json:"preferred_encoding"`

	// PreferredResolution is the recommended resolution
	PreferredResolution VideoResolution `json:"preferred_resolution"`

	// PreferredFrameRate is the recommended frame rate
	PreferredFrameRate int `json:"preferred_frame_rate"`
}

VideoStreamingCapabilities describes video streaming support.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL