Documentation
¶
Overview ¶
Package bifrost provides a unified LLM interface using the Bifrost gateway library. This package wraps Bifrost to implement the llm.LanguageModel interface, allowing the plugin to use multiple LLM providers through a single, consistent API.
Index ¶
- Constants
- func FetchModels(cfg FetchModelsConfig) ([]llm.ModelInfo, error)
- func FetchModelsForService(svc llm.ServiceConfig) ([]llm.ModelInfo, error)
- func FetchModelsForServiceType(serviceType, apiKey, apiURL, orgID string) ([]llm.ModelInfo, error)
- func IsSupported(serviceType string) bool
- func MapServiceTypeToProvider(serviceType string) (schemas.ModelProvider, error)
- func Ptr[T any](v T) *T
- func SupportsNativeTools(serviceType string) bool
- type Config
- type EmbeddingConfig
- type EmbeddingProvider
- type FallbackEntry
- type FetchModelsConfig
- type LLM
- func (b *LLM) ChatCompletion(ctx context.Context, request llm.CompletionRequest, ...) (*llm.TextStreamResult, error)
- func (b *LLM) ChatCompletionNoStream(ctx context.Context, request llm.CompletionRequest, ...) (string, error)
- func (b *LLM) CountTokens(ctx context.Context, request llm.CompletionRequest, ...) (int, error)
- func (b *LLM) GetDefaultConfig() llm.LanguageModelConfig
- func (b *LLM) InputTokenLimit() int
- func (b *LLM) OutputTokenLimit() int
- func (b *LLM) Shutdown()
- type ProviderSettings
- type Transcriber
- type TranscriptionConfig
Constants ¶
const ( DefaultMaxTokens = 8192 MaxToolResolutionDepth = 10 DefaultStreamingTimeout = 5 * time.Minute // CountTokensTimeout caps the count-tokens preflight so a wedged provider // can't block the request handler. CountTokensTimeout = 30 * time.Second )
Variables ¶
This section is empty.
Functions ¶
func FetchModels ¶
func FetchModels(cfg FetchModelsConfig) ([]llm.ModelInfo, error)
FetchModels retrieves the list of available models from a provider using Bifrost.
func FetchModelsForService ¶
func FetchModelsForService(svc llm.ServiceConfig) ([]llm.ModelInfo, error)
FetchModelsForService fetches models for a given service configuration. This handles provider-specific credentials (for example, Vertex AI's project ID, region, and service-account JSON) that cannot be expressed as a single API key.
func FetchModelsForServiceType ¶
FetchModelsForServiceType fetches models for a given service type string. This variant is kept for services that only require API-key style credentials (OpenAI, Anthropic, Azure, OpenAI-compatible, Gemini, Cohere, Mistral). Use FetchModelsForService for Vertex AI and other providers that need structured credentials beyond a single API key.
func IsSupported ¶
IsSupported returns true if the service type is supported by Bifrost.
func MapServiceTypeToProvider ¶
func MapServiceTypeToProvider(serviceType string) (schemas.ModelProvider, error)
MapServiceTypeToProvider maps our service type strings to Bifrost provider constants.
func SupportsNativeTools ¶
SupportsNativeTools reports whether the given service type can use provider native tools (currently, web search). This gates both request-time filtering and the effective-behavior checks used by built-in Mattermost tools so that built-in fallbacks do not get suppressed when native tools would be stripped.
Types ¶
type Config ¶
type Config struct {
ProviderSettings
InputTokenLimit int
OutputTokenLimit int
// Native tools and reasoning configuration
EnabledNativeTools []string
ReasoningEnabled bool
ReasoningEffort string
ThinkingBudget int
// UseResponsesAPI enables OpenAI Responses API for native tools support
UseResponsesAPI bool
// Fallbacks is the ordered list of providers Bifrost tries sequentially
// when the primary provider fails.
Fallbacks []FallbackEntry
}
Config holds the configuration for creating a LLM instance.
type EmbeddingConfig ¶
type EmbeddingConfig struct {
Provider schemas.ModelProvider
APIKey string
APIURL string
Model string
Dimensions int
}
EmbeddingConfig holds the configuration for creating a EmbeddingProvider.
type EmbeddingProvider ¶
type EmbeddingProvider struct {
// contains filtered or unexported fields
}
EmbeddingProvider implements the embeddings.EmbeddingProvider interface using Bifrost.
func NewEmbeddingProvider ¶
func NewEmbeddingProvider(cfg EmbeddingConfig) (*EmbeddingProvider, error)
NewEmbeddingProvider creates a new EmbeddingProvider.
func (*EmbeddingProvider) BatchCreateEmbeddings ¶
func (p *EmbeddingProvider) BatchCreateEmbeddings(ctx context.Context, texts []string) ([][]float32, error)
BatchCreateEmbeddings generates embeddings for any number of texts, transparently splitting into multiple requests to stay within the provider's per-request limits.
func (*EmbeddingProvider) CreateEmbedding ¶
CreateEmbedding generates an embedding for the given text.
func (*EmbeddingProvider) Dimensions ¶
func (p *EmbeddingProvider) Dimensions() int
Dimensions returns the dimensionality of the embeddings.
func (*EmbeddingProvider) Shutdown ¶
func (p *EmbeddingProvider) Shutdown()
Shutdown gracefully shuts down the Bifrost client.
type FallbackEntry ¶
type FallbackEntry struct {
ProviderSettings
// ID is the source service ID, used to mint a unique custom-provider name
// when this fallback shares a base provider type with another service.
ID string
// IsKeyLess marks a fallback that authenticates without an API key (e.g. a
// local Ollama server).
IsKeyLess bool
// ChatOnly marks an OpenAI-base fallback whose endpoint lacks the Responses
// API; it is registered chat-only so Bifrost downgrades Responses-API
// requests to chat completions for it.
ChatOnly bool
}
FallbackEntry holds the settings for a single fallback in the chain.
type FetchModelsConfig ¶
type FetchModelsConfig struct {
Provider schemas.ModelProvider
APIKey string
APIURL string
OrgID string
// Region applies to providers that require a region to list models
// (Vertex AI, Bedrock).
Region string
// Vertex AI credentials. Empty AuthCredentials signals ADC / attached IAM.
VertexProjectID string
VertexProjectNumber string
VertexAuthCredentials string
}
FetchModelsConfig holds configuration for fetching models.
type LLM ¶
type LLM struct {
// contains filtered or unexported fields
}
LLM implements the llm.LanguageModel interface using the Bifrost gateway.
func New ¶
New creates a new LLM instance with the given configuration. It errors when a fallback cannot be registered, so a misconfigured fallback chain fails at setup instead of silently shrinking.
func NewFromServiceConfig ¶
func NewFromServiceConfig(serviceConfig llm.ServiceConfig, botConfig llm.BotConfig, fallbackServices []llm.ServiceConfig) (*LLM, error)
NewFromServiceConfig creates a LLM instance from ServiceConfig and BotConfig. fallbackServices is an ordered slice of fallback services resolved from the primary service's fallback chain (see llm.ResolveFallbackChain). Each fallback service's DefaultModel is used as the fallback model.
func (*LLM) ChatCompletion ¶
func (b *LLM) ChatCompletion(ctx context.Context, request llm.CompletionRequest, opts ...llm.LanguageModelOption) (*llm.TextStreamResult, error)
ChatCompletion performs a streaming chat completion request.
func (*LLM) ChatCompletionNoStream ¶
func (b *LLM) ChatCompletionNoStream(ctx context.Context, request llm.CompletionRequest, opts ...llm.LanguageModelOption) (string, error)
ChatCompletionNoStream performs a non-streaming chat completion request.
func (*LLM) CountTokens ¶
func (b *LLM) CountTokens(ctx context.Context, request llm.CompletionRequest, opts ...llm.LanguageModelOption) (int, error)
CountTokens returns llm.ErrUnsupportedTokenCount when the provider lacks a count-tokens endpoint, signaling callers to fall back to llm.EstimateTokens.
func (*LLM) GetDefaultConfig ¶
func (b *LLM) GetDefaultConfig() llm.LanguageModelConfig
GetDefaultConfig returns the default language model configuration. MaxGeneratedTokens substitutes DefaultMaxTokens when unset because some providers (Anthropic) require it.
func (*LLM) InputTokenLimit ¶
InputTokenLimit returns the configured maximum number of input tokens. Zero means "no client-side truncation" — the provider's own limit applies.
func (*LLM) OutputTokenLimit ¶
OutputTokenLimit returns the configured maximum number of output tokens. Zero means the request-building layer falls back to DefaultMaxTokens.
type ProviderSettings ¶
type ProviderSettings struct {
Provider schemas.ModelProvider
APIKey string
APIURL string // Custom base URL (for Azure, OpenAI Compatible, etc.)
OrgID string
Region string // For AWS Bedrock and Vertex
AWSAccessKeyID string
AWSSecretAccessKey string
// Vertex AI (GCP). VertexAuthCredentials holds the service-account JSON;
// empty falls back to ADC/IAM.
VertexProjectID string
VertexProjectNumber string
VertexAuthCredentials string
DefaultModel string
StreamingTimeout time.Duration
}
ProviderSettings holds the connection and credential fields needed to reach one provider. It is shared by the primary Config and every FallbackEntry in the chain.
type Transcriber ¶
type Transcriber struct {
// contains filtered or unexported fields
}
Transcriber implements transcription using the Bifrost gateway.
func NewTranscriber ¶
func NewTranscriber(cfg TranscriptionConfig) (*Transcriber, error)
NewTranscriber creates a new Transcriber.
func (*Transcriber) Shutdown ¶
func (t *Transcriber) Shutdown()
Shutdown gracefully shuts down the Bifrost client.
func (*Transcriber) Transcribe ¶
Transcribe converts audio to text using Bifrost.
type TranscriptionConfig ¶
type TranscriptionConfig struct {
Provider schemas.ModelProvider
APIKey string
APIURL string
Model string // e.g., "whisper-1"
}
TranscriptionConfig holds configuration for creating a Transcriber.
Source Files