bifrost

package module
v1.5.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 28, 2026 License: Apache-2.0 Imports: 56 Imported by: 0

Documentation

Overview

Package bifrost provides the core implementation of the Bifrost system. Bifrost is a unified interface for interacting with various AI model providers, managing concurrent requests, and handling provider-specific configurations.

Package bifrost provides the core implementation of the Bifrost system.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CanProviderKeyValueBeEmpty 

func CanProviderKeyValueBeEmpty(providerKey schemas.ModelProvider) bool

CanProviderKeyValueBeEmpty returns true if the given provider allows the API key to be empty. Some providers like Vertex and Bedrock have their credentials in additional key configs. Ollama and SGL are keyless (API Key is optional) but use per-key server URLs.

func GetBoolFromContext 

func GetBoolFromContext(ctx context.Context, key any) bool

GetBoolFromContext safely extracts a bool value from context

func GetErrorMessage 

func GetErrorMessage(err *schemas.BifrostError) string

func GetIntFromContext 

func GetIntFromContext(ctx context.Context, key any) int

GetIntFromContext safely extracts an int value from context

func GetResponseFields 

func GetResponseFields(result *schemas.BifrostResponse, err *schemas.BifrostError) (requestType schemas.RequestType, provider schemas.ModelProvider, originalModel string, resolvedModel string)

GetResponseFields extracts the request type, provider, original model, and resolved model from the result or error.

func GetStringFromContext 

func GetStringFromContext(ctx context.Context, key any) string

GetStringFromContext safely extracts a string value from context

func GetTracerFromContext 

func GetTracerFromContext(ctx *schemas.BifrostContext) (schemas.Tracer, string, error)

func IsCodemodeTool 

func IsCodemodeTool(toolName string) bool

IsCodemodeTool returns true if the given tool name is a codemode tool.

func IsFinalChunk 

func IsFinalChunk(ctx *schemas.BifrostContext) bool

IsFinalChunk returns true if the given context is a final chunk.

func IsRateLimitErrorMessage 

func IsRateLimitErrorMessage(errorMessage string) bool

IsRateLimitErrorMessage checks if an error message indicates a rate limit issue

func IsStandardProvider 

func IsStandardProvider(providerKey schemas.ModelProvider) bool

IsStandardProvider reports whether providerKey is a built-in (non-custom) provider.

func IsStreamRequestType 

func IsStreamRequestType(reqType schemas.RequestType) bool

IsStreamRequestType returns true if the given request type is a stream request.

func IsSupportedBaseProvider 

func IsSupportedBaseProvider(providerKey schemas.ModelProvider) bool

IsSupportedBaseProvider reports whether providerKey is allowed as a base provider for custom providers.

func MarshalUnsafe 

func MarshalUnsafe(v any) string

MarshalUnsafe marshals the given value to a JSON string without escaping HTML characters. Returns empty string if marshaling fails.

func NewNoOpLogger 

func NewNoOpLogger() schemas.Logger

NewNoOpLogger creates a new NoOpLogger instance.

func Ptr 

func Ptr[T any](v T) *T

Ptr returns a pointer to the given value.

func RedactSensitiveString 

func RedactSensitiveString(s string) string

RedactSensitiveString redacts sensitive information in a string

func ValidateExternalURL 

func ValidateExternalURL(urlStr string) error

ValidateExternalURL validates a URL for security concerns (SSRF protection)

Types

type Bifrost 

type Bifrost struct {
	MCPManager mcp.MCPManagerInterface // MCP integration manager (nil if MCP not configured)
	// contains filtered or unexported fields
}

Bifrost manages providers and maintains specified open channels for concurrent processing. It handles request routing, provider management, and response processing.

func Init 

func Init(ctx context.Context, config schemas.BifrostConfig) (*Bifrost, error)

Init initializes a new Bifrost instance with the given configuration. It sets up the account, plugins, object pools, and initializes providers. Returns an error if initialization fails. Initial Memory Allocations happens here as per the initial pool size.

func (*Bifrost) AddMCPClient 

func (bifrost *Bifrost) AddMCPClient(config *schemas.MCPClientConfig) error

AddMCPClient adds a new MCP client to the Bifrost instance. This allows for dynamic MCP client management at runtime.

Parameters:

  • config: MCP client configuration

Returns:

  • error: Any registration error

Example: 

err := bifrost.AddMCPClient(schemas.MCPClientConfig{
    Name: "my-mcp-client",
    ConnectionType: schemas.MCPConnectionTypeHTTP,
    ConnectionString: &url,
})

func (*Bifrost) BatchCancelRequest 

func (bifrost *Bifrost) BatchCancelRequest(ctx *schemas.BifrostContext, req *schemas.BifrostBatchCancelRequest) (*schemas.BifrostBatchCancelResponse, *schemas.BifrostError)

BatchCancelRequest cancels a batch job.

func (*Bifrost) BatchCreateRequest 

func (bifrost *Bifrost) BatchCreateRequest(ctx *schemas.BifrostContext, req *schemas.BifrostBatchCreateRequest) (*schemas.BifrostBatchCreateResponse, *schemas.BifrostError)

BatchCreateRequest creates a new batch job for asynchronous processing.

func (*Bifrost) BatchDeleteRequest 

func (bifrost *Bifrost) BatchDeleteRequest(ctx *schemas.BifrostContext, req *schemas.BifrostBatchDeleteRequest) (*schemas.BifrostBatchDeleteResponse, *schemas.BifrostError)

BatchDeleteRequest deletes a batch job.

func (*Bifrost) BatchListRequest 

func (bifrost *Bifrost) BatchListRequest(ctx *schemas.BifrostContext, req *schemas.BifrostBatchListRequest) (*schemas.BifrostBatchListResponse, *schemas.BifrostError)

BatchListRequest lists batch jobs for the specified provider.

func (*Bifrost) BatchResultsRequest 

func (bifrost *Bifrost) BatchResultsRequest(ctx *schemas.BifrostContext, req *schemas.BifrostBatchResultsRequest) (*schemas.BifrostBatchResultsResponse, *schemas.BifrostError)

BatchResultsRequest retrieves results from a completed batch job.

func (*Bifrost) BatchRetrieveRequest 

func (bifrost *Bifrost) BatchRetrieveRequest(ctx *schemas.BifrostContext, req *schemas.BifrostBatchRetrieveRequest) (*schemas.BifrostBatchRetrieveResponse, *schemas.BifrostError)

BatchRetrieveRequest retrieves a specific batch job.

func (*Bifrost) ChatCompletionRequest 

func (bifrost *Bifrost) ChatCompletionRequest(ctx *schemas.BifrostContext, req *schemas.BifrostChatRequest) (*schemas.BifrostChatResponse, *schemas.BifrostError)

ChatCompletionRequest sends a chat completion request to the specified provider.

func (*Bifrost) ChatCompletionStreamRequest 

func (bifrost *Bifrost) ChatCompletionStreamRequest(ctx *schemas.BifrostContext, req *schemas.BifrostChatRequest) (chan *schemas.BifrostStreamChunk, *schemas.BifrostError)

ChatCompletionStreamRequest sends a chat completion stream request to the specified provider.

func (*Bifrost) ContainerCreateRequest 

func (bifrost *Bifrost) ContainerCreateRequest(ctx *schemas.BifrostContext, req *schemas.BifrostContainerCreateRequest) (*schemas.BifrostContainerCreateResponse, *schemas.BifrostError)

ContainerCreateRequest creates a new container.

func (*Bifrost) ContainerDeleteRequest 

func (bifrost *Bifrost) ContainerDeleteRequest(ctx *schemas.BifrostContext, req *schemas.BifrostContainerDeleteRequest) (*schemas.BifrostContainerDeleteResponse, *schemas.BifrostError)

ContainerDeleteRequest deletes a container.

func (*Bifrost) ContainerFileContentRequest 

func (bifrost *Bifrost) ContainerFileContentRequest(ctx *schemas.BifrostContext, req *schemas.BifrostContainerFileContentRequest) (*schemas.BifrostContainerFileContentResponse, *schemas.BifrostError)

ContainerFileContentRequest retrieves the content of a file from a container.

func (*Bifrost) ContainerFileCreateRequest 

func (bifrost *Bifrost) ContainerFileCreateRequest(ctx *schemas.BifrostContext, req *schemas.BifrostContainerFileCreateRequest) (*schemas.BifrostContainerFileCreateResponse, *schemas.BifrostError)

ContainerFileCreateRequest creates a file in a container.

func (*Bifrost) ContainerFileDeleteRequest 

func (bifrost *Bifrost) ContainerFileDeleteRequest(ctx *schemas.BifrostContext, req *schemas.BifrostContainerFileDeleteRequest) (*schemas.BifrostContainerFileDeleteResponse, *schemas.BifrostError)

ContainerFileDeleteRequest deletes a file from a container.

func (*Bifrost) ContainerFileListRequest 

func (bifrost *Bifrost) ContainerFileListRequest(ctx *schemas.BifrostContext, req *schemas.BifrostContainerFileListRequest) (*schemas.BifrostContainerFileListResponse, *schemas.BifrostError)

ContainerFileListRequest lists files in a container.

func (*Bifrost) ContainerFileRetrieveRequest 

func (bifrost *Bifrost) ContainerFileRetrieveRequest(ctx *schemas.BifrostContext, req *schemas.BifrostContainerFileRetrieveRequest) (*schemas.BifrostContainerFileRetrieveResponse, *schemas.BifrostError)

ContainerFileRetrieveRequest retrieves a file from a container.

func (*Bifrost) ContainerListRequest 

func (bifrost *Bifrost) ContainerListRequest(ctx *schemas.BifrostContext, req *schemas.BifrostContainerListRequest) (*schemas.BifrostContainerListResponse, *schemas.BifrostError)

ContainerListRequest lists containers.

func (*Bifrost) ContainerRetrieveRequest 

func (bifrost *Bifrost) ContainerRetrieveRequest(ctx *schemas.BifrostContext, req *schemas.BifrostContainerRetrieveRequest) (*schemas.BifrostContainerRetrieveResponse, *schemas.BifrostError)

ContainerRetrieveRequest retrieves a specific container.

func (*Bifrost) CountTokensRequest 

func (bifrost *Bifrost) CountTokensRequest(ctx *schemas.BifrostContext, req *schemas.BifrostResponsesRequest) (*schemas.BifrostCountTokensResponse, *schemas.BifrostError)

CountTokensRequest sends a count tokens request to the specified provider.

func (*Bifrost) EmbeddingRequest 

func (bifrost *Bifrost) EmbeddingRequest(ctx *schemas.BifrostContext, req *schemas.BifrostEmbeddingRequest) (*schemas.BifrostEmbeddingResponse, *schemas.BifrostError)

EmbeddingRequest sends an embedding request to the specified provider.

func (*Bifrost) ExecuteChatMCPTool 

func (bifrost *Bifrost) ExecuteChatMCPTool(ctx *schemas.BifrostContext, toolCall *schemas.ChatAssistantMessageToolCall) (*schemas.ChatMessage, *schemas.BifrostError)

ExecuteChatMCPTool executes an MCP tool call and returns the result as a chat message. This is the main public API for manual MCP tool execution in Chat format.

Parameters:

  • ctx: Execution context
  • toolCall: The tool call to execute (from assistant message)

Returns:

  • *schemas.ChatMessage: Tool message with execution result
  • *schemas.BifrostError: Any execution error

func (*Bifrost) ExecuteResponsesMCPTool 

func (bifrost *Bifrost) ExecuteResponsesMCPTool(ctx *schemas.BifrostContext, toolCall *schemas.ResponsesToolMessage) (*schemas.ResponsesMessage, *schemas.BifrostError)

ExecuteResponsesMCPTool executes an MCP tool call and returns the result as a responses message. This is the main public API for manual MCP tool execution in Responses format.

Parameters:

  • ctx: Execution context
  • toolCall: The tool call to execute (from assistant message)

Returns:

  • *schemas.ResponsesMessage: Tool message with execution result
  • *schemas.BifrostError: Any execution error

func (*Bifrost) FileContentRequest 

func (bifrost *Bifrost) FileContentRequest(ctx *schemas.BifrostContext, req *schemas.BifrostFileContentRequest) (*schemas.BifrostFileContentResponse, *schemas.BifrostError)

FileContentRequest downloads file content from the specified provider.

func (*Bifrost) FileDeleteRequest 

func (bifrost *Bifrost) FileDeleteRequest(ctx *schemas.BifrostContext, req *schemas.BifrostFileDeleteRequest) (*schemas.BifrostFileDeleteResponse, *schemas.BifrostError)

FileDeleteRequest deletes a file from the specified provider.

func (*Bifrost) FileListRequest 

func (bifrost *Bifrost) FileListRequest(ctx *schemas.BifrostContext, req *schemas.BifrostFileListRequest) (*schemas.BifrostFileListResponse, *schemas.BifrostError)

FileListRequest lists files from the specified provider.

func (*Bifrost) FileRetrieveRequest 

func (bifrost *Bifrost) FileRetrieveRequest(ctx *schemas.BifrostContext, req *schemas.BifrostFileRetrieveRequest) (*schemas.BifrostFileRetrieveResponse, *schemas.BifrostError)

FileRetrieveRequest retrieves file metadata from the specified provider.

func (*Bifrost) FileUploadRequest 

func (bifrost *Bifrost) FileUploadRequest(ctx *schemas.BifrostContext, req *schemas.BifrostFileUploadRequest) (*schemas.BifrostFileUploadResponse, *schemas.BifrostError)

FileUploadRequest uploads a file to the specified provider.

func (*Bifrost) GetAvailableMCPTools 

func (bifrost *Bifrost) GetAvailableMCPTools(ctx *schemas.BifrostContext) []schemas.ChatTool

GetAvailableTools returns the available tools for the given context.

Returns:

  • []schemas.ChatTool: List of available tools

func (*Bifrost) GetConfiguredProviders 

func (bifrost *Bifrost) GetConfiguredProviders() ([]schemas.ModelProvider, error)

GetConfiguredProviders returns the configured providers.

Returns:

  • []schemas.ModelProvider: List of configured providers
  • error: Any error that occurred during the retrieval process

Example: 

providers, err := bifrost.GetConfiguredProviders()
if err != nil {
	return nil, err
}
fmt.Println(providers)

func (*Bifrost) GetDropExcessRequests 

func (bifrost *Bifrost) GetDropExcessRequests() bool

GetDropExcessRequests returns the current value of DropExcessRequests

func (*Bifrost) GetMCPClients 

func (bifrost *Bifrost) GetMCPClients() ([]schemas.MCPClient, error)

GetMCPClients returns all MCP clients managed by the Bifrost instance.

Returns:

  • []schemas.MCPClient: List of all MCP clients
  • error: Any retrieval error

func (*Bifrost) GetProviderByKey 

func (bifrost *Bifrost) GetProviderByKey(providerKey schemas.ModelProvider) schemas.Provider

GetProviderByKey returns the provider instance for the given provider key. Returns nil if no provider with the given key exists.

func (*Bifrost) ImageEditRequest 

func (bifrost *Bifrost) ImageEditRequest(ctx *schemas.BifrostContext, req *schemas.BifrostImageEditRequest) (*schemas.BifrostImageGenerationResponse, *schemas.BifrostError)

ImageEditRequest sends an image edit request to the specified provider.

func (*Bifrost) ImageEditStreamRequest 

func (bifrost *Bifrost) ImageEditStreamRequest(ctx *schemas.BifrostContext, req *schemas.BifrostImageEditRequest) (chan *schemas.BifrostStreamChunk, *schemas.BifrostError)

ImageEditStreamRequest sends an image edit stream request to the specified provider.

func (*Bifrost) ImageGenerationRequest 

func (bifrost *Bifrost) ImageGenerationRequest(ctx *schemas.BifrostContext,
	req *schemas.BifrostImageGenerationRequest,
) (*schemas.BifrostImageGenerationResponse, *schemas.BifrostError)

ImageGenerationRequest sends an image generation request to the specified provider.

func (*Bifrost) ImageGenerationStreamRequest 

func (bifrost *Bifrost) ImageGenerationStreamRequest(ctx *schemas.BifrostContext,
	req *schemas.BifrostImageGenerationRequest,
) (chan *schemas.BifrostStreamChunk, *schemas.BifrostError)

ImageGenerationStreamRequest sends an image generation stream request to the specified provider.

func (*Bifrost) ImageVariationRequest 

func (bifrost *Bifrost) ImageVariationRequest(ctx *schemas.BifrostContext, req *schemas.BifrostImageVariationRequest) (*schemas.BifrostImageGenerationResponse, *schemas.BifrostError)

ImageVariationRequest sends an image variation request to the specified provider.

func (*Bifrost) ListAllModels 

func (bifrost *Bifrost) ListAllModels(ctx *schemas.BifrostContext, req *schemas.BifrostListModelsRequest) (*schemas.BifrostListModelsResponse, *schemas.BifrostError)

ListAllModels lists all models from all configured providers. It accumulates responses from all providers with a limit of 1000 per provider to get all results.

func (*Bifrost) ListModelsRequest 

func (bifrost *Bifrost) ListModelsRequest(ctx *schemas.BifrostContext, req *schemas.BifrostListModelsRequest) (*schemas.BifrostListModelsResponse, *schemas.BifrostError)

ListModelsRequest sends a list models request to the specified provider.

func (*Bifrost) OCRRequest added in v1.4.20 

func (bifrost *Bifrost) OCRRequest(ctx *schemas.BifrostContext, req *schemas.BifrostOCRRequest) (*schemas.BifrostOCRResponse, *schemas.BifrostError)

OCRRequest sends an OCR request to the specified provider.

func (*Bifrost) Passthrough 

func (bifrost *Bifrost) Passthrough(
	ctx *schemas.BifrostContext,
	provider schemas.ModelProvider,
	req *schemas.BifrostPassthroughRequest,
) (*schemas.BifrostPassthroughResponse, *schemas.BifrostError)

func (*Bifrost) PassthroughStream 

func (bifrost *Bifrost) PassthroughStream(
	ctx *schemas.BifrostContext,
	provider schemas.ModelProvider,
	req *schemas.BifrostPassthroughRequest,
) (chan *schemas.BifrostStreamChunk, *schemas.BifrostError)

func (*Bifrost) ReconnectMCPClient 

func (bifrost *Bifrost) ReconnectMCPClient(id string) error

ReconnectMCPClient attempts to reconnect an MCP client if it is disconnected.

Parameters:

  • id: ID of the client to reconnect

Returns:

  • error: Any reconnection error

func (*Bifrost) RegisterMCPTool 

func (bifrost *Bifrost) RegisterMCPTool(name, description string, handler func(args any) (string, error), toolSchema schemas.ChatTool) error

RegisterMCPTool registers a typed tool handler with the MCP integration. This allows developers to easily add custom tools that will be available to all LLM requests processed by this Bifrost instance.

Parameters:

  • name: Unique tool name
  • description: Human-readable tool description
  • handler: Function that handles tool execution
  • toolSchema: Bifrost tool schema for function calling

Returns:

  • error: Any registration error

Example: 

type EchoArgs struct {
    Message string `json:"message"`
}

err := bifrost.RegisterMCPTool("echo", "Echo a message",
    func(args EchoArgs) (string, error) {
        return args.Message, nil
    }, toolSchema)

func (*Bifrost) ReloadConfig 

func (bifrost *Bifrost) ReloadConfig(config schemas.BifrostConfig) error

ReloadConfig reloads the config from DB Currently we update account, drop excess requests, and plugin lists We will keep on adding other aspects as required

func (*Bifrost) ReloadPlugin 

func (bifrost *Bifrost) ReloadPlugin(plugin schemas.BasePlugin, pluginTypes []schemas.PluginType) error

ReloadPlugin reloads a plugin with new instance During the reload - it's stop the world phase where we take a global lock on the plugin mutex

func (*Bifrost) RemoveMCPClient 

func (bifrost *Bifrost) RemoveMCPClient(id string) error

RemoveMCPClient removes an MCP client from the Bifrost instance. This allows for dynamic MCP client management at runtime.

Parameters:

  • id: ID of the client to remove

Returns:

  • error: Any removal error

Example: 

err := bifrost.RemoveMCPClient("my-mcp-client-id")
if err != nil {
    log.Fatalf("Failed to remove MCP client: %v", err)
}

func (*Bifrost) RemovePlugin 

func (bifrost *Bifrost) RemovePlugin(name string, pluginTypes []schemas.PluginType) error

RemovePlugin removes a plugin from the server.

func (*Bifrost) RemoveProvider 

func (bifrost *Bifrost) RemoveProvider(providerKey schemas.ModelProvider) error

RemoveProvider removes a provider from the server. This method gracefully stops all workers for the provider, closes the request queue, and removes the provider from the providers slice.

Parameters:

  • providerKey: The provider to remove

Returns:

  • error: Any error that occurred during the removal process

func (*Bifrost) ReorderPlugins 

func (bifrost *Bifrost) ReorderPlugins(orderedNames []string)

ReorderPlugins reorders all plugin slices (LLM, MCP) to match the given base plugin name ordering. This should be called after SortAndRebuildPlugins on the config layer to sync the core's execution order. Plugins not in the ordering are appended at the end (defensive).

func (*Bifrost) RerankRequest 

func (bifrost *Bifrost) RerankRequest(ctx *schemas.BifrostContext, req *schemas.BifrostRerankRequest) (*schemas.BifrostRerankResponse, *schemas.BifrostError)

RerankRequest sends a rerank request to the specified provider.

func (*Bifrost) ResponsesRequest 

func (bifrost *Bifrost) ResponsesRequest(ctx *schemas.BifrostContext, req *schemas.BifrostResponsesRequest) (*schemas.BifrostResponsesResponse, *schemas.BifrostError)

ResponsesRequest sends a responses request to the specified provider.

func (*Bifrost) ResponsesStreamRequest 

func (bifrost *Bifrost) ResponsesStreamRequest(ctx *schemas.BifrostContext, req *schemas.BifrostResponsesRequest) (chan *schemas.BifrostStreamChunk, *schemas.BifrostError)

ResponsesStreamRequest sends a responses stream request to the specified provider.

func (*Bifrost) RunRealtimeTurnPreHooks added in v1.5.5 

func (bifrost *Bifrost) RunRealtimeTurnPreHooks(ctx *schemas.BifrostContext, req *schemas.BifrostRequest) (*RealtimeTurnHooks, *schemas.BifrostError)

RunRealtimeTurnPreHooks acquires a plugin pipeline and runs LLM pre-hooks for a single realtime turn. Unlike generic stream hooks, realtime turns do not support short-circuit responses in v1 because the transports cannot yet emit a fully synthetic assistant turn without an upstream generation.

func (*Bifrost) RunStreamPreHooks 

func (bifrost *Bifrost) RunStreamPreHooks(ctx *schemas.BifrostContext, req *schemas.BifrostRequest) (*WSStreamHooks, *schemas.BifrostError)

RunStreamPreHooks acquires a plugin pipeline, sets up tracing context, runs PreLLMHooks, and returns a PostHookRunner for per-chunk post-processing. Used by WebSocket handlers that bypass the normal inference path but still need plugin hooks.

func (*Bifrost) SelectKeyForProviderRequestType added in v1.5.5 

func (bifrost *Bifrost) SelectKeyForProviderRequestType(ctx *schemas.BifrostContext, requestType schemas.RequestType, providerKey schemas.ModelProvider, model string) (schemas.Key, error)

SelectKeyForProviderRequestType selects an API key for the given provider, request type, and model. Used by WebSocket handlers that need a key for upstream connections while honoring request-specific AllowedRequests gates such as realtime-only support.

func (*Bifrost) SetClientTools added in v1.5.5 

func (bifrost *Bifrost) SetClientTools(clientID string, tools map[string]schemas.ChatTool, toolNameMapping map[string]string)

SetClientTools delegates to the MCP manager to update the tool map for an existing MCP client.

func (*Bifrost) SetMCPManager 

func (bifrost *Bifrost) SetMCPManager(manager mcp.MCPManagerInterface)

SetMCPManager sets the MCP manager for this Bifrost instance. This allows injecting a custom MCP manager implementation (e.g., for enterprise features). If the provided manager is a concrete *mcp.MCPManager, Bifrost's plugin pipeline is injected into the manager's CodeMode so that nested tool calls run through the plugin hooks.

Parameters:

  • manager: The MCP manager to set (must implement MCPManagerInterface)

func (*Bifrost) SetTracer 

func (bifrost *Bifrost) SetTracer(tracer schemas.Tracer)

SetTracer sets the tracer for the Bifrost instance.

func (*Bifrost) Shutdown 

func (bifrost *Bifrost) Shutdown()

Shutdown gracefully stops all workers when triggered. It closes all request channels and waits for workers to exit.

func (*Bifrost) SpeechRequest 

func (bifrost *Bifrost) SpeechRequest(ctx *schemas.BifrostContext, req *schemas.BifrostSpeechRequest) (*schemas.BifrostSpeechResponse, *schemas.BifrostError)

SpeechRequest sends a speech request to the specified provider.

func (*Bifrost) SpeechStreamRequest 

func (bifrost *Bifrost) SpeechStreamRequest(ctx *schemas.BifrostContext, req *schemas.BifrostSpeechRequest) (chan *schemas.BifrostStreamChunk, *schemas.BifrostError)

SpeechStreamRequest sends a speech stream request to the specified provider.

func (*Bifrost) TextCompletionRequest 

func (bifrost *Bifrost) TextCompletionRequest(ctx *schemas.BifrostContext, req *schemas.BifrostTextCompletionRequest) (*schemas.BifrostTextCompletionResponse, *schemas.BifrostError)

TextCompletionRequest sends a text completion request to the specified provider.

func (*Bifrost) TextCompletionStreamRequest 

func (bifrost *Bifrost) TextCompletionStreamRequest(ctx *schemas.BifrostContext, req *schemas.BifrostTextCompletionRequest) (chan *schemas.BifrostStreamChunk, *schemas.BifrostError)

TextCompletionStreamRequest sends a streaming text completion request to the specified provider.

func (*Bifrost) TranscriptionRequest 

func (bifrost *Bifrost) TranscriptionRequest(ctx *schemas.BifrostContext, req *schemas.BifrostTranscriptionRequest) (*schemas.BifrostTranscriptionResponse, *schemas.BifrostError)

TranscriptionRequest sends a transcription request to the specified provider.

func (*Bifrost) TranscriptionStreamRequest 

func (bifrost *Bifrost) TranscriptionStreamRequest(ctx *schemas.BifrostContext, req *schemas.BifrostTranscriptionRequest) (chan *schemas.BifrostStreamChunk, *schemas.BifrostError)

TranscriptionStreamRequest sends a transcription stream request to the specified provider.

func (*Bifrost) UpdateDropExcessRequests 

func (bifrost *Bifrost) UpdateDropExcessRequests(value bool)

UpdateDropExcessRequests updates the DropExcessRequests setting at runtime. This allows for hot-reloading of this configuration value.

func (*Bifrost) UpdateMCPClient 

func (bifrost *Bifrost) UpdateMCPClient(id string, updatedConfig *schemas.MCPClientConfig) error

UpdateMCPClient updates the MCP client. This allows for dynamic MCP client tool management at runtime.

Parameters:

  • id: ID of the client to edit
  • updatedConfig: Updated MCP client configuration

Returns:

  • error: Any edit error

Example: 

err := bifrost.UpdateMCPClient("my-mcp-client-id", schemas.MCPClientConfig{
    Name:           "my-mcp-client-name",
    ToolsToExecute: []string{"tool1", "tool2"},
})

func (*Bifrost) UpdateProvider 

func (bifrost *Bifrost) UpdateProvider(providerKey schemas.ModelProvider) error

UpdateProvider dynamically updates a provider with new configuration. This method gracefully recreates the provider instance with updated settings, stops existing workers, creates a new queue with updated settings, and starts new workers with the updated provider and concurrency configuration.

Parameters:

  • providerKey: The provider to update

Returns:

  • error: Any error that occurred during the update process

Note: This operation will temporarily pause request processing for the specified provider while the transition occurs. In-flight requests will complete before workers are stopped. Buffered requests in the old queue will be transferred to the new queue to prevent loss.

Concurrency safety — no-worker window: UpdateProvider holds a per-provider write lock (providerMutex.Lock) for its entire duration. All producer paths (tryRequest, tryStreamRequest) acquire the corresponding read lock inside getProviderQueue before they can look up or enqueue into any queue. This means no producer can observe or enqueue into newPq until UpdateProvider returns and releases the write lock — at which point new workers are already running and consuming newPq. There is therefore no window where newPq is visible to producers but has zero workers.

func (*Bifrost) UpdateToolManagerConfig 

func (bifrost *Bifrost) UpdateToolManagerConfig(maxAgentDepth int, toolExecutionTimeoutInSeconds int, codeModeBindingLevel string, disableAutoToolInject bool) error

UpdateToolManagerConfig updates the tool manager config for the MCP manager. This allows for hot-reloading of the tool manager config at runtime. Pass the current value of disableAutoToolInject whenever only other fields change so the flag is never silently reset to its zero value.

func (*Bifrost) VerifyPerUserOAuthConnection added in v1.5.5 

func (bifrost *Bifrost) VerifyPerUserOAuthConnection(ctx context.Context, config *schemas.MCPClientConfig, accessToken string) (map[string]schemas.ChatTool, map[string]string, error)

VerifyPerUserOAuthConnection delegates to the MCP manager to verify an MCP server using a temporary access token and discover available tools. The connection is closed after verification. If the MCP manager is not yet initialized, it is lazily created (same as AddMCPClient).

func (*Bifrost) VideoDeleteRequest 

func (bifrost *Bifrost) VideoDeleteRequest(ctx *schemas.BifrostContext, req *schemas.BifrostVideoDeleteRequest) (*schemas.BifrostVideoDeleteResponse, *schemas.BifrostError)

func (*Bifrost) VideoDownloadRequest 

func (bifrost *Bifrost) VideoDownloadRequest(ctx *schemas.BifrostContext, req *schemas.BifrostVideoDownloadRequest) (*schemas.BifrostVideoDownloadResponse, *schemas.BifrostError)

VideoDownloadRequest downloads video content from the provider.

func (*Bifrost) VideoGenerationRequest 

func (bifrost *Bifrost) VideoGenerationRequest(ctx *schemas.BifrostContext,
	req *schemas.BifrostVideoGenerationRequest,
) (*schemas.BifrostVideoGenerationResponse, *schemas.BifrostError)

VideoGenerationRequest sends a video generation request to the specified provider.

func (*Bifrost) VideoListRequest 

func (bifrost *Bifrost) VideoListRequest(ctx *schemas.BifrostContext, req *schemas.BifrostVideoListRequest) (*schemas.BifrostVideoListResponse, *schemas.BifrostError)

func (*Bifrost) VideoRemixRequest 

func (bifrost *Bifrost) VideoRemixRequest(ctx *schemas.BifrostContext, req *schemas.BifrostVideoRemixRequest) (*schemas.BifrostVideoGenerationResponse, *schemas.BifrostError)

func (*Bifrost) VideoRetrieveRequest 

func (bifrost *Bifrost) VideoRetrieveRequest(ctx *schemas.BifrostContext, req *schemas.BifrostVideoRetrieveRequest) (*schemas.BifrostVideoGenerationResponse, *schemas.BifrostError)

type ChannelMessage 

type ChannelMessage struct {
	schemas.BifrostRequest
	Context        *schemas.BifrostContext
	Response       chan *schemas.BifrostResponse
	ResponseStream chan chan *schemas.BifrostStreamChunk
	Err            chan schemas.BifrostError
}

ChannelMessage represents a message passed through the request channel. It contains the request, response and error channels, and the request type.

type DefaultLogger 

type DefaultLogger struct {
	// contains filtered or unexported fields
}

DefaultLogger implements the Logger interface with stdout/stderr printing. It provides a simple logging implementation that writes to standard output and error streams with formatted timestamps and log levels. It is used as the default logger if no logger is provided in the BifrostConfig.

func NewDefaultLogger 

func NewDefaultLogger(level schemas.LogLevel) *DefaultLogger

NewDefaultLogger creates a new DefaultLogger instance with the specified log level. The log level determines which messages will be output based on their severity.

func (*DefaultLogger) Debug 

func (logger *DefaultLogger) Debug(msg string, args ...any)

Debug logs a debug level message to stdout. Messages are only output if the logger's level is set to LogLevelDebug.

func (*DefaultLogger) Error 

func (logger *DefaultLogger) Error(msg string, args ...any)

Error logs an error level message to stderr. Error messages are always output regardless of the logger's level.

func (*DefaultLogger) Fatal 

func (logger *DefaultLogger) Fatal(msg string, args ...any)

Fatal logs a fatal-level message to stderr. Fatal messages are always output regardless of the logger's level.

func (*DefaultLogger) Info 

func (logger *DefaultLogger) Info(msg string, args ...any)

Info logs an info level message to stdout. Messages are output if the logger's level is LogLevelDebug or LogLevelInfo.

func (*DefaultLogger) LogHTTPRequest 

func (logger *DefaultLogger) LogHTTPRequest(level schemas.LogLevel, msg string) schemas.LogEventBuilder

LogHTTPRequest returns a LogEventBuilder for structured HTTP access logging. We are exposing the zerolog loggers directly to allow for more flexibility in logging and also to reduce the number of allocations we do in the logger.

func (*DefaultLogger) SetLevel 

func (logger *DefaultLogger) SetLevel(level schemas.LogLevel)

SetLevel sets the logging level for the logger. This determines which messages will be output based on their severity.

func (*DefaultLogger) SetOutputType 

func (logger *DefaultLogger) SetOutputType(outputType schemas.LoggerOutputType)

SetOutputType sets the output type for the logger. This determines the format of the log output. If the output type is unknown, it defaults to JSON

func (*DefaultLogger) Warn 

func (logger *DefaultLogger) Warn(msg string, args ...any)

Warn logs a warning level message to stdout. Messages are output if the logger's level is LogLevelDebug, LogLevelInfo, or LogLevelWarn.

type NoOpLogger 

type NoOpLogger struct{}

NoOpLogger is a no-op implementation of schemas.Logger.

func (*NoOpLogger) Debug 

func (l *NoOpLogger) Debug(string, ...any)

func (*NoOpLogger) Error 

func (l *NoOpLogger) Error(string, ...any)

func (*NoOpLogger) Fatal 

func (l *NoOpLogger) Fatal(string, ...any)

func (*NoOpLogger) Info 

func (l *NoOpLogger) Info(string, ...any)

func (*NoOpLogger) LogHTTPRequest 

func (l *NoOpLogger) LogHTTPRequest(schemas.LogLevel, string) schemas.LogEventBuilder

func (*NoOpLogger) SetLevel 

func (l *NoOpLogger) SetLevel(schemas.LogLevel)

func (*NoOpLogger) SetOutputType 

func (l *NoOpLogger) SetOutputType(schemas.LoggerOutputType)

func (*NoOpLogger) Warn 

func (l *NoOpLogger) Warn(string, ...any)

type PluginPipeline 

type PluginPipeline struct {
	// contains filtered or unexported fields
}

PluginPipeline encapsulates the execution of plugin PreHooks and PostHooks, tracks how many plugins ran, and manages short-circuiting and error aggregation.

func (*PluginPipeline) FinalizeStreamingPostHookSpans 

func (p *PluginPipeline) FinalizeStreamingPostHookSpans(ctx context.Context)

FinalizeStreamingPostHookSpans creates aggregated spans for each plugin after streaming completes. This should be called once at the end of streaming to create one span per plugin with average timing. Spans are nested to mirror the pre-hook hierarchy (each post-hook is a child of the previous one).

func (*PluginPipeline) GetChunkCount 

func (p *PluginPipeline) GetChunkCount() int

GetChunkCount returns the number of chunks processed during streaming

func (*PluginPipeline) RunLLMPreHooks 

func (p *PluginPipeline) RunLLMPreHooks(ctx *schemas.BifrostContext, req *schemas.BifrostRequest) (*schemas.BifrostRequest, *schemas.LLMPluginShortCircuit, int)

RunLLMPreHooks executes PreHooks in order, tracks how many ran, and returns the final request, any short-circuit decision, and the count.

func (*PluginPipeline) RunMCPPostHooks 

func (p *PluginPipeline) RunMCPPostHooks(ctx *schemas.BifrostContext, mcpResp *schemas.BifrostMCPResponse, bifrostErr *schemas.BifrostError, runFrom int) (*schemas.BifrostMCPResponse, *schemas.BifrostError)

RunMCPPostHooks executes MCP PostHooks in reverse order for the plugins whose PreMCPHook ran. Accepts the MCP response and error, and allows plugins to transform either (e.g., recover from error, or invalidate a response). Returns the final MCP response and error after all hooks. If both are set, error takes precedence unless error is nil. runFrom is the count of plugins whose PreHooks ran; PostHooks will run in reverse from index (runFrom - 1) down to 0

func (*PluginPipeline) RunMCPPreHooks 

func (p *PluginPipeline) RunMCPPreHooks(ctx *schemas.BifrostContext, req *schemas.BifrostMCPRequest) (*schemas.BifrostMCPRequest, *schemas.MCPPluginShortCircuit, int)

RunMCPPreHooks executes MCP PreHooks in order for all registered MCP plugins. Returns the modified request, any short-circuit decision, and the count of hooks that ran. If a plugin short-circuits, only PostHooks for plugins up to and including that plugin will run.

func (*PluginPipeline) RunPostLLMHooks 

func (p *PluginPipeline) RunPostLLMHooks(ctx *schemas.BifrostContext, resp *schemas.BifrostResponse, bifrostErr *schemas.BifrostError, runFrom int) (*schemas.BifrostResponse, *schemas.BifrostError)

RunPostLLMHooks executes PostHooks in reverse order for the plugins whose PreLLMHook ran. Accepts the response and error, and allows plugins to transform either (e.g., recover from error, or invalidate a response). Returns the final response and error after all hooks. If both are set, error takes precedence unless error is nil. runFrom is the count of plugins whose PreHooks ran; PostHooks will run in reverse from index (runFrom - 1) down to 0 For streaming requests, it accumulates timing per plugin instead of creating individual spans per chunk.

type ProviderQueue 

type ProviderQueue struct {
	// contains filtered or unexported fields
}

ProviderQueue wraps a provider's request channel with lifecycle management to prevent "send on closed channel" panics during provider removal/update. Producers must check the closing flag or select on the done channel before sending.

Why pq.queue is NEVER closed:

Closing a channel in Go causes any concurrent send to that channel to panic ("send on closed channel"). There is always a TOCTOU window between a producer's isClosing() check and its select { case pq.queue <- msg: ... }: the producer could pass isClosing() while the queue is open, get preempted, and resume only after the queue is closed. Go's selectgo evaluates select cases in a random order, so even having case <-pq.done: in the same select does not protect against this — if selectgo evaluates the send case first on a closed channel it panics immediately via goto sclose, before reaching done.

To close pq.queue safely you would need a sender-side WaitGroup so that signalClosing could wait for every in-flight producer to finish. That adds non-trivial overhead on the hot request path.

Instead, pq.done is the sole shutdown signal. Receiving from a closed channel is always safe (returns the zero value immediately), so:

  • Workers exit via case <-pq.done: — safe
  • Producers bail via case <-pq.done: — safe
  • drainQueueWithErrors handles any messages that slip through the TOCTOU window

pq.queue is garbage collected automatically:

  • RemoveProvider calls requestQueues.Delete, dropping the map's reference.
  • UpdateProvider calls requestQueues.Store with a new queue, dropping the map's reference to oldPq. Shutdown does not Delete at all — the whole Bifrost instance is torn down. In all cases, once no producer goroutine holds a reference to the ProviderQueue, both the struct and pq.queue are eligible for GC. No explicit close is needed.

type RealtimeTurnHooks added in v1.5.5 

type RealtimeTurnHooks struct {
	PostHookRunner schemas.PostHookRunner
	Cleanup        func()
}

RealtimeTurnHooks mirrors RunStreamPreHooks but is explicitly scoped to a single realtime turn rather than one long-lived transport connection.

type WSStreamHooks 

type WSStreamHooks struct {
	PostHookRunner       schemas.PostHookRunner
	Cleanup              func()
	ShortCircuitResponse *schemas.BifrostResponse
}

WSStreamHooks holds the post-hook runner and cleanup function returned by RunStreamPreHooks. Call PostHookRunner for each streaming chunk, setting StreamEndIndicator on the final chunk. Call Cleanup when done to release the pipeline back to the pool. If ShortCircuitResponse is non-nil, a plugin short-circuited with a cached response — the caller should write this response to the client and skip the upstream call.

Source Files

View all Source files

Directories

Path Synopsis
internal
llmtests
Package llmtests provides comprehensive test account and configuration management for the Bifrost system.
 Package llmtests provides comprehensive test account and configuration management for the Bifrost system.
mcptests
mcp
codemode/starlark
Package starlark provides a Starlark-based implementation of the CodeMode interface.
 Package starlark provides a Starlark-based implementation of the CodeMode interface.
utils
Package network provides centralized HTTP client management with proxy support.
 Package network provides centralized HTTP client management with proxy support.
providers
anthropic
Package anthropic implements the Anthropic provider for the Bifrost API.
 Package anthropic implements the Anthropic provider for the Bifrost API.
azure
Package azure implements the Azure provider.
 Package azure implements the Azure provider.
bedrock
cerebras
Package cerebras implements the Cerebras LLM provider.
 Package cerebras implements the Cerebras LLM provider.
cohere
deepseek
Package providers implements various LLM providers and their utility functions.
 Package providers implements various LLM providers and their utility functions.
elevenlabs
fireworks
Package fireworks implements the Fireworks AI provider and its utility functions.
 Package fireworks implements the Fireworks AI provider and its utility functions.
gemini
Package gemini provides types and structures for interacting with Google's Gemini API.
 Package gemini provides types and structures for interacting with Google's Gemini API.
glm
Package providers implements various LLM providers and their utility functions.
 Package providers implements various LLM providers and their utility functions.
groq
Package groq implements the Groq provider and its utility functions.
 Package groq implements the Groq provider and its utility functions.
huggingface
Package huggingface provides a HuggingFace chat provider.
 Package huggingface provides a HuggingFace chat provider.
minimax
Package providers implements various LLM providers and their utility functions.
 Package providers implements various LLM providers and their utility functions.
mistral
Package mistral implements the Mistral provider.
 Package mistral implements the Mistral provider.
moonshot
Package providers implements various LLM providers and their utility functions.
 Package providers implements various LLM providers and their utility functions.
nebius
Package nebius implements the Nebius LLM provider.
 Package nebius implements the Nebius LLM provider.
ollama
Package providers implements various LLM providers and their utility functions.
 Package providers implements various LLM providers and their utility functions.
openai
Package openai provides the OpenAI provider implementation for the Bifrost framework.
 Package openai provides the OpenAI provider implementation for the Bifrost framework.
openrouter
Package openrouter implements the OpenRouter LLM provider.
 Package openrouter implements the OpenRouter LLM provider.
parasail
Package providers implements various LLM providers and their utility functions.
 Package providers implements various LLM providers and their utility functions.
perplexity
Package providers implements various LLM providers and their utility functions.
 Package providers implements various LLM providers and their utility functions.
qwen
Package providers implements various LLM providers and their utility functions.
 Package providers implements various LLM providers and their utility functions.
replicate
Package providers implements various LLM providers and their utility functions.
 Package providers implements various LLM providers and their utility functions.
runway
Package providers implements various LLM providers and their utility functions.
 Package providers implements various LLM providers and their utility functions.
sgl
Package providers implements various LLM providers and their utility functions.
 Package providers implements various LLM providers and their utility functions.
utils
Package utils provides common utility functions used across different provider implementations.
 Package utils provides common utility functions used across different provider implementations.
vertex
vllm
Package vllm implements the vLLM LLM provider (OpenAI-compatible).
 Package vllm implements the vLLM LLM provider (OpenAI-compatible).
volcengine
Package providers implements various LLM providers and their utility functions.
 Package providers implements various LLM providers and their utility functions.
xai
Package providers implements various LLM providers and their utility functions.
 Package providers implements various LLM providers and their utility functions.
Package schemas defines the core schemas and types used by the Bifrost system.
 Package schemas defines the core schemas and types used by the Bifrost system.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.