bifrost

package module
v1.7.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 21, 2026 License: Apache-2.0 Imports: 58 Imported by: 22

Documentation

Overview

Package bifrost provides the core implementation of the Bifrost system. Bifrost is a unified interface for interacting with various AI model providers, managing concurrent requests, and handling provider-specific configurations.

Package bifrost provides the core implementation of the Bifrost system.

Index

Constants

View Source 
const (
	ProviderAutoResolveErrorMessage = "could not auto resolve a provider for the request, please specify a provider explicitly"
	ModelAutoResolveErrorMessage    = "could not auto resolve a model for the request, please specify a model explicitly"
)

Variables

This section is empty.

Functions

func CanProviderKeyValueBeEmpty added in v1.4.5 

func CanProviderKeyValueBeEmpty(providerKey schemas.ModelProvider) bool

CanProviderKeyValueBeEmpty returns true if the given provider allows the API key to be empty. Some providers like Vertex and Bedrock have their credentials in additional key configs. Ollama and SGL are keyless (API Key is optional) but use per-key server URLs.

func ClearContextForInternalRequest added in v1.6.3 

func ClearContextForInternalRequest(ctx *schemas.BifrostContext)

ClearContextForInternalRequest clears context state that is specific to the caller's original request, so a context derived from it can carry an internal sub-request (e.g. a plugin generating an embedding for its own use) that must behave like a fresh top-level request.

Two categories are cleared:

  • Key routing: key-selection state resolved for the caller's provider (governance key allow-list, pinned/direct keys, key-selection skip). An internal request typically targets a different provider, and when it skips the plugin pipeline this state is never re-resolved — inherited, it is applied against the wrong provider's key pool and rejects every key ("no keys found for provider").
  • Body transport: raw-body passthrough and large-payload/large-response streaming state, plus caller-forwarded extra headers and the caller's URL-path override. Inherited, these make providers send the caller's raw or streamed body instead of marshaling the internal request, route it to the caller's endpoint path instead of the internal request's own, and forward the caller's headers on a call the caller doesn't own.

Deliberately not cleared: tracing/observability keys (the sub-request should stay tied to the caller's trace) and BifrostContextKeySkipPluginPipeline (whether the internal request runs the plugin pipeline is the caller's decision).

func GetBoolFromContext added in v1.4.3 

func GetBoolFromContext(ctx context.Context, key any) bool

GetBoolFromContext safely extracts a bool value from context

func GetErrorMessage added in v1.2.20 

func GetErrorMessage(err *schemas.BifrostError) string

// [Deprecated] use err.GetErrorString() instead. Will be removed in a future release.

func GetIntFromContext added in v1.2.21 

func GetIntFromContext(ctx context.Context, key any) int

GetIntFromContext safely extracts an int value from context

func GetResponseFields added in v1.2.7 

func GetResponseFields(result *schemas.BifrostResponse, err *schemas.BifrostError) (requestType schemas.RequestType, provider schemas.ModelProvider, originalModel string, resolvedModel string)

GetResponseFields extracts the request type, provider, original model, and resolved model from the result or error.

func GetResponseRoutingInfo added in v1.5.19 

func GetResponseRoutingInfo(result *schemas.BifrostResponse, err *schemas.BifrostError) schemas.RoutingInfo

GetResponseRoutingInfo extracts the RoutingInfo recorded on a completed attempt — from the accumulated response, or the error when the attempt failed.

func GetStringFromContext added in v1.2.21 

func GetStringFromContext(ctx context.Context, key any) string

GetStringFromContext safely extracts a string value from context

func GetTracerFromContext added in v1.3.4 

func GetTracerFromContext(ctx *schemas.BifrostContext) (schemas.Tracer, string, error)

func IsCodemodeTool added in v1.4.0 

func IsCodemodeTool(toolName string) bool

IsCodemodeTool returns true if the given tool name is a codemode tool.

func IsFinalChunk added in v1.1.31 

func IsFinalChunk(ctx *schemas.BifrostContext) bool

IsFinalChunk returns true if the given context is a final chunk.

func IsRateLimitErrorMessage added in v1.2.18 

func IsRateLimitErrorMessage(errorMessage string) bool

IsRateLimitErrorMessage checks if an error message indicates a rate limit issue

func IsStandardProvider added in v1.1.26 

func IsStandardProvider(providerKey schemas.ModelProvider) bool

IsStandardProvider reports whether providerKey is a built-in (non-custom) provider.

func IsStreamRequestType added in v1.1.31 

func IsStreamRequestType(reqType schemas.RequestType) bool

IsStreamRequestType returns true if the given request type is a stream request.

func IsSupportedBaseProvider added in v1.1.26 

func IsSupportedBaseProvider(providerKey schemas.ModelProvider) bool

IsSupportedBaseProvider reports whether providerKey is allowed as a base provider for custom providers.

func MarshalUnsafe added in v1.2.17 

func MarshalUnsafe(v any) string

MarshalUnsafe marshals the given value to a JSON string without escaping HTML characters. Returns empty string if marshaling fails.

func NewNoOpLogger added in v1.4.1 

func NewNoOpLogger() schemas.Logger

NewNoOpLogger creates a new NoOpLogger instance.

func Ptr added in v1.0.4 

func Ptr[T any](v T) *T

Ptr returns a pointer to the given value.

func RedactSensitiveString added in v1.2.30 

func RedactSensitiveString(s string) string

RedactSensitiveString redacts sensitive information in a string

func ValidateExternalURL added in v1.2.30 

func ValidateExternalURL(urlStr string, allowPrivateNetwork bool) error

ValidateExternalURL validates a URL for security concerns (SSRF protection). When allowPrivateNetwork is true, RFC 1918 private IPs are permitted (for k8s/LAN deployments). Link-local addresses (169.254.x.x, fe80::) are always blocked regardless of allowPrivateNetwork.

Types

type Bifrost 

type Bifrost struct {
	MCPManager mcp.MCPManagerInterface // MCP integration manager (nil if MCP not configured)
	// contains filtered or unexported fields
}

Bifrost manages providers and maintains specified open channels for concurrent processing. It handles request routing, provider management, and response processing.

func Init 

func Init(ctx context.Context, config schemas.BifrostConfig) (*Bifrost, error)

Init initializes a new Bifrost instance with the given configuration. It sets up the account, plugins, object pools, and initializes providers. Returns an error if initialization fails. Initial Memory Allocations happens here as per the initial pool size.

func (*Bifrost) AddMCPClient added in v1.1.7 

func (bifrost *Bifrost) AddMCPClient(ctx context.Context, config *schemas.MCPClientConfig) error

func (*Bifrost) BatchCancelRequest added in v1.2.38 

func (bifrost *Bifrost) BatchCancelRequest(ctx *schemas.BifrostContext, req *schemas.BifrostBatchCancelRequest) (*schemas.BifrostBatchCancelResponse, *schemas.BifrostError)

BatchCancelRequest cancels a batch job.

func (*Bifrost) BatchCreateRequest added in v1.2.38 

func (bifrost *Bifrost) BatchCreateRequest(ctx *schemas.BifrostContext, req *schemas.BifrostBatchCreateRequest) (*schemas.BifrostBatchCreateResponse, *schemas.BifrostError)

BatchCreateRequest creates a new batch job for asynchronous processing.

func (*Bifrost) BatchDeleteRequest added in v1.4.8 

func (bifrost *Bifrost) BatchDeleteRequest(ctx *schemas.BifrostContext, req *schemas.BifrostBatchDeleteRequest) (*schemas.BifrostBatchDeleteResponse, *schemas.BifrostError)

BatchDeleteRequest deletes a batch job.

func (*Bifrost) BatchListRequest added in v1.2.38 

func (bifrost *Bifrost) BatchListRequest(ctx *schemas.BifrostContext, req *schemas.BifrostBatchListRequest) (*schemas.BifrostBatchListResponse, *schemas.BifrostError)

BatchListRequest lists batch jobs for the specified provider.

func (*Bifrost) BatchResultsRequest added in v1.2.38 

func (bifrost *Bifrost) BatchResultsRequest(ctx *schemas.BifrostContext, req *schemas.BifrostBatchResultsRequest) (*schemas.BifrostBatchResultsResponse, *schemas.BifrostError)

BatchResultsRequest retrieves results from a completed batch job.

func (*Bifrost) BatchRetrieveRequest added in v1.2.38 

func (bifrost *Bifrost) BatchRetrieveRequest(ctx *schemas.BifrostContext, req *schemas.BifrostBatchRetrieveRequest) (*schemas.BifrostBatchRetrieveResponse, *schemas.BifrostError)

BatchRetrieveRequest retrieves a specific batch job.

func (*Bifrost) CachedContentCreateRequest added in v1.5.8 

func (bifrost *Bifrost) CachedContentCreateRequest(ctx *schemas.BifrostContext, req *schemas.BifrostCachedContentCreateRequest) (*schemas.BifrostCachedContentCreateResponse, *schemas.BifrostError)

CachedContentCreateRequest creates a new cached content (Gemini / Vertex AI named cache lifecycle).

func (*Bifrost) CachedContentDeleteRequest added in v1.5.8 

func (bifrost *Bifrost) CachedContentDeleteRequest(ctx *schemas.BifrostContext, req *schemas.BifrostCachedContentDeleteRequest) (*schemas.BifrostCachedContentDeleteResponse, *schemas.BifrostError)

CachedContentDeleteRequest deletes a cached content by name.

func (*Bifrost) CachedContentListRequest added in v1.5.8 

func (bifrost *Bifrost) CachedContentListRequest(ctx *schemas.BifrostContext, req *schemas.BifrostCachedContentListRequest) (*schemas.BifrostCachedContentListResponse, *schemas.BifrostError)

CachedContentListRequest lists cached contents.

func (*Bifrost) CachedContentRetrieveRequest added in v1.5.8 

func (bifrost *Bifrost) CachedContentRetrieveRequest(ctx *schemas.BifrostContext, req *schemas.BifrostCachedContentRetrieveRequest) (*schemas.BifrostCachedContentRetrieveResponse, *schemas.BifrostError)

CachedContentRetrieveRequest retrieves a single cached content by name.

func (*Bifrost) CachedContentUpdateRequest added in v1.5.8 

func (bifrost *Bifrost) CachedContentUpdateRequest(ctx *schemas.BifrostContext, req *schemas.BifrostCachedContentUpdateRequest) (*schemas.BifrostCachedContentUpdateResponse, *schemas.BifrostError)

CachedContentUpdateRequest updates expiration on a cached content.

func (*Bifrost) ChatCompletionRequest 

func (bifrost *Bifrost) ChatCompletionRequest(ctx *schemas.BifrostContext, req *schemas.BifrostChatRequest) (*schemas.BifrostChatResponse, *schemas.BifrostError)

ChatCompletionRequest sends a chat completion request to the specified provider.

func (*Bifrost) ChatCompletionStreamRequest added in v1.1.8 

func (bifrost *Bifrost) ChatCompletionStreamRequest(ctx *schemas.BifrostContext, req *schemas.BifrostChatRequest) (chan *schemas.BifrostStreamChunk, *schemas.BifrostError)

ChatCompletionStreamRequest sends a chat completion stream request to the specified provider.

func (*Bifrost) CompactionRequest added in v1.5.17 

func (bifrost *Bifrost) CompactionRequest(ctx *schemas.BifrostContext, req *schemas.BifrostCompactionRequest) (*schemas.BifrostCompactionResponse, *schemas.BifrostError)

CompactionRequest compacts a conversation context window via providers that implement the OpenAI-compatible /v1/responses/compact flow (OpenAI, Azure OpenAI, xAI). Providers without compaction support return an unsupported-operation error.

func (*Bifrost) ComputeRawStorageForProvider added in v1.5.11 

func (bifrost *Bifrost) ComputeRawStorageForProvider(ctx *schemas.BifrostContext, providerKey schemas.ModelProvider) bool

ComputeRawStorageForProvider determines whether raw request/response payloads should be captured and stored in log records for the given provider. This is the same computation performed inside executeRequest (lines 5675-5713), exported for callers that bypass the normal inference path (e.g. realtime WebSocket/WebRTC sessions).

func (*Bifrost) ConnectConfiguredMCPClients added in v1.5.22 

func (bifrost *Bifrost) ConnectConfiguredMCPClients(ctx context.Context)

AddMCPClient adds a new MCP client to the Bifrost instance. This allows for dynamic MCP client management at runtime.

Parameters:

  • config: MCP client configuration

Returns:

  • error: Any registration error

Example: 

err := bifrost.AddMCPClient(ctx, &schemas.MCPClientConfig{
    Name: "my-mcp-client",
    ConnectionType: schemas.MCPConnectionTypeHTTP,
    ConnectionString: &url,
})

ConnectConfiguredMCPClients dials the MCP clients supplied to Init. Construction no longer auto-connects, so callers invoke this after all plugins are registered (so every PreMCPConnectionHook participates in the connection). No-op if MCP is not configured.

func (*Bifrost) ContainerCreateRequest added in v1.3.12 

func (bifrost *Bifrost) ContainerCreateRequest(ctx *schemas.BifrostContext, req *schemas.BifrostContainerCreateRequest) (*schemas.BifrostContainerCreateResponse, *schemas.BifrostError)

ContainerCreateRequest creates a new container.

func (*Bifrost) ContainerDeleteRequest added in v1.3.12 

func (bifrost *Bifrost) ContainerDeleteRequest(ctx *schemas.BifrostContext, req *schemas.BifrostContainerDeleteRequest) (*schemas.BifrostContainerDeleteResponse, *schemas.BifrostError)

ContainerDeleteRequest deletes a container.

func (*Bifrost) ContainerFileContentRequest added in v1.3.12 

func (bifrost *Bifrost) ContainerFileContentRequest(ctx *schemas.BifrostContext, req *schemas.BifrostContainerFileContentRequest) (*schemas.BifrostContainerFileContentResponse, *schemas.BifrostError)

ContainerFileContentRequest retrieves the content of a file from a container.

func (*Bifrost) ContainerFileCreateRequest added in v1.3.12 

func (bifrost *Bifrost) ContainerFileCreateRequest(ctx *schemas.BifrostContext, req *schemas.BifrostContainerFileCreateRequest) (*schemas.BifrostContainerFileCreateResponse, *schemas.BifrostError)

ContainerFileCreateRequest creates a file in a container.

func (*Bifrost) ContainerFileDeleteRequest added in v1.3.12 

func (bifrost *Bifrost) ContainerFileDeleteRequest(ctx *schemas.BifrostContext, req *schemas.BifrostContainerFileDeleteRequest) (*schemas.BifrostContainerFileDeleteResponse, *schemas.BifrostError)

ContainerFileDeleteRequest deletes a file from a container.

func (*Bifrost) ContainerFileListRequest added in v1.3.12 

func (bifrost *Bifrost) ContainerFileListRequest(ctx *schemas.BifrostContext, req *schemas.BifrostContainerFileListRequest) (*schemas.BifrostContainerFileListResponse, *schemas.BifrostError)

ContainerFileListRequest lists files in a container.

func (*Bifrost) ContainerFileRetrieveRequest added in v1.3.12 

func (bifrost *Bifrost) ContainerFileRetrieveRequest(ctx *schemas.BifrostContext, req *schemas.BifrostContainerFileRetrieveRequest) (*schemas.BifrostContainerFileRetrieveResponse, *schemas.BifrostError)

ContainerFileRetrieveRequest retrieves a file from a container.

func (*Bifrost) ContainerListRequest added in v1.3.12 

func (bifrost *Bifrost) ContainerListRequest(ctx *schemas.BifrostContext, req *schemas.BifrostContainerListRequest) (*schemas.BifrostContainerListResponse, *schemas.BifrostError)

ContainerListRequest lists containers.

func (*Bifrost) ContainerRetrieveRequest added in v1.3.12 

func (bifrost *Bifrost) ContainerRetrieveRequest(ctx *schemas.BifrostContext, req *schemas.BifrostContainerRetrieveRequest) (*schemas.BifrostContainerRetrieveResponse, *schemas.BifrostError)

ContainerRetrieveRequest retrieves a specific container.

func (*Bifrost) CountTokensRequest added in v1.2.43 

func (bifrost *Bifrost) CountTokensRequest(ctx *schemas.BifrostContext, req *schemas.BifrostResponsesRequest) (*schemas.BifrostCountTokensResponse, *schemas.BifrostError)

CountTokensRequest sends a count tokens request to the specified provider.

func (*Bifrost) DisableMCPClient added in v1.5.8 

func (bifrost *Bifrost) DisableMCPClient(id string) error

DisableMCPClient shuts down an MCP client's connection, health monitor, and tool syncer without removing it. The client entry is kept in a "disabled" state so it can be re-enabled via EnableMCPClient.

func (*Bifrost) EmbeddingRequest added in v1.1.7 

func (bifrost *Bifrost) EmbeddingRequest(ctx *schemas.BifrostContext, req *schemas.BifrostEmbeddingRequest) (*schemas.BifrostEmbeddingResponse, *schemas.BifrostError)

EmbeddingRequest sends an embedding request to the specified provider.

func (*Bifrost) EnableMCPClient added in v1.5.8 

func (bifrost *Bifrost) EnableMCPClient(id string) error

EnableMCPClient reconnects a previously disabled MCP client and restarts its health monitor and tool syncer.

func (*Bifrost) ExecuteChatMCPTool added in v1.3.0 

func (bifrost *Bifrost) ExecuteChatMCPTool(ctx *schemas.BifrostContext, toolCall *schemas.ChatAssistantMessageToolCall) (*schemas.ChatMessage, *schemas.BifrostError)

ExecuteChatMCPTool executes an MCP tool call and returns the result as a chat message. This is the main public API for manual MCP tool execution in Chat format. All the real work — request pooling, plugin gate (PreMCPHook / PostMCPHook), short-circuit handling, error enrichment — lives on MCPManager.ExecuteChatTool.

func (*Bifrost) ExecuteResponsesMCPTool added in v1.3.0 

func (bifrost *Bifrost) ExecuteResponsesMCPTool(ctx *schemas.BifrostContext, toolCall *schemas.ResponsesToolMessage) (*schemas.ResponsesMessage, *schemas.BifrostError)

ExecuteResponsesMCPTool executes an MCP tool call and returns the result as a responses message. Thin delegator — see ExecuteChatMCPTool for the rationale.

func (*Bifrost) FileContentRequest added in v1.2.38 

func (bifrost *Bifrost) FileContentRequest(ctx *schemas.BifrostContext, req *schemas.BifrostFileContentRequest) (*schemas.BifrostFileContentResponse, *schemas.BifrostError)

FileContentRequest downloads file content from the specified provider.

func (*Bifrost) FileDeleteRequest added in v1.2.38 

func (bifrost *Bifrost) FileDeleteRequest(ctx *schemas.BifrostContext, req *schemas.BifrostFileDeleteRequest) (*schemas.BifrostFileDeleteResponse, *schemas.BifrostError)

FileDeleteRequest deletes a file from the specified provider.

func (*Bifrost) FileListRequest added in v1.2.38 

func (bifrost *Bifrost) FileListRequest(ctx *schemas.BifrostContext, req *schemas.BifrostFileListRequest) (*schemas.BifrostFileListResponse, *schemas.BifrostError)

FileListRequest lists files from the specified provider.

func (*Bifrost) FileRetrieveRequest added in v1.2.38 

func (bifrost *Bifrost) FileRetrieveRequest(ctx *schemas.BifrostContext, req *schemas.BifrostFileRetrieveRequest) (*schemas.BifrostFileRetrieveResponse, *schemas.BifrostError)

FileRetrieveRequest retrieves file metadata from the specified provider.

func (*Bifrost) FileUploadRequest added in v1.2.38 

func (bifrost *Bifrost) FileUploadRequest(ctx *schemas.BifrostContext, req *schemas.BifrostFileUploadRequest) (*schemas.BifrostFileUploadResponse, *schemas.BifrostError)

FileUploadRequest uploads a file to the specified provider.

func (*Bifrost) GetAvailableMCPTools added in v1.3.0 

func (bifrost *Bifrost) GetAvailableMCPTools(ctx *schemas.BifrostContext) []schemas.ChatTool

GetAvailableTools returns the available tools for the given context.

Returns:

  • []schemas.ChatTool: List of available tools

func (*Bifrost) GetConfiguredProviders added in v1.2.16 

func (bifrost *Bifrost) GetConfiguredProviders() ([]schemas.ModelProvider, error)

GetConfiguredProviders returns the configured providers.

Returns:

  • []schemas.ModelProvider: List of configured providers
  • error: Any error that occurred during the retrieval process

Example: 

providers, err := bifrost.GetConfiguredProviders()
if err != nil {
	return nil, err
}
fmt.Println(providers)

func (*Bifrost) GetDropExcessRequests added in v1.1.7 

func (bifrost *Bifrost) GetDropExcessRequests() bool

GetDropExcessRequests returns the current value of DropExcessRequests

func (*Bifrost) GetMCPClients added in v1.1.7 

func (bifrost *Bifrost) GetMCPClients() ([]schemas.MCPClient, error)

GetMCPClients returns all MCP clients managed by the Bifrost instance.

Returns:

  • []schemas.MCPClient: List of all MCP clients
  • error: Any retrieval error

func (*Bifrost) GetProviderByKey added in v1.4.8 

func (bifrost *Bifrost) GetProviderByKey(providerKey schemas.ModelProvider) schemas.Provider

GetProviderByKey returns the provider instance for the given provider key. Returns nil if no provider with the given key exists.

func (*Bifrost) ImageEditRequest added in v1.4.0 

func (bifrost *Bifrost) ImageEditRequest(ctx *schemas.BifrostContext, req *schemas.BifrostImageEditRequest) (*schemas.BifrostImageGenerationResponse, *schemas.BifrostError)

ImageEditRequest sends an image edit request to the specified provider.

func (*Bifrost) ImageEditStreamRequest added in v1.4.0 

func (bifrost *Bifrost) ImageEditStreamRequest(ctx *schemas.BifrostContext, req *schemas.BifrostImageEditRequest) (chan *schemas.BifrostStreamChunk, *schemas.BifrostError)

ImageEditStreamRequest sends an image edit stream request to the specified provider.

func (*Bifrost) ImageGenerationRequest added in v1.3.9 

func (bifrost *Bifrost) ImageGenerationRequest(ctx *schemas.BifrostContext,
	req *schemas.BifrostImageGenerationRequest,
) (*schemas.BifrostImageGenerationResponse, *schemas.BifrostError)

ImageGenerationRequest sends an image generation request to the specified provider.

func (*Bifrost) ImageGenerationStreamRequest added in v1.3.9 

func (bifrost *Bifrost) ImageGenerationStreamRequest(ctx *schemas.BifrostContext,
	req *schemas.BifrostImageGenerationRequest,
) (chan *schemas.BifrostStreamChunk, *schemas.BifrostError)

ImageGenerationStreamRequest sends an image generation stream request to the specified provider.

func (*Bifrost) ImageVariationRequest added in v1.4.0 

func (bifrost *Bifrost) ImageVariationRequest(ctx *schemas.BifrostContext, req *schemas.BifrostImageVariationRequest) (*schemas.BifrostImageGenerationResponse, *schemas.BifrostError)

ImageVariationRequest sends an image variation request to the specified provider.

func (*Bifrost) ListAllModels added in v1.2.14 

func (bifrost *Bifrost) ListAllModels(ctx *schemas.BifrostContext, req *schemas.BifrostListModelsRequest) (*schemas.BifrostListModelsResponse, *schemas.BifrostError)

ListAllModels lists all models from all configured providers. It accumulates responses from all providers with a limit of 1000 per provider to get all results.

func (*Bifrost) ListModelsRequest added in v1.2.14 

func (bifrost *Bifrost) ListModelsRequest(ctx *schemas.BifrostContext, req *schemas.BifrostListModelsRequest) (*schemas.BifrostListModelsResponse, *schemas.BifrostError)

ListModelsRequest sends a list models request to the specified provider.

func (*Bifrost) OCRRequest added in v1.4.18 

func (bifrost *Bifrost) OCRRequest(ctx *schemas.BifrostContext, req *schemas.BifrostOCRRequest) (*schemas.BifrostOCRResponse, *schemas.BifrostError)

OCRRequest sends an OCR request to the specified provider.

func (*Bifrost) Passthrough added in v1.4.8 

func (bifrost *Bifrost) Passthrough(
	ctx *schemas.BifrostContext,
	provider schemas.ModelProvider,
	req *schemas.BifrostPassthroughRequest,
) (*schemas.BifrostPassthroughResponse, *schemas.BifrostError)

func (*Bifrost) PassthroughStream added in v1.4.8 

func (bifrost *Bifrost) PassthroughStream(
	ctx *schemas.BifrostContext,
	provider schemas.ModelProvider,
	req *schemas.BifrostPassthroughRequest,
) (chan *schemas.BifrostStreamChunk, *schemas.BifrostError)

func (*Bifrost) ReconnectMCPClient added in v1.1.7 

func (bifrost *Bifrost) ReconnectMCPClient(id string) error

ReconnectMCPClient attempts to reconnect an MCP client if it is disconnected.

Parameters:

  • id: ID of the client to reconnect

Returns:

  • error: Any reconnection error

func (*Bifrost) RegisterMCPTool added in v1.1.4 

func (bifrost *Bifrost) RegisterMCPTool(name, description string, handler func(args any) (string, error), toolSchema schemas.ChatTool) error

RegisterMCPTool registers a typed tool handler with the MCP integration. This allows developers to easily add custom tools that will be available to all LLM requests processed by this Bifrost instance.

Parameters:

  • name: Unique tool name
  • description: Human-readable tool description
  • handler: Function that handles tool execution
  • toolSchema: Bifrost tool schema for function calling

Returns:

  • error: Any registration error

Example: 

type EchoArgs struct {
    Message string `json:"message"`
}

err := bifrost.RegisterMCPTool("echo", "Echo a message",
    func(args EchoArgs) (string, error) {
        return args.Message, nil
    }, toolSchema)

func (*Bifrost) ReloadConfig added in v1.2.3 

func (bifrost *Bifrost) ReloadConfig(config schemas.BifrostConfig) error

ReloadConfig reloads the config from DB Currently we update account, drop excess requests, and plugin lists We will keep on adding other aspects as required

func (*Bifrost) ReloadPlugin added in v1.2.0 

func (bifrost *Bifrost) ReloadPlugin(plugin schemas.BasePlugin, pluginTypes []schemas.PluginType) error

ReloadPlugin reloads a plugin with new instance During the reload - it's stop the world phase where we take a global lock on the plugin mutex

func (*Bifrost) RemoveMCPClient added in v1.1.7 

func (bifrost *Bifrost) RemoveMCPClient(id string) error

RemoveMCPClient removes an MCP client from the Bifrost instance. This allows for dynamic MCP client management at runtime.

Parameters:

  • id: ID of the client to remove

Returns:

  • error: Any removal error

Example: 

err := bifrost.RemoveMCPClient("my-mcp-client-id")
if err != nil {
    log.Fatalf("Failed to remove MCP client: %v", err)
}

func (*Bifrost) RemovePlugin added in v1.2.0 

func (bifrost *Bifrost) RemovePlugin(name string, pluginTypes []schemas.PluginType) error

RemovePlugin removes a plugin from the server.

func (*Bifrost) RemoveProvider added in v1.3.13 

func (bifrost *Bifrost) RemoveProvider(providerKey schemas.ModelProvider) error

RemoveProvider removes a provider from the server. This method gracefully stops all workers for the provider, closes the request queue, and removes the provider from the providers slice.

Parameters:

  • providerKey: The provider to remove

Returns:

  • error: Any error that occurred during the removal process

func (*Bifrost) ReorderPlugins added in v1.4.9 

func (bifrost *Bifrost) ReorderPlugins(orderedNames []string)

ReorderPlugins reorders all plugin slices (LLM, MCP) to match the given base plugin name ordering. This should be called after SortAndRebuildPlugins on the config layer to sync the core's execution order. Plugins not in the ordering are appended at the end (defensive).

func (*Bifrost) RerankRequest added in v1.4.4 

func (bifrost *Bifrost) RerankRequest(ctx *schemas.BifrostContext, req *schemas.BifrostRerankRequest) (*schemas.BifrostRerankResponse, *schemas.BifrostError)

RerankRequest sends a rerank request to the specified provider.

func (*Bifrost) ResponsesCancelRequest added in v1.6.3 

func (bifrost *Bifrost) ResponsesCancelRequest(ctx *schemas.BifrostContext, req *schemas.BifrostResponsesCancelRequest) (*schemas.BifrostResponsesResponse, *schemas.BifrostError)

ResponsesCancelRequest cancels an in-flight stored response (OpenAI POST /v1/responses/{id}/cancel).

func (*Bifrost) ResponsesDeleteRequest added in v1.6.3 

func (bifrost *Bifrost) ResponsesDeleteRequest(ctx *schemas.BifrostContext, req *schemas.BifrostResponsesDeleteRequest) (*schemas.BifrostResponsesDeleteResponse, *schemas.BifrostError)

ResponsesDeleteRequest deletes a stored response (OpenAI DELETE /v1/responses/{id}).

func (*Bifrost) ResponsesInputItemsRequest added in v1.6.3 

func (bifrost *Bifrost) ResponsesInputItemsRequest(ctx *schemas.BifrostContext, req *schemas.BifrostResponsesInputItemsRequest) (*schemas.BifrostResponsesInputItemsResponse, *schemas.BifrostError)

ResponsesInputItemsRequest lists input items for a stored response (OpenAI GET /v1/responses/{id}/input_items).

func (*Bifrost) ResponsesRequest added in v1.2.0 

func (bifrost *Bifrost) ResponsesRequest(ctx *schemas.BifrostContext, req *schemas.BifrostResponsesRequest) (*schemas.BifrostResponsesResponse, *schemas.BifrostError)

ResponsesRequest sends a responses request to the specified provider.

func (*Bifrost) ResponsesRetrieveRequest added in v1.6.3 

func (bifrost *Bifrost) ResponsesRetrieveRequest(ctx *schemas.BifrostContext, req *schemas.BifrostResponsesRetrieveRequest) (*schemas.BifrostResponsesResponse, *schemas.BifrostError)

ResponsesRetrieveRequest retrieves a stored response by ID (OpenAI GET /v1/responses/{id}).

func (*Bifrost) ResponsesStreamRequest added in v1.2.0 

func (bifrost *Bifrost) ResponsesStreamRequest(ctx *schemas.BifrostContext, req *schemas.BifrostResponsesRequest) (chan *schemas.BifrostStreamChunk, *schemas.BifrostError)

ResponsesStreamRequest sends a responses stream request to the specified provider.

func (*Bifrost) RunPreRequestHooks added in v1.5.19 

func (bifrost *Bifrost) RunPreRequestHooks(ctx *schemas.BifrostContext, req *schemas.BifrostRequest)

RunPreRequestHooks acquires a plugin pipeline and runs PreRequestHook on each LLM plugin for callers that do not flow through handleRequest/handleStreamRequest — primarily realtime WebSocket upgrades, where the upgrade itself is the routing decision (once per WS connection) but the per-turn pipeline handles PreLLMHook/PostLLMHook separately.

Mutations to req.Provider/req.Model/req.Fallbacks made by PreRequestHook plugins are committed to the shared *BifrostRequest. Plugin errors are non-blocking — they are logged as warnings and the pipeline continues to the next plugin (same semantics as RunLLMPreHooks). Callers should validate req.Provider after this returns if a provider is required.

func (*Bifrost) RunRealtimeTurnPreHooks added in v1.5.1 

func (bifrost *Bifrost) RunRealtimeTurnPreHooks(ctx *schemas.BifrostContext, req *schemas.BifrostRequest) (*RealtimeTurnHooks, *schemas.BifrostError)

RunRealtimeTurnPreHooks acquires a plugin pipeline and runs LLM pre-hooks for a single realtime turn. Unlike generic stream hooks, realtime turns do not support short-circuit responses in v1 because the transports cannot yet emit a fully synthetic assistant turn without an upstream generation.

func (*Bifrost) RunStreamPreHooks added in v1.4.8 

func (bifrost *Bifrost) RunStreamPreHooks(ctx *schemas.BifrostContext, req *schemas.BifrostRequest) (*WSStreamHooks, *schemas.BifrostError)

RunStreamPreHooks acquires a plugin pipeline, sets up tracing context, runs PreLLMHooks, and returns a PostHookRunner for per-chunk post-processing. Used by WebSocket handlers that bypass the normal inference path but still need plugin hooks.

func (*Bifrost) SelectKeyForProviderRequestType added in v1.5.1 

func (bifrost *Bifrost) SelectKeyForProviderRequestType(ctx *schemas.BifrostContext, requestType schemas.RequestType, providerKey schemas.ModelProvider, model string) (schemas.Key, error)

SelectKeyForProviderRequestType selects an API key for the given provider, request type, and model. Used by WebSocket handlers that need a key for upstream connections while honoring request-specific AllowedRequests gates such as realtime-only support.

func (*Bifrost) SetClientTools added in v1.5.1 

func (bifrost *Bifrost) SetClientTools(clientID string, tools map[string]schemas.ChatTool, toolNameMapping map[string]string)

SetClientTools delegates to the MCP manager to update the tool map for an existing MCP client.

func (*Bifrost) SetMCPManager added in v1.4.0 

func (bifrost *Bifrost) SetMCPManager(manager mcp.MCPManagerInterface)

SetMCPManager sets the MCP manager for this Bifrost instance. This allows injecting a custom MCP manager implementation. If the provided manager is a concrete *mcp.MCPManager, Bifrost's plugin pipeline is injected into the manager's CodeMode so that nested tool calls run through the plugin hooks.

Parameters:

  • manager: The MCP manager to set (must implement MCPManagerInterface)

func (*Bifrost) SetTracer added in v1.3.0 

func (bifrost *Bifrost) SetTracer(tracer schemas.Tracer)

SetTracer sets the tracer for the Bifrost instance.

func (*Bifrost) Shutdown 

func (bifrost *Bifrost) Shutdown()

Shutdown gracefully stops all workers when triggered. It closes all request channels and waits for workers to exit.

func (*Bifrost) SpeechRequest added in v1.1.11 

func (bifrost *Bifrost) SpeechRequest(ctx *schemas.BifrostContext, req *schemas.BifrostSpeechRequest) (*schemas.BifrostSpeechResponse, *schemas.BifrostError)

SpeechRequest sends a speech request to the specified provider.

func (*Bifrost) SpeechStreamRequest added in v1.1.11 

func (bifrost *Bifrost) SpeechStreamRequest(ctx *schemas.BifrostContext, req *schemas.BifrostSpeechRequest) (chan *schemas.BifrostStreamChunk, *schemas.BifrostError)

SpeechStreamRequest sends a speech stream request to the specified provider.

func (*Bifrost) TextCompletionRequest 

func (bifrost *Bifrost) TextCompletionRequest(ctx *schemas.BifrostContext, req *schemas.BifrostTextCompletionRequest) (*schemas.BifrostTextCompletionResponse, *schemas.BifrostError)

TextCompletionRequest sends a text completion request to the specified provider.

func (*Bifrost) TextCompletionStreamRequest added in v1.2.1 

func (bifrost *Bifrost) TextCompletionStreamRequest(ctx *schemas.BifrostContext, req *schemas.BifrostTextCompletionRequest) (chan *schemas.BifrostStreamChunk, *schemas.BifrostError)

TextCompletionStreamRequest sends a streaming text completion request to the specified provider.

func (*Bifrost) TranscriptionRequest added in v1.1.11 

func (bifrost *Bifrost) TranscriptionRequest(ctx *schemas.BifrostContext, req *schemas.BifrostTranscriptionRequest) (*schemas.BifrostTranscriptionResponse, *schemas.BifrostError)

TranscriptionRequest sends a transcription request to the specified provider.

func (*Bifrost) TranscriptionStreamRequest added in v1.1.11 

func (bifrost *Bifrost) TranscriptionStreamRequest(ctx *schemas.BifrostContext, req *schemas.BifrostTranscriptionRequest) (chan *schemas.BifrostStreamChunk, *schemas.BifrostError)

TranscriptionStreamRequest sends a transcription stream request to the specified provider.

func (*Bifrost) UpdateDropExcessRequests added in v1.1.7 

func (bifrost *Bifrost) UpdateDropExcessRequests(value bool)

UpdateDropExcessRequests updates the DropExcessRequests setting at runtime. This allows for hot-reloading of this configuration value.

func (*Bifrost) UpdateMCPClient added in v1.4.0 

func (bifrost *Bifrost) UpdateMCPClient(id string, updatedConfig *schemas.MCPClientConfig) error

UpdateMCPClient updates the MCP client. This allows for dynamic MCP client tool management at runtime.

Parameters:

  • id: ID of the client to edit
  • updatedConfig: Updated MCP client configuration

Returns:

  • error: Any edit error

Example: 

err := bifrost.UpdateMCPClient("my-mcp-client-id", schemas.MCPClientConfig{
    Name:           "my-mcp-client-name",
    ToolsToExecute: []string{"tool1", "tool2"},
})

func (*Bifrost) UpdateMCPClientConnection added in v1.5.8 

func (bifrost *Bifrost) UpdateMCPClientConnection(id string, newConfig *schemas.MCPClientConfig) error

UpdateMCPClientConnection reconnects an existing MCP client using updated headers

func (*Bifrost) UpdateProvider added in v1.2.16 

func (bifrost *Bifrost) UpdateProvider(providerKey schemas.ModelProvider) error

UpdateProvider dynamically updates a provider with new configuration. This method gracefully recreates the provider instance with updated settings, stops existing workers, creates a new queue with updated settings, and starts new workers with the updated provider and concurrency configuration.

Parameters:

  • providerKey: The provider to update

Returns:

  • error: Any error that occurred during the update process

Note: This operation will temporarily pause request processing for the specified provider while the transition occurs. In-flight requests will complete before workers are stopped. Buffered requests in the old queue will be transferred to the new queue to prevent loss.

Concurrency safety — update handoff: UpdateProvider holds a per-provider write lock only while publishing the new provider instance, queue, and workers. It starts the new workers before the old queue is signalled closed, then releases the lock before old workers are waited on. This avoids high-load updates blocking new requests behind the provider read lock while slow in-flight old-worker requests finish.

func (*Bifrost) UpdateToolManagerConfig added in v1.3.0 

func (bifrost *Bifrost) UpdateToolManagerConfig(maxAgentDepth int, toolExecutionTimeoutInSeconds int, codeModeBindingLevel string, disableAutoToolInject bool) error

UpdateToolManagerConfig updates the tool manager config for the MCP manager. This allows for hot-reloading of the tool manager config at runtime. Pass the current value of disableAutoToolInject whenever only other fields change so the flag is never silently reset to its zero value.

func (*Bifrost) VerifyHeadersConnection added in v1.5.14 

func (bifrost *Bifrost) VerifyHeadersConnection(ctx context.Context, config *schemas.MCPClientConfig, userHeaders map[string]string) (map[string]schemas.ChatTool, map[string]string, error)

VerifyHeadersConnection delegates to the MCP manager to verify an MCP server using caller-supplied header values (admin sample or user-submitted) and discover available tools. Mirrors VerifyPerUserOAuthConnection's lazy MCP-manager init.

func (*Bifrost) VerifyPerUserOAuthConnection added in v1.5.1 

func (bifrost *Bifrost) VerifyPerUserOAuthConnection(ctx context.Context, config *schemas.MCPClientConfig, accessToken string) (map[string]schemas.ChatTool, map[string]string, error)

VerifyPerUserOAuthConnection delegates to the MCP manager to verify an MCP server using a temporary access token and discover available tools. The connection is closed after verification. If the MCP manager is not yet initialized, it is lazily created (same as AddMCPClient).

func (*Bifrost) VideoDeleteRequest added in v1.4.4 

func (bifrost *Bifrost) VideoDeleteRequest(ctx *schemas.BifrostContext, req *schemas.BifrostVideoDeleteRequest) (*schemas.BifrostVideoDeleteResponse, *schemas.BifrostError)

func (*Bifrost) VideoDownloadRequest added in v1.4.4 

func (bifrost *Bifrost) VideoDownloadRequest(ctx *schemas.BifrostContext, req *schemas.BifrostVideoDownloadRequest) (*schemas.BifrostVideoDownloadResponse, *schemas.BifrostError)

VideoDownloadRequest downloads video content from the provider.

func (*Bifrost) VideoGenerationRequest added in v1.4.4 

func (bifrost *Bifrost) VideoGenerationRequest(ctx *schemas.BifrostContext,
	req *schemas.BifrostVideoGenerationRequest,
) (*schemas.BifrostVideoGenerationResponse, *schemas.BifrostError)

VideoGenerationRequest sends a video generation request to the specified provider.

func (*Bifrost) VideoListRequest added in v1.4.4 

func (bifrost *Bifrost) VideoListRequest(ctx *schemas.BifrostContext, req *schemas.BifrostVideoListRequest) (*schemas.BifrostVideoListResponse, *schemas.BifrostError)

func (*Bifrost) VideoRemixRequest added in v1.4.4 

func (bifrost *Bifrost) VideoRemixRequest(ctx *schemas.BifrostContext, req *schemas.BifrostVideoRemixRequest) (*schemas.BifrostVideoGenerationResponse, *schemas.BifrostError)

func (*Bifrost) VideoRetrieveRequest added in v1.4.4 

func (bifrost *Bifrost) VideoRetrieveRequest(ctx *schemas.BifrostContext, req *schemas.BifrostVideoRetrieveRequest) (*schemas.BifrostVideoGenerationResponse, *schemas.BifrostError)

type ChannelMessage 

type ChannelMessage struct {
	schemas.BifrostRequest
	Context        *schemas.BifrostContext
	Response       chan *schemas.BifrostResponse
	ResponseStream chan chan *schemas.BifrostStreamChunk
	Err            chan schemas.BifrostError
}

ChannelMessage represents a message passed through the request channel. It contains the request, response and error channels, and the request type.

type DefaultLogger 

type DefaultLogger struct {
	// contains filtered or unexported fields
}

DefaultLogger implements the Logger interface with stdout/stderr printing. It provides a simple logging implementation that writes to standard output and error streams with formatted timestamps and log levels. It is used as the default logger if no logger is provided in the BifrostConfig.

func NewDefaultLogger 

func NewDefaultLogger(level schemas.LogLevel) *DefaultLogger

NewDefaultLogger creates a new DefaultLogger instance with the specified log level. The log level determines which messages will be output based on their severity.

func (*DefaultLogger) Debug 

func (logger *DefaultLogger) Debug(msg string, args ...any)

Debug logs a debug level message to stdout. Messages are only output if the logger's level is set to LogLevelDebug.

func (*DefaultLogger) Error 

func (logger *DefaultLogger) Error(msg string, args ...any)

Error logs an error level message to stderr. Error messages are always output regardless of the logger's level.

func (*DefaultLogger) Fatal added in v1.1.16 

func (logger *DefaultLogger) Fatal(msg string, args ...any)

Fatal logs a fatal-level message to stderr. Fatal messages are always output regardless of the logger's level.

func (*DefaultLogger) Info 

func (logger *DefaultLogger) Info(msg string, args ...any)

Info logs an info level message to stdout. Messages are output if the logger's level is LogLevelDebug or LogLevelInfo.

func (*DefaultLogger) LogHTTPRequest added in v1.4.1 

func (logger *DefaultLogger) LogHTTPRequest(level schemas.LogLevel, msg string) schemas.LogEventBuilder

LogHTTPRequest returns a LogEventBuilder for structured HTTP access logging. We are exposing the zerolog loggers directly to allow for more flexibility in logging and also to reduce the number of allocations we do in the logger.

func (*DefaultLogger) SetLevel 

func (logger *DefaultLogger) SetLevel(level schemas.LogLevel)

SetLevel sets the logging level for the logger. This determines which messages will be output based on their severity.

func (*DefaultLogger) SetOutputType added in v1.1.16 

func (logger *DefaultLogger) SetOutputType(outputType schemas.LoggerOutputType)

SetOutputType sets the output type for the logger. This determines the format of the log output. If the output type is unknown, it defaults to JSON

func (*DefaultLogger) Warn 

func (logger *DefaultLogger) Warn(msg string, args ...any)

Warn logs a warning level message to stdout. Messages are output if the logger's level is LogLevelDebug, LogLevelInfo, or LogLevelWarn.

type NoOpLogger added in v1.4.1 

type NoOpLogger struct{}

NoOpLogger is a no-op implementation of schemas.Logger.

func (*NoOpLogger) Debug added in v1.4.1 

func (l *NoOpLogger) Debug(string, ...any)

func (*NoOpLogger) Error added in v1.4.1 

func (l *NoOpLogger) Error(string, ...any)

func (*NoOpLogger) Fatal added in v1.4.1 

func (l *NoOpLogger) Fatal(string, ...any)

func (*NoOpLogger) Info added in v1.4.1 

func (l *NoOpLogger) Info(string, ...any)

func (*NoOpLogger) LogHTTPRequest added in v1.4.1 

func (l *NoOpLogger) LogHTTPRequest(schemas.LogLevel, string) schemas.LogEventBuilder

func (*NoOpLogger) SetLevel added in v1.4.1 

func (l *NoOpLogger) SetLevel(schemas.LogLevel)

func (*NoOpLogger) SetOutputType added in v1.4.1 

func (l *NoOpLogger) SetOutputType(schemas.LoggerOutputType)

func (*NoOpLogger) Warn added in v1.4.1 

func (l *NoOpLogger) Warn(string, ...any)

type PluginPipeline added in v1.1.2 

type PluginPipeline struct {
	// contains filtered or unexported fields
}

PluginPipeline encapsulates the execution of plugin PreHooks and PostHooks, tracks how many plugins ran, and manages short-circuiting and error aggregation.

func (*PluginPipeline) FinalizeStreamingPostHookSpans added in v1.3.0 

func (p *PluginPipeline) FinalizeStreamingPostHookSpans(ctx context.Context)

FinalizeStreamingPostHookSpans creates aggregated spans for each plugin after streaming completes. This should be called once at the end of streaming to create one span per plugin with average timing. Spans are nested to mirror the pre-hook hierarchy (each post-hook is a child of the previous one).

func (*PluginPipeline) GetChunkCount added in v1.3.0 

func (p *PluginPipeline) GetChunkCount() int

GetChunkCount returns the number of chunks processed during streaming

func (*PluginPipeline) RunLLMPreHooks added in v1.4.0 

func (p *PluginPipeline) RunLLMPreHooks(ctx *schemas.BifrostContext, req *schemas.BifrostRequest) (*schemas.BifrostRequest, *schemas.LLMPluginShortCircuit, int)

RunLLMPreHooks executes PreHooks in order, tracks how many ran, and returns the final request, any short-circuit decision, and the count.

func (*PluginPipeline) RunMCPPostConnectionHooks added in v1.5.10 

func (p *PluginPipeline) RunMCPPostConnectionHooks(ctx *schemas.BifrostContext, resp *schemas.BifrostMCPConnectResponse, bifrostErr *schemas.BifrostError, runFrom int) (*schemas.BifrostMCPConnectResponse, *schemas.BifrostError)

RunMCPPostConnectionHooks executes typed Connect PostHooks in reverse order for the plugins whose PreMCPConnectionHook ran. Plugins that only implement MCPPlugin are skipped (they didn't run in PreHook, they don't run in PostHook).

func (*PluginPipeline) RunMCPPostHooks added in v1.4.0 

func (p *PluginPipeline) RunMCPPostHooks(ctx *schemas.BifrostContext, mcpResp *schemas.BifrostMCPResponse, bifrostErr *schemas.BifrostError, runFrom int) (*schemas.BifrostMCPResponse, *schemas.BifrostError)

RunMCPPostHooks executes MCP PostHooks in reverse order for the envelope-based pipeline (Ping / ListTools / ExecuteTool variants). Connect responses do NOT flow through here — they use RunMCPPostConnectionHooks.

func (*PluginPipeline) RunMCPPreConnectionHooks added in v1.5.10 

func (p *PluginPipeline) RunMCPPreConnectionHooks(ctx *schemas.BifrostContext, req *schemas.BifrostMCPConnectRequest) (*schemas.BifrostMCPConnectRequest, *schemas.MCPConnectionShortCircuit, int)

RunMCPPreConnectionHooks executes typed Connect PreHooks in order for plugins implementing MCPConnectionPlugin. Plugins that only implement MCPPlugin (no typed Connect methods) are silently skipped — they cannot observe or intercept the connection lifecycle.

Returns the (possibly mutated) typed sub-request, any short-circuit decision, and the count of hooks that executed (for matching PostHook dispatch).

func (*PluginPipeline) RunMCPPreHooks added in v1.4.0 

func (p *PluginPipeline) RunMCPPreHooks(ctx *schemas.BifrostContext, req *schemas.BifrostMCPRequest) (*schemas.BifrostMCPRequest, *schemas.MCPPluginShortCircuit, int)

RunMCPPreHooks executes MCP PreHooks in order for all registered MCP plugins. Handles the envelope-based MCP pipeline (Ping / ListTools / ExecuteTool variants). Connect requests do NOT flow through here — they use RunMCPPreConnectionHooks with typed signatures.

func (*PluginPipeline) RunPostLLMHooks added in v1.4.3 

func (p *PluginPipeline) RunPostLLMHooks(ctx *schemas.BifrostContext, resp *schemas.BifrostResponse, bifrostErr *schemas.BifrostError, runFrom int) (*schemas.BifrostResponse, *schemas.BifrostError)

RunPostLLMHooks executes PostHooks in reverse order for the plugins whose PreLLMHook ran. Accepts the response and error, and allows plugins to transform either (e.g., recover from error, or invalidate a response). Returns the final response and error after all hooks. If both are set, error takes precedence unless error is nil. runFrom is the count of plugins whose PreHooks ran; PostHooks will run in reverse from index (runFrom - 1) down to 0 For streaming requests, it accumulates timing per plugin instead of creating individual spans per chunk.

func (*PluginPipeline) RunPreRequestHooks added in v1.5.19 

func (p *PluginPipeline) RunPreRequestHooks(ctx *schemas.BifrostContext, req *schemas.BifrostRequest)

RunPreRequestHooks executes PreRequestHook on each LLM plugin in registration order, once per top-level request. Plugins mutate req.Provider, req.Model, req.Fallbacks (and any other field they choose); mutations are committed to the shared *BifrostRequest and observed by every subsequent plugin, the provider call, and every fallback attempt. There is no short-circuit and errors are non-blocking — same semantics as RunLLMPreHooks: errors are logged as warnings and accumulated in p.preHookErrors, then the pipeline continues to the next plugin. The empty- provider validation in handleRequest/handleStreamRequest catches the case where no plugin successfully resolved a provider.

Per-request semantics: unlike PreLLMHook (which runs again on every fallback), PreRequestHook runs exactly once at the top of handleRequest/handleStreamRequest, before any fan-out.

type ProviderQueue added in v1.3.13 

type ProviderQueue struct {
	// contains filtered or unexported fields
}

ProviderQueue wraps a provider's request channel with lifecycle management to prevent "send on closed channel" panics during provider removal/update. Producers must check the closing flag or select on the done channel before sending.

Why pq.queue is NEVER closed:

Closing a channel in Go causes any concurrent send to that channel to panic ("send on closed channel"). There is always a TOCTOU window between a producer's isClosing() check and its select { case pq.queue <- msg: ... }: the producer could pass isClosing() while the queue is open, get preempted, and resume only after the queue is closed. Go's selectgo evaluates select cases in a random order, so even having case <-pq.done: in the same select does not protect against this — if selectgo evaluates the send case first on a closed channel it panics immediately via goto sclose, before reaching done.

To close pq.queue safely you would need a sender-side WaitGroup so that signalClosing could wait for every in-flight producer to finish. That adds non-trivial overhead on the hot request path.

Instead, pq.done is the sole shutdown signal. Receiving from a closed channel is always safe (returns the zero value immediately), so:

  • Workers exit via case <-pq.done: — safe
  • Producers bail via case <-pq.done: — safe
  • drainQueueWithErrors handles any messages that slip through the TOCTOU window

pq.queue is garbage collected automatically:

  • RemoveProvider calls requestQueues.Delete, dropping the map's reference.
  • UpdateProvider calls requestQueues.Store with a new queue, dropping the map's reference to oldPq. Shutdown does not Delete at all — the whole Bifrost instance is torn down. In all cases, once no producer goroutine holds a reference to the ProviderQueue, both the struct and pq.queue are eligible for GC. No explicit close is needed.

type RealtimeTurnHooks added in v1.5.1 

type RealtimeTurnHooks struct {
	PostHookRunner schemas.PostHookRunner
	Cleanup        func()
}

RealtimeTurnHooks mirrors RunStreamPreHooks but is explicitly scoped to a single realtime turn rather than one long-lived transport connection.

type WSStreamHooks added in v1.4.8 

type WSStreamHooks struct {
	PostHookRunner       schemas.PostHookRunner
	Cleanup              func()
	ShortCircuitResponse *schemas.BifrostResponse
}

WSStreamHooks holds the post-hook runner and cleanup function returned by RunStreamPreHooks. Call PostHookRunner for each streaming chunk, setting StreamEndIndicator on the final chunk. Call Cleanup when done to release the pipeline back to the pool. If ShortCircuitResponse is non-nil, a plugin short-circuited with a cached response — the caller should write this response to the client and skip the upstream call.

Source Files

View all Source files

Directories

Path Synopsis
internal
llmtests
Package llmtests provides comprehensive test account and configuration management for the Bifrost system.
 Package llmtests provides comprehensive test account and configuration management for the Bifrost system.
mcptests
mcp
codemode/starlark
Package starlark provides a Starlark-based implementation of the CodeMode interface.
 Package starlark provides a Starlark-based implementation of the CodeMode interface.
credstore
Package credstore implements schemas.CredentialStore: it routes credential resolution for MCP tool execution by auth type.
 Package credstore implements schemas.CredentialStore: it routes credential resolution for MCP tool execution by auth type.
utils
Package network provides centralized HTTP client management with proxy support.
 Package network provides centralized HTTP client management with proxy support.
providers
anthropic
Package anthropic implements the Anthropic provider for the Bifrost API.
 Package anthropic implements the Anthropic provider for the Bifrost API.
azure
Package azure implements the Azure provider.
 Package azure implements the Azure provider.
bedrock
bedrockmantle
Package bedrockmantle implements the Bedrock Mantle LLM provider.
 Package bedrockmantle implements the Bedrock Mantle LLM provider.
cerebras
Package cerebras implements the Cerebras LLM provider.
 Package cerebras implements the Cerebras LLM provider.
cohere
deepseek
Package deepseek implements the DeepSeek LLM provider.
 Package deepseek implements the DeepSeek LLM provider.
elevenlabs
fireworks
Package fireworks implements the Fireworks AI provider and its utility functions.
 Package fireworks implements the Fireworks AI provider and its utility functions.
gemini
Package gemini provides types and structures for interacting with Google's Gemini API.
 Package gemini provides types and structures for interacting with Google's Gemini API.
groq
Package groq implements the Groq provider and its utility functions.
 Package groq implements the Groq provider and its utility functions.
huggingface
Package huggingface provides a HuggingFace chat provider.
 Package huggingface provides a HuggingFace chat provider.
mistral
Package mistral implements the Mistral provider.
 Package mistral implements the Mistral provider.
nebius
Package nebius implements the Nebius LLM provider.
 Package nebius implements the Nebius LLM provider.
ollama
Package providers implements various LLM providers and their utility functions.
 Package providers implements various LLM providers and their utility functions.
openai
Package openai provides the OpenAI provider implementation for the Bifrost framework.
 Package openai provides the OpenAI provider implementation for the Bifrost framework.
opencode
Package opencode implements the Opencode Zen and Go AI gateway providers.
 Package opencode implements the Opencode Zen and Go AI gateway providers.
openrouter
Package openrouter implements the OpenRouter LLM provider.
 Package openrouter implements the OpenRouter LLM provider.
parasail
Package providers implements various LLM providers and their utility functions.
 Package providers implements various LLM providers and their utility functions.
perplexity
Package providers implements various LLM providers and their utility functions.
 Package providers implements various LLM providers and their utility functions.
replicate
Package providers implements various LLM providers and their utility functions.
 Package providers implements various LLM providers and their utility functions.
runware
Package runware implements the Runware provider for Bifrost.
 Package runware implements the Runware provider for Bifrost.
runway
Package providers implements various LLM providers and their utility functions.
 Package providers implements various LLM providers and their utility functions.
sarvam
Package sarvam implements the Sarvam AI provider.
 Package sarvam implements the Sarvam AI provider.
sgl
Package providers implements various LLM providers and their utility functions.
 Package providers implements various LLM providers and their utility functions.
utils
Package utils provides common utility functions used across different provider implementations.
 Package utils provides common utility functions used across different provider implementations.
vertex
vllm
Package vllm implements the vLLM LLM provider (OpenAI-compatible).
 Package vllm implements the vLLM LLM provider (OpenAI-compatible).
wafer
Package wafer implements the Wafer LLM provider.
 Package wafer implements the Wafer LLM provider.
xai
Package providers implements various LLM providers and their utility functions.
 Package providers implements various LLM providers and their utility functions.
Package schemas defines the core schemas and types used by the Bifrost system.
 Package schemas defines the core schemas and types used by the Bifrost system.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.