Documentation
¶
Overview ¶
Package bifrost embeds the Bifrost AI gateway (github.com/maximhq/bifrost) inside the Obot process and exposes it to the rest of Obot as an http.Handler.
Bifrost runs its full HTTP transport (inference routes, governance, logging, telemetry, routing and caching plugins) in-process. It never listens on a network socket: it is served over an in-memory listener, so the only way to reach it is through Obot, which authenticates callers and applies its own policies before forwarding requests.
Index ¶
- Constants
- func ApplyProviderDefaults(provider map[string]any)
- func CostFromMetadata(metadata map[string]string) (types.ModelCost, bool)
- func DerivePostgresDSN(ctx context.Context, obotDSN string) (string, error)
- func DialectForModel(provider, modelID string) llmtypes.Dialect
- func FillDefaultModelAliases(ctx context.Context, c kclient.Client) error
- func HasSecretValue(raw []byte) bool
- func IsMaskedSecret(raw []byte) bool
- func IsSecretReference(raw []byte) bool
- func IsStoreBusy(statusCode int, body []byte) bool
- func LegacyImportDone(mp *v1.ModelProvider) bool
- func MigrateLegacyModels(ctx context.Context, c kclient.Client, importDone bool) error
- func ModelProviderResource() *v1.ModelProvider
- func PreferredModels(alias types.DefaultModelAliasType) []string
- func ScrubEngineName(message string) string
- func ScrubEngineNameBytes(body []byte) []byte
- func SecretValue(raw []byte) string
- func TriggerModelSync(ctx context.Context, c kclient.Client) error
- func TrimProviderBaseURLVersion(provider map[string]any, providerType string) bool
- type APIError
- type Engine
- func (e *Engine) AnthropicIsDirect(ctx context.Context) bool
- func (e *Engine) ApplyEgressProxy(ctx context.Context) (changed bool)
- func (e *Engine) BaseURL() url.URL
- func (e *Engine) Change(ctx context.Context, change func() (changed, applied bool), ...) error
- func (e *Engine) ClaudeSubscriptionModels(ctx context.Context) []ListedModel
- func (e *Engine) Client() *bifrostcore.Bifrost
- func (e *Engine) ConfiguredProviders(ctx context.Context) ([]string, error)
- func (e *Engine) Do(ctx context.Context, method, path string, body any, out any) error
- func (e *Engine) Err() error
- func (e *Engine) Fail(err error)
- func (e *Engine) GuardProviderChange(ctx context.Context, method, path string, body []byte, removeKeys bool) (ProviderChange, error)
- func (e *Engine) HasUpstreamProviders(ctx context.Context) bool
- func (engine *Engine) ImportLegacyModelProviders(ctx context.Context, _ kclient.Client, gatewayClient *gateway.Client) ([]LegacyImportResult, error)
- func (e *Engine) LLMPluginOrder() []string
- func (e *Engine) ListModels(ctx context.Context) (*openai.ModelsList, error)
- func (e *Engine) MaxRequestBodyBytes() int64
- func (e *Engine) Ready() bool
- func (e *Engine) Reload(revision int64) (bool, error)
- func (e *Engine) RunStartupTasks(ctx context.Context, c kclient.Client, gatewayClient *gateway.Client, ...)
- func (e *Engine) ServeHTTP(w http.ResponseWriter, r *http.Request)
- func (e *Engine) Shutdown()
- func (e *Engine) Start(ctx context.Context, opts Options) error
- func (e *Engine) Transport() http.RoundTripper
- func (e *Engine) Wait(ctx context.Context) error
- type LegacyImportResult
- type ListedModel
- type Options
- type ProviderChange
- type SharedKeyStore
Constants ¶
const ( MetadataProvider = "bifrostProvider" MetadataInputCost = "bifrostInputCostPerToken" MetadataOutputCost = "bifrostOutputCostPerToken" MetadataCacheReadCost = "bifrostCacheReadCostPerToken" MetadataCacheWriteCost = "bifrostCacheWriteCostPerToken" MetadataContextLength = "bifrostContextLength" MetadataMaxOutputTokens = "bifrostMaxOutputTokens" )
Metadata keys set on listed models. The provider back-population controller reads "displayName", "usage" and "dialect"; the remaining keys carry Bifrost pricing so the caller can derive ModelInfo costs.
const (
// Version is reported to Bifrost as its runtime version.
Version = "obot-embedded"
)
Variables ¶
This section is empty.
Functions ¶
func ApplyProviderDefaults ¶
ApplyProviderDefaults fills in the network settings a new provider leaves unset: retries with backoff, which also let requests fail over between the provider's keys, and the request timeout.
func CostFromMetadata ¶
CostFromMetadata converts Bifrost per-token pricing into Obot's per-million-token cost.
func DerivePostgresDSN ¶
DerivePostgresDSN returns a DSN for a database named after Obot's own, on the same Postgres server, creating the database if it doesn't exist. Bifrost's tables can't share Obot's database, but sharing the server keeps every Obot replica on the same gateway state without extra configuration. Connection failures are retried for up to a minute; a database that can't be created fails at once, with an error that says how to create it.
func DialectForModel ¶
DialectForModel picks the API dialect Obot's agents should use for a model. Bifrost translates every dialect to every provider, so this only selects the most faithful one: Anthropic models speak Messages natively, everything else speaks Responses.
func FillDefaultModelAliases ¶
FillDefaultModelAliases sets unset default model aliases to the most preferred gateway model that is available, so a new installation works as soon as a provider is added.
func HasSecretValue ¶
HasSecretValue reports whether a JSON value the engine accepts as an optional secret, such as a CA certificate, holds a value. JSON null leaves the setting unset.
func IsMaskedSecret ¶
IsMaskedSecret reports whether a JSON value is a secret as the engine lists it, redacted, which the engine takes to mean "keep the stored value" when it is sent back in an update.
func IsSecretReference ¶
IsSecretReference reports whether the engine would read a JSON value from the host's environment or a vault, rather than take it as given, wherever it accepts a secret. It decodes the value with the engine's own decoder, so it can't disagree with the engine on spelling, such as key names in another case.
func IsStoreBusy ¶
IsStoreBusy reports whether a response from the engine's API failed because its SQLite store was locked by another write. The write was not made, and can be sent again.
func LegacyImportDone ¶
func LegacyImportDone(mp *v1.ModelProvider) bool
LegacyImportDone reports whether the credentials of legacy model providers have been imported into the gateway, whose ModelProvider is mp.
func MigrateLegacyModels ¶
MigrateLegacyModels moves what refers to legacy model providers' models over to the gateway's models: default model aliases and model access policies that name a legacy model are re-pointed to the gateway model with the same upstream model, and edits made to the legacy model (deactivating it, an alias, a cost override) carry over. It runs after every gateway model sync, so references move as soon as the gateway lists the models.
Once legacy provider credentials have been imported (importDone), legacy models that were migrated or that nothing refers to are deleted, and then legacy model providers left without models. A legacy model that is still referred to but that the gateway doesn't serve is kept, so the reference doesn't break, until the gateway lists its model.
func ModelProviderResource ¶
func ModelProviderResource() *v1.ModelProvider
ModelProviderResource is the ModelProvider resource that represents the embedded gateway. Unlike registry providers it has no daemon and no credential: providers and keys are configured in Bifrost itself.
func PreferredModels ¶
func PreferredModels(alias types.DefaultModelAliasType) []string
PreferredModels returns the gateway models (provider/model) preferred for a default model alias, most preferred first.
func ScrubEngineName ¶
ScrubEngineName replaces the engine's name in an error message with the product's own, so errors read as coming from Obot's LLM gateway.
func ScrubEngineNameBytes ¶
ScrubEngineNameBytes is ScrubEngineName for a response body.
func SecretValue ¶
SecretValue returns the value the engine reads from a JSON value it accepts as a secret, such as a region, which may be a string or an object holding one.
func TriggerModelSync ¶
TriggerModelSync flips the sync annotation on the gateway's ModelProvider so the provider controller re-lists models. The provider controller updates the same resource while it syncs, so a conflicting write is retried against the latest version.
func TrimProviderBaseURLVersion ¶
TrimProviderBaseURLVersion removes a trailing "/v1" from the base URL of a provider of type providerType (the provider's own name for a built-in provider, or the type a custom provider is built on) when the engine adds the version to every path itself. OpenAI-compatible servers usually document their base URL with "/v1", which the engine would otherwise turn into "/v1/v1/chat/completions". It reports whether it changed the base URL.
Types ¶
type Engine ¶
type Engine struct {
// contains filtered or unexported fields
}
Engine is a running, embedded Bifrost gateway.
func (*Engine) AnthropicIsDirect ¶
AnthropicIsDirect reports whether the engine's Anthropic provider sends requests straight to Anthropic over verified TLS: no other base URL, and no setting that would let another server present a certificate for Anthropic's host, whether in the provider's network or proxy settings or in the gateway's global proxy. A caller's Claude subscription token is only forwarded when it does, so an administrator can't redirect users' tokens to a server of their own. A proxy without such settings, such as the environment's egress proxy, only relays the TLS connection to Anthropic, so it doesn't stop the token being forwarded.
func (*Engine) ApplyEgressProxy ¶
ApplyEgressProxy routes providers through the HTTP(S) proxy configured in Obot's environment. Bifrost's HTTP client does not read HTTPS_PROXY/NO_PROXY itself, so without this an Obot deployed behind an egress proxy could not reach any provider. Providers that already have a proxy configured, and those whose endpoint NO_PROXY excludes (such as in-cluster vLLM or Ollama servers), are left alone.
func (*Engine) Change ¶
func (e *Engine) Change(ctx context.Context, change func() (changed, applied bool), announce func(context.Context) (int64, error)) error
Change makes a configuration change through this engine and tells the other replicas. change applies it and reports whether anything changed, and whether this engine is sure to have the change in memory (false when, say, it failed partway); announce then advances the shared configuration revision, and returns it. Reloads wait while a change is made, so this engine has exactly the configuration up to the new revision when it had the one before; otherwise it reloads to pick up what it missed. An error from announce is returned: the change is saved, but other replicas only see it after a restart or a later change.
func (*Engine) ClaudeSubscriptionModels ¶
func (e *Engine) ClaudeSubscriptionModels(ctx context.Context) []ListedModel
ClaudeSubscriptionModels returns the Anthropic models listed from the model catalog while the Anthropic provider is used only for Claude subscriptions, which Obot's model list has too (see claudeSubscriptionModels). The gateway's own /v1/models endpoint lists a provider's models with its keys, so it lists none of them.
func (*Engine) Client ¶
func (e *Engine) Client() *bifrostcore.Bifrost
Client exposes the embedded Bifrost core client for direct, in-process calls.
func (*Engine) ConfiguredProviders ¶
ConfiguredProviders returns the names of the upstream providers configured in the gateway.
func (*Engine) Do ¶
Do sends a request to the embedded server's API and decodes a JSON response into out. A write the store refused because it was busy is sent again, for up to 30 seconds (see IsStoreBusy).
func (*Engine) Fail ¶
Fail marks the engine as failed to start, with err as the reason, when it can't be started at all. The gateway then answers with that error instead of serving.
func (*Engine) GuardProviderChange ¶
func (e *Engine) GuardProviderChange(ctx context.Context, method, path string, body []byte, removeKeys bool) (ProviderChange, error)
GuardProviderChange checks a change to a provider or provider key made through the admin API, where path is the engine API path ("providers", "providers/<name>", "providers/<name>/keys" or "providers/<name>/keys/<id>"):
- Every address the provider connects to (its base URL and its keys' server URLs) may be a loopback or private network address only when the provider allows private network addresses. Link-local and cloud metadata addresses are never allowed. The engine itself lets loopback addresses through, and doesn't know every cloud metadata address.
- A provider's base URL can't change while it has keys, so stored keys aren't sent to a new address. With removeKeys, the keys are removed instead (see ProviderChange).
- A key's server URL can't change without its secrets being entered again.
- An update that leaves out a custom provider's custom_provider_config keeps it. The engine treats a missing custom_provider_config as removing it, which it refuses.
func (*Engine) HasUpstreamProviders ¶
HasUpstreamProviders reports whether Bifrost has at least one usable provider. A failed check reports the last known answer, so a transient error doesn't unconfigure the gateway.
func (*Engine) ImportLegacyModelProviders ¶
func (engine *Engine) ImportLegacyModelProviders(ctx context.Context, _ kclient.Client, gatewayClient *gateway.Client) ([]LegacyImportResult, error)
ImportLegacyModelProviders copies the credentials of configured legacy model providers into the gateway, so an existing installation can switch over without re-entering keys. Providers the gateway already has are left alone, and so are those configured from the environment (see importEnvProviders). Two legacy providers that map to the same gateway provider (the Bedrock or Azure variants) become two keys of one provider.
func (*Engine) LLMPluginOrder ¶
LLMPluginOrder returns the names of the gateway's LLM plugins in execution order.
func (*Engine) ListModels ¶
ListModels returns every model Bifrost can currently route to, in the shape the provider back-population controller expects.
func (*Engine) MaxRequestBodyBytes ¶
MaxRequestBodyBytes is the largest request body the gateway accepts, as configured in its settings (max_request_body_size_mb). Obot enforces it before forwarding a request, so a request over it gets a proper error, and a change takes effect at once.
func (*Engine) Reload ¶
Reload rebuilds the gateway from its stores when revision is newer than what it has loaded. Bifrost loads its configuration into memory at startup and only updates that copy for changes made through it, so with several replicas each must reload when another changes the configuration. New requests go to the rebuilt gateway at once; requests already in flight finish on the old one. It reports whether it reloaded. Callers hold the lock every replica takes to change the gateway's configuration (see Options.WithLock), since booting writes the configuration it read back to the shared stores.
func (*Engine) RunStartupTasks ¶
func (e *Engine) RunStartupTasks(ctx context.Context, c kclient.Client, gatewayClient *gateway.Client, currentRevision, announce func(context.Context) (int64, error))
RunStartupTasks waits for the engine to start, imports legacy model provider credentials once, and then keeps Obot's model list in step with Bifrost's. announce tells the other replicas when it changed the gateway's configuration, so they reload it.
Replicas that start together take turns: each holds a lock every replica shares while it makes its changes, and first reloads the configuration the others announced (currentRevision returns the latest), so it never writes back a stale copy of it.
func (*Engine) ServeHTTP ¶
func (e *Engine) ServeHTTP(w http.ResponseWriter, r *http.Request)
ServeHTTP forwards the request to the embedded Bifrost server unchanged. Callers are responsible for authenticating the request and rewriting the path to a Bifrost route.
func (*Engine) Start ¶
Start boots Bifrost in the background. It returns once bootstrap completes or fails.
func (*Engine) Transport ¶
func (e *Engine) Transport() http.RoundTripper
Transport returns a RoundTripper that sends requests to the embedded server regardless of the request's host.
type LegacyImportResult ¶
type LegacyImportResult struct {
ModelProvider string `json:"modelProvider"`
GatewayProvider string `json:"gatewayProvider,omitempty"`
Status string `json:"status"`
Message string `json:"message,omitempty"`
}
LegacyImportResult reports what ImportLegacyProviders did for each legacy provider.
type ListedModel ¶
type ListedModel struct {
// ID is the model's provider-qualified ID, such as "anthropic/claude-sonnet-4-5".
ID string
Name string
}
ListedModel is a model the gateway's model listing endpoint doesn't list itself.
type Options ¶
type Options struct {
// DataDir holds the generated config.json and, when no Postgres DSN is supplied, the
// SQLite config and log stores.
DataDir string
// PostgresDSN, when set, is used for Bifrost's config and log stores instead of SQLite.
// It must point to a database that is not shared with Obot,
// because Bifrost's table names overlap with Obot's.
PostgresDSN string
// EncryptionKey encrypts provider keys at rest. When empty, the key comes from
// SharedKeys if set, and is otherwise generated and persisted in DataDir.
EncryptionKey string
// the stores are shared (Postgres), so all replicas encrypt with the same key.
SharedKeys SharedKeyStore
// LogLevel is the Bifrost logger level (debug, info, warn, error).
LogLevel string
// FirstPlugin and LastPlugin, when set, are registered before the gateway serves any
// request (see registerLLMPlugins). The gateway fails to start if they can't be, so it
// never serves traffic without them.
FirstPlugin, LastPlugin schemas.LLMPlugin
// CurrentRevision, when set, returns the shared configuration revision. The engine reads
// it before it boots, so it knows it already has everything up to that revision and
// doesn't boot a second time to catch up.
CurrentRevision func(context.Context) (int64, error)
// WithLock, when set, runs fn while holding the lock every replica takes to change the
// gateway's configuration (see gateway.Client.WithLLMGatewayLock). The engine boots while
// holding it: booting writes every provider it read back to the shared stores, and on
// empty stores the engine configures providers from the environment on its own, which
// Obot then removes (see removeAutoDetectedKeys). Without the lock, those writes could
// undo or remove what another replica configures meanwhile.
WithLock func(ctx context.Context, fn func() error) error
// ClaudeSubscriptions, when set, reports whether Claude Code users may use their own
// Claude subscriptions through the gateway. While they may, an Anthropic provider without
// keys lists its models from the gateway's model catalog (see ListModels).
ClaudeSubscriptions func(context.Context) (bool, error)
// ModelDataURL is where model pricing and capabilities are refreshed from (see
// modeldata.go). Empty leaves the snapshot bundled with Obot.
ModelDataURL string
}
Options configures the embedded engine.
type ProviderChange ¶
type ProviderChange struct {
// Body is the request body to send to the engine.
Body []byte
// RemoveKeys are the IDs of the provider's keys to remove before the change: its base URL
// changes, and the caller asked for its keys to be removed rather than sent to the new
// address.
RemoveKeys []string
}
ProviderChange is a provider change made through the admin API, after GuardProviderChange checked it.
type SharedKeyStore ¶
SharedKeyStore stores the gateway's encryption key for every replica.
func CredentialKeyStore ¶
func CredentialKeyStore(ctx context.Context, c *gatewayclient.Client) SharedKeyStore
CredentialKeyStore keeps the shared encryption key in Obot's credential store, which is encrypted with Obot's own encryption configuration when one is set.