llmengine

package
v0.28.0-alpha3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 22, 2026 License: MIT Imports: 49 Imported by: 0

Documentation

Overview

Package bifrost embeds the Bifrost AI gateway (github.com/maximhq/bifrost) inside the Obot process and exposes it to the rest of Obot as an http.Handler.

Bifrost runs its full HTTP transport (inference routes, governance, logging, telemetry, routing and caching plugins) in-process. It never listens on a network socket: it is served over an in-memory listener, so the only way to reach it is through Obot, which authenticates callers and applies its own policies before forwarding requests.

Index

Constants

View Source
const (
	MetadataProvider        = "bifrostProvider"
	MetadataInputCost       = "bifrostInputCostPerToken"
	MetadataOutputCost      = "bifrostOutputCostPerToken"
	MetadataCacheReadCost   = "bifrostCacheReadCostPerToken"
	MetadataCacheWriteCost  = "bifrostCacheWriteCostPerToken"
	MetadataContextLength   = "bifrostContextLength"
	MetadataMaxOutputTokens = "bifrostMaxOutputTokens"
)

Metadata keys set on listed models. The provider back-population controller reads "displayName", "usage" and "dialect"; the remaining keys carry Bifrost pricing so the caller can derive ModelInfo costs.

View Source
const (

	// Version is reported to Bifrost as its runtime version.
	Version = "obot-embedded"
)

Variables

This section is empty.

Functions

func ApplyProviderDefaults

func ApplyProviderDefaults(provider map[string]any)

ApplyProviderDefaults fills in the network settings a new provider leaves unset: retries with backoff, which also let requests fail over between the provider's keys, and the request timeout.

func CostFromMetadata

func CostFromMetadata(metadata map[string]string) (types.ModelCost, bool)

CostFromMetadata converts Bifrost per-token pricing into Obot's per-million-token cost.

func DerivePostgresDSN

func DerivePostgresDSN(ctx context.Context, obotDSN string) (string, error)

DerivePostgresDSN returns a DSN for a database named after Obot's own, on the same Postgres server, creating the database if it doesn't exist. Bifrost's tables can't share Obot's database, but sharing the server keeps every Obot replica on the same gateway state without extra configuration. Connection failures are retried for up to a minute; a database that can't be created fails at once, with an error that says how to create it.

func DialectForModel

func DialectForModel(provider, modelID string) llmtypes.Dialect

DialectForModel picks the API dialect Obot's agents should use for a model. Bifrost translates every dialect to every provider, so this only selects the most faithful one: Anthropic models speak Messages natively, everything else speaks Responses.

func FillDefaultModelAliases

func FillDefaultModelAliases(ctx context.Context, c kclient.Client) error

FillDefaultModelAliases sets unset default model aliases to the most preferred gateway model that is available, so a new installation works as soon as a provider is added.

func HasSecretValue

func HasSecretValue(raw []byte) bool

HasSecretValue reports whether a JSON value the engine accepts as an optional secret, such as a CA certificate, holds a value. JSON null leaves the setting unset.

func IsMaskedSecret

func IsMaskedSecret(raw []byte) bool

IsMaskedSecret reports whether a JSON value is a secret as the engine lists it, redacted, which the engine takes to mean "keep the stored value" when it is sent back in an update.

func IsSecretReference

func IsSecretReference(raw []byte) bool

IsSecretReference reports whether the engine would read a JSON value from the host's environment or a vault, rather than take it as given, wherever it accepts a secret. It decodes the value with the engine's own decoder, so it can't disagree with the engine on spelling, such as key names in another case.

func IsStoreBusy

func IsStoreBusy(statusCode int, body []byte) bool

IsStoreBusy reports whether a response from the engine's API failed because its SQLite store was locked by another write. The write was not made, and can be sent again.

func LegacyImportDone

func LegacyImportDone(mp *v1.ModelProvider) bool

LegacyImportDone reports whether the credentials of legacy model providers have been imported into the gateway, whose ModelProvider is mp.

func MigrateLegacyModels

func MigrateLegacyModels(ctx context.Context, c kclient.Client, importDone bool) error

MigrateLegacyModels moves what refers to legacy model providers' models over to the gateway's models: default model aliases and model access policies that name a legacy model are re-pointed to the gateway model with the same upstream model, and edits made to the legacy model (deactivating it, an alias, a cost override) carry over. It runs after every gateway model sync, so references move as soon as the gateway lists the models.

Once legacy provider credentials have been imported (importDone), legacy models that were migrated or that nothing refers to are deleted, and then legacy model providers left without models. A legacy model that is still referred to but that the gateway doesn't serve is kept, so the reference doesn't break, until the gateway lists its model.

func ModelProviderResource

func ModelProviderResource() *v1.ModelProvider

ModelProviderResource is the ModelProvider resource that represents the embedded gateway. Unlike registry providers it has no daemon and no credential: providers and keys are configured in Bifrost itself.

func PreferredModels

func PreferredModels(alias types.DefaultModelAliasType) []string

PreferredModels returns the gateway models (provider/model) preferred for a default model alias, most preferred first.

func ScrubEngineName

func ScrubEngineName(message string) string

ScrubEngineName replaces the engine's name in an error message with the product's own, so errors read as coming from Obot's LLM gateway.

func ScrubEngineNameBytes

func ScrubEngineNameBytes(body []byte) []byte

ScrubEngineNameBytes is ScrubEngineName for a response body.

func SecretValue

func SecretValue(raw []byte) string

SecretValue returns the value the engine reads from a JSON value it accepts as a secret, such as a region, which may be a string or an object holding one.

func TriggerModelSync

func TriggerModelSync(ctx context.Context, c kclient.Client) error

TriggerModelSync flips the sync annotation on the gateway's ModelProvider so the provider controller re-lists models. The provider controller updates the same resource while it syncs, so a conflicting write is retried against the latest version.

func TrimProviderBaseURLVersion

func TrimProviderBaseURLVersion(provider map[string]any, providerType string) bool

TrimProviderBaseURLVersion removes a trailing "/v1" from the base URL of a provider of type providerType (the provider's own name for a built-in provider, or the type a custom provider is built on) when the engine adds the version to every path itself. OpenAI-compatible servers usually document their base URL with "/v1", which the engine would otherwise turn into "/v1/v1/chat/completions". It reports whether it changed the base URL.

Types

type APIError

type APIError struct {
	StatusCode int
	Message    string
}

APIError is returned by Do for non-2xx responses from Bifrost.

func (*APIError) Error

func (e *APIError) Error() string

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine is a running, embedded Bifrost gateway.

func New

func New() *Engine

New creates an engine. Call Start to boot it.

func (*Engine) AnthropicIsDirect

func (e *Engine) AnthropicIsDirect(ctx context.Context) bool

AnthropicIsDirect reports whether the engine's Anthropic provider sends requests straight to Anthropic over verified TLS: no other base URL, and no setting that would let another server present a certificate for Anthropic's host, whether in the provider's network or proxy settings or in the gateway's global proxy. A caller's Claude subscription token is only forwarded when it does, so an administrator can't redirect users' tokens to a server of their own. A proxy without such settings, such as the environment's egress proxy, only relays the TLS connection to Anthropic, so it doesn't stop the token being forwarded.

func (*Engine) ApplyEgressProxy

func (e *Engine) ApplyEgressProxy(ctx context.Context) (changed bool)

ApplyEgressProxy routes providers through the HTTP(S) proxy configured in Obot's environment. Bifrost's HTTP client does not read HTTPS_PROXY/NO_PROXY itself, so without this an Obot deployed behind an egress proxy could not reach any provider. Providers that already have a proxy configured, and those whose endpoint NO_PROXY excludes (such as in-cluster vLLM or Ollama servers), are left alone.

func (*Engine) BaseURL

func (e *Engine) BaseURL() url.URL

BaseURL is the URL requests to the embedded server should be addressed to.

func (*Engine) Change

func (e *Engine) Change(ctx context.Context, change func() (changed, applied bool), announce func(context.Context) (int64, error)) error

Change makes a configuration change through this engine and tells the other replicas. change applies it and reports whether anything changed, and whether this engine is sure to have the change in memory (false when, say, it failed partway); announce then advances the shared configuration revision, and returns it. Reloads wait while a change is made, so this engine has exactly the configuration up to the new revision when it had the one before; otherwise it reloads to pick up what it missed. An error from announce is returned: the change is saved, but other replicas only see it after a restart or a later change.

func (*Engine) ClaudeSubscriptionModels

func (e *Engine) ClaudeSubscriptionModels(ctx context.Context) []ListedModel

ClaudeSubscriptionModels returns the Anthropic models listed from the model catalog while the Anthropic provider is used only for Claude subscriptions, which Obot's model list has too (see claudeSubscriptionModels). The gateway's own /v1/models endpoint lists a provider's models with its keys, so it lists none of them.

func (*Engine) Client

func (e *Engine) Client() *bifrostcore.Bifrost

Client exposes the embedded Bifrost core client for direct, in-process calls.

func (*Engine) ConfiguredProviders

func (e *Engine) ConfiguredProviders(ctx context.Context) ([]string, error)

ConfiguredProviders returns the names of the upstream providers configured in the gateway.

func (*Engine) Do

func (e *Engine) Do(ctx context.Context, method, path string, body any, out any) error

Do sends a request to the embedded server's API and decodes a JSON response into out. A write the store refused because it was busy is sent again, for up to 30 seconds (see IsStoreBusy).

func (*Engine) Err

func (e *Engine) Err() error

Err returns the startup error, if any.

func (*Engine) Fail

func (e *Engine) Fail(err error)

Fail marks the engine as failed to start, with err as the reason, when it can't be started at all. The gateway then answers with that error instead of serving.

func (*Engine) GuardProviderChange

func (e *Engine) GuardProviderChange(ctx context.Context, method, path string, body []byte, removeKeys bool) (ProviderChange, error)

GuardProviderChange checks a change to a provider or provider key made through the admin API, where path is the engine API path ("providers", "providers/<name>", "providers/<name>/keys" or "providers/<name>/keys/<id>"):

  • Every address the provider connects to (its base URL and its keys' server URLs) may be a loopback or private network address only when the provider allows private network addresses. Link-local and cloud metadata addresses are never allowed. The engine itself lets loopback addresses through, and doesn't know every cloud metadata address.
  • A provider's base URL can't change while it has keys, so stored keys aren't sent to a new address. With removeKeys, the keys are removed instead (see ProviderChange).
  • A key's server URL can't change without its secrets being entered again.
  • An update that leaves out a custom provider's custom_provider_config keeps it. The engine treats a missing custom_provider_config as removing it, which it refuses.

func (*Engine) HasUpstreamProviders

func (e *Engine) HasUpstreamProviders(ctx context.Context) bool

HasUpstreamProviders reports whether Bifrost has at least one usable provider. A failed check reports the last known answer, so a transient error doesn't unconfigure the gateway.

func (*Engine) ImportLegacyModelProviders

func (engine *Engine) ImportLegacyModelProviders(ctx context.Context, _ kclient.Client, gatewayClient *gateway.Client) ([]LegacyImportResult, error)

ImportLegacyModelProviders copies the credentials of configured legacy model providers into the gateway, so an existing installation can switch over without re-entering keys. Providers the gateway already has are left alone, and so are those configured from the environment (see importEnvProviders). Two legacy providers that map to the same gateway provider (the Bedrock or Azure variants) become two keys of one provider.

func (*Engine) LLMPluginOrder

func (e *Engine) LLMPluginOrder() []string

LLMPluginOrder returns the names of the gateway's LLM plugins in execution order.

func (*Engine) ListModels

func (e *Engine) ListModels(ctx context.Context) (*openai.ModelsList, error)

ListModels returns every model Bifrost can currently route to, in the shape the provider back-population controller expects.

func (*Engine) MaxRequestBodyBytes

func (e *Engine) MaxRequestBodyBytes() int64

MaxRequestBodyBytes is the largest request body the gateway accepts, as configured in its settings (max_request_body_size_mb). Obot enforces it before forwarding a request, so a request over it gets a proper error, and a change takes effect at once.

func (*Engine) Ready

func (e *Engine) Ready() bool

Ready reports whether the engine has started successfully.

func (*Engine) Reload

func (e *Engine) Reload(revision int64) (bool, error)

Reload rebuilds the gateway from its stores when revision is newer than what it has loaded. Bifrost loads its configuration into memory at startup and only updates that copy for changes made through it, so with several replicas each must reload when another changes the configuration. New requests go to the rebuilt gateway at once; requests already in flight finish on the old one. It reports whether it reloaded. Callers hold the lock every replica takes to change the gateway's configuration (see Options.WithLock), since booting writes the configuration it read back to the shared stores.

func (*Engine) RunStartupTasks

func (e *Engine) RunStartupTasks(ctx context.Context, c kclient.Client, gatewayClient *gateway.Client, currentRevision, announce func(context.Context) (int64, error))

RunStartupTasks waits for the engine to start, imports legacy model provider credentials once, and then keeps Obot's model list in step with Bifrost's. announce tells the other replicas when it changed the gateway's configuration, so they reload it.

Replicas that start together take turns: each holds a lock every replica shares while it makes its changes, and first reloads the configuration the others announced (currentRevision returns the latest), so it never writes back a stale copy of it.

func (*Engine) ServeHTTP

func (e *Engine) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP forwards the request to the embedded Bifrost server unchanged. Callers are responsible for authenticating the request and rewriting the path to a Bifrost route.

func (*Engine) Shutdown

func (e *Engine) Shutdown()

Shutdown stops the embedded server.

func (*Engine) Start

func (e *Engine) Start(ctx context.Context, opts Options) error

Start boots Bifrost in the background. It returns once bootstrap completes or fails.

func (*Engine) Transport

func (e *Engine) Transport() http.RoundTripper

Transport returns a RoundTripper that sends requests to the embedded server regardless of the request's host.

func (*Engine) Wait

func (e *Engine) Wait(ctx context.Context) error

Wait blocks until the engine has finished starting (successfully or not).

type LegacyImportResult

type LegacyImportResult struct {
	ModelProvider   string `json:"modelProvider"`
	GatewayProvider string `json:"gatewayProvider,omitempty"`
	Status          string `json:"status"`
	Message         string `json:"message,omitempty"`
}

LegacyImportResult reports what ImportLegacyProviders did for each legacy provider.

type ListedModel

type ListedModel struct {
	// ID is the model's provider-qualified ID, such as "anthropic/claude-sonnet-4-5".
	ID   string
	Name string
}

ListedModel is a model the gateway's model listing endpoint doesn't list itself.

type Options

type Options struct {
	// DataDir holds the generated config.json and, when no Postgres DSN is supplied, the
	// SQLite config and log stores.
	DataDir string
	// PostgresDSN, when set, is used for Bifrost's config and log stores instead of SQLite.
	// It must point to a database that is not shared with Obot,
	// because Bifrost's table names overlap with Obot's.
	PostgresDSN string
	// EncryptionKey encrypts provider keys at rest. When empty, the key comes from
	// SharedKeys if set, and is otherwise generated and persisted in DataDir.
	EncryptionKey string
	// SharedKeys stores the encryption key where every replica can read it. Set it whenever
	// the stores are shared (Postgres), so all replicas encrypt with the same key.
	SharedKeys SharedKeyStore
	// LogLevel is the Bifrost logger level (debug, info, warn, error).
	LogLevel string
	// FirstPlugin and LastPlugin, when set, are registered before the gateway serves any
	// request (see registerLLMPlugins). The gateway fails to start if they can't be, so it
	// never serves traffic without them.
	FirstPlugin, LastPlugin schemas.LLMPlugin
	// CurrentRevision, when set, returns the shared configuration revision. The engine reads
	// it before it boots, so it knows it already has everything up to that revision and
	// doesn't boot a second time to catch up.
	CurrentRevision func(context.Context) (int64, error)
	// WithLock, when set, runs fn while holding the lock every replica takes to change the
	// gateway's configuration (see gateway.Client.WithLLMGatewayLock). The engine boots while
	// holding it: booting writes every provider it read back to the shared stores, and on
	// empty stores the engine configures providers from the environment on its own, which
	// Obot then removes (see removeAutoDetectedKeys). Without the lock, those writes could
	// undo or remove what another replica configures meanwhile.
	WithLock func(ctx context.Context, fn func() error) error
	// ClaudeSubscriptions, when set, reports whether Claude Code users may use their own
	// Claude subscriptions through the gateway. While they may, an Anthropic provider without
	// keys lists its models from the gateway's model catalog (see ListModels).
	ClaudeSubscriptions func(context.Context) (bool, error)
	// ModelDataURL is where model pricing and capabilities are refreshed from (see
	// modeldata.go). Empty leaves the snapshot bundled with Obot.
	ModelDataURL string
}

Options configures the embedded engine.

type ProviderChange

type ProviderChange struct {
	// Body is the request body to send to the engine.
	Body []byte
	// RemoveKeys are the IDs of the provider's keys to remove before the change: its base URL
	// changes, and the caller asked for its keys to be removed rather than sent to the new
	// address.
	RemoveKeys []string
}

ProviderChange is a provider change made through the admin API, after GuardProviderChange checked it.

type SharedKeyStore

type SharedKeyStore interface {
	// EnsureEncryptionKey stores key unless a key is already stored, and returns the
	// stored key.
	EnsureEncryptionKey(key string) (string, error)
}

SharedKeyStore stores the gateway's encryption key for every replica.

func CredentialKeyStore

func CredentialKeyStore(ctx context.Context, c *gatewayclient.Client) SharedKeyStore

CredentialKeyStore keeps the shared encryption key in Obot's credential store, which is encrypted with Obot's own encryption configuration when one is set.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL