llmengine

package
v0.28.1-rc2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 13, 2026 License: MIT Imports: 55 Imported by: 0

Documentation

Overview

Package llmengine adapts llmcore to Obot's services and administrative API. It owns model catalog refresh, legacy imports, and admin writes. Inference, routing, and runtime configuration reconciliation belong to llmcore; the catalog and plugins are injected so llmcore does not depend on Obot's service lifecycle.

Index

Constants

View Source
const (
	MetadataProvider        = "bifrostProvider"
	MetadataInputCost       = "bifrostInputCostPerToken"
	MetadataOutputCost      = "bifrostOutputCostPerToken"
	MetadataCacheReadCost   = "bifrostCacheReadCostPerToken"
	MetadataCacheWriteCost  = "bifrostCacheWriteCostPerToken"
	MetadataContextLength   = "bifrostContextLength"
	MetadataMaxOutputTokens = "bifrostMaxOutputTokens"
)

Metadata keys set on listed models. The provider back-population controller reads "displayName", "usage" and "dialect"; the remaining keys carry Bifrost pricing so the caller can derive ModelInfo costs.

View Source
const (
	Version = "obot-core"
)

Variables

This section is empty.

Functions

func ApplyProviderDefaults

func ApplyProviderDefaults(provider map[string]any)

ApplyProviderDefaults fills in the network settings a new provider leaves unset: retries with backoff, which also let requests fail over between the provider's keys, and the request timeout.

func CostFromMetadata

func CostFromMetadata(metadata map[string]string) (types.ModelCost, bool)

CostFromMetadata converts Bifrost per-token pricing into Obot's per-million-token cost.

func DialectForModel

func DialectForModel(provider, modelID string) llmtypes.Dialect

DialectForModel picks the API dialect Obot's agents should use for a model. Bifrost translates every dialect to every provider, so this only selects the most faithful one: Anthropic models speak Messages natively, everything else speaks Responses.

func FillDefaultModelAliases

func FillDefaultModelAliases(ctx context.Context, c kclient.Client) error

FillDefaultModelAliases sets unset default model aliases to the most preferred gateway model that is available, so a new installation works as soon as a provider is added.

func HasSecretValue

func HasSecretValue(raw []byte) bool

HasSecretValue reports whether a JSON value the engine accepts as an optional secret, such as a CA certificate, holds a value. JSON null leaves the setting unset.

func IsMaskedSecret

func IsMaskedSecret(raw []byte) bool

IsMaskedSecret reports whether a JSON value is a secret as the engine lists it, redacted, which the engine takes to mean "keep the stored value" when it is sent back in an update.

func IsSecretReference

func IsSecretReference(raw []byte) bool

IsSecretReference reports whether the engine would read a JSON value from the host's environment or a vault, rather than take it as given, wherever it accepts a secret. It decodes the value with the engine's own decoder, so it can't disagree with the engine on spelling, such as key names in another case.

func LegacyImportDone

func LegacyImportDone(mp *v1.ModelProvider) bool

LegacyImportDone reports whether the credentials of legacy model providers have been imported into the gateway, whose ModelProvider is mp.

func LegacyProvider added in v0.28.1

func LegacyProvider(ctx context.Context) string

func MigrateLegacyModels

func MigrateLegacyModels(ctx context.Context, c kclient.Client, provider *v1.ModelProvider) error

MigrateLegacyModels rewrites default aliases and Profile references after normal discovery publishes gateway models. It never changes source or destination models.

func ModelProviderResource

func ModelProviderResource() *v1.ModelProvider

ModelProviderResource is the ModelProvider resource that represents the embedded gateway. Unlike registry providers it has no daemon and no credential: providers and keys are configured in Bifrost itself.

func PreferredModels

func PreferredModels(alias types.DefaultModelAliasType) []string

PreferredModels returns the gateway models (provider/model) preferred for a default model alias, most preferred first.

func ScrubEngineName

func ScrubEngineName(message string) string

ScrubEngineName replaces the engine's name in an error message with the product's own, so errors read as coming from Obot's LLM gateway.

func ScrubEngineNameBytes

func ScrubEngineNameBytes(body []byte) []byte

ScrubEngineNameBytes is ScrubEngineName for a response body.

func SecretValue

func SecretValue(raw []byte) string

SecretValue returns the value the engine reads from a JSON value it accepts as a secret, such as a region, which may be a string or an object holding one.

func TriggerModelSync

func TriggerModelSync(ctx context.Context, c kclient.Client) error

TriggerModelSync flips the sync annotation on the gateway's ModelProvider so the provider controller re-lists models. The provider controller updates the same resource while it syncs, so a conflicting write is retried against the latest version.

func TrimProviderBaseURLVersion

func TrimProviderBaseURLVersion(provider map[string]any, providerType string) bool

TrimProviderBaseURLVersion removes a trailing "/v1" from the base URL of a provider of type providerType (the provider's own name for a built-in provider, or the type a custom provider is built on) when the engine adds the version to every path itself. OpenAI-compatible servers usually document their base URL with "/v1", which the engine would otherwise turn into "/v1/v1/chat/completions". It reports whether it changed the base URL.

func ValidateAzureKey

func ValidateAzureKey(body []byte) error

ValidateAzureKey restricts ambient Azure identity to Azure AI endpoints and scopes. Call it on the effective key after merging partial updates and restoring masked secrets.

func WithLegacySource added in v0.28.1

func WithLegacySource(ctx context.Context, source string) context.Context

WithLegacySource retains the old credential family even when providers were merged.

Types

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

func New

func New() *Engine

func (*Engine) AnthropicIsDirect

func (e *Engine) AnthropicIsDirect(ctx context.Context) bool

AnthropicIsDirect reports whether the engine's Anthropic provider sends requests straight to Anthropic over verified TLS: no other base URL, and no setting that would let another server present a certificate for Anthropic's host, whether in the provider's network or proxy settings or in the gateway's global proxy. A caller's Claude subscription token is only forwarded when it does, so an administrator can't redirect users' tokens to a server of their own. A proxy without such settings, such as the environment's egress proxy, only relays the TLS connection to Anthropic, so it doesn't stop the token being forwarded.

func (*Engine) ApplyEgressProxy

func (e *Engine) ApplyEgressProxy(ctx context.Context) bool

ApplyEgressProxy routes providers through the HTTP(S) proxy configured in Obot's environment. Bifrost's HTTP client does not read HTTPS_PROXY/NO_PROXY itself, so without this an Obot deployed behind an egress proxy could not reach any provider. Providers that already have a proxy configured, and those whose endpoint NO_PROXY excludes (such as in-cluster vLLM or Ollama servers), are left alone. This only persists desired settings; startup and model controllers apply them through Sync.

func (*Engine) ChatCompletion

func (e *Engine) ChatCompletion(ctx context.Context, model string, messages []schemas.ChatMessage) (*schemas.BifrostChatResponse, error)

ChatCompletion runs trusted message-policy inference through the configured core.

func (*Engine) ClaudeSubscriptionModels

func (e *Engine) ClaudeSubscriptionModels(ctx context.Context) []ListedModel

ClaudeSubscriptionModels returns the Anthropic models listed from the model catalog while the Anthropic provider is used only for Claude subscriptions, which Obot's model list has too (see claudeSubscriptionModels). The gateway's own /v1/models endpoint lists a provider's models with its keys, so it lists none of them.

func (*Engine) Client

func (e *Engine) Client() *bifrostcore.Bifrost

func (*Engine) ConfiguredProviders

func (e *Engine) ConfiguredProviders(ctx context.Context) ([]string, error)

ConfiguredProviders returns the names of the upstream providers configured in the gateway.

func (*Engine) CreateKey

func (e *Engine) CreateKey(ctx context.Context, provider string, body json.RawMessage) (Key, error)

func (*Engine) CreatePricingOverride

func (e *Engine) CreatePricingOverride(ctx context.Context, body json.RawMessage) (tables.TablePricingOverride, error)

func (*Engine) CreateProvider

func (e *Engine) CreateProvider(ctx context.Context, body json.RawMessage) (ProviderResponse, error)

func (*Engine) CreateRoutingRule

func (e *Engine) CreateRoutingRule(ctx context.Context, body json.RawMessage) (tables.TableRoutingRule, error)

func (*Engine) DeleteKey

func (e *Engine) DeleteKey(ctx context.Context, provider, id string) error

func (*Engine) DeletePricingOverride

func (e *Engine) DeletePricingOverride(ctx context.Context, id string) (tables.TablePricingOverride, error)

func (*Engine) DeleteProvider

func (e *Engine) DeleteProvider(ctx context.Context, provider string) error

func (*Engine) DeleteRoutingRule

func (e *Engine) DeleteRoutingRule(ctx context.Context, id string) error

func (*Engine) Err

func (e *Engine) Err() error

func (*Engine) Fail

func (e *Engine) Fail(err error)

func (*Engine) GetClientConfig

func (e *Engine) GetClientConfig(ctx context.Context) (map[string]any, error)

func (*Engine) GetKey

func (e *Engine) GetKey(ctx context.Context, provider, id string) (Key, error)

func (*Engine) GetProvider

func (e *Engine) GetProvider(ctx context.Context, provider string) (ProviderResponse, error)

func (*Engine) GetProxyConfig

func (e *Engine) GetProxyConfig(ctx context.Context) (map[string]any, error)

func (*Engine) GetRoutingRule

func (e *Engine) GetRoutingRule(ctx context.Context, id string) (tables.TableRoutingRule, error)

func (*Engine) GuardProviderChange

func (e *Engine) GuardProviderChange(ctx context.Context, method, path string, body []byte, removeKeys bool) (ProviderChange, error)

GuardProviderChange checks a change to a provider or provider key made through the admin API, where path is the engine API path ("providers", "providers/<name>", "providers/<name>/keys" or "providers/<name>/keys/<id>"):

  • Every address the provider connects to (its base URL and its keys' server URLs) may be a loopback or private network address only when the provider allows private network addresses. Link-local and cloud metadata addresses are never allowed. The engine itself lets loopback addresses through, and doesn't know every cloud metadata address.
  • A provider's base URL can't change while it has keys, so stored keys aren't sent to a new address. With removeKeys, the keys are removed instead (see ProviderChange).
  • A key's server URL can't change without its secrets being entered again.
  • An update that leaves out a custom provider's custom_provider_config keeps it. The engine treats a missing custom_provider_config as removing it, which it refuses.

func (*Engine) HasUpstreamProviders

func (e *Engine) HasUpstreamProviders(ctx context.Context) bool

HasUpstreamProviders reports whether Bifrost has at least one usable provider. A failed check reports the last known answer, so a transient error doesn't unconfigure the gateway.

func (*Engine) ImportLegacyModelProviders

func (e *Engine) ImportLegacyModelProviders(ctx context.Context, c kclient.Client, gatewayClient *gateway.Client, allowPrivateNetwork bool) error

ImportLegacyModelProviders imports each provider family once, without changing models or references.

func (*Engine) LLMPluginOrder

func (e *Engine) LLMPluginOrder() []string

func (*Engine) LegacyKey added in v0.28.1

func (e *Engine) LegacyKey(ctx context.Context, target string) (string, error)

LegacyKey prefers an eligible imported credential; other requests use the normal key pool.

func (*Engine) ListCatalogModels

func (e *Engine) ListCatalogModels(ctx context.Context, provider, query string) ([]map[string]any, error)

func (*Engine) ListKeys

func (e *Engine) ListKeys(ctx context.Context) ([]Key, error)

func (*Engine) ListModels

func (e *Engine) ListModels(ctx context.Context) (*openai.ModelsList, error)

ListModels returns every model Bifrost can currently route to, in the shape the provider back-population controller expects.

func (*Engine) ListPricingOverrides

func (e *Engine) ListPricingOverrides(ctx context.Context) ([]tables.TablePricingOverride, error)

func (*Engine) ListProviderKeys

func (e *Engine) ListProviderKeys(ctx context.Context, provider string) ([]Key, error)

func (*Engine) ListProviders

func (e *Engine) ListProviders(ctx context.Context) ([]ProviderResponse, error)

func (*Engine) ListRoutingRules

func (e *Engine) ListRoutingRules(ctx context.Context) ([]tables.TableRoutingRule, error)

func (*Engine) MaxRequestBodyBytes

func (e *Engine) MaxRequestBodyBytes() int64

func (*Engine) Ready

func (e *Engine) Ready() bool

func (*Engine) RefreshAzureDeployments

func (e *Engine) RefreshAzureDeployments(ctx context.Context) error

RefreshAzureDeployments is called only by the leader-elected model-sync controller.

func (*Engine) RefreshPricing

func (e *Engine) RefreshPricing(ctx context.Context) error

func (*Engine) RefreshProviderModels

func (e *Engine) RefreshProviderModels(ctx context.Context, name string) (pending bool, err error)

RefreshProviderModels queues Azure discovery on the leader; other providers refresh synchronously.

func (*Engine) RunStartupTasks

func (e *Engine) RunStartupTasks(ctx context.Context, c kclient.Client)

RunStartupTasks configures environment keys and periodically asks Obot to refresh models.

func (*Engine) ServeHTTP

func (e *Engine) ServeHTTP(w http.ResponseWriter, r *http.Request)

func (*Engine) ServeMetrics

func (e *Engine) ServeMetrics(w http.ResponseWriter, r *http.Request)

ServeMetrics exposes this replica's gateway inference metrics.

func (*Engine) Shutdown

func (e *Engine) Shutdown(ctx context.Context) error

Shutdown bounds the caller's wait while core cleanup continues before the catalog stops.

func (*Engine) Start

func (e *Engine) Start(ctx context.Context, opts Options) error

func (*Engine) Sync

func (e *Engine) Sync(ctx context.Context) (bool, error)

func (*Engine) UnavailableModels

func (e *Engine) UnavailableModels(ctx context.Context, ids []string) (map[string]bool, error)

UnavailableModels identifies models that resource retention must not restore. Unknown models remain eligible for retention during upstream listing failures.

func (*Engine) UpdateClientConfig

func (e *Engine) UpdateClientConfig(ctx context.Context, body json.RawMessage) (map[string]any, error)

func (*Engine) UpdateKey

func (e *Engine) UpdateKey(ctx context.Context, provider, id string, body json.RawMessage) (Key, error)

func (*Engine) UpdatePricingOverride

func (e *Engine) UpdatePricingOverride(ctx context.Context, id string, body json.RawMessage) (tables.TablePricingOverride, error)

func (*Engine) UpdateProvider

func (e *Engine) UpdateProvider(ctx context.Context, provider string, body json.RawMessage, removeKeys bool) (ProviderResponse, error)

func (*Engine) UpdateProxyConfig

func (e *Engine) UpdateProxyConfig(ctx context.Context, body json.RawMessage) error

func (*Engine) UpdateRoutingRule

func (e *Engine) UpdateRoutingRule(ctx context.Context, id string, body json.RawMessage) (tables.TableRoutingRule, error)

func (*Engine) Wait

func (e *Engine) Wait(ctx context.Context) error

type Key

type Key struct {
	schemas.Key
	llmcoreclient.AzureAccount
}

ProviderResponse adds Obot's admin identifiers to Bifrost's redacted configuration.

type ListedModel

type ListedModel struct {
	// ID is the model's provider-qualified ID, such as "anthropic/claude-sonnet-4-5".
	ID   string
	Name string
}

ListedModel is a model the gateway's model listing endpoint doesn't list itself.

type Options

type Options struct {
	Store                   *llmcoreclient.Client
	TriggerModelSync        func(context.Context) error
	DataDir                 string
	LogLevel                string
	FirstPlugin, LastPlugin schemas.LLMPlugin
	ClaudeSubscriptions     func(context.Context) (bool, error)
	ModelDataURL            string
}

type ProviderChange

type ProviderChange struct {
	// Body is the validated partial patch, with custom provider identity preserved.
	Body []byte
	// RemoveKeys are the IDs of the provider's keys to remove before the change: its base URL
	// changes, and the caller asked for its keys to be removed rather than sent to the new
	// address.
	RemoveKeys []string
}

ProviderChange is a provider change made through the admin API, after GuardProviderChange checked it.

type ProviderResponse

type ProviderResponse struct {
	Keys []Key `json:"keys"`
	*configstore.ProviderConfig
	Name           schemas.ModelProvider `json:"name"`
	Provider       schemas.ModelProvider `json:"provider"`
	ProviderStatus string                `json:"provider_status"`
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL