Documentation
¶
Overview ¶
Package llmcore implements Obot's LLM gateway using the Bifrost core library (github.com/maximhq/bifrost/core) instead of Bifrost's HTTP transport.
Providers, keys, routing rules and settings live in Obot resources. Each replica's engine reconciles their desired configuration against its successfully applied state. Resource watches trigger Sync without rebuilding the engine.
Index ¶
- func ProviderProxy(pc schemas.ProviderConfig, raw json.RawMessage) *schemas.ProxyConfig
- func WithInitialKey(ctx context.Context, keyID string) context.Context
- func WithResponseObserver(ctx context.Context, observe func(schemas.BifrostResponseExtraFields)) context.Context
- func WithStreamObserver(ctx context.Context, ...) context.Context
- type Engine
- func (e *Engine) ChatCompletion(parent context.Context, model string, messages []schemas.ChatMessage) (resp *schemas.BifrostChatResponse, err error)
- func (e *Engine) Client() *bifrost.Bifrost
- func (e *Engine) Err() error
- func (e *Engine) MaxRequestBodyBytes() int64
- func (e *Engine) ProviderKeyID(ctx context.Context, provider schemas.ModelProvider, name, model string) (string, bool, error)
- func (e *Engine) ServeHTTP(w http.ResponseWriter, r *http.Request)
- func (e *Engine) Setting(name string) json.RawMessage
- func (e *Engine) Shutdown(ctx context.Context) error
- func (e *Engine) Sync(ctx context.Context) (bool, error)
- type Options
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ProviderProxy ¶
func ProviderProxy(pc schemas.ProviderConfig, raw json.RawMessage) *schemas.ProxyConfig
ProviderProxy returns the effective inference proxy, including provider overrides. Compare this value during reconciliation so unrelated or cosmetic global edits do not restart providers whose network behavior has not changed.
func WithInitialKey ¶ added in v0.28.1
WithInitialKey selects the initial credential for a trusted HTTP request; routing rules and fallback attempts may replace it. It is never populated from caller headers.
func WithResponseObserver ¶
func WithResponseObserver(ctx context.Context, observe func(schemas.BifrostResponseExtraFields)) context.Context
WithResponseObserver reports metadata for unary responses and every stream chunk, including final latency metadata. Cost and usage reporting belong to plugin hooks.
func WithStreamObserver ¶
func WithStreamObserver(ctx context.Context, observe func(*schemas.BifrostContext, *schemas.BifrostStreamChunk, *streaming.Accumulator)) context.Context
WithStreamObserver observes chunks after guardrail replay and successful client encoding. The accumulator belongs to the engine; observers must clean up their per-request state.
Types ¶
type Engine ¶
type Engine struct {
// contains filtered or unexported fields
}
Engine is one replica's gateway.
func New ¶
New starts an engine with the configuration in the store. Individual provider failures leave a degraded engine available; Err reports them until Sync recovers.
func (*Engine) ChatCompletion ¶
func (e *Engine) ChatCompletion(parent context.Context, model string, messages []schemas.ChatMessage) (resp *schemas.BifrostChatResponse, err error)
ChatCompletion is for trusted internal callers that consume a complete result. It shares HTTP model normalization and the configured routing/plugin chain.
func (*Engine) Client ¶
Client exposes the trusted in-process core API. Public requests must use the authenticated proxy, which admits subscriptions and strips other caller credentials.
func (*Engine) Err ¶
Err reports replica-local load/application failures and routing diagnostics, including degraded startup. A missing routing dependency alone does not fail Sync.
func (*Engine) MaxRequestBodyBytes ¶
func (*Engine) ProviderKeyID ¶ added in v0.28.1
func (e *Engine) ProviderKeyID(ctx context.Context, provider schemas.ModelProvider, name, model string) (string, bool, error)
ProviderKeyID prefers an enabled key that supports model, using the same runtime key view as inference (including Azure discovery). The bool reports provider existence.
func (*Engine) ServeHTTP ¶
func (e *Engine) ServeHTTP(w http.ResponseWriter, r *http.Request)
ServeHTTP serves the OpenAI Responses API at /v1/responses and the Anthropic Messages API at /v1/messages (and /anthropic/v1/messages, the path Obot's proxy uses for the engine today). Either API can reach any provider's models: the engine translates.
func (*Engine) Setting ¶
func (e *Engine) Setting(name string) json.RawMessage
Setting returns an immutable copy of a setting from the loaded configuration.
func (*Engine) Shutdown ¶
Shutdown starts shutdown and waits for completion or ctx cancellation. Cancellation bounds only the caller's wait: cleanup continues in order because Bifrost provider removal is not cancellable. Later calls join the same shutdown. The caller must stop serving requests before shutting down the engine.
func (*Engine) Sync ¶
Sync brings the engine up to the store's configuration, if it changed. Only what changed is touched: a changed provider is gated and drained before its replacement starts; key-only changes take effect on the next request, and rules are replaced one by one. Waiting for drains has a shared one-second budget; unfinished removals stay gated for retry while unrelated configuration advances.
type Options ¶
type Options struct {
// Plugins are registered in order, with routing inserted after Plugins[0]. The first
// plugin sees the original request in PreRequestHook; the remaining plugins
// see the routed request. All PreLLMHook calls run per attempt after routing.
Plugins []schemas.LLMPlugin
Logger schemas.Logger
// Catalog supplies request pricing and model metadata to Obot plugins.
Catalog schemas.ModelInfoProvider
}
Options configure an Engine.