llmcore

package
v0.28.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 7, 2026 License: MIT Imports: 33 Imported by: 0

Documentation

Overview

Package llmcore implements Obot's LLM gateway using the Bifrost core library (github.com/maximhq/bifrost/core) instead of Bifrost's HTTP transport.

Providers, keys, routing rules and settings live in Obot resources. Each replica's engine reconciles their desired configuration against its successfully applied state. Resource watches trigger Sync without rebuilding the engine.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ProviderProxy

func ProviderProxy(pc schemas.ProviderConfig, raw json.RawMessage) *schemas.ProxyConfig

ProviderProxy returns the effective inference proxy, including provider overrides. Compare this value during reconciliation so unrelated or cosmetic global edits do not restart providers whose network behavior has not changed.

func WithResponseObserver

func WithResponseObserver(ctx context.Context, observe func(schemas.BifrostResponseExtraFields)) context.Context

WithResponseObserver reports metadata for unary responses and every stream chunk, including final latency metadata. Cost and usage reporting belong to plugin hooks.

func WithStreamObserver

WithStreamObserver observes chunks after guardrail replay and successful client encoding. The accumulator belongs to the engine; observers must clean up their per-request state.

Types

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine is one replica's gateway.

func New

func New(ctx context.Context, store *llmcoreclient.Client, opts Options) (*Engine, error)

New starts an engine with the configuration in the store. Individual provider failures leave a degraded engine available; Err reports them until Sync recovers.

func (*Engine) ChatCompletion

func (e *Engine) ChatCompletion(parent context.Context, model string, messages []schemas.ChatMessage) (resp *schemas.BifrostChatResponse, err error)

ChatCompletion is for trusted internal callers that consume a complete result. It shares HTTP model normalization and the configured routing/plugin chain.

func (*Engine) Client

func (e *Engine) Client() *bifrost.Bifrost

Client exposes the trusted in-process core API. Public requests must use the authenticated proxy, which admits subscriptions and strips other caller credentials.

func (*Engine) Err

func (e *Engine) Err() error

Err reports replica-local load/application failures and routing diagnostics, including degraded startup. A missing routing dependency alone does not fail Sync.

func (*Engine) MaxRequestBodyBytes

func (e *Engine) MaxRequestBodyBytes() int64

func (*Engine) ServeHTTP

func (e *Engine) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP serves the OpenAI Responses API at /v1/responses and the Anthropic Messages API at /v1/messages (and /anthropic/v1/messages, the path Obot's proxy uses for the engine today). Either API can reach any provider's models: the engine translates.

func (*Engine) Setting

func (e *Engine) Setting(name string) json.RawMessage

Setting returns an immutable copy of a setting from the loaded configuration.

func (*Engine) Shutdown

func (e *Engine) Shutdown(ctx context.Context) error

Shutdown starts shutdown and waits for completion or ctx cancellation. Cancellation bounds only the caller's wait: cleanup continues in order because Bifrost provider removal is not cancellable. Later calls join the same shutdown. The caller must stop serving requests before shutting down the engine.

func (*Engine) Sync

func (e *Engine) Sync(ctx context.Context) (bool, error)

Sync brings the engine up to the store's configuration, if it changed. Only what changed is touched: a changed provider is gated and drained before its replacement starts; key-only changes take effect on the next request, and rules are replaced one by one. Waiting for drains has a shared one-second budget; unfinished removals stay gated for retry while unrelated configuration advances.

type Options

type Options struct {
	// Plugins are registered in order, with routing inserted after Plugins[0]. The first
	// plugin sees the original request in PreRequestHook; the remaining plugins
	// see the routed request. All PreLLMHook calls run per attempt after routing.
	Plugins []schemas.LLMPlugin
	Logger  schemas.Logger
	// Catalog supplies request pricing and model metadata to Obot plugins.
	Catalog schemas.ModelInfoProvider
}

Options configure an Engine.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL