providers

package
v0.1.67 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 1, 2026 License: MIT Imports: 39 Imported by: 0

Documentation

Overview

Package providers provides a factory for creating provider instances.

Package providers provides model registry and routing for LLM providers.

Package providers provides a router for multiple LLM providers.

Index

Constants

View Source 
const (
	CredentialFieldAPIKeys                  = "api_keys"
	CredentialFieldSessionStickyKeys        = "session_sticky_keys"
	CredentialFieldBaseURL                  = "base_url"
	CredentialFieldAPIVersion               = "api_version"
	CredentialFieldBackend                  = "backend"
	CredentialFieldAuthType                 = "auth_type"
	CredentialFieldAPIMode                  = "api_mode"
	CredentialFieldVertexProject            = "vertex_project"
	CredentialFieldVertexLocation           = "vertex_location"
	CredentialFieldServiceAccountFile       = "service_account_file"
	CredentialFieldServiceAccountJSON       = "service_account_json"
	CredentialFieldServiceAccountJSONBase64 = "service_account_json_base64"
	CredentialFieldGCPScope                 = "gcp_scope"
	CredentialFieldModels                   = "models"
)

Credential form field names. They match the admin API's upsert payload keys and the ManagedProviderCredential fields they fill, so a schema entry can be used verbatim as an error `param` and as a dashboard form field id.

Variables

View Source 
var ErrCredentialNotFound = errors.New("provider credential not found")

ErrCredentialNotFound indicates a requested admin-managed provider credential row was not found.

View Source 
var ErrRegistryNotInitialized = fmt.Errorf("model registry has no models: ensure Initialize() or LoadFromCache() is called before using the router")

ErrRegistryNotInitialized is returned when the router is used before the registry has any models.

Functions

func AdaptReasoningEffortRequest 

func AdaptReasoningEffortRequest(req *core.ChatRequest, effort string) (*core.ChatRequest, error)

AdaptReasoningEffortRequest rewrites GoModel's common nested reasoning shape into the flat "reasoning_effort" string extension used by several OpenAI-compatible providers (Gemini, DeepSeek). It shallow-copies the typed request and merges the effort into ExtraFields, so the body is marshaled only once, by the HTTP client. Other reasoning fields (e.g. budget_tokens) are dropped: these providers accept the flat string only.

func BuildResponsesOutputItems 

func BuildResponsesOutputItems(msg core.ResponseMessage) []core.ResponsesOutputItem

BuildResponsesOutputItems converts a response message into Responses API output items.

func CloneHTTPHeaders 

func CloneHTTPHeaders(src http.Header) map[string][]string

CloneHTTPHeaders returns a detached copy of an http.Header map.

func ConvertChatResponseToResponses 

func ConvertChatResponseToResponses(resp *core.ChatResponse) *core.ResponsesResponse

ConvertChatResponseToResponses converts a ChatResponse to a ResponsesResponse.

func ConvertResponsesContentToChatContent 

func ConvertResponsesContentToChatContent(content any) (any, bool)

ConvertResponsesContentToChatContent maps Responses input content to Chat content. Text-only arrays are flattened to strings for broader provider compatibility. Any non-text part preserves the array form so multimodal payloads survive routing.

func ConvertResponsesInputToMessages 

func ConvertResponsesInputToMessages(input any) ([]core.Message, error)

ConvertResponsesInputToMessages converts a Responses API input payload into Chat API messages.

func ConvertResponsesRequestToChat 

func ConvertResponsesRequestToChat(req *core.ResponsesRequest) (*core.ChatRequest, error)

ConvertResponsesRequestToChat converts a ResponsesRequest to a ChatRequest. It also validates the supported Responses input shapes and returns an error when the request cannot be converted safely.

func CreateOpenAICompatibleFile 

func CreateOpenAICompatibleFile(ctx context.Context, client *llmclient.Client, req *core.FileCreateRequest) (*core.FileObject, error)

CreateOpenAICompatibleFile uploads a file using the OpenAI-compatible multipart files API.

func CreateOpenAICompatibleFileWithPreparer 

func CreateOpenAICompatibleFileWithPreparer(ctx context.Context, client *llmclient.Client, req *core.FileCreateRequest, prepare openAICompatibleRequestPreparer) (*core.FileObject, error)

func DeleteOpenAICompatibleFile 

func DeleteOpenAICompatibleFile(ctx context.Context, client *llmclient.Client, id string) (*core.FileDeleteResponse, error)

DeleteOpenAICompatibleFile deletes a file object by id after normalizing the incoming id via validatedOpenAICompatibleFileID. Missing response ID and Object fields are synthesized on the returned delete response.

func DeleteOpenAICompatibleFileWithPreparer 

func DeleteOpenAICompatibleFileWithPreparer(ctx context.Context, client *llmclient.Client, id string, prepare openAICompatibleRequestPreparer) (*core.FileDeleteResponse, error)

func EnsureChatCompletionSSE 

func EnsureChatCompletionSSE(stream io.ReadCloser) io.ReadCloser

EnsureChatCompletionSSE normalizes a chat completions stream so the client always receives well-formed Server-Sent Events terminated by data: [DONE].

Some OpenAI-compatible upstreams ignore stream:true and reply with a single buffered application/json completion (no data: framing, no [DONE]). Forwarding that verbatim under a text/event-stream content type leaves SSE clients waiting forever for an end-of-stream marker that never arrives. When the upstream body is detected as a buffered JSON object it is re-emitted as one SSE chunk plus a terminal [DONE]; genuine SSE streams pass through untouched with no buffering.

func EnsureProviderBatchID 

func EnsureProviderBatchID(resp *core.BatchResponse)

EnsureProviderBatchID defaults the provider-facing batch ID to the response ID when an OpenAI-compatible upstream does not return a distinct one. No-op for a nil response or one that already carries a provider batch ID.

func EnsureProviderBatchIDs 

func EnsureProviderBatchIDs(resp *core.BatchListResponse)

EnsureProviderBatchIDs applies EnsureProviderBatchID to every batch in a list response.

func EnsureResponsesDone 

func EnsureResponsesDone(stream io.ReadCloser) io.ReadCloser

EnsureResponsesDone normalizes Responses API streams so clients always receive a terminal data: [DONE] marker when the upstream stream reaches a completed Responses event but closes at EOF before sending the final marker.

func ExtractContentFromInput 

func ExtractContentFromInput(content any) string

ExtractContentFromInput extracts text content from responses input.

func FetchBatchResultsFromOutputFile 

func FetchBatchResultsFromOutputFile(ctx context.Context, client *llmclient.Client, providerName, batchID string) (*core.BatchResultsResponse, error)

FetchBatchResultsFromOutputFile adapts OpenAI-compatible batch output files to gateway batch results.

func FetchBatchResultsFromOutputFileWithPreparer 

func FetchBatchResultsFromOutputFileWithPreparer(ctx context.Context, client *llmclient.Client, providerName, batchID string, prepare openAICompatibleRequestPreparer) (*core.BatchResultsResponse, error)

func FormatChatChunkSSE 

func FormatChatChunkSSE(id string, created int64, model, provider string, delta map[string]any, finishReason any, usage map[string]any) string

FormatChatChunkSSE renders a single-choice OpenAI chat.completion.chunk as one SSE data line. It defines the chunk envelope emitted by native-protocol stream converters (Anthropic, Bedrock) so the OpenAI-compatible wire shape lives in one place. A nil usage omits the member; finishReason may be nil.

func GetOpenAICompatibleFile 

func GetOpenAICompatibleFile(ctx context.Context, client *llmclient.Client, id string) (*core.FileObject, error)

GetOpenAICompatibleFile retrieves a file object by id after normalizing the incoming id via validatedOpenAICompatibleFileID. Missing response ID and Object fields are synthesized on the returned file object.

func GetOpenAICompatibleFileContent 

func GetOpenAICompatibleFileContent(ctx context.Context, client *llmclient.Client, id string) (*core.FileContentResponse, error)

GetOpenAICompatibleFileContent fetches file bytes via /files/{id}/content after normalizing the incoming id via validatedOpenAICompatibleFileID. The returned response always includes the normalized file ID.

func GetOpenAICompatibleFileContentWithPreparer 

func GetOpenAICompatibleFileContentWithPreparer(ctx context.Context, client *llmclient.Client, id string, prepare openAICompatibleRequestPreparer) (*core.FileContentResponse, error)

func GetOpenAICompatibleFileWithPreparer 

func GetOpenAICompatibleFileWithPreparer(ctx context.Context, client *llmclient.Client, id string, prepare openAICompatibleRequestPreparer) (*core.FileObject, error)

func HasResolvedProviderValue 

func HasResolvedProviderValue(value string) bool

HasResolvedProviderValue reports whether a provider-config field carries a usable string value. It returns false for empty/whitespace input and false when the value still contains a literal `${` substring — that signals an unresolved YAML environment-variable placeholder such as `${OPENAI_API_KEY}` which the env-substitution pass failed to fill in. Provider builders use this to drop providers whose credentials never resolved.

func IsValidClientRequestID 

func IsValidClientRequestID(id string) bool

IsValidClientRequestID reports whether id may be forwarded as a client request-ID header value: upstreams that accept one (OpenAI, OpenRouter, Azure) require printable ASCII and reject oversized values with a 400.

func ListOpenAICompatibleFiles 

func ListOpenAICompatibleFiles(ctx context.Context, client *llmclient.Client, purpose string, limit int, after string) (*core.FileListResponse, error)

ListOpenAICompatibleFiles lists files using OpenAI-compatible files API.

func ListOpenAICompatibleFilesWithPreparer 

func ListOpenAICompatibleFilesWithPreparer(ctx context.Context, client *llmclient.Client, purpose string, limit int, after string, prepare openAICompatibleRequestPreparer) (*core.FileListResponse, error)

func OpenAIRealtimeAttachURL 

func OpenAIRealtimeAttachURL(baseURL, callID string) (string, error)

OpenAIRealtimeAttachURL derives the websocket URL that attaches to an existing realtime call as a sideband channel: https://host/v1 -> wss://host/v1/realtime?call_id=... The call already owns a model, so no model parameter is sent.

func OpenAIRealtimeHTTPURL 

func OpenAIRealtimeHTTPURL(baseURL, endpoint string) (string, error)

OpenAIRealtimeHTTPURL derives an OpenAI-style realtime HTTP signaling URL from an HTTP(S) base URL: https://host/v1 + "calls" -> https://host/v1/realtime/calls. It is the HTTP sibling of OpenAIRealtimeURL for the WebRTC SDP exchange and client secret endpoints; ws/wss base schemes map back to http/https.

func OpenAIRealtimeURL 

func OpenAIRealtimeURL(baseURL, model string) (string, error)

OpenAIRealtimeURL derives an OpenAI-style realtime websocket URL from an HTTP(S) base URL: https://host/v1 -> wss://host/v1/realtime?model=... It maps the scheme to ws/wss and appends the realtime path and model query parameter.

It is shared by providers whose realtime endpoint mirrors OpenAI's exact shape (OpenAI, xAI). Providers whose realtime endpoint differs (e.g. Bailian's /api-ws/v1/realtime) build their own target instead.

func PaginatedEndpoint 

func PaginatedEndpoint(path string, limit int, cursorParam, cursor string) string

PaginatedEndpoint appends optional limit and pagination-cursor query parameters to an endpoint path. The cursor parameter name varies by provider (e.g. "after" for OpenAI-compatible APIs, "before_id" for Anthropic).

func PassthroughEndpoint 

func PassthroughEndpoint(endpoint string) string

PassthroughEndpoint normalizes a provider-relative passthrough endpoint into an absolute path fragment suitable for baseURL + endpoint request building.

func PassthroughEndpointPath 

func PassthroughEndpointPath(info *core.PassthroughRouteInfo) string

PassthroughEndpointPath returns the normalized path portion of a provider passthrough endpoint, preferring a semantic normalized endpoint when present.

func ResolveAPIVersion 

func ResolveAPIVersion(apiVersion, fallback string) string

ResolveAPIVersion returns the configured API version when present, otherwise the provider default.

func ResolveBaseURL 

func ResolveBaseURL(baseURL, fallback string) string

ResolveBaseURL returns the configured base URL when present, otherwise the provider default.

func ResponsesFunctionCallCallID 

func ResponsesFunctionCallCallID(callID string) string

ResponsesFunctionCallCallID returns the call id if present or generates one.

func ResponsesFunctionCallItemID 

func ResponsesFunctionCallItemID(callID string) string

ResponsesFunctionCallItemID returns a stable function-call item id.

func ResponsesViaChat 

func ResponsesViaChat(ctx context.Context, p ChatProvider, req *core.ResponsesRequest) (*core.ResponsesResponse, error)

ResponsesViaChat implements the Responses API by converting to/from Chat format.

func SetAuthHeaders 

func SetAuthHeaders(req *http.Request, apiKey string, cfg AuthHeaderConfig)

SetAuthHeaders applies cfg to req for the given API key. It is safe to use directly as an llmclient header hook or as CompatibleProviderConfig.SetHeaders.

func StreamResponsesViaChat 

func StreamResponsesViaChat(ctx context.Context, p ChatProvider, req *core.ResponsesRequest, providerName string) (io.ReadCloser, error)

StreamResponsesViaChat implements streaming Responses API by converting to/from Chat format.

Types

type AuthHeaderConfig 

type AuthHeaderConfig struct {
	// AuthHeader carries the credential. Defaults to "Authorization".
	AuthHeader string
	// AuthScheme prefixes the credential, e.g. "Bearer ". Empty for raw values.
	AuthScheme string
	// RequestIDHeader, when non-empty, forwards the context request ID under
	// this header name. When empty, no request ID is forwarded.
	RequestIDHeader string
	// ValidateRequestID, when set, gates request-ID forwarding (e.g. ASCII and
	// length checks required by some upstreams).
	ValidateRequestID func(string) bool
	// OptionalAPIKey skips the auth header entirely when the API key is empty,
	// for providers that allow unauthenticated access (e.g. local Ollama/vLLM).
	OptionalAPIKey bool
}

AuthHeaderConfig describes how a provider populates outbound request headers. It captures the few axes along which OpenAI-compatible providers differ so the shared logic (empty-key handling, request-ID forwarding, validation) lives in one place and each provider declares only its variations as data.

type CategoryCount 

type CategoryCount struct {
	Category    core.ModelCategory `json:"category"`
	DisplayName string             `json:"display_name"`
	Count       int                `json:"count"`
}

CategoryCount holds a model category and the number of models in it.

type ChatProvider 

type ChatProvider interface {
	ChatCompletion(ctx context.Context, req *core.ChatRequest) (*core.ChatResponse, error)
	StreamChatCompletion(ctx context.Context, req *core.ChatRequest) (io.ReadCloser, error)
}

ChatProvider is the minimal interface needed by the shared Responses-to-Chat adapter. Any provider that supports ChatCompletion and StreamChatCompletion can use the ResponsesViaChat and StreamResponsesViaChat helpers to implement the Responses API.

type CredentialField added in v0.1.60 

type CredentialField struct {
	Name     string
	Required bool
	// Advanced marks a field most operators leave at its default for this
	// provider type, so the form can fold it away behind a disclosure.
	Advanced bool
	// Options are the canonical values of an enumerated field, for a form to
	// offer as a choice. It is deliberately not a whitelist: providers accept
	// further spellings of each value (`useNativeAPI`, `NormalizeAuthType`,
	// ...), and rejecting those would make the admin API stricter than the
	// config.yaml and env vars it exists to replace. An empty value always
	// means "provider default", so it is never listed.
	Options []string
}

CredentialField describes one field of a provider type's credential form. Only the fields a type actually reads are described, so a dashboard can render a form that fits the provider instead of every field the wire format happens to carry.

type CredentialFieldError added in v0.1.60 

type CredentialFieldError struct {
	Field   string
	Message string
}

CredentialFieldError reports a credential row the operator has to fix, naming the credential field at fault so the admin API can point the dashboard at that input instead of at the form as a whole.

func (*CredentialFieldError) Error added in v0.1.60 

func (e *CredentialFieldError) Error() string

type CredentialSchema added in v0.1.60 

type CredentialSchema struct {
	Type           string
	DefaultBaseURL string
	Fields         []CredentialField
}

CredentialSchema is the credential form one provider type accepts.

func (CredentialSchema) Accepts added in v0.1.60 

func (s CredentialSchema) Accepts(name string) bool

Accepts reports whether this provider type uses the named credential field.

func (CredentialSchema) Field added in v0.1.60 

func (s CredentialSchema) Field(name string) (CredentialField, bool)

Field returns the schema entry for name, and whether the type accepts it.

type CredentialStore added in v0.1.56 

type CredentialStore interface {
	List(ctx context.Context) ([]ManagedProviderCredential, error)
	Get(ctx context.Context, name string) (*ManagedProviderCredential, error)
	Upsert(ctx context.Context, cred ManagedProviderCredential) error
	Delete(ctx context.Context, name string) error
	Close() error
}

CredentialStore persists admin-managed provider credentials.

type CredentialsResult added in v0.1.56 

type CredentialsResult struct {
	Service *CredentialsService
	Store   CredentialStore
	// contains filtered or unexported fields
}

CredentialsResult holds the initialized provider-credentials subsystem and any resources it owns.

func NewCredentialsStore added in v0.1.56 

func NewCredentialsStore(ctx context.Context, shared storage.Storage, factory *ProviderFactory, registry *ModelRegistry, declaredNames []string, resilience config.ResilienceConfig) (*CredentialsResult, error)

NewCredentialsStore creates the provider-credentials subsystem using an existing storage connection.

func (*CredentialsResult) Close added in v0.1.56 

func (r *CredentialsResult) Close() error

Close releases resources held by the provider-credentials subsystem.

type CredentialsService added in v0.1.56 

type CredentialsService struct {
	// contains filtered or unexported fields
}

CredentialsService merges declarative (config.yaml / env var) provider names with the admin-managed credential store and reconciles the result into the live factory/registry — the provider-credentials equivalent of mcpgateway.Service. Declarative names are read-only here (shadowed), mirroring every other config-store precedence rule in the gateway.

func NewCredentialsService added in v0.1.56 

func NewCredentialsService(ctx context.Context, factory *ProviderFactory, registry *ModelRegistry, store CredentialStore, declaredNames []string, resilience config.ResilienceConfig) (*CredentialsService, error)

NewCredentialsService builds the service and applies every currently stored, enabled, non-shadowed credential to the registry before returning.

func (*CredentialsService) CredentialSchema added in v0.1.60 

func (s *CredentialsService) CredentialSchema(providerType string) CredentialSchema

CredentialSchema returns one provider type's credential form.

func (*CredentialsService) CredentialSchemas added in v0.1.60 

func (s *CredentialsService) CredentialSchemas() []CredentialSchema

CredentialSchemas returns the credential form of every registered provider type, so the admin create/edit form can offer exactly the fields the selected type uses.

func (*CredentialsService) Delete added in v0.1.56 

func (s *CredentialsService) Delete(ctx context.Context, name string) error

Delete removes one admin-managed provider credential and unregisters it from the live registry.

func (*CredentialsService) Get added in v0.1.56 

func (s *CredentialsService) Get(ctx context.Context, name string) (*ManagedProviderCredential, error)

Get returns one admin-managed credential row.

func (*CredentialsService) IsManaged added in v0.1.56 

func (s *CredentialsService) IsManaged(name string) bool

IsManaged reports whether name is declared in config.yaml/env, making it read-only from the admin API's point of view.

func (*CredentialsService) List added in v0.1.56 

func (s *CredentialsService) List(ctx context.Context) ([]ManagedProviderCredential, error)

List returns every admin-managed credential row (secrets included; callers at the HTTP boundary are responsible for redaction).

func (*CredentialsService) RegisteredTypes added in v0.1.56 

func (s *CredentialsService) RegisteredTypes() []string

RegisteredTypes lists every provider type the factory can construct, for the admin create-form's type picker.

func (*CredentialsService) Reload added in v0.1.56 

func (s *CredentialsService) Reload(ctx context.Context) error

Reload re-registers every enabled, non-shadowed stored credential into the registry (pure in-memory work) and then kicks a non-blocking model-inventory fetch so the newly registered providers' models become routable without delaying the caller — the same async startup contract providers.Init uses for declarative providers, so adding the credentials store never turns gateway startup into a synchronous network round trip to every provider. Declarative providers are untouched: providers.Init already registered them.

func (*CredentialsService) Upsert added in v0.1.56 

func (s *CredentialsService) Upsert(ctx context.Context, cred ManagedProviderCredential) error

Upsert validates, persists, and hot-applies one admin-managed provider credential. A disabled row is persisted but unregistered from the live registry, the same end state as deleting it.

type DiscoveryConfig 

type DiscoveryConfig struct {
	DefaultBaseURL     string
	RequireBaseURL     bool
	AllowAPIKeyless    bool
	SupportsAPIVersion bool
	NameSeparator      string

	// CredentialFields declares the credential form of provider types that do
	// not fit the plain "API key against one endpoint" shape (Google's
	// project/service-account auth, an endpoint mode selector, ...), in display
	// order. Leave it nil to derive the form from the flags above; the model
	// list is always appended. See credentialSchema.
	CredentialFields []CredentialField
}

DiscoveryConfig describes how a provider participates in config resolution, and — from the same facts — which credential fields it accepts, so the admin form and env/YAML resolution can never disagree about what a type needs. Env var names are derived by convention from Registration.Type.

type InitResult 

type InitResult struct {
	Registry *ModelRegistry
	Router   *Router
	Cache    modelcache.Cache
	Factory  *ProviderFactory

	// ConfiguredProviders is the effective, admin-safe provider inventory keyed
	// by configured provider name.
	ConfiguredProviders []SanitizedProviderConfig

	// CredentialResolvedProviders is the env-merged, credential-filtered providers
	// map (same keys as Router). Keys match top-level providers YAML names.
	CredentialResolvedProviders map[string]config.RawProviderConfig
	// contains filtered or unexported fields
}

InitResult holds the initialized provider infrastructure and cleanup functions.

func Init 

func Init(ctx context.Context, result *config.LoadResult, factory *ProviderFactory) (*InitResult, error)

Init initializes the provider registry, cache, and router.

It performs:

  1. Provider config resolution (env var overlay, filtering, resilience merging)
  2. Cache initialization (local or Redis based on config)
  3. Provider instantiation and registration
  4. Async model loading (from cache first, then network refresh)
  5. Best-effort background model-list fetch (goroutine with ~45s timeout that calls modeldata.Fetch, registry.EnrichModels, and SaveToCache)
  6. Background refresh scheduling (interval from cfg.Cache.RefreshInterval)
  7. Router creation

The caller must call InitResult.Close() during shutdown.

func (*InitResult) Close 

func (r *InitResult) Close() error

Close releases all resources and stops background goroutines. Safe to call multiple times (but stopRefresh is only called once).

type Keyring 

type Keyring struct {
	// contains filtered or unexported fields
}

Keyring holds the API keys configured for a single provider instance and hands them out one at a time, round robin.

A provider is built once and serves every request, so the credential can no longer be a string captured at construction: it is resolved per outbound request by calling Next from the provider's header hook. One Keyring is shared by all of a provider's HTTP clients, so the rotation is even across every endpoint that provider serves.

The zero value is not useful; build one with NewKeyring. A nil *Keyring is safe to call and behaves as an empty ring, which lets keyless providers (Ollama, vLLM) and direct test constructors skip it entirely.

func NewKeyring 

func NewKeyring(keys ...string) *Keyring

NewKeyring returns a Keyring over keys, preserving order while dropping empty and duplicate entries. Duplicates are dropped so that a key repeated across `OPENAI_API_KEY` and `OPENAI_API_KEY_1` does not take a double share of the rotation. It returns nil when no usable key remains, so callers can treat "no credentials" and "no keyring" identically.

func NewKeyringWithSessionStickiness added in v0.1.67 

func NewKeyringWithSessionStickiness(sessionSticky bool, keys ...string) *Keyring

NewKeyringWithSessionStickiness builds a keyring whose identified sessions deterministically select one credential. Sessionless traffic always keeps the historical round-robin behavior.

func (*Keyring) Len 

func (k *Keyring) Len() int

Len reports how many distinct keys back the rotation.

func (*Keyring) Next 

func (k *Keyring) Next() string

Next returns the key to authenticate the next outbound request, advancing the rotation. It is safe for concurrent use. Next returns "" for an empty ring, matching the unconfigured-credential behaviour providers already handle (see AuthHeaderConfig.OptionalAPIKey).

Rotation advances per outbound HTTP request, which includes retries: a request retried after a 429 is re-sent under the next key rather than hammering the one that was just throttled.

func (*Keyring) NextForContext added in v0.1.67 

func (k *Keyring) NextForContext(ctx context.Context) string

NextForContext returns the key pinned to the request's GoModel session. When no session is present, or session stickiness was disabled for this provider, it advances the ordinary round-robin sequence.

func (*Keyring) NextForSession added in v0.1.67 

func (k *Keyring) NextForSession(sessionID string) string

NextForSession deterministically maps one non-empty session to one key using rendezvous hashing. Adding or removing a key therefore remaps only sessions assigned to the changed key, rather than invalidating every warm cache. The digest and credential material remain in-process and are never exposed.

func (*Keyring) Primary 

func (k *Keyring) Primary() string

Primary returns the first configured key without advancing the rotation. It is the key to use where a stable identity matters more than spreading load, and where an empty ring must stay empty.

func (*Keyring) Rotates 

func (k *Keyring) Rotates() bool

Rotates reports whether more than one key is configured. Identified sessions may still remain pinned to one of those keys when session stickiness is on.

type ManagedProviderCredential added in v0.1.56 

type ManagedProviderCredential struct {
	Name string
	Type string

	// APIKeys is the ordered credential rotation set. Keyless providers
	// (Ollama, vLLM) and Vertex/Bedrock (which authenticate a different way)
	// may leave this empty.
	APIKeys []string
	// SessionStickyKeys is nil for legacy/default rows and therefore means
	// enabled. A non-nil false value is the explicit dashboard/YAML opt-out.
	SessionStickyKeys *bool

	BaseURL                  string
	APIVersion               string
	Backend                  string
	AuthType                 string
	APIMode                  string
	VertexProject            string
	VertexLocation           string
	ServiceAccountFile       string
	ServiceAccountJSON       string
	ServiceAccountJSONBase64 string
	GCPScope                 string
	Models                   []string

	// Enabled controls whether this credential is applied to the running
	// registry. Disabling one keeps the row (and its keys) on file without
	// routing traffic to it — the same effect as deleting it, without losing
	// the configuration.
	Enabled bool

	CreatedAt time.Time
	UpdatedAt time.Time
}

ManagedProviderCredential is one admin-managed provider instance: the dashboard equivalent of a `providers.<name>:` entry in config.yaml. Name is the configured provider instance name (what shows up as the qualifier in "name/model" selectors); Type selects the registered adapter (openai, anthropic, ollama, vertex, ...).

type ModelInfo 

type ModelInfo struct {
	Model        core.Model
	Provider     core.Provider
	ProviderName string
	ProviderType string
}

ModelInfo holds information about a model and its provider

type ModelRegistry 

type ModelRegistry struct {
	// contains filtered or unexported fields
}

ModelRegistry manages the mapping of models to their providers. It fetches models from providers on startup and caches them in memory. Supports loading from a cache (local file or Redis) for instant startup.

func NewModelRegistry 

func NewModelRegistry() *ModelRegistry

NewModelRegistry creates a new model registry

func (*ModelRegistry) EnrichModels 

func (r *ModelRegistry) EnrichModels()

EnrichModels re-applies model list metadata to all currently registered models. Call this after SetModelList to update existing models with the new metadata. Holds the write lock for the entire operation and replaces published ModelInfo entries instead of mutating them in place so concurrent readers can safely keep using older snapshots after unlocking.

func (*ModelRegistry) FailedProviderNames 

func (r *ModelRegistry) FailedProviderNames() []string

FailedProviderNames returns configured provider names whose latest model refresh attempt or availability probe failed. The background recheck loop uses this to re-probe only the providers that are currently down. The availability side matters: a request-time refresh can fail the availability gate (marking the inventory stale) without ever attempting a model fetch, and such a provider must still be re-probed to detect its recovery.

func (*ModelRegistry) GetCategoryCounts 

func (r *ModelRegistry) GetCategoryCounts() []CategoryCount

GetCategoryCounts returns model counts per category, in display order. A model with multiple categories is counted in each.

func (*ModelRegistry) GetModel 

func (r *ModelRegistry) GetModel(model string) *ModelInfo

GetModel returns the registry-backed model info for the given model, or nil if not found. Callers must treat the returned data as read-only.

func (*ModelRegistry) GetModelMetadata 

func (r *ModelRegistry) GetModelMetadata(modelID string) *core.ModelMetadata

GetModelMetadata returns the metadata for a model, or nil if not found or not enriched.

func (*ModelRegistry) GetProvider 

func (r *ModelRegistry) GetProvider(model string) core.Provider

GetProvider returns the provider for the given model, or nil if not found

func (*ModelRegistry) GetProviderName 

func (r *ModelRegistry) GetProviderName(model string) string

GetProviderName returns the concrete configured provider instance name for the given model selector. Returns empty string if the model is not found.

func (*ModelRegistry) GetProviderNameForType 

func (r *ModelRegistry) GetProviderNameForType(providerType string) string

GetProviderNameForType returns the first registered configured provider name for the given provider type. This follows the same first-registered behavior used when provider-typed routes resolve a concrete provider instance.

func (*ModelRegistry) GetProviderType 

func (r *ModelRegistry) GetProviderType(model string) string

GetProviderType returns the provider type string for the given model. Returns empty string if the model is not found.

func (*ModelRegistry) GetProviderTypeForName 

func (r *ModelRegistry) GetProviderTypeForName(providerName string) string

GetProviderTypeForName returns the provider type for the given concrete configured provider instance name.

func (*ModelRegistry) Initialize 

func (r *ModelRegistry) Initialize(ctx context.Context) error

Initialize fetches models from all registered providers and populates the registry. This should be called on application startup.

func (*ModelRegistry) InitializeAsync 

func (r *ModelRegistry) InitializeAsync(ctx context.Context)

InitializeAsync starts model fetching in a background goroutine. It first loads any cached models for immediate availability, then refreshes from network. Returns immediately after loading cache. The background goroutine will update models and save to cache when network fetch completes.

func (*ModelRegistry) IsInitialized 

func (r *ModelRegistry) IsInitialized() bool

IsInitialized returns true if at least one successful network fetch has completed. This can be used to check if the registry has fresh data or is only serving from cache.

func (*ModelRegistry) ListModels 

func (r *ModelRegistry) ListModels() []core.Model

ListModels returns all models in the registry, sorted by model ID for consistent ordering. The sorted slice is cached and rebuilt only when the underlying models change. Returns a defensive copy so callers cannot mutate the internal cache.

func (*ModelRegistry) ListModelsWithProvider 

func (r *ModelRegistry) ListModelsWithProvider() []ModelWithProvider

ListModelsWithProvider returns all provider-backed models with provider metadata, sorted by public selector. The sorted slice is cached and rebuilt only when the underlying models change. Returns a defensive copy so callers cannot mutate the internal cache.

func (*ModelRegistry) ListModelsWithProviderByCategory 

func (r *ModelRegistry) ListModelsWithProviderByCategory(category core.ModelCategory) []ModelWithProvider

ListModelsWithProviderByCategory returns provider-backed models filtered by category, sorted by public selector. If category is CategoryAll, returns all models (same as ListModelsWithProvider). Results for known categories are cached and rebuilt only when the underlying models change. Returns a defensive copy so callers cannot mutate the internal cache.

func (*ModelRegistry) ListPublicModels 

func (r *ModelRegistry) ListPublicModels() []core.Model

ListPublicModels returns all provider-backed models as public selectors in providerName/modelID form, sorted by public model ID. Models the owning provider cannot actually serve (audio-only models on providers without audio support) are not advertised.

func (*ModelRegistry) LoadFromCache 

func (r *ModelRegistry) LoadFromCache(ctx context.Context) (int, error)

LoadFromCache loads the model list from the cache backend. Returns the number of models loaded and any error encountered.

func (*ModelRegistry) LookupModel 

func (r *ModelRegistry) LookupModel(model string) (*core.Model, bool)

LookupModel returns a shallow copy of the concrete model for the given selector. Qualified selectors use the configured provider name prefix when present.

func (*ModelRegistry) ModelAvailable 

func (r *ModelRegistry) ModelAvailable(model string) bool

ModelAvailable reports whether the model is registered AND its provider's inventory is fresh (latest refresh succeeded). Virtual-model load balancing uses this to skip providers whose upstream is failing, while Supports keeps resolving stale models so direct requests still reach the provider and fail with an honest 502/503 instead of "model not found".

func (*ModelRegistry) ModelCount 

func (r *ModelRegistry) ModelCount() int

ModelCount returns the number of registered models

func (*ModelRegistry) ProviderByName 

func (r *ModelRegistry) ProviderByName(providerName string) core.Provider

ProviderByName returns the registered provider for a configured provider instance name.

func (*ModelRegistry) ProviderByType 

func (r *ModelRegistry) ProviderByType(providerType string) core.Provider

ProviderByType returns the first registered provider for the given provider type. This lookup is independent of discovered models so provider-typed routes keep working even when a provider currently exposes zero models.

func (*ModelRegistry) ProviderCount 

func (r *ModelRegistry) ProviderCount() int

ProviderCount returns the number of registered providers

func (*ModelRegistry) ProviderNames 

func (r *ModelRegistry) ProviderNames() []string

ProviderNames returns the configured provider instance names in registration order.

func (*ModelRegistry) ProviderRuntimeSnapshots 

func (r *ModelRegistry) ProviderRuntimeSnapshots() []ProviderRuntimeSnapshot

ProviderRuntimeSnapshots returns runtime diagnostics for configured providers keyed by configured provider name.

func (*ModelRegistry) ProviderTypes 

func (r *ModelRegistry) ProviderTypes() []string

ProviderTypes returns the unique registered provider types in sorted order. This inventory is independent of discovered models.

func (*ModelRegistry) RecordAvailabilityCheck 

func (r *ModelRegistry) RecordAvailabilityCheck(providerName string, err error)

RecordAvailabilityCheck stores the latest startup or explicit availability probe result for a configured provider name.

func (*ModelRegistry) Refresh 

func (r *ModelRegistry) Refresh(ctx context.Context) error

Refresh updates the model registry by fetching fresh model lists from providers. This can be called periodically to keep the registry up to date.

func (*ModelRegistry) RefreshModelList 

func (r *ModelRegistry) RefreshModelList(ctx context.Context, url string) (int, error)

RefreshModelList fetches the external model metadata list and re-enriches all currently registered models. It does not persist the model cache; callers that want durable startup data should call SaveToCache after this succeeds.

func (*ModelRegistry) RefreshProviderModels 

func (r *ModelRegistry) RefreshProviderModels(ctx context.Context, providerSelector string) (int, error)

RefreshProviderModels refreshes model inventory for a configured provider name, or all providers matching a provider type. It is intended for request-time recovery when startup discovery failed before a provider was reachable.

func (*ModelRegistry) RegisterProvider 

func (r *ModelRegistry) RegisterProvider(provider core.Provider)

RegisterProvider adds a provider to the registry

func (*ModelRegistry) RegisterProviderWithNameAndType 

func (r *ModelRegistry) RegisterProviderWithNameAndType(provider core.Provider, providerName, providerType string)

RegisterProviderWithNameAndType adds a provider with a configured provider instance name and type. Name is used for unambiguous provider/model selection (e.g. "provider/model") and cache persistence.

func (*ModelRegistry) RegisterProviderWithType 

func (r *ModelRegistry) RegisterProviderWithType(provider core.Provider, providerType string)

RegisterProviderWithType adds a provider to the registry with its type string. The type is used for cache persistence to re-associate models with providers on startup.

func (*ModelRegistry) ResolveMetadata 

func (r *ModelRegistry) ResolveMetadata(providerType, modelID string) *core.ModelMetadata

ResolveMetadata resolves metadata for a model directly via the stored model list, bypassing the registry key lookup. This handles cases where the usage DB stores a response model ID (e.g., "gpt-4o-2024-08-06") that differs from the registry key (e.g., "gpt-4o") by using the reverse index in the model list.

func (*ModelRegistry) ResolvePricing 

func (r *ModelRegistry) ResolvePricing(model, providerType string) *core.ModelPricing

ResolvePricing returns the pricing metadata for a model, trying the registry first and falling back to a reverse-index lookup via the model list. Returns nil if no pricing is available.

func (*ModelRegistry) ResolveProviderSelector 

func (r *ModelRegistry) ResolveProviderSelector(segment, modelID string) (core.ModelSelector, bool)

ResolveProviderSelector resolves a qualified "<segment>/<modelID>" selector, where segment is a provider instance name or a provider type, to the concrete provider-name-qualified selector. Provider-name matches take precedence over provider-type matches, mirroring catalog-scan resolution. Returns ok=false when the segment+model pair is not a direct name/type match so callers can fall back to slower resolution for raw slash-shaped IDs and other edge cases.

This is O(1) and exists so the per-request routing path does not copy and linearly scan the entire model catalog.

func (*ModelRegistry) SaveToCache 

func (r *ModelRegistry) SaveToCache(ctx context.Context) error

SaveToCache saves the current model list to the cache backend.

func (*ModelRegistry) SetCache 

func (r *ModelRegistry) SetCache(c modelcache.Cache)

SetCache sets the cache backend for persistent model storage. The cache can be a local file-based cache or a Redis cache.

func (*ModelRegistry) SetConfiguredProviderModelsMode 

func (r *ModelRegistry) SetConfiguredProviderModelsMode(mode config.ConfiguredProviderModelsMode)

SetConfiguredProviderModelsMode controls how configured provider model lists affect the final registry inventory.

func (*ModelRegistry) SetModelList 

func (r *ModelRegistry) SetModelList(list *modeldata.ModelList, raw json.RawMessage)

SetModelList stores the parsed model list and its raw bytes for cache persistence.

func (*ModelRegistry) SetProviderConfiguredModels 

func (r *ModelRegistry) SetProviderConfiguredModels(providerName string, models []string)

SetProviderConfiguredModels records the explicit model inventory declared for a configured provider instance. Call with an empty/nil slice to clear it.

func (*ModelRegistry) SetProviderMetadataOverrides 

func (r *ModelRegistry) SetProviderMetadataOverrides(providerName string, overrides map[string]*core.ModelMetadata)

SetProviderMetadataOverrides records per-model metadata overrides declared in config.yaml for the given provider instance name. Overrides are merged onto remote-registry enrichment each time the registry re-enriches its models.

Call with an empty/nil map to clear any prior overrides for that provider.

func (*ModelRegistry) StartBackgroundRefresh 

func (r *ModelRegistry) StartBackgroundRefresh(interval, recheckInterval time.Duration, modelListURL string) func()

StartBackgroundRefresh starts a goroutine that periodically refreshes the model registry. If modelListURL is non-empty, the model list is also re-fetched on each tick. A positive recheckInterval additionally re-probes only the providers whose latest refresh failed, so outages and recoveries are detected without waiting for the next full refresh. The returned stop function is blocking: it cancels the refresh loop and waits for the goroutine to exit before returning, so callers should expect it to block during shutdown until any in-flight refresh work unwinds.

func (*ModelRegistry) Supports 

func (r *ModelRegistry) Supports(model string) bool

Supports returns true if the registry has a provider for the given model

func (*ModelRegistry) UnregisterProvider added in v0.1.56 

func (r *ModelRegistry) UnregisterProvider(providerName string)

UnregisterProvider removes every provider instance registered under name (normally at most one) and immediately removes its models from the live catalog. Callers may still Refresh afterward to update the remaining providers, but a failed or slow refresh must not leave deleted-provider models visible. Safe to call for a name that was never registered.

type ModelWithProvider 

type ModelWithProvider struct {
	Model        core.Model `json:"model"`
	ProviderType string     `json:"provider_type"`
	ProviderName string     `json:"provider_name"`
	Selector     string     `json:"selector"`
}

ModelWithProvider holds a model alongside provider metadata and its public selector.

type MongoDBCredentialStore added in v0.1.56 

type MongoDBCredentialStore struct {
	// contains filtered or unexported fields
}

MongoDBCredentialStore stores admin-managed provider credentials in MongoDB.

func NewMongoDBCredentialStore added in v0.1.56 

func NewMongoDBCredentialStore(database *mongo.Database) (*MongoDBCredentialStore, error)

NewMongoDBCredentialStore creates collection indexes if needed.

func (*MongoDBCredentialStore) Close added in v0.1.56 

func (s *MongoDBCredentialStore) Close() error

func (*MongoDBCredentialStore) Delete added in v0.1.56 

func (s *MongoDBCredentialStore) Delete(ctx context.Context, name string) error

func (*MongoDBCredentialStore) Get added in v0.1.56 

func (s *MongoDBCredentialStore) Get(ctx context.Context, name string) (*ManagedProviderCredential, error)

func (*MongoDBCredentialStore) List added in v0.1.56 

func (s *MongoDBCredentialStore) List(ctx context.Context) ([]ManagedProviderCredential, error)

func (*MongoDBCredentialStore) Upsert added in v0.1.56 

func (s *MongoDBCredentialStore) Upsert(ctx context.Context, cred ManagedProviderCredential) error

type OpenAIResponsesStreamConverter 

type OpenAIResponsesStreamConverter struct {
	// contains filtered or unexported fields
}

OpenAIResponsesStreamConverter wraps an OpenAI-compatible SSE stream and converts it to Responses API format. Used by providers that have OpenAI-compatible streaming (Groq, Gemini, etc.)

func NewOpenAIResponsesStreamConverter 

func NewOpenAIResponsesStreamConverter(reader io.ReadCloser, model, provider string) *OpenAIResponsesStreamConverter

NewOpenAIResponsesStreamConverter creates a new converter that transforms OpenAI-format SSE streams to Responses API format.

func (*OpenAIResponsesStreamConverter) Close 

func (sc *OpenAIResponsesStreamConverter) Close() error

func (*OpenAIResponsesStreamConverter) Read 

func (sc *OpenAIResponsesStreamConverter) Read(p []byte) (n int, err error)

type PassthroughEndpointSemantics 

type PassthroughEndpointSemantics struct {
	Operation string
	AuditPath string
}

PassthroughEndpointSemantics names the semantic operation and audit path for one provider passthrough endpoint.

type ProviderConfig 

type ProviderConfig struct {
	Type string
	// APIKey is the provider's primary credential: the first entry of APIKeys,
	// or "" for keyless providers. Prefer APIKeys for anything that
	// authenticates a request, so rotation is honoured.
	APIKey string
	// APIKeys is the provider's full, ordered, de-duplicated key set. Identified
	// sessions stay on one key when SessionStickyKeys is true; sessionless
	// traffic rotates round robin.
	APIKeys                  []string
	SessionStickyKeys        bool
	BaseURL                  string
	APIVersion               string
	Backend                  string
	AuthType                 string
	APIMode                  string
	VertexProject            string
	VertexLocation           string
	ServiceAccountFile       string
	ServiceAccountJSON       string
	ServiceAccountJSONBase64 string
	GCPScope                 string
	Models                   []string
	// ModelMetadataOverrides holds operator-supplied metadata keyed by raw model
	// ID (as it appears in the provider's /models response). The registry merges
	// these onto remote-registry metadata after enrichment; non-zero fields here
	// win. Empty/nil when no per-model metadata is declared in YAML.
	ModelMetadataOverrides map[string]*core.ModelMetadata
	Resilience             config.ResilienceConfig
}

ProviderConfig holds the fully resolved provider configuration after merging global defaults with per-provider overrides.

type ProviderConstructor 

type ProviderConstructor func(cfg ProviderConfig, opts ProviderOptions) core.Provider

ProviderConstructor is the constructor signature for providers.

type ProviderFactory 

type ProviderFactory struct {
	// contains filtered or unexported fields
}

ProviderFactory manages provider registration and creation.

func NewProviderFactory 

func NewProviderFactory() *ProviderFactory

NewProviderFactory creates a new provider factory instance.

func (*ProviderFactory) Add 

func (f *ProviderFactory) Add(reg Registration)

Add adds a provider constructor to the factory. Panics if reg.Type is empty or reg.New is nil — both are programming errors caught at startup, not runtime conditions.

func (*ProviderFactory) AddHooks 

func (f *ProviderFactory) AddHooks(hooks llmclient.Hooks)

AddHooks composes additional hooks with any already configured, affecting providers created after the call.

func (*ProviderFactory) Create 

func (f *ProviderFactory) Create(cfg ProviderConfig) (core.Provider, error)

Create instantiates a provider based on its resolved configuration.

func (*ProviderFactory) CredentialSchemas added in v0.1.60 

func (f *ProviderFactory) CredentialSchemas() []CredentialSchema

CredentialSchemas returns the credential form of every registered provider type, ordered by type name.

func (*ProviderFactory) PassthroughSemanticEnrichers 

func (f *ProviderFactory) PassthroughSemanticEnrichers() []core.PassthroughSemanticEnricher

PassthroughSemanticEnrichers returns registered passthrough semantic enrichers in deterministic provider-type order.

func (*ProviderFactory) RegisteredTypes 

func (f *ProviderFactory) RegisteredTypes() []string

RegisteredTypes returns a list of all registered provider types.

func (*ProviderFactory) SetHooks 

func (f *ProviderFactory) SetHooks(hooks llmclient.Hooks)

SetHooks configures observability hooks for all providers created by this factory.

type ProviderOptions 

type ProviderOptions struct {
	Hooks      llmclient.Hooks
	Models     []string
	Resilience config.ResilienceConfig
	// Keys carries every API key configured for this provider instance. It is
	// nil for keyless providers and for constructors invoked outside the
	// factory; use the Keyring method rather than reading it directly.
	Keys *Keyring
}

ProviderOptions bundles runtime settings passed from the factory to provider constructors.

func (ProviderOptions) Keyring 

func (o ProviderOptions) Keyring(apiKey string) *Keyring

Keyring returns the key source a provider should authenticate with, falling back to a single-key ring over apiKey when the factory supplied none. Every provider constructor takes an API key and ProviderOptions, so this one call gives a provider rotation support without changing its signature, and keeps constructors invoked outside the factory (tests, the NewWithHTTPClient variants) working unchanged.

type ProviderRuntimeSnapshot 

type ProviderRuntimeSnapshot struct {
	Name                    string     `json:"name"`
	Type                    string     `json:"type"`
	Registered              bool       `json:"registered"`
	RegistryInitialized     bool       `json:"registry_initialized"`
	DiscoveredModelCount    int        `json:"discovered_model_count"`
	UsingCachedModels       bool       `json:"using_cached_models"`
	LastModelFetchAt        *time.Time `json:"last_model_fetch_at,omitempty"`
	LastModelFetchSuccessAt *time.Time `json:"last_model_fetch_success_at,omitempty"`
	LastModelFetchError     string     `json:"last_model_fetch_error,omitempty"`
	LastAvailabilityCheckAt *time.Time `json:"last_availability_check_at,omitempty"`
	LastAvailabilityOKAt    *time.Time `json:"last_availability_ok_at,omitempty"`
	LastAvailabilityError   string     `json:"last_availability_error,omitempty"`
	InventoryStale          bool       `json:"inventory_stale,omitempty"`
}

ProviderRuntimeSnapshot describes runtime diagnostics for a configured provider.

type Registration 

type Registration struct {
	Type                        string
	New                         ProviderConstructor
	PassthroughSemanticEnricher core.PassthroughSemanticEnricher
	Discovery                   DiscoveryConfig
}

Registration contains metadata for registering a provider with the factory.

type ResponsesOutputEventState 

type ResponsesOutputEventState struct {
	// contains filtered or unexported fields
}

ResponsesOutputEventState manages assistant/tool output items for Responses streams.

func NewResponsesOutputEventState 

func NewResponsesOutputEventState(responseID string) *ResponsesOutputEventState

NewResponsesOutputEventState creates a new Responses output-item state manager.

func (*ResponsesOutputEventState) AppendAssistantText 

func (s *ResponsesOutputEventState) AppendAssistantText(text string)

AppendAssistantText appends assistant text content to the output item buffer.

func (*ResponsesOutputEventState) AppendReasoningDelta added in v0.1.67 

func (s *ResponsesOutputEventState) AppendReasoningDelta(outputIndex int, delta string) string

AppendReasoningDelta starts the reasoning item if needed and emits a raw reasoning-text delta.

func (*ResponsesOutputEventState) AssistantDone 

func (s *ResponsesOutputEventState) AssistantDone() bool

AssistantDone reports whether the assistant output item has been completed.

func (*ResponsesOutputEventState) AssistantMessageItem 

func (s *ResponsesOutputEventState) AssistantMessageItem(status string, includeContent bool) map[string]any

AssistantMessageItem renders the assistant message output item payload.

func (*ResponsesOutputEventState) AssistantReserved 

func (s *ResponsesOutputEventState) AssistantReserved() bool

AssistantReserved reports whether the assistant output item has been reserved.

func (*ResponsesOutputEventState) AssistantStarted 

func (s *ResponsesOutputEventState) AssistantStarted() bool

AssistantStarted reports whether the assistant output item has been emitted.

func (*ResponsesOutputEventState) CompleteAssistantOutput 

func (s *ResponsesOutputEventState) CompleteAssistantOutput(outputIndex int) string

CompleteAssistantOutput emits the assistant message output_item.done event once.

func (*ResponsesOutputEventState) CompleteReasoningOutput added in v0.1.67 

func (s *ResponsesOutputEventState) CompleteReasoningOutput(outputIndex int) string

CompleteReasoningOutput emits the raw reasoning completion and output_item.done events once.

func (*ResponsesOutputEventState) CompleteToolCall 

func (s *ResponsesOutputEventState) CompleteToolCall(state *ResponsesOutputToolCallState, includePlaceholder bool) string

CompleteToolCall emits the argument completion and output_item.done events once.

func (*ResponsesOutputEventState) ReasoningDone added in v0.1.67 

func (s *ResponsesOutputEventState) ReasoningDone() bool

ReasoningDone reports whether the reasoning output item has been completed.

func (*ResponsesOutputEventState) ReasoningItem added in v0.1.67 

func (s *ResponsesOutputEventState) ReasoningItem(status string, includeContent bool) map[string]any

ReasoningItem renders a raw reasoning output item. Provider reasoning_content is chain-of-thought text, not an OpenAI-generated summary, so it belongs in content while summary remains empty.

func (*ResponsesOutputEventState) ReasoningReserved added in v0.1.67 

func (s *ResponsesOutputEventState) ReasoningReserved() bool

ReasoningReserved reports whether a reasoning output item has been reserved.

func (*ResponsesOutputEventState) RenderToolCallItem 

func (s *ResponsesOutputEventState) RenderToolCallItem(state *ResponsesOutputToolCallState, status string, includePlaceholder bool) map[string]any

RenderToolCallItem renders a function_call output item payload.

func (*ResponsesOutputEventState) ReserveAssistant 

func (s *ResponsesOutputEventState) ReserveAssistant()

ReserveAssistant marks that the assistant message output item occupies index 0.

func (*ResponsesOutputEventState) ReserveReasoning added in v0.1.67 

func (s *ResponsesOutputEventState) ReserveReasoning()

ReserveReasoning marks that a reasoning output item occupies index 0, ahead of the assistant message and any tool calls.

func (*ResponsesOutputEventState) StartAssistantOutput 

func (s *ResponsesOutputEventState) StartAssistantOutput(outputIndex int) string

StartAssistantOutput emits the assistant message output_item.added event once.

func (*ResponsesOutputEventState) StartReasoningOutput added in v0.1.67 

func (s *ResponsesOutputEventState) StartReasoningOutput(outputIndex int) string

StartReasoningOutput emits the reasoning output_item.added event once before any reasoning_text delta references the item.

func (*ResponsesOutputEventState) StartToolCall 

func (s *ResponsesOutputEventState) StartToolCall(state *ResponsesOutputToolCallState, includePlaceholder bool) string

StartToolCall emits the function_call output_item.added event once the item metadata is available.

func (*ResponsesOutputEventState) ToolCallArguments 

func (s *ResponsesOutputEventState) ToolCallArguments(state *ResponsesOutputToolCallState) string

ToolCallArguments returns the serialized argument payload for a function_call item.

func (*ResponsesOutputEventState) WriteEvent 

func (s *ResponsesOutputEventState) WriteEvent(eventName string, payload map[string]any) string

WriteEvent renders one SSE event in Responses API format.

type ResponsesOutputToolCallState 

type ResponsesOutputToolCallState struct {
	ItemID            string
	CallID            string
	Name              string
	OutputIndex       int
	Arguments         strings.Builder
	Started           bool
	Completed         bool
	PlaceholderObject bool
}

ResponsesOutputToolCallState tracks one function_call item in a Responses stream.

type Router 

type Router struct {
	// contains filtered or unexported fields
}

Router routes requests to the appropriate provider based on the model lookup. It uses a dynamic model-to-provider mapping that is populated at startup by fetching available models from each provider's /models endpoint.

func NewRouter 

func NewRouter(lookup core.ModelLookup) (*Router, error)

NewRouter creates a new provider router with a model lookup. The lookup must be initialized (via Initialize() or LoadFromCache()) before using the router. Returns an error if the lookup is nil.

func (*Router) CancelBatch 

func (r *Router) CancelBatch(ctx context.Context, providerType, id string) (*core.BatchResponse, error)

CancelBatch routes native batch cancellation to a provider type.

func (*Router) CancelResponse 

func (r *Router) CancelResponse(ctx context.Context, providerType, id string) (*core.ResponsesResponse, error)

CancelResponse routes native response cancellation to a provider type.

func (*Router) ChatCompletion 

func (r *Router) ChatCompletion(ctx context.Context, req *core.ChatRequest) (*core.ChatResponse, error)

ChatCompletion routes the request to the appropriate provider. Returns ErrRegistryNotInitialized if the lookup has no models loaded.

func (*Router) ClearBatchResultHints 

func (r *Router) ClearBatchResultHints(providerType, batchID string)

ClearBatchResultHints clears transient provider-side batch result hints once they have been persisted by the gateway.

func (*Router) CompactResponse 

func (r *Router) CompactResponse(ctx context.Context, providerType string, req *core.ResponsesRequest) (*core.ResponseCompactResponse, error)

CompactResponse routes native response compaction to a provider type.

func (*Router) CountResponseInputTokens 

func (r *Router) CountResponseInputTokens(ctx context.Context, providerType string, req *core.ResponsesRequest) (*core.ResponseInputTokensResponse, error)

CountResponseInputTokens routes native response input token counting to a provider type.

func (*Router) CreateBatch 

func (r *Router) CreateBatch(ctx context.Context, providerType string, req *core.BatchRequest) (*core.BatchResponse, error)

CreateBatch routes native batch creation to a provider type.

func (*Router) CreateBatchWithHints 

func (r *Router) CreateBatchWithHints(ctx context.Context, providerType string, req *core.BatchRequest) (*core.BatchResponse, map[string]string, error)

CreateBatchWithHints routes native batch creation and returns any provider batch-result shaping hints that need gateway persistence.

func (*Router) CreateFile 

func (r *Router) CreateFile(ctx context.Context, providerType string, req *core.FileCreateRequest) (*core.FileObject, error)

CreateFile routes file upload to a provider type.

func (*Router) CreateSpeech 

func (r *Router) CreateSpeech(ctx context.Context, req *core.AudioSpeechRequest) (*core.AudioResponse, error)

CreateSpeech routes a text-to-speech request to the provider that owns the model.

func (*Router) CreateTranscription 

func (r *Router) CreateTranscription(ctx context.Context, req *core.AudioTranscriptionRequest) (*core.AudioResponse, error)

CreateTranscription routes a speech-to-text request to the provider that owns the model.

func (*Router) DeleteBatch added in v0.1.53 

func (r *Router) DeleteBatch(ctx context.Context, providerType, id string) error

DeleteBatch routes native batch deletion to a provider type. It reports core.ErrNativeBatchDeleteUnsupported for providers whose upstream batch API has no delete operation, so callers can fall back to gateway-local deletion.

func (*Router) DeleteFile 

func (r *Router) DeleteFile(ctx context.Context, providerType, id string) (*core.FileDeleteResponse, error)

DeleteFile routes file deletion to a provider type.

func (*Router) DeleteResponse 

func (r *Router) DeleteResponse(ctx context.Context, providerType, id string) (*core.ResponseDeleteResponse, error)

DeleteResponse routes native response deletion to a provider type.

func (*Router) Embeddings 

func (r *Router) Embeddings(ctx context.Context, req *core.EmbeddingRequest) (*core.EmbeddingResponse, error)

Embeddings routes the embeddings request to the appropriate provider.

func (*Router) GetBatch 

func (r *Router) GetBatch(ctx context.Context, providerType, id string) (*core.BatchResponse, error)

GetBatch routes native batch lookup to a provider type.

func (*Router) GetBatchResults 

func (r *Router) GetBatchResults(ctx context.Context, providerType, id string) (*core.BatchResultsResponse, error)

GetBatchResults routes native batch results lookup to a provider type.

func (*Router) GetBatchResultsWithHints 

func (r *Router) GetBatchResultsWithHints(ctx context.Context, providerType, id string, endpointByCustomID map[string]string) (*core.BatchResultsResponse, error)

GetBatchResultsWithHints routes native batch results lookup with persisted per-item endpoint hints when the provider supports them.

func (*Router) GetFile 

func (r *Router) GetFile(ctx context.Context, providerType, id string) (*core.FileObject, error)

GetFile routes file retrieval to a provider type.

func (*Router) GetFileContent 

func (r *Router) GetFileContent(ctx context.Context, providerType, id string) (*core.FileContentResponse, error)

GetFileContent routes file content retrieval to a provider type.

func (*Router) GetProviderName 

func (r *Router) GetProviderName(model string) string

GetProviderName returns the concrete configured provider instance name for the given model selector. Returns empty string when unavailable.

func (*Router) GetProviderNameForType 

func (r *Router) GetProviderNameForType(providerType string) string

GetProviderNameForType returns the concrete configured provider instance name chosen for a provider-typed route.

func (*Router) GetProviderType 

func (r *Router) GetProviderType(model string) string

GetProviderType returns the provider type string for the given model. Returns empty string if the model is not found.

func (*Router) GetProviderTypeForName 

func (r *Router) GetProviderTypeForName(providerName string) string

GetProviderTypeForName returns the provider type for a concrete configured provider instance name.

func (*Router) GetResponse 

func (r *Router) GetResponse(ctx context.Context, providerType, id string, params core.ResponseRetrieveParams) (*core.ResponsesResponse, error)

GetResponse routes native response retrieval to a provider type.

func (*Router) ListBatches 

func (r *Router) ListBatches(ctx context.Context, providerType string, limit int, after string) (*core.BatchListResponse, error)

ListBatches routes native batch listing to a provider type.

func (*Router) ListFiles 

func (r *Router) ListFiles(ctx context.Context, providerType, purpose string, limit int, after string) (*core.FileListResponse, error)

ListFiles routes file listing to a provider type.

func (*Router) ListModels 

func (r *Router) ListModels(_ context.Context) (*core.ModelsResponse, error)

ListModels returns all models from the lookup. Returns ErrRegistryNotInitialized if the lookup has no models loaded.

func (*Router) ListResponseInputItems 

func (r *Router) ListResponseInputItems(ctx context.Context, providerType, id string, params core.ResponseInputItemsParams) (*core.ResponseInputItemListResponse, error)

ListResponseInputItems routes native response input item listing to a provider type.

func (*Router) ModelCount 

func (r *Router) ModelCount() int

ModelCount returns the number of models currently loaded into the router lookup.

func (*Router) NativeBatchProviderTypes 

func (r *Router) NativeBatchProviderTypes() []string

NativeBatchProviderTypes returns the registered provider types that support native batch operations.

func (*Router) NativeFileProviderTypes 

func (r *Router) NativeFileProviderTypes() []string

NativeFileProviderTypes returns the registered provider types that support native file operations.

func (*Router) NativeResponseProviderTypes 

func (r *Router) NativeResponseProviderTypes() []string

NativeResponseProviderTypes returns the registered provider types that support native Responses lifecycle operations.

func (*Router) Passthrough 

func (r *Router) Passthrough(ctx context.Context, providerType string, req *core.PassthroughRequest) (*core.PassthroughResponse, error)

Passthrough routes an opaque provider-native request by provider type. If req.ProviderName is set, routing prefers the named provider instance over the first registered provider of the given type.

func (*Router) RealtimeCallTarget 

func (r *Router) RealtimeCallTarget(ctx context.Context, req *core.RealtimeRequest) (*core.RealtimeHTTPTarget, error)

RealtimeCallTarget resolves the upstream HTTP endpoint for the WebRTC SDP exchange, requiring the model's provider to implement core.RealtimeCallProvider.

func (*Router) RealtimeClientSecretTarget 

func (r *Router) RealtimeClientSecretTarget(ctx context.Context, req *core.RealtimeRequest) (*core.RealtimeHTTPTarget, error)

RealtimeClientSecretTarget resolves the upstream HTTP endpoint for minting ephemeral realtime client secrets.

func (*Router) RealtimeTarget 

func (r *Router) RealtimeTarget(ctx context.Context, req *core.RealtimeRequest) (*core.RealtimeTarget, error)

RealtimeTarget resolves the upstream realtime websocket for the model's owning provider, requiring it to implement core.RealtimeProvider. It mirrors the audio routing: resolve the model, narrow to the capability, and forward the bare provider model id.

func (*Router) RefreshProviderModels 

func (r *Router) RefreshProviderModels(ctx context.Context, providerSelector string) (int, error)

RefreshProviderModels refreshes a configured provider's model inventory when the backing lookup supports request-time provider refreshes.

func (*Router) ResolveModel 

func (r *Router) ResolveModel(requested core.RequestedModelSelector) (core.ModelSelector, bool, error)

ResolveModel canonicalizes a requested selector into the concrete provider-name-qualified selector used for execution.

Resolution precedence is:

  1. configured provider name + model ID
  2. provider type + model ID
  3. raw slash-shaped model ID (only when provider was not explicit)
  4. default normalization fallback

func (*Router) Responses 

func (r *Router) Responses(ctx context.Context, req *core.ResponsesRequest) (*core.ResponsesResponse, error)

Responses routes the Responses API request to the appropriate provider. Returns ErrRegistryNotInitialized if the lookup has no models loaded.

func (*Router) StreamChatCompletion 

func (r *Router) StreamChatCompletion(ctx context.Context, req *core.ChatRequest) (io.ReadCloser, error)

StreamChatCompletion routes the streaming request to the appropriate provider. Returns ErrRegistryNotInitialized if the lookup has no models loaded.

func (*Router) StreamResponses 

func (r *Router) StreamResponses(ctx context.Context, req *core.ResponsesRequest) (io.ReadCloser, error)

StreamResponses routes the streaming Responses API request to the appropriate provider. Returns ErrRegistryNotInitialized if the lookup has no models loaded.

func (*Router) Supports 

func (r *Router) Supports(model string) bool

Supports returns true if any provider supports the given model. Returns false if the lookup has no models loaded.

type SQLCredentialStore added in v0.1.60 

type SQLCredentialStore struct {
	// contains filtered or unexported fields
}

SQLCredentialStore stores admin-managed provider credentials in a SQL database.

func NewSQLCredentialStore added in v0.1.60 

func NewSQLCredentialStore(ctx context.Context, db sqlx.DB) (*SQLCredentialStore, error)

NewSQLCredentialStore creates the provider_credentials table and indexes if needed.

func (*SQLCredentialStore) Close added in v0.1.60 

func (s *SQLCredentialStore) Close() error

func (*SQLCredentialStore) Delete added in v0.1.60 

func (s *SQLCredentialStore) Delete(ctx context.Context, name string) error

func (*SQLCredentialStore) Get added in v0.1.60 

func (s *SQLCredentialStore) Get(ctx context.Context, name string) (*ManagedProviderCredential, error)

func (*SQLCredentialStore) List added in v0.1.60 

func (s *SQLCredentialStore) List(ctx context.Context) ([]ManagedProviderCredential, error)

func (*SQLCredentialStore) Upsert added in v0.1.60 

func (s *SQLCredentialStore) Upsert(ctx context.Context, cred ManagedProviderCredential) error

type SanitizedCircuitBreakerConfig 

type SanitizedCircuitBreakerConfig struct {
	FailureThreshold int    `json:"failure_threshold"`
	SuccessThreshold int    `json:"success_threshold"`
	Timeout          string `json:"timeout"`
}

SanitizedCircuitBreakerConfig exposes effective circuit-breaker settings.

type SanitizedProviderConfig 

type SanitizedProviderConfig struct {
	Name              string                    `json:"name"`
	Type              string                    `json:"type"`
	BaseURL           string                    `json:"base_url,omitempty"`
	APIVersion        string                    `json:"api_version,omitempty"`
	Models            []string                  `json:"models,omitempty"`
	SessionStickyKeys bool                      `json:"session_sticky_keys"`
	Resilience        SanitizedResilienceConfig `json:"resilience"`
}

SanitizedProviderConfig is the admin-safe provider configuration view.

func SanitizeProviderConfigs 

func SanitizeProviderConfigs(configs map[string]ProviderConfig) []SanitizedProviderConfig

SanitizeProviderConfigs converts effective provider configs into a stable, admin-safe slice keyed by configured provider name.

type SanitizedResilienceConfig 

type SanitizedResilienceConfig struct {
	Retry          SanitizedRetryConfig          `json:"retry"`
	CircuitBreaker SanitizedCircuitBreakerConfig `json:"circuit_breaker"`
}

SanitizedResilienceConfig exposes effective resilience settings.

type SanitizedRetryConfig 

type SanitizedRetryConfig struct {
	MaxRetries     int     `json:"max_retries"`
	InitialBackoff string  `json:"initial_backoff"`
	MaxBackoff     string  `json:"max_backoff"`
	BackoffFactor  float64 `json:"backoff_factor"`
	JitterFactor   float64 `json:"jitter_factor"`
}

SanitizedRetryConfig exposes effective retry settings without secrets.

type SemanticEnricher 

type SemanticEnricher struct {
	// contains filtered or unexported fields
}

SemanticEnricher implements core.PassthroughSemanticEnricher from a static endpoint table. Endpoints missing from the table keep their audit path, or fall back to the generic /p/{provider}/... form when none is set.

func NewSemanticEnricher 

func NewSemanticEnricher(providerType string, endpoints map[string]PassthroughEndpointSemantics) SemanticEnricher

NewSemanticEnricher builds a SemanticEnricher for a provider type from its endpoint table; keys are normalized endpoint paths such as "/embeddings".

func (SemanticEnricher) Enrich 

func (e SemanticEnricher) Enrich(_ *core.RequestSnapshot, _ *core.WhiteBoxPrompt, info *core.PassthroughRouteInfo) *core.PassthroughRouteInfo

Enrich annotates passthrough route info with the provider's semantic operation and audit path for known endpoints.

func (SemanticEnricher) ProviderType 

func (e SemanticEnricher) ProviderType() string

ProviderType returns the provider type this enricher serves.

Source Files

View all Source files

Directories

Path Synopsis
Package anthropic provides Anthropic API integration for the LLM gateway.
 Package anthropic provides Anthropic API integration for the LLM gateway.
Package bailian provides the Alibaba Cloud Bailian (百炼 / DashScope) provider.
 Package bailian provides the Alibaba Cloud Bailian (百炼 / DashScope) provider.
Package bedrock provides Amazon Bedrock integration for the LLM gateway.
 Package bedrock provides Amazon Bedrock integration for the LLM gateway.
Package bedrockmantle provides direct access to Amazon Bedrock's OpenAI-compatible Mantle API.
 Package bedrockmantle provides direct access to Amazon Bedrock's OpenAI-compatible Mantle API.
Package cohere provides Cohere API v2 integration for the LLM gateway.
 Package cohere provides Cohere API v2 integration for the LLM gateway.
Package deepseek provides DeepSeek API integration for the LLM gateway.
 Package deepseek provides DeepSeek API integration for the LLM gateway.
Package fireworks provides Fireworks AI API integration for the LLM gateway.
 Package fireworks provides Fireworks AI API integration for the LLM gateway.
Package gemini provides Google Gemini API integration for the LLM gateway.
 Package gemini provides Google Gemini API integration for the LLM gateway.
Package googlecommon holds infrastructure shared by GoModel's Google-backed providers (Gemini AI Studio + Vertex AI).
 Package googlecommon holds infrastructure shared by GoModel's Google-backed providers (Gemini AI Studio + Vertex AI).
Package groq provides Groq API integration for the LLM gateway.
 Package groq provides Groq API integration for the LLM gateway.
Package health tracks recent request outcomes per provider and model.
 Package health tracks recent request outcomes per provider and model.
Package kilo provides Kilo AI Gateway integration for the LLM gateway.
 Package kilo provides Kilo AI Gateway integration for the LLM gateway.
Package kimicode provides Kimi Code API integration for the LLM gateway.
 Package kimicode provides Kimi Code API integration for the LLM gateway.
Package meta provides Meta Model API integration for the LLM gateway.
 Package meta provides Meta Model API integration for the LLM gateway.
Package minimax provides MiniMax API integration for the LLM gateway.
 Package minimax provides MiniMax API integration for the LLM gateway.
Package ollama provides Ollama API integration for the LLM gateway.
 Package ollama provides Ollama API integration for the LLM gateway.
Package openai provides OpenAI API integration for the LLM gateway.
 Package openai provides OpenAI API integration for the LLM gateway.
Package opencodego provides OpenCode Zen (Go subscription) integration for the LLM gateway.
 Package opencodego provides OpenCode Zen (Go subscription) integration for the LLM gateway.
Package vertex provides Google Vertex AI Gemini integration.
 Package vertex provides Google Vertex AI Gemini integration.
Package vllm provides vLLM OpenAI-compatible API integration for the LLM gateway.
 Package vllm provides vLLM OpenAI-compatible API integration for the LLM gateway.
Package xai provides xAI (Grok) API integration for the LLM gateway.
 Package xai provides xAI (Grok) API integration for the LLM gateway.
Package xiaomi provides Xiaomi MiMo API integration for the LLM gateway.
 Package xiaomi provides Xiaomi MiMo API integration for the LLM gateway.
Package zai provides Z.ai API integration for the LLM gateway.
 Package zai provides Z.ai API integration for the LLM gateway.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.