Documentation
¶
Overview ¶
Package provider defines the unified interface for LLM providers.
Index ¶
- Variables
- func NewChannelAdapter(ctx context.Context, s Stream, opts ChannelOptions) <-chan StreamChunk
- func OpenAIToolChoice(v any) (any, error)
- func WithCredential(ctx context.Context, c Credential) context.Context
- type Accumulator
- type AudioChunk
- type BiStream
- type Capabilities
- type ChannelOptions
- type Choice
- type Citation
- type ClientEvent
- type CompletionRequest
- type CompletionResponse
- type ContentPart
- type Credential
- type CredentialAware
- type CredentialProvider
- type CredentialResolver
- type Delta
- type DimensionsAware
- type EmbeddingRequest
- type EmbeddingResponse
- type EnvCredential
- type EventKind
- type FreeOfCharge
- type HealthStats
- type HealthTracker
- type ImageChunk
- type JSONSchemaDef
- type Message
- type Model
- type NativeContent
- type Pricing
- type Provider
- type Registry
- type ResponseFormat
- type StaticCredential
- type Stream
- type StreamChunk
- func Drain(ch <-chan StreamChunk) []StreamChunk
- func NewErrorChunk(providerName string, err error) *StreamChunk
- func NewHeartbeatChunk(providerName string) *StreamChunk
- func NewReasoningChunk(providerName, model, reasoning string) *StreamChunk
- func NewUsageChunk(providerName, model string, usage *Usage) *StreamChunk
- type StreamEvents
- type ThinkingConfig
- type Tool
- type ToolCall
- type ToolCallFunc
- type ToolChoice
- type ToolChoiceMode
- type ToolFunction
- type TransportWrapper
- type Usage
Constants ¶
This section is empty.
Variables ¶
var ErrCredentialNotFound = errors.New("nexus: provider credential not found")
ErrCredentialNotFound means there is no credential for the provider: the request is served with the provider's own configured key. nexus.ErrCredentialNotFound is the same value.
var ErrMissingAPIKey = errors.New("nexus: provider API key is required")
ErrMissingAPIKey is returned when a provider is invoked without a credential. Providers should return this (wrapped with their name) before making any network call, so the failure is a clear local error rather than a cryptic upstream 401 such as Anthropic's "x-api-key header is required".
var ErrMissingBaseURL = errors.New("nexus: provider base URL is required")
ErrMissingBaseURL is returned, before any network call, by a provider asked to send a request when neither its configuration nor the request's credential gives it a base URL.
var ErrNotSupported = errors.New("nexus: operation not supported by provider")
ErrNotSupported is returned when a provider does not support an operation.
Functions ¶
func NewChannelAdapter ¶
func NewChannelAdapter(ctx context.Context, s Stream, opts ChannelOptions) <-chan StreamChunk
NewChannelAdapter spawns a goroutine that reads chunks from s and publishes them on the returned channel. The channel closes when:
- Stream.Next returns io.EOF (clean drain), or
- ctx is canceled (channel closes immediately, Stream.Close is called), or
- Stream.Next returns an error (an EventError chunk is emitted first if ChannelOptions.EmitErrorChunk is set, then the channel closes).
The caller MUST drain the channel or cancel ctx — leaking the goroutine otherwise. Streams from this adapter must not be re-consumed via Next() in parallel; the adapter owns the iterator.
func OpenAIToolChoice ¶
OpenAIToolChoice returns v in the shape OpenAI's chat API takes, or nil when v is nil. Every shape ParseToolChoice reads is converted. An object with a "type" this package does not model, such as OpenAI's {"type":"allowed_tools",...} or {"type":"custom",...}, is OpenAI's own and is returned unchanged, so a newer OpenAI tool_choice keeps working. A value that is malformed (an unknown string, an object with no type, a function with no name) fails with an error that wraps ErrNotSupported.
func WithCredential ¶
func WithCredential(ctx context.Context, c Credential) context.Context
WithCredential returns a context that carries c for the provider call. An empty c clears any credential the parent context carried.
Types ¶
type Accumulator ¶
type Accumulator struct {
// contains filtered or unexported fields
}
Accumulator is the public, incremental form of the merge logic.
func NewAccumulator ¶
func NewAccumulator() *Accumulator
NewAccumulator creates an incremental accumulator that callers can feed chunks into one at a time, then call Finalize to produce the merged CompletionResponse. Useful for middleware that needs to forward chunks to the consumer while also assembling the merged response for hooks/cache.
func (*Accumulator) Add ¶
func (a *Accumulator) Add(c *StreamChunk)
Add merges a single chunk's contributions into the in-progress response. Safe to call with a nil chunk (no-op).
func (*Accumulator) Finalize ¶
func (a *Accumulator) Finalize(usageFallback func() *Usage) *CompletionResponse
Finalize returns the merged CompletionResponse. If usageFallback is non-nil and no usage chunk was seen, its return value is used. Pass `s.Usage` (the bound method) to fall back to the underlying stream's terminal Usage().
type AudioChunk ¶
type AudioChunk struct {
Format string `json:"format,omitempty"` // "pcm16","opus","mp3"
SampleRate int `json:"sample_rate,omitempty"`
Data []byte `json:"data,omitempty"` // raw bytes; encoders may base64
Transcript string `json:"transcript,omitempty"` // partial transcript for realtime
}
AudioChunk is a multi-modal audio fragment.
type BiStream ¶
type BiStream interface {
Stream
// Send pushes a client event upstream. May return ErrNotSupported on
// streams that don't accept the requested event type.
Send(ctx context.Context, evt ClientEvent) error
}
BiStream extends Stream with a client → server channel for true bidirectional streaming providers (OpenAI Realtime, Gemini Live).
Unidirectional providers do not implement BiStream; the WebSocket handler in package httpstream type-asserts the inner stream and only forwards client multi-modal envelopes when the assertion succeeds.
type Capabilities ¶
type Capabilities struct {
Chat bool `json:"chat"` // Chat completions
Streaming bool `json:"streaming"` // SSE streaming
Embeddings bool `json:"embeddings"` // Text embeddings
Images bool `json:"images"` // Image generation
Vision bool `json:"vision"` // Image input in messages
Tools bool `json:"tools"` // Function/tool calling
JSON bool `json:"json"` // Structured JSON output
Audio bool `json:"audio"` // Audio input/output
Thinking bool `json:"thinking"` // Extended thinking / reasoning
Batch bool `json:"batch"` // Batch API support
// Streaming-specific capabilities — describe what the provider can
// surface incrementally during a stream. Use these for feature
// detection at the gateway (e.g. only forward reasoning deltas to
// clients that explicitly opted in).
StreamingReasoning bool `json:"streaming_reasoning,omitempty"` // emits delta.reasoning
StreamingTools bool `json:"streaming_tools,omitempty"` // emits incremental tool-call deltas
StreamingAudio bool `json:"streaming_audio,omitempty"` // emits delta.audio chunks
StreamingCitations bool `json:"streaming_citations,omitempty"` // emits delta.citations
RealtimeAudio bool `json:"realtime_audio,omitempty"` // bidirectional realtime audio (OpenAI Realtime)
RealtimeVideo bool `json:"realtime_video,omitempty"` // bidirectional realtime video
LiveBidi bool `json:"live_bidi,omitempty"` // bidirectional live API (Gemini Live)
}
Capabilities describes what a provider can do.
func (Capabilities) Supports ¶
func (c Capabilities) Supports(capability string) bool
Supports checks if a capability is available by name.
type ChannelOptions ¶
type ChannelOptions struct {
// Buffer is the channel capacity. Default: 16. Use 0 for an unbuffered
// channel (slowest consumer fully gates the producer).
Buffer int
// EmitErrorChunk, when true, sends a final EventError chunk over the
// channel before closing on a non-EOF error. Consumers that select on
// the channel will see the error in-band rather than having to call
// Stream.Next themselves. Default: true.
EmitErrorChunk bool
}
ChannelOptions tunes the channel adapter.
type Choice ¶
type Choice struct {
Index int `json:"index"`
Message Message `json:"message"`
FinishReason string `json:"finish_reason"`
}
Choice represents one completion choice.
type Citation ¶
type Citation struct {
URL string `json:"url,omitempty"`
Title string `json:"title,omitempty"`
Quoted string `json:"quoted,omitempty"`
StartIdx int `json:"start_idx,omitempty"`
EndIdx int `json:"end_idx,omitempty"`
}
Citation is a single source/grounding reference.
type ClientEvent ¶
type ClientEvent struct {
// Type identifies the variant. Common values:
// "audio_chunk" — Audio populated; appends a partial audio buffer
// "image" — Image populated; uploads a still frame
// "commit" — finalize the current input turn (Realtime API)
// "cancel" — abort the current response generation
// "control" — provider-specific Data payload
Type string
Audio *AudioChunk
Image *ImageChunk
// Data is a free-form payload for provider-specific control messages.
// JSON-marshalable; the upstream provider deserializes it.
Data any
}
ClientEvent is a unified envelope for inbound (client → server) events on a bidirectional stream. The discriminator is Type; consumers populate the matching pointer field.
type CompletionRequest ¶
type CompletionRequest struct {
// Routing
Model string `json:"model"`
Provider string `json:"provider,omitempty"` // force specific provider
// Messages
Messages []Message `json:"messages"`
System string `json:"system,omitempty"` // system prompt (Anthropic-style)
// Parameters
MaxTokens int `json:"max_tokens,omitempty"`
Temperature *float64 `json:"temperature,omitempty"`
TopP *float64 `json:"top_p,omitempty"`
Stop []string `json:"stop,omitempty"`
Stream bool `json:"stream,omitempty"`
// Tool calling
Tools []Tool `json:"tools,omitempty"`
ToolChoice any `json:"tool_choice,omitempty"` // honoured by anthropic, openai (and providers built on it) and azureopenai; others drop it
// Structured output
ResponseFormat *ResponseFormat `json:"response_format,omitempty"`
// Extended thinking / reasoning
Thinking *ThinkingConfig `json:"thinking,omitempty"`
// Nexus metadata (not sent to provider)
TenantID string `json:"-"`
KeyID string `json:"-"`
Metadata map[string]string `json:"-"` // user-defined labels
}
CompletionRequest is the unified request type across all providers.
type CompletionResponse ¶
type CompletionResponse struct {
ID string `json:"id"`
Provider string `json:"provider"`
Model string `json:"model"`
Created time.Time `json:"created"`
// Content
Choices []Choice `json:"choices"`
// Usage
Usage Usage `json:"usage"`
// Nexus metadata
Cached bool `json:"cached,omitempty"`
Latency time.Duration `json:"latency,omitempty"`
Cost *money.USD `json:"cost,omitempty"` // exact; nil until something prices the request
// Extended thinking
ThinkingContent string `json:"thinking_content,omitempty"`
ThinkingTokens int `json:"thinking_tokens,omitempty"`
// State is used to pass metadata between middleware layers.
State map[string]any `json:"-"`
}
CompletionResponse is the unified response type.
func Accumulate ¶
func Accumulate(ctx context.Context, s Stream) (*CompletionResponse, error)
Accumulate drains a Stream and merges its chunks into a single CompletionResponse — the same shape Provider.Complete would have produced.
Merge rules:
- Content: concatenated into Choices[0].Message.Content.
- Reasoning: concatenated into ThinkingContent.
- ToolCalls: merged by ID first, falling back to slot index. function.name keeps the first non-empty value; function.arguments are concatenated.
- Citations: appended.
- Audio: bytes concatenated; latest non-empty Transcript wins.
- Image: last non-Partial chunk wins; otherwise the latest seen.
- Usage: prefer the EventUsage / final-delta chunk; otherwise s.Usage().
- FinishReason: last non-empty across chunks.
- Native: the last non-nil Delta.Native becomes Message.Native.
Accumulate stops on io.EOF and closes the stream. It does NOT close on error — callers may inspect the partial response and handle cleanup.
type ContentPart ¶
type ContentPart struct {
Type string `json:"type"` // text, image_url, image_base64
Text string `json:"text,omitempty"`
ImageURL string `json:"image_url,omitempty"`
Data string `json:"data,omitempty"` // base64
MimeType string `json:"mime_type,omitempty"`
}
ContentPart for multimodal messages.
type Credential ¶
type Credential struct {
// APIKey replaces the provider's configured key for the request. It is
// a secret: nothing logs it or puts it in an error, and Credential's
// String and GoString leave it out.
APIKey string
// BaseURL replaces the provider's configured base URL for the request.
// "" keeps the configured one. A credential with a BaseURL brings its
// own APIKey: the provider sends that key, empty or not, and never its
// configured one, to the host this URL names.
//
// Nexus trusts the URL as given and sends the request body there. A
// host that lets tenants choose it must validate it first: allow only
// https (or the schemes it means to), refuse private, loopback,
// link-local and cloud metadata addresses, and guard against DNS
// rebinding by checking the address the name resolves to at dial time,
// not only when the URL is saved.
BaseURL string
}
Credential is what one request authenticates with: the API key and, optionally, the endpoint. The gateway resolves one per provider call (see CredentialResolver) and hands it to the provider on the context.
func CredentialFromContext ¶
func CredentialFromContext(ctx context.Context) (Credential, bool)
CredentialFromContext returns the credential ctx carries. ok is false when there is none or it is empty.
func (Credential) GoString ¶
func (c Credential) GoString() string
GoString is String, so %#v leaves the key out too.
func (Credential) IsZero ¶
func (c Credential) IsZero() bool
IsZero reports whether c carries neither a key nor a base URL, which counts as no credential at all.
type CredentialAware ¶
type CredentialAware interface {
AcceptsCredentials() bool
}
CredentialAware is implemented by providers that apply a per-request Credential from the context: anthropic, openai and opencompat. The gateway refuses to send a tenant credential to a provider that does not answer true, rather than serve the tenant on the operator's key.
type CredentialProvider ¶
type CredentialProvider interface {
// GetCredential returns the current credential (API key) for a
// provider, or ErrCredentialNotFound when it has none.
GetCredential(ctx context.Context, providerName string) (string, error)
}
CredentialProvider retrieves API keys or tokens for providers at runtime. This allows integration with secret managers (Vault, AWS Secrets, etc.) instead of hard-coding API keys. Given to the gateway (nexus.WithCredentialProvider), it is asked before each provider call; a provider that also implements CredentialResolver is asked through that instead, so it can choose the base URL too.
type CredentialResolver ¶
type CredentialResolver interface {
ResolveCredential(ctx context.Context, providerName string) (Credential, error)
}
CredentialResolver is a CredentialProvider that can also choose the endpoint: a host that keeps each workspace's own key, and for an OpenAI-compatible server its URL, implements it. The gateway calls it after routing, once per provider attempt, with pipeline.TenantID(ctx) set to the request's nexus tenant ("" for an unattributed request).
Return ErrCredentialNotFound (wrapped or not) to serve the request with the provider's own key. Any other error refuses the request: nothing is sent and the gateway does not fall back to the provider's key. The error must not contain a key.
A resolver that returns a BaseURL is choosing where the gateway sends the request, its body and the key. Nexus does not check that URL; validate it as Credential.BaseURL describes before returning it.
type Delta ¶
type Delta struct {
Role string `json:"role,omitempty"`
Content string `json:"content,omitempty"`
ToolCalls []ToolCall `json:"tool_calls,omitempty"`
// Reasoning carries extended-thinking content (o1, Claude thinking_delta,
// DeepSeek R1 reasoning_content). Emitted alongside (or before) Content.
Reasoning string `json:"reasoning,omitempty"`
// Refusal is an OpenAI-style structured refusal string.
Refusal string `json:"refusal,omitempty"`
// Citations carries grounding / source references.
Citations []Citation `json:"citations,omitempty"`
// Audio carries a multi-modal audio chunk (raw bytes + optional transcript).
Audio *AudioChunk `json:"audio,omitempty"`
// Image carries a multi-modal image chunk (raw bytes or URL).
Image *ImageChunk `json:"image,omitempty"`
// Transcript is a partial speech transcript (realtime / live APIs).
Transcript string `json:"transcript,omitempty"`
// Native is the whole reply in the provider's own format, set on the
// EventMessageStop chunk. The accumulator copies it to Message.Native.
Native *NativeContent `json:"-"`
}
Delta contains the incremental content in a stream chunk.
Role/Content/ToolCalls are the legacy fields. The remaining fields are optional and populated by providers that surface richer streams.
type DimensionsAware ¶
DimensionsAware is implemented by a provider that can return embeddings of a requested width (EmbeddingRequest.Dimensions). SupportsDimensions reports whether it can for model. The gateway refuses a request with Dimensions set when the routed provider does not implement this or answers false, so the provider is never called with a field it would drop.
type EmbeddingRequest ¶
type EmbeddingRequest struct {
Model string `json:"model"`
Provider string `json:"provider,omitempty"` // force specific provider
Input []string `json:"input"`
// Dimensions asks for vectors of exactly this many numbers; 0 keeps the
// model's default. The gateway refuses it for a provider that cannot
// honour it (see DimensionsAware) and fails a response whose vectors
// have another length.
Dimensions int `json:"dimensions,omitempty"`
TenantID string `json:"-"`
KeyID string `json:"-"`
// Metadata is stored on the usage record, where it can be filtered by
// (see usage.ValidateMetadata for the limits).
Metadata map[string]string `json:"-"`
}
EmbeddingRequest for text embeddings.
type EmbeddingResponse ¶
type EmbeddingResponse struct {
Provider string `json:"provider"`
Model string `json:"model"`
Embeddings [][]float64 `json:"embeddings"`
Usage Usage `json:"usage"`
}
EmbeddingResponse for embeddings.
type EnvCredential ¶
type EnvCredential struct {
// contains filtered or unexported fields
}
EnvCredential reads credentials from environment variables.
func NewEnvCredential ¶
func NewEnvCredential(prefix string) *EnvCredential
NewEnvCredential creates a credential provider that reads from env vars. The environment variable name is formed as: {prefix}{PROVIDER_NAME}_API_KEY.
func (*EnvCredential) GetCredential ¶
type EventKind ¶
type EventKind string
EventKind identifies the semantic category of a StreamChunk.
The zero value (EventDelta) is the legacy form: Delta carries Role/Content/ ToolCalls and consumers ignore Kind. New kinds let providers surface richer signals (reasoning, message boundaries, usage, citations, multi-modal, etc.) without breaking existing consumers — kinds they don't recognize can be dropped or forwarded as-is.
const ( // EventDelta is the default — incremental Role/Content/ToolCalls. EventDelta EventKind = "" // EventMessageStart marks the beginning of an assistant message. EventMessageStart EventKind = "message_start" // EventMessageStop marks the end of an assistant message. EventMessageStop EventKind = "message_stop" // EventReasoning carries an extended-thinking / reasoning content delta. // Used for OpenAI o-series, Anthropic extended thinking, DeepSeek R1, etc. EventReasoning EventKind = "reasoning" // EventToolCallDelta carries an incremental tool-call fragment // (function name, partial arguments JSON, or accumulator-keyed index). EventToolCallDelta EventKind = "tool_call_delta" // EventAudio carries a multi-modal audio chunk (raw bytes or transcript). EventAudio EventKind = "audio" // EventImage carries a multi-modal image chunk. EventImage EventKind = "image" // EventCitation carries a single citation reference. EventCitation EventKind = "citation" // EventUsage carries a token-usage snapshot — typically the final chunk. EventUsage EventKind = "usage" // EventHeartbeat is an upstream keepalive ping (e.g. Anthropic "ping"). // Encoders may translate this to a wire-level keepalive comment. EventHeartbeat EventKind = "heartbeat" // EventError signals a recoverable, in-band provider error. Consumers // should propagate to the wire but may continue reading further events. EventError EventKind = "error" )
type FreeOfCharge ¶
type FreeOfCharge interface {
FreeOfCharge() bool
}
FreeOfCharge is implemented by a provider whose every model, listed or not, costs exactly $0, such as local inference.
type HealthStats ¶
type HealthStats struct {
TotalRequests int `json:"total_requests"`
Successes int `json:"successes"`
Failures int `json:"failures"`
SuccessRate float64 `json:"success_rate"`
AvgLatency time.Duration `json:"avg_latency"`
P99Latency time.Duration `json:"p99_latency"`
LastSuccess time.Time `json:"last_success,omitempty"`
LastFailure time.Time `json:"last_failure,omitempty"`
LastError string `json:"last_error,omitempty"`
}
HealthStats contains health metrics for a provider.
type HealthTracker ¶
type HealthTracker interface {
// RecordSuccess records a successful provider call.
RecordSuccess(providerName string, latency time.Duration)
// RecordFailure records a failed provider call.
RecordFailure(providerName string, err error)
// Stats returns health statistics for a provider.
Stats(providerName string) *HealthStats
// IsHealthy returns true if the provider is considered healthy.
IsHealthy(providerName string) bool
// AllStats returns stats for all tracked providers.
AllStats() map[string]*HealthStats
}
HealthTracker monitors provider health for routing decisions.
func NewHealthTracker ¶
func NewHealthTracker() HealthTracker
NewHealthTracker creates a new in-memory health tracker.
type ImageChunk ¶
type ImageChunk struct {
MimeType string `json:"mime_type,omitempty"`
Data []byte `json:"data,omitempty"`
URL string `json:"url,omitempty"`
Partial bool `json:"partial,omitempty"` // true while incremental, false on final
}
ImageChunk is a multi-modal image fragment.
type JSONSchemaDef ¶
type JSONSchemaDef struct {
// Name is a free-form identifier OpenAI requires (any non-empty string).
Name string `json:"name"`
// Description is optional human-readable hint for the model.
Description string `json:"description,omitempty"`
// Schema is the JSON Schema document (object with type/properties/required).
Schema any `json:"schema"`
// Strict enables strict mode — model output must validate against Schema.
Strict bool `json:"strict,omitempty"`
}
JSONSchemaDef wraps a JSON schema for OpenAI-style structured outputs. The wire format is `{"name": ..., "strict": true, "schema": {...}}` nested under `response_format.json_schema`.
type Message ¶
type Message struct {
Role string `json:"role"` // system, user, assistant, tool
Content any `json:"content"`
Name string `json:"name,omitempty"`
ToolCalls []ToolCall `json:"tool_calls,omitempty"`
ToolCallID string `json:"tool_call_id,omitempty"`
// Native is the assistant reply in the provider's own format. Keep it
// with the message and pass it back unchanged; see NativeContent.
Native *NativeContent `json:"-"`
}
Message represents a conversation message.
type Model ¶
type Model struct {
ID string `json:"id"` // e.g., "gpt-4o"
Provider string `json:"provider"` // e.g., "openai"
Name string `json:"name"` // human-readable
Capabilities Capabilities `json:"capabilities"`
ContextWindow int `json:"context_window"`
MaxOutput int `json:"max_output"`
Pricing Pricing `json:"pricing"`
}
Model describes an available LLM model.
type NativeContent ¶
type NativeContent struct {
// Provider names the provider whose wire format Content is in.
Provider string `json:"provider"`
// Model is the model that produced the reply.
Model string `json:"model"`
// Content is the reply's content as that provider returned it.
Content json.RawMessage `json:"content"`
}
NativeContent is an assistant reply in the provider's own wire format, kept beside the unified fields so the turn can be replayed exactly.
Some models reject a history whose earlier turns were edited, signed thinking blocks included. Store Choices[0].Message.Native with the assistant message and send it back unchanged on the next request: a provider that recognises it sends Content as is while the message still matches it, and rebuilds the turn from the unified fields otherwise.
It is never serialized, and the response and stream caches drop it on every backend, so a cached reply carries none. An output guard that rewrites a message drops it too.
type Pricing ¶
type Pricing struct {
InputPerMillion money.USD `json:"input_per_million"`
OutputPerMillion money.USD `json:"output_per_million"`
EmbeddingPerMillion money.USD `json:"embedding_per_million,omitzero"`
CacheReadPerMillion money.USD `json:"cache_read_per_million,omitzero"`
CacheWritePerMillion money.USD `json:"cache_write_per_million,omitzero"` // 5-minute cache writes
CacheWriteLongPerMillion money.USD `json:"cache_write_long_per_million,omitzero"` // 1-hour cache writes
LongContextAbove int `json:"long_context_above,omitempty"` // prompt tokens; 0 means no tier
LongContext *Pricing `json:"long_context,omitempty"`
Free bool `json:"free,omitempty"`
}
Pricing is a model's list price per million tokens. When Free is set the model costs exactly $0. Otherwise a model with both InputPerMillion and OutputPerMillion at zero is unpriced (its cost is unknown, never $0); when only one of them is set, the other kind of token is charged at $0. Embeddings use EmbeddingPerMillion.
Cache reads, 5-minute cache writes and 1-hour cache writes have their own prices. A cache price left at zero charges those tokens at InputPerMillion; a 1-hour write with no price of its own is charged at CacheWritePerMillion first. When LongContextAbove is set and a request's prompt tokens exceed it, every token of that request, cache reads and writes included, is charged at LongContext instead.
type Provider ¶
type Provider interface {
// Name returns the provider identifier (e.g., "openai", "anthropic").
Name() string
// Capabilities returns what this provider supports.
Capabilities() Capabilities
// Models returns the list of available models.
Models(ctx context.Context) ([]Model, error)
// Complete sends a chat completion request.
Complete(ctx context.Context, req *CompletionRequest) (*CompletionResponse, error)
// CompleteStream sends a streaming chat completion request.
CompleteStream(ctx context.Context, req *CompletionRequest) (Stream, error)
// Embed sends an embedding request (if supported).
Embed(ctx context.Context, req *EmbeddingRequest) (*EmbeddingResponse, error)
// Healthy returns true if the provider is reachable.
Healthy(ctx context.Context) bool
}
Provider abstracts an LLM provider (OpenAI, Anthropic, Ollama, etc.)
type Registry ¶
type Registry interface {
// Register adds a provider to the registry.
Register(p Provider)
// Get returns a provider by name.
Get(name string) (Provider, bool)
// All returns all registered providers.
All() []Provider
// Count returns the number of registered providers.
Count() int
// ForModel returns the providers whose Models list the model, in
// registration order. A provider whose Models fails is left out.
ForModel(model string) []Provider
// WithCapability returns providers with a specific capability.
WithCapability(capability string) []Provider
// Healthy returns only healthy providers.
Healthy(ctx context.Context) []Provider
}
Registry manages registered LLM providers.
type ResponseFormat ¶
type ResponseFormat struct {
Type string `json:"type"` // text, json_object, json_schema
Schema any `json:"schema,omitempty"`
JSONSchema *JSONSchemaDef `json:"json_schema,omitempty"`
}
ResponseFormat specifies the desired output format.
For OpenAI-compatible structured outputs (Type = "json_schema"), populate JSONSchema with the schema wrapped in a JSONSchemaDef — that is the only shape OpenAI, LM Studio, and other OAI-compatible servers accept. The legacy Schema field is preserved for providers that take a bare schema (most do not) and for backward compatibility.
type StaticCredential ¶
type StaticCredential struct {
// contains filtered or unexported fields
}
StaticCredential always returns the same key. Useful for testing.
func NewStaticCredential ¶
func NewStaticCredential(credentials map[string]string) *StaticCredential
NewStaticCredential creates a credential provider from a static map.
func (*StaticCredential) GetCredential ¶
type Stream ¶
type Stream interface {
// Next returns the next chunk. Returns io.EOF when done.
Next(ctx context.Context) (*StreamChunk, error)
// Close releases resources.
Close() error
// Usage returns final usage after stream completes.
Usage() *Usage
}
Stream represents a streaming response from a provider.
type StreamChunk ¶
type StreamChunk struct {
ID string `json:"id"`
Provider string `json:"provider"`
Model string `json:"model"`
Kind EventKind `json:"kind,omitempty"`
Delta Delta `json:"delta"`
FinishReason string `json:"finish_reason,omitempty"`
// Optional payloads — populated based on Kind.
Usage *Usage `json:"usage,omitempty"` // EventUsage / final delta
Err string `json:"error,omitempty"` // EventError
Created int64 `json:"created,omitempty"` // unix ms; used for cache-replay timing
}
StreamChunk is a single piece of a streamed response.
Kind discriminates the payload: legacy chunks have Kind == "" (EventDelta) and read Delta.Content / Delta.ToolCalls. Newer kinds populate the matching optional fields (Usage, Err) or extended Delta fields (Reasoning, Audio…).
func Drain ¶
func Drain(ch <-chan StreamChunk) []StreamChunk
Drain reads everything from a channel adapter into a slice. Useful for tests; not recommended in production where you want to process events as they arrive.
func NewErrorChunk ¶
func NewErrorChunk(providerName string, err error) *StreamChunk
NewErrorChunk builds an in-band error frame. The provider field tags which component produced the error so wire encoders can attribute it.
func NewHeartbeatChunk ¶
func NewHeartbeatChunk(providerName string) *StreamChunk
NewHeartbeatChunk builds a keepalive frame. Encoders may translate to a wire-level keepalive (SSE comment, WS ping) instead of forwarding the JSON.
func NewReasoningChunk ¶
func NewReasoningChunk(providerName, model, reasoning string) *StreamChunk
NewReasoningChunk builds a reasoning/thinking delta.
func NewUsageChunk ¶
func NewUsageChunk(providerName, model string, usage *Usage) *StreamChunk
NewUsageChunk builds a chunk that carries final/intermediate usage. Used by providers that surface usage as a separate SSE event (OpenAI with stream_options.include_usage, Anthropic message_delta) and by middleware that synthesizes a usage frame for downstream encoders.
type StreamEvents ¶
type StreamEvents interface {
Events(ctx context.Context) <-chan StreamChunk
}
StreamEvents is an optional interface streams may implement to expose a channel-based consumer surface in addition to the iterator-style Next(). Use provider.NewChannelAdapter as a default fallback for streams that only implement Stream.
type ThinkingConfig ¶
type ThinkingConfig struct {
Enabled bool `json:"enabled,omitempty"`
BudgetTokens int `json:"budget_tokens,omitempty"`
IncludeThinking bool `json:"include_thinking,omitempty"`
// Effort steers how much the model thinks and spends: "low", "medium",
// "high", "xhigh" or "max". Empty keeps the model's default. A provider
// that knows the model rejects a level the model does not accept.
Effort string `json:"effort,omitempty"`
}
ThinkingConfig controls extended thinking / reasoning behavior.
type Tool ¶
type Tool struct {
Type string `json:"type"` // function
Function ToolFunction `json:"function"`
}
Tool definition.
type ToolCall ¶
type ToolCall struct {
ID string `json:"id"`
Type string `json:"type"`
Function ToolCallFunc `json:"function"`
}
ToolCall represents a tool invocation from the assistant.
type ToolCallFunc ¶
ToolCallFunc is the function call details within a ToolCall.
type ToolChoice ¶
type ToolChoice struct {
Mode ToolChoiceMode
Function string // set when Mode is ToolChoiceFunction
}
ToolChoice is the parsed form of CompletionRequest.ToolChoice.
func ParseToolChoice ¶
func ParseToolChoice(v any) (tc ToolChoice, ok bool, err error)
ParseToolChoice parses a CompletionRequest.ToolChoice value. It reads OpenAI's shape ("auto", "none", "required", or {"type":"function","function":{"name":...}}) and, for callers that already speak Anthropic, {"type":"auto"|"any"|"none"} and {"type":"tool","name":...}. The bare string "any" is read as required. ok is false for nil. A value it cannot read fails with an error that wraps ErrNotSupported, so the gateway refuses the request as the caller's mistake and does not retry it.
func (ToolChoice) OpenAI ¶
func (tc ToolChoice) OpenAI() any
OpenAI returns tc in OpenAI's tool_choice shape: the string "auto", "none" or "required", or {"type":"function","function":{"name":...}}. It returns nil for a mode it does not know, such as the zero ToolChoice.
type ToolChoiceMode ¶
type ToolChoiceMode string
ToolChoiceMode is how a request constrains the model's use of tools.
const ( // ToolChoiceAuto lets the model decide whether to call a tool. ToolChoiceAuto ToolChoiceMode = "auto" // ToolChoiceNone forbids tool calls. ToolChoiceNone ToolChoiceMode = "none" // ToolChoiceRequired forces the model to call some tool. ToolChoiceRequired ToolChoiceMode = "required" // ToolChoiceFunction forces the model to call one named tool. ToolChoiceFunction ToolChoiceMode = "function" )
type ToolFunction ¶
type ToolFunction struct {
Name string `json:"name"`
Description string `json:"description,omitempty"`
Parameters any `json:"parameters,omitempty"` // JSON Schema
}
ToolFunction describes a function tool.
type TransportWrapper ¶
type TransportWrapper interface {
WrapTransport(wrap func(http.RoundTripper) http.RoundTripper)
}
TransportWrapper is implemented by a provider that sends its requests through an HTTP client it owns and lets the gateway wrap that client's transport, to trace every outbound call for example. Every HTTP provider in this repo implements it; the WebSocket ones (geminilive, openairealtime) do not.
The gateway calls WrapTransport once, from Initialize, with the function given to nexus.WithProviderTransport. wrap receives the client's current transport (nil means http.DefaultTransport) and returns the one to use. A nil result leaves the transport as it was. Call it before the first request, never while one is in flight.
type Usage ¶
type Usage struct {
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
ThinkingTokens int `json:"thinking_tokens,omitempty"`
CacheReadTokens int `json:"cache_read_tokens,omitempty"`
CacheWriteTokens int `json:"cache_write_tokens,omitempty"`
CacheWriteLongTokens int `json:"cache_write_long_tokens,omitempty"`
TotalTokens int `json:"total_tokens"`
}
Usage tracks token consumption. PromptTokens counts every input token, cache reads and cache writes included. CacheReadTokens, CacheWriteTokens and CacheWriteLongTokens are subsets of it: CacheWriteTokens counts every cache write, and CacheWriteLongTokens is the part of those written for an hour rather than five minutes.