provider

package
v1.9.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 28, 2026 License: MIT Imports: 11 Imported by: 0

Documentation

Overview

Package provider defines the unified interface for LLM providers.

Index

Constants

This section is empty.

Variables

View Source
var ErrCredentialNotFound = errors.New("nexus: provider credential not found")

ErrCredentialNotFound means there is no credential for the provider: the request is served with the provider's own configured key. nexus.ErrCredentialNotFound is the same value.

View Source
var ErrMissingAPIKey = errors.New("nexus: provider API key is required")

ErrMissingAPIKey is returned when a provider is invoked without a credential. Providers should return this (wrapped with their name) before making any network call, so the failure is a clear local error rather than a cryptic upstream 401 such as Anthropic's "x-api-key header is required".

View Source
var ErrMissingBaseURL = errors.New("nexus: provider base URL is required")

ErrMissingBaseURL is returned, before any network call, by a provider asked to send a request when neither its configuration nor the request's credential gives it a base URL.

View Source
var ErrNotSupported = errors.New("nexus: operation not supported by provider")

ErrNotSupported is returned when a provider does not support an operation.

Functions

func NewChannelAdapter

func NewChannelAdapter(ctx context.Context, s Stream, opts ChannelOptions) <-chan StreamChunk

NewChannelAdapter spawns a goroutine that reads chunks from s and publishes them on the returned channel. The channel closes when:

  • Stream.Next returns io.EOF (clean drain), or
  • ctx is canceled (channel closes immediately, Stream.Close is called), or
  • Stream.Next returns an error (an EventError chunk is emitted first if ChannelOptions.EmitErrorChunk is set, then the channel closes).

The caller MUST drain the channel or cancel ctx — leaking the goroutine otherwise. Streams from this adapter must not be re-consumed via Next() in parallel; the adapter owns the iterator.

func OpenAIToolChoice

func OpenAIToolChoice(v any) (any, error)

OpenAIToolChoice returns v in the shape OpenAI's chat API takes, or nil when v is nil. Every shape ParseToolChoice reads is converted. An object with a "type" this package does not model, such as OpenAI's {"type":"allowed_tools",...} or {"type":"custom",...}, is OpenAI's own and is returned unchanged, so a newer OpenAI tool_choice keeps working. A value that is malformed (an unknown string, an object with no type, a function with no name) fails with an error that wraps ErrNotSupported.

func WithCredential

func WithCredential(ctx context.Context, c Credential) context.Context

WithCredential returns a context that carries c for the provider call. An empty c clears any credential the parent context carried.

Types

type Accumulator

type Accumulator struct {
	// contains filtered or unexported fields
}

Accumulator is the public, incremental form of the merge logic.

func NewAccumulator

func NewAccumulator() *Accumulator

NewAccumulator creates an incremental accumulator that callers can feed chunks into one at a time, then call Finalize to produce the merged CompletionResponse. Useful for middleware that needs to forward chunks to the consumer while also assembling the merged response for hooks/cache.

func (*Accumulator) Add

func (a *Accumulator) Add(c *StreamChunk)

Add merges a single chunk's contributions into the in-progress response. Safe to call with a nil chunk (no-op).

func (*Accumulator) Finalize

func (a *Accumulator) Finalize(usageFallback func() *Usage) *CompletionResponse

Finalize returns the merged CompletionResponse. If usageFallback is non-nil and no usage chunk was seen, its return value is used. Pass `s.Usage` (the bound method) to fall back to the underlying stream's terminal Usage().

type AudioChunk

type AudioChunk struct {
	Format     string `json:"format,omitempty"` // "pcm16","opus","mp3"
	SampleRate int    `json:"sample_rate,omitempty"`
	Data       []byte `json:"data,omitempty"`       // raw bytes; encoders may base64
	Transcript string `json:"transcript,omitempty"` // partial transcript for realtime
}

AudioChunk is a multi-modal audio fragment.

type BiStream

type BiStream interface {
	Stream

	// Send pushes a client event upstream. May return ErrNotSupported on
	// streams that don't accept the requested event type.
	Send(ctx context.Context, evt ClientEvent) error
}

BiStream extends Stream with a client → server channel for true bidirectional streaming providers (OpenAI Realtime, Gemini Live).

Unidirectional providers do not implement BiStream; the WebSocket handler in package httpstream type-asserts the inner stream and only forwards client multi-modal envelopes when the assertion succeeds.

type Capabilities

type Capabilities struct {
	Chat       bool `json:"chat"`       // Chat completions
	Streaming  bool `json:"streaming"`  // SSE streaming
	Embeddings bool `json:"embeddings"` // Text embeddings
	Images     bool `json:"images"`     // Image generation
	Vision     bool `json:"vision"`     // Image input in messages
	Tools      bool `json:"tools"`      // Function/tool calling
	JSON       bool `json:"json"`       // Structured JSON output
	Audio      bool `json:"audio"`      // Audio input/output
	Thinking   bool `json:"thinking"`   // Extended thinking / reasoning
	Batch      bool `json:"batch"`      // Batch API support

	// Streaming-specific capabilities — describe what the provider can
	// surface incrementally during a stream. Use these for feature
	// detection at the gateway (e.g. only forward reasoning deltas to
	// clients that explicitly opted in).
	StreamingReasoning bool `json:"streaming_reasoning,omitempty"` // emits delta.reasoning
	StreamingTools     bool `json:"streaming_tools,omitempty"`     // emits incremental tool-call deltas
	StreamingAudio     bool `json:"streaming_audio,omitempty"`     // emits delta.audio chunks
	StreamingCitations bool `json:"streaming_citations,omitempty"` // emits delta.citations
	RealtimeAudio      bool `json:"realtime_audio,omitempty"`      // bidirectional realtime audio (OpenAI Realtime)
	RealtimeVideo      bool `json:"realtime_video,omitempty"`      // bidirectional realtime video
	LiveBidi           bool `json:"live_bidi,omitempty"`           // bidirectional live API (Gemini Live)
}

Capabilities describes what a provider can do.

func (Capabilities) Supports

func (c Capabilities) Supports(capability string) bool

Supports checks if a capability is available by name.

type ChannelOptions

type ChannelOptions struct {
	// Buffer is the channel capacity. Default: 16. Use 0 for an unbuffered
	// channel (slowest consumer fully gates the producer).
	Buffer int

	// EmitErrorChunk, when true, sends a final EventError chunk over the
	// channel before closing on a non-EOF error. Consumers that select on
	// the channel will see the error in-band rather than having to call
	// Stream.Next themselves. Default: true.
	EmitErrorChunk bool
}

ChannelOptions tunes the channel adapter.

type Choice

type Choice struct {
	Index        int     `json:"index"`
	Message      Message `json:"message"`
	FinishReason string  `json:"finish_reason"`
}

Choice represents one completion choice.

type Citation

type Citation struct {
	URL      string `json:"url,omitempty"`
	Title    string `json:"title,omitempty"`
	Quoted   string `json:"quoted,omitempty"`
	StartIdx int    `json:"start_idx,omitempty"`
	EndIdx   int    `json:"end_idx,omitempty"`
}

Citation is a single source/grounding reference.

type ClientEvent

type ClientEvent struct {
	// Type identifies the variant. Common values:
	//   "audio_chunk" — Audio populated; appends a partial audio buffer
	//   "image"       — Image populated; uploads a still frame
	//   "commit"      — finalize the current input turn (Realtime API)
	//   "cancel"      — abort the current response generation
	//   "control"     — provider-specific Data payload
	Type string

	Audio *AudioChunk
	Image *ImageChunk

	// Data is a free-form payload for provider-specific control messages.
	// JSON-marshalable; the upstream provider deserializes it.
	Data any
}

ClientEvent is a unified envelope for inbound (client → server) events on a bidirectional stream. The discriminator is Type; consumers populate the matching pointer field.

type CompletionRequest

type CompletionRequest struct {
	// Routing
	Model    string `json:"model"`
	Provider string `json:"provider,omitempty"` // force specific provider

	// Messages
	Messages []Message `json:"messages"`
	System   string    `json:"system,omitempty"` // system prompt (Anthropic-style)

	// Parameters
	MaxTokens   int      `json:"max_tokens,omitempty"`
	Temperature *float64 `json:"temperature,omitempty"`
	TopP        *float64 `json:"top_p,omitempty"`
	Stop        []string `json:"stop,omitempty"`
	Stream      bool     `json:"stream,omitempty"`

	// Tool calling
	Tools      []Tool `json:"tools,omitempty"`
	ToolChoice any    `json:"tool_choice,omitempty"` // honoured by anthropic, openai (and providers built on it) and azureopenai; others drop it

	// Structured output
	ResponseFormat *ResponseFormat `json:"response_format,omitempty"`

	// Extended thinking / reasoning
	Thinking *ThinkingConfig `json:"thinking,omitempty"`

	// Nexus metadata (not sent to provider)
	TenantID string            `json:"-"`
	KeyID    string            `json:"-"`
	Metadata map[string]string `json:"-"` // user-defined labels
}

CompletionRequest is the unified request type across all providers.

type CompletionResponse

type CompletionResponse struct {
	ID       string    `json:"id"`
	Provider string    `json:"provider"`
	Model    string    `json:"model"`
	Created  time.Time `json:"created"`

	// Content
	Choices []Choice `json:"choices"`

	// Usage
	Usage Usage `json:"usage"`

	// Nexus metadata
	Cached  bool          `json:"cached,omitempty"`
	Latency time.Duration `json:"latency,omitempty"`
	Cost    *money.USD    `json:"cost,omitempty"` // exact; nil until something prices the request

	// Extended thinking
	ThinkingContent string `json:"thinking_content,omitempty"`
	ThinkingTokens  int    `json:"thinking_tokens,omitempty"`

	// State is used to pass metadata between middleware layers.
	State map[string]any `json:"-"`
}

CompletionResponse is the unified response type.

func Accumulate

func Accumulate(ctx context.Context, s Stream) (*CompletionResponse, error)

Accumulate drains a Stream and merges its chunks into a single CompletionResponse — the same shape Provider.Complete would have produced.

Merge rules:

  • Content: concatenated into Choices[0].Message.Content.
  • Reasoning: concatenated into ThinkingContent.
  • ToolCalls: merged by ID first, falling back to slot index. function.name keeps the first non-empty value; function.arguments are concatenated.
  • Citations: appended.
  • Audio: bytes concatenated; latest non-empty Transcript wins.
  • Image: last non-Partial chunk wins; otherwise the latest seen.
  • Usage: prefer the EventUsage / final-delta chunk; otherwise s.Usage().
  • FinishReason: last non-empty across chunks.
  • Native: the last non-nil Delta.Native becomes Message.Native.

Accumulate stops on io.EOF and closes the stream. It does NOT close on error — callers may inspect the partial response and handle cleanup.

type ContentPart

type ContentPart struct {
	Type     string `json:"type"` // text, image_url, image_base64
	Text     string `json:"text,omitempty"`
	ImageURL string `json:"image_url,omitempty"`
	Data     string `json:"data,omitempty"` // base64
	MimeType string `json:"mime_type,omitempty"`
}

ContentPart for multimodal messages.

type Credential

type Credential struct {
	// APIKey replaces the provider's configured key for the request. It is
	// a secret: nothing logs it or puts it in an error, and Credential's
	// String and GoString leave it out.
	APIKey string
	// BaseURL replaces the provider's configured base URL for the request.
	// "" keeps the configured one. A credential with a BaseURL brings its
	// own APIKey: the provider sends that key, empty or not, and never its
	// configured one, to the host this URL names.
	//
	// Nexus trusts the URL as given and sends the request body there. A
	// host that lets tenants choose it must validate it first: allow only
	// https (or the schemes it means to), refuse private, loopback,
	// link-local and cloud metadata addresses, and guard against DNS
	// rebinding by checking the address the name resolves to at dial time,
	// not only when the URL is saved.
	BaseURL string
}

Credential is what one request authenticates with: the API key and, optionally, the endpoint. The gateway resolves one per provider call (see CredentialResolver) and hands it to the provider on the context.

func CredentialFromContext

func CredentialFromContext(ctx context.Context) (Credential, bool)

CredentialFromContext returns the credential ctx carries. ok is false when there is none or it is empty.

func (Credential) GoString

func (c Credential) GoString() string

GoString is String, so %#v leaves the key out too.

func (Credential) IsZero

func (c Credential) IsZero() bool

IsZero reports whether c carries neither a key nor a base URL, which counts as no credential at all.

func (Credential) String

func (c Credential) String() string

String describes c without its key.

type CredentialAware

type CredentialAware interface {
	AcceptsCredentials() bool
}

CredentialAware is implemented by providers that apply a per-request Credential from the context: anthropic, openai and opencompat. The gateway refuses to send a tenant credential to a provider that does not answer true, rather than serve the tenant on the operator's key.

type CredentialProvider

type CredentialProvider interface {
	// GetCredential returns the current credential (API key) for a
	// provider, or ErrCredentialNotFound when it has none.
	GetCredential(ctx context.Context, providerName string) (string, error)
}

CredentialProvider retrieves API keys or tokens for providers at runtime. This allows integration with secret managers (Vault, AWS Secrets, etc.) instead of hard-coding API keys. Given to the gateway (nexus.WithCredentialProvider), it is asked before each provider call; a provider that also implements CredentialResolver is asked through that instead, so it can choose the base URL too.

type CredentialResolver

type CredentialResolver interface {
	ResolveCredential(ctx context.Context, providerName string) (Credential, error)
}

CredentialResolver is a CredentialProvider that can also choose the endpoint: a host that keeps each workspace's own key, and for an OpenAI-compatible server its URL, implements it. The gateway calls it after routing, once per provider attempt, with pipeline.TenantID(ctx) set to the request's nexus tenant ("" for an unattributed request).

Return ErrCredentialNotFound (wrapped or not) to serve the request with the provider's own key. Any other error refuses the request: nothing is sent and the gateway does not fall back to the provider's key. The error must not contain a key.

A resolver that returns a BaseURL is choosing where the gateway sends the request, its body and the key. Nexus does not check that URL; validate it as Credential.BaseURL describes before returning it.

type Delta

type Delta struct {
	Role      string     `json:"role,omitempty"`
	Content   string     `json:"content,omitempty"`
	ToolCalls []ToolCall `json:"tool_calls,omitempty"`

	// Reasoning carries extended-thinking content (o1, Claude thinking_delta,
	// DeepSeek R1 reasoning_content). Emitted alongside (or before) Content.
	Reasoning string `json:"reasoning,omitempty"`

	// Refusal is an OpenAI-style structured refusal string.
	Refusal string `json:"refusal,omitempty"`

	// Citations carries grounding / source references.
	Citations []Citation `json:"citations,omitempty"`

	// Audio carries a multi-modal audio chunk (raw bytes + optional transcript).
	Audio *AudioChunk `json:"audio,omitempty"`

	// Image carries a multi-modal image chunk (raw bytes or URL).
	Image *ImageChunk `json:"image,omitempty"`

	// Transcript is a partial speech transcript (realtime / live APIs).
	Transcript string `json:"transcript,omitempty"`

	// Native is the whole reply in the provider's own format, set on the
	// EventMessageStop chunk. The accumulator copies it to Message.Native.
	Native *NativeContent `json:"-"`
}

Delta contains the incremental content in a stream chunk.

Role/Content/ToolCalls are the legacy fields. The remaining fields are optional and populated by providers that surface richer streams.

type DimensionsAware

type DimensionsAware interface {
	SupportsDimensions(model string) bool
}

DimensionsAware is implemented by a provider that can return embeddings of a requested width (EmbeddingRequest.Dimensions). SupportsDimensions reports whether it can for model. The gateway refuses a request with Dimensions set when the routed provider does not implement this or answers false, so the provider is never called with a field it would drop.

type EmbeddingRequest

type EmbeddingRequest struct {
	Model    string   `json:"model"`
	Provider string   `json:"provider,omitempty"` // force specific provider
	Input    []string `json:"input"`
	// Dimensions asks for vectors of exactly this many numbers; 0 keeps the
	// model's default. The gateway refuses it for a provider that cannot
	// honour it (see DimensionsAware) and fails a response whose vectors
	// have another length.
	Dimensions int    `json:"dimensions,omitempty"`
	TenantID   string `json:"-"`
	KeyID      string `json:"-"`
	// Metadata is stored on the usage record, where it can be filtered by
	// (see usage.ValidateMetadata for the limits).
	Metadata map[string]string `json:"-"`
}

EmbeddingRequest for text embeddings.

type EmbeddingResponse

type EmbeddingResponse struct {
	Provider   string      `json:"provider"`
	Model      string      `json:"model"`
	Embeddings [][]float64 `json:"embeddings"`
	Usage      Usage       `json:"usage"`
}

EmbeddingResponse for embeddings.

type EnvCredential

type EnvCredential struct {
	// contains filtered or unexported fields
}

EnvCredential reads credentials from environment variables.

func NewEnvCredential

func NewEnvCredential(prefix string) *EnvCredential

NewEnvCredential creates a credential provider that reads from env vars. The environment variable name is formed as: {prefix}{PROVIDER_NAME}_API_KEY.

func (*EnvCredential) GetCredential

func (ec *EnvCredential) GetCredential(_ context.Context, providerName string) (string, error)

type EventKind

type EventKind string

EventKind identifies the semantic category of a StreamChunk.

The zero value (EventDelta) is the legacy form: Delta carries Role/Content/ ToolCalls and consumers ignore Kind. New kinds let providers surface richer signals (reasoning, message boundaries, usage, citations, multi-modal, etc.) without breaking existing consumers — kinds they don't recognize can be dropped or forwarded as-is.

const (
	// EventDelta is the default — incremental Role/Content/ToolCalls.
	EventDelta EventKind = ""

	// EventMessageStart marks the beginning of an assistant message.
	EventMessageStart EventKind = "message_start"

	// EventMessageStop marks the end of an assistant message.
	EventMessageStop EventKind = "message_stop"

	// EventReasoning carries an extended-thinking / reasoning content delta.
	// Used for OpenAI o-series, Anthropic extended thinking, DeepSeek R1, etc.
	EventReasoning EventKind = "reasoning"

	// EventToolCallDelta carries an incremental tool-call fragment
	// (function name, partial arguments JSON, or accumulator-keyed index).
	EventToolCallDelta EventKind = "tool_call_delta"

	// EventAudio carries a multi-modal audio chunk (raw bytes or transcript).
	EventAudio EventKind = "audio"

	// EventImage carries a multi-modal image chunk.
	EventImage EventKind = "image"

	// EventCitation carries a single citation reference.
	EventCitation EventKind = "citation"

	// EventUsage carries a token-usage snapshot — typically the final chunk.
	EventUsage EventKind = "usage"

	// EventHeartbeat is an upstream keepalive ping (e.g. Anthropic "ping").
	// Encoders may translate this to a wire-level keepalive comment.
	EventHeartbeat EventKind = "heartbeat"

	// EventError signals a recoverable, in-band provider error. Consumers
	// should propagate to the wire but may continue reading further events.
	EventError EventKind = "error"
)

type FreeOfCharge

type FreeOfCharge interface {
	FreeOfCharge() bool
}

FreeOfCharge is implemented by a provider whose every model, listed or not, costs exactly $0, such as local inference.

type HealthStats

type HealthStats struct {
	TotalRequests int           `json:"total_requests"`
	Successes     int           `json:"successes"`
	Failures      int           `json:"failures"`
	SuccessRate   float64       `json:"success_rate"`
	AvgLatency    time.Duration `json:"avg_latency"`
	P99Latency    time.Duration `json:"p99_latency"`
	LastSuccess   time.Time     `json:"last_success,omitempty"`
	LastFailure   time.Time     `json:"last_failure,omitempty"`
	LastError     string        `json:"last_error,omitempty"`
}

HealthStats contains health metrics for a provider.

type HealthTracker

type HealthTracker interface {
	// RecordSuccess records a successful provider call.
	RecordSuccess(providerName string, latency time.Duration)

	// RecordFailure records a failed provider call.
	RecordFailure(providerName string, err error)

	// Stats returns health statistics for a provider.
	Stats(providerName string) *HealthStats

	// IsHealthy returns true if the provider is considered healthy.
	IsHealthy(providerName string) bool

	// AllStats returns stats for all tracked providers.
	AllStats() map[string]*HealthStats
}

HealthTracker monitors provider health for routing decisions.

func NewHealthTracker

func NewHealthTracker() HealthTracker

NewHealthTracker creates a new in-memory health tracker.

type ImageChunk

type ImageChunk struct {
	MimeType string `json:"mime_type,omitempty"`
	Data     []byte `json:"data,omitempty"`
	URL      string `json:"url,omitempty"`
	Partial  bool   `json:"partial,omitempty"` // true while incremental, false on final
}

ImageChunk is a multi-modal image fragment.

type JSONSchemaDef

type JSONSchemaDef struct {
	// Name is a free-form identifier OpenAI requires (any non-empty string).
	Name string `json:"name"`
	// Description is optional human-readable hint for the model.
	Description string `json:"description,omitempty"`
	// Schema is the JSON Schema document (object with type/properties/required).
	Schema any `json:"schema"`
	// Strict enables strict mode — model output must validate against Schema.
	Strict bool `json:"strict,omitempty"`
}

JSONSchemaDef wraps a JSON schema for OpenAI-style structured outputs. The wire format is `{"name": ..., "strict": true, "schema": {...}}` nested under `response_format.json_schema`.

type Message

type Message struct {
	Role       string     `json:"role"` // system, user, assistant, tool
	Content    any        `json:"content"`
	Name       string     `json:"name,omitempty"`
	ToolCalls  []ToolCall `json:"tool_calls,omitempty"`
	ToolCallID string     `json:"tool_call_id,omitempty"`

	// Native is the assistant reply in the provider's own format. Keep it
	// with the message and pass it back unchanged; see NativeContent.
	Native *NativeContent `json:"-"`
}

Message represents a conversation message.

type Model

type Model struct {
	ID            string       `json:"id"`       // e.g., "gpt-4o"
	Provider      string       `json:"provider"` // e.g., "openai"
	Name          string       `json:"name"`     // human-readable
	Capabilities  Capabilities `json:"capabilities"`
	ContextWindow int          `json:"context_window"`
	MaxOutput     int          `json:"max_output"`
	Pricing       Pricing      `json:"pricing"`
}

Model describes an available LLM model.

type NativeContent

type NativeContent struct {
	// Provider names the provider whose wire format Content is in.
	Provider string `json:"provider"`
	// Model is the model that produced the reply.
	Model string `json:"model"`
	// Content is the reply's content as that provider returned it.
	Content json.RawMessage `json:"content"`
}

NativeContent is an assistant reply in the provider's own wire format, kept beside the unified fields so the turn can be replayed exactly.

Some models reject a history whose earlier turns were edited, signed thinking blocks included. Store Choices[0].Message.Native with the assistant message and send it back unchanged on the next request: a provider that recognises it sends Content as is while the message still matches it, and rebuilds the turn from the unified fields otherwise.

It is never serialized, and the response and stream caches drop it on every backend, so a cached reply carries none. An output guard that rewrites a message drops it too.

type Pricing

type Pricing struct {
	InputPerMillion          money.USD `json:"input_per_million"`
	OutputPerMillion         money.USD `json:"output_per_million"`
	EmbeddingPerMillion      money.USD `json:"embedding_per_million,omitzero"`
	CacheReadPerMillion      money.USD `json:"cache_read_per_million,omitzero"`
	CacheWritePerMillion     money.USD `json:"cache_write_per_million,omitzero"`      // 5-minute cache writes
	CacheWriteLongPerMillion money.USD `json:"cache_write_long_per_million,omitzero"` // 1-hour cache writes
	LongContextAbove         int       `json:"long_context_above,omitempty"`          // prompt tokens; 0 means no tier
	LongContext              *Pricing  `json:"long_context,omitempty"`
	Free                     bool      `json:"free,omitempty"`
}

Pricing is a model's list price per million tokens. When Free is set the model costs exactly $0. Otherwise a model with both InputPerMillion and OutputPerMillion at zero is unpriced (its cost is unknown, never $0); when only one of them is set, the other kind of token is charged at $0. Embeddings use EmbeddingPerMillion.

Cache reads, 5-minute cache writes and 1-hour cache writes have their own prices. A cache price left at zero charges those tokens at InputPerMillion; a 1-hour write with no price of its own is charged at CacheWritePerMillion first. When LongContextAbove is set and a request's prompt tokens exceed it, every token of that request, cache reads and writes included, is charged at LongContext instead.

type Provider

type Provider interface {
	// Name returns the provider identifier (e.g., "openai", "anthropic").
	Name() string

	// Capabilities returns what this provider supports.
	Capabilities() Capabilities

	// Models returns the list of available models.
	Models(ctx context.Context) ([]Model, error)

	// Complete sends a chat completion request.
	Complete(ctx context.Context, req *CompletionRequest) (*CompletionResponse, error)

	// CompleteStream sends a streaming chat completion request.
	CompleteStream(ctx context.Context, req *CompletionRequest) (Stream, error)

	// Embed sends an embedding request (if supported).
	Embed(ctx context.Context, req *EmbeddingRequest) (*EmbeddingResponse, error)

	// Healthy returns true if the provider is reachable.
	Healthy(ctx context.Context) bool
}

Provider abstracts an LLM provider (OpenAI, Anthropic, Ollama, etc.)

type Registry

type Registry interface {
	// Register adds a provider to the registry.
	Register(p Provider)

	// Get returns a provider by name.
	Get(name string) (Provider, bool)

	// All returns all registered providers.
	All() []Provider

	// Count returns the number of registered providers.
	Count() int

	// ForModel returns the providers whose Models list the model, in
	// registration order. A provider whose Models fails is left out.
	ForModel(model string) []Provider

	// WithCapability returns providers with a specific capability.
	WithCapability(capability string) []Provider

	// Healthy returns only healthy providers.
	Healthy(ctx context.Context) []Provider
}

Registry manages registered LLM providers.

func NewRegistry

func NewRegistry() Registry

NewRegistry creates an empty provider registry.

type ResponseFormat

type ResponseFormat struct {
	Type       string         `json:"type"` // text, json_object, json_schema
	Schema     any            `json:"schema,omitempty"`
	JSONSchema *JSONSchemaDef `json:"json_schema,omitempty"`
}

ResponseFormat specifies the desired output format.

For OpenAI-compatible structured outputs (Type = "json_schema"), populate JSONSchema with the schema wrapped in a JSONSchemaDef — that is the only shape OpenAI, LM Studio, and other OAI-compatible servers accept. The legacy Schema field is preserved for providers that take a bare schema (most do not) and for backward compatibility.

type StaticCredential

type StaticCredential struct {
	// contains filtered or unexported fields
}

StaticCredential always returns the same key. Useful for testing.

func NewStaticCredential

func NewStaticCredential(credentials map[string]string) *StaticCredential

NewStaticCredential creates a credential provider from a static map.

func (*StaticCredential) GetCredential

func (sc *StaticCredential) GetCredential(_ context.Context, providerName string) (string, error)

type Stream

type Stream interface {
	// Next returns the next chunk. Returns io.EOF when done.
	Next(ctx context.Context) (*StreamChunk, error)

	// Close releases resources.
	Close() error

	// Usage returns final usage after stream completes.
	Usage() *Usage
}

Stream represents a streaming response from a provider.

type StreamChunk

type StreamChunk struct {
	ID           string    `json:"id"`
	Provider     string    `json:"provider"`
	Model        string    `json:"model"`
	Kind         EventKind `json:"kind,omitempty"`
	Delta        Delta     `json:"delta"`
	FinishReason string    `json:"finish_reason,omitempty"`

	// Optional payloads — populated based on Kind.
	Usage   *Usage `json:"usage,omitempty"`   // EventUsage / final delta
	Err     string `json:"error,omitempty"`   // EventError
	Created int64  `json:"created,omitempty"` // unix ms; used for cache-replay timing
}

StreamChunk is a single piece of a streamed response.

Kind discriminates the payload: legacy chunks have Kind == "" (EventDelta) and read Delta.Content / Delta.ToolCalls. Newer kinds populate the matching optional fields (Usage, Err) or extended Delta fields (Reasoning, Audio…).

func Drain

func Drain(ch <-chan StreamChunk) []StreamChunk

Drain reads everything from a channel adapter into a slice. Useful for tests; not recommended in production where you want to process events as they arrive.

func NewErrorChunk

func NewErrorChunk(providerName string, err error) *StreamChunk

NewErrorChunk builds an in-band error frame. The provider field tags which component produced the error so wire encoders can attribute it.

func NewHeartbeatChunk

func NewHeartbeatChunk(providerName string) *StreamChunk

NewHeartbeatChunk builds a keepalive frame. Encoders may translate to a wire-level keepalive (SSE comment, WS ping) instead of forwarding the JSON.

func NewReasoningChunk

func NewReasoningChunk(providerName, model, reasoning string) *StreamChunk

NewReasoningChunk builds a reasoning/thinking delta.

func NewUsageChunk

func NewUsageChunk(providerName, model string, usage *Usage) *StreamChunk

NewUsageChunk builds a chunk that carries final/intermediate usage. Used by providers that surface usage as a separate SSE event (OpenAI with stream_options.include_usage, Anthropic message_delta) and by middleware that synthesizes a usage frame for downstream encoders.

type StreamEvents

type StreamEvents interface {
	Events(ctx context.Context) <-chan StreamChunk
}

StreamEvents is an optional interface streams may implement to expose a channel-based consumer surface in addition to the iterator-style Next(). Use provider.NewChannelAdapter as a default fallback for streams that only implement Stream.

type ThinkingConfig

type ThinkingConfig struct {
	Enabled         bool `json:"enabled,omitempty"`
	BudgetTokens    int  `json:"budget_tokens,omitempty"`
	IncludeThinking bool `json:"include_thinking,omitempty"`
	// Effort steers how much the model thinks and spends: "low", "medium",
	// "high", "xhigh" or "max". Empty keeps the model's default. A provider
	// that knows the model rejects a level the model does not accept.
	Effort string `json:"effort,omitempty"`
}

ThinkingConfig controls extended thinking / reasoning behavior.

type Tool

type Tool struct {
	Type     string       `json:"type"` // function
	Function ToolFunction `json:"function"`
}

Tool definition.

type ToolCall

type ToolCall struct {
	ID       string       `json:"id"`
	Type     string       `json:"type"`
	Function ToolCallFunc `json:"function"`
}

ToolCall represents a tool invocation from the assistant.

type ToolCallFunc

type ToolCallFunc struct {
	Name      string `json:"name"`
	Arguments string `json:"arguments"`
}

ToolCallFunc is the function call details within a ToolCall.

type ToolChoice

type ToolChoice struct {
	Mode     ToolChoiceMode
	Function string // set when Mode is ToolChoiceFunction
}

ToolChoice is the parsed form of CompletionRequest.ToolChoice.

func ParseToolChoice

func ParseToolChoice(v any) (tc ToolChoice, ok bool, err error)

ParseToolChoice parses a CompletionRequest.ToolChoice value. It reads OpenAI's shape ("auto", "none", "required", or {"type":"function","function":{"name":...}}) and, for callers that already speak Anthropic, {"type":"auto"|"any"|"none"} and {"type":"tool","name":...}. The bare string "any" is read as required. ok is false for nil. A value it cannot read fails with an error that wraps ErrNotSupported, so the gateway refuses the request as the caller's mistake and does not retry it.

func (ToolChoice) OpenAI

func (tc ToolChoice) OpenAI() any

OpenAI returns tc in OpenAI's tool_choice shape: the string "auto", "none" or "required", or {"type":"function","function":{"name":...}}. It returns nil for a mode it does not know, such as the zero ToolChoice.

type ToolChoiceMode

type ToolChoiceMode string

ToolChoiceMode is how a request constrains the model's use of tools.

const (
	// ToolChoiceAuto lets the model decide whether to call a tool.
	ToolChoiceAuto ToolChoiceMode = "auto"
	// ToolChoiceNone forbids tool calls.
	ToolChoiceNone ToolChoiceMode = "none"
	// ToolChoiceRequired forces the model to call some tool.
	ToolChoiceRequired ToolChoiceMode = "required"
	// ToolChoiceFunction forces the model to call one named tool.
	ToolChoiceFunction ToolChoiceMode = "function"
)

type ToolFunction

type ToolFunction struct {
	Name        string `json:"name"`
	Description string `json:"description,omitempty"`
	Parameters  any    `json:"parameters,omitempty"` // JSON Schema
}

ToolFunction describes a function tool.

type TransportWrapper

type TransportWrapper interface {
	WrapTransport(wrap func(http.RoundTripper) http.RoundTripper)
}

TransportWrapper is implemented by a provider that sends its requests through an HTTP client it owns and lets the gateway wrap that client's transport, to trace every outbound call for example. Every HTTP provider in this repo implements it; the WebSocket ones (geminilive, openairealtime) do not.

The gateway calls WrapTransport once, from Initialize, with the function given to nexus.WithProviderTransport. wrap receives the client's current transport (nil means http.DefaultTransport) and returns the one to use. A nil result leaves the transport as it was. Call it before the first request, never while one is in flight.

type Usage

type Usage struct {
	PromptTokens         int `json:"prompt_tokens"`
	CompletionTokens     int `json:"completion_tokens"`
	ThinkingTokens       int `json:"thinking_tokens,omitempty"`
	CacheReadTokens      int `json:"cache_read_tokens,omitempty"`
	CacheWriteTokens     int `json:"cache_write_tokens,omitempty"`
	CacheWriteLongTokens int `json:"cache_write_long_tokens,omitempty"`
	TotalTokens          int `json:"total_tokens"`
}

Usage tracks token consumption. PromptTokens counts every input token, cache reads and cache writes included. CacheReadTokens, CacheWriteTokens and CacheWriteLongTokens are subsets of it: CacheWriteTokens counts every cache write, and CacheWriteLongTokens is the part of those written for an hour rather than five minutes.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL