provider

package
v1.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 20, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Overview

Package provider implements the real model execution layer behind the Model Gateway decision chain: an HTTP executor covering two wire dialects.

The default dialect is the OpenAI-compatible /chat/completions spoken by vLLM, Qwen, DeepSeek, GLM and any compatible endpoint. The Anthropic Messages dialect (/v1/messages) is selected per provider with Config.Protocol. Both dialects share the same guarantees: streaming, bounded retries, a per-provider circuit breaker, exact usage extraction and provider request-id capture.

Credentials never cross this boundary in either direction: the API key is resolved from the configured environment variable per request, is attached only to the outbound authentication header of the selected dialect, and is stripped from every error message and telemetry surface.

Index

Constants

View Source 
const (
	// ProtocolOpenAI is the /chat/completions dialect spoken by OpenAI
	// itself and by vLLM, Qwen, DeepSeek, GLM and other compatible
	// endpoints. It is the default.
	ProtocolOpenAI = "openai"
	// ProtocolAnthropic is the Anthropic Messages dialect: /v1/messages
	// with a top-level system prompt, tool_use/tool_result content blocks
	// and typed server-sent events.
	ProtocolAnthropic = "anthropic"
)

Protocol identifiers accepted in provider configuration.

Variables

View Source 
var (
	// ErrProviderUnavailable reports a provider endpoint that could not be
	// reached after the bounded retry budget, or one whose circuit breaker
	// is open (fast fail).
	ErrProviderUnavailable = errors.New("model provider unavailable")
	// ErrProviderRejected reports a non-retryable provider refusal (4xx other
	// than 408/429): authentication, permission, malformed request or model.
	ErrProviderRejected = errors.New("model provider rejected the request")
	// ErrStreamAborted reports a stream that failed after content had already
	// been delivered; it is never retried, because a retry would duplicate
	// billed output.
	ErrStreamAborted = errors.New("model provider stream aborted after delivery")
	// ErrCircuitOpen is returned by a shared breaker backend when another
	// gateway instance has opened or is probing the provider circuit.
	ErrCircuitOpen = errors.New("model provider circuit breaker open")
)

Sentinel errors the invoker maps onto ledger outcomes.

Functions

func UsageKnownZero 

func UsageKnownZero(err error) bool

UsageKnownZero reports whether an executor error proves the provider did not accept billable generation (for example a definitive 4xx or 429). Every ambiguous transport/5xx/partial-stream outcome returns false.

Types

type CapabilityProfile 

type CapabilityProfile struct {
	StreamUsage    *bool  `json:"streamUsage,omitempty"`
	ToolCalling    *bool  `json:"toolCalling,omitempty"`
	JSONSchema     *bool  `json:"jsonSchema,omitempty"`
	Reasoning      *bool  `json:"reasoning,omitempty"`
	Usage          *bool  `json:"usage,omitempty"`
	RequestID      *bool  `json:"requestId,omitempty"`
	MaxTokensField string `json:"maxTokensField,omitempty"` // max_tokens|max_completion_tokens
}

CapabilityProfile makes OpenAI-compatible dialect differences explicit. Pointer booleans distinguish an omitted field (safe default) from false.

type CircuitAcquire 

type CircuitAcquire struct {
	Provider  string
	Threshold int
	Cooldown  time.Duration
	ProbeTTL  time.Duration
	Now       time.Time
}

type CircuitBreaker 

type CircuitBreaker interface {
	AcquireCircuit(context.Context, CircuitAcquire) (CircuitPermit, error)
	RecordCircuit(context.Context, CircuitRecord) error
}

CircuitBreaker is the optional distributed provider-health coordination surface. PostgreSQL implements it so every gateway instance observes the same open/half-open state; a nil breaker retains the process-local fallback.

type CircuitPermit 

type CircuitPermit struct{ ProbeToken string }

type CircuitRecord 

type CircuitRecord struct {
	Provider   string
	ProbeToken string
	Success    bool
	Retryable  bool
	Threshold  int
	Now        time.Time
}

type Config 

type Config struct {
	Name                string            `json:"name"`
	BaseURL             string            `json:"baseUrl"`
	APIKeyEnv           string            `json:"apiKeyEnv"`
	APIKey              string            `json:"-"` // programmatic (tests); file loads must use APIKeyEnv
	Timeout             time.Duration     `json:"-"`
	TimeoutMs           int64             `json:"timeoutMs"`
	MaxAttempts         int               `json:"maxAttempts"`
	BreakerOpens        int               `json:"breakerOpens"`
	BreakerCoolMs       int64             `json:"breakerCooldownMs"`
	MaxConcurrent       int               `json:"maxConcurrent"`
	SupportsIdempotency bool              `json:"supportsIdempotency"`
	IdempotencyHeader   string            `json:"idempotencyHeader"`
	RetryAmbiguous      bool              `json:"retryAmbiguous"`
	HealthPath          string            `json:"healthPath"`
	HealthMethod        string            `json:"healthMethod"`
	Capabilities        CapabilityProfile `json:"capabilities,omitempty"`
	// Tokenizer names the token estimator used for this provider's budget
	// reservations: "heuristic" (default, script-aware) or "conservative"
	// (uncharacterized tokenizers). Provider-reported usage remains the
	// authoritative settlement.
	Tokenizer string `json:"tokenizer,omitempty"`
	// Protocol selects the wire dialect: "openai" (the default,
	// /chat/completions) or "anthropic" (/v1/messages). An empty value
	// keeps the OpenAI-compatible behaviour, so existing configurations do
	// not change meaning.
	Protocol string `json:"protocol,omitempty"`
	// ExtraHeaders are added to every outbound request before the
	// dialect's own headers, so the authentication header of the selected
	// dialect always wins. They carry provider hints (for example the
	// OpenRouter attribution headers) and never hold credentials.
	ExtraHeaders map[string]string `json:"extraHeaders,omitempty"`
}

Config is one provider endpoint. APIKeyEnv names the environment variable the key is resolved from on every call; the key itself is never stored.

type Executor 

type Executor struct {
	// contains filtered or unexported fields
}

Executor executes invocations against one provider endpoint.

func NewExecutor 

func NewExecutor(config Config, client *http.Client) *Executor

NewExecutor builds the executor for one provider endpoint.

func (*Executor) Complete 

func (e *Executor) Complete(ctx context.Context, invocation Invocation) (Result, error)

Complete performs one non-streaming invocation with bounded retries.

func (*Executor) Health 

func (e *Executor) Health(ctx context.Context) error

Health probes the endpoint with a short-lived request against the provider's model list. It is the half-open probe of the circuit breaker as well as the readiness signal.

func (*Executor) Name 

func (e *Executor) Name() string

Name returns the provider name model references resolve against.

func (*Executor) Stream 

func (e *Executor) Stream(ctx context.Context, invocation Invocation, onDelta func(string)) (Result, error)

Stream performs one streaming invocation, delivering content deltas to onDelta as they arrive. Retries happen only before the first delta is delivered; after that, a failure aborts the stream without retry so no output is duplicated.

func (*Executor) StreamObserved 

func (e *Executor) StreamObserved(ctx context.Context, invocation Invocation, observer StreamObserver) (Result, error)

StreamObserved performs one streaming invocation with complete generation metering and retry-delivery semantics.

func (*Executor) Tokenizer 

func (e *Executor) Tokenizer() string

Tokenizer returns the configured token-estimator name for budget reservations on this provider.

func (*Executor) WithCircuitBreaker 

func (e *Executor) WithCircuitBreaker(shared CircuitBreaker) *Executor

WithCircuitBreaker installs a distributed breaker backend.

type Invocation 

type Invocation struct {
	ModelName       string           `json:"model"`
	Messages        []Message        `json:"messages"`
	Temperature     *float64         `json:"temperature,omitempty"`
	MaxOutputTokens int32            `json:"max_tokens,omitempty"`
	Stream          bool             `json:"stream"`
	Tools           []ToolDefinition `json:"tools,omitempty"`
	IdempotencyKey  string           `json:"-"`
	TraceID         string           `json:"-"`
	TaskID          string           `json:"-"`
	AttemptID       string           `json:"-"`
	ModelCallID     string           `json:"-"`
}

Invocation is one provider request.

type Message 

type Message struct {
	Role       string     `json:"role"`
	Content    string     `json:"content"`
	ToolCallID string     `json:"tool_call_id,omitempty"`
	ToolCalls  []ToolCall `json:"tool_calls,omitempty"`
}

Message is one chat turn. Role is system|user|assistant|tool; ToolCalls carries OpenAI-style function calls on assistant turns; ToolCallID binds a tool-role turn to its call.

type Registry 

type Registry struct {
	// contains filtered or unexported fields
}

Registry resolves provider names (the first path segment of a model reference, e.g. "deepseek" in deepseek/deepseek-chat) to their executors. An empty registry fails closed: every reference resolves to an explicit "no execution endpoint configured" error rather than silently bypassing the Model Gateway.

func LoadRegistryFile 

func LoadRegistryFile(path string) (*Registry, error)

LoadRegistryFile reads a provider configuration file. An empty path yields an empty (fail-closed) registry.

func NewRegistry 

func NewRegistry() *Registry

NewRegistry builds an empty registry. client may be nil (a default HTTP client is used per executor).

func (*Registry) Names 

func (r *Registry) Names() []string

Names lists registered provider names in stable order.

func (*Registry) Register 

func (r *Registry) Register(config Config) error

Register adds one provider endpoint configuration. A provider name is declared once: registering an existing name fails instead of silently replacing the endpoint (a duplicate in static configuration is a mistake, not a rotation).

func (*Registry) RegisterRoute 

func (r *Registry) RegisterRoute(route Route) error

RegisterRoute adds one explicit logical-model route. A modelRef is routed once: re-declaring it fails instead of silently overriding the previous route.

func (*Registry) Resolve 

func (r *Registry) Resolve(name string) (*Executor, error)

Resolve returns the executor for one provider name.

func (*Registry) ResolveModel 

func (r *Registry) ResolveModel(modelRef string) (*Executor, string, error)

ResolveModel resolves an exact logical route, falling back to the legacy provider/model convention for backwards compatibility.

func (*Registry) UseCircuitBreaker 

func (r *Registry) UseCircuitBreaker(breaker CircuitBreaker)

UseCircuitBreaker applies one distributed breaker to current and future provider executors.

type Result 

type Result struct {
	Content           string     `json:"content"`
	ToolCalls         []ToolCall `json:"toolCalls,omitempty"`
	FinishReason      string     `json:"finishReason"`
	InputTokens       int64      `json:"inputTokens"`
	OutputTokens      int64      `json:"outputTokens"`
	ProviderRequestID string     `json:"providerRequestId"`
	UsageReported     bool       `json:"usageReported"`
}

Result is the outcome of one invocation. Usage is the exact provider-reported token counts (never estimated by the caller).

type Route 

type Route struct {
	ModelRef string `json:"modelRef"`
	Provider string `json:"provider"`
	Model    string `json:"model"`
}

Route separates a tenant-visible logical model reference from the provider endpoint and wire model selected by operations.

type StreamObserver 

type StreamObserver struct {
	OnContent   func(string)
	OnGenerated func(string)
	OnThinking  func(string)
}

StreamObserver separates user-visible content from all provider-generated bytes. Budget meters observe content, tool names and tool arguments while callers only receive textual content.

OnThinking receives the provider's reasoning channel separately: the Anthropic thinking blocks and the reasoning fields of OpenAI-compatible endpoints. Reasoning is provider-generated text that is not part of the answer, so it is metered like any other generated byte but never delivered as content.

type ToolCall 

type ToolCall struct {
	ID        string `json:"id"`
	Name      string `json:"name"`
	Arguments string `json:"arguments"`
}

ToolCall is one function call the model requested.

type ToolDefinition 

type ToolDefinition struct {
	Name        string          `json:"name"`
	Description string          `json:"description"`
	Parameters  json.RawMessage `json:"parameters"`
}

ToolDefinition declares one callable tool to the model. The internal shape stays flat; encodeInvocation converts it into the OpenAI-compatible function-tool wire shape ({"type":"function","function":{…}}) at the request boundary.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.