providers

package
v2.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 6, 2026 License: MIT Imports: 24 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ErrExtensionSignInRequired = errors.New("sign in to this provider first")

ErrExtensionSignInRequired reports a provider whose credential comes from a sign-in that has not happened.

View Source 
var ExtensionDaemonSecret = func() (string, error) {
	credential, err := auth.GetCredential(auth.ExtensionDaemonKey)
	if err != nil || credential == nil {
		return "", err
	}
	return credential.AccessToken, nil
}

ExtensionDaemonSecret returns the extension daemon's shared secret. It reads the auth store on every call, so a secret changed in the UI applies to the next provider built. Tests replace it.

View Source 
var ExtensionRejectedCredential = func(ctx context.Context, endpoint, provider, key string, rejected *core.Credential) (*core.Credential, error) {
	coordinator, err := extensionCoordinator(endpoint, provider)
	if err != nil {
		return nil, err
	}
	current, err := auth.DefaultTokenStore().Load(ctx, key)
	if errors.Is(err, tokenstore.ErrNotFound) {
		return nil, ErrExtensionSignInRequired
	}
	if err != nil {
		return nil, err
	}
	if rejected != nil && current.AccessToken != "" && current.AccessToken != rejected.Token {
		return core.CredentialFromRecord(key, current), nil
	}
	record, err := coordinator.Rejected(ctx, key, current)
	if err != nil {
		return nil, err
	}
	return core.CredentialFromRecord(key, record), nil
}

ExtensionRejectedCredential returns the credential to use instead of rejected, which the upstream refused with 401 although it had not expired: one another request has refreshed meanwhile, or else a refreshed one. As with ExtensionSignedInCredential, a refresh token is spent once. Tests replace it.

View Source 
var ExtensionSignedInCredential = func(ctx context.Context, endpoint, provider, key string) (*core.Credential, error) {
	coordinator, err := extensionCoordinator(endpoint, provider)
	if err != nil {
		return nil, err
	}
	record, err := coordinator.Token(ctx, key)
	if errors.Is(err, tokenstore.ErrNotFound) {
		return nil, ErrExtensionSignInRequired
	}
	if err != nil {
		return nil, err
	}
	return core.CredentialFromRecord(key, record), nil
}

ExtensionSignedInCredential returns the current credential stored under key for the daemon provider the owner signed in to, refreshing it through the daemon when due. Every process shares the auth store's leases, so a refresh token is spent once. Tests replace it.

Functions

func CheckExtensionEndpoint 

func CheckExtensionEndpoint(endpoint, secret string) error

CheckExtensionEndpoint reports an extension daemon endpoint Compa must not send the daemon's shared secret to: anything but an http(s) URL, plain http to a host other than this machine's loopback interface, where the secret would cross the network in clear text, and a daemon on another machine without a secret, which would serve any caller.

func Classify 

func Classify(err error) core.ProviderErrorClassification

Classify returns err's classification as Compa's routing reads it: core's, except for these statuses:

  • 401, 402, 403 and 404 move on to the next target, which may hold a valid key, a balance, access or the model, without counting against the instance's health: it answered.
  • 529, Anthropic's "overloaded", is transient as a 503 is: it may be retried and fail over, and counts against the target.

400 and 422, a request every target would reject, stay terminal, and a failure after output reached the caller stays terminal whatever its status.

func ExtensionControlClient 

func ExtensionControlClient() *http.Client

ExtensionControlClient returns the client for a daemon's control routes (info, models, refresh and sign-in): it never goes through a proxy and never follows redirects.

func HealthPolicy 

func HealthPolicy() execution.HealthPolicy

HealthPolicy is the health policy of every instance. Retry-After is honoured: an instance the upstream asked to wait is not tried until then.

func InstanceCredential 

func InstanceCredential(instance *config.ProviderInstanceConfig, secret string) (coretransport.CredentialSource, error)

InstanceCredential returns the source of the credential each request of instance carries. secret is what the instance's auth_connection_ref resolved to: an API key, or an extension provider's token. A signed-in extension provider's credential is read, and refreshed when due, per request.

func InstanceCredentialRefresher 

func InstanceCredentialRefresher(instance *config.ProviderInstanceConfig) coretransport.CredentialRefresher

InstanceCredentialRefresher returns how a credential of instance that the upstream rejected with 401 is replaced, or nil when it cannot be: only a signed-in extension provider's OAuth credential is refreshed.

func InstanceRuntimeType 

func InstanceRuntimeType(instance *config.ProviderInstanceConfig) (string, bool)

InstanceRuntimeType returns the core runtime type that serves instance, or false when Compa runs none for it. Extension instances run on the extension daemon and report "extension".

func ModelKey 

func ModelKey(provider, model string) string

ModelKey returns a canonical "provider/model" key for deduplication.

func NewCoreProvider 

func NewCoreProvider(instance *config.ProviderInstanceConfig) (core.Provider, error)

NewCoreProvider builds the core provider of instance, able to serve Chat Completions: a provider that does not speak it natively is wrapped in a translation.Adapter. Its HTTP client carries the instance's headers and honours its proxy and request timeout.

func NewHealthTracker 

func NewHealthTracker(now func() time.Time) *execution.HealthTracker

NewHealthTracker returns an instance health tracker with HealthPolicy and core's default observation of errors. now is its clock; nil uses time.Now.

func NormalizeProvider 

func NormalizeProvider(provider string) string

NormalizeProvider returns the canonical llmgw-core registry id of provider, resolving the registry's aliases. An identifier the registry does not know comes back lower-cased and trimmed.

func RegistryAdapter 

func RegistryAdapter(entry coreproviders.RegistryEntry) (string, bool)

RegistryAdapter returns the adapter an instance of a registry entry records: the compatible adapter of its wire, or native for any other runtime Compa runs.

func RetryDelay 

func RetryDelay(err error, backoff, maxWait time.Duration) (time.Duration, bool)

RetryDelay reports whether a request that failed with err may be repeated and how long to wait first: only a retryable failure may be, after the larger of backoff and the upstream's Retry-After. A Retry-After longer than maxWait is not waited out: the failure stands.

Types

type Attachment 

type Attachment = protocoltypes.Attachment

type CacheControl 

type CacheControl = protocoltypes.CacheControl

type CallSpec 

type CallSpec struct {
	// DisplayName names the target, "instance-id/model-id".
	DisplayName string
	// ModelID is the model the request names.
	ModelID string
	// Streaming turns provider streaming on. It is on unless the instance
	// turns it off.
	Streaming bool
	// ThinkingLevel is the extended-thinking level; empty leaves it to the
	// agent.
	ThinkingLevel string
	// RPM caps requests per minute; 0 is unlimited.
	RPM int
	// MaxTokensField names the request's max-tokens field.
	MaxTokensField string
	// ToolSchemaTransform adapts tool schemas ("" or "simple").
	ToolSchemaTransform string
	// ExtraBody holds fields merged into every request body.
	ExtraBody map[string]any
}

CallSpec is what the agent pipeline reads about one model call: one model on one provider instance, with the instance's runtime settings. It is never persisted and carries no credential.

type ContentBlock 

type ContentBlock = protocoltypes.ContentBlock

type ExtraContent 

type ExtraContent = protocoltypes.ExtraContent

type Failover 

type Failover struct {
	// contains filtered or unexported fields
}

Failover runs a request on its candidates with llmgw-core's execution primitives. Core's classification says whether a failure may be repeated, may move on to the next candidate or ends the request, and what Retry-After the upstream asked for; Classify adjusts it for a few statuses (see there).

  • execution.Execute tries the candidates in order. A retryable or failover failure moves on to the next candidate; a terminal one - an invalid request, an unclassified error, or a failure after output reached the caller - ends the request.
  • A shared execution.HealthTracker keeps each target's health, a model on an instance (FallbackCandidate.HealthKey): a streak of circuit failures opens its circuit, and a Retry-After cools it down. A candidate that is unavailable is skipped, and when every candidate is, the request fails with an *execution.UnavailableError. A request with a single candidate has nowhere else to go, so it is never held back: its outcome is only recorded, and the caller's retry honours the failure's Retry-After.
  • Streaming pushes output to the caller as it arrives, so a run whose stream failed after visible output returns an *execution.AfterOutputError: terminal and neutral for health, so nothing fails over once output was seen.
  • Core has no client-side request rate limit, so each instance's RPM limit stays Studio's: a candidate whose limit is saturated moves on unrecorded while another candidate remains, and the last candidate waits for its token.

A nil *Failover tries each candidate once without health or rate limits.

func NewFailover 

func NewFailover(limits *RateLimiterRegistry) *Failover

NewFailover returns a Failover with a fresh health tracker and limits, which may be nil for no rate limits.

func NewFailoverWith 

func NewFailoverWith(health *execution.HealthTracker, limits *RateLimiterRegistry, now func() time.Time) *Failover

NewFailoverWith returns a Failover on health and limits with the clock now, which must be health's clock; nil uses time.Now.

func (*Failover) Execute 

func (f *Failover) Execute(
	ctx context.Context,
	candidates []FallbackCandidate,
	run func(ctx context.Context, candidate FallbackCandidate) (*LLMResponse, error),
) (FailoverResult, error)

Execute runs the request on candidates in order until one serves.

The error is nil when a candidate served. Otherwise it is the context's error, the error of the last candidate tried, an *execution.UnavailableError when every candidate's instance was unavailable, or execution.ErrNoCandidates.

func (*Failover) Health 

func (f *Failover) Health() *execution.HealthTracker

Health returns the instance health tracker, nil for a nil Failover.

func (*Failover) WithLimits 

func (f *Failover) WithLimits(limits *RateLimiterRegistry) *Failover

WithLimits returns a Failover that shares f's instance health and limits requests with limits, so a config reload keeps what health has learned.

type FailoverResult 

type FailoverResult = execution.Result[FallbackCandidate, *LLMResponse]

FailoverResult is a request's outcome: the serving candidate, its response, and the trace of every candidate reached.

type Failure 

type Failure struct {
	// Class names the kind of failure: the class the error gives itself,
	// else the one its status or a transport failure implies. It is empty
	// for an error core cannot classify, such as a cancellation.
	Class core.ProviderErrorClass
	// StatusCode is the upstream's HTTP status, zero when none was received.
	StatusCode int
	// RetryAfter is how long the upstream, or health, asked callers to wait.
	RetryAfter time.Duration
	// Disposition is what the error permits: retry, failover or nothing.
	Disposition core.Disposition
	// AfterOutput reports a stream that failed after output reached the
	// caller; the other fields describe that stream's own error, but the
	// disposition is terminal.
	AfterOutput bool
	// Unavailable reports that every candidate's instance was unavailable,
	// its circuit open or cooling down, so none was tried.
	Unavailable bool
	// Timeout reports a transport failure that was a timeout.
	Timeout bool
}

Failure is what core's classification says about a request's error, for the retry decision and the message a user reads.

func DescribeFailure 

func DescribeFailure(err error) Failure

DescribeFailure reads err's classification.

type FallbackCandidate 

type FallbackCandidate struct {
	Provider    string
	Model       string
	DisplayName string // selection label shown and persisted, such as "instance-id/model-id"
	RPM         int    // requests per minute; 0 means unrestricted
	IdentityKey string // optional stable identity, such as the owning instance
	ConfigKey   string // optional stable configuration identity, such as the exact target
}

FallbackCandidate is one target a request may run on: an instance's model, in the order a route lists its targets.

func (FallbackCandidate) HealthKey 

func (c FallbackCandidate) HealthKey() string

HealthKey returns the key of the candidate's health: its model on its instance. A model an aggregator cannot serve for a while holds back that model only, not the instance's other models.

func (FallbackCandidate) InstanceKey 

func (c FallbackCandidate) InstanceKey() string

InstanceKey returns the key of the instance the candidate runs on. It keys the instance's RPM limit, which the instance's runtime settings set. A candidate without an instance identity falls back to its stable key.

func (FallbackCandidate) StableKey 

func (c FallbackCandidate) StableKey() string

StableKey returns the candidate's config-level identity when available, otherwise it falls back to the runtime provider/model key. It keys the provider built for the candidate.

type FunctionCall 

type FunctionCall = protocoltypes.FunctionCall

type GoogleExtra 

type GoogleExtra = protocoltypes.GoogleExtra

type InstanceCatalog 

type InstanceCatalog struct {
	InstanceID string
	Models     []string
}

InstanceCatalog is the model identity snapshot owned by one configured provider instance. Callers load this from their catalog persistence layer.

type InstanceCredentialResolver 

type InstanceCredentialResolver func(ref string) (string, error)

InstanceCredentialResolver resolves an instance auth connection reference.

type InstanceProviderFactory 

type InstanceProviderFactory func(
	instance *config.ProviderInstanceConfig,
	modelID string,
	secret string,
) (LLMProvider, error)

InstanceProviderFactory creates a runtime provider from one exact target's owning instance. secret is resolved from that instance's auth reference.

type InstanceResolution 

type InstanceResolution struct {
	Candidates []FallbackCandidate
	// contains filtered or unexported fields
}

InstanceResolution contains ordered fallback candidates and their exact instance-owned providers, keyed by FallbackCandidate.StableKey().

func ResolveInstanceTargetOrRoute 

func ResolveInstanceTargetOrRoute(
	cfg *config.Config,
	catalogs map[string]InstanceCatalog,
	selection string,
	resolveCredential InstanceCredentialResolver,
	createProvider InstanceProviderFactory,
) (*InstanceResolution, error)

ResolveInstanceTargetOrRoute resolves either an exact instance-id/model-id target or a named route. Every target runs on its own instance's endpoint, credential and runtime settings. Resolution checks each target's instance and catalog but neither resolves credentials nor creates providers; those happen on the first ProviderForCandidate call for a candidate.

func (*InstanceResolution) CallSpecForCandidate 

func (r *InstanceResolution) CallSpecForCandidate(candidate FallbackCandidate) (*CallSpec, error)

CallSpecForCandidate returns the call spec of candidate: its model on its own instance, with the instance's runtime settings but no credential.

func (*InstanceResolution) ProviderForCandidate 

func (r *InstanceResolution) ProviderForCandidate(candidate FallbackCandidate) (LLMProvider, error)

type LLMProvider 

type LLMProvider interface {
	Chat(
		ctx context.Context,
		messages []Message,
		tools []ToolDefinition,
		model string,
		options map[string]any,
	) (*LLMResponse, error)
	GetDefaultModel() string
}

func CreateProviderFromInstance 

func CreateProviderFromInstance(
	instance *config.ProviderInstanceConfig,
	modelID string,
	secret string,
) (LLMProvider, error)

CreateProviderFromInstance builds the chat provider of one model on one provider instance from that instance alone: Compa's OpenAI-compatible chat client, shaping requests with the instance's runtime settings, sending them through the instance's llmgw-core provider (see NewCoreProvider) with the credential secret - what the instance's auth_connection_ref resolved to - provides.

type LLMResponse 

type LLMResponse = protocoltypes.LLMResponse

type Message 

type Message = protocoltypes.Message

type NativeSearchCapable 

type NativeSearchCapable interface {
	SupportsNativeSearch() bool
}

NativeSearchCapable is an optional interface for providers that support built-in web search during LLM inference (e.g. OpenAI web_search_preview, xAI Grok search). When the active provider implements this interface and returns true, the agent loop can hide the client-side web_search tool to avoid duplicate search surfaces and use the provider's native search instead.

type RateLimiter 

type RateLimiter struct {
	// contains filtered or unexported fields
}

RateLimiter implements a token-bucket rate limiter for a single key. Allows up to RPM requests per minute with a burst equal to RPM. Thread-safe.

func (*RateLimiter) TryAcquire 

func (rl *RateLimiter) TryAcquire() bool

TryAcquire attempts to consume a token without blocking.

func (*RateLimiter) Wait 

func (rl *RateLimiter) Wait(ctx context.Context) error

Wait blocks until a token is available or ctx is canceled. Returns ctx.Err() if canceled while waiting.

type RateLimiterRegistry 

type RateLimiterRegistry struct {
	// contains filtered or unexported fields
}

RateLimiterRegistry holds per-instance rate limiters, keyed by FallbackCandidate.InstanceKey. Core has no client-side request limiter, so this stays Studio's. Candidates with RPM=0 are unrestricted. Thread-safe for concurrent reads/writes.

func NewRateLimiterRegistry 

func NewRateLimiterRegistry() *RateLimiterRegistry

NewRateLimiterRegistry creates an empty registry.

func (*RateLimiterRegistry) Register 

func (r *RateLimiterRegistry) Register(key string, rpm int)

Register adds a rate limiter for the given key at the given RPM. If rpm <= 0, no limiter is registered (unrestricted).

func (*RateLimiterRegistry) RegisterCandidates 

func (r *RateLimiterRegistry) RegisterCandidates(candidates []FallbackCandidate)

RegisterCandidates registers rate limiters for all candidates that have RPM > 0. Candidates with RPM == 0 are ignored (no restriction).

func (*RateLimiterRegistry) TryAcquire 

func (r *RateLimiterRegistry) TryAcquire(key string) bool

TryAcquire attempts to consume a token for the given key without blocking. If no limiter is registered for key, it returns true.

func (*RateLimiterRegistry) Wait 

func (r *RateLimiterRegistry) Wait(ctx context.Context, key string) error

Wait acquires a token for the given key, blocking if needed. If no limiter is registered for key, returns immediately.

type StatefulProvider 

type StatefulProvider interface {
	LLMProvider
	Close()
}

type StreamChunk 

type StreamChunk = protocoltypes.StreamChunk

type StreamingEventProvider 

type StreamingEventProvider interface {
	ChatStreamEvents(
		ctx context.Context,
		messages []Message,
		tools []ToolDefinition,
		model string,
		options map[string]any,
		onChunk func(StreamChunk),
	) (*LLMResponse, error)
}

type StreamingProvider 

type StreamingProvider interface {
	ChatStream(
		ctx context.Context,
		messages []Message,
		tools []ToolDefinition,
		model string,
		options map[string]any,
		onChunk func(accumulated string),
	) (*LLMResponse, error)
}

StreamingProvider is an optional interface for providers that support token streaming. onChunk receives the accumulated text so far (not individual deltas). The returned LLMResponse is the same complete response for compatibility with tool-call handling.

type ThinkingCapable 

type ThinkingCapable interface {
	SupportsThinking() bool
}

ThinkingCapable is an optional interface for providers that support extended thinking (e.g. Anthropic). Used by the agent loop to warn when thinking_level is configured but the active provider cannot use it.

type ToolCall 

type ToolCall = protocoltypes.ToolCall

func NormalizeToolCall 

func NormalizeToolCall(tc ToolCall) ToolCall

NormalizeToolCall normalizes a ToolCall to ensure all fields are properly populated. It handles cases where Name/Arguments might be in different locations (top-level vs Function) and ensures both are populated consistently.

type ToolDefinition 

type ToolDefinition = protocoltypes.ToolDefinition

type ToolFunctionDefinition 

type ToolFunctionDefinition = protocoltypes.ToolFunctionDefinition

type UsageInfo 

type UsageInfo = protocoltypes.UsageInfo

Source Files

View all Source files

Directories

Path Synopsis
Package common provides shared utilities used by multiple LLM provider implementations (openai_compat, azure, etc.).
 Package common provides shared utilities used by multiple LLM provider implementations (openai_compat, azure, etc.).
Package coretransport hands the surface requests Compa's chat client makes to an llmgw-core provider.
 Package coretransport hands the surface requests Compa's chat client makes to an llmgw-core provider.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.