router

package
v0.3.7 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 1, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package router implements the Router Service.

Responsibilities:

  • Resolving a ModelId to the correct Provider
  • Fetching live Credentials from the Credential Pool
  • Single-pass delegation to Lua plugins or built-in Go adapters
  • Translating backend-specific errors back to OpenAI-compatible ones

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Service

type Service struct {
	// contains filtered or unexported fields
}

Service routes validated API requests to the appropriate provider.

func New

func New(providerSvc *provider.Service, credSvc *credential.Service, modelInfoSvc *modelinfo.Service, exhaustedSvc *exhausted.Service, logger *slog.Logger) *Service

New constructs a new router Service. exhaustedSvc may be nil (no exhausted filtering); production always wires it.

func (*Service) Complete

Complete routes a non-streaming chat completion request. The backend tries the credential pool in order, at most once per key; the first success wins and the last error is returned as-is.

func (*Service) CompleteStream

func (s *Service) CompleteStream(
	ctx context.Context,
	req *models.ChatCompletionRequest,
	w io.Writer,
	token *models.RouterToken,
) error

CompleteStream routes a streaming chat completion request. Server-Sent Events are written directly to w.

func (*Service) Embed added in v0.0.5

Embed routes a POST /v1/embeddings request in a single backend pass over the credential pool.

func (*Service) GenerateImage added in v0.0.5

GenerateImage routes a POST /v1/images/generations request in a single backend pass over the credential pool.

func (*Service) GetProviderIDForModel

func (s *Service) GetProviderIDForModel(ctx context.Context, modelID models.ModelId) (string, error)

GetProviderIDForModel returns the composite provider ID for a given model.

func (*Service) HasUsableCredential added in v0.2.0

func (s *Service) HasUsableCredential(model models.ModelId) bool

HasUsableCredential reports whether at least one credential for the model is not rate-limited in the exhausted store. It checks per-account-model keys for every credential in the pool. A true result means the model is worth attempting; a false result means every credential is in cooldown. The last-resort fallback in dropExhausted still applies: a fully-limited pool is kept as a last resort, so this is a pre-check, not a guarantee.

func (*Service) LikelyExhausted added in v0.2.0

func (s *Service) LikelyExhausted(model models.ModelId) bool

LikelyExhausted reports whether model is already known to be entirely unusable: a model-wide limit key exists for its resolved provider, independent of which account or proxy would be tried. It is a cheap, best-effort pre-check for a caller iterating several candidate models (virtual-model fan-out) deciding whether a member is worth attempting at all — it never touches the credential pool or the exhausted store's account/proxy dimensions, so it cannot detect "every account happens to be limited" short of an explicit model-wide mark. A false result is not proof of success: dropExhausted's own last-resort fallback can still let a fully-filtered pool through. Callers must not skip the last remaining candidate on a true result; a stale mark must never fully deny a request.

func (*Service) ProbeCapabilities

func (s *Service) ProbeCapabilities(ctx context.Context, modelID models.ModelId) ([]string, error)

ProbeCapabilities detects model features with live probe requests. Each probe consumes a small amount of quota. Unprobed capabilities are simply absent from the result.

func (*Service) Speech added in v0.0.5

Speech routes a POST /v1/audio/speech request in a single backend pass over the credential pool.

func (*Service) TestCredential

func (s *Service) TestCredential(ctx context.Context, providerID, credentialID string, overrideModel string) TestResult

TestCredential runs a single probe request pinned to one credential, bypassing pool rotation. The probe model is the provider's first listed model, or overrideModel when given.

func (*Service) TestEmbeddings added in v0.1.0

func (s *Service) TestEmbeddings(ctx context.Context, modelID models.ModelId) TestResult

TestEmbeddings runs a minimal embeddings probe, bypassing manual disable.

func (*Service) TestImageGeneration added in v0.1.0

func (s *Service) TestImageGeneration(ctx context.Context, modelID models.ModelId) TestResult

TestImageGeneration runs a minimal image probe, bypassing manual disable.

func (*Service) TestModel

func (s *Service) TestModel(ctx context.Context, modelID models.ModelId) TestResult

TestModel runs a probe through the normal routing path (credential pool included), as an internal unrestricted call. Manually disabled models stay testable: the admin override gates routing, not probing.

func (*Service) TestSpeech added in v0.1.0

func (s *Service) TestSpeech(ctx context.Context, modelID models.ModelId) TestResult

TestSpeech runs a minimal speech probe, bypassing manual disable.

func (*Service) TestTranscribe added in v0.1.0

func (s *Service) TestTranscribe(ctx context.Context, modelID models.ModelId) TestResult

TestTranscribe runs a minimal transcription probe, bypassing manual disable.

func (*Service) TestVision added in v0.1.0

func (s *Service) TestVision(ctx context.Context, modelID models.ModelId) TestResult

TestVision runs a minimal image-input probe, bypassing manual disable.

func (*Service) Transcribe added in v0.0.5

Transcribe routes a POST /v1/audio/transcriptions request in a single backend pass over the credential pool.

type TestResult

type TestResult struct {
	OK            bool   `json:"ok"`
	Latency       int64  `json:"latency_ms"`
	Error         string `json:"error,omitempty"`
	Response      string `json:"response,omitempty"`
	QuotaExceeded bool   `json:"quota_exceeded,omitempty"`
	Code          string `json:"code,omitempty"`
	Summary       string `json:"summary,omitempty"`
	Proxy         string `json:"proxy,omitempty"`
}

TestResult reports the outcome of a credential or model probe. QuotaExceeded marks temporary rate-limit failures: the model is alive, the quota is not. Callers must not disable models over it. Code is the machine-readable wire code for the failure (empty on success); Summary is a short human string for toasts, the full error goes to logs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL