Documentation
¶
Overview ¶
Package router implements the Router Service.
Responsibilities:
- Resolving a ModelId to the correct Provider
- Fetching live Credentials from the Credential Pool
- Single-pass delegation to Lua plugins or built-in Go adapters
- Translating backend-specific errors back to OpenAI-compatible ones
Index ¶
- type Service
- func (s *Service) Complete(ctx context.Context, req *models.ChatCompletionRequest, ...) (*models.ChatCompletionResponse, error)
- func (s *Service) CompleteStream(ctx context.Context, req *models.ChatCompletionRequest, w io.Writer, ...) error
- func (s *Service) Embed(ctx context.Context, req *models.EmbeddingsRequest, token *models.RouterToken) (*models.EmbeddingsResponse, error)
- func (s *Service) GenerateImage(ctx context.Context, req *models.ImageGenerationRequest, ...) (*models.ImageGenerationResponse, error)
- func (s *Service) GetProviderIDForModel(ctx context.Context, modelID models.ModelId) (string, error)
- func (s *Service) HasUsableCredential(model models.ModelId) bool
- func (s *Service) LikelyExhausted(model models.ModelId) bool
- func (s *Service) ProbeCapabilities(ctx context.Context, modelID models.ModelId) ([]string, error)
- func (s *Service) Speech(ctx context.Context, req *models.SpeechRequest, token *models.RouterToken) (*models.SpeechResponse, error)
- func (s *Service) TestCredential(ctx context.Context, providerID, credentialID string, overrideModel string) TestResult
- func (s *Service) TestEmbeddings(ctx context.Context, modelID models.ModelId) TestResult
- func (s *Service) TestImageGeneration(ctx context.Context, modelID models.ModelId) TestResult
- func (s *Service) TestModel(ctx context.Context, modelID models.ModelId) TestResult
- func (s *Service) TestSpeech(ctx context.Context, modelID models.ModelId) TestResult
- func (s *Service) TestTranscribe(ctx context.Context, modelID models.ModelId) TestResult
- func (s *Service) TestVision(ctx context.Context, modelID models.ModelId) TestResult
- func (s *Service) Transcribe(ctx context.Context, req *models.TranscriptionRequest, ...) (*models.TranscriptionResponse, error)
- type TestResult
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service routes validated API requests to the appropriate provider.
func New ¶
func New(providerSvc *provider.Service, credSvc *credential.Service, modelInfoSvc *modelinfo.Service, exhaustedSvc *exhausted.Service, logger *slog.Logger) *Service
New constructs a new router Service. exhaustedSvc may be nil (no exhausted filtering); production always wires it.
func (*Service) Complete ¶
func (s *Service) Complete( ctx context.Context, req *models.ChatCompletionRequest, token *models.RouterToken, ) (*models.ChatCompletionResponse, error)
Complete routes a non-streaming chat completion request. The backend tries the credential pool in order, at most once per key; the first success wins and the last error is returned as-is.
func (*Service) CompleteStream ¶
func (s *Service) CompleteStream( ctx context.Context, req *models.ChatCompletionRequest, w io.Writer, token *models.RouterToken, ) error
CompleteStream routes a streaming chat completion request. Server-Sent Events are written directly to w.
func (*Service) Embed ¶ added in v0.0.5
func (s *Service) Embed( ctx context.Context, req *models.EmbeddingsRequest, token *models.RouterToken, ) (*models.EmbeddingsResponse, error)
Embed routes a POST /v1/embeddings request in a single backend pass over the credential pool.
func (*Service) GenerateImage ¶ added in v0.0.5
func (s *Service) GenerateImage( ctx context.Context, req *models.ImageGenerationRequest, token *models.RouterToken, ) (*models.ImageGenerationResponse, error)
GenerateImage routes a POST /v1/images/generations request in a single backend pass over the credential pool.
func (*Service) GetProviderIDForModel ¶
func (s *Service) GetProviderIDForModel(ctx context.Context, modelID models.ModelId) (string, error)
GetProviderIDForModel returns the composite provider ID for a given model.
func (*Service) HasUsableCredential ¶ added in v0.2.0
HasUsableCredential reports whether at least one credential for the model is not rate-limited in the exhausted store. It checks per-account-model keys for every credential in the pool. A true result means the model is worth attempting; a false result means every credential is in cooldown. The last-resort fallback in dropExhausted still applies: a fully-limited pool is kept as a last resort, so this is a pre-check, not a guarantee.
func (*Service) LikelyExhausted ¶ added in v0.2.0
LikelyExhausted reports whether model is already known to be entirely unusable: a model-wide limit key exists for its resolved provider, independent of which account or proxy would be tried. It is a cheap, best-effort pre-check for a caller iterating several candidate models (virtual-model fan-out) deciding whether a member is worth attempting at all — it never touches the credential pool or the exhausted store's account/proxy dimensions, so it cannot detect "every account happens to be limited" short of an explicit model-wide mark. A false result is not proof of success: dropExhausted's own last-resort fallback can still let a fully-filtered pool through. Callers must not skip the last remaining candidate on a true result; a stale mark must never fully deny a request.
func (*Service) ProbeCapabilities ¶
ProbeCapabilities detects model features with live probe requests. Each probe consumes a small amount of quota. Unprobed capabilities are simply absent from the result.
func (*Service) Speech ¶ added in v0.0.5
func (s *Service) Speech( ctx context.Context, req *models.SpeechRequest, token *models.RouterToken, ) (*models.SpeechResponse, error)
Speech routes a POST /v1/audio/speech request in a single backend pass over the credential pool.
func (*Service) TestCredential ¶
func (s *Service) TestCredential(ctx context.Context, providerID, credentialID string, overrideModel string) TestResult
TestCredential runs a single probe request pinned to one credential, bypassing pool rotation. The probe model is the provider's first listed model, or overrideModel when given.
func (*Service) TestEmbeddings ¶ added in v0.1.0
TestEmbeddings runs a minimal embeddings probe, bypassing manual disable.
func (*Service) TestImageGeneration ¶ added in v0.1.0
TestImageGeneration runs a minimal image probe, bypassing manual disable.
func (*Service) TestModel ¶
TestModel runs a probe through the normal routing path (credential pool included), as an internal unrestricted call. Manually disabled models stay testable: the admin override gates routing, not probing.
func (*Service) TestSpeech ¶ added in v0.1.0
TestSpeech runs a minimal speech probe, bypassing manual disable.
func (*Service) TestTranscribe ¶ added in v0.1.0
TestTranscribe runs a minimal transcription probe, bypassing manual disable.
func (*Service) TestVision ¶ added in v0.1.0
TestVision runs a minimal image-input probe, bypassing manual disable.
func (*Service) Transcribe ¶ added in v0.0.5
func (s *Service) Transcribe( ctx context.Context, req *models.TranscriptionRequest, token *models.RouterToken, ) (*models.TranscriptionResponse, error)
Transcribe routes a POST /v1/audio/transcriptions request in a single backend pass over the credential pool.
type TestResult ¶
type TestResult struct {
OK bool `json:"ok"`
Latency int64 `json:"latency_ms"`
Error string `json:"error,omitempty"`
Response string `json:"response,omitempty"`
QuotaExceeded bool `json:"quota_exceeded,omitempty"`
Code string `json:"code,omitempty"`
Summary string `json:"summary,omitempty"`
Proxy string `json:"proxy,omitempty"`
}
TestResult reports the outcome of a credential or model probe. QuotaExceeded marks temporary rate-limit failures: the model is alive, the quota is not. Callers must not disable models over it. Code is the machine-readable wire code for the failure (empty on success); Summary is a short human string for toasts, the full error goes to logs.