Documentation
¶
Overview ¶
Package cache owns response-cache eligibility, semantic identity, versioned canonical records, and stream replay.
Index ¶
- Constants
- Variables
- func ChatKey(identity ChatIdentity) (string, error)
- func CompleteStream(events []inference.StreamEvent) (inference.ChatResponse, error)
- func EmbeddingKey(identity EmbeddingIdentity) (string, error)
- func StreamEvents(response inference.ChatResponse, options inference.StreamOptions) ([]inference.StreamEvent, error)
- type AccountPolicy
- type ChatIdentity
- type Clock
- type EmbeddingIdentity
- type Policy
- type ProviderAccess
- type ProviderPolicy
- type Repository
- type Store
Constants ¶
const RecordSchemaVersion = 1
RecordSchemaVersion identifies the canonical response-cache record schema.
const SemanticKeyVersion = 6
SemanticKeyVersion identifies the canonical cache-identity encoding. The key payload embeds inference.ChatRequest, and a canonical type carries no transport tag, so a field added to that struct reaches the hash even when no caller sets it. Raise this constant in the same change: the entries a running gateway holds keep their own prefix, and none of them is read back under an encoding that did not write it.
Version 3 added the output modality and audio output request fields. Version 4 added the stored document reference. Version 5 added the document parser the caller asked for. The same bytes read by two engines are two different inputs to the model. Version 6 renamed the tenant identity to account, which renamed the tagged account_id and policy fields the payload encodes.
Variables ¶
var ( // ErrIneligible reports a request whose identity is not cache-safe. ErrIneligible = errors.New("request is not eligible for response caching") // ErrAccountRequired reports a request without account identity. ErrAccountRequired = errors.New("cache account ID is required") // ErrGenerationRequired reports a request without catalog identity. ErrGenerationRequired = errors.New("catalog generation ID is required") // ErrMutableMedia reports a remote media input that can change without a // new key. ErrMutableMedia = errors.New("remote media input is mutable") // ErrUnknownExtension reports provider semantics outside the canonical identity. ErrUnknownExtension = errors.New("provider extension semantics are not cache-safe") // ErrInvalidJSONContract reports an invalid tool or output schema. ErrInvalidJSONContract = errors.New("request JSON contract is invalid") )
var ( // ErrStoreRequired reports an absent response-cache byte store. ErrStoreRequired = errors.New("response-cache store is required") // ErrCorruptRecord reports invalid durable response-cache data. ErrCorruptRecord = errors.New("response-cache record is invalid") // ErrKindMismatch reports a record payload that does not match its kind. ErrKindMismatch = errors.New("response-cache record kind does not match") // ErrEmptyResponse reports a completion with nothing a later request // can reuse. Caching one would replay a provider failure as a success. ErrEmptyResponse = errors.New("response-cache refuses an empty completion") )
var ( // ErrNoStreamEvents reports an empty canonical event sequence. ErrNoStreamEvents = errors.New("stream has no canonical events") // ErrUnsupportedContent reports a content part that canonical replay // cannot represent. A part now replays whatever kind it names, so the // remaining case is a malformed part: one that claims to be text while // carrying a media payload, which no reader can resolve without // guessing which half to believe. ErrUnsupportedContent = errors.New("cached stream content part is malformed") )
Functions ¶
func ChatKey ¶
func ChatKey(identity ChatIdentity) (string, error)
ChatKey returns the versioned semantic key for an eligible chat request.
func CompleteStream ¶
func CompleteStream(events []inference.StreamEvent) (inference.ChatResponse, error)
CompleteStream builds one canonical completed result from stream events.
func EmbeddingKey ¶
func EmbeddingKey(identity EmbeddingIdentity) (string, error)
EmbeddingKey returns the versioned semantic key for an eligible embedding request.
func StreamEvents ¶
func StreamEvents(response inference.ChatResponse, options inference.StreamOptions) ([]inference.StreamEvent, error)
StreamEvents reconstructs canonical stream events from a completed result.
Types ¶
type AccountPolicy ¶ added in v1.1.0
type AccountPolicy struct {
AllowedModels []string `json:"allowed_models,omitempty"`
AllowedProviders []string `json:"allowed_providers,omitempty"`
Access []ProviderAccess `json:"access,omitempty"`
RateLimitTier string `json:"rate_limit_tier,omitempty"`
CredentialStrategy string `json:"credential_strategy,omitempty"`
}
AccountPolicy contains account-scoped route restrictions.
type ChatIdentity ¶
type ChatIdentity struct {
AccountID string
CatalogGeneration string
Request inference.ChatRequest
Policy Policy
}
ChatIdentity contains all semantic inputs for one chat cache entry.
type EmbeddingIdentity ¶
type EmbeddingIdentity struct {
AccountID string
CatalogGeneration string
Request inference.EmbeddingRequest
Policy Policy
}
EmbeddingIdentity contains all semantic inputs for one embedding cache entry.
type Policy ¶
type Policy struct {
Provider ProviderPolicy `json:"provider"`
Account AccountPolicy `json:"account"`
}
Policy contains all routing policy that can change a response.
type ProviderAccess ¶ added in v1.1.0
type ProviderAccess struct {
Provider string `json:"provider"`
Models []string `json:"models,omitempty"`
}
ProviderAccess is one paired provider and model grant. It changes which route may serve a response, so it is part of the cache identity.
type ProviderPolicy ¶
type ProviderPolicy struct {
Order []string `json:"order,omitempty"`
Only []string `json:"only,omitempty"`
Ignore []string `json:"ignore,omitempty"`
AllowFallbacks bool `json:"allow_fallbacks"`
Route string `json:"route,omitempty"`
ModelOverrides map[string]string `json:"model_overrides,omitempty"`
// Sort and the price caps change which route serves the response, so
// they are part of the cache identity.
Sort string `json:"sort,omitempty"`
MaxPromptPricePer1M float64 `json:"max_prompt_price_per_1m,omitempty"`
MaxCompletionPricePer1M float64 `json:"max_completion_price_per_1m,omitempty"`
}
ProviderPolicy contains request-scoped provider routing semantics.
type Repository ¶
type Repository interface {
GetChat(context.Context, string) (inference.ChatResponse, time.Time, bool, error)
PutChat(context.Context, string, inference.ChatResponse) error
GetEmbedding(context.Context, string) (inference.EmbeddingResponse, time.Time, bool, error)
PutEmbedding(context.Context, string, inference.EmbeddingResponse) error
}
Repository stores versioned canonical inference results.