Documentation
¶
Overview ¶
Package cache owns response-cache eligibility, semantic identity, versioned canonical records, and stream replay.
Index ¶
- Constants
- Variables
- func ChatKey(identity ChatIdentity) (string, error)
- func CompleteStream(events []inference.StreamEvent) (inference.ChatResponse, error)
- func Cosine(a, b []float32) float64
- func EmbeddingKey(identity EmbeddingIdentity) (string, error)
- func SemanticScope(identity ChatIdentity) (scopeKey, promptText string, err error)
- func StreamEvents(response inference.ChatResponse, options inference.StreamOptions) ([]inference.StreamEvent, error)
- type AccountPolicy
- type ChatIdentity
- type Clock
- type EmbeddingIdentity
- type Policy
- type ProviderAccess
- type ProviderPolicy
- type Repository
- type SemanticIndex
- type SemanticMatch
- type Store
Constants ¶
const ( // DefaultSemanticThreshold is the minimum cosine similarity that // answers when the deployment does not configure one. It is set high: // a wrong answer served confidently costs more than a provider call. DefaultSemanticThreshold = 0.95 // DefaultSemanticMaxEntries bounds the vectors one similarity scope // holds when the deployment does not configure a bound. The scope // embeds the account, so this is also the per-account scan bound. DefaultSemanticMaxEntries = 128 )
const RecordSchemaVersion = 1
RecordSchemaVersion identifies the canonical response-cache record schema.
const SemanticIndexSchemaVersion = 1
SemanticIndexSchemaVersion identifies the stored vector-index encoding.
const SemanticKeyVersion = 6
SemanticKeyVersion identifies the canonical cache-identity encoding. The key payload embeds inference.ChatRequest, and a canonical type carries no transport tag, so a field added to that struct reaches the hash even when no caller sets it. Raise this constant in the same change: the entries a running gateway holds keep their own prefix, and none of them is read back under an encoding that did not write it.
Version 3 added the output modality and audio output request fields. Version 4 added the stored document reference. Version 5 added the document parser the caller asked for. The same bytes read by two engines are two different inputs to the model. Version 6 renamed the tenant identity to account, which renamed the tagged account_id and policy fields the payload encodes.
Variables ¶
var ( // ErrIneligible reports a request whose identity is not cache-safe. ErrIneligible = errors.New("request is not eligible for response caching") // ErrAccountRequired reports a request without account identity. ErrAccountRequired = errors.New("cache account ID is required") // ErrGenerationRequired reports a request without catalog identity. ErrGenerationRequired = errors.New("catalog generation ID is required") // ErrMutableMedia reports a remote media input that can change without a // new key. ErrMutableMedia = errors.New("remote media input is mutable") // ErrUnknownExtension reports provider semantics outside the canonical identity. ErrUnknownExtension = errors.New("provider extension semantics are not cache-safe") // ErrInvalidJSONContract reports an invalid tool or output schema. ErrInvalidJSONContract = errors.New("request JSON contract is invalid") )
var ( // ErrStoreRequired reports an absent response-cache byte store. ErrStoreRequired = errors.New("response-cache store is required") // ErrCorruptRecord reports invalid durable response-cache data. ErrCorruptRecord = errors.New("response-cache record is invalid") // ErrKindMismatch reports a record payload that does not match its kind. ErrKindMismatch = errors.New("response-cache record kind does not match") // ErrEmptyResponse reports a completion with nothing a later request // can reuse. Caching one would replay a provider failure as a success. ErrEmptyResponse = errors.New("response-cache refuses an empty completion") )
var ( // ErrNoStreamEvents reports an empty canonical event sequence. ErrNoStreamEvents = errors.New("stream has no canonical events") // ErrUnsupportedContent reports a content part that canonical replay // cannot represent. A part now replays whatever kind it names, so the // remaining case is a malformed part: one that claims to be text while // carrying a media payload, which no reader can resolve without // guessing which half to believe. ErrUnsupportedContent = errors.New("cached stream content part is malformed") )
var ( // ErrNoPromptText reports a request with no text to embed. A purely // non-text request has no paraphrase, so the layer has nothing to do. ErrNoPromptText = errors.New("semantic cache needs prompt text") )
Functions ¶
func ChatKey ¶
func ChatKey(identity ChatIdentity) (string, error)
ChatKey returns the versioned semantic key for an eligible chat request.
func CompleteStream ¶
func CompleteStream(events []inference.StreamEvent) (inference.ChatResponse, error)
CompleteStream builds one canonical completed result from stream events.
func Cosine ¶ added in v1.2.0
Cosine returns the cosine similarity of two vectors, or 0 when the two cannot be compared: different lengths, empty, or zero magnitude.
func EmbeddingKey ¶
func EmbeddingKey(identity EmbeddingIdentity) (string, error)
EmbeddingKey returns the versioned semantic key for an eligible embedding request.
func SemanticScope ¶ added in v1.2.0
func SemanticScope(identity ChatIdentity) (scopeKey, promptText string, err error)
SemanticScope derives the similarity scope for one eligible chat request: the exact identity with every text part blanked, plus the prompt text those parts held. The scope key hashes everything the exact key hashes except the text, so entries under one scope differ only in wording.
func StreamEvents ¶
func StreamEvents(response inference.ChatResponse, options inference.StreamOptions) ([]inference.StreamEvent, error)
StreamEvents reconstructs canonical stream events from a completed result.
Types ¶
type AccountPolicy ¶ added in v1.1.0
type AccountPolicy struct {
AllowedModels []string `json:"allowed_models,omitempty"`
AllowedProviders []string `json:"allowed_providers,omitempty"`
Access []ProviderAccess `json:"access,omitempty"`
RateLimitTier string `json:"rate_limit_tier,omitempty"`
CredentialStrategy string `json:"credential_strategy,omitempty"`
}
AccountPolicy contains account-scoped route restrictions.
type ChatIdentity ¶
type ChatIdentity struct {
AccountID string
CatalogGeneration string
Request inference.ChatRequest
Policy Policy
}
ChatIdentity contains all semantic inputs for one chat cache entry.
type EmbeddingIdentity ¶
type EmbeddingIdentity struct {
AccountID string
CatalogGeneration string
Request inference.EmbeddingRequest
Policy Policy
}
EmbeddingIdentity contains all semantic inputs for one embedding cache entry.
type Policy ¶
type Policy struct {
Provider ProviderPolicy `json:"provider"`
Account AccountPolicy `json:"account"`
}
Policy contains all routing policy that can change a response.
type ProviderAccess ¶ added in v1.1.0
type ProviderAccess struct {
Provider string `json:"provider"`
Models []string `json:"models,omitempty"`
}
ProviderAccess is one paired provider and model grant. It changes which route may serve a response, so it is part of the cache identity.
type ProviderPolicy ¶
type ProviderPolicy struct {
Order []string `json:"order,omitempty"`
Only []string `json:"only,omitempty"`
Ignore []string `json:"ignore,omitempty"`
AllowFallbacks bool `json:"allow_fallbacks"`
Route string `json:"route,omitempty"`
ModelOverrides map[string]string `json:"model_overrides,omitempty"`
// Sort and the price caps change which route serves the response, so
// they are part of the cache identity.
Sort string `json:"sort,omitempty"`
MaxPromptPricePer1M float64 `json:"max_prompt_price_per_1m,omitempty"`
MaxCompletionPricePer1M float64 `json:"max_completion_price_per_1m,omitempty"`
}
ProviderPolicy contains request-scoped provider routing semantics.
type Repository ¶
type Repository interface {
GetChat(context.Context, string) (inference.ChatResponse, time.Time, bool, error)
PutChat(context.Context, string, inference.ChatResponse) error
GetEmbedding(context.Context, string) (inference.EmbeddingResponse, time.Time, bool, error)
PutEmbedding(context.Context, string, inference.EmbeddingResponse) error
}
Repository stores versioned canonical inference results.
type SemanticIndex ¶ added in v1.2.0
type SemanticIndex interface {
// Lookup answers the best entry whose cosine similarity to the vector
// clears the threshold, or reports no match.
Lookup(ctx context.Context, scopeKey string, vector []float32) (SemanticMatch, bool, error)
// Add records a vector pointing at one exact entry, bounded per scope:
// the oldest vector leaves when the scope is full.
Add(ctx context.Context, scopeKey string, vector []float32, exactKey string) error
// Drop removes the vector pointing at one exact entry. The caller uses
// it when the entry has left the store, so the vector goes with it.
Drop(ctx context.Context, scopeKey, exactKey string) error
}
SemanticIndex holds prompt vectors per similarity scope and answers the nearest entry above the threshold.
func OpenSemanticIndex ¶ added in v1.2.0
func OpenSemanticIndex(store Store, threshold float64, maxEntries int) (SemanticIndex, error)
OpenSemanticIndex creates a vector index over the response-cache store. A zero threshold or bound takes the built-in default.
type SemanticMatch ¶ added in v1.2.0
SemanticMatch is one similarity answer: the exact entry it points at and the cosine similarity that cleared the threshold.
Source Files