Documentation
¶
Overview ¶
Package cache defines the caching interface for Nexus.
Index ¶
- func Key(req *provider.CompletionRequest) string
- func KindOf(c any) string
- func StreamKey(req *provider.CompletionRequest) string
- type Cache
- type ReplayMode
- type SemanticCache
- func (sc *SemanticCache) Clear(ctx context.Context) error
- func (sc *SemanticCache) Delete(ctx context.Context, key string) error
- func (sc *SemanticCache) Get(ctx context.Context, key string) (*provider.CompletionResponse, error)
- func (sc *SemanticCache) Set(ctx context.Context, key string, resp *provider.CompletionResponse) error
- type SemanticMatcher
- type Service
- type Stats
- type StreamCache
- type StreamCacheOptions
- type StreamFrame
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Key ¶
func Key(req *provider.CompletionRequest) string
Key generates a deterministic cache key from a request. The key is a SHA-256 hash of every request field that can change the answer: the tenant, the named provider, the model, the system prompt, each message (role, content, name, tool calls and tool call id), the sampling parameters, the tools, the tool choice, the thinking settings and the response format. Tenants never share entries, and neither do requests routed to different providers.
Each field is written as JSON, whose encoder sorts map keys, so the same request always gives the same key. A tool choice is keyed by what it means, so "required" and Anthropic's {"type":"any"} share an entry.
Key returns "" when a field cannot be encoded. Such a request must not be cached.
func KindOf ¶
KindOf reports a cache's optional Kind capability. Custom caches need not implement it; unknown means the backend cannot be identified.
func StreamKey ¶
func StreamKey(req *provider.CompletionRequest) string
StreamKey derives a deterministic cache key for a streaming request. Adds a "stream:" prefix to the standard request key so streaming and non-streaming caches don't collide if a single backend hosts both. Like Key, it returns "" for a request that must not be cached.
Types ¶
type Cache ¶
type Cache interface {
Get(ctx context.Context, key string) (*provider.CompletionResponse, error)
Set(ctx context.Context, key string, resp *provider.CompletionResponse) error
Delete(ctx context.Context, key string) error
Clear(ctx context.Context) error
}
Cache is the core caching interface.
type ReplayMode ¶
type ReplayMode int
ReplayMode controls how a cached stream is played back.
const ( // ReplayBurst emits all cached frames immediately. Fastest; loses the // original timing. Default mode for Nexus. ReplayBurst ReplayMode = iota // ReplayPaced sleeps between frames to reproduce the original cadence. // Useful for demos / matching real-time UX. ReplayPaced // ReplayFastForward applies a divisor to the original gaps. // Use ReplayFastForwardN for divisor configuration. ReplayFastForward )
type SemanticCache ¶
type SemanticCache struct {
// contains filtered or unexported fields
}
SemanticCache wraps a Cache with semantic matching capabilities.
func NewSemanticCache ¶
func NewSemanticCache(cache Cache, matcher SemanticMatcher, threshold float64) *SemanticCache
NewSemanticCache creates a cache that falls back to semantic matching when exact key lookup misses. The threshold (0.0-1.0) controls the minimum similarity score for a semantic match.
func (*SemanticCache) Delete ¶
func (sc *SemanticCache) Delete(ctx context.Context, key string) error
func (*SemanticCache) Get ¶
func (sc *SemanticCache) Get(ctx context.Context, key string) (*provider.CompletionResponse, error)
func (*SemanticCache) Set ¶
func (sc *SemanticCache) Set(ctx context.Context, key string, resp *provider.CompletionResponse) error
type SemanticMatcher ¶
type SemanticMatcher interface {
// Match returns the best matching cache key and a similarity score.
// A score of 1.0 is an exact match; 0.0 is no match.
// Returns ("", 0, nil) if no similar entry is found.
Match(ctx context.Context, key string, threshold float64) (matchedKey string, score float64, err error)
// Index adds a key to the semantic index.
Index(ctx context.Context, key string) error
// Remove deletes a key from the semantic index.
Remove(ctx context.Context, key string) error
}
SemanticMatcher finds semantically similar cache keys. This enables cache hits even when the exact prompt differs but the meaning is equivalent.
type Stats ¶
type Stats struct {
Hits int64 `json:"hits"`
Misses int64 `json:"misses"`
Size int64 `json:"size"` // entries
Bytes int64 `json:"bytes"` // memory usage
HitRate float64 `json:"hit_rate"`
}
Stats reports cache performance.
type StreamCache ¶
type StreamCache interface {
GetStream(ctx context.Context, key string) ([]StreamFrame, error)
SetStream(ctx context.Context, key string, frames []StreamFrame, ttl time.Duration) error
DeleteStream(ctx context.Context, key string) error
}
StreamCache stores ordered captures of streamed responses, keyed independently from the non-streaming Cache. Implementations need not support both tiers — a deployment may register a memory stream cache even when its standard cache is Redis.
type StreamCacheOptions ¶
type StreamCacheOptions struct {
TTL time.Duration // entry lifetime; 0 = unbounded (subject to backend)
MaxFrames int // abandon recording after this many frames; 0 = unlimited
MaxBytes int // abandon recording when total payload exceeds; 0 = unlimited
Mode ReplayMode // replay strategy on hit
FFDivisor float64 // ReplayFastForward divisor (default 4x)
}
StreamCacheOptions tunes recording and replay caps.
type StreamFrame ¶
type StreamFrame struct {
Chunk *provider.StreamChunk
OffsetMs int64
}
StreamFrame is one captured chunk plus its arrival offset since the start of the recording. OffsetMs is used by paced replay to reproduce the original cadence.
Source Files
Directories