Documentation
¶
Overview ¶
Pluggable distributed cache backend interface.
CacheBackend abstracts the storage layer behind the in-process cache so that callers can swap in a distributed store (e.g. Redis) without changing the surrounding code. The default is an in-memory implementation; callers that do not opt in keep the existing in-process behavior unchanged.
Everything here is stdlib-only. The Redis backend speaks RESP directly over a net.Conn rather than pulling in a redis client dependency.
Index ¶
- Constants
- Variables
- type CacheBackend
- type CacheStats
- type CacheWarmer
- func (cw *CacheWarmer) CacheBreakpoints(systemPrompt string, conversationPrefix []Message) []int
- func (cw *CacheWarmer) EstimateSavings(inputTokens int, requestCount int) float64
- func (cw *CacheWarmer) ShouldWarm() bool
- func (cw *CacheWarmer) Start(ctx context.Context) error
- func (cw *CacheWarmer) Stop()
- func (cw *CacheWarmer) Warm(ctx context.Context) error
- type ChatOptions
- type MemoryBackend
- type Message
- type RedisBackend
Constants ¶
const ( // DefaultWarmInterval is the default interval between cache warming pings. // Anthropic's prompt cache TTL is 5 minutes; we warm at 4 minutes to stay ahead. DefaultWarmInterval = 4 * time.Minute )
Default cache warmer settings.
Variables ¶
var ErrRedisNil = errors.New("eyrie: redis nil reply")
ErrRedisNil is returned by the RESP client for a nil bulk-string reply (i.e. a missing key). It is handled internally and never surfaced from Get.
Functions ¶
This section is empty.
Types ¶
type CacheBackend ¶
type CacheBackend interface {
// Get returns the value for key. The second return value reports whether the
// key was present (and unexpired). A miss is (nil, false, nil), not an error.
Get(ctx context.Context, key string) ([]byte, bool, error)
// Set stores val under key. A non-positive ttl means "no expiry".
Set(ctx context.Context, key string, val []byte, ttl time.Duration) error
// Delete removes key. Deleting a missing key is not an error.
Delete(ctx context.Context, key string) error
}
CacheBackend is the storage abstraction used by the cache. Implementations must be safe for concurrent use by multiple goroutines.
type CacheStats ¶
type CacheStats struct {
WarmingRequests int64
CacheHits int64
CacheMisses int64
EstimatedSavingsUSD float64
LastWarmedAt time.Time
}
CacheStats tracks cache warming statistics.
type CacheWarmer ¶
type CacheWarmer struct {
Provider string
Model string
SystemPrompt string
Interval time.Duration
Enabled bool
Stats CacheStats
// ChatFn is the function used to send warming requests.
// It should send a chat completion request to the provider.
ChatFn func(ctx context.Context, messages []Message, opts ChatOptions) error
// contains filtered or unexported fields
}
CacheWarmer keeps Anthropic's prompt cache warm by periodically sending minimal requests with the system prompt, ensuring subsequent real requests get cache hits at a 90% discount.
func NewCacheWarmer ¶
func NewCacheWarmer(chatFn func(ctx context.Context, messages []Message, opts ChatOptions) error, systemPrompt, provider, model string) *CacheWarmer
NewCacheWarmer creates a new CacheWarmer configured for the given provider. The chatFn is called to send warming pings; it should make a real API call.
func (*CacheWarmer) CacheBreakpoints ¶
func (cw *CacheWarmer) CacheBreakpoints(systemPrompt string, conversationPrefix []Message) []int
CacheBreakpoints suggests where to place cache breakpoints in a message list. Anthropic allows up to 4 breakpoints. The strategy is:
- After system prompt (index 0 in returned slice signals "system prompt")
- After the first user message
- After large context blocks (messages with content > 200 chars)
Returns indices into the messages array where breakpoints should be placed. The special index -1 indicates the system prompt should be cached.
func (*CacheWarmer) EstimateSavings ¶
func (cw *CacheWarmer) EstimateSavings(inputTokens int, requestCount int) float64
EstimateSavings calculates the cost savings from prompt caching in USD.
Without cache: inputTokens * requestCount * price_per_token With cache: (inputTokens * price_per_token * 1.25 first time) +
(inputTokens * (requestCount-1) * price_per_token * 0.1)
Returns the difference (savings amount).
func (*CacheWarmer) ShouldWarm ¶
func (cw *CacheWarmer) ShouldWarm() bool
ShouldWarm returns true if the cache has likely expired (more than 4 minutes since the last warming request) and should be refreshed.
func (*CacheWarmer) Start ¶
func (cw *CacheWarmer) Start(ctx context.Context) error
Start begins the background cache warming loop. It sends a warming ping immediately, then repeats every Interval until Stop is called or the context is cancelled. For non-Anthropic providers this is a no-op.
func (*CacheWarmer) Stop ¶
func (cw *CacheWarmer) Stop()
Stop stops the cache warmer background loop.
type ChatOptions ¶
ChatOptions is a minimal options struct for cache warmer use.
type MemoryBackend ¶
type MemoryBackend struct {
// contains filtered or unexported fields
}
MemoryBackend is the default in-process CacheBackend. It stores values in a map guarded by a mutex and lazily evicts expired entries on access.
func NewMemoryBackend ¶
func NewMemoryBackend() *MemoryBackend
NewMemoryBackend creates an empty in-memory CacheBackend.
func (*MemoryBackend) Delete ¶
func (m *MemoryBackend) Delete(_ context.Context, key string) error
Delete implements CacheBackend.
type RedisBackend ¶
type RedisBackend struct {
// contains filtered or unexported fields
}
RedisBackend is a minimal, dependency-free CacheBackend backed by Redis.
It implements only the subset of commands the cache needs:
GET <key> SET <key> <val> [PX <ttl-ms>] DEL <key>
(EXPIRE is expressed via SET ... PX rather than a separate command.)
The client speaks RESP over a single net.Conn protected by a mutex. This is a SKELETON: it is intentionally simple (one connection, no pooling, no pipelining, no AUTH/SELECT) and is meant as a starting point. For production use, extend with connection pooling, AUTH/db selection, and automatic reconnection.
func NewRedisBackend ¶
func NewRedisBackend(addr string, timeout time.Duration) *RedisBackend
NewRedisBackend creates a RedisBackend targeting addr (host:port). The connection is established lazily on first use. A non-positive timeout falls back to a 5s default applied per operation.
func (*RedisBackend) Close ¶
func (r *RedisBackend) Close() error
Close closes the underlying connection, if any.
func (*RedisBackend) Delete ¶
func (r *RedisBackend) Delete(ctx context.Context, key string) error
Delete implements CacheBackend.
Source Files