README
¶
AI Cache Service (ai/cache)
The cache package provides a two-layer caching mechanism to optimize AI Agent response speed and reduce API costs. It includes a generic in-memory LRU cache and a dedicated Semantic Cache.
Architecture
classDiagram
class CacheService {
<<interface>>
+Get(ctx, key)
+Set(ctx, key, val, ttl)
+Invalidate(ctx, pattern)
}
class LRUCache~K,V~ {
-cache map[K]*entry
-order *list.Element
+Get(key)
+Set(key, val, ttl)
}
class SemanticCache {
-exactCache *ByteLRUCache
-semanticCache map
-stats SemanticCacheStats
+Get(text)
+Set(text, embedding)
}
CacheService <|.. LRUCache : implements
SemanticCache --* LRUCache : uses (L1)
1. LRU Cache (LRUCache[K, V])
- Generic: Implemented using Go generics, supports any key-value types.
- Thread-safe: Uses
sync.RWMutexfor concurrent access safety. - TTL Support: Each cache entry has an expiration time for automatic cleanup.
2. Semantic Cache (SemanticCache)
Implements a two-layer caching strategy for retrieving similar text embeddings and reducing redundant embedding calculations.
- Level 1: Exact Match
- Uses SHA256 to hash input text.
- Stored in
ByteLRUCachefor fast access.
- Level 2: Semantic Match
- Stores text Vector Embeddings.
- Finds similar cached entries using Cosine Similarity.
Algorithm
flowchart TD
Start([Input Text]) --> L1{L1: Exact Match?}
L1 -- Hit (SHA256) --> Return[Return Cached Embedding]
L1 -- Miss --> Embed[Call EmbeddingService]
Embed --> L2{L2: Semantic Match?}
L2 -- Hit (Similarity > 0.95) --> Return
L2 -- Miss --> Store[Store in L1 & L2 Cache]
Store --> Return
- Input: Text to embed.
- L1 Lookup: Calculate SHA256 hash, check table. Return cached embedding on hit (similarity 1.0).
- L2 Lookup:
- Call
EmbeddingServiceto generate embedding vector. - Iterate through all valid entries in semantic cache.
- Calculate Cosine Similarity between current and cached vectors.
- Record max similarity
max_simand corresponding entry.
- Call
- Threshold: If
max_sim>SimilarityThreshold(default 0.95), treat as semantic hit. - Update: If both miss, store new embedding in L1 and L2 cache.
Statistics
The semantic cache tracks detailed statistics for monitoring and optimization:
| Metric | Description |
|---|---|
ExactHits |
Number of exact cache hits |
ExactMisses |
Number of exact cache misses |
SemanticHits |
Number of semantic cache hits |
SemanticMisses |
Number of semantic cache misses |
SemanticSize |
Current number of entries |
SimilarityDistribution |
Distribution of similarity scores (buckets: 0.00-0.90, 0.90-0.95, 0.95-1.00) |
Configuration
| Config | Default | Description |
|---|---|---|
MaxEntries |
1000 | Maximum cache entries |
SimilarityThreshold |
0.95 | Semantic similarity threshold (0-1) |
TTL |
24h | Cache entry time-to-live |
Documentation
¶
Overview ¶
Package cache provides the cache service interface for AI agents. This interface is consumed by Team B (Assistant+Schedule) and Team C (Memo Enhancement).
Package cache provides semantic caching for AI agents. Issue #91: 语义缓存层实现 - 基于 Embedding 相似度匹配
Index ¶
- type ByteLRUCache
- type CacheService
- type EmbeddingService
- type LRUCache
- func (c *LRUCache[K, V]) Capacity() int
- func (c *LRUCache[K, V]) CleanupExpired() int
- func (c *LRUCache[K, V]) Clear()
- func (c *LRUCache[K, V]) Contains(key K) bool
- func (c *LRUCache[K, V]) Get(key K) (V, bool)
- func (c *LRUCache[K, V]) Invalidate(pattern string) int
- func (c *LRUCache[K, V]) Remove(key K) bool
- func (c *LRUCache[K, V]) Set(key K, value V, ttl time.Duration)
- func (c *LRUCache[K, V]) SetWithDefaultTTL(key K, value V)
- func (c *LRUCache[K, V]) Size() int
- type MockCacheService
- func (m *MockCacheService) Clear()
- func (m *MockCacheService) Get(ctx context.Context, key string) ([]byte, bool)
- func (m *MockCacheService) Invalidate(ctx context.Context, pattern string) error
- func (m *MockCacheService) Set(ctx context.Context, key string, value []byte, ttl time.Duration) error
- func (m *MockCacheService) Size() int
- type SemanticCache
- type SemanticCacheConfig
- type SemanticCacheStats
- type Service
- func (s *Service) Clear()
- func (s *Service) Close()
- func (s *Service) Get(_ context.Context, key string) ([]byte, bool)
- func (s *Service) Invalidate(_ context.Context, pattern string) error
- func (s *Service) Set(_ context.Context, key string, value []byte, ttl time.Duration) error
- func (s *Service) Size() int
- type ServiceConfig
- type StringLRUCache
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ByteLRUCache ¶ added in v0.100.0
ByteLRUCache is a type alias for LRUCache with string keys and []byte values. Provided for backward compatibility with existing code that uses []byte values.
func NewByteLRUCache ¶ added in v0.100.0
func NewByteLRUCache(capacity int, defaultTTL time.Duration) *ByteLRUCache
NewByteLRUCache creates a new LRU cache with string keys and []byte values. This is a convenience function for the common case of caching byte slices.
type CacheService ¶
type CacheService interface {
// Get retrieves a value from cache.
// Returns: value, whether it exists
Get(ctx context.Context, key string) ([]byte, bool)
// Set stores a value in cache.
// ttl: expiration time
Set(ctx context.Context, key string, value []byte, ttl time.Duration) error
// Invalidate invalidates cache entries.
// pattern: supports wildcards (user:123:*)
Invalidate(ctx context.Context, pattern string) error
}
Consumers: Team B (Assistant+Schedule), Team C (Memo Enhancement).
type EmbeddingService ¶ added in v0.94.0
type EmbeddingService interface {
// Embed generates a vector embedding for the given text.
Embed(ctx context.Context, text string) ([]float32, error)
// EmbedBatch generates vector embeddings for multiple texts.
EmbedBatch(ctx context.Context, texts []string) ([][]float32, error)
}
EmbeddingService defines the interface for generating vector embeddings. This is a local interface to avoid circular dependencies.
type LRUCache ¶
type LRUCache[K comparable, V any] struct { // contains filtered or unexported fields }
LRUCache implements an LRU cache with TTL support and generics. LRUCache 实现支持 TTL 和泛型的 LRU 缓存。
func NewLRUCache ¶
NewLRUCache creates a new LRU cache.
func (*LRUCache[K, V]) Capacity ¶ added in v0.94.0
Capacity returns the maximum capacity of the cache.
func (*LRUCache[K, V]) CleanupExpired ¶
CleanupExpired removes all expired entries. Returns the number of entries removed.
func (*LRUCache[K, V]) Clear ¶
func (c *LRUCache[K, V]) Clear()
Clear removes all entries from the cache.
func (*LRUCache[K, V]) Contains ¶ added in v0.100.0
Contains checks if a key exists in the cache (without updating access order).
IMPORTANT: Unlike Get, this method does NOT remove expired entries. It only checks if the key exists AND has not expired. This means Contains() may return true while a subsequent Get() returns false (if the entry expired between the two calls).
This "read-only" semantics is intentional for performance. If you need consistent behavior with Get, call Get instead.
Example:
// DON'T: Check then Get (race condition possible)
if cache.Contains(key) {
val, ok := cache.Get(key) // ok may be false!
}
// DO: Just call Get directly
if val, ok := cache.Get(key); ok {
// use val
}
func (*LRUCache[K, V]) Get ¶
Get retrieves a value from the cache. Uses a two-phase locking strategy: RLock for read, upgrade to Lock only if modification needed.
func (*LRUCache[K, V]) Invalidate ¶
Invalidate removes entries matching the pattern. Supports * wildcard at the end (e.g., "user:123:*"). Note: This method only works for string keys. For other key types, use Remove.
func (*LRUCache[K, V]) SetWithDefaultTTL ¶ added in v0.100.0
func (c *LRUCache[K, V]) SetWithDefaultTTL(key K, value V)
SetWithDefaultTTL stores a value using the default TTL.
type MockCacheService ¶
type MockCacheService struct {
// contains filtered or unexported fields
}
MockCacheService is a mock implementation of CacheService for testing.
func NewMockCacheService ¶
func NewMockCacheService() *MockCacheService
NewMockCacheService creates a new MockCacheService.
func (*MockCacheService) Clear ¶
func (m *MockCacheService) Clear()
Clear removes all items from the cache (for testing).
func (*MockCacheService) Invalidate ¶
func (m *MockCacheService) Invalidate(ctx context.Context, pattern string) error
Invalidate invalidates cache entries.
func (*MockCacheService) Set ¶
func (m *MockCacheService) Set(ctx context.Context, key string, value []byte, ttl time.Duration) error
Set stores a value in cache.
func (*MockCacheService) Size ¶
func (m *MockCacheService) Size() int
Size returns the number of items in the cache (for testing).
type SemanticCache ¶ added in v0.94.0
type SemanticCache struct {
// contains filtered or unexported fields
}
SemanticCache provides two-layer caching: exact match (SHA256) and semantic match (cosine similarity).
func NewSemanticCache ¶ added in v0.94.0
func NewSemanticCache(cfg SemanticCacheConfig) *SemanticCache
NewSemanticCache creates a new semantic cache.
func (*SemanticCache) Clear ¶ added in v0.94.0
func (c *SemanticCache) Clear()
Clear clears all cache entries.
func (*SemanticCache) Get ¶ added in v0.94.0
Get retrieves a cached embedding. Returns (embedding, found, similarity, isExactMatch).
func (*SemanticCache) GetStats ¶ added in v0.94.0
func (c *SemanticCache) GetStats() SemanticCacheStats
GetStats returns the cache statistics.
type SemanticCacheConfig ¶ added in v0.94.0
type SemanticCacheConfig struct {
// MaxEntries is the maximum number of entries in the cache.
MaxEntries int
// SimilarityThreshold is the minimum cosine similarity for a match (0-1).
SimilarityThreshold float32
// TTL is the time-to-live for cache entries.
TTL time.Duration
// EmbeddingService is the vector embedding service.
EmbeddingService EmbeddingService
}
SemanticCacheConfig configures the semantic cache.
func DefaultSemanticCacheConfig ¶ added in v0.94.0
func DefaultSemanticCacheConfig() SemanticCacheConfig
DefaultSemanticCacheConfig returns the default configuration.
type SemanticCacheStats ¶ added in v0.94.0
type SemanticCacheStats struct {
ExactHits int64
ExactMisses int64
SemanticHits int64
SemanticMisses int64
SemanticSize int
SimilarityDistribution map[string]int64
}
SemanticCacheStats represents cache statistics.
func (SemanticCacheStats) MarshalJSON ¶ added in v0.94.0
func (s SemanticCacheStats) MarshalJSON() ([]byte, error)
MarshalJSON implements json.Marshaler for stats.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service implements CacheService with LRU eviction.
func NewService ¶
func NewService(cfg ServiceConfig) *Service
NewService creates a new cache service.
func (*Service) Invalidate ¶
Invalidate invalidates cache entries matching the pattern.
type ServiceConfig ¶
type ServiceConfig struct {
Capacity int // Maximum number of entries (default: 1000)
DefaultTTL time.Duration // Default TTL for entries (default: 5 minutes)
CleanupInterval time.Duration // Interval for expired entry cleanup (default: 1 minute)
}
ServiceConfig configures the cache service.
func DefaultServiceConfig ¶
func DefaultServiceConfig() ServiceConfig
DefaultServiceConfig returns default cache service configuration.
type StringLRUCache ¶ added in v0.100.0
StringLRUCache is a type alias for LRUCache with string keys and string values.
func NewStringLRUCache ¶ added in v0.100.0
func NewStringLRUCache(capacity int, defaultTTL time.Duration) *StringLRUCache
NewStringLRUCache creates a new LRU cache with string keys and string values.
Source Files