Documentation
¶
Overview ¶
Package cache provides generic, thread-safe cache implementations with configurable eviction policies.
Strategies supported through one parameterized implementation (memCache):
- StrategySimple: No eviction policy (stores items indefinitely)
- StrategyLRU: Least Recently Used eviction based on max_size
- StrategyTTL: Time-To-Live eviction based on ttl and cleanup_interval
- StrategyHybrid: Combined LRU and TTL eviction
All stateful caches are thread-safe with built-in atomic statistics (always enabled for observability) and optional Prometheus metrics integration via functional options.
Package cache provides high-performance, thread-safe caching implementations with multiple eviction policies, built-in statistics tracking, and optional Prometheus metrics integration.
Overview ¶
The cache package offers one stateful in-memory cache implementation (plus a no-op cache) supporting four eviction strategies:
- Simple: No eviction (manual cleanup only)
- LRU: Least Recently Used eviction
- TTL: Time-To-Live expiration
- Hybrid: Combines LRU and TTL policies
All strategies are generic, thread-safe, and provide comprehensive observability through always-on statistics and optional metrics.
Quick Start ¶
Every constructor except NewNoop returns (Cache[V], error), and the cache is nil when the error is not: check the error before using the cache.
Simple cache creation:
names, err := cache.NewSimple[string]()
if err != nil {
return err
}
if _, err := names.Set("key", "value"); err != nil {
return err
}
value, ok := names.Get("key")
LRU cache with capacity limit:
users, err := cache.NewLRU[*User](1000)
if err != nil {
return err
}
TTL cache with expiration:
sessions, err := cache.NewTTL[*Session](ctx, 30*time.Minute, 5*time.Minute)
if err != nil {
return err
}
Hybrid cache with both LRU and TTL, built from a Config (there is no direct hybrid constructor):
responses, err := cache.NewFromConfig[[]byte](ctx, cache.Config{
Enabled: true, Strategy: cache.StrategyHybrid,
MaxSize: 5000, TTL: 10 * time.Minute, CleanupInterval: 1 * time.Minute,
}, cache.WithMetrics[[]byte](registry, "api_cache"))
if err != nil {
return err
}
Cache Types and Eviction Policies ¶
Simple Cache (No Eviction):
Items remain in cache until explicitly deleted or cache is cleared. Best for small, stable datasets where manual control is desired.
c, err := cache.NewSimple[V]()
LRU Cache (Capacity-Based):
Evicts least recently used items when maximum capacity is reached. Best for fixed-size caches where recent access patterns indicate importance.
c, err := cache.NewLRU[V](maxSize)
TTL Cache (Time-Based):
Items expire after a time-to-live period. Background cleanup goroutine removes expired items. Best for time-sensitive data like sessions or tokens.
c, err := cache.NewTTL[V](ctx, ttl, cleanupInterval)
Hybrid Cache (Capacity + Time):
Combines LRU and TTL - items are evicted if they're either least recently used OR expired. Best for production caches requiring both size and time limits.
c, err := cache.NewFromConfig[V](ctx, cache.Config{
Enabled: true, Strategy: cache.StrategyHybrid,
MaxSize: maxSize, TTL: ttl, CleanupInterval: cleanupInterval,
})
In each case check err before using c, as in Quick Start. NewFromConfig with Enabled false returns the no-op cache from NewNoop, which stores nothing and whose Stats is nil.
Observability Architecture ¶
The cache package implements a dual-tracking pattern for comprehensive observability:
Statistics (Always On):
- Tracks all operations using atomic counters
- Zero configuration required
- Available via cache.Stats()
- Provides computed metrics (hit ratio, requests/sec)
- No external dependencies
Prometheus Metrics (Optional):
- Enabled via WithMetrics() option
- Exports to Prometheus for time-series monitoring
- Includes component labels for instance identification
- Standard metric types (Counter, Gauge)
Design Decision: Dual Tracking Pattern ¶
Both Statistics and Metrics track operations independently, which appears redundant but serves distinct operational purposes:
Why Track Twice?
1. Independence: Statistics work without Prometheus dependency
- Always available for debugging, even in minimal deployments
- No external infrastructure required for basic observability
- Critical for tests and local development
2. Computed Metrics: Statistics provide derived values not available in raw Prometheus
- Hit ratio (hits / total requests)
- Requests per second with built-in timing
- Miss ratio (misses / total requests)
3. Different Use Cases:
- Statistics: Programmatic access, debugging, tests, runtime inspection
- Metrics: Time-series analysis, Grafana dashboards, alerting, production monitoring
4. Performance Trade-off:
- Overhead: ~50-100ns per operation for dual tracking
- At 1M ops/sec: ~0.5-1% total overhead
- Cost is negligible compared to observability value
Alternative Considered: Metrics-Based Statistics
We considered reading Statistics from Prometheus metrics to avoid duplication:
func (s *Statistics) Hits() int64 {
dto := &dto.Metric{}
s.metrics.hits.Write(dto)
return int64(dto.Counter.GetValue())
}
Rejected because:
- Creates Prometheus dependency for basic stats
- Reading from Prometheus is significantly slower (~10x) than atomic operations
- Breaks Statistics when metrics are disabled
- Violates separation of concerns (stats vs monitoring)
- Makes testing more complex (requires mock metrics)
Performance Impact ¶
Dual tracking overhead per operation:
- 1x atomic increment (Statistics)
- 1x atomic increment (Prometheus counter) if enabled
- 1x gauge set (Prometheus) if enabled
Benchmarks (M1 MacBook Pro):
- LRU Get with stats only: ~226ns/op
- LRU Get with stats + metrics: ~238ns/op (~5% overhead)
- LRU Set with stats only: ~361ns/op
- LRU Set with stats + metrics: ~379ns/op (~5% overhead)
At high throughput (1M ops/sec), dual tracking adds ~50-100ms/sec of overhead, which is acceptable for the operational visibility gained.
Functional Options Pattern ¶
The package uses functional options for clean, composable configuration:
c, err := cache.NewLRU[V](capacity,
cache.WithMetrics[V](registry, "component"),
)
if err != nil {
return err
}
Available options:
- WithMetrics: Enable Prometheus metrics export
This pattern provides:
- Clear intent with named functions
- Easy composition of features
- Backward compatibility when adding options
- Type-safe configuration with generics
Thread Safety ¶
All cache operations are thread-safe for concurrent use:
- Cache map, LRU list, and size transitions are synchronized under a per-cache mutex
- Size updates to Statistics happen under the cache mutex so Stats().CurrentSize() matches Size()
- Statistics fields use lock-free atomic operations (counters, currentSize, CAS maxSize, atomic.Pointer[time.Time] startTime)
- Metrics use Prometheus atomic types
- TTL and hybrid cleanup runs in a background goroutine joined by Close()
Performance Characteristics ¶
Simple Cache:
- Get: O(1) map lookup
- Set: O(1) map insert
- Delete: O(1) map delete
- Memory: O(n) where n is number of items
LRU Cache:
- Get: O(1) map lookup + list move
- Set: O(1) map insert + list append/evict
- Delete: O(1) map delete + list remove
- Memory: O(n) map + list overhead
TTL Cache:
- Get: O(1) map lookup + expiry check
- Set: O(1) map insert
- Delete: O(1) map delete
- Cleanup: O(n) periodic scan (background)
- Memory: O(n) map + expiry tracking
Hybrid Cache:
- Get: O(1) map lookup + list move + expiry check
- Set: O(1) map insert + list append/evict
- Delete: O(1) map delete + list remove
- Cleanup: O(n) periodic scan (background)
- Memory: O(n) map + list + expiry tracking
Generic Type Support ¶
Caches are fully generic and work with any Go type: the type parameter names the value type, as in cache.NewSimple[string](), cache.NewLRU[int](100), cache.NewTTL[*User](ctx, 5*time.Minute, 1*time.Minute) or cache.NewFromConfig[[]byte](ctx, cfg). Each returns an error to check, as in Quick Start.
Type constraints:
- Keys are always strings (for consistent hashing and comparison)
- Values can be any type V
- No serialization required - stores values directly in memory
Common Use Cases ¶
Every example below continues with `if err != nil { return err }` before the cache is used.
API Response Caching:
responses, err := cache.NewFromConfig[*Response](ctx, cache.Config{
Enabled: true, Strategy: cache.StrategyHybrid,
MaxSize: 5000, TTL: 30 * time.Minute, CleanupInterval: 5 * time.Minute,
}, cache.WithMetrics[*Response](registry, "api_cache"))
Session Storage:
sessions, err := cache.NewTTL[*Session](ctx, 2*time.Hour, 10*time.Minute)
Entity Caching (Two-Level), one error check per constructor:
hot, err := cache.NewLRU[*Entity](1000) // Hot entities
if err != nil {
return err
}
recent, err := cache.NewTTL[*Entity](ctx, 1*time.Hour, 5*time.Minute) // All recent entities
if err != nil {
return err
}
Computed Results:
results, err := cache.NewLRU[*Result](500, cache.WithMetrics[*Result](registry, "computation_cache"), )
Context and Cleanup ¶
TTL and Hybrid caches run background cleanup goroutines. Always pass a context that will be canceled when cleanup should stop:
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
c, err := cache.NewTTL[V](ctx, ttl, cleanupInterval)
if err != nil {
return err
}
// Cleanup goroutine stops when ctx is canceled
Close also stops the cleanup goroutine, and returns once it has exited. NewTTL and NewFromConfig refuse a nil context with an error.
For Simple and LRU caches, no background goroutines are created.
CoalescingSet's goroutine runs the caller's callback, so it stops through Shutdown(ctx): Shutdown returns once the goroutine has exited, or ctx.Err() if ctx ends while the callback is still running; the goroutine then exits when the callback returns. NewCoalescingSet refuses a nil context, a nil callback, or a panic counter the registry refuses with an error, before it starts the goroutine. A panic in the callback is recovered: the batch is dropped, the panic is logged with the batch size (WithPanicLogger) and counted when WithCoalescingMetrics names a registry, and later batches still fire.
Testing ¶
The package includes comprehensive tests with race detection:
go test -race ./internal/cache
Benchmarks are available to validate performance:
go test -bench=. ./internal/cache
Statistics make testing cache behavior easy:
c, err := cache.NewSimple[int]()
require.NoError(t, err)
_, err = c.Set("key", 42)
require.NoError(t, err)
_, _ = c.Get("key")
_, _ = c.Get("missing")
assert.Equal(t, int64(1), c.Stats().Hits())
assert.Equal(t, int64(1), c.Stats().Misses())
assert.Equal(t, 0.5, c.Stats().HitRatio())
Examples ¶
The tests in cache_test.go show each cache in use; the package has no Example functions.
Index ¶
- type Cache
- func NewFromConfig[V any](ctx context.Context, config Config, options ...Option[V]) (Cache[V], error)
- func NewLRU[V any](maxSize int, options ...Option[V]) (Cache[V], error)
- func NewNoop[V any]() Cache[V]
- func NewSimple[V any](options ...Option[V]) (Cache[V], error)
- func NewTTL[V any](ctx context.Context, ttl, cleanupInterval time.Duration, options ...Option[V]) (Cache[V], error)
- type CoalescingOption
- type CoalescingSet
- type Config
- type Option
- type Statistics
- func (s *Statistics) CurrentSize() int64
- func (s *Statistics) Delete()
- func (s *Statistics) Deletes() int64
- func (s *Statistics) Eviction()
- func (s *Statistics) Evictions() int64
- func (s *Statistics) Hit()
- func (s *Statistics) HitRatio() float64
- func (s *Statistics) Hits() int64
- func (s *Statistics) MaxSize() int64
- func (s *Statistics) Miss()
- func (s *Statistics) MissRatio() float64
- func (s *Statistics) Misses() int64
- func (s *Statistics) RequestsPerSecond() float64
- func (s *Statistics) Reset()
- func (s *Statistics) Set()
- func (s *Statistics) Sets() int64
- func (s *Statistics) Summary() StatsSummary
- func (s *Statistics) UpdateSize(size int64)
- func (s *Statistics) Uptime() time.Duration
- type StatsSummary
- type Strategy
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Cache ¶
type Cache[V any] interface { // Get retrieves a value by key. Returns the value and true if found, zero value and false otherwise. Get(key string) (V, bool) // Set stores a value with the given key. Returns true if a new entry was created, false if updated. // Returns an error if the operation fails (e.g., invalid key). Set(key string, value V) (bool, error) // Delete removes an entry by key. Returns true if the key existed and was deleted. // Returns an error if the operation fails. Delete(key string) (bool, error) // Clear removes all entries from the cache. Clear() // Size returns the current number of entries in the cache. Size() int // Keys returns a slice of all keys currently in the cache. Keys() []string // Stats returns cache statistics if enabled, nil otherwise. Stats() *Statistics // Close shuts down the cache and releases any resources (e.g., background goroutines). // After Close, Get, Set and Delete keep working; only background expiry stops. Close() error }
Cache represents a generic cache interface that all cache implementations must satisfy. The cache is parameterized by value type V for type safety.
func NewFromConfig ¶
func NewFromConfig[V any](ctx context.Context, config Config, options ...Option[V]) (Cache[V], error)
NewFromConfig creates a cache based on the provided configuration. Returns a disabled cache (NoopCache) if config.Enabled is false. Additional functional options can be passed to configure metrics, callbacks, etc.
func NewLRU ¶
NewLRU creates a new LRU cache with the specified maximum size. Stats are always enabled for observability. Use WithMetrics() to also export as Prometheus metrics.
func NewNoop ¶
NewNoop creates a cache that does nothing (always returns cache misses). This is useful when caching is disabled via configuration.
func NewSimple ¶
NewSimple creates a new Simple cache with no eviction policy. Stats are always enabled for observability. Use WithMetrics() to also export as Prometheus metrics.
func NewTTL ¶
func NewTTL[V any](ctx context.Context, ttl, cleanupInterval time.Duration, options ...Option[V]) (Cache[V], error)
NewTTL creates a new TTL cache with the specified TTL and cleanup interval. Stats are always enabled for observability. Use WithMetrics() to also export as Prometheus metrics.
type CoalescingOption ¶
type CoalescingOption func(*coalescingOptions)
CoalescingOption configures a CoalescingSet.
func WithCoalescingMetrics ¶
func WithCoalescingMetrics(registry *metric.MetricsRegistry, prefix string) CoalescingOption
WithCoalescingMetrics counts recovered callback panics on semengine_cache_coalescing_callback_panics_total{component=prefix}, registered on registry the way WithMetrics registers a cache's collectors. A nil registry or an empty prefix is ignored, as WithMetrics ignores them: a panic is then logged but not counted.
func WithPanicLogger ¶
func WithPanicLogger(logger *slog.Logger) CoalescingOption
WithPanicLogger sets the logger that records a recovered callback panic. A nil logger keeps the default, slog.Default(). The package has no other logger path.
type CoalescingSet ¶
type CoalescingSet struct {
// contains filtered or unexported fields
}
CoalescingSet collects keys over a time window and fires a callback with the batch. It deduplicates keys automatically using a map-based set structure. Its background goroutine runs the caller's callback, so it stops through Shutdown(ctx), which bounds the wait for that callback by the caller's context (background-work shape 3).
func NewCoalescingSet ¶
func NewCoalescingSet( ctx context.Context, window time.Duration, callback func([]string), opts ...CoalescingOption, ) (*CoalescingSet, error)
NewCoalescingSet creates a new CoalescingSet that fires the callback every window duration with the collected (deduplicated) keys. The background goroutine stops when ctx is cancelled or when Shutdown is called. A nil ctx or a nil callback is refused with an invalid-argument error, and a panic counter the registry refuses with a transient error, as the cache constructors return a metrics-registration error; in each case no goroutine starts. A panic in the callback is recovered: the set is a helper inside a service, not a root process. The batch the callback was handed is dropped, the panic is logged at error level with the batch size and counted when WithCoalescingMetrics names a registry, and later batches still fire.
func (*CoalescingSet) Add ¶
func (c *CoalescingSet) Add(key string) bool
Add adds a key to the pending set and reports whether it was newly inserted. If the key already exists, it is deduplicated. Thread-safe.
func (*CoalescingSet) Drain ¶
func (c *CoalescingSet) Drain() []string
Drain removes and returns every pending key without invoking the callback. Callers use this after Shutdown when queued work owns external resources that must be released during shutdown.
func (*CoalescingSet) PendingCount ¶
func (c *CoalescingSet) PendingCount() int
PendingCount returns the number of keys currently pending in the set. Thread-safe.
func (*CoalescingSet) Remove ¶
func (c *CoalescingSet) Remove(key string) bool
Remove removes a key from the pending set and reports whether it existed. This is useful when an entity is deleted before the window expires. Thread-safe.
func (*CoalescingSet) RemovePrefix ¶
func (c *CoalescingSet) RemovePrefix(prefix string) int
RemovePrefix removes every pending key with the supplied prefix and returns the number removed. This is useful when several provenance-bearing work items belong to the same logical entity and a delete must retire all of them atomically.
func (*CoalescingSet) Shutdown ¶
func (c *CoalescingSet) Shutdown(ctx context.Context) error
Shutdown stops the background goroutine and waits for it to exit, including a callback it is running. It returns nil once the goroutine has exited, and ctx.Err() if ctx ends first: the goroutine then exits when the callback returns, and a later Shutdown returns nil. It is safe to call more than once and from several goroutines. A nil ctx is refused with an error; the goroutine is not stopped.
type Config ¶
type Config struct {
// Enabled determines if caching is enabled.
Enabled bool `json:"enabled" schema:"editable,type:bool,description:Enable caching"`
// Strategy determines the eviction strategy.
Strategy Strategy `json:"strategy" schema:"editable,type:enum,description:Cache eviction strategy,enum:simple|lru|ttl|hybrid"`
// MaxSize is the maximum number of entries (for LRU and Hybrid caches).
MaxSize int `json:"max_size" schema:"editable,type:int,description:Maximum number of cache entries (for LRU and Hybrid),min:1"`
// TTL is the time-to-live for entries (for TTL and Hybrid caches).
TTL time.Duration `json:"ttl" schema:"editable,type:string,description:Time-to-live for entries (for TTL and Hybrid)"`
// CleanupInterval is how often to run background cleanup (for TTL and Hybrid caches).
CleanupInterval time.Duration `json:"cleanup_interval" schema:"editable,type:string,description:How often to run background cleanup (for TTL and Hybrid)"`
}
Config contains configuration for cache creation.
func DefaultConfig ¶
func DefaultConfig() Config
DefaultConfig returns a default cache configuration.
func (*Config) UnmarshalJSON ¶
UnmarshalJSON implements custom JSON unmarshaling for Config to support duration strings (e.g., "1h", "5m", "30s") in addition to nanosecond integers.
type Option ¶
type Option[V any] func(*cacheOptions[V])
Option configures cache behavior using the functional options pattern. This provides a clean, extensible API for configuring caches.
func WithMetrics ¶
func WithMetrics[V any](registry *metric.MetricsRegistry, prefix string) Option[V]
WithMetrics enables Prometheus metrics export for cache statistics. If registry is nil, this option is ignored. Registry should not be nil in normal usage - this handles edge cases gracefully.
type Statistics ¶
type Statistics struct {
// contains filtered or unexported fields
}
Statistics tracks cache performance metrics using lock-free atomic counters.
func NewStatistics ¶
func NewStatistics() *Statistics
NewStatistics creates a new statistics tracker.
func (*Statistics) CurrentSize ¶
func (s *Statistics) CurrentSize() int64
CurrentSize returns the current number of entries in the cache.
func (*Statistics) Deletes ¶
func (s *Statistics) Deletes() int64
Deletes returns the total number of delete operations.
func (*Statistics) Evictions ¶
func (s *Statistics) Evictions() int64
Evictions returns the total number of evictions.
func (*Statistics) HitRatio ¶
func (s *Statistics) HitRatio() float64
HitRatio returns the cache hit ratio as a percentage (0.0 to 1.0).
func (*Statistics) Hits ¶
func (s *Statistics) Hits() int64
Hits returns the total number of cache hits.
func (*Statistics) MaxSize ¶
func (s *Statistics) MaxSize() int64
MaxSize returns the maximum number of entries the cache has held.
func (*Statistics) MissRatio ¶
func (s *Statistics) MissRatio() float64
MissRatio returns the cache miss ratio as a percentage (0.0 to 1.0).
func (*Statistics) Misses ¶
func (s *Statistics) Misses() int64
Misses returns the total number of cache misses.
func (*Statistics) RequestsPerSecond ¶
func (s *Statistics) RequestsPerSecond() float64
RequestsPerSecond returns the average number of requests (hits + misses) per second.
func (*Statistics) Reset ¶
func (s *Statistics) Reset()
Reset resets all statistics to zero. Callers must ensure cache mutations (UpdateSize) are quiescent when calling Reset; Reset does not provide an atomic multi-field reset across concurrent size updates.
func (*Statistics) Sets ¶
func (s *Statistics) Sets() int64
Sets returns the total number of set operations.
func (*Statistics) Summary ¶
func (s *Statistics) Summary() StatsSummary
Summary returns a snapshot of all statistics.
func (*Statistics) UpdateSize ¶
func (s *Statistics) UpdateSize(size int64)
UpdateSize updates the current cache size and advances MaxSize when size exceeds it.
func (*Statistics) Uptime ¶
func (s *Statistics) Uptime() time.Duration
Uptime returns how long the cache has been running using the stored start time's monotonic clock reading.
type StatsSummary ¶
type StatsSummary struct {
Hits int64 `json:"hits"`
Misses int64 `json:"misses"`
Sets int64 `json:"sets"`
Deletes int64 `json:"deletes"`
Evictions int64 `json:"evictions"`
CurrentSize int64 `json:"current_size"`
MaxSize int64 `json:"max_size"`
HitRatio float64 `json:"hit_ratio"`
MissRatio float64 `json:"miss_ratio"`
RequestsPerSecond float64 `json:"requests_per_second"`
Uptime time.Duration `json:"uptime"`
}
StatsSummary returns a snapshot of all statistics.
type Strategy ¶
type Strategy string
Strategy defines the eviction strategy for the cache.
const ( // StrategySimple uses no eviction policy. StrategySimple Strategy = "simple" // StrategyLRU uses Least Recently Used eviction based on size. StrategyLRU Strategy = "lru" // StrategyTTL uses Time-To-Live eviction based on expiry. StrategyTTL Strategy = "ttl" // StrategyHybrid uses combined LRU and TTL eviction. StrategyHybrid Strategy = "hybrid" )