Help us shape the future of Go! Take the 2026 Go Developer Survey

storage

package module
v1.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 12, 2026 License: Apache-2.0 Imports: 27 Imported by: 0

Documentation

Index

Constants

View Source
const (
	// Default disk path
	DefaultDiskPath = "/var/cache"

	// Default TTL (seconds) when no key-level TTL is set
	DefaultTTL = 0

	// Default inline threshold (bytes) for small objects that are inlined in RocksDB
	DefaultInlineThreshold = 64 * 1024 // 64KB

	// Default compact threshold (bytes) for objects that are compacted to segments
	DefaultCompactThreshold = 64 * 1024 * 1024 // 64MB

	// Default segment size (bytes)
	DefaultSegmentSize = 256 * 1024 * 1024 // 256MB

	// Default file descriptor cache size (entries)
	DefaultFdCacheSize = 10000

	// Default max disk usage (bytes)
	DefaultMaxDiskUsage = 0

	// Default compaction threads
	DefaultCompactionThreads = 1 // Default to single thread for backwards compatibility

	// Default TTL cleanup interval
	DefaultTTLCleanupInterval = 1 * time.Minute

	// Default access update buffer size, batch interval and delay
	DefaultAccessUpdateBufferSize = 100000
	DefaultAccessUpdateInterval   = 1 * time.Second
	DefaultAccessUpdateDelay      = 5 * time.Minute

	// Default queue config
	DefaultDeleteBatchSize = 1000 // Number of deletions to process per batch

	// Default segment recompaction settings
	DefaultRecompactionDisabled          = false
	DefaultFragmentationThreshold        = 0.5             // Recompact when dead space exceeds 50%
	DefaultMinSegmentAgeForRecompaction  = 2 * time.Hour   // Don't recompact segments younger than 2 hours (ensures they're cold)
	DefaultMinSegmentsBeforeRecompaction = 2               // Minimum number of segments required for recompaction
	DefaultRecompactionInterval          = 1 * time.Minute // Interval between segment recompaction runs

	// Default delete queue settings
	DeleteProcessInterval = time.Second      // Interval between batch processing
	DeletePruneAge        = 24 * time.Hour   // Age after which entries are pruned
	DeleteRetryDelay      = 30 * time.Second // Backoff before retrying a failed deletion (bounds retry churn for stuck files)

	// Default number of parallel workers for startup file recovery
	DefaultRecoveryWorkers = files.MaxWorkers

	// Default RocksDB configuration
	DefaultMetadataCacheSize      = metadata.DefaultRocksDBBlockCacheSize
	DefaultMetadataBackgroundJobs = metadata.DefaultRocksDBMaxBackgroundJobs
)
View Source
const (
	// BucketDuration defines the time window for each bucket (1 hour)
	BucketDuration = time.Hour
)
View Source
const MaxListValueSize int64 = 1 << 20 // 1 MiB

MaxListValueSize caps the per-value bytes returned by List-with-values. A value larger than this is not read from disk; the entry is returned with the value omitted (and its size reported) so a List over large objects cannot buffer an object-sized allocation or exceed the gRPC message limit (#165).

Variables

This section is empty.

Functions

func GetOldAccessBucketPrefixes

func GetOldAccessBucketPrefixes(olderThan time.Duration) []string

GetOldAccessBucketPrefixes returns a list of bucket prefixes older than the given duration that can be deleted during maintenance

func GetOldestAccessBucketPrefix

func GetOldestAccessBucketPrefix() []byte

GetOldestBucket returns the prefix for iterating the oldest bucket This starts from the beginning of the access index

Types

type Cleaner

type Cleaner struct {
	// contains filtered or unexported fields
}

Cleaner is responsible for background TTL cleanup and LRU eviction

func NewCleaner

func NewCleaner(storage *Storage, interval time.Duration, maxDiskUsage int64) *Cleaner

NewCleaner creates a new Cleaner for background TTL cleanup and LRU eviction

func (*Cleaner) Close

func (c *Cleaner) Close()

Close stops the background cleanup loop and waits for it to exit

func (*Cleaner) Start

func (c *Cleaner) Start()

Start launches the background cleanup goroutine It performs an initial size calculation synchronously to establish accurate baseline before any concurrent operations can modify the size

func (*Cleaner) Stats

func (c *Cleaner) Stats() (cleaned, evicted int64)

Stats returns cleaner statistics

func (*Cleaner) UpdateSize

func (c *Cleaner) UpdateSize(delta int64)

UpdateSize updates the tracked total size when keys are added/removed

func (*Cleaner) WaitForInitialization

func (c *Cleaner) WaitForInitialization()

WaitForInitialization waits until the cleaner has completed its initial size calculation

type KeyValue

type KeyValue struct {
	Key   string
	Value []byte
	// ValueLength is the size of the value in bytes, set even when the value was
	// omitted for exceeding MaxListValueSize.
	ValueLength int64
	// ValueOmitted is true when the value was not read because it exceeds
	// MaxListValueSize; Value is nil in that case.
	ValueOmitted bool
}

KeyValue holds a key and its associated value bytes.

type SegmentDeleteStats

type SegmentDeleteStats struct {
	SegmentPath    string
	DeletedEntries int64
	DeletedBytes   int64
}

SegmentDeleteStats holds deletion statistics for a segment

type Storage

type Storage struct {
	// contains filtered or unexported fields
}

Storage wraps all RocksDB access and related logic It provides methods to store, retrieve, delete, and list keys

Value encoding format: For small objects (in-memory):

"S|" + [8-byte big-endian expiry] + '|' + [data bytes]

For large objects (spilled to disk):

"L|" + [8-byte big-endian expiry] + '|' + [file path as bytes]

If no TTL is set, expiry and separator are omitted:

"S|" + [data bytes] or "L|" + [file path as bytes]

The separator '|' after the expiry ensures robust parsing, even if the data or file path contains '|'.

On read, expiry is checked (if present) and expired keys are deleted and not returned.

func NewStorageWithConfig

func NewStorageWithConfig(config *StorageConfig) (*Storage, error)

NewStorageWithConfig creates a new isolated Storage instance with the given config.

func (*Storage) CleanerStats

func (s *Storage) CleanerStats() (cleaned, evicted int64)

CleanerStats returns statistics from the background cleaner

func (*Storage) Close

func (s *Storage) Close()

Close closes this storage instance and releases all resources. This is safe to call on isolated instances created with NewStorageWithConfig.

func (*Storage) DeleteKey

func (s *Storage) DeleteKey(key string) error

DeleteKey removes metadata and spills for a key

func (*Storage) FlushAccessUpdates

func (s *Storage) FlushAccessUpdates()

FlushAccessUpdates forces all pending access time updates to be written immediately This is mainly useful for testing to ensure deterministic behavior

func (*Storage) Get

func (s *Storage) Get(key string, start, end int64) (io.Reader, bool, error)

Get retrieves the value for the given key from the database and returns an io.Reader for streaming Supports byte-range requests via start and end parameters (0 means no limit)

func (*Storage) GetDeleteIndexStats

func (s *Storage) GetDeleteIndexStats(segmentPath string) (deletedEntries, deletedBytes int64, err error)

GetDeleteIndexStats returns the delete index statistics for a segment

func (*Storage) IsClosed

func (s *Storage) IsClosed() bool

IsClosed returns true if this storage instance has been closed. This can be used to check if it's safe to call other Storage methods.

func (*Storage) ListKeyValuesWithPagination

func (s *Storage) ListKeyValuesWithPagination(userPrefix string, startKey string, limit int) ([]KeyValue, string, bool, error)

ListKeyValuesWithPagination returns paginated, sorted key-value pairs from RocksDB. For inline values the data is returned directly; for file/segment values the data is read from disk. Returns: (entries, lastKey, hasMore, error).

func (*Storage) ListKeys

func (s *Storage) ListKeys(userPrefix string) ([]string, error)

ListKeys returns all non-expired keys in the RocksDB instance that match the given prefix Note: Expired keys are skipped but not deleted - deletion is handled by the background cleaner

func (*Storage) ListKeysWithPagination

func (s *Storage) ListKeysWithPagination(userPrefix string, startKey string, limit int) ([]string, string, bool, error)

ListKeysWithPagination returns paginated, sorted keys from RocksDB Returns: (keys, lastKey, hasMore, error) - keys: Up to 'limit' keys starting after 'startKey' - lastKey: The last key in this page (for continuation) - hasMore: True if more keys exist beyond this page

func (*Storage) ListSegmentDeleteStats

func (s *Storage) ListSegmentDeleteStats() ([]SegmentDeleteStats, error)

ListSegmentDeleteStats returns deletion statistics for all segments in the delete index

func (*Storage) Put

func (s *Storage) Put(key string, body io.Reader, ttl int) error

Put streams the body into spillWriter, stores metadata, and handles TTL

func (*Storage) RemoveDeleteIndex

func (s *Storage) RemoveDeleteIndex(segmentPath string) error

RemoveDeleteIndex removes the delete index entry for a segment (used when segment is removed)

func (*Storage) SetAccessTime

func (s *Storage) SetAccessTime(key string, accessTime int64)

SetAccessTime sets a specific access time for a key This is mainly useful for testing to create predictable LRU scenarios

type StorageConfig

type StorageConfig struct {
	DiskPath             string        // Directory for on-disk cache data
	TTL                  int           // Default TTL when no key-level TTL is set (seconds)
	InlineThreshold      int           // Threshold for small objects that are inlined in RocksDB (bytes)
	CompactThreshold     int64         // Objects less than this size are compacted to segments (bytes)
	SegmentSize          int64         // Segment size (bytes)
	FdCacheSize          int           // Size of the file descriptor cache
	MaxDiskUsage         int64         // Maximum disk usage in bytes (0 = unlimited)
	CompactionThreads    int           // Number of compaction threads
	FragThreshold        float64       // Fragmentation threshold for segment recompaction (0.0-1.0)
	MinSegmentAge        time.Duration // Minimum age for segment recompaction
	MinSegments          int           // Minimum number of segments for recompaction
	DisableRecompaction  bool          // Disable automatic segment recompaction
	RecompactionInterval time.Duration // Interval between segment recompaction runs
	CleanupInterval      time.Duration // Cleanup interval
	AccessUpdateDelay    time.Duration // Access update delay
	RecoveryWorkers      int           // Number of parallel workers for startup file recovery (<= 0 = default)
	DeleteBatchSize      int           // Number of file deletions processed per deletion-queue batch (<= 0 = default)

	// RocksDB-specific configuration
	MetadataCacheSize      int64 // RocksDB Block cache size in bytes (0 = use default)
	MetadataBackgroundJobs int   // Max concurrent RocksDB background jobs, compactions+flushes (0 = use default)
}

StorageConfig holds all configuration parameters for initializing storage

type TimeBucketedAccessIndex

type TimeBucketedAccessIndex struct {
}

TimeBucketedAccessIndex manages access times using time-bucketed keys for LRU eviction without loading all keys into memory

Directories

Path Synopsis
Package merge implements a multiplexing merge operator for RocksDB that supports different merge strategies based on key types.
Package merge implements a multiplexing merge operator for RocksDB that supports different merge strategies based on key types.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL