Documentation
¶
Overview ¶
Package persistence adds durable storage to ConcurrentOrderedMap. It uses a Write-Ahead Log (WAL) for crash-safe individual writes and periodic binary snapshots for fast recovery.
Design goals:
- Zero external dependencies (stdlib only: os, encoding/gob, compress/zlib)
- Non-blocking reads; writers hold the lock only during encode+append
- Recovery is deterministic: snapshot base + WAL replay in sequence order
- Optional fsync-on-write for mission-critical durability
Index ¶
- type FlatVectorIndex
- type OpType
- type Options
- type PersistentConcurrentOrderedMap
- func (p *PersistentConcurrentOrderedMap[K, V]) Close() error
- func (p *PersistentConcurrentOrderedMap[K, V]) Delete(key K) error
- func (p *PersistentConcurrentOrderedMap[K, V]) Exists(key K) bool
- func (p *PersistentConcurrentOrderedMap[K, V]) Get(key K) (V, bool)
- func (p *PersistentConcurrentOrderedMap[K, V]) Inner() *com.ConcurrentOrderedMap[K, V]
- func (p *PersistentConcurrentOrderedMap[K, V]) Len() int
- func (p *PersistentConcurrentOrderedMap[K, V]) Set(key K, value V) error
- func (p *PersistentConcurrentOrderedMap[K, V]) SetWithVector(key K, value V, vec []float64) error
- func (p *PersistentConcurrentOrderedMap[K, V]) Snapshot() error
- func (p *PersistentConcurrentOrderedMap[K, V]) Sync() error
- func (p *PersistentConcurrentOrderedMap[K, V]) VectorSearch(queryVec []float64, topK int, filter func(K) bool) ([]VectorResult[K], error)
- type VectorResult
- type WALWriter
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type FlatVectorIndex ¶
type FlatVectorIndex[K comparable] struct { // contains filtered or unexported fields }
FlatVectorIndex is an in-memory brute-force cosine-similarity index.
Suitable for up to ~500 000 vectors. For larger corpora consider a hierarchical index (HNSW) as a drop-in replacement behind the same interface.
All operations are safe for concurrent use.
func NewFlatVectorIndex ¶
func NewFlatVectorIndex[K comparable](dims int) *FlatVectorIndex[K]
NewFlatVectorIndex creates an empty index that expects vectors of `dims` dimensions.
func (*FlatVectorIndex[K]) Delete ¶
func (idx *FlatVectorIndex[K]) Delete(key K)
Delete removes a key from the index.
func (*FlatVectorIndex[K]) Len ¶
func (idx *FlatVectorIndex[K]) Len() int
Len returns the number of vectors in the index.
func (*FlatVectorIndex[K]) Search ¶
func (idx *FlatVectorIndex[K]) Search( queryVec []float64, topK int, filter func(K) bool, ) ([]VectorResult[K], error)
Search returns the topK nearest neighbours of queryVec by cosine similarity. filter is an optional key-predicate; pass nil to search all entries.
func (*FlatVectorIndex[K]) Upsert ¶
func (idx *FlatVectorIndex[K]) Upsert(key K, vec []float64) error
Upsert adds or replaces a vector for key. The vector is normalised internally so raw embeddings (e.g. BGE-M3 outputs) can be passed directly.
type Options ¶
type Options struct {
// SnapshotInterval is how often a compaction snapshot is written.
// Set to 0 to disable automatic snapshots (you can call Snapshot() manually).
SnapshotInterval time.Duration
// MaxWALEntries triggers a snapshot + WAL truncation once the WAL grows
// beyond this many entries. 0 = disabled.
MaxWALEntries int
// SyncOnWrite calls fdatasync after every WAL append.
// Safer but slower. Recommended for critical data.
SyncOnWrite bool
// VectorDims is the embedding dimensionality. Set >0 to enable the vector
// index. Vectors are stored alongside values using SetWithVector.
VectorDims int
}
Options configures the persistent map.
type PersistentConcurrentOrderedMap ¶
type PersistentConcurrentOrderedMap[K comparable, V any] struct { // contains filtered or unexported fields }
PersistentConcurrentOrderedMap wraps ConcurrentOrderedMap with WAL-based durability and an optional in-memory vector index for semantic search.
Files created under `dir`:
- data.wal — append-only write-ahead log
- data.snap — latest binary snapshot
- data.snap.tmp — transient during snapshot write (atomic rename)
func New ¶
func New[K comparable, V any](dir string, opts Options) (*PersistentConcurrentOrderedMap[K, V], error)
New opens or creates a PersistentConcurrentOrderedMap in `dir`. If existing data is found it is loaded before the function returns.
func (*PersistentConcurrentOrderedMap[K, V]) Close ¶
func (p *PersistentConcurrentOrderedMap[K, V]) Close() error
Close flushes, takes a final snapshot, and releases all resources.
func (*PersistentConcurrentOrderedMap[K, V]) Delete ¶
func (p *PersistentConcurrentOrderedMap[K, V]) Delete(key K) error
Delete removes key/value and appends a WAL delete entry.
func (*PersistentConcurrentOrderedMap[K, V]) Exists ¶
func (p *PersistentConcurrentOrderedMap[K, V]) Exists(key K) bool
Exists reports whether key is present.
func (*PersistentConcurrentOrderedMap[K, V]) Get ¶
func (p *PersistentConcurrentOrderedMap[K, V]) Get(key K) (V, bool)
Get retrieves a value by key.
func (*PersistentConcurrentOrderedMap[K, V]) Inner ¶
func (p *PersistentConcurrentOrderedMap[K, V]) Inner() *com.ConcurrentOrderedMap[K, V]
Inner exposes the underlying ConcurrentOrderedMap for read operations and similarity searches (Levenshtein, vector, etc.).
func (*PersistentConcurrentOrderedMap[K, V]) Len ¶
func (p *PersistentConcurrentOrderedMap[K, V]) Len() int
Len returns the number of stored entries.
func (*PersistentConcurrentOrderedMap[K, V]) Set ¶
func (p *PersistentConcurrentOrderedMap[K, V]) Set(key K, value V) error
Set stores key/value and appends a WAL entry.
func (*PersistentConcurrentOrderedMap[K, V]) SetWithVector ¶
func (p *PersistentConcurrentOrderedMap[K, V]) SetWithVector(key K, value V, vec []float64) error
SetWithVector stores key/value and its embedding vector. The vector is indexed for semantic search (requires VectorDims > 0).
func (*PersistentConcurrentOrderedMap[K, V]) Snapshot ¶
func (p *PersistentConcurrentOrderedMap[K, V]) Snapshot() error
Snapshot writes a compacted snapshot and truncates the WAL.
func (*PersistentConcurrentOrderedMap[K, V]) Sync ¶
func (p *PersistentConcurrentOrderedMap[K, V]) Sync() error
Sync flushes WAL buffers to OS.
func (*PersistentConcurrentOrderedMap[K, V]) VectorSearch ¶
func (p *PersistentConcurrentOrderedMap[K, V]) VectorSearch( queryVec []float64, topK int, filter func(K) bool, ) ([]VectorResult[K], error)
VectorSearch performs a cosine-similarity search over all stored vectors. Requires VectorDims > 0 in Options.
type VectorResult ¶
type VectorResult[K comparable] struct { Key K Score float64 }
VectorResult is a single hit from a vector search.
type WALWriter ¶
type WALWriter[K comparable, V any] struct { // contains filtered or unexported fields }
WALWriter appends entries to a WAL file. It is safe for concurrent use.
Source Files