archive

go-cache-archive

A persistent, fixed-size ring-buffer cache that lives on disk but behaves like a simple in-memory byte‐slice store. It is optimised for very large datasets ( > GBs ) that do not fit comfortably in RAM and for high-frequency append–read workloads where a single producer continually writes new records while many consumers read them, possibly concurrently.

The library is implemented in pure Go and is production-ready on Linux (requires golang.org/x/sys/unix).

Key Features

Why use a disk-backed ring buffer?

Typical RAM usage

go-cache-archive stores exactly recordSize + 4 bytes per entry on disk. With mmap enabled the kernel will only keep hot pages resident. Cold pages are purged without affecting the process’s RSS.

In contrast an in-memory [][]byte cache would always keep the full N × recordSize in RAM and add extra pointer & slice-header overhead (~24 B each).

Theoretical Throughput

* numbers assume SSD/NVMe and the page already loaded in the VFS cache.

In benchmarks with 4 kB records on an NVMe drive:

• Writes: 400–500 MB/s sustained (msync batched every 64 records).
• Reads (cold): 2–3 GB/s sequential thanks to the kernel readahead.

Producer / Consumer Model

flowchart LR
    subgraph Producer/Cache
        P[Producer]
        C[RingBufferCache]
        P -- "Write(ID++, payload)" --> C
    end

    subgraph Consumers
        direction TB
        Consumer1[Consumer 1]
        ConsumerN[... Many Consumers ...]
    end

    C -- "Read(lastID-N..∞)" --> ConsumerN
    Consumer1 -- "Read(ID)" --> C
    C --> Disk[(Cold Storage)]

1. Producer appends one record every second (or faster).
2. Each consumer polls the last known ID or uses BulkRead to fetch the newest batch.
3. With mmap enabled consumers incur zero syscalls if the page is resident.
4. Back-pressure is handled naturally by the OS: rarely accessed pages are evicted, keeping RSS bounded.

Quick Start

import "github.com/luhtfiimanal/go-cache-archive"

// Create a cache allowing IDs 100..5000 (wraps automatically)
opts := archive.DefaultOptions()
opts.MinIDAlloc = 100
opts.MaxIDAlloc = 5000

cache, err := archive.NewRingBufferCacheWithOptions("/var/lib/myapp/cache.dat", opts)
if err != nil { panic(err) }

defer cache.Close()

// producer
payload := make([]byte, 128)
cache.Write(1, payload, true) // flush immediately

// consumer
p, _ := cache.Read(1)

See archive_test.go for more examples.

Auto-sequenced writes (producer convenience)

Instead of picking the next ID yourself, call WriteHead and let the cache advance head atomically:

id, err := cache.WriteHead(payload, true) // flush to disk

// and you can get the current head with
head := cache.Head()

// and tail with
tail := cache.Tail()

head, tail, and Head() / Tail() give you visibility into the current range. They are persisted in a side-car .meta file so the cache resumes correctly after restart.

Project File Layout

Roadmap / Ideas

• Eviction or TTL‐based trimming for infinite streams.
• Compression codecs per record.
• Windows / macOS support.

Contributions welcome — open an issue or PR!