backend

type Backend ¶

type Backend interface {
	// IsEphemeral reports whether the backend stores data only in RAM (dropped on
	// process exit). [Memory] is ephemeral; file and s3 are not.
	IsEphemeral() bool

	// PutIfAbsent stores data under key only if the key does not already exist. It
	// returns true if the write happened, false if the key was already present (no
	// change). Like [Backend.Write] it is atomic per object. It is the compare-and-swap
	// primitive for manifest commits.
	PutIfAbsent(ctx context.Context, key string, data []byte) (bool, error)

	// Write stores data under key, overwriting any existing value. The write is
	// atomic per object: a reader never observes a partially written value. The
	// implementation takes ownership semantics by copying data as needed; callers may
	// reuse the buffer after Write returns.
	Write(ctx context.Context, key string, data []byte) error

	// Read returns the value stored under key. It returns an error satisfying
	// errors.Is(err, [ErrNotExist]) if the key is absent. The returned slice is owned
	// by the caller (implementations return a fresh copy, never aliased state).
	Read(ctx context.Context, key string) ([]byte, error)

	// List returns, sorted ascending, every key with the given prefix (empty prefix
	// lists all keys).
	List(ctx context.Context, prefix string) ([]string, error)

	// Delete removes key. It returns an error satisfying errors.Is(err, [ErrNotExist])
	// if the key is absent.
	Delete(ctx context.Context, key string) error
}

Backend is the L1 storage seam (DESIGN.md §3, §5): a common interface over interchangeable implementations — memory (ephemeral, the reference), file, and (later) s3. The same engine code path runs over all three.

Data is addressed by an opaque, slash-delimited string key (e.g. a time-bucketed object path or a file-relative path). Values are whole objects: the part format (`block`) maps one part to a key prefix and one object per column/marks/manifest, so whole-object Read/Write is sufficient and gives per-object write atomicity. All methods are safe for concurrent use.

Ranged reads are not part of this interface, but they are available as the optional ReaderAt capability: the multi-key layout gives projection pushdown (read only the referenced column objects), and ReaderAt gives the pushdown *within* a column that keeps a query touching a few granules from paying for the whole thing.

Backend.PutIfAbsent is the conditional-write primitive (added in M5) on which atomic manifest / block-list commits build: a versioned manifest key is written only if no writer has claimed that version, so single-writer-wins coordination needs no Raft (it maps to S3 If-None-Match, a filesystem exclusive create, and a guarded map insert).

func Cached(inner Backend, maxBytes int64) Backend

func Memory() Backend

type CacheStats ¶ added in v0.6.0

type CacheStats struct {
	Hits, Misses int64
	Bytes        int64 // resident value bytes
	Items        int   // resident objects
}

CacheStats is a snapshot of a cached backend's effectiveness.

type NodeLocal ¶ added in v0.39.0

type NodeLocal interface {
	IsNodeLocal() bool
}

NodeLocal is the optional capability of reporting that a backend's objects live on a medium private to the process that writes them, so a peer cannot read them. Memory and a local directory tree implement it; an object store does not.

It reports the medium, not the deployment, and is therefore a heuristic in exactly one direction: a `file` backend rooted on a shared mount (NFS, a clustered filesystem) is a legitimate shared store and still answers true. Read a true answer as "assume private unless the operator declares otherwise", never as proof — a false answer is exact.

A wrapper around a Backend must forward it, or the capability is silently lost.

type ObjectCreator ¶ added in v0.37.0

type ObjectCreator interface {
	// CreateObject returns a writer building the object stored under key. Nothing is stored until
	// the writer commits.
	CreateObject(ctx context.Context, key string) (ObjectWriter, error)
}

ObjectCreator is an optional Backend capability: build an object incrementally rather than handing it over whole. It exists so a writer producing an object far larger than it wants resident — a merged part's column — can hand finished bytes to the backend as they are produced instead of holding the whole object in RAM.

Use CreateObject rather than asserting directly: it falls back to buffering into a Backend.Write for backends without the capability, so callers stay correct everywhere. Use StreamsWrites to ask whether that fallback would be taken — the caller that *sizes* its output against memory needs to know, since the fallback puts the object back in RAM.

type ObjectWriter ¶ added in v0.37.0

type ObjectWriter interface {
	io.Writer

	// Commit publishes everything written so far under the writer's key.
	Commit(ctx context.Context) error

	// Abort discards the writer's bytes and releases whatever it holds. It is idempotent and a
	// no-op once Commit has succeeded.
	Abort()
}

ObjectWriter builds one object incrementally. Bytes appended with Write are not visible under the object's key until ObjectWriter.Commit; until then the key reads as it did before (absent, or the previous value), so the "manifest written last" commit rule still holds.

Commit publishes the bytes atomically, exactly like Backend.Write. Abort discards them and is safe to call after Commit (where it does nothing), so `defer w.Abort()` is the correct cleanup. A writer must not be used from more than one goroutine, but several writers may target the same key concurrently — only the one that commits wins, which is how a part's rival-codec columns race.

func CreateObject(ctx context.Context, b Backend, key string) (ObjectWriter, error)

type ReaderAt ¶ added in v0.37.0

type ReaderAt interface {
	// ReadAt returns the bytes of key in [off, off+n), clamped to the object's end. See [ReadAt] for
	// the contract implementations must honor.
	ReadAt(ctx context.Context, key string, off, n int64) ([]byte, error)
}

ReaderAt is an optional Backend capability: read a byte range of an object instead of the whole thing. It is the read counterpart of ObjectCreator, and it exists for the same reason — a part stores one object per column, so without it touching any block of a column costs the whole column, and part size becomes a bound on process memory rather than on disk.

Both durable backends have it natively (`pread` for file, the `Range` header for s3). Use ReadAt rather than asserting directly: it falls back to reading the whole object and slicing for backends without the capability, so callers stay correct everywhere and a wrapper that forgets to forward costs bytes rather than correctness.

type Sizer ¶ added in v0.12.0

type Sizer interface {
	// Size returns the stored byte size of key, or an [ErrNotExist]-wrapping error if absent.
	Size(ctx context.Context, key string) (int64, error)
}

Sizer is an optional Backend capability: report an object's stored byte size without reading its contents. Backends that can answer cheaply implement it (memory: the in-RAM length; file: os.Stat). Use SizeOf rather than asserting directly — it falls back to a full Read for backends that do not implement Sizer, so callers stay correct everywhere.

type SpaceReporter ¶ added in v0.37.0

type SpaceReporter interface {
	FreeSpace(ctx context.Context) (int64, error)
}

SpaceReporter is the optional capability of reporting how much room a backend has left, so the merge engine can size its output parts against the disk they land on. A backend over a bounded local medium implements it; Memory and object stores do not.

A wrapper around a Backend must forward it, or the capability is silently lost.

type Viewer ¶ added in v0.25.0

type Viewer interface {
	// ReadView returns the value stored under key as a read-only view (do not mutate). Absent keys
	// error like [Backend.Read].
	ReadView(ctx context.Context, key string) ([]byte, error)
}

Viewer is an optional Backend capability: ReadView returns the value stored under key as a **read-only view** that may alias shared state (a cache entry, the in-memory store) instead of Backend.Read's defensive copy. The caller MUST NOT mutate the returned slice; it MAY retain it indefinitely — a stored value is never mutated in place (Write/Delete replace or drop the map entry, they never rewrite the old array), so a view stays valid even after the key is overwritten, evicted, or deleted. It exists for hot read paths (part column objects, read once per query per column) where the copy is a measured allocation cost; use ReadView rather than asserting directly so callers stay correct over any backend.

type ViewerAt ¶ added in v0.37.0

type ViewerAt interface {
	// ReadViewAt returns key's [off, off+n) range as a read-only view, clamped like [ReadAt].
	ReadViewAt(ctx context.Context, key string, off, n int64) ([]byte, error)
}

ViewerAt is ReaderAt's no-copy counterpart, the ranged form of Viewer: it returns the range as a **read-only view** that may alias shared state instead of a caller-owned copy. The same contract applies — never mutate it, and it stays valid indefinitely because a stored value is never mutated in place.

Only a backend already holding the object in memory can offer it (Memory, and the read cache over a resident entry); file and s3 must materialize the bytes to return them, so for those ReadViewAt falls through to ReadAt.