backend

type Backend ¶

type Backend interface {
	// IsEphemeral reports whether the backend stores data only in RAM (dropped on
	// process exit). [Memory] is ephemeral; file and s3 are not.
	IsEphemeral() bool

	// PutIfAbsent stores data under key only if the key does not already exist. It
	// returns true if the write happened, false if the key was already present (no
	// change). Like [Backend.Write] it is atomic per object. It is the compare-and-swap
	// primitive for manifest commits.
	PutIfAbsent(ctx context.Context, key string, data []byte) (bool, error)

	// CompareAndSwap stores data under key only if the key's current version is expected,
	// atomically. Pass [VersionAbsent] to demand that the key not exist yet, so a first commit
	// and every later one are the same call. It returns the version the stored data now has,
	// which the committer holds for its next commit without re-reading the object.
	//
	// A lost race is (false, nil), not an error: nothing failed and nothing is broken, the
	// caller simply holds a stale version and must reload and retry. Reporting it as an error
	// would put it on the path every caller already funnels into "the backend is unhealthy",
	// which is how a contended commit becomes an outage. A returned error means the operation
	// could not be evaluated at all, and says nothing about whether the write landed.
	//
	// Both failing cases are conditional, never destructive: an absent key with a version
	// expected, and a present key with [VersionAbsent] expected, both report (false, nil).
	CompareAndSwap(ctx context.Context, key string, expected Version, data []byte) (Version, bool, error)

	// ReadVersioned returns the value stored under key together with the [Version] identifying
	// it — the token a later [Backend.CompareAndSwap] conditions on. An absent key is
	// ([]byte(nil), [VersionAbsent], nil): absence is a version, and the value a first
	// committer conditions on. Errors are reserved for a backend that could not answer.
	ReadVersioned(ctx context.Context, key string) ([]byte, Version, error)

	// Write stores data under key, overwriting any existing value. The write is
	// atomic per object: a reader never observes a partially written value. The
	// implementation takes ownership semantics by copying data as needed; callers may
	// reuse the buffer after Write returns.
	Write(ctx context.Context, key string, data []byte) error

	// Read returns the value stored under key. It returns an error satisfying
	// errors.Is(err, [ErrNotExist]) if the key is absent. The returned slice is owned
	// by the caller (implementations return a fresh copy, never aliased state).
	Read(ctx context.Context, key string) ([]byte, error)

	// List returns, sorted ascending, every key with the given prefix (empty prefix
	// lists all keys).
	List(ctx context.Context, prefix string) ([]string, error)

	// Delete removes key. It returns an error satisfying errors.Is(err, [ErrNotExist])
	// if the key is absent.
	Delete(ctx context.Context, key string) error
}

Backend is the L1 storage seam (DESIGN.md §3, §5): a common interface over interchangeable implementations — memory (ephemeral, the reference), file, and (later) s3. The same engine code path runs over all three.

Data is addressed by an opaque, slash-delimited string key (e.g. a time-bucketed object path or a file-relative path). Values are whole objects: the part format (`block`) maps one part to a key prefix and one object per column/marks/manifest, so whole-object Read/Write is sufficient and gives per-object write atomicity. All methods are safe for concurrent use.

Ranged reads are not part of this interface, but they are available as the optional ReaderAt capability: the multi-key layout gives projection pushdown (read only the referenced column objects), and ReaderAt gives the pushdown *within* a column that keeps a query touching a few granules from paying for the whole thing.

Backend.PutIfAbsent and Backend.CompareAndSwap are the conditional-write primitives on which atomic manifest / block-list / bucket-index commits build, so multi-writer coordination over one prefix needs no Raft. PutIfAbsent claims an *absent* key (S3 If-None-Match, a filesystem exclusive create, a guarded map insert); CompareAndSwap replaces an *existing* one only if it still holds the version the committer read (S3 If-Match, a digest checked under the store's own lock).

func Cached(inner Backend, maxBytes int64) Backend

func Memory() Backend

type CacheStats ¶ added in v0.6.0

type CacheStats struct {
	Hits, Misses int64
	Bytes        int64 // resident value bytes
	Items        int   // resident objects
}

CacheStats is a snapshot of a cached backend's effectiveness.

type InodeReporter ¶ added in v0.40.0

type InodeReporter interface {
	FreeInodes(ctx context.Context) (int64, error)
}

InodeReporter is the optional capability of reporting how many objects a backend can still create, whatever their size. It is a separate axis from SpaceReporter, not a refinement of it: a part is many small objects, so a filesystem with terabytes free can still fail every create once its inode table is exhausted, and the two failures are identical from a byte-accounting point of view.

A wrapper around a Backend must forward it, or the capability is silently lost.

type NodeLocal ¶ added in v0.39.0

type NodeLocal interface {
	IsNodeLocal() bool
}

NodeLocal is the optional capability of reporting that a backend's objects live on a medium private to the process that writes them, so a peer cannot read them. Memory and a local directory tree implement it; an object store does not.

It reports the medium, not the deployment. A `file` backend rooted on a shared mount (NFS, a clustered filesystem) answers true as well, which is the right answer rather than a false positive: a shared mount is not a supported shared store, because Backend.CompareAndSwap over a directory tree is process-local (see backend/file) and two nodes over one mount would lose index commits to each other in silence. A backend that genuinely is shared — an object store, or a wrapper declaring itself so — answers false, and that answer is exact.

A wrapper around a Backend must forward it, or the capability is silently lost.

type ObjectCreator ¶ added in v0.37.0

type ObjectCreator interface {
	// CreateObject returns a writer building the object stored under key. Nothing is stored until
	// the writer commits.
	CreateObject(ctx context.Context, key string) (ObjectWriter, error)
}

ObjectCreator is an optional Backend capability: build an object incrementally rather than handing it over whole. It exists so a writer producing an object far larger than it wants resident — a merged part's column — can hand finished bytes to the backend as they are produced instead of holding the whole object in RAM.

Use CreateObject rather than asserting directly: it falls back to buffering into a Backend.Write for backends without the capability, so callers stay correct everywhere. Use StreamsWrites to ask whether that fallback would be taken — the caller that *sizes* its output against memory needs to know, since the fallback puts the object back in RAM.

type ObjectWriter ¶ added in v0.37.0

type ObjectWriter interface {
	io.Writer

	// Commit publishes everything written so far under the writer's key.
	Commit(ctx context.Context) error

	// Abort discards the writer's bytes and releases whatever it holds. It is idempotent and a
	// no-op once Commit has succeeded.
	Abort()
}

ObjectWriter builds one object incrementally. Bytes appended with Write are not visible under the object's key until ObjectWriter.Commit; until then the key reads as it did before (absent, or the previous value), so the "manifest written last" commit rule still holds.

Commit publishes the bytes atomically, exactly like Backend.Write. Abort discards them and is safe to call after Commit (where it does nothing), so `defer w.Abort()` is the correct cleanup. A writer must not be used from more than one goroutine, but several writers may target the same key concurrently — only the one that commits wins, which is how a part's rival-codec columns race.

func CreateObject(ctx context.Context, b Backend, key string) (ObjectWriter, error)

type ReaderAt ¶ added in v0.37.0

type ReaderAt interface {
	// ReadAt returns the bytes of key in [off, off+n), clamped to the object's end. See [ReadAt] for
	// the contract implementations must honor.
	ReadAt(ctx context.Context, key string, off, n int64) ([]byte, error)
}

ReaderAt is an optional Backend capability: read a byte range of an object instead of the whole thing. It is the read counterpart of ObjectCreator, and it exists for the same reason — a part stores one object per column, so without it touching any block of a column costs the whole column, and part size becomes a bound on process memory rather than on disk.

Both durable backends have it natively (`pread` for file, the `Range` header for s3). Use ReadAt rather than asserting directly: it falls back to reading the whole object and slicing for backends without the capability, so callers stay correct everywhere and a wrapper that forgets to forward costs bytes rather than correctness.

type Sizer ¶ added in v0.12.0

type Sizer interface {
	// Size returns the stored byte size of key, or an [ErrNotExist]-wrapping error if absent.
	Size(ctx context.Context, key string) (int64, error)
}

Sizer is an optional Backend capability: report an object's stored byte size without reading its contents. Backends that can answer cheaply implement it (memory: the in-RAM length; file: os.Stat). Use SizeOf rather than asserting directly — it falls back to a full Read for backends that do not implement Sizer, so callers stay correct everywhere.

type SpaceReporter ¶ added in v0.37.0

type SpaceReporter interface {
	FreeSpace(ctx context.Context) (int64, error)
}

SpaceReporter is the optional capability of reporting how much room a backend has left, so the merge engine can size its output parts against the disk they land on. A backend over a bounded local medium implements it; Memory and object stores do not.

A wrapper around a Backend must forward it, or the capability is silently lost.

type Version ¶ added in v0.40.0

type Version string

Version is an opaque token identifying the exact bytes stored under a key. It is produced by the backend (Backend.ReadVersioned, Backend.CompareAndSwap) and consumed by it; its form is implementation-defined (an object store's ETag, a content digest) and no other meaning may be read into it — two versions are only ever compared for equality, never ordered.

A version identifies *contents*, not a write: rewriting a key with the bytes it already holds may leave the version unchanged. That is what makes the token safe against the ABA problem here, rather than exposed to it — the state a committer conditions on is the object itself, so a value that came back is the value it read.

const VersionAbsent Version = ""

func ContentVersion(data []byte) Version

type Viewer ¶ added in v0.25.0

type Viewer interface {
	// ReadView returns the value stored under key as a read-only view (do not mutate). Absent keys
	// error like [Backend.Read].
	ReadView(ctx context.Context, key string) ([]byte, error)
}

Viewer is an optional Backend capability: ReadView returns the value stored under key as a **read-only view** that may alias shared state (a cache entry, the in-memory store) instead of Backend.Read's defensive copy. The caller MUST NOT mutate the returned slice; it MAY retain it indefinitely — a stored value is never mutated in place (Write/Delete replace or drop the map entry, they never rewrite the old array), so a view stays valid even after the key is overwritten, evicted, or deleted. It exists for hot read paths (part column objects, read once per query per column) where the copy is a measured allocation cost; use ReadView rather than asserting directly so callers stay correct over any backend.

type ViewerAt ¶ added in v0.37.0

type ViewerAt interface {
	// ReadViewAt returns key's [off, off+n) range as a read-only view, clamped like [ReadAt].
	ReadViewAt(ctx context.Context, key string, off, n int64) ([]byte, error)
}

ViewerAt is ReaderAt's no-copy counterpart, the ranged form of Viewer: it returns the range as a **read-only view** that may alias shared state instead of a caller-owned copy. The same contract applies — never mutate it, and it stays valid indefinitely because a stored value is never mutated in place.

Only a backend already holding the object in memory can offer it (Memory, and the read cache over a resident entry); file and s3 must materialize the bytes to return them, so for those ReadViewAt falls through to ReadAt.