fsstore

package module
v0.5.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 30, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

README

fsstore

fsstore implements storage's five storage primitives — Ledger, Leaser, KV, Blobs, and OrderedIndex — over the local filesystem, giving a durable, single-host backend for session and workspace persistence. The concrete on-disk formats live here, behind storage's neutral contracts: the ledger is an append-only frame log (length-prefixed, CRC-32C-checked frames) with automatic torn-tail recovery on reopen; leases are flock-fenced lock files carrying a durable, strictly-increasing epoch counter; KV is one revision-CAS'd file per key (atomic temp-file + rename writes); Blobs are content-addressed immutable byte objects; and OrderedIndex uses one checksummed append-only mutation log plus warm in-memory order, rank, and due views per namespace. It depends only on the Go standard library and github.com/looprig/storage.

Ledger and KV assume single-writer-per-name: their reads take no advisory lock and trust that no one rewrites committed bytes concurrently. Enforcing that discipline is the caller's responsibility — acquire the name's lease via the Leaser before writing those entries. OrderedIndex instead serializes each operation with both an in-process namespace mutex and a cross-process advisory lock because a read may recover a torn tail. Path containment is lexical (validated names, filepath.Clean, a root-prefix check); it does not resolve symlinks below the store root, so owning that directory is the deployment's responsibility.

Usage

Open wires the five backends under one root directory and returns them bundled as a complete *storage.Composite (the primitives collide on method names, so they are composed as a field-bundle, not a single all-five type — reach each as store.Ledger, store.Leaser, store.KV, store.Blobs, store.OrderedIndex). Hand the bundle to a consumer such as sessionstore.Open:

store, err := fsstore.Open(fsstore.Options{Root: "/var/lib/looprig"})
if err != nil {
    return err
}
defer store.Close() // releases in-process state; idempotent

sess, err := sessionstore.Open(store.Backend()) // or store.Composite

Root is required (an empty Root is rejected with an *OptionsError) and is created at 0700 if absent. Its layout is streams/, leases/, kv/, blobs/, and ordered/. Close releases the ledger cache and OrderedIndex namespace views and is idempotent; after Close the Store must not be reused.

OrderedIndex warm pages use their maintained views without walking the filesystem or replaying already-applied frames. If another handle appends to a namespace, the next operation reads and applies only the new complete tail. Recovery truncates an all-zero unwritten tail, including the post-crash case where the filesystem persisted the journal size before its data blocks. A zero region that is not at the tail fails closed with *OrderedLogCorruptError; no acknowledged record is discarded.

An OrderedIndex operation that returns cancellation before filesystem setup leaves no path. If it returns cancellation after a fresh namespace directory or log is created but before advisory-lock acquisition, it may leave that inert empty artifact, but never a committed or partial frame and never a truncation. Once the exclusive advisory lock is held, recovery or mutation completes without abandoning durability-critical work midway.

Blob reader close behavior

The planned v0.5.1 release makes each successful Blobs.Get return a concrete, independent close-aware reader. Read and Close are race-safe; the first Close publishes closed state before closing its file, repeated Close calls return the same result, and every later Read returns zero bytes with fs.ErrClosed rather than ordinary EOF. Closing one reader never changes stored bytes or another reader.

fsstore intentionally does not implement Storage's optional bounded Blob reader lifecycle capability. Portable ordinary-file I/O generally has no deadline support, so a slow kernel, network-filesystem, or FUSE Read cannot be force-cancelled within a finite bound. Close may therefore wait for an active Read to return; callers requiring strictly bounded reader shutdown must select a provider that advertises that capability.

Documentation

Index

Constants

View Source
const (

	// MaxFramePayload caps a single frame's payload at 16 MiB. storage only
	// requires backends to accept 1 MiB payloads; the 16 MiB ceiling is
	// deliberate headroom. It also bounds the allocation a decoder performs for
	// any declared length: a larger declared length is rejected as corruption
	// before any buffer is sized to it, so a bogus header cannot force a huge
	// allocation.
	MaxFramePayload = 16 << 20
)

Variables

This section is empty.

Functions

This section is empty.

Types

type BlobIOError

type BlobIOError struct {
	Op    string
	Path  string
	Cause error
}

BlobIOError wraps an underlying filesystem failure (mkdir, open, read, write, fsync, rename, remove, walk) with the operation and path for diagnosis. Cause is the os/syscall error and is exposed via Unwrap.

func (*BlobIOError) Error

func (e *BlobIOError) Error() string

func (*BlobIOError) Unwrap

func (e *BlobIOError) Unwrap() error

Unwrap exposes the underlying filesystem error.

type BlobPathError

type BlobPathError struct {
	Key  string
	Path string
}

BlobPathError reports that a blob key mapped to a filesystem location outside the store's blobs directory. It is defense in depth: ValidateName already forbids the '..' and empty segments that could escape, so a valid key never triggers it — a triggered BlobPathError means an unvalidated key reached path mapping.

func (*BlobPathError) Error

func (e *BlobPathError) Error() string

type BlobRootError

type BlobRootError struct {
	Root   string
	Reason string
}

BlobRootError reports that newBlobStore was given an unusable root (for example, an empty path).

func (*BlobRootError) Error

func (e *BlobRootError) Error() string

type FlockError

type FlockError struct {
	Op    string
	Path  string
	Cause error
}

FlockError reports that acquiring or releasing a cross-process advisory lock failed. Op is "lock" or "unlock"; Cause carries the underlying syscall error (or errFlockUnsupported on a platform without advisory locking). Callers classify with errors.As(&FlockError), never by string.

func (*FlockError) Error

func (e *FlockError) Error() string

func (*FlockError) Unwrap

func (e *FlockError) Unwrap() error

Unwrap exposes the underlying syscall error (or errFlockUnsupported).

type FrameError

type FrameError struct {
	Fault FrameFault
	// Have is the number of bytes available (Torn faults).
	Have int
	// Need is the number of bytes the frame required (Torn faults).
	Need int
	// Length is the declared or requested payload length (Oversize fault).
	Length uint64
}

FrameError is the single typed error the frame codec returns. Its Fault names the exact cause; IsTorn / IsCorrupt classify it into the two recovery modes the ledger acts on:

  • Torn — a clean truncation at a byte boundary (an interrupted final write). Recoverable: the ledger truncates back to the last good frame.
  • Corrupt — every declared byte is present but the frame is internally inconsistent (bad CRC, or a length beyond the ceiling). Not recoverable by truncation; the ledger must refuse to open and fail closed.

Callers classify with errors.As(&FrameError) and the predicates, never by string. The Error() text carries only integers, never payload bytes.

func (*FrameError) Error

func (e *FrameError) Error() string

func (*FrameError) IsCorrupt

func (e *FrameError) IsCorrupt() bool

IsCorrupt reports a present-but-invalid frame: a CRC mismatch or a length beyond MaxFramePayload. The ledger cannot recover by truncation and fails closed.

func (*FrameError) IsTorn

func (e *FrameError) IsTorn() bool

IsTorn reports a clean truncation: the header or payload is incomplete. The ledger recovers by truncating the file to the previous whole frame.

type FrameFault

type FrameFault uint8

FrameFault identifies the precise reason a frame failed to encode or decode. It is the machine-classifiable cause; callers fold faults into the two recovery modes with FrameError.IsTorn / FrameError.IsCorrupt.

const (
	// FaultShortHeader: fewer than frameHeaderSize bytes are present, so the
	// header itself is incomplete. Torn.
	FaultShortHeader FrameFault = iota + 1
	// FaultShortPayload: the header is complete and declares a payload length,
	// but fewer than that many payload bytes are present. Torn.
	FaultShortPayload
	// FaultOversize: a declared or requested payload length exceeds
	// MaxFramePayload. Corrupt — rejected before any buffer is sized to it.
	FaultOversize
	// FaultCRCMismatch: every declared byte is present but the payload CRC does
	// not match the header. Corrupt.
	FaultCRCMismatch
)

type KVCorruptError

type KVCorruptError struct {
	Path  string
	Cause error
}

KVCorruptError reports that a key's file could not be trusted: it is present but too short to hold the revision header. Both Get and Put fail closed rather than treat the file as absent (rev 0), which would let a torn file silently reset the revision counter. Cause is errShortRevHeader.

func (*KVCorruptError) Error

func (e *KVCorruptError) Error() string

func (*KVCorruptError) Unwrap

func (e *KVCorruptError) Unwrap() error

Unwrap exposes the underlying cause (errShortRevHeader).

type KVIOError

type KVIOError struct {
	Op    string
	Path  string
	Cause error
}

KVIOError wraps an underlying filesystem failure (mkdir, open, read, write, fsync, rename, remove, walk) with the operation and path for diagnosis. Cause is the os/syscall error and is exposed via Unwrap.

func (*KVIOError) Error

func (e *KVIOError) Error() string

func (*KVIOError) Unwrap

func (e *KVIOError) Unwrap() error

Unwrap exposes the underlying filesystem error.

type KVPathError

type KVPathError struct {
	Key  string
	Path string
}

KVPathError reports that a KV key mapped to a filesystem location outside the store's kv directory. It is defense in depth: ValidateName already forbids the '..' and empty segments that could escape, so a valid key never triggers it — a triggered KVPathError means an unvalidated key reached path mapping.

func (*KVPathError) Error

func (e *KVPathError) Error() string

type KVRootError

type KVRootError struct {
	Root   string
	Reason string
}

KVRootError reports that newKVStore was given an unusable root (for example, an empty path).

func (*KVRootError) Error

func (e *KVRootError) Error() string

type LeaseCorruptError

type LeaseCorruptError struct {
	Path  string
	Cause error
}

LeaseCorruptError reports that a lock file's persisted epoch counter could not be trusted: it is present but not a valid decimal uint64. Acquire fails closed rather than reset the counter to 0 (which would let a stale holder's epoch be reissued, breaking the strictly-increasing guarantee). Cause is errBadEpoch.

func (*LeaseCorruptError) Error

func (e *LeaseCorruptError) Error() string

func (*LeaseCorruptError) Unwrap

func (e *LeaseCorruptError) Unwrap() error

Unwrap exposes the underlying cause (errBadEpoch).

type LeaseIOError

type LeaseIOError struct {
	Op    string
	Path  string
	Cause error
}

LeaseIOError wraps an underlying filesystem failure (mkdir, open, read, write, truncate, fsync, close) with the operation and path for diagnosis. Cause is the os/syscall error and is exposed via Unwrap.

func (*LeaseIOError) Error

func (e *LeaseIOError) Error() string

func (*LeaseIOError) Unwrap

func (e *LeaseIOError) Unwrap() error

Unwrap exposes the underlying filesystem error.

type LeasePathError

type LeasePathError struct {
	Name string
	Path string
}

LeasePathError reports that a lease Name mapped to a filesystem location outside the store's leases directory. It is defense in depth: ValidateName already forbids the '..' and empty segments that could escape, so a valid name never triggers it — a triggered LeasePathError means an unvalidated name reached path mapping.

func (*LeasePathError) Error

func (e *LeasePathError) Error() string

type LeaseRootError

type LeaseRootError struct {
	Root   string
	Reason string
}

LeaseRootError reports that newLeaserStore was given an unusable root (for example, an empty path).

func (*LeaseRootError) Error

func (e *LeaseRootError) Error() string

type LedgerCorruptError

type LedgerCorruptError struct {
	Path  string
	Seq   uint64
	Cause error
}

LedgerCorruptError reports that a ledger file could not be trusted: a frame at Seq is present-but-invalid (CRC mismatch, over-ceiling length) or the sequence is non-contiguous. Unlike a torn tail, this is not recoverable by truncation, so every operation fails closed rather than silently drop data. Cause carries the underlying frame fault or errNonContiguousSeq.

func (*LedgerCorruptError) Error

func (e *LedgerCorruptError) Error() string

func (*LedgerCorruptError) Unwrap

func (e *LedgerCorruptError) Unwrap() error

Unwrap exposes the underlying frame fault (a *FrameError) or errNonContiguousSeq.

type LedgerIOError

type LedgerIOError struct {
	Op    string
	Path  string
	Cause error
}

LedgerIOError wraps an underlying filesystem failure (open, read, write, fsync, truncate, mkdir, stat) with the operation and path for diagnosis. Cause is the os/syscall error and is exposed via Unwrap.

func (*LedgerIOError) Error

func (e *LedgerIOError) Error() string

func (*LedgerIOError) Unwrap

func (e *LedgerIOError) Unwrap() error

Unwrap exposes the underlying filesystem error.

type LedgerPathError

type LedgerPathError struct {
	Name string
	Path string
}

LedgerPathError reports that a ledger Name mapped to a filesystem location outside the store root. It is defense in depth: ValidateName already forbids the '..' and empty segments that could escape, so a valid name never triggers it — a triggered LedgerPathError means an unvalidated name reached path mapping.

func (*LedgerPathError) Error

func (e *LedgerPathError) Error() string

type LedgerRootError

type LedgerRootError struct {
	Root   string
	Reason string
}

LedgerRootError reports that newLedgerStore was given an unusable root (for example, an empty path).

func (*LedgerRootError) Error

func (e *LedgerRootError) Error() string

type Options

type Options struct {
	// Root is the store root directory. It is created at 0700 if absent. Required:
	// an empty Root is rejected with an *OptionsError.
	Root string
}

Options configures Open. Root is the single directory under which the store lays out its five backends (streams/, leases/, kv/, blobs/, ordered/); it is required.

type OptionsError

type OptionsError struct {
	Field  string
	Reason string
	Cause  error
}

OptionsError reports an invalid or unusable Open option. Field names the option (currently always "Root") and Reason explains the fault; Cause carries the underlying filesystem error (or errRootNotDir) when the fault was not a plain validation failure, and is exposed via Unwrap for errors.Is/As.

func (*OptionsError) Error

func (e *OptionsError) Error() string

func (*OptionsError) Unwrap

func (e *OptionsError) Unwrap() error

Unwrap exposes the underlying cause (a filesystem error or errRootNotDir), or nil for a pure validation failure such as an empty Root.

type OrderedCodecError added in v0.5.0

type OrderedCodecError struct {
	Fault OrderedCodecFault
	// Field names the implicated field ("stable_key", "value", ...) when the
	// fault is attributable to one.
	Field string
	// Have and Need are byte counts for a short payload.
	Have int
	Need int
	// Length is the declared or supplied field length for an oversize fault.
	Length uint64
	// Max is the ceiling Length exceeded.
	Max uint64
	// Cause is the underlying storage validation error, when there is one.
	Cause error
}

OrderedCodecError is the single typed error the ordered mutation codec returns. Fault names the exact cause and Cause, when set, carries the underlying storage validation error (for example *storage.InvalidNameError or *storage.InvalidStableKeyError) so callers can classify with errors.As at either level. Field names the offending field where one is implicated.

The Error() text carries only field names and integers, never payload bytes: a StableKey may be arbitrary user data and must not be echoed into logs.

Classification precedence

This one type reports two different things, and the wrapper is what tells them apart. An append REJECTING a caller's mutation returns a BARE *OrderedCodecError; a replay finding the same fault ON DISK returns it wrapped in an *OrderedLogCorruptError. errors.As(&OrderedCodecError) therefore matches both, and a caller that must distinguish invalid input from a damaged log has to test *OrderedLogCorruptError FIRST.

A public API whose contract names a specific error for bad input — such as OrderedIndex.CreateOrdered, which owes *storage.InvalidNameError or *storage.InvalidStableKeyError for a malformed OrderedID — must not rely on this precedence at all. It should validate the caller's ID up front with storage.ValidateOrderedID and return THAT error unwrapped, leaving the codec's validation as the internal backstop it is.

func (*OrderedCodecError) Error added in v0.5.0

func (e *OrderedCodecError) Error() string

func (*OrderedCodecError) Unwrap added in v0.5.0

func (e *OrderedCodecError) Unwrap() error

Unwrap exposes the underlying storage validation error, or nil when the fault is purely structural.

type OrderedCodecFault added in v0.5.0

type OrderedCodecFault uint8

OrderedCodecFault identifies the precise reason an ordered mutation payload failed to encode or decode. Every fault is fail-closed: unlike a frame fault, none of them is recoverable by truncating the log, because the outer frame checksum has already established that the bytes are complete and unaltered.

const (
	// OrderedFaultShortPayload: the payload is shorter than the fixed header, or
	// shorter than the field widths its own header declares.
	OrderedFaultShortPayload OrderedCodecFault = iota + 1
	// OrderedFaultUnknownVersion: the version byte is not orderedMutationVersion.
	OrderedFaultUnknownVersion
	// OrderedFaultUnknownOp: the op byte is not one of the defined orderedOp
	// values.
	OrderedFaultUnknownOp
	// OrderedFaultFieldOversize: a declared (or supplied) field length exceeds
	// its contractual ceiling. Checked before any buffer is sized to it.
	OrderedFaultFieldOversize
	// OrderedFaultTrailingBytes: the payload contains bytes beyond the fields its
	// header declares, so the encoding is not canonical.
	OrderedFaultTrailingBytes
	// OrderedFaultInvalidIdentity: the ordering scope or stable key violates the
	// OrderedID rules. Cause carries the storage validation error.
	OrderedFaultInvalidIdentity
	// OrderedFaultInvalidRecord: the mutation's non-identity state is illegal —
	// an invalid ranking scope or due state, a zero revision or order, a
	// non-boolean rank flag, or a tombstone that is still ranked or due.
	OrderedFaultInvalidRecord
)

type OrderedLogClosedError added in v0.5.0

type OrderedLogClosedError struct {
	Root      string
	Namespace string
}

OrderedLogClosedError reports an operation on a closed ordered log store. Close drops the namespace->orderedLog registry, so a later logFor would mint a FRESH orderedLog with a NEW mutex for a namespace another goroutine may still be appending to under the old one — two mutexes guarding one file, surviving only because the advisory lock happens to serialize the descriptors. Failing closed is the same discipline Store.Close already applies.

func (*OrderedLogClosedError) Error added in v0.5.0

func (e *OrderedLogClosedError) Error() string

type OrderedLogCorruptError added in v0.5.0

type OrderedLogCorruptError struct {
	Path  string
	Seq   uint64
	Cause error
}

OrderedLogCorruptError reports that a namespace's mutation log could not be trusted: the frame at Seq is present-but-invalid (a checksum mismatch or an over-ceiling length), decodes to an illegal mutation, or breaks the sequence. Unlike a torn tail this is not recoverable by truncation, so every operation fails closed. Cause carries the underlying *FrameError, *OrderedCodecError, or errOrderedNonContiguousSeq.

PRECEDENCE: because a codec fault read off the disk is wrapped in this type, errors.As finds BOTH *OrderedLogCorruptError and *OrderedCodecError on one disk-corruption error, while a caller's invalid mutation carries only the codec type. A caller separating "your input was bad" from "the log is damaged" must test *OrderedLogCorruptError FIRST.

func (*OrderedLogCorruptError) Error added in v0.5.0

func (e *OrderedLogCorruptError) Error() string

func (*OrderedLogCorruptError) Unwrap added in v0.5.0

func (e *OrderedLogCorruptError) Unwrap() error

Unwrap exposes the underlying frame fault, codec fault, or errOrderedNonContiguousSeq.

type OrderedLogIOError added in v0.5.0

type OrderedLogIOError struct {
	Op    string
	Path  string
	Cause error
}

OrderedLogIOError wraps an underlying filesystem failure (open, read, write, fsync, truncate, mkdir, stat, seek) with the operation and path for diagnosis. Cause is the os/syscall error and is exposed via Unwrap.

func (*OrderedLogIOError) Error added in v0.5.0

func (e *OrderedLogIOError) Error() string

func (*OrderedLogIOError) Unwrap added in v0.5.0

func (e *OrderedLogIOError) Unwrap() error

Unwrap exposes the underlying filesystem error.

type OrderedLogPathError added in v0.5.0

type OrderedLogPathError struct {
	Namespace string
	Path      string
}

OrderedLogPathError reports that a namespace mapped to a filesystem location outside the ordered log directory. It is defense in depth: ValidateName already forbids the '..' and empty segments that could escape, so a valid namespace never triggers it — a triggered OrderedLogPathError means an unvalidated name reached path mapping.

func (*OrderedLogPathError) Error added in v0.5.0

func (e *OrderedLogPathError) Error() string

type OrderedLogRootError added in v0.5.0

type OrderedLogRootError struct {
	Root   string
	Reason string
}

OrderedLogRootError reports that newOrderedLogStore was given an unusable root (for example, an empty or blank path).

func (*OrderedLogRootError) Error added in v0.5.0

func (e *OrderedLogRootError) Error() string

type Store

type Store struct {
	*storage.Composite
	// contains filtered or unexported fields
}

Store is an open fsstore rooted at a single directory on the local filesystem. It embeds *storage.Composite, so a caller reaches each primitive as a promoted field — store.Ledger, store.Leaser, store.KV, store.Blobs, store.OrderedIndex — and hands the whole bundle to a consumer as store.Composite or via Backend. The five primitives collide on method names, so no single type can implement all five (see the file-level comment); embedding the field-bundle is how Store sidesteps that.

func Open

func Open(opts Options) (*Store, error)

Open assembles a filesystem-backed storage bundle under opts.Root. It requires a non-empty Root (empty -> *OptionsError), creates it at 0700, and verifies it resolves to a directory. It then wires the five backends under that root — <root>/streams, <root>/leases, <root>/kv, <root>/blobs, <root>/ordered, each created at 0700 by its constructor — and bundles them with storage.NewCompositeWithOrderedIndex.

On any backend-construction failure Open returns that backend's typed error and no Store; because every fs backend only creates directories (none holds an open file descriptor until first use), a partial failure leaks no handles and needs no unwind.

func (*Store) Backend

func (s *Store) Backend() *storage.Composite

Backend returns the assembled five-primitive bundle to hand to a consumer such as sessionstore.Open. It is the embedded *storage.Composite; callers may read store.Composite directly instead.

func (*Store) Close

func (s *Store) Close() error

Close releases the store's in-process state. Concretely it drops the ledger backend's cached name->file registry and the ordered index's namespace views. The fs backends hold no long-lived file descriptors of their own — the ledger opens and closes its fd within each Append/Delete (the per-append advisory-lock model), the KV and Blobs writers do the same, and a Leaser grant's fd is owned and released by the caller via Lease.Release — so there are no open handles for Close to reclaim beyond that in-memory registry.

Close is idempotent: a second call is a no-op returning nil. After Close the Store must not be reused; open a fresh Store on the same root to use it again.

func (*Store) StoragePaths

func (s *Store) StoragePaths() []string

StoragePaths reports the canonical local root that contains every primitive. The result is newly allocated so callers cannot mutate subsequent reports.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL