filestore

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 28, 2026 License: AGPL-3.0 Imports: 25 Imported by: 0

Documentation

Overview

Package filestore is the single-node durable store: the in-memory store (store/memory) is the engine, and every committed change is appended to an fsync'd JSON Lines journal before it is applied. A compaction folds the journal into a snapshot; Open loads the snapshot, replays the journal and restores the version counter. Transcripts are files.

<dir>/                        0700
  snapshot.json               {"format":1,"storeVersion":"N","objects":[…]} (atomic rename)
  journal.jsonl               one line per change above the snapshot, fsync'd
  transcripts/<run>.jsonl.gz  one file per run
  LOCK                        flock: one store per dir

Design: design/workstreams/dispatcher-runtime/LLD/store.md §File store.

Index

Constants

View Source
const (
	DefaultCompactBytes = 64 << 20
	DefaultCompactLines = 50_000
)

Compaction defaults: the journal is folded into the snapshot once it passes either.

View Source
const Format = journal.Format

Format is the snapshot format this package reads and writes.

Variables

View Source
var (
	// ErrLocked: another store has the state dir open.
	ErrLocked = errors.New("state dir is locked by another store")
	// ErrInsecure: the state dir or a file in it is readable or writable by group or others,
	// or owned by another user.
	ErrInsecure = errors.New("state dir is not private")
	// ErrCorrupt: the snapshot or a journal line (other than a torn last one) can't be loaded.
	ErrCorrupt = errors.New("state is corrupt")
	// ErrFormat: the snapshot has a format this package doesn't read.
	ErrFormat = errors.New("unknown state format")
	// ErrClosed: a write after Close.
	ErrClosed = errors.New("store is closed")
	// ErrObjectStoreDir: the dir holds an object store's log (log/ or snapshots/: the file://
	// bucket of an object store, P3), which the file store must never share a dir with.
	ErrObjectStoreDir = errors.New("dir holds an object store's state")
	// ErrJournal: an earlier journal fsync (or rollback) failed, so the journal can't be
	// trusted; every write fails until the store is reopened.
	ErrJournal = errors.New("journal unusable after an earlier failure; reopen the store")
)

Errors Open and the writes return (test with errors.Is).

Functions

func LockPath

func LockPath(dir string) string

LockPath returns the lock file of the state dir dir: a store holds an exclusive flock on it while it has the dir open.

func MoveAside

func MoveAside(dir, dst string) error

MoveAside renames the state dir dir to dst (which must not exist), for a clean start that keeps the old state as evidence. It holds the dir's lock while it renames, so it fails with ErrLocked while a store has the dir open, and no store opens the dir midway.

Types

type JournalEntry

type JournalEntry struct {
	Version uint64
	At      time.Time
	Op      memory.Op
	Kind    string
	Name    string
	// Obj is the object as stored by the change (nil for delete).
	Obj json.RawMessage
}

JournalEntry is one journal line: one committed change, in version order.

type LoadInfo

type LoadInfo struct {
	// Version is the store version restored (0 for a new dir).
	Version uint64
	// Objects counts the objects loaded, by kind.
	Objects map[string]int
	// Snapshot is whether there was a snapshot; JournalLines counts the journal lines
	// replayed above it.
	Snapshot     bool
	JournalLines int
	// TornLine is whether a torn last journal line (a crash mid-append) was dropped.
	TornLine bool
	// Transcripts counts the transcript files found.
	Transcripts int
}

LoadInfo is what Open loaded from the state dir, for the caller's start-up log.

type Option

type Option func(*config)

Option configures Open.

func WithCompaction

func WithCompaction(maxBytes int64, maxLines int) Option

WithCompaction sets the journal size, in bytes, and length, in lines, past which a write compacts it into the snapshot (defaults DefaultCompactBytes and DefaultCompactLines).

func WithKind

func WithKind(kind string, proto store.Object) Option

WithKind registers the Go type of kind, a stored kind outside v1alpha1 (the internal Lease, Session, Delivery and the like): proto is a pointer to its zero value. Open needs the type to load the kind's objects, and writes of a kind without one are ErrInvalid, so a kind that could not be loaded back is never stored. The v1alpha1 runtime kinds are built in.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is the file-backed store. Use Open, and Close when done.

func Open

func Open(dir string, opts store.Options, options ...Option) (*Store, error)

Open opens (creating it if missing) the state dir and loads it: the snapshot, then the journal lines above the snapshot's version. A torn last journal line (a crash mid-append) is dropped with a warning and the journal truncated to the last good line; anything else that can't be loaded fails Open with ErrCorrupt and where. The dir must be private (0700, files 0600, owned by this user: ErrInsecure says how to fix it) and not open in another store (ErrLocked). opts are the memory store's.

func (*Store) Close

func (s *Store) Close() error

Close compacts the journal into the snapshot and releases the state dir. Writes after Close fail with ErrClosed; reads still answer from memory. It is idempotent.

func (*Store) Create

func (s *Store) Create(ctx context.Context, obj store.Object) error

Create implements store.Store. The change is journaled before it is applied.

func (*Store) Delete

func (s *Store) Delete(ctx context.Context, k store.Key) error

Delete implements store.Store. The change is journaled before it is applied; a Run's transcript goes with it.

func (*Store) DeleteIfUnchanged added in v0.2.0

func (s *Store) DeleteIfUnchanged(ctx context.Context, k store.Key, resourceVersion string) error

DeleteIfUnchanged implements store.UnchangedDeleter: through the journal, as Delete.

func (*Store) Get

func (s *Store) Get(ctx context.Context, k store.Key, out store.Object) error

Get implements store.Store.

func (*Store) GetTranscript

func (s *Store) GetTranscript(ctx context.Context, run string) ([]byte, error)

GetTranscript implements store.TranscriptStore.

func (*Store) List

func (s *Store) List(ctx context.Context, kind string, opts store.ListOptions, out any) (store.ListMeta, error)

List implements store.Store.

func (*Store) Loaded

func (s *Store) Loaded() LoadInfo

Loaded returns what Open loaded.

func (*Store) Lock

func (s *Store) Lock(ctx context.Context, name string, ttl time.Duration) (store.Unlock, error)

Lock implements store.Store. Locks are not persisted: they are advisory and lease-based, and a restart frees them.

func (*Store) PutTranscript

func (s *Store) PutTranscript(ctx context.Context, run string, data []byte) error

PutTranscript implements store.TranscriptStore: the blob is written to a tmp file, fsync'd and renamed into transcripts/. The caps are the memory store's: over the per-transcript cap is ErrTooLarge, and the oldest transcripts are evicted (logged) to stay under the total.

func (*Store) Sweep

func (s *Store) Sweep(ctx context.Context) (store.SweepResult, error)

Sweep implements store.Sweeper. Its deletes are journaled like any other.

func (*Store) Update

func (s *Store) Update(ctx context.Context, obj store.Object) error

Update implements store.Store. The change is journaled before it is applied.

func (*Store) Watch

func (s *Store) Watch(ctx context.Context, kind string, opts store.WatchOptions) (store.Watcher, error)

Watch implements store.Store. After a reopen, watches start from the restored version: the history isn't persisted, so an older SinceVersion is ErrWatchExpired.

type View

type View struct {
	// contains filtered or unexported fields
}

View is a state dir loaded read-only, by Read: a memory store holding what the snapshot and the journal held when they were read, the journal lines above the snapshot, and the transcripts, read from their files on demand. Nothing in the dir is written, locked or removed, so a View can be taken while a store (a running serve) has the dir open, and of a copy of one.

func Read

func Read(dir string, opts store.Options, options ...Option) (*View, error)

Read loads the state dir read-only: the snapshot, then the journal lines above its version. It writes nothing and takes no lock, so it works while a store has the dir open and on a copied dir:

  • a journal line being appended (or torn by a crash) is the last one and has no trailing newline: it is left out, and the file left as it is;
  • a compaction that replaces the snapshot while Read reads makes it start over (the snapshot it holds open is checked against the file's after the journal is read), so the snapshot and the journal it loads always fit together.

A dir that isn't private is read, with a warning (Warnings): only a store that writes refuses it. A missing dir is an error wrapping os.ErrNotExist; an unknown format is ErrFormat and corruption ErrCorrupt, as for Open. options register the internal kinds (WithKind), as for Open; WithCompaction is ignored.

func (*View) Dir

func (v *View) Dir() string

Dir returns the state dir read.

func (*View) Get

func (v *View) Get(ctx context.Context, k store.Key, out store.Object) error

Get reads one object, as store.Store's Get.

func (*View) Journal

func (v *View) Journal() []JournalEntry

Journal returns the journal lines above the snapshot, in version order. Changes folded into the snapshot by a compaction are no longer there.

func (*View) List

func (v *View) List(ctx context.Context, kind string, opts store.ListOptions, out any) (store.ListMeta, error)

List lists a kind, as store.Store's List.

func (*View) Loaded

func (v *View) Loaded() LoadInfo

Loaded returns what was read: the version, the objects per kind, whether there was a snapshot, the journal lines above it, whether a torn last line was left out and the transcripts.

func (*View) Transcript

func (v *View) Transcript(run string) ([]byte, error)

Transcript reads run's transcript file (gzip'd JSON Lines, as the agent uploaded it): store.ErrNotFound when there is none.

func (*View) Transcripts

func (v *View) Transcripts() []string

Transcripts returns the runs with a transcript, sorted.

func (*View) Warnings

func (v *View) Warnings() []string

Warnings returns what Read noticed but read anyway: a dir that isn't private, a torn last journal line.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL