Documentation
¶
Overview ¶
Package filestore is the single-node durable store: the in-memory store (store/memory) is the engine, and every committed change is appended to an fsync'd JSON Lines journal before it is applied. A compaction folds the journal into a snapshot; Open loads the snapshot, replays the journal and restores the version counter. Transcripts are files.
<dir>/ 0700
snapshot.json {"format":1,"storeVersion":"N","objects":[…]} (atomic rename)
journal.jsonl one line per change above the snapshot, fsync'd
transcripts/<run>.jsonl.gz one file per run
LOCK flock: one store per dir
Design: design/workstreams/dispatcher-runtime/LLD/store.md §File store.
Index ¶
- Constants
- Variables
- func LockPath(dir string) string
- func MoveAside(dir, dst string) error
- type JournalEntry
- type LoadInfo
- type Option
- type Store
- func (s *Store) Close() error
- func (s *Store) Create(ctx context.Context, obj store.Object) error
- func (s *Store) Delete(ctx context.Context, k store.Key) error
- func (s *Store) Get(ctx context.Context, k store.Key, out store.Object) error
- func (s *Store) GetTranscript(ctx context.Context, run string) ([]byte, error)
- func (s *Store) List(ctx context.Context, kind string, opts store.ListOptions, out any) (store.ListMeta, error)
- func (s *Store) Loaded() LoadInfo
- func (s *Store) Lock(ctx context.Context, name string, ttl time.Duration) (store.Unlock, error)
- func (s *Store) PutTranscript(ctx context.Context, run string, data []byte) error
- func (s *Store) Sweep(ctx context.Context) (store.SweepResult, error)
- func (s *Store) Update(ctx context.Context, obj store.Object) error
- func (s *Store) Watch(ctx context.Context, kind string, opts store.WatchOptions) (store.Watcher, error)
- type View
- func (v *View) Dir() string
- func (v *View) Get(ctx context.Context, k store.Key, out store.Object) error
- func (v *View) Journal() []JournalEntry
- func (v *View) List(ctx context.Context, kind string, opts store.ListOptions, out any) (store.ListMeta, error)
- func (v *View) Loaded() LoadInfo
- func (v *View) Transcript(run string) ([]byte, error)
- func (v *View) Transcripts() []string
- func (v *View) Warnings() []string
Constants ¶
const ( DefaultCompactBytes = 64 << 20 DefaultCompactLines = 50_000 )
Compaction defaults: the journal is folded into the snapshot once it passes either.
const Format = 1
Format is the snapshot format this package reads and writes.
Variables ¶
var ( // ErrLocked: another store has the state dir open. ErrLocked = errors.New("state dir is locked by another store") // ErrInsecure: the state dir or a file in it is readable or writable by group or others, // or owned by another user. ErrInsecure = errors.New("state dir is not private") // ErrCorrupt: the snapshot or a journal line (other than a torn last one) can't be loaded. ErrCorrupt = errors.New("state is corrupt") // ErrFormat: the snapshot has a format this package doesn't read. ErrFormat = errors.New("unknown state format") // ErrClosed: a write after Close. ErrClosed = errors.New("store is closed") // ErrJournal: an earlier journal fsync (or rollback) failed, so the journal can't be // trusted; every write fails until the store is reopened. ErrJournal = errors.New("journal unusable after an earlier failure; reopen the store") )
Errors Open and the writes return (test with errors.Is).
Functions ¶
func LockPath ¶
LockPath returns the lock file of the state dir dir: a store holds an exclusive flock on it while it has the dir open.
Types ¶
type JournalEntry ¶
type JournalEntry struct {
Version uint64
At time.Time
Op memory.Op
Kind string
Name string
// Obj is the object as stored by the change (nil for delete).
Obj json.RawMessage
}
JournalEntry is one journal line: one committed change, in version order.
type LoadInfo ¶
type LoadInfo struct {
// Version is the store version restored (0 for a new dir).
Version uint64
// Objects counts the objects loaded, by kind.
Objects map[string]int
// Snapshot is whether there was a snapshot; JournalLines counts the journal lines
// replayed above it.
Snapshot bool
JournalLines int
// TornLine is whether a torn last journal line (a crash mid-append) was dropped.
TornLine bool
// Transcripts counts the transcript files found.
Transcripts int
}
LoadInfo is what Open loaded from the state dir, for the caller's start-up log.
type Option ¶
type Option func(*config)
Option configures Open.
func WithCompaction ¶
WithCompaction sets the journal size, in bytes, and length, in lines, past which a write compacts it into the snapshot (defaults DefaultCompactBytes and DefaultCompactLines).
func WithKind ¶
WithKind registers the Go type of kind, a stored kind outside v1alpha1 (the internal Lease, Session, Delivery and the like): proto is a pointer to its zero value. Open needs the type to load the kind's objects, and writes of a kind without one are ErrInvalid, so a kind that could not be loaded back is never stored. The v1alpha1 runtime kinds are built in.
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is the file-backed store. Use Open, and Close when done.
func Open ¶
Open opens (creating it if missing) the state dir and loads it: the snapshot, then the journal lines above the snapshot's version. A torn last journal line (a crash mid-append) is dropped with a warning and the journal truncated to the last good line; anything else that can't be loaded fails Open with ErrCorrupt and where. The dir must be private (0700, files 0600, owned by this user: ErrInsecure says how to fix it) and not open in another store (ErrLocked). opts are the memory store's.
func (*Store) Close ¶
Close compacts the journal into the snapshot and releases the state dir. Writes after Close fail with ErrClosed; reads still answer from memory. It is idempotent.
func (*Store) Delete ¶
Delete implements store.Store. The change is journaled before it is applied; a Run's transcript goes with it.
func (*Store) GetTranscript ¶
GetTranscript implements store.TranscriptStore.
func (*Store) List ¶
func (s *Store) List(ctx context.Context, kind string, opts store.ListOptions, out any) (store.ListMeta, error)
List implements store.Store.
func (*Store) Lock ¶
Lock implements store.Store. Locks are not persisted: they are advisory and lease-based, and a restart frees them.
func (*Store) PutTranscript ¶
PutTranscript implements store.TranscriptStore: the blob is written to a tmp file, fsync'd and renamed into transcripts/. The caps are the memory store's: over the per-transcript cap is ErrTooLarge, and the oldest transcripts are evicted (logged) to stay under the total.
type View ¶
type View struct {
// contains filtered or unexported fields
}
View is a state dir loaded read-only, by Read: a memory store holding what the snapshot and the journal held when they were read, the journal lines above the snapshot, and the transcripts, read from their files on demand. Nothing in the dir is written, locked or removed, so a View can be taken while a store (a running serve) has the dir open, and of a copy of one.
func Read ¶
Read loads the state dir read-only: the snapshot, then the journal lines above its version. It writes nothing and takes no lock, so it works while a store has the dir open and on a copied dir:
- a journal line being appended (or torn by a crash) is the last one and has no trailing newline: it is left out, and the file left as it is;
- a compaction that replaces the snapshot while Read reads makes it start over (the snapshot it holds open is checked against the file's after the journal is read), so the snapshot and the journal it loads always fit together.
A dir that isn't private is read, with a warning (Warnings): only a store that writes refuses it. A missing dir is an error wrapping os.ErrNotExist; an unknown format is ErrFormat and corruption ErrCorrupt, as for Open. options register the internal kinds (WithKind), as for Open; WithCompaction is ignored.
func (*View) Journal ¶
func (v *View) Journal() []JournalEntry
Journal returns the journal lines above the snapshot, in version order. Changes folded into the snapshot by a compaction are no longer there.
func (*View) List ¶
func (v *View) List(ctx context.Context, kind string, opts store.ListOptions, out any) (store.ListMeta, error)
List lists a kind, as store.Store's List.
func (*View) Loaded ¶
Loaded returns what was read: the version, the objects per kind, whether there was a snapshot, the journal lines above it, whether a torn last line was left out and the transcripts.
func (*View) Transcript ¶
Transcript reads run's transcript file (gzip'd JSON Lines, as the agent uploaded it): store.ErrNotFound when there is none.
func (*View) Transcripts ¶
Transcripts returns the runs with a transcript, sorted.