docstore

package
v0.26.21 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 25, 2026 License: MIT Imports: 25 Imported by: 0

README

docstore (service)

A typed, keyed store of JSON documents with unique and multi-valued secondary indexes, kept in memory and, given a core/data/vfs filesystem, persisted to it: one snapshot at rest, one small overlay entry per write — durable before the write returns, whatever the store holds — folded back into the snapshot as the overlay grows; or over SQL, in a database the caller owns (ADR 0139). Either keeps, when asked, the last versions of each document in the same write as the document (ADR 0143). Public facade: pkg/v1/data/docstore. ADR 0110. See CLAUDE.md and BENCH.md.

Documentation

Overview

Package docstore — the store's construction parameters, and the refusals a configuration no store could honour gets, the index declarations' included: an IndexSpec is core/data/docstore's, which both engines take (ADR 0160).

Package docstore is a typed, keyed store of JSON documents with secondary indexes, kept in memory and, when given a filesystem, persisted to it — every write durable before it returns. ADR 0110.

What it is

A Store[T] holds documents of one Go type, each under the key a function of the document returns. Every read returns a copy and every write stores one: a document is kept as its JSON encoding, so what a caller holds can never alias what the store holds, and the memory and file backends behave alike. Writes come in three modes — Put creates or replaces, Insert refuses an existing key, Replace refuses a missing one and never resurrects a deleted document — plus Update, a read-modify-write, and Delete.

Secondary indexes are declared at Open: Unique files at most one document per key and refuses a write that would file a second, Index files any number. Lookup reads a unique index, Find reads any index, Filter reads every document. The indexes are rebuilt when the store opens, and a load whose documents break a unique index is refused rather than served. So is a load holding a document the type no longer decodes, or one stored under another key than its own Key gives: a store opens only over documents it can serve.

How it persists, and what that costs

The resting state is ONE file, the snapshot: a JSON object mapping each key to its document. A write does not rewrite it. It publishes one small file — an overlay entry, named by the SHA-256 of the key, holding the key and its document or its deletion — atomically, through vfs.AtomicWriter, before it returns. A write therefore costs what one document costs, whatever the store holds. When the overlay holds as many entries as the store holds documents (at least DefaultFoldAt), the write that reaches that count folds it: the snapshot is rewritten once and the entries it now contains are removed. Folding N documents once per N writes keeps the average write a constant. Close folds, so a store that was closed rests as a single snapshot a person can read.

Loading reads the snapshot and replays the overlay on top of it. Replay order does not matter — one entry per key, each holding that key's whole latest state — and an entry the last fold already folded holds exactly what the snapshot holds, so a crash anywhere in a fold, or between a fold and the removals that follow it, loads the same documents.

Concurrency

Readers never wait for a disk. Writers are serialised by one lock and do their filesystem work under it alone; the documents and the indexes are changed only once the write is durable, under a second lock readers share. A reader sees the state before the write or after it, never a document and an index that disagree.

What it is not

One process. The store detects no second process opening the same files, and two would each believe their own memory. It keeps every document in memory and sorts the keys for every List. It has no transaction across documents. It is a store for the documents a service owns, not a database.

The same store over SQL

OpenSQL builds the second engine, SQLStore, for when a store must outgrow that: the same documents, keys, index declarations, write modes, hooks and refusals, kept in two tables of a PostgreSQL, MySQL or SQLite database the caller owns and hands over as a core/data/sql Transactor. Every call takes a context and runs on the transaction it carries — a write in a savepoint of it, its hooks after its commit — and nothing is held in memory. ADR 0139.

Package docstore — hosts the compile-time interface assertions, keeping them out of the production source so the runtime binary carries no diagnostic-only declarations.

Package docstore — the functions told about a write or a deletion once it is durable, outside every lock.

Package docstore — the secondary indexes, kept beside the documents under the store's own locks.

An index is rebuilt from the documents when the store opens, and every write — Put, Insert, Replace, Update, Delete — maintains it in the same critical section that changes the document. A reader therefore never sees a document and an index that disagree, and a write a unique index refuses leaves both exactly as they were.

Package docstore — opening a persistent store: the snapshot and the versions file read, the overlay replayed on top, and the resting state restored.

Package docstore — opening a store: the configuration checked, the indexes declared, and a persistent store loaded.

Package docstore — the files: one snapshot at rest, and an overlay of one entry per key written since, folded back into the snapshot.

Package docstore — the SQL store's construction parameters, and the refusals a configuration no SQL store could honour gets.

Package docstore — the only place the SQL store renders SQL. Every statement it sends is built here, once, at OpenSQL, for its dialect and its tables, spelled with the vocabulary core/data/sql's Dialect owns: the bind markers, the quoting and the row lock.

Package docstore — the SQL store's tables, as a migration the caller runs under its own version table.

Package docstore — the SQL store's index rows rebuilt from its documents, for when the declarations that filed them changed.

Package docstore — the SQL store: the same documents, keys, indexes and refusals as Store, kept in a database the caller owns, each call on the transaction its context carries (ADR 0139).

Package docstore — the SQL store's versions: a third table, written by the statements of the write that stores the document, in its transaction (ADR 0143).

Package docstore — the SQL store's writes: each one atomic on its own, in a transaction of the store's or a savepoint of the caller's, and announced once the transaction that holds it has committed.

Package docstore — a document's versions: the current one, which is the document itself, and the former ones kept beside it, recorded and pruned in the write that stores the document (ADR 0143).

Package docstore — a document's trash, on a store that keeps versions: a trashed document leaves every read and keeps its versions and its number, until Restore brings it back or Purge empties it (ADR 0143, amended for ADR 0178). The file engine keeps it in the document's versions, the SQL engine in a version row numbered 0.

Package docstore — the writes: prepared outside every lock, checked and persisted under the writers' lock, applied under both.

Index

Constants

View Source 
const DefaultFoldAt int = 1024

DefaultFoldAt is the least number of overlay entries that folds the overlay into the snapshot, when Config.FoldAt is zero.

A fold rewrites the whole snapshot, so how often it runs decides what a write costs on average. Folding once the overlay holds as many entries as the store holds documents keeps that average a constant — one fold of N documents per N writes — and the floor keeps a small store from rewriting its snapshot every few writes for nothing: a thousand small files in one directory is not a cost worth avoiding.

View Source 
const MaxSQLIndexNameLen int = 64

MaxSQLIndexNameLen is the longest index name a SQL store accepts, in bytes. An index's name is stored in every one of its rows, beside its keys.

View Source 
const MaxSQLKeyLen int = 1024

MaxSQLKeyLen is the longest store key, and the longest index key as it reaches the table, in bytes. An index row carries an index key and a store key in one primary key, and MySQL's InnoDB caps an index entry at 3072 bytes: two keys of this length and an index name fit in it, with room left.

View Source 
const MaxSQLTableLen int = 58

MaxSQLTableLen is the longest table name a SQL store accepts, in bytes: PostgreSQL's identifier limit of 63, less the five bytes of the longest names the store derives from it — its index table, the name followed by "___ix", and its versions table, followed by "___vs". A caller whose names can be longer shortens them, with a digest, before they reach the store.

Variables

This section is empty.

Functions

func SQLMigration 

func SQLMigration(dialect coresql.Dialect, table string, version uint64) (coresql.MigrationValue, error)

SQLMigration returns the migration that creates the two tables a SQL store named table keeps on dialect — its documents' and its index rows' — as a core/data/sql MigrationValue numbered version, for the caller's Migrator to run under the caller's version table. Its Down drops both tables, and every document in them.

The SDK numbers nothing: a migration is a value its consumer constructs (ADR 0055 §D12), and only the consumer knows where the store's tables sit in its own history.

Every statement does nothing when its table already exists — or is already gone — so a run MySQL's implicit commit stopped halfway completes when it runs again, and applying the migration over tables a previous deployment made changes nothing (ADR 0139).

func SQLVersionsMigration 

func SQLVersionsMigration(dialect coresql.Dialect, table string, version uint64) (coresql.MigrationValue, error)

SQLVersionsMigration returns the migration that creates the table a SQL store named table keeps its versions in on dialect — <table>___vs, one row per version — as a core/data/sql MigrationValue numbered version, beside the one SQLMigration returns. A store opened with SQLConfig.Versions needs it; one without never touches it. Its Down drops the table, and every version in it: run it to turn versions off for good (ADR 0143).

Its one statement does nothing when the table already exists, or is already gone, as SQLMigration's do.

Types

type Config 

type Config[T any] struct {
	// Key returns the key a document is stored under. It must be a pure
	// function of the document: the store calls it to file every write and
	// compares its answers to refuse an update that would rename a document.
	Key func(T) string
	// FS is where the store persists, and Path is the snapshot's name inside
	// it — an io/fs name such as "shop/items.json". The overlay lives beside
	// the snapshot, in the directory Path + ".d". A nil FS keeps the store in
	// memory: nothing survives the process, and nothing is written anywhere.
	FS corevfs.FullFS
	// Clock stamps each version with the instant of the write that made it.
	// Nil is the system clock.
	Clock clock.Clock
	// Held reports whether the document stored under key is held — a legal
	// hold — so that no write prunes its versions: they pile up beyond
	// Versions until a write finds the document released, and that write
	// prunes them. It is called with the document's key by a write that would
	// prune, under the writers' lock, so it may read this store and must not
	// write to it; a panic in it reaches the writer, and nothing is written.
	// Nil holds nothing. It is refused without Versions.
	Held func(key string) bool
	// Path is the snapshot's name inside FS. It is required with FS and
	// refused without it, because a path with nowhere to be written is a
	// store that would silently keep nothing.
	Path string
	// FoldAt is how many overlay entries fold the overlay into the snapshot
	// on the write that reaches it. Zero is the default rule: as many entries
	// as the store holds documents, and at least [DefaultFoldAt]. A negative
	// value never folds on a write; Fold, Open and Close still do.
	FoldAt int
	// Versions is how many former versions each document keeps beside its
	// current one, the newest: a write that changes a document makes a new
	// version, and prunes the oldest beyond Versions in the same durable
	// write. Zero keeps no versions — the store writes exactly the files it
	// wrote before versions existed — and a negative value is refused. A
	// persistent store keeps them in the file Path + ".versions" once a fold
	// ran, and a store opened with zero over files that keep versions is
	// refused rather than left to drop or corrupt them (ADR 0143). A store
	// that keeps versions has a trash, Trash's; one that keeps none has none.
	Versions int
}

Config configures Open. Key is required; every other field is optional, and leaving all of them empty opens a working store kept in memory. The secondary indexes are Open's other arguments.

type SQLConfig 

type SQLConfig[T any] struct {
	// Key returns the key a document is stored under, as for [Open]: a pure
	// function of the document, compared byte for byte.
	Key func(T) string
	// Clock stamps each version with the instant of the write that made it.
	// Nil is the system clock.
	Clock clock.Clock
	// Held reports whether the document stored under key is held — a legal
	// hold — so that no write prunes its versions, as for [Open]. It is
	// called by a write that would prune, inside that write's transaction
	// and with its context, so a read it makes through the same transactor
	// joins it. Nil holds nothing. It is refused without Versions.
	Held func(ctx context.Context, key string) bool
	// Transactor is the transaction manager of the database the store lives
	// in — the one the caller's own transactions are opened with, so a call
	// made under one of them joins it. It must also be a core/data/sql Joiner and
	// Deferrer, which the SDK's own transactor is.
	Transactor coresql.Transactor
	// IndexKey, when set, transforms every index key before it reaches the
	// database — on a write, and on every Lookup and Find — so a table holds
	// what it returns and never the key itself. An index is an equality
	// lookup, which a keyed hash keeps: a framework passes an HMAC under a key
	// of its own, and an e-mail address never reaches a table. It is called
	// with the index's name and a non-empty key; an empty answer files
	// nothing, as an empty key does. Nil stores index keys as they are.
	IndexKey func(index, key string) []byte
	// Table names the documents' table. The store derives its index table's
	// name from it by appending "___ix". It is a lower-case SQL identifier of
	// at most [MaxSQLTableLen] bytes, holds no three underscores in a row,
	// and does not start with "sqlite_". [SQLMigration] creates both tables.
	Table string
	// Dialect is the engine's: the SQL the store speaks to it. It must be the
	// Transactor's own.
	Dialect coresql.Dialect
	// Versions is how many former versions each document keeps beside its
	// current one, as for [Open]: a write that changes a document makes a
	// new version and prunes the oldest beyond Versions, in its own
	// transaction. They live in a third table, <Table>___vs, which
	// [SQLVersionsMigration] creates. Zero keeps no versions and sends
	// exactly the statements the store sent before versions existed; a
	// negative value is refused. The store cannot see a versions table it was
	// told nothing about: turned off, it leaves the table as it is, and a
	// document deleted and written again meanwhile would find the rows of the
	// one it replaced — so drop the table with the migration's Down (ADR 0143).
	// A store that keeps versions has a trash, Trash's, in the same table; one
	// that keeps none has none.
	Versions int
}

SQLConfig configures OpenSQL. Key, Transactor, Dialect and Table are required. The secondary indexes are OpenSQL's other arguments, declared with coredocstore.Unique and coredocstore.Index exactly as for Open.

type SQLStore 

type SQLStore[T any] struct {
	// contains filtered or unexported fields
}

SQLStore is a typed, keyed collection of JSON documents with secondary indexes, kept in two tables of a SQL database. Build one with OpenSQL. It is safe for concurrent use, and holds nothing but its configuration: the database is the source of truth, and every call is a round trip.

Every call takes a context, and runs on the transaction that context carries for the store's transactor, else on the pool. A write runs in a transaction of its own — a savepoint inside the caller's — so it is atomic on its own either way, and its hooks run once the transaction that holds it has committed.

func OpenSQL 

func OpenSQL[T any](cfg SQLConfig[T], indexes ...coredocstore.IndexSpec[T]) (*SQLStore[T], error)

OpenSQL builds a SQL store from cfg, with the secondary indexes given. It sends no statement: its tables are SQLMigration's, run before the store is used. 

cases, err := docstore.OpenSQL(
    docstore.SQLConfig[Case]{Key: Case.Key, Transactor: tm, Dialect: sql.DialectPostgres, Table: "moderation_intake__cases"},
    coredocstore.Unique("reference", func(c Case) string { return c.Reference }),
)

func (*SQLStore[T]) Count 

func (s *SQLStore[T]) Count(ctx context.Context) (int, error)

Count returns how many documents the store holds.

func (*SQLStore[T]) Delete 

func (s *SQLStore[T]) Delete(ctx context.Context, key string) error

Delete removes the document stored under key, or returns DocumentNotFound. Its versions go with it, in the same transaction: a deletion never reaches the trash, which is Trash's.

func (*SQLStore[T]) Entries 

func (s *SQLStore[T]) Entries(ctx context.Context, limit int) (entries []coredocstore.EntryValue, err error)

Entries returns up to limit stored documents as JSON, in key order — every one when limit is not positive — for a caller that shows documents rather than decoding them. Each JSON is the bytes the store wrote.

func (*SQLStore[T]) Filter 

func (s *SQLStore[T]) Filter(ctx context.Context, keep func(T) bool) ([]T, error)

Filter returns the documents keep accepts, in key order. It reads every document; an index is the way not to.

func (*SQLStore[T]) Find 

func (s *SQLStore[T]) Find(ctx context.Context, index, key string) ([]T, error)

Find returns the documents filed under key in the index named index, in store-key order. It reads any index, unique or not, and no document is an empty slice, not an error.

func (*SQLStore[T]) Get 

func (s *SQLStore[T]) Get(ctx context.Context, key string) (T, error)

Get returns the document stored under key, or DocumentNotFound.

func (*SQLStore[T]) Insert 

func (s *SQLStore[T]) Insert(ctx context.Context, v T) error

Insert stores v, which must be new: DocumentExists when its key is taken.

func (*SQLStore[T]) InsertStamped 

func (s *SQLStore[T]) InsertStamped(ctx context.Context, v T, stamp coredocstore.StampValue) error

InsertStamped is Insert, with what the write says about the version it makes: version 1, whatever stamp.InPlace says — or, under the key of a document in the trash, a write over that document.

func (*SQLStore[T]) Keys added in v0.6.0 

func (s *SQLStore[T]) Keys(ctx context.Context, after string) iter.Seq2[string, error]

Keys yields the key of every document the store holds that sorts after after — every key when after is empty — in key order, byte order, and no document. It reads them in pages of a few hundred, each page one statement on the primary key that resumes after the last key yielded, on the transaction ctx carries, else on the pool: a document written during the range after the last key read is seen, one written before it is not. A failed statement is yielded once, as StatementFailed, and ends the listing.

func (*SQLStore[T]) List 

func (s *SQLStore[T]) List(ctx context.Context) ([]T, error)

List returns every document, in key order — byte order, as Go compares strings.

func (*SQLStore[T]) Lookup 

func (s *SQLStore[T]) Lookup(ctx context.Context, index, key string) (T, error)

Lookup returns the document holding key in the unique index named index, DocumentNotFound when none does, IndexUnknown for an undeclared index and IndexNotUnique for an index that may hold several: read that one with Find.

func (*SQLStore[T]) OnDelete 

func (s *SQLStore[T]) OnDelete(fn func(key string)) (remove func())

OnDelete registers fn to be told the key of every document Delete or Trash removed, on the same terms as OnWrite.

func (*SQLStore[T]) OnWrite 

func (s *SQLStore[T]) OnWrite(fn func(key string)) (remove func())

OnWrite registers fn to be told the key of every document Put, Insert, Replace, Update or Restore stored, once the transaction that holds the write has committed — never for a write that is rolled back — on the goroutine that committed it, outside every lock. It returns the function that removes the registration.

func (*SQLStore[T]) Purge added in v0.5.0 

func (s *SQLStore[T]) Purge(ctx context.Context, key string) error

Purge empties the trash of the document trashed under key: the trashed document and every version row of its key go, in one transaction, and a document written later under the key starts again at version 1. It calls no hook and asks Held nothing, as Delete does not. DocumentNotTrashed when no document is trashed under key.

func (*SQLStore[T]) Put 

func (s *SQLStore[T]) Put(ctx context.Context, v T) error

Put stores v under its key, creating it or replacing whatever was there. On a store that keeps versions, it makes one, stamped with nothing.

func (*SQLStore[T]) PutDocument added in v0.8.0 

func (s *SQLStore[T]) PutDocument(ctx context.Context, key string, versions []coredocstore.VersionValue, trashedAt time.Time) error

PutDocument writes the document key with its versions as they are given, in one transaction — a savepoint of the caller's when ctx carries one —, as core/data/docstore.VersionWriterContext says: the document's row and its index rows unless it is trashed, and a version row for each version — the trash's row, numbered 0, at trashedAt when it is set. It calls OnWrite once that transaction commits, for a stored document. DocumentExists when anything is under key — a document, the trash, or version rows —, VersionsInvalid, DocumentUndecodable, KeyTooLong or UniqueKeyTaken for what cannot be written as given, VersionsNotKept on a store that keeps none; nothing changes on a refusal.

func (*SQLStore[T]) PutStamped 

func (s *SQLStore[T]) PutStamped(ctx context.Context, v T, stamp coredocstore.StampValue) error

PutStamped is Put, with what the write says about the version it makes.

func (*SQLStore[T]) Reindex 

func (s *SQLStore[T]) Reindex(ctx context.Context) error

Reindex rebuilds every index row from the stored documents, in one transaction — a savepoint of the caller's when ctx carries one.

The SQL store keeps its index rows, where the memory and file store rebuild theirs at every Open. So a change to the declarations — an index added, a key function or IndexKey changed, an index made unique — files nothing for the documents already stored until Reindex runs. It drops every row, including those of an index no longer declared, reads the documents in key order a page at a time, and files each one's keys again.

Documents that break a unique index refuse the rebuild with INDEX_BROKEN, naming the index and never the key, and so does a key function that panics; a document the type no longer fits is DOCUMENT_UNDECODABLE. Nothing changes then. Run it when no write to the store runs beside it: a write in flight files its keys under whichever rows it finds.

func (*SQLStore[T]) Replace 

func (s *SQLStore[T]) Replace(ctx context.Context, v T) error

Replace stores v over the document already under its key: DocumentNotFound when there is none, so a document deleted meanwhile is never brought back.

func (*SQLStore[T]) ReplaceStamped 

func (s *SQLStore[T]) ReplaceStamped(ctx context.Context, v T, stamp coredocstore.StampValue) error

ReplaceStamped is Replace, with what the write says about the version it makes.

func (*SQLStore[T]) Restore added in v0.5.0 

func (s *SQLStore[T]) Restore(ctx context.Context, key string) error

Restore brings the document trashed under key back exactly as it was — the same JSON, the same current version — files it again and calls OnWrite once its transaction commits. DocumentNotTrashed when no document is trashed under key, UniqueKeyTaken when another document took one of its unique keys meanwhile, DocumentUndecodable when it no longer decodes into the store's type; nothing changes on a refusal.

func (*SQLStore[T]) RewriteVersions 

func (s *SQLStore[T]) RewriteVersions(ctx context.Context, key string, fn func(former []coredocstore.VersionValue) ([]coredocstore.VersionValue, error)) error

RewriteVersions rewrites the former versions of the document stored under key — every version but the current one, which is the document and which only a write changes — in one transaction, a savepoint of the caller's when ctx carries one, the document locked as a write locks it. fn receives copies of them, newest first, and returns those to keep, in the same order, under the rules of Store.RewriteVersions: VersionsRewriteRefused otherwise, and fn's own error returned as it is, nothing changed either way. It calls no hook: the document did not change.

func (*SQLStore[T]) Tombstones added in v0.5.0 

func (s *SQLStore[T]) Tombstones(ctx context.Context) (tombstones []coredocstore.TombstoneValue, err error)

Tombstones returns every document in the trash, in key order: its key and the instant it was trashed, read in one statement on the transaction ctx carries. The statement scans the versions table, as List scans the documents'.

func (*SQLStore[T]) Trash added in v0.5.0 

func (s *SQLStore[T]) Trash(ctx context.Context, key string) error

Trash moves the document stored under key to the trash, in one transaction — a savepoint of the caller's when ctx carries one: its row and its index rows go, and a version row numbered 0 — no version's number — holds the document and the instant it was trashed beside its versions, which stay. It calls OnDelete once that transaction commits. DocumentNotFound when no document is stored under key, VersionsNotKept when the store keeps no versions: only a store that keeps versions has a trash.

func (*SQLStore[T]) Update 

func (s *SQLStore[T]) Update(ctx context.Context, key string, fn func(*T) error) (T, error)

Update applies fn to a copy of the document stored under key and stores the result, atomically: the document is locked from the read to the write — its row on PostgreSQL and MySQL, the database's write lock on SQLite — so no other write lands between them. An error from fn changes nothing and is returned for errors.Is to find, and a result whose key is not key is DocumentKeyChanged. It returns the stored result.

fn runs inside the write's transaction. It may read the store; a read under a context carrying no transaction runs on the pool, and so needs a second connection. It must not write the document it updates, which is locked by the very write that is waiting for fn.

func (*SQLStore[T]) UpdateStamped 

func (s *SQLStore[T]) UpdateStamped(ctx context.Context, key string, stamp coredocstore.StampValue, fn func(*T) error) (T, error)

UpdateStamped is Update, with what the write says about the version it makes.

func (*SQLStore[T]) Version 

func (s *SQLStore[T]) Version(ctx context.Context, key string, number uint64) (coredocstore.VersionValue, error)

Version returns the version numbered number of the document stored under key, VersionNotFound when the document keeps none of that number — never made, or pruned — and what Versions returns otherwise.

func (*SQLStore[T]) VersionWeight added in v0.9.0 

func (s *SQLStore[T]) VersionWeight(ctx context.Context) (coredocstore.VersionWeightValue, error)

VersionWeight reports what the store keeps beyond its documents, as core/data/docstore.VersionWeigherContext says: one aggregate statement over the versions table — its former rows, its trash rows numbered 0, and the length of the documents they hold —, on the transaction ctx carries, else on the pool. The database scans the table and sends one row. VersionsNotKept on a store that keeps none, StatementFailed when the database does not answer.

func (*SQLStore[T]) Versions 

func (s *SQLStore[T]) Versions(ctx context.Context, key string) (versions []coredocstore.VersionValue, err error)

Versions returns the versions the store keeps of the document stored under key, newest first: the current one — the document as it is now — then the former ones, read in one statement on the transaction ctx carries; a document in the trash answers the versions it kept, the trashed document first, in a second one. DocumentNotFound when no document is stored or trashed under key, VersionsNotKept when the store keeps no versions.

type StatsValue 

type StatsValue struct {
	// FoldError is the last automatic fold's failure, nil once a fold
	// succeeds. The write that triggered the fold succeeded either way — its
	// own entry was already durable — and the entries stay until a fold works.
	FoldError error
	// Documents is how many documents the store holds.
	Documents int
	// Pending is how many overlay entries the snapshot does not contain yet:
	// zero in memory, and after a fold.
	Pending int
	// Folds is how many folds ran since the store opened, Open's own included.
	Folds int
	// Versions is what the store's versions weigh — its former versions, its
	// trashed documents and their bytes —, a counter every write moves:
	// zero on a store that keeps none (ADR 0143).
	Versions coredocstore.VersionWeightValue
}

StatsValue is what a store can say about itself: how many documents it holds, how far its overlay has grown, how its folds have gone, and what its versions weigh.

type Store 

type Store[T any] struct {
	// contains filtered or unexported fields
}

Store is a typed, keyed collection of JSON documents with secondary indexes. Build one with Open. It is safe for concurrent use.

func Open 

func Open[T any](cfg Config[T], indexes ...coredocstore.IndexSpec[T]) (*Store[T], error)

Open builds a store from cfg, with the secondary indexes given: in memory without a filesystem, otherwise loaded from it — the snapshot, the overlay replayed on top, the indexes rebuilt, and the overlay folded when it holds anything. It creates the directories the store lives in, 0700, and writes its files 0600. An open it refuses writes no data: the fold comes after every check, so the files an operator must repair are left as they were. 

accounts, err := docstore.Open(
    docstore.Config[Account]{Key: Account.ID, FS: data, Path: "members/accounts.json"},
    coredocstore.Unique("email", func(a Account) string { return a.Email }),
    coredocstore.Index("team", func(a Account) []string { return a.Teams }),
)

func (*Store[T]) Close 

func (s *Store[T]) Close() error

Close folds the overlay and closes the store: every later call is StoreClosed, except Stats and Close, which returns nil. A fold that fails leaves its entries where they are — the next Open replays them — and is returned; the store closes either way.

func (*Store[T]) Delete 

func (s *Store[T]) Delete(key string) error

Delete removes the document stored under key, or returns DocumentNotFound. Its versions go with it: a deletion never reaches the trash, which is Trash's.

func (*Store[T]) Entries 

func (s *Store[T]) Entries(limit int) ([]coredocstore.EntryValue, error)

Entries returns up to limit stored documents as JSON, in key order — every one when limit is not positive — for a caller that shows documents rather than decoding them. Each JSON is the caller's own copy.

func (*Store[T]) Filter 

func (s *Store[T]) Filter(keep func(T) bool) ([]T, error)

Filter returns the documents keep accepts, in key order. It decodes every document; an index is the way not to.

func (*Store[T]) Find 

func (s *Store[T]) Find(index, key string) ([]T, error)

Find returns the documents filed under key in the index named index, in store-key order. It reads any index, unique or not, and no document is an empty slice, not an error.

func (*Store[T]) Fold 

func (s *Store[T]) Fold() error

Fold rewrites the snapshot with every document and removes the overlay entries it now contains. A write folds by itself when the overlay has grown enough, and Open and Close fold; Fold is for a caller that wants the resting state now — before a backup, say. A memory store has nothing to fold.

func (*Store[T]) Get 

func (s *Store[T]) Get(key string) (T, error)

Get returns the document stored under key, or DocumentNotFound.

func (*Store[T]) Insert 

func (s *Store[T]) Insert(v T) error

Insert stores v, which must be new: DocumentExists when its key is taken.

func (*Store[T]) InsertStamped 

func (s *Store[T]) InsertStamped(v T, stamp coredocstore.StampValue) error

InsertStamped is Insert, with what the write says about the version it makes: version 1, whatever stamp.InPlace says — or, under the key of a document in the trash, a write over that document.

func (*Store[T]) Keys added in v0.6.0 

func (s *Store[T]) Keys(after string) iter.Seq2[string, error]

Keys yields the key of every document the store holds that sorts after after — every key when after is empty — in key order, and no document. The keys are taken once, under the read lock, when the range starts: a write made during the range is not seen by it. What it costs is the keys' sort, never a document's copy. A closed store yields StoreClosed once.

func (*Store[T]) List 

func (s *Store[T]) List() ([]T, error)

List returns every document, in key order.

func (*Store[T]) Lookup 

func (s *Store[T]) Lookup(index, key string) (T, error)

Lookup returns the document holding key in the unique index named index, DocumentNotFound when none does, IndexUnknown for an undeclared index and IndexNotUnique for an index that may hold several: read that one with Find.

func (*Store[T]) OnDelete 

func (s *Store[T]) OnDelete(fn func(key string)) (remove func())

OnDelete registers fn to be told the key of every document Delete or Trash removed, on the same terms as OnWrite.

func (*Store[T]) OnWrite 

func (s *Store[T]) OnWrite(fn func(key string)) (remove func())

OnWrite registers fn to be told the key of every document Put, Insert, Replace, Update or Restore stored, once the write is durable, on the writer's goroutine and outside every lock. It returns the function that removes the registration. A panic in fn reaches the writer after its write stood.

func (*Store[T]) Purge added in v0.5.0 

func (s *Store[T]) Purge(key string) error

Purge empties the trash of the document trashed under key: the trashed document and every version it kept go, in one durable write, and a document written later under the key starts again at version 1. It calls no hook and asks Held nothing, as Delete does not. DocumentNotTrashed when no document is trashed under key.

func (*Store[T]) Put 

func (s *Store[T]) Put(v T) error

Put stores v under its key, creating it or replacing whatever was there. On a store that keeps versions, it makes one, stamped with nothing.

func (*Store[T]) PutDocument added in v0.8.0 

func (s *Store[T]) PutDocument(key string, versions []coredocstore.VersionValue, trashedAt time.Time) error

PutDocument writes the document key with its versions as they are given, in one durable write under the writers' lock, as core/data/docstore.VersionWriter says: the current version and the former ones with their numbers, instants and metadata, and — when trashedAt is set — the document in the trash at that instant. Stored, it is filed and OnWrite is told; trashed, it is neither. DocumentExists when anything is under key, VersionsInvalid, DocumentUndecodable or UniqueKeyTaken for what cannot be written as given, VersionsNotKept on a store that keeps none; nothing changes on a refusal.

func (*Store[T]) PutStamped 

func (s *Store[T]) PutStamped(v T, stamp coredocstore.StampValue) error

PutStamped is Put, with what the write says about the version it makes.

func (*Store[T]) Replace 

func (s *Store[T]) Replace(v T) error

Replace stores v over the document already under its key: DocumentNotFound when there is none, so a document deleted meanwhile is never brought back.

func (*Store[T]) ReplaceStamped 

func (s *Store[T]) ReplaceStamped(v T, stamp coredocstore.StampValue) error

ReplaceStamped is Replace, with what the write says about the version it makes.

func (*Store[T]) Restore added in v0.5.0 

func (s *Store[T]) Restore(key string) error

Restore brings the document trashed under key back exactly as it was — the same JSON, the same current version — files it again and calls OnWrite. DocumentNotTrashed when no document is trashed under key, UniqueKeyTaken when another document took one of its unique keys meanwhile, DocumentUndecodable when it no longer decodes into the store's type; nothing changes on a refusal.

func (*Store[T]) RewriteVersions 

func (s *Store[T]) RewriteVersions(key string, fn func(former []coredocstore.VersionValue) ([]coredocstore.VersionValue, error)) error

RewriteVersions rewrites the former versions of the document stored under key — every version but the current one, which is the document and which only a write changes — in one durable write, under the writers' lock. fn receives copies of them, newest first, and returns those to keep, in the same order: it may drop a version, and change the document one holds and what its writer said, and nothing else — VersionsRewriteRefused for a version it was not given, out of order, made at another instant or that is not JSON. An error from fn is returned as it is and changes nothing. It calls no hook: the document did not change.

It is what an erasure needs: the former versions cleared as the document is, or dropped. fn runs under the writers' lock, like Update's.

func (*Store[T]) Stats 

func (s *Store[T]) Stats() StatsValue

Stats reports what the store holds and where its overlay stands. It answers after Close too, with the state the store closed in.

func (*Store[T]) Tombstones added in v0.5.0 

func (s *Store[T]) Tombstones() ([]coredocstore.TombstoneValue, error)

Tombstones returns every document in the trash, in key order: its key and the instant it was trashed. It reads every document's versions, as List reads every document.

func (*Store[T]) Trash added in v0.5.0 

func (s *Store[T]) Trash(key string) error

Trash moves the document stored under key to the trash, in one durable write under the writers' lock: it leaves every read and its index keys are freed, while its versions are kept beside the trashed document and the instant it was trashed. It calls OnDelete. DocumentNotFound when no document is stored under key, VersionsNotKept when the store keeps no versions: only a store that keeps versions has a trash.

func (*Store[T]) Update 

func (s *Store[T]) Update(key string, fn func(*T) error) (T, error)

Update applies fn to a copy of the document stored under key and stores the result, atomically with respect to every other write. An error from fn is returned as it is and changes nothing, and so does a result whose key is not key (DocumentKeyChanged). It returns the stored result.

fn runs under the writers' lock: it may read this store, and must not write to it — the write would wait for the lock fn holds, forever.

func (*Store[T]) UpdateStamped 

func (s *Store[T]) UpdateStamped(key string, stamp coredocstore.StampValue, fn func(*T) error) (T, error)

UpdateStamped is Update, with what the write says about the version it makes.

func (*Store[T]) Version 

func (s *Store[T]) Version(key string, number uint64) (coredocstore.VersionValue, error)

Version returns the version numbered number of the document stored under key, VersionNotFound when the document keeps none of that number — never made, or pruned — and what Versions returns otherwise.

func (*Store[T]) VersionWeight added in v0.9.0 

func (s *Store[T]) VersionWeight() (coredocstore.VersionWeightValue, error)

VersionWeight reports what the store keeps beyond its documents — its former versions, its trashed documents and the bytes of their JSON — as core/data/docstore.VersionWeigher says. It reads the counter every write moves, which Stats also reports, and no document. VersionsNotKept on a store that keeps none, StoreClosed after Close.

func (*Store[T]) Versions 

func (s *Store[T]) Versions(key string) ([]coredocstore.VersionValue, error)

Versions returns the versions the store keeps of the document stored under key, newest first: the current one — the document as it is now — then the former ones; a document in the trash answers the versions it kept, the trashed document first. It returns DocumentNotFound when no document is stored or trashed under key, and VersionsNotKept when the store keeps no versions.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.