docstore

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 14, 2026 License: MIT Imports: 4 Imported by: 0

README

docstore

import "github.com/kitsunium/sdk/pkg/v1/data/docstore"

Package docstore is the public facade for the SDK's document store: typed, keyed JSON documents with unique and multi-valued secondary indexes, kept in memory or persisted to a filesystem, every write durable before it returns.

type Account struct {
    ID    string   `json:"id"`
    Email string   `json:"email"`
    Teams []string `json:"teams"`
}

data, err := vfs.NewOS("/var/lib/app") // or vfs.NewMem(), or nil FS for memory
accounts, err := docstore.Open(
    docstore.Config[Account]{
        Key:  func(a Account) string { return a.ID },
        FS:   data,
        Path: "members/accounts.json",
    },
    docstore.Unique("email", func(a Account) string { return a.Email }),
    docstore.Index("team", func(a Account) []string { return a.Teams }),
)
defer accounts.Close()

err = accounts.Insert(Account{ID: "acc_1", Email: "ada@example.com"})
ada, err := accounts.Lookup("email", "ada@example.com")
red, err := accounts.Find("team", "red")
Documents, keys and the write modes

A document is kept as its JSON encoding, so every read returns a copy and a caller can never alias what the store holds. Its key is what Config.Key returns; an empty key is refused. Put creates or replaces; Insert refuses a key that is taken (DocumentExists); Replace refuses a key that is not (DocumentNotFound), so a document deleted meanwhile is never brought back. Update applies a function to a copy and stores the result atomically with respect to every other write — its function may read the store, and must not write to it. Delete removes.

Indexes

Unique files at most one document per key: a write that would file a second is refused with UniqueKeyTaken and changes nothing. Index files any number. Empty keys are not indexed. Lookup reads a unique index; Find reads any index, in store-key order, and answers an empty slice rather than an error; Filter reads every document. The indexes are rebuilt when the store opens, and documents that break a unique index refuse the open (IndexBroken) rather than answer lies.

No refusal ever quotes a key: a store key is routinely an e-mail address, an index key the hash of a token.

How it persists, and what a write costs

At rest the store is ONE file, the snapshot — a JSON object from key to document, the file a person can open. A write publishes one small file beside it, in the directory Path + ".d", atomically, before it returns — so a write costs one document, whatever the store holds. When that overlay holds as many entries as the store holds documents (at least DefaultFoldAt), the write that got it there folds it back into the snapshot: one rewrite per N writes, a constant on average. Close folds too. Opening replays the overlay over the snapshot, and a crash at any point of a fold loads the same documents. BENCH.md in internal/service/data/docstore has the numbers: a write at 100 000 documents costs what it costs at 100, where rewriting the whole file costs 70 ms.

A publication that fails is PersistFailed and changes nothing. One whose rename happened but whose directory flush failed is WriteUnconfirmed: the write stands — the file shows it, and so does the store — and only its survival across a power loss is in doubt.

Concurrency, and what it is not

A Store is safe for concurrent use, and a reader never waits for a disk: writers are serialised among themselves and do their filesystem work alone, and the documents change only once a write is durable. OnWrite and OnDelete tell a caller about each write once it is durable, outside every lock.

It is one process's store: a second process opening the same files is not detected. It keeps every document in memory. It has no transaction across documents. It is a store for the documents a service owns, not a database.

The same store over SQL

OpenSQL keeps the same documents — the same keys, indexes, write modes, hooks and refusals, the same codes — in two tables of a PostgreSQL, MySQL or SQLite database the caller opened and hands over as a [sql.Transactor]. Nothing is held in memory: the database is the source of truth, and every call is a round trip that takes a context (ADR 0139).

tm, err := sql.NewTransactor(sql.Config{DB: pool, Dialect: sql.DialectPostgres, Pool: sql.PoolConfig{MaxOpen: 10}})
create, err := docstore.SQLMigration(sql.DialectPostgres, "members__accounts", 20260927120000) // run by your Migrator
accounts, err := docstore.OpenSQL(
    docstore.SQLConfig[Account]{Key: func(a Account) string { return a.ID },
        Transactor: tm, Dialect: sql.DialectPostgres, Table: "members__accounts"},
    docstore.Unique("email", func(a Account) string { return a.Email }),
)
err = sql.Transact(ctx, tm, func(ctx context.Context, _ sql.Executor) error {
    return accounts.Insert(ctx, Account{ID: "acc_1", Email: "ada@example.com"}) // joins this transaction
})

A call runs on the transaction its context carries for that transactor, and on the pool otherwise. A write runs in a savepoint of the caller's transaction — or a transaction of its own — so it is atomic on its own, and a refused or failed one leaves the caller's transaction usable. Its hooks run once the transaction that holds it has committed, and never for one rolled back.

The document is kept as the bytes the store encoded, never as the engine's JSON type, which reorders members and respells numbers, so it reads back byte for byte. Keys are compared as bytes — binary key columns, no collation — so "a", "A" and "a " are three keys, as they are in Go. A key longer than MaxSQLKeyLen bytes is refused with KeyTooLong. Index keys may reach the table hashed: SQLConfig.IndexKey transforms every one at a write and at every Lookup and Find, and an index is an equality lookup, which a keyed hash keeps. Index rows are kept, not rebuilt at every open: after a declaration changes, SQLStore.Reindex files the stored documents again.

A failure of the database is StatementFailed, which a caller answers with a 503: the driver's error is joined beside it for errors.Is and errors.As, and withheld from its text, because a driver quotes the row a constraint refused and that row holds a key.

The ports, for a double

Store and SQLStore are engines. A caller that must stand a double in for one depends instead on the port it implements: Collection and Versioned for a Store, whose calls take no context, CollectionContext and VersionedContext for a SQLStore, whose every call takes one, and Announcer — OnWrite and OnDelete — for either. Each pair carries the same methods, the same documents and the same refusals; there is no single port over both, because the file engine waits on nothing a context could abandon while the SQL engine waits on a database (ADR 0160).

Versions

A store opened with Config.Versions — or SQLConfig.Versions — keeps the last versions of each document, as a content system keeps a page's revisions (ADR 0143):

pages, err := docstore.Open(docstore.Config[Page]{Key: Page.Key, FS: data, Path: "cms/pages.json", Versions: 20})
err = pages.PutStamped(page, docstore.Stamp{Meta: map[string]string{"by": userID, "command": "pages.Edit"}})
revisions, err := pages.Versions(page.ID) // newest first: the page as it is, then the 20 before it
third, err := pages.Version(page.ID, revisions[3].Number)

A creation is version 1, and every write that changes the document makes the next one, stamped with the instant of Config.Clock and the metadata the Stamped writes carry — who, which command; the plain writes carry none. A number is never given twice while the document exists. A write storing the JSON already stored makes no version, and neither does one stamped InPlace: the document changes and its current version keeps its number. The former versions beyond Versions are pruned by the write that makes a newer one, in the same durable write as the document — the same overlay entry, the same transaction — so no crash ever leaves the versions ahead of or behind their document. Config.Held keeps a document's versions from pruning, for a legal hold, until a write finds it released. RewriteVersions rewrites a document's former versions under the writers' lock, for an erasure: it may clear or drop them, never renumber or add one. A deletion takes every version with it. Versions are never indexed: Lookup and Find read the current version.

The file store keeps them in memory, in each write's overlay entry and, at rest, in the file Path + ".versions"; a store opened without versions over files that keep them is refused, rather than left to drop them. The SQL store keeps them in a third table, which SQLVersionsMigration creates. Version.JSON is compact JSON; a version predating a change of the type may no longer decode into it, which is why it is JSON and not a T.

Index

Constants

const CodeDocumentExists errs.Code = 217090

CodeDocumentExists: Insert over a taken key.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.CodeDocumentExists.

Code 0.3.80.2

const CodeDocumentKeyChanged errs.Code = 217093

CodeDocumentKeyChanged: an Update's function changed the key.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.CodeDocumentKeyChanged.

Code 0.3.80.5

const CodeDocumentKeyEmpty errs.Code = 217092

CodeDocumentKeyEmpty: the key function returned the empty string.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.CodeDocumentKeyEmpty.

Code 0.3.80.4

const CodeDocumentNotFound errs.Code = 217089

CodeDocumentNotFound: no document under the key, or nobody holds the key in a unique index.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.CodeDocumentNotFound.

Code 0.3.80.1

const CodeDocumentUndecodable errs.Code = 217096

CodeDocumentUndecodable: a stored document no longer fits the type.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.CodeDocumentUndecodable.

Code 0.3.80.8

const CodeDocumentUnencodable errs.Code = 217097

CodeDocumentUnencodable: encoding/json refused the value.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.CodeDocumentUnencodable.

Code 0.3.80.9

const CodeIndexBroken errs.Code = 217100

CodeIndexBroken: the stored documents break a unique index.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.CodeIndexBroken.

Code 0.3.80.12

const CodeIndexNotUnique errs.Code = 217095

CodeIndexNotUnique: Lookup on a multi-valued index.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.CodeIndexNotUnique.

Code 0.3.80.7

const CodeIndexUnknown errs.Code = 217094

CodeIndexUnknown: a read named an index the store never declared.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.CodeIndexUnknown.

Code 0.3.80.6

const CodeKeyTooLong errs.Code = 217105

CodeKeyTooLong: a key is longer than the SQL store's key columns hold.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.CodeKeyTooLong.

Code 0.3.80.17

const CodeLoadFailed errs.Code = 217099

CodeLoadFailed: the store's files cannot be read or are not a store's.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.CodeLoadFailed.

Code 0.3.80.11

const CodePersistFailed errs.Code = 217098

CodePersistFailed: the filesystem refused the write; nothing changed.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.CodePersistFailed.

Code 0.3.80.10

const CodeStatementFailed errs.Code = 217104

CodeStatementFailed: the SQL store's database did not complete a call.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.CodeStatementFailed.

Code 0.3.80.16

const CodeStoreClosed errs.Code = 217101

CodeStoreClosed: a call after Close.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.CodeStoreClosed.

Code 0.3.80.13

const CodeStoreMisconfigured errs.Code = 217102

CodeStoreMisconfigured: Open refused its configuration.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.CodeStoreMisconfigured.

Code 0.3.80.14

const CodeUniqueKeyTaken errs.Code = 217091

CodeUniqueKeyTaken: a unique index refused the write.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.CodeUniqueKeyTaken.

Code 0.3.80.3

const CodeVersionNotFound errs.Code = 217107

CodeVersionNotFound: the document keeps no version of that number.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.CodeVersionNotFound.

Code 0.3.80.19

const CodeVersionsNotKept errs.Code = 217106

CodeVersionsNotKept: a call on versions to a store that keeps none.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.CodeVersionsNotKept.

Code 0.3.80.18

const CodeVersionsRewriteRefused errs.Code = 217108

CodeVersionsRewriteRefused: a rewrite returned what a rewrite may not.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.CodeVersionsRewriteRefused.

Code 0.3.80.20

const CodeWriteUnconfirmed errs.Code = 217103

CodeWriteUnconfirmed: the write stands, its durability is in doubt.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.CodeWriteUnconfirmed.

Code 0.3.80.15

const DefaultFoldAt int = 1024

DefaultFoldAt is the least number of overlay entries that folds the overlay into the snapshot when Config.FoldAt is zero.

Initialised with github.com/kitsunium/sdk/internal/service/data/docstore.DefaultFoldAt.

const MaxSQLIndexNameLen int = 64

MaxSQLIndexNameLen is the longest index name a SQL store accepts, in bytes.

Initialised with github.com/kitsunium/sdk/internal/service/data/docstore.MaxSQLIndexNameLen.

const MaxSQLKeyLen int = 1024

MaxSQLKeyLen is the longest store key, and the longest index key as it reaches the table, a SQL store accepts, in bytes.

Initialised with github.com/kitsunium/sdk/internal/service/data/docstore.MaxSQLKeyLen.

const MaxSQLTableLen int = 58

MaxSQLTableLen is the longest table name OpenSQL and SQLMigration accept, in bytes: PostgreSQL's limit of 63, less what the store appends to derive its index table's name.

Initialised with github.com/kitsunium/sdk/internal/service/data/docstore.MaxSQLTableLen.

Variables

var DocumentExists *errs.Error

DocumentExists refuses an insertion over a taken key.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.DocumentExists.

Code 0.3.80.2 · reason DOCUMENT_EXISTS · public "A document already has this key"

var DocumentKeyChanged *errs.Error

DocumentKeyChanged refuses an update that renames.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.DocumentKeyChanged.

Code 0.3.80.5 · reason DOCUMENT_KEY_CHANGED · public "An update cannot change a document's key"

var DocumentKeyEmpty *errs.Error

DocumentKeyEmpty refuses a document whose key is empty.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.DocumentKeyEmpty.

Code 0.3.80.4 · reason DOCUMENT_KEY_EMPTY · public "A document's key cannot be empty"

var DocumentNotFound *errs.Error

DocumentNotFound is a miss.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.DocumentNotFound.

Code 0.3.80.1 · reason DOCUMENT_NOT_FOUND · public "No document has this key"

var DocumentUndecodable *errs.Error

DocumentUndecodable reports a stored document the type no longer fits.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.DocumentUndecodable.

Code 0.3.80.8 · reason DOCUMENT_UNDECODABLE · public "A stored document does not match its type"

var DocumentUnencodable *errs.Error

DocumentUnencodable refuses a value encoding/json cannot encode.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.DocumentUnencodable.

Code 0.3.80.9 · reason DOCUMENT_UNENCODABLE · public "The document cannot be encoded"

var IndexBroken *errs.Error

IndexBroken refuses documents that break a unique index.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.IndexBroken.

Code 0.3.80.12 · reason INDEX_BROKEN · public "The stored documents break a unique index"

var IndexNotUnique *errs.Error

IndexNotUnique refuses a Lookup on a multi-valued index.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.IndexNotUnique.

Code 0.3.80.7 · reason INDEX_NOT_UNIQUE · public "The index is not unique: read it with Find"

var IndexUnknown *errs.Error

IndexUnknown refuses a read of an undeclared index.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.IndexUnknown.

Code 0.3.80.6 · reason INDEX_UNKNOWN · public "The store has no index of this name"

var KeyTooLong *errs.Error

KeyTooLong refuses a key longer than the SQL store's key columns hold.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.KeyTooLong.

Code 0.3.80.17 · reason KEY_TOO_LONG · public "A key is longer than the store can keep"

var LoadFailed *errs.Error

LoadFailed refuses files that are not a store's.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.LoadFailed.

Code 0.3.80.11 · reason LOAD_FAILED · public "The store's files could not be loaded"

var PersistFailed *errs.Error

PersistFailed reports a write the filesystem refused.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.PersistFailed.

Code 0.3.80.10 · reason PERSIST_FAILED · public "The document could not be written to storage"

var StatementFailed *errs.Error

StatementFailed reports a call the SQL store's database did not complete; the driver's error is joined beside it, its text withheld.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.StatementFailed.

Code 0.3.80.16 · reason STATEMENT_FAILED · public "The store's database did not complete the request"

var StoreClosed *errs.Error

StoreClosed refuses a call after Close.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.StoreClosed.

Code 0.3.80.13 · reason STORE_CLOSED · public "The store is closed"

var StoreMisconfigured *errs.Error

StoreMisconfigured refuses a configuration no store could honour.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.StoreMisconfigured.

Code 0.3.80.14 · reason STORE_MISCONFIGURED · public "The document store cannot be opened as configured"

var UniqueKeyTaken *errs.Error

UniqueKeyTaken refuses a write a unique index cannot file.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.UniqueKeyTaken.

Code 0.3.80.3 · reason UNIQUE_KEY_TAKEN · public "Another document already has this key in a unique index"

var VersionNotFound *errs.Error

VersionNotFound is a miss on a version: never made, or pruned.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.VersionNotFound.

Code 0.3.80.19 · reason VERSION_NOT_FOUND · public "The document keeps no version of this number"

var VersionsNotKept *errs.Error

VersionsNotKept refuses a call on versions to a store that keeps none.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.VersionsNotKept.

Code 0.3.80.18 · reason VERSIONS_NOT_KEPT · public "This store keeps no versions"

var VersionsRewriteRefused *errs.Error

VersionsRewriteRefused refuses what a rewrite's function returned.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.VersionsRewriteRefused.

Code 0.3.80.20 · reason VERSIONS_REWRITE_REFUSED · public "A rewrite of versions may only drop or rewrite the versions it was given"

var WriteUnconfirmed *errs.Error

WriteUnconfirmed reports a write that stands but may not survive a power loss.

Initialised with github.com/kitsunium/sdk/internal/core/data/docstore.WriteUnconfirmed.

Code 0.3.80.15 · reason WRITE_UNCONFIRMED · public "The document was stored, but its durability could not be confirmed"

func Index

func Index[T any](name string, keys func(T) []string) IndexSpec[T]

Index declares an index where a document may have several keys and a key several documents. Empty keys are not indexed.

func Open

func Open[T any](cfg Config[T], indexes ...IndexSpec[T]) (*Store[T], error)

Open builds a store from cfg with the secondary indexes given: in memory without a filesystem, otherwise loaded from it — the snapshot, the overlay replayed on top, the indexes rebuilt. It creates the directories the store lives in, 0700, and writes its files 0600.

func OpenSQL

func OpenSQL[T any](cfg SQLConfig[T], indexes ...IndexSpec[T]) (*SQLStore[T], error)

OpenSQL builds a store over SQL from cfg with the secondary indexes given, declared as for Open. It sends no statement: its tables are SQLMigration's, run by the caller's Migrator before the store is used.

func SQLMigration

func SQLMigration(dialect sql.Dialect, table string, version uint64) (sql.Migration, error)

SQLMigration returns the migration that creates the two tables a SQL store named table keeps on dialect, numbered version for the caller's own version table. Its Down drops both, and every document in them. Every statement does nothing when its table exists, so a run MySQL's implicit commit stopped halfway completes when it runs again.

func SQLVersionsMigration

func SQLVersionsMigration(dialect sql.Dialect, table string, version uint64) (sql.Migration, error)

SQLVersionsMigration returns the migration that creates the table a SQL store named table keeps its versions in on dialect, numbered version for the caller's own version table, beside SQLMigration's. A store opened with SQLConfig.Versions needs it. Its Down drops the table, and every version in it: run it to turn versions off for good.

func Unique

func Unique[T any](name string, key func(T) string) IndexSpec[T]

Unique declares a unique index over the one key key returns. An empty key is not indexed, so any number of documents may have none.

type Announcer

Announcer is the public alias for the pair of hooks both engines answer exactly as they are: OnWrite and OnDelete.

type Announcer = coredocstore.Announcer

Announcer is an alias of github.com/kitsunium/sdk/internal/core/data/docstore.Announcer, declared as:

type Announcer interface {
	// OnDelete registers fn to be told the key of every document Delete
	// removed, on the same terms as OnWrite.
	OnDelete(fn func(key string)) (remove func())
	// OnWrite registers fn to be told the key of every document Put, Insert,
	// Replace or Update stored, and returns the function that removes the
	// registration. A nil fn registers nothing. A panic in fn reaches the
	// writer after its write stood.
	OnWrite(fn func(key string)) (remove func())
}

func (Announcer) OnDelete
func (Announcer) OnDelete(fn func(key string)) (remove func())

OnDelete registers fn to be told the key of every document Delete removed, on the same terms as OnWrite.

func (Announcer) OnWrite
func (Announcer) OnWrite(fn func(key string)) (remove func())

OnWrite registers fn to be told the key of every document Put, Insert, Replace or Update stored, and returns the function that removes the registration. A nil fn registers nothing. A panic in fn reaches the writer after its write stood.

type Collection

Collection is the public alias for the port of a store whose calls take no context, which Store implements: Get, List, Filter, Entries, Lookup and Find; Put, Insert, Replace, Update and Delete; OnWrite and OnDelete. It is what a caller depends on to stand a double in for a Store; Stats, Fold and Close are the engine's own.

type Collection[T any] = coredocstore.Collection[T]

Collection is an alias of github.com/kitsunium/sdk/internal/core/data/docstore.Collection, declared as:

type Collection[T any] interface {
	Delete(key string) error
	Entries(limit int) ([]EntryValue, error)
	Filter(keep func(T) bool) ([]T, error)
	Find(index string, key string) ([]T, error)
	Get(key string) (T, error)
	Insert(v T) error
	List() ([]T, error)
	Lookup(index string, key string) (T, error)
	Put(v T) error
	Replace(v T) error
	Update(key string, fn func(*T) error) (T, error)
	Announcer
}

func (Collection) Delete
func (Collection) Delete(key string) error

Delete removes the document stored under key, or returns DOCUMENT_NOT_FOUND. Its versions go with it.

func (Collection) Entries
func (Collection) Entries(limit int) ([]EntryValue, error)

Entries returns up to limit stored documents as JSON, in key order — every one when limit is not positive — each a copy the caller owns.

func (Collection) Filter
func (Collection) Filter(keep func(T) bool) ([]T, error)

Filter returns the documents keep accepts, in key order. It decodes every document; an index is the way not to.

func (Collection) Find
func (Collection) Find(index string, key string) ([]T, error)

Find returns the documents filed under key in the index named index, in store-key order. It reads any index, unique or not, and no document is an empty slice, not an error.

func (Collection) Get
func (Collection) Get(key string) (T, error)

Get returns the document stored under key, or DOCUMENT_NOT_FOUND.

func (Collection) Insert
func (Collection) Insert(v T) error

Insert stores v, which must be new: DOCUMENT_EXISTS when its key is taken.

func (Collection) List
func (Collection) List() ([]T, error)

List returns every document, in key order.

func (Collection) Lookup
func (Collection) Lookup(index string, key string) (T, error)

Lookup returns the document holding key in the unique index named index: DOCUMENT_NOT_FOUND when none does, INDEX_UNKNOWN for an index the store never declared, INDEX_NOT_UNIQUE for one that may file several.

func (Collection) Put
func (Collection) Put(v T) error

Put stores v under its key, creating it or replacing whatever was there.

func (Collection) Replace
func (Collection) Replace(v T) error

Replace stores v over the document already under its key: DOCUMENT_NOT_FOUND when there is none, so a document deleted meanwhile is never brought back.

func (Collection) Update
func (Collection) Update(key string, fn func(*T) error) (T, error)

Update applies fn to a copy of the document stored under key and stores the result, atomically with respect to every other write, and returns it. An error from fn is returned as it is and changes nothing, and so does a result whose key is not key (DOCUMENT_KEY_CHANGED). fn may read the store and must not write to it.

type CollectionContext

CollectionContext is the public alias for Collection with a context on every call, which SQLStore implements; Count and Reindex are the engine's own.

type CollectionContext[T any] = coredocstore.CollectionContext[T]

CollectionContext is an alias of github.com/kitsunium/sdk/internal/core/data/docstore.CollectionContext, declared as:

type CollectionContext[T any] interface {
	Delete(ctx context.Context, key string) error
	Entries(ctx context.Context, limit int) ([]EntryValue, error)
	Filter(ctx context.Context, keep func(T) bool) ([]T, error)
	Find(ctx context.Context, index string, key string) ([]T, error)
	Get(ctx context.Context, key string) (T, error)
	Insert(ctx context.Context, v T) error
	List(ctx context.Context) ([]T, error)
	Lookup(ctx context.Context, index string, key string) (T, error)
	Put(ctx context.Context, v T) error
	Replace(ctx context.Context, v T) error
	Update(ctx context.Context, key string, fn func(*T) error) (T, error)
	Announcer
}

func (CollectionContext) Delete
func (CollectionContext) Delete(ctx context.Context, key string) error

Delete is Collection.Delete under ctx.

func (CollectionContext) Entries
func (CollectionContext) Entries(ctx context.Context, limit int) ([]EntryValue, error)

Entries is Collection.Entries under ctx.

func (CollectionContext) Filter
func (CollectionContext) Filter(ctx context.Context, keep func(T) bool) ([]T, error)

Filter is Collection.Filter under ctx.

func (CollectionContext) Find
func (CollectionContext) Find(ctx context.Context, index string, key string) ([]T, error)

Find is Collection.Find under ctx.

func (CollectionContext) Get
func (CollectionContext) Get(ctx context.Context, key string) (T, error)

Get is Collection.Get under ctx.

func (CollectionContext) Insert
func (CollectionContext) Insert(ctx context.Context, v T) error

Insert is Collection.Insert under ctx.

func (CollectionContext) List
func (CollectionContext) List(ctx context.Context) ([]T, error)

List is Collection.List under ctx.

func (CollectionContext) Lookup
func (CollectionContext) Lookup(ctx context.Context, index string, key string) (T, error)

Lookup is Collection.Lookup under ctx.

func (CollectionContext) Put
func (CollectionContext) Put(ctx context.Context, v T) error

Put is Collection.Put under ctx.

func (CollectionContext) Replace
func (CollectionContext) Replace(ctx context.Context, v T) error

Replace is Collection.Replace under ctx.

func (CollectionContext) Update
func (CollectionContext) Update(ctx context.Context, key string, fn func(*T) error) (T, error)

Update is Collection.Update under ctx: fn runs inside the write's transaction, and must not write the document it updates.

type Config

Config is the public alias for a store's configuration: Key (required), FS and Path (both, or neither for a memory store), FoldAt, and Versions with its Clock and Held.

type Config[T any] = svcdocstore.Config[T]

Config is an alias of github.com/kitsunium/sdk/internal/service/data/docstore.Config, declared as:

type Config[T any] struct {
	// Key returns the key a document is stored under. It must be a pure
	// function of the document: the store calls it to file every write and
	// compares its answers to refuse an update that would rename a document.
	Key      func(T) string
	// FS is where the store persists, and Path is the snapshot's name inside
	// it — an io/fs name such as "shop/items.json". The overlay lives beside
	// the snapshot, in the directory Path + ".d". A nil FS keeps the store in
	// memory: nothing survives the process, and nothing is written anywhere.
	FS       corevfs.FullFS
	// Clock stamps each version with the instant of the write that made it.
	// Nil is the system clock.
	Clock    clock.Clock
	// Held reports whether the document stored under key is held — a legal
	// hold — so that no write prunes its versions: they pile up beyond
	// Versions until a write finds the document released, and that write
	// prunes them. It is called with the document's key by a write that would
	// prune, under the writers' lock, so it may read this store and must not
	// write to it; a panic in it reaches the writer, and nothing is written.
	// Nil holds nothing. It is refused without Versions.
	Held     func(key string) bool
	// Path is the snapshot's name inside FS. It is required with FS and
	// refused without it, because a path with nowhere to be written is a
	// store that would silently keep nothing.
	Path     string
	// FoldAt is how many overlay entries fold the overlay into the snapshot
	// on the write that reaches it. Zero is the default rule: as many entries
	// as the store holds documents, and at least [DefaultFoldAt]. A negative
	// value never folds on a write; Fold, Open and Close still do.
	FoldAt   int
	// Versions is how many former versions each document keeps beside its
	// current one, the newest: a write that changes a document makes a new
	// version, and prunes the oldest beyond Versions in the same durable
	// write. Zero keeps no versions — the store writes exactly the files it
	// wrote before versions existed — and a negative value is refused. A
	// persistent store keeps them in the file Path + ".versions" once a fold
	// ran, and a store opened with zero over files that keep versions is
	// refused rather than left to drop or corrupt them (ADR 0143).
	Versions int
}

type Entry

Entry is the public alias for one stored document as JSON, with its key.

type Entry = coredocstore.EntryValue

Entry is an alias of github.com/kitsunium/sdk/internal/core/data/docstore.EntryValue, declared as:

type Entry struct {
	// Key is the document's key.
	Key  string
	// JSON is the document, a copy the caller owns.
	JSON json.RawMessage
}

type IndexSpec

IndexSpec is the public alias for one secondary index's declaration, built by Unique or Index and given to Open.

type IndexSpec[T any] = coredocstore.IndexSpec[T]

IndexSpec is an alias of github.com/kitsunium/sdk/internal/core/data/docstore.IndexSpec, declared as:

type IndexSpec[T any] struct {
	// Keys returns a document's keys in the index. Empty keys are not
	// indexed, and a key listed twice is filed once.
	Keys   func(T) []string
	// Name reads the index in Lookup and Find, and names it in a refusal.
	Name   string
	// Unique refuses a write that would file two documents under one key.
	Unique bool
}

type SQLConfig

SQLConfig is the public alias for a SQL store's configuration: Key, Transactor, Dialect and Table (required), IndexKey, and Versions with its Clock and Held.

type SQLConfig[T any] = svcdocstore.SQLConfig[T]

SQLConfig is an alias of github.com/kitsunium/sdk/internal/service/data/docstore.SQLConfig, declared as:

type SQLConfig[T any] struct {
	// Key returns the key a document is stored under, as for [Open]: a pure
	// function of the document, compared byte for byte.
	Key        func(T) string
	// Clock stamps each version with the instant of the write that made it.
	// Nil is the system clock.
	Clock      clock.Clock
	// Held reports whether the document stored under key is held — a legal
	// hold — so that no write prunes its versions, as for [Open]. It is
	// called by a write that would prune, inside that write's transaction
	// and with its context, so a read it makes through the same transactor
	// joins it. Nil holds nothing. It is refused without Versions.
	Held       func(ctx context.Context, key string) bool
	// Transactor is the transaction manager of the database the store lives
	// in — the one the caller's own transactions are opened with, so a call
	// made under one of them joins it. It must also be a core/data/sql Joiner and
	// Deferrer, which the SDK's own transactor is.
	Transactor coresql.Transactor
	// IndexKey, when set, transforms every index key before it reaches the
	// database — on a write, and on every Lookup and Find — so a table holds
	// what it returns and never the key itself. An index is an equality
	// lookup, which a keyed hash keeps: a framework passes an HMAC under a key
	// of its own, and an e-mail address never reaches a table. It is called
	// with the index's name and a non-empty key; an empty answer files
	// nothing, as an empty key does. Nil stores index keys as they are.
	IndexKey   func(index string, key string) []byte
	// Table names the documents' table. The store derives its index table's
	// name from it by appending "___ix". It is a lower-case SQL identifier of
	// at most [MaxSQLTableLen] bytes, holds no three underscores in a row,
	// and does not start with "sqlite_". [SQLMigration] creates both tables.
	Table      string
	// Dialect is the engine's: the SQL the store speaks to it. It must be the
	// Transactor's own.
	Dialect    coresql.Dialect
	// Versions is how many former versions each document keeps beside its
	// current one, as for [Open]: a write that changes a document makes a
	// new version and prunes the oldest beyond Versions, in its own
	// transaction. They live in a third table, <Table>___vs, which
	// [SQLVersionsMigration] creates. Zero keeps no versions and sends
	// exactly the statements the store sent before versions existed; a
	// negative value is refused. The store cannot see a versions table it was
	// told nothing about: turned off, it leaves the table as it is, and a
	// document deleted and written again meanwhile would find the rows of the
	// one it replaced — so drop the table with the migration's Down (ADR 0143).
	Versions   int
}

type SQLStore

SQLStore is the public alias for the document store over SQL: Get, List, Filter, Entries, Count, Lookup and Find read; Put, Insert, Replace, Update and Delete write, and their Stamped forms say what the version they make records; Versions, Version and RewriteVersions as for Store; OnWrite and OnDelete announce once the write's transaction commits; Reindex files the stored documents again. Every call takes a context.

type SQLStore[T any] = svcdocstore.SQLStore[T]

SQLStore is an alias of github.com/kitsunium/sdk/internal/service/data/docstore.SQLStore, declared as:

type SQLStore[T any] struct{}

func (SQLStore) Count
func (*SQLStore) Count(ctx context.Context) (int, error)

Count returns how many documents the store holds.

func (SQLStore) Delete
func (*SQLStore) Delete(ctx context.Context, key string) error

Delete removes the document stored under key, or returns DocumentNotFound. Its versions go with it, in the same transaction.

func (SQLStore) Entries
func (*SQLStore) Entries(ctx context.Context, limit int) (entries []coredocstore.EntryValue, err error)

Entries returns up to limit stored documents as JSON, in key order — every one when limit is not positive — for a caller that shows documents rather than decoding them. Each JSON is the bytes the store wrote.

func (SQLStore) Filter
func (*SQLStore) Filter(ctx context.Context, keep func(T) bool) ([]T, error)

Filter returns the documents keep accepts, in key order. It reads every document; an index is the way not to.

func (SQLStore) Find
func (*SQLStore) Find(ctx context.Context, index string, key string) ([]T, error)

Find returns the documents filed under key in the index named index, in store-key order. It reads any index, unique or not, and no document is an empty slice, not an error.

func (SQLStore) Get
func (*SQLStore) Get(ctx context.Context, key string) (T, error)

Get returns the document stored under key, or DocumentNotFound.

func (SQLStore) Insert
func (*SQLStore) Insert(ctx context.Context, v T) error

Insert stores v, which must be new: DocumentExists when its key is taken.

func (SQLStore) InsertStamped
func (*SQLStore) InsertStamped(ctx context.Context, v T, stamp coredocstore.StampValue) error

InsertStamped is Insert, with what the write says about the version it makes: version 1, whatever stamp.InPlace says.

func (SQLStore) List
func (*SQLStore) List(ctx context.Context) ([]T, error)

List returns every document, in key order — byte order, as Go compares strings.

func (SQLStore) Lookup
func (*SQLStore) Lookup(ctx context.Context, index string, key string) (T, error)

Lookup returns the document holding key in the unique index named index, DocumentNotFound when none does, IndexUnknown for an undeclared index and IndexNotUnique for an index that may hold several: read that one with Find.

func (SQLStore) OnDelete
func (*SQLStore) OnDelete(fn func(key string)) (remove func())

OnDelete registers fn to be told the key of every document Delete removed, on the same terms as OnWrite.

func (SQLStore) OnWrite
func (*SQLStore) OnWrite(fn func(key string)) (remove func())

OnWrite registers fn to be told the key of every document Put, Insert, Replace or Update stored, once the transaction that holds the write has committed — never for a write that is rolled back — on the goroutine that committed it, outside every lock. It returns the function that removes the registration.

func (SQLStore) Put
func (*SQLStore) Put(ctx context.Context, v T) error

Put stores v under its key, creating it or replacing whatever was there. On a store that keeps versions, it makes one, stamped with nothing.

func (SQLStore) PutStamped
func (*SQLStore) PutStamped(ctx context.Context, v T, stamp coredocstore.StampValue) error

PutStamped is Put, with what the write says about the version it makes.

func (SQLStore) Reindex
func (*SQLStore) Reindex(ctx context.Context) error

Reindex rebuilds every index row from the stored documents, in one transaction — a savepoint of the caller's when ctx carries one.

The SQL store keeps its index rows, where the memory and file store rebuild theirs at every Open. So a change to the declarations — an index added, a key function or IndexKey changed, an index made unique — files nothing for the documents already stored until Reindex runs. It drops every row, including those of an index no longer declared, reads the documents in key order a page at a time, and files each one's keys again.

Documents that break a unique index refuse the rebuild with INDEX_BROKEN, naming the index and never the key, and so does a key function that panics; a document the type no longer fits is DOCUMENT_UNDECODABLE. Nothing changes then. Run it when no write to the store runs beside it: a write in flight files its keys under whichever rows it finds.

func (SQLStore) Replace
func (*SQLStore) Replace(ctx context.Context, v T) error

Replace stores v over the document already under its key: DocumentNotFound when there is none, so a document deleted meanwhile is never brought back.

func (SQLStore) ReplaceStamped
func (*SQLStore) ReplaceStamped(ctx context.Context, v T, stamp coredocstore.StampValue) error

ReplaceStamped is Replace, with what the write says about the version it makes.

func (SQLStore) RewriteVersions
func (*SQLStore) RewriteVersions(ctx context.Context, key string, fn func(former []coredocstore.VersionValue) ([]coredocstore.VersionValue, error)) error

RewriteVersions rewrites the former versions of the document stored under key — every version but the current one, which is the document and which only a write changes — in one transaction, a savepoint of the caller's when ctx carries one, the document locked as a write locks it. fn receives copies of them, newest first, and returns those to keep, in the same order, under the rules of Store.RewriteVersions: VersionsRewriteRefused otherwise, and fn's own error returned as it is, nothing changed either way. It calls no hook: the document did not change.

func (SQLStore) Update
func (*SQLStore) Update(ctx context.Context, key string, fn func(*T) error) (T, error)

Update applies fn to a copy of the document stored under key and stores the result, atomically: the document is locked from the read to the write — its row on PostgreSQL and MySQL, the database's write lock on SQLite — so no other write lands between them. An error from fn changes nothing and is returned for errors.Is to find, and a result whose key is not key is DocumentKeyChanged. It returns the stored result.

fn runs inside the write's transaction. It may read the store; a read under a context carrying no transaction runs on the pool, and so needs a second connection. It must not write the document it updates, which is locked by the very write that is waiting for fn.

func (SQLStore) UpdateStamped
func (*SQLStore) UpdateStamped(ctx context.Context, key string, stamp coredocstore.StampValue, fn func(*T) error) (T, error)

UpdateStamped is Update, with what the write says about the version it makes.

func (SQLStore) Version
func (*SQLStore) Version(ctx context.Context, key string, number uint64) (coredocstore.VersionValue, error)

Version returns the version numbered number of the document stored under key, VersionNotFound when the document keeps none of that number — never made, or pruned — and what Versions returns otherwise.

func (SQLStore) Versions
func (*SQLStore) Versions(ctx context.Context, key string) (versions []coredocstore.VersionValue, err error)

Versions returns the versions the store keeps of the document stored under key, newest first: the current one — the document as it is now — then the former ones, read in one statement on the transaction ctx carries. DocumentNotFound when no document is stored under key, VersionsNotKept when the store keeps no versions.

type Stamp

Stamp is the public alias for what a Stamped write says about the version it makes: the Meta the version records, or InPlace — no version.

type Stamp = coredocstore.StampValue

Stamp is an alias of github.com/kitsunium/sdk/internal/core/data/docstore.StampValue, declared as:

type Stamp struct {
	// Meta is kept with the version the write makes — who made it, which
	// command — and read back with it. The store copies it and never reads
	// it.
	Meta    map[string]string
	// InPlace makes the write change the document without making a version:
	// the current version keeps its number, its instant and its Meta, and
	// holds the new document. A framework stamps so a workflow's transition,
	// which its journal already records. A creation is version 1 all the
	// same.
	InPlace bool
}

type Stats

Stats is the public alias for what a store says about itself: documents, pending overlay entries, folds, and the last automatic fold's failure.

type Stats = svcdocstore.StatsValue

Stats is an alias of github.com/kitsunium/sdk/internal/service/data/docstore.StatsValue, declared as:

type Stats struct {
	// FoldError is the last automatic fold's failure, nil once a fold
	// succeeds. The write that triggered the fold succeeded either way — its
	// own entry was already durable — and the entries stay until a fold works.
	FoldError error
	// Documents is how many documents the store holds.
	Documents int
	// Pending is how many overlay entries the snapshot does not contain yet:
	// zero in memory, and after a fold.
	Pending   int
	// Folds is how many folds ran since the store opened, Open's own included.
	Folds     int
}

type Store

Store is the public alias for the document store: Get, List, Filter, Entries, Lookup, Find and Stats read; Put, Insert, Replace, Update and Delete write, and their Stamped forms say what the version they make records; Versions and Version read a document's versions, RewriteVersions rewrites its former ones; OnWrite and OnDelete announce; Fold and Close bring it to rest.

type Store[T any] = svcdocstore.Store[T]

Store is an alias of github.com/kitsunium/sdk/internal/service/data/docstore.Store, declared as:

type Store[T any] struct{}

func (Store) Close
func (*Store) Close() error

Close folds the overlay and closes the store: every later call is StoreClosed, except Stats and Close, which returns nil. A fold that fails leaves its entries where they are — the next Open replays them — and is returned; the store closes either way.

func (Store) Delete
func (*Store) Delete(key string) error

Delete removes the document stored under key, or returns DocumentNotFound. Its versions go with it.

func (Store) Entries
func (*Store) Entries(limit int) ([]coredocstore.EntryValue, error)

Entries returns up to limit stored documents as JSON, in key order — every one when limit is not positive — for a caller that shows documents rather than decoding them. Each JSON is the caller's own copy.

func (Store) Filter
func (*Store) Filter(keep func(T) bool) ([]T, error)

Filter returns the documents keep accepts, in key order. It decodes every document; an index is the way not to.

func (Store) Find
func (*Store) Find(index string, key string) ([]T, error)

Find returns the documents filed under key in the index named index, in store-key order. It reads any index, unique or not, and no document is an empty slice, not an error.

func (Store) Fold
func (*Store) Fold() error

Fold rewrites the snapshot with every document and removes the overlay entries it now contains. A write folds by itself when the overlay has grown enough, and Open and Close fold; Fold is for a caller that wants the resting state now — before a backup, say. A memory store has nothing to fold.

func (Store) Get
func (*Store) Get(key string) (T, error)

Get returns the document stored under key, or DocumentNotFound.

func (Store) Insert
func (*Store) Insert(v T) error

Insert stores v, which must be new: DocumentExists when its key is taken.

func (Store) InsertStamped
func (*Store) InsertStamped(v T, stamp coredocstore.StampValue) error

InsertStamped is Insert, with what the write says about the version it makes: version 1, whatever stamp.InPlace says.

func (Store) List
func (*Store) List() ([]T, error)

List returns every document, in key order.

func (Store) Lookup
func (*Store) Lookup(index string, key string) (T, error)

Lookup returns the document holding key in the unique index named index, DocumentNotFound when none does, IndexUnknown for an undeclared index and IndexNotUnique for an index that may hold several: read that one with Find.

func (Store) OnDelete
func (*Store) OnDelete(fn func(key string)) (remove func())

OnDelete registers fn to be told the key of every document Delete removed, on the same terms as OnWrite.

func (Store) OnWrite
func (*Store) OnWrite(fn func(key string)) (remove func())

OnWrite registers fn to be told the key of every document Put, Insert, Replace or Update stored, once the write is durable, on the writer's goroutine and outside every lock. It returns the function that removes the registration. A panic in fn reaches the writer after its write stood.

func (Store) Put
func (*Store) Put(v T) error

Put stores v under its key, creating it or replacing whatever was there. On a store that keeps versions, it makes one, stamped with nothing.

func (Store) PutStamped
func (*Store) PutStamped(v T, stamp coredocstore.StampValue) error

PutStamped is Put, with what the write says about the version it makes.

func (Store) Replace
func (*Store) Replace(v T) error

Replace stores v over the document already under its key: DocumentNotFound when there is none, so a document deleted meanwhile is never brought back.

func (Store) ReplaceStamped
func (*Store) ReplaceStamped(v T, stamp coredocstore.StampValue) error

ReplaceStamped is Replace, with what the write says about the version it makes.

func (Store) RewriteVersions
func (*Store) RewriteVersions(key string, fn func(former []coredocstore.VersionValue) ([]coredocstore.VersionValue, error)) error

RewriteVersions rewrites the former versions of the document stored under key — every version but the current one, which is the document and which only a write changes — in one durable write, under the writers' lock. fn receives copies of them, newest first, and returns those to keep, in the same order: it may drop a version, and change the document one holds and what its writer said, and nothing else — VersionsRewriteRefused for a version it was not given, out of order, made at another instant or that is not JSON. An error from fn is returned as it is and changes nothing. It calls no hook: the document did not change.

It is what an erasure needs: the former versions cleared as the document is, or dropped. fn runs under the writers' lock, like Update's.

func (Store) Stats
func (*Store) Stats() StatsValue

Stats reports what the store holds and where its overlay stands. It answers after Close too, with the state the store closed in.

func (Store) Update
func (*Store) Update(key string, fn func(*T) error) (T, error)

Update applies fn to a copy of the document stored under key and stores the result, atomically with respect to every other write. An error from fn is returned as it is and changes nothing, and so does a result whose key is not key (DocumentKeyChanged). It returns the stored result.

fn runs under the writers' lock: it may read this store, and must not write to it — the write would wait for the lock fn holds, forever.

func (Store) UpdateStamped
func (*Store) UpdateStamped(key string, stamp coredocstore.StampValue, fn func(*T) error) (T, error)

UpdateStamped is Update, with what the write says about the version it makes.

func (Store) Version
func (*Store) Version(key string, number uint64) (coredocstore.VersionValue, error)

Version returns the version numbered number of the document stored under key, VersionNotFound when the document keeps none of that number — never made, or pruned — and what Versions returns otherwise.

func (Store) Versions
func (*Store) Versions(key string) ([]coredocstore.VersionValue, error)

Versions returns the versions the store keeps of the document stored under key, newest first: the current one — the document as it is now — then the former ones. It returns DocumentNotFound when no document is stored under key, and VersionsNotKept when the store keeps no versions.

type Version

Version is the public alias for one version of a document: its Number from 1, the instant At the write that made it ran, the Meta that write carried, and the document as JSON.

type Version = coredocstore.VersionValue

Version is an alias of github.com/kitsunium/sdk/internal/core/data/docstore.VersionValue, declared as:

type Version struct {
	// At is when the write that made the version ran, in UTC, read from the
	// store's clock and kept to the nanosecond — by the file engine for the
	// years 0 to 9999, which RFC 3339 writes, and by the SQL engine for any
	// instant. It is zero for the first version of a document the store held
	// before it kept versions: when that one was written is not known.
	At     time.Time
	// Meta is what the write's caller said about it — who made it, which
	// command — as its [StampValue] gave it; nil when it said nothing. The
	// store keeps it and never reads it.
	Meta   map[string]string
	// JSON is the document as the version holds it, compact, a copy the
	// caller owns. The newest version's is the document itself.
	JSON   json.RawMessage
	// Number counts a document's versions from 1. No number is given twice
	// while the document exists, pruning included; a document deleted and
	// written again is a new document, and starts again at 1.
	Number uint64
}

type Versioned

Versioned is the public alias for the sibling of Collection that reads and writes a document's versions without a context, which Store implements: the four Stamped writes, Versions, Version and RewriteVersions.

type Versioned[T any] = coredocstore.Versioned[T]

Versioned is an alias of github.com/kitsunium/sdk/internal/core/data/docstore.Versioned, declared as:

type Versioned[T any] interface {
	InsertStamped(v T, stamp StampValue) error
	PutStamped(v T, stamp StampValue) error
	ReplaceStamped(v T, stamp StampValue) error
	RewriteVersions(key string, fn func(former []VersionValue) ([]VersionValue, error)) error
	UpdateStamped(key string, stamp StampValue, fn func(*T) error) (T, error)
	Version(key string, number uint64) (VersionValue, error)
	Versions(key string) ([]VersionValue, error)
}

func (Versioned) InsertStamped
func (Versioned) InsertStamped(v T, stamp StampValue) error

InsertStamped is Insert, with what the write says about the version it makes: version 1, whatever stamp.InPlace says.

func (Versioned) PutStamped
func (Versioned) PutStamped(v T, stamp StampValue) error

PutStamped is Put, with what the write says about the version it makes.

func (Versioned) ReplaceStamped
func (Versioned) ReplaceStamped(v T, stamp StampValue) error

ReplaceStamped is Replace, with what the write says about the version it makes.

func (Versioned) RewriteVersions
func (Versioned) RewriteVersions(key string, fn func(former []VersionValue) ([]VersionValue, error)) error

RewriteVersions rewrites the former versions of the document stored under key in one durable write: fn receives copies of them, newest first, and returns those to keep, in the same order. It may drop a version and change what one holds and what its writer said, and nothing else — VERSIONS_REWRITE_REFUSED otherwise. An error from fn is returned as it is and changes nothing. It calls no hook: the document did not change.

func (Versioned) UpdateStamped
func (Versioned) UpdateStamped(key string, stamp StampValue, fn func(*T) error) (T, error)

UpdateStamped is Update, with what the write says about the version it makes.

func (Versioned) Version
func (Versioned) Version(key string, number uint64) (VersionValue, error)

Version returns the version numbered number of the document stored under key: VERSION_NOT_FOUND when the document keeps none of that number — never made, or pruned.

func (Versioned) Versions
func (Versioned) Versions(key string) ([]VersionValue, error)

Versions returns the versions kept of the document stored under key, newest first: the current one — the document as it is now — then the former ones. DOCUMENT_NOT_FOUND when no document is stored under key.

type VersionedContext

VersionedContext is the public alias for Versioned with a context on every call, which SQLStore implements beside CollectionContext.

type VersionedContext[T any] = coredocstore.VersionedContext[T]

VersionedContext is an alias of github.com/kitsunium/sdk/internal/core/data/docstore.VersionedContext, declared as:

type VersionedContext[T any] interface {
	InsertStamped(ctx context.Context, v T, stamp StampValue) error
	PutStamped(ctx context.Context, v T, stamp StampValue) error
	ReplaceStamped(ctx context.Context, v T, stamp StampValue) error
	RewriteVersions(ctx context.Context, key string, fn func(former []VersionValue) ([]VersionValue, error)) error
	UpdateStamped(ctx context.Context, key string, stamp StampValue, fn func(*T) error) (T, error)
	Version(ctx context.Context, key string, number uint64) (VersionValue, error)
	Versions(ctx context.Context, key string) ([]VersionValue, error)
}

func (VersionedContext) InsertStamped
func (VersionedContext) InsertStamped(ctx context.Context, v T, stamp StampValue) error

InsertStamped is Versioned.InsertStamped under ctx.

func (VersionedContext) PutStamped
func (VersionedContext) PutStamped(ctx context.Context, v T, stamp StampValue) error

PutStamped is Versioned.PutStamped under ctx.

func (VersionedContext) ReplaceStamped
func (VersionedContext) ReplaceStamped(ctx context.Context, v T, stamp StampValue) error

ReplaceStamped is Versioned.ReplaceStamped under ctx.

func (VersionedContext) RewriteVersions
func (VersionedContext) RewriteVersions(ctx context.Context, key string, fn func(former []VersionValue) ([]VersionValue, error)) error

RewriteVersions is Versioned.RewriteVersions under ctx.

func (VersionedContext) UpdateStamped
func (VersionedContext) UpdateStamped(ctx context.Context, key string, stamp StampValue, fn func(*T) error) (T, error)

UpdateStamped is Versioned.UpdateStamped under ctx.

func (VersionedContext) Version
func (VersionedContext) Version(ctx context.Context, key string, number uint64) (VersionValue, error)

Version is Versioned.Version under ctx.

func (VersionedContext) Versions
func (VersionedContext) Versions(ctx context.Context, key string) ([]VersionValue, error)

Versions is Versioned.Versions under ctx.

Generated from docs/api by tools/genindex.

Documentation

Overview

Package docstore is the public facade for the SDK's document store: typed, keyed JSON documents with unique and multi-valued secondary indexes, kept in memory or persisted to a filesystem, every write durable before it returns. 

type Account struct {
    ID    string   `json:"id"`
    Email string   `json:"email"`
    Teams []string `json:"teams"`
}

data, err := vfs.NewOS("/var/lib/app") // or vfs.NewMem(), or nil FS for memory
accounts, err := docstore.Open(
    docstore.Config[Account]{
        Key:  func(a Account) string { return a.ID },
        FS:   data,
        Path: "members/accounts.json",
    },
    docstore.Unique("email", func(a Account) string { return a.Email }),
    docstore.Index("team", func(a Account) []string { return a.Teams }),
)
defer accounts.Close()

err = accounts.Insert(Account{ID: "acc_1", Email: "ada@example.com"})
ada, err := accounts.Lookup("email", "ada@example.com")
red, err := accounts.Find("team", "red")

Documents, keys and the write modes

A document is kept as its JSON encoding, so every read returns a copy and a caller can never alias what the store holds. Its key is what Config.Key returns; an empty key is refused. Put creates or replaces; Insert refuses a key that is taken (DocumentExists); Replace refuses a key that is not (DocumentNotFound), so a document deleted meanwhile is never brought back. Update applies a function to a copy and stores the result atomically with respect to every other write — its function may read the store, and must not write to it. Delete removes.

Indexes

Unique files at most one document per key: a write that would file a second is refused with UniqueKeyTaken and changes nothing. Index files any number. Empty keys are not indexed. Lookup reads a unique index; Find reads any index, in store-key order, and answers an empty slice rather than an error; Filter reads every document. The indexes are rebuilt when the store opens, and documents that break a unique index refuse the open (IndexBroken) rather than answer lies.

No refusal ever quotes a key: a store key is routinely an e-mail address, an index key the hash of a token.

How it persists, and what a write costs

At rest the store is ONE file, the snapshot — a JSON object from key to document, the file a person can open. A write publishes one small file beside it, in the directory Path + ".d", atomically, before it returns — so a write costs one document, whatever the store holds. When that overlay holds as many entries as the store holds documents (at least DefaultFoldAt), the write that got it there folds it back into the snapshot: one rewrite per N writes, a constant on average. Close folds too. Opening replays the overlay over the snapshot, and a crash at any point of a fold loads the same documents. BENCH.md in internal/service/data/docstore has the numbers: a write at 100 000 documents costs what it costs at 100, where rewriting the whole file costs 70 ms.

A publication that fails is PersistFailed and changes nothing. One whose rename happened but whose directory flush failed is WriteUnconfirmed: the write stands — the file shows it, and so does the store — and only its survival across a power loss is in doubt.

Concurrency, and what it is not

A Store is safe for concurrent use, and a reader never waits for a disk: writers are serialised among themselves and do their filesystem work alone, and the documents change only once a write is durable. OnWrite and OnDelete tell a caller about each write once it is durable, outside every lock.

It is one process's store: a second process opening the same files is not detected. It keeps every document in memory. It has no transaction across documents. It is a store for the documents a service owns, not a database.

The same store over SQL

OpenSQL keeps the same documents — the same keys, indexes, write modes, hooks and refusals, the same codes — in two tables of a PostgreSQL, MySQL or SQLite database the caller opened and hands over as a sql.Transactor. Nothing is held in memory: the database is the source of truth, and every call is a round trip that takes a context (ADR 0139). 

tm, err := sql.NewTransactor(sql.Config{DB: pool, Dialect: sql.DialectPostgres, Pool: sql.PoolConfig{MaxOpen: 10}})
create, err := docstore.SQLMigration(sql.DialectPostgres, "members__accounts", 20260927120000) // run by your Migrator
accounts, err := docstore.OpenSQL(
    docstore.SQLConfig[Account]{Key: func(a Account) string { return a.ID },
        Transactor: tm, Dialect: sql.DialectPostgres, Table: "members__accounts"},
    docstore.Unique("email", func(a Account) string { return a.Email }),
)
err = sql.Transact(ctx, tm, func(ctx context.Context, _ sql.Executor) error {
    return accounts.Insert(ctx, Account{ID: "acc_1", Email: "ada@example.com"}) // joins this transaction
})

A call runs on the transaction its context carries for that transactor, and on the pool otherwise. A write runs in a savepoint of the caller's transaction — or a transaction of its own — so it is atomic on its own, and a refused or failed one leaves the caller's transaction usable. Its hooks run once the transaction that holds it has committed, and never for one rolled back.

The document is kept as the bytes the store encoded, never as the engine's JSON type, which reorders members and respells numbers, so it reads back byte for byte. Keys are compared as bytes — binary key columns, no collation — so "a", "A" and "a " are three keys, as they are in Go. A key longer than MaxSQLKeyLen bytes is refused with KeyTooLong. Index keys may reach the table hashed: SQLConfig.IndexKey transforms every one at a write and at every Lookup and Find, and an index is an equality lookup, which a keyed hash keeps. Index rows are kept, not rebuilt at every open: after a declaration changes, SQLStore.Reindex files the stored documents again.

A failure of the database is StatementFailed, which a caller answers with a 503: the driver's error is joined beside it for errors.Is and errors.As, and withheld from its text, because a driver quotes the row a constraint refused and that row holds a key.

The ports, for a double

Store and SQLStore are engines. A caller that must stand a double in for one depends instead on the port it implements: Collection and Versioned for a Store, whose calls take no context, CollectionContext and VersionedContext for a SQLStore, whose every call takes one, and Announcer — OnWrite and OnDelete — for either. Each pair carries the same methods, the same documents and the same refusals; there is no single port over both, because the file engine waits on nothing a context could abandon while the SQL engine waits on a database (ADR 0160).

Versions

A store opened with Config.Versions — or SQLConfig.Versions — keeps the last versions of each document, as a content system keeps a page's revisions (ADR 0143): 

pages, err := docstore.Open(docstore.Config[Page]{Key: Page.Key, FS: data, Path: "cms/pages.json", Versions: 20})
err = pages.PutStamped(page, docstore.Stamp{Meta: map[string]string{"by": userID, "command": "pages.Edit"}})
revisions, err := pages.Versions(page.ID) // newest first: the page as it is, then the 20 before it
third, err := pages.Version(page.ID, revisions[3].Number)

A creation is version 1, and every write that changes the document makes the next one, stamped with the instant of Config.Clock and the metadata the Stamped writes carry — who, which command; the plain writes carry none. A number is never given twice while the document exists. A write storing the JSON already stored makes no version, and neither does one stamped InPlace: the document changes and its current version keeps its number. The former versions beyond Versions are pruned by the write that makes a newer one, in the same durable write as the document — the same overlay entry, the same transaction — so no crash ever leaves the versions ahead of or behind their document. Config.Held keeps a document's versions from pruning, for a legal hold, until a write finds it released. RewriteVersions rewrites a document's former versions under the writers' lock, for an erasure: it may clear or drop them, never renumber or add one. A deletion takes every version with it. Versions are never indexed: Lookup and Find read the current version.

The file store keeps them in memory, in each write's overlay entry and, at rest, in the file Path + ".versions"; a store opened without versions over files that keep them is refused, rather than left to drop them. The SQL store keeps them in a third table, which SQLVersionsMigration creates. Version.JSON is compact JSON; a version predating a change of the type may no longer decode into it, which is why it is JSON and not a T.

Index

Constants

View Source 
const (
	// CodeDocumentNotFound: no document under the key, or nobody holds the
	// key in a unique index.
	CodeDocumentNotFound errs.Code = coredocstore.CodeDocumentNotFound
	// CodeDocumentExists: Insert over a taken key.
	CodeDocumentExists errs.Code = coredocstore.CodeDocumentExists
	// CodeUniqueKeyTaken: a unique index refused the write.
	CodeUniqueKeyTaken errs.Code = coredocstore.CodeUniqueKeyTaken
	// CodeDocumentKeyEmpty: the key function returned the empty string.
	CodeDocumentKeyEmpty errs.Code = coredocstore.CodeDocumentKeyEmpty
	// CodeDocumentKeyChanged: an Update's function changed the key.
	CodeDocumentKeyChanged errs.Code = coredocstore.CodeDocumentKeyChanged
	// CodeIndexUnknown: a read named an index the store never declared.
	CodeIndexUnknown errs.Code = coredocstore.CodeIndexUnknown
	// CodeIndexNotUnique: Lookup on a multi-valued index.
	CodeIndexNotUnique errs.Code = coredocstore.CodeIndexNotUnique
	// CodeDocumentUndecodable: a stored document no longer fits the type.
	CodeDocumentUndecodable errs.Code = coredocstore.CodeDocumentUndecodable
	// CodeDocumentUnencodable: encoding/json refused the value.
	CodeDocumentUnencodable errs.Code = coredocstore.CodeDocumentUnencodable
	// CodePersistFailed: the filesystem refused the write; nothing changed.
	CodePersistFailed errs.Code = coredocstore.CodePersistFailed
	// CodeLoadFailed: the store's files cannot be read or are not a store's.
	CodeLoadFailed errs.Code = coredocstore.CodeLoadFailed
	// CodeIndexBroken: the stored documents break a unique index.
	CodeIndexBroken errs.Code = coredocstore.CodeIndexBroken
	// CodeStoreClosed: a call after Close.
	CodeStoreClosed errs.Code = coredocstore.CodeStoreClosed
	// CodeStoreMisconfigured: Open refused its configuration.
	CodeStoreMisconfigured errs.Code = coredocstore.CodeStoreMisconfigured
	// CodeWriteUnconfirmed: the write stands, its durability is in doubt.
	CodeWriteUnconfirmed errs.Code = coredocstore.CodeWriteUnconfirmed
	// CodeStatementFailed: the SQL store's database did not complete a call.
	CodeStatementFailed errs.Code = coredocstore.CodeStatementFailed
	// CodeKeyTooLong: a key is longer than the SQL store's key columns hold.
	CodeKeyTooLong errs.Code = coredocstore.CodeKeyTooLong
	// CodeVersionsNotKept: a call on versions to a store that keeps none.
	CodeVersionsNotKept errs.Code = coredocstore.CodeVersionsNotKept
	// CodeVersionNotFound: the document keeps no version of that number.
	CodeVersionNotFound errs.Code = coredocstore.CodeVersionNotFound
	// CodeVersionsRewriteRefused: a rewrite returned what a rewrite may not.
	CodeVersionsRewriteRefused errs.Code = coredocstore.CodeVersionsRewriteRefused
)

The codes of the store's refusals (0.3.80.*), for errs.HasCode.

View Source 
const DefaultFoldAt int = svcdocstore.DefaultFoldAt

DefaultFoldAt is the least number of overlay entries that folds the overlay into the snapshot when Config.FoldAt is zero.

View Source 
const MaxSQLIndexNameLen int = svcdocstore.MaxSQLIndexNameLen

MaxSQLIndexNameLen is the longest index name a SQL store accepts, in bytes.

View Source 
const MaxSQLKeyLen int = svcdocstore.MaxSQLKeyLen

MaxSQLKeyLen is the longest store key, and the longest index key as it reaches the table, a SQL store accepts, in bytes.

View Source 
const MaxSQLTableLen int = svcdocstore.MaxSQLTableLen

MaxSQLTableLen is the longest table name OpenSQL and SQLMigration accept, in bytes: PostgreSQL's limit of 63, less what the store appends to derive its index table's name.

Variables

View Source 
var (
	// DocumentNotFound is a miss.
	DocumentNotFound = coredocstore.DocumentNotFound
	// DocumentExists refuses an insertion over a taken key.
	DocumentExists = coredocstore.DocumentExists
	// UniqueKeyTaken refuses a write a unique index cannot file.
	UniqueKeyTaken = coredocstore.UniqueKeyTaken
	// DocumentKeyEmpty refuses a document whose key is empty.
	DocumentKeyEmpty = coredocstore.DocumentKeyEmpty
	// DocumentKeyChanged refuses an update that renames.
	DocumentKeyChanged = coredocstore.DocumentKeyChanged
	// IndexUnknown refuses a read of an undeclared index.
	IndexUnknown = coredocstore.IndexUnknown
	// IndexNotUnique refuses a Lookup on a multi-valued index.
	IndexNotUnique = coredocstore.IndexNotUnique
	// DocumentUndecodable reports a stored document the type no longer fits.
	DocumentUndecodable = coredocstore.DocumentUndecodable
	// DocumentUnencodable refuses a value encoding/json cannot encode.
	DocumentUnencodable = coredocstore.DocumentUnencodable
	// PersistFailed reports a write the filesystem refused.
	PersistFailed = coredocstore.PersistFailed
	// LoadFailed refuses files that are not a store's.
	LoadFailed = coredocstore.LoadFailed
	// IndexBroken refuses documents that break a unique index.
	IndexBroken = coredocstore.IndexBroken
	// StoreClosed refuses a call after Close.
	StoreClosed = coredocstore.StoreClosed
	// StoreMisconfigured refuses a configuration no store could honour.
	StoreMisconfigured = coredocstore.StoreMisconfigured
	// WriteUnconfirmed reports a write that stands but may not survive a
	// power loss.
	WriteUnconfirmed = coredocstore.WriteUnconfirmed
	// StatementFailed reports a call the SQL store's database did not
	// complete; the driver's error is joined beside it, its text withheld.
	StatementFailed = coredocstore.StatementFailed
	// KeyTooLong refuses a key longer than the SQL store's key columns hold.
	KeyTooLong = coredocstore.KeyTooLong
	// VersionsNotKept refuses a call on versions to a store that keeps none.
	VersionsNotKept = coredocstore.VersionsNotKept
	// VersionNotFound is a miss on a version: never made, or pruned.
	VersionNotFound = coredocstore.VersionNotFound
	// VersionsRewriteRefused refuses what a rewrite's function returned.
	VersionsRewriteRefused = coredocstore.VersionsRewriteRefused
)

The store's sentinels, for errors.Is.

Functions

func SQLMigration 

func SQLMigration(dialect sql.Dialect, table string, version uint64) (sql.Migration, error)

SQLMigration returns the migration that creates the two tables a SQL store named table keeps on dialect, numbered version for the caller's own version table. Its Down drops both, and every document in them. Every statement does nothing when its table exists, so a run MySQL's implicit commit stopped halfway completes when it runs again.

func SQLVersionsMigration 

func SQLVersionsMigration(dialect sql.Dialect, table string, version uint64) (sql.Migration, error)

SQLVersionsMigration returns the migration that creates the table a SQL store named table keeps its versions in on dialect, numbered version for the caller's own version table, beside SQLMigration's. A store opened with SQLConfig.Versions needs it. Its Down drops the table, and every version in it: run it to turn versions off for good.

Types

type Announcer 

type Announcer = coredocstore.Announcer

Announcer is the public alias for the pair of hooks both engines answer exactly as they are: OnWrite and OnDelete.

type Collection 

type Collection[T any] = coredocstore.Collection[T]

Collection is the public alias for the port of a store whose calls take no context, which Store implements: Get, List, Filter, Entries, Lookup and Find; Put, Insert, Replace, Update and Delete; OnWrite and OnDelete. It is what a caller depends on to stand a double in for a Store; Stats, Fold and Close are the engine's own.

type CollectionContext 

type CollectionContext[T any] = coredocstore.CollectionContext[T]

CollectionContext is the public alias for Collection with a context on every call, which SQLStore implements; Count and Reindex are the engine's own.

type Config 

type Config[T any] = svcdocstore.Config[T]

Config is the public alias for a store's configuration: Key (required), FS and Path (both, or neither for a memory store), FoldAt, and Versions with its Clock and Held.

type Entry 

type Entry = coredocstore.EntryValue

Entry is the public alias for one stored document as JSON, with its key.

type IndexSpec 

type IndexSpec[T any] = coredocstore.IndexSpec[T]

IndexSpec is the public alias for one secondary index's declaration, built by Unique or Index and given to Open.

func Index 

func Index[T any](name string, keys func(T) []string) IndexSpec[T]

Index declares an index where a document may have several keys and a key several documents. Empty keys are not indexed.

func Unique 

func Unique[T any](name string, key func(T) string) IndexSpec[T]

Unique declares a unique index over the one key key returns. An empty key is not indexed, so any number of documents may have none.

type SQLConfig 

type SQLConfig[T any] = svcdocstore.SQLConfig[T]

SQLConfig is the public alias for a SQL store's configuration: Key, Transactor, Dialect and Table (required), IndexKey, and Versions with its Clock and Held.

type SQLStore 

type SQLStore[T any] = svcdocstore.SQLStore[T]

SQLStore is the public alias for the document store over SQL: Get, List, Filter, Entries, Count, Lookup and Find read; Put, Insert, Replace, Update and Delete write, and their Stamped forms say what the version they make records; Versions, Version and RewriteVersions as for Store; OnWrite and OnDelete announce once the write's transaction commits; Reindex files the stored documents again. Every call takes a context.

func OpenSQL 

func OpenSQL[T any](cfg SQLConfig[T], indexes ...IndexSpec[T]) (*SQLStore[T], error)

OpenSQL builds a store over SQL from cfg with the secondary indexes given, declared as for Open. It sends no statement: its tables are SQLMigration's, run by the caller's Migrator before the store is used.

type Stamp 

type Stamp = coredocstore.StampValue

Stamp is the public alias for what a Stamped write says about the version it makes: the Meta the version records, or InPlace — no version.

type Stats 

type Stats = svcdocstore.StatsValue

Stats is the public alias for what a store says about itself: documents, pending overlay entries, folds, and the last automatic fold's failure.

type Store 

type Store[T any] = svcdocstore.Store[T]

Store is the public alias for the document store: Get, List, Filter, Entries, Lookup, Find and Stats read; Put, Insert, Replace, Update and Delete write, and their Stamped forms say what the version they make records; Versions and Version read a document's versions, RewriteVersions rewrites its former ones; OnWrite and OnDelete announce; Fold and Close bring it to rest.

func Open 

func Open[T any](cfg Config[T], indexes ...IndexSpec[T]) (*Store[T], error)

Open builds a store from cfg with the secondary indexes given: in memory without a filesystem, otherwise loaded from it — the snapshot, the overlay replayed on top, the indexes rebuilt. It creates the directories the store lives in, 0700, and writes its files 0600.

type Version 

type Version = coredocstore.VersionValue

Version is the public alias for one version of a document: its Number from 1, the instant At the write that made it ran, the Meta that write carried, and the document as JSON.

type Versioned 

type Versioned[T any] = coredocstore.Versioned[T]

Versioned is the public alias for the sibling of Collection that reads and writes a document's versions without a context, which Store implements: the four Stamped writes, Versions, Version and RewriteVersions.

type VersionedContext 

type VersionedContext[T any] = coredocstore.VersionedContext[T]

VersionedContext is the public alias for Versioned with a context on every call, which SQLStore implements beside CollectionContext.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.