Documentation
¶
Overview ¶
Package docdb maintains indexed SQLite views of document streams.
The unit of storage is a JSON document. A doc table keeps each document whole in its raw column and exposes it relationally. A few real columns, such as the primary key and the update clock, are extracted by the write statement itself, and every other column is GENERATED from raw, so a column can never disagree with the document it derives from. Writes are guarded upserts that keep the newer document, which makes them idempotent and order-independent: replaying a write, or applying old and new in either order, converges on the same row.
A derived table is the other kind: a SELECT over the database's other tables, materialized once when the database is built.
Changes travel between databases as segments, gzip JSONL files of bare documents that replay through the same guarded upserts. Cache serves a local copy of a published database, applying new segments as they land and downloading the database afresh when it is replaced wholesale.
Index ¶
- func ApplyDocs(db *sqlite3.Conn, td TableDef, docs []json.RawMessage) (err error)
- func ApplySegment(db *sqlite3.Conn, src billy.Basic, defs []TableDef, name string) error
- func Doc(path string) string
- func DocTime(path string) string
- func EnsureDocTables(db *sqlite3.Conn, defs []TableDef) error
- func ListSegments(src billy.Filesystem, prefix, start string) ([]string, error)
- func OneofKey(path string) string
- func Raw(path string) string
- func RawSeconds(path string) string
- func RawTime(path string) string
- func SegmentName(prefix string, t time.Time) string
- func SegmentTime(name string) (time.Time, error)
- func StoreDocs(db *sqlite3.Conn, td TableDef, docs []json.RawMessage) error
- func StoreQuery(db *sqlite3.Conn, td TableDef) (int, error)
- func WriteSegment(dest billy.Basic, prefix string, t time.Time, defs []TableDef, ...) (_ string, err error)
- type Cache
- type Col
- type GenCol
- type TableDef
- type Upstream
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ApplySegment ¶
ApplySegment applies one of src's segments to db with the same guarded upserts the table writes use, so replaying a segment, applying one twice, or applying out of order can never regress a row.
func EnsureDocTables ¶
EnsureDocTables creates the doc tables among defs (and their indexes) that db does not already have, making an empty or partial database writable in place. Derived tables are skipped: they materialize from queries, not writes. NOTE: Existing tables are not reconciled with defs. Columns added to the registry appear only on the next full rebuild, since SQLite cannot ALTER in a STORED generated column.
func ListSegments ¶
func ListSegments(src billy.Filesystem, prefix, start string) ([]string, error)
ListSegments returns the segment names src holds under prefix at or after start (a segment name, empty for all of them) in write order. A prefix that does not exist yet lists empty.
func OneofKey ¶
OneofKey names the single field present in a oneof object of the document being written (empty when none): the oneof's JSON tags double as labels.
func RawSeconds ¶
RawSeconds converts a Go duration field (integer nanoseconds) to seconds.
func SegmentName ¶
SegmentName returns the object name for a segment written at t under a destination prefix. Owners version the prefix with their schema so a reader only ever consumes segments matching its base's era.
func SegmentTime ¶
SegmentTime parses a segment object name back to its write time.
func StoreDocs ¶
StoreDocs creates td's doc table, upserts the given documents, and builds its declared indexes.
func StoreQuery ¶
StoreQuery materializes a derived table from its defining query and builds its declared indexes, returning the row count. Materializing at build keeps reads cheap and snapshot-consistent. It refreshes only by rebuild.
func WriteSegment ¶
func WriteSegment(dest billy.Basic, prefix string, t time.Time, defs []TableDef, tables map[string][]json.RawMessage) (_ string, err error)
WriteSegment writes documents (doc table name to document list) into dest as one gzip JSONL segment named for t under prefix. When every list is empty nothing is written and the returned name is empty.
Types ¶
type Cache ¶
type Cache struct {
// contains filtered or unexported fields
}
Cache serves an updated local copy of an upstream database. Contents are refreshed at the specified interval, applying new delta segments incrementally and rehydrating wholesale when a newer full base becomes available. Query serializes access to the current connection.
func OpenCache ¶
OpenCache hydrates a cache of up's base object and refreshes it in the background every interval until Close. A base from any era but up.Schema is refused.
type Col ¶
Col is a real column of a doc table, extracted from the bound document by the write statement itself. Unlike generated columns, real columns can be primary keys and carry the upsert guard.
type GenCol ¶
GenCol is a column generated from the stored raw document. Stored columns cost disk and compute at write when they are updated. Virtual ones cost nothing on write and compute on read, so columns in hot filters or orderings should be stored (or indexed, which also stores the computed values).
type TableDef ¶
type TableDef struct {
Name string
Cols []Col
PK []string
GenCols []GenCol
Query string
Indexes [][]string
}
TableDef describes one table. A table is either a doc table (Cols set: real columns extracted from each written document, an implicit raw column holding the document, and optionally GenCols computed from raw) or a derived table (Query set: a SELECT over the database's other tables, materialized by StoreQuery). Indexes lists the column sets to index after load. A real column named updated becomes the upsert guard clock, and without one replays are last-write-wins. NOTE: Names and expressions are trusted SQL fragments. They should be exclusively sourced from compile-time registries.
type Upstream ¶
type Upstream struct {
FS billy.Filesystem
Object string
Deltas string
Defs []TableDef
Schema int
Watermark func(*sqlite3.Conn) (time.Time, error)
}
Upstream names the published database a Cache follows: the filesystem holding it, its gzip base object, the prefix its delta segments land under, the doc tables those segments replay into, and the schema era this binary reads. Watermark reads the time through which the base is complete, which is where segment replay resumes.
Source Files