Documentation
¶
Overview ¶
Package store owns pi-durable's SQLite format inside the core: the SQL table, the rows every commit writes (CONTRACT section 3.1, row identity with pi-durable main), the single-writer guard, the in-memory metadata, cold open of the live state (ADR D5), loads of committed non-live records, and the pig_ sidecar (ADR D6). Transcripts (entries, forks, head markers, context derivation) live in package history; the store writes entry rows that history encodes and serves the reads history asks for (history.Need).
The store never performs I/O. It emits statements into the session's abi.Builder: commits, and reads that the host answers with a rows event. A caller that needs state the store has not loaded calls the matching Ensure method; when it returns ErrPending the caller stops handling its event, the root core keeps the event, and re-delivers it after the rows arrive (see core.Session). No other package issues SQL.
Rows written per commit are exactly pi-durable main's (src/storage/sqlite/storage.ts commit): the same statements in the same order, except that reads Pi makes (record_ids lookups, the metadata read, existence checks) are answered from the store's own state, and `INSERT OR IGNORE INTO record_ids` is left out for a task or submission the store has already claimed. The metadata update carries the single-writer guard `AND next_seq = ?` (ADR D13).
Owner: dcore-store. Consumers: session (all writes and loads), turn (through session).
Index ¶
- Constants
- Variables
- func SQL() []string
- type Dump
- type DumpEntry
- type DumpSubmission
- type DumpTask
- type JSON
- type Live
- type LiveDocument
- type Options
- type RejectedError
- type RequestError
- type Revision
- type SchemaNewerError
- type Sidecar
- type Store
- func (s *Store) Begin() *Tx
- func (s *Store) Commit(b *abi.Builder, tx *Tx) (seq int64, err error)
- func (s *Store) EnsureDocument(b *abi.Builder, id int64, at int64) (*LiveDocument, error)
- func (s *Store) EnsureScope(b *abi.Builder, kind rec.ScopeKind, owner int64) ([]*LiveDocument, error)
- func (s *Store) EnsureTask(b *abi.Builder, id int64) (*rec.Task, error)
- func (s *Store) Footprint() (docs uint64)
- func (s *Store) ForgetScope(kind rec.ScopeKind, owner int64)
- func (s *Store) Idle(b *abi.Builder)
- func (s *Store) Load(b *abi.Builder, h *history.Store, need *history.Need) error
- func (s *Store) LoadAll(b *abi.Builder, live bool) (*Dump, error)
- func (s *Store) MintID() int64
- func (s *Store) NextID() int64
- func (s *Store) NextSeq() int64
- func (s *Store) ObserveNextID(next int64)
- func (s *Store) Open(b *abi.Builder) (*Live, error)
- func (s *Store) ReleaseDump()
- func (s *Store) Rows(b *abi.Builder, answers []abi.ReadRows) error
- type Tx
- func (tx *Tx) Conversation(c *rec.Conversation)
- func (tx *Tx) ConversationRow(row history.ConversationRow)
- func (tx *Tx) DocumentBase(id int64, version int, base JSON)
- func (tx *Tx) DocumentCopy(d *rec.Document, source int64, at int64)
- func (tx *Tx) DocumentCreate(d *rec.Document, version int, base JSON)
- func (tx *Tx) DocumentDelta(id int64, version int, ops []byte)
- func (tx *Tx) DocumentRetire(id int64)
- func (tx *Tx) Entry(id, conversationID, head int64, record []byte)
- func (tx *Tx) EntryRow(row history.EntryRow)
- func (tx *Tx) RecordText(k int) []byte
- func (tx *Tx) Submission(sub *rec.Submission, text []byte)
- func (tx *Tx) Task(t *rec.Task, text []byte)
Constants ¶
const ( TextContextSummary = "SELECT pig_context_summary(?, ?, ?, ?, ?)" TextContextMessages = "SELECT pig_context_messages(?, ?, ?, ?)" TextContextRelease = "SELECT pig_context_release(?)" )
The texts of the context reads. A host that answers them from the context it derived compares a statement's text with these, as it reads the table the core published (a probe core publishes its own).
const ( // SQLTaskByID reads one task's record. Parameters: id. SQLTaskByID = stmtTaskByID // SQLSubmissionByID reads one submission's record. Parameters: id. SQLSubmissionByID = stmtSubmissionByID // SQLSubmissionByRequest reads a conversation's submission with a request ID. Parameters: conversation id, // JSON.stringify(requestId). SQLSubmissionByRequest = stmtSubmissionByRequest // SQLTasksPage and SQLSubmissionsPage read every task or submission a page at a time, as the dump does. Parameters: the // id the page follows. SQLTasksPage = stmtDumpTasks SQLSubmissionsPage = stmtDumpSubmissions // SQLTasksDesc reads a page of tasks newest first, with the columns of SQLTasksPage. Parameters: the id the page precedes. SQLTasksDesc = stmtTasksDesc // SQLRecordType reads the record_ids row of an id. Parameters: id. SQLRecordType = stmtRecordType // SQLEntryByID reads one entry: conversation_id, commit_seq, record. Parameters: id. SQLEntryByID = stmtEntryByID // SQLEntryIDsDesc reads a page of a conversation's entry ids in a range, newest first. Parameters: conversation id, // lowest id, highest id, page size. SQLEntryIDsDesc = stmtEntryIDsDesc // SQLEntryBounds is history.SQLBounds: a conversation's newest entry, newest head marker and its head at or below a // bound. SQLEntryBounds = stmtBounds // SQLEntriesPage reads a page of a conversation's entries in an id range: id, commit_seq, record. Parameters: // conversation id, the id the page follows, highest id. SQLEntriesPage = stmtEntriesPage // SQLEntryCount counts a conversation's entries in an id range, so a load reserves its index once. Parameters: // conversation id, lowest id, highest id. SQLEntryCount = stmtEntryCount // SQLEntriesDesc reads a page of a conversation's entries in a range, newest first: id, commit_seq, record. Parameters: // conversation id, lowest id, highest id, page size. SQLEntriesDesc = stmtEntriesDesc // SQLContextSummary is the host's summary of a conversation's context through an entry: what a request preparation reads // of it. Parameters: conversation id, the newest entry (tail), the newest head marker at or below it (0: none) and that // marker's head, 1 when the estimate of the compaction check is wanted else 0. One row, one BLOB column: the summary JSON // `{tail, head, entries?, system, tokens}`. SQLContextSummary = stmtContextSummary // SQLContextMessages is the host's model messages of the same context. Parameters: conversation id, tail, marker, // marker's head. One row, one BLOB column: the JSON array of the messages. SQLContextMessages = stmtContextMessages // SQLContextRelease tells the host the conversation's kept context is dropped. Parameter: conversation id. No rows. SQLContextRelease = stmtContextRelease // PageRows is how many rows one page read returns at most. PageRows = dumpPage )
The statements a core with a synchronous host read (ABI section 4.2, pig.read) runs when a step needs a record it did not load at open. Each reads the columns its dump statement reads, so one row decoder serves both.
const SupportedSchemaVersion = 2
SupportedSchemaVersion is the newest durable_schema.version this build writes and reads (migrations.ts CURRENT_SQLITE_SCHEMA_VERSION).
Variables ¶
var ErrPending = errors.New("store: waiting for rows")
ErrPending means the store emitted reads the caller's operation depends on. Read-only.
Functions ¶
Types ¶
type Dump ¶
type Dump struct {
Conversations []*jv.Object
Entries []DumpEntry
Tasks []DumpTask
Submissions []DumpSubmission
Documents []*jv.Object
Revisions map[int64][]Revision // by document ID, in seq order
}
Dump is every record of the store, in ID order: what LoadAll reads after cold open.
type DumpEntry ¶
DumpEntry is one entries row. Raw is the stored record text, which the dump holds without parsing it: opening a long history costs a copy of its bytes, not a tree per record.
type DumpSubmission ¶
type DumpSubmission struct {
ID, Conversation int64
Request string
HasRequest bool
Status string
Raw []byte
}
DumpSubmission is one submissions row: Request is the requestId, when the row has one.
type DumpTask ¶
type DumpTask struct {
ID, Conversation int64
Kind, Status string
Abort, Background bool
Raw []byte
}
DumpTask is one tasks row: the columns every scan filters on, and the stored record text.
type JSON ¶
type JSON = any
JSON is a value of package jv (nil, bool, float64, string, []any or *jv.Object; ECMAScript property order).
type Live ¶
type Live struct {
NextID, NextSeq int64
Conversations []*rec.Conversation // the ones open needs: root and every conversation of a live task or submission
Tasks []*rec.Task // status != terminal, ID order
Submissions []*rec.Submission // queued or placed, ID order
Documents []*LiveDocument // current, alive documents of the session, the loaded conversations and the live tasks
}
Live is the live state cold open loads (ADR D5) and hands to the session once: everything non-terminal.
type LiveDocument ¶
LiveDocument is one materialised document (base plus delta tail, spec §10 document()).
type Options ¶
type Options struct {
Sidecar Sidecar
CoreID string // layout identity the sidecar must carry; empty means the build's own
Debug bool // rebuild from rows and compare on every cold open (RISKS R11)
EmptyNew bool // an empty database is initialised with Pi's schema (migrations.ts) instead of rejected
}
Options are the open-time choices the open event carries.
type RejectedError ¶
type RejectedError struct{ Message string }
RejectedError is pi-durable's StorageRejected: a commit the store refused before it changed anything.
func (*RejectedError) Error ¶
func (e *RejectedError) Error() string
type RequestError ¶
type RequestError struct {
// contains filtered or unexported fields
}
RequestError is Pi's StorageRequestError: a read the caller's request got wrong, which fails only that read.
func (*RequestError) Error ¶
func (e *RequestError) Error() string
type Revision ¶
type Revision struct {
Seq int64
Base bool
Version int
Content any // the base value, or the delta's Chord Op[] JSON value
}
Revision is one document_revisions row.
type SchemaNewerError ¶
type SchemaNewerError struct{ Version int64 }
SchemaNewerError is the refusal applySqliteMigrations gives a store whose durable_schema.version is newer than SupportedSchemaVersion.
func (SchemaNewerError) Error ¶
func (e SchemaNewerError) Error() string
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is one Session's view of its database.
func (*Store) Commit ¶
Commit validates tx as SqliteStorage.commit does (ID claims, document actions; a failure is Pi's StorageRejected and changes nothing), emits one abi.Commit with Pi's rows in Pi's order and the guarded metadata update, applies the commit to the store's in-memory state (next_id, next_seq, documents), and returns the sequence it consumed. It returns ErrPending, having emitted reads and no commit, when a document copy needs a source the store has not loaded; the caller returns it unchanged and runs the whole handler again after the rows.
func (*Store) EnsureDocument ¶
EnsureDocument returns incarnation id materialised at a point (at <= 0 is "current"), for a historical read or a document of a scope the store has not loaded. A nil document with a nil error means absent or not alive at the point. ErrPending means reads were emitted.
func (*Store) EnsureScope ¶
func (s *Store) EnsureScope(b *abi.Builder, kind rec.ScopeKind, owner int64) ([]*LiveDocument, error)
EnsureScope loads the live (current, not retired) documents of a scope: the Session, a conversation or a task. It returns ErrPending after emitting reads, and the documents, ordered by ID, once every revision has arrived. A scope the store created itself (a new conversation or task) is loaded and empty.
func (*Store) EnsureTask ¶
EnsureTask loads a committed task that is not live (a terminal task for getTask or outcomes). ErrPending when reads were emitted. A task the store has never seen and the database lacks returns (nil, nil).
func (*Store) ForgetScope ¶
ForgetScope drops the loaded documents of a scope that is no longer in use (an unloaded conversation).
func (*Store) Idle ¶
Idle is called by the session at an idle boundary (a run settled, nothing live): the store writes due sidecar chunks as a sidecar-only commit (seq -1).
func (*Store) Load ¶
Load emits the reads of a history.Need (the transcript range history asks for) and returns ErrPending; the rows return through Rows, which hands them to history.Store.AddRow and SupplyBounds.
func (*Store) LoadAll ¶
LoadAll reads every Pi table after Open and seeds the store's commit validation with it (record_ids claims of live tasks and submissions, every conversation, every document incarnation with its current value). With live set it reads no entries and only the live tasks and submissions: a core that reads the rest on demand (ABI section 4.2, pig.read). It returns ErrPending after emitting reads, and the dump once every table has arrived. A store Open initialised needs no reads.
func (*Store) MintID ¶
MintID allocates from the global ID namespace like SqliteStorage.mintId: an in-memory counter that the next commit persists as max(stored next_id, counter, max written id + 1).
func (*Store) ObserveNextID ¶
ObserveNextID raises the in-memory ID counter to a caller's: the Session mints IDs itself, and the next commit persists max(stored next_id, the counter, max written id + 1).
func (*Store) Open ¶
Open runs cold open: it emits the first reads into b and returns ErrPending, or returns the loaded Live once the rows it asked for have been delivered through Rows. A newer schema returns SchemaNewerError. Open reads the live state only: the metadata row, the non-terminal tasks and submissions, the conversations they live in and the live documents of the Session, those conversations and those tasks (ADR D5). Transcripts load per conversation through Load.
func (*Store) ReleaseDump ¶
func (s *Store) ReleaseDump()
ReleaseDump drops the rows LoadAll gathered, once the caller has taken what it keeps from them.
type Tx ¶
type Tx struct {
// contains filtered or unexported fields
}
Tx is one Pi commit under construction: the StorageWrite batch of spec §10, in call order.
func (*Tx) Conversation ¶
func (tx *Tx) Conversation(c *rec.Conversation)
Conversation stages a conversation insert (the "conversation" StorageWrite).
func (*Tx) ConversationRow ¶
func (tx *Tx) ConversationRow(row history.ConversationRow)
ConversationRow stages a conversation insert from the bytes history.AppendConversation produced.
func (*Tx) DocumentBase ¶
DocumentBase stores a new complete base of an existing incarnation (the "document.change" StorageWrite, base).
func (*Tx) DocumentCopy ¶
DocumentCopy copies a source at a point (a fork; the "document.copy" StorageWrite). at <= 0 selects "current".
func (*Tx) DocumentCreate ¶
DocumentCreate writes a new incarnation with its base (the "document.create" StorageWrite); d is the DocumentCreate record, without createdAt and retiredAt.
func (*Tx) DocumentDelta ¶
DocumentDelta stores a delta of an existing incarnation; ops is Chord Op[] JSON (the "document.change" write, delta).
func (*Tx) DocumentRetire ¶
DocumentRetire retires an incarnation (the "document.retire" StorageWrite).
func (*Tx) Entry ¶
Entry writes one entry row; record is the bytes history.AppendEntryRecord produced. After Commit, the session appends the same record to the history.Store with the returned seq.
func (*Tx) EntryRow ¶
EntryRow stages an entry insert from history.NewEntryRow, which reads the id, conversation and head columns from the record the way SqliteStorage binds them (`head ?? null`).
func (*Tx) RecordText ¶
RecordText returns the record text the k-th conversation, entry, task or submission write staged in tx binds, in call order: an entry's, task's or submission's JSON text, nil for a conversation. The mirror keeps it as the row's stored text (mirror.Commit), so a commit needs no list of the texts of its own.
func (*Tx) Submission ¶
func (tx *Tx) Submission(sub *rec.Submission, text []byte)
Submission stages a submission upsert (the "submission" StorageWrite); text is the record's JSON text (rec.Submission.Append).