store

package
v0.5.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 22, 2026 License: MIT Imports: 8 Imported by: 0

Documentation

Overview

Package store owns pi-durable's SQLite format inside the core: the SQL table, the rows every commit writes (CONTRACT section 3.1, row identity with pi-durable main), the single-writer guard, the in-memory metadata, cold open of the live state (ADR D5), loads of committed non-live records, and the pig_ sidecar (ADR D6). Transcripts (entries, forks, head markers, context derivation) live in package history; the store writes entry rows that history encodes and serves the reads history asks for (history.Need).

The store never performs I/O. It emits statements into the session's abi.Builder: commits, and reads that the host answers with a rows event. A caller that needs state the store has not loaded calls the matching Ensure method; when it returns ErrPending the caller stops handling its event, the root core keeps the event, and re-delivers it after the rows arrive (see core.Session). No other package issues SQL.

Rows written per commit are exactly pi-durable main's (src/storage/sqlite/storage.ts commit): the same statements in the same order, except that reads Pi makes (record_ids lookups, the metadata read, existence checks) are answered from the store's own state, and `INSERT OR IGNORE INTO record_ids` is left out for a task or submission the store has already claimed. The metadata update carries the single-writer guard `AND next_seq = ?` (ADR D13).

Owner: dcore-store. Consumers: session (all writes and loads), turn (through session).

Index

Constants

View Source
const (
	TextContextSummary  = "SELECT pig_context_summary(?, ?, ?, ?, ?)"
	TextContextMessages = "SELECT pig_context_messages(?, ?, ?, ?)"
	TextContextRelease  = "SELECT pig_context_release(?)"
)

The texts of the context reads. A host that answers them from the context it derived compares a statement's text with these, as it reads the table the core published (a probe core publishes its own).

View Source
const (
	// SQLTaskByID reads one task's record. Parameters: id.
	SQLTaskByID = stmtTaskByID
	// SQLSubmissionByID reads one submission's record. Parameters: id.
	SQLSubmissionByID = stmtSubmissionByID
	// SQLSubmissionByRequest reads a conversation's submission with a request ID. Parameters: conversation id,
	// JSON.stringify(requestId).
	SQLSubmissionByRequest = stmtSubmissionByRequest
	// SQLTasksPage and SQLSubmissionsPage read every task or submission a page at a time, as the dump does. Parameters: the
	// id the page follows.
	SQLTasksPage       = stmtDumpTasks
	SQLSubmissionsPage = stmtDumpSubmissions
	// SQLTasksDesc reads a page of tasks newest first, with the columns of SQLTasksPage. Parameters: the id the page precedes.
	SQLTasksDesc = stmtTasksDesc
	// SQLRecordType reads the record_ids row of an id. Parameters: id.
	SQLRecordType = stmtRecordType
	// SQLEntryByID reads one entry: conversation_id, commit_seq, record. Parameters: id.
	SQLEntryByID = stmtEntryByID
	// SQLEntryIDsDesc reads a page of a conversation's entry ids in a range, newest first. Parameters: conversation id,
	// lowest id, highest id, page size.
	SQLEntryIDsDesc = stmtEntryIDsDesc
	// SQLEntryBounds is history.SQLBounds: a conversation's newest entry, newest head marker and its head at or below a
	// bound.
	SQLEntryBounds = stmtBounds
	// SQLEntriesPage reads a page of a conversation's entries in an id range: id, commit_seq, record. Parameters:
	// conversation id, the id the page follows, highest id.
	SQLEntriesPage = stmtEntriesPage
	// SQLEntryCount counts a conversation's entries in an id range, so a load reserves its index once. Parameters:
	// conversation id, lowest id, highest id.
	SQLEntryCount = stmtEntryCount
	// SQLEntriesDesc reads a page of a conversation's entries in a range, newest first: id, commit_seq, record. Parameters:
	// conversation id, lowest id, highest id, page size.
	SQLEntriesDesc = stmtEntriesDesc
	// SQLContextSummary is the host's summary of a conversation's context through an entry: what a request preparation reads
	// of it. Parameters: conversation id, the newest entry (tail), the newest head marker at or below it (0: none) and that
	// marker's head, 1 when the estimate of the compaction check is wanted else 0. One row, one BLOB column: the summary JSON
	// `{tail, head, entries?, system, tokens}`.
	SQLContextSummary = stmtContextSummary
	// SQLContextMessages is the host's model messages of the same context. Parameters: conversation id, tail, marker,
	// marker's head. One row, one BLOB column: the JSON array of the messages.
	SQLContextMessages = stmtContextMessages
	// SQLContextRelease tells the host the conversation's kept context is dropped. Parameter: conversation id. No rows.
	SQLContextRelease = stmtContextRelease
	// PageRows is how many rows one page read returns at most.
	PageRows = dumpPage
)

The statements a core with a synchronous host read (ABI section 4.2, pig.read) runs when a step needs a record it did not load at open. Each reads the columns its dump statement reads, so one row decoder serves both.

View Source
const SupportedSchemaVersion = 2

SupportedSchemaVersion is the newest durable_schema.version this build writes and reads (migrations.ts CURRENT_SQLITE_SCHEMA_VERSION).

Variables

View Source
var ErrPending = errors.New("store: waiting for rows")

ErrPending means the store emitted reads the caller's operation depends on. Read-only.

Functions

func SQL

func SQL() []string

SQL returns the sql_table: every statement the core issues, by statement ID. The texts of Pi's writes are copied from pi-durable's storage.ts so DO and node:sqlite plan them the same way; reads of record columns use CAST(record AS BLOB) (ABI section 4.3).

Types

type Dump

type Dump struct {
	Conversations []*jv.Object
	Entries       []DumpEntry
	Tasks         []DumpTask
	Submissions   []DumpSubmission
	Documents     []*jv.Object
	Revisions     map[int64][]Revision // by document ID, in seq order
}

Dump is every record of the store, in ID order: what LoadAll reads after cold open.

type DumpEntry

type DumpEntry struct {
	ID, Conversation, Seq int64
	Head                  bool
	Raw                   []byte
}

DumpEntry is one entries row. Raw is the stored record text, which the dump holds without parsing it: opening a long history costs a copy of its bytes, not a tree per record.

type DumpSubmission

type DumpSubmission struct {
	ID, Conversation int64
	Request          string
	HasRequest       bool
	Status           string
	Raw              []byte
}

DumpSubmission is one submissions row: Request is the requestId, when the row has one.

type DumpTask

type DumpTask struct {
	ID, Conversation  int64
	Kind, Status      string
	Abort, Background bool
	Raw               []byte
}

DumpTask is one tasks row: the columns every scan filters on, and the stored record text.

type JSON

type JSON = any

JSON is a value of package jv (nil, bool, float64, string, []any or *jv.Object; ECMAScript property order).

type Live

type Live struct {
	NextID, NextSeq int64
	Conversations   []*rec.Conversation // the ones open needs: root and every conversation of a live task or submission
	Tasks           []*rec.Task         // status != terminal, ID order
	Submissions     []*rec.Submission   // queued or placed, ID order
	Documents       []*LiveDocument     // current, alive documents of the session, the loaded conversations and the live tasks
}

Live is the live state cold open loads (ADR D5) and hands to the session once: everything non-terminal.

func (*Live) LoadInto

func (l *Live) LoadInto(h *history.Store)

LoadInto registers the loaded conversation records with the transcript store.

type LiveDocument

type LiveDocument struct {
	Record          *rec.Document
	Version         int
	Value           *jv.Object
	DeltasSinceBase int
}

LiveDocument is one materialised document (base plus delta tail, spec §10 document()).

type Options

type Options struct {
	Sidecar  Sidecar
	CoreID   string // layout identity the sidecar must carry; empty means the build's own
	Debug    bool   // rebuild from rows and compare on every cold open (RISKS R11)
	EmptyNew bool   // an empty database is initialised with Pi's schema (migrations.ts) instead of rejected
}

Options are the open-time choices the open event carries.

type RejectedError

type RejectedError struct{ Message string }

RejectedError is pi-durable's StorageRejected: a commit the store refused before it changed anything.

func (*RejectedError) Error

func (e *RejectedError) Error() string

type RequestError

type RequestError struct {
	// contains filtered or unexported fields
}

RequestError is Pi's StorageRequestError: a read the caller's request got wrong, which fails only that read.

func (*RequestError) Error

func (e *RequestError) Error() string

type Revision

type Revision struct {
	Seq     int64
	Base    bool
	Version int
	Content any // the base value, or the delta's Chord Op[] JSON value
}

Revision is one document_revisions row.

type SchemaNewerError

type SchemaNewerError struct{ Version int64 }

SchemaNewerError is the refusal applySqliteMigrations gives a store whose durable_schema.version is newer than SupportedSchemaVersion.

func (SchemaNewerError) Error

func (e SchemaNewerError) Error() string

type Sidecar

type Sidecar uint8

Sidecar selects the cold-open path of ADR D6.

const (
	SidecarOff Sidecar = iota
	SidecarIndex
	SidecarSnapshot
)

Sidecar modes.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is one Session's view of its database.

func New

func New(opts Options) *Store

New returns a store that has read nothing.

func (*Store) Begin

func (s *Store) Begin() *Tx

Begin starts a Pi commit.

func (*Store) Commit

func (s *Store) Commit(b *abi.Builder, tx *Tx) (seq int64, err error)

Commit validates tx as SqliteStorage.commit does (ID claims, document actions; a failure is Pi's StorageRejected and changes nothing), emits one abi.Commit with Pi's rows in Pi's order and the guarded metadata update, applies the commit to the store's in-memory state (next_id, next_seq, documents), and returns the sequence it consumed. It returns ErrPending, having emitted reads and no commit, when a document copy needs a source the store has not loaded; the caller returns it unchanged and runs the whole handler again after the rows.

func (*Store) EnsureDocument

func (s *Store) EnsureDocument(b *abi.Builder, id int64, at int64) (*LiveDocument, error)

EnsureDocument returns incarnation id materialised at a point (at <= 0 is "current"), for a historical read or a document of a scope the store has not loaded. A nil document with a nil error means absent or not alive at the point. ErrPending means reads were emitted.

func (*Store) EnsureScope

func (s *Store) EnsureScope(b *abi.Builder, kind rec.ScopeKind, owner int64) ([]*LiveDocument, error)

EnsureScope loads the live (current, not retired) documents of a scope: the Session, a conversation or a task. It returns ErrPending after emitting reads, and the documents, ordered by ID, once every revision has arrived. A scope the store created itself (a new conversation or task) is loaded and empty.

func (*Store) EnsureTask

func (s *Store) EnsureTask(b *abi.Builder, id int64) (*rec.Task, error)

EnsureTask loads a committed task that is not live (a terminal task for getTask or outcomes). ErrPending when reads were emitted. A task the store has never seen and the database lacks returns (nil, nil).

func (*Store) Footprint

func (s *Store) Footprint() (docs uint64)

Footprint reports resident bytes of live documents for mem_stats.

func (*Store) ForgetScope

func (s *Store) ForgetScope(kind rec.ScopeKind, owner int64)

ForgetScope drops the loaded documents of a scope that is no longer in use (an unloaded conversation).

func (*Store) Idle

func (s *Store) Idle(b *abi.Builder)

Idle is called by the session at an idle boundary (a run settled, nothing live): the store writes due sidecar chunks as a sidecar-only commit (seq -1).

func (*Store) Load

func (s *Store) Load(b *abi.Builder, h *history.Store, need *history.Need) error

Load emits the reads of a history.Need (the transcript range history asks for) and returns ErrPending; the rows return through Rows, which hands them to history.Store.AddRow and SupplyBounds.

func (*Store) LoadAll

func (s *Store) LoadAll(b *abi.Builder, live bool) (*Dump, error)

LoadAll reads every Pi table after Open and seeds the store's commit validation with it (record_ids claims of live tasks and submissions, every conversation, every document incarnation with its current value). With live set it reads no entries and only the live tasks and submissions: a core that reads the rest on demand (ABI section 4.2, pig.read). It returns ErrPending after emitting reads, and the dump once every table has arrived. A store Open initialised needs no reads.

func (*Store) MintID

func (s *Store) MintID() int64

MintID allocates from the global ID namespace like SqliteStorage.mintId: an in-memory counter that the next commit persists as max(stored next_id, counter, max written id + 1).

func (*Store) NextID

func (s *Store) NextID() int64

NextID is the ID the next MintID returns.

func (*Store) NextSeq

func (s *Store) NextSeq() int64

NextSeq is the next_seq the next Pi commit consumes.

func (*Store) ObserveNextID

func (s *Store) ObserveNextID(next int64)

ObserveNextID raises the in-memory ID counter to a caller's: the Session mints IDs itself, and the next commit persists max(stored next_id, the counter, max written id + 1).

func (*Store) Open

func (s *Store) Open(b *abi.Builder) (*Live, error)

Open runs cold open: it emits the first reads into b and returns ErrPending, or returns the loaded Live once the rows it asked for have been delivered through Rows. A newer schema returns SchemaNewerError. Open reads the live state only: the metadata row, the non-terminal tasks and submissions, the conversations they live in and the live documents of the Session, those conversations and those tasks (ADR D5). Transcripts load per conversation through Load.

func (*Store) ReleaseDump

func (s *Store) ReleaseDump()

ReleaseDump drops the rows LoadAll gathered, once the caller has taken what it keeps from them.

func (*Store) Rows

func (s *Store) Rows(b *abi.Builder, answers []abi.ReadRows) error

Rows delivers a rows event. It returns ErrPending when it emitted further reads, nil when every pending load is complete.

type Tx

type Tx struct {
	// contains filtered or unexported fields
}

Tx is one Pi commit under construction: the StorageWrite batch of spec §10, in call order.

func (*Tx) Conversation

func (tx *Tx) Conversation(c *rec.Conversation)

Conversation stages a conversation insert (the "conversation" StorageWrite).

func (*Tx) ConversationRow

func (tx *Tx) ConversationRow(row history.ConversationRow)

ConversationRow stages a conversation insert from the bytes history.AppendConversation produced.

func (*Tx) DocumentBase

func (tx *Tx) DocumentBase(id int64, version int, base JSON)

DocumentBase stores a new complete base of an existing incarnation (the "document.change" StorageWrite, base).

func (*Tx) DocumentCopy

func (tx *Tx) DocumentCopy(d *rec.Document, source int64, at int64)

DocumentCopy copies a source at a point (a fork; the "document.copy" StorageWrite). at <= 0 selects "current".

func (*Tx) DocumentCreate

func (tx *Tx) DocumentCreate(d *rec.Document, version int, base JSON)

DocumentCreate writes a new incarnation with its base (the "document.create" StorageWrite); d is the DocumentCreate record, without createdAt and retiredAt.

func (*Tx) DocumentDelta

func (tx *Tx) DocumentDelta(id int64, version int, ops []byte)

DocumentDelta stores a delta of an existing incarnation; ops is Chord Op[] JSON (the "document.change" write, delta).

func (*Tx) DocumentRetire

func (tx *Tx) DocumentRetire(id int64)

DocumentRetire retires an incarnation (the "document.retire" StorageWrite).

func (*Tx) Entry

func (tx *Tx) Entry(id, conversationID, head int64, record []byte)

Entry writes one entry row; record is the bytes history.AppendEntryRecord produced. After Commit, the session appends the same record to the history.Store with the returned seq.

func (*Tx) EntryRow

func (tx *Tx) EntryRow(row history.EntryRow)

EntryRow stages an entry insert from history.NewEntryRow, which reads the id, conversation and head columns from the record the way SqliteStorage binds them (`head ?? null`).

func (*Tx) RecordText

func (tx *Tx) RecordText(k int) []byte

RecordText returns the record text the k-th conversation, entry, task or submission write staged in tx binds, in call order: an entry's, task's or submission's JSON text, nil for a conversation. The mirror keeps it as the row's stored text (mirror.Commit), so a commit needs no list of the texts of its own.

func (*Tx) Submission

func (tx *Tx) Submission(sub *rec.Submission, text []byte)

Submission stages a submission upsert (the "submission" StorageWrite); text is the record's JSON text (rec.Submission.Append).

func (*Tx) Task

func (tx *Tx) Task(t *rec.Task, text []byte)

Task stages a task upsert (the "task" StorageWrite); the tasks row columns come from the record, and text is its JSON text (rec.Task.Append).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL