storage

package
v0.50.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 3, 2026 License: AGPL-3.0 Imports: 36 Imported by: 0

Documentation

Overview

Package storage: tamper-evident audit log.

What: an append-only log of every statement Execute runs (see AttachAuditLog/DB.AuditLog and internal/engine/audit.go for the hook), where each entry's hash covers its own fields plus the previous entry's hash — a hash chain, the same integrity primitive used by blockchains and git commit history. Altering, reordering, or deleting any entry breaks the chain from that point forward in a way Verify detects.

Why this matters for the intended use cases (industry compliance, scientific reproducibility, security-sensitive deployments): a plain log file can be edited by anyone with filesystem access, silently. A hash chain doesn't prevent that access, but it makes silent editing detectable — Verify recomputes every hash and reports exactly where the chain first diverges from what's recorded, which is what audit/compliance tooling actually needs (proof of tampering, not prevention of it — true tamper-*prevention* needs signing with a key the log's own host doesn't hold, which is out of scope for an embedded database).

Package storage provides persistence primitives and metadata structures used by tinySQL. This file implements a lightweight in-memory system catalog used for introspection (tables, columns, views, functions) and simple job scheduling metadata.

Reading and writing a whole-database snapshot: to a file, a writer, or a byte slice. The file variant writes to a temporary file and renames it, so a crash during a save leaves the previous snapshot intact rather than a half-written one.

The column type system: storage types, SQLite type affinities, column constraints and referential actions. These are the vocabulary the schema is written in, so they are kept apart from the database that stores it.

Package storage provides the durable data structures for tinySQL.

What: An in-memory multi-tenant catalog of tables with column metadata, rows, and basic typing. It includes snapshot cloning for MVCC-light, GOB-based checkpoints, and an append-only Write-Ahead Log (WAL) for crash recovery and durability. How: Tables store rows as [][]any for compactness; a lower-cased column index accelerates name lookups. Save/Load serialize the catalog to a file, writing JSON for JSON columns. The WAL logs whole-table changes and drops; recovery replays committed records and truncates partial tails. Why: Favor a simple, explicit model over complex page managers: it keeps the code understandable, testable, and sufficient for embedded/edge use cases.

The on-disk representation of a database: the serializable forms of columns, tables and the system catalog, and the conversions to and from them. This is the format compatibility surface, so a change here changes what older files mean.

Converting one table to and from its on-disk form, including the row ranges and row sets the write-ahead log writes as deltas.

Package storage: encryption at rest for disk-based backends.

What: AES-256-GCM authenticated encryption of table files (ModeDisk, ModeJSON) and the backend catalog file. GCM was chosen over a plain block cipher mode specifically because it's authenticated — decrypting with the wrong key, or decrypting data that was altered after being written, fails closed with an error instead of silently returning garbage or (worse) subtly-wrong-but-plausible data. That gives encryption at rest a side benefit of the same tamper-detection property the audit log provides for statement history, applied to the data files themselves.

Why Argon2id for passphrase-based keys: PBKDF2 (also available in golang.org/x/crypto) is checkable purely with CPU, which makes brute-force guessing cheap on GPUs/ASICs at scale. Argon2id is memory-hard — an attacker also needs the configured amount of RAM per guess, which is the current best-practice recommendation (OWASP, and the Password Hashing Competition it won) for deriving a key from a human-chosen passphrase.

The operational surface: health, recovery status, syncing to the backend, closing, evicting a table from memory, and migrating an in-memory database onto a backend.

Opening a database: allocating an empty one, and OpenDB's per-mode construction of the backend, write-ahead log and checkpoint that a storage mode implies. Also loading a GOB checkpoint and reading the watermark that says how far it reflects the log.

Package storage: role-based access control (RBAC).

What: Users authenticate with a bcrypt-hashed password and are members of one or more roles; roles carry grants (a permission on a schema/table pair, with "*" as a wildcard for "all schemas"/"all tables"). Execute() in internal/engine checks the acting user's resolved permissions before running a statement.

Why opt-in: tinySQL has always been usable with zero setup (NewDB(), start running SQL). Requiring every caller to create a user before their first query would break that and every existing embedder. So RBAC enforcement only activates once at least one user has been created via CreateUser — see HasUsers/IsRBACEnabled. Before that, every permission check is a no-op allow, matching tinySQL's historical behavior exactly. This mirrors the same opt-in pattern already used for read-only mode (SetReadOnly) and cmd/server's optional auth token.

Copies of a database: the deep clone and the transaction snapshot pair the SQL driver runs a transaction against, and the row-less metadata snapshot the engine diffs for write-ahead logging.

Every clone must carry the runtime state that is not tenant data — see DB.copyRuntimeState in db.go. Hand-copying individual fields here is what once left a promoted clone without its write-ahead log.

Package storage – StorageBackend interface and StorageMode definitions.

What: Pluggable storage backends that decouple data management from the in-memory catalog. Each backend decides where table data lives (RAM, disk, or a combination) and how it is persisted. How: The DB struct optionally delegates Get/Put/Drop operations to an attached StorageBackend. Backends may lazily load tables, cache hot data, and flush dirty tables on Sync/Close. Why: Supporting multiple storage modes turns tinySQL into a realistic alternative to SQLite – from purely in-memory analytics right through to disk-resident databases that exceed available RAM.

Table: the rows of one table plus the schema and statistics describing them, and the dirty tracking that lets the write-ahead log write a delta instead of a whole table.

Finding, adding and removing tables within a tenant, including the did-you-mean suggestion an unknown table name produces and the lazy load a disk-backed mode performs on a cache miss.

The unit of work the write-ahead log records: which tables a change touched, how to compute that from two database states, and how to apply it. Shared by the log itself and by the SQL driver's commit path.

Primitives for a one-way replication feed built on top of AdvancedWAL: reading committed-only records since some LSN watermark, applying a received record on the subscriber side, and taking a snapshot tagged with the watermark a feed should resume from.

This file deliberately does not touch Recover (wal_advanced.go): its BEGIN/COMMIT/ABORT bookkeeping is duplicated here for ReadCommittedSince rather than factored out, so existing crash-recovery behavior and its tests are zero-risk to this addition.

Index

Examples

Constants

View Source 
const EncryptionKeySize = 32

EncryptionKeySize is the required AES-256 key size in bytes.

View Source 
const EncryptionSaltSize = 16

EncryptionSaltSize is the recommended salt size for DeriveKeyFromPassphrase.

Variables

View Source 
var (
	ErrTxNotActive          = fmt.Errorf("transaction is not active")
	ErrSerializationFailure = fmt.Errorf("could not serialize access due to concurrent update")
	ErrRowNotFound          = fmt.Errorf("row not found")
)

Errors

View Source 
var ErrAdvancedWALCheckpointActiveTransactions = errors.New("advanced WAL checkpoint requires all transactions to be committed or aborted")

ErrAdvancedWALCheckpointActiveTransactions means a checkpoint was requested while a transaction is still open. Saving a snapshot in that state can make an uncommitted in-memory mutation indistinguishable from committed state on recovery, so callers must commit or abort first. The automatic scheduler treats this as "try again later" rather than an operational failure.

View Source 
var ErrIndexRangeUnsupported = errors.New("secondary index cannot serve this range")

ErrIndexRangeUnsupported reports that this index cannot answer the requested range, so the caller must fall back to a scan. It is returned rather than a partial result: silently dropping an entry the walk cannot compare would return wrong rows, which is worse than being slow.

View Source 
var ErrReadOnlyStorage = errors.New("storage is read-only")

ErrReadOnlyStorage is returned by a persistence backend when a caller tries to modify an artifact opened for serving. It is deliberately distinct from an OS permission error so callers can surface an actionable configuration error before any file is touched.

View Source 
var ErrReplicaEpochMismatch = errors.New("replica WAL epoch does not match its bootstrap snapshot")

ErrReplicaEpochMismatch rejects applying a change batch from a different primary WAL incarnation than the snapshot the receiver bootstrapped from. The server normally catches this before it reaches storage, but enforcing it here as well makes the package API safe for callers that consume a feed directly rather than through cmd/server.

View Source 
var ErrReplicaTooFarBehind = errors.New("replica requested WAL records already checkpointed away; a full re-bootstrap is required")

ErrReplicaTooFarBehind is returned by ReadCommittedSince when sinceLSN is below the highest LSN a checkpoint has actually captured (see checkpointDataWatermark's doc comment in wal_advanced.go for exactly what that is and why it is not simply wal.checkpointWatermark): the caller is asking for a range that was already checkpointed and truncated out of the live WAL file, so it can no longer be served incrementally without silently skipping every record between sinceLSN and that boundary. That silent gap is exactly what the checkpointWatermark skip inside ReadCommittedSince's decode loop would otherwise produce for any request in this state -- returning this error up front instead makes the condition observable, so a caller (see cmd/server's GetChangesSince handler, which maps this to a distinguishable gRPC status) can react by re-bootstrapping the replica from a fresh snapshot instead of quietly missing data.

View Source 
var ErrReplicaWALOrder = errors.New("replica WAL records are not strictly ordered")

ErrReplicaWALOrder rejects a malformed or out-of-order committed-record batch. A feed may legitimately omit BEGIN/COMMIT marker LSNs, so this is a strict ordering check rather than a false requirement for contiguous LSNs.

View Source 
var ErrSaveToFileWouldBypassWALWatermark = errors.New("tinysql: SaveToFile would overwrite a live WAL-mode database's own checkpoint file without a watermark; use WALManager.Checkpoint/AdvancedWAL.Checkpoint (or DB.Close) instead")

ErrSaveToFileWouldBypassWALWatermark is returned by SaveToFile when filename is the checkpoint path of a WALManager or AdvancedWAL currently attached to db and the call carries no watermark (see rejectLiveWALPath).

Functions

func AppendCanonicalIndexValue added in v0.28.0 

func AppendCanonicalIndexValue(dst []byte, value any) []byte

AppendCanonicalIndexValue appends one value using the durable index-key encoding. Typed hot paths can provide stack-backed scratch space and avoid the growing allocations incurred by CanonicalIndexKey's general []any API.

func ApplyReplicaWALRecords added in v0.46.0 

func ApplyReplicaWALRecords(db *DB, epoch uint64, records []WALRecord) (int, error)

ApplyReplicaWALRecords applies one committed feed batch atomically with respect to readers and tracks its source LSN for idempotent retries. The input is expected to contain only INSERT/UPDATE/DELETE records from ReadCommittedSince; marker records are rejected rather than silently treating an unexpected transport payload as a no-op.

The full statement snapshot is deliberately used only for a non-empty batch of new records. A replication response may touch several tables, and a later bad record or secondary-index rebuild failure must not leave the replica half advanced while its resume LSN stays unchanged. Normal responses are small and the safety property matters more than optimizing that rare failure path.

func AsBigRat added in v0.6.0 

func AsBigRat(v any) (*big.Rat, bool)

AsBigRat returns the value as *big.Rat if it is already a rational type (either *big.Rat or big.Rat). Unlike DecimalFromAny this does not coerce ints/floats/strings and is suitable for preserving existing numeric semantics in the engine.

func CanonicalIndexKey added in v0.20.0 

func CanonicalIndexKey(values []any) []byte

CanonicalIndexKey returns the durable, type-tagged composite encoding used by materialized secondary indexes. Paged storage uses exactly the same encoding so a SQL-bound key and an on-disk B+Tree key compare identically.

func CanonicalIndexValueEqual added in v0.22.0 

func CanonicalIndexValueEqual(left, right any) bool

CanonicalIndexValueEqual reports whether two individual values have the same durable index encoding. The hot primitive forms avoid constructing temporary []any and []byte values, while the fallback deliberately uses the canonical encoder so newly supported value types retain identical behavior.

func DecimalAdd added in v0.6.0 

func DecimalAdd(a, b any) (*big.Rat, error)

DecimalAdd returns the sum of two decimal-like values as *big.Rat.

func DecimalFromAny added in v0.6.0 

func DecimalFromAny(v any) (*big.Rat, bool)

DecimalFromAny attempts to convert a value to *big.Rat.

func DecimalToString added in v0.6.0 

func DecimalToString(r *big.Rat) string

DecimalToString returns a plain decimal string representation.

func DeriveKeyFromPassphrase added in v0.13.0 

func DeriveKeyFromPassphrase(passphrase string, salt []byte) []byte

DeriveKeyFromPassphrase derives a 32-byte AES-256 key from a passphrase using Argon2id. salt must be a fixed, persisted value of at least EncryptionSaltSize bytes (see NewEncryptionSalt) — reusing a random salt on every call would derive a different key each time and permanently lock the caller out of data encrypted under the previous key.

The cost parameters (time=1, memory=64 MiB, threads=4) follow OWASP's Argon2id baseline recommendation: expensive enough to resist large-scale offline guessing, cheap enough to run once at database-open time rather than per-operation.

func EstimateColumnSize 

func EstimateColumnSize(typ ColType) int64

EstimateColumnSize estimates the average size of a column type.

func EstimateTableSize 

func EstimateTableSize(t *Table) int64

EstimateTableSize estimates memory usage of a table in bytes.

func EstimateValueSize 

func EstimateValueSize(val any) int64

EstimateValueSize estimates the memory size of a value.

func GenerateRandomPassword added in v0.12.0 

func GenerateRandomPassword(n int) (string, error)

GenerateRandomPassword returns a cryptographically random hex-encoded password of the given byte length (before hex encoding, so the resulting string is 2*n characters) — a convenience for callers provisioning service accounts that don't need a human-memorable password.

func IndexRangeComponentEncodable added in v0.26.0 

func IndexRangeComponentEncodable(v any) bool

IndexRangeComponentEncodable reports whether v encodes as a fixed-width numeric component, i.e. whether it can serve as a range bound. Text and BLOB values cannot: their framing puts a length ahead of the payload, so byte order is not value order.

func InitializeReplicaState added in v0.46.0 

func InitializeReplicaState(db *DB, epoch, watermark uint64) error

InitializeReplicaState records the epoch and LSN watermark that accompanied a Bootstrap snapshot. It must run immediately after LoadFromBytes and before applying changes; together with ApplyReplicaWALRecords it makes a retry of a successfully received gRPC response idempotent instead of duplicating its INSERT operations.

func JSONMarshal added in v0.6.0 

func JSONMarshal(v any) ([]byte, error)

JSONMarshal marshals v after converting internal types to JSON-friendly representations (e.g. big.Rat -> string).

func NewEncryptionSalt added in v0.13.0 

func NewEncryptionSalt() ([]byte, error)

NewEncryptionSalt returns a fresh, cryptographically random salt sized for DeriveKeyFromPassphrase. Persist it (e.g. in the backend's manifest) — the same salt must be supplied on every subsequent open for the same passphrase to derive the same key.

func ParseUUID added in v0.6.0 

func ParseUUID(s string) (uuid.UUID, error)

ParseUUID parses a UUID string into uuid.UUID.

func ReadCheckpointDataWatermark added in v0.46.0 

func ReadCheckpointDataWatermark(filename string) (uint64, error)

ReadCheckpointDataWatermark returns the feed-resume boundary a checkpoint snapshot contains. AdvancedWAL writes it after the recovery marker and epoch (see AdvancedWAL.Checkpoint). For ordinary row work this is the highest committed INSERT/UPDATE/DELETE LSN, deliberately separate from the later checkpoint marker because feed clients resume at operation LSNs. For a metadata-only checkpoint it is the marker itself, forcing an older replica to bootstrap rather than miss unrepresentable DDL/catalog state. Older checkpoints have no fourth trailing field and return zero; OpenAdvancedWAL then falls back conservatively to the marker.

func ReadCheckpointEpoch added in v0.30.0 

func ReadCheckpointEpoch(filename string) (uint64, error)

ReadCheckpointEpoch reads the WAL epoch identifier a checkpoint file may carry right after its table dump, catalog, and watermark (see SaveToFile's extra parameter and AdvancedWAL.Checkpoint, which writes watermark then epoch, in that order) -- see AdvancedWAL.epoch's doc comment for what the epoch identifies. Returns 0 with no error if the file doesn't exist, is empty, or simply predates the epoch field (an older checkpoint, or one written before this feature existed) -- all of which mean "no epoch recorded," the reserved sentinel OpenAdvancedWAL treats as "nothing to compare against, do not manufacture epoch churn for a plain restart."

func ReadCheckpointWatermark added in v0.22.0 

func ReadCheckpointWatermark(filename string) (uint64, error)

ReadCheckpointWatermark reads the trailing uint64 a checkpoint file may carry after its table dump and catalog (see SaveToFile's extra parameter) — the LSN/Seq up to which that checkpoint already reflects every operation, used by AdvancedWAL/WALManager to skip re-applying already-checkpointed WAL records on recovery (see each Checkpoint/Recover pair). Returns 0 with no error if the file doesn't exist, is empty, or simply predates this watermark (an older checkpoint format) — all of which mean "nothing to skip," the safe default.

func ReplicaApplyWatermark added in v0.46.0 

func ReplicaApplyWatermark(db *DB) (epoch, appliedLSN uint64)

ReplicaApplyWatermark returns the in-memory source epoch and highest applied operation/Bootstrap LSN. It is primarily useful to package users exposing replication health; v1 replicas intentionally bootstrap again on process restart, so this state is not persisted separately from their snapshot.

func SaveToBytes 

func SaveToBytes(db *DB, extra ...any) ([]byte, error)

SaveToBytes serializes the database snapshot to a byte slice. extra is passed through to SaveToWriter unchanged.

func SaveToFile 

func SaveToFile(db *DB, filename string, extra ...any) error

SaveToFile writes a snapshot of the database to a file. If the filename ends with .gz, the snapshot is gzip-compressed to reduce size.

The snapshot is written atomically: data goes to a temporary file in the same directory, is fsynced, and is then renamed over the target. A crash mid-checkpoint therefore never corrupts or truncates the previous snapshot. SaveToFile writes an atomic snapshot of db (every table plus the catalog) to filename, then gob-encodes each value in extra, in order, immediately after — letting a caller persist small auxiliary state (e.g. a WAL checkpoint's last-applied LSN/Seq watermark, see AdvancedWAL.Checkpoint/WALManager.Checkpoint) atomically with the snapshot itself, via the same temp-file-then-rename step, rather than a separate file whose write could complete independently of this one and leave the two inconsistent after a crash. Existing callers that pass no extra values are unaffected; the file format is unchanged for them.

func SaveToWriter 

func SaveToWriter(db *DB, w io.Writer, extra ...any) error

SaveToWriter writes a snapshot of the database to an arbitrary writer. It does not attach or alter WAL configuration.

Like SaveToFile, it gob-encodes each value in extra, in order, immediately after the table dump and catalog — letting a caller carry small auxiliary state (e.g. a replication watermark LSN, see SnapshotWithWatermark) alongside the snapshot in the same encoded stream. Existing callers that pass no extra values are unaffected; the encoded format for them is unchanged.

func SnapshotWithWatermark added in v0.30.0 

func SnapshotWithWatermark(db *DB, wal *AdvancedWAL) (data []byte, watermark uint64, epoch uint64, err error)

SnapshotWithWatermark takes a consistent snapshot of db together with the WAL LSN watermark up to which that snapshot is known to be complete and wal's current epoch identifier, for a replication feed to use as its starting point (hand the bytes to LoadFromBytes, then call ReadCommittedSince(wal, watermark) for everything after). A caller should remember epoch alongside watermark and compare it against whatever epoch every later GetChangesSince response carries (see cmd/server's Bootstrap/ GetChangesSince handlers) -- a mismatch means the primary's WAL/checkpoint files were wiped or restored from backup, and calling this again for a fresh snapshot is the only safe response (see AdvancedWAL.epoch's doc comment).

It holds DB's content read lock first and then wal's lock for the whole encode. SQL mutations hold the same content lock in write mode across their in-memory change and WAL append/commit, so no snapshot can now land between those two steps. The lock order matches Checkpoint exactly and is important: acquiring wal first and then waiting for DB content would deadlock against a writer that already holds content and is trying to append its WAL record. The watermark and epoch are read directly from wal's fields because the exported accessors take the same non-reentrant mutex.

func UUIDToBytes added in v0.6.0 

func UUIDToBytes(u uuid.UUID) []byte

UUIDToBytes returns the 16-byte representation of a uuid.UUID.

Types

type AdvancedWAL 

type AdvancedWAL struct {
	// contains filtered or unexported fields
}

AdvancedWAL manages row-level write-ahead logging with full ACID guarantees.

func OpenAdvancedWAL 

func OpenAdvancedWAL(config AdvancedWALConfig) (*AdvancedWAL, error)

OpenAdvancedWAL creates or opens a WAL with full ACID semantics.

func (*AdvancedWAL) Checkpoint 

func (w *AdvancedWAL) Checkpoint(db *DB) error

Checkpoint creates a consistent snapshot and truncates the WAL.

The lock order is deliberately DB content lock first, WAL lock second. SQL mutations already hold the content write lock while they append and commit their WAL records, so this makes the three values a replica/recovery cares about -- table bytes, snapshot watermark, and WAL history -- one atomic cut. In particular, a checkpoint can no longer serialize a row that has been mutated in memory but whose WAL record has not been appended yet.

Callers must not invoke Checkpoint while manually holding DB's content lock; the public method owns that lock so it can be safely used by the automatic scheduler and ordinary administrative callers without lock-order inversions.

func (*AdvancedWAL) CheckpointRequested added in v0.46.0 

func (w *AdvancedWAL) CheckpointRequested() bool

CheckpointRequested reports whether a metadata-only mutation asked for an immediate checkpoint that has not yet completed. The SQL driver consumes it after it has atomically published a shadow transaction's local DDL/catalog mutation to the live database; a rollback never gets to set this shared request.

func (*AdvancedWAL) CheckpointWhileContentLocked added in v0.46.0 

func (w *AdvancedWAL) CheckpointWhileContentLocked(db *DB) error

CheckpointWhileContentLocked is Checkpoint's variant for the one narrow case where a caller already owns db's content read or write lock. It exists so a successful schema/catalog statement can make its snapshot durable before returning without attempting to recursively acquire sync.RWMutex. Callers that do not already hold that lock must use Checkpoint instead.

func (*AdvancedWAL) Close 

func (w *AdvancedWAL) Close() error

Close flushes and closes the WAL.

func (*AdvancedWAL) Epoch added in v0.30.0 

func (w *AdvancedWAL) Epoch() uint64

Epoch returns this WAL's current epoch identifier -- see AdvancedWAL.epoch's doc comment for what it identifies and why a replication feed compares it across calls instead of relying on LSNs alone.

func (*AdvancedWAL) GetCommittedLSN 

func (w *AdvancedWAL) GetCommittedLSN() LSN

GetCommittedLSN returns the LSN of the last committed transaction.

func (*AdvancedWAL) GetFlushedLSN 

func (w *AdvancedWAL) GetFlushedLSN() LSN

GetFlushedLSN returns the LSN of the last flushed record.

func (*AdvancedWAL) GetNextLSN 

func (w *AdvancedWAL) GetNextLSN() LSN

GetNextLSN returns the next LSN to be assigned.

func (*AdvancedWAL) LogAbort 

func (w *AdvancedWAL) LogAbort(txID TxID) (LSN, error)

LogAbort logs a transaction abort.

func (*AdvancedWAL) LogBegin 

func (w *AdvancedWAL) LogBegin(txID TxID) (LSN, error)

LogBegin logs the start of a transaction.

func (*AdvancedWAL) LogCommit 

func (w *AdvancedWAL) LogCommit(txID TxID) (LSN, error)

LogCommit logs a transaction commit.

func (*AdvancedWAL) LogDelete 

func (w *AdvancedWAL) LogDelete(txID TxID, tenant, table string, rowID int64, before []any, cols []Column) (LSN, error)

LogDelete logs a row deletion.

cols is accepted for API symmetry with LogInsert/LogUpdate (existing callers pass it) but deliberately never stored on the record: applyOperation (below) only ever reads record.Columns in its bootstrap branch for WALOpInsert/WALOpUpdate -- a delete falls through to "ignore delete/update for non-existent table" and never creates anything, so no delete record, first-for-its-table or not, ever needs its schema.

func (*AdvancedWAL) LogInsert 

func (w *AdvancedWAL) LogInsert(txID TxID, tenant, table string, rowID int64, data []any, cols []Column) (LSN, error)

LogInsert logs a row insertion.

func (*AdvancedWAL) LogUpdate 

func (w *AdvancedWAL) LogUpdate(txID TxID, tenant, table string, rowID int64, before, after []any, cols []Column) (LSN, error)

LogUpdate logs a row update with before/after images.

func (*AdvancedWAL) NewAutoTxID added in v0.12.0 

func (w *AdvancedWAL) NewAutoTxID() TxID

NewAutoTxID returns a fresh transaction ID for the engine to use as an implicit single-statement transaction (one INSERT/UPDATE/DELETE statement = one WAL transaction, autocommitted). Safe for concurrent use.

func (*AdvancedWAL) Recover 

func (w *AdvancedWAL) Recover(db *DB) (int, error)

Recover replays the WAL to restore database state after a crash.

func (*AdvancedWAL) RequestCheckpoint added in v0.46.0 

func (w *AdvancedWAL) RequestCheckpoint()

RequestCheckpoint asks the DB-owned scheduler to persist a successful metadata/DDL mutation that has no row-level AdvancedWAL record of its own. It is non-blocking with respect to snapshot I/O: a checkpoint runs in the background after the caller's statement has released DB's content write lock. Calling it for an already-logged mutation is harmless, but the engine limits it to operations that did not open a statement WAL transaction so the configured row/byte thresholds remain meaningful for ordinary DML.

func (*AdvancedWAL) ShouldCheckpoint 

func (w *AdvancedWAL) ShouldCheckpoint() bool

ShouldCheckpoint checks if a checkpoint is needed.

type AdvancedWALConfig 

type AdvancedWALConfig struct {
	Path               string
	CheckpointPath     string
	CheckpointEvery    uint64        // Checkpoint after N records
	CheckpointInterval time.Duration // Checkpoint after duration
	CheckpointMaxBytes int64         // Checkpoint once WAL exceeds this size (0 = 64 MB default, <0 disables)
	BufferSize         int           // Buffer size for writing
	// SyncMode controls the commit flush. Its zero value, WALSyncFull,
	// preserves the historic strongest-flush behavior.
	SyncMode WALSyncMode
}

AdvancedWALConfig configures the advanced WAL.

There is deliberately no Compress option here: the live WAL log is a continuously-appended, crash-recoverable stream, and compressing it safely would need a materially more complex design (resumable framing, truncation handling that doesn't regress the per-record checksum's corruption detection). Compression for ModeAdvancedWAL instead applies to the periodic checkpoint snapshot only, via CheckpointPath's ".gz" suffix (see OpenDB's ModeAdvancedWAL case) — a checkpoint is a whole snapshot written once and read back whole, so it can reuse SaveToFile/ loadGOBInto's existing, already-tested gzip support with no new risk.

type AuditEntry added in v0.13.0 

type AuditEntry struct {
	Seq       uint64
	Timestamp time.Time
	Tenant    string
	User      string // empty if RBAC isn't enabled / no user was set
	Statement string // original SQL text if available, else a best-effort fallback
	Success   bool
	Error     string // populated only when !Success
	PrevHash  string // hex-encoded SHA-256 of the previous entry, "" for the first entry
	Hash      string // hex-encoded SHA-256 of this entry's own fields + PrevHash
}

AuditEntry is one recorded statement execution.

type AuditLog added in v0.13.0 

type AuditLog struct {
	// contains filtered or unexported fields
}

AuditLog is an in-memory, optionally file-backed hash-chained log. Safe for concurrent use.

func NewAuditLog added in v0.13.0 

func NewAuditLog() *AuditLog

NewAuditLog creates an in-memory-only audit log (not persisted to disk — useful for tests, or callers that only care about the current process's lifetime).

func OpenAuditLog added in v0.13.0 

func OpenAuditLog(path string) (*AuditLog, error)

OpenAuditLog opens (creating if necessary) an append-only JSONL file — one JSON-encoded AuditEntry per line, chosen over a binary format for the same reason ModeJSON exists: human-readable, diffable, greppable, and inspectable with ordinary tools during an incident, at the cost of some size on disk. If the file already has entries, they're replayed and the chain is verified as part of opening — a corrupted or tampered existing log is caught immediately at startup rather than silently trusted.

func (*AuditLog) Append added in v0.13.0 

func (a *AuditLog) Append(tenant, user, statement string, success bool, errMsg string) AuditEntry

Append records one statement execution and returns the entry, hash chain already computed. statement should be the original SQL text when available; user may be empty (RBAC not enabled, or no user in context).

func (*AuditLog) Close added in v0.13.0 

func (a *AuditLog) Close() error

Close flushes and closes the backing file, if any. A no-op for an in-memory-only log (NewAuditLog).

func (*AuditLog) Entries added in v0.13.0 

func (a *AuditLog) Entries() []AuditEntry

Entries returns a copy of every logged entry, in order.

func (*AuditLog) Verify added in v0.13.0 

func (a *AuditLog) Verify() error

Verify walks the whole chain and recomputes every hash, returning an error identifying the first entry (by Seq) where the recorded hash doesn't match — either the entry's own fields were altered, or PrevHash no longer matches its predecessor (an entry was inserted, removed, or reordered).

type BackendStats added in v0.5.4 

type BackendStats struct {
	Mode             StorageMode
	TablesInMemory   int
	TablesOnDisk     int
	MemoryUsedBytes  int64
	MemoryLimitBytes int64
	DiskUsedBytes    int64
	CacheHitRate     float64
	SyncCount        int64
	LoadCount        int64
	EvictionCount    int64
	// AdmissionRejects and LargestRejectedBytes are populated by the
	// table-cache backends (ModeHybrid / ModeIndex). A nonzero
	// AdmissionRejects means at least one table does not fit
	// MemoryLimitBytes even after eviction and is therefore decoded from
	// disk on every access; raising the budget past LargestRejectedBytes is
	// what makes it cacheable. They stay zero for backends without a table
	// cache.
	AdmissionRejects     int64
	LargestRejectedBytes int64
	// PageReads and cache counters are populated by page-oriented backends.
	// They are zero for legacy table-file backends which do not expose pages.
	PageReads   int64
	CacheHits   int64
	CacheMisses int64
	CachedPages int
	PinnedPages int
	// Transient pages are read-only query scratch that could not be admitted
	// while all cache frames were pinned. They are never retained by the LRU.
	TransientPages  int
	TransientFrames int
	MaxCachePages   int
}

BackendStats provides observability into storage backend behaviour.

type BufferPool 

type BufferPool struct {
	// contains filtered or unexported fields
}

BufferPool manages in-memory tables with configurable eviction.

func NewBufferPool 

func NewBufferPool(policy *MemoryPolicy) *BufferPool

NewBufferPool creates a buffer pool with the given policy.

func (*BufferPool) Get 

func (bp *BufferPool) Get(tenant, name string) (*Table, bool)

Get retrieves a table from the buffer pool.

func (*BufferPool) GetMemoryLimit 

func (bp *BufferPool) GetMemoryLimit() int64

GetMemoryLimit returns the configured memory limit.

func (*BufferPool) GetMemoryUsage 

func (bp *BufferPool) GetMemoryUsage() int64

GetMemoryUsage returns current memory usage in bytes.

func (*BufferPool) GetStats 

func (bp *BufferPool) GetStats() CacheStats

GetStats returns a copy of current statistics. HitRate is computed fresh from the live hit/miss counters rather than a stored field: those counters change on every Get (see recordHit/recordMiss), far more often than updateStats runs (Put/Remove only), so a stored HitRate would read stale after any Get-only stretch.

func (*BufferPool) Put 

func (bp *BufferPool) Put(tenant, name string, table *Table) error

Put adds or updates a table in the buffer pool.

func (*BufferPool) Remove 

func (bp *BufferPool) Remove(tenant, name string)

Remove removes a table from the buffer pool.

func (*BufferPool) SetEvictionSaver added in v0.23.1 

func (bp *BufferPool) SetEvictionSaver(f func(tenant, name string, table *Table))

SetEvictionSaver installs the callback evictLRU invokes on each table it is about to drop from the cache. See the evictionSaver field for the no-reentrancy constraint.

func (*BufferPool) Snapshot added in v0.23.1 

func (bp *BufferPool) Snapshot() []PooledEntry

Snapshot returns every table currently resident in the pool. The returned Table pointers are shared with the cache (and with whatever caller is still holding the lease DB.Get returned) — callers must not assume exclusive access to them.

type CacheStats 

type CacheStats struct {
	MemoryUsed        int64
	MemoryLimit       int64
	MemoryUtilization float64

	CacheHits   int64
	CacheMisses int64
	HitRate     float64

	EvictionCount int64
	EvictionSize  int64

	// AdmissionRejects counts table caching attempts refused because the
	// table exceeds the budget even after eviction; LargestRejectedBytes is
	// the largest size refused. A nonzero AdmissionRejects means some table
	// is being re-read from disk on every single access, which a larger
	// budget fixes only once it clears LargestRejectedBytes.
	AdmissionRejects     int64
	LargestRejectedBytes int64

	TablesInMemory int
	TablesOnDisk   int
	// contains filtered or unexported fields
}

CacheStats tracks buffer pool performance metrics.

type CacheStrategy 

type CacheStrategy int

CacheStrategy defines the eviction policy for the buffer pool. 

const (
	StrategyNone CacheStrategy = iota // No eviction (full in-memory)
	StrategyLRU                       // Least Recently Used
	StrategyLFU                       // Least Frequently Used (future)
	StrategyARC                       // Adaptive Replacement Cache (future)
)

func (CacheStrategy) String 

func (s CacheStrategy) String() string

type CachedTable 

type CachedTable struct {
	Table       *Table
	Size        int64
	LoadedAt    time.Time
	LastAccess  time.Time
	AccessCount int64
	Pinned      bool

	// Disk location (for lazy loading - future)
	OnDisk     bool
	DiskOffset int64
	// contains filtered or unexported fields
}

CachedTable wraps a table with caching metadata.

type CatalogColumn added in v0.5.1 

type CatalogColumn struct {
	Schema       string
	TableName    string
	Name         string
	Position     int
	DataType     string
	IsNullable   bool
	DefaultValue *string
}

CatalogColumn represents metadata for a table column CatalogColumn represents a column in a catalog table including its position and declared data type. `DefaultValue` may be nil if none is defined.

type CatalogDependency added in v0.11.0 

type CatalogDependency struct {
	Schema          string
	ObjectName      string
	ObjectType      string
	DependsOnSchema string
	DependsOnName   string
	DependsOnType   string
	DependencyType  string
	CreatedAt       time.Time
}

CatalogDependency records that one catalog object depends on another object.

type CatalogFunction added in v0.5.1 

type CatalogFunction struct {
	Schema          string
	Name            string
	FunctionType    string // 'SCALAR', 'TABLE', 'AGGREGATE', 'WINDOW'
	ArgTypes        []string
	ReturnType      string
	Language        string // 'BUILTIN', 'SQL', 'GO'
	IsDeterministic bool
	Description     string
}

CatalogFunction represents metadata for scalar and table-valued functions CatalogFunction describes registered functions (builtin or user defined). `FunctionType` categorizes the function, and `Language` indicates the implementation origin.

type CatalogIndex added in v0.18.0 

type CatalogIndex struct {
	// Tenant scopes the index definition to one logical database tenant.
	// Unlike views and functions, materialized secondary indexes belong to a
	// physical tenant table, so catalog metadata must never be shared across
	// tenants with the same schema/name.
	Tenant    string
	Schema    string
	Name      string
	Table     string
	Columns   []string
	Unique    bool
	CreatedAt time.Time
}

CatalogIndex stores metadata for a CREATE INDEX statement. tinySQL records index definitions for introspection, but the query planner does not consume them yet.

type CatalogJob added in v0.5.1 

type CatalogJob struct {
	Name         string
	SQLText      string
	ScheduleType string     // 'CRON', 'INTERVAL', 'ONCE'
	CronExpr     string     // For CRON schedule
	IntervalMs   int64      // For INTERVAL schedule (milliseconds)
	RunAt        *time.Time // For ONCE schedule
	Timezone     string
	Enabled      bool
	CatchUp      bool // Run missed executions
	NoOverlap    bool // Prevent concurrent runs
	MaxRuntimeMs int64
	LastRunAt    *time.Time
	NextRunAt    *time.Time
	CreatedAt    time.Time
	UpdatedAt    time.Time
}

CatalogJob represents a scheduled job CatalogJob describes a scheduled job. The fields provide flexible scheduling options (CRON, interval, or single-run) and execution metadata for bookkeeping and scheduling decisions.

type CatalogJobHistory added in v0.11.0 

type CatalogJobHistory struct {
	RunID        int64
	JobName      string
	StartedAt    time.Time
	FinishedAt   time.Time
	DurationMs   int64
	Status       string // 'SUCCEEDED', 'FAILED', 'SKIPPED', 'CANCELED'
	ErrorMessage string
}

CatalogJobHistory records one completed, failed, skipped, or canceled job run.

type CatalogManager added in v0.5.1 

type CatalogManager struct {
	// contains filtered or unexported fields
}

CatalogManager manages system catalog tables (`catalog.tables`, `catalog.columns`, etc.) and provides thread-safe registration and lookup helpers used by the rest of the system for introspection and scheduling. CatalogManager is safe for concurrent use.

func NewCatalogManager added in v0.5.1 

func NewCatalogManager() *CatalogManager

NewCatalogManager allocates and returns an initialized CatalogManager.

func (*CatalogManager) AddJobHistory added in v0.11.0 

func (c *CatalogManager) AddJobHistory(run *CatalogJobHistory) error

AddJobHistory appends a job execution history row and assigns a run id.

func (*CatalogManager) Authenticate added in v0.12.0 

func (c *CatalogManager) Authenticate(name, password string) bool

Authenticate verifies a password against the stored bcrypt hash in constant time (bcrypt.CompareHashAndPassword is itself constant-time with respect to the password content). Returns false for an unknown or disabled user without distinguishing why, so a caller can't use timing or error content to enumerate valid usernames.

func (*CatalogManager) CreateRole added in v0.12.0 

func (c *CatalogManager) CreateRole(name string) error

CreateRole adds a new, initially grant-less role.

func (*CatalogManager) CreateUser added in v0.12.0 

func (c *CatalogManager) CreateUser(name, password string, roles []string) error

CreateUser adds a new user with a bcrypt-hashed password. Returns an error if the name is already taken (case-insensitive) or the password is empty (an empty password would make Authenticate meaningless — bcrypt would happily hash "" too, but rejecting it here catches the mistake of e.g. an unpopulated config field at the source instead of silently creating a passwordless account).

func (*CatalogManager) DeleteIndex added in v0.18.0 

func (c *CatalogManager) DeleteIndex(schema, name string) error

DeleteIndex removes an index definition from the default tenant. New engine code should use DeleteIndexForTenant so tenant ownership is explicit.

func (*CatalogManager) DeleteIndexForTenant added in v0.22.0 

func (c *CatalogManager) DeleteIndexForTenant(tenant, schema, name string) error

DeleteIndexForTenant removes a stored index definition from one tenant.

func (*CatalogManager) DeleteIndexesForTable added in v0.18.0 

func (c *CatalogManager) DeleteIndexesForTable(table string)

DeleteIndexesForTable removes default-tenant indexes registered for a table. New engine code should use DeleteIndexesForTenantTable.

func (*CatalogManager) DeleteIndexesForTenantTable added in v0.22.0 

func (c *CatalogManager) DeleteIndexesForTenantTable(tenant, table string)

DeleteIndexesForTenantTable removes all indexes registered for a table in one tenant only.

func (*CatalogManager) DeleteJob added in v0.5.1 

func (c *CatalogManager) DeleteJob(name string) error

DeleteJob removes a job from the catalog, returning an error when the job does not exist.

func (*CatalogManager) DeleteMaterializedView added in v0.11.0 

func (c *CatalogManager) DeleteMaterializedView(schema, name string) error

DeleteMaterializedView removes a materialized view definition.

func (*CatalogManager) DeleteView added in v0.11.0 

func (c *CatalogManager) DeleteView(schema, name string) error

DeleteView removes a view definition and its catalog table entry.

func (*CatalogManager) DropRole added in v0.12.0 

func (c *CatalogManager) DropRole(name string) error

DropRole removes a role. Existing users keep a dangling membership reference (harmless: GrantRoleToUser/permission resolution both treat a reference to a since-dropped role as granting nothing), consistent with how DropTrigger etc. don't cascade either.

func (*CatalogManager) DropTrigger added in v0.9.0 

func (c *CatalogManager) DropTrigger(name string) error

DropTrigger removes a named trigger from the catalog. It returns an error when the trigger does not exist.

func (*CatalogManager) DropUser added in v0.12.0 

func (c *CatalogManager) DropUser(name string) error

DropUser removes a user. Returns an error if it doesn't exist.

func (*CatalogManager) FinishMaterializedViewRefresh added in v0.11.0 

func (c *CatalogManager) FinishMaterializedViewRefresh(schema, name string, refreshedAt time.Time, durationMs int64, rowCount int64, errMsg string) error

FinishMaterializedViewRefresh updates refresh bookkeeping.

func (*CatalogManager) GetAllColumns added in v0.5.1 

func (c *CatalogManager) GetAllColumns() []CatalogColumn

GetAllColumns aggregates and returns columns registered for every table in the catalog. The returned slice is a concatenation of the internal per-table column lists.

func (*CatalogManager) GetColumns added in v0.5.1 

func (c *CatalogManager) GetColumns(schema, tableName string) []CatalogColumn

GetColumns returns the column metadata for `schema.tableName`. If the table is unknown an empty slice is returned.

func (*CatalogManager) GetDependencies added in v0.11.0 

func (c *CatalogManager) GetDependencies() []CatalogDependency

GetDependencies returns all recorded catalog dependencies.

func (*CatalogManager) GetFunctions added in v0.5.1 

func (c *CatalogManager) GetFunctions() []*CatalogFunction

GetFunctions returns metadata for all registered functions.

func (*CatalogManager) GetIndex added in v0.18.0 

func (c *CatalogManager) GetIndex(schema, name string) (*CatalogIndex, bool)

GetIndex retrieves a default-tenant index definition by schema and name. New engine code should use GetIndexForTenant.

func (*CatalogManager) GetIndexForTenant added in v0.22.0 

func (c *CatalogManager) GetIndexForTenant(tenant, schema, name string) (*CatalogIndex, bool)

GetIndexForTenant retrieves one tenant's index definition by schema and name. Legacy unscoped catalog records intentionally do not match: guessing their owner would let one tenant read or delete another tenant's metadata.

func (*CatalogManager) GetIndexes added in v0.18.0 

func (c *CatalogManager) GetIndexes() []*CatalogIndex

GetIndexes returns all registered index definitions, including legacy unscoped records. It exists for administrative callers; query execution and virtual tables must use GetIndexesForTenant.

func (*CatalogManager) GetIndexesForTenant added in v0.22.0 

func (c *CatalogManager) GetIndexesForTenant(tenant string) []*CatalogIndex

GetIndexesForTenant returns only index definitions owned by tenant. Legacy records with an empty tenant are deliberately excluded rather than being guessed as a particular tenant.

func (*CatalogManager) GetJob added in v0.5.1 

func (c *CatalogManager) GetJob(name string) (*CatalogJob, error)

GetJob retrieves a job by name, returning an error if not found.

func (*CatalogManager) GetMaterializedView added in v0.11.0 

func (c *CatalogManager) GetMaterializedView(schema, name string) (*CatalogMaterializedView, bool)

GetMaterializedView retrieves a materialized view definition by schema/name.

func (*CatalogManager) GetMaterializedViews added in v0.11.0 

func (c *CatalogManager) GetMaterializedViews() []*CatalogMaterializedView

GetMaterializedViews returns all materialized view definitions.

func (*CatalogManager) GetTables added in v0.5.1 

func (c *CatalogManager) GetTables() []*CatalogTable

GetTables returns a slice with metadata for all registered tables and views.

func (*CatalogManager) GetTriggers added in v0.9.0 

func (c *CatalogManager) GetTriggers(table string, timing TriggerTiming, event TriggerEvent) []*CatalogTrigger

GetTriggers returns all triggers for the given table, timing, and event.

func (*CatalogManager) GetTriggersForEvent added in v0.23.1 

func (c *CatalogManager) GetTriggersForEvent(table string, event TriggerEvent) (before, after []*CatalogTrigger)

GetTriggersForEvent returns the BEFORE and AFTER row-trigger lists for one table/event pair while holding the catalog lock only once. DML executors resolve both timings together before entering their row loop, avoiding two full catalog scans per statement and any lookup work per affected row.

func (*CatalogManager) GetUser added in v0.12.0 

func (c *CatalogManager) GetUser(name string) (CatalogUser, bool)

GetUser returns a copy of a user's metadata (never the password hash's caller-mutable backing, since Roles is copied) for introspection, or false if it doesn't exist.

func (*CatalogManager) GetView added in v0.11.0 

func (c *CatalogManager) GetView(schema, name string) (*CatalogView, bool)

GetView retrieves a view definition by schema and name.

func (*CatalogManager) GetViews added in v0.5.1 

func (c *CatalogManager) GetViews() []*CatalogView

GetViews returns metadata for all registered views.

func (*CatalogManager) GrantPermission added in v0.12.0 

func (c *CatalogManager) GrantPermission(roleName string, perm Permission, schema, table string) error

GrantPermission adds a grant to an existing role. schema/table accept "*" as a wildcard.

func (*CatalogManager) GrantRoleToUser added in v0.12.0 

func (c *CatalogManager) GrantRoleToUser(userName, roleName string) error

GrantRoleToUser adds a role membership to an existing user. Idempotent — granting a role the user already has is not an error.

func (*CatalogManager) HasMaterializedViews added in v0.33.0 

func (c *CatalogManager) HasMaterializedViews() bool

HasMaterializedViews reports whether any materialized view is registered.

It exists for the DML path, which asks after every successful mutating statement whether that statement invalidated a materialized view. Databases with none — the overwhelming majority — should not pay for splitting the table name and building a dependency key to find that out.

func (*CatalogManager) HasPermission added in v0.12.0 

func (c *CatalogManager) HasPermission(userName string, perm Permission, schema, table string) bool

HasPermission resolves userName's roles to their grants and reports whether any grant authorizes perm on (schema, table). Returns false for an unknown user (fail closed).

func (*CatalogManager) HasUsers added in v0.12.0 

func (c *CatalogManager) HasUsers() bool

HasUsers reports whether any user has been created. Note this ignores SetRBACEnabled's override — use IsRBACEnabled to answer "is enforcement actually active right now", which is what Execute checks.

func (*CatalogManager) IsRBACEnabled added in v0.13.0 

func (c *CatalogManager) IsRBACEnabled() bool

IsRBACEnabled reports whether Execute currently enforces permissions: true if SetRBACEnabled(false) was never called (or was reverted via SetRBACEnabled(true)) and at least one user exists.

func (*CatalogManager) ListEnabledJobs added in v0.5.1 

func (c *CatalogManager) ListEnabledJobs() []*CatalogJob

ListEnabledJobs returns only jobs whose `Enabled` flag is true.

func (*CatalogManager) ListJobHistory added in v0.11.0 

func (c *CatalogManager) ListJobHistory() []*CatalogJobHistory

ListJobHistory returns all job execution history rows.

func (*CatalogManager) ListJobs added in v0.5.1 

func (c *CatalogManager) ListJobs() []*CatalogJob

ListJobs returns a slice containing all registered jobs.

func (*CatalogManager) ListRoles added in v0.12.0 

func (c *CatalogManager) ListRoles() []CatalogRole

ListRoles returns all roles, sorted by name.

func (*CatalogManager) ListTriggers added in v0.9.0 

func (c *CatalogManager) ListTriggers() []*CatalogTrigger

ListTriggers returns all registered trigger definitions.

func (*CatalogManager) ListUsers added in v0.12.0 

func (c *CatalogManager) ListUsers() []CatalogUser

ListUsers returns all users (without password hashes), sorted by name.

func (*CatalogManager) MarkMaterializedViewsStaleByDependency added in v0.11.0 

func (c *CatalogManager) MarkMaterializedViewsStaleByDependency(schema, changedName string) []string

MarkMaterializedViewsStaleByDependency marks opt-in materialized views stale when they depend on the changed object. It returns the affected view names.

func (*CatalogManager) RegisterFunction added in v0.5.1 

func (c *CatalogManager) RegisterFunction(fn *CatalogFunction) error

RegisterFunction registers or updates a function's metadata.

func (*CatalogManager) RegisterIndex added in v0.18.0 

func (c *CatalogManager) RegisterIndex(idx *CatalogIndex) error

RegisterIndex stores index metadata in the default tenant. New engine code should use RegisterIndexForTenant so tenant ownership is explicit.

func (*CatalogManager) RegisterIndexForTenant added in v0.22.0 

func (c *CatalogManager) RegisterIndexForTenant(tenant string, idx *CatalogIndex) error

RegisterIndexForTenant stores index metadata for one tenant. If an index with the same tenant/schema/name exists, it is replaced.

func (*CatalogManager) RegisterJob added in v0.5.1 

func (c *CatalogManager) RegisterJob(job *CatalogJob) error

RegisterJob adds a new scheduled job or updates an existing entry. Job names must be non-empty.

func (*CatalogManager) RegisterMaterializedView added in v0.11.0 

func (c *CatalogManager) RegisterMaterializedView(mv *CatalogMaterializedView) error

RegisterMaterializedView registers or updates a materialized view definition.

func (*CatalogManager) RegisterTable added in v0.5.1 

func (c *CatalogManager) RegisterTable(schema, name string, cols []Column) error

RegisterTable registers a table and its columns in the system catalog. The provided `cols` slice is converted to `CatalogColumn` entries and stored under the key `schema.name`.

func (*CatalogManager) RegisterTrigger added in v0.9.0 

func (c *CatalogManager) RegisterTrigger(t *CatalogTrigger) error

RegisterTrigger stores a trigger definition in the catalog. If a trigger with the same name already exists it is replaced.

func (*CatalogManager) RegisterView added in v0.5.1 

func (c *CatalogManager) RegisterView(schema, name, sqlText string) error

RegisterView registers a view definition under `schema.name`.

func (*CatalogManager) RenameIndexColumnForTenant added in v0.43.0 

func (c *CatalogManager) RenameIndexColumnForTenant(tenant, table, from, to string)

RenameIndexColumnForTenant updates the introspection definition of every index on one table. Materialized index entries live on storage.Table and are updated by Table.RenameColumn; this method keeps sys.indexes and persistence metadata in step with that physical change.

func (*CatalogManager) Revision added in v0.24.0 

func (c *CatalogManager) Revision() uint64

Revision returns a counter that increases on every catalog mutation. It is only meaningful for comparison against an earlier reading of the same catalog instance (or of a snapshot taken from it), never across instances.

func (*CatalogManager) RevokePermission added in v0.12.0 

func (c *CatalogManager) RevokePermission(roleName string, perm Permission, schema, table string) error

RevokePermission removes every grant on a role matching perm/schema/table exactly (same wildcard values as originally granted — this is a simple exact-match revoke, not a partial-overlap subtraction).

func (*CatalogManager) RevokeRoleFromUser added in v0.12.0 

func (c *CatalogManager) RevokeRoleFromUser(userName, roleName string) error

RevokeRoleFromUser removes a role membership. Not an error if the user didn't have that role.

func (*CatalogManager) SetDependencies added in v0.11.0 

func (c *CatalogManager) SetDependencies(schema, objectName, objectType string, deps []CatalogDependency)

SetDependencies replaces the dependency list for a catalog object.

func (*CatalogManager) SetRBACEnabled added in v0.13.0 

func (c *CatalogManager) SetRBACEnabled(enabled bool)

SetRBACEnabled overrides the default opt-in-via-CreateUser behavior. Pass false to force RBAC off — every Execute call is permitted regardless of context, even if users/roles already exist — useful for a setup that provisions accounts ahead of time but isn't ready to enforce yet, or a dev/test environment that wants audit-log attribution (see WithUser) without access checks getting in the way. Pass true to restore the default: enforcement follows HasUsers() exactly as if SetRBACEnabled had never been called.

This is a deliberate escape hatch matching SetReadOnly's toggle pattern elsewhere in this package — RBAC being opt-in-by-default already covers "not needed in every setup" for anyone who simply never calls CreateUser; this method is for the narrower case of a database that *has* users defined but shouldn't enforce against them right now.

func (*CatalogManager) SetUserDisabled added in v0.12.0 

func (c *CatalogManager) SetUserDisabled(name string, disabled bool) error

SetUserDisabled enables/disables a user without deleting their account (preserving audit-log attribution history for anything they did while enabled). A disabled user fails Authenticate.

func (*CatalogManager) TryBeginMaterializedViewRefresh added in v0.11.0 

func (c *CatalogManager) TryBeginMaterializedViewRefresh(schema, name string) bool

TryBeginMaterializedViewRefresh marks a materialized view as refreshing.

func (*CatalogManager) UpdateJobRuntime added in v0.5.1 

func (c *CatalogManager) UpdateJobRuntime(name string, lastRun, nextRun time.Time) error

UpdateJobRuntime updates runtime bookkeeping fields for a named job. It sets `LastRunAt`, `NextRunAt` and marks the job as recently updated.

func (*CatalogManager) UpdateJobRuntimePtr added in v0.11.0 

func (c *CatalogManager) UpdateJobRuntimePtr(name string, lastRun time.Time, nextRun *time.Time) error

UpdateJobRuntimePtr updates runtime bookkeeping fields for a named job. Passing nil for nextRun clears NextRunAt, which is expected for completed one-shot jobs.

type CatalogMaterializedView added in v0.11.0 

type CatalogMaterializedView struct {
	Schema             string
	Name               string
	SQLText            string
	CacheTableName     string
	StaleAfterMs       int64
	RefreshEveryMs     int64
	DailyAt            string
	Timezone           string
	WithData           bool
	InvalidateOnChange bool
	IsStale            bool
	LastRefreshAt      *time.Time
	LastDurationMs     int64
	LastError          string
	IsRefreshing       bool
	CreatedAt          time.Time
	UpdatedAt          time.Time
}

CatalogMaterializedView stores a saved query with physical refresh metadata.

type CatalogRole added in v0.12.0 

type CatalogRole struct {
	Name      string
	Grants    []Grant
	CreatedAt time.Time
}

CatalogRole is a named bundle of grants that one or more users can hold.

type CatalogSnapshot added in v0.24.0 

type CatalogSnapshot struct {
	// contains filtered or unexported fields
}

CatalogSnapshot is an opaque, comparable copy of a catalog's contents. It exists so the SQL driver can tell "this transaction created a view" from "this transaction ran an INSERT that happened to touch catalog bookkeeping", which a revision counter alone cannot distinguish.

func (CatalogSnapshot) Equal added in v0.24.0 

func (s CatalogSnapshot) Equal(other CatalogSnapshot) bool

Equal reports whether two snapshots hold the same catalog contents.

type CatalogTable added in v0.5.1 

type CatalogTable struct {
	Schema    string
	Name      string
	Type      string // 'TABLE', 'VIEW', 'SYSTEM'
	RowCount  int64
	CreatedAt time.Time
	UpdatedAt time.Time
}

CatalogTable represents metadata for a single table CatalogTable holds basic metadata for a registered table or view.

type CatalogTrigger added in v0.9.0 

type CatalogTrigger struct {
	Name       string
	Table      string
	Timing     TriggerTiming
	Event      TriggerEvent
	ForEachRow bool
	WhenExpr   string // optional WHEN clause SQL text
	Body       string // SQL text of trigger body (semicolon-separated stmts)
	CreatedAt  time.Time
}

CatalogTrigger holds a stored trigger definition.

type CatalogUser added in v0.12.0 

type CatalogUser struct {
	Name         string
	PasswordHash string
	Roles        []string
	Disabled     bool
	CreatedAt    time.Time
}

CatalogUser is an authenticatable principal. Passwords are never stored in plaintext — only a bcrypt hash, which is itself salted and safe to treat as opaque (bcrypt embeds its own per-hash random salt).

type CatalogView added in v0.5.1 

type CatalogView struct {
	Schema    string
	Name      string
	SQLText   string
	CreatedAt time.Time
}

CatalogView represents a saved view definition CatalogView stores the definition of a saved view.

type ColType 

type ColType int

ColType enumerates supported column data types. 

const (
	// IntType is a generic integer column type.
	IntType ColType = iota
	// Int8Type is an 8-bit signed integer column type.
	Int8Type
	// Int16Type is a 16-bit signed integer column type.
	Int16Type
	// Int32Type is a 32-bit signed integer column type.
	Int32Type
	// Int64Type is a 64-bit signed integer column type.
	Int64Type
	// UintType is an unsigned integer column type.
	UintType
	// Uint8Type is an 8-bit unsigned integer column type.
	Uint8Type
	// Uint16Type is a 16-bit unsigned integer column type.
	Uint16Type
	// Uint32Type is a 32-bit unsigned integer column type.
	Uint32Type
	// Uint64Type is a 64-bit unsigned integer column type.
	Uint64Type

	// Float32Type is a 32-bit floating point column type.
	Float32Type
	// Float64Type is a 64-bit floating point column type.
	Float64Type
	// FloatType is an alias for Float64Type.
	FloatType // alias for Float64Type

	// StringType represents a variable-length UTF-8 string column.
	StringType
	// TextType is an alias for StringType intended for long text.
	TextType // alias for StringType
	// RuneType stores single Unicode code points.
	RuneType
	// ByteType stores raw byte data.
	ByteType

	// BoolType represents a boolean column (true/false).
	BoolType

	// TimeType stores time-of-day values.
	TimeType
	// DateType stores date-only values.
	DateType
	// DateTimeType stores combined date and time values.
	DateTimeType
	// TimestampType stores an absolute point in time.
	TimestampType
	// DurationType stores a time duration.
	DurationType

	// DecimalType stores arbitrary-precision decimal numbers.
	DecimalType
	// MoneyType is a convenience alias for DecimalType used for monetary values.
	MoneyType
	// UUIDType stores RFC-4122 UUID values.
	UUIDType
	// BlobType stores binary large objects.
	BlobType
	// XMLType stores XML text.
	XMLType
	// IntervalType stores SQL-like intervals (parsed to time.Duration when possible).
	IntervalType

	// JsonType stores JSON text.
	JsonType
	// JsonbType stores binary JSON representations.
	JsonbType
	// MapType stores map-like complex values.
	MapType
	// SliceType stores slice-like complex values.
	SliceType
	// ArrayType stores array-like complex values.
	ArrayType

	// Complex64Type stores complex64 numeric values.
	Complex64Type
	// Complex128Type stores complex128 numeric values.
	Complex128Type
	// ComplexType is an alias for Complex128Type.
	ComplexType // alias for Complex128Type
	// PointerType represents a pointer/reference to another object.
	PointerType
	// InterfaceType represents an arbitrary Go interface value.
	InterfaceType

	// VectorType represents a vector/embedding column used by RAG features.
	VectorType
	// GeometryType stores spatial geometry values (GeoJSON/WKB) as JSONB or binary payload.
	GeometryType

	// YAMLType stores YAML-formatted text data.
	YAMLType
	// URLType stores URL/URI values with optional validation.
	URLType
	// HASHType stores cryptographic hash digests (hex-encoded).
	HASHType
	// BitmapType stores roaring-bitmap or bitset values as a byte slice.
	BitmapType
)

func (ColType) String 

func (t ColType) String() string

type Column 

type Column struct {
	Name string
	Type ColType
	// DeclaredType retains the source SQL spelling (for example VARCHAR(80)
	// or DOUBLE PRECISION). It is intentionally metadata; the physical value
	// representation remains ColType plus SQLiteAffinity.
	DeclaredType string
	// Affinity is populated for SQLite-style declarations. AffinityDefault
	// means this column was declared using a native tinySQL type.
	Affinity SQLiteAffinity
	// NotNull and default metadata are independent of Constraint because SQL
	// permits combinations such as "PRIMARY KEY NOT NULL DEFAULT 0".
	NotNull      bool
	HasDefault   bool
	DefaultValue any
	Constraint   ConstraintType
	ForeignKey   *ForeignKeyRef // Only used if Constraint == ForeignKey
	PointerTable string         // Target table for POINTER type
}

Column holds column schema information in a table.

type ColumnStats added in v0.21.0 

type ColumnStats struct {
	NullCount     int
	DistinctCount int
	Min           string
	Max           string
	HasMinMax     bool
}

ColumnStats summarizes one column as of TableStats.AnalyzedAt. Min and Max are display values for introspection; the planner currently uses row and distinct counts, which remain meaningful across all supported column types.

type ConstraintType 

type ConstraintType int

ConstraintType enumerates supported column constraints. 

const (
	NoConstraint ConstraintType = iota
	PrimaryKey
	ForeignKey
	Unique
)

func (ConstraintType) String 

func (c ConstraintType) String() string

type DB 

type DB struct {
	// contains filtered or unexported fields
}

DB is an in-memory, multi-tenant database catalog with full MVCC support. It optionally delegates storage to a StorageBackend for disk-based or hybrid persistence strategies.

func LoadFromBytes 

func LoadFromBytes(b []byte, extra ...any) (*DB, error)

LoadFromBytes loads a database from a byte slice. extra is passed through to LoadFromReader unchanged.

func LoadFromFile 

func LoadFromFile(filename string) (*DB, error)

LoadFromFile loads a database snapshot from a file. It auto-detects gzip compression based on the .gz suffix and attaches a WAL if a path is given.

func LoadFromReader 

func LoadFromReader(r io.Reader, extra ...any) (*DB, error)

LoadFromReader loads a database snapshot from an arbitrary reader. The returned DB has no WAL attached.

extra mirrors SaveToWriter's extra parameter: each element must be a pointer, and is decoded in order from whatever trailing values follow the table dump and catalog in the stream (see SaveToWriter). If the stream has fewer trailing values than len(extra) — e.g. it was written by an older caller that passed no extra values — decoding stops at the first EOF and any remaining pointers in extra are left untouched, matching ReadCheckpointWatermark's "predates this field: nothing to fill in" behavior rather than erroring.

func NewDB 

func NewDB() *DB

NewDB creates a new empty database catalog with MVCC support. The database operates in ModeMemory (pure in-memory) by default. Use OpenDB for disk-backed or hybrid storage modes.

func OpenDB added in v0.5.4 

func OpenDB(cfg StorageConfig) (*DB, error)

OpenDB creates or opens a database with the specified storage configuration. For ModeMemory this is equivalent to NewDB (with optional save-on-close). For ModeDisk/ModeJSON/ModeHybrid/ModeIndex, tables are stored as individual files in the configured directory. For ModeWAL, the existing WAL mechanism is configured automatically.

func (*DB) AbortAmbientWALTx added in v0.24.0 

func (db *DB) AbortAmbientWALTx() error

AbortAmbientWALTx writes the abort record for the ambient transaction and clears it, so recovery discards whatever the rolled-back block logged.

func (*DB) AdoptCatalog added in v0.24.0 

func (db *DB) AdoptCatalog(catalog *CatalogManager)

AdoptCatalog installs a catalog produced elsewhere as this database's own. The SQL driver calls it at COMMIT to publish the catalog work a transaction did on its private copy — creating a view or trigger, granting a permission, marking a materialized view stale. Those changes are not table data and so are invisible to ApplyWALChanges; before this existed they were silently discarded by a successful commit.

The caller is responsible for having established that no concurrent change is being overwritten (see conn.commitTx's revision check). The revision counter is carried forward so it stays monotonic for the next comparison.

func (*DB) AdvancedWAL 

func (db *DB) AdvancedWAL() *AdvancedWAL

AdvancedWAL returns the configured advanced WAL manager (may be nil).

func (*DB) AdvancedWALMetadataDirty added in v0.46.0 

func (db *DB) AdvancedWALMetadataDirty() bool

AdvancedWALMetadataDirty reports whether this transaction shadow needs a post-COMMIT metadata checkpoint. It is intentionally read-only: a failed durability tail may be retried while the driver still owns the shadow.

func (*DB) AmbientWALTx added in v0.24.0 

func (db *DB) AmbientWALTx() (TxID, bool)

AmbientWALTx returns the ambient AdvancedWAL transaction, if one is open.

func (*DB) AppendRowsFast added in v0.26.2 

func (db *DB) AppendRowsFast(tenant, tableName string, newRows [][]any) (ok bool, err error)

AppendRowsFast appends newRows to tenant/tableName, maintaining every declared unique secondary index, without materializing rows already on disk -- when the attached backend is ModePagedIndex (see PagedIndexBackend.AppendRows). ok is false when there is no such backend attached, or the table has no catalog entry yet (it must already exist, e.g. via Put/CreateTable): the caller should fall back to its normal db.Get-then-append path. A true result with a non-nil error means the fast path was attempted and failed -- do not also apply newRows through the fallback path in that case, or they will be double-inserted.

ImportMBTiles (internal/importer) is the motivating caller: without this, importing a tileset larger than memory needs the whole destination table resident by the time the import finishes, for every storage mode including paged, because nothing durable happens until that in-memory slice is eventually saved -- and PagedIndexBackend.SaveTable's "replace everything" rebuild would make every batch after the first cost the size of everything imported so far, not just its own size.

func (*DB) ApplyWALChanges added in v0.18.0 

func (db *DB) ApplyWALChanges(changes []WALChange) error

ApplyWALChanges applies a set of table-level changes to the database. It is used by the SQL driver to commit a transaction by merging the transaction's delta into the latest shared database instead of replacing the whole DB with an older snapshot.

func (*DB) AttachAdvancedWAL 

func (db *DB) AttachAdvancedWAL(wal *AdvancedWAL)

AttachAdvancedWAL attaches an advanced WAL to the database.

func (*DB) AttachAuditLog added in v0.13.0 

func (db *DB) AttachAuditLog(log *AuditLog)

AttachAuditLog attaches a tamper-evident audit log to the database. Once attached, internal/engine.Execute records every statement to it (see internal/engine/audit.go). Pass nil to detach (stop logging); Execute treats a nil audit log as "logging disabled", matching the opt-in pattern used throughout this session's hardening work (RBAC, read-only mode, the security warning in cmd/server).

func (*DB) AuditLog added in v0.13.0 

func (db *DB) AuditLog() *AuditLog

AuditLog returns the attached audit log, or nil if none is configured.

func (*DB) Backend added in v0.5.4 

func (db *DB) Backend() StorageBackend

Backend returns the attached StorageBackend (may be nil for pure in-memory databases created with NewDB).

func (*DB) BackendStats added in v0.5.4 

func (db *DB) BackendStats() BackendStats

BackendStats returns statistics from the storage backend. Returns a zero-value BackendStats if no backend is attached.

func (*DB) BeginAmbientWALTx added in v0.24.0 

func (db *DB) BeginAmbientWALTx() (bool, error)

BeginAmbientWALTx opens one AdvancedWAL transaction that every subsequent statement executed against db joins, instead of each statement logging its own implicitly-committed transaction. The SQL driver calls it on a transaction shadow at BEGIN so that recovery replays the block only if CommitAmbientWALTx ran; AbortAmbientWALTx marks it rolled back.

It is idempotent, so the driver can call it lazily before the first write in a block instead of writing a begin record for every read-only BEGIN. It is a no-op returning false when no AdvancedWAL is attached.

func (*DB) Catalog added in v0.5.1 

func (db *DB) Catalog() *CatalogManager

Catalog returns the CatalogManager attached to the DB, creating one lazily if necessary. The catalog is reached by every SQL statement for RBAC checks, so its one-time construction must be synchronized when a database/sql connector opens several physical reader connections at once.

func (*DB) Close added in v0.5.4 

func (db *DB) Close() error

Close persists all data and releases resources. For ModeMemory with a configured path, this saves a final GOB snapshot. For ModeDisk/ModeJSON/ ModeHybrid, dirty tables are flushed. WAL and Advanced WAL resources are closed.

func (*DB) CommitAmbientWALTx added in v0.24.0 

func (db *DB) CommitAmbientWALTx() error

CommitAmbientWALTx writes the commit record for the ambient transaction and clears it. Recovery replays the transaction's operations only after seeing this record.

func (*DB) Config added in v0.5.4 

func (db *DB) Config() *StorageConfig

Config returns the StorageConfig used to open this database. Returns nil for databases created with NewDB().

func (*DB) DeepClone 

func (db *DB) DeepClone() *DB

DeepClone creates a full copy of the database (MVCC-light snapshot). Note: This is not copy-on-write; it creates a full copy (simple but O(n)).

The result is a shadow: it carries the full runtime state and schema of the original, but statements executed against it do not write to the WAL. Call PromoteShadow if it is going to replace the live database.

func (*DB) DeleteJob added in v0.11.0 

func (db *DB) DeleteJob(name string) error

DeleteJob deletes a job and unschedules/cancels it when present.

func (*DB) DetachPinnedTableForSchemaWrite added in v0.46.0 

func (db *DB) DetachPinnedTableForSchemaWrite(tenant, name string) bool

DetachPinnedTableForSchemaWrite is the schema-mutation companion to DetachPinnedTableForWrite. ALTER TABLE can edit existing row cells in place while widening or shrinking them, so it deliberately retains a deep clone.

func (*DB) DetachPinnedTableForWrite added in v0.46.0 

func (db *DB) DetachPinnedTableForWrite(tenant, name string) bool

DetachPinnedTableForWrite makes the current in-memory table mapping private to a writer when one or more streams still read its old instance. It must be called with contentMu's write side held and before statement planning or a rollback snapshot captures the table pointer. The caller can then freely mutate the table returned by Get while existing streams continue to read the old, immutable instance.

A false result means no current table was pinned (or this DB uses an evictable backend), so no copy was needed.

func (*DB) DiscardCachedTable added in v0.26.2 

func (db *DB) DiscardCachedTable(tenant, name string) bool

DiscardCachedTable drops tenant/name from the DB's in-memory table cache without persisting it first, then reports whether it was present.

Put always caches the table it creates in db.tenants, even for the evictable backends (ModeIndex/ModeHybrid/ModePagedIndex) that otherwise keep loaded tables out of db.tenants precisely so they can be evicted (see Get). A caller that then writes directly to the backend -- bypassing that cached pointer entirely, as AppendRowsFast does -- leaves it stale: it still reads back whatever Put originally wrote (typically an empty table), not what is now on disk. DiscardCachedTable clears that stale pointer so the next Get falls through to the backend and sees current data.

This is only safe when the backend is already known to be authoritative for tenant/name. Calling it on a table whose in-memory copy has mutations the backend does not yet have loses those mutations.

func (*DB) Drop 

func (db *DB) Drop(tn, name string) error

Drop removes a table from the tenant (and from the backend if attached).

func (*DB) Evict added in v0.5.4 

func (db *DB) Evict(tenant, name string) error

Evict removes a table from the in-memory cache without deleting it from the backend. This is only meaningful for disk-backed modes; in ModeMemory the data would be lost. Returns an error if no backend is attached.

func (*DB) Extensions added in v0.21.0 

func (db *DB) Extensions() []ExtensionInfo

Extensions returns a deterministic snapshot of extensions active for this DB. Mutating the returned values cannot change the database registry.

func (*DB) Get 

func (db *DB) Get(tn, name string) (*Table, error)

Get returns a table by name for the given tenant. When a StorageBackend is attached, tables not found in memory are loaded from the backend on demand (lazy loading).

func (*DB) HealthCheck added in v0.11.0 

func (db *DB) HealthCheck() DBHealth

HealthCheck returns a production-oriented lifecycle and storage snapshot.

func (*DB) IsClosed added in v0.24.0 

func (db *DB) IsClosed() bool

IsClosed reports whether Close has completed for this database. The SQL driver uses it to let the connection pool discard connections whose database went away underneath them, rather than returning errors from each one.

func (*DB) IsRBACEnabled added in v0.13.0 

func (db *DB) IsRBACEnabled() bool

IsRBACEnabled reports whether Execute currently enforces RBAC permissions. A convenience delegate to db.Catalog().IsRBACEnabled.

func (*DB) IsReadOnly added in v0.11.0 

func (db *DB) IsReadOnly() bool

IsReadOnly reports whether the database is in read-only mode.

func (*DB) IsShadow added in v0.24.0 

func (db *DB) IsShadow() bool

IsShadow reports whether this database is an uncommitted working copy.

func (*DB) IsTablePinnedForStream added in v0.46.0 

func (db *DB) IsTablePinnedForStream(table *Table) bool

IsTablePinnedForStream reports whether table is still the source of a live stream. Statement rollback uses it to avoid restoring state in place into a table a stream is concurrently reading.

func (*DB) JobScheduler added in v0.11.0 

func (db *DB) JobScheduler() *Scheduler

JobScheduler returns the active database job scheduler, if any.

func (*DB) ListTables 

func (db *DB) ListTables(tn string) []*Table

ListTables returns the tables in a tenant sorted by name. When a StorageBackend is attached, tables that exist on disk but are not currently loaded into memory are loaded on demand.

func (*DB) ListTenants added in v0.5.4 

func (db *DB) ListTenants() []string

ListTenants returns the names of all tenants that have at least one table.

func (*DB) LocatePagedIndex added in v0.28.0 

func (db *DB) LocatePagedIndex(tenant, table, indexName string) (*PagedIndexLocator, bool, error)

LocatePagedIndex resolves one immutable physical index for a typed serving path. The boolean is false for non-paged backends or missing indexes.

func (*DB) LockContentForRead added in v0.12.0 

func (db *DB) LockContentForRead()

LockContentForRead acquires the database's content lock for a read-only statement (SELECT/EXPLAIN/PRAGMA). Multiple readers may hold this concurrently; it excludes concurrent LockContentForWrite callers. Callers must call UnlockContentForRead exactly once, typically via defer, and must not call it again re-entrantly on the same goroutine (sync.RWMutex is not reentrant) — nested statement execution within the engine must bypass this lock rather than re-acquire it.

func (*DB) LockContentForWrite added in v0.12.0 

func (db *DB) LockContentForWrite()

LockContentForWrite acquires the database's content lock exclusively, for any statement that may mutate table rows/columns (INSERT/UPDATE/DELETE and all DDL). See LockContentForRead for the re-entrancy caveat.

func (*DB) MVCC 

func (db *DB) MVCC() *MVCCManager

MVCC returns the MVCC manager.

func (*DB) MarkAdvancedWALMetadataDirty added in v0.46.0 

func (db *DB) MarkAdvancedWALMetadataDirty()

MarkAdvancedWALMetadataDirty records that an uncommitted shadow made a schema/catalog mutation with no row-level AdvancedWAL representation. The SQL driver consumes this fact only after it has published the shadow at a successful COMMIT; a discarded shadow therefore never schedules a live checkpoint or invalidates replica feed state.

func (*DB) MetaSnapshot added in v0.24.0 

func (db *DB) MetaSnapshot() *DB

MetaSnapshot captures every table's identity and Version but none of its rows. It is the "before" image the engine diffs against to decide what one statement changed for WALManager logging.

It replaces the previous approach of reusing the statement's full rollback snapshot for that diff, which forced a deep copy of the entire database on every INSERT into a WAL-backed database and still could not see a CREATE or DROP TABLE, because DDL takes no rollback snapshot at all. A metadata snapshot costs O(number of tables) and detects created, dropped and mutated tables alike.

func (*DB) MigrateToBackend added in v0.5.4 

func (db *DB) MigrateToBackend(b StorageBackend) error

MigrateToBackend copies all in-memory tables to the given backend and attaches it. This enables migrating a ModeMemory database to ModeDisk or ModeHybrid at runtime.

func (*DB) OpenPagedIndexRangeCursor added in v0.46.0 

func (db *DB) OpenPagedIndexRangeCursor(tenant, table, indexName string, startKey, endKey []byte) (cursor *PagedRowCursor, ok bool, err error)

OpenPagedIndexRangeCursor starts an ordered index cursor over inclusive raw index-key bounds. The keys are copied because planner scratch buffers must not outlive plan construction. ok is false for non-paged storage.

func (*DB) OpenPagedTableCursor added in v0.46.0 

func (db *DB) OpenPagedTableCursor(tenant, table string) (cursor *PagedRowCursor, ok bool, err error)

OpenPagedTableCursor starts an ordered table-row cursor. ok is false for a database without the ModePagedIndex backend.

func (*DB) PagedIndexMetadata added in v0.20.0 

func (db *DB) PagedIndexMetadata(tenant, table string) (*Table, bool, error)

PagedIndexMetadata returns a schema-only table for the immutable ModePagedIndex backend. The returned table has column and secondary-index metadata but no rows; it lets a planner select an on-disk index before a full-table compatibility load is considered.

func (*DB) PagedIndexRows added in v0.20.0 

func (db *DB) PagedIndexRows(tenant, table, indexName string, values []any) ([][]any, bool, error)

PagedIndexRows performs an exact composite seek in a ModePagedIndex artifact. The boolean is false when no physical index with that name exists; a true result with an empty row slice is a valid negative lookup.

func (*DB) PinTableForStream added in v0.46.0 

func (db *DB) PinTableForStream(table *Table) (release func(), ok bool)

PinTableForStream pins table as the immutable source of one direct result stream. The caller must already hold contentMu's read side while it obtains and pins the table, so a writer cannot slip between planning and the pin.

The returned release function is idempotent. It must run when the producer no longer reads the table. A pin is intentionally unavailable for evictable backends: their query-scoped table leases live in backend-owned pools, which cannot be atomically swapped through DB.tenants. Paged read-only artifacts use their own cursor source instead of a Table pin.

func (*DB) PromoteShadow added in v0.24.0 

func (db *DB) PromoteShadow()

PromoteShadow clears the shadow flag, declaring that this database's contents are committed and that statements executed against it from now on are responsible for their own write-ahead logging. Only the SQL driver calls it, after it has durably logged the transaction that produced the shadow.

func (*DB) Put 

func (db *DB) Put(tn string, t *Table) error

Put adds a new table to the tenant; returns error if it already exists. When a StorageBackend is attached, the table is also checked against the backend to prevent duplicates, and optionally persisted immediately when SyncOnMutate is configured.

func (*DB) RegisterJob added in v0.11.0 

func (db *DB) RegisterJob(job *CatalogJob) error

RegisterJob registers a job and refreshes the live scheduler when present.

func (*DB) ReleaseStatementSnapshot added in v0.33.0 

func (db *DB) ReleaseStatementSnapshot(snapshot *StatementSnapshot)

ReleaseStatementSnapshot discards a snapshot that will not be used, because its statement succeeded. Callers must invoke it exactly once per snapshot they take, whether or not they also called RestoreStatementSnapshot: it is what stops the next statement's catalog writes from being captured into this statement's rollback point.

func (*DB) Replace added in v0.41.0 

func (db *DB) Replace(tn string, t *Table) error

Replace overwrites tenant/name with t in one step, unlike a Drop followed by a Put: those are two separate lock acquisitions, so a concurrent reader calling Get in between them observes the table as briefly not existing at all, even though conceptually it never stopped having data (see refreshMaterializedView, the caller this exists for -- a materialized view's cache table is rebuilt by dropping the old one and putting the new one back under the same name). Replace instead reassigns the table map's entry directly under a single db.mu.Lock, so there is no window where tenant/name is absent. Unlike Put it never rejects an existing table -- that check exists for CREATE TABLE's "already exists" error, which does not apply to a caller that is deliberately replacing content.

func (*DB) RestartJobScheduler added in v0.11.0 

func (db *DB) RestartJobScheduler(executor JobExecutor) error

RestartJobScheduler restarts the database job scheduler with the executor.

func (*DB) RestoreStatementSnapshot added in v0.21.0 

func (db *DB) RestoreStatementSnapshot(snapshot *StatementSnapshot)

RestoreStatementSnapshot rolls a database back to snapshot. It restores pre-existing tables in place and removes tables created by the failed statement. The caller must hold DB's content write lock.

func (*DB) ScanPagedIndexRange added in v0.27.1 

func (db *DB) ScanPagedIndexRange(tenant, table, indexName string, startValues, endValues []any, fn func([]any) bool) (bool, error)

ScanPagedIndexRange streams rows selected by an ordered composite index. startValues and endValues use the same value types as PagedIndexRows.

func (*DB) ScanRowsFast added in v0.26.2 

func (db *DB) ScanRowsFast(tenant, tableName string, fn func(row []any) bool) (ok bool, err error)

ScanRowsFast streams tenant/tableName's rows to fn without materializing the whole table -- when the attached backend is ModePagedIndex (see PagedIndexBackend.ScanRows). ok is false when there is no such backend attached; the caller should fall back to Get and range over table.Rows. fn returning false stops the scan early.

ExportMBTiles (internal/importer) is the motivating caller: writing a tileset larger than memory back out to a .mbtiles file should not first need to load it into memory, which is what Get would otherwise force.

func (*DB) SetBackend added in v0.5.4 

func (db *DB) SetBackend(b StorageBackend)

SetBackend attaches a StorageBackend and sets the storage mode. This is primarily used internally by OpenDB; calling it on a running database should be done with care.

func (*DB) SetRBACEnabled added in v0.13.0 

func (db *DB) SetRBACEnabled(enabled bool)

SetRBACEnabled overrides RBAC's default opt-in-via-CreateUser behavior; see CatalogManager.SetRBACEnabled for the full explanation. A convenience delegate to db.Catalog().SetRBACEnabled, provided directly on DB to mirror SetReadOnly above (the same "toggle a behavior" shape).

func (*DB) SetReadOnly added in v0.11.0 

func (db *DB) SetReadOnly(ro bool)

SetReadOnly toggles read-only mode. While enabled, the SQL engine rejects INSERT/UPDATE/DELETE/DDL with an error; SELECT, EXPLAIN, and PRAGMA still work. Safe to call concurrently with running queries.

func (*DB) SnapshotCatalog added in v0.24.0 

func (db *DB) SnapshotCatalog() CatalogSnapshot

SnapshotCatalog copies this database's catalog contents for later comparison.

func (*DB) SnapshotForAppendOnlyTableStatement added in v0.22.0 

func (db *DB) SnapshotForAppendOnlyTableStatement(tenant, name string) (*StatementSnapshot, error)

SnapshotForAppendOnlyTableStatement captures the lightweight rollback state for a statement that can only append rows to one table. It is intended for the executor's triggerless INSERT fast path (see appendOnlySnapshotTarget in package engine) and works whether or not the table carries secondary indexes -- restoreAppendOnlyTable removes an appended row's index entries before truncating it away. Other callers must use SnapshotForTableStatement or SnapshotForStatement.

func (*DB) SnapshotForRowDeleteStatement added in v0.25.0 

func (db *DB) SnapshotForRowDeleteStatement(tenant, name string, rowIDs []int) (*StatementSnapshot, error)

SnapshotForRowDeleteStatement captures the single candidate row and table metadata needed to undo a PRIMARY KEY/UNIQUE point delete. The caller must use a table/full snapshot when triggers, foreign keys, or secondary indexes can observe or mutate additional state.

func (*DB) SnapshotForRowUpdateStatement added in v0.25.0 

func (db *DB) SnapshotForRowUpdateStatement(tenant, name string, rowIDs []int) (*StatementSnapshot, error)

SnapshotForRowUpdateStatement captures a lightweight rollback point for an UPDATE whose candidate row IDs are already known through a PRIMARY KEY or UNIQUE constraint seek. Callers must use a table/full snapshot when triggers or foreign-key actions can mutate state beyond these rows. Secondary-index entries for the bounded rows are restored directly on rollback.

func (*DB) SnapshotForStatement added in v0.21.0 

func (db *DB) SnapshotForStatement() *StatementSnapshot

SnapshotForStatement captures all table and catalog state needed to undo a failed mutating statement, including mutations issued by its triggers. The caller must already hold DB's content write lock.

func (*DB) SnapshotForTableStatement added in v0.22.0 

func (db *DB) SnapshotForTableStatement(tenant, name string) (*StatementSnapshot, error)

SnapshotForTableStatement captures one table and the catalog state needed to roll back a statement known to mutate only that table. Callers must use SnapshotForStatement instead when triggers or foreign-key actions could affect other tables. The caller must already hold DB's content write lock.

func (*DB) SnapshotForTx added in v0.21.0 

func (db *DB) SnapshotForTx() (base *DB, shadow *DB)

SnapshotForTx creates the pair of snapshots a SQL transaction needs while copying row data only once instead of twice.

shadow is a full deep clone that receives the transaction's writes. base is a lightweight snapshot that records each table's identity and Version but no rows: the only consumers of the base — CollectWALChanges and the driver's conflict detection — read Table.Version and table existence exclusively and never inspect rows. Copying rows into the base (as DeepClonePair does) would therefore waste memory proportional to the entire database on every Begin.

func (*DB) StartJobScheduler added in v0.11.0 

func (db *DB) StartJobScheduler(executor JobExecutor) error

StartJobScheduler starts the database job scheduler with the given executor.

func (*DB) StatementAdvancedWAL added in v0.24.0 

func (db *DB) StatementAdvancedWAL() *AdvancedWAL

StatementAdvancedWAL is the AdvancedWAL counterpart of StatementWAL. Unlike the basic WALManager, AdvancedWAL records row-level operations grouped into explicit transactions, so a shadow does not have to stop logging: it logs into the ambient transaction the driver opened at BEGIN (see BeginAmbientWALTx), which recovery only replays once a matching commit record exists.

func (*DB) StatementWAL added in v0.24.0 

func (db *DB) StatementWAL() *WALManager

StatementWAL returns the WALManager that a statement executing against db must append to, or nil when statement-level logging is not this database's responsibility.

It differs from WAL in exactly one case: a transaction shadow, where it returns nil. A statement inside BEGIN…COMMIT is not durable until the transaction commits, so logging it as it runs would leave a committed-looking record on disk that recovery replays even when the transaction was rolled back. The driver logs the transaction as a whole against the live database instead.

func (*DB) StopJobScheduler added in v0.11.0 

func (db *DB) StopJobScheduler()

StopJobScheduler stops the database job scheduler if it is running.

func (*DB) StorageMode added in v0.5.4 

func (db *DB) StorageMode() StorageMode

StorageMode returns the active storage mode.

func (*DB) Sync added in v0.5.4 

func (db *DB) Sync() error

Sync flushes all dirty in-memory tables to the storage backend. For ModeMemory and ModeWAL this is a no-op (those modes use SaveToFile / WAL checkpoints respectively). For ModeDisk, ModeJSON, ModeHybrid, and ModeIndex, tables whose version has changed since the last save are written to disk.

func (*DB) SyncTable added in v0.5.4 

func (db *DB) SyncTable(tenant string, t *Table) error

SyncTable flushes a single table to the backend. This is called by the engine after mutations when SyncOnMutate is enabled.

func (*DB) TableExists added in v0.5.4 

func (db *DB) TableExists(tenant, name string) bool

TableExists reports whether the named table exists, checking both in-memory tables and the storage backend.

func (*DB) UnlockContentForRead added in v0.12.0 

func (db *DB) UnlockContentForRead()

UnlockContentForRead releases a lock taken by LockContentForRead.

func (*DB) UnlockContentForWrite added in v0.12.0 

func (db *DB) UnlockContentForWrite()

UnlockContentForWrite releases a lock taken by LockContentForWrite.

func (*DB) Use added in v0.21.0 

func (db *DB) Use(extension Extension) error

Use activates one statically linked Go extension for this database. An extension name can be activated only once per DB instance. Extensions are not unloadable because their registration may affect executing queries or other database state.

func (*DB) WAL 

func (db *DB) WAL() *WALManager

WAL returns the configured WAL manager (may be nil).

type DBHealth added in v0.11.0 

type DBHealth struct {
	OK                bool
	Mode              StorageMode
	ModeName          string
	Path              string
	Closed            bool
	Closing           bool
	ReadOnly          bool
	SchedulerRunning  bool
	WALActive         bool
	AdvancedWALActive bool
	Tenants           int
	Tables            int
	BackendStats      BackendStats
	LastSyncAt        time.Time
	LastCloseAt       time.Time
	Recovery          RecoveryStatus
	Error             string
}

DBHealth is a point-in-time operational snapshot for production probes.

type DerivedCloner added in v0.41.0 

type DerivedCloner interface {
	CloneDerived() any
}

DerivedCloner is implemented by derived state (see the field above) that knows how to produce an independent copy of itself. cloneTable checks for it via a type assertion so storage never has to know the concrete type stored there — today, only internal/engine's constraint-index cache implements it. Derived state that does not implement DerivedCloner is simply dropped on clone, exactly as before this interface existed.

CloneDerived must return a value that shares no mutable state with the receiver: the clone and the table it was cloned from immediately start diverging independently (new rows appended, rows rolled back, etc.), so anything shared between them (e.g. a map, or a []int slice that either side might later append to) would let one side's later mutation corrupt the other's cached state.

type DiskBackend added in v0.5.4 

type DiskBackend struct {
	// contains filtered or unexported fields
}

DiskBackend stores each table as an individual file under a directory tree: 

<dir>/<tenant>/<tablename>.tbl        (GOB-encoded diskTable, default)
<dir>/<tenant>/<tablename>.json       (JSON-encoded diskTable, ModeJSON)
<dir>/manifest.json                   (lightweight metadata index)

Tables are loaded into memory on demand (via LoadTable) and written back on SaveTable / Sync. This minimises RAM usage while keeping the file format compatible with the existing GOB serialisation (or, for ModeJSON, a JSON encoding of the same diskTable structure).

func NewDiskBackend added in v0.5.4 

func NewDiskBackend(dir string, compress bool) (*DiskBackend, error)

NewDiskBackend opens (or creates) a disk-backed storage directory using GOB-encoded per-table files (ModeDisk).

func NewJSONBackend added in v0.12.0 

func NewJSONBackend(dir string, compress bool) (*DiskBackend, error)

NewJSONBackend opens (or creates) a disk-backed storage directory using JSON-encoded per-table files (ModeJSON). Functionally identical to NewDiskBackend — same manifest, same lazy loading, same dirty tracking — only the per-table file encoding differs. JSON files are larger than GOB and big.Rat/uuid.UUID values round-trip as plain strings (see normalizeForJSON), but the files can be read, diffed, or hand-edited with any text tool, which GOB files cannot.

func (*DiskBackend) Close added in v0.5.4 

func (b *DiskBackend) Close() error

Close is like Sync but also cleans up any resources.

func (*DiskBackend) DeleteTable added in v0.5.4 

func (b *DiskBackend) DeleteTable(tenant, name string) error

DeleteTable removes a table file from disk and updates the manifest.

func (*DiskBackend) GetMeta added in v0.5.4 

func (b *DiskBackend) GetMeta(tenant, name string) *TableMeta

GetMeta returns table metadata without loading the full table. Returns nil if the table is unknown.

func (*DiskBackend) ImportFromDB added in v0.5.4 

func (b *DiskBackend) ImportFromDB(db *DB) error

ImportFromDB takes all tables from an existing in-memory DB and writes them as individual table files. This is the migration path from ModeMemory / GOB checkpoints to ModeDisk.

func (*DiskBackend) IsDirty added in v0.5.4 

func (b *DiskBackend) IsDirty(tenant, name string, currentVersion int) bool

IsDirty reports whether the named table has been modified since it was last saved. Uses the version field for comparison.

func (*DiskBackend) IsReadOnly added in v0.20.0 

func (b *DiskBackend) IsReadOnly() bool

IsReadOnly reports whether persistence has been disabled for this backend.

func (*DiskBackend) ListTableNames added in v0.5.4 

func (b *DiskBackend) ListTableNames(tenant string) ([]string, error)

ListTableNames returns all known table names for a tenant (on disk or in memory).

func (*DiskBackend) LoadTable added in v0.5.4 

func (b *DiskBackend) LoadTable(tenant, name string) (*Table, error)

LoadTable reads a table from its GOB file on disk. Returns (nil, nil) if the table does not exist.

func (*DiskBackend) Mode added in v0.5.4 

func (b *DiskBackend) Mode() StorageMode

func (*DiskBackend) SaveTable added in v0.5.4 

func (b *DiskBackend) SaveTable(tenant string, t *Table) error

SaveTable writes a table to its GOB file on disk and updates the manifest.

func (*DiskBackend) SetEncryptor added in v0.13.0 

func (b *DiskBackend) SetEncryptor(enc *Encryptor)

SetEncryptor enables (or, passing nil, disables) AES-256-GCM encryption at rest for every table file this backend writes from this point on. Existing files already on disk are re-encrypted (or decrypted) lazily, the next time each is saved — SetEncryptor doesn't rewrite files that aren't touched again, so a backend directory can end up with a mix of encrypted and unencrypted files if the encryptor is changed mid-lifetime. For a clean transition, set the encryptor once at open time (see StorageConfig.EncryptionKey) rather than toggling it on a live backend.

func (*DiskBackend) SetReadOnly added in v0.20.0 

func (b *DiskBackend) SetReadOnly(readOnly bool)

SetReadOnly makes the backend observational. It is normally set by OpenDB before the backend is exposed to callers.

func (*DiskBackend) Stats added in v0.5.4 

func (b *DiskBackend) Stats() BackendStats

func (*DiskBackend) Sync added in v0.5.4 

func (b *DiskBackend) Sync() error

Sync writes the manifest (individual table files are written immediately by SaveTable). The caller should call SaveTable for each dirty table before calling Sync, or use DB.Sync which does this automatically.

func (*DiskBackend) TableExists added in v0.5.4 

func (b *DiskBackend) TableExists(tenant, name string) bool

TableExists reports whether the table is known to the backend.

type Encryptor added in v0.13.0 

type Encryptor struct {
	// contains filtered or unexported fields
}

Encryptor performs authenticated AES-256-GCM encryption/decryption of byte blobs, used to encrypt table files at rest.

func NewEncryptor added in v0.13.0 

func NewEncryptor(key []byte) (*Encryptor, error)

NewEncryptor creates an Encryptor from a 32-byte AES-256 key (typically from DeriveKeyFromPassphrase, or any caller-supplied 32 cryptographically random bytes for key-file-based setups).

func (*Encryptor) Decrypt added in v0.13.0 

func (e *Encryptor) Decrypt(blob []byte) ([]byte, error)

Decrypt reverses Encrypt. It returns an error — rather than corrupted or wrong plaintext — if blob is truncated, was encrypted under a different key, or was altered after encryption: GCM's authentication tag check fails closed on any of those.

func (*Encryptor) Encrypt added in v0.13.0 

func (e *Encryptor) Encrypt(plaintext []byte) ([]byte, error)

Encrypt returns nonce||ciphertext||tag as a single self-contained blob safe to write directly to disk (Decrypt expects exactly this layout). A fresh random nonce is generated per call, which is required for GCM's security guarantees — reusing a nonce under the same key breaks confidentiality — and cheap enough to pay per file write.

type Extension added in v0.21.0 

type Extension interface {
	ExtensionInfo() ExtensionInfo
	Register(db *DB) error
}

Extension is the stable entry point for statically linked Go extensions. An extension package is imported by the application and explicitly enabled with db.Use. tinySQL intentionally does not use Go's plugin package: its shared-object ABI is platform/toolchain-dependent and unavailable to many tinySQL targets, including WebAssembly and TinyGo.

Register may register metadata, functions, table functions, storage helpers, or other supported public extension points on db. Returning an error leaves the extension absent from DB.Extensions and sys.extensions.

Register runs with the full privileges of the host process, unsandboxed, exactly like any other imported Go package's init-time code: DB.Use is not a permission boundary, so only activate extensions whose source you trust.

type ExtensionCapability added in v0.21.0 

type ExtensionCapability string

ExtensionCapability is a self-reported, unverified resource category an extension declares it may need. tinySQL never checks it against what the extension's code actually does, and never will: a statically linked Go extension runs as ordinary code in the same process with the same privileges as the rest of the application, so there is no boundary to enforce a capability against. Declarations exist purely so a caller can review what an extension claims to need -- via ExtensionInfo.Capabilities or sys.extensions -- before calling DB.Use, the same way you would read a new dependency's imports before adding it. 

const (
	CapabilityFilesystem ExtensionCapability = "filesystem"
	CapabilityNetwork    ExtensionCapability = "network"
	CapabilityWrite      ExtensionCapability = "write"
	CapabilitySecrets    ExtensionCapability = "secrets"
)

type ExtensionInfo added in v0.21.0 

type ExtensionInfo struct {
	Name         string
	Version      string
	Description  string
	Capabilities []ExtensionCapability
	LoadedAt     time.Time
}

ExtensionInfo describes one statically linked Go extension activated for a database instance. LoadedAt is set by DB.Use and is not supplied by the extension implementation.

type FTSDocument added in v0.45.0 

type FTSDocument struct {
	TermStart  int32
	TermCount  int32
	TokenStart int32
	TokenCount int32
	DocLen     float64
	Valid      bool
}

FTSDocument is the compact per-row directory for a persisted FTS index. The offsets address the index's contiguous term/frequency/token arenas.

type FTSIndex added in v0.45.0 

type FTSIndex struct {
	Format        int
	Version       int
	StructVersion int
	BuiltRows     int
	Docs          []FTSDocument
	AvgDocLen     float64
	TotalDocLen   float64
	Postings      map[string][]int32
	NumDocs       int
	TermIDs       map[string]int32
	DocTermIDs    []int32
	DocTermCounts []int32
	DocTokenIDs   []int32
	// Stale* count arena entries superseded by UPDATE-only maintenance. They
	// let the executor compact before a long update-heavy workload retains an
	// unbounded trail of obsolete document terms/tokens.
	StaleTerms  int
	StaleTokens int
}

FTSIndex is the backend-neutral persisted representation of the BM25 document index used by FTS_SEARCH. Format allows future encoding changes.

type ForeignKeyRef 

type ForeignKeyRef struct {
	Table    string
	Column   string
	OnDelete ReferentialAction
	OnUpdate ReferentialAction
}

ForeignKeyRef describes a foreign key reference target.

type Grant added in v0.12.0 

type Grant struct {
	Permission Permission
	Schema     string
	Table      string
}

Grant authorizes Permission on every table in Schema (or every schema, if Schema is "*") named Table (or every table, if Table is "*").

type HybridBackend added in v0.5.4 

type HybridBackend struct {
	// contains filtered or unexported fields
}

HybridBackend keeps frequently-accessed tables in an LRU memory cache backed by individual table files on disk. When the cache exceeds the configured memory limit, cold tables are evicted. This gives near in- memory performance for the working set while allowing the full database to exceed available RAM.

For ModeIndex, the same structure is used with a much smaller memory limit and aggressive eviction – essentially only table schemas (loaded as metadata from the manifest) stay resident.

func NewHybridBackend added in v0.5.4 

func NewHybridBackend(dir string, maxMemoryBytes int64, compress bool, mode StorageMode) (*HybridBackend, error)

NewHybridBackend creates a HybridBackend.

  • dir: directory for table files
  • maxMemoryBytes: memory limit (0 = 256 MB default)
  • compress: gzip table files
  • mode: ModeHybrid or ModeIndex

func (*HybridBackend) Close added in v0.5.4 

func (h *HybridBackend) Close() error

Close flushes dirty tables and closes the disk backend.

func (*HybridBackend) DeleteTable added in v0.5.4 

func (h *HybridBackend) DeleteTable(tenant, name string) error

DeleteTable removes from both disk and cache.

func (*HybridBackend) Disk added in v0.5.4 

func (h *HybridBackend) Disk() *DiskBackend

Disk returns the underlying DiskBackend for advanced operations (e.g. migration, metadata access).

func (*HybridBackend) IsDirty added in v0.23.1 

func (h *HybridBackend) IsDirty(tenant, name string, currentVersion int) bool

IsDirty delegates to the underlying DiskBackend's version-based dirty check, satisfying the dirtyTracker interface DB.Sync uses for both db.tenants entries and PooledTables leases.

func (*HybridBackend) ListTableNames added in v0.5.4 

func (h *HybridBackend) ListTableNames(tenant string) ([]string, error)

ListTableNames returns names from the disk manifest (authoritative source).

func (*HybridBackend) LoadTable added in v0.5.4 

func (h *HybridBackend) LoadTable(tenant, name string) (*Table, error)

LoadTable tries the in-memory cache first, then loads from disk.

func (*HybridBackend) MarkDirty added in v0.5.4 

func (h *HybridBackend) MarkDirty(tenant, name string)

MarkDirty records that a table has been modified in memory and needs to be flushed on the next Sync. Called by DB after mutations.

func (*HybridBackend) Mode added in v0.5.4 

func (h *HybridBackend) Mode() StorageMode

Mode reports the configured hybrid storage mode.

func (*HybridBackend) PooledTables added in v0.23.1 

func (h *HybridBackend) PooledTables() []PooledTableRef

PooledTables returns every table currently resident in the buffer pool, satisfying the pooledTableSource interface. DB.Sync/DB.Close use this to find and flush a table that DB.Get loaded lazily for this evictable storage mode (ModeHybrid/ModeIndex) without registering it in DB.tenants.

func (*HybridBackend) SaveTable added in v0.5.4 

func (h *HybridBackend) SaveTable(tenant string, t *Table) error

SaveTable writes to disk and updates the cache.

func (*HybridBackend) SetEncryptor added in v0.18.0 

func (h *HybridBackend) SetEncryptor(enc *Encryptor)

SetEncryptor forwards table-file encryption to the underlying DiskBackend.

func (*HybridBackend) SetReadOnly added in v0.20.0 

func (h *HybridBackend) SetReadOnly(readOnly bool)

SetReadOnly forwards the serving-only contract to the durable backend.

func (*HybridBackend) Stats added in v0.5.4 

func (h *HybridBackend) Stats() BackendStats

Stats returns combined disk and in-memory cache statistics.

func (*HybridBackend) Sync added in v0.5.4 

func (h *HybridBackend) Sync() error

Sync writes all dirty in-memory tables to disk.

func (*HybridBackend) TableExists added in v0.5.4 

func (h *HybridBackend) TableExists(tenant, name string) bool

TableExists checks the disk manifest.

type IndexEntry added in v0.19.0 

type IndexEntry struct {
	Key    []byte
	RowIDs []int
}

IndexEntry is one canonical composite key and the table row positions that match it. Entries are sorted by Key, making equality and prefix seeks a binary-search operation rather than a table scan.

type IndexRangeBound added in v0.26.0 

type IndexRangeBound struct {
	Value     any
	Inclusive bool
	Absent    bool
}

IndexRangeBound is one end of a range. Absent means unbounded on that side.

type IsolationLevel 

type IsolationLevel uint8

IsolationLevel defines transaction isolation semantics. 

const (
	ReadCommitted IsolationLevel = iota
	RepeatableRead
	SnapshotIsolation
	Serializable
)

type JobExecutor added in v0.5.1 

type JobExecutor interface {
	ExecuteSQL(ctx context.Context, sql string) (interface{}, error)
}

JobExecutor interface allows the scheduler to execute SQL without circular dependencies

type LRUNode 

type LRUNode struct {
	// contains filtered or unexported fields
}

LRUNode is a node in the LRU doubly-linked list.

type LRUQueue 

type LRUQueue struct {
	// contains filtered or unexported fields
}

LRUQueue implements least-recently-used eviction.

func NewLRUQueue 

func NewLRUQueue() *LRUQueue

NewLRUQueue creates an empty LRU queue.

func (*LRUQueue) Access 

func (lru *LRUQueue) Access(key string, table *CachedTable)

Access moves a node to the front (most recent).

func (*LRUQueue) Add 

func (lru *LRUQueue) Add(key string, table *CachedTable)

Add adds a node to the LRU queue (most recent).

func (*LRUQueue) Remove 

func (lru *LRUQueue) Remove(key string)

Remove removes a node from the queue.

func (*LRUQueue) RemoveLRU 

func (lru *LRUQueue) RemoveLRU() (string, *CachedTable)

RemoveLRU removes and returns the least recently used node.

type LSN 

type LSN uint64

LSN (Log Sequence Number) provides total ordering of log records.

type MVCCManager 

type MVCCManager struct {
	// contains filtered or unexported fields
}

MVCCManager coordinates transaction IDs, timestamps, and visibility.

func NewMVCCManager 

func NewMVCCManager() *MVCCManager

NewMVCCManager creates a new MVCC coordinator.

func (*MVCCManager) AbortTx 

func (m *MVCCManager) AbortTx(tx *TxContext)

AbortTx marks a transaction as aborted.

func (*MVCCManager) BeginTx 

func (m *MVCCManager) BeginTx(level IsolationLevel) *TxContext

BeginTx starts a new transaction and returns its context.

func (*MVCCManager) CommitTx 

func (m *MVCCManager) CommitTx(tx *TxContext) (Timestamp, error)

CommitTx marks a transaction as committed and records its commit timestamp.

func (*MVCCManager) GCWatermark 

func (m *MVCCManager) GCWatermark() Timestamp

GCWatermark returns the timestamp before which row versions can be garbage collected.

func (*MVCCManager) IsVisible 

func (m *MVCCManager) IsVisible(tx *TxContext, rv *RowVersion) bool

IsVisible determines if a row version is visible to a transaction.

type MVCCTable 

type MVCCTable struct {
	*Table
	// contains filtered or unexported fields
}

MVCCTable extends Table with version chains.

func NewMVCCTable 

func NewMVCCTable(name string, cols []Column, isTemp bool) *MVCCTable

NewMVCCTable creates a table with MVCC support.

func (*MVCCTable) DeleteVersion 

func (t *MVCCTable) DeleteVersion(tx *TxContext, rowID int64) error

DeleteVersion marks a row version as deleted.

func (*MVCCTable) GarbageCollect 

func (t *MVCCTable) GarbageCollect(watermark Timestamp) int

GarbageCollect removes old row versions that are no longer visible.

func (*MVCCTable) GetVisibleVersion 

func (t *MVCCTable) GetVisibleVersion(mvcc *MVCCManager, tx *TxContext, rowID int64) *RowVersion

GetVisibleVersion returns the visible version of a row for the given transaction.

func (*MVCCTable) InsertVersion 

func (t *MVCCTable) InsertVersion(tx *TxContext, data []any) int64

InsertVersion adds a new row version.

func (*MVCCTable) UpdateVersion 

func (t *MVCCTable) UpdateVersion(tx *TxContext, rowID int64, newData []any) error

UpdateVersion creates a new version for an update.

type MemoryBackend added in v0.5.4 

type MemoryBackend struct {
	// contains filtered or unexported fields
}

MemoryBackend is a no-op backend used by ModeMemory. All data lives exclusively in the DB's in-memory tenants map; the backend simply reports that tables don't exist on any backing store (so the DB never tries to load from disk). Save/Sync are intentional no-ops – persistence is the caller's responsibility (via SaveToFile or the WAL).

func NewMemoryBackend added in v0.5.4 

func NewMemoryBackend(savePath string) *MemoryBackend

NewMemoryBackend creates a MemoryBackend. If savePath is non-empty, Close performs a final SaveToFile to that path.

func (*MemoryBackend) Close added in v0.5.4 

func (m *MemoryBackend) Close() error

func (*MemoryBackend) DeleteTable added in v0.5.4 

func (m *MemoryBackend) DeleteTable(_, _ string) error

DeleteTable is a no-op because memory mode has no durable table files.

func (*MemoryBackend) ListTableNames added in v0.5.4 

func (m *MemoryBackend) ListTableNames(_ string) ([]string, error)

ListTableNames reports no durable table names in memory mode.

func (*MemoryBackend) LoadTable added in v0.5.4 

func (m *MemoryBackend) LoadTable(_, _ string) (*Table, error)

LoadTable reports no durable table because memory mode has no backing store.

func (*MemoryBackend) Mode added in v0.5.4 

func (m *MemoryBackend) Mode() StorageMode

func (*MemoryBackend) SaveTable added in v0.5.4 

func (m *MemoryBackend) SaveTable(_ string, _ *Table) error

SaveTable is a no-op because memory mode keeps rows in the owning DB.

func (*MemoryBackend) Stats added in v0.5.4 

func (m *MemoryBackend) Stats() BackendStats

func (*MemoryBackend) Sync added in v0.5.4 

func (m *MemoryBackend) Sync() error

Sync is a no-op because memory mode has no durable writes to flush.

func (*MemoryBackend) TableExists added in v0.5.4 

func (m *MemoryBackend) TableExists(_, _ string) bool

TableExists reports false because memory mode has no durable table files.

type MemoryPolicy 

type MemoryPolicy struct {
	// Maximum memory usage in bytes (0 = unlimited)
	MaxMemoryBytes int64

	// Cache eviction strategy
	Strategy CacheStrategy

	// Start evicting when memory usage exceeds this ratio (0.0-1.0)
	EvictionThreshold float64

	// Tables that should always stay in memory
	PinnedTables []string

	// Tables that should never be cached
	IgnoreTables []string

	// Enable eviction (if false, OOM when limit reached)
	EnableEviction bool

	// Number of tables to evict in one batch
	EvictionBatchSize int

	// Track access patterns for better eviction decisions
	TrackAccessPatterns bool

	// Time window for access tracking
	AccessWindow time.Duration
}

MemoryPolicy defines memory management configuration.

func DefaultMemoryPolicy 

func DefaultMemoryPolicy() *MemoryPolicy

DefaultMemoryPolicy returns a sensible default configuration.

func LimitedMemoryPolicy 

func LimitedMemoryPolicy(maxMB int64) *MemoryPolicy

LimitedMemoryPolicy returns a policy with memory limits.

type PagedIndexBackend added in v0.20.0 

type PagedIndexBackend struct {
	// contains filtered or unexported fields
}

PagedIndexBackend adapts the pager B+Tree format to StorageBackend. Unlike ModeIndex's legacy GOB table files, it persists table rows and materialized secondary indexes in distinct B+Trees. Exact seeks can therefore read an index key, its row locator and only the requested rows/BLOBs.

It is deliberately a separate mode: existing ModeIndex directories retain their GOB compatibility, while published `paged_index` artifacts have an explicit format boundary and can be opened safely read-only.

func NewPagedIndexBackend added in v0.20.0 

func NewPagedIndexBackend(dir string, maxMemoryBytes int64, readOnly bool) (*PagedIndexBackend, error)

func (*PagedIndexBackend) AppendRows added in v0.26.2 

func (b *PagedIndexBackend) AppendRows(tenant, name string, newRows [][]any, newIndexKeys map[string][][]byte) (int64, error)

AppendRows adds newRows (and their unique-index keys) to an existing table's on-disk B+Trees without reading or rewriting rows already there -- see pager.PageBackend.AppendRows. Memory and I/O cost are proportional to len(newRows), not to how much of the table already exists, which is what makes building a tileset larger than memory possible: SaveTable cannot do this, since it always frees and rebuilds the whole tree from a complete row set. ImportMBTiles (internal/importer) uses this for a PagedIndexBackend destination.

The table must already exist (created once via SaveTable/CreateTable) and declare its unique index; AppendRows only appends.

func (*PagedIndexBackend) Close added in v0.20.0 

func (b *PagedIndexBackend) Close() error

func (*PagedIndexBackend) DeleteTable added in v0.20.0 

func (b *PagedIndexBackend) DeleteTable(tenant, name string) error

func (*PagedIndexBackend) IndexMetadata added in v0.20.0 

func (b *PagedIndexBackend) IndexMetadata(tenant, name string) (*Table, error)

IndexMetadata creates a schema-only Table with physical index metadata. It is a cheap operation (catalog B+Tree only) and lets the SQL planner choose a page index before calling LoadTable.

func (*PagedIndexBackend) IsDirty added in v0.23.1 

func (b *PagedIndexBackend) IsDirty(tenant, name string, currentVersion int) bool

IsDirty reports whether name's in-memory version has diverged from the version last saved to the pager. Satisfies the dirtyTracker interface.

func (*PagedIndexBackend) IsReadOnly added in v0.20.0 

func (b *PagedIndexBackend) IsReadOnly() bool

func (*PagedIndexBackend) ListTableNames added in v0.20.0 

func (b *PagedIndexBackend) ListTableNames(tenant string) ([]string, error)

func (*PagedIndexBackend) LoadTable added in v0.20.0 

func (b *PagedIndexBackend) LoadTable(tenant, name string) (*Table, error)

func (*PagedIndexBackend) LocateIndex added in v0.28.0 

func (b *PagedIndexBackend) LocateIndex(tenant, table, indexName string) (*PagedIndexLocator, bool, error)

LocateIndex resolves one physical index without reading table rows.

func (*PagedIndexBackend) LookupIndexRows added in v0.20.0 

func (b *PagedIndexBackend) LookupIndexRows(tenant, table, indexName string, values []any) ([][]any, bool, error)

LookupIndexRows resolves a complete canonical composite key through the on-disk index and returns only matching rows. The caller owns the returned slices and can safely project/copy BLOBs without exposing page-cache memory.

func (*PagedIndexBackend) Mode added in v0.20.0 

func (b *PagedIndexBackend) Mode() StorageMode

func (*PagedIndexBackend) PooledTables added in v0.23.1 

func (b *PagedIndexBackend) PooledTables() []PooledTableRef

PooledTables returns every table currently resident in the table-object pool, satisfying the pooledTableSource interface. DB.Sync/DB.Close use this to find and flush a table that DB.Get loaded lazily for ModePagedIndex without registering it in DB.tenants.

func (*PagedIndexBackend) SaveTable added in v0.20.0 

func (b *PagedIndexBackend) SaveTable(tenant string, t *Table) error

func (*PagedIndexBackend) ScanIndexRowsRange added in v0.27.1 

func (b *PagedIndexBackend) ScanIndexRowsRange(tenant, table, indexName string, startValues, endValues []any, fn func([]any) bool) error

ScanIndexRowsRange streams rows in an ordered composite-index interval.

func (*PagedIndexBackend) ScanRows added in v0.26.2 

func (b *PagedIndexBackend) ScanRows(tenant, name string, fn func(row []any) bool) error

ScanRows streams a table's rows to fn without materializing all of them at once -- see pager.PageBackend.ScanTableRows. This bypasses both LoadTable and the table-object pool entirely: a caller using it wants to read a table that may be far larger than the pool's memory budget, so pooling it (or any part of it) would defeat the point.

func (*PagedIndexBackend) SetReadOnly added in v0.20.0 

func (b *PagedIndexBackend) SetReadOnly(readOnly bool)

func (*PagedIndexBackend) Stats added in v0.20.0 

func (b *PagedIndexBackend) Stats() BackendStats

func (*PagedIndexBackend) Sync added in v0.20.0 

func (b *PagedIndexBackend) Sync() error

func (*PagedIndexBackend) TableExists added in v0.20.0 

func (b *PagedIndexBackend) TableExists(tenant, name string) bool

type PagedIndexLocator added in v0.28.0 

type PagedIndexLocator struct {
	// contains filtered or unexported fields
}

PagedIndexLocator is an immutable resolved table/index pair. It is valid for the lifetime of its read-only PagedIndexBackend and lets typed serving paths bypass repeated catalog and metadata-map work.

func (*PagedIndexLocator) ContainsUnique added in v0.28.1 

func (locator *PagedIndexLocator) ContainsUnique(key []byte) (bool, error)

ContainsUnique proves that a unique index key resolves to an existing table row without loading that row's value or any overflow BLOB pages.

func (*PagedIndexLocator) LookupUniqueBytesColumn added in v0.33.0 

func (locator *PagedIndexLocator) LookupUniqueBytesColumn(key []byte, column int, visit func([]byte) error) (bool, error)

LookupUniqueBytesColumn performs an exact unique seek and decodes one BLOB column without converting it through any. It is the low-allocation route for immutable artifact payloads.

func (*PagedIndexLocator) LookupUniqueColumn added in v0.28.0 

func (locator *PagedIndexLocator) LookupUniqueColumn(key []byte, column int, visit func(any) error) (bool, error)

LookupUniqueColumn performs an exact seek through a unique index and invokes visit with one decoded column from the matching table row.

type PagedRowCursor added in v0.46.0 

type PagedRowCursor struct {
	// contains filtered or unexported fields
}

PagedRowCursor advances through one immutable PagedIndex table or index interval in bounded decoded batches. It is intentionally single-consumer: ResultStream owns one producer goroutine, and retaining a mutex here would put it on the per-batch hot path for no useful concurrency.

Each NextBatch call holds the pager read lock only while it decodes that batch. Callers must process/send the returned rows after NextBatch returns, so slow downstream consumers cannot retain a pager lock.

func (*PagedRowCursor) Done added in v0.46.0 

func (c *PagedRowCursor) Done() bool

Done reports whether the cursor has reached EOF. It is valid after a successful NextBatch call.

func (*PagedRowCursor) NextBatch added in v0.46.0 

func (c *PagedRowCursor) NextBatch(limit int) ([][]any, error)

NextBatch returns up to limit decoded rows and advances the cursor. A nil error with an empty result means EOF. limit must be positive.

type Permission added in v0.12.0 

type Permission string

Permission enumerates the operations a grant can authorize. 

const (
	PermSelect Permission = "SELECT"
	PermInsert Permission = "INSERT"
	PermUpdate Permission = "UPDATE"
	PermDelete Permission = "DELETE"
	// PermDDL covers CREATE/DROP/ALTER TABLE/INDEX/VIEW/TRIGGER/JOB and
	// similar schema-changing statements.
	PermDDL Permission = "DDL"
	// PermAll grants every permission above at once.
	PermAll Permission = "ALL"
)

func ParsePermission added in v0.12.0 

func ParsePermission(s string) (Permission, error)

ParsePermission validates and normalizes a permission keyword from SQL (case-insensitive).

type PooledEntry added in v0.23.1 

type PooledEntry struct {
	Tenant string
	Name   string
	Table  *Table
}

PooledEntry is one resident cache entry returned by Snapshot.

type PooledTableRef added in v0.23.1 

type PooledTableRef struct {
	Tenant string
	Table  *Table
}

PooledTableRef identifies one table object currently resident in a storage backend's own bounded cache — outside DB.tenants — for an evictable storage mode (ModeIndex, ModeHybrid, ModePagedIndex; see DB.backendTablesEvictable). DB.Get returns exactly this *Table pointer as a query-scoped lease without registering it in DB.tenants, so a mutation applied in place (as the SQL engine's INSERT/UPDATE/DELETE do) is only reachable through the backend's own pool. DB.Sync and DB.Close use pooledTableSource.PooledTables to find and flush these leases; without it a table that was never re-registered in DB.tenants could be mutated, "successfully" synced/closed, and then lose the mutation on next open.

type RecoveryStatus added in v0.11.0 

type RecoveryStatus struct {
	Mode                  StorageMode
	Path                  string
	CheckpointLoaded      bool
	RecoveredTransactions uint64
	RecoveredOperations   int
	Truncated             bool
	RecoveredAt           time.Time
}

RecoveryStatus describes the last recovery pass performed while opening a DB.

type ReferentialAction added in v0.12.0 

type ReferentialAction int

ReferentialAction enumerates what happens to a child row when the parent row it references is deleted (ON DELETE) or its referenced column value changes (ON UPDATE). 

const (
	// NoAction means no ON DELETE/ON UPDATE clause was given. tinySQL checks
	// foreign key constraints immediately (it has no deferred-constraint
	// mode), so NoAction and Restrict behave identically here — the SQL
	// standard only distinguishes them for deferred checking.
	NoAction ReferentialAction = iota
	// Restrict blocks the parent-side mutation while a referencing child row exists.
	Restrict
	// Cascade propagates the parent-side delete/update to matching child rows.
	Cascade
	// SetNull nulls the child row's foreign key column instead of deleting/blocking.
	SetNull
)

func (ReferentialAction) String added in v0.12.0 

func (a ReferentialAction) String() string

type RowVersion 

type RowVersion struct {
	// Transaction that created this version
	XMin TxID

	// Transaction that deleted/updated this version (0 if still valid)
	XMax TxID

	// Creation timestamp
	CreatedAt Timestamp

	// Deletion/update timestamp (0 if still valid)
	DeletedAt Timestamp

	// Actual row data
	Data []any

	// Pointer to next version (for version chain)
	NextVersion *RowVersion
}

RowVersion contains MVCC metadata for a single row version.

type SQLiteAffinity added in v0.19.0 

type SQLiteAffinity uint8

SQLiteAffinity is the five-class type system used by SQLite declarations. It is schema metadata, not another runtime value type: tinySQL continues to store NULL, integer, real, text and binary values directly. Keeping the declared affinity separate lets imported SQLite schemas retain their lossless-coercion behaviour without multiplying ColType values. 

const (
	// AffinityDefault retains tinySQL's native, strongly typed coercion rules.
	AffinityDefault SQLiteAffinity = iota
	AffinityInteger
	AffinityText
	AffinityNumeric
	AffinityReal
	AffinityBlob
)

func (SQLiteAffinity) String added in v0.19.0 

func (a SQLiteAffinity) String() string

type SQLiteBackend added in v0.27.0 

type SQLiteBackend struct{}

SQLiteBackend is an opaque placeholder without the sqliteimport build tag: NewSQLiteBackend above always fails, so no method on it is ever reachable.

func NewSQLiteBackend added in v0.27.0 

func NewSQLiteBackend(path string) (*SQLiteBackend, error)

NewSQLiteBackend reports that ModeSQLite requires the sqliteimport build tag, mirroring internal/importer/mbtiles_unsupported.go's fallback for the same underlying dependency (modernc.org/sqlite).

func (*SQLiteBackend) Close added in v0.27.0 

func (b *SQLiteBackend) Close() error

func (*SQLiteBackend) DeleteTable added in v0.27.0 

func (b *SQLiteBackend) DeleteTable(tenant, name string) error

func (*SQLiteBackend) ListTableNames added in v0.27.0 

func (b *SQLiteBackend) ListTableNames(tenant string) ([]string, error)

func (*SQLiteBackend) LoadTable added in v0.27.0 

func (b *SQLiteBackend) LoadTable(tenant, name string) (*Table, error)

func (*SQLiteBackend) Mode added in v0.27.0 

func (b *SQLiteBackend) Mode() StorageMode

func (*SQLiteBackend) SaveTable added in v0.27.0 

func (b *SQLiteBackend) SaveTable(tenant string, t *Table) error

func (*SQLiteBackend) SetReadOnly added in v0.27.0 

func (b *SQLiteBackend) SetReadOnly(ro bool)

func (*SQLiteBackend) Stats added in v0.27.0 

func (b *SQLiteBackend) Stats() BackendStats

func (*SQLiteBackend) Sync added in v0.27.0 

func (b *SQLiteBackend) Sync() error

func (*SQLiteBackend) TableExists added in v0.27.0 

func (b *SQLiteBackend) TableExists(tenant, name string) bool

type Scheduler added in v0.5.1 

type Scheduler struct {
	// contains filtered or unexported fields
}

Scheduler manages scheduled job execution

func NewScheduler added in v0.5.1 

func NewScheduler(db *DB, executor JobExecutor) *Scheduler

NewScheduler creates a new job scheduler

func (*Scheduler) AddJob added in v0.5.1 

func (s *Scheduler) AddJob(job *CatalogJob) error

AddJob registers a new job and schedules it immediately if enabled

func (*Scheduler) RemoveJob added in v0.5.1 

func (s *Scheduler) RemoveJob(name string) error

RemoveJob unregisters a job and stops its execution

func (*Scheduler) SetMaxConcurrentJobs added in v0.39.0 

func (s *Scheduler) SetMaxConcurrentJobs(n int)

SetMaxConcurrentJobs changes how many jobs may execute their SQL at once. Replaces the semaphore outright, so it only affects jobs that acquire a slot after this call -- safe to call before Start, or while running (jobs already holding a slot on the old channel still release it there, which is harmless since nothing reads that channel anymore). n <= 0 is ignored.

func (*Scheduler) Start added in v0.5.1 

func (s *Scheduler) Start() error

Start begins the scheduler loop

func (*Scheduler) Stop added in v0.5.1 

func (s *Scheduler) Stop()

Stop halts the scheduler and cancels all running jobs

func (*Scheduler) UpsertJob added in v0.11.0 

func (s *Scheduler) UpsertJob(job *CatalogJob) error

UpsertJob registers or updates a job and refreshes its live schedule.

type SecondaryIndex added in v0.19.0 

type SecondaryIndex struct {
	Name    string
	Columns []string
	Unique  bool
	Entries []IndexEntry
	// contains filtered or unexported fields
}

SecondaryIndex is a materialized, persistent secondary index. It is kept alongside table rows so GOB snapshots, disk and hybrid backends preserve the index itself, not merely CREATE INDEX catalog metadata.

Entries is the GOB/JSON wire format and nothing else: it is the byte- identical, backward-compatible on-disk shape older saved databases already use. fast is the live, runtime-only backing structure -- a skip list (see skiplist.go) that gives every insert/lookup/delete/range-scan O(log n) expected-case cost instead of Entries' O(n) sorted-slice insert. Being unexported, fast is automatically skipped by gob, so this costs zero wire format changes.

Entries is therefore no longer kept in sync on every mutation -- doing so would defeat the whole point of adding fast. It is only ever refreshed by materialize, called immediately before this index crosses a persistence boundary (GOB/JSON encode, or the paged-index backend's B+Tree writer). Between one materialize and the next, Entries can be arbitrarily stale; nothing except a persistence boundary is allowed to read it directly.

mu guards exactly two operations, hydrate and materialize, which are the only things introduced by fast that mutate a SecondaryIndex from what used to be a read-only path: DB.Get and ordinary query execution take only DB.mu's read lock (see db.go), so multiple SELECTs -- and multiple concurrent checkpoints -- against the same index can run genuinely in parallel. Every other operation (Insert/Remove/Get/Range on an already-hydrated fast) takes no lock at all, matching this package's existing convention that table mutations are already serialized by the caller holding DB's write lock for the whole statement.

func (*SecondaryIndex) Len added in v0.32.0 

func (idx *SecondaryIndex) Len() int

Len reports the number of distinct composite keys this index currently holds, hydrating from Entries first if nothing has touched it yet since it was loaded. It exists for introspection and tests that want the materialized key count without reaching into Entries directly -- which, unlike before, is not kept in sync on every mutation and so cannot be trusted outside a persistence boundary.

type SkipList added in v0.32.0 

type SkipList struct {
	// contains filtered or unexported fields
}

SkipList maps a canonically-encoded []byte key to a sorted []int of row IDs, ordered by bytes.Compare. It is the runtime-only counterpart to the GOB-persisted []IndexEntry slice: same ordering, same key/value shape, but O(log n) expected-case insert/lookup/delete instead of the slice's O(n) insert.

Example (Usage) 
s := NewSkipList()
s.Insert(intKey(3), 30)
s.Insert(intKey(1), 10)
s.Insert(intKey(2), 20)
for _, e := range s.All() {
	fmt.Println(e.RowIDs)
}

Output:
[10]
[20]
[30]

func NewSkipList added in v0.32.0 

func NewSkipList() *SkipList

NewSkipList returns an empty skip list ready for use.

func (*SkipList) All added in v0.32.0 

func (s *SkipList) All() []IndexEntry

All returns every entry in ascending key order, as freshly allocated IndexEntry values -- independent copies, safe to hand to a GOB encoder or otherwise retain past the next skip-list mutation. This is what materializes the persisted Entries []IndexEntry slice from the live skip list at a persistence boundary (checkpoint/SaveTable).

func (*SkipList) Clone added in v0.32.0 

func (s *SkipList) Clone() *SkipList

Clone returns a deep, independent copy of the skip list: same keys, same RowIDs, same tower heights, but entirely separate nodes -- mutating the clone never affects the original, or vice versa.

This is O(n), not O(n log n): rather than re-inserting every key (which would re-run randomLevel and a fresh search per key), it walks the original once in ascending order while keeping a "last node seen at each level" cursor, exactly as Insert's own update[] array would look after seeking to the end of the list. Reusing each source node's existing height (rather than drawing a new random one) means no search is needed at all: the next node at level i is simply linked after whatever node last occupied level i.

func (*SkipList) Get added in v0.32.0 

func (s *SkipList) Get(key []byte) (rowIDs []int, found bool)

Get performs a point lookup, returning the RowIDs slice for key. The returned slice aliases the skip list's internal storage and must be treated as read-only by the caller -- it remains valid only until the next mutation, mirroring the existing aliasing contract documented on SecondaryIndex.lookup/LookupSecondaryIndexPoint in secondary_index.go.

func (*SkipList) Insert added in v0.32.0 

func (s *SkipList) Insert(key []byte, rowID int)

Insert adds rowID under key. If key does not exist yet, a new entry is created holding just rowID. If key already exists, rowID is merged into its existing RowIDs via insertRowIDSorted -- the same sorted-insert-with- dedup behavior insertSecondaryIndexRowID already applies for both unique and non-unique indexes (a unique index's uniqueness is enforced earlier, by CheckSecondaryIndexConstraints; by the time Insert is called the value being appended is already known-valid, exactly as today).

func (*SkipList) Len added in v0.32.0 

func (s *SkipList) Len() int

Len returns the number of distinct keys currently stored.

func (*SkipList) Range added in v0.32.0 

func (s *SkipList) Range(startKey []byte, fn func(key []byte, rowIDs []int) bool)

Range walks entries in ascending key order starting at the first key >= startKey (an empty/nil startKey starts at the very first entry), calling fn for each one. fn's key and rowIDs alias internal storage exactly like Get's return value -- read-only, valid only until the next mutation. Range stops as soon as fn returns false, letting a caller implement equality (stop after the first non-matching key), prefix (stop once bytes.HasPrefix fails) and bounded-range (stop once past the upper bound) seeks the same way secondary_index_range.go's slice-based walk already does, without Range needing to know which shape is in play.

func (*SkipList) Remove added in v0.32.0 

func (s *SkipList) Remove(key []byte, rowID int) (removed bool)

Remove removes rowID from key's RowIDs. If RowIDs becomes empty the whole key entry is removed. Removing a rowID from a key that does not exist, or a rowID that is not present under an existing key, is a safe no-op that reports removed=false -- matching removeSecondaryIndexRowID's existing exact semantics.

type StatementSnapshot added in v0.21.0 

type StatementSnapshot struct {
	// contains filtered or unexported fields
}

StatementSnapshot is an in-memory rollback point for one SQL statement. It is deliberately internal storage machinery: the engine holds DB's content lock for the whole statement, so a snapshot does not need to solve concurrent writers or provide an application-level transaction API.

Table pointers that existed when the snapshot was taken are retained and restored in place. This keeps a caller holding a *Table obtained through DB.Get from observing a failed statement's half-applied row changes.

func (*StatementSnapshot) RestoredTables added in v0.33.0 

func (s *StatementSnapshot) RestoredTables() ([]TableRef, bool)

RestoredTables reports the tables RestoreStatementSnapshot can put back. ok is false for a whole-database snapshot, where the caller must assume any table may have changed.

It exists so a failed statement's cache cleanup is scoped to what that statement could actually have touched. The executor used to walk every tenant and every table on any failure and drop each one's constraint, vector and geo caches — so one rejected INSERT threw away warm caches belonging to tables it never looked at, in the process, for every other caller. A snapshot narrow enough to restore one table is by construction narrow enough to have changed only that one: the compact and table-scoped shapes are only chosen when no trigger and no foreign-key action can write elsewhere.

type StorageBackend added in v0.5.4 

type StorageBackend interface {
	// LoadTable retrieves a table from the backing store. It returns nil, nil
	// when the table does not exist (not an error – the table may simply not
	// have been created yet).
	LoadTable(tenant, name string) (*Table, error)

	// SaveTable persists a single table to the backing store.
	SaveTable(tenant string, t *Table) error

	// DeleteTable removes a table from the backing store.
	DeleteTable(tenant, name string) error

	// ListTableNames returns all table names for a tenant that exist in the
	// backing store (including tables not currently loaded in memory).
	ListTableNames(tenant string) ([]string, error)

	// TableExists reports whether the named table exists in the backing store
	// without loading it into memory.
	TableExists(tenant, name string) bool

	// Sync flushes any pending / dirty data to durable storage.
	Sync() error

	// Close releases all resources. Implementations should call Sync
	// internally if there is unsaved data.
	Close() error

	// Mode returns the StorageMode this backend implements.
	Mode() StorageMode

	// Stats returns operational statistics about the backend.
	Stats() BackendStats
}

StorageBackend abstracts the underlying table storage mechanism.

Implementations are responsible for loading tables into *Table structs that the engine can mutate directly. After mutations, Sync writes dirty tables back to durable storage. Close persists pending data and releases resources.

type StorageConfig added in v0.5.4 

type StorageConfig struct {
	// Mode selects the storage strategy. Defaults to ModeMemory.
	Mode StorageMode

	// Path is the root directory (or file path) for persistent storage.
	// Required for all modes except ModeMemory.
	Path string

	// MaxMemoryBytes limits in-memory table data. Used by ModeHybrid and
	// ModeIndex. Zero means use a sensible default (256 MB).
	MaxMemoryBytes int64

	// SyncOnMutate forces a disk write after every INSERT / UPDATE / DELETE.
	// Slower but provides immediate durability for ModeDisk / ModeHybrid.
	SyncOnMutate bool

	// CompressFiles enables gzip compression for table files on disk
	// (ModeDisk, ModeJSON, ModeHybrid, ModeIndex). For ModeAdvancedWAL it
	// instead compresses the periodic checkpoint snapshot only — the live
	// WAL log itself is never compressed, since it is a continuously
	// appended, crash-recoverable stream rather than a point-in-time
	// artifact (see OpenDB's ModeAdvancedWAL case).
	CompressFiles bool

	// CheckpointEvery controls how many WAL work units trigger an automatic
	// checkpoint: committed transactions in ModeWAL and row-operation records
	// in ModeAdvancedWAL. Zero means the mode default (1000).
	CheckpointEvery uint64

	// CheckpointInterval controls the maximum time between checkpoints. Zero
	// means the mode default (30 s for ModeWAL, 5 min for ModeAdvancedWAL).
	CheckpointInterval time.Duration

	// CheckpointMaxBytes forces a checkpoint once the WAL file exceeds this
	// size, bounding WAL growth independently of transaction count and time
	// (ModeWAL / ModeAdvancedWAL). Zero means default (64 MB); negative
	// disables the size trigger.
	CheckpointMaxBytes int64

	// WALSync controls the durability barrier for ModeWAL and ModeAdvancedWAL
	// commits. The default WALSyncFull retains the historic strongest-flush
	// behavior. WALSyncNormal uses a regular fsync, matching SQLite
	// synchronous=FULL on macOS unless SQLite's separate fullfsync pragma is
	// enabled.
	WALSync WALSyncMode

	// ReadOnly opens the database in read-only mode: the SQL engine rejects
	// all mutating statements. Ideal for serve-only phases (e.g. load at
	// night, query during the day) — vector index and column caches can never
	// be invalidated and the WAL is never appended to.
	ReadOnly bool

	// EncryptionKey, if set, must be exactly EncryptionKeySize (32) bytes
	// and enables AES-256-GCM encryption at rest for table files (ModeDisk,
	// ModeJSON, ModeHybrid, ModeIndex). Derive it from a passphrase with
	// DeriveKeyFromPassphrase (persist the salt yourself — this package
	// doesn't manage key storage) or supply 32 cryptographically random
	// bytes directly for a key-file-based setup. Leave nil for the default,
	// unencrypted behavior. Not yet supported for ModeWAL/ModeAdvancedWAL —
	// see README's Limitations section.
	EncryptionKey []byte
}

StorageConfig configures database storage behaviour.

func DefaultStorageConfig added in v0.5.4 

func DefaultStorageConfig(mode StorageMode) StorageConfig

DefaultStorageConfig returns a StorageConfig with sensible defaults for the given mode. Path must be supplied by the caller afterwards.

type StorageMode added in v0.5.4 

type StorageMode int

StorageMode defines how the database manages data between memory and disk. 

const (
	// ModeMemory keeps all data in RAM. Persistence only occurs via explicit
	// SaveToFile calls or when DB.Close is invoked. Fastest mode.
	ModeMemory StorageMode = iota

	// ModeWAL keeps all data in RAM and writes a Write-Ahead Log for crash
	// recovery. Periodic checkpoints create full GOB snapshots. Good
	// balance of speed and durability.
	ModeWAL

	// ModeDisk stores each table as a separate GOB file on disk. Tables
	// are loaded into memory on demand and flushed back on Sync/Close.
	// Minimises RAM usage at the cost of disk I/O.
	ModeDisk

	// ModeIndex keeps table schemas (columns, row-counts) permanently in
	// RAM while row data resides on disk. Rows are loaded on demand and
	// evicted aggressively. Memory usage is proportional to schema size,
	// not data size.
	ModeIndex

	// ModeHybrid uses an LRU buffer pool with a configurable memory limit.
	// Hot tables stay in RAM; cold tables spill to disk. Best for mixed
	// workloads where the working set fits in a bounded amount of memory.
	ModeHybrid

	// ModeAdvancedWAL keeps all data in RAM and uses a row-level Write-Ahead Log
	// for full ACID transaction durability, crash recovery, and point-in-time recovery.
	ModeAdvancedWAL

	// ModeJSON stores each table as a separate human-readable JSON file on
	// disk (same lazy-load/dirty-tracking behaviour as ModeDisk, which uses
	// GOB instead). Larger on disk than GOB and big.Rat/uuid.UUID values
	// round-trip as plain strings, but files can be read, diffed, or
	// hand-edited with any text tool.
	ModeJSON

	// ModePagedIndex is an immutable-page-oriented artifact for read-mostly
	// lookups. It stores rows and materialized secondary indexes in separate
	// B+Trees, so an exact index seek can load only index and referenced row
	// pages instead of decoding a complete table file.
	ModePagedIndex

	// ModeSQLite stores every table as a native table in a real SQLite
	// database file (via the pure-Go modernc.org/sqlite driver), so the
	// resulting file is directly readable by any SQLite tool — sqlite3,
	// DB Browser for SQLite, etc. Requires the sqliteimport build tag; see
	// backend_sqlite_unsupported.go for the error returned without it.
	ModeSQLite
)

func ParseStorageMode added in v0.5.4 

func ParseStorageMode(s string) (StorageMode, error)

ParseStorageMode converts a string representation back to a StorageMode. It is case-insensitive and returns an error for unknown values.

func (StorageMode) String added in v0.5.4 

func (m StorageMode) String() string

String returns a human-readable label for the StorageMode.

type Table 

type Table struct {
	Name string
	Cols []Column
	Rows [][]any
	// Indexes contains materialized secondary and composite indexes keyed by
	// lower-case SQL index name. Unlike catalog metadata these entries are
	// used by the executor and persisted with table snapshots.
	Indexes map[string]*SecondaryIndex
	// FTSIndexes contains persisted, executor-built full-text indexes keyed by
	// the searched column positions (for example "1,2"). They are derived
	// from Rows, but unlike the process-local hot cache they survive database
	// reopen. Version and StructVersion let the executor either reuse an index,
	// extend it for append-only INSERTs, or rebuild it after UPDATE/DELETE/DDL.
	FTSIndexes map[string]*FTSIndex
	// VectorIndexes contains persisted ANN structures keyed by the executor's
	// stable index key (kind, column position and metric).  Like FTSIndexes,
	// these are derived from Rows but deliberately live with the table so a
	// reopened database can use a validated graph rather than rebuilding it on
	// its first vector query.
	VectorIndexes map[string]*VectorIndex

	IsTemp bool

	Version int
	// Stats is populated by ANALYZE and persisted with the table. DML marks it
	// stale rather than trying to estimate distinct values incrementally.
	Stats *TableStats
	// contains filtered or unexported fields
}

Table stores rows along with column metadata and indexes.

func ApplyWALRecord added in v0.30.0 

func ApplyWALRecord(db *DB, record *WALRecord) (*Table, error)

ApplyWALRecord applies one already-committed row operation under DB's content write lock. It remains as the compatibility API for embedders; feed consumers should prefer ApplyReplicaWALRecords so one response becomes one reader-visible atomic batch and duplicate delivery is harmless.

func NewTable 

func NewTable(name string, cols []Column, isTemp bool) *Table

NewTable creates a new Table with case-insensitive column lookup indices.

func (*Table) AddColumn added in v0.33.0 

func (t *Table) AddColumn(col Column) error

AddColumn appends a column to the table's schema and backfills every existing row with a NULL for it.

Cols and the lower-cased name index ColIndex answers from are two representations of the same thing, and only this package can keep them in step — colPos is unexported and is built once, by NewTable. A caller that appends to Cols directly (which is what ALTER TABLE ADD COLUMN did) ends up with a column that is present in the schema and in every row but that no lookup can resolve.

The duplicate check is case-insensitive, matching ColIndex: two columns whose names differ only in case would be indistinguishable to every lookup, and the second would shadow the first in colPos.

func (*Table) Analyze added in v0.21.0 

func (t *Table) Analyze() *TableStats

Analyze computes exact cardinality, null and simple range summaries for the current table contents. The first statistics implementation scans all rows deliberately: transparent and correct inputs are more useful than a sampled model whose accuracy would need separate policy and tuning.

func (*Table) AnalyzeSummary added in v0.43.0 

func (t *Table) AnalyzeSummary() (rowCount, columnCount int, analyzedAt time.Time)

AnalyzeSummary refreshes the persisted statistics and returns just the values needed by callers that report ANALYZE progress. It avoids cloning every per-column statistic when the detailed result will not be inspected. The caller must already provide the same table-content synchronization that is required for Analyze.

func (*Table) CheckSecondaryIndexConstraints added in v0.19.0 

func (t *Table) CheckSecondaryIndexConstraints(row []any, skipRow int) error

CheckSecondaryIndexConstraints rejects a duplicate before a new row is appended. skipRow is used by UPDATE to ignore a row's current key.

func (*Table) ClearSecondaryIndexes added in v0.21.0 

func (t *Table) ClearSecondaryIndexes()

ClearSecondaryIndexes removes all RowIDs while retaining CREATE INDEX metadata, as required after DELETE without a WHERE clause.

func (*Table) ColIndex 

func (t *Table) ColIndex(name string) (int, error)

ColIndex returns the zero-based index of the named column.

func (*Table) CreateSecondaryIndex added in v0.19.0 

func (t *Table) CreateSecondaryIndex(name string, columns []string, unique bool) error

CreateSecondaryIndex builds an index over both existing and future table rows. Names are case-insensitive, matching SQL identifiers.

func (*Table) DeleteSecondaryIndexRow added in v0.25.0 

func (t *Table) DeleteSecondaryIndexRow(rowID int)

DeleteSecondaryIndexRow removes one stable row position and shifts every later RowID down by one. Point DELETE uses this to avoid allocating an old-to-new map proportional to the whole table.

Like ReindexSecondaryIndexRows, every key must be visited, so this rebuilds fast wholesale via one ascending walk rather than trying to patch it in place.

func (*Table) Derived added in v0.33.0 

func (t *Table) Derived() any

Derived returns the executor state stored on this table, or nil. The caller must hold DerivedLock.

func (*Table) DerivedLock added in v0.33.0 

func (t *Table) DerivedLock()

DerivedLock locks this table's derived-state slot. Derived and SetDerived may only be called while it is held; pair it with DerivedUnlock.

The lock is per-table by design: two statements writing different tables never meet on it, which is the entire reason the state moved here. Two touching the *same* table are already serialized by DB's content lock — this exists for the paths that hold only its read side, where two concurrent SELECTs can both find a cold cache and try to build it.

func (*Table) DerivedUnlock added in v0.33.0 

func (t *Table) DerivedUnlock()

DerivedUnlock releases the lock taken by DerivedLock.

func (*Table) DirtyFrom added in v0.5.4 

func (t *Table) DirtyFrom() int

DirtyFrom returns the first dirty row index, or -1 if non-append-only.

func (*Table) DirtyRows added in v0.24.0 

func (t *Table) DirtyRows() ([]int, bool)

DirtyRows returns the in-place updated row indices and whether that list is known to be complete. A false second result means the caller must treat the whole table as changed.

func (*Table) DropSecondaryIndex added in v0.19.0 

func (t *Table) DropSecondaryIndex(name string) bool

DropSecondaryIndex removes the materialized structure from the table.

func (*Table) FTSIndexesPersistenceState added in v0.45.0 

func (t *Table) FTSIndexesPersistenceState() (generation int, dirty bool)

FTSIndexesPersistenceState atomically returns the current FTS generation and whether it has been flushed to the storage backend.

func (*Table) FindSecondaryIndex added in v0.19.0 

func (t *Table) FindSecondaryIndex(columns []string) *SecondaryIndex

FindSecondaryIndex returns an index whose leading columns exactly match the requested equality predicates. The caller may provide a prefix shorter than the full composite index, enabling prefix seeks.

func (*Table) InsertSecondaryIndexRow added in v0.21.0 

func (t *Table) InsertSecondaryIndexRow(rowID int, row []any, names []string) error

InsertSecondaryIndexRow adds one appended table row to every materialized secondary index. It avoids rebuilding unaffected keys after every INSERT. Call it only after the row has been appended to t.Rows and constraints have been checked.

names is the table's index names in sorted order (see SortedIndexNames). The caller computes it once per statement, since the index set never changes mid-statement, instead of paying a rebuild-from-map-plus-sort on every single row of a multi-row INSERT.

func (*Table) InvalidateStats added in v0.21.0 

func (t *Table) InvalidateStats()

InvalidateStats marks the previous ANALYZE result stale after a mutation. RowCount remains useful for observability while distinct/range values are excluded from planner decisions until ANALYZE is run again.

func (*Table) LookupSecondaryIndexPoint added in v0.19.0 

func (t *Table) LookupSecondaryIndexPoint(idx *SecondaryIndex, values []any) ([]int, error)

LookupSecondaryIndexPoint returns the immutable RowID run for a complete composite key. Unlike a prefix seek it neither merges nor sorts entries: RebuildSecondaryIndexes appends RowIDs in table order, so the entry is already in the observable order of a table scan. The returned slice aliases the index and is read-only; it remains valid until the table is mutated.

This is intentionally a separate API from LookupSecondaryIndexPrefix. A general caller may need an owned prefix result, while the engine's locked read path can safely avoid an allocation on every point lookup.

func (*Table) LookupSecondaryIndexPrefix added in v0.19.0 

func (t *Table) LookupSecondaryIndexPrefix(idx *SecondaryIndex, values []any) ([]int, error)

LookupSecondaryIndexPrefix performs a skip-list seek to the first key >= the prefix, followed by a compact prefix walk. Returned row IDs are sorted in table order to preserve the observable order of a scan when a query has no ORDER BY clause.

func (*Table) LookupSecondaryIndexRange added in v0.26.0 

func (t *Table) LookupSecondaryIndexRange(idx *SecondaryIndex, prefix []any, lo, hi IndexRangeBound) ([]int, error)

LookupSecondaryIndexRange returns the row IDs whose index key matches every value in prefix and whose next component falls within lo..hi.

prefix supplies equality-matched leading columns and may be empty. The range applies to the component at position len(prefix), which must be a numeric column; trailing index columns are ignored, so the result is a superset the caller still filters. Row IDs come back in table order, matching the observable order of a scan with no ORDER BY.

func (*Table) MarkDirtyFrom added in v0.5.4 

func (t *Table) MarkDirtyFrom(idx int)

MarkDirtyFrom records the first row index that was modified. If an earlier index is already set, it is kept. Use -1 for non-append mutations (UPDATE, DELETE) to force a full-table WAL entry.

The -1 (full-table) sentinel is sticky: once a mutation within a transaction forces a full-table entry, a later append-only INSERT must not downgrade it to a delta, or the earlier UPDATE/DELETE would be lost on WAL recovery. Any call also gives up on the in-place row list: an append shifts nothing but adds rows the list cannot describe, and an explicit -1 from a caller that is not reporting individual rows means the shape of the change is unknown.

func (*Table) MarkFTSIndexesChanged added in v0.45.0 

func (t *Table) MarkFTSIndexesChanged()

MarkFTSIndexesChanged records a persist-relevant FTS metadata change. The caller must hold DerivedLock, matching direct access to FTSIndexes.

func (*Table) MarkFTSIndexesPersisted added in v0.45.0 

func (t *Table) MarkFTSIndexesPersisted(generation int)

MarkFTSIndexesPersisted marks generation durable unless a newer FTS build completed while the backend was saving the older snapshot.

func (*Table) MarkRowUpdated added in v0.24.0 

func (t *Table) MarkRowUpdated(idx int)

MarkRowUpdated records that row idx was replaced in place, so the WAL can log that row instead of the entire table. UPDATE is the only shape this fits: it changes row contents without changing how many rows there are or where they sit.

Callers must report every row they change. Reporting none is safe — the WAL writes the whole table, as it always did. Reporting some but not all is not, so a mutation path that cannot enumerate its rows must call MarkDirtyFrom(-1) instead, which permanently gives up the list for this dirty window.

The list is bounded in two ways, because ResetDirty is only reached at a WAL checkpoint: a database with no WAL attached never resets it at all, and one with a WAL still accumulates between checkpoints. Every rollback snapshot carries this list, so an unbounded one makes repeated UPDATEs quadratic.

func (*Table) MarkVectorIndexesChanged added in v0.46.0 

func (t *Table) MarkVectorIndexesChanged()

MarkVectorIndexesChanged records a persist-relevant ANN metadata change. The caller must hold DerivedLock, matching direct access to VectorIndexes.

func (*Table) MarkVectorIndexesPersisted added in v0.46.0 

func (t *Table) MarkVectorIndexesPersisted(generation int)

MarkVectorIndexesPersisted marks ANN metadata durable unless a newer build completed while the backend was saving an older snapshot.

func (*Table) RebuildSecondaryIndexes added in v0.19.0 

func (t *Table) RebuildSecondaryIndexes() error

RebuildSecondaryIndexes rebuilds every materialized index from table rows. It is called after DML, during recovery and before persistence boundaries so index/table versions cannot diverge across snapshots or WAL replay.

The rebuild targets fast directly via bulk insert rather than going through the old build-a-map-then-sort-into-Entries path: it produces an equivalent end state (same keys, same RowIDs, same uniqueness enforcement) while leaving Entries to be derived from fast by materialize only when this index is actually persisted.

func (*Table) ReindexSecondaryIndexRows added in v0.21.0 

func (t *Table) ReindexSecondaryIndexRows(oldToNew map[int]int)

ReindexSecondaryIndexRows applies the old-to-new row-position mapping made by DELETE. Keys stay the same and are not recomputed; only RowIDs belonging to removed rows disappear, and surviving RowIDs are renumbered. This is deliberately named "reindex" rather than "rebuild": it preserves the materialized key structures instead of recomputing keys from row data.

Every key is visited regardless of backing structure (there is no reverse rowID-to-key index to consult instead), so this rebuilds fast wholesale via one ascending walk of the old list -- the same O(total entries + total RowIDs) cost ReindexSecondaryIndexRows already had before fast existed.

func (*Table) RenameColumn added in v0.43.0 

func (t *Table) RenameColumn(from, to string) error

RenameColumn changes one column name while preserving its physical position and every stored value. Secondary-index keys are positional, so their live entries remain valid; their column metadata and the lookup index are updated to make future writes and query planning use the new name.

func (*Table) ResetDirty added in v0.5.4 

func (t *Table) ResetDirty()

ResetDirty marks the table as clean (called after WAL checkpoint).

func (*Table) SetDerived added in v0.33.0 

func (t *Table) SetDerived(v any)

SetDerived stores executor state on this table; nil discards it. The caller must hold DerivedLock.

func (*Table) SortedIndexNames added in v0.30.0 

func (t *Table) SortedIndexNames() []string

SortedIndexNames returns this table's secondary index names (the same lower-cased keys used by t.Indexes) in sorted order, for callers that just want the list (e.g. tests). engine's INSERT/UPDATE hot paths deliberately do not call this: sort.Strings inside the function body keeps the compiler from inlining it, which forces the returned slice to escape to heap on every call. They instead inline the equivalent of this function (see rawIndexNames in internal/engine) and call sort.Strings themselves at the call site, which the compiler can prove non-escaping and keep on the stack -- one heap allocation saved per statement instead of per row.

func (*Table) Statistics added in v0.21.0 

func (t *Table) Statistics() *TableStats

Statistics returns a defensive copy of the latest ANALYZE result.

func (*Table) StructVersion added in v0.31.0 

func (t *Table) StructVersion() int

StructVersion returns the current structural-change counter. See noteStructuralChange's doc comment. Equality across two observations proves no row indexed at the earlier observation has since changed content or been removed — the table may only have grown by appending.

func (*Table) SwapRemoveSecondaryIndexRow added in v0.31.0 

func (t *Table) SwapRemoveSecondaryIndexRow(deleteRowID int, deletedRow []any, lastRowID int, lastRow []any, names []string) error

SwapRemoveSecondaryIndexRow updates every materialized secondary index after a point DELETE removes deleteRowID via swap-and-pop: the caller moves its current last row (lastRowID, with values lastRow) into deleteRowID's slot and truncates, instead of shifting every subsequent row down by one. Only two row positions are affected, so this patches exactly those entries instead of rescanning the whole index the way ReindexSecondaryIndexRows (bulk shift) or DeleteSecondaryIndexRow (single shift-compact) do.

deletedRow is the row being removed (its values before the swap). lastRow is the row that swap-and-pop relocates from lastRowID to deleteRowID; when deleteRowID == lastRowID the delete removed the table's last row and nothing moves. names is the table's index names in sorted order (see SortedIndexNames), computed once per statement by the caller.

func (*Table) UpdateSecondaryIndexRow added in v0.21.0 

func (t *Table) UpdateSecondaryIndexRow(rowID int, before, after []any, names []string) error

UpdateSecondaryIndexRow moves one stable row position between composite keys. Row positions do not change during UPDATE, so this is O(indexes · log(keys)) instead of rescanning the table.

names is the table's index names in sorted order (see SortedIndexNames), computed once per statement by the caller. Without it this rebuilt and sorted the same name list from t.Indexes twice per row -- once for the before key, once for the after key -- even though the index set is identical for both and does not change mid-statement.

func (*Table) UpdatedRowsSince added in v0.46.0 

func (t *Table) UpdatedRowsSince(version int) (rows []int, ok bool)

UpdatedRowsSince returns the rows changed exclusively by UPDATEs since a prior structural version. ok is false when a DELETE, reorder, schema change, or an evicted delta history occurred, in which case index users must rebuild.

func (*Table) VectorIndexesPersistenceState added in v0.46.0 

func (t *Table) VectorIndexesPersistenceState() (generation int, dirty bool)

VectorIndexesPersistenceState atomically returns the current ANN generation and whether it has crossed a persistence boundary.

type TableMeta added in v0.5.4 

type TableMeta struct {
	Tenant   string   `json:"tenant"`
	Name     string   `json:"name"`
	Cols     []Column `json:"cols"`
	RowCount int      `json:"row_count"`
	Version  int      `json:"version"`
	// DiskSize is the file size in bytes on the backing store.
	DiskSize int64 `json:"disk_size"`
	// FilePath is the relative path inside the database directory.
	FilePath string `json:"file_path"`
}

TableMeta stores lightweight metadata for a table that is (potentially) on disk. The disk backend uses this to answer ListTableNames, TableExists, and schema-level queries without loading row data.

type TableRef added in v0.33.0 

type TableRef struct {
	Tenant string
	Table  string
}

TableRef names one table for a caller that has to act on it by name, such as the executor purging its name-keyed caches after a rollback.

type TableStats added in v0.21.0 

type TableStats struct {
	RowCount   int
	Columns    map[string]ColumnStats // lower-cased column name → statistics
	AnalyzedAt time.Time
	Stale      bool
}

TableStats is the persisted result of ANALYZE for one table.

type Timestamp 

type Timestamp uint64

Timestamp represents a logical timestamp for MVCC visibility.

type TriggerEvent added in v0.9.0 

type TriggerEvent string

TriggerEvent indicates what event fires the trigger. 

const (
	// TriggerInsert fires on INSERT.
	TriggerInsert TriggerEvent = "INSERT"
	// TriggerUpdate fires on UPDATE.
	TriggerUpdate TriggerEvent = "UPDATE"
	// TriggerDelete fires on DELETE.
	TriggerDelete TriggerEvent = "DELETE"
)

type TriggerTiming added in v0.9.0 

type TriggerTiming string

TriggerTiming indicates when a trigger fires. 

const (
	// TriggerBefore fires before the DML operation.
	TriggerBefore TriggerTiming = "BEFORE"
	// TriggerAfter fires after the DML operation.
	TriggerAfter TriggerTiming = "AFTER"
	// TriggerInsteadOf replaces the DML operation (views).
	TriggerInsteadOf TriggerTiming = "INSTEAD OF"
)

type TxContext 

type TxContext struct {
	ID             TxID
	StartTime      Timestamp
	Status         TxStatus
	ReadSnapshot   Timestamp                      // Snapshot timestamp for reads
	WriteSet       map[string]map[int64]bool      // table -> row IDs modified
	ReadSet        map[string]map[int64]Timestamp // table -> row IDs read with version
	IsolationLevel IsolationLevel
	// contains filtered or unexported fields
}

TxContext holds the state of an active transaction.

func (*TxContext) RecordRead 

func (tx *TxContext) RecordRead(table string, rowID int64, version Timestamp)

RecordRead tracks a read operation for conflict detection.

func (*TxContext) RecordWrite 

func (tx *TxContext) RecordWrite(table string, rowID int64)

RecordWrite tracks a write operation.

type TxID 

type TxID uint64

TxID represents a unique transaction identifier.

type TxStatus 

type TxStatus uint8

TxStatus represents the current state of a transaction. 

const (
	TxStatusInProgress TxStatus = iota
	TxStatusCommitted
	TxStatusAborted
)

type VectorIndex added in v0.46.0 

type VectorIndex struct {
	Format        int
	Kind          string
	Column        int
	Metric        string
	Version       int
	StructVersion int
	BuiltRows     int
	Dims          int
	Entry         int
	MaxLevel      int
	Levels        []int
	Neighbors     [][][]int
}

VectorIndex is the backend-neutral persisted representation of an ANN index. The current format stores HNSW topology; the Kind field makes the envelope forward-compatible with additional vector index families without coupling storage to the execution engine's runtime types.

Row positions are physical table row positions. StructVersion and BuiltRows distinguish append-only growth (which can extend a graph) from a mutation of an existing row (which must rebuild it).

type WALChange 

type WALChange struct {
	Tenant string
	Name   string
	Table  *Table
	Drop   bool
}

WALChange describes a persistent change that will be written to the WAL.

func CollectWALChanges 

func CollectWALChanges(prev, next *DB) []WALChange

CollectWALChanges computes the delta between two MVCC snapshots.

type WALConfig 

type WALConfig struct {
	Path               string
	CheckpointEvery    uint64
	CheckpointInterval time.Duration
	// SyncMode controls the commit flush. Its zero value, WALSyncFull,
	// preserves ModeWAL's historic strongest-flush behavior.
	SyncMode WALSyncMode
	// CheckpointMaxBytes forces a checkpoint once the WAL file exceeds this
	// size, bounding WAL growth independently of transaction count and time.
	// Zero means default (64 MB); negative disables the size trigger.
	CheckpointMaxBytes int64
}

WALConfig configures WAL and checkpoint behavior.

type WALManager 

type WALManager struct {
	// contains filtered or unexported fields
}

WALManager encapsulates WAL append, recovery, and checkpoints.

func OpenWAL 

func OpenWAL(db *DB, cfg WALConfig) (*WALManager, error)

ready-to-use manager. It attaches no WAL when Path is empty.

func (*WALManager) Checkpoint 

func (w *WALManager) Checkpoint(db *DB) error

Checkpoint writes a DB snapshot and resets the WAL file.

func (*WALManager) Close 

func (w *WALManager) Close() error

Close flushes, syncs, and closes the WAL resources.

func (*WALManager) LogTransaction 

func (w *WALManager) LogTransaction(changes []WALChange) (bool, error)

LogTransaction appends all changes atomically to the WAL. It returns true when a checkpoint is recommended.

type WALOperationType 

type WALOperationType uint8

WALOperationType defines the type of WAL operation. 

const (
	WALOpBegin WALOperationType = iota + 1
	WALOpInsert
	WALOpUpdate
	WALOpDelete
	WALOpCommit
	WALOpAbort
	WALOpCheckpoint
)

func (WALOperationType) String 

func (t WALOperationType) String() string

type WALRecord 

type WALRecord struct {
	// Log Sequence Number - globally unique, monotonically increasing
	LSN LSN

	// Transaction ID
	TxID TxID

	// Operation type
	OpType WALOperationType

	// Tenant and table
	Tenant string
	Table  string

	// Row ID (for row-level operations)
	RowID int64

	// UNDO image (before state) - for rollback
	BeforeImage []any

	// REDO image (after state) - for recovery
	AfterImage []any

	// Column information (for schema tracking)
	Columns []Column

	// Timestamp
	Timestamp time.Time

	// Checksum for corruption detection
	Checksum uint32
}

WALRecord represents a single log entry with before/after images.

func ReadCommittedSince added in v0.30.0 

func ReadCommittedSince(wal *AdvancedWAL, sinceLSN uint64) ([]WALRecord, uint64, error)

ReadCommittedSince returns every WALRecord belonging to a COMMITTED transaction in wal's on-disk log whose LSN is greater than sinceLSN, in LSN order, plus the resume LSN a caller should pass as sinceLSN on its next call (the highest LSN returned, or sinceLSN unchanged if nothing new was found).

Records belonging to a transaction that aborts, or that never reaches a terminal BEGIN/COMMIT/ABORT state (e.g. a crash mid-transaction, or a transaction still open at the moment this is called), are never returned — operations are buffered per transaction as they're read and only released into the result once that transaction's COMMIT record is seen, mirroring exactly how Recover distinguishes committed from pending/aborted transactions before replaying anything. A replication feed must never apply an uncommitted operation, so this filtering is not optional.

Like Recover, a record at or below wal's checkpoint watermark is skipped: it is already reflected in the checkpoint snapshot loaded when the WAL was opened, and would only still be on disk because of a crash between that snapshot's save and the WAL truncation that normally removes it.

ReadCommittedSince holds wal's lock for the duration of the read, the same as Recover, so it does not observe a WAL file mid-write.

type WALSyncMode added in v0.34.0 

type WALSyncMode uint8

WALSyncMode controls the durability barrier used for each committed WAL transaction. Its zero value deliberately preserves the historical strongest available behavior. 

const (
	// WALSyncFull asks the operating system for its strongest available file
	// flush. On macOS this is F_FULLFSYNC; on other platforms it is File.Sync.
	// It is the default because it preserves the durability behavior WAL modes
	// exposed before sync policy became configurable.
	WALSyncFull WALSyncMode = iota

	// WALSyncNormal uses a regular fsync. On macOS this matches SQLite's
	// synchronous=FULL behavior unless SQLite's separate fullfsync pragma is
	// enabled. A committed transaction is still flushed before it is
	// acknowledged, but sudden-power-loss guarantees depend on the filesystem
	// and hardware write cache.
	WALSyncNormal
)

func ParseWALSyncMode added in v0.34.0 

func ParseWALSyncMode(s string) (WALSyncMode, error)

ParseWALSyncMode parses a WAL sync policy. Keep this separate from storage modes: both affect durability, but one chooses a backend and the other chooses its per-commit flush strength.

func (WALSyncMode) String added in v0.34.0 

func (m WALSyncMode) String() string

String returns the canonical configuration spelling for a WAL sync mode.

type WALTxState 

type WALTxState struct {
	TxID       TxID
	StartLSN   LSN
	Operations []LSN
	Status     TxStatus
}

WALTxState tracks the state of a transaction in the WAL.

Source Files

View all Source files

Directories

Path Synopsis
Package pager — PageBackend integrates the page-based storage engine with the tinySQL StorageBackend interface.
 Package pager — PageBackend integrates the page-based storage engine with the tinySQL StorageBackend interface.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.