capture

package
v0.1.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 5, 2026 License: MIT Imports: 21 Imported by: 0

Documentation

Overview

Package capture copies each change of a database in WAL mode into LTX files in a local staging folder, from which the replicator uploads them, and runs the database's checkpoints, so that no frame is checkpointed before it is copied.

Every Interval the capturer opens a short read transaction and reads the WAL header SQLite sees in it: the generation of the WAL (its salts), its last committed frame (mxFrame) and a count of the commits made (iChange, one more for each commit of any process, kept when the WAL starts over). It reads the frames after the last one it staged up to that frame, checks each as SQLite's recovery does, and stages the pages as one LTX file of the next TXID, synced before it counts. A frame after mxFrame, of a transaction that has not committed, is never read.

The count of commits tells whether the WAL went on in a way the capturer saw: the commit frames it reads must be exactly the commits made since the last read, also across a new generation of the WAL. When they are not, another process wrote and checkpointed, or the WAL went back, and the capturer stages a full copy of the database as the next TXID instead.

Connections of the driver to the database have no automatic checkpoints and cannot run a checkpoint (replhook.Attach). The capturer checkpoints once the WAL holds CheckpointFrames frames: it takes the driver's write lock of the file, reads the WAL to its end and checkpoints, so that no frame of this process is checkpointed before it is staged.

On start, the last staged file says where in the WAL staging stopped and the checksum of the database there. The capturer reads the database, works out its checksum at that point, and goes on from there only if it is the same. Otherwise it stages a full copy.

Index

Constants

This section is empty.

Variables

View Source
var ErrFullCopy = errors.New("capture: staged a full copy of the database")

ErrFullCopy is under the error reported for a full copy of the database staged because the WAL went on in a way the capturer could not follow, or the database is not where the staged files end.

View Source
var ErrStagingFull = errors.New("capture: the staged files not uploaded reached the limit")

ErrStagingFull is under the error reported when staging stops at Options.Limit, and the error of Sync while it is stopped.

Functions

func SnapshotDir

func SnapshotDir(path string) string

SnapshotDir is the folder of the snapshots of the database path.

func Stage

func Stage(path, from string, min, max ltx.TXID) error

Stage moves the LTX file from, which holds TXIDs min to max and leaves the database path as it is now, into the staging folder of path, so that Open goes on from it. The folder must hold no LTX file. The folders it makes are synced, up to the folder of path.

func StageDir

func StageDir(path string) string

StageDir is the folder of the level 0 files of the database path.

Types

type Capturer

type Capturer struct {
	// contains filtered or unexported fields
}

A Capturer stages the changes of one database file.

func Open

func Open(name string, opt Options) (*Capturer, error)

Open starts staging the changes of the database name, before the driver opens it: it fails when the driver of this process has it open. The database is put in WAL mode. Open stages a copy of the whole database first, unless the staged files end where the database is.

func (*Capturer) Close

func (c *Capturer) Close() error

Close stages what is left, checkpoints and stops. When a write transaction keeps the driver's write lock longer than closeWait, the WAL is left as it is, staged, for the next Open. Connections the driver opens to the database after Close are as before replication.

func (*Capturer) Dir

func (c *Capturer) Dir() string

Dir is the staging folder of the level 0 files.

func (*Capturer) Done

func (c *Capturer) Done(txid ltx.TXID) error

Done records that the staged files up to txid are uploaded, or never will be, and removes them but the last staged file, which the capturer goes on from after a restart.

func (*Capturer) Durable

func (c *Capturer) Durable() ltx.TXID

Durable is the last TXID staged whose name is synced: a crash keeps the files up to it.

func (*Capturer) FullCopies

func (c *Capturer) FullCopies() int

FullCopies is how many full copies were staged after the first.

func (*Capturer) Pending

func (c *Capturer) Pending() int64

Pending is the size of the staged files not uploaded yet.

func (*Capturer) Snapshot

func (c *Capturer) Snapshot(ctx context.Context) (snap string, _ ltx.TXID, err error)

Snapshot writes a snapshot of the database, the LTX file of TXIDs 1 to the last staged, into SnapshotDir, and returns its name and that TXID. Under the driver's write lock it stages the WAL to its end and opens a read transaction on a connection of its own. The pages are read after the lock, so that writers do not wait for them, and their checksum must be the one of the staged files. Checkpoints cannot reset the WAL while the pages are read.

func (*Capturer) Staged

func (c *Capturer) Staged() <-chan struct{}

Staged gets a value when Durable moves on.

func (*Capturer) Sync

func (c *Capturer) Sync(ctx context.Context) error

Sync stages what was committed up to now. ctx bounds the wait for the driver's write lock, which a checkpoint takes.

func (*Capturer) TXID

func (c *Capturer) TXID() ltx.TXID

TXID is the TXID of the last staged file.

type Options

type Options struct {
	// Interval is the time between two reads of the WAL. Default 1s.
	Interval time.Duration
	// CheckpointFrames is how many frames the WAL holds before the capturer
	// checkpoints it. Default 1000, as SQLite's automatic checkpoints.
	CheckpointFrames uint32
	// Limit is the most bytes of level 0 files staged and not uploaded yet
	// (see Done): past it, staging stops until they are uploaded, and a full
	// copy of the database follows them. Checkpoints go on, so the WAL
	// stays bounded. The last full copy, which staging goes on from, does
	// not count, and the step that passes the limit stages no more after
	// that one read of the WAL. 0 is no limit.
	Limit int64
	// Uploaded is the last TXID uploaded when Open runs.
	Uploaded ltx.TXID
	// OnError, if set, gets the errors of the reads made in the background,
	// and an error under ErrFullCopy for each full copy staged because the
	// WAL could not be followed.
	OnError func(error)
	// Start, if set, is called once Open has the database, before it
	// resumes, with the staged files in TXID order (their headers and
	// trailers, not checked). An error stops Open. A Restart drops the
	// staged files and starts over from it.
	Start func(staged []StagedFile) (*Restart, error)
}

Options are the options of Open.

type Restart

type Restart struct {
	TXID ltx.TXID
	Post ltx.Checksum
}

A Restart starts the staged files over: a full copy of the database as TXID+1, whose pre-apply checksum is Post (a snapshot for TXID 0).

type StagedFile

type StagedFile struct {
	Min, TXID ltx.TXID     // the first and last TXID it holds: one but after a restore
	Pre, Post ltx.Checksum // the checksums of the database before and after it
}

A StagedFile is a staged LTX file, of one TXID.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL