Documentation
¶
Overview ¶
Package capture copies each change of a database in WAL mode into LTX files in a local staging folder, from which the replicator uploads them, and runs the database's checkpoints, so that no frame is checkpointed before it is copied.
Every Interval the capturer opens a short read transaction and reads the WAL header SQLite sees in it: the generation of the WAL (its salts), its last committed frame (mxFrame) and a count of the commits made (iChange, one more for each commit of any process, kept when the WAL starts over). It reads the frames after the last one it staged up to that frame, checks each as SQLite's recovery does, and stages the pages as one LTX file of the next TXID, synced before it counts. A frame after mxFrame, of a transaction that has not committed, is never read.
The count of commits tells whether the WAL went on in a way the capturer saw: the commit frames it reads must be exactly the commits made since the last read, also across a new generation of the WAL. When they are not, another process wrote and checkpointed, or the WAL went back, and the capturer stages a full copy of the database as the next TXID instead.
Connections of the driver to the database have no automatic checkpoints and cannot run a checkpoint (replhook.Attach). The capturer checkpoints once the WAL holds CheckpointFrames frames: it takes the driver's write lock of the file, reads the WAL to its end and checkpoints, so that no frame of this process is checkpointed before it is staged.
On start, the last staged file says where in the WAL staging stopped and the checksum of the database there. The capturer reads the database, works out its checksum at that point, and goes on from there only if it is the same. Otherwise it stages a full copy.
Index ¶
- Variables
- func SnapshotDir(path string) string
- func Stage(path, from string, min, max ltx.TXID) error
- func StageDir(path string) string
- type Capturer
- func (c *Capturer) Close() error
- func (c *Capturer) Dir() string
- func (c *Capturer) Done(txid ltx.TXID) error
- func (c *Capturer) Durable() ltx.TXID
- func (c *Capturer) FullCopies() int
- func (c *Capturer) Pending() int64
- func (c *Capturer) Snapshot(ctx context.Context) (snap string, _ ltx.TXID, err error)
- func (c *Capturer) Staged() <-chan struct{}
- func (c *Capturer) Sync(ctx context.Context) error
- func (c *Capturer) TXID() ltx.TXID
- type Options
- type Restart
- type StagedFile
Constants ¶
This section is empty.
Variables ¶
var ErrFullCopy = errors.New("capture: staged a full copy of the database")
ErrFullCopy is under the error reported for a full copy of the database staged because the WAL went on in a way the capturer could not follow, or the database is not where the staged files end.
var ErrStagingFull = errors.New("capture: the staged files not uploaded reached the limit")
ErrStagingFull is under the error reported when staging stops at Options.Limit, and the error of Sync while it is stopped.
Functions ¶
func SnapshotDir ¶
SnapshotDir is the folder of the snapshots of the database path.
Types ¶
type Capturer ¶
type Capturer struct {
// contains filtered or unexported fields
}
A Capturer stages the changes of one database file.
func Open ¶
Open starts staging the changes of the database name, before the driver opens it: it fails when the driver of this process has it open. The database is put in WAL mode. Open stages a copy of the whole database first, unless the staged files end where the database is.
func (*Capturer) Close ¶
Close stages what is left, checkpoints and stops. When a write transaction keeps the driver's write lock longer than closeWait, the WAL is left as it is, staged, for the next Open. Connections the driver opens to the database after Close are as before replication.
func (*Capturer) Done ¶
Done records that the staged files up to txid are uploaded, or never will be, and removes them but the last staged file, which the capturer goes on from after a restart.
func (*Capturer) Durable ¶
Durable is the last TXID staged whose name is synced: a crash keeps the files up to it.
func (*Capturer) FullCopies ¶
FullCopies is how many full copies were staged after the first.
func (*Capturer) Snapshot ¶
Snapshot writes a snapshot of the database, the LTX file of TXIDs 1 to the last staged, into SnapshotDir, and returns its name and that TXID. Under the driver's write lock it stages the WAL to its end and opens a read transaction on a connection of its own. The pages are read after the lock, so that writers do not wait for them, and their checksum must be the one of the staged files. Checkpoints cannot reset the WAL while the pages are read.
func (*Capturer) Staged ¶
func (c *Capturer) Staged() <-chan struct{}
Staged gets a value when Durable moves on.
type Options ¶
type Options struct {
// Interval is the time between two reads of the WAL. Default 1s.
Interval time.Duration
// CheckpointFrames is how many frames the WAL holds before the capturer
// checkpoints it. Default 1000, as SQLite's automatic checkpoints.
CheckpointFrames uint32
// Limit is the most bytes of level 0 files staged and not uploaded yet
// (see Done): past it, staging stops until they are uploaded, and a full
// copy of the database follows them. Checkpoints go on, so the WAL
// stays bounded. The last full copy, which staging goes on from, does
// not count, and the step that passes the limit stages no more after
// that one read of the WAL. 0 is no limit.
Limit int64
// Uploaded is the last TXID uploaded when Open runs.
Uploaded ltx.TXID
// OnError, if set, gets the errors of the reads made in the background,
// and an error under ErrFullCopy for each full copy staged because the
// WAL could not be followed.
OnError func(error)
// Start, if set, is called once Open has the database, before it
// resumes, with the staged files in TXID order (their headers and
// trailers, not checked). An error stops Open. A Restart drops the
// staged files and starts over from it.
Start func(staged []StagedFile) (*Restart, error)
}
Options are the options of Open.