Documentation
¶
Overview ¶
Package capture copies each change of a database in WAL mode into LTX files in a local staging folder, from which the replicator uploads them, and runs the database's checkpoints, so that no frame is checkpointed before it is copied.
Every Interval the capturer opens a short read transaction and reads the WAL header SQLite sees in it: the generation of the WAL (its salts), its last committed frame (mxFrame) and a count of the commits made (iChange, one more for each commit of any process, kept when the WAL starts over). It reads the frames after the last one it staged up to that frame, checks each as SQLite's recovery does, and stages the pages as one LTX file of the next TXID, synced before it counts. A frame after mxFrame, of a transaction that has not committed, is never read.
The count of commits tells whether the WAL went on in a way the capturer saw: the commit frames it reads must be exactly the commits made since the last read, also across a new generation of the WAL. When they are not, another process wrote and checkpointed, or the WAL went back, and the capturer stages a full copy of the database as the next TXID instead.
Connections of the driver to the database have no automatic checkpoints and cannot run a checkpoint (replhook.Attach). The capturer checkpoints once the WAL holds CheckpointFrames frames, holding the driver's write lock of the file: it reads the WAL to its end, stages it and checkpoints, so that no frame of this process is checkpointed before it is staged. A writer of the driver does it as it gives the lock back (committed), as SQLite's automatic checkpoints run after a commit, so the WAL stays short. A step does it too when it finds the WAL that long.
On start, the last staged file says where in the WAL staging stopped and the checksum of the database there. The capturer reads the database, works out its checksum at that point, and goes on from there only if it is the same. Otherwise it stages a full copy.
Index ¶
- Variables
- func CompactDir(path string) string
- func SnapshotDir(path string) string
- func Stage(path, from string, min, max ltx.TXID) error
- func StageDir(path string) string
- func Verify(r io.Reader) (h ltx.Header, t ltx.Trailer, err error)
- type Capturer
- func (c *Capturer) Close() error
- func (c *Capturer) CompactDir() string
- func (c *Capturer) Dir() string
- func (c *Capturer) Done(txid ltx.TXID) error
- func (c *Capturer) Durable() ltx.TXID
- func (c *Capturer) FullCopies() int
- func (c *Capturer) Pending() int64
- func (c *Capturer) Snapshot(ctx context.Context) (snap string, _ ltx.TXID, err error)
- func (c *Capturer) Staged() <-chan struct{}
- func (c *Capturer) Sync(ctx context.Context) error
- func (c *Capturer) TXID() ltx.TXID
- func (c *Capturer) Unlock()
- type Options
- type Restart
- type StagedFile
Constants ¶
This section is empty.
Variables ¶
var ErrFullCopy = errors.New("capture: staged a full copy of the database")
ErrFullCopy is under the error reported for a full copy of the database staged because the WAL went on in a way the capturer could not follow, or the database is not where the staged files end.
var ErrStagingFull = errors.New("capture: the staged files not uploaded reached the limit")
ErrStagingFull is under the error reported when staging stops at Options.Limit, and the error of Sync while it is stopped.
Functions ¶
func CompactDir ¶ added in v0.1.4
CompactDir is the folder the replicator compacts in. Open removes it.
func SnapshotDir ¶
SnapshotDir is the folder of the snapshots of the database path.
func Stage ¶
Stage moves the LTX file from, which holds TXIDs min to max and leaves the database path as it is now, into the staging folder of path, so that Open goes on from it. The folder must hold no LTX file. The folders it makes are synced, up to the folder of path.
Types ¶
type Capturer ¶
type Capturer struct {
// contains filtered or unexported fields
}
A Capturer stages the changes of one database file.
func Open ¶
Open starts staging the changes of the database name, before the driver opens it: it fails when the driver of this process has it open. The database is put in WAL mode. Open stages a copy of the whole database first, unless the staged files end where the database is.
func (*Capturer) Close ¶
Close stages what is left, checkpoints and stops. With staging stopped at the limit, it stages nothing and gives an error under ErrStagingFull. When a write transaction keeps the driver's write lock longer than closeWait, the WAL is left as it is, staged, for the next Open. Connections the driver opens to the database after Close are as before replication.
func (*Capturer) CompactDir ¶ added in v0.1.4
CompactDir is the folder the replicator compacts in, for the database as the capturer found it.
func (*Capturer) Done ¶
Done records that the staged files up to txid are uploaded, or never will be, and removes them but the last staged file, which the capturer goes on from after a restart.
func (*Capturer) Durable ¶
Durable is the last TXID staged whose name is synced: a crash keeps the files up to it.
func (*Capturer) FullCopies ¶
FullCopies is how many full copies were staged after the first.
func (*Capturer) Snapshot ¶
Snapshot writes a snapshot of the database, the LTX file of TXIDs 1 to the last staged, into SnapshotDir, and returns its name and that TXID. Under the driver's write lock it stages the WAL to its end and opens a read transaction on a connection of its own. The pages are read after the lock, so that writers do not wait for them, and their checksum must be the one of the staged files. Checkpoints cannot reset the WAL while the pages are read. ctx bounds the waits, and so does Close.
func (*Capturer) Staged ¶
func (c *Capturer) Staged() <-chan struct{}
Staged gets a value when Durable moves on.
func (*Capturer) Sync ¶
Sync stages what was committed up to now. ctx bounds the waits for the capturer and for the driver's write lock, which a checkpoint takes, and so does Close.
type Options ¶
type Options struct {
// Interval is the time between two reads of the WAL. Default 1s.
Interval time.Duration
// CheckpointFrames is how many frames the WAL holds before the capturer
// checkpoints it. Default 4000 (about 16 MB of 4 KiB pages): at 1000, as
// SQLite's automatic checkpoints, a write costs about a tenth more.
CheckpointFrames uint32
// Limit is the most bytes of level 0 files staged and not uploaded yet
// (see Done): past it, staging stops until they are uploaded, and a full
// copy of the database follows them. Checkpoints go on, so the WAL
// stays bounded. The last full copy, which staging goes on from, does
// not count, and the step that passes the limit stages no more after
// that one read of the WAL. 0 is no limit.
Limit int64
// Uploaded is the last TXID uploaded when Open runs.
Uploaded ltx.TXID
// OnError, if set, gets the errors of the reads made in the background,
// and an error under ErrFullCopy for each full copy staged because the
// WAL could not be followed. Open, Close and the goroutine of the
// background steps call it with no lock of the capturer held. It should
// return soon, as a log does, and must not call Close, which waits for
// that goroutine, nor wait for a write to the database: the caller of
// Close may hold the write lock.
OnError func(error)
// Start, if set, is called once Open has the database and the lock of
// its staging, before it resumes, with the staged files in TXID order
// (their headers and trailers, not checked). An error stops Open. A
// Restart drops the staged files and starts over from it.
Start func(staged []StagedFile) (*Restart, error)
// Hold keeps the staging folder locked after Close, until Unlock, so
// that the caller can upload what Close staged, and call Done, before
// another process stages the database.
Hold bool
}
Options are the options of Open.