Documentation
¶
Overview ¶
Package replica copies a database of the driver to S3 (AWS S3, Cloudflare R2, MinIO and the other stores that speak its API) as it changes, so that it can be restored after the machine is lost.
Start it before the program opens the database, then open the database as usual:
r, err := replica.New(ctx, "app.db", replica.Config{
URL: "s3://bucket/app.db?endpoint=https://<account>.r2.cloudflarestorage.com®ion=auto",
})
if err != nil {
return err
}
defer r.Close()
db, err := sql.Open("sqlite", "app.db")
Each commit is copied into a local staging folder beside the database within SyncInterval and uploaded from there. Writes never wait for S3. The replica is laid out as Litestream lays it out, so `litestream restore` reads it too.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrConflict = errors.New("replica: another replicator writes to the replica")
ErrConflict is under the error of an upload the store refused because the replica has another file of that TXID: another replicator writes to the same path. The replicator stops uploading. Staging and checkpoints go on, and the program's writes are not touched.
var ErrDiverged = errors.New("replica: the replica and the database went apart")
ErrDiverged is under the error of New when the replica does not end where the staged files do. Config.FreshStart goes on from the replica.
var ErrNoConditionalWrites = errors.New("replica: the store does not refuse a second write of a key (If-None-Match)")
ErrNoConditionalWrites is the error of New on a store that does not refuse a second write of a key with If-None-Match, which keeps two replicators from writing over each other.
var ErrNoReplica = errors.New("replica: there is no replica at the URL")
ErrNoReplica is the error of Restore when the replica holds no file.
var ErrStagingFull = capture.ErrStagingFull
ErrStagingFull is under the error given to Config.OnError when staging stops at Config.StagingLimit.
Functions ¶
func Restore ¶
Restore rebuilds the database name from the replica of cfg.URL, and returns the TXID it holds. Only URL, Credentials and AllowHTTP of cfg are used. Every file is checked: its checksum, and the checksum of the database before and after it. Restore writes neither over a database nor beside a WAL, journal or staged file of name that is there. It leaves the replica's last file in the staging folder of name, so that New goes on from the replica: after a restore to the latest state, New goes on with no full copy, and after a restore to an earlier one New fails with ErrDiverged unless Config.FreshStart.
Types ¶
type Config ¶
type Config struct {
// URL is s3://bucket/path. Query parameters: endpoint, the scheme and
// host of the store (default https://s3.<region>.amazonaws.com), and
// region (default AWS_REGION, else us-east-1). R2 takes region=auto.
URL string
// Credentials sign the requests. nil reads AWS_ACCESS_KEY_ID,
// AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN.
Credentials CredentialsProvider
// SyncInterval is the most time between a commit and its copy into
// the staging folder. Default 1s.
SyncInterval time.Duration
// AllowHTTP takes an http:// endpoint, such as a local MinIO.
AllowHTTP bool
// SingleWriter says the program makes sure only one replicator writes
// to the replica, so New does not check that the store refuses a second
// write of a key. Without it, a store that does not is refused.
SingleWriter bool
// FreshStart lets New go on when the replica does not end where the
// staged files do (another machine wrote it, or the database was
// replaced): it uploads a full copy of the database as the replica's
// next TXID. Without it New fails with ErrDiverged.
FreshStart bool
// Retention is how long the replica keeps the database's past: a
// restore goes back that far at least. A snapshot of the database is
// uploaded every Retention or every day, the more often. Default 24h.
Retention time.Duration
// StagingLimit is the most bytes of changes staged and not uploaded
// yet, such as while the store is down. Past it, staging stops (an error
// under ErrStagingFull) and keeps what it has: checkpoints go on, so the
// WAL stays bounded, and writes never wait. Once what is staged is
// uploaded, a full copy of the database follows it. The changes in
// between have no point of their own to restore to. The last full
// copy, which staging goes on from, is not counted: the staging folder
// may hold one database more. Default 1 GiB.
StagingLimit int64
// OnError, if set, gets the errors of the work done in the background.
OnError func(error)
// contains filtered or unexported fields
}
Config says where the replica goes and how.
type Credentials ¶
type Credentials = s3.Credentials
Credentials are the keys requests are signed with. Printing them never shows the secret or the session token.
type CredentialsProvider ¶
type CredentialsProvider = s3.CredentialsProvider
A CredentialsProvider gives the credentials of each request, such as the AWS SDK's provider of an instance role, wrapped.
func StaticCredentials ¶
func StaticCredentials(c Credentials) CredentialsProvider
StaticCredentials gives the same credentials to every request.
type Replica ¶
type Replica struct {
// contains filtered or unexported fields
}
A Replica copies one database file to S3.
func New ¶
New starts replicating the database file name, which the driver of this process must not have open yet. It checks that the store refuses a second write of a key (unless Config.SingleWriter), reads where the replica ends, and goes on from there when the staged files do, else fails with ErrDiverged (unless Config.FreshStart). An empty replica starts with a full copy. ctx bounds the requests New makes.
type RestoreOptions ¶
type RestoreOptions struct {
// TXID, if set, is the TXID to restore.
TXID uint64
// At, if set, restores the database as it was before that time: the
// files of the replica made before it, as litestream restore does. A
// file is made when its commits are staged, within SyncInterval of
// them, and may hold several commits.
At time.Time
}
RestoreOptions says which state of the database Restore brings back: the latest, the one at a TXID, or the one at a time.
type Status ¶
type Status struct {
Staged uint64 // the last TXID in the staging folder
Uploaded uint64 // the last TXID in the replica
Pending int64 // the bytes staged and not uploaded yet
// Err is the error of the last upload, nil once one succeeds. It is
// under ErrConflict when uploads stopped for good.
Err error
}
Status is where replication is.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
internal
|
|
|
capture
Package capture copies each change of a database in WAL mode into LTX files in a local staging folder, from which the replicator uploads them, and runs the database's checkpoints, so that no frame is checkpointed before it is copied.
|
Package capture copies each change of a database in WAL mode into LTX files in a local staging folder, from which the replicator uploads them, and runs the database's checkpoints, so that no frame is checkpointed before it is copied. |
|
s3
Package s3 is the small client of the S3 protocol the replicator uses: PUT, multipart upload, ranged GET, ListObjectsV2 and DELETE, signed with SigV4, against AWS S3 and the stores that speak its protocol (Cloudflare R2, MinIO and the like).
|
Package s3 is the small client of the S3 protocol the replicator uses: PUT, multipart upload, ranged GET, ListObjectsV2 and DELETE, signed with SigV4, against AWS S3 and the stores that speak its protocol (Cloudflare R2, MinIO and the like). |
|
s3test
Package s3test gives tests an S3 store:
|
Package s3test gives tests an S3 store: |