litestream

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 31, 2020 License: GPL-3.0 Imports: 25 Imported by: 13

README

litestream

Streaming replication for SQLite.

Questions

  • How to avoid WAL checkpointing on close?

Notes

-- Disable autocheckpointing.
PRAGMA wal_autocheckpoint = 0

Documentation

Index

Constants

View Source 
const (
	DefaultMonitorInterval    = 1 * time.Second
	DefaultCheckpointInterval = 1 * time.Minute
	DefaultMinCheckpointPageN = 1000
)

Default DB settings.

View Source 
const (
	MetaDirSuffix = "-litestream"

	WALDirName  = "wal"
	WALExt      = ".wal"
	SnapshotExt = ".snapshot"

	GenerationNameLen = 16
)
View Source 
const (
	CheckpointModePassive  = "PASSIVE"
	CheckpointModeFull     = "FULL"
	CheckpointModeRestart  = "RESTART"
	CheckpointModeTruncate = "TRUNCATE"
)

SQLite checkpoint modes.

View Source 
const (
	// WALHeaderSize is the size of the WAL header, in bytes.
	WALHeaderSize = 32

	// WALFrameHeaderSize is the size of the WAL frame header, in bytes.
	WALFrameHeaderSize = 24
)
View Source 
const WALFrameHeaderChecksumOffset = 16
View Source 
const WALHeaderChecksumOffset = 24

Variables

View Source 
var (
	ErrNoSnapshots      = errors.New("no snapshots available")
	ErrChecksumMismatch = errors.New("invalid replica, checksum mismatch")
)

Litestream errors.

Functions

func Checksum 

func Checksum(bo binary.ByteOrder, s0, s1 uint32, b []byte) (uint32, uint32)

Checksum computes a running SQLite checksum over a byte slice.

func FormatWALPath added in v0.2.0 

func FormatWALPath(index int) string

FormatWALPath formats a WAL filename with a given index.

func FormatWALPathWithOffset added in v0.2.0 

func FormatWALPathWithOffset(index int, offset int64) string

FormatWALPathWithOffset formats a WAL filename with a given index & offset.

func HexDump 

func HexDump(b []byte) string

HexDump returns hexdump output but with duplicate lines removed.

func IsGenerationName 

func IsGenerationName(s string) bool

IsGenerationName returns true if s is the correct length and is only lowercase hex characters.

func IsSnapshotPath 

func IsSnapshotPath(s string) bool

IsSnapshotPath returns true if s is a path to a snapshot file.

func IsWALPath 

func IsWALPath(s string) bool

IsWALPath returns true if s is a path to a WAL file.

func ParseSnapshotPath 

func ParseSnapshotPath(s string) (index int, typ, ext string, err error)

ParseSnapshotPath returns the index for the snapshot. Returns an error if the path is not a valid snapshot path.

func ParseWALPath 

func ParseWALPath(s string) (index int, offset int64, ext string, err error)

ParseWALPath returns the index & offset for the WAL file. Returns an error if the path is not a valid snapshot path.

Types

type DB 

type DB struct {

	// Minimum threshold of WAL size, in pages, before a passive checkpoint.
	// A passive checkpoint will attempt a checkpoint but fail if there are
	// active transactions occurring at the same time.
	MinCheckpointPageN int

	// Maximum threshold of WAL size, in pages, before a forced checkpoint.
	// A forced checkpoint will block new transactions and wait for existing
	// transactions to finish before issuing a checkpoint and resetting the WAL.
	//
	// If zero, no checkpoints are forced. This can cause the WAL to grow
	// unbounded if there are always read transactions occurring.
	MaxCheckpointPageN int

	// Time between automatic checkpoints in the WAL. This is done to allow
	// more fine-grained WAL files so that restores can be performed with
	// better precision.
	CheckpointInterval time.Duration

	// List of replicas for the database.
	// Must be set before calling Open().
	Replicas []Replica

	// Frequency at which to perform db sync.
	MonitorInterval time.Duration
	// contains filtered or unexported fields
}

DB represents a managed instance of a SQLite database in the file system.

func NewDB 

func NewDB(path string) *DB

NewDB returns a new instance of DB for a given path.

func (*DB) CRC32C added in v0.2.0 

func (db *DB) CRC32C() (uint32, Pos, error)

CRC32C returns a CRC-32C checksum of the database and its current position.

This function obtains a read lock so it prevents syncs from occuring until the operation is complete. The database will still be usable but it will be unable to checkpoint during this time.

func (*DB) Close 

func (db *DB) Close() (err error)

Close releases the read lock & closes the database. This method should only be called by tests as it causes the underlying database to be checkpointed.

func (*DB) CurrentGeneration 

func (db *DB) CurrentGeneration() (string, error)

CurrentGeneration returns the name of the generation saved to the "generation" file in the meta data directory. Returns empty string if none exists.

func (*DB) CurrentShadowWALIndex 

func (db *DB) CurrentShadowWALIndex(generation string) (index int, size int64, err error)

CurrentShadowWALIndex returns the current WAL index & total size.

func (*DB) CurrentShadowWALPath 

func (db *DB) CurrentShadowWALPath(generation string) (string, error)

CurrentShadowWALPath returns the path to the last shadow WAL in a generation.

func (*DB) GenerationNamePath 

func (db *DB) GenerationNamePath() string

GenerationNamePath returns the path of the name of the current generation.

func (*DB) GenerationPath 

func (db *DB) GenerationPath(generation string) string

GenerationPath returns the path of a single generation.

func (*DB) Init 

func (db *DB) Init() (err error)

Init initializes the connection to the database. Skipped if already initialized or if the database file does not exist.

func (*DB) MetaPath 

func (db *DB) MetaPath() string

MetaPath returns the path to the database metadata.

func (*DB) Notify 

func (db *DB) Notify() <-chan struct{}

Notify returns a channel that closes when the shadow WAL changes.

func (*DB) Open 

func (db *DB) Open() (err error)

func (*DB) PageSize 

func (db *DB) PageSize() int

PageSize returns the page size of the underlying database. Only valid after database exists & Init() has successfully run.

func (*DB) Path 

func (db *DB) Path() string

Path returns the path to the database.

func (*DB) Pos added in v0.2.0 

func (db *DB) Pos() (Pos, error)

Pos returns the current position of the database.

func (*DB) Replica added in v0.2.0 

func (db *DB) Replica(name string) Replica

Replica returns a replica by name.

func (*DB) Restore 

func (db *DB) Restore(ctx context.Context, opt RestoreOptions) error

Restore restores the database from a replica based on the options given. This method will restore into opt.OutputPath, if specified, or into the DB's original database path. It can optionally restore from a specific replica or generation or it will automatically choose the best one. Finally, a timestamp can be specified to restore the database to a specific point-in-time.

func (*DB) ShadowWALDir added in v0.2.0 

func (db *DB) ShadowWALDir(generation string) string

ShadowWALDir returns the path of the shadow wal directory.

func (*DB) ShadowWALPath 

func (db *DB) ShadowWALPath(generation string, index int) string

ShadowWALPath returns the path of a single shadow WAL file.

func (*DB) ShadowWALReader 

func (db *DB) ShadowWALReader(pos Pos) (r *ShadowWALReader, err error)

ShadowWALReader opens a reader for a shadow WAL file at a given position. If the reader is at the end of the file, it attempts to return the next file.

The caller should check Pos() & Size() on the returned reader to check offset.

func (*DB) Snapshots added in v0.2.0 

func (db *DB) Snapshots(ctx context.Context) ([]*SnapshotInfo, error)

Snapshots returns a list of all snapshots across all replicas.

func (*DB) SoftClose 

func (db *DB) SoftClose() (err error)

SoftClose closes everything but the underlying db connection. This method is available because the binary needs to avoid closing the database on exit to prevent autocheckpointing.

func (*DB) Sync 

func (db *DB) Sync() (err error)

Sync copies pending data from the WAL to the shadow WAL.

func (*DB) UpdatedAt 

func (db *DB) UpdatedAt() (time.Time, error)

UpdatedAt returns the last modified time of the database or WAL file.

func (*DB) Validate added in v0.2.0 

func (db *DB) Validate(ctx context.Context, replicaName string, opt RestoreOptions) error

Validate restores the most recent data from a replica and validates that the resulting database matches the current database.

func (*DB) WALPath 

func (db *DB) WALPath() string

WALPath returns the path to the database's WAL file.

func (*DB) WALs added in v0.2.0 

func (db *DB) WALs(ctx context.Context) ([]*WALInfo, error)

WALs returns a list of all WAL files across all replicas.

type FileReplica 

type FileReplica struct {
	// contains filtered or unexported fields
}

FileReplica is a replica that replicates a DB to a local file path.

func NewFileReplica 

func NewFileReplica(db *DB, name, dst string) *FileReplica

NewFileReplica returns a new instance of FileReplica.

func (*FileReplica) CalcPos added in v0.2.0 

func (r *FileReplica) CalcPos(generation string) (pos Pos, err error)

CalcPos returns the position for the replica for the current generation. Returns a zero value if there is no active generation.

func (*FileReplica) GenerationStats 

func (r *FileReplica) GenerationStats(ctx context.Context, generation string) (stats GenerationStats, err error)

GenerationStats returns stats for a generation.

func (*FileReplica) Generations 

func (r *FileReplica) Generations(ctx context.Context) ([]string, error)

Generations returns a list of available generation names.

func (*FileReplica) LastPos added in v0.2.0 

func (r *FileReplica) LastPos() Pos

LastPos returns the last successfully replicated position.

func (*FileReplica) MaxSnapshotIndex added in v0.2.0 

func (r *FileReplica) MaxSnapshotIndex(generation string) (int, error)

MaxSnapshotIndex returns the highest index for the snapshots.

func (*FileReplica) MaxWALSubdirName added in v0.2.0 

func (r *FileReplica) MaxWALSubdirName(generation string) (string, error)

MaxWALSubdirName returns the highest WAL subdirectory group name.

func (*FileReplica) Name 

func (r *FileReplica) Name() string

Name returns the name of the replica. Returns the type if no name set.

func (*FileReplica) SnapshotDir 

func (r *FileReplica) SnapshotDir(generation string) string

SnapshotDir returns the path to a generation's snapshot directory.

func (*FileReplica) SnapshotIndexAt 

func (r *FileReplica) SnapshotIndexAt(ctx context.Context, generation string, timestamp time.Time) (int, error)

SnapsotIndexAt returns the highest index for a snapshot within a generation that occurs before timestamp. If timestamp is zero, returns the latest snapshot.

func (*FileReplica) SnapshotPath 

func (r *FileReplica) SnapshotPath(generation string, index int) string

SnapshotPath returns the path to a snapshot file.

func (*FileReplica) SnapshotReader 

func (r *FileReplica) SnapshotReader(ctx context.Context, generation string, index int) (io.ReadCloser, error)

SnapshotReader returns a reader for snapshot data at the given generation/index. Returns os.ErrNotExist if no matching index is found.

func (*FileReplica) Snapshots added in v0.2.0 

func (r *FileReplica) Snapshots(ctx context.Context) ([]*SnapshotInfo, error)

Snapshots returns a list of available snapshots in the replica.

func (*FileReplica) Start 

func (r *FileReplica) Start(ctx context.Context)

Start starts replication for a given generation.

func (*FileReplica) Stop 

func (r *FileReplica) Stop()

Stop cancels any outstanding replication and blocks until finished.

func (*FileReplica) Sync added in v0.2.0 

func (r *FileReplica) Sync(ctx context.Context) (err error)

func (*FileReplica) Type 

func (r *FileReplica) Type() string

Type returns the type of replica.

func (*FileReplica) WALDir 

func (r *FileReplica) WALDir(generation string) string

WALDir returns the path to a generation's WAL directory

func (*FileReplica) WALIndexAt 

func (r *FileReplica) WALIndexAt(ctx context.Context, generation string, maxIndex int, timestamp time.Time) (int, error)

Returns the highest index for a WAL file that occurs before maxIndex & timestamp. If timestamp is zero, returns the highest WAL index.

func (*FileReplica) WALPath 

func (r *FileReplica) WALPath(generation string, index int) string

WALPath returns the path to a WAL file.

func (*FileReplica) WALReader 

func (r *FileReplica) WALReader(ctx context.Context, generation string, index int) (io.ReadCloser, error)

WALReader returns a reader for WAL data at the given index. Returns os.ErrNotExist if no matching index is found.

func (*FileReplica) WALSubdir added in v0.2.0 

func (r *FileReplica) WALSubdir(generation string, index int) string

WALSubdir returns the directory used for grouping WAL files.

func (*FileReplica) WALSubdirNames added in v0.2.0 

func (r *FileReplica) WALSubdirNames(generation string) ([]string, error)

WALSubdirNames returns a list of all WAL subdirectory group names.

func (*FileReplica) WALs added in v0.2.0 

func (r *FileReplica) WALs(ctx context.Context) ([]*WALInfo, error)

WALs returns a list of available WAL files in the replica.

type GenerationStats 

type GenerationStats struct {
	// Count of snapshot & WAL files.
	SnapshotN int
	WALN      int

	// Time range for the earliest snapshot & latest WAL file update.
	CreatedAt time.Time
	UpdatedAt time.Time
}

GenerationStats represents high level stats for a single generation.

type Pos 

type Pos struct {
	Generation string // generation name
	Index      int    // wal file index
	Offset     int64  // offset within wal file
}

Pos is a position in the WAL for a generation.

func (Pos) IsZero 

func (p Pos) IsZero() bool

IsZero returns true if p is the zero value.

func (Pos) String 

func (p Pos) String() string

String returns a string representation.

type Replica 

type Replica interface {
	// The name of the replica. Defaults to type if no name specified.
	Name() string

	// String identifier for the type of replica ("file", "s3", etc).
	Type() string

	// Starts replicating in a background goroutine.
	Start(ctx context.Context)

	// Stops all replication processing. Blocks until processing stopped.
	Stop()

	// Returns the last replication position.
	LastPos() Pos

	// Returns the computed position of the replica for a given generation.
	CalcPos(generation string) (Pos, error)

	// Returns a list of generation names for the replica.
	Generations(ctx context.Context) ([]string, error)

	// Returns basic information about a generation including the number of
	// snapshot & WAL files as well as the time range covered.
	GenerationStats(ctx context.Context, generation string) (GenerationStats, error)

	// Returns a list of available snapshots in the replica.
	Snapshots(ctx context.Context) ([]*SnapshotInfo, error)

	// Returns a list of available WAL files in the replica.
	WALs(ctx context.Context) ([]*WALInfo, error)

	// Returns the highest index for a snapshot within a generation that occurs
	// before timestamp. If timestamp is zero, returns the latest snapshot.
	SnapshotIndexAt(ctx context.Context, generation string, timestamp time.Time) (int, error)

	// Returns the highest index for a WAL file that occurs before timestamp.
	// If timestamp is zero, returns the highest WAL index.
	WALIndexAt(ctx context.Context, generation string, maxIndex int, timestamp time.Time) (int, error)

	// Returns a reader for snapshot data at the given generation/index.
	SnapshotReader(ctx context.Context, generation string, index int) (io.ReadCloser, error)

	// Returns a reader for WAL data at the given position.
	WALReader(ctx context.Context, generation string, index int) (io.ReadCloser, error)
}

Replica represents a remote destination to replicate the database & WAL.

type RestoreOptions 

type RestoreOptions struct {
	// Target path to restore into.
	// If blank, the original DB path is used.
	OutputPath string

	// Specific replica to restore from.
	// If blank, all replicas are considered.
	ReplicaName string

	// Specific generation to restore from.
	// If blank, all generations considered.
	Generation string

	// Specific index to restore from.
	// Set to math.MaxInt64 to ignore index.
	Index int

	// Point-in-time to restore database.
	// If zero, database restore to most recent state available.
	Timestamp time.Time

	// If true, no actual restore is performed.
	// Only equivalent log output for a regular restore.
	DryRun bool

	// Logger used to print status to.
	Logger *log.Logger
}

RestoreOptions represents options for DB.Restore().

func NewRestoreOptions added in v0.2.0 

func NewRestoreOptions() RestoreOptions

type ShadowWALReader 

type ShadowWALReader struct {
	// contains filtered or unexported fields
}

ShadowWALReader represents a reader for a shadow WAL file that tracks WAL position.

func (*ShadowWALReader) Close 

func (r *ShadowWALReader) Close() error

Close closes the underlying WAL file handle.

func (*ShadowWALReader) N 

func (r *ShadowWALReader) N() int64

N returns the remaining bytes in the reader.

func (*ShadowWALReader) Pos 

func (r *ShadowWALReader) Pos() Pos

Pos returns the current WAL position.

func (*ShadowWALReader) Read 

func (r *ShadowWALReader) Read(p []byte) (n int, err error)

Read reads bytes into p, updates the position, and returns the bytes read. Returns io.EOF at the end of the available section of the WAL.

type SnapshotInfo added in v0.2.0 

type SnapshotInfo struct {
	Name       string
	Replica    string
	Generation string
	Index      int
	Size       int64
	CreatedAt  time.Time
}

SnapshotInfo represents file information about a snapshot.

type WALInfo added in v0.2.0 

type WALInfo struct {
	Name       string
	Replica    string
	Generation string
	Index      int
	Offset     int64
	Size       int64
	CreatedAt  time.Time
}

WALInfo represents file information about a WAL file.

Source Files

View all Source files

Directories

Path Synopsis
cmd
litestream command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.