litestream

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 29, 2020 License: GPL-3.0 Imports: 21 Imported by: 13

README

litestream

Streaming replication for SQLite.

Questions

  • How to avoid WAL checkpointing on close?

Notes

-- Disable autocheckpointing.
PRAGMA wal_autocheckpoint = 0

Documentation

Index

Constants

View Source 
const (
	DefaultMonitorInterval    = 1 * time.Second
	DefaultMinCheckpointPageN = 1000
)

Default DB settings.

View Source 
const (
	MetaDirSuffix = "-litestream"

	WALDirName  = "wal"
	WALExt      = ".wal"
	SnapshotExt = ".snapshot"

	GenerationNameLen = 16
)
View Source 
const (
	// WALHeaderSize is the size of the WAL header, in bytes.
	WALHeaderSize = 32

	// WALFrameHeaderSize is the size of the WAL frame header, in bytes.
	WALFrameHeaderSize = 24
)
View Source 
const WALFrameHeaderChecksumOffset = 16
View Source 
const WALHeaderChecksumOffset = 24

Variables

View Source 
var (
	ErrNoSnapshots = errors.New("no snapshots available")
)

Litestream errors.

Functions

func Checksum 

func Checksum(bo binary.ByteOrder, s0, s1 uint32, b []byte) (uint32, uint32)

Checksum computes a running SQLite checksum over a byte slice.

func FormatWALFilename 

func FormatWALFilename(index int) string

func HexDump 

func HexDump(b []byte) string

HexDump returns hexdump output but with duplicate lines removed.

func IsGenerationName 

func IsGenerationName(s string) bool

IsGenerationName returns true if s is the correct length and is only lowercase hex characters.

func IsSnapshotPath 

func IsSnapshotPath(s string) bool

IsSnapshotPath returns true if s is a path to a snapshot file.

func IsWALPath 

func IsWALPath(s string) bool

IsWALPath returns true if s is a path to a WAL file.

func ParseSnapshotPath 

func ParseSnapshotPath(s string) (index int, typ, ext string, err error)

ParseSnapshotPath returns the index for the snapshot. Returns an error if the path is not a valid snapshot path.

func ParseWALFilename 

func ParseWALFilename(name string) (index int, err error)

func ParseWALPath 

func ParseWALPath(s string) (index int, offset int64, ext string, err error)

ParseWALPath returns the index & offset for the WAL file. Returns an error if the path is not a valid snapshot path.

Types

type DB 

type DB struct {

	// Minimum threshold of WAL size, in pages, before a passive checkpoint.
	// A passive checkpoint will attempt a checkpoint but fail if there are
	// active transactions occurring at the same time.
	MinCheckpointPageN int

	// Maximum threshold of WAL size, in pages, before a forced checkpoint.
	// A forced checkpoint will block new transactions and wait for existing
	// transactions to finish before issuing a checkpoint and resetting the WAL.
	//
	// If zero, no checkpoints are forced. This can cause the WAL to grow
	// unbounded if there are always read transactions occurring.
	MaxCheckpointPageN int

	// List of replicas for the database.
	// Must be set before calling Open().
	Replicas []Replica

	// Frequency at which to perform db sync.
	MonitorInterval time.Duration
	// contains filtered or unexported fields
}

DB represents a managed instance of a SQLite database in the file system.

func NewDB 

func NewDB(path string) *DB

NewDB returns a new instance of DB for a given path.

func (*DB) Close 

func (db *DB) Close() (err error)

Close releases the read lock & closes the database. This method should only be called by tests as it causes the underlying database to be checkpointed.

func (*DB) CurrentGeneration 

func (db *DB) CurrentGeneration() (string, error)

CurrentGeneration returns the name of the generation saved to the "generation" file in the meta data directory. Returns empty string if none exists.

func (*DB) CurrentShadowWALIndex 

func (db *DB) CurrentShadowWALIndex(generation string) (int, error)

CurrentShadowWALIndex returns the current WAL index for a given generation.

func (*DB) CurrentShadowWALPath 

func (db *DB) CurrentShadowWALPath(generation string) (string, error)

CurrentShadowWALPath returns the path to the last shadow WAL in a generation.

func (*DB) GenerationNamePath 

func (db *DB) GenerationNamePath() string

GenerationNamePath returns the path of the name of the current generation.

func (*DB) GenerationPath 

func (db *DB) GenerationPath(generation string) string

GenerationPath returns the path of a single generation.

func (*DB) Init 

func (db *DB) Init() (err error)

Init initializes the connection to the database. Skipped if already initialized or if the database file does not exist.

func (*DB) MetaPath 

func (db *DB) MetaPath() string

MetaPath returns the path to the database metadata.

func (*DB) Notify 

func (db *DB) Notify() <-chan struct{}

Notify returns a channel that closes when the shadow WAL changes.

func (*DB) Open 

func (db *DB) Open() (err error)

func (*DB) PageSize 

func (db *DB) PageSize() int

PageSize returns the page size of the underlying database. Only valid after database exists & Init() has successfully run.

func (*DB) Path 

func (db *DB) Path() string

Path returns the path to the database.

func (*DB) Restore 

func (db *DB) Restore(ctx context.Context, opt RestoreOptions) error

Restore restores the database from a replica based on the options given. This method will restore into opt.OutputPath, if specified, or into the DB's original database path. It can optionally restore from a specific replica or generation or it will automatically choose the best one. Finally, a timestamp can be specified to restore the database to a specific point-in-time.

func (*DB) ShadowWALPath 

func (db *DB) ShadowWALPath(generation string, index int) string

ShadowWALPath returns the path of a single shadow WAL file.

func (*DB) ShadowWALReader 

func (db *DB) ShadowWALReader(pos Pos) (r *ShadowWALReader, err error)

ShadowWALReader opens a reader for a shadow WAL file at a given position. If the reader is at the end of the file, it attempts to return the next file.

The caller should check Pos() & Size() on the returned reader to check offset.

func (*DB) SoftClose 

func (db *DB) SoftClose() (err error)

SoftClose closes everything but the underlying db connection. This method is available because the binary needs to avoid closing the database on exit to prevent autocheckpointing.

func (*DB) Sync 

func (db *DB) Sync() (err error)

Sync copies pending data from the WAL to the shadow WAL.

func (*DB) UpdatedAt 

func (db *DB) UpdatedAt() (time.Time, error)

UpdatedAt returns the last modified time of the database or WAL file.

func (*DB) WALPath 

func (db *DB) WALPath() string

WALPath returns the path to the database's WAL file.

type FileReplica 

type FileReplica struct {
	// contains filtered or unexported fields
}

FileReplica is a replica that replicates a DB to a local file path.

func NewFileReplica 

func NewFileReplica(db *DB, name, dst string) *FileReplica

NewFileReplica returns a new instance of FileReplica.

func (*FileReplica) GenerationStats 

func (r *FileReplica) GenerationStats(ctx context.Context, generation string) (stats GenerationStats, err error)

GenerationStats returns stats for a generation.

func (*FileReplica) Generations 

func (r *FileReplica) Generations(ctx context.Context) ([]string, error)

Generations returns a list of available generation names.

func (*FileReplica) Name 

func (r *FileReplica) Name() string

Name returns the name of the replica. Returns the type if no name set.

func (*FileReplica) SnapshotDir 

func (r *FileReplica) SnapshotDir(generation string) string

SnapshotDir returns the path to a generation's snapshot directory.

func (*FileReplica) SnapshotIndexAt 

func (r *FileReplica) SnapshotIndexAt(ctx context.Context, generation string, timestamp time.Time) (int, error)

SnapsotIndexAt returns the highest index for a snapshot within a generation that occurs before timestamp. If timestamp is zero, returns the latest snapshot.

func (*FileReplica) SnapshotPath 

func (r *FileReplica) SnapshotPath(generation string, index int) string

SnapshotPath returns the path to a snapshot file.

func (*FileReplica) SnapshotReader 

func (r *FileReplica) SnapshotReader(ctx context.Context, generation string, index int) (io.ReadCloser, error)

SnapshotReader returns a reader for snapshot data at the given generation/index. Returns os.ErrNotExist if no matching index is found.

func (*FileReplica) Start 

func (r *FileReplica) Start(ctx context.Context)

Start starts replication for a given generation.

func (*FileReplica) Stop 

func (r *FileReplica) Stop()

Stop cancels any outstanding replication and blocks until finished.

func (*FileReplica) Type 

func (r *FileReplica) Type() string

Type returns the type of replica.

func (*FileReplica) WALDir 

func (r *FileReplica) WALDir(generation string) string

WALDir returns the path to a generation's WAL directory

func (*FileReplica) WALIndexAt 

func (r *FileReplica) WALIndexAt(ctx context.Context, generation string, timestamp time.Time) (int, error)

Returns the highest index for a WAL file that occurs before timestamp. If timestamp is zero, returns the highest WAL index.

func (*FileReplica) WALPath 

func (r *FileReplica) WALPath(generation string, index int) string

WALPath returns the path to a WAL file.

func (*FileReplica) WALReader 

func (r *FileReplica) WALReader(ctx context.Context, generation string, index int) (io.ReadCloser, error)

WALReader returns a reader for WAL data at the given index. Returns os.ErrNotExist if no matching index is found.

type GenerationStats 

type GenerationStats struct {
	SnapshotN int
	WALN      int
	CreatedAt time.Time
	UpdatedAt time.Time
}

type Pos 

type Pos struct {
	Generation string // generation name
	Index      int    // wal file index
	Offset     int64  // offset within wal file
}

Pos is a position in the WAL for a generation.

func (Pos) IsZero 

func (p Pos) IsZero() bool

IsZero returns true if p is the zero value.

func (Pos) String 

func (p Pos) String() string

String returns a string representation.

type Replica 

type Replica interface {
	// The name of the replica. Defaults to type if no name specified.
	Name() string

	// String identifier for the type of replica ("file", "s3", etc).
	Type() string

	// Starts replicating in a background goroutine.
	Start(ctx context.Context)

	// Stops all replication processing. Blocks until processing stopped.
	Stop()

	// Returns a list of generation names for the replica.
	Generations(ctx context.Context) ([]string, error)

	// Returns basic information about a generation including the number of
	// snapshot & WAL files as well as the time range covered.
	GenerationStats(ctx context.Context, generation string) (GenerationStats, error)

	// Returns the highest index for a snapshot within a generation that occurs
	// before timestamp. If timestamp is zero, returns the latest snapshot.
	SnapshotIndexAt(ctx context.Context, generation string, timestamp time.Time) (int, error)

	// Returns the highest index for a WAL file that occurs before timestamp.
	// If timestamp is zero, returns the highest WAL index.
	WALIndexAt(ctx context.Context, generation string, timestamp time.Time) (int, error)

	// Returns a reader for snapshot data at the given generation/index.
	SnapshotReader(ctx context.Context, generation string, index int) (io.ReadCloser, error)

	// Returns a reader for WAL data at the given position.
	WALReader(ctx context.Context, generation string, index int) (io.ReadCloser, error)
}

Replica represents a remote destination to replicate the database & WAL.

type RestoreOptions 

type RestoreOptions struct {
	// Target path to restore into.
	// If blank, the original DB path is used.
	OutputPath string

	// Specific replica to restore from.
	// If blank, all replicas are considered.
	ReplicaName string

	// Specific generation to restore from.
	// If blank, all generations considered.
	Generation string

	// Point-in-time to restore database.
	// If zero, database restore to most recent state available.
	Timestamp time.Time

	// If true, no actual restore is performed.
	// Only equivalent log output for a regular restore.
	DryRun bool

	// Logger used to print status to.
	Logger *log.Logger
}

RestoreOptions represents options for DB.Restore().

type ShadowWALReader 

type ShadowWALReader struct {
	// contains filtered or unexported fields
}

ShadowWALReader represents a reader for a shadow WAL file that tracks WAL position.

func (*ShadowWALReader) Close 

func (r *ShadowWALReader) Close() error

Close closes the underlying WAL file handle.

func (*ShadowWALReader) N 

func (r *ShadowWALReader) N() int64

N returns the remaining bytes in the reader.

func (*ShadowWALReader) Pos 

func (r *ShadowWALReader) Pos() Pos

Pos returns the current WAL position.

func (*ShadowWALReader) Read 

func (r *ShadowWALReader) Read(p []byte) (n int, err error)

Read reads bytes into p, updates the position, and returns the bytes read. Returns io.EOF at the end of the available section of the WAL.

Source Files

View all Source files

Directories

Path Synopsis
cmd
litestream command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.