walship

package
v0.9.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 6, 2026 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Overview

Package walship is an opt-in prototype of continuous SQLite replication by WAL frame shipping (ADR 039). It is not wired into the control plane.

Index

Constants

View Source
const DefaultCheckpointBytes = 16 << 20

DefaultCheckpointBytes is the WAL size at which the shipper checkpoints and lets SQLite restart the WAL.

View Source
const (

	// DefaultObjectTimeout bounds one object-store call; callers override it from env.
	DefaultObjectTimeout = 2 * time.Minute
)

Variables

View Source
var ErrNoIdentity = errors.New("walship: sink has no age identity to decrypt with")

ErrNoIdentity means a Get was attempted on a sink built without identities.

Functions

This section is empty.

Types

type Bucket

type Bucket interface {
	Put(ctx context.Context, key string, body io.ReadSeeker, contentType, contentEncoding string) error
	Get(ctx context.Context, key string) (io.ReadCloser, int64, error)
	List(ctx context.Context, prefix, token string, limit int32) (ListPage, error)
}

Bucket is the slice of an object-store client the sink needs.

type DirSink

type DirSink struct{ Dir string }

DirSink stores replica objects as files under Dir.

func (DirSink) Get

func (d DirSink) Get(name string) (io.ReadCloser, error)

Get opens name.

func (DirSink) List

func (d DirSink) List(prefix string) ([]string, error)

List returns object names with the prefix, sorted.

func (DirSink) Put

func (d DirSink) Put(name string, r io.Reader) error

Put writes name atomically.

type ListPage

type ListPage struct {
	Keys []string
	Next string
}

ListPage is one page of keys; local so walship stays off objectstore's import graph and its large fan-in.

type ObjectSink

type ObjectSink struct {
	Bucket     Bucket
	Prefix     string
	Recipients []age.Recipient
	Identities []age.Identity
	// Timeout bounds each call; zero means DefaultObjectTimeout.
	Timeout time.Duration
	// TempDir holds ciphertext while uploading; empty uses the OS default.
	TempDir string
}

ObjectSink stores each replica object as one age-encrypted S3 object under Prefix, so the bucket only ever sees ciphertext. The shipping side needs only Recipients; the standby adds Identities to read.

func (*ObjectSink) Get

func (o *ObjectSink) Get(name string) (io.ReadCloser, error)

Get downloads and decrypts name. The timeout covers the whole read.

func (*ObjectSink) List

func (o *ObjectSink) List(prefix string) ([]string, error)

List returns object names (prefix stripped) that start with prefix, sorted.

func (*ObjectSink) Put

func (o *ObjectSink) Put(name string, r io.Reader) error

Put encrypts r to a temp file (the SDK needs a seekable body) and uploads it.

type Reader

type Reader struct{ Sink Sink }

Reader lists and fetches what a Shipper wrote; promote and status build on it.

func (Reader) Fetch

func (r Reader) Fetch(name string) ([]byte, error)

Fetch returns the decoded bytes of one object.

func (Reader) Generations

func (r Reader) Generations() ([]int, error)

Generations returns generation numbers that have a snapshot, ascending.

func (Reader) Segments

func (r Reader) Segments(gen int) ([]Segment, error)

Segments lists one generation's segments ordered by epoch then offset.

type RestoreResult

type RestoreResult struct {
	Generation int
	Epochs     int
	FrameBytes int64
	// Truncated is true when a missing segment stopped replay early.
	Truncated bool
}

RestoreResult describes what Restore replayed.

func Restore

func Restore(ctx context.Context, sink Sink, dest string) (RestoreResult, error)

Restore rebuilds the database at dest from the newest generation in sink.

type Segment

type Segment struct {
	Name       string
	Generation int
	Seq        uint32
	Offset     int64
}

Segment is one shipped WAL segment found in a sink.

type Shipper

type Shipper struct {
	// contains filtered or unexported fields
}

Shipper copies new WAL frames of one SQLite database to a Sink. It holds a read transaction between syncs so SQLite cannot restart the WAL under it.

func New

func New(path string, sink Sink, checkpointBytes int64) (*Shipper, error)

New opens a shipper for the database at path. checkpointBytes <= 0 uses DefaultCheckpointBytes.

func (*Shipper) Close

func (s *Shipper) Close() error

Close releases the read hold and the connection pool.

func (*Shipper) Run

func (s *Shipper) Run(ctx context.Context, interval time.Duration) error

Run syncs every interval until ctx ends.

func (*Shipper) Stats

func (s *Shipper) Stats() Stats

Stats returns a copy of the counters.

func (*Shipper) Sync

func (s *Shipper) Sync(ctx context.Context) error

Sync ships every committed frame not yet shipped, then checkpoints when the WAL is large.

type Sink

type Sink interface {
	Put(name string, r io.Reader) error
	Get(name string) (io.ReadCloser, error)
	List(prefix string) ([]string, error)
}

Sink is where the replica lives (a directory here; an object store later).

type Stats

type Stats struct {
	Generations   int
	Segments      int
	SnapshotBytes int64
	SegmentBytes  int64
	FrameBytes    int64
	Checkpoints   int
}

Stats counts what a Shipper has done since it was created.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL