Documentation
¶
Overview ¶
Package wal is Atlas's write-ahead log: a segmented, append-only record store with group commit.
The log is the single source of truth (ADR-0001). Durability comes from fsync, but one fsync per record caps throughput at a few thousand per second (ADR-0005), so the WAL separates buffering from flushing: Log.Append stages records in memory and Log.Sync writes the whole batch and issues exactly one fsync. A processor appends every event of a batch, then calls Sync once — the "durable before visible" boundary (invariant I2).
Entries are opaque byte slices; the WAL does not interpret them, which keeps it decoupled from the record model. Each entry is framed with a length and a CRC32C so forward iteration can detect a torn tail left by a crash mid-batch and stop cleanly at the last durable record.
A Log is owned by a single goroutine (the partition's writer, invariant I3) and is not safe for concurrent use.
Index ¶
- type Cursor
- type Log
- func (l *Log) Append(data []byte) error
- func (l *Log) Close() error
- func (l *Log) Compact(after uint64, positionOf func(data []byte) (uint64, error)) (int, error)
- func (l *Log) Replay(fn func(data []byte) error) error
- func (l *Log) ReplayFrom(after uint64, positionOf func(data []byte) (uint64, error), ...) error
- func (l *Log) Sync() error
- type Options
- type Tailer
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Cursor ¶ added in v0.2.0
type Cursor struct {
// contains filtered or unexported fields
}
Cursor marks a resume point in the log for a Tailer: the segment (by its index in append order) and a byte offset at a frame boundary within it. The zero Cursor is the start of the log (genesis). Its fields are unexported — a caller only ever stores a Cursor returned by Tailer.Read and passes it back; it is valid within one process run (segments are append-only today, never deleted), and a restart resumes from genesis by design (ADR-0114).
type Log ¶
type Log struct {
// contains filtered or unexported fields
}
Log is a segmented append-only write-ahead log.
func Open ¶
Open opens (or creates) the log in opts.Dir. If the last segment has a torn tail from a crash mid-batch, it is truncated to the last valid frame so subsequent appends remain readable.
func (*Log) Append ¶
Append stages data as the next record. It is buffered, not durable, until Sync returns. The data is copied into the WAL's buffer, so the caller may reuse its slice immediately.
Records must be non-empty: a zero-length frame is indistinguishable from the zeroed tail a crash can leave, so the reader treats length zero as end of log. Real records always carry a header, so this is not a practical limit.
func (*Log) Close ¶
Close closes the active segment. Records staged but not yet Synced are discarded — by contract they were never durable.
func (*Log) Compact ¶ added in v0.2.0
Compact deletes the segments a replay after `after` would never open, returning how many were removed. It is the disk-bounding half of ADR-0131.
The set it deletes is computed by the very same rule ReplayFrom uses to skip, via the same helper: a segment goes only when the next one provably starts past its last record, so "deleted" and "skipped" can never drift apart. Two consequences fall out structurally rather than by a check:
- the **active segment is never deleted** — the skip rule can never advance past the last segment, because nothing follows it to bound its extent;
- a segment is deleted only when *every* record in it is at or below after.
Deletion runs oldest-first, so an interruption leaves a contiguous suffix — still a valid log, just less compacted. The caller owns the safety question of *what* after may be: it must be a position that no future recovery, and no log consumer, still needs (Processor.CompactLog derives it). after == 0 deletes nothing.
func (*Log) Replay ¶
Replay calls fn for every durable record across all segments, in append order. It reads only what is on disk, so records staged but not yet Synced are not visible. Replay is the recovery entry point: a processor folds these records through applyToState to rebuild state (ADR-0001).
The data slice passed to fn is owned by the caller for the duration of the call; it is freshly allocated per record, so it remains valid after fn returns. If fn returns an error, Replay stops and returns it.
func (*Log) ReplayFrom ¶ added in v0.2.0
func (l *Log) ReplayFrom(after uint64, positionOf func(data []byte) (uint64, error), fn func(data []byte) error) error
ReplayFrom is Replay restricted to the suffix of the log holding records past after. It is what lets recovery skip a prefix a checkpoint already covers (ADR-0131) instead of reading the whole log from genesis.
Log positions increase monotonically in append order, so a segment is entirely at or below after whenever the *next* segment starts at or below it. ReplayFrom uses that to skip whole segment files, reading just the first record of each to learn where it starts; positionOf extracts a record's log position (the wal package does not decode records itself). The surviving segments are replayed in full, so fn still sees the few records at or below after that share the boundary segment — filtering those is the caller's job, exactly as with Replay.
after == 0 means "everything", which is plain Replay. A segment whose first record cannot be read stops the skipping conservatively: replay starts no later than it. The final segment has no successor to bound its extent, so it is always replayed.
type Options ¶
type Options struct {
// Dir is the directory holding segment files. Created if absent.
Dir string
// MaxSegmentSize is the soft cap after which the next Sync rolls to a new
// segment. Zero means the default (64 MiB). A single batch is never split
// across segments, so a segment may exceed this by up to one batch.
MaxSegmentSize int64
}
Options configures a Log.
type Tailer ¶ added in v0.2.0
type Tailer struct {
// contains filtered or unexported fields
}
Tailer reads durable records forward from a Cursor, across segment rolls, resuming where a previous read left off. Unlike Log.Replay (one shot, from genesis), a Tailer is stateless apart from the Cursor the caller threads through, so it can poll a growing log for newly-appended records.
A Tailer opens segment files read-only and independently of the writing Log, so it is safe to run from another goroutine while the log is being appended (invariant I3). It reads only whole, CRC-valid frames and stops cleanly at a torn or not-yet-written tail — the same discipline recovery relies on — so it never observes a partially-written record. Records staged by Log.Append but not yet made durable by Log.Sync are not on disk and are invisible to it.
func (*Tailer) Read ¶ added in v0.2.0
Read invokes fn for each durable record at or after `from`, in append order. If fn returns stop=true, Read halts immediately and the stopping record is **not** consumed: the returned Cursor points at it, so a later Read from that Cursor re-delivers it. This lets a caller bound how far it reads by a limit it computes from the record itself (e.g. a durable-position watermark, ADR-0114) and resume at exactly that record next time.
When fn never stops, Read consumes every currently-durable record and returns the Cursor at the log's durable end. The []byte passed to fn is freshly allocated per record, so it remains valid after fn returns. On an I/O error Read returns the last safe Cursor and the error; the caller should not adopt a Cursor from a failed Read.
Source Files