Documentation
¶
Overview ¶
Package wal is Atlas's write-ahead log: a segmented, append-only record store with group commit.
The log is the single source of truth (ADR-0001). Durability comes from fsync, but one fsync per record caps throughput at a few thousand per second (ADR-0005), so the WAL separates buffering from flushing: Log.Append stages records in memory and Log.Sync writes the whole batch and issues exactly one fsync. A processor appends every event of a batch, then calls Sync once — the "durable before visible" boundary (invariant I2).
Entries are opaque byte slices; the WAL does not interpret them, which keeps it decoupled from the record model. The unit of framing is the *batch*, not the record: one length and one CRC32C cover everything a Sync writes, so a write torn anywhere inside it fails as a whole and the batch is discarded entire. Framing each record separately made every prefix of that one write look like a shorter valid log, which let a crash leave half a command's events behind (ADR-0285).
Every segment opens with a 16-byte header naming the format, so a file written by an older build is recognised rather than misread. Those segments — one record per frame, no header — are still replayed, and are never appended to.
A Log is owned by a single goroutine (the partition's writer, invariant I3) and is not safe for concurrent use.
Index ¶
- type Cursor
- type Log
- func (l *Log) Append(data []byte) error
- func (l *Log) AppendContinuation(data []byte) error
- func (l *Log) Close() error
- func (l *Log) Compact(after uint64, positionOf func(data []byte) (uint64, error)) (int, error)
- func (l *Log) EarliestPosition(positionOf func(data []byte) (uint64, error)) (uint64, bool, error)
- func (l *Log) Replay(fn func(data []byte) error) error
- func (l *Log) ReplayForRecovery(after uint64, positionOf func(data []byte) (uint64, error), ...) ([]byte, error)
- func (l *Log) ReplayFrom(after uint64, positionOf func(data []byte) (uint64, error), ...) error
- func (l *Log) Sync() error
- type Options
- type Tailer
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Cursor ¶ added in v0.2.0
type Cursor struct {
// contains filtered or unexported fields
}
Cursor marks a resume point in the log for a Tailer: the segment (by its index in append order), a byte offset at a batch boundary within it, and how many of that batch's records have already been consumed. The zero Cursor is the start of the log (genesis). Its fields are unexported — a caller only ever stores a Cursor returned by Tailer.Read and passes it back; it is valid within one process run (segments are append-only today, never deleted), and a restart resumes from genesis by design (ADR-0114).
The record index is what keeps "resume at exactly the stopping record" true now that several records share one framed batch: without it, stopping part-way through a batch would have to resume at the batch's start and re-deliver the records before the stopping one.
type Log ¶
type Log struct {
// contains filtered or unexported fields
}
Log is a segmented append-only write-ahead log.
func Open ¶
Open opens (or creates) the log in opts.Dir. If the last segment has a torn tail from a crash mid-batch, it is truncated to the last valid frame so subsequent appends remain readable.
func (*Log) Append ¶
Append stages data as the next record. It is buffered, not durable, until Sync returns. The data is copied into the WAL's buffer, so the caller may reuse its slice immediately.
Records must be non-empty: a zero-length frame is indistinguishable from the zeroed tail a crash can leave, so the reader treats length zero as end of log. Real records always carry a header, so this is not a practical limit.
func (*Log) AppendContinuation ¶ added in v0.5.0
AppendContinuation stages the batch's outstanding work as a non-record entry. It is durable exactly when the batch's events are — the same frame, the same fsync — which is the point: the obligation to continue must not be able to survive or vanish separately from the events that created it.
At most one continuation belongs in a batch, and the newest one supersedes every earlier one, since each describes the whole queue rather than a delta.
func (*Log) Close ¶
Close closes the active segment. Records staged but not yet Synced are discarded — by contract they were never durable.
func (*Log) Compact ¶ added in v0.2.0
Compact deletes the segments a replay after `after` would never open, returning how many were removed. It is the disk-bounding half of ADR-0131.
The set it deletes is computed by the very same rule ReplayFrom uses to skip, via the same helper: a segment goes only when the next one provably starts past its last record, so "deleted" and "skipped" can never drift apart. Two consequences fall out structurally rather than by a check:
- the **active segment is never deleted** — the skip rule can never advance past the last segment, because nothing follows it to bound its extent;
- a segment is deleted only when *every* record in it is at or below after.
Deletion runs oldest-first, so an interruption leaves a contiguous suffix — still a valid log, just less compacted. The caller owns the safety question of *what* after may be: it must be a position that no future recovery, and no log consumer, still needs (Processor.CompactLog derives it). after == 0 deletes nothing.
func (*Log) EarliestPosition ¶ added in v0.5.0
EarliestPosition reports the log position of the oldest record still on disk, and whether there is one at all.
It is what lets a caller prove that the log still contains the prefix its state needs. Compaction deletes whole segments (ADR-0131), so after one the log no longer starts at genesis — and a replay of what remains looks exactly like a replay of everything unless somebody checks where "what remains" begins.
Segments that hold no readable record are skipped: a freshly rolled one carries only its header, which says nothing about position.
func (*Log) Replay ¶
Replay calls fn for every durable record across all segments, in append order. It reads only what is on disk, so records staged but not yet Synced are not visible. Replay is the recovery entry point: a processor folds these records through applyToState to rebuild state (ADR-0001).
The data slice passed to fn is owned by the caller for the duration of the call; it is freshly allocated per record, so it remains valid after fn returns. If fn returns an error, Replay stops and returns it.
func (*Log) ReplayForRecovery ¶ added in v0.5.0
func (l *Log) ReplayForRecovery(after uint64, positionOf func(data []byte) (uint64, error), onRecord func(data []byte) error) ([]byte, error)
ReplayForRecovery is Log.ReplayFrom plus the continuation the last batch in the log carried — the work that batch had scheduled and not yet done.
Recovery is the one reader that needs it. Every other reader folds events, and a continuation is not an event: it never reaches applyToState and replaying it would be replaying an intention (invariant I6). It is handed back separately so the caller can seed its queue with it and nothing else (ADR-0271).
The last one wins because each continuation describes the whole outstanding queue rather than a change to it, so an earlier one is a strictly older answer to the same question. nil means the log ends owing nothing.
func (*Log) ReplayFrom ¶ added in v0.2.0
func (l *Log) ReplayFrom(after uint64, positionOf func(data []byte) (uint64, error), fn func(data []byte) error) error
ReplayFrom is Replay restricted to the suffix of the log holding records past after. It is what lets recovery skip a prefix a checkpoint already covers (ADR-0131) instead of reading the whole log from genesis.
Log positions increase monotonically in append order, so a segment is entirely at or below after whenever the *next* segment starts at or below it. ReplayFrom uses that to skip whole segment files, reading just the first record of each to learn where it starts; positionOf extracts a record's log position (the wal package does not decode records itself). The surviving segments are replayed in full, so fn still sees the few records at or below after that share the boundary segment — filtering those is the caller's job, exactly as with Replay.
after == 0 means "everything", which is plain Replay. A segment whose first record cannot be read stops the skipping conservatively: replay starts no later than it. The final segment has no successor to bound its extent, so it is always replayed.
type Options ¶
type Options struct {
// Dir is the directory holding segment files. Created if absent.
Dir string
// MaxSegmentSize is the soft cap after which the next Sync rolls to a new
// segment. Zero means the default (64 MiB). A single batch is never split
// across segments, so a segment may exceed this by up to one batch.
MaxSegmentSize int64
// NoFsync writes each batch but does not force it to the platter, so a crash
// can lose the tail of the log.
//
// It exists for a log nobody will ever recover: the Playground's sandbox is
// discarded when the run ends, and its whole point is to be cheap. Turning it
// on for anything a process instance depends on breaks "durable before
// visible" (invariant I2), which is the one thing this log is for — so it is
// named after what it gives up rather than after the speed it buys.
NoFsync bool
}
Options configures a Log.
type Tailer ¶ added in v0.2.0
type Tailer struct {
// contains filtered or unexported fields
}
Tailer reads durable records forward from a Cursor, across segment rolls, resuming where a previous read left off. Unlike Log.Replay (one shot, from genesis), a Tailer is stateless apart from the Cursor the caller threads through, so it can poll a growing log for newly-appended records.
A Tailer opens segment files read-only and independently of the writing Log, so it is safe to run from another goroutine while the log is being appended (invariant I3). It reads only whole, CRC-valid frames and stops cleanly at a torn or not-yet-written tail — the same discipline recovery relies on — so it never observes a partially-written record. Records staged by Log.Append but not yet made durable by Log.Sync are not on disk and are invisible to it.
func (*Tailer) Read ¶ added in v0.2.0
Read invokes fn for each durable record at or after `from`, in append order. If fn returns stop=true, Read halts immediately and the stopping record is **not** consumed: the returned Cursor points at it, so a later Read from that Cursor re-delivers it. This lets a caller bound how far it reads by a limit it computes from the record itself (e.g. a durable-position watermark, ADR-0114) and resume at exactly that record next time.
When fn never stops, Read consumes every currently-durable record and returns the Cursor at the log's durable end. The []byte passed to fn is freshly allocated per record, so it remains valid after fn returns. On an I/O error Read returns the last safe Cursor and the error; the caller should not adopt a Cursor from a failed Read.
Source Files