Documentation
¶
Overview ¶
Package wal implements a versioned, length-prefixed, CRC32C-checksummed Write-Ahead Log for the gograph durability stack.
The on-disk format is documented in FORMAT.md alongside this package. Each frame is self-describing; readers stop cleanly at the first torn or corrupted frame and report the byte offset where the cut occurred, leaving the file otherwise untouched.
Example ¶
Example shows the core write-ahead-log loop: open a writer, append a few opaque payload frames, Sync them durably, then reopen the file with a Reader and replay every frame back in append order.
package main
import (
"fmt"
"os"
"path/filepath"
"github.com/FlavioCFOliveira/GoGraph/store/wal"
)
func main() {
dir, err := os.MkdirTemp("", "wal-example")
if err != nil {
panic(err)
}
defer func() { _ = os.RemoveAll(dir) }()
path := filepath.Join(dir, "wal")
// Append three records. A WAL payload is opaque bytes as far as the
// log is concerned; the durability stack above it (store/txn) gives
// them meaning. Group-commit: several Appends, then a single Sync.
w, err := wal.Open(path)
if err != nil {
panic(err)
}
for _, rec := range [][]byte{[]byte("alpha"), []byte("bravo"), []byte("charlie")} {
if err := w.Append(rec); err != nil {
panic(err)
}
}
if err := w.Sync(); err != nil {
panic(err)
}
if err := w.Close(); err != nil {
panic(err)
}
// Replay: a fresh Reader iterates the frames in the order they were
// appended, stopping cleanly at the first torn frame (none here).
r, err := wal.OpenReader(path)
if err != nil {
panic(err)
}
defer func() { _ = r.Close() }()
count := 0
err = r.Replay(func(f wal.Frame) error {
count++
fmt.Printf("frame %d: %s\n", count, f.Payload)
return nil
})
if err != nil {
panic(err)
}
fmt.Printf("replayed %d frames\n", count)
}
Output: frame 1: alpha frame 2: bravo frame 3: charlie replayed 3 frames
Index ¶
- Constants
- Variables
- func Encode(w io.Writer, f Frame) (int, error)
- type Frame
- type Reader
- type Stats
- type WALFile
- type Writer
- func (w *Writer) Append(payload []byte) error
- func (w *Writer) AppendCtx(ctx context.Context, payload []byte) error
- func (w *Writer) AppendRun(fn func(emit func([]byte) error) error) (int64, error)
- func (w *Writer) Close() error
- func (w *Writer) DurableOffset() int64
- func (w *Writer) Poisoned() error
- func (w *Writer) Stats() Stats
- func (w *Writer) Sync() error
- func (w *Writer) SyncBuffered() error
- func (w *Writer) SyncCtx(ctx context.Context) error
- func (w *Writer) SyncGroup(target int64) error
- func (w *Writer) Truncate() (int64, error)
- func (w *Writer) TruncatePrefix(upTo int64) (int64, error)
Examples ¶
Constants ¶
const CurrentVersion uint16 = 1
CurrentVersion is the WAL format version this package writes. Readers must accept all versions <= CurrentVersion; older versions are intentionally permitted so a fresh build can replay archives produced by previous releases.
const HeaderSize = 4 + 2 + 4 + 4
HeaderSize is the fixed number of bytes occupying the frame header (magic + version + length + crc32c).
Variables ¶
var ( // ErrBadMagic indicates the next four bytes did not match Magic. ErrBadMagic = errors.New("wal: bad frame magic") // ErrUnsupportedVersion indicates the frame version is newer // than this build knows how to parse. ErrUnsupportedVersion = errors.New("wal: unsupported frame version") // ErrCRCMismatch indicates the frame's CRC32C did not match the // re-computed value. ErrCRCMismatch = errors.New("wal: crc32c mismatch") // ErrTornFrame indicates the underlying reader returned EOF // before the frame was fully read. ErrTornFrame = errors.New("wal: torn frame at end of input") // ErrTornFrameMasksData indicates a frame's declared payload length // over-declared past the end of input AND the bytes it would have // consumed contain at least one further valid (CRC-checking) frame. // This is genuine mid-stream corruption masquerading as a benign torn // tail: a corrupt length field swallowed durable frames that follow it. // Unlike [ErrTornFrame] (a benign final partial write), this is a hard // error — it MUST fail-stop so the durable frames the bad length hid are // never silently dropped. It is a DISTINCT sentinel (it deliberately does // not wrap [ErrTornFrame]) so recovery's corruption classifier treats it // as corruption rather than a benign tail. ErrTornFrameMasksData = errors.New("wal: torn frame hides later valid frames (corrupt length)") // ErrFrameTooLarge indicates the frame's declared payload length // exceeds maxFrameSize. A length field this large is treated as // corruption: the frame is rejected before any allocation, so a // crafted or corrupted length cannot force a large one-shot make. ErrFrameTooLarge = errors.New("wal: frame payload length exceeds maximum") )
Errors returned by the reader.
var ErrDurabilityFailed = errors.New("wal: durability failed; the un-synced suffix was discarded and this writer is poisoned")
ErrDurabilityFailed marks every error a POISONED writer returns: the write-ahead log could not be made durable, the un-synced suffix has been discarded, and this writer will refuse every further append and sync.
It wraps the underlying I/O error, so `errors.Is(err, ErrDurabilityFailed)` identifies the class and `errors.Unwrap` still reaches the cause.
Why it exists, and what it is NOT ¶
It is NOT retriable, and that is the whole point of naming it. A group commit is FAIL-ALL: when the leader's fsync fails, every member's frames and OpCommit markers are discarded together, so a transaction that did nothing wrong fails because another transaction's I/O failed. While commits were serialised that was unremarkable — the batch was one unit. Once writers are independent (rmp #2306) a caller needs to tell "MY transaction lost a conflict, retry it" from "the storage substrate failed, everything in flight is gone, and retrying will not help", because the two demand opposite responses and both used to arrive as an undistinguished error.
Fail-all is kept rather than softened, because the alternative is to acknowledge a commit whose durability is unknown, which the module's ACID mandate forbids outright. It is also the LENIENT end of the prior art: PostgreSQL does not fail the transaction, it fails the PROCESS — `issue_xlog_fsync` carries the comment "PANIC if failed to fsync" and calls `ereport(PANIC, …)` (postgres/postgres, branch master, read 2026-08-04 at commit 69ed7fd7e9da1cff2f04af04f630287971fe99fe; src/backend/access/transam/xlog.c). GoGraph cannot take that route: it is a library embedded in the caller's process, killing the host is not its decision to make, and the reliability mandate forbids the library from crashing. So the handle dies and says so, which is PostgreSQL's conclusion scoped to what a library owns.
var ErrPrefixTruncateUnsupported = errors.New("wal: TruncatePrefix requires a path-backed writer (use Open, not OpenWith)")
ErrPrefixTruncateUnsupported is returned by Writer.TruncatePrefix on a Writer created via OpenWith (a synthetic, path-less file handle). The crash-safe prefix truncation works by writing the surviving suffix to a sibling temp file and atomically renaming it over the WAL path, so it requires a real filesystem path — which only Open records.
var ErrWALLocked = errors.New("wal: WAL directory is locked by another process")
ErrWALLocked is returned by Open when another process already holds the exclusive OS-level lock on the WAL directory. It signals that the WAL is in active use and the caller must not open a second writer against it — doing so would silently interleave frames and corrupt the log.
var ErrWriterClosed = errors.New("wal: writer is closed")
ErrWriterClosed is returned by methods on a Writer that has already been closed.
var Magic = [4]byte{'G', 'G', 'W', 'A'}
Magic is the 4-byte identifier prefix of every WAL frame: ASCII "GGWA".
Functions ¶
Types ¶
type Frame ¶
Frame is the in-memory representation of one WAL frame.
A Frame carries no synchronisation of its own, so its concurrency contract follows the ownership of Payload. A Frame returned by Decode owns its Payload outright — the decoder allocates a fresh slice per frame and never aliases the reader's buffer — so it is safe to hand to another goroutine and to read concurrently. A Frame passed to Encode merely borrows the caller's Payload for the duration of that call, which is what lets the transaction layer re-use one pooled scratch buffer for every op; such a Frame must not be retained or shared past the call that consumed it.
func Decode ¶
Decode reads the next frame from r. It returns ErrTornFrame when the reader ends mid-frame (clean tail truncation), ErrBadMagic on a missing magic, ErrUnsupportedVersion on a newer-than-supported version, and ErrCRCMismatch on integrity failure. Any other error is propagated from the underlying reader.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader iterates the frames of a WAL file. It is read-only and stops cleanly at the first torn or corrupted frame, reporting the byte offset where the cut occurred via Reader.TailOffset.
Reader is not safe for concurrent use; create one Reader per goroutine that wishes to iterate.
func NewReader ¶
NewReader builds a Reader over an io.Reader. closer may be nil if the caller owns the resource.
func OpenReader ¶
OpenReader opens path for read-only frame iteration.
func (*Reader) Close ¶
Close releases any underlying resource passed to NewReader or OpenReader.
func (*Reader) Frames ¶
Frames returns an iterator over every frame in the WAL. The iterator stops at the first error; call Reader.TailError / Reader.TailOffset after iteration to inspect why.
func (*Reader) Replay ¶
Replay applies apply to every frame in the WAL in order. If apply returns an error, replay stops with that error returned to the caller. After Replay returns, TailOffset/TailError describe where and why iteration stopped (frame-level errors).
func (*Reader) TailError ¶
TailError returns the error that ended iteration (typically ErrTornFrame, ErrCRCMismatch, or ErrBadMagic), or nil when iteration ended at clean EOF.
func (*Reader) TailOffset ¶
TailOffset returns the byte offset (from the start of the input) where iteration stopped. After a successful iteration to EOF this equals the file size; after a torn frame this equals the start of the torn frame.
type Stats ¶
type Stats struct {
Frames uint64 // total frames appended
Bytes uint64 // total bytes appended (header + payload)
Syncs uint64 // total Sync calls
// SyncFailed counts Sync calls that failed at the flush/fsync
// I/O layer. Calls rejected because the writer was already
// poisoned by an earlier failure are not counted (mirroring how
// context-cancelled calls are not counted).
SyncFailed uint64
}
Stats is a snapshot of a Writer's lifetime counters. Counters are monotonic; subtract two snapshots to compute deltas. Values are read with sync/atomic.LoadUint64, so they may race slightly behind in-flight operations but never observe a torn value. The four counters are loaded one at a time, so a Stats is a per-field snapshot rather than a single atomic view across all four.
The value Writer.Stats returns is a detached copy of plain integers, so a Stats is safe for concurrent reads and Writer.Stats is safe to call concurrently with Writer.Append and Writer.Sync.
type WALFile ¶ added in v0.6.0
type WALFile interface {
io.Writer
// Reader is required by [Writer.TruncatePrefix], which reads the
// surviving suffix of the file (the frames committed after the
// captured watermark) before atomically replacing the file with a
// suffix-only copy. *os.File and *testfs.FaultFile both satisfy it.
io.Reader
io.Seeker
// Sync flushes OS write buffers to durable storage.
Sync() error
// Truncate resizes the file to size bytes.
Truncate(size int64) error
// Close releases underlying OS resources.
Close() error
}
WALFile is the minimal open-handle interface that Writer requires of its underlying file. *os.File and *testfs.FaultFile both satisfy it, which lets fault-injection tests substitute a synthetic file without touching production paths.
It is exported so an external filesystem backend (the deterministic- simulation harness, internal/sim) can name it as the return type of its [walFS].OpenFile method and thereby satisfy the unexported walFS interface, exactly as github.com/FlavioCFOliveira/GoGraph/store/snapshot.File is exported for the snapshot seam. Production callers open WAL files via Open (which wraps *os.File) and never reference this type directly; tests and the simulator reach for OpenWith / OpenFS.
Concurrency: a WALFile is used by a single Writer whose own mutex serialises every access; any implementation's further guarantees are its own.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer is a single-writer append-only log file. Callers append frames with Writer.Append and durably commit them with Writer.Sync; group-commit is achieved by appending several frames before a single Sync.
Writer is safe for concurrent calls to Writer.Append / Sync / Stats; all mutations serialise on an internal mutex.
A Writer fail-stops on commit failure: the first flush or fsync error in Writer.Sync permanently poisons the writer. The un-synced suffix of the file — which may hold the flushed frames (including the commit marker) of the very transaction whose Sync just failed — is physically discarded, and every subsequent Append/Sync returns the original error. Without the poison, a later transaction's successful fsync would make the failed transaction's frames durable even though its commit was never acknowledged, and recovery would replay it: a phantom commit violating Atomicity and Durability. A poisoned Writer accepts only Writer.Close; the owner must discard it and re-open the WAL, which re-validates the tail.
func Open ¶
Open opens or creates the WAL file at path for append-only writing. The file is created with mode 0o600 (owner read/write only) if it does not already exist; existing complete frames are preserved and new frames are appended after them. The restrictive mode keeps the full graph mutation stream from being world-readable.
When the existing file ends in a benign torn frame (ErrTornFrame — the crash-mid-write-after-last-fsync case), Open truncates the file to the last durable frame boundary and fsyncs it before returning, so new frames are never appended after torn junk that every reader would stop at; see discardTornTail. Files whose scan stops at genuine corruption (for example ErrCRCMismatch or ErrBadMagic) are left byte-for-byte intact.
func OpenFS ¶ added in v0.6.0
OpenFS builds a path-backed Writer over a caller-supplied filesystem backend. Unlike OpenWith (which produces a path-less Writer that rejects Writer.TruncatePrefix with ErrPrefixTruncateUnsupported), OpenFS records the path and routes the truncation's temp-write/rename/remove/parent-dir-fsync through fsys, so a Checkpointer can reclaim the WAL prefix over the injected filesystem. The caller transfers ownership of the opened handle: Writer.Close will close it.
OpenFS is the seam the deterministic-simulation harness (internal/sim) uses to run the full snapshot + WAL + checkpoint stack against its in-memory [SimDisk]; production code uses Open.
Unlike Open, OpenFS does NOT acquire the OS-level WAL directory lock (flock(2)/O_EXCL has no analogue on an injected backend, and OpenFS callers are single-writer by contract) and does NOT scan for or discard a benign torn tail. The caller MUST therefore pre-truncate any benign torn tail to the last durable frame boundary (recovery reports it as github.com/FlavioCFOliveira/GoGraph/store/recovery.ReplayResult.WALTailOffset) BEFORE calling OpenFS, exactly as the production Open does internally via discardTornTail; appending after un-discarded torn junk would strand every new frame behind bytes that every reader stops at — a Durability violation.
func OpenWith ¶
OpenWith builds a Writer over an already-open file handle. The caller transfers ownership: Writer.Close will call f.Close().
This constructor exists primarily for tests that inject a *testfs.FaultFile; production code should use Open.
func (*Writer) Append ¶
Append writes one frame with the given opaque payload to the underlying file. The frame is buffered in process memory; call Writer.Sync to durably commit.
On a writer poisoned by an earlier Sync failure, Append rejects the frame and returns the original sync error; see the Writer type documentation.
func (*Writer) AppendCtx ¶
AppendCtx is the context-aware variant of Writer.Append. ctx.Err() is checked before acquiring the internal mutex and again before writing; on cancellation returns the wrapped ctx.Err.
func (*Writer) AppendRun ¶ added in v0.11.0
AppendRun appends every frame fn emits as ONE CONTIGUOUS RUN: no other appender's frame can land between them.
Why this exists — rmp #2302, audit finding E5 ¶
Crash recovery commits the ops carrying a marker's own TxnSeq and discards the buffered prefix as orphaned (store/recovery/recovery.go:1421-1429), and that reading is correct ONLY IF a transaction's frames are contiguous. Recovery says so in its own words: "The store serialises commits (single writer), so a transaction's frames are contiguous and never interleave with another's."
That contiguity did not come from here. Writer.AppendCtx serialises INDIVIDUAL appends; the run was held together by the store's single-writer semaphore two layers up (store/txn). The instant two writers append concurrently, interleaved frames make recovery drop COMMITTED ops — an Atomicity and Durability violation whose only symptom is the store.recovery.openCodec.orphanedOps counter.
So contiguity moves into the component that owns the file. Recovery's assumption stays TRUE rather than being relaxed, which is why this needs no on-disk format change, no new frame field, and no change to store/recovery at all.
The lock this holds, and why it is LESS than what it replaces ¶
w.mu is taken once, before fn, and released after it — so fn runs with the writer exclusively held. That is a longer critical section than one Append, and a strictly SHORTER one than the store semaphore it replaces, which spans a commit's encoding, its append loop and everything else it does. Every other writer can proceed with all of that; only the WAL append is exclusive.
Group commit is unaffected: Writer.SyncGroup coalesces on Sync, not on Append, and a run of appends followed by one Sync is the shape it already coalesces.
Contract ¶
The emit closure handed to fn is valid ONLY for the duration of the call; retaining it and calling it later writes into a writer this goroutine no longer holds. fn MUST NOT call any other method on this Writer — w.mu is not re-entrant and doing so deadlocks. Keep fn to encoding and appending.
An error from fn is returned unchanged, and frames already emitted stay in the buffer: they are an un-marked, incomplete transaction, which recovery discards for atomicity exactly as it does for a crash between the data frames and the commit marker. An error from append itself is the same fail-stop as Writer.AppendCtx's — a partial frame poisons the next Sync.
Prior art: PostgreSQL's XLogInsertRecord does the expensive work (assembling and CRCing the record) outside its insertion lock and holds it only for the copy (postgres/postgres, master, src/backend/access/transam/xlog.c). This is that insight at the granularity GoGraph needs: the per-op encoding stays with the caller, and the lock covers only the framing. # Return value — the run's own durability watermark
AppendRun returns the writer offset immediately after the run's last frame. That offset is the run's OWN watermark, and it is what the caller must hand to Writer.SyncGroup to make this run durable.
The caller cannot derive it afterwards. The writer's accepted offset is shared mutable state: another appender advances it, and a durability failure REWINDS it ([Writer.poison] resets appendedSize to durableSize). A committer that reads it later is asking about somebody else's frames — which is exactly how an unacknowledged transaction was resurrected after recovery (rmp #2322).
func (*Writer) Close ¶
Close flushes any buffered frames, calls Sync once, and releases the underlying file.
On a writer poisoned by an earlier Sync failure, Close skips the flush (which would buffer new data) but performs a second-chance truncation followed by a best-effort fsync. The truncation to durableSize discards any suffix that poison() may have failed to discard on a device in transient distress. The subsequent fsync makes the reduced inode metadata durable; it is safe to issue because the Truncate has already shrunk the file — the fsync can only confirm the reduced EOF, never resurrect data above it.
func (*Writer) DurableOffset ¶ added in v0.3.1
DurableOffset returns the file size, in bytes, covered by the last successful fsync — the byte length of every frame durably committed so far. The value always lands on a frame boundary: a transaction commits only after its [OpCommit] marker has been appended and fsynced, so the durable prefix is always a whole number of complete frames.
It is the watermark a non-blocking checkpoint captures (see store/checkpoint): taken under the store's quiesce boundary ([txn.Store.RunUnderCommitLock], which drains in-flight group commits) the returned offset equals appendedSize — every committed frame is durable and none is mid-flight — so it is exactly the prefix a self-sufficient snapshot folds and Writer.TruncatePrefix may discard.
Concurrency: safe for concurrent use; it reads durableSize under the internal mutex.
func (*Writer) Poisoned ¶ added in v0.8.0
Poisoned reports the writer's fail-stop state: it returns the sticky commit-failure error when a prior Writer.Sync / Writer.SyncGroup flush or fsync has permanently poisoned the writer (see the Writer type doc), and nil while the writer is healthy. The returned error is the same sentinel every subsequent Append/Sync returns.
It is the WAL-health probe a non-blocking checkpoint consults under the store's quiesce boundary ([txn.Store.RunUnderCommitLock]) BEFORE it captures and publishes a snapshot (rmp #1919). A concurrent schema DDL (CREATE/DROP CONSTRAINT or INDEX) whose commit fails at fsync poisons the writer and discards its frame — Writer.DurableOffset then excludes it — yet the engine's in-memory registry still reflects the attempted change until the DDL's out-of-lock compensator unwinds it. Because the poison is applied inside SyncGroup BEFORE the committer's in-flight token is released (store/txn markInflight/doneInflight), and the checkpoint captures only after RunUnderCommitLock drains in-flight commits to zero, a writer observed poisoned at capture time means exactly that transient window: folding the registry into constraints.bin / indexdefs.bin there would persist a non-acknowledged schema change across restart, violating Atomicity. The checkpoint therefore aborts (never publishing) when this returns non-nil, instead of only discovering the poison at the post-publish phase-2 Sync.
Concurrency: safe for concurrent use; it reads syncErr under the internal mutex.
func (*Writer) Sync ¶
Sync flushes the buffered frames to the OS and then issues the per-commit data sync (fdatasync(2) on Linux, os.File.Sync / fsync elsewhere; see dataSync) so the appended frames and the grown file size reach durable storage before returning. It must be invoked at every transaction commit boundary.
The first flush or fsync failure permanently poisons the writer: the un-synced suffix of the file is discarded and every subsequent Append/Sync returns the original error; see the Writer type documentation.
func (*Writer) SyncBuffered ¶ added in v0.11.0
SyncBuffered makes durable everything the writer has accepted at the moment of the call, coalescing with any concurrent group round exactly as Writer.SyncGroup does.
It is a FLUSH, not a commit acknowledgement, and a committer must NOT use it to learn whether its own frames are durable. The accepted offset is shared mutable state — another appender advances it, and [Writer.poison] rewinds it — so "everything accepted now" is not the caller's own watermark. Deciding a commit's fate from it resurrected an unacknowledged transaction after recovery (rmp #2322). Use Writer.AppendRun's returned watermark with Writer.SyncGroup for that. This exists for the callers that have nothing of their own to acknowledge and merely want any buffered tail on disk.
func (*Writer) SyncCtx ¶
SyncCtx is the context-aware variant of Writer.Sync. ctx.Err() is checked before acquiring the internal mutex; on cancellation returns the wrapped ctx.Err without flushing.
func (*Writer) SyncGroup ¶ added in v0.3.1
SyncGroup durably commits the caller's already-appended frames, coalescing the fsync with those of every other committer whose frames are buffered at the same time — PostgreSQL-XLogFlush-style group commit. It returns nil only after a single data sync (fdatasync on Linux, fsync elsewhere; see dataSync) has made durable every byte up to and including the caller's last appended frame (its OpCommit marker); a caller therefore acknowledges its commit only once the marker is on stable storage, exactly as Writer.Sync does, but without paying a private fsync per commit.
Contract ¶
SyncGroup must be called AFTER the caller has appended all of its frames, with target set to the watermark Writer.AppendRun returned for that run — the offset immediately after the caller's last frame (its OpCommit marker). It is the durability barrier for exactly those frames, then:
- If a previous sync has already advanced durableSize to target (a concurrent leader covered it), it returns nil without any I/O — the follower fast path.
- Otherwise, if the writer is poisoned, it returns the sticky error (the un-synced suffix, including this caller's frames, was discarded by an earlier failed sync).
- Otherwise, if no leader is flushing, the caller becomes the LEADER: it flushes the buffer and fsyncs once, covering its own and every other buffered committer's frames, publishes the new durableSize, and wakes the followers. If a leader is already flushing, the caller waits on the group condition until durableSize covers its watermark or the writer poisons.
Durability, atomicity, and failure semantics ¶
- DURABILITY: success is returned only after the fsync covering the caller's marker completes. Because all appends serialise (the buffer is FIFO and O_APPEND lands every write at EOF), a marker whose end offset is <= the flushed appendedSize is made durable by that flush's fsync; there is no prefix-only fsync on a local file system.
- ATOMICITY: the on-disk frame stream is unchanged from the per-commit path — each transaction's ops are contiguous and followed by its OpCommit marker — so a crash mid-leader-fsync recovers each fully-marked transaction and discards the unmarked tail exactly as before.
- FAIL-ALL: if the leader's flush or fsync fails, [Writer.poison] discards the entire un-synced suffix (every group member's frames and markers) and broadcasts; every waiter then observes the sticky error and fails its own commit. No member may believe it committed when the shared fsync failed.
Cancellation ¶
SyncGroup is intentionally NOT context-aware. Once a committer's frames are in the shared buffer they cannot be un-appended (later committers' frames sit after them, and the transaction sequence is consumed), so abandoning the wait while the group still fsyncs the frames would make the transaction durable — recovery replays a fully-marked transaction — while returning an error to the caller, risking a double apply on retry. The caller's deadline is honoured earlier, at the cancellable single-writer acquire ([Store.BeginCtx]); after the append point the commit is in-flight-durable and the wait for its covering fsync runs to completion (the marker either becomes durable or the writer poisons and fails it). This matches PostgreSQL: a backend cannot un-write WAL it has already inserted.
Concurrency: safe for concurrent calls; it serialises on the same internal mutex as Append/Sync and guarantees a single leader per fsync round.
func (*Writer) Truncate ¶
Truncate empties the WAL: flushes any buffered frames, truncates the underlying file to zero bytes, and fsyncs the result so the empty state is durable on disk before returning. Subsequent Writer.Append calls write to offset 0 of the freshly-empty file.
Truncate is intended to be called by the checkpointer after a snapshot covering all WAL frames has been durably persisted; on success every frame previously durable in the WAL is logically folded into the snapshot.
Lifetime counters in Writer.Stats are NOT reset; the returned int64 reports the number of bytes that were in the file at the moment of truncation (after the in-memory buffer was flushed), which is the canonical measure of WAL bytes freed by this call.
On error the WAL may be in a partially-truncated state; callers should not continue using the Writer.
func (*Writer) TruncatePrefix ¶ added in v0.3.1
TruncatePrefix crash-safely discards the WAL bytes in [0, upTo) and preserves every byte in [upTo, end) — the frames committed after the watermark a checkpoint captured. It is the WAL-prefix-reclamation primitive a non-blocking checkpoint uses: the snapshot folds the prefix [0, upTo), the suffix [upTo, end) holds transactions committed concurrently while the snapshot was written lock-free, and recovery replays that surviving suffix on top of the self-sufficient snapshot.
Why a copy-and-rename, not an in-place rewrite ¶
The WAL is opened O_APPEND, so writes cannot be repositioned; more importantly, rewriting the file in place (truncate to zero, then write the suffix back) has a fatal crash window: a crash after the truncate but before the suffix is rewritten leaves the suffix — committed transactions NOT present in the snapshot — permanently lost, a Durability violation. Instead TruncatePrefix writes the surviving suffix to a sibling temp file, fsyncs it, then atomically renames it over the WAL path and fsyncs the parent directory. rename(2) is atomic: a crash before the rename leaves the original full WAL intact (recovery = snapshot + full replay); a crash after it leaves the suffix-only WAL (recovery = snapshot + suffix replay); both reconstruct the exact committed state. This mirrors the file-granularity WAL reclamation of RocksDB (delete whole log files below the flush point) and PostgreSQL (recycle whole segments below the redo LSN), adapted to GoGraph's single un-segmented WAL file.
Contract and ordering ¶
The caller MUST hold the store's quiesce boundary ([txn.Store.RunUnderCommitLock]) so no concurrent Writer.Append races the rename or the durableSize/appendedSize/buffer reset, and MUST have made the covering snapshot fully durable (data fsync + snapshot publish + parent-dir fsync) BEFORE calling this. upTo must be a value previously returned by Writer.DurableOffset (a frame boundary) and must satisfy 0 <= upTo <= durableSize; an out-of-range upTo is rejected without touching the file. upTo == 0 is a no-op (nothing to reclaim).
On success the returned int64 is the number of bytes reclaimed (upTo) and the Writer continues against the suffix-only file. On a path-less Writer (OpenWith) it returns ErrPrefixTruncateUnsupported.
Error handling splits on the atomic rename:
- A failure BEFORE the rename (suffix copy, or the rename itself) leaves the ORIGINAL full WAL intact and the Writer usable; the error is returned and the caller may retry the checkpoint — the prefix is still present, so nothing is lost.
- A failure AFTER the rename (parent-dir fsync, or the reopen of the new inode) cannot be undone: the on-disk state has already advanced to the durable suffix-only WAL and the old inode is unlinked. The Writer is therefore POISONED (fail-stop): the error is returned and every subsequent Append/Sync returns it, so the owner must discard the Writer and re-open the WAL, which re-validates the already-correct on-disk suffix. The committed data is safe on disk; only the in-memory handle is abandoned.
Source Files