Documentation
¶
Overview ¶
Package wal implements a versioned, length-prefixed, CRC32C-checksummed Write-Ahead Log for the gograph durability stack.
The on-disk format is documented in FORMAT.md alongside this package. Each frame is self-describing; readers stop cleanly at the first torn or corrupted frame and report the byte offset where the cut occurred, leaving the file otherwise untouched.
Example ¶
Example shows the core write-ahead-log loop: open a writer, append a few opaque payload frames, Sync them durably, then reopen the file with a Reader and replay every frame back in append order.
package main
import (
"fmt"
"os"
"path/filepath"
"github.com/FlavioCFOliveira/GoGraph/store/wal"
)
func main() {
dir, err := os.MkdirTemp("", "wal-example")
if err != nil {
panic(err)
}
defer func() { _ = os.RemoveAll(dir) }()
path := filepath.Join(dir, "wal")
// Append three records. A WAL payload is opaque bytes as far as the
// log is concerned; the durability stack above it (store/txn) gives
// them meaning. Group-commit: several Appends, then a single Sync.
w, err := wal.Open(path)
if err != nil {
panic(err)
}
for _, rec := range [][]byte{[]byte("alpha"), []byte("bravo"), []byte("charlie")} {
if err := w.Append(rec); err != nil {
panic(err)
}
}
if err := w.Sync(); err != nil {
panic(err)
}
if err := w.Close(); err != nil {
panic(err)
}
// Replay: a fresh Reader iterates the frames in the order they were
// appended, stopping cleanly at the first torn frame (none here).
r, err := wal.OpenReader(path)
if err != nil {
panic(err)
}
defer func() { _ = r.Close() }()
count := 0
err = r.Replay(func(f wal.Frame) error {
count++
fmt.Printf("frame %d: %s\n", count, f.Payload)
return nil
})
if err != nil {
panic(err)
}
fmt.Printf("replayed %d frames\n", count)
}
Output: frame 1: alpha frame 2: bravo frame 3: charlie replayed 3 frames
Index ¶
- Constants
- Variables
- func Encode(w io.Writer, f Frame) (int, error)
- type Frame
- type Reader
- type Stats
- type Writer
- func (w *Writer) Append(payload []byte) error
- func (w *Writer) AppendCtx(ctx context.Context, payload []byte) error
- func (w *Writer) Close() error
- func (w *Writer) Stats() Stats
- func (w *Writer) Sync() error
- func (w *Writer) SyncCtx(ctx context.Context) error
- func (w *Writer) Truncate() (int64, error)
Examples ¶
Constants ¶
const CurrentVersion uint16 = 1
CurrentVersion is the WAL format version this package writes. Readers must accept all versions <= CurrentVersion; older versions are intentionally permitted so a fresh build can replay archives produced by previous releases.
const HeaderSize = 4 + 2 + 4 + 4
HeaderSize is the fixed number of bytes occupying the frame header (magic + version + length + crc32c).
Variables ¶
var ( // ErrBadMagic indicates the next four bytes did not match Magic. ErrBadMagic = errors.New("wal: bad frame magic") // ErrUnsupportedVersion indicates the frame version is newer // than this build knows how to parse. ErrUnsupportedVersion = errors.New("wal: unsupported frame version") // ErrCRCMismatch indicates the frame's CRC32C did not match the // re-computed value. ErrCRCMismatch = errors.New("wal: crc32c mismatch") // ErrTornFrame indicates the underlying reader returned EOF // before the frame was fully read. ErrTornFrame = errors.New("wal: torn frame at end of input") // ErrFrameTooLarge indicates the frame's declared payload length // exceeds maxFrameSize. A length field this large is treated as // corruption: the frame is rejected before any allocation, so a // crafted or corrupted length cannot force a large one-shot make. ErrFrameTooLarge = errors.New("wal: frame payload length exceeds maximum") )
Errors returned by the reader.
var ErrWriterClosed = errors.New("wal: writer is closed")
ErrWriterClosed is returned by methods on a Writer that has already been closed.
var Magic = [4]byte{'G', 'G', 'W', 'A'}
Magic is the 4-byte identifier prefix of every WAL frame: ASCII "GGWA".
Functions ¶
Types ¶
type Frame ¶
Frame is the in-memory representation of one WAL frame.
func Decode ¶
Decode reads the next frame from r. It returns ErrTornFrame when the reader ends mid-frame (clean tail truncation), ErrBadMagic on a missing magic, ErrUnsupportedVersion on a newer-than-supported version, and ErrCRCMismatch on integrity failure. Any other error is propagated from the underlying reader.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader iterates the frames of a WAL file. It is read-only and stops cleanly at the first torn or corrupted frame, reporting the byte offset where the cut occurred via Reader.TailOffset.
Reader is not safe for concurrent use; create one Reader per goroutine that wishes to iterate.
func NewReader ¶
NewReader builds a Reader over an io.Reader. closer may be nil if the caller owns the resource.
func OpenReader ¶
OpenReader opens path for read-only frame iteration.
func (*Reader) Close ¶
Close releases any underlying resource passed to NewReader or OpenReader.
func (*Reader) Frames ¶
Frames returns an iterator over every frame in the WAL. The iterator stops at the first error; call Reader.TailError / Reader.TailOffset after iteration to inspect why.
func (*Reader) Replay ¶
Replay applies apply to every frame in the WAL in order. If apply returns an error, replay stops with that error returned to the caller. After Replay returns, TailOffset/TailError describe where and why iteration stopped (frame-level errors).
func (*Reader) TailError ¶
TailError returns the error that ended iteration (typically ErrTornFrame, ErrCRCMismatch, or ErrBadMagic), or nil when iteration ended at clean EOF.
func (*Reader) TailOffset ¶
TailOffset returns the byte offset (from the start of the input) where iteration stopped. After a successful iteration to EOF this equals the file size; after a torn frame this equals the start of the torn frame.
type Stats ¶
type Stats struct {
Frames uint64 // total frames appended
Bytes uint64 // total bytes appended (header + payload)
Syncs uint64 // total Sync calls
SyncFailed uint64 // Sync calls that returned an error
}
Stats is a snapshot of a Writer's lifetime counters. Counters are monotonic; subtract two snapshots to compute deltas. Values are read with sync/atomic.LoadUint64, so they may race slightly behind in-flight operations but never observe a torn value.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer is a single-writer append-only log file. Callers append frames with Writer.Append and durably commit them with Writer.Sync; group-commit is achieved by appending several frames before a single Sync.
Writer is safe for concurrent calls to Writer.Append / Sync / Stats; all mutations serialise on an internal mutex.
func Open ¶
Open opens or creates the WAL file at path for append-only writing. The file is created with mode 0o600 (owner read/write only) if it does not already exist; existing data is preserved and new frames are appended. The restrictive mode keeps the full graph mutation stream from being world-readable.
func OpenWith ¶
OpenWith builds a Writer over an already-open file handle. The caller transfers ownership: Writer.Close will call f.Close().
This constructor exists primarily for tests that inject a *testfs.FaultFile; production code should use Open.
func (*Writer) Append ¶
Append writes one frame with the given opaque payload to the underlying file. The frame is buffered in process memory; call Writer.Sync to durably commit.
func (*Writer) AppendCtx ¶
AppendCtx is the context-aware variant of Writer.Append. ctx.Err() is checked before acquiring the internal mutex and again before writing; on cancellation returns the wrapped ctx.Err.
func (*Writer) Close ¶
Close flushes any buffered frames, calls Sync once, and releases the underlying file.
func (*Writer) Sync ¶
Sync flushes the buffered frames to the OS and then calls os.File.Sync so the data reaches durable storage before returning. It must be invoked at every transaction commit boundary.
func (*Writer) SyncCtx ¶
SyncCtx is the context-aware variant of Writer.Sync. ctx.Err() is checked before acquiring the internal mutex; on cancellation returns the wrapped ctx.Err without flushing.
func (*Writer) Truncate ¶
Truncate empties the WAL: flushes any buffered frames, truncates the underlying file to zero bytes, and fsyncs the result so the empty state is durable on disk before returning. Subsequent Writer.Append calls write to offset 0 of the freshly-empty file.
Truncate is intended to be called by the checkpointer after a snapshot covering all WAL frames has been durably persisted; on success every frame previously durable in the WAL is logically folded into the snapshot.
Lifetime counters in Writer.Stats are NOT reset; the returned int64 reports the number of bytes that were in the file at the moment of truncation (after the in-memory buffer was flushed), which is the canonical measure of WAL bytes freed by this call.
On error the WAL may be in a partially-truncated state; callers should not continue using the Writer.
Source Files