Documentation
¶
Overview ¶
Package wal implements a tiny write-ahead log so that state.Store.WithTransaction can commit MULTIPLE on-disk files atomically across a process crash.
Motivating problem. A single-file write is already atomic + durable via state/atomic_write.go (write-tmp → fsync → rename → fsync-dir). But a transaction body that writes two or three files in sequence — score handler: pool-matches.csv + competitor-status.yaml + lineups.yaml — has a window between rename N and rename N+1 where a crash can leave the half-committed state on disk. The first file reflects the new write; the rest still reflect the old. Replay on restart can't infer the intended end-state from the partial result.
Design. Two-phase commit, with the intent log stored on disk:
fn runs. Each tx-internal write goes through WAL.Append, which stages a FileIntent (full target path + full new bytes + mode) in memory. NO target file is touched. Append coalesces by path: two Append calls for the same target leave only the LAST bytes in the intent slice. This matches the score-handler rollback pattern where pool-matches.csv may be written twice in one tx (write new result → fail eligibility check → write old result back). Last-write-wins is the correct semantics.
fn returns nil. Caller calls Commit, which atomic-writes the intent log to <walDir>/<txID>.json via the same atomic-rename dance the target files use. After Commit returns success, the transaction is committed AT THE LOG LEVEL: a crash from this point forward will replay the intents on restart.
Caller calls Apply, which iterates intents in order and atomic-writes each one to its target path. If Apply returns an error mid-way, the WAL stays on disk and the next Scan-replay finishes the job. Apply is idempotent — replaying a fully- committed WAL just re-writes the same bytes to the same paths.
Caller calls Done, which removes the WAL file. After Done the transaction is complete and leaves no on-disk trace.
Replay. Store.NewStore scans walDir at startup. Any WAL file found represents a committed-but-not-finished transaction: Apply it, Done it. Uncommitted WALs leave nothing on disk because Append stages everything in memory.
What this does NOT provide. No undo log: there's no way to roll back an already-committed WAL after Apply has partially run. The model is forward-only "complete what we promised" recovery, not "restore the pre-transaction state". This matches the deferred- write usage in state.WithTransaction where fn never observes its own writes (it only reads from disk, mutates in memory, then stages the result via Append).
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type FileIntent ¶
type FileIntent struct {
Path string `json:"path"`
Data []byte `json:"data"`
Mode os.FileMode `json:"mode"`
}
FileIntent is a single staged target write. Persisted to disk in the WAL file as JSON; Data is base64-encoded by encoding/json's default []byte handling. Mode is the file-permission bits the target file should be created with.
type WAL ¶
type WAL struct {
// contains filtered or unexported fields
}
WAL is the in-memory transaction handle.
Append-only intents (modulo coalescing by path). State machine:
- BeginTx → opened, intents=[], file not yet on disk
- Append… → intents grow / coalesce
- Commit → intents serialized to <walDir>/<id>.json
- Apply → each intent written to its target
- Done → WAL file removed
func BeginTx ¶
BeginTx opens a fresh WAL with the given id. walDir is created if it doesn't exist. The write function will be called by Apply for each FileIntent — typically state.atomicWriteFile wired in by the state package.
No on-disk state is created here. Commit is what publishes the intent log to walDir; an uncommitted WAL leaves no trace.
func Scan ¶
Scan reads every WAL file in walDir and returns a WAL for each. Files are returned in id-sorted order so replay is deterministic across crash recoveries. Uncommitted WALs leave no file on disk, so anything Scan finds is committed and needs Apply + Done.
The write function is wired into every returned WAL. A missing walDir returns (nil, nil) — that's the "fresh data folder" case.
Files that fail to parse are skipped with their path included in the returned error chain (so a corrupted log doesn't block other pending transactions); the caller is expected to log and continue. Per-file errors don't abort the scan.
func (*WAL) Append ¶
func (w *WAL) Append(intent FileIntent)
Append stages a single target write. Coalesces by path: a second call with the same FileIntent.Path replaces the first (last-write- wins). The data + mode are copied/captured directly — callers don't need to defensively-clone before appending.
Returns no error: in-memory operation only.
func (*WAL) Apply ¶
Apply writes each staged intent to its target path via the configured write function, in insertion order. Returns the first error encountered, in which case the WAL file remains on disk so the next Scan can resume.
Idempotent: re-running Apply on a WAL whose targets have already been written produces the same on-disk state (atomicWriteFile is itself idempotent for identical bytes).
func (*WAL) Commit ¶
Commit serializes the staged intents to <walDir>/<id>.json via the configured write function. After Commit returns success the transaction is durable: a crash from this point on will replay the intents on restart.
A second Commit re-writes the file with the current intent set — callers should not Append after Commit, but if they do, a second Commit will produce a WAL file with the additional intents.
func (*WAL) Done ¶
Done removes the WAL file. Call after Apply succeeds. A subsequent Scan will not see this WAL.
Safe to call when the WAL file doesn't exist (returns nil) — that branch covers the "uncommitted WAL" cleanup case in the abort path of state.WithTransaction.
func (*WAL) Intents ¶
func (w *WAL) Intents() []FileIntent
Intents returns a snapshot of the currently-staged intents in insertion order (after coalescing). Provided for tests + diagnostic logging in the state-package replay path; production code should not mutate the returned slice.
func (*WAL) PendingBytes ¶
PendingBytes returns the currently-staged bytes for path (after coalescing) and ok=true; if no intent has been staged for path, returns (nil, false).
State-package storeTx loaders use this for read-your-own-writes inside a transaction body: a tx that saves pool-matches and then loads pool-matches must see the just-saved data, not the stale on-disk version (which won't update until Apply runs after fn returns). Without this peek the load would return the pre-save state and a tx body that reads-after-writes would silently see stale data.
Returns a fresh copy of the bytes so callers can mutate without disturbing the staged intent.
func (*WAL) WriteFn ¶
WriteFn returns a function that wraps Append in the WriteFn shape. The state package uses this to redirect savers — the savers think they're writing to disk, but the WAL captures the bytes for deferred commit.
The returned function always returns nil — Append cannot fail (it's in-memory).
Source Files