README
¶
directory: ledger-sqlite purpose: The durable ledger event store — a SQLite-backed app.Store, and the adapter tests that prove a fact survives process exit. owner: "@FabioCaffarello" allowed:
- An app.Store implementation backed by SQLite through modernc.org/sqlite
- Schema, migrations, and the index over the total order ADR-0009 defines
- The driver dependency and what it transitively requires
- The shared store conformance suite, run against this implementation
- Adapter tests for durability, crash-safety and index rebuild forbidden:
- Business rules or financial calculations — those live in libs/ledger/domain
- Storing derived state: a position, a balance, or a resolution result
- An index that cannot be rebuilt from the facts alone
- Consulting anything outside the ledger to answer a read
- cgo, or any dependency that requires it
- Assigning knowledge time, or accepting one from a caller
libs/ledger-sqlite
The store that makes the ledger outlive the process. Until this module exists, everything FDOS asserts is true only until the program ends.
A separate module, not a package inside libs/ledger, and the reason is
dependency resolution rather than taste (ADR-0013). Go resolves dependencies per
module: a driver inside libs/ledger would land in the go.sum of every
consumer that imports libs/ledger/domain, including consumers that never touch
storage.
What it implements
app.Store as ADR-0034 defines it — Load, and Append carrying an
Expectation. There is no whole-stream write, because a stream is only ever
extended and the operation that wrote one entire is what lost facts.
Three obligations this module inherits, each of which is a test below rather than a promise:
| Obligation | From |
|---|---|
The store assigns the sequence, so two writers cannot compute one Ref |
ADR-0034 |
| A stale read is refused, not applied | ADR-0034 |
| Knowledge time is monotonic per stream | ADR-0009 |
The index is derived state
The store indexes (stream, effective_from, knowledge_time, sequence) so an
as-of read is a range scan rather than a full scan.
Constitution §1 governs it: the index is not a second source of truth. It must be rebuildable from the facts alone, and a test proves rebuilding gives identical answers. An index that cannot be rebuilt is a second copy of the ledger that will eventually disagree with the first, with nothing to say which is right.
The driver, and the risk that came with it
modernc.org/sqlite, chosen on a measured audit and recorded in ADR-0035. That
ADR also records what the audit could not fix: the driver cannot be
independently re-derived from upstream SQLite. go.sum pins what was received,
not that what was received is SQLite. Accepted, unmitigated, and stated here so
that nobody reading this module assumes otherwise.
CGO_ENABLED=0 is pinned in the Makefile. A cgo dependency would make the build
depend on the host C toolchain and put make repro-check at the mercy of a
system compiler.
Test plan
Two layers. The first is not specific to SQLite and should not live here.
The shared store conformance suite
ADR-0034 anticipated a second engine — "one conformance suite, two
implementations" — so the suite that defines what implementing app.Store
means cannot live inside either adapter, or Postgres would depend on SQLite to
be tested.
Home: libs/ledger/storetest — confirmed and written. An exported package in
the context module, needing domain and app and nothing else, the same shape
as the standard library's testing/fstest. It is public rather than internal for
the same reason app.Store is: the port is public API, so what implementing it
means is public API too, and an out-of-tree adapter that cannot run the suite is
one nobody can hold to the contract.
The suite takes a factory and runs every case against it — the in-memory store
included, so the two implementations are held to one definition. The ten cases
below all exist and pass against adapters/memory:
| # | Case | Proves |
|---|---|---|
| 1 | Two appends receive sequences 1 and 2, never equal | the measured defect stays fixed |
| 2 | Load of an unknown stream is ErrStreamNotFound |
"we know nothing" ≠ "we hold nothing" |
| 3 | The first append creates the stream | a stream is the facts in it |
| 4 | AtLength(n) succeeds when the stream is at n |
the precondition does not fire spuriously |
| 5 | AtLength(n) is ErrStaleRead otherwise, and the fact is not appended |
a refusal refuses |
| 6 | Any() succeeds regardless of what landed since |
admission is never blocked by another writer |
| 7 | An equal or earlier knowledge time is ErrNonMonotonicKnowledge |
ADR-0009's axis is ordered |
| 8 | Facts reload with identical refs, envelopes and payloads | nothing is lost in the round trip |
| 9 | An as-of read matches the same projection computed in memory | the index does not change answers |
| 10 | Concurrent appends under -race: N goroutines, N facts, refs 1…N |
the serialisation point serialises |
Case 5's second clause is the one worth writing carefully. A store that returns the error and appends anyway passes a naive version of this test, and that is precisely the failure the M10 gate measured in a different disguise.
Adapter tests, which belong here
All five exist and pass. TestTheSchemaRefusesADuplicateSequence was
mutation-checked — removing the primary key turns it red — so the constraint is
demonstrably doing the work rather than the Go code doing it alone.
| # | Case | Proves |
|---|---|---|
| 11 | Append, close the database, reopen it, read the facts back | the point of the module — and the case the in-memory store cannot satisfy |
| 12 | Drop the index, rebuild from the facts, answers are identical | the index is derived, not a second truth |
| 13 | Replay a stream into a fresh database: same answers and same derivation content addresses | ADR-0034's reproducibility clause |
| 14 | A transaction interrupted mid-append leaves no partial fact | crash-safety — the gap ADR-0035 named as unaudited |
| 15 | The schema rejects a duplicate (stream, sequence) |
the sequence is unique at the storage layer, not only in Go |
Case 14 is the one ADR-0035 flagged as the thing it would want tested before trusting this with a ledger, and it is the hardest to write portably. At minimum it must assert the durability settings the driver is opened with, rather than assuming a default.
Case 13 is the storage analogue of make repro-check. It stays a test rather
than a make target because it needs a database, and repro-check deliberately
needs only a compiler.
Not proven by any of these
That the store reads the ledger and nothing else. A store that dialled out would pass every case above. ADR-0034 records it at rung 6, and no mechanism is proposed here — saying so is better than implying coverage that does not exist.
Release chain
Closed. This module pins libs/ledger v0.4.0 and libs/ledger-wire v0.4.0,
both released, which is the two-to-three coordinated releases ADR-0004 predicted
for a change spanning modules:
libs/kernel v0.7.0 the canonicalisation ruleset
libs/ledger v0.4.0 the port change
libs/ledger-wire v0.4.0 bumped to see it
libs/ledger-sqlite this module
make verify runs each module with GOWORK=off, which is what makes a missing
link in that chain a build failure rather than something go.work hides until
CI.
Documentation
¶
Overview ¶
Package sqlite is the durable ledger event store (ADR-0034, ADR-0035).
An adapter, so the constructs the domain forbids are legitimate here: I/O, `context`, mutable state, a driver. That asymmetry is the architecture working — a purity rule that fired in this package would be a rule nobody kept.
A separate module rather than a package inside `libs/ledger`, because Go resolves dependencies per module: the driver would otherwise land in the `go.sum` of every consumer that imports `libs/ledger/domain`, including consumers that never touch storage (ADR-0013).
Index ¶
- Variables
- type Store
- func (s *Store) Append(ctx context.Context, name string, expect app.Expectation, ...) (ref domain.Ref, err error)
- func (s *Store) Close() error
- func (s *Store) Load(ctx context.Context, name string) (domain.Stream, error)
- func (s *Store) Serialise(ctx context.Context, name string, fn func(context.Context, app.Store) error) (err error)
Constants ¶
This section is empty.
Variables ¶
var ErrEncodingVersion = errors.New("sqlite: database uses an older temporal encoding")
ErrEncodingVersion is returned when a database was written under an encoding this build cannot order correctly.
Refusing rather than migrating is the decision, not a limitation (ADR-0040): a store whose facts this build cannot order is one it must not answer an as-of query from, and an as-of answer is the only thing the ledger is for.
var ErrGap = errors.New("sqlite: stream has a sequence gap")
ErrGap is returned when a stream's stored sequences are not 1..N contiguous.
An append-only stream whose sequence is assigned by the store (ADR-0034) has no legitimate source of gaps: Append never reserves a number it might not use, which is what makes gaplessness free here and expensive in a database sequence. So a gap is not a condition to tolerate — it is evidence that rows were deleted or the file was altered out of band, and the only safe response is to refuse the stream and say which sequence is missing.
Tolerating one is worse than it sounds. Replay assigns refs by position, so a single missing row silently re-points every later ref at different content: a FactCorrected naming s#3 would correct whatever landed at position 3 instead.
Functions ¶
This section is empty.
Types ¶
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
Store is a durable app.Store backed by SQLite.
func Open ¶
Open opens or creates the database at dsn and applies the schema.
`dsn` is a file path. There is deliberately no in-memory convenience constructor: `libs/ledger/adapters/memory` is the in-memory store, and a second one here would be a second implementation of the same thing whose divergence nobody would notice.
func (*Store) Append ¶
func (s *Store) Append( ctx context.Context, name string, expect app.Expectation, envelope domain.Envelope, kind domain.Kind, payload domain.Payload, ) (ref domain.Ref, err error)
Append records one fact and returns the reference the store assigned.
Everything happens inside one immediate transaction: reading the current length, checking the caller's expectation, checking knowledge-time monotonicity, and the insert. That is the whole point of moving the append here — the sequence is assigned where writes serialise, so two writers cannot compute the same Ref (ADR-0034).
The transaction is IMMEDIATE, set on the connection in Open. A deferred transaction — the default — takes its write lock at the first write, which would leave the read that decides the sequence outside the lock: the same time-of-check gap this design exists to close, reintroduced one layer down.
func (*Store) Load ¶
Load rebuilds the stream from its facts, or returns app.ErrStreamNotFound.
The stream is replayed through `domain.Stream.Append` rather than reconstructed field by field, so a decoded fact goes through the same constructor a new one does. A store that assembled a Stream directly could produce one the domain would refuse to build.
Replay assigns each ref from the stream's current length, so it reproduces the stored sequences exactly when they are 1..N contiguous — and silently renumbers them when they are not. The stored sequence is therefore read and compared rather than discarded, which is what makes the replay's assumption checked instead of assumed. The sequence is stored twice, in the column and inside the encoded fact's own ref, and both are compared: they cannot disagree unless the row was written by something other than Append.
func (*Store) Serialise ¶ added in v0.3.0
func (s *Store) Serialise( ctx context.Context, name string, fn func(context.Context, app.Store) error, ) (err error)
Serialise runs fn holding this database's write lock (ADR-0041).
The lock is the transaction ¶
`BEGIN IMMEDIATE` takes SQLite's write lock at the statement rather than at the first write, and holds it until commit or rollback. So a region is a transaction held open across fn, and the caller's clock read happens inside it — which is what closes the window ADR-0036 closed for one process and could not close for two.
`name` is ignored, and that is the honest answer rather than a shortcut ¶
SQLite has one writer per *database*, not per stream. There is no per-name lock to take, so a region over `acct-1` excludes a writer to `acct-2` as well. ADR-0041 records this as the cost that ADR-0042's per-stream advisory locks exist to pay down. The parameter stays in the signature because the port has it and a second engine uses it; pretending it were honoured here would be worse than ignoring it visibly.
Why this serialises processes, which a mutex could not ¶
The lock is the file's, so it is held against every process that has the database open, not only every goroutine in this one. Measured before this existed: 128 concurrent admissions to one stream admitted 128 from a single process and 106 from sixteen, and what kept the number as high as it was is this same lock being taken by Append — just too late to cover the clock read.
SetMaxOpenConns(1) makes misuse a deadlock rather than a stale read ¶
fn must use the Store it is given. Reaching past it to the outer Store would ask for a second connection, and there is only one — so the mistake hangs instead of silently reading outside the region. Hence [regional], whose methods run on this transaction and whose Serialise refuses.
Source Files