Documentation
¶
Overview ¶
Package outbox holds the ordered event streams that peers exchange and the hybrid logical clock that orders their events: the port that stores this instance's outbox, each peer's acknowledged position in it, the receiver's applied positions and the blob fetches that failed while applying, with the values it takes and returns.
Role ¶
platform layer: the ordered event streams of decision 0021 and the clock that orders them. The target layout puts the outbox in the platform, so that a module appends its events in its own transaction and a module's api package may import it. OutboxStore takes and returns the clock and the values, so they are declared beside it and the package imports no module. The platform's store adapters (platform/store/sql and platform/store/inmem) implement OutboxStore, and the metadata store's Outbox returns theirs; no service appends yet. The package imports only the standard library.
Inputs ¶
- Entries appended inside the transaction of the change they record: origin, stream, clock time, event type, payload and, for a content event, the blob's SHA-256 and size.
- Peers' acknowledged positions, the receiver's applied positions and the blob fetches that failed while applying.
- For the clock, a physical time (Next and Merge) or the text form (ParseHLC).
Outputs ¶
- OutboxStore: Append, Read, LastSeq, Streams and Prune for the entries; PutCursor, Cursor, Cursors and DeleteCursors for peers' positions; Applied and SetApplied for the receiver's; and RecordFailedFetch, FailedFetches and ClearFailedFetch for failed blob fetches.
- HLC, with Compare, Next, Merge and String, and ParseHLC with its sentinel ErrInvalidHLC.
- OutboxEntry; PeerCursor, with Behind; AppliedMarker, with Expect; and FailedBlobFetch.
Invariants ¶
- Seq is gap-free per origin and stream, and concurrent appends to one stream are serialised, never given the same Seq. An append rolled back with its transaction gives its number back, and pruning never resets the numbering.
- The HLC text form is fixed-width hex, so it sorts as Compare does and can be stored in an indexed column; ParseHLC reads back every value that String writes.
- The clock never runs behind anything it has seen: Merge takes the largest wall time, and the logical counter orders events that share one.
- SetApplied runs in the transaction that applies the entry, which makes applying idempotent: a retried batch starts at the next expected sequence, AppliedMarker.Expect.
References ¶
- Decision 0021: docs/research/decisions/0021-peer-trust-and-sync-protocol.md ("Ordered outbox": streams, sequence numbers, acknowledgements, applied positions and failed fetches; "Conflict rule": the clock time and the origin), with Kulkarni et al., "Logical Physical Clocks", 2014.
- Decision 0044: docs/research/decisions/0044-modular-monolith-migration.md ("Dependency rules", rule 5: a module's api may import platform/outbox).
- Research 15: docs/research/15-data-model.md, lines 703 to 706 (the outbox, peer cursor, applied marker and failed fetch tables) and line 763 (the clock's two columns).
- Target layout: docs/research/architecture-options/5-modular-monolith.md, lines 140 and 141 (platform/outbox, from the store's OutboxStore, generalised in P11), lines 720 to 724 (P11's Event, Append and per-consumer cursors) and lines 341 and 342 (the federation module's core/port, where the target layout keeps the peer types).
- Plan: docs/implementation/plan.md#p2-stores, #p13-peer-trust-and-federated-repositories and #p9d-modular-monolith-migration, epic PLT (https://github.com/weaveplatform/artifactweave/issues/938).
- https://github.com/weaveplatform/artifactweave/issues/989
Status ¶
Implemented P2: the clock, the outbox values and OutboxStore, with the metadata store's SQL and in-memory stores (pull request 3). Moved here in P9d. Roadmap: P11 generalises the outbox to Append(ctx, Event) for every consumer, and P13 delivers its streams to peers.
Testing ¶
- outbox_test.go: TestUnitHLC (Next ahead of and behind physical time, the four Merge cases, Compare and the text form agree on order, ParseHLC reads back what String writes and refuses five malformed values with ErrInvalidHLC) and TestUnitCursorsAndMarkers (Behind and Expect).
The platform's outbox contract (the Outbox, OutboxConcurrent, PeerCursors, AppliedMarkers and FailedFetches subtests of platform/store/storetest: sequences per origin and stream, reads after a sequence and up to a limit, pruning that keeps the numbering, a rolled-back append, concurrent appends, cursor replacement and deletion, applied positions per origin, failed fetch counting and clearing, and cancelled contexts) runs on the platform's in-memory and SQLite stores and on the metadata store's in-memory, SQLite and PostgreSQL stores. The federation model's TestUnitStamp orders stamps through Compare.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrInvalidHLC = errors.New("invalid hybrid logical clock value")
ErrInvalidHLC reports a hybrid logical clock value that does not parse.
Functions ¶
This section is empty.
Types ¶
type AppliedMarker ¶
AppliedMarker is the receiver's record of the last sequence applied from one origin's stream, updated in the apply transaction so a retried batch is idempotent.
func (AppliedMarker) Expect ¶
func (m AppliedMarker) Expect() int64
Expect is the sequence the next batch must start at.
type FailedBlobFetch ¶
type FailedBlobFetch struct {
Stream string
SHA256 string
SHA1 string
RepoKey string
FirstFailed time.Time
Failures int
LastError string
}
FailedBlobFetch records content a receiver could not fetch from a peer; it backs the federation failed-binary APIs and is retried with back-off.
type HLC ¶
HLC is a hybrid logical clock timestamp (Kulkarni et al., 2014): physical milliseconds plus a logical counter. It orders events across instances whose wall clocks disagree, which last-writer-wins needs (decision 0021).
func ParseHLC ¶
ParseHLC reads the String form. Text that is not 16 and 8 hex digits around a dot gives an error that wraps ErrInvalidHLC.
func (HLC) Merge ¶
Merge is the timestamp of receiving remote at physical time now, so the local clock never runs behind anything it has seen.
type OutboxEntry ¶
type OutboxEntry struct {
Origin string
Stream string
Seq int64
Time HLC
EventType string
Payload []byte
BlobSHA256 string
BlobSize int64
Created time.Time
}
OutboxEntry is one change queued for peers, written in the same transaction as the change it records. (Origin, Stream, Seq) is unique and Seq is gap-free per stream, which lets a receiver detect a missing batch. BlobSHA256 is set for content events; receivers pull the blob.
type OutboxStore ¶
type OutboxStore interface {
// Append stores e as the next entry of (e.Origin, e.Stream) and returns
// it with Seq assigned, one more than the stream's last. Call it inside
// the transaction that makes the change it records; concurrent appends
// to one stream are serialised, never given the same Seq.
Append(ctx context.Context, e OutboxEntry) (OutboxEntry, error)
// Read returns up to limit entries of a stream with Seq above after, in
// Seq order.
Read(
ctx context.Context,
origin, stream string,
after int64,
limit int,
) ([]OutboxEntry, error)
// LastSeq returns the highest Seq of a stream, 0 when it has none.
LastSeq(ctx context.Context, origin, stream string) (int64, error)
// Streams returns the streams origin has entries or a sequence in,
// ordered.
Streams(ctx context.Context, origin string) ([]string, error)
// Prune removes entries of a stream with Seq at most upTo and returns how
// many. The stream's sequence is not reset.
Prune(ctx context.Context, origin, stream string, upTo int64) (int64, error)
// PutCursor creates or replaces a peer's position in a stream.
PutCursor(ctx context.Context, c PeerCursor) error
// Cursor returns a peer's position in a stream; kernel.ErrNotFound when
// the peer has none there.
Cursor(ctx context.Context, peer, stream string) (PeerCursor, error)
// Cursors returns a peer's positions ("" for every peer) ordered by
// (peer, stream).
Cursors(ctx context.Context, peer string) ([]PeerCursor, error)
// DeleteCursors removes every position of a peer; a peer with none is
// not an error.
DeleteCursors(ctx context.Context, peer string) error
// Applied returns the receiver's position in a peer's stream; a zero
// LastSeq and no error when nothing was applied yet.
Applied(ctx context.Context, origin, stream string) (AppliedMarker, error)
// SetApplied records the position, in the transaction that applies the
// entry, which is what makes applying idempotent.
SetApplied(ctx context.Context, m AppliedMarker) error
// RecordFailedFetch stores a failed blob fetch with Failures 1, or
// increments Failures and replaces LastError of the existing one, keeping
// its other fields, and returns the record as stored. Failures counts
// the calls; the value in f is ignored.
RecordFailedFetch(ctx context.Context, f FailedBlobFetch) (FailedBlobFetch, error)
// FailedFetches returns a stream's failed fetches ("" for every stream)
// ordered by (stream, sha256).
FailedFetches(ctx context.Context, stream string) ([]FailedBlobFetch, error)
// ClearFailedFetch removes one after a successful retry;
// kernel.ErrNotFound when there is none.
ClearFailedFetch(ctx context.Context, stream, sha256 string) error
}
OutboxStore persists the ordered event streams of decision 0021: this instance's outbox, each peer's acknowledged position in it, the receiver's applied positions, and blob fetches that failed while applying.
type PeerCursor ¶
type PeerCursor struct {
PeerInstanceID string
Stream string
AckedSeq int64
LastSent time.Time
LastError string
}
PeerCursor is the highest sequence a peer has acknowledged on a stream.
func (PeerCursor) Behind ¶
func (c PeerCursor) Behind(head int64) int64
Behind is how many entries the peer still has to apply, given the stream's head sequence.