Documentation
¶
Overview ¶
Package writelane owns the daemon's single serialized lifecycle write lane: the ONE budget splitter (WriteLaneBudget + ExceedsWriteBatch) and the ONE goroutine that commits admitted records.
The lane exists because the contended resource is SQLite's single writer slot, not throughput: every hook process opened pasture.db and asked for a write lock at least twice per event, so a convoy of short-lived writers starved one another and each hook reported a fault it could not interpret. Putting the lane in front of the store serializes those writes by construction and bounds how long any one turn holds the writer, so the engine's and the CLI's own writers are never starved past their busy window.
ONE SPLITTER. Every place that groups records into a batch asks ExceedsWriteBatch; no site re-derives a count or byte bound. The sibling repository's ingest lane (internal/ingest/pipeline.go ExceedsWriteBudget) is the pattern this package adapts, and the reason for a single splitter is the same one stated there: two sites spelling out the same terms drift.
Index ¶
- Variables
- func ContextWithTurnScope(ctx context.Context, scope *TurnScope) context.Context
- func ExceedsWriteBatch(b WriteLaneBudget, pendingRecords int, pendingBytes, nextBytes int64) bool
- func OutcomeIsDurable(outcome provenance.BatchRecordOutcome) bool
- func Report(outcome provenance.BatchRecordResult) error
- type BatchApply
- type Lane
- type Record
- type Stats
- type TurnScope
- type WriteLaneBudget
Constants ¶
This section is empty.
Variables ¶
var ErrLaneClosed = errors.New("the lifecycle write lane is closed")
ErrLaneClosed is returned by Submit once the lane has been closed: no record was admitted, so the caller may honestly report that nothing was committed for it.
var ErrNotAdmitted = errors.New("the lifecycle write lane did not admit the record")
ErrNotAdmitted marks a Submit that staged NOTHING: the lane was closed, the caller's own context ended while the queue was at its bound, or the request itself was malformed. It is the evidence for a provably not-recorded claim, exactly as ErrLifecycleBeforeDurableWrite is on the direct path, and the served handler converts it into that sentinel. An error from the COMMIT is never wrapped in it.
Functions ¶
func ContextWithTurnScope ¶
ContextWithTurnScope returns a context carrying scope for one commit turn.
func ExceedsWriteBatch ¶
func ExceedsWriteBatch(b WriteLaneBudget, pendingRecords int, pendingBytes, nextBytes int64) bool
ExceedsWriteBatch reports whether the pending batch must be flushed BEFORE the next record joins it. It is the ONE splitter: the single home of the count and byte terms.
A record larger than the whole batch budget is still admitted, alone: the batch it would join is flushed first and the empty-batch term then admits it. Refusing it instead would lose an event that happened.
func OutcomeIsDurable ¶
func OutcomeIsDurable(outcome provenance.BatchRecordOutcome) bool
OutcomeIsDurable reports whether a batch outcome means the record is durably recorded after the call: BatchRecordCommitted (this batch's transaction wrote it) or BatchRecordShortCircuited (an earlier transaction did). Every other outcome means nothing this batch folded is durable for the record.
func Report ¶
func Report(outcome provenance.BatchRecordResult) error
Report maps one record's batch outcome to the error the lane reports for that record. It is the ONE home of the mapping, so the lane and the served path cannot drift about what an outcome means.
- nil means the record is durably recorded: committed by this batch, or short-circuited because an EARLIER transaction already committed it. A caller that gets nil may claim durability; a short-circuit is NOT a failure.
- Every other outcome (conflict, failed, not-recorded) returns a typed, actionable error. A record the batch rolled back is never reported as a commit, and the whole-batch failure is never reported as a partial commit.
Types ¶
type BatchApply ¶
type BatchApply func(ctx context.Context, inputs []provenance.OperationInput) (provenance.BatchResult, error)
BatchApply commits a flush's prepared operations in ONE transaction and returns one outcome per input, aligned by index. It is the lane's whole reason for phase 2: the per-record lane ran one transaction per record, and the pinned Provenance BatchJournal folds the batch through the same per-operation reducer step in a single BEGIN IMMEDIATE.
type Lane ¶
type Lane struct {
// contains filtered or unexported fields
}
Lane is the ONE serialized writer for lifecycle records: a bounded staging queue of admitted records, one goroutine that commits them in bounded batches, and per-record completion so a caller learns whether ITS record committed.
THE CALLER LEARNS NOTHING BEFORE THE COMMIT. Submit returns only after the record's Settle has run and reported its result, so there is no path on which a hook is told "recorded" for a record that has not committed. That is the property the whole daemon-fronted lane exists to preserve, and it is a property of Submit's shape rather than of any discipline at the call site.
func New ¶
func New(budget WriteLaneBudget, apply BatchApply) (*Lane, error)
New constructs the lane and starts its commit goroutine. The budget is validated here, so a lane that exists is a lane whose bounds hold; the batch applier is required, because a lane with no way to commit a flush would admit records it could never make durable.
func (*Lane) Budget ¶
func (l *Lane) Budget() WriteLaneBudget
Budget returns the validated budget this lane runs on.
func (*Lane) Close ¶
Close stops admission and drains what is already staged inside the ctx budget, then stops the commit goroutine. A drain that does not finish in time returns the ctx error; the records still staged are lost, which is why the daemon reports every hook that was still waiting as record-unknown rather than recorded.
func (*Lane) Submit ¶
Submit admits one record and blocks until it has committed or failed.
ADMISSION BLOCKS RATHER THAN REFUSING. When the queue is at its record or byte bound the caller waits for room inside its own context; the connection handler is what waits, so no further request is read from the socket and the pressure is passed back to the hook, which abandons at its own hook-invocation deadline exactly as it does against a slow direct write. No new refusal path is introduced.
A record larger than the whole batch byte budget is admitted when the queue is empty and committed ALONE; refusing it would lose an event that happened.
type Record ¶
type Record interface {
Prepare(ctx context.Context) (provenance.OperationInput, error)
Settle(ctx context.Context, outcome provenance.BatchRecordResult) error
}
Record is one admitted record's durable work, expressed in two phases so the lane can commit a whole flush in ONE provenance transaction:
- Prepare runs everything that must happen BEFORE the shared write transaction — the payload blob write, identity resolution, and canonicalization — and returns the record's canonical journal operation. A Prepare error is this record's own: it contributes no operation to the batch and the lane reports that error to its caller. Anything durable it left behind is a reclaimable payload blob, never a journal row.
- Settle reports this record's own outcome to its caller and runs any work that must follow the shared commit. It is called exactly once for every record whose Prepare succeeded, with that record's own BatchRecordResult, whatever the batch did.
The context both phases receive is the LANE's, not the submitter's: a record a hook abandoned while it waited in the queue is still committed (evidence over silence), so neither phase may be cancelled by the hook's own deadline.
type Stats ¶
type Stats struct {
Records int64
Batches int64
MaxBatchRecords int64
MaxHold time.Duration
QueueDepth int64
HoldExceeded int64
}
Stats is a read-only snapshot of what the lane has done. It exists so an operator, a test and the startup log line can read the same numbers without reaching into the lane's state.
type TurnScope ¶
type TurnScope struct {
// contains filtered or unexported fields
}
TurnScope carries state shared by every record prepared in one commit turn. A turn is one trip through the lane's prepare-every-record loop: the lane creates a scope, prepares every record of the flush under it, applies the batch, and drops it. Records use it to share preparation reads that are constant within a turn — content-addressed owner facts resolved from monotonic journal truth, where every record of the turn would otherwise repeat the identical indexed lookup and resolution against the same rows.
The scope never outlives the batch apply, so no invalidation or cross-turn staleness exists: the next turn re-resolves everything. A failed share is never stored, so one record's pre-transaction fault cannot poison the turn. Records prepared outside a lane (the direct path) see no scope and prepare standalone, exactly as before.
func TurnScopeFrom ¶
TurnScopeFrom returns the commit-turn share carried by ctx, or nil when the record is prepared outside a lane turn.
type WriteLaneBudget ¶
type WriteLaneBudget struct {
// QueueRecords bounds the number of admitted-but-uncommitted records. When
// the queue is at this bound a new request BLOCKS in the lane until a
// commit frees room; the connection handler is what blocks, so a further
// request is not read from the socket (natural backpressure). It is a
// bound on staged memory and an admission brake, never a refusal path: an
// invocation waiting here is abandoned by its own hook-invocation deadline
// exactly as a slow direct write is today.
QueueRecords int
// QueueBytes bounds the staged payload bytes across the whole queue, the
// byte half of the same brake.
QueueBytes int64
// BatchRecords bounds ONE lane turn's record count. The turn is what holds
// the store writer, so this count times the per-record commit cost must
// stay inside HoldTarget.
BatchRecords int
// BatchBytes bounds ONE lane turn's staged payload bytes on the same terms.
BatchBytes int64
// FlushInterval is the IDLE FALLBACK only: the longest a non-empty partial
// batch waits for a companion record before it is committed anyway. It must
// stay well below the hook-invocation tier, because every record in the
// batch is still waiting for its acknowledgement.
FlushInterval time.Duration
// HoldTarget bounds how long ONE lane turn may hold the SQLite writer, so
// the engine's own writes and every CLI writer keep their measured wait.
// It is derived below the other writers' busy tier.
HoldTarget time.Duration
}
WriteLaneBudget is every bound the lane enforces, in one value. It is constructor-validated: the daemon resolves it once from the timeout profile (BudgetFromProfile) and an operator may override named knobs, but the value the lane runs on always passes Validate, so an inverted combination is refused with a named error rather than clamped.
func BudgetFromProfile ¶
func BudgetFromProfile(profile timeouts.Profile) (WriteLaneBudget, error)
BudgetFromProfile derives the lane budget from one validated timeout profile: the profile is the single home of the tiers, so the lane joins it rather than carrying a second, independent set of durations.
The derivation, and why each term is what it is:
- HoldTarget is four fifths of the profile's SQLite busy tier, capped by maxDerivedHoldTarget. The busy tier is the window every OTHER writer waits for the lock, so a lane turn that stays inside four fifths of it leaves scheduling headroom instead of becoming the holder that starves them.
- BatchRecords is HoldTarget divided by assumedPerRecordCommitCost, floored at one, so the turn's record count is sized against the hold bound rather than picked.
- BatchBytes is 8 MiB: eight times the 1 MiB ingress payload cap, so a queue of concurrent maximum-size payloads cannot itself become the memory bound (the design's stated start).
- QueueRecords/QueueBytes are the staging bounds: 256 records and four times BatchBytes, so the queue is deeper than one turn but still bounded.
- FlushInterval is 25 ms, the design's stated start, and never more than a quarter of HoldTarget. It is the ONLY wait a lone record can add to the hook path, so it is bounded hard: it stays well below the hook-invocation tier, and it is below a hold bound that is itself below the busy tier.
func (WriteLaneBudget) Validate ¶
func (b WriteLaneBudget) Validate() error
Validate refuses an unusable budget with an error naming the knob and the relationship that failed, so an operator who set one via the environment is told which one and what it collides with. It never clamps.
Source Files