Documentation
¶
Overview ¶
Package engine is the durable-execution adapter for the pasture epoch lifecycle. It owns the shared modernc SQLite handle, registers and drives the pure-Go EpochStateMachine over durable steps, persists an EpochState projection each transition, and records forensic rows exactly once.
The state machine itself lives in pkg/protocol and has no substrate dependency; this package is the impure adapter around it.
Package engine — queue.go defines the DBOS WorkflowQueue for concurrency-limited slice and review sub-workflow dispatch.
Sub-workflows that drive individual implementation slices and review cycles are dispatched through a shared DBOS WorkflowQueue with a configurable per-executor concurrency limit K. Bounded concurrency is the primary control point for the single-writer WAL bottleneck: 30+ unbounded sub-workflows would thrash the shared SQLite connection, so K is tuned to the write throughput of the pasture.db file.
Queue lifecycle:
- newSliceQueue must be called BEFORE dbos.Launch (NewWorkflowQueue panics after Launch). Engine.New calls it as part of construction.
- Sub-workflows are enqueued via Engine.EnqueueSlice / Engine.EnqueueReview, each of which calls dbos.RunWorkflow with dbos.WithQueue(SliceQueueName).
- DBOS dequeues and starts sub-workflows up to K at a time; excess are held in the queues table until a running sub-workflow completes and frees a slot. Multi-process crash recovery and automatic retry across restarts are tracked as a separate follow-up item.
Index ¶
- Constants
- func ReadProjection(db *sql.DB, epochId string) (*protocol.EpochState, error)
- func ResolveSliceConcurrency(flagVal int) (int, error)
- func WriteProjection(ctx context.Context, db *sql.DB, state *protocol.EpochState, nowUnixNano int64) error
- type ActivitySink
- type AdvanceStep
- type Config
- type ControlInput
- type Engine
- func (e *Engine) ControlQueue() dbos.WorkflowQueue
- func (e *Engine) DB() *sql.DB
- func (e *Engine) DBOS() dbos.DBOSContext
- func (e *Engine) EnqueueReview(in ReviewInput) (dbos.WorkflowHandle[ReviewResult], error)
- func (e *Engine) EnqueueSlice(in SliceInput) (dbos.WorkflowHandle[SliceResult], error)
- func (e *Engine) EpochControlWorkflow(ctx dbos.DBOSContext, in ControlInput) (protocol.EpochState, error)
- func (e *Engine) EpochWorkflow(ctx dbos.DBOSContext, in EpochInput) (protocol.EpochState, error)
- func (e *Engine) Launch() error
- func (e *Engine) ReadProjection(epochId string) (*protocol.EpochState, error)
- func (e *Engine) ReviewSubWorkflow(ctx dbos.DBOSContext, in ReviewInput) (ReviewResult, error)
- func (e *Engine) Shutdown(timeout time.Duration)
- func (e *Engine) SliceConcurrency() int
- func (e *Engine) SliceQueue() dbos.WorkflowQueue
- func (e *Engine) SliceSubWorkflow(ctx dbos.DBOSContext, in SliceInput) (SliceResult, error)
- func (e *Engine) Trail() audit.Trail
- type EpochInput
- type ReviewInput
- type ReviewResult
- type SliceInput
- type SliceResult
Constants ¶
const ( // EngineAgentName is the stable software-agent name the engine attributes // its phase-transition activities to. It is exported so callers that record // audit events attributed to the engine (e.g. the terminate handler) can // use the same stable name without duplicating the string. It is deliberately // NOT one of the well-known automaton agents, so adding it does not change // the well-known agent count or its registration tests. EngineAgentName = "pasture/automaton/epoch-engine" // ActivityKindPhaseTransition is the discriminator the engine passes to // protocol.DedupKey for a phase-transition activity. It is DELIBERATELY // distinct from the audit tier's event_type ("PhaseTransition"): an activity // is a different PROV-O entity (a unit of work owned by the engine's software // agent) than a system audit event, so they occupy independent id-spaces. // Both tiers use the SAME derivation mechanism (this one DedupKey encoder), // but the distinct kind makes the activity id differ from the audit dedup_key // for the same transition — id-equality across tiers would be a fragile // implicit join. Exactly-once still holds: each table is keyed on its own // kind. Exported so a cross-tier replay test derives the identical activity // id from the same const. ActivityKindPhaseTransition = "activity:phase-transition" )
const ControlQueueName = "pasture-control-queue"
ControlQueueName is the canonical DBOS queue name for epoch control workflows. CLI lifecycle commands enqueue onto this queue through a DBOS client; pastured hosts the registered workflow and dequeues it.
const DefaultAppName = "pasture"
DefaultAppName is the pinned DBOS application name.
const DefaultApplicationVersion = "1"
DefaultApplicationVersion is the pinned DBOS recovery COHORT marker.
Role: DBOS filters crash-recovery by (ExecutorID, ApplicationVersion) and otherwise defaults the version to a per-build binary hash. Pinning a stable, build-independent value here is precisely what lets a REBUILT binary still recover epochs that an earlier build left in flight — without it, every rebuild would start a new cohort and silently orphan the previous build's in-flight epochs.
Bump criteria: increment this ONLY on an incompatible change to the EpochWorkflow / EpochControlWorkflow shape that makes already-in-flight workflows non-resumable. A routine rebuild MUST NOT bump it (that would abandon in-flight epochs); after a deliberate bump, old in-flight workflows are resumed manually rather than auto-recovered.
Cross-binary invariant: every pasture process that opens the engine — the local CLI (epoch start) and the daemon — MUST pin this same value (together with DefaultExecutorID and DefaultAppName). If the CLI and the daemon use different values, each silently fails to recover the other's in-flight epochs.
const DefaultExecutorID = "pasture"
DefaultExecutorID is the pinned DBOS executor id. DBOS filters crash-recovery by ExecutorID + ApplicationVersion; pinning the executor id keeps recovery attributable to "pasture" across restarts rather than a per-process default.
const DefaultSliceQueueConcurrency = 8
DefaultSliceQueueConcurrency is the default per-executor concurrency limit for the slice queue. The value balances SQLite WAL write throughput against parallel-agent utilisation:
- SQLite WAL serialises writers: only one write transaction commits at a time. Under 30+ unbounded writers the commit queue grows faster than it drains and busy_timeout errors accumulate.
- K=8 allows up to 8 sub-workflows to hold a write transaction concurrently. This is a conservative default chosen to stay within the WAL commit throughput of a typical single-disk host; the right value depends on your storage — lower K (e.g. 4) on HDD or network-attached storage, higher K (e.g. 16) on NVMe-backed hosts with idle I/O headroom.
- A benchmark validating a specific K for your setup is the authoritative guide; measure with your actual storage before changing this default.
Override via --slice-concurrency / PASTURE_SLICE_CONCURRENCY or the engine Config.SliceConcurrency field.
const EpochControlWorkflowName = "pasture.epoch_control.v1"
EpochControlWorkflowName is the stable DBOS workflow name used by clients that enqueue epoch control work without linking to the engine implementation.
const SliceConcurrencyEnv = "PASTURE_SLICE_CONCURRENCY"
SliceConcurrencyEnv is the environment variable that overrides the per-executor concurrency limit for the slice queue. When set, its integer value is used instead of DefaultSliceQueueConcurrency.
const SliceQueueName = "pasture-slice-queue"
SliceQueueName is the canonical DBOS queue name for slice and review sub-workflow dispatch. A single shared queue keeps the concurrency budget unified across both sub-workflow kinds — slices and reviews compete for the same K slots, so the total in-flight count across both is bounded by K.
Variables ¶
This section is empty.
Functions ¶
func ReadProjection ¶
ReadProjection returns the projected EpochState for epochId. It returns (nil, nil) when no projection exists yet (the epoch has not advanced), so callers can distinguish "unknown epoch" from a read error.
Callers that need available transitions recompute them from the returned state via protocol.NewEpochStateMachineFromState(...).AvailableTransitions(); the projection stores raw state, not derived views.
func ResolveSliceConcurrency ¶
ResolveSliceConcurrency resolves the effective per-executor concurrency limit K from the three override sources, highest-priority first:
- flagVal > 0: the caller-supplied CLI flag value (--slice-concurrency).
- $PASTURE_SLICE_CONCURRENCY env var (non-empty, parses as a positive int).
- DefaultSliceQueueConcurrency (8).
If the env var is set but not a valid positive integer, the function returns an actionable validation error (the caller should surface it and exit 1). A zero or negative flagVal is treated as "not set" (fall through to env/default).
This function is the single resolution rule shared by pastured and any other process that constructs an Engine; call it once at startup and pass the result to engine.Config.SliceConcurrency.
func WriteProjection ¶
func WriteProjection(ctx context.Context, db *sql.DB, state *protocol.EpochState, nowUnixNano int64) error
WriteProjection upserts the serialized EpochState for state.EpochId. It is idempotent (last-write-wins) — safe to re-run when a durable step replays, because the projection is a cache of the authoritative FSM state, not an append-only log. nowUnixNano timestamps the row's freshness.
Types ¶
type ActivitySink ¶
type ActivitySink interface {
// RegisterSoftwareAgent find-or-creates is the caller's concern; the engine
// only registers its own stable agent once if absent.
RegisterSoftwareAgent(namespace, name, version, source string) (provenance.SoftwareAgent, error)
// StartActivityWithID records an activity under a caller-supplied id with
// ON CONFLICT(id) DO NOTHING, so a replayed emission collapses to one row.
StartActivityWithID(id provenance.ActivityID, agentID provenance.AgentID, phase provenance.Phase, stage provenance.Stage, notes string) (provenance.Activity, error)
}
ActivitySink is the narrow provenance surface the engine needs to record activities idempotently. protocol.TaskTracker satisfies it (via the embedded provenance.Tracker), as does provenance.Tracker directly.
type AdvanceStep ¶
type AdvanceStep struct {
// ToPhase is the target phase for this transition.
ToPhase protocol.PhaseId
// TriggeredBy identifies who/what drove the transition (recorded as the
// forensic row's role; defaults to the epoch role when empty).
TriggeredBy string
// ConditionMet describes the satisfied transition condition.
ConditionMet string
// Votes are recorded (in order) before the advance, to satisfy the
// consensus gate at p4/p10.
Votes []protocol.ReviewVoteSignal
// BlockerDelta adjusts the blocker count before the advance: a positive
// value records that many new blockers, a negative value resolves that
// many. Used to exercise the p10 blocker gate.
BlockerDelta int
}
AdvanceStep is one scripted transition in an epoch plan. It carries the votes and blocker delta to apply (deterministically, before the advance) so a single plan can exercise the consensus and blocker gates without an external signal source — the signal-driven control surface is a later slice.
type Config ¶
type Config struct {
// DBPath is the unified pasture.db path. Required.
DBPath string
// ApplicationVersion is the pinned DBOS application version. REQUIRED:
// DBOS recovery is filtered by it, and it defaults to a binary hash, so a
// rebuilt binary would skip recovery of an in-flight epoch unless this is
// pinned to a stable value across builds. New rejects an empty value.
ApplicationVersion string
// ExecutorID overrides DefaultExecutorID. Pinned across restarts.
ExecutorID string
// AppName overrides DefaultAppName.
AppName string
// Trail is the forensic sink for one audit row per transition. When nil,
// New opens an owned SQLite trail on DBPath (also migrating the file to the
// current schema, which creates the dedup_key column).
Trail audit.Trail
// SkipMigrations opens DBPath as a pre-migrated database when Trail is nil.
// The audit layer still asserts the schema version. This is intended for
// tests that copy a current golden database; production callers should leave
// it false so the real migrator runs.
SkipMigrations bool
// Specs overrides the canonical phase transition table (for tests). nil →
// protocol.PhaseSpecs.
Specs map[protocol.PhaseId]protocol.PhaseSpec
// Logger is the DBOS logger. nil → slog.Default().
Logger *slog.Logger
// OnTransition, when set, runs INSIDE the durable step for each successful
// transition, AFTER the projection + forensic audit row are written and
// BEFORE the step returns. It is the step-bracketing seam: idempotent
// activity recording wires here (it shares the step's replay semantics, so
// any external write it makes must be idempotent — e.g. a deterministic-id
// ON CONFLICT insert). Returning an error fails the step (and so the
// transition's durable commit).
//
// stepSeq is the deterministic per-transition step sequence (the same value
// the audit dedup key is derived from). It is threaded in from the workflow
// body because it cannot be recovered inside the hook: DBOS exposes it only
// in the workflow body, and a replay re-runs only the crashed step, so a
// hook-local counter would not be replay-stable. Hooks derive their own
// deterministic keys from it via protocol.DedupKey.
OnTransition func(ctx context.Context, epochId string, rec *protocol.TransitionRecord, stepSeq string) error
// Tracker, when set, makes the engine record one PROV-O activity per
// transition with a deterministic id (exactly-once across replay). nil ⇒
// activities are not recorded and the engine behaves as it did without this
// field. The engine resolves a stable software-agent id at New() so the
// deterministic insert always references a present agent row.
Tracker ActivitySink
// SliceConcurrency is the per-executor concurrency limit K for the slice
// queue. It bounds the number of slice and review sub-workflows that the
// local executor runs concurrently, providing backpressure on the single
// SQLite WAL writer bottleneck. <= 0 uses DefaultSliceQueueConcurrency.
//
// See DefaultSliceQueueConcurrency in internal/engine/queue.go for the
// full trade-off rationale and tuning guidance.
SliceConcurrency int
// QueueBasePollingInterval overrides the DBOS queue base polling interval.
// Zero keeps the DBOS production default. Tests may set a shorter interval
// to keep bounded-concurrency assertions fast without changing production
// queue cadence.
QueueBasePollingInterval time.Duration
// HooksMgr, when set, receives slice lifecycle events (SliceStarted,
// SliceCompleted, SliceFailed) dispatched by slice sub-workflows. nil ⇒
// hook dispatch is skipped (no observability events; the sub-workflow still
// runs correctly).
//
// pastured wires HooksMgr when it hosts the engine. Callers that don't need
// slice lifecycle observability (e.g. the local CLI, unit tests) may leave
// this nil.
HooksMgr *hooks.Manager
Timeouts timeouts.Profile
}
Config configures an Engine.
type ControlInput ¶
type ControlInput struct {
EpochId string
}
ControlInput is the EpochControlWorkflow input: the epoch id whose lifecycle this durable workflow drives. The workflow ID is set to the epoch ID by the caller, so senders address signals to the epoch by its own id.
type Engine ¶
type Engine struct {
// contains filtered or unexported fields
}
Engine owns the shared modernc handle, the DBOS context, and the forensic trail. It registers and drives the EpochStateMachine over durable steps.
Lifecycle: New → Launch → (run workflows) → Shutdown.
func New ¶
New constructs an Engine: opens the shared handle with the WAL/busy-timeout DSN, ensures the projection table, opens (or adopts) the forensic trail, creates the DBOS context with the shared handle as SqliteSystemDB and the pinned ExecutorID + ApplicationVersion, and registers EpochWorkflow.
The returned Engine is NOT yet launched; call Launch to run the recovery sweep and accept work. Always call Shutdown to release handles.
func (*Engine) ControlQueue ¶
func (e *Engine) ControlQueue() dbos.WorkflowQueue
ControlQueue returns the DBOS WorkflowQueue used for epoch control workflows.
func (*Engine) DBOS ¶
func (e *Engine) DBOS() dbos.DBOSContext
DBOS returns the underlying DBOS context so callers (and later slices) can RunWorkflow / Send / ListWorkflows against the engine's registered workflow.
func (*Engine) EnqueueReview ¶
func (e *Engine) EnqueueReview(in ReviewInput) (dbos.WorkflowHandle[ReviewResult], error)
EnqueueReview dispatches a ReviewSubWorkflow via the slice queue. The workflow id is derived from the epoch id, phase id, and review round so that each round of a review cycle runs a fresh sub-workflow rather than returning the memoized result of a prior round.
The caller should retain the returned handle to send submit_vote signals and to wait for the result. To compute the same workflow id for vote delivery, call protocol.ReviewWorkflowID(epochId, phaseId, round).
func (*Engine) EnqueueSlice ¶
func (e *Engine) EnqueueSlice(in SliceInput) (dbos.WorkflowHandle[SliceResult], error)
EnqueueSlice dispatches a SliceSubWorkflow via the slice queue, giving it the supplied workflow id (sliceId) so start_slice / complete_slice signals can address it. The caller supplies the epoch context needed for hook dispatch and parent progress signalling. The returned handle is live; callers may call GetResult to wait for the slice to complete.
func (*Engine) EpochControlWorkflow ¶
func (e *Engine) EpochControlWorkflow(ctx dbos.DBOSContext, in ControlInput) (protocol.EpochState, error)
EpochControlWorkflow is the signal-driven durable driver for one epoch.
Unlike the scripted EpochWorkflow (which replays a fixed plan), this workflow advances only in response to durable signals delivered by topic:
- advance_phase drives one FSM transition.
- submit_vote records a phase-scoped review vote (consumed before the gated advance that needs it).
- register_session registers a session (idempotent by session id).
- slice_progress appends a slice-progress event.
The slice-level start_slice / complete_slice topics are consumed by the slice sub-workflows, not this epoch loop.
Each loop blocks for the next advance_phase signal. When one arrives it first drains the three side-channel topics (non-blocking) — so any votes sent just before the advance are recorded and the consensus gate sees them — then applies the advance. On an idle timeout it drains the side channels anyway so sessions and slice progress reported between advances reach the projection. Every successful transition funnels through commitTransition, so the projection, the exactly-once forensic emit, and the activity hook are identical to the scripted driver. The loop ends when the FSM reaches the terminal phase.
func (*Engine) EpochWorkflow ¶
func (e *Engine) EpochWorkflow(ctx dbos.DBOSContext, in EpochInput) (protocol.EpochState, error)
EpochWorkflow is the durable workflow that drives the 12-phase epoch.
For each planned transition it (1) records votes and the blocker delta and runs EpochStateMachine.Advance in the workflow BODY — pure, deterministic, so the phase sequence replays identically — then (2) performs the I/O in ONE durable step: persist the EpochState projection and record exactly one forensic row keyed by the deterministic dedup key. One step per transition means one forensic emission per (kind, step), preserving the dedup invariant.
A failed advance (gate violation) is recorded as a failed transition and the plan continues; the durable step is skipped for that entry.
func (*Engine) Launch ¶
Launch runs the DBOS recovery sweep (resuming any in-flight epochs) and makes the engine ready to run new workflows. Call exactly once after New.
func (*Engine) ReadProjection ¶
func (e *Engine) ReadProjection(epochId string) (*protocol.EpochState, error)
ReadProjection returns the projected EpochState for epochId, or (nil, nil) if the epoch has not advanced yet. This is the read side of the projection that query and status surfaces consume.
func (*Engine) ReviewSubWorkflow ¶
func (e *Engine) ReviewSubWorkflow(ctx dbos.DBOSContext, in ReviewInput) (ReviewResult, error)
ReviewSubWorkflow is the DBOS sub-workflow for a single P4/P10 review phase.
Lifecycle:
- Dispatched via Engine.EnqueueReview to the slice queue; starts when a queue slot is free (bounded by K).
- Receives submit_vote signals (ReviewVoteSignal) via a polling Recv loop until all three ReviewAxis members have voted.
- Returns a ReviewResult with the collected per-axis vote map.
The submit_vote signals are addressed to this sub-workflow by the id assigned by Engine.EnqueueReview (protocol.ReviewWorkflowID(epochId, phaseId, round)).
Idempotency: if the same axis votes twice, the later vote overwrites the earlier one (last-writer-wins per ReviewAxis key).
func (*Engine) Shutdown ¶
Shutdown stops the DBOS context (waiting up to timeout for in-flight steps), then closes the shared handle and the owned trail. Safe to call once.
func (*Engine) SliceConcurrency ¶
SliceConcurrency returns the effective per-executor concurrency limit K that was used to configure the slice queue. This is the resolved value (after applying the DefaultSliceQueueConcurrency fallback) stored once in New — not re-derived from the config to avoid having two copies of the fallback logic that could drift.
func (*Engine) SliceQueue ¶
func (e *Engine) SliceQueue() dbos.WorkflowQueue
SliceQueue returns the DBOS WorkflowQueue used for slice and review sub-workflow dispatch. Tests may inspect the queue name to verify wiring.
func (*Engine) SliceSubWorkflow ¶
func (e *Engine) SliceSubWorkflow(ctx dbos.DBOSContext, in SliceInput) (SliceResult, error)
SliceSubWorkflow is the DBOS sub-workflow for a single implementation slice.
Lifecycle:
- Dispatched via Engine.EnqueueSlice to the slice queue; starts when a queue slot is free (bounded by the configured concurrency limit K).
- Receives a start_slice signal (SliceStartSignal) via dbos.Recv before deciding the execution mode. If no signal arrives within the deadline the sub-workflow records an honest failure (Success=false) and returns; no completion hook fires and the parent projection receives Completed=false.
- Executes the slice in the chosen mode (mock / tmux / subprocess) inside a durable step.
- Receives an optional complete_slice signal (SliceCompleteSignal) that overrides the computed outcome.
- Dispatches hook events (SliceStarted / SliceCompleted / SliceFailed) through the engine's hook manager inside durable steps (memoized; not re-fired on crash recovery).
- Sends a slice_progress signal to the parent epoch workflow.
The start_slice and complete_slice signals are addressed to the sub-workflow by its sliceId (which is its DBOS workflow id).
type EpochInput ¶
type EpochInput struct {
EpochId string
Advances []AdvanceStep
}
EpochInput is the EpochWorkflow input: the epoch id and the ordered plan of transitions to drive.
type ReviewInput ¶
type ReviewInput struct {
// EpochId is the parent epoch this review belongs to.
EpochId string `json:"epochId"`
// PhaseId identifies which review phase this is (e.g. "review" or "code-review").
PhaseId string `json:"phaseId"`
// Round is the review-cycle counter for this (epochId, phaseId) pair. It
// starts at 1 and increments each time a review returns REVISE and the
// protocol re-enters the review phase. Supplying the round ensures each
// re-review runs a fresh DBOS sub-workflow (a different id) rather than
// returning the memoized result of a prior round.
//
// The round value MUST come from a deterministic, replay-stable counter
// tracked in workflow state, NOT from wall-clock time or a random value.
// Default 0 is treated as round 1 by EnqueueReview for backwards
// compatibility (existing callers that don't set Round still get the
// correct first-round workflow id).
Round int `json:"round,omitempty"`
}
ReviewInput is the input to a review sub-workflow.
type ReviewResult ¶
type ReviewResult struct {
// PhaseId echoes the input for correlation on the parent side.
PhaseId string `json:"phaseId"`
// Success is true when all review axes received an ACCEPT vote.
Success bool `json:"success"`
// VoteResult is the per-axis vote map collected by the sub-workflow.
VoteResult map[protocol.ReviewAxis]protocol.VoteType `json:"voteResult"`
}
ReviewResult is the output of a review sub-workflow.
type SliceInput ¶
type SliceInput struct {
// EpochId is the parent epoch this slice belongs to. Used for hook dispatch
// and the parent progress signal.
EpochId string `json:"epochId"`
// SliceId is the unique identifier for this slice. It doubles as the
// sub-workflow's id so start_slice / complete_slice signals address it.
SliceId string `json:"sliceId"`
// ParentWorkflowId is the id of the epoch control workflow that dispatched
// this slice. When non-empty, the sub-workflow delivers a slice_progress
// signal to it on completion.
ParentWorkflowId string `json:"parentWorkflowId"`
}
SliceInput is the input to a slice sub-workflow.
type SliceResult ¶
type SliceResult struct {
// SliceId echoes the input for correlation on the parent side.
SliceId string `json:"sliceId"`
// Success is true when the slice completed without error.
Success bool `json:"success"`
// Output holds a human-readable success message (non-empty when Success is true).
Output string `json:"output,omitempty"`
// Error holds the failure reason (non-empty when Success is false).
Error *string `json:"error,omitempty"`
}
SliceResult is the output of a slice sub-workflow.