Documentation
¶
Overview ¶
Package journal is the engine's durable, append-only event log.
One journal exists per session at <session_root>/journal/. Every event dispatched on the bus is recorded with a monotonic per-session sequence number, the dispatching event's parent sequence (best-effort), and the vetoed flag for before:* events. The journal is the source of truth for crash recovery, deterministic replay, and observability projections — the in-memory bus ring is only for the boot-time pre-subscription gap.
Index ¶
- Constants
- func IsSideEffect(eventType string) bool
- func ListArchives(journalDir string) ([]string, error)
- func PayloadAs[T any](payload any) (T, error)
- func ProjectFile(dir string, types []string, handler func(Envelope)) error
- func Restore(journalDir, archiveName string) error
- func Sweep(root string, retainDays int) error
- type Coordinator
- func (c *Coordinator) AttachTurnSync(subscribe func(handler func(seq uint64)) func())
- func (c *Coordinator) CompletedInputs() []Envelope
- func (c *Coordinator) Inputs() []Envelope
- func (c *Coordinator) IsPartialTurn() bool
- func (c *Coordinator) LastSeq() uint64
- func (c *Coordinator) LastTurnBoundary() (uint64, bool)
- func (c *Coordinator) PartialInput() (Envelope, bool)
- func (c *Coordinator) Run(ctx context.Context) error
- type CoordinatorOptions
- type Envelope
- type EventEmitter
- type FsyncMode
- type Header
- type PayloadConverter
- type Reader
- type ReplayController
- type RewindResult
- type SeqSource
- type SnapshotFile
- type SubscribingBus
- type Writer
- func (w *Writer) Append(env *Envelope)
- func (w *Writer) Barrier(ctx context.Context) error
- func (w *Writer) Close(ctx context.Context) error
- func (w *Writer) Closed() bool
- func (w *Writer) JournalDir() string
- func (w *Writer) Snapshot(stageDir string) ([]SnapshotFile, error)
- func (w *Writer) SubscribeProjection(types []string, handler func(Envelope)) func()
- type WriterOptions
Constants ¶
const SchemaVersion = "1"
SchemaVersion is the on-disk format version. Readers reject journals with any other value rather than attempting silent migration.
Variables ¶
This section is empty.
Functions ¶
func IsSideEffect ¶
IsSideEffect reports whether the event type is recorded as a side effect. Centralized so providers and tools can extend the list as new categories arrive without each writer site re-implementing the rule.
func ListArchives ¶
ListArchives returns the archive directory names under journalDir, sorted ascending by directory name (timestamp). An empty slice is returned when no archive directory has ever been created.
func PayloadAs ¶
PayloadAs round-trips a journal-decoded payload (map[string]any after JSON unmarshal) back into a typed value via JSON. Side-effecting plugins use this to re-materialize their stashed responses.
Generic helper rather than per-event-type: the journal package has no dependency on plugins/events, so providers call it with their own type.
func ProjectFile ¶
ProjectFile walks an existing journal directory and fires the handler for every envelope whose type is in the filter set. Pass an empty types slice to receive every envelope.
Use case: regenerate a derived view from the journal alone — e.g. rebuild a per-plugin JSONL file after it was deleted, or feed a projection plugin that did not exist when the original session ran. Pairs with Writer.SubscribeProjection — same handler signature, same filter semantics — so the same code path drives both live and post-mortem projection.
Errors from Open or Iter are returned to the caller; per-envelope handler panics propagate (this is a tooling helper, not a production hot path — callers can recover at the boundary if they want to keep going).
func Restore ¶
Restore swaps the live journal for a previously archived snapshot. The current live journal is itself archived first (under a fresh timestamp) so the operation is reversible.
archiveName is the directory name (not full path) reported by Rewind or visible under <journalDir>/archive/.
func Sweep ¶
Sweep removes session journal directories whose enclosing session was last modified more than retainDays ago. Called from engine boot in a goroutine so a slow filesystem does not delay startup.
root is the sessions root (e.g. ~/.nexus/sessions). retainDays <= 0 is a no-op so accidental misconfiguration cannot purge a freshly-cleaned home.
Types ¶
type Coordinator ¶
type Coordinator struct {
// contains filtered or unexported fields
}
Coordinator drives deterministic replay of a session journal.
At Run(), it scans the journal once to seed the replay state's per-event queues with journaled responses (llm.response, tool.result, hitl.responded), then re-emits io.input events from the journal in seq order. The live agent reacts to each io.input as if it were a fresh turn; the side-effecting plugins (providers, tools) check the engine's replay flag and pop the next stashed response instead of calling out.
Synchronization between turns is via a transient subscription to agent.turn.end. The coordinator does not own the engine's lifecycle — the caller must boot the engine before calling Run.
func NewCoordinator ¶
func NewCoordinator(journalDir string, bus EventEmitter, state ReplayController, opts CoordinatorOptions) (*Coordinator, error)
NewCoordinator builds a coordinator for a session's journal directory. Returns ErrEmptyJournal if the journal contains no replayable inputs.
func (*Coordinator) AttachTurnSync ¶
func (c *Coordinator) AttachTurnSync(subscribe func(handler func(seq uint64)) func())
AttachTurnSync subscribes to agent.turn.end on the supplied bus so Run can synchronize re-emitted io.inputs against turn completion. Caller supplies a subscribe closure to avoid the journal package importing engine. The returned unsubscribe func runs on Run completion.
If unset, Run emits inputs back-to-back without per-turn synchronization — useful for tests where the system processes inputs synchronously.
func (*Coordinator) CompletedInputs ¶
func (c *Coordinator) CompletedInputs() []Envelope
CompletedInputs returns inputs whose enclosing turn ended — i.e. input envelopes with seq <= LastTurnBoundary. The Phase 2 / 3 replay path uses this when it wants to replay the well-formed prefix of a journal without spilling into the partial-turn tail.
func (*Coordinator) Inputs ¶
func (c *Coordinator) Inputs() []Envelope
Inputs returns the io.input envelopes the coordinator will replay. Visible for tests.
func (*Coordinator) IsPartialTurn ¶
func (c *Coordinator) IsPartialTurn() bool
IsPartialTurn reports whether the journal ends mid-turn — an agent.turn.start without a matching agent.turn.end. Phase 3 crash-resume uses this to decide whether to replay-then-continue or just replay.
func (*Coordinator) LastSeq ¶
func (c *Coordinator) LastSeq() uint64
LastSeq returns the highest seq seen in the source journal.
func (*Coordinator) LastTurnBoundary ¶
func (c *Coordinator) LastTurnBoundary() (uint64, bool)
LastTurnBoundary returns the seq of the most recent agent.turn.end in the scanned journal, or (0, false) if none.
func (*Coordinator) PartialInput ¶
func (c *Coordinator) PartialInput() (Envelope, bool)
PartialInput returns the io.input envelope that started the unfinished turn — i.e. the input with seq greater than LastTurnBoundary. Returns (Envelope{}, false) when no partial turn exists or no io.input above the boundary was journaled (the crash happened before the input was recorded — nothing to re-fire).
type CoordinatorOptions ¶
type CoordinatorOptions struct {
// TurnTimeout caps how long the coordinator waits between re-emitting
// an io.input and observing the next agent.turn.end. Replay aborts
// with a clear error when exceeded — typically signals a provider
// short-circuit miss or a stalled handler.
TurnTimeout time.Duration
// Logger is optional; nil silently uses slog.Default().
Logger *slog.Logger
// PayloadConverter, when set, is applied to every payload before the
// coordinator emits it on the bus or pushes it onto a stash queue.
// Engines wire this to a per-event-type type-restoration helper.
PayloadConverter PayloadConverter
}
CoordinatorOptions tune replay behavior.
type Envelope ¶
type Envelope struct {
// Seq is monotonic per-session, starts at 1, gap-free.
Seq uint64 `json:"seq"`
// Ts is when the event was dispatched on the bus, not when it landed
// on disk. Replay tools key off this.
Ts time.Time `json:"ts"`
// Type is the event type (e.g. "llm.request", "tool.result").
Type string `json:"type"`
// EventID is the bus-assigned random ID. Carried so external tools can
// correlate journal entries with otel spans and logger output.
EventID string `json:"event_id,omitempty"`
// Source is the plugin ID that emitted the event, when known.
Source string `json:"source,omitempty"`
// TraceID is reserved for cross-system correlation (otel). Empty for now.
TraceID string `json:"trace_id,omitempty"`
// ParentSeq is the seq of the event whose handler emitted this one,
// best-effort. Zero means no detectable parent.
ParentSeq uint64 `json:"parent_seq,omitempty"`
// ParentID is the EventID of the event whose handler emitted this one.
// Empty for root events. Mirrors ParentSeq via the event identifier so
// replay can rebuild the causation DAG without re-resolving seq → ID.
ParentID string `json:"parent_id,omitempty"`
// SessionID is the session the event belongs to. Carried per-envelope
// so external aggregators that flatten multi-session journals retain
// session attribution without re-reading header.json.
SessionID string `json:"session_id,omitempty"`
// AgentID identifies the agent that produced the event. For sub-agent
// activity this is the sub-agent's identity, not the parent's.
AgentID string `json:"agent_id,omitempty"`
// Depth is the sub-agent recursion depth at emission time. Zero for
// the top-level agent.
Depth int `json:"depth,omitempty"`
// SideEffect marks events whose handlers performed real-world I/O
// (LLM call billed, file written, shell run). Replay must short-circuit
// these. Computed from event type, not the handler.
SideEffect bool `json:"side_effect,omitempty"`
// Vetoed is true for before:* events that a handler vetoed.
Vetoed bool `json:"vetoed,omitempty"`
// VetoReason is the handler-supplied reason when Vetoed is true.
VetoReason string `json:"veto_reason,omitempty"`
// Payload is the event payload, marshaled as-is. Marshalling errors
// fall back to a placeholder so the envelope still records the event.
Payload any `json:"payload,omitempty"`
}
Envelope is the on-disk record for a single dispatched event. One envelope is one JSONL line in events.jsonl (or events-NNN.jsonl.zst once rotated).
Field naming uses snake_case to match the rest of the on-disk JSON in the session workspace and to keep tail-style external tooling readable.
type EventEmitter ¶
EventEmitter is the subset of the bus that the coordinator needs. The journal package cannot import engine without creating a cycle, so this minimal interface is enough.
type FsyncMode ¶
type FsyncMode string
FsyncMode controls how often the writer flushes the kernel buffer to disk.
const ( // FsyncEveryEvent fsyncs after every appended envelope. Strongest crash // guarantee, slowest throughput. FsyncEveryEvent FsyncMode = "every-event" // FsyncTurnBoundary fsyncs only when an agent.turn.end envelope is // appended. Default — most turns are 5–30 events, amortizing the disk // flush across the burst. FsyncTurnBoundary FsyncMode = "turn-boundary" // FsyncNone never fsyncs explicitly. The OS buffer is still flushed on // Close. Suitable for ephemeral test runs. FsyncNone FsyncMode = "none" )
func ParseFsyncMode ¶
ParseFsyncMode normalizes a config string to a known mode. Unknown values fall back to FsyncTurnBoundary so a typo cannot silently disable durability.
type Header ¶
type Header struct {
SchemaVersion string `json:"schema_version"`
CreatedAt time.Time `json:"created_at"`
FsyncMode string `json:"fsync_mode"`
SessionID string `json:"session_id,omitempty"`
}
Header is the journal directory's manifest. Written once at construction, re-read on every open to validate compatibility.
type PayloadConverter ¶
PayloadConverter optionally re-types a journal-deserialized payload (typically map[string]any after JSON round-trip) into whatever struct type the live subscribers expect. The journal package does not import any plugin-specific types, so callers wire the per-event-type conversions in. A nil converter passes the payload through unchanged.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader opens a journal directory and streams envelopes in seq order. It is read-only; concurrent open with an active Writer on the same dir is fine for the rotated segments but the active segment's tail may be incomplete.
func Open ¶
Open validates the header and indexes the segments. Iter does the actual streaming; this constructor is cheap.
func (*Reader) Iter ¶
Iter streams every envelope in seq order. The yield func returns false to stop iteration; the underlying readers are closed before Iter returns.
Malformed JSONL lines are skipped; partial trailing lines (no newline) are also skipped — the writer crashed mid-write and the next Append will either complete or overwrite that line on resume. Behavior matches Go's bufio.Scanner default.
func (*Reader) LastSeq ¶
LastSeq returns the highest seq seen across all segments. Returns 0 if the journal is empty.
func (*Reader) LastTurnBoundary ¶
LastTurnBoundary returns the seq of the most recent agent.turn.end envelope and ok=true if found. Used by the Phase 2 coordinator to detect a partial turn at boot — the unfinished turn lives between this seq and LastSeq.
type ReplayController ¶
ReplayController is the subset of engine.ReplayState the coordinator drives. Same anti-cycle reason as EventEmitter.
type RewindResult ¶
type RewindResult struct {
// ArchiveName is the directory name (not full path) of the archived
// pre-rewind journal under <journalDir>/archive/. Pass it back to
// Restore to swap it in.
ArchiveName string
// TruncatedSeq is the highest seq that survived the rewind. Equals
// the requested ToSeq when the journal contained an envelope with
// that seq; otherwise it is the largest seq <= ToSeq that was found.
TruncatedSeq uint64
// EventsKept is how many envelopes ended up in the new live journal.
EventsKept int
// EventsArchived is how many envelopes were in the pre-rewind
// journal (kept + dropped).
EventsArchived int
}
RewindResult describes the outcome of a successful Rewind call.
func Rewind ¶
func Rewind(journalDir string, toSeq uint64) (RewindResult, error)
Rewind rewrites journalDir so its live journal ends at the envelope with seq == toSeq (inclusive). The original journal is preserved verbatim under <journalDir>/archive/<timestamp>/, where it can be inspected or swapped back in via Restore.
The session must not have an active Writer attached when Rewind runs; concurrent writes against a journal being rewound produce undefined state. Engine callers tear down the Writer before invoking.
Rewind is idempotent in the sense that calling it twice with the same toSeq leaves the live journal in the same shape; each call simply produces another archive snapshot.
type SeqSource ¶
SeqSource is the contract the bus implements so journal handlers can retrieve the seq + parent_seq the bus assigned for the current dispatch.
Every EmitEvent / EmitVetoable assigns a seq via an atomic counter and pushes (seq, eventID) onto a per-goroutine stack at dispatch entry. While a wildcard handler runs (the writer's hook), CurrentSeq returns the top of the stack and ParentSeq returns the next entry below — the seq of the event whose handler triggered the current emit.
The interface lives in the journal package (not the engine package) because the writer does not know the engine type, and the engine wires the dependency at construction time.
type SnapshotFile ¶
type SnapshotFile struct {
// Name is the base name within the journal directory. Callers turn it
// into a destination path or object key themselves; the journal package
// knows nothing about either.
Name string
// Path is where the bytes actually live. For an immutable segment this is
// the live file; for the active segment it is the staged copy.
Path string
// Staged reports whether Path is a copy Snapshot made (and the caller is
// responsible for cleaning up) rather than a file in the journal dir.
Staged bool
// Size is the file size at capture time.
Size int64
}
SnapshotFile names one file in a consistent instant of a journal directory.
type SubscribingBus ¶
type SubscribingBus interface {
EventEmitter
Subscribe(eventType string, handler func(eventType string, payload any), priority int) (unsubscribe func())
}
SubscribingBus is the bus capability the coordinator needs to install the agent.turn.end synchronization handler. The engine's bus implements it; tests can supply a stub.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer is the durable JSONL appender for a single session's journal.
Object-store disposition: turn-boundary-only ¶
The Writer puts bytes under the session tree (journal/events.jsonl) with raw os.* calls and announces nothing on the bus. Unlike every other bypassing writer, that is not a cost/benefit judgement — it is the only option. This writer's *input* is every event dispatched on the bus, so an announcement per journal write would produce an envelope, which would produce another announcement, forever. The Writer holding no bus reference at all is what makes that loop impossible by construction rather than by discipline, and is the reason to leave it that way.
The journal still syncs: the turn-boundary snapshot captures it through Snapshot below, which takes one consistent instant across the active and rotated segments so a snapshot can never contain the same events twice. Recorded in SessionTreeWriters (pkg/engine/session_writers.go).
Writes are decoupled from the bus dispatch goroutine via a buffered channel: the bus's wildcard handler builds an Envelope (with seq + parent_seq supplied by the bus) and pushes it on the channel; a drain goroutine orders envelopes by seq (since wildcard order is child-before-parent for nested emits) and appends them to events.jsonl. Fsync policy is honored per-envelope.
func NewWriter ¶
func NewWriter(dir string, opts WriterOptions) (*Writer, error)
NewWriter creates the journal directory if absent, writes (or validates) the header, opens events.jsonl for append, and starts the drain goroutine.
Multiple writers must not target the same dir; the engine creates exactly one writer per session and that constraint is enforced at the call site.
func (*Writer) Append ¶
Append queues an envelope for asynchronous write. Blocks if the channel is full so callers feel back-pressure rather than silently losing events. Safe to call concurrently with Close — once closed, subsequent Appends drop.
func (*Writer) Barrier ¶
Barrier blocks until the drain goroutine has written, flushed and fsynced every envelope queued before the call.
Appends are asynchronous: the bus's wildcard handler hands an envelope to a channel and returns, so a bus handler running at a turn boundary observes a journal that trails the turn it is reacting to by however many envelopes are still queued. The object-store turn snapshot is exactly such a handler, and without this barrier every snapshot would capture a journal ending mid-turn — making a resumed session re-fire its last input on every single resume.
One caveat, deliberate: the drain writes envelopes in contiguous seq order, so a seq assigned by the bus but not yet handed to Append leaves a gap that blocks everything behind it. Barrier does not wait for such a gap to close — it cannot, since the missing envelope may never arrive — so it guarantees "everything queued before the call, up to the first gap", not "everything the bus has ever numbered".
Returns nil once the writer is closed: Close already flushed and fsynced.
func (*Writer) Close ¶
Close drains in-flight envelopes, fsyncs, and shuts the file. Safe to call multiple times; subsequent calls return nil.
func (*Writer) Closed ¶
Closed reports whether Close has run. Once true, Appends drop and the drain goroutine is on its way out (it has exited once Close returns nil).
func (*Writer) JournalDir ¶
JournalDir returns the directory passed to NewWriter. Used by Coordinator (Phase 2) and by tests.
func (*Writer) Snapshot ¶
func (w *Writer) Snapshot(stageDir string) ([]SnapshotFile, error)
Snapshot captures a consistent instant of the journal's segment files, staging the mutable active segment into stageDir.
Subdirectories are ignored: the tool-result cache lives at journal/cache/ and is ordinary, independently-written data that a caller walking the session tree picks up on its own. Only the top-level segment files — which rotation mutates as a set — need this treatment.
The caller owns stageDir and must remove it when done. A nil Writer yields no files and no error, so a caller need not special-case a session that has no journal.
func (*Writer) SubscribeProjection ¶
SubscribeProjection registers a handler that fires synchronously on the writer's drain goroutine after each envelope lands on disk. Pass an empty types slice to receive every envelope; otherwise the handler only fires when env.Type is in the filter set.
Projections run AFTER the disk write, so a handler observing an envelope can rely on it being durable. The trade-off is that a slow handler stalls the drain — keep handlers cheap (append-to-file is fine) and offload anything heavier to a goroutine the handler launches.
Returns an unsubscribe function. Safe to call from any goroutine.
type WriterOptions ¶
type WriterOptions struct {
FsyncMode FsyncMode
RotateBytes int64
BufferSize int
SchemaVersion string
SessionID string
// InitialSeq is the lowest seq the drain expects to flush. Defaults
// to 1 for fresh sessions; the engine sets this to LastSeq+1 on
// recall so the reorder buffer does not stall waiting for a seq the
// new run will never produce.
InitialSeq uint64
}
WriterOptions tune the writer.