journal

package
v0.32.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 1, 2026 License: MIT Imports: 18 Imported by: 0

Documentation

Overview

Package journal is the engine's durable, append-only event log.

One journal exists per session at <session_root>/journal/. Every event dispatched on the bus is recorded with a monotonic per-session sequence number, the dispatching event's parent sequence (best-effort), and the vetoed flag for before:* events. The journal is the source of truth for crash recovery, deterministic replay, and observability projections — the in-memory bus ring is only for the boot-time pre-subscription gap.

Index

Constants

View Source
const SchemaVersion = "1"

SchemaVersion is the on-disk format version. Readers reject journals with any other value rather than attempting silent migration.

Variables

This section is empty.

Functions

func IsSideEffect

func IsSideEffect(eventType string) bool

IsSideEffect reports whether the event type is recorded as a side effect. Centralized so providers and tools can extend the list as new categories arrive without each writer site re-implementing the rule.

func ListArchives

func ListArchives(journalDir string) ([]string, error)

ListArchives returns the archive directory names under journalDir, sorted ascending by directory name (timestamp). An empty slice is returned when no archive directory has ever been created.

func PayloadAs

func PayloadAs[T any](payload any) (T, error)

PayloadAs round-trips a journal-decoded payload (map[string]any after JSON unmarshal) back into a typed value via JSON. Side-effecting plugins use this to re-materialize their stashed responses.

Generic helper rather than per-event-type: the journal package has no dependency on plugins/events, so providers call it with their own type.

func ProjectFile

func ProjectFile(dir string, types []string, handler func(Envelope)) error

ProjectFile walks an existing journal directory and fires the handler for every envelope whose type is in the filter set. Pass an empty types slice to receive every envelope.

Use case: regenerate a derived view from the journal alone — e.g. rebuild a per-plugin JSONL file after it was deleted, or feed a projection plugin that did not exist when the original session ran. Pairs with Writer.SubscribeProjection — same handler signature, same filter semantics — so the same code path drives both live and post-mortem projection.

Errors from Open or Iter are returned to the caller; per-envelope handler panics propagate (this is a tooling helper, not a production hot path — callers can recover at the boundary if they want to keep going).

func Restore

func Restore(journalDir, archiveName string) error

Restore swaps the live journal for a previously archived snapshot. The current live journal is itself archived first (under a fresh timestamp) so the operation is reversible.

archiveName is the directory name (not full path) reported by Rewind or visible under <journalDir>/archive/.

func Sweep

func Sweep(root string, retainDays int) error

Sweep removes session journal directories whose enclosing session was last modified more than retainDays ago. Called from engine boot in a goroutine so a slow filesystem does not delay startup.

root is the sessions root (e.g. ~/.nexus/sessions). retainDays <= 0 is a no-op so accidental misconfiguration cannot purge a freshly-cleaned home.

Types

type Coordinator

type Coordinator struct {
	// contains filtered or unexported fields
}

Coordinator drives deterministic replay of a session journal.

At Run(), it scans the journal once to seed the replay state's per-event queues with journaled responses (llm.response, tool.result, hitl.responded), then re-emits io.input events from the journal in seq order. The live agent reacts to each io.input as if it were a fresh turn; the side-effecting plugins (providers, tools) check the engine's replay flag and pop the next stashed response instead of calling out.

Synchronization between turns is via a transient subscription to agent.turn.end. The coordinator does not own the engine's lifecycle — the caller must boot the engine before calling Run.

func NewCoordinator

func NewCoordinator(journalDir string, bus EventEmitter, state ReplayController, opts CoordinatorOptions) (*Coordinator, error)

NewCoordinator builds a coordinator for a session's journal directory. Returns ErrEmptyJournal if the journal contains no replayable inputs.

func (*Coordinator) AttachTurnSync

func (c *Coordinator) AttachTurnSync(subscribe func(handler func(seq uint64)) func())

AttachTurnSync subscribes to agent.turn.end on the supplied bus so Run can synchronize re-emitted io.inputs against turn completion. Caller supplies a subscribe closure to avoid the journal package importing engine. The returned unsubscribe func runs on Run completion.

If unset, Run emits inputs back-to-back without per-turn synchronization — useful for tests where the system processes inputs synchronously.

func (*Coordinator) CompletedInputs

func (c *Coordinator) CompletedInputs() []Envelope

CompletedInputs returns inputs whose enclosing turn ended — i.e. input envelopes with seq <= LastTurnBoundary. The Phase 2 / 3 replay path uses this when it wants to replay the well-formed prefix of a journal without spilling into the partial-turn tail.

func (*Coordinator) Inputs

func (c *Coordinator) Inputs() []Envelope

Inputs returns the io.input envelopes the coordinator will replay. Visible for tests.

func (*Coordinator) IsPartialTurn

func (c *Coordinator) IsPartialTurn() bool

IsPartialTurn reports whether the journal ends mid-turn — an agent.turn.start without a matching agent.turn.end. Phase 3 crash-resume uses this to decide whether to replay-then-continue or just replay.

func (*Coordinator) LastSeq

func (c *Coordinator) LastSeq() uint64

LastSeq returns the highest seq seen in the source journal.

func (*Coordinator) LastTurnBoundary

func (c *Coordinator) LastTurnBoundary() (uint64, bool)

LastTurnBoundary returns the seq of the most recent agent.turn.end in the scanned journal, or (0, false) if none.

func (*Coordinator) PartialInput

func (c *Coordinator) PartialInput() (Envelope, bool)

PartialInput returns the io.input envelope that started the unfinished turn — i.e. the input with seq greater than LastTurnBoundary. Returns (Envelope{}, false) when no partial turn exists or no io.input above the boundary was journaled (the crash happened before the input was recorded — nothing to re-fire).

func (*Coordinator) Run

func (c *Coordinator) Run(ctx context.Context) error

Run executes the replay. The engine must already be booted; the caller is responsible for tearing it down. Returns when all journaled io.inputs have been replayed and their turns have settled, or on first error.

type CoordinatorOptions

type CoordinatorOptions struct {
	// TurnTimeout caps how long the coordinator waits between re-emitting
	// an io.input and observing the next agent.turn.end. Replay aborts
	// with a clear error when exceeded — typically signals a provider
	// short-circuit miss or a stalled handler.
	TurnTimeout time.Duration
	// Logger is optional; nil silently uses slog.Default().
	Logger *slog.Logger
	// PayloadConverter, when set, is applied to every payload before the
	// coordinator emits it on the bus or pushes it onto a stash queue.
	// Engines wire this to a per-event-type type-restoration helper.
	PayloadConverter PayloadConverter
}

CoordinatorOptions tune replay behavior.

type Envelope

type Envelope struct {
	// Seq is monotonic per-session, starts at 1, gap-free.
	Seq uint64 `json:"seq"`
	// Ts is when the event was dispatched on the bus, not when it landed
	// on disk. Replay tools key off this.
	Ts time.Time `json:"ts"`
	// Type is the event type (e.g. "llm.request", "tool.result").
	Type string `json:"type"`
	// EventID is the bus-assigned random ID. Carried so external tools can
	// correlate journal entries with otel spans and logger output.
	EventID string `json:"event_id,omitempty"`
	// Source is the plugin ID that emitted the event, when known.
	Source string `json:"source,omitempty"`
	// TraceID is reserved for cross-system correlation (otel). Empty for now.
	TraceID string `json:"trace_id,omitempty"`
	// ParentSeq is the seq of the event whose handler emitted this one,
	// best-effort. Zero means no detectable parent.
	ParentSeq uint64 `json:"parent_seq,omitempty"`
	// ParentID is the EventID of the event whose handler emitted this one.
	// Empty for root events. Mirrors ParentSeq via the event identifier so
	// replay can rebuild the causation DAG without re-resolving seq → ID.
	ParentID string `json:"parent_id,omitempty"`
	// SessionID is the session the event belongs to. Carried per-envelope
	// so external aggregators that flatten multi-session journals retain
	// session attribution without re-reading header.json.
	SessionID string `json:"session_id,omitempty"`
	// AgentID identifies the agent that produced the event. For sub-agent
	// activity this is the sub-agent's identity, not the parent's.
	AgentID string `json:"agent_id,omitempty"`
	// Depth is the sub-agent recursion depth at emission time. Zero for
	// the top-level agent.
	Depth int `json:"depth,omitempty"`
	// SideEffect marks events whose handlers performed real-world I/O
	// (LLM call billed, file written, shell run). Replay must short-circuit
	// these. Computed from event type, not the handler.
	SideEffect bool `json:"side_effect,omitempty"`
	// Vetoed is true for before:* events that a handler vetoed.
	Vetoed bool `json:"vetoed,omitempty"`
	// VetoReason is the handler-supplied reason when Vetoed is true.
	VetoReason string `json:"veto_reason,omitempty"`
	// Payload is the event payload, marshaled as-is. Marshalling errors
	// fall back to a placeholder so the envelope still records the event.
	Payload any `json:"payload,omitempty"`
}

Envelope is the on-disk record for a single dispatched event. One envelope is one JSONL line in events.jsonl (or events-NNN.jsonl.zst once rotated).

Field naming uses snake_case to match the rest of the on-disk JSON in the session workspace and to keep tail-style external tooling readable.

type EventEmitter

type EventEmitter interface {
	Emit(eventType string, payload any) error
}

EventEmitter is the subset of the bus that the coordinator needs. The journal package cannot import engine without creating a cycle, so this minimal interface is enough.

type FsyncMode

type FsyncMode string

FsyncMode controls how often the writer flushes the kernel buffer to disk.

const (
	// FsyncEveryEvent fsyncs after every appended envelope. Strongest crash
	// guarantee, slowest throughput.
	FsyncEveryEvent FsyncMode = "every-event"
	// FsyncTurnBoundary fsyncs only when an agent.turn.end envelope is
	// appended. Default — most turns are 5–30 events, amortizing the disk
	// flush across the burst.
	FsyncTurnBoundary FsyncMode = "turn-boundary"
	// FsyncNone never fsyncs explicitly. The OS buffer is still flushed on
	// Close. Suitable for ephemeral test runs.
	FsyncNone FsyncMode = "none"
)

func ParseFsyncMode

func ParseFsyncMode(s string) FsyncMode

ParseFsyncMode normalizes a config string to a known mode. Unknown values fall back to FsyncTurnBoundary so a typo cannot silently disable durability.

type Header struct {
	SchemaVersion string    `json:"schema_version"`
	CreatedAt     time.Time `json:"created_at"`
	FsyncMode     string    `json:"fsync_mode"`
	SessionID     string    `json:"session_id,omitempty"`
}

Header is the journal directory's manifest. Written once at construction, re-read on every open to validate compatibility.

type PayloadConverter

type PayloadConverter func(eventType string, payload any) (any, error)

PayloadConverter optionally re-types a journal-deserialized payload (typically map[string]any after JSON round-trip) into whatever struct type the live subscribers expect. The journal package does not import any plugin-specific types, so callers wire the per-event-type conversions in. A nil converter passes the payload through unchanged.

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader opens a journal directory and streams envelopes in seq order. It is read-only; concurrent open with an active Writer on the same dir is fine for the rotated segments but the active segment's tail may be incomplete.

func Open

func Open(dir string) (*Reader, error)

Open validates the header and indexes the segments. Iter does the actual streaming; this constructor is cheap.

func (*Reader) Header

func (r *Reader) Header() Header

Header returns the parsed manifest. Useful for diagnostics.

func (*Reader) Iter

func (r *Reader) Iter(yield func(Envelope) bool) error

Iter streams every envelope in seq order. The yield func returns false to stop iteration; the underlying readers are closed before Iter returns.

Malformed JSONL lines are skipped; partial trailing lines (no newline) are also skipped — the writer crashed mid-write and the next Append will either complete or overwrite that line on resume. Behavior matches Go's bufio.Scanner default.

func (*Reader) LastSeq

func (r *Reader) LastSeq() (uint64, error)

LastSeq returns the highest seq seen across all segments. Returns 0 if the journal is empty.

func (*Reader) LastTurnBoundary

func (r *Reader) LastTurnBoundary() (uint64, bool, error)

LastTurnBoundary returns the seq of the most recent agent.turn.end envelope and ok=true if found. Used by the Phase 2 coordinator to detect a partial turn at boot — the unfinished turn lives between this seq and LastSeq.

type ReplayController

type ReplayController interface {
	SetActive(bool)
	Push(eventType string, payload any)
	Reset()
}

ReplayController is the subset of engine.ReplayState the coordinator drives. Same anti-cycle reason as EventEmitter.

type RewindResult

type RewindResult struct {
	// ArchiveName is the directory name (not full path) of the archived
	// pre-rewind journal under <journalDir>/archive/. Pass it back to
	// Restore to swap it in.
	ArchiveName string
	// TruncatedSeq is the highest seq that survived the rewind. Equals
	// the requested ToSeq when the journal contained an envelope with
	// that seq; otherwise it is the largest seq <= ToSeq that was found.
	TruncatedSeq uint64
	// EventsKept is how many envelopes ended up in the new live journal.
	EventsKept int
	// EventsArchived is how many envelopes were in the pre-rewind
	// journal (kept + dropped).
	EventsArchived int
}

RewindResult describes the outcome of a successful Rewind call.

func Rewind

func Rewind(journalDir string, toSeq uint64) (RewindResult, error)

Rewind rewrites journalDir so its live journal ends at the envelope with seq == toSeq (inclusive). The original journal is preserved verbatim under <journalDir>/archive/<timestamp>/, where it can be inspected or swapped back in via Restore.

The session must not have an active Writer attached when Rewind runs; concurrent writes against a journal being rewound produce undefined state. Engine callers tear down the Writer before invoking.

Rewind is idempotent in the sense that calling it twice with the same toSeq leaves the live journal in the same shape; each call simply produces another archive snapshot.

type SeqSource

type SeqSource interface {
	CurrentSeq() uint64
	ParentSeq() uint64
}

SeqSource is the contract the bus implements so journal handlers can retrieve the seq + parent_seq the bus assigned for the current dispatch.

Every EmitEvent / EmitVetoable assigns a seq via an atomic counter and pushes (seq, eventID) onto a per-goroutine stack at dispatch entry. While a wildcard handler runs (the writer's hook), CurrentSeq returns the top of the stack and ParentSeq returns the next entry below — the seq of the event whose handler triggered the current emit.

The interface lives in the journal package (not the engine package) because the writer does not know the engine type, and the engine wires the dependency at construction time.

type SnapshotFile

type SnapshotFile struct {
	// Name is the base name within the journal directory. Callers turn it
	// into a destination path or object key themselves; the journal package
	// knows nothing about either.
	Name string
	// Path is where the bytes actually live. For an immutable segment this is
	// the live file; for the active segment it is the staged copy.
	Path string
	// Staged reports whether Path is a copy Snapshot made (and the caller is
	// responsible for cleaning up) rather than a file in the journal dir.
	Staged bool
	// Size is the file size at capture time.
	Size int64
}

SnapshotFile names one file in a consistent instant of a journal directory.

type SubscribingBus

type SubscribingBus interface {
	EventEmitter
	Subscribe(eventType string, handler func(eventType string, payload any), priority int) (unsubscribe func())
}

SubscribingBus is the bus capability the coordinator needs to install the agent.turn.end synchronization handler. The engine's bus implements it; tests can supply a stub.

type TeardownFilter added in v0.31.0

type TeardownFilter struct {
	// contains filtered or unexported fields
}

TeardownFilter classifies journal envelopes, in seq order, as belonging to a run's teardown or not.

The engine journals its own teardown: core.shutdown, whatever the bus drain dispatches, and everything plugins emit from Shutdown. Consumers that model a run as "what the session did" — replay's response stash, eval promote's generated assertions — must not mistake those trailing envelopes for session work.

An envelope is teardown when it falls between a core.shutdown (inclusive) and the next core.boot (exclusive) — a recalled run appends to the same journal and starts with core.boot — UNLESS it belongs to a turn that was still open when core.shutdown fired. A turn that finishes during the drain is real work: its llm.response / tool.result / hitl.responded answer a request the replayed turn will make again, and its closing agent.turn.end is what marks it complete. Once every such turn has ended, the rest of the window is teardown. A turn that starts inside the window is teardown too.

Open turns are counted by agent.turn.start / agent.turn.end depth, reset at core.boot: a new process has no turns open, whatever an earlier crashed run left unmatched.

The zero value is ready to use. Not safe for concurrent use.

func (*TeardownFilter) Observe added in v0.31.0

func (f *TeardownFilter) Observe(e Envelope) bool

Observe advances the filter past e and reports whether e is teardown. Envelopes must be observed in journal (seq) order.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer is the durable JSONL appender for a single session's journal.

Object-store disposition: turn-boundary-only

The Writer puts bytes under the session tree (journal/events.jsonl) with raw os.* calls and announces nothing on the bus. Unlike every other bypassing writer, that is not a cost/benefit judgement — it is the only option. This writer's *input* is every event dispatched on the bus, so an announcement per journal write would produce an envelope, which would produce another announcement, forever. The Writer holding no bus reference at all is what makes that loop impossible by construction rather than by discipline, and is the reason to leave it that way.

The journal still syncs: the turn-boundary snapshot captures it through Snapshot below, which takes one consistent instant across the active and rotated segments so a snapshot can never contain the same events twice. Recorded in SessionTreeWriters (pkg/engine/session_writers.go).

Writes are decoupled from the bus dispatch goroutine via a buffered channel: the bus's wildcard handler builds an Envelope (with seq + parent_seq supplied by the bus) and pushes it on the channel; a drain goroutine orders envelopes by seq (since wildcard order is child-before-parent for nested emits) and appends them to events.jsonl. Fsync policy is honored per-envelope.

func NewWriter

func NewWriter(dir string, opts WriterOptions) (*Writer, error)

NewWriter creates the journal directory if absent, writes (or validates) the header, opens events.jsonl for append, and starts the drain goroutine.

Multiple writers must not target the same dir; the engine creates exactly one writer per session and that constraint is enforced at the call site.

func (*Writer) Append

func (w *Writer) Append(env *Envelope)

Append queues an envelope for asynchronous write. Blocks if the channel is full so callers feel back-pressure rather than silently losing events. Safe to call concurrently with Close — once closed, subsequent Appends drop.

func (*Writer) Barrier

func (w *Writer) Barrier(ctx context.Context) error

Barrier blocks until the drain goroutine has written, flushed and fsynced every envelope queued before the call.

Appends are asynchronous: the bus's wildcard handler hands an envelope to a channel and returns, so a bus handler running at a turn boundary observes a journal that trails the turn it is reacting to by however many envelopes are still queued. The object-store turn snapshot is exactly such a handler, and without this barrier every snapshot would capture a journal ending mid-turn — making a resumed session re-fire its last input on every single resume.

One caveat, deliberate: the drain writes envelopes in contiguous seq order, so a seq assigned by the bus but not yet handed to Append leaves a gap that blocks everything behind it. Barrier does not wait for such a gap to close — it cannot, since the missing envelope may never arrive — so it guarantees "everything queued before the call, up to the first gap", not "everything the bus has ever numbered".

Returns nil once the writer is closed: Close already flushed and fsynced.

func (*Writer) Close

func (w *Writer) Close(ctx context.Context) error

Close drains in-flight envelopes, fsyncs, and shuts the file. Safe to call multiple times; subsequent calls return nil.

func (*Writer) Closed

func (w *Writer) Closed() bool

Closed reports whether Close has run. Once true, Appends drop and the drain goroutine is on its way out (it has exited once Close returns nil).

func (*Writer) JournalDir

func (w *Writer) JournalDir() string

JournalDir returns the directory passed to NewWriter. Used by Coordinator (Phase 2) and by tests.

func (*Writer) Snapshot

func (w *Writer) Snapshot(stageDir string) ([]SnapshotFile, error)

Snapshot captures a consistent instant of the journal's segment files, staging the mutable active segment into stageDir.

Subdirectories are ignored: the tool-result cache lives at journal/cache/ and is ordinary, independently-written data that a caller walking the session tree picks up on its own. Only the top-level segment files — which rotation mutates as a set — need this treatment.

The caller owns stageDir and must remove it when done. A nil Writer yields no files and no error, so a caller need not special-case a session that has no journal.

func (*Writer) SubscribeProjection

func (w *Writer) SubscribeProjection(types []string, handler func(Envelope)) func()

SubscribeProjection registers a handler that fires synchronously on the writer's drain goroutine after each envelope lands on disk. Pass an empty types slice to receive every envelope; otherwise the handler only fires when env.Type is in the filter set.

Projections run AFTER the disk write, so a handler observing an envelope can rely on it being durable. The trade-off is that a slow handler stalls the drain — keep handlers cheap (append-to-file is fine) and offload anything heavier to a goroutine the handler launches.

Returns an unsubscribe function. Safe to call from any goroutine.

type WriterOptions

type WriterOptions struct {
	FsyncMode     FsyncMode
	RotateBytes   int64
	BufferSize    int
	SchemaVersion string
	SessionID     string
	// InitialSeq is the lowest seq the drain expects to flush. Defaults
	// to 1 for fresh sessions; the engine sets this to LastSeq+1 on
	// recall so the reorder buffer does not stall waiting for a seq the
	// new run will never produce.
	InitialSeq uint64
}

WriterOptions tune the writer.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL