Documentation
¶
Overview ¶
Package runtime is the flow engine: it executes one flow over one batch of event-log messages and produces the CommitBatch that batch is worth.
It runs in Go, in-process, and it is the whole of graph execution — topological order, port routing, filter matching, script evaluation, terminal sink tagging and node-run accounting. Nothing about it is transport-specific, which is the point: the editor preview, a CLI dry-run and an MCP tool all reach the same execution rather than the desktop window owning one of them.
The engine never writes the database and never opens its own connection. Run takes the messages it is given, reads durable node KV only through the KVReader port the caller supplies, and returns what should be committed — including buffered KV writes — for the caller to apply. That keeps a dry-run (no-op reader, discarded batch) and a live tick the same code path, differing only in what the ports are wired to.
Index ¶
- Constants
- func SyntheticMsgID(index int) string
- type CommitStore
- type ConsoleLine
- type ConsoleSink
- type DryRunResult
- type Engine
- type EngineOptions
- type Flows
- type Graph
- type InboxReader
- type KVReader
- type LogStore
- type MemoryKV
- type NodeError
- type NodeKV
- type NodeTrace
- type Options
- type PortEmission
- type Runner
- func (r *Runner) Close()
- func (r *Runner) DryRun(ctx context.Context, entryNodeID string, batch []models.Msg) (DryRunResult, error)
- func (r *Runner) Run(ctx context.Context, batch []models.Msg) (models.CommitBatch, error)
- func (r *Runner) RunReplay(ctx context.Context, batch []models.Msg) (models.CommitBatch, error)
- type ScriptError
- type ScriptErrorKind
- type ScriptInstance
- type ScriptPool
- type ScriptRegistry
- type ScriptRuntime
Constants ¶
const DefaultMaxConcurrentScripts = 32
DefaultMaxConcurrentScripts bounds how many script evaluations may be in flight at once, process-wide.
const DefaultPageSize = 500
DefaultPageSize is how many log records one drain pass reads at a time.
const DefaultScriptLanguage = "javascript"
DefaultScriptLanguage is the language a function node is written in. There is no `runtime:` field on the node yet — the port and registry are what make a second language possible, and the config field can follow when one exists.
const UnroutedNodeID = "$unrouted"
UnroutedNodeID is the Discard.NodeID recorded for a message that matched no entry node — its topic belongs to another flow's source, say. It is still accounted for, because a message nobody claims must not stop the consumer offset advancing past it.
Variables ¶
This section is empty.
Functions ¶
func SyntheticMsgID ¶
SyntheticMsgID is the id given to an injected message the caller left unidentified, suffixed with its position in the batch. A message needs some id for the discards a drop records to name it; a dry run commits nothing, so unlike a live offset it only has to be distinct within the run.
Types ¶
type CommitStore ¶
type CommitStore interface {
// Commit applies one run's outputs and advances the offset, atomically.
Commit(ctx context.Context, batch models.CommitBatch) error
// ActivateReplay installs a prepared replay: claims, removed structure,
// node-KV reconciliation, and the consumer checkpoint, in one transaction.
ActivateReplay(ctx context.Context, profileID string, tail int64, claims []models.FeedClaim, feedIDs, sourceIDs, kvNodeIDs []string) error
}
type ConsoleLine ¶
ConsoleLine is one console.* call from a function node.
type ConsoleSink ¶
type ConsoleSink func(level, text string)
ConsoleSink receives one console.* call from a script. A nil sink discards the line, which is what a live run passes: console output is a debugging affordance for a dry run, not a second log stream on the hot path. It is invoked from the evaluating goroutine, which for a wedged script may be one the engine has already abandoned — an implementation must be safe to call after the evaluation it belongs to has returned.
func (ConsoleSink) Log ¶
func (s ConsoleSink) Log(level, text string)
Log writes one line, tolerating a nil sink so callers never branch.
type DryRunResult ¶
type DryRunResult struct {
// Nodes is one entry per node a message reached, in execution order.
Nodes []NodeTrace `json:"nodes"`
// Outputs is what the terminals would have committed — feed memberships,
// notifications and enqueued actions — had this been a live run.
Outputs []models.Output `json:"outputs"`
// FeedSnapshots is the reconciliation scope each snapshot input declared.
// An empty snapshot still produces one: it is how a clean poll clears a
// feed, so a dry run has to show it.
FeedSnapshots []models.FeedSnapshot `json:"feedSnapshots"`
// KVMutations is what the run would have written to durable node KV. The
// values are the sandbox's, so this reads as "given the kv you seeded, here
// is what the flow would store".
KVMutations []models.KVMutation `json:"kvMutations"`
// ConsoleTruncated reports that the run produced more console output than
// it kept.
ConsoleTruncated bool `json:"consoleTruncated,omitempty"`
}
DryRunResult is not a models.CommitBatch, so callers cannot accidentally commit a preview.
type Engine ¶
type Engine struct {
// contains filtered or unexported fields
}
Engine runs every enabled flow against the event log. One per process: the producer and the output worker are singletons for the same reason, and a second engine would double-execute every flow.
It is level-triggered. Wake and Reload set a latch rather than queueing, so a signal that arrives while a pass is already running is serviced by the next pass instead of being dropped or piling up. That is the property the browser runtime had to reconstruct by hand, and losing it means a message sits unrouted until something else happens to wake the engine.
func NewEngine ¶
func NewEngine(opts EngineOptions) *Engine
NewEngine builds the engine. Nothing runs until Start.
func (*Engine) Reload ¶
func (e *Engine) Reload()
Reload requests that the flow set be re-read and every runner reinstalled. It implies a drain.
func (*Engine) Start ¶
Start installs every enabled flow and then runs the drain loop.
Installation is synchronous, before Start returns. Everything that can append to the log — the producer, the webhook listener, the e2e harness — is started after this, so there is no window in which an append can arrive before the engine is able to route it. The browser runtime had exactly that window, and closing it needed a readiness latch the frontend published for tests to gate on.
type EngineOptions ¶
type EngineOptions struct {
Log LogStore
Items InboxReader
Commits CommitStore
KV KVReader
Flows Flows
Scripts *ScriptRegistry
Logger zerolog.Logger
// PageSize bounds one read. Zero means DefaultPageSize.
PageSize int
// Events receives InboxUpdated after a pass commits at least one flow.
// Appending an unrouted log message does not emit this event.
Events *events.Bus
// Recorder reports install failures while the engine keeps the last known-good
// flow active.
Recorder activity.Recorder
}
EngineOptions configure an Engine. Only Logger and PageSize are optional.
type Flows ¶
Flows is the engine's view of the flow set: whatever loaded successfully, which is what it can run.
type Graph ¶
type Graph struct {
// contains filtered or unexported fields
}
Graph is the execution-order index derived once from a flow: which nodes are entry points, which wires leave each output port, and the topological order Run walks. Building it is the only place a flow's shape is inspected, so Run itself never searches the node or wire slices.
Wires with either endpoint unknown are dropped here rather than reported: flow.LoadFlow owns reference diagnostics, and an engine that refused to run on a dangling wire would take a flow offline for a problem its editor already surfaces.
func NewGraph ¶
NewGraph indexes f for execution. It returns an error only for a cycle: a flow that is not a DAG has no execution order at all, and looping forever on one is worse than refusing it.
type InboxReader ¶
type InboxReader interface {
ListUnarchived(ctx context.Context, profileID string) ([]stores.InboxItem, error)
}
InboxReader excludes archived items because replay must preserve their frozen memberships.
type KVReader ¶
type KVReader interface {
Get(ctx context.Context, flowID, nodeID, key string, now int64) (value string, found bool, err error)
Keys(ctx context.Context, flowID, nodeID, prefix string, now int64) ([]string, error)
}
KVReader provides durable node-scoped reads. Callers pass one Unix-millisecond cutoff per Run so expiry is deterministic.
type LogStore ¶
type LogStore interface {
// ReadForConsumer returns the next page after a consumer's committed
// offset, with payloads only for the routed topics.
ReadForConsumer(ctx context.Context, consumer string, limit int, routed models.RoutedTopics) ([]models.Msg, error)
// TailOffset is the log's high-water mark, the point a replay
// fast-forwards its consumer to.
TailOffset(ctx context.Context) (int64, error)
// ListLatestSnapshots returns each source's newest authoritative snapshot
// at or before an offset.
ListLatestSnapshots(ctx context.Context, profileID string, throughOffset int64) ([]models.Msg, error)
}
type MemoryKV ¶
MemoryKV is a KVReader over an in-memory seed: node id -> key -> the JSON text NodeKV stores. It is what makes a dry run's `kv` a sandbox — reads see only what the caller seeded, and the writes a run produces come back as mutations instead of reaching the database.
Expiry is not modelled: a seed is a starting state a caller wrote by hand, so there is nothing older than the run itself to expire.
type NodeError ¶
type NodeError struct {
Message string `json:"message"`
Kind ScriptErrorKind `json:"kind,omitempty"`
Line int `json:"line,omitempty"`
Column int `json:"column,omitempty"`
Stack string `json:"stack,omitempty"`
}
NodeError is a node failure as a dry run reports it. Kind, Line and Column are set only for a script failure — everything else carries Message alone.
type NodeKV ¶
type NodeKV interface {
Get(ctx context.Context, key string) (value string, found bool, err error)
Set(ctx context.Context, key, value string, ttlSeconds int64) error
Has(ctx context.Context, key string) (bool, error)
Delete(ctx context.Context, key string) error
Keys(ctx context.Context, prefix string) ([]string, error)
}
NodeKV is the node-scoped surface a function node's `kv` object is bound to. Values are opaque JSON text; the js layer owns (de)serialization.
type NodeTrace ¶
type NodeTrace struct {
NodeID string `json:"nodeId"`
Type string `json:"type"`
// Disabled marks a node that dropped everything without running, which is
// otherwise indistinguishable from a node that chose to drop everything.
Disabled bool `json:"disabled,omitempty"`
// In, Out and Dropped are the same counters a live run records: messages
// received, messages emitted or committed, and messages this node ended.
In int `json:"in"`
Out int `json:"out"`
Dropped int `json:"dropped"`
// OK is false when at least one message failed here.
OK bool `json:"ok"`
// Error is the last failure, structured when the node is a script.
Error *NodeError `json:"error,omitempty"`
// DurationMs is the total time spent in this node's processor.
DurationMs int64 `json:"durationMs"`
// Received is every message that arrived at this node's input, snapshots
// already expanded into their items.
Received []models.Msg `json:"received"`
// Emitted is what the node put on each of its output ports, in port order.
// A port with no wire behind it is still reported — what the node produced
// does not depend on what is listening. Terminals emit nothing; read
// Received and the run's Outputs for those.
Emitted []PortEmission `json:"emitted"`
// Console is the node's console output, oldest first.
Console []ConsoleLine `json:"console,omitempty"`
}
NodeTrace is what one node did during a dry run.
type Options ¶
type Options struct {
// Scripts is the registry function nodes resolve their language through.
Scripts *ScriptRegistry
// KV is the durable node-KV read port; nil means every read misses
// (preview, dry-run, tests).
KV KVReader
}
Options configure a Runner. The zero value is usable: it runs a flow with no script runtimes registered, which is correct for a flow with no function nodes and an explicit error for one that has them.
type PortEmission ¶
PortEmission is one output port's messages.
type Runner ¶
type Runner struct {
// contains filtered or unexported fields
}
Runner executes one flow, repeatedly. It owns the per-node-instance state that has to survive between batches — a function node's `state` object, most importantly — so a flow that is pumped page by page behaves as one continuous run rather than resetting on every page.
A Runner is not safe for concurrent use. One flow is one consumer of one ordered log, so running two batches of the same flow at once would reorder it; the caller serialises.
func NewRunner ¶
NewRunner indexes f and builds its processors. It fails on a flow that is not a DAG, on a node type the engine cannot execute, and on a function node whose script does not compile — all three are conditions under which running the flow at all would be worse than reporting it.
func (*Runner) DryRun ¶
func (r *Runner) DryRun(ctx context.Context, entryNodeID string, batch []models.Msg) (DryRunResult, error)
DryRun executes the flow over messages injected at entryNodeID and reports what every node downstream of it did, without producing anything the caller could commit.
Injection is by node id rather than by topic on purpose: it is what lets one function node be exercised against a captured payload without standing up the source in front of it. A source node is a relay at run time — the core ingests for it — so injecting at one is the same as that source having polled and returned exactly these messages, and nothing fetches.
The run is as read-only as the ports it is given: durable KV is whatever Options.KV reads (seed a MemoryKV to sandbox it), and the mutations and outputs it would have produced come back in the result instead of being applied. Call it on a Runner built for this run alone — a Runner carries each function node's `state` object between messages, so dry-running through the engine's installed one would mutate live node state.
func (*Runner) Run ¶
Run executes the flow over batch and returns the commit it is worth.
Every input message is accounted for exactly once — as a tagged terminal output, as a discard, or (when a node failed) as an errored discard — so the consumer offset in the returned batch can always advance past the whole page. That is the invariant the commit protocol's idempotency rests on: a message the graph has no use for still moves the cursor.
Run only prepares a batch; it does not persist outputs or advance the consumer.
func (*Runner) RunReplay ¶
RunReplay recomputes membership with a fully inert KV — durable and overlay reads miss, staged writes are discarded — so replay stays a pure function of (current snapshots, current graph) and never suppresses items via dedup history. It resets the processors on the way out, so state mutated during the recompute never reaches the next live Run.
type ScriptError ¶
type ScriptError struct {
Kind ScriptErrorKind `json:"kind"`
Message string `json:"message"`
Line int `json:"line,omitempty"`
Column int `json:"column,omitempty"`
// Stack is the language's own trace, kept whole for the editor. It is
// never parsed.
Stack string `json:"stack,omitempty"`
}
ScriptError is the normalized failure every ScriptRuntime reports. Line and Column are 1-based and zero when the language could not attribute the failure to a position.
func (*ScriptError) Error ¶
func (e *ScriptError) Error() string
type ScriptErrorKind ¶
type ScriptErrorKind string
ScriptErrorKind classifies a script failure. The editor renders every language's diagnostics identically, so the vocabulary is fixed here rather than being whatever a given engine happens to report.
const ( // ScriptErrorCompile is a syntax or compile-time failure. ScriptErrorCompile ScriptErrorKind = "compile" // ScriptErrorRuntime is a value thrown while the script ran. ScriptErrorRuntime ScriptErrorKind = "runtime" // ScriptErrorTimeout is an evaluation that outlived the node's timeout. ScriptErrorTimeout ScriptErrorKind = "timeout" // ScriptErrorResult is a return value the language adapter could not // resolve into port-indexed messages. ScriptErrorResult ScriptErrorKind = "result" // evaluation because too many are already wedged. ScriptErrorUnavailable ScriptErrorKind = "unavailable" )
type ScriptInstance ¶
type ScriptInstance interface {
// OnMessage evaluates the script for msg and returns its outputs indexed
// by output port. config is the node's own config, passed as the script's
// `node` argument; kv is the node's per-message durable-KV handle and
// console its per-message console sink. A returned error is always a
// *ScriptError.
OnMessage(ctx context.Context, msg models.Msg, config any, kv NodeKV, console ConsoleSink) ([][]models.Msg, error)
// Close releases the instance. An instance whose evaluation timed out may
// still be running, in which case Close interrupts it and returns without
// waiting.
Close()
}
ScriptInstance is one node instance's live script: its compiled program and the state object that survives across messages for as long as the instance does. Not safe for concurrent use — the engine evaluates one message at a time per instance.
type ScriptPool ¶
type ScriptPool struct {
// contains filtered or unexported fields
}
ScriptPool bounds concurrent script evaluations. One pool is shared by every registered ScriptRuntime, which is why it is passed to them at construction rather than owned by any one of them.
A cooperative interrupt cannot stop every pathological script — a tight loop that neither allocates nor calls back into the host may keep running after its timeout fires. A runtime therefore evaluates on a goroutine it is willing to abandon, and holds its slot until that goroutine actually returns. Bounding the pool converts "one bad node hangs the process" into "one bad node exhausts a fixed budget and every later evaluation fails fast", which is a limitation the user can see rather than a hang they cannot.
func NewScriptPool ¶
func NewScriptPool(maxConcurrent int) *ScriptPool
NewScriptPool returns a pool admitting at most max concurrent evaluations. A non-positive max uses DefaultMaxConcurrentScripts.
func (*ScriptPool) Acquire ¶
func (p *ScriptPool) Acquire() bool
Acquire takes a slot, or reports false immediately when none is free. It never blocks: an evaluation that has to queue behind wedged goroutines would just add its own timeout to the caller's.
func (*ScriptPool) Available ¶
func (p *ScriptPool) Available() int
Available reports how many evaluations could start right now.
func (*ScriptPool) Release ¶
func (p *ScriptPool) Release()
Release returns a slot. It is only ever called by an evaluation goroutine that actually finished.
type ScriptRegistry ¶
type ScriptRegistry struct {
// contains filtered or unexported fields
}
ScriptRegistry maps a language name to its runtime. It is an explicit registry rather than init() self-registration: gochecknoinits is enabled, and a registry that is assembled in one visible place is the point.
func NewScriptRegistry ¶
func NewScriptRegistry() *ScriptRegistry
NewScriptRegistry returns an empty registry.
func (*ScriptRegistry) Lookup ¶
func (r *ScriptRegistry) Lookup(name string) (ScriptRuntime, bool)
Lookup returns the runtime registered under name.
func (*ScriptRegistry) Register ¶
func (r *ScriptRegistry) Register(rt ScriptRuntime)
Register adds rt under its own name. Registering a name twice is a programming error and panics at wiring time rather than silently shadowing.
type ScriptRuntime ¶
type ScriptRuntime interface {
// Name is the language this runtime registers under.
Name() string
// Check reports whether src compiles, without running it. The editor's
// live syntax check is a core concern, so it goes through the same
// compiler the engine uses rather than a second parser.
Check(src string) error
// New compiles src into an instance with its own isolated state. outputs
// is the node's declared output-port count: it is what disambiguates a
// language's overloaded return shapes, so the engine only ever receives
// port-indexed values.
New(src string, outputs int) (ScriptInstance, error)
}
ScriptRuntime is a driven port: one language a function node can be written in. It is defined here because the engine is its consumer; implementations live in their own packages (runtime/js is the goja one) and satisfy it structurally.
Adding a language is an implementation plus one Register call. The engine never learns a second language's name, so nothing in run.go changes.