Help us shape the future of Go! Take the 2026 Go Developer Survey

runtime

package
v0.66.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 24, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package runtime is the flow engine: it executes one flow over one batch of event-log messages and produces the CommitBatch that batch is worth.

It runs in Go, in-process, and it is the whole of graph execution — topological order, port routing, filter matching, script evaluation, terminal sink tagging and node-run accounting. Nothing about it is transport-specific, which is the point: the editor preview, a CLI dry-run and an MCP tool all reach the same execution rather than the desktop window owning one of them.

The engine never writes the database and never opens its own connection. Run takes the messages it is given, reads durable node KV only through the KVReader port the caller supplies, and returns what should be committed — including buffered KV writes — for the caller to apply. That keeps a dry-run (no-op reader, discarded batch) and a live tick the same code path, differing only in what the ports are wired to.

Index

Constants

View Source
const DefaultMaxConcurrentScripts = 32

DefaultMaxConcurrentScripts bounds how many script evaluations may be in flight at once, process-wide.

View Source
const DefaultPageSize = 500

DefaultPageSize is how many log records one drain pass reads at a time.

View Source
const DefaultScriptLanguage = "javascript"

DefaultScriptLanguage is the language a function node is written in. There is no `runtime:` field on the node yet — the port and registry are what make a second language possible, and the config field can follow when one exists.

View Source
const UnroutedNodeID = "$unrouted"

UnroutedNodeID is the Discard.NodeID recorded for a message that matched no entry node — its topic belongs to another flow's source, say. It is still accounted for, because a message nobody claims must not stop the consumer offset advancing past it.

Variables

This section is empty.

Functions

func SyntheticMsgID

func SyntheticMsgID(index int) string

SyntheticMsgID is the id given to an injected message the caller left unidentified, suffixed with its position in the batch. A message needs some id for the discards a drop records to name it; a dry run commits nothing, so unlike a live offset it only has to be distinct within the run.

Types

type CommitStore

type CommitStore interface {
	// Commit applies one run's outputs and advances the offset, atomically.
	Commit(ctx context.Context, batch models.CommitBatch) error
	// ActivateReplay installs a prepared replay: claims, removed structure,
	// node-KV reconciliation, and the consumer checkpoint, in one transaction.
	ActivateReplay(ctx context.Context, profileID string, tail int64, claims []models.FeedClaim, feedIDs, sourceIDs, kvNodeIDs []string) error
}

type ConsoleLine

type ConsoleLine struct {
	Level string `json:"level"`
	Text  string `json:"text"`
}

ConsoleLine is one console.* call from a function node.

type ConsoleSink

type ConsoleSink func(level, text string)

ConsoleSink receives one console.* call from a script. A nil sink discards the line, which is what a live run passes: console output is a debugging affordance for a dry run, not a second log stream on the hot path. It is invoked from the evaluating goroutine, which for a wedged script may be one the engine has already abandoned — an implementation must be safe to call after the evaluation it belongs to has returned.

func (ConsoleSink) Log

func (s ConsoleSink) Log(level, text string)

Log writes one line, tolerating a nil sink so callers never branch.

type DryRunResult

type DryRunResult struct {
	// Nodes is one entry per node a message reached, in execution order.
	Nodes []NodeTrace `json:"nodes"`
	// Outputs is what the terminals would have committed — feed memberships,
	// notifications and enqueued actions — had this been a live run.
	Outputs []models.Output `json:"outputs"`
	// FeedSnapshots is the reconciliation scope each snapshot input declared.
	// An empty snapshot still produces one: it is how a clean poll clears a
	// feed, so a dry run has to show it.
	FeedSnapshots []models.FeedSnapshot `json:"feedSnapshots"`
	// KVMutations is what the run would have written to durable node KV. The
	// values are the sandbox's, so this reads as "given the kv you seeded, here
	// is what the flow would store".
	KVMutations []models.KVMutation `json:"kvMutations"`
	// ConsoleTruncated reports that the run produced more console output than
	// it kept.
	ConsoleTruncated bool `json:"consoleTruncated,omitempty"`
}

DryRunResult is not a models.CommitBatch, so callers cannot accidentally commit a preview.

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine runs every enabled flow against the event log. One per process: the producer and the output worker are singletons for the same reason, and a second engine would double-execute every flow.

It is level-triggered. Wake and Reload set a latch rather than queueing, so a signal that arrives while a pass is already running is serviced by the next pass instead of being dropped or piling up. That is the property the browser runtime had to reconstruct by hand, and losing it means a message sits unrouted until something else happens to wake the engine.

func NewEngine

func NewEngine(opts EngineOptions) *Engine

NewEngine builds the engine. Nothing runs until Start.

func (*Engine) Reload

func (e *Engine) Reload()

Reload requests that the flow set be re-read and every runner reinstalled. It implies a drain.

func (*Engine) Start

func (e *Engine) Start(ctx context.Context) error

Start installs every enabled flow and then runs the drain loop.

Installation is synchronous, before Start returns. Everything that can append to the log — the producer, the webhook listener, the e2e harness — is started after this, so there is no window in which an append can arrive before the engine is able to route it. The browser runtime had exactly that window, and closing it needed a readiness latch the frontend published for tests to gate on.

func (*Engine) Stop

func (e *Engine) Stop()

Stop ends the drain loop and releases every runner. An in-flight page finishes; no next page starts.

func (*Engine) Wake

func (e *Engine) Wake()

Wake requests a drain. It never blocks: the request is a latch, so a wake arriving mid-pass is serviced by the next one.

type EngineOptions

type EngineOptions struct {
	Log     LogStore
	Items   InboxReader
	Commits CommitStore
	KV      KVReader
	Flows   Flows
	Scripts *ScriptRegistry
	Logger  zerolog.Logger
	// PageSize bounds one read. Zero means DefaultPageSize.
	PageSize int
	// Events receives InboxUpdated after a pass commits at least one flow.
	// Appending an unrouted log message does not emit this event.
	Events *events.Bus
	// Recorder reports install failures while the engine keeps the last known-good
	// flow active.
	Recorder activity.Recorder
}

EngineOptions configure an Engine. Only Logger and PageSize are optional.

type Flows

type Flows interface {
	List() []flow.Flow
}

Flows is the engine's view of the flow set: whatever loaded successfully, which is what it can run.

type Graph

type Graph struct {
	// contains filtered or unexported fields
}

Graph is the execution-order index derived once from a flow: which nodes are entry points, which wires leave each output port, and the topological order Run walks. Building it is the only place a flow's shape is inspected, so Run itself never searches the node or wire slices.

Wires with either endpoint unknown are dropped here rather than reported: flow.LoadFlow owns reference diagnostics, and an engine that refused to run on a dangling wire would take a flow offline for a problem its editor already surfaces.

func NewGraph

func NewGraph(f flow.Flow) (*Graph, error)

NewGraph indexes f for execution. It returns an error only for a cycle: a flow that is not a DAG has no execution order at all, and looping forever on one is worse than refusing it.

func (*Graph) Entries

func (g *Graph) Entries() []string

Entries returns the entry node ids in flow declaration order.

func (*Graph) Node

func (g *Graph) Node(id string) *flow.Node

Node returns the node with this id, or nil.

func (*Graph) Order

func (g *Graph) Order() []string

Order returns the topological execution order.

func (*Graph) Wires

func (g *Graph) Wires(nodeID string, port int) []flow.Wire

Wires returns the wires leaving nodeID on port.

type InboxReader

type InboxReader interface {
	ListUnarchived(ctx context.Context, profileID string) ([]stores.InboxItem, error)
}

InboxReader excludes archived items because replay must preserve their frozen memberships.

type KVReader

type KVReader interface {
	Get(ctx context.Context, flowID, nodeID, key string, now int64) (value string, found bool, err error)
	Keys(ctx context.Context, flowID, nodeID, prefix string, now int64) ([]string, error)
}

KVReader provides durable node-scoped reads. Callers pass one Unix-millisecond cutoff per Run so expiry is deterministic.

type LogStore

type LogStore interface {
	// ReadForConsumer returns the next page after a consumer's committed
	// offset, with payloads only for the routed topics.
	ReadForConsumer(ctx context.Context, consumer string, limit int, routed models.RoutedTopics) ([]models.Msg, error)
	// TailOffset is the log's high-water mark, the point a replay
	// fast-forwards its consumer to.
	TailOffset(ctx context.Context) (int64, error)
	// ListLatestSnapshots returns each source's newest authoritative snapshot
	// at or before an offset.
	ListLatestSnapshots(ctx context.Context, profileID string, throughOffset int64) ([]models.Msg, error)
}

type MemoryKV

type MemoryKV map[string]map[string]string

MemoryKV is a KVReader over an in-memory seed: node id -> key -> the JSON text NodeKV stores. It is what makes a dry run's `kv` a sandbox — reads see only what the caller seeded, and the writes a run produces come back as mutations instead of reaching the database.

Expiry is not modelled: a seed is a starting state a caller wrote by hand, so there is nothing older than the run itself to expire.

func (MemoryKV) Get

func (m MemoryKV) Get(_ context.Context, _, nodeID, key string, _ int64) (string, bool, error)

func (MemoryKV) Keys

func (m MemoryKV) Keys(_ context.Context, _, nodeID, prefix string, _ int64) ([]string, error)

type NodeError

type NodeError struct {
	Message string          `json:"message"`
	Kind    ScriptErrorKind `json:"kind,omitempty"`
	Line    int             `json:"line,omitempty"`
	Column  int             `json:"column,omitempty"`
	Stack   string          `json:"stack,omitempty"`
}

NodeError is a node failure as a dry run reports it. Kind, Line and Column are set only for a script failure — everything else carries Message alone.

type NodeKV

type NodeKV interface {
	Get(ctx context.Context, key string) (value string, found bool, err error)
	Set(ctx context.Context, key, value string, ttlSeconds int64) error
	Has(ctx context.Context, key string) (bool, error)
	Delete(ctx context.Context, key string) error
	Keys(ctx context.Context, prefix string) ([]string, error)
}

NodeKV is the node-scoped surface a function node's `kv` object is bound to. Values are opaque JSON text; the js layer owns (de)serialization.

type NodeTrace

type NodeTrace struct {
	NodeID string `json:"nodeId"`
	Type   string `json:"type"`
	// Disabled marks a node that dropped everything without running, which is
	// otherwise indistinguishable from a node that chose to drop everything.
	Disabled bool `json:"disabled,omitempty"`
	// In, Out and Dropped are the same counters a live run records: messages
	// received, messages emitted or committed, and messages this node ended.
	In      int `json:"in"`
	Out     int `json:"out"`
	Dropped int `json:"dropped"`
	// OK is false when at least one message failed here.
	OK bool `json:"ok"`
	// Error is the last failure, structured when the node is a script.
	Error *NodeError `json:"error,omitempty"`
	// DurationMs is the total time spent in this node's processor.
	DurationMs int64 `json:"durationMs"`
	// Received is every message that arrived at this node's input, snapshots
	// already expanded into their items.
	Received []models.Msg `json:"received"`
	// Emitted is what the node put on each of its output ports, in port order.
	// A port with no wire behind it is still reported — what the node produced
	// does not depend on what is listening. Terminals emit nothing; read
	// Received and the run's Outputs for those.
	Emitted []PortEmission `json:"emitted"`
	// Console is the node's console output, oldest first.
	Console []ConsoleLine `json:"console,omitempty"`
}

NodeTrace is what one node did during a dry run.

type Options

type Options struct {
	// Scripts is the registry function nodes resolve their language through.
	Scripts *ScriptRegistry
	// KV is the durable node-KV read port; nil means every read misses
	// (preview, dry-run, tests).
	KV KVReader
}

Options configure a Runner. The zero value is usable: it runs a flow with no script runtimes registered, which is correct for a flow with no function nodes and an explicit error for one that has them.

type PortEmission

type PortEmission struct {
	Port     int          `json:"port"`
	Messages []models.Msg `json:"messages"`
}

PortEmission is one output port's messages.

type Runner

type Runner struct {
	// contains filtered or unexported fields
}

Runner executes one flow, repeatedly. It owns the per-node-instance state that has to survive between batches — a function node's `state` object, most importantly — so a flow that is pumped page by page behaves as one continuous run rather than resetting on every page.

A Runner is not safe for concurrent use. One flow is one consumer of one ordered log, so running two batches of the same flow at once would reorder it; the caller serialises.

func NewRunner

func NewRunner(f flow.Flow, opts Options) (*Runner, error)

NewRunner indexes f and builds its processors. It fails on a flow that is not a DAG, on a node type the engine cannot execute, and on a function node whose script does not compile — all three are conditions under which running the flow at all would be worse than reporting it.

func (*Runner) Close

func (r *Runner) Close()

Close releases every processor's resources.

func (*Runner) DryRun

func (r *Runner) DryRun(ctx context.Context, entryNodeID string, batch []models.Msg) (DryRunResult, error)

DryRun executes the flow over messages injected at entryNodeID and reports what every node downstream of it did, without producing anything the caller could commit.

Injection is by node id rather than by topic on purpose: it is what lets one function node be exercised against a captured payload without standing up the source in front of it. A source node is a relay at run time — the core ingests for it — so injecting at one is the same as that source having polled and returned exactly these messages, and nothing fetches.

The run is as read-only as the ports it is given: durable KV is whatever Options.KV reads (seed a MemoryKV to sandbox it), and the mutations and outputs it would have produced come back in the result instead of being applied. Call it on a Runner built for this run alone — a Runner carries each function node's `state` object between messages, so dry-running through the engine's installed one would mutate live node state.

func (*Runner) Run

func (r *Runner) Run(ctx context.Context, batch []models.Msg) (models.CommitBatch, error)

Run executes the flow over batch and returns the commit it is worth.

Every input message is accounted for exactly once — as a tagged terminal output, as a discard, or (when a node failed) as an errored discard — so the consumer offset in the returned batch can always advance past the whole page. That is the invariant the commit protocol's idempotency rests on: a message the graph has no use for still moves the cursor.

Run only prepares a batch; it does not persist outputs or advance the consumer.

func (*Runner) RunReplay

func (r *Runner) RunReplay(ctx context.Context, batch []models.Msg) (models.CommitBatch, error)

RunReplay recomputes membership with a fully inert KV — durable and overlay reads miss, staged writes are discarded — so replay stays a pure function of (current snapshots, current graph) and never suppresses items via dedup history. It resets the processors on the way out, so state mutated during the recompute never reaches the next live Run.

type ScriptError

type ScriptError struct {
	Kind    ScriptErrorKind `json:"kind"`
	Message string          `json:"message"`
	Line    int             `json:"line,omitempty"`
	Column  int             `json:"column,omitempty"`
	// Stack is the language's own trace, kept whole for the editor. It is
	// never parsed.
	Stack string `json:"stack,omitempty"`
}

ScriptError is the normalized failure every ScriptRuntime reports. Line and Column are 1-based and zero when the language could not attribute the failure to a position.

func (*ScriptError) Error

func (e *ScriptError) Error() string

type ScriptErrorKind

type ScriptErrorKind string

ScriptErrorKind classifies a script failure. The editor renders every language's diagnostics identically, so the vocabulary is fixed here rather than being whatever a given engine happens to report.

const (
	// ScriptErrorCompile is a syntax or compile-time failure.
	ScriptErrorCompile ScriptErrorKind = "compile"
	// ScriptErrorRuntime is a value thrown while the script ran.
	ScriptErrorRuntime ScriptErrorKind = "runtime"
	// ScriptErrorTimeout is an evaluation that outlived the node's timeout.
	ScriptErrorTimeout ScriptErrorKind = "timeout"
	// ScriptErrorResult is a return value the language adapter could not
	// resolve into port-indexed messages.
	ScriptErrorResult ScriptErrorKind = "result"
	// ScriptErrorUnavailable is the script pool refusing to start another
	// evaluation because too many are already wedged.
	ScriptErrorUnavailable ScriptErrorKind = "unavailable"
)

type ScriptInstance

type ScriptInstance interface {
	// OnMessage evaluates the script for msg and returns its outputs indexed
	// by output port. config is the node's own config, passed as the script's
	// `node` argument; kv is the node's per-message durable-KV handle and
	// console its per-message console sink. A returned error is always a
	// *ScriptError.
	OnMessage(ctx context.Context, msg models.Msg, config any, kv NodeKV, console ConsoleSink) ([][]models.Msg, error)
	// Close releases the instance. An instance whose evaluation timed out may
	// still be running, in which case Close interrupts it and returns without
	// waiting.
	Close()
}

ScriptInstance is one node instance's live script: its compiled program and the state object that survives across messages for as long as the instance does. Not safe for concurrent use — the engine evaluates one message at a time per instance.

type ScriptPool

type ScriptPool struct {
	// contains filtered or unexported fields
}

ScriptPool bounds concurrent script evaluations. One pool is shared by every registered ScriptRuntime, which is why it is passed to them at construction rather than owned by any one of them.

A cooperative interrupt cannot stop every pathological script — a tight loop that neither allocates nor calls back into the host may keep running after its timeout fires. A runtime therefore evaluates on a goroutine it is willing to abandon, and holds its slot until that goroutine actually returns. Bounding the pool converts "one bad node hangs the process" into "one bad node exhausts a fixed budget and every later evaluation fails fast", which is a limitation the user can see rather than a hang they cannot.

func NewScriptPool

func NewScriptPool(maxConcurrent int) *ScriptPool

NewScriptPool returns a pool admitting at most max concurrent evaluations. A non-positive max uses DefaultMaxConcurrentScripts.

func (*ScriptPool) Acquire

func (p *ScriptPool) Acquire() bool

Acquire takes a slot, or reports false immediately when none is free. It never blocks: an evaluation that has to queue behind wedged goroutines would just add its own timeout to the caller's.

func (*ScriptPool) Available

func (p *ScriptPool) Available() int

Available reports how many evaluations could start right now.

func (*ScriptPool) Release

func (p *ScriptPool) Release()

Release returns a slot. It is only ever called by an evaluation goroutine that actually finished.

type ScriptRegistry

type ScriptRegistry struct {
	// contains filtered or unexported fields
}

ScriptRegistry maps a language name to its runtime. It is an explicit registry rather than init() self-registration: gochecknoinits is enabled, and a registry that is assembled in one visible place is the point.

func NewScriptRegistry

func NewScriptRegistry() *ScriptRegistry

NewScriptRegistry returns an empty registry.

func (*ScriptRegistry) Lookup

func (r *ScriptRegistry) Lookup(name string) (ScriptRuntime, bool)

Lookup returns the runtime registered under name.

func (*ScriptRegistry) Register

func (r *ScriptRegistry) Register(rt ScriptRuntime)

Register adds rt under its own name. Registering a name twice is a programming error and panics at wiring time rather than silently shadowing.

type ScriptRuntime

type ScriptRuntime interface {
	// Name is the language this runtime registers under.
	Name() string
	// Check reports whether src compiles, without running it. The editor's
	// live syntax check is a core concern, so it goes through the same
	// compiler the engine uses rather than a second parser.
	Check(src string) error
	// New compiles src into an instance with its own isolated state. outputs
	// is the node's declared output-port count: it is what disambiguates a
	// language's overloaded return shapes, so the engine only ever receives
	// port-indexed values.
	New(src string, outputs int) (ScriptInstance, error)
}

ScriptRuntime is a driven port: one language a function node can be written in. It is defined here because the engine is its consumer; implementations live in their own packages (runtime/js is the goja one) and satisfy it structurally.

Adding a language is an implementation plus one Register call. The engine never learns a second language's name, so nothing in run.go changes.

Directories

Path Synopsis
Package js implements the engine's ScriptRuntime port with goja, a pure-Go ECMAScript interpreter.
Package js implements the engine's ScriptRuntime port with goja, a pure-Go ECMAScript interpreter.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL