Help us shape the future of Go! Take the 2026 Go Developer Survey

flow

package
v1.0.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 19, 2026 License: Apache-2.0 Imports: 19 Imported by: 0

Documentation

Overview

Package flow defines packtrail flow definitions: a declarative graph of nodes (task, choice, fanout, join, await, map, subflow) plus the typed state channels the nodes write to. A Flow is pure data: it is parsed from YAML (or built in Go), validated, hashed and stored immutably; every execution is bound to the hash of the definition it started with.

Validation reports every problem at once; each is a *ValidationError naming the node and YAML field at fault (see ValidationErrors).

packtrail is agnostic: a task is a unit of work executed by a worker of the given kind, whatever that worker does.

Index

Examples

Constants

View Source
const (
	NodeTask    = "task"
	NodeChoice  = "choice"
	NodeFanout  = "fanout"
	NodeJoin    = "join"
	NodeAwait   = "await"
	NodeMap     = "map"
	NodeSubflow = "subflow"
)

Node types.

View Source
const (
	JoinAll    = "all"
	JoinAny    = "any"
	JoinQuorum = "quorum"
	JoinMin    = "min"
)

Join policies.

View Source
const (
	BackoffFixed       = "fixed"
	BackoffLinear      = "linear"
	BackoffExponential = "exponential"
)

Retry backoff kinds.

View Source
const (
	ReducerReplace = "replace"
	ReducerAppend  = "append"
	ReducerMerge   = "merge"
	ReducerSum     = "sum"
)

Channel reducers.

View Source
const (
	// ExpireArchive moves the execution to the archive: it stays readable but
	// can no longer be driven.
	ExpireArchive = "archive"
	// ExpireDelete removes the execution and everything derived from it; its
	// id is free again.
	ExpireDelete = "delete"
)

What expiry (Flow.Retention) does to a terminal execution.

View Source
const (
	OnErrorDefault = ""
	OnErrorFail    = "fail"
)

Choice on_error modes.

View Source
const (
	ParentCloseCancel  = "cancel"
	ParentCloseAbandon = "abandon"
)

Subflow on_parent_close policies.

View Source
const (
	// MaxRetryAttempts bounds retry.max_attempts, keeping the exponential
	// backoff shift far from overflow.
	MaxRetryAttempts = 64
	// DefaultMaxSteps is the default recursion limit: the total number of node
	// entries an execution may perform before it fails with "max_steps".
	DefaultMaxSteps = 1000
	// DefaultRetryDelay is the base delay of a retry policy without delay.
	DefaultRetryDelay = time.Second
	// DefaultMapParallel is the max_parallel of a map node that sets none: it
	// also keeps a single decision well under the decision events limit.
	DefaultMapParallel = 100
	// MaxParallel bounds max_parallel and MaxBranches bounds a fan-out: with
	// the events each scheduled task needs (entered, scheduled, timeout timer),
	// one decision stays under the 1 000-event decision limit (F-04).
	MaxParallel = 256
	MaxBranches = 256
	// DefaultRetryMaxDelay caps a retry backoff without max_delay.
	DefaultRetryMaxDelay = 5 * time.Minute
)

Defaults and bounds.

View Source
const End = "$end"

End is the reserved choice target that completes the execution, with the output of the flow's output expression. It is not a node: it is never entered, counted in steps or visited. In a child execution it ends the child, and the parent continues.

View Source
const SupportedVersion = "1"

SupportedVersion is the flow-definition schema version this build accepts. An omitted version means SupportedVersion.

Variables

This section is empty.

Functions

func LoadDir

func LoadDir(dir string) (map[string]*Flow, error)

LoadDir parses every *.yaml / *.yml file in dir, keyed by flow name.

Types

type Cache

type Cache struct {
	TTL Duration `yaml:"ttl" json:"ttl"`
}

Cache enables the result cache of a task node. A result is stored once the node completes, so concurrent runs with the same key all miss and all run the job (no single-flight).

type Channel

type Channel struct {
	// Reducer folds writes: replace (default), append, merge or sum.
	Reducer string `yaml:"reducer,omitempty" json:"reducer,omitempty"`
	// Default is the initial value.
	Default any `yaml:"default,omitempty" json:"default,omitempty"`
}

Channel declares one state channel.

func (Channel) ReducerOrDefault

func (c Channel) ReducerOrDefault() string

ReducerOrDefault returns the reducer, defaulting to replace.

type Concurrency

type Concurrency struct {
	Key string `yaml:"key" json:"key"`
	Max int    `yaml:"max" json:"max"`
}

Concurrency bounds how many jobs of a node run at once per key, across all executions.

type Duration

type Duration time.Duration

Duration is a time.Duration that (un)marshals as a string like "30s".

func (Duration) D

func (d Duration) D() time.Duration

D returns the value as a time.Duration.

func (Duration) MarshalJSON

func (d Duration) MarshalJSON() ([]byte, error)

MarshalJSON encodes the duration as a string.

func (Duration) MarshalText

func (d Duration) MarshalText() ([]byte, error)

MarshalText encodes the duration as a string.

func (*Duration) UnmarshalJSON

func (d *Duration) UnmarshalJSON(b []byte) error

UnmarshalJSON decodes a duration string ("30s") or a number of nanoseconds.

func (*Duration) UnmarshalText

func (d *Duration) UnmarshalText(b []byte) error

UnmarshalText decodes a duration string; it is what yaml.v3 uses.

type Flow

type Flow struct {
	Version     string `yaml:"version,omitempty" json:"version,omitempty"`
	Name        string `yaml:"name" json:"name"`
	Description string `yaml:"description,omitempty" json:"description,omitempty"`
	// Start names the entry node. When empty, the entry node is the unique node
	// with no inbound transition.
	Start string `yaml:"start,omitempty" json:"start,omitempty"`
	// Channels declares the typed state of the flow. Nodes write deltas to
	// channels and each channel folds them with its reducer.
	Channels map[string]Channel `yaml:"channels,omitempty" json:"channels,omitempty"`
	// Output is an expression, evaluated on the context when the execution
	// completes, that yields its output: an object or null. It picks what the
	// execution returns, e.g. "{answer: channels.answer}" to keep working
	// memory out, or "results[last_node]" for the last node's result. When
	// empty the output is every channel, or the last node's result if the flow
	// declares no channels.
	Output string `yaml:"output,omitempty" json:"output,omitempty"`
	Nodes  []Node `yaml:"nodes" json:"nodes"`
	// Budget caps generic usage counters reported by workers (and the
	// built-in "steps" counter): exceeding one fails the execution with reason
	// budget_exceeded.
	Budget map[string]float64 `yaml:"budget,omitempty" json:"budget,omitempty"`
	// MaxSteps is the recursion limit (total node entries); 0 = DefaultMaxSteps.
	MaxSteps int `yaml:"max_steps,omitempty" json:"max_steps,omitempty"`
	// SearchAttributes are expressions evaluated on the start input and
	// indexed for List/Query.
	SearchAttributes map[string]string `yaml:"search_attributes,omitempty" json:"search_attributes,omitempty"`
	// Retention is how long a terminal execution is kept before it expires
	// (0 = forever). What happens then is OnExpire.
	Retention Duration `yaml:"retention,omitempty" json:"retention,omitempty"`
	// OnExpire is what expiry does to the execution: ExpireArchive (the
	// default) moves it to the archive, where it stays readable; ExpireDelete
	// removes it for good, archive included. It needs a Retention.
	OnExpire string `yaml:"on_expire,omitempty" json:"on_expire,omitempty"`
	// ArchiveRetention deletes an archived execution this long after it was
	// archived, by Retention or by hand (0 = keep the archive forever). It
	// does not apply with ExpireDelete, which leaves no archive.
	ArchiveRetention Duration `yaml:"archive_retention,omitempty" json:"archive_retention,omitempty"`
	// Triggers start this flow when a message arrives on a subject.
	Triggers []Trigger `yaml:"triggers,omitempty" json:"triggers,omitempty"`
	// contains filtered or unexported fields
}

Flow is a flow definition.

func Parse

func Parse(data []byte) (*Flow, error)

Parse decodes and validates a flow definition from YAML. Decoding is strict: an unknown field (a typo like "retires:") is an error rather than a silently dropped setting, and extra YAML documents are rejected.

Example
package main

import (
	"fmt"

	"github.com/henomis/packtrail/flow"
)

func main() {
	def, err := flow.Parse([]byte(`
name: greet
channels: {log: {reducer: append}}
nodes:
  - {id: hello, type: task, kind: greeter, next: route}
  - id: route
    type: choice
    rules:
      - {when: "results.hello.polite", to: bye}
      - {default: true, to: hello}
  - {id: bye, type: task, kind: greeter}
start: hello
`))
	if err != nil {
		fmt.Println(err)

		return
	}

	fmt.Println(def.Name, def.StartNode(), def.Node("route").Type, def.Channels["log"].ReducerOrDefault())
}
Output:
greet hello choice append

func ParseFile

func ParseFile(path string) (*Flow, error)

ParseFile reads and parses a flow definition file.

func ParseJSON

func ParseJSON(data []byte) (*Flow, error)

ParseJSON decodes and validates a flow definition from its JSON encoding (the form stored in the flows bucket). Unknown fields are rejected.

func (*Flow) AttrProgram

func (f *Flow) AttrProgram(name string) *expr.Program

AttrProgram returns the compiled search attribute expression.

func (*Flow) Clone

func (f *Flow) Clone() (*Flow, error)

Clone returns a deep copy of the definition, validated.

func (*Flow) DeletesOnExpire added in v1.0.1

func (f *Flow) DeletesOnExpire() bool

DeletesOnExpire reports whether an expired execution is deleted instead of archived.

func (*Flow) FanoutOf

func (f *Flow) FanoutOf(branch string) string

FanoutOf returns the fanout owning branch id, or "".

func (*Flow) Hash

func (f *Flow) Hash() (string, error)

Hash returns the content hash identifying this version of the definition.

func (*Flow) JSON

func (f *Flow) JSON() ([]byte, error)

JSON returns the canonical JSON encoding of the definition (map keys are sorted by encoding/json, so equal definitions encode equally).

func (*Flow) JoinOf

func (f *Flow) JoinOf(fanout string) string

JoinOf returns the join node of fanout id.

func (*Flow) MaxStepsOrDefault

func (f *Flow) MaxStepsOrDefault() int

MaxStepsOrDefault returns the recursion limit.

func (*Flow) Node

func (f *Flow) Node(id string) *Node

Node returns the node with the given id, or nil. Valid after Validate.

func (*Flow) OutputProgram

func (f *Flow) OutputProgram() *expr.Program

OutputProgram returns the compiled output expression (nil = the default output).

func (*Flow) StartNode

func (f *Flow) StartNode() string

StartNode returns the entry node id. Valid after Validate.

func (*Flow) Validate

func (f *Flow) Validate() error

Validate checks structural and semantic correctness of the flow and builds the indexes and compiled expressions the engine uses. It is called by Parse and ParseJSON; a Flow built in Go must be validated before use.

Validate reports every problem it finds, each a *ValidationError, joined with errors.Join (list them with ValidationErrors), in a stable order: flow-level problems, then nodes in declaration order, then the graph. Graph checks (fans, start, reachability) run only once every node is valid on its own and every reference resolves.

func (*Flow) WaitFor

func (f *Flow) WaitFor(join string) []string

WaitFor returns the branches a join waits for: its wait_for list, or every branch of its fanout when wait_for is omitted. Valid after Validate.

type Node

type Node struct {
	ID   string `yaml:"id" json:"id"`
	Type string `yaml:"type" json:"type"`
	// Next is the static successor. Empty on a terminal node.
	Next string `yaml:"next,omitempty" json:"next,omitempty"`
	// OnFailure routes a permanent failure of the node (task, map, subflow,
	// join) to another node instead of failing the execution: retries
	// exhausted, a permanent error, an invalid output, a failed child, a join
	// whose policy was not met. The handler sees errors.<node>.
	OnFailure string `yaml:"on_failure,omitempty" json:"on_failure,omitempty"`
	// Meta is free-form configuration for whoever runs or shows the node: the
	// engine ignores it, but it is part of the definition, so it is versioned
	// with the flow hash and handed to the worker of each job (Job.Meta). It
	// must encode as JSON.
	Meta map[string]any `yaml:"meta,omitempty" json:"meta,omitempty"`

	// task, map: the worker kind and per-attempt policies.
	Kind         string       `yaml:"kind,omitempty" json:"kind,omitempty"`
	Timeout      Duration     `yaml:"timeout,omitempty" json:"timeout,omitempty"`
	Retry        *Retry       `yaml:"retry,omitempty" json:"retry,omitempty"`
	OutputSchema any          `yaml:"output_schema,omitempty" json:"output_schema,omitempty"`
	Cache        *Cache       `yaml:"cache,omitempty" json:"cache,omitempty"`
	Concurrency  *Concurrency `yaml:"concurrency,omitempty" json:"concurrency,omitempty"`
	// Dynamic lists the nodes a task worker may route to by returning `next`
	// (dynamic edges, LangGraph Command(goto)).
	Dynamic []string `yaml:"dynamic,omitempty" json:"dynamic,omitempty"`

	// choice
	Rules   []Rule `yaml:"rules,omitempty" json:"rules,omitempty"`
	OnError string `yaml:"on_error,omitempty" json:"on_error,omitempty"`

	// fanout
	Branches []string `yaml:"branches,omitempty" json:"branches,omitempty"`

	// join
	WaitFor []string `yaml:"wait_for,omitempty" json:"wait_for,omitempty"`
	// Policy is all (default), any or quorum:N, which settle as soon as the
	// outcome is known and cancel the branches still running, or min:N,
	// which waits for every branch and succeeds when N of them did.
	Policy string `yaml:"policy,omitempty" json:"policy,omitempty"`

	// await (Timeout is required)
	Signal    string `yaml:"signal,omitempty" json:"signal,omitempty"`
	OnTimeout string `yaml:"on_timeout,omitempty" json:"on_timeout,omitempty"`

	// map
	Over        string `yaml:"over,omitempty" json:"over,omitempty"`
	MaxParallel int    `yaml:"max_parallel,omitempty" json:"max_parallel,omitempty"`

	// subflow
	Flow          string `yaml:"flow,omitempty" json:"flow,omitempty"`
	Input         string `yaml:"input,omitempty" json:"input,omitempty"`
	OnParentClose string `yaml:"on_parent_close,omitempty" json:"on_parent_close,omitempty"`
	// contains filtered or unexported fields
}

Node is a single node of the graph. Fields are type-specific; Validate enforces which are required for each type.

func (*Node) AllowsDynamic

func (n *Node) AllowsDynamic(target string) bool

AllowsDynamic reports whether target is a declared dynamic successor.

func (*Node) Backoff

func (n *Node) Backoff(attempt int) time.Duration

Backoff returns the delay before attempt (2, 3, …). It never returns less than the base delay nor more than the cap: an overflowed shift must not wrap into a too-short delay.

func (*Node) ConcurrencyProgram

func (n *Node) ConcurrencyProgram() *expr.Program

ConcurrencyProgram returns the compiled concurrency key expression.

func (*Node) InputProgram

func (n *Node) InputProgram() *expr.Program

InputProgram returns the compiled subflow input expression (nil = pass the parent input through).

func (*Node) JoinKind

func (n *Node) JoinKind() (kind string, count int)

JoinKind returns the join policy and, for quorum and min, the required count (-1 when it is not a number).

func (*Node) MaxAttempts

func (n *Node) MaxAttempts() int

MaxAttempts returns the attempt budget of a task (at least 1).

func (*Node) MetaJSON

func (n *Node) MetaJSON() json.RawMessage

MetaJSON returns the JSON encoding of Meta (nil when there is none).

func (*Node) OverProgram

func (n *Node) OverProgram() *expr.Program

OverProgram returns the compiled map source expression.

func (*Node) ParentClose

func (n *Node) ParentClose() string

ParentClose returns the subflow on_parent_close policy (default cancel).

func (*Node) RulePrograms

func (n *Node) RulePrograms() []*expr.Program

RulePrograms returns the compiled choice predicates (nil for the default).

func (*Node) ValidateOutput

func (n *Node) ValidateOutput(raw []byte) error

ValidateOutput checks an output against the node's output_schema. A node without a schema accepts any output.

type Retry

type Retry struct {
	// MaxAttempts is the total number of attempts (1 = no retry).
	MaxAttempts int      `yaml:"max_attempts" json:"max_attempts"`
	Backoff     string   `yaml:"backoff,omitempty" json:"backoff,omitempty"`
	Delay       Duration `yaml:"delay,omitempty" json:"delay,omitempty"`
	MaxDelay    Duration `yaml:"max_delay,omitempty" json:"max_delay,omitempty"`
}

Retry is a task retry policy.

type Rule

type Rule struct {
	// Name labels the rule in ChoiceEvaluated events and timelines; optional,
	// unique within its node.
	Name    string `yaml:"name,omitempty" json:"name,omitempty"`
	When    string `yaml:"when,omitempty" json:"when,omitempty"`
	Default bool   `yaml:"default,omitempty" json:"default,omitempty"`
	// To is the node the rule routes to, or End to complete the execution.
	To string `yaml:"to" json:"to"`
}

Rule is one branch of a choice node. Exactly one of When / Default is set.

type Trigger

type Trigger struct {
	Subject string `yaml:"subject" json:"subject"`
	Stream  string `yaml:"stream,omitempty" json:"stream,omitempty"`
}

Trigger starts the flow when a message arrives on Subject. When Stream is set the trigger is a durable JetStream consumer of that stream (at-least once; a message that cannot be started is eventually dead-lettered); otherwise a core NATS queue subscription (at-most-once). The execution id is "<flow>-<Nats-Msg-Id>", so a duplicate message starts nothing twice and every flow triggered by the same message runs; without a Msg-Id it is "<flow>-<stream>-<sequence>" (stream) or a fresh id (core).

type ValidationError

type ValidationError struct {
	// Flow is the name of the flow being validated.
	Flow string
	// Node is the id of the node at fault, as written; empty for a
	// flow-level problem.
	Node string
	// Field is the YAML path of the offending field inside Node, or inside
	// the flow when Node is empty: "retry.max_attempts", "rules[1].when",
	// "channels.c.reducer", "start". Empty when no single field applies.
	Field string
	// Msg describes the problem, without the flow, node and field.
	Msg string
	// Err is the underlying cause (an invalid name, an expression or schema
	// compile error), or nil.
	Err error
}

ValidationError is one problem found by Validate. Validate reports every problem it finds, joined with errors.Join; ValidationErrors lists them.

func ValidationErrors

func ValidationErrors(err error) []*ValidationError

ValidationErrors returns every *ValidationError in err's tree, in order, looking through errors.Join and fmt.Errorf wrapping (such as the file path ParseFile adds). It returns nil if there are none.

Example
package main

import (
	"fmt"

	"github.com/henomis/packtrail/flow"
)

func main() {
	_, err := flow.Parse([]byte(`
name: greet
nodes:
  - {id: hello, type: task, kind: greeter, retry: {max_attempts: 100}, next: bye}
  - {id: bye, type: task}
`))

	for _, e := range flow.ValidationErrors(err) {
		fmt.Printf("%s [%s] %s\n", e.Node, e.Field, e.Msg)
	}
}
Output:
hello [retry.max_attempts] must be between 0 and 64
bye [kind] invalid worker kind "": must match [A-Za-z0-9_-]{1,128}

func (*ValidationError) Error

func (e *ValidationError) Error() string

Error renders the problem as `flow "x": node "a": retry.max_attempts: msg`, omitting the empty parts.

func (*ValidationError) Unwrap

func (e *ValidationError) Unwrap() error

Unwrap returns the underlying cause.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL