flow

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 4, 2026 License: Apache-2.0 Imports: 19 Imported by: 0

Documentation

Overview

Package flow defines packtrail flow definitions: a declarative graph of nodes (task, choice, fanout, join, await, map, subflow) plus the typed state channels the nodes write to. A Flow is pure data: it is parsed from YAML (or built in Go), validated, hashed and stored immutably; every execution is bound to the hash of the definition it started with.

Validation reports every problem at once; each is a *ValidationError naming the node and YAML field at fault (see ValidationErrors).

packtrail is agnostic: a task is a unit of work executed by a worker of the given kind, whatever that worker does.

Index

Examples

Constants

View Source
const (
	NodeTask    = "task"
	NodeChoice  = "choice"
	NodeFanout  = "fanout"
	NodeJoin    = "join"
	NodeAwait   = "await"
	NodeMap     = "map"
	NodeSubflow = "subflow"
)

Node types.

View Source
const (
	JoinAll    = "all"
	JoinAny    = "any"
	JoinQuorum = "quorum"
)

Join policies.

View Source
const (
	BackoffFixed       = "fixed"
	BackoffLinear      = "linear"
	BackoffExponential = "exponential"
)

Retry backoff kinds.

View Source
const (
	ReducerReplace = "replace"
	ReducerAppend  = "append"
	ReducerMerge   = "merge"
	ReducerSum     = "sum"
)

Channel reducers.

View Source
const (
	OnErrorDefault = ""
	OnErrorFail    = "fail"
)

Choice on_error modes.

View Source
const (
	ParentCloseCancel  = "cancel"
	ParentCloseAbandon = "abandon"
)

Subflow on_parent_close policies.

View Source
const (
	// MaxRetryAttempts bounds retry.max_attempts, keeping the exponential
	// backoff shift far from overflow.
	MaxRetryAttempts = 64
	// DefaultMaxSteps is the default recursion limit: the total number of node
	// entries an execution may perform before it fails with "max_steps".
	DefaultMaxSteps = 1000
	// DefaultRetryDelay is the base delay of a retry policy without delay.
	DefaultRetryDelay = time.Second
	// DefaultMapParallel is the max_parallel of a map node that sets none: it
	// also keeps a single decision well under the decision events limit.
	DefaultMapParallel = 100
	// MaxParallel bounds max_parallel and MaxBranches bounds a fan-out: with
	// the events each scheduled task needs (entered, scheduled, timeout timer),
	// one decision stays under the 1 000-event decision limit (F-04).
	MaxParallel = 256
	MaxBranches = 256
	// DefaultRetryMaxDelay caps a retry backoff without max_delay.
	DefaultRetryMaxDelay = 5 * time.Minute
)

Defaults and bounds.

View Source
const End = "$end"

End is the reserved choice target that completes the execution, with the output of the flow's output expression. It is not a node: it is never entered, counted in steps or visited. In a child execution it ends the child, and the parent continues.

View Source
const SupportedVersion = "1"

SupportedVersion is the flow-definition schema version this build accepts. An omitted version means SupportedVersion.

Variables

This section is empty.

Functions

func LoadDir

func LoadDir(dir string) (map[string]*Flow, error)

LoadDir parses every *.yaml / *.yml file in dir, keyed by flow name.

Types

type Cache

type Cache struct {
	TTL Duration `yaml:"ttl" json:"ttl"`
}

Cache enables the result cache of a task node. A result is stored once the node completes, so concurrent runs with the same key all miss and all run the job (no single-flight).

type Channel

type Channel struct {
	// Reducer folds writes: replace (default), append, merge or sum.
	Reducer string `yaml:"reducer,omitempty" json:"reducer,omitempty"`
	// Default is the initial value.
	Default any `yaml:"default,omitempty" json:"default,omitempty"`
}

Channel declares one state channel.

func (Channel) ReducerOrDefault

func (c Channel) ReducerOrDefault() string

ReducerOrDefault returns the reducer, defaulting to replace.

type Concurrency

type Concurrency struct {
	Key string `yaml:"key" json:"key"`
	Max int    `yaml:"max" json:"max"`
}

Concurrency bounds how many jobs of a node run at once per key, across all executions.

type Duration

type Duration time.Duration

Duration is a time.Duration that (un)marshals as a string like "30s".

func (Duration) D

func (d Duration) D() time.Duration

D returns the value as a time.Duration.

func (Duration) MarshalJSON

func (d Duration) MarshalJSON() ([]byte, error)

MarshalJSON encodes the duration as a string.

func (Duration) MarshalText

func (d Duration) MarshalText() ([]byte, error)

MarshalText encodes the duration as a string.

func (*Duration) UnmarshalJSON

func (d *Duration) UnmarshalJSON(b []byte) error

UnmarshalJSON decodes a duration string ("30s") or a number of nanoseconds.

func (*Duration) UnmarshalText

func (d *Duration) UnmarshalText(b []byte) error

UnmarshalText decodes a duration string; it is what yaml.v3 uses.

type Flow

type Flow struct {
	Version     string `yaml:"version,omitempty" json:"version,omitempty"`
	Name        string `yaml:"name" json:"name"`
	Description string `yaml:"description,omitempty" json:"description,omitempty"`
	// Start names the entry node. When empty, the entry node is the unique node
	// with no inbound transition.
	Start string `yaml:"start,omitempty" json:"start,omitempty"`
	// Channels declares the typed state of the flow. Nodes write deltas to
	// channels and each channel folds them with its reducer.
	Channels map[string]Channel `yaml:"channels,omitempty" json:"channels,omitempty"`
	// Output is an expression, evaluated on the context when the execution
	// completes, that yields its output: an object or null. It picks what the
	// execution returns, e.g. "{answer: channels.answer}" to keep working
	// memory out, or "results[last_node]" for the last node's result. When
	// empty the output is every channel, or the last node's result if the flow
	// declares no channels.
	Output string `yaml:"output,omitempty" json:"output,omitempty"`
	Nodes  []Node `yaml:"nodes" json:"nodes"`
	// Budget caps generic usage counters reported by workers (and the
	// built-in "steps" counter): exceeding one fails the execution with reason
	// budget_exceeded.
	Budget map[string]float64 `yaml:"budget,omitempty" json:"budget,omitempty"`
	// MaxSteps is the recursion limit (total node entries); 0 = DefaultMaxSteps.
	MaxSteps int `yaml:"max_steps,omitempty" json:"max_steps,omitempty"`
	// SearchAttributes are expressions evaluated on the start input and
	// indexed for List/Query.
	SearchAttributes map[string]string `yaml:"search_attributes,omitempty" json:"search_attributes,omitempty"`
	// Retention archives a terminal execution after this long (0 = never).
	Retention Duration `yaml:"retention,omitempty" json:"retention,omitempty"`
	// Triggers start this flow when a message arrives on a subject.
	Triggers []Trigger `yaml:"triggers,omitempty" json:"triggers,omitempty"`
	// contains filtered or unexported fields
}

Flow is a flow definition.

func Parse

func Parse(data []byte) (*Flow, error)

Parse decodes and validates a flow definition from YAML. Decoding is strict: an unknown field (a typo like "retires:") is an error rather than a silently dropped setting, and extra YAML documents are rejected.

Example
package main

import (
	"fmt"

	"github.com/henomis/packtrail/flow"
)

func main() {
	def, err := flow.Parse([]byte(`
name: greet
channels: {log: {reducer: append}}
nodes:
  - {id: hello, type: task, kind: greeter, next: route}
  - id: route
    type: choice
    rules:
      - {when: "results.hello.polite", to: bye}
      - {default: true, to: hello}
  - {id: bye, type: task, kind: greeter}
start: hello
`))
	if err != nil {
		fmt.Println(err)

		return
	}

	fmt.Println(def.Name, def.StartNode(), def.Node("route").Type, def.Channels["log"].ReducerOrDefault())
}
Output:
greet hello choice append

func ParseFile

func ParseFile(path string) (*Flow, error)

ParseFile reads and parses a flow definition file.

func ParseJSON

func ParseJSON(data []byte) (*Flow, error)

ParseJSON decodes and validates a flow definition from its JSON encoding (the form stored in the flows bucket). Unknown fields are rejected.

func (*Flow) AttrProgram

func (f *Flow) AttrProgram(name string) *expr.Program

AttrProgram returns the compiled search attribute expression.

func (*Flow) Clone

func (f *Flow) Clone() (*Flow, error)

Clone returns a deep copy of the definition, validated.

func (*Flow) FanoutOf

func (f *Flow) FanoutOf(branch string) string

FanoutOf returns the fanout owning branch id, or "".

func (*Flow) Hash

func (f *Flow) Hash() (string, error)

Hash returns the content hash identifying this version of the definition.

func (*Flow) JSON

func (f *Flow) JSON() ([]byte, error)

JSON returns the canonical JSON encoding of the definition (map keys are sorted by encoding/json, so equal definitions encode equally).

func (*Flow) JoinOf

func (f *Flow) JoinOf(fanout string) string

JoinOf returns the join node of fanout id.

func (*Flow) MaxStepsOrDefault

func (f *Flow) MaxStepsOrDefault() int

MaxStepsOrDefault returns the recursion limit.

func (*Flow) Node

func (f *Flow) Node(id string) *Node

Node returns the node with the given id, or nil. Valid after Validate.

func (*Flow) OutputProgram

func (f *Flow) OutputProgram() *expr.Program

OutputProgram returns the compiled output expression (nil = the default output).

func (*Flow) StartNode

func (f *Flow) StartNode() string

StartNode returns the entry node id. Valid after Validate.

func (*Flow) Validate

func (f *Flow) Validate() error

Validate checks structural and semantic correctness of the flow and builds the indexes and compiled expressions the engine uses. It is called by Parse and ParseJSON; a Flow built in Go must be validated before use.

Validate reports every problem it finds, each a *ValidationError, joined with errors.Join (list them with ValidationErrors), in a stable order: flow-level problems, then nodes in declaration order, then the graph. Graph checks (fans, start, reachability) run only once every node is valid on its own and every reference resolves.

func (*Flow) WaitFor

func (f *Flow) WaitFor(join string) []string

WaitFor returns the branches a join waits for: its wait_for list, or every branch of its fanout when wait_for is omitted. Valid after Validate.

type Node

type Node struct {
	ID   string `yaml:"id" json:"id"`
	Type string `yaml:"type" json:"type"`
	// Next is the static successor. Empty on a terminal node.
	Next string `yaml:"next,omitempty" json:"next,omitempty"`
	// OnFailure routes a permanent failure of the node (task, map, subflow,
	// join) to another node instead of failing the execution: retries
	// exhausted, a permanent error, an invalid output, a failed child, a join
	// whose policy was not met. The handler sees errors.<node>.
	OnFailure string `yaml:"on_failure,omitempty" json:"on_failure,omitempty"`
	// Meta is free-form configuration for whoever runs or shows the node: the
	// engine ignores it, but it is part of the definition, so it is versioned
	// with the flow hash and handed to the worker of each job (Job.Meta). It
	// must encode as JSON.
	Meta map[string]any `yaml:"meta,omitempty" json:"meta,omitempty"`

	// task, map: the worker kind and per-attempt policies.
	Kind         string       `yaml:"kind,omitempty" json:"kind,omitempty"`
	Timeout      Duration     `yaml:"timeout,omitempty" json:"timeout,omitempty"`
	Retry        *Retry       `yaml:"retry,omitempty" json:"retry,omitempty"`
	OutputSchema any          `yaml:"output_schema,omitempty" json:"output_schema,omitempty"`
	Cache        *Cache       `yaml:"cache,omitempty" json:"cache,omitempty"`
	Concurrency  *Concurrency `yaml:"concurrency,omitempty" json:"concurrency,omitempty"`
	// Dynamic lists the nodes a task worker may route to by returning `next`
	// (dynamic edges, LangGraph Command(goto)).
	Dynamic []string `yaml:"dynamic,omitempty" json:"dynamic,omitempty"`

	// choice
	Rules   []Rule `yaml:"rules,omitempty" json:"rules,omitempty"`
	OnError string `yaml:"on_error,omitempty" json:"on_error,omitempty"`

	// fanout
	Branches []string `yaml:"branches,omitempty" json:"branches,omitempty"`

	// join
	WaitFor []string `yaml:"wait_for,omitempty" json:"wait_for,omitempty"`
	Policy  string   `yaml:"policy,omitempty" json:"policy,omitempty"`

	// await (Timeout is required)
	Signal    string `yaml:"signal,omitempty" json:"signal,omitempty"`
	OnTimeout string `yaml:"on_timeout,omitempty" json:"on_timeout,omitempty"`

	// map
	Over        string `yaml:"over,omitempty" json:"over,omitempty"`
	MaxParallel int    `yaml:"max_parallel,omitempty" json:"max_parallel,omitempty"`

	// subflow
	Flow          string `yaml:"flow,omitempty" json:"flow,omitempty"`
	Input         string `yaml:"input,omitempty" json:"input,omitempty"`
	OnParentClose string `yaml:"on_parent_close,omitempty" json:"on_parent_close,omitempty"`
	// contains filtered or unexported fields
}

Node is a single node of the graph. Fields are type-specific; Validate enforces which are required for each type.

func (*Node) AllowsDynamic

func (n *Node) AllowsDynamic(target string) bool

AllowsDynamic reports whether target is a declared dynamic successor.

func (*Node) Backoff

func (n *Node) Backoff(attempt int) time.Duration

Backoff returns the delay before attempt (2, 3, …). It never returns less than the base delay nor more than the cap: an overflowed shift must not wrap into a too-short delay.

func (*Node) ConcurrencyProgram

func (n *Node) ConcurrencyProgram() *expr.Program

ConcurrencyProgram returns the compiled concurrency key expression.

func (*Node) InputProgram

func (n *Node) InputProgram() *expr.Program

InputProgram returns the compiled subflow input expression (nil = pass the parent input through).

func (*Node) JoinKind

func (n *Node) JoinKind() (kind string, quorum int)

JoinKind returns the join policy and, for quorum, the required count.

func (*Node) MaxAttempts

func (n *Node) MaxAttempts() int

MaxAttempts returns the attempt budget of a task (at least 1).

func (*Node) MetaJSON

func (n *Node) MetaJSON() json.RawMessage

MetaJSON returns the JSON encoding of Meta (nil when there is none).

func (*Node) OverProgram

func (n *Node) OverProgram() *expr.Program

OverProgram returns the compiled map source expression.

func (*Node) ParentClose

func (n *Node) ParentClose() string

ParentClose returns the subflow on_parent_close policy (default cancel).

func (*Node) RulePrograms

func (n *Node) RulePrograms() []*expr.Program

RulePrograms returns the compiled choice predicates (nil for the default).

func (*Node) ValidateOutput

func (n *Node) ValidateOutput(raw []byte) error

ValidateOutput checks an output against the node's output_schema. A node without a schema accepts any output.

type Retry

type Retry struct {
	// MaxAttempts is the total number of attempts (1 = no retry).
	MaxAttempts int      `yaml:"max_attempts" json:"max_attempts"`
	Backoff     string   `yaml:"backoff,omitempty" json:"backoff,omitempty"`
	Delay       Duration `yaml:"delay,omitempty" json:"delay,omitempty"`
	MaxDelay    Duration `yaml:"max_delay,omitempty" json:"max_delay,omitempty"`
}

Retry is a task retry policy.

type Rule

type Rule struct {
	When    string `yaml:"when,omitempty" json:"when,omitempty"`
	Default bool   `yaml:"default,omitempty" json:"default,omitempty"`
	// To is the node the rule routes to, or End to complete the execution.
	To string `yaml:"to" json:"to"`
}

Rule is one branch of a choice node. Exactly one of When / Default is set.

type Trigger

type Trigger struct {
	Subject string `yaml:"subject" json:"subject"`
	Stream  string `yaml:"stream,omitempty" json:"stream,omitempty"`
}

Trigger starts the flow when a message arrives on Subject. When Stream is set the trigger is a durable JetStream consumer of that stream (at-least once; a message that cannot be started is eventually dead-lettered); otherwise a core NATS queue subscription (at-most-once). The execution id is "<flow>-<Nats-Msg-Id>", so a duplicate message starts nothing twice and every flow triggered by the same message runs; without a Msg-Id it is "<flow>-<stream>-<sequence>" (stream) or a fresh id (core).

type ValidationError

type ValidationError struct {
	// Flow is the name of the flow being validated.
	Flow string
	// Node is the id of the node at fault, as written; empty for a
	// flow-level problem.
	Node string
	// Field is the YAML path of the offending field inside Node, or inside
	// the flow when Node is empty: "retry.max_attempts", "rules[1].when",
	// "channels.c.reducer", "start". Empty when no single field applies.
	Field string
	// Msg describes the problem, without the flow, node and field.
	Msg string
	// Err is the underlying cause (an invalid name, an expression or schema
	// compile error), or nil.
	Err error
}

ValidationError is one problem found by Validate. Validate reports every problem it finds, joined with errors.Join; ValidationErrors lists them.

func ValidationErrors

func ValidationErrors(err error) []*ValidationError

ValidationErrors returns every *ValidationError in err's tree, in order, looking through errors.Join and fmt.Errorf wrapping (such as the file path ParseFile adds). It returns nil if there are none.

Example
package main

import (
	"fmt"

	"github.com/henomis/packtrail/flow"
)

func main() {
	_, err := flow.Parse([]byte(`
name: greet
nodes:
  - {id: hello, type: task, kind: greeter, retry: {max_attempts: 100}, next: bye}
  - {id: bye, type: task}
`))

	for _, e := range flow.ValidationErrors(err) {
		fmt.Printf("%s [%s] %s\n", e.Node, e.Field, e.Msg)
	}
}
Output:
hello [retry.max_attempts] must be between 0 and 64
bye [kind] invalid worker kind "": must match [A-Za-z0-9_-]{1,128}

func (*ValidationError) Error

func (e *ValidationError) Error() string

Error renders the problem as `flow "x": node "a": retry.max_attempts: msg`, omitting the empty parts.

func (*ValidationError) Unwrap

func (e *ValidationError) Unwrap() error

Unwrap returns the underlying cause.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL