Documentation
¶
Overview ¶
Package flow defines packtrail flow definitions: a declarative graph of nodes (task, choice, fanout, join, await, map, subflow) plus the typed state channels the nodes write to. A Flow is pure data: it is parsed from YAML (or built in Go), validated, hashed and stored immutably; every execution is bound to the hash of the definition it started with.
Validation reports every problem at once; each is a *ValidationError naming the node and YAML field at fault (see ValidationErrors).
packtrail is agnostic: a task is a unit of work executed by a worker of the given kind, whatever that worker does.
Index ¶
- Constants
- func LoadDir(dir string) (map[string]*Flow, error)
- type Cache
- type Channel
- type Concurrency
- type Duration
- type Flow
- func (f *Flow) AttrProgram(name string) *expr.Program
- func (f *Flow) Clone() (*Flow, error)
- func (f *Flow) DeletesOnExpire() bool
- func (f *Flow) FanoutOf(branch string) string
- func (f *Flow) Hash() (string, error)
- func (f *Flow) JSON() ([]byte, error)
- func (f *Flow) JoinOf(fanout string) string
- func (f *Flow) MaxStepsOrDefault() int
- func (f *Flow) Node(id string) *Node
- func (f *Flow) OutputProgram() *expr.Program
- func (f *Flow) StartNode() string
- func (f *Flow) Validate() error
- func (f *Flow) WaitFor(join string) []string
- type Node
- func (n *Node) AllowsDynamic(target string) bool
- func (n *Node) Backoff(attempt int) time.Duration
- func (n *Node) ConcurrencyProgram() *expr.Program
- func (n *Node) InputProgram() *expr.Program
- func (n *Node) JoinKind() (kind string, quorum int)
- func (n *Node) MaxAttempts() int
- func (n *Node) MetaJSON() json.RawMessage
- func (n *Node) OverProgram() *expr.Program
- func (n *Node) ParentClose() string
- func (n *Node) RulePrograms() []*expr.Program
- func (n *Node) ValidateOutput(raw []byte) error
- type Retry
- type Rule
- type Trigger
- type ValidationError
Examples ¶
Constants ¶
const ( NodeTask = "task" NodeChoice = "choice" NodeFanout = "fanout" NodeJoin = "join" NodeAwait = "await" NodeMap = "map" NodeSubflow = "subflow" )
Node types.
const ( JoinAll = "all" JoinAny = "any" JoinQuorum = "quorum" )
Join policies.
const ( BackoffFixed = "fixed" BackoffLinear = "linear" BackoffExponential = "exponential" )
Retry backoff kinds.
const ( ReducerReplace = "replace" ReducerAppend = "append" ReducerMerge = "merge" ReducerSum = "sum" )
Channel reducers.
const ( // ExpireArchive moves the execution to the archive: it stays readable but // can no longer be driven. ExpireArchive = "archive" // ExpireDelete removes the execution and everything derived from it; its // id is free again. ExpireDelete = "delete" )
What expiry (Flow.Retention) does to a terminal execution.
const ( OnErrorDefault = "" OnErrorFail = "fail" )
Choice on_error modes.
const ( ParentCloseCancel = "cancel" ParentCloseAbandon = "abandon" )
Subflow on_parent_close policies.
const ( // MaxRetryAttempts bounds retry.max_attempts, keeping the exponential // backoff shift far from overflow. MaxRetryAttempts = 64 // DefaultMaxSteps is the default recursion limit: the total number of node // entries an execution may perform before it fails with "max_steps". DefaultMaxSteps = 1000 // DefaultRetryDelay is the base delay of a retry policy without delay. DefaultRetryDelay = time.Second // DefaultMapParallel is the max_parallel of a map node that sets none: it // also keeps a single decision well under the decision events limit. DefaultMapParallel = 100 // MaxParallel bounds max_parallel and MaxBranches bounds a fan-out: with // the events each scheduled task needs (entered, scheduled, timeout timer), // one decision stays under the 1 000-event decision limit (F-04). MaxParallel = 256 MaxBranches = 256 // DefaultRetryMaxDelay caps a retry backoff without max_delay. DefaultRetryMaxDelay = 5 * time.Minute )
Defaults and bounds.
const End = "$end"
End is the reserved choice target that completes the execution, with the output of the flow's output expression. It is not a node: it is never entered, counted in steps or visited. In a child execution it ends the child, and the parent continues.
const SupportedVersion = "1"
SupportedVersion is the flow-definition schema version this build accepts. An omitted version means SupportedVersion.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Cache ¶
type Cache struct {
TTL Duration `yaml:"ttl" json:"ttl"`
}
Cache enables the result cache of a task node. A result is stored once the node completes, so concurrent runs with the same key all miss and all run the job (no single-flight).
type Channel ¶
type Channel struct {
// Reducer folds writes: replace (default), append, merge or sum.
Reducer string `yaml:"reducer,omitempty" json:"reducer,omitempty"`
// Default is the initial value.
Default any `yaml:"default,omitempty" json:"default,omitempty"`
}
Channel declares one state channel.
func (Channel) ReducerOrDefault ¶
ReducerOrDefault returns the reducer, defaulting to replace.
type Concurrency ¶
Concurrency bounds how many jobs of a node run at once per key, across all executions.
type Duration ¶
Duration is a time.Duration that (un)marshals as a string like "30s".
func (Duration) MarshalJSON ¶
MarshalJSON encodes the duration as a string.
func (Duration) MarshalText ¶
MarshalText encodes the duration as a string.
func (*Duration) UnmarshalJSON ¶
UnmarshalJSON decodes a duration string ("30s") or a number of nanoseconds.
func (*Duration) UnmarshalText ¶
UnmarshalText decodes a duration string; it is what yaml.v3 uses.
type Flow ¶
type Flow struct {
Version string `yaml:"version,omitempty" json:"version,omitempty"`
Name string `yaml:"name" json:"name"`
Description string `yaml:"description,omitempty" json:"description,omitempty"`
// Start names the entry node. When empty, the entry node is the unique node
// with no inbound transition.
Start string `yaml:"start,omitempty" json:"start,omitempty"`
// Channels declares the typed state of the flow. Nodes write deltas to
// channels and each channel folds them with its reducer.
Channels map[string]Channel `yaml:"channels,omitempty" json:"channels,omitempty"`
// Output is an expression, evaluated on the context when the execution
// completes, that yields its output: an object or null. It picks what the
// execution returns, e.g. "{answer: channels.answer}" to keep working
// memory out, or "results[last_node]" for the last node's result. When
// empty the output is every channel, or the last node's result if the flow
// declares no channels.
Output string `yaml:"output,omitempty" json:"output,omitempty"`
Nodes []Node `yaml:"nodes" json:"nodes"`
// Budget caps generic usage counters reported by workers (and the
// built-in "steps" counter): exceeding one fails the execution with reason
// budget_exceeded.
Budget map[string]float64 `yaml:"budget,omitempty" json:"budget,omitempty"`
// MaxSteps is the recursion limit (total node entries); 0 = DefaultMaxSteps.
MaxSteps int `yaml:"max_steps,omitempty" json:"max_steps,omitempty"`
// SearchAttributes are expressions evaluated on the start input and
// indexed for List/Query.
SearchAttributes map[string]string `yaml:"search_attributes,omitempty" json:"search_attributes,omitempty"`
// Retention is how long a terminal execution is kept before it expires
// (0 = forever). What happens then is OnExpire.
Retention Duration `yaml:"retention,omitempty" json:"retention,omitempty"`
// OnExpire is what expiry does to the execution: ExpireArchive (the
// default) moves it to the archive, where it stays readable; ExpireDelete
// removes it for good, archive included. It needs a Retention.
OnExpire string `yaml:"on_expire,omitempty" json:"on_expire,omitempty"`
// ArchiveRetention deletes an archived execution this long after it was
// archived, by Retention or by hand (0 = keep the archive forever). It
// does not apply with ExpireDelete, which leaves no archive.
ArchiveRetention Duration `yaml:"archive_retention,omitempty" json:"archive_retention,omitempty"`
// Triggers start this flow when a message arrives on a subject.
Triggers []Trigger `yaml:"triggers,omitempty" json:"triggers,omitempty"`
// contains filtered or unexported fields
}
Flow is a flow definition.
func Parse ¶
Parse decodes and validates a flow definition from YAML. Decoding is strict: an unknown field (a typo like "retires:") is an error rather than a silently dropped setting, and extra YAML documents are rejected.
Example ¶
package main
import (
"fmt"
"github.com/henomis/packtrail/flow"
)
func main() {
def, err := flow.Parse([]byte(`
name: greet
channels: {log: {reducer: append}}
nodes:
- {id: hello, type: task, kind: greeter, next: route}
- id: route
type: choice
rules:
- {when: "results.hello.polite", to: bye}
- {default: true, to: hello}
- {id: bye, type: task, kind: greeter}
start: hello
`))
if err != nil {
fmt.Println(err)
return
}
fmt.Println(def.Name, def.StartNode(), def.Node("route").Type, def.Channels["log"].ReducerOrDefault())
}
Output: greet hello choice append
func ParseJSON ¶
ParseJSON decodes and validates a flow definition from its JSON encoding (the form stored in the flows bucket). Unknown fields are rejected.
func (*Flow) AttrProgram ¶
AttrProgram returns the compiled search attribute expression.
func (*Flow) DeletesOnExpire ¶
DeletesOnExpire reports whether an expired execution is deleted instead of archived.
func (*Flow) JSON ¶
JSON returns the canonical JSON encoding of the definition (map keys are sorted by encoding/json, so equal definitions encode equally).
func (*Flow) MaxStepsOrDefault ¶
MaxStepsOrDefault returns the recursion limit.
func (*Flow) OutputProgram ¶
OutputProgram returns the compiled output expression (nil = the default output).
func (*Flow) Validate ¶
Validate checks structural and semantic correctness of the flow and builds the indexes and compiled expressions the engine uses. It is called by Parse and ParseJSON; a Flow built in Go must be validated before use.
Validate reports every problem it finds, each a *ValidationError, joined with errors.Join (list them with ValidationErrors), in a stable order: flow-level problems, then nodes in declaration order, then the graph. Graph checks (fans, start, reachability) run only once every node is valid on its own and every reference resolves.
type Node ¶
type Node struct {
ID string `yaml:"id" json:"id"`
Type string `yaml:"type" json:"type"`
// Next is the static successor. Empty on a terminal node.
Next string `yaml:"next,omitempty" json:"next,omitempty"`
// OnFailure routes a permanent failure of the node (task, map, subflow,
// join) to another node instead of failing the execution: retries
// exhausted, a permanent error, an invalid output, a failed child, a join
// whose policy was not met. The handler sees errors.<node>.
OnFailure string `yaml:"on_failure,omitempty" json:"on_failure,omitempty"`
// Meta is free-form configuration for whoever runs or shows the node: the
// engine ignores it, but it is part of the definition, so it is versioned
// with the flow hash and handed to the worker of each job (Job.Meta). It
// must encode as JSON.
Meta map[string]any `yaml:"meta,omitempty" json:"meta,omitempty"`
// task, map: the worker kind and per-attempt policies.
Kind string `yaml:"kind,omitempty" json:"kind,omitempty"`
Timeout Duration `yaml:"timeout,omitempty" json:"timeout,omitempty"`
Retry *Retry `yaml:"retry,omitempty" json:"retry,omitempty"`
OutputSchema any `yaml:"output_schema,omitempty" json:"output_schema,omitempty"`
Cache *Cache `yaml:"cache,omitempty" json:"cache,omitempty"`
Concurrency *Concurrency `yaml:"concurrency,omitempty" json:"concurrency,omitempty"`
// Dynamic lists the nodes a task worker may route to by returning `next`
// (dynamic edges, LangGraph Command(goto)).
Dynamic []string `yaml:"dynamic,omitempty" json:"dynamic,omitempty"`
// choice
Rules []Rule `yaml:"rules,omitempty" json:"rules,omitempty"`
OnError string `yaml:"on_error,omitempty" json:"on_error,omitempty"`
// fanout
Branches []string `yaml:"branches,omitempty" json:"branches,omitempty"`
// join
WaitFor []string `yaml:"wait_for,omitempty" json:"wait_for,omitempty"`
Policy string `yaml:"policy,omitempty" json:"policy,omitempty"`
// await (Timeout is required)
Signal string `yaml:"signal,omitempty" json:"signal,omitempty"`
OnTimeout string `yaml:"on_timeout,omitempty" json:"on_timeout,omitempty"`
// map
Over string `yaml:"over,omitempty" json:"over,omitempty"`
MaxParallel int `yaml:"max_parallel,omitempty" json:"max_parallel,omitempty"`
// subflow
Flow string `yaml:"flow,omitempty" json:"flow,omitempty"`
Input string `yaml:"input,omitempty" json:"input,omitempty"`
OnParentClose string `yaml:"on_parent_close,omitempty" json:"on_parent_close,omitempty"`
// contains filtered or unexported fields
}
Node is a single node of the graph. Fields are type-specific; Validate enforces which are required for each type.
func (*Node) AllowsDynamic ¶
AllowsDynamic reports whether target is a declared dynamic successor.
func (*Node) Backoff ¶
Backoff returns the delay before attempt (2, 3, …). It never returns less than the base delay nor more than the cap: an overflowed shift must not wrap into a too-short delay.
func (*Node) ConcurrencyProgram ¶
ConcurrencyProgram returns the compiled concurrency key expression.
func (*Node) InputProgram ¶
InputProgram returns the compiled subflow input expression (nil = pass the parent input through).
func (*Node) MaxAttempts ¶
MaxAttempts returns the attempt budget of a task (at least 1).
func (*Node) MetaJSON ¶
func (n *Node) MetaJSON() json.RawMessage
MetaJSON returns the JSON encoding of Meta (nil when there is none).
func (*Node) OverProgram ¶
OverProgram returns the compiled map source expression.
func (*Node) ParentClose ¶
ParentClose returns the subflow on_parent_close policy (default cancel).
func (*Node) RulePrograms ¶
RulePrograms returns the compiled choice predicates (nil for the default).
func (*Node) ValidateOutput ¶
ValidateOutput checks an output against the node's output_schema. A node without a schema accepts any output.
type Retry ¶
type Retry struct {
// MaxAttempts is the total number of attempts (1 = no retry).
MaxAttempts int `yaml:"max_attempts" json:"max_attempts"`
Backoff string `yaml:"backoff,omitempty" json:"backoff,omitempty"`
Delay Duration `yaml:"delay,omitempty" json:"delay,omitempty"`
MaxDelay Duration `yaml:"max_delay,omitempty" json:"max_delay,omitempty"`
}
Retry is a task retry policy.
type Rule ¶
type Rule struct {
When string `yaml:"when,omitempty" json:"when,omitempty"`
Default bool `yaml:"default,omitempty" json:"default,omitempty"`
// To is the node the rule routes to, or End to complete the execution.
To string `yaml:"to" json:"to"`
}
Rule is one branch of a choice node. Exactly one of When / Default is set.
type Trigger ¶
type Trigger struct {
Subject string `yaml:"subject" json:"subject"`
Stream string `yaml:"stream,omitempty" json:"stream,omitempty"`
}
Trigger starts the flow when a message arrives on Subject. When Stream is set the trigger is a durable JetStream consumer of that stream (at-least once; a message that cannot be started is eventually dead-lettered); otherwise a core NATS queue subscription (at-most-once). The execution id is "<flow>-<Nats-Msg-Id>", so a duplicate message starts nothing twice and every flow triggered by the same message runs; without a Msg-Id it is "<flow>-<stream>-<sequence>" (stream) or a fresh id (core).
type ValidationError ¶
type ValidationError struct {
// Flow is the name of the flow being validated.
Flow string
// Node is the id of the node at fault, as written; empty for a
// flow-level problem.
Node string
// Field is the YAML path of the offending field inside Node, or inside
// the flow when Node is empty: "retry.max_attempts", "rules[1].when",
// "channels.c.reducer", "start". Empty when no single field applies.
Field string
// Msg describes the problem, without the flow, node and field.
Msg string
// Err is the underlying cause (an invalid name, an expression or schema
// compile error), or nil.
Err error
}
ValidationError is one problem found by Validate. Validate reports every problem it finds, joined with errors.Join; ValidationErrors lists them.
func ValidationErrors ¶
func ValidationErrors(err error) []*ValidationError
ValidationErrors returns every *ValidationError in err's tree, in order, looking through errors.Join and fmt.Errorf wrapping (such as the file path ParseFile adds). It returns nil if there are none.
Example ¶
package main
import (
"fmt"
"github.com/henomis/packtrail/flow"
)
func main() {
_, err := flow.Parse([]byte(`
name: greet
nodes:
- {id: hello, type: task, kind: greeter, retry: {max_attempts: 100}, next: bye}
- {id: bye, type: task}
`))
for _, e := range flow.ValidationErrors(err) {
fmt.Printf("%s [%s] %s\n", e.Node, e.Field, e.Msg)
}
}
Output: hello [retry.max_attempts] must be between 0 and 64 bye [kind] invalid worker kind "": must match [A-Za-z0-9_-]{1,128}
func (*ValidationError) Error ¶
func (e *ValidationError) Error() string
Error renders the problem as `flow "x": node "a": retry.max_attempts: msg`, omitting the empty parts.
func (*ValidationError) Unwrap ¶
func (e *ValidationError) Unwrap() error
Unwrap returns the underlying cause.