Documentation
¶
Overview ¶
Package flow defines packtrail flow definitions: a declarative graph of nodes (task, choice, fanout, join, await, map, subflow) plus the typed state channels the nodes write to. A Flow is pure data: it is parsed from YAML (or built in Go), validated, hashed and stored immutably; every execution is bound to the hash of the definition it started with.
packtrail is agnostic: a task is a unit of work executed by a worker of the given kind, whatever that worker does.
Index ¶
- Constants
- func LoadDir(dir string) (map[string]*Flow, error)
- type Cache
- type Channel
- type Concurrency
- type Duration
- type Flow
- func (f *Flow) AttrProgram(name string) *expr.Program
- func (f *Flow) Clone() (*Flow, error)
- func (f *Flow) FanoutOf(branch string) string
- func (f *Flow) Hash() (string, error)
- func (f *Flow) JSON() ([]byte, error)
- func (f *Flow) JoinOf(fanout string) string
- func (f *Flow) MaxStepsOrDefault() int
- func (f *Flow) Node(id string) *Node
- func (f *Flow) StartNode() string
- func (f *Flow) Validate() error
- func (f *Flow) WaitFor(join string) []string
- type Node
- func (n *Node) AllowsDynamic(target string) bool
- func (n *Node) Backoff(attempt int) time.Duration
- func (n *Node) ConcurrencyProgram() *expr.Program
- func (n *Node) InputProgram() *expr.Program
- func (n *Node) JoinKind() (kind string, quorum int)
- func (n *Node) MaxAttempts() int
- func (n *Node) MetaJSON() json.RawMessage
- func (n *Node) OverProgram() *expr.Program
- func (n *Node) ParentClose() string
- func (n *Node) RulePrograms() []*expr.Program
- func (n *Node) ValidateOutput(raw []byte) error
- type Retry
- type Rule
- type Trigger
Examples ¶
Constants ¶
const ( NodeTask = "task" NodeChoice = "choice" NodeFanout = "fanout" NodeJoin = "join" NodeAwait = "await" NodeMap = "map" NodeSubflow = "subflow" )
Node types.
const ( JoinAll = "all" JoinAny = "any" JoinQuorum = "quorum" )
Join policies.
const ( BackoffFixed = "fixed" BackoffLinear = "linear" BackoffExponential = "exponential" )
Retry backoff kinds.
const ( ReducerReplace = "replace" ReducerAppend = "append" ReducerMerge = "merge" ReducerSum = "sum" )
Channel reducers.
const ( OnErrorDefault = "" OnErrorFail = "fail" )
Choice on_error modes.
const ( ParentCloseCancel = "cancel" ParentCloseAbandon = "abandon" )
Subflow on_parent_close policies.
const ( // MaxRetryAttempts bounds retry.max_attempts, keeping the exponential // backoff shift far from overflow. MaxRetryAttempts = 64 // DefaultMaxSteps is the default recursion limit: the total number of node // entries an execution may perform before it fails with "max_steps". DefaultMaxSteps = 1000 // DefaultRetryDelay is the base delay of a retry policy without delay. DefaultRetryDelay = time.Second // DefaultMapParallel is the max_parallel of a map node that sets none: it // also keeps a single decision well under the decision events limit. DefaultMapParallel = 100 // MaxParallel bounds max_parallel and MaxBranches bounds a fan-out: with // the events each scheduled task needs (entered, scheduled, timeout timer), // one decision stays under the 1 000-event decision limit (F-04). MaxParallel = 256 MaxBranches = 256 // DefaultRetryMaxDelay caps a retry backoff without max_delay. DefaultRetryMaxDelay = 5 * time.Minute )
Defaults and bounds.
const SupportedVersion = "1"
SupportedVersion is the flow-definition schema version this build accepts. An omitted version means SupportedVersion.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Cache ¶
type Cache struct {
TTL Duration `yaml:"ttl" json:"ttl"`
}
Cache enables the result cache of a task node. A result is stored once the node completes, so concurrent runs with the same key all miss and all run the job (no single-flight).
type Channel ¶
type Channel struct {
// Reducer folds writes: replace (default), append, merge or sum.
Reducer string `yaml:"reducer,omitempty" json:"reducer,omitempty"`
// Default is the initial value.
Default any `yaml:"default,omitempty" json:"default,omitempty"`
}
Channel declares one state channel.
func (Channel) ReducerOrDefault ¶
ReducerOrDefault returns the reducer, defaulting to replace.
type Concurrency ¶
Concurrency bounds how many jobs of a node run at once per key, across all executions.
type Duration ¶
Duration is a time.Duration that (un)marshals as a string like "30s".
func (Duration) MarshalJSON ¶
MarshalJSON encodes the duration as a string.
func (Duration) MarshalText ¶
MarshalText encodes the duration as a string.
func (*Duration) UnmarshalJSON ¶
UnmarshalJSON decodes a duration string ("30s") or a number of nanoseconds.
func (*Duration) UnmarshalText ¶
UnmarshalText decodes a duration string; it is what yaml.v3 uses.
type Flow ¶
type Flow struct {
Version string `yaml:"version,omitempty" json:"version,omitempty"`
Name string `yaml:"name" json:"name"`
Description string `yaml:"description,omitempty" json:"description,omitempty"`
// Start names the entry node. When empty, the entry node is the unique node
// with no inbound transition.
Start string `yaml:"start,omitempty" json:"start,omitempty"`
// Channels declares the typed state of the flow. Nodes write deltas to
// channels and each channel folds them with its reducer.
Channels map[string]Channel `yaml:"channels,omitempty" json:"channels,omitempty"`
Nodes []Node `yaml:"nodes" json:"nodes"`
// Budget caps generic usage counters reported by workers (and the
// built-in "steps" counter): exceeding one fails the execution with reason
// budget_exceeded.
Budget map[string]float64 `yaml:"budget,omitempty" json:"budget,omitempty"`
// MaxSteps is the recursion limit (total node entries); 0 = DefaultMaxSteps.
MaxSteps int `yaml:"max_steps,omitempty" json:"max_steps,omitempty"`
// SearchAttributes are expressions evaluated on the start input and
// indexed for List/Query.
SearchAttributes map[string]string `yaml:"search_attributes,omitempty" json:"search_attributes,omitempty"`
// Retention archives a terminal execution after this long (0 = never).
Retention Duration `yaml:"retention,omitempty" json:"retention,omitempty"`
// Triggers start this flow when a message arrives on a subject.
Triggers []Trigger `yaml:"triggers,omitempty" json:"triggers,omitempty"`
// contains filtered or unexported fields
}
Flow is a flow definition.
func Parse ¶
Parse decodes and validates a flow definition from YAML. Decoding is strict: an unknown field (a typo like "retires:") is an error rather than a silently dropped setting, and extra YAML documents are rejected.
Example ¶
package main
import (
"fmt"
"github.com/henomis/packtrail/flow"
)
func main() {
def, err := flow.Parse([]byte(`
name: greet
channels: {log: {reducer: append}}
nodes:
- {id: hello, type: task, kind: greeter, next: route}
- id: route
type: choice
rules:
- {when: "results.hello.polite", to: bye}
- {default: true, to: hello}
- {id: bye, type: task, kind: greeter}
start: hello
`))
if err != nil {
fmt.Println(err)
return
}
fmt.Println(def.Name, def.StartNode(), def.Node("route").Type, def.Channels["log"].ReducerOrDefault())
}
Output: greet hello choice append
func ParseJSON ¶
ParseJSON decodes and validates a flow definition from its JSON encoding (the form stored in the flows bucket). Unknown fields are rejected.
func (*Flow) AttrProgram ¶
AttrProgram returns the compiled search attribute expression.
func (*Flow) JSON ¶
JSON returns the canonical JSON encoding of the definition (map keys are sorted by encoding/json, so equal definitions encode equally).
func (*Flow) MaxStepsOrDefault ¶
MaxStepsOrDefault returns the recursion limit.
type Node ¶
type Node struct {
ID string `yaml:"id" json:"id"`
Type string `yaml:"type" json:"type"`
// Next is the static successor. Empty on a terminal node.
Next string `yaml:"next,omitempty" json:"next,omitempty"`
// OnFailure routes a permanent failure of the node (task, map, subflow,
// join) to another node instead of failing the execution: retries
// exhausted, a permanent error, an invalid output, a failed child, a join
// whose policy was not met. The handler sees errors.<node>.
OnFailure string `yaml:"on_failure,omitempty" json:"on_failure,omitempty"`
// Meta is free-form configuration for whoever runs or shows the node: the
// engine ignores it, but it is part of the definition, so it is versioned
// with the flow hash and handed to the worker of each job (Job.Meta). It
// must encode as JSON.
Meta map[string]any `yaml:"meta,omitempty" json:"meta,omitempty"`
// task, map: the worker kind and per-attempt policies.
Kind string `yaml:"kind,omitempty" json:"kind,omitempty"`
Timeout Duration `yaml:"timeout,omitempty" json:"timeout,omitempty"`
Retry *Retry `yaml:"retry,omitempty" json:"retry,omitempty"`
OutputSchema any `yaml:"output_schema,omitempty" json:"output_schema,omitempty"`
Cache *Cache `yaml:"cache,omitempty" json:"cache,omitempty"`
Concurrency *Concurrency `yaml:"concurrency,omitempty" json:"concurrency,omitempty"`
// Dynamic lists the nodes a task worker may route to by returning `next`
// (dynamic edges, LangGraph Command(goto)).
Dynamic []string `yaml:"dynamic,omitempty" json:"dynamic,omitempty"`
// choice
Rules []Rule `yaml:"rules,omitempty" json:"rules,omitempty"`
OnError string `yaml:"on_error,omitempty" json:"on_error,omitempty"`
// fanout
Branches []string `yaml:"branches,omitempty" json:"branches,omitempty"`
// join
WaitFor []string `yaml:"wait_for,omitempty" json:"wait_for,omitempty"`
Policy string `yaml:"policy,omitempty" json:"policy,omitempty"`
// await (Timeout is required)
Signal string `yaml:"signal,omitempty" json:"signal,omitempty"`
OnTimeout string `yaml:"on_timeout,omitempty" json:"on_timeout,omitempty"`
// map
Over string `yaml:"over,omitempty" json:"over,omitempty"`
MaxParallel int `yaml:"max_parallel,omitempty" json:"max_parallel,omitempty"`
// subflow
Flow string `yaml:"flow,omitempty" json:"flow,omitempty"`
Input string `yaml:"input,omitempty" json:"input,omitempty"`
OnParentClose string `yaml:"on_parent_close,omitempty" json:"on_parent_close,omitempty"`
// contains filtered or unexported fields
}
Node is a single node of the graph. Fields are type-specific; Validate enforces which are required for each type.
func (*Node) AllowsDynamic ¶
AllowsDynamic reports whether target is a declared dynamic successor.
func (*Node) Backoff ¶
Backoff returns the delay before attempt (2, 3, …). It never returns less than the base delay nor more than the cap: an overflowed shift must not wrap into a too-short delay.
func (*Node) ConcurrencyProgram ¶
ConcurrencyProgram returns the compiled concurrency key expression.
func (*Node) InputProgram ¶
InputProgram returns the compiled subflow input expression (nil = pass the parent input through).
func (*Node) MaxAttempts ¶
MaxAttempts returns the attempt budget of a task (at least 1).
func (*Node) MetaJSON ¶
func (n *Node) MetaJSON() json.RawMessage
MetaJSON returns the JSON encoding of Meta (nil when there is none).
func (*Node) OverProgram ¶
OverProgram returns the compiled map source expression.
func (*Node) ParentClose ¶
ParentClose returns the subflow on_parent_close policy (default cancel).
func (*Node) RulePrograms ¶
RulePrograms returns the compiled choice predicates (nil for the default).
func (*Node) ValidateOutput ¶
ValidateOutput checks an output against the node's output_schema. A node without a schema accepts any output.
type Retry ¶
type Retry struct {
// MaxAttempts is the total number of attempts (1 = no retry).
MaxAttempts int `yaml:"max_attempts" json:"max_attempts"`
Backoff string `yaml:"backoff,omitempty" json:"backoff,omitempty"`
Delay Duration `yaml:"delay,omitempty" json:"delay,omitempty"`
MaxDelay Duration `yaml:"max_delay,omitempty" json:"max_delay,omitempty"`
}
Retry is a task retry policy.
type Rule ¶
type Rule struct {
When string `yaml:"when,omitempty" json:"when,omitempty"`
Default bool `yaml:"default,omitempty" json:"default,omitempty"`
To string `yaml:"to" json:"to"`
}
Rule is one branch of a choice node. Exactly one of When / Default is set.
type Trigger ¶
type Trigger struct {
Subject string `yaml:"subject" json:"subject"`
Stream string `yaml:"stream,omitempty" json:"stream,omitempty"`
}
Trigger starts the flow when a message arrives on Subject. When Stream is set the trigger is a durable JetStream consumer of that stream (at-least once; a message that cannot be started is eventually dead-lettered); otherwise a core NATS queue subscription (at-most-once). The execution id is "<flow>-<Nats-Msg-Id>", so a duplicate message starts nothing twice and every flow triggered by the same message runs; without a Msg-Id it is "<flow>-<stream>-<sequence>" (stream) or a fresh id (core).