pipeline

package
v0.0.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 24, 2026 License: MIT Imports: 15 Imported by: 0

Documentation

Overview

Package pipeline implements the tcli pipeline (YAML) execution engine. See docs/pipeline.md for the schema reference.

Index

Constants

View Source
const (
	OnFailureCancel = "cancel"
	OnFailureDrain  = "drain"
)

Variables

This section is empty.

Functions

func BuildArgs

func BuildArgs(s *Step) ([]string, error)

BuildArgs turns a resolved Step into the CLI argument vector tcli expects. Exposed so callers writing custom runners can reuse the mapping.

Types

type Concurrency

type Concurrency struct {
	MaxParallel int    `yaml:"maxParallel,omitempty"`
	OnFailure   string `yaml:"onFailure,omitempty"`
}

type DAG

type DAG struct {
	Nodes []*Node // topologically sorted; roots first
	// contains filtered or unexported fields
}

DAG is the fully-resolved execution graph for a pipeline.

func BuildDAG

func BuildDAG(p *Pipeline) (*DAG, error)

BuildDAG derives the execution graph from a pipeline. Edges come from:

  • inputFrom on a step
  • explicit dependsOn entries
  • ${{ steps.<name>.xx }} references anywhere in the step's values

Returns an error if the graph contains a cycle. Callers should validate the pipeline before calling BuildDAG (Load and Parse do this automatically).

func (*DAG) Node

func (d *DAG) Node(name string) *Node

Node looks up a graph node by step name. Returns nil if not found.

type Executor

type Executor struct {
	Runner Runner
}

Executor walks the pipeline DAG, resolves interpolations, and dispatches each step to the configured Runner. Zero value is not usable; use NewExecutor.

func NewExecutor

func NewExecutor(r Runner) *Executor

NewExecutor returns an Executor that dispatches steps to r.

func (*Executor) Run

func (e *Executor) Run(ctx context.Context, p *Pipeline) (*State, error)

Run executes p to completion and returns the shared runtime state. If any non-continueOnError step fails, Run returns the first such error (all other steps are still reflected in the returned State). If the underlying DAG build fails, Run returns that error directly.

type Node

type Node struct {
	Step       *Step
	Deps       []*Node // steps this one waits on
	Dependents []*Node // steps waiting on this one
}

Node is a single step's position in the resolved execution graph, holding upstream (deps) and downstream (dependents) edges plus a back-pointer to the step definition.

type Pipeline

type Pipeline struct {
	Name        string         `yaml:"name"`
	Description string         `yaml:"description,omitempty"`
	Variables   map[string]any `yaml:"variables,omitempty"`
	Concurrency *Concurrency   `yaml:"concurrency,omitempty"`
	Defaults    *StepDefaults  `yaml:"defaults,omitempty"`
	Steps       []*Step        `yaml:"steps"`
}

func Load

func Load(path string) (*Pipeline, error)

Load reads and parses a pipeline YAML file, then runs full validation (schema, references, DAG cycles). A returned *Pipeline is safe to execute.

func Parse

func Parse(data []byte) (*Pipeline, error)

Decodes YAML bytes into a *Pipeline.

func (*Pipeline) Validate

func (p *Pipeline) Validate() error

Validate runs all structural checks on a pipeline. It returns the first error encountered; callers can inspect the wrapped error for detail. A pipeline that passes Validate is safe to build a DAG from.

type Record

type Record = map[string]any

Record is one JSON object flowing between steps.

type Reference

type Reference struct {
	Raw  string   // the literal "${{ ... }}" including braces, for replacement
	Path []string // dotted path split on "."
}

Reference is one parsed ${{ ... }} occurrence in a pipeline value.

func FindReferences

func FindReferences(s string) []Reference

FindReferences returns every ${{ }} occurrence inside s.

func (Reference) Kind

func (r Reference) Kind() string

Kind reports the top-level scope of a reference path: "variables", "steps", "env", or "" for anything else. Callers use this to route lookup.

func (Reference) StepDep

func (r Reference) StepDep() string

StepDep returns the step name a reference depends on, or "" if the reference does not target a step. Used by the DAG builder to derive edges from ${{ steps.<name>.xx }} occurrences.

type Resolver

type Resolver struct {
	State *State
	Env   func(string) string // defaults to os.Getenv when nil
}

Resolver evaluates ${{ }} references against a State. Runtime environment lookups go through Env; tests can substitute a fake to keep them hermetic.

func NewResolver

func NewResolver(state *State) *Resolver

NewResolver returns a Resolver bound to state, using os.Getenv for ${{ env.X }} lookups.

func (*Resolver) ResolveAny

func (r *Resolver) ResolveAny(v any) (any, error)

ResolveAny walks a value: for strings, interpolate; for maps/slices, recurse. Non-string leaves pass through unchanged.

func (*Resolver) ResolveString

func (r *Resolver) ResolveString(s string) (string, error)

ResolveString replaces every ${{ }} occurrence in s with its string value. Missing references return an error rather than silently substituting empty; missing values in configuration are almost always bugs.

func (*Resolver) ResolveValue

func (r *Resolver) ResolveValue(s string) (any, error)

ResolveValue returns the typed value when s is exactly one ${{ }} reference (so bools stay bools, numbers stay numbers), and falls back to string interpolation otherwise. Used where the resulting type matters — notably `condition:`.

type Runner

type Runner interface {
	Run(ctx context.Context, s *Step, input []Record) ([]Record, error)
}

Runner is the boundary between the pipeline engine and the tcli command execution layer. Production code uses SubprocessRunner (which shells out to the tcli binary); tests substitute an in-memory fake.

A Runner receives the fully-resolved step (interpolation done, defaults merged) plus any records streamed in from an upstream step, and returns the records this step produced.

type State

type State struct {
	Variables map[string]any
	// contains filtered or unexported fields
}

State is the shared runtime state for one pipeline execution: variables (immutable snapshot from the YAML) and per-step results (mutated as steps complete). Reads and writes are safe from multiple goroutines.

func NewState

func NewState(p *Pipeline) *State

NewState seeds pending StepResults for every step so ${{ steps.X.xx }} references never hit a nil lookup.

func (*State) Get

func (s *State) Get(name string) *StepResult

Get returns the result for one step. Returns nil if the step name is unknown.

func (*State) Set

func (s *State) Set(name string, r *StepResult)

Set replaces a step's result atomically.

func (*State) Snapshot

func (s *State) Snapshot() map[string]*StepResult

Snapshot returns a shallow copy of the step results map. Useful for end-of-run reporting.

type Step

type Step struct {
	Name    string         `yaml:"name"`
	Command string         `yaml:"command"`
	Params  map[string]any `yaml:"params,omitempty"`
	Body    any            `yaml:"body,omitempty"`
	Format  string         `yaml:"format,omitempty"`

	InputFrom string            `yaml:"inputFrom,omitempty"`
	Outputs   map[string]string `yaml:"outputs,omitempty"`

	DependsOn       []string `yaml:"dependsOn,omitempty"`
	Condition       string   `yaml:"condition,omitempty"`
	ContinueOnError bool     `yaml:"continueOnError,omitempty"`

	Count        int     `yaml:"count,omitempty"`
	Parallelism  *int    `yaml:"parallelism,omitempty"`
	RetryCount   *int    `yaml:"retryCount,omitempty"`
	IgnoreErrors *bool   `yaml:"ignoreErrors,omitempty"`
	StatusCode   string  `yaml:"statusCode,omitempty"`
	Verbose      *bool   `yaml:"verbose,omitempty"`
	BasePath     *string `yaml:"basePath,omitempty"`
	Scheme       *string `yaml:"scheme,omitempty"`
	Server       *string `yaml:"server,omitempty"`
	Jwt          *string `yaml:"jwt,omitempty"`
}

type StepDefaults

type StepDefaults struct {
	Verbose      *bool   `yaml:"verbose,omitempty"`
	IgnoreErrors *bool   `yaml:"ignoreErrors,omitempty"`
	RetryCount   *int    `yaml:"retryCount,omitempty"`
	Parallelism  *int    `yaml:"parallelism,omitempty"`
	StatusCode   *string `yaml:"statusCode,omitempty"`
	BasePath     *string `yaml:"basePath,omitempty"`
	Scheme       *string `yaml:"scheme,omitempty"`
	Server       *string `yaml:"server,omitempty"`
	Jwt          *string `yaml:"jwt,omitempty"`
	Doc          *string `yaml:"doc,omitempty"`
}

StepDefaults holds pipeline-wide defaults. Fields are pointers so an unset default can be distinguished from an explicit zero value (e.g. `retryCount: 0` is not the same as "not specified").

type StepResult

type StepResult struct {
	Status  StepStatus
	Records []Record       // records the step emitted (may be nil if none)
	Outputs map[string]any // extracted per the step's outputs: block
	Err     error
}

StepResult captures everything about one completed step that later steps or the caller can observe.

func (*StepResult) Ok

func (r *StepResult) Ok() bool

Ok reports whether the step reached StatusSucceeded. Used for ${{ steps.X.ok }} references and parent-status checks.

type StepStatus

type StepStatus int

StepStatus is the terminal (or in-progress) status of one step during a run.

const (
	StatusPending StepStatus = iota
	StatusRunning
	StatusSucceeded
	StatusFailed
	StatusSkipped
)

func (StepStatus) String

func (s StepStatus) String() string

type SubprocessRunner

type SubprocessRunner struct {
	// Binary is the path to the tcli executable. When empty, os.Executable()
	// is used, which is correct when the current process itself is tcli.
	Binary string

	// Stderr, if set, is where the child's stderr is copied. Defaults to
	// os.Stderr so tcli's logs surface during a pipeline run.
	Stderr io.Writer
}

SubprocessRunner executes each step by invoking the tcli binary as a subprocess. This deliberately reuses the CLI code path same argument-parsing, same module/command dispatch, same stdout format so scenario behavior is identical to what the equivalent bash pipe would do today. In-process execution is deferred to a later iteration.

The pipeline's `parallelism` field is translated best-effort: for any value other than unset/1 we pass `-parallel` (tcli's boolean flag). Exact worker-count control requires an eventual `-parallelism <n>` addition to pkg/cmd.

func (*SubprocessRunner) Run

func (r *SubprocessRunner) Run(ctx context.Context, s *Step, input []Record) ([]Record, error)

Run invokes tcli for one step, streams input records on stdin, captures stdout, and parses the JSON stream back into records.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL