docs

package
v0.2.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 25, 2026 License: MIT Imports: 5 Imported by: 0

README

Documentation

Read the page for what you're doing. Nothing here needs to be read in order. New to steps: the quick start (install, a task, a get:, an agent step), then resources.md, agents.md, and web.md for the daemon and its UI.

Every YAML example in these docs is a complete pipeline, extracted and executed by the test suite (e2e/docs_test.go). An example that needs docker, the network, a CLI, or real credentials says which one; everything else runs exactly as shown, and a doc example that stops working fails the build. Read them via steps docs in a terminal, /docs in the web UI, or the files themselves.

The assert: blocks you see in them are load-bearing, not decoration. Every executed example verifies its own behavior — which steps ran in what order, what a command printed, what a model decided, what a step wrote — so the examples are regression tests for the features they document, and the claims in the prose beside them are checked rather than asserted by an author. A separate mutation suite proves those assertions can actually fail: it rewrites each one to something the pipeline does not satisfy and requires the build to go red. See assert:.

Writing pipelines

Page What it covers Length
resources.md Resources and resource types: the built-ins git, cron, the GitHub and Slack ones, the check/in/out contract, version:, trigger: short
webhooks.md type: webhook: deliveries as versions, the provider table, filter:/id:/version:, custom schemes, tunnels, --deliver medium
expr.md Expression resource types: expr:, the batched http(), env()/file(), when to use it instead of shell medium
control-flow.md when: guards, hooks, to:/verdicts: routing, do:/in_parallel:/race:/across:, assert:, approvals long
agents.md Agent steps: tools, prompts, verdicts, sub-agents, fix:, budgets, failover, CLI agents, ensembles long
attempts-timeout.md attempts: and timeout:, and how they interact short
workspace.md inputs:/outputs:, per-step isolation (always on), read-modify-write artifacts, the resource and step caches (volatile:), task caches (caches:), --resume short
infra.md Containers (image:, network:, user:, env:, limits), remote workers (tags:), the artifact store, build metadata, and job scheduling: trigger:, passed:, serial:, max_in_flight:, interruptible:, the circuit breaker, run_history: long
templating.md {{ }} in resource commands and custom tools, shellquote, ((var)), load_var: short
mcp.md MCP servers as tool sources and as resource-type backends medium
complete.md A full pipeline putting the pieces together short

Reference

Page What it covers
web.md The daemon: steps pipeline set, the browser UI, run transcripts, the dependency graph, live runs, triggering, and one state database for every pipeline it holds
observability.md OpenTelemetry traces and metrics from the OTEL_* environment: what turns them on, what is traced, what is measured
authentication.md Accounts on steps web (Basic, local users, GitHub/Google sign-in), roles and a pipeline's access:, the exempt routes, and how the CLI carries credentials
agents-internals.md How agent steps work underneath: transport, tool-call repair, compaction, caching
aws-workers.md Standing up an aws:// worker by hand with the AWS CLI: IAM, security group, launch template, instance, bucket — and running a pipeline on it
gcp-workers.md Standing up a gcp:// worker by hand with gcloud: the IAP firewall rule, instance template, instance — and running a pipeline on it
do-workers.md A do:// worker: the droplet defaults, how its host key is trusted, the token and firewall to set up — and running a pipeline on it
conformance.md Which Concourse behaviors steps matches, which it doesn't, and which are verified

Commands

steps run <pipeline>        run one job (--resume <id> continues a failed one,
                            --rerun <id> runs it again against its own versions,
                            --replay <id> --from <step> re-runs one step of one)
                            --worker <tag>=<url> places tags: steps on a machine
                            --progress tty|plain|auto draws it live or as lines
steps test <pipeline>       run every job and check assert: directives
steps validate <pipeline>   check the file, and that this machine can run it
steps plan <pipeline>       show what a run would execute vs skip
steps validate --live       also probe the models and MCP servers themselves
                            (--job <name> narrows it to one job)

steps web                   the daemon: serve the UI, hold the pipelines set
                            into it, poll trigger: true resources, run jobs
steps pipeline set -c f.yml upload a pipeline into a daemon — the only way a
                            served pipeline changes (list|get|pause|unpause|
                            rename|destroy for the rest; --target names one)

steps runs -p <name>        what ran, newest first, each row naming the
                            configuration it executed (steps|queue|cost|where
                            for the other four views; runs steps says why,
                            runs steps <run> what a --resume of it skips)
steps runs                  with no -p: every pipeline in the state file
steps runs follow -p <name> [run]
                            watch a run to its end (live on a terminal;
                            Ctrl-C detaches, never aborts)
steps runs abort -p <name> <run>
                            stop a run on a daemon (--queued <job> drops
                            a queued one instead)
steps jobs -p <name>        list jobs the circuit breaker is holding
                            (steps jobs release <job> -p <name> clears one)
steps approvals -p <name>   list approval: steps waiting for a decision
                            (steps approvals approve|reject <id> -p <name>)
steps questions -p <name>   list ask_user questions waiting for an answer
                            (steps questions answer <id> <answer> -p <name>)
steps memory -p <name>      list what agent steps remember, by scope
                            (--scope <s> for its entries; steps memory rm
                            --scope <s> <id>...|--all -p <name> deletes)
steps clear-task-cache <pipeline> -j <job> -s <step>
                            empty a step's caches: here and on its workers
                            (-c <path> for one; --worker as steps run takes it)
steps mcp list|tools|login  inspect or authorize mcp_servers: entries
                            (login <server> -c f.yml authorizes this machine;
                            -p <name> --target <url> authorizes a daemon)
steps docs [page]           read these docs in the terminal

The read commands default to a daemon's .steps/steps.db. A local steps run or steps test keeps its state beside the YAML instead, so reaching it takes --db — which a parked approval: or question prints for you: steps approvals approve 1 -p pipeline --db .steps/pipeline.yml.db.

How a run looks while it runs. On a terminal, steps run and steps test draw a live view: a region at the bottom shows each running step with its clock, the last lines it printed and an agent's tool calls, and each step that finishes leaves one line above it in your scrollback. A failed run ends with the failed step's full output under the summary. Anywhere else — a pipe, a file, CI (CI set), TERM=dumb, or an agent calling steps — it prints one plain line per event instead, which --progress plain forces and --progress tty overrides. Both are drawn from the same events the web transcript is, so the notes a run prints (a worker started, an image pulled, a try: tolerated) are on its run page too. steps runs follow -p <name> [run] attaches the same view to a run another process is executing — a daemon's, or a steps run in another terminal — by reading what it records: it replays from the start, shows a running step's output as the step records it, and exits as the run did (an errored run reads as failed, since only failed is recorded). Nothing to follow is an error, never an empty success. With no run named it takes the newest run that has started: a build still in steps runs queue has no run yet, so following right after a trigger can attach to the previous one — name the run once steps runs lists it.

Two of these answer most "why is it doing that?" questions: steps plan explains what the cache would skip, and steps runs steps shows what previous runs actually did.

A third answers "did the pipeline change?" — steps runs carries a CONFIG column, the hash of the configuration each run was started from. Two runs of one file agree; an edit between them does not, and a run whose rows disagree with the last green one is a run that executed something else. The hash covers everything the configuration is made of: the pipeline file after ((var)) substitution — so one file under two --vars-files is two configurations — and every file it includes, so editing a run_file: script is editing the pipeline. A run started by something that loaded no file at all shows -.

steps validate answers a fourth: will this run at all? It checks the file — syntax, references, field placement — and then the things the file depends on that live outside it:

  • every model name resolves to a known provider (a typo like opencoder/ for opencode/ is a load error, not a failed run)
  • every api_key_env: the pipeline names is actually set
  • every variable an expr-backed resource the plan uses reads by literal name (env("LINEAR_API_KEY")) is actually set: a computed name is a runtime answer and is not checked, and a type's env: list alone requires nothing, since it only allows
  • every stdio mcp_servers: command is actually on PATH

It reports all of them at once, because finding them one run at a time is the problem:

$ steps validate pipeline.yml
steps: error: pipeline.yml cannot run here:
  agent "coder"  $OPENROUTER_API_KEY is not set (source.api_key_env)
  mcp "gopls"    command "gopls" not found on PATH

steps validate --live answers the run-time version of the same question: not "is this pipeline runnable" but "is it runnable right now". It sends a minimal request to every model the pipeline reaches and starts every MCP server it grants; --job <name> narrows that to one job.

steps run does this automatically, before any step executes. Without it, a plan like plan -> code -> check -> review -> publish discovers a dead model half an hour in, with everything before it paid for and thrown away. With it, the run fails in seconds, saying explicitly that nothing ran. It is a run all the same — errored, with no steps, the reason in its head — so under steps web the job turns red, the job's on_error fires, and Retry on that run builds what it refused:

$ steps run pipeline.yml --job self-build
steps: error: job "self-build": preflight failed, no steps were run:
  agent "coder": model "deepseek-v4-flash": no response within 30s
    (other models on this endpoint responded — the model itself looks unavailable, not the endpoint or the key)

Tuning, all optional:

defaults:
  preflight:
    disabled: false   # true skips it entirely
    timeout: 30s      # per check
    cache: 5m         # a target verified this recently is trusted

agents:
- name: coder
  preflight: false    # opt one agent out — e.g. a local model slow to WAKE

The cache is what makes this usable under steps web: without it every poll interval would pay for a probe request against every model. --no-preflight skips it for one invocation. What it does not do: preflight catches "broken before we start", not "breaks halfway through" — failing over mid-run is fallback:'s territory.

Add --syntax-only to steps validate to check the file alone — the right flag for a pre-commit hook or a CI lint that should not need the pipeline's production credentials on hand.

Editor support

steps.schema.json is a JSON Schema for the pipeline format. Point your editor at it with a modeline on the first line of a pipeline:

# yaml-language-server: $schema=./steps.schema.json

That gives completion and inline errors while you type. steps validate remains the authority — it checks rules a schema can't express, like whether a to: target exists in the same segment — but the schema catches misspelled keys at the keystroke rather than the run.

Re-running one step: --replay

An agent step makes its chain unskippable, and unskippable propagates forward — so editing the last agent step's prompt re-runs every step before it, at full price, except those the step cache can answer (a durable workspace.root:, declared outputs:, and none of the exclusions it lists, such as verdicts: or hooks). That is the single most expensive thing about authoring an agent pipeline.

steps run pipeline.yml --replay r-8f2a1c --from synthesizer

That forks the recorded run and executes from synthesizer onward. It does not consult the merkle cache: state comes from the source run's workspace (the artifacts earlier steps produced are already on disk), its recorded run_context, and its step record.

  • It forks, never mutates. The source run stays exactly as it was, so the thing you are comparing against still exists — two prompt variants become two runs you can read side by side. steps runs cost prices both.
  • --from names a step, matched against the current plan. The pipeline has almost certainly changed since the source run; that is why you are replaying.
  • A source run that never completed an earlier step is refused, naming it.
  • A get: before --from is refused. Every step after a get runs once per version, in a build of its own, and a replay has no way to say which of those builds it means.
  • It needs the source workspace, so the run being replayed must have been kept (--keep-workspace). A reaped tree is a clear error, not a silent full re-run.

Documentation

Overview

Package docs embeds the user-facing documentation (docs/*.md) and extracts its fenced ```yaml blocks, which are the repo's tested example corpus: docs_test.go (root package) schema-validates and executes them, so a doc example cannot drift from what the runner actually accepts.

The fence info string is a tiny grammar deciding how a block is tested:

```yaml                a complete pipeline; schema-validated and executed
```yaml test=<id>      executed against the fake LLM scenario named <id>
                       (docs_scenarios_test.go) — required for agent steps
```yaml mcp=<id>       executed with every mcp_servers: entry pointed at
                       the in-process fake MCP fixture named <id>
                       (docs_mcp_test.go) — required for mcp examples
                       that run
```yaml github=<id>    executed with every github-* resource's endpoint
                       pointed at the in-process fake GitHub fixture
                       named <id> (docs_github_test.go)
```yaml noexec=<why>   schema-validated and loaded, not executed, because
                       this host cannot run it — the reason is mandatory
                       and drawn from a fixed vocabulary (see NoexecReason)
```yaml fragment       rendered only; not a complete pipeline

A bare `noexec` is deliberately NOT a mode: an unexecuted example is the one that can silently rot, so opting out has to name which capability is missing rather than being the cheapest thing to type.

Everything here is reference material for what steps DOES. A proposal for what it might do is a GitHub issue, not a page: sketches are not promises, and one living next to the docs reads like one.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NoexecReasons

func NoexecReasons() []string

NoexecReasons is the closed vocabulary a noexec block may name: the capability this host does not have. Closed rather than free text so the set stays auditable — "which examples would run in CI if we gave it docker?" has to be a question you can answer by grepping.

func Page

func Page(name string) ([]byte, error)

Page returns one page's markdown source.

func Pages

func Pages() []string

Pages lists the embedded doc pages, README.md first (it is the index), then alphabetically.

func Slug

func Slug(heading string) string

Slug converts a heading's text to its GitHub-style anchor id: lowercased, punctuation dropped, spaces hyphenated, underscores and hyphens kept.

This is the algorithm the corpus's hand-written #fragment links were authored against (the pages are read on GitHub too), so the web renderer generates ids with THIS rather than goldmark's default — which folds "_" into "-" and silently strands every anchor containing a field name like max_visits. TestDocsAnchors holds the two in agreement.

Types

type Block

type Block struct {
	Page string // file name, e.g. "agents.md"
	Line int    // 1-based line number of the opening fence
	Info string // fence info string after "yaml", e.g. "test=verdicts"
	Body string
}

Block is one fenced ```yaml example lifted from a doc page.

func Blocks

func Blocks() ([]Block, error)

Blocks extracts every ```yaml block from every page. An unterminated fence is an authoring error and reported as one, not silently swallowed.

func (Block) DeliverID

func (b Block) DeliverID() string

DeliverID is the captured webhook delivery this block runs with (docs_deliveries_test.go), "" when none.

func (Block) GitHubID

func (b Block) GitHubID() string

GitHubID is the fake GitHub fixture this block's github-* resources are pointed at during execution (docs_github_test.go), "" when none.

func (Block) MCPID

func (b Block) MCPID() string

MCPID is the fake MCP fixture this block's mcp_servers: are pointed at during execution (docs_mcp_test.go), "" when none.

func (Block) Mode

func (b Block) Mode() string

Mode reports how the block is tested: "run", "noexec", or "fragment". A bare `noexec` reports "noexec" too — it is a mode with no reason, which TestDocsNoexecReasons rejects; treating it as "run" instead would fail the authoring mistake as a broken pipeline rather than a missing reason.

func (Block) Name

func (b Block) Name() string

Name identifies the block in test output: page, line, and scenario or MCP fixture if any.

func (Block) NoexecReason

func (b Block) NoexecReason() string

NoexecReason is the reason a noexec block names, "" when it names none.

func (Block) TestID

func (b Block) TestID() string

TestID is the fake-LLM scenario this block runs against, "" when none.

type Group

type Group struct {
	Title string
	Pages []string
}

Group is a curated section of the corpus, in reading order — the same order docs/README.md's own tables teach. Pages() stays alphabetical for enumeration; Groups() is what a navigation surface should render, so the rail agrees with the curriculum instead of sorting agents-internals ahead of agents.

func Groups

func Groups() []Group

Groups returns every page but the index, grouped and ordered for reading. TestDocsGroupsComplete (root package) keeps this table complete: a new page must be placed in a group, or the build is red.

type Heading

type Heading struct {
	Level int
	Text  string
	ID    string
}

Heading is one heading on a page, with the anchor id it renders under.

func Headings

func Headings(body string) []Heading

Headings returns every heading on a page in order, each with its GitHub-style id, deduplicated the way GitHub does (repeats get -1, -2, ...). Fenced code blocks are skipped — a "# comment" inside one is not a heading. This is the id set the web renderer emits and the anchor test verifies against, so all three surfaces agree by construction.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL