execution

package
v0.13.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 11, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package execution (application) walks the graph and runs its nodes.

This is the LOCAL version: no queue, no persistence, no scheduler -- those pieces have phases of their own in the plan (§37, phases 2 and 4). What lives here is enough for `brevis run file.yaml` to run on the instance itself, which is what was asked for.

Index

Constants

View Source
const LogCeiling = 128 << 10

LogCeiling is how much of a step's output goes to the database.

128 KB comfortably covers a dbt run with 60 nodes (~25 KB of text) and still holds a noisy backfill. The ceiling exists because a `while true; do echo` in any workflow must not be able to fill Postgres's disk.

Variables

This section is empty.

Functions

This section is empty.

Types

type ContextPersister added in v0.8.0

type ContextPersister interface {
	RecordContext(ctx context.Context, runID uuid.UUID, step run.StepKey, attempt int,
		published json.RawMessage) error
}

ContextPersister is optional, so a Persister written before this feature still satisfies the runner.

The alternative -- adding the method to Persister -- would break every implementation at once, including the fakes in this package's own tests, for a capability most of them do not need.

type ContextReader added in v0.8.0

type ContextReader interface {
	PublishedContext(ctx context.Context, runID uuid.UUID) (map[string]json.RawMessage, error)
}

ContextReader is optional, for the same reason ContextPersister is: a History written before this feature still satisfies the runner.

type History added in v0.7.0

type History interface {
	StepHasSucceeded(ctx context.Context, workflowSlug, nodeID string, exceto uuid.UUID) (bool, error)

	// AlreadySucceeded returns the INSTANCES of THIS run that finished well in
	// an earlier attempt of it, so a retry re-runs only what failed.
	//
	// Keyed by instance and not by node, because a mapped step's twenty
	// partitions do not fail together: a retry has to redo the two that broke
	// and leave the other eighteen alone.
	//
	// A different question from StepHasSucceeded, which asks about EARLIER
	// RUNS. Confusing the two would make a step skip itself forever after its
	// first good day, so they are separate methods rather than one with a
	// flag.
	AlreadySucceeded(ctx context.Context, runID uuid.UUID) (map[run.StepKey]bool, error)
}

Persistidor records each step's state. Optional: the local `brevis run` has no database, and requiring one would make an ad-hoc execution depend on infrastructure. Historico answers whether a step has ever succeeded. It is what decides whether this is its FIRST run -- something the step does not know and only the engine holds.

A small interface, declared here in the consumer rather than in the package that implements it.

The question is per (workflow, step), not per workflow: a workflow with three fetchers writing to three tables would create only the first step's if the answer covered the whole workflow, and the other two would fail in silence.

type LoadNumbers added in v0.13.0

type LoadNumbers struct {
	Rows, Records, Ignored, BytesOut int64
	LoadMs                           int64

	BytesIn             int64
	Pages, HTTPAttempts int
	ExtractMs           int64
}

LoadNumbers is what one attempt of an SDK pipeline measured, flattened out of the phases so it can be stored as a row and read as a trend.

Every field here already crossed the `@brevis:` pipe and is already in `task_runs.etapas`. This does not measure anything new; it makes the numbers answerable without reading a year of JSONB.

A table rather than a query over the JSONB, because it was measured: on a probe seeded with a year of hourly runs -- 350,000 runs across 40 workflows -- ninety days of one workflow grouped by day costs 14,913 buffer reads and 22 ms through jsonb_array_elements, against 2,243 reads and 5 ms on this narrow indexed table. Six and a half times the reads and four times the latency, for one panel on one page.

func LoadNumbersFrom added in v0.13.0

func LoadNumbersFrom(stages []Stage) (LoadNumbers, bool)

LoadNumbersFrom is the same reading, over the phases as they are STORED.

It takes the recorded shape rather than the collector because that is the shape the other implementation of this rule works on: migration 00012 backfills the same numbers out of `task_runs.etapas`, in SQL, for history this code never saw. Two implementations of one rule is the shape that drifts, and having the Go one answerable from the stored shape is what lets a test put the same phases through both and compare.

type Persister added in v0.7.0

type Persister interface {
	IniciarTask(ctx context.Context, runID uuid.UUID, step run.StepKey, attempt int) error
	TerminarTask(ctx context.Context, runID uuid.UUID, step run.StepKey, attempt int,
		status run.Status, exit *int, failure string, log string) error

	// RecordStages records the phases of an SDK step while it runs. It is
	// what makes the screen advance before the step finishes.
	RecordStages(ctx context.Context, runID uuid.UUID, step run.StepKey, attempt int,
		sdkVersion string, stages json.RawMessage) error

	// RecordLoad keeps what one attempt of an SDK pipeline measured, so the
	// trend can be read without unpacking a year of JSONB.
	//
	// Called once, when the step ends, and only for a step that ran a load to
	// completion -- unlike RecordStages, which runs on every marked line
	// because the screen has to advance while the step is alive.
	RecordLoad(ctx context.Context, runID uuid.UUID, step run.StepKey,
		workflow string, n LoadNumbers) error

	// MarkSkipped records a step whose trigger rule was not satisfied, with the
	// reason. Required rather than optional: a skipped step that leaves no row
	// is invisible, and "did not run and nobody can tell why" is the state this
	// whole feature exists to remove.
	MarkSkipped(ctx context.Context, runID uuid.UUID, step run.StepKey, attempt int,
		reason string) error
}

The step is identified by a run.StepKey and not by a bare node id, because a mapped step (`for_each:`) has one row per instance. It is a struct rather than a fifth positional int next to `attempt`, which is how the wrong one gets passed.

type RecordedStage added in v0.7.0

type RecordedStage = Stage

knownStages is a closed list on purpose: a phase this engine does not know is ignored, rather than becoming a meaningless box on the screen. RecordedStage is the shape an Etapa takes in the JSONB column.

type Reporter

type Reporter interface {
	Evento(execution.Event)
}

Reporter receives the execution's events. A small interface so the CLI, the tests and the persister can all watch the same stream.

type Runner

type Runner struct {
	Processo execution.Executor // serves `run:`; may be nil when there are only Go tasks
	Go       execution.Executor // serves `action:`; may be nil

	WorkDir string
	Env     map[string]string
	Report  Reporter

	// Timeout per node. Zero means no limit.
	Timeout time.Duration

	// MaxAttempts per node. Zero or 1 means a single attempt.
	MaxAttempts int
	BackoffBase time.Duration

	// Persist and RunID are used together: without both, per-step state is not
	// recorded and the DAG in the UI shows up with no execution state.
	Persist Persister

	// ContextDir is where a step's published context is written on the ENGINE's
	// filesystem, for the executors that read a real file.
	//
	// Empty is the normal case and does NOT turn the feature off: Run creates a
	// temporary directory per run and removes it at the end. It was "empty
	// means off" for one commit, and the consequence was that the whole feature
	// worked in tests and did nothing for a user, because nothing outside a
	// test ever set it. A capability that has to be switched on by a field
	// nobody knows about is a capability nobody has.
	//
	// Set it to pin the location -- a test that wants to read the files back.
	ContextDir string

	RunID uuid.UUID

	// Params are this run's values. They reach the step's command through a
	// template (see execution.Render) and the step's environment, so a
	// fetcher using the SDK sees them without being handed an argument.
	Params map[string]string

	// Trigger says why this Run exists: schedule, manual or backfill.
	Trigger string

	// LogicalDate is the slot this Run stands for. Nil on a manual trigger.
	LogicalDate *time.Time

	// Auto are the run's automatic params: the clock to read instead of now(),
	// the window it covers, how late it was, and whether the run before it
	// failed. The engine works them out so no pipeline has to.
	Auto run.AutoParams

	// Historico decides whether a step is running for the first time. Nil means
	// there is no way to know -- and then the step gets first=false, because
	// creating a table without being sure is worse than not creating it.
	History History

	// Vagas caps how many STEPS run at once -- in Kubernetes, how many pods
	// exist simultaneously. Nil means no limit.
	//
	// It has to be shared across every Runner in the process, which is why it
	// is injected rather than created here: the ceiling belongs to the CLUSTER,
	// not to one workflow. Without it, the dispatcher's concurrency limit
	// counted RUNS -- five runs with three parallel steps each gave fifteen
	// pods, not five.
	Slots chan struct{}

	// TentativaDoRun is this RUN's attempt, counted by the dispatcher. It goes
	// into the pod name so a retry does not find the previous attempt's pod.
	RunAttempt int

	// Pods runs steps as pods in Kubernetes. When present it serves every step
	// that declares `image:` -- and the same DAG runs as a pod in the cluster
	// and as a process on a laptop, with no change to the YAML.
	Pods execution.Executor

	// Hosts serves the steps that declare `host:`, keyed by the name in the
	// YAML. Empty is the normal case.
	//
	// A MAP and not a single executor, because one installation talks to
	// several machines and each is a different agent at a different address.
	// The YAML names one of them, and a name with no entry is refused with the
	// list of what exists -- an installation-level fact belongs in an
	// installation-level message, not in a step that silently ran somewhere
	// else.
	Hosts map[string]execution.Executor

	// Metrics records per-step numbers. Nil means nothing is measured, which is
	// what `brevis run` on a laptop wants: it has no endpoint to scrape.
	//
	// The STEP is measured here and the RUN is measured by the dispatcher,
	// which is the only place that knows a failure was the last attempt rather
	// than one of three.
	Metrics *metrics.Metrics
	// contains filtered or unexported fields
}

Runner runs a whole workflow.

It holds TWO executors and picks per node: `run:` goes to the process one, `action:` resolves in the Go registry. The choice belongs to the runner and not to the executor, so each executor can go on ignoring that the other exists.

func (Runner) Run

func (r Runner) Run(ctx context.Context, w wf.Workflow) error

Run walks the graph by levels: everything inside a level runs in parallel, and the next level only starts once the previous one closes entirely.

It walks the WHOLE graph, and a failure does not stop it. What a failure stops is the branch below it: every step whose dependencies did not all succeed is marked `skipped`, with the step responsible named. A branch that does not touch the failure keeps going -- Airflow's rule, and what anybody arriving from it expects.

It used to abort at the first failure, and the reason was a real one: carrying on after an error produced a partial result that looked complete, and a pipeline ran 28 days late without anyone seeing it. What replaces that protection is not nothing. The run still FAILS, with the same error; the failed step is red and the skipped ones are their own colour; and the alert still goes out. A partial result no longer looks complete because the run says it is not.

The cost is real and worth naming: more steps run on a failing run. What it does NOT cost is paying for them twice -- a retry re-runs only what failed; see alreadySucceeded.

type Stage added in v0.7.0

type Stage struct {
	Index    int            `json:"indice"`
	TaskName string         `json:"nome"`
	State    string         `json:"estado"`
	Ms       *int64         `json:"ms,omitempty"`
	At       string         `json:"em"`
	Numbers  map[string]any `json:"numeros,omitempty"`
}

Etapa is one phase of an SDK step, as it stands now.

Indice is what identifies it, and not Nome: a pipeline with two Map stages announces `map` twice, and keying by name would make the second overwrite the first -- three declared stages collapsing into two boxes on the screen, with no warning.

RecordedStage is the same type under the name it travels through the database with: it is what a test outside this package needs to check what was recorded.

type StepError added in v0.7.0

type StepError struct {
	NodeID   string
	ExitCode int
	Message  string

	// Saida is the last few lines of stderr. Only the last ones, and not all of
	// them, because a chatty process would fill the database's error column --
	// and the cause is almost always at the end.
	Output []string
}

StepError is a step's failure, with the context needed to understand it without opening a log: the exit code, what it means, and the last lines the process wrote to stderr.

Before, all that survived was "exited with code 127" -- technically correct and useless. The cause (`/bin/sh: python: not found`) went through the events as a log line and was dropped right there, so the screen showed the symptom without the explanation.

func (*StepError) Error added in v0.7.0

func (e *StepError) Error() string

type StepMetric added in v0.12.0

type StepMetric struct {
	Name  string
	Kind  string
	Value float64
}

StepMetric is one value a step reported through the @brevis: protocol.

Kind is "gauge" (last value wins) or "counter" (adds up). The engine supplies the labels; there is deliberately no field for the step's own.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL