Documentation
¶
Overview ¶
Package execution (application) walks the graph and runs its nodes.
This is the LOCAL version: no queue, no persistence, no scheduler -- those pieces have phases of their own in the plan (§37, phases 2 and 4). What lives here is enough for `brevis run file.yaml` to run on the instance itself, which is what was asked for.
Index ¶
Constants ¶
const LogCeiling = 128 << 10
LogCeiling is how much of a step's output goes to the database.
128 KB comfortably covers a dbt run with 60 nodes (~25 KB of text) and still holds a noisy backfill. The ceiling exists because a `while true; do echo` in any workflow must not be able to fill Postgres's disk.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type History ¶ added in v0.7.0
type History interface {
StepHasSucceeded(ctx context.Context, workflowSlug, nodeID string, exceto uuid.UUID) (bool, error)
}
Persistidor records each step's state. Optional: the local `brevis run` has no database, and requiring one would make an ad-hoc execution depend on infrastructure. Historico answers whether a step has ever succeeded. It is what decides whether this is its FIRST run -- something the step does not know and only the engine holds.
A small interface, declared here in the consumer rather than in the package that implements it.
The question is per (workflow, step), not per workflow: a workflow with three fetchers writing to three tables would create only the first step's if the answer covered the whole workflow, and the other two would fail in silence.
type Persister ¶ added in v0.7.0
type Persister interface {
IniciarTask(ctx context.Context, runID uuid.UUID, nodeID string, attempt int) error
TerminarTask(ctx context.Context, runID uuid.UUID, nodeID string, attempt int,
status run.Status, exit *int, failure string, log string) error
// RecordStages records the phases of an SDK step while it runs. It is
// what makes the screen advance before the step finishes.
RecordStages(ctx context.Context, runID uuid.UUID, nodeID string, attempt int,
sdkVersion string, stages json.RawMessage) error
}
type RecordedStage ¶ added in v0.7.0
type RecordedStage = Stage
knownStages is a closed list on purpose: a phase this engine does not know is ignored, rather than becoming a meaningless box on the screen. RecordedStage is the shape an Etapa takes in the JSONB column.
type Reporter ¶
Reporter receives the execution's events. A small interface so the CLI, the tests and the persister can all watch the same stream.
type Runner ¶
type Runner struct {
Processo execution.Executor // serves `run:`; may be nil when there are only Go tasks
Go execution.Executor // serves `action:`; may be nil
WorkDir string
Env map[string]string
Report Reporter
// Timeout per node. Zero means no limit.
Timeout time.Duration
// MaxAttempts per node. Zero or 1 means a single attempt.
MaxAttempts int
BackoffBase time.Duration
// Persist and RunID are used together: without both, per-step state is not
// recorded and the DAG in the UI shows up with no execution state.
Persist Persister
RunID uuid.UUID
// Params are this run's values. They reach the step's command through a
// template (see execution.Render) and the step's environment, so a
// fetcher using the SDK sees them without being handed an argument.
Params map[string]string
// Trigger says why this Run exists: schedule, manual or backfill.
Trigger string
// LogicalDate is the slot this Run stands for. Nil on a manual trigger.
LogicalDate *time.Time
// Historico decides whether a step is running for the first time. Nil means
// there is no way to know -- and then the step gets first=false, because
// creating a table without being sure is worse than not creating it.
History History
// Vagas caps how many STEPS run at once -- in Kubernetes, how many pods
// exist simultaneously. Nil means no limit.
//
// It has to be shared across every Runner in the process, which is why it
// is injected rather than created here: the ceiling belongs to the CLUSTER,
// not to one workflow. Without it, the dispatcher's concurrency limit
// counted RUNS -- five runs with three parallel steps each gave fifteen
// pods, not five.
Slots chan struct{}
// TentativaDoRun is this RUN's attempt, counted by the dispatcher. It goes
// into the pod name so a retry does not find the previous attempt's pod.
RunAttempt int
// Pods runs steps as pods in Kubernetes. When present it serves every step
// that declares `image:` -- and the same DAG runs as a pod in the cluster
// and as a process on a laptop, with no change to the YAML.
Pods execution.Executor
}
Runner runs a whole workflow.
It holds TWO executors and picks per node: `run:` goes to the process one, `action:` resolves in the Go registry. The choice belongs to the runner and not to the executor, so each executor can go on ignoring that the other exists.
func (Runner) Run ¶
Run walks the graph by levels: everything inside a level runs in parallel, and the next level only starts once the previous one closes entirely.
It stops at the FIRST failure in a level, without starting the next. Carrying on after an error would produce a partial result that looks complete -- which is how a pipeline ran 28 days late without anyone seeing it, in the system this one replaces.
type Stage ¶ added in v0.7.0
type Stage struct {
Index int `json:"indice"`
TaskName string `json:"nome"`
State string `json:"estado"`
Ms *int64 `json:"ms,omitempty"`
At string `json:"em"`
Numbers map[string]any `json:"numeros,omitempty"`
}
Etapa is one phase of an SDK step, as it stands now.
Indice is what identifies it, and not Nome: a pipeline with two Map stages announces `map` twice, and keying by name would make the second overwrite the first -- three declared stages collapsing into two boxes on the screen, with no warning.
RecordedStage is the same type under the name it travels through the database with: it is what a test outside this package needs to check what was recorded.
type StepError ¶ added in v0.7.0
type StepError struct {
NodeID string
ExitCode int
Message string
// Saida is the last few lines of stderr. Only the last ones, and not all of
// them, because a chatty process would fill the database's error column --
// and the cause is almost always at the end.
Output []string
}
StepError is a step's failure, with the context needed to understand it without opening a log: the exit code, what it means, and the last lines the process wrote to stderr.
Before, all that survived was "exited with code 127" -- technically correct and useless. The cause (`/bin/sh: python: not found`) went through the events as a log line and was dropped right there, so the screen showed the symptom without the explanation.