Documentation
¶
Overview ¶
Package execution defines the contract for executing tasks.
The interface is section 13's, with one difference: `Execute` returns a channel of events instead of blocking. A pod that runs for twenty minutes has to report progress before it ends, and the same holds for a local process writing to stdout.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( ErrShutdown = errors.New("the scheduler is shutting down") ErrClaimLost = errors.New("this scheduler lost the run's claim to another") // ErrCanceled: someone asked to cancel the run. Like a lost claim, and // unlike a shutdown, it is never handed over: the steps stop. ErrCanceled = errors.New("someone canceled the run") )
Why a run's context ended, read with context.Cause by an executor that has a choice to make (#125). Both cancel the run; they differ in what may be left behind. A SHUTDOWN may hand a step over to the next scheduler -- a `host:` step goes on running on its agent. A LOST CLAIM may not: another scheduler holds the run now, and leaving the step running is running it twice (#121).
var ( // ErrUnknownExecution: where the step ran has no execution by that id -- // it restarted, or kept it for less time than it was gone. Nothing runs // under that id, so the step may start again with it. ErrUnknownExecution = errors.New("the agent has no execution by that id") // ErrLostTrack: the execution is known but its output can no longer be // read from where it was left. It may still be running, so it is stopped, // and the attempt fails. ErrLostTrack = errors.New("the step's output can no longer be read from where it was left") )
Why a handed-over step cannot be followed (#125), kept apart because the runner does a different thing for each. An executor wraps one of them; any other error means it could not reach where the step runs.
var ErrNotStarted = errors.New("the step was not started")
ErrNotStarted is a step that nothing ran: where it was sent refused it before starting anything, for a reason that is not the step's -- every agent of a pool draining (#146). The runner does not spend the step's attempts on it, and a run whose only failures are these is one the dispatcher sends back to the queue without spending the run's.
Functions ¶
func Render ¶
Render substitutes the params into a step's command line.
The stdlib's `text/template`, with `missingkey=error`: a param with a typo in the YAML fails HERE, naming what was missing, instead of becoming an empty string and producing a silently wrong command — `--select ` with no target, or `--date` with no date.
So only the command is rendered. `image:` is deliberately NOT templatable: whoever triggers a run would be choosing the image the pod runs, which is choosing the code that executes.
Types ¶
type Event ¶
type Event struct {
Kind EventKind
NodeID string
Message string
Stream string // "stdout" | "stderr"
ExitCode int
Err error
}
Event is one occurrence during the run. `Stream` tells stdout from stderr: merging the two loses the information about where the message came from, and that is exactly what made dbt's final summary show up as an error in Leoflow.
type EventKind ¶
type EventKind string
EventKind classifies what the executor reports.
const ( EventStarted EventKind = "started" EventLog EventKind = "log" EventSucceeded EventKind = "succeeded" // EventContext carries what the step published, in Message. // // It travels as an event for the same reason the phases do: the runner // collects it without knowing whether it came from a file on this disk or // from a pod's termination message, so a third executor costs the runner // nothing. EventContext EventKind = "context" EventFailed EventKind = "failed" // EventHandover says the step was NOT stopped when its run's context // ended: it goes on where it runs, and Message carries a Handover (JSON) // saying how to find it again (#125). Only an executor whose steps // outlive the scheduler sends it -- the remote one, on a shutdown. EventHandover EventKind = "handover" )
type Executor ¶
type Executor interface {
Name() string
Execute(ctx context.Context, t TaskExec) (<-chan Event, error)
Cancel(ctx context.Context, execID string) error
}
Executor runs a task and reports what happens.
type FuncTask ¶
FuncTask adapts a function to the Task interface, for the cases where a type of its own would be ceremony with no gain.
type Handover ¶
type Handover struct {
Instance string `json:"instance,omitempty"`
ExecutionID string `json:"execution_id"`
Attempt int `json:"attempt"`
After int64 `json:"after"`
}
Handover is where a step that outlived its scheduler can be found again: the agent instance that holds it, its execution id, the attempt it is, and the last line the engine read of it.
type Input ¶
type Input struct {
NodeID string
With map[string]any
// Log emits one line into the event stream. It exists so the task can report
// progress without knowing about channels or the executor.
Log func(msg string)
}
Input is what the task receives.
type Reattacher ¶
type Reattacher interface {
Reattach(ctx context.Context, t TaskExec, h Handover) (<-chan Event, error)
}
Reattacher is an executor that can follow a step a previous scheduler handed over, instead of starting it (#125). Optional, asked for by type assertion: only an executor whose steps outlive the scheduler has anything to follow.
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry holds the available tasks. Safe for concurrent use because the dispatcher queries it from several goroutines.
func NewRegistry ¶
func NewRegistry() *Registry
func (*Registry) MustRegister ¶
MustRegister registra e entra em panico se falhar.
For use in `init()` or at boot: an invalid registration is a programming error, and failing at start beats finding out on the first scheduled run.
type Task ¶
Task is a unit of work written in Go, compiled into the binary.
The rule is categorical: "Do not execute arbitrary code received through the API. Local tasks must be compiled and registered in the runtime". The registry exists to make that structural — the YAML can only name something that is already in the binary, never supply the code.
type TaskExec ¶
type TaskExec struct {
ExecutionID string
NodeID string
// Workflow, RunID and Attempt do not change the run — they identify it.
// In Kubernetes they become the pod's labels, and they are what makes it
// possible to find "that run's pods" without searching by name.
Workflow string
RunID string
// The STEP's attempt, within one execution of the run.
Attempt int
// TentativaDoRun is the RUN's, counted by the dispatcher. Both go into the
// pod's name: without the second, a dispatcher retry recreates the run from
// scratch (the step at attempt 0 again) and runs into the previous pod.
RunAttempt int
// MapIndex is which instance of a `for_each:` step this is, or -1
// (run.Unmapped) for a step that is not mapped. Its zero value is a real
// index, so whoever builds a TaskExec sets it -- `build` always does.
MapIndex int
Command string // a shell line, for the ProcessExecutor
Action string // the name in the registry, for the GoExecutor
With map[string]any
// Image is this step's runtime. Empty in local mode (the command runs on the
// instance itself); required in Kubernetes, where it IS the pod.
Image string
// Shell chooses between `sh -c "line"` and plain argv. It matters for a
// distroless image, which has no shell at all.
Shell bool
// The pod's resources, in Kubernetes's format. Ignored in local mode, where
// a process's limit is the machine's.
CPU, Memoria string
CPUMax, MemoriaMax string
WorkDir string
Env map[string]string
// Secrets are variables whose value the engine does NOT carry: the map is
// variable-name -> `secret-name/key`, and the executor is what resolves it.
//
// They are kept apart from Env for exactly that reason. If the value arrived
// resolved here, it would pass through the dispatcher, through the task
// assembly log and through any TaskExec dump somebody writes later.
Secrets map[string]string
// OutputPath is where the step writes what it publishes for the steps that
// depend on it. It reaches the step as BREVIS_OUTPUT.
//
// The RUNNER picks it and the EXECUTOR may override it, because only the
// executor knows what a path means in its world: a temporary file in the
// engine's filesystem is meaningless inside a pod, where the answer is
// /dev/termination-log -- which the engine already reads to get the exit
// code.
OutputPath string
// A zero Timeout means no limit. A timeout was asked for early on, and
// leaving the default open is deliberate — imposing an arbitrary limit would
// kill legitimately long tasks.
Timeout time.Duration
}
TaskExec is what gets asked to run. Deliberately poor: the executor knows nothing of workflows, dependencies or schedules.
`Command` and `Action` are exclusive: the first goes to the ProcessExecutor, the second resolves in the Go task registry. The runner is what chooses the executor, not the executor itself.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package kubernetes runs each step of a workflow as a POD of its own.
|
Package kubernetes runs each step of a workflow as a POD of its own. |
|
Package local implements running processes on the host.
|
Package local implements running processes on the host. |
|
Package remote runs a step on a host the engine does not manage.
|
Package remote runs a step on a host the engine does not manage. |