Documentation
¶
Overview ¶
Package remote runs a step on a host the engine does not manage.
The third executor. `local` runs a process on this machine and `kubernetes` creates a pod; this one hands the work to an agent on a host somebody else administers -- an EC2 instance, a VM, a box in another cluster, a licensed tool that cannot be containerised, a GPU machine.
What made it possible ¶
Nothing in this package returns context to the engine, and that is deliberate. A pod publishes through the kubelet's termination message and a host outside the cluster has no equivalent, which was the blocker on this whole executor. It was answered elsewhere: a step publishes on stdout as a `@brevis:` marked line, on the pipe that already carries its phases. The runner reads it out of the log stream, so this executor only has to stream logs faithfully and the return path follows.
The four decisions ¶
Each is a place where a thin version would ship and then hurt. Three of them resolve to something this codebase already does; the reasoning is in the commit that introduced this package.
STREAMING. NDJSON with a sequence number per line, and a bounded ring on the agent so a dropped connection resumes rather than failing a forty-minute load. The sequence number is not tidiness: replayed log lines, phases, published context and gauges are all idempotent, but a COUNTER metric is not -- the engine's registry does `Add`, so a replayed line adds twice and reads high while nothing fails.
CANCEL. See Cancel.
SECRETS. The agent resolves them. The engine has never transported a secret value -- kubernetes emits a secretKeyRef for the kubelet, local passes along variables already in its own environment -- and this executor does not become the first. See the `Secrets` field of the request below.
LIVENESS. An agent that died is indistinguishable from one running a long step. The stream carries a periodic `alive` line, which is the lease: the same claim-renew-sweep shape as queue.Recover and alerts.Outbox.Recover, riding a channel that already exists rather than opening a second one.
Index ¶
- Constants
- Variables
- type Agent
- type Executor
- func (e *Executor) Cancel(ctx context.Context, execID string) error
- func (e *Executor) Execute(ctx context.Context, t execution.TaskExec) (<-chan execution.Event, error)
- func (e *Executor) Name() string
- func (e *Executor) Reattach(ctx context.Context, t execution.TaskExec, h execution.Handover) (<-chan execution.Event, error)
- type GapError
- type HTTPAgent
- func (a HTTPAgent) At(instance string) string
- func (a HTTPAgent) Cancel(ctx context.Context, instance, execID string) error
- func (a HTTPAgent) Resume(ctx context.Context, instance, execID string, r Resume) (io.ReadCloser, error)
- func (a HTTPAgent) Start(ctx context.Context, r StartRequest) (Session, error)
- type Line
- type Replicas
- type Resume
- type Session
- type StartRequest
Constants ¶
const ( // KindStarted says the process exists on the host. Nothing before it can be // attributed to the step. KindStarted = "started" // KindLog is one line the step wrote. `Stream` keeps stdout apart from // stderr, which the pod executor cannot do at all -- Kubernetes merges // them -- so this executor is strictly better informed there. KindLog = "log" // KindAlive is the lease renewal. It carries no information about the step; // its arrival IS the information. A step can legitimately print nothing for // an hour, so silence cannot mean death and this is what tells the two // apart. KindAlive = "alive" // KindExit ends the stream. Everything after it is ignored. KindExit = "exit" // KindFailed is the agent reporting it could not run the step at all -- // an unresolvable secret, a missing working directory, a refused protocol // version. Distinct from an exit code because "the step ran and failed" and // "the step never ran" are different facts. KindFailed = "failed" )
Line kinds. The agent emits one JSON object per line, and the line is the framing: the engine's scanner breaks on newlines, the `@brevis:` protocol inside a step's output is line-oriented, and SSE's own framing would buy nothing on top of that.
const ( DefaultAliveEvery = 10 * time.Second DefaultLeaseLimit = 30 * time.Second )
DefaultAliveEvery is how often an agent should renew its lease, and DefaultLeaseLimit is how long the engine waits before deciding it died.
Three renewals of headroom, which is the ratio the queue's own recovery uses: a limit that is a small multiple of the renewal survives one slow moment and still catches a dead process quickly. Too short kills a step whose host was merely busy; too long is the thing this prevents.
const HeaderInstance = "Brevis-Instance"
HeaderInstance is how an agent behind a shared address says where to come back to.
An agent is STATEFUL PER EXECUTION -- a ring and a process handle in its own memory -- so Resume and Cancel only mean anything at the instance that took the work. One agent behind one address needs none of this; a POOL behind one address, which is what a Deployment with an HPA is, breaks without it: the engine resumes through the address, reaches a different replica, is told "not running here", and fails a step that is still running somewhere else.
It rides a RESPONSE HEADER on the start, and the choice matters. A header arrives before any byte of the body, so a connection that dies before the first line still leaves the engine knowing whom to ask. Carrying it on the `started` line instead would leave exactly that window open, and it is the window where an orphan process is created.
The value is a base URL -- `https://dbt-runner-2.dbt.svc:9443` -- which the agent is TOLD, not one it discovers: only the deployment knows which of its names is routable from the engine. An agent that advertises nothing is one agent at one address, and the engine keeps using the address it has.
const HeaderRefusal = "Brevis-Refusal"
HeaderRefusal names why a start that ran nothing was refused (#146).
The status alone cannot: 503 is what a draining agent answers, and also what a balancer with nobody behind it answers. Only the first has a replica that would run the step, so only the first is a reason to send it elsewhere rather than to spend an attempt.
const Protocol = 1
Protocol is the wire version. It goes in every request, and an agent that does not recognise it must refuse rather than guess: a silent mismatch between two programs on different release cycles is the failure this number exists to make loud.
const RefusalDraining = "draining"
RefusalDraining is HeaderRefusal's value for an agent on its way out.
Variables ¶
var ErrAgentDraining = fmt.Errorf("%w: it is draining", errUnavailable)
ErrAgentDraining is a 503 from an agent on its way out (#146), which another replica of the same host would run. It wraps errUnavailable, so it is sent again like any start that ran nothing.
Kept apart from a bare 503 because that one is also a balancer with nobody behind it -- a host that is down -- and only a refusal that says it is draining is a reason to look for another replica instead of spending an attempt.
var ErrUnknownExecution = execution.ErrUnknownExecution
ErrUnknownExecution is an agent saying it has no execution by that id. Said apart on the wire (404) so the engine can tell it from an agent it cannot reach. The contract's own error, named here too because the wire is where it is born.
Functions ¶
This section is empty.
Types ¶
type Agent ¶
type Agent interface {
// Start hands over the work and returns the stream, with the instance that
// took it when the agent advertises one.
Start(ctx context.Context, req StartRequest) (Session, error)
// Resume picks the same execution's stream back up after `after`, AT THE
// INSTANCE THAT TOOK IT. It returns a GapError when its buffer no longer
// reaches that far.
Resume(ctx context.Context, instance, execID string, r Resume) (io.ReadCloser, error)
// Cancel asks the host to stop the process, at that same instance. See
// Executor.Cancel for what its error means, which is the part that matters.
Cancel(ctx context.Context, instance, execID string) error
}
Agent is the host's side, as this executor needs it.
An interface rather than an HTTP client directly, and that is not ceremony: the contract has to be settled by tests before a second program exists to keep in sync with it. An agent written first would become the specification by accident, and the specification would then be whatever that program happened to do.
type Executor ¶
type Executor struct {
Agent Agent
Host string
// LeaseLimit is how long the stream may go without any line -- including
// the agent's periodic `alive` -- before the step is failed. Zero takes
// DefaultLeaseLimit.
LeaseLimit time.Duration
// HandOver lets a shutdown leave the step running on its agent, to be
// reattached by the next scheduler (#125), instead of cancelling it. Only a
// shutdown: any other end of the run's context still cancels. Off is
// #122's behaviour.
HandOver bool
// Retries is how many times a dropped connection is resumed before the step
// is failed. Zero takes defaultRetries.
//
// It is bounded rather than endless because an agent that drops the
// connection every second is not a network blip, and retrying forever turns
// a broken host into a step that never ends.
Retries int
// contains filtered or unexported fields
}
Executor runs steps on one host.
func (*Executor) Cancel ¶
Cancel asks the host to stop the process, and is honest about what it achieved.
Decision 2, and the only one of the four with no precedent in this codebase -- nothing here has ever had to cancel something it did not spawn.
Two things this does NOT hide:
- the local context is cancelled first, so the engine stops reading the stream whatever the agent does. The step's own outcome is then decided by the caller, not by this stream going quiet.
- if the agent is unreachable, the error says so plainly. The step is cancelled HERE and may well still be running THERE, which is a real state and worth naming rather than showing as a spinner that never resolves.
func (*Executor) Execute ¶
func (e *Executor) Execute(ctx context.Context, t execution.TaskExec) (<-chan execution.Event, error)
Execute hands the step to the agent and turns its stream into events.
func (*Executor) Reattach ¶
func (e *Executor) Reattach(ctx context.Context, t execution.TaskExec, h execution.Handover) (<-chan execution.Event, error)
Reattach follows a step a previous scheduler handed over (#125): the same execution, on the instance that holds it, through the same pump a dropped connection is resumed by -- lease, retries and verdict included.
From the START of the agent's ring, so the log comes back whole: the scheduler that handed it over never wrote the lines it had read. When the ring no longer reaches its start, from the last line that scheduler read -- the head of the log is lost, the outcome is not.
An error says why it cannot, and the three are kept apart because the runner does a different thing for each: a GapError (not even the last line read is left), ErrUnknownExecution (the agent lost it), or anything else (the agent could not be reached).
type GapError ¶
GapError is the agent saying it can no longer replay from where the engine stopped: its ring has moved past that point.
It is a distinct answer rather than a truncated stream because the difference matters. A step whose output is missing an unknown number of lines is worse than a step that failed -- its counters are wrong by an unknown amount and nothing says so.
type HTTPAgent ¶
type HTTPAgent struct {
BaseURL string
Token string
Client *http.Client
// StartBackoff is the wait before each further try of a start that ran
// nothing (#136); its length + 1 is how many tries. Nil takes
// defaultStartBackoff. A test passes milliseconds.
StartBackoff []time.Duration
// Replicas reaches a pool behind one name, and steers a start away from a
// replica that refused one for draining (#146). Nil dials the name as the
// system resolves it, which is one agent at one address.
Replicas *Replicas
}
HTTPAgent is the real agent, over HTTP.
Deliberately thin: everything that decides behaviour is in Executor and is tested against a fake. This is transport.
func (HTTPAgent) At ¶
At is the address to use for an execution already under way.
The instance when it is one the installation already trusts, and the configured address otherwise. Empty is not a degraded case: it is one agent at one address, which is every deployment that does not put a pool behind a Service.
BOUNDED, because the advertisement makes this engine POST to an address the AGENT chose. The token is not the new exposure -- every agent already receives it on every request -- but an arbitrary outbound POST from the engine is a capability that did not exist before the mechanism.
SCHEME AND PORT ARE NOT A BOUND, and a test is what said so: `http://attacker.example.com:9443` shares both with the configured address, so a rule checking only those allows every host on earth. It looks like a bound and is one comparison away from nothing.
What separates a replica from a stranger is the DOMAIN. A pool differs in the first DNS label -- `brevis-agent-2` for `brevis-agent-0` -- and shares everything after it. So: same scheme, same port, same domain. No second configuration, because the bound is derived from the address an operator already wrote into BREVIS_HOSTS.
An address with no domain to share -- a bare host, an IP -- allows only itself. There is no first label to vary, so there is no pool to allow.
Exported for the test that pins all of it. A refused advertisement is IGNORED rather than fatal: the engine falls back to the address it has, and on a pool that surfaces as the agent's own "not running here" rather than as a silence.
func (HTTPAgent) Start ¶
Start sends a step, again when the answer says nothing ran (#136).
Two answers say that: a 503 -- a draining agent refuses before it reserves the id, and a balancer answers it with nobody behind -- and a refused connection, which a pod published before it binds gives. During a rolling restart of a pool both are ordinary, and failing an attempt over them would spend a workflow's retries on a deploy.
Sending again cannot run a step twice. If a first start did reach an agent after all, the second gets "already running" (#130), not a second process.
A start ONLY. A resume or a cancel is about an execution one particular agent holds, and the engine's reconnect loop already decides about those.
type Line ¶
type Line struct {
// Seq is monotonic from 1, per execution, and never reused. It is what
// makes a resumed stream safe: see the package comment, decision 1.
Seq int64 `json:"seq"`
Kind string `json:"kind"`
Stream string `json:"stream,omitempty"`
Message string `json:"message,omitempty"`
// Code is the process's exit status, on a KindExit line.
Code int `json:"code,omitempty"`
}
Line is one NDJSON record from the agent.
type Replicas ¶
type Replicas struct {
// Lookup resolves a host to the addresses (host:port) to dial. Nil takes
// the system resolver. A test passes a map.
Lookup func(ctx context.Context, host, port string) ([]string, error)
// TLSClientConfig is for an agent behind a certificate the system does not
// trust. Nil is the system's roots, as without Replicas.
TLSClientConfig *tls.Config
// contains filtered or unexported fields
}
Replicas is how the engine reaches a host that is a POOL of agents behind one name (#146).
The name is a Service's, and it resolves to every pod of the pool, a draining one included -- it has to, because the engine comes back to a draining agent to resume the step it handed over (publishNotReadyAddresses, #136). What the name cannot say is which of them takes new work. So the engine dials the addresses itself, and after one refuses a start for draining, it goes to the others first until drainingSkip has passed.
One per host, shared by every copy of its HTTPAgent: it holds the connections and what the addresses last answered.
type Resume ¶
type Resume struct {
Protocol int `json:"protocol"`
After int64 `json:"after"`
Reason string `json:"reason,omitempty"`
}
Resume is what the engine sends to pick a stream back up.
type Session ¶
type Session struct {
Instance string
Stream io.ReadCloser
}
Session is what Start answers: the stream, and where to come back to.
`Instance` is empty for an agent that advertises nothing, which is one agent at one address and every deployment that exists today. The engine then keeps using the address it was configured with, and nothing about this type changes what happens. See HeaderInstance.
type StartRequest ¶
type StartRequest struct {
Protocol int `json:"protocol"`
ExecutionID string `json:"execution_id"`
NodeID string `json:"node_id"`
// Identifying, not behavioural: they let an operator on the host see which
// run a process belongs to, which is the whole reason the pod carries them
// as labels.
Workflow string `json:"workflow"`
RunID string `json:"run_id"`
Attempt int `json:"attempt"`
Command string `json:"command"`
WorkDir string `json:"workdir,omitempty"`
Env map[string]string `json:"env,omitempty"`
// Secrets is variable-name -> `secret-name/key`, UNRESOLVED, and it is the
// point of decision 3.
//
// The engine does not hold these values and does not learn them. The agent
// resolves each coordinate against its own store, under its own allowlist,
// on the host that administers them. That is what the kubelet does for a
// pod and what the OS does for a local process; sending resolved values
// over mTLS instead would protect the wire and do nothing about the
// dispatcher, the task assembly log, or any TaskExec dump somebody writes
// later -- which is what `TaskExec.Secrets`' own comment was written about.
//
// An agent that cannot resolve one must FAIL the step naming the
// coordinate. Starting a step with a variable silently unset is how a 401
// three layers down gets blamed on the wrong service.
Secrets map[string]string `json:"secrets,omitempty"`
// TimeoutSeconds is zero for no limit, matching TaskExec.Timeout.
TimeoutSeconds int `json:"timeout_seconds,omitempty"`
}
StartRequest is what the engine asks the agent to run.
It is TaskExec's fields, minus everything that means nothing on a host the engine does not own -- an image, a pod's CPU limits -- and it is deliberately no richer than that. The agent knows nothing of workflows, dependencies or schedules, exactly as the Executor interface says of every executor.