remote

package
v0.13.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 11, 2026 License: MIT Imports: 11 Imported by: 0

Documentation

Overview

Package remote runs a step on a host the engine does not manage.

The third executor. `local` runs a process on this machine and `kubernetes` creates a pod; this one hands the work to an agent on a host somebody else administers -- an EC2 instance, a VM, a box in another cluster, a licensed tool that cannot be containerised, a GPU machine.

What made it possible

Nothing in this package returns context to the engine, and that is deliberate. A pod publishes through the kubelet's termination message and a host outside the cluster has no equivalent, which was the blocker on this whole executor. It was answered elsewhere: a step publishes on stdout as a `@brevis:` marked line, on the pipe that already carries its phases. The runner reads it out of the log stream, so this executor only has to stream logs faithfully and the return path follows.

The four decisions

Each is a place where a thin version would ship and then hurt. Three of them resolve to something this codebase already does; the reasoning is in the commit that introduced this package.

  1. STREAMING. NDJSON with a sequence number per line, and a bounded ring on the agent so a dropped connection resumes rather than failing a forty-minute load. The sequence number is not tidiness: replayed log lines, phases, published context and gauges are all idempotent, but a COUNTER metric is not -- the engine's registry does `Add`, so a replayed line adds twice and reads high while nothing fails.

  2. CANCEL. See Cancel.

  3. SECRETS. The agent resolves them. The engine has never transported a secret value -- kubernetes emits a secretKeyRef for the kubelet, local passes along variables already in its own environment -- and this executor does not become the first. See the `Secrets` field of the request below.

  4. LIVENESS. An agent that died is indistinguishable from one running a long step. The stream carries a periodic `alive` line, which is the lease: the same claim-renew-sweep shape as queue.Recover and alerts.Outbox.Recover, riding a channel that already exists rather than opening a second one.

Index

Constants

View Source
const (
	// KindStarted says the process exists on the host. Nothing before it can be
	// attributed to the step.
	KindStarted = "started"

	// KindLog is one line the step wrote. `Stream` keeps stdout apart from
	// stderr, which the pod executor cannot do at all -- Kubernetes merges
	// them -- so this executor is strictly better informed there.
	KindLog = "log"

	// KindAlive is the lease renewal. It carries no information about the step;
	// its arrival IS the information. A step can legitimately print nothing for
	// an hour, so silence cannot mean death and this is what tells the two
	// apart.
	KindAlive = "alive"

	// KindExit ends the stream. Everything after it is ignored.
	KindExit = "exit"

	// KindFailed is the agent reporting it could not run the step at all --
	// an unresolvable secret, a missing working directory, a refused protocol
	// version. Distinct from an exit code because "the step ran and failed" and
	// "the step never ran" are different facts.
	KindFailed = "failed"
)

Line kinds. The agent emits one JSON object per line, and the line is the framing: the engine's scanner breaks on newlines, the `@brevis:` protocol inside a step's output is line-oriented, and SSE's own framing would buy nothing on top of that.

View Source
const (
	DefaultAliveEvery = 10 * time.Second
	DefaultLeaseLimit = 30 * time.Second
)

DefaultAliveEvery is how often an agent should renew its lease, and DefaultLeaseLimit is how long the engine waits before deciding it died.

Three renewals of headroom, which is the ratio the queue's own recovery uses: a limit that is a small multiple of the renewal survives one slow moment and still catches a dead process quickly. Too short kills a step whose host was merely busy; too long is the thing this prevents.

View Source
const Protocol = 1

Protocol is the wire version. It goes in every request, and an agent that does not recognise it must refuse rather than guess: a silent mismatch between two programs on different release cycles is the failure this number exists to make loud.

Variables

This section is empty.

Functions

This section is empty.

Types

type Agent

type Agent interface {
	// Start hands over the work and returns the stream of NDJSON lines.
	Start(ctx context.Context, req StartRequest) (io.ReadCloser, error)

	// Resume picks the same execution's stream back up after `after`.
	// It returns a GapError when its buffer no longer reaches that far.
	Resume(ctx context.Context, execID string, r Resume) (io.ReadCloser, error)

	// Cancel asks the host to stop the process. See Executor.Cancel for what
	// its error means, which is the part that matters.
	Cancel(ctx context.Context, execID string) error
}

Agent is the host's side, as this executor needs it.

An interface rather than an HTTP client directly, and that is not ceremony: the contract has to be settled by tests before a second program exists to keep in sync with it. An agent written first would become the specification by accident, and the specification would then be whatever that program happened to do.

type Executor

type Executor struct {
	Agent Agent
	Host  string

	// LeaseLimit is how long the stream may go without any line -- including
	// the agent's periodic `alive` -- before the step is failed. Zero takes
	// DefaultLeaseLimit.
	LeaseLimit time.Duration

	// Retries is how many times a dropped connection is resumed before the step
	// is failed. Zero takes defaultRetries.
	//
	// It is bounded rather than endless because an agent that drops the
	// connection every second is not a network blip, and retrying forever turns
	// a broken host into a step that never ends.
	Retries int
	// contains filtered or unexported fields
}

Executor runs steps on one host.

func (*Executor) Cancel

func (e *Executor) Cancel(ctx context.Context, execID string) error

Cancel asks the host to stop the process, and is honest about what it achieved.

Decision 2, and the only one of the four with no precedent in this codebase -- nothing here has ever had to cancel something it did not spawn.

Two things this does NOT hide:

  • the local context is cancelled first, so the engine stops reading the stream whatever the agent does. The step's own outcome is then decided by the caller, not by this stream going quiet.
  • if the agent is unreachable, the error says so plainly. The step is cancelled HERE and may well still be running THERE, which is a real state and worth naming rather than showing as a spinner that never resolves.

func (*Executor) Execute

func (e *Executor) Execute(ctx context.Context, t execution.TaskExec) (<-chan execution.Event, error)

Execute hands the step to the agent and turns its stream into events.

func (*Executor) Name

func (e *Executor) Name() string

type GapError

type GapError struct {
	Wanted    int64 `json:"wanted"`
	Available int64 `json:"available"`
}

GapError is the agent saying it can no longer replay from where the engine stopped: its ring has moved past that point.

It is a distinct answer rather than a truncated stream because the difference matters. A step whose output is missing an unknown number of lines is worse than a step that failed -- its counters are wrong by an unknown amount and nothing says so.

func (GapError) Error

func (g GapError) Error() string

type HTTPAgent

type HTTPAgent struct {
	BaseURL string
	Token   string
	Client  *http.Client
}

HTTPAgent is the real agent, over HTTP.

Deliberately thin: everything that decides behaviour is in Executor and is tested against a fake. This is transport.

func (HTTPAgent) Cancel

func (a HTTPAgent) Cancel(ctx context.Context, execID string) error

func (HTTPAgent) Resume

func (a HTTPAgent) Resume(ctx context.Context, execID string, r Resume) (io.ReadCloser, error)

func (HTTPAgent) Start

type Line

type Line struct {
	// Seq is monotonic from 1, per execution, and never reused. It is what
	// makes a resumed stream safe: see the package comment, decision 1.
	Seq int64 `json:"seq"`

	Kind    string `json:"kind"`
	Stream  string `json:"stream,omitempty"`
	Message string `json:"message,omitempty"`

	// Code is the process's exit status, on a KindExit line.
	Code int `json:"code,omitempty"`
}

Line is one NDJSON record from the agent.

func Decode

func Decode(raw []byte) (Line, error)

Decode reads one NDJSON line.

func (Line) Validate

func (l Line) Validate() error

Validate refuses a line that would be worse to act on than to reject.

The agent is a separate program on a separate release cycle, on a host this engine does not administer. Trusting its output shape is how a bad deploy over there becomes a confusing failure over here.

type Resume

type Resume struct {
	Protocol int    `json:"protocol"`
	After    int64  `json:"after"`
	Reason   string `json:"reason,omitempty"`
}

Resume is what the engine sends to pick a stream back up.

type StartRequest

type StartRequest struct {
	Protocol    int    `json:"protocol"`
	ExecutionID string `json:"execution_id"`
	NodeID      string `json:"node_id"`

	// Identifying, not behavioural: they let an operator on the host see which
	// run a process belongs to, which is the whole reason the pod carries them
	// as labels.
	Workflow string `json:"workflow"`
	RunID    string `json:"run_id"`
	Attempt  int    `json:"attempt"`

	Command string            `json:"command"`
	WorkDir string            `json:"workdir,omitempty"`
	Env     map[string]string `json:"env,omitempty"`

	// Secrets is variable-name -> `secret-name/key`, UNRESOLVED, and it is the
	// point of decision 3.
	//
	// The engine does not hold these values and does not learn them. The agent
	// resolves each coordinate against its own store, under its own allowlist,
	// on the host that administers them. That is what the kubelet does for a
	// pod and what the OS does for a local process; sending resolved values
	// over mTLS instead would protect the wire and do nothing about the
	// dispatcher, the task assembly log, or any TaskExec dump somebody writes
	// later -- which is what `TaskExec.Secrets`' own comment was written about.
	//
	// An agent that cannot resolve one must FAIL the step naming the
	// coordinate. Starting a step with a variable silently unset is how a 401
	// three layers down gets blamed on the wrong service.
	Secrets map[string]string `json:"secrets,omitempty"`

	// TimeoutSeconds is zero for no limit, matching TaskExec.Timeout.
	TimeoutSeconds int `json:"timeout_seconds,omitempty"`
}

StartRequest is what the engine asks the agent to run.

It is TaskExec's fields, minus everything that means nothing on a host the engine does not own -- an image, a pod's CPU limits -- and it is deliberately no richer than that. The agent knows nothing of workflows, dependencies or schedules, exactly as the Executor interface says of every executor.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL