Documentation
¶
Overview ¶
Package docker runs Podium tasks as containers on a local Docker engine and streams ordered lifecycle events for them.
Index ¶
- Constants
- Variables
- type AdoptRequest
- type ArtifactPayload
- type ArtifactStored
- type ArtifactUpload
- type ArtifactUploader
- type EgressPolicy
- type ErrorPayload
- type Event
- type Executor
- func (e *Executor) Adopt(ctx context.Context, req AdoptRequest, events chan<- Event) (Result, error)
- func (e *Executor) Cancel(taskID string)
- func (e *Executor) Close() error
- func (e *Executor) Images() *ImageCache
- func (e *Executor) Inject(taskID, text string) error
- func (e *Executor) ListOwned(ctx context.Context) ([]OwnedContainer, error)
- func (e *Executor) PruneImages(ctx context.Context, req PruneRequest) ([]ImageRecord, error)
- func (e *Executor) Run(ctx context.Context, req Request, events chan<- Event) (Result, error)
- func (e *Executor) ServerVersion() string
- func (e *Executor) TaskDir(taskID string) string
- func (e *Executor) Teardown(ctx context.Context, taskID string, keepWorkspace bool) error
- type ExitedPayload
- type FinishedPayload
- type ImageCache
- type ImageRecord
- type LogPayload
- type MessagePayload
- type Options
- type OwnedContainer
- type PruneRequest
- type PullingPayload
- type RegistryCredential
- type Request
- type Result
- type Secret
- type StepPayload
- type Usage
Constants ¶
const ( KindProvisioning = "provisioning" KindPulling = "pulling" KindStarted = "started" KindLog = "log" KindStep = "step" KindArtifact = "artifact" KindExited = "exited" KindFinished = "finished" KindError = "error" KindMessage = "message" )
Event kinds emitted by Executor.Run. They are the canonical lowercase TaskEvent kind names from the wire contract; the node daemon maps them onto podiumv1.TaskEvent.
const ( StreamStdout = "stdout" StreamStderr = "stderr" StreamSidecar = "sidecar" )
Log stream names carried by LogPayload.
const ( LabelTask = "podium.task" LabelLease = "podium.lease" LabelRole = "podium.role" LabelSidecar = "podium.sidecar" // RoleTask marks the single task container of a run. RoleTask = "task" // RoleSidecar marks a sibling container the task talks to by name. RoleSidecar = "sidecar" )
Container labels applied to every resource the executor creates.
const AutoArtifactDir = "/workspace/.podium/artifacts"
AutoArtifactDir is collected from the task container when it exits: every regular file under it becomes an artifact named by its path relative to the directory. It is inside the workspace volume, so a sidecar or an earlier step can put things there too.
const ImageCacheFile = "images.json"
ImageCacheFile is where the node records the images it pulled itself, inside the data dir. It is both the LRU bookkeeping and — far more importantly — the allow-list.
const LowWatermarkMargin = 0.10
LowWatermarkMargin is how far below the high watermark a prune drives disk usage before it stops, so a node at the line does not prune on every sweep.
const MaxArtifactBytes int64 = 512 << 20
MaxArtifactBytes mirrors the control plane's own limit. The node checks it first so a 600 MB file is refused before it crosses the wire rather than after; the server's check is the authoritative one.
const StepReattached = "node/reattached"
StepReattached is the marker an adopted run puts in a task's history at the point where one daemon incarnation handed the container to the next.
Variables ¶
var ErrNotPodiumImage = errors.New("docker: image was not pulled by podium; refusing to remove it")
ErrNotPodiumImage is what the remover answers for an image Podium did not pull.
This is the load-bearing safety property of the whole file, so it is worth saying why it exists rather than a comment on the caller. A Podium node shares its Docker engine with everything else on the machine: on a developer's laptop that is their own work, and on a shared build host it is somebody else's. "Remove images no running task references, least recently used first" would, on such an engine, delete images that have nothing to do with Podium and cannot be recovered without whatever built them. So Podium's cache is not "the engine's images"; it is "the images Podium pulled", recorded at the moment it pulled them, and nothing else is ever a candidate. The check lives in the one function that calls ImageRemove, not in the policy above it, so a future caller cannot get it wrong.
Functions ¶
This section is empty.
Types ¶
type AdoptRequest ¶
type AdoptRequest struct {
TaskID string
LeaseID string
// FromSeq is the highest sequence number the previous incarnation of the
// daemon assigned to this task. Adopt numbers its events from FromSeq+1 so
// they never collide with rows the server already stored.
FromSeq uint64
// SkipStdout and SkipStderr are how many bytes of each stream the server has
// already acknowledged. Docker replays a container's whole output on every
// attach, so those bytes are read and discarded rather than sent twice.
SkipStdout int64
SkipStderr int64
}
AdoptRequest re-attaches to a task container that outlived the daemon.
type ArtifactPayload ¶
type ArtifactPayload struct {
ArtifactID string
Name string
ObjectKey string
SizeBytes int64
ContentType string
}
ArtifactPayload reports one artifact the control plane has stored. The node turns it into a TaskEvent of kind artifact.
type ArtifactStored ¶
ArtifactStored is what the control plane says it kept.
type ArtifactUpload ¶
type ArtifactUpload struct {
TaskID string
LeaseID string
Name string
ContentType string
Size int64
Body io.Reader
}
ArtifactUpload is one file on its way from a container to the control plane.
type ArtifactUploader ¶
type ArtifactUploader interface {
UploadArtifact(ctx context.Context, up ArtifactUpload) (ArtifactStored, error)
}
ArtifactUploader hands an artifact to the control plane. The node daemon implements it with NodeService.UploadArtifact; a nil one means this node cannot store artifacts, which a task must survive.
type EgressPolicy ¶
type EgressPolicy interface {
Apply(ctx context.Context, taskID, networkID string, allow []string) error
Revoke(ctx context.Context, taskID, networkID string) error
}
EgressPolicy is the hook a future egress allow-list plugs into.
Today a task's network is a plain bridge with `internal: false`: the task and its sidecars reach each other, the internet, and whatever else the node's host routes. A tailnet is included — a container leaves through the host and inherits its routes, so on a tailnet node a task can open the control plane's own 443, which is how a task drives a browser against a live stack. Nothing here restricts that; what confines a worker today is the Tailscale ACL, not this file. Restricting the internet half means programming the node's firewall for the task's subnet, which is host-specific (nftables on Linux, nothing usable on Docker Desktop) and is deliberately not part of this step.
When it arrives, Apply is called after the network is created and before any container joins it, and Revoke during Teardown after the last container is gone. An implementation that cannot enforce the list must return an error rather than silently allowing everything: a task that believes it is confined and is not is worse than one that fails.
type ErrorPayload ¶
ErrorPayload reports something that went wrong. Retryable is true for transient pull and engine errors and false for a spec, or an image, no retry can fix. AbortsRun is true when the error ended the run, which is what tells the control plane the task is nobody's until it decides otherwise; a failure the run survived — an artifact that could not be stored — leaves it false.
type Event ¶
Event is one ordered observation about a running task. Seq starts at 1 for each task and increases by exactly one per event.
type Executor ¶
type Executor struct {
// contains filtered or unexported fields
}
Executor runs task containers on one Docker engine.
func New ¶
New connects to the Docker engine, verifies it is usable (API >= 1.43, cgroup v2) and prepares the executor's state directory.
func (*Executor) Adopt ¶
func (e *Executor) Adopt(ctx context.Context, req AdoptRequest, events chan<- Event) (Result, error)
Adopt re-attaches to the container of a task this engine is already running, which is how a restarted daemon finishes a task it started before the restart. It emits the tail of the run: the output the caller says has not been delivered yet, then exited and finished.
Like Run, Adopt does not close events and leaves Teardown to the caller.
func (*Executor) Cancel ¶
Cancel asks the task's container to stop: SIGTERM, then SIGKILL after a 30s grace period. It is idempotent, returns immediately, and does nothing for a task this executor is not running. The run reports the resulting exit normally; deciding that the task was "cancelled" is the caller's job.
func (*Executor) Close ¶
Close releases the Docker client and the private socket directory, if this executor needed one. It does not stop running tasks.
func (*Executor) Images ¶
func (e *Executor) Images() *ImageCache
Images is the node's image cache, so the daemon can record use and drive the prune.
func (*Executor) Inject ¶
Inject delivers one human message into a running task's inbox. It does not cancel the task. A task this executor is not running is an error, so the caller can fall back to queueing the message as a new turn.
func (*Executor) ListOwned ¶
func (e *Executor) ListOwned(ctx context.Context) ([]OwnedContainer, error)
ListOwned returns every container on this engine labelled podium.task, running or not. Step 12's reconciliation uses it to find orphans after a daemon restart.
func (*Executor) PruneImages ¶
func (e *Executor) PruneImages(ctx context.Context, req PruneRequest) ([]ImageRecord, error)
PruneImages removes the least recently used of *Podium's own* images until the disk is back under the low watermark. It never touches an image Podium did not pull, never runs a bulk prune of any kind, and reports what it removed.
The whole feature is off unless the operator turned it on: see Config.ImageCachePrune.
func (*Executor) Run ¶
Run executes req to completion, emitting ordered events on events. It returns a Result for any run that produced an exit code, even a non-zero one; an error means the task could not be run at all, in which case an error event was emitted first and every resource this call created has been removed.
Run never closes events; the channel belongs to the caller. On success the caller is responsible for calling Teardown.
func (*Executor) ServerVersion ¶
ServerVersion is the Docker Engine version this executor negotiated with, which the node reports at enrollment.
func (*Executor) TaskDir ¶
TaskDir is the per-task state directory inside the data dir. It is created by Run and removed by Teardown; the node daemon keeps its sequence-space bookmark there.
func (*Executor) Teardown ¶
Teardown removes everything a task owns, in the order the design requires: its sidecars first (stop signal, 10s grace, SIGKILL, then remove), then its task container, then its network, then its workspace volume unless keepWorkspace, then its state directory. Sidecars go first because the task container is the thing that talks to them; killing a database out from under a still-running client is how you get confusing logs on the way out.
It is idempotent — resources that are already gone are not an error.
type ExitedPayload ¶
ExitedPayload reports the container's exit as observed from the Docker API.
type FinishedPayload ¶
FinishedPayload closes out a task run with its exit code and resource usage.
type ImageCache ¶
type ImageCache struct {
// contains filtered or unexported fields
}
ImageCache is the node's record of the images it pulled, and the only thing that may authorise removing one. It is safe for concurrent use.
func NewImageCache ¶
func NewImageCache(dir string) *ImageCache
NewImageCache loads the cache from dir, creating an empty one when there is no file. A corrupt file yields an empty cache: forgetting which images are ours makes the prune do nothing, which is the safe direction to fail in.
func (*ImageCache) Owns ¶
func (c *ImageCache) Owns(ref string) bool
Owns reports whether ref is an image Podium pulled. It is the allow-list test.
func (*ImageCache) Pulled ¶
func (c *ImageCache) Pulled(ref, id string, size int64)
Pulled records that Podium pulled ref, which is the only way an image ever becomes a prune candidate.
func (*ImageCache) Records ¶
func (c *ImageCache) Records() []ImageRecord
Records is a copy of the cache, oldest use first.
func (*ImageCache) Used ¶
func (c *ImageCache) Used(ref string)
Used marks an image as recently used, so the LRU order reflects what tasks actually run. An image Podium never pulled is not recorded: being used does not make it ours.
type ImageRecord ¶
type ImageRecord struct {
// Ref is what was pulled, exactly as the spec named it.
Ref string `json:"ref"`
// ID is the engine's image id, which is what actually gets removed. A ref that has
// since been retagged elsewhere would otherwise take an unrelated image with it.
ID string `json:"id"`
// SizeBytes is what the engine reported at pull time; it is the estimate the prune
// uses to decide how many images to remove.
SizeBytes int64 `json:"size_bytes"`
PulledAt time.Time `json:"pulled_at"`
LastUsedAt time.Time `json:"last_used_at"`
}
ImageRecord is one image Podium pulled.
func SelectForPrune ¶
func SelectForPrune(records []ImageRecord, req PruneRequest) []ImageRecord
SelectForPrune is the whole LRU policy, as a pure function so it can be tested without an engine: given how full the disk is and what is in use, which of *Podium's own* images should go, least recently used first, and how far down that gets us.
It returns nothing at all when usage is below the high watermark, when nothing is in the cache, or when everything in the cache is in use.
type LogPayload ¶
type LogPayload struct {
Stream string
Sidecar string
Bytes []byte
// Offset is how many bytes of this source the container had produced by the end
// of this chunk, counting from the start of the container's life and including
// anything an adopting run skipped. It is the cursor Docker does not give us:
// ContainerLogs replays the whole stream on every attach, so a daemon adopting a
// container has to be told where to resume, and the control plane can only say so
// in these terms.
Offset int64
}
LogPayload carries one demultiplexed chunk of container output. Bytes is owned by the receiver and is never reused. Sidecar names which sidecar the chunk came from and is empty for the task container itself; it is only ever set when Stream is StreamSidecar.
type MessagePayload ¶
MessagePayload is the body of a KindMessage event, forwarded verbatim from the runner. Attachments are artifact names the reader should resolve; the node does not check them, and the text is untrusted content authored by the task.
type Options ¶
type Options struct {
// DataDir is the node's state directory. It must live on the same host as
// the Docker daemon; a remote DOCKER_HOST is unsupported.
DataDir string
// DockerHost overrides the daemon endpoint. Empty means use the
// environment (DOCKER_HOST, DOCKER_CONTEXT, then the default socket).
DockerHost string
// AllowPrivilegedSidecars honours a spec's privileged sidecar on this engine. Off by
// default: it hands that container root on the node's kernel, which is a decision
// about the machine and never a spec author's to make. See internal/node.Config.
AllowPrivilegedSidecars bool
// Logger defaults to slog.Default().
Logger *slog.Logger
}
Options configures New.
type OwnedContainer ¶
type OwnedContainer struct {
ID string
Name string
TaskID string
LeaseID string
Role string
Image string
State string
Status string
CreatedAt time.Time
}
OwnedContainer is a container this node created for a task, as reported by Executor.ListOwned.
type PruneRequest ¶
type PruneRequest struct {
// DiskUsedFraction is how full the filesystem holding the data dir is, 0..1.
DiskUsedFraction float64
// HighWatermark is the fraction above which pruning starts.
HighWatermark float64
// TotalBytes is the size of that filesystem, used to turn "how much to free" into
// bytes. Zero disables the size arithmetic and prunes one image per sweep.
TotalBytes int64
// InUse are the image references running tasks need. They are never candidates,
// whatever the cache says.
InUse []string
}
PruneRequest is one prune decision's inputs.
type PullingPayload ¶
PullingPayload reports image pull progress. Current and Total are bytes and are zero for status lines that carry no progress detail.
type RegistryCredential ¶
RegistryCredential is the login for one registry host, as the Assign delivered it. Password is plaintext; it lives in node memory for the length of the pulls and is zeroed with the run's secrets.
type Request ¶
type Request struct {
TaskID string
LeaseID string
Spec spec.TaskSpec
// Secrets are the resolved values of Spec.Secrets. SENSITIVE: they are plaintext,
// they are never logged, and Run zeroes them once the container has started.
Secrets []Secret
// Registries are the logins for the registries Spec's images are pulled from, as the
// Assign delivered them. SENSITIVE: plaintext, never logged, zeroed with Secrets.
Registries []RegistryCredential
// Artifacts is how files the task produces reach the control plane. A nil one means
// this node stores no artifacts: the task still runs, and anything it asks to keep
// becomes a retryable error event.
Artifacts ArtifactUploader
}
Request is one assignment to execute.
type Secret ¶
Secret is one resolved secret on its way into a task container. Value is plaintext; it exists in node memory for the length of one container start and is zeroed afterwards.
type StepPayload ¶
StepPayload carries a structured event: a sidecar reaching a lifecycle milestone ("sidecar/<name>" with status started, ready or failed), or anything the runner reported over its event socket that this node does not otherwise understand.