Documentation
¶
Overview ¶
Package runtime abstracts the container engine. The implementation is the embedded containerd (package containerd, ADR 0011).
Index ¶
- Constants
- Variables
- func CanDial(d Driver) bool
- func CanExec(d Driver) bool
- func CanMountVolumes(d Driver) bool
- func CanRunMembers(d Driver) bool
- func CanRunSystem(d Driver) bool
- func CanSandbox(d Driver) bool
- func CanStats(d Driver) bool
- func CheckContainerSpec(s ContainerSpec) error
- func CheckExecCommand(cmd []string) error
- func CheckFileMount(f FileMount) error
- func CheckSandboxID(id string) error
- func CheckSystemMount(m Mount) error
- func CheckVolumeMount(v VolumeMount) error
- func ContainsFold(line, needle []byte) bool
- func HasBridge(d Driver) bool
- func ValidMemberName(name string) bool
- type AgentStarter
- type Bridge
- type BridgeInspector
- type Container
- type ContainerDialer
- type ContainerSpec
- type Driver
- type ExecOptions
- type Execer
- type FileMount
- type Image
- type ImagePruner
- type LogFrame
- type LogOptions
- type LogPruner
- type LogQuery
- type LogRecord
- type LogSearcher
- type LogStream
- type LogTail
- type LogTailer
- type Member
- type MemberSpec
- type Mount
- type NetworkReleaser
- type NetworkSandboxer
- type OutputActivity
- type PlatformError
- type RegistryAuth
- type Sandbox
- type Stats
- type StatsReader
- type Stream
- type SystemRunner
- type SystemSpec
- type TermSize
- type VolumeMount
Constants ¶
const ( LabelManaged = "armada.managed" LabelApp = "armada.app" LabelRevision = "armada.revision" LabelReplica = "armada.replica" LabelSpecHash = "armada.spec-hash" // LabelFenceAfter is how many seconds without the control plane stop // the container (ADR 0041); absent: never. LabelFenceAfter = "armada.fence-after" )
Labels set on every container Armada manages.
const ( StateCreated = "created" StateRunning = "running" StateRestarting = "restarting" StateExited = "exited" StatePaused = "paused" StateDead = "dead" // StateStarting is a replica's container whose init containers or // sidecars the driver is still starting (ADR 0038). StateStarting = "starting" )
Container states.
const ( RoleInit = "init" RoleSidecar = "sidecar" )
Roles of the containers of a replica, in the LabelRole label.
const ( LabelRole = "armada.role" LabelContainer = "armada.container" )
Labels on init containers and sidecars, besides the replica's own: their role and name. A container without LabelRole is a replica's own.
const ( LogMaxSize = 10 << 20 LogMaxFiles = 3 LogRate = 1 << 20 LogBurst = 4 << 20 )
Log limits for container output: rotation by size, and a write rate per container (bytes per second, with a burst) past which output is dropped so one container cannot take a node's disk bandwidth.
const ( MaxExecArgs = 64 MaxExecArgLen = 4096 // MaxExecChunk is the most data one exec message carries. MaxExecChunk = 32 << 10 )
Exec limits (ADR 0015).
const LabelSystem = "armada.system"
LabelSystem marks Armada's own infrastructure containers (e.g. HAProxy).
Variables ¶
var DroppedCaps = []string{"NET_RAW", "MKNOD", "AUDIT_WRITE"}
DroppedCaps are removed from the engine's default capability set for app containers.
var ErrNotFound = errors.New("not found")
ErrNotFound is returned when a container or image does not exist.
Functions ¶
func CanMountVolumes ¶
CanMountVolumes reports whether d mounts volumes into app containers.
func CanRunMembers ¶
CanRunMembers reports whether d runs init containers and sidecars.
func CanRunSystem ¶
CanRunSystem reports whether d runs system containers.
func CanSandbox ¶
CanSandbox reports whether d keeps network sandboxes, like CanExec.
func CheckContainerSpec ¶
func CheckContainerSpec(s ContainerSpec) error
CheckContainerSpec enforces the rules every driver applies before running an app container: read-only regular-file mounts, private DNS servers, and a port published only on a private address.
func CheckExecCommand ¶
CheckExecCommand checks a command for ExecApp: at least one argument, at most MaxExecArgs of at most MaxExecArgLen bytes each, and no NUL bytes, which cannot reach a process's arguments. Errors never repeat the arguments, which may hold secret values.
func CheckFileMount ¶
CheckFileMount allows only absolute, clean paths and a regular host file, so a mount can never expose an engine socket or a host directory.
func CheckSandboxID ¶
CheckSandboxID refuses an ID made of anything but an app ID's characters: it names a file and network plugin state on the node.
func CheckSystemMount ¶
CheckSystemMount accepts absolute paths without characters that engines treat as separators.
func CheckVolumeMount ¶
func CheckVolumeMount(v VolumeMount) error
CheckVolumeMount allows a directory (not a link) at an absolute, clean host path, mounted at a target the spec would accept for a volume.
func ContainsFold ¶
ContainsFold reports whether line contains needle, ignoring case; needle must already be lower case. ASCII needles are matched without allocating, so filtering a busy log stays cheap.
func ValidMemberName ¶
ValidMemberName reports whether name can name an init container or a sidecar: a valid spec name of at most 30 characters, other than the replica's own container's.
Types ¶
type AgentStarter ¶
AgentStarter is told when the agent starts, after Ping: it does what only the agent should, never a command like doctor that also pings the runtime. It is optional; an error is only logged.
type Bridge ¶
type Bridge struct {
// Gateway is the host's address on the bridge, reachable only from the
// node itself and its containers.
Gateway netip.Addr
Subnet netip.Prefix
}
Bridge describes the network app containers are attached to.
type BridgeInspector ¶
BridgeInspector reports the app container network. It is optional: internal ingress and DNS need it.
type Container ¶
type Container struct {
ID string
Name string
Image string
Labels map[string]string
State string
ExitCode int
HostIP string
HostPort int
// IP is the container's address on its bridge network, reachable from
// the node for health checks; empty if unknown.
IP string
StartedAt time.Time
// Members are the replica's init containers and sidecars (ADR 0038),
// listed with the replica's own container rather than on their own. An
// init container or sidecar whose replica's container is gone is listed
// by itself, with LabelRole set, to be removed.
Members []Member
// StartError says why the driver gave up starting a replica's init
// containers or sidecars (one could not be created, for example); empty
// otherwise. It names no values from the spec.
StartError string
}
Container is the observed state of a container.
func (Container) FailedInit ¶
FailedInit returns the init container of c that exited with another code than 0, if any.
func (Container) StoppedSidecar ¶
StoppedSidecar returns a sidecar of c that no longer runs, if any.
type ContainerDialer ¶
type ContainerDialer interface {
// DialContainer dials address, an IP and port, from the network
// namespace of the running container id.
DialContainer(ctx context.Context, id, network, address string) (net.Conn, error)
}
ContainerDialer opens connections from inside an app container's network namespace, so the connection reaches only what the app could reach itself (queue rules, ADR 0034). It is optional.
type ContainerSpec ¶
type ContainerSpec struct {
Name string
Image string
Entrypoint []string
Cmd []string
Env []string
Labels map[string]string
Port int // container TCP port to publish; 0 for none
HostIP string // address to publish on; must be a private IP
MemoryBytes int64
NanoCPUs int64
// CPUReserveMillis sets the CPU weight: under contention, containers
// share CPU in proportion to what they reserve. 0 keeps the default.
CPUReserveMillis int64
PidsLimit int64
// Files are mounted read-only; the driver refuses sources that are not
// regular files (e.g. a runtime socket).
Files []FileMount
// DNS overrides the container's nameservers (private IPs only); empty
// keeps the engine default.
DNS []string
// DNSSearch are search domains put before the host's: the internal
// zone of the app's space (spec.ValidSearchDomain), so its apps
// resolve by name alone.
DNSSearch []string
// Volumes are mounted read-write (ADR 0023); only drivers for which
// CanMountVolumes reports true accept them.
Volumes []VolumeMount
// Init containers run in order, each to completion, before Sidecars
// and then the container itself start; all of them share its network
// namespace (ADR 0038). Only drivers for which CanRunMembers reports
// true accept them.
Init []MemberSpec
Sidecars []MemberSpec
}
ContainerSpec describes a container to run. Security defaults (no-new-privileges, dropped capabilities, no privileged mode) are applied by the driver and cannot be loosened through this struct.
type Driver ¶
type Driver interface {
// Ping verifies that the engine is reachable and supported.
Ping(ctx context.Context) error
// Pull fetches image for this node's platform. It returns a
// *PlatformError when the image has no variant for it.
Pull(ctx context.Context, image string, auth *RegistryAuth) (Image, error)
// Run creates and starts a container. One with init containers or
// sidecars (ContainerSpec.Init) is created and returned at once, and
// listed as StateStarting while the driver runs the init containers in
// order and starts the sidecars, then starts it. When an init container
// fails, it is left exited, never started, and Container.FailedInit
// names the init container, whose output stays readable until Remove.
// A container the driver stopped starting (it went away) is listed as
// stopped.
Run(ctx context.Context, spec ContainerSpec) (string, error)
// Stop stops a container (ending its start if it is still starting),
// then its sidecars, each with grace.
Stop(ctx context.Context, id string, grace time.Duration) error
// Remove removes a container with its init containers and sidecars.
Remove(ctx context.Context, id string) error
// List returns containers (running or not) carrying all given labels.
List(ctx context.Context, labels map[string]string) ([]Container, error)
Inspect(ctx context.Context, id string) (Container, error)
Logs(ctx context.Context, id string, opts LogOptions) (LogStream, error)
}
Driver runs containers on the local node.
type ExecOptions ¶
type ExecOptions struct {
// Cmd is the command and its arguments (CheckExecCommand).
Cmd []string
// TTY allocates a terminal; output then arrives on Stdout only.
TTY bool
// Size is the terminal's initial size; Resize carries changes until
// the command ends.
Size TermSize
Resize <-chan TermSize
// Stdin is the command's standard input; nil gives it none. EOF closes
// the command's standard input.
Stdin io.Reader
// Stdout and Stderr receive the command's output.
Stdout, Stderr io.Writer
}
ExecOptions describes a command to run in an app container (ADR 0015).
type Execer ¶
type Execer interface {
// ExecApp runs opts.Cmd in a running app container and returns its exit
// code once it has ended and its output has been written. When ctx
// ends while output may still be on its way, it returns ctx's error
// (with the exit code, if known), so a cut output is never taken for
// a whole one.
ExecApp(ctx context.Context, id string, opts ExecOptions) (exitCode int, err error)
}
Execer runs commands in app containers (armada app exec, ADR 0015). The command gets the container's own security context: user, capabilities, no-new-privileges, seccomp, limits, and namespaces. It is optional.
type FileMount ¶
type FileMount struct {
Source string // absolute path of a regular file on the node
Target string // absolute path in the container
}
FileMount bind-mounts a single host file read-only into a container.
type Image ¶
type Image struct {
// Ref is the digest-pinned reference ("repo@sha256:..."), or "" if the
// engine reports no digest.
Ref string
// Platforms lists the image's platforms as "os/arch" (sorted, no
// duplicates); empty if the engine cannot tell.
Platforms []string
}
Image is a pulled image.
type ImagePruner ¶
type ImagePruner interface {
// PruneImages removes every image that no container uses, that keep
// does not name (by reference or digest), and that was not pulled or
// updated within grace. It returns the names of the removed images.
PruneImages(ctx context.Context, keep []string, grace time.Duration) ([]string, error)
}
ImagePruner removes images nothing needs any more. It is optional.
type LogFrame ¶
type LogFrame struct {
Stream Stream
Data []byte
// Time is when the line was written; zero when the runtime does not say.
Time time.Time
}
LogFrame is one chunk of container output.
type LogOptions ¶
type LogOptions struct {
Follow bool
Tail int // 0 means all, negative none (with Follow: only new output)
// Grep keeps only lines containing it, ignoring case (lower case; empty
// for all); Tail counts matching lines.
Grep string
}
LogOptions controls log retrieval.
type LogPruner ¶
type LogPruner interface {
PruneLogs(ctx context.Context, retention time.Duration, maxBytes int64) error
}
LogPruner removes log history older than retention, then the oldest until it fits maxBytes (0 means the runtime's default).
type LogQuery ¶
type LogQuery struct {
App string
Replica string // empty for every replica
Container string // empty for the app's own container
// Since and Until bound the entries' times; a zero Until means now.
Since, Until time.Time
// Grep keeps entries containing it, ignoring case; never a pattern.
Grep string
// Limit keeps the newest entries; MaxBytes bounds what is read.
Limit int
MaxBytes int64
}
LogQuery asks for a window of an app's output from the node's log history and live logs (ADR 0043).
type LogRecord ¶
type LogRecord struct {
Time time.Time
Stream Stream
Data []byte
Replica string
Container string
// App and Revision are set by LogTailer.
App, Revision string
}
LogRecord is one entry a log search found.
type LogSearcher ¶
type LogSearcher interface {
SearchLogs(ctx context.Context, q LogQuery) ([]LogRecord, string, error)
}
LogSearcher is implemented by runtimes that keep a log history (ADR 0043). Search returns the newest q.Limit matching entries, oldest first, and a notice when it stopped early.
type LogTail ¶
LogTail is app output a LogTailer read; Commit records it as sent, so the next tail starts after it.
type LogTailer ¶
type LogTailer interface {
TailLogs(ctx context.Context, target string, maxBytes int64) (LogTail, error)
}
LogTailer reads app containers' output written since the last committed tail, at most about maxBytes of it, also across restarts (ADR 0044). target names the destination: for a new one, containers that already exist start at the end of their output.
type Member ¶
type Member struct {
ID string
Name string
Role string
State string
ExitCode int
StartedAt time.Time
}
Member is the observed state of an init container or a sidecar.
type MemberSpec ¶
type MemberSpec struct {
// Name is unique within the replica and a valid container name
// (spec.Container).
Name string
Image string
Entrypoint []string
Cmd []string
Env []string
MemoryBytes int64
NanoCPUs int64
CPUReserveMillis int64
PidsLimit int64
Files []FileMount
Volumes []VolumeMount
}
MemberSpec is an init container or a sidecar of a replica (ADR 0038). It gets the same security defaults as the replica's own container, joins its network namespace and its host files, and mounts only the files and volumes it lists, which must be among the replica's.
type NetworkReleaser ¶
NetworkReleaser frees the network of a stopped container that is kept only for its logs. Releasing twice is harmless. It is optional.
type NetworkSandboxer ¶
type NetworkSandboxer interface {
// EnsureSandbox creates the sandbox id, or returns the one that exists.
EnsureSandbox(ctx context.Context, id string) (Sandbox, error)
// RemoveSandbox releases the sandbox's address and removes it; one that
// is gone is not an error.
RemoveSandbox(ctx context.Context, id string) error
// ListSandboxes returns the node's sandboxes, so that an agent that
// restarts adopts or removes them.
ListSandboxes(ctx context.Context) ([]Sandbox, error)
// DialSandbox dials address, an IP and port, from the sandbox.
DialSandbox(ctx context.Context, id, network, address string) (net.Conn, error)
}
NetworkSandboxer keeps network sandboxes, from which the agent dials what a job's containers could reach while the job has none (queue triggers, ADR 0040). It is optional.
type OutputActivity ¶
type OutputActivity struct {
// contains filtered or unexported fields
}
OutputActivity tells when a command's output last moved. Runtimes wrap the writers they copy a command's output to with Wrap and, once the command exited, keep copying while output moves (WaitQuiet): a child the command left in the background can hold the output open for as long as it lives, but output still on its way to a slow reader is never cut. The zero value is ready to use.
func (*OutputActivity) WaitQuiet ¶
func (a *OutputActivity) WaitQuiet(ctx context.Context, done <-chan struct{}, grace time.Duration) error
WaitQuiet waits until done is closed, or until no output has moved for grace, counted from the call at the earliest. It returns ctx.Err() when ctx ends first, since output may then have been cut.
type PlatformError ¶
type PlatformError struct {
Image string
// Want is the node's platform, Have the image's, both as "os/arch".
Want string
Have []string
}
PlatformError reports an image without a variant for the node's platform.
func (*PlatformError) Error ¶
func (e *PlatformError) Error() string
type RegistryAuth ¶
RegistryAuth holds credentials for a private registry. It prints without the password.
func (RegistryAuth) GoString ¶
func (a RegistryAuth) GoString() string
GoString implements fmt.GoStringer without the password.
func (RegistryAuth) LogValue ¶
func (a RegistryAuth) LogValue() slog.Value
LogValue implements slog.LogValuer without the password.
func (RegistryAuth) String ¶
func (a RegistryAuth) String() string
String implements fmt.Stringer without the password.
type Sandbox ¶
type Sandbox struct {
// ID names the sandbox (a job's ID).
ID string
// IP is its address on the container bridge.
IP string
}
Sandbox is a network namespace wired into the node's container network like a container's, with no container and no process in it (ADR 0040).
type Stats ¶
type Stats struct {
// CPU is the CPU time the container used since it started.
CPU time.Duration
// MemoryBytes is the memory in use, without page cache the kernel can
// reclaim.
MemoryBytes int64
// EgressBytes is what the container sent over its network interface
// since it started; 0 when unknown.
EgressBytes int64
// NetworkLost is true when the container's network namespace has no
// interface left: it can reach nothing until it is recreated.
NetworkLost bool
// OOMKills counts the processes the kernel killed at the container's
// memory limit since it started. It stays readable after the container
// exits, until it is removed.
OOMKills int64
}
Stats is what an app container uses (ADR 0016).
type StatsReader ¶
StatsReader samples what app containers use (armada app top, ADR 0016). It is optional.
type SystemRunner ¶
type SystemRunner interface {
// RunSystem creates and starts a host-network container with all
// capabilities dropped except CapAdd.
RunSystem(ctx context.Context, spec SystemSpec) (string, error)
// Exec runs cmd in a running container and returns combined output.
Exec(ctx context.Context, id string, cmd []string) (output string, exitCode int, err error)
// Signal sends a signal (e.g. "SIGUSR2") to the container's main process.
Signal(ctx context.Context, id, signal string) error
}
SystemRunner runs and controls Armada's infrastructure containers. It is separate from Driver so the app code path can never reach host networking or bind mounts.
type SystemSpec ¶
type SystemSpec struct {
Name string
Image string
Cmd []string
Labels map[string]string
Mounts []Mount
// CapAdd lists the only capabilities granted; all others are dropped.
CapAdd []string
User string
}
SystemSpec describes an Armada infrastructure container. Unlike ContainerSpec it allows host networking and bind mounts, so it must never be built from user input.
type TermSize ¶
type TermSize struct {
Rows, Cols uint16
}
TermSize is a terminal's size in character cells.
type VolumeMount ¶
type VolumeMount struct {
Source string // absolute path of a directory on the node
Target string // absolute path in the container
// Shared marks a mount other containers, on this node or others, use
// too (an NFS export, ADR 0042): it keeps the label it was mounted
// with instead of getting the container's.
ReadOnly bool
}
VolumeMount is a persistent volume: a directory the agent created for the app on this node, mounted read-write without setuid or device files.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package conformance tests that the container runtime behaves as Armada needs (ADR 0011): the security defaults, checked from inside containers, the container lifecycle, labels, publishing on a private address, logs, exec, stats, volumes, network sandboxes, and errors.
|
Package conformance tests that the container runtime behaves as Armada needs (ADR 0011): the security defaults, checked from inside containers, the container lifecycle, labels, publishing on a private address, logs, exec, stats, volumes, network sandboxes, and errors. |
|
Package containerd implements runtime.Driver on the containerd instance that Armada embeds (ADR 0011).
|
Package containerd implements runtime.Driver on the containerd instance that Armada embeds (ADR 0011). |
|
Package embedded runs containers on Armada's own containerd instance (ADR 0011).
|
Package embedded runs containers on Armada's own containerd instance (ADR 0011). |
|
mkbundle
command
Command mkbundle downloads the artifacts pinned in embedded.Lock, checks their SHA-256, and writes the runtime pack that release builds embed.
|
Command mkbundle downloads the artifacts pinned in embedded.Lock, checks their SHA-256, and writes the runtime pack that release builds embed. |
|
Package fake provides an in-memory runtime.Driver for tests.
|
Package fake provides an in-memory runtime.Driver for tests. |
|
Package imagecheck checks that an image's layers are what its config says they are before a runtime reuses an unpacked layer for it.
|
Package imagecheck checks that an image's layers are what its config says they are before a runtime reuses an unpacked layer for it. |
|
Package supervisor is the container runtime of ADR 0046: images and container records kept by the agent itself, containers supervised by armada-shim.
|
Package supervisor is the container runtime of ADR 0046: images and container records kept by the agent itself, containers supervised by armada-shim. |