Documentation
¶
Overview ¶
Package dind implements a fake Docker Engine API server.
Each job gets its own server instance listening on a Unix socket. The socket is bind-mounted into the job container at /var/run/docker.sock. Docker CLI calls inside the container are translated into containerd operations on the host — no real Docker daemon.
Sibling containers created through this API can opt into the full docker --privileged elevation stack (all caps, all devices, seccomp and apparmor unconfined, writable sysfs/cgroupfs) when the host is configured to allow it. See Server.allowPrivileged and config.DindConfig.AllowPrivileged for the threat model — the short version is that on Windows and macOS hosts the dind containerd lives inside a managed Linux VM, so an escape stays in that VM; on a Linux host with no VM fence the default is to deny privileged requests.
Index ¶
- Constants
- func BuildScopePrefix(jobID string) string
- func CacheNamespace(provider, repo string) string
- func CachePrune(ctx context.Context, c *client.Client, maxAge time.Duration, log *slog.Logger) error
- func CleanupJobBuildRecords(ctx context.Context, c *client.Client, buildkitNS, jobID string, ...) int
- func CleanupJobNamespace(ctx context.Context, c *client.Client, ns string, log *slog.Logger)
- func CleanupStaleDindNamespaces(ctx context.Context, c *client.Client, log *slog.Logger)
- func DeadBuildRecords(names []string, live map[string]struct{}) []string
- func MirrorImageToCache(ctx context.Context, c *client.Client, jobNS, cacheNS, imageName string, ...) error
- func PruneDeadBuildRecords(ctx context.Context, c *client.Client, buildkitNS string, ...) (int, error)
- func RefreshLastAccessed(ctx context.Context, c *client.Client, cacheNS, imageName string, ...)
- func SweepBrokenImageChains(ctx context.Context, c *client.Client, nss []string, snapshotter string, ...)
- func SweepStagedBinds(dataDir string, log *slog.Logger)
- type Config
- type Server
- func (s *Server) Endpoint() string
- func (s *Server) EndpointPort() int
- func (s *Server) SetRunnerIP(containerIP string) error
- func (s *Server) SetRunnerNetNS(netnsPath string)
- func (s *Server) SetRunnerRootfs(snapshotKey string, runnerRootfsPath string, bindMappings map[string]string)
- func (s *Server) SocketPath() string
- func (s *Server) Start() error
- func (s *Server) Stop()
- func (s *Server) Transport() Transport
- type Transport
Constants ¶
const DindCacheNamespacePrefix = "ephemerd-dind-cache-"
DindCacheNamespacePrefix prefixes every per-repo image cache namespace.
Full namespace name format:
ephemerd-dind-cache-<provider>-<sanitized-repo>
Examples:
ephemerd-dind-cache-github-ephpm_ephpm ephemerd-dind-cache-gitea-ephpm_ephpm (distinct from the github one) ephemerd-dind-cache-gitlab-acme_platform_api (nested GitLab groups OK)
Provider + repo together form the privacy boundary: two different forges with same-named repos do NOT share a cache, and two different orgs on the same forge get separate caches keyed by the full `owner/repo` path.
const DindNamespacePrefix = "ephemerd-dind-"
DindNamespacePrefix is the prefix every per-job containerd namespace the dind subsystem creates. Each running ephpm-style job has its own namespace (e.g. "ephemerd-dind-ephemerd-github-ephpm-fast_shannon") so containers, images, and leases from one job can't pin disk against another's.
const LastAccessedLabel = imagegc.LastAccessedLabel
LastAccessedLabel records the most recent time an Image record in a cache namespace was touched (pull or container-create). The pruner uses this for LRU eviction. RFC3339-formatted, UTC.
Aliased to pkg/imagegc, which owns the label now that the same LRU key is stamped on the main "ephemerd" runtime namespace too.
Variables ¶
This section is empty.
Functions ¶
func BuildScopePrefix ¶ added in v0.1.8
BuildScopePrefix returns the image-name prefix every build result from jobID is stored under in the shared buildkit containerd namespace. It is the prefix half of scopedBuildRef.
Empty jobID returns "" — callers must treat that as "matches nothing", never as "matches everything", or a teardown would wipe the whole namespace.
func CacheNamespace ¶
CacheNamespace returns the containerd namespace name used to cache image metadata for a given (provider, repo) pair. Both inputs are sanitized so the result is always a valid containerd namespace identifier (regex: ^[A-Za-z0-9]+(?:[._-]+[A-Za-z0-9]+)*$).
Provider should be the value from providers.Provider.Name() (e.g. "github", "gitea"). Repo is the forge-native repo path (e.g. "owner/repo" on GitHub or "group/subgroup/project" on GitLab); path separators are mapped to underscores so the namespace identifier stays valid. Empty provider or repo returns "" — callers should treat that as "caching disabled for this job".
func CachePrune ¶
func CachePrune(ctx context.Context, c *client.Client, maxAge time.Duration, log *slog.Logger) error
CachePrune walks every per-repo cache namespace and evicts Image records whose LastAccessedLabel (or UpdatedAt fallback for records pre-dating the label) is older than maxAge, then deletes any cache namespace left empty. Containerd's content GC reclaims the unreferenced blobs after this runs.
maxAge <= 0 skips eviction and runs only the empty-namespace reap. That is now the default: age is an optional backstop, and disk-pressure-triggered LRU collection (pkg/imagegc, wired from [image_gc]) is the primary mechanism for keeping these namespaces bounded. Keeping the reap unconditional means a node that disables the age backstop still doesn't accumulate one stale metadata bucket per repo that ever ran a job.
The candidate listing, protection and ordering all come from pkg/imagegc so this path and the pressure collector cannot drift apart.
Returns nil and logs warnings on partial failures — the next pass will retry whatever didn't clean up this time.
func CleanupJobBuildRecords ¶ added in v0.1.8
func CleanupJobBuildRecords(ctx context.Context, c *client.Client, buildkitNS, jobID string, log *slog.Logger) int
CleanupJobBuildRecords deletes the build result records job jobID left in the shared buildkit namespace. Called from Server.Stop, so a job's build output is released as soon as the job ends.
It deliberately does NOT touch BuildKit's own cache records, leases or snapshots. Those belong to BuildKit's bbolt cache DB, which keeps its own references to them; deleting them behind BuildKit's back leaves the snapshots pinned (reclaiming nothing) and corrupts the cache index. Bounding that half is BuildKit's GC policy's job — see buildkit.GCConfig, which ephemerd previously left unset.
Returns the number of records deleted.
func CleanupJobNamespace ¶
CleanupJobNamespace removes everything inside a per-job dind namespace and then the namespace metadata bucket itself. Safe to call multiple times and safe to call on a namespace that contains stragglers from prior crashes — each step logs and continues on error rather than bailing partway through.
Order matters:
- Containers (with their tasks + snapshots) — releases the overlayfs upperdirs that hold container rootfs writes.
- Images — drops the gc.ref labels that pin manifest+config+layer blobs.
- Leases — releases any explicit content holds (buildkit etc. take these during pulls/builds).
- Snapshots — orphan snapshots from layers that were unpinned in step 2 don't get reclaimed by containerd's async GC fast enough; the NamespaceService.Delete in step 6 would fail with FailedPrecondition until those snapshots are gone. Walk the snapshotter and remove them explicitly.
- Content blobs — same story as snapshots; the async GC won't have swept by the time we try to delete the namespace.
- NamespaceService.Delete — drops the metadata bucket. Will only succeed if 1-5 left the namespace truly empty; on failure we log and leave the bucket so a subsequent boot's CleanupStaleDindNamespaces can retry.
func CleanupStaleDindNamespaces ¶
CleanupStaleDindNamespaces enumerates every namespace matching DindNamespacePrefix and runs CleanupJobNamespace on each. Intended to be called once at ephemerd worker-mode startup to clean up after crashed or killed jobs from a previous boot — the same Server.Stop path would have done this on a graceful shutdown but a runner timeout / SIGKILL skips it.
func DeadBuildRecords ¶ added in v0.1.8
DeadBuildRecords returns the job-scoped build result records in names whose job is not in live, together with the job each belongs to.
This is the decision half of the buildkit-namespace leak fix, kept pure so the "which records are garbage" rule is testable on its own.
Why these records are garbage by construction: every `docker build` in a job exports its image into the ONE shared buildkit namespace under a name scoped by that job's unique ID (see scopedBuildRef — the scoping exists to stop concurrent jobs racing on the same tag). Because the ID is unique per job, nothing ever overwrites or reuses those names, and nothing deleted them either. A production node accumulated 76 such records across 49 long-dead jobs, whose gc.flat leases then pinned ~44 GB of content that containerd's GC could never reclaim.
live holds container IDs as ephemerd knows them; matching is done on the lowercased form because scopedBuildRef lowercases the ID to satisfy Docker's reference grammar.
func MirrorImageToCache ¶
func MirrorImageToCache(ctx context.Context, c *client.Client, jobNS, cacheNS, imageName string, log *slog.Logger) error
MirrorImageToCache copies an image from the per-job namespace into the per-repo cache namespace (creating it if needed): every content reference in the image's DAG first, then the Image record itself, with the LastAccessedLabel refreshed.
Copying the *content references* — not just the Image record — is what makes the cache a cache. containerd's GC works on (namespace, digest) nodes, and `metadata.contentStore.garbageCollect` deletes a backing blob as soon as no namespace holds a blob bucket for it. An Image record's target and gc.ref labels only nominate nodes; if the cache namespace has no bucket for a digest there is nothing for the mark phase to keep, so dropping the per-job namespace made the whole image collectable and the next job re-pulled it over the network.
The copy costs no bytes and no network. containerd's default content sharing policy is "shared", so opening a writer in the cache namespace for a digest that is already in the backing store short-circuits (metadata.contentStore.Writer's `cs.shared` branch) and Commit only creates the bucket. Under an "isolated" policy there is no short-circuit, so we stream the bytes from the job namespace — still local disk, never the registry.
Once the buckets exist, the next job's pull into a fresh namespace hits the same short-circuit in remotes.fetch (`ws.Offset == desc.Size`) and downloads nothing.
Returns nil if the cache namespace name is empty (no provider/repo set).
func PruneDeadBuildRecords ¶ added in v0.1.8
func PruneDeadBuildRecords(ctx context.Context, c *client.Client, buildkitNS string, live map[string]struct{}, log *slog.Logger) (int, error)
PruneDeadBuildRecords removes every job-scoped build record in the buildkit namespace whose job is not in live. It is the sweep that catches what CleanupJobBuildRecords misses: jobs killed by SIGKILL, a host reboot, or a daemon that predates per-job cleanup entirely.
Safe to run while jobs are in flight — records belonging to a live job are skipped.
func RefreshLastAccessed ¶
func RefreshLastAccessed(ctx context.Context, c *client.Client, cacheNS, imageName string, log *slog.Logger)
RefreshLastAccessed bumps the LastAccessedLabel on a cached image. Called from the container-create path when a job references an image that's already in the cache (no pull happens, but the image is in use). Silently no-ops if the image isn't in the cache.
Delegates to imagegc.Touch, which does the same thing for the main runtime namespace — one writer for the LRU key.
func SweepBrokenImageChains ¶ added in v0.1.10
func SweepBrokenImageChains(ctx context.Context, c *client.Client, nss []string, snapshotter string, pinned []string, log *slog.Logger)
sweepBrokenImageChains evicts image records anywhere on the node whose layer snapshot has gone missing. Runs on the node disk sweeper's timer so a node repairs itself in the background, without waiting for a job to trip over the broken record first.
Deliberately independent of whether image GC is enabled: this is a correctness repair, not a capacity policy, and a node with GC turned off is exactly the node most likely to be carrying a broken store.
func SweepStagedBinds ¶ added in v0.2.4
SweepStagedBinds removes bind staging mounts and directories left behind by a previous ephemerd process. STARTUP ONLY: it does not know which jobs are live, and unmounting a running job's staged bind would not break that job's already-running containers but would break any container it starts next.
A hard kill (SIGKILL, panic, node reset) skips every teardown path, and the leaked mounts are not merely untidy: each one pins the runner container's rootfs mount, so containerd cannot delete the snapshot and the node accumulates undeletable snapshots until it runs out of disk.
Types ¶
type Config ¶
type Config struct {
// JobID is the unique job identifier.
JobID string
// Provider is the forge provider name ("github", "gitea", "forgejo",
// "gitlab", "woodpecker") for the job. Used together with Repo to
// build the per-repo image cache namespace; if empty, image caching
// across jobs is disabled and every pull is cold for this job.
Provider string
// Repo is the forge-native repo path (e.g. "owner/repo" on GitHub
// or "group/subgroup/project" on GitLab). Used together with Provider
// to build the per-repo image cache namespace; if empty, image
// caching across jobs is disabled.
Repo string
// DataDir is the ephemerd data directory. The socket and temp layers
// are stored under <DataDir>/jobs/<JobID>/docker/.
DataDir string
// Client is the containerd client for image pulls and container ops.
Client *client.Client
// Network is the networking manager for attaching sibling containers
// to the CNI bridge. May be nil if networking is not available.
Network *networking.Manager
// BuildKit is the shared embedded BuildKit solver. When non-nil,
// POST /build routes through handleImageBuildBuildkit. When nil, the
// platform default (buildah on Linux, 501 elsewhere) is used.
BuildKit *buildkit.Server
// RunnerNetNS is the path to the runner container's net namespace
// (e.g. /proc/<pid>/ns/net). Required for port forwarding — when a
// dind sibling exposes ports via -p, we install iptables DNAT rules
// in this namespace so the runner sees 127.0.0.1:hostPort routed to
// the sibling's container IP. Empty disables port forwarding.
RunnerNetNS string
// AllowPrivileged controls whether sibling containers may opt into
// the full elevation stack via HostConfig.Privileged, via
// HostConfig.CapAdd, or via the HostConfig.SecurityOpt values that
// switch off seccomp or AppArmor. When false, requests carrying any of
// them are rejected with HTTP 403. See config.DindConfig.AllowPrivileged
// for the threat model.
AllowPrivileged bool
// RegistryMirror routes this job's image pulls through a LAN
// pull-through cache. Nil disables mirroring, leaving the pulls
// identical to what they were before the feature existed.
RegistryMirror *registrymirror.Mirror
// Transport selects how the Docker API is exposed to the job container:
// a bind-mounted unix socket (the default and only supported option
// under runc) or a TCP port on the container network's gateway. Leave it
// zero for the platform default; set TransportTCP when the job container
// is VM-isolated and therefore has its own kernel, which is what makes a
// bind-mounted socket unusable. Ignored on Windows, which is always TCP.
// See listen.go.
Transport Transport
Log *slog.Logger
}
Config for creating a per-job fake Docker daemon.
type Server ¶
type Server struct {
// contains filtered or unexported fields
}
Server is a per-job fake Docker daemon.
func (*Server) Endpoint ¶
Endpoint returns the value a container should set DOCKER_HOST to in order to reach this fake daemon. On the unix-socket transport that is the host socket path (the container instead sees it bind-mounted at the standard location, so DOCKER_HOST is not normally needed); on the TCP transport it is a "tcp://<gateway-ip>:<port>" URI the guest reaches over IP.
func (*Server) EndpointPort ¶ added in v0.2.2
EndpointPort returns the TCP port this server's listener bound, or 0 on the unix-socket transport. Exported for tests and diagnostics that need to assert the firewall scope matches the listener.
func (*Server) SetRunnerIP ¶ added in v0.2.2
SetRunnerIP records the address the network assigned to this job's runner container and opens the firewall allow that lets exactly that container — and nothing else — reach this server's TCP listener.
Applies to every TCP-transport server: BOTH Windows paths — L2Bridge and the default NAT network, where the Docker API listens on the bridge gateway and the host firewall default-denies the container's inbound connection to it (#162) — and the Linux VM-isolated (Kata) path, where the allow is an iptables pair scoped to containerIP/32. A no-op only on the unix-socket transport, which binds no TCP port at all (listenPort stays 0).
SECURITY. The allow is scoped to containerIP/32 rather than to the container pool because the Docker API this port serves does no authentication whatsoever — it accepts any credentials. A pool-scoped allow (what this used to be) let any job container port-scan the host's LAN address, find another job's dind endpoint, and drive that job's daemon: run and exec containers, bind-mount host paths, read its build context. The firewall scope is the only thing standing between two concurrent jobs here.
FAIL CLOSED. An empty or malformed containerIP returns an error and opens nothing; the caller is expected to fail the job. Falling back to a pool-wide allow would trade this job's inconvenience for every other job's isolation.
ORDERING. Must be called after the container's network endpoint exists (the address is allocated by networking.Manager.Setup) and before the container is created, so the container never runs without its allow in place. Pairs with SetRunnerNetNS / SetRunnerRootfs, which are the other post-setup handoffs into this server.
func (*Server) SetRunnerNetNS ¶
SetRunnerNetNS records the runner container's net namespace path so the dind server can install iptables DNAT rules for port-bound siblings. Must be called after the runner task starts (PID is known) and before any docker create from inside the runner.
func (*Server) SetRunnerRootfs ¶
func (s *Server) SetRunnerRootfs(snapshotKey string, runnerRootfsPath string, bindMappings map[string]string)
SetRunnerRootfs registers the runner container's snapshot, rootfs mount path, and the non-rootfs bind table ephemerd installed into it, so that subsequent docker create requests from inside the runner can have their -v sources translated from the runner's mount namespace to real host paths.
snapshotKey is the containerd snapshot name (typically "<runnerID>-snapshot" in the runtime's "ephemerd" namespace). Kept for the (now-fallback) layer walk used by the unit tests; production resolution goes through runnerRootfsPath.
runnerRootfsPath is the host-namespace path where runc has mounted the runner's merged overlay. Caller typically obtains this by calling os.Readlink("/proc/<runner-pid>/root") after the runner task starts — runc sets up the overlay before exec, and the symlink reads back to the bundle's rootfs in the host's namespace. Required for the merged view: without it, sibling binds for paths split across image layers (e.g. /home/runner/externals) bind incomplete trees.
bindMappings keys are container destination paths (what the runner sees, e.g. "/var/run/docker.sock"); values are host source paths (what the dind daemon hands to containerd). The map is copied so the caller may continue to mutate it.
Must be called after the runner task starts (rootfs is mounted) and before any docker create from inside the runner. Pairs with SetRunnerNetNS in the runtime, called at the same point in startup.
func (*Server) SocketPath ¶
SocketPath returns the host-side Unix socket path to bind-mount into the job container at /var/run/docker.sock.
Empty whenever this server is on the TCP transport — always on Windows, and on Linux when the job container is VM-isolated. An empty result is the runtime's signal to inject DOCKER_HOST from Endpoint() instead of mounting anything; it is not an error.
func (*Server) Start ¶
Start begins serving the fake Docker API on the server's transport.
Unix socket (default on Linux/macOS under runc): a per-job socket at <DataDir>/jobs/<jobID>/docker/d.sock, bind-mounted into the container.
TCP (always on Windows; on Linux when the job container is VM-isolated): an ephemeral port on the container network's gateway, handed to the job as DOCKER_HOST. A guest with its own kernel cannot use a bind-mounted socket, and runhcs supports neither that bind nor named-pipe sharing.
type Transport ¶ added in v0.2.2
type Transport string
Transport selects how a per-job fake Docker daemon is exposed to the job container. It is a runtime choice, not a compile-time one, because the right answer depends on whether the job container shares the host kernel.
- A runc container shares the host's kernel and mount namespace plumbing, so a bind-mounted unix socket works and is the cheapest, most Docker-native option: the CLI finds /var/run/docker.sock by itself and the socket is reachable by nothing outside that container's mount namespace.
- A VM-isolated container (Kata on Linux, Hyper-V isolation on Windows) has its OWN kernel. A bind-mounted unix socket arrives in the guest as an inode with no listening endpoint behind it, so every connect(2) returns ECONNREFUSED. There is nothing to fix in the socket transport — the socket's connectability lives in the host kernel and does not cross the VM boundary. Such jobs get a TCP listener on the bridge gateway instead, handed over as DOCKER_HOST, which the guest reaches over IP like any other network service.
Nothing about the served API differs between the two; only the wire.
const ( // TransportAuto lets New pick the platform default: a unix socket on // Linux/macOS, TCP on Windows. The zero value, so a construction site // that does not set Transport keeps today's behavior. TransportAuto Transport = "" // TransportUnixSocket serves on a per-job unix socket which the caller // bind-mounts into the container at /var/run/docker.sock. TransportUnixSocket Transport = "unix" // TransportTCP serves on an ephemeral TCP port bound to the container // network's gateway address. The caller injects DOCKER_HOST and must // call SetRunnerIP once the container's address is known, so the port // is firewalled to that one container. TransportTCP Transport = "tcp" )
Source Files