Documentation
¶
Overview ¶
Package buildkit wires an in-process BuildKit solver into ephemerd.
The server type holds a *control.Controller configured with:
- a containerd worker pointed at ephemerd's embedded containerd
- the Dockerfile frontend plus the gateway.v0 frontend
- bbolt-backed cache and history stores under <dataDir>/buildkit
Callers interact with the server through the Build method, which accepts a high-level BuildOpts describing a Docker-style build request and returns a progress stream. The Docker-API translation layer lives in pkg/dind.
Index ¶
- func DefaultSnapshotter() string
- type Config
- type DanglingSnapshot
- type GCConfig
- type HealAction
- type HealReport
- type Healer
- type Server
- func (s *Server) Build(ctx context.Context, opt client.SolveOpt, statusCh chan *client.SolveStatus) (*client.SolveResponse, error)
- func (s *Server) Client(ctx context.Context) (*client.Client, error)
- func (s *Server) Close() error
- func (s *Server) ContainerdNamespace() string
- func (s *Server) DataDir() string
- func (s *Server) Healer() *Healer
- func (s *Server) Prune(ctx context.Context, rule client.PruneInfo) (int64, error)
- func (s *Server) PruneAll(ctx context.Context) (int64, error)
- func (s *Server) Rebuild(ctx context.Context) error
- func (s *Server) SessionManager() *session.Manager
- func (s *Server) Snapshotter() string
- type SnapshotKind
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DefaultSnapshotter ¶ added in v0.1.10
func DefaultSnapshotter() string
DefaultSnapshotter is the containerd snapshotter this platform's BuildKit worker uses when Config.Snapshotter is empty. Exported because the node's image ↔ snapshot repair pass needs the same name to build the `containerd.io/gc.ref.snapshot.<snapshotter>` label key, and it must agree with the solver even on nodes where BuildKit never started.
Types ¶
type Config ¶
type Config struct {
// DataDir is where BuildKit stores its cache, history, and content.
// Typically <ephemerd data dir>/buildkit.
DataDir string
// ContainerdAddress is the address of ephemerd's embedded containerd
// gRPC endpoint. On Linux this is a unix socket path; on Windows it's
// a named pipe (e.g. "npipe:////./pipe/ephemerd-containerd").
ContainerdAddress string
// ContainerdNamespace is the containerd namespace buildkit should use
// for image and content storage. Defaults to "buildkit" if empty.
ContainerdNamespace string
// Snapshotter selects the containerd snapshotter. Defaults to "overlayfs"
// on Linux and "windows" on Windows.
Snapshotter string
// Network manages container networking. Required on Windows so build
// containers get an HCN NAT endpoint (otherwise RUN steps that hit the
// internet exit immediately). Ignored on platforms where buildkit's
// default network providers already work.
Network *networking.Manager
// CNIConfigPath points the Linux BuildKit worker at ephemerd's CNI
// conflist so build `RUN` steps attach to the per-job CNI bridge — and
// are therefore subject to the egress firewall (which jumps on the
// container subnet) — instead of BuildKit's default fallback to the HOST
// network namespace, which bypasses the firewall entirely. When set, the
// worker uses network mode "cni" and fails init if the config is missing,
// rather than silently degrading to host networking. Empty preserves
// BuildKit's default ("auto") provider selection. Linux only; ignored on
// Windows (which uses the HCN NAT path via Network) and unused on macOS.
CNIConfigPath string
// CNIBinDir is the directory holding the CNI plugin binaries (bridge,
// host-local, portmap) the worker invokes for build-step networking.
// Paired with CNIConfigPath. Linux only.
CNIBinDir string
// GC bounds the on-disk build cache. The zero value produces NO prune
// rules, which is how BuildKit reads "never garbage-collect" — see
// GCConfig for what that cost us in production. Callers should pass a
// configured policy.
GC GCConfig
// Log receives structured logging from the buildkit server.
Log *slog.Logger
}
Config configures an embedded BuildKit Server.
type DanglingSnapshot ¶ added in v0.1.10
type DanglingSnapshot struct {
// ID is the containerd snapshot key from the error message.
ID string
// Kind is what that key addresses. See SnapshotKind.
Kind SnapshotKind
}
DanglingSnapshot identifies a snapshot BuildKit believes in and containerd does not.
func DanglingSnapshotFromError ¶ added in v0.1.10
func DanglingSnapshotFromError(err error) (DanglingSnapshot, bool)
DanglingSnapshotFromError is ParseDanglingSnapshot over an error value. Nil errors report false.
func ParseDanglingSnapshot ¶ added in v0.1.10
func ParseDanglingSnapshot(msg string) (DanglingSnapshot, bool)
ParseDanglingSnapshot extracts the offending snapshot key from an error message, reporting false when the message is some other failure. Pure — it is the whole detection rule, so the signature we act on is testable without standing up BuildKit.
Deliberately narrow. Auto-repair throws away the node's shared build cache; firing it on a merely similar message (a missing *content* blob, a permission error, a full disk) would turn an unrelated build failure into a cache wipe. Only "snapshot X does not exist" qualifies.
type GCConfig ¶ added in v0.1.8
type GCConfig struct {
// Disabled turns BuildKit's garbage collection off entirely, restoring
// the previous (unbounded) behavior. Only useful for debugging a
// cache-correctness problem.
Disabled bool
// ReservedBytes is cache that is never collected, even when idle. This
// is the warm floor: below it BuildKit keeps everything, so the common
// "same repo builds again an hour later" case still hits cache.
ReservedBytes int64
// MaxUsedBytes is the hard ceiling on total build cache. Anything above
// it is collected regardless of age.
MaxUsedBytes int64
// MinFreeBytes makes GC collect whatever it must to keep at least this
// much free space on the filesystem, overriding ReservedBytes. This is
// the arm that saves the node when something else (runner images, job
// workdirs) is consuming the disk.
MinFreeBytes int64
// KeepDuration is the age after which cache records are collected once
// usage is above ReservedBytes.
KeepDuration time.Duration
// EphemeralKeepDuration and EphemeralMaxUsedBytes bound the cheaply
// reproducible record types — local build contexts, RUN --mount=cache
// mounts and git checkouts. Re-creating those costs a copy, not a
// network round trip, so they are collected far more eagerly than
// layer cache. Mirrors the first rule of BuildKit's own default policy.
EphemeralKeepDuration time.Duration
EphemeralMaxUsedBytes int64
}
GCConfig bounds the on-disk BuildKit build cache.
WHY THIS EXISTS: BuildKit only garbage-collects when its worker is constructed with a non-empty GC policy — control.Controller's gc() is literally `if policy := w.GCPolicy(); len(policy) > 0 { w.Prune(...) }`. ephemerd built its worker without one, so the shared "buildkit" containerd namespace grew without bound: every `docker build` in every CI job added cache records, snapshots and `containerd.io/gc.flat` leases that nothing ever released. Measured on a production node: 76 images, 302 snapshots, 481 leases and ~44 GB of a 116 GB disk, spanning 49 long-dead jobs going back two and a half weeks. That, not runner image retention, is what filled the disk until QEMU froze the VM.
The goal is a WARM BUT BOUNDED cache. Pruning aggressively would defeat the point of a build cache and push the cost onto the network, which we are separately trying to reduce; leaving it unbounded is what caused the outage. So: a floor that is never collected, a ceiling that always is, and a free-space guard that overrides both.
func (GCConfig) Enabled ¶ added in v0.1.8
Enabled reports whether this config produces any prune rule at all — i.e. whether the build cache is bounded.
func (GCConfig) PruneInfo ¶ added in v0.1.8
PruneInfo renders the config as the BuildKit prune rules a worker's GC policy is made of. Pure — no disk access, no clock — so the resulting rule set is unit-testable without standing up a worker.
Returns nil when collection is disabled or nothing is configured, which is exactly the "never collect" state that caused the leak; callers that want a bounded cache must supply a non-zero bound.
Rules are evaluated in order and are complementary: the first keeps ephemeral records from ever becoming a large share of the cache, the second ages records out, and the third is the unconditional size bound.
THE THIRD RULE IS LOAD-BEARING AND MUST NOT CARRY A KeepDuration. BuildKit applies KeepDuration as an absolute exemption within a rule — cache/manager.go's pruneOnce skips every record whose lastUsedAt is newer than now-KeepDuration BEFORE it looks at MaxUsedSpace. So a rule that sets both an age and a size bound does not mean "collect anything over the cap, and additionally anything older than the age"; it means "collect only records older than the age, and only once over the cap". With the default 168h that leaves a week's worth of build cache completely exempt from the ceiling, which on a node that rebuilds the same images several times a day is indistinguishable from having no ceiling at all. Measured in a Linux VM against a real containerd + BuildKit: 18 builds of a unique 200 MB layer with reserved=1 GiB, max_used=2 GiB and keep_duration=168h grew the cache to 4.4 GB and climbing; the identical workload with the size bound applied unconditionally settled at 2.1 GB. BuildKit's own DefaultGCPolicy has the same shape — its last two rules carry no KeepDuration for exactly this reason.
type HealAction ¶ added in v0.1.10
type HealAction int
HealAction is what the healer wants done about a dangling snapshot.
const ( // HealNone means do nothing — this is not a repairable signature. HealNone HealAction = iota // HealPrune evicts the image records whose layer chain is broken and // prunes BuildKit's cache so the stale metadata record goes away. Cheap // and surgical enough to run inline, mid-build. HealPrune // HealRebuild discards BuildKit's cache metadata store entirely and // reconstructs the solver against the live containerd. This is the // escalation for a store whose corruption survived HealPrune: the two // databases disagree in a way we cannot enumerate, so the derived one // is rebuilt from the authoritative one. HealRebuild // HealGiveUp means repair has already been tried at both levels for // this key and did not stick. Fail the build loudly rather than loop. HealGiveUp )
func (HealAction) String ¶ added in v0.1.10
func (a HealAction) String() string
type HealReport ¶ added in v0.1.10
type HealReport struct {
// Snapshot is the key that triggered the repair.
Snapshot DanglingSnapshot
// Action is the rung of the ladder that was executed.
Action HealAction
// ImagesEvicted counts containerd image records dropped because their
// layer chain resolved to the missing snapshot.
ImagesEvicted int
// BytesReleased is what BuildKit's prune reported freeing.
BytesReleased int64
// Rebuilt reports that the BuildKit metadata store was quarantined and
// the solver reconstructed.
Rebuilt bool
}
HealReport summarises one repair attempt, for logging and for the message surfaced to the job when repair fails.
func (HealReport) String ¶ added in v0.1.10
func (r HealReport) String() string
String renders the report for a log line or an error message.
type Healer ¶ added in v0.1.10
type Healer struct {
// contains filtered or unexported fields
}
Healer decides how hard to try when a dangling snapshot is seen, and remembers what it has already tried.
The escalation ladder exists because the cheap repair is not always enough. HealPrune clears the records BuildKit is willing to enumerate; if the same key comes back, BuildKit's metadata is inconsistent in a way its own prune cannot reach (a record it refuses to size because the snapshot behind it is missing, for instance) and only rebuilding the derived store fixes it.
State is per-daemon and in-memory on purpose. A restart re-arms the ladder, which is correct: after a restart the store may be a different store.
The zero Healer is ready to use. Safe for concurrent use — several jobs can hit the same poisoned record at once.
func (*Healer) Forget ¶ added in v0.1.10
Forget drops the remembered escalation state for a key. Called once a build succeeds after a repair, so a key that recurs weeks later starts from the cheap rung again rather than jumping straight to a rebuild.
func (*Healer) Next ¶ added in v0.1.10
func (h *Healer) Next(d DanglingSnapshot) HealAction
Next reports the action to take for d and records it, so a second sighting of the same key escalates instead of repeating a repair that did not work.
type Server ¶
type Server struct {
// contains filtered or unexported fields
}
Server hosts an in-process BuildKit Controller and the supporting objects (session manager, worker controller, caches) it needs. Callers interact with the controller through a buildkit client.Client obtained from the Client() method; the client dials an in-process bufconn listener that the Controller serves on, so no network socket is exposed.
func NewServer ¶
NewServer constructs and initializes an embedded BuildKit server. The returned Server is ready to accept Build calls but does not expose a network listener; it is used in-process only.
func (*Server) Build ¶
func (s *Server) Build(ctx context.Context, opt client.SolveOpt, statusCh chan *client.SolveStatus) (*client.SolveResponse, error)
Build performs a Docker-style build using the embedded BuildKit solver. Progress events are written to statusCh as they arrive; statusCh is closed by the underlying solve when the build terminates. Build itself blocks until the solve completes and returns the solve response.
The caller constructs SolveOpt from Docker build options (this is the translation layer pkg/dind owns). def is nil when using a frontend like dockerfile.v0 — the frontend loads the definition from the build context supplied via SolveOpt.LocalMounts.
func (*Server) Client ¶
Client returns a buildkit client.Client connected to the in-process Controller via bufconn. The returned Client is not safe for concurrent use across different callers — construct one per request/goroutine and Close it when done.
func (*Server) Close ¶
Close signals the Controller to shut down gracefully, stops the in-process gRPC server, and releases worker resources. Safe to call multiple times.
func (*Server) ContainerdNamespace ¶ added in v0.1.8
ContainerdNamespace reports the containerd namespace build results and cache records land in. Exposed so per-job teardown (pkg/dind) can find and remove that job's build artifacts without hard-coding "buildkit".
func (*Server) Healer ¶ added in v0.1.10
Healer exposes the per-daemon repair-escalation state. See heal.go.
func (*Server) Prune ¶ added in v0.1.8
Prune runs BuildKit's cache prune through its own cache manager and returns the number of bytes released.
It must go through BuildKit rather than deleting containerd records directly: BuildKit's bbolt cache DB keeps its own references to the snapshots backing each cache record, so containerd-level deletion leaves the snapshots pinned and reclaims nothing. (Confirmed the hard way on a production node — image records and leases were gone and the space did not come back until the snapshots were removed too.)
rule is a single BuildKit prune rule. The zero rule prunes everything not currently in use (a full `docker builder prune`); passing GCConfig.PruneInfo()'s bounding rule performs the same bounded collection the worker does automatically, on demand.
BuildKit's Prune RPC takes one rule per call, so callers wanting a multi-rule policy applied should call this once per rule.
func (*Server) PruneAll ¶ added in v0.1.10
PruneAll drops every build-cache record not currently in use — the equivalent of `docker builder prune -af`, but executed against the SHARED host-side store rather than the namespaced view a job can reach.
This is the cheap rung of the repair ladder: a stale record naming a snapshot containerd no longer has is unreferenced by definition (the build that would have referenced it just failed), so a full prune clears it.
func (*Server) Rebuild ¶ added in v0.1.10
Rebuild discards BuildKit's cache metadata store and reconstructs the solver against the live containerd.
WHEN THIS IS THE RIGHT ANSWER. BuildKit's bbolt store is a DERIVED view of containerd: every record in it describes a snapshot and content that containerd owns. When the two disagree and a prune cannot reconcile them, the derived view is the one that is wrong, and there is no supported API to delete one bad record from it. Throwing the whole store away costs cache warmth — the next few builds re-pull and re-run their layers — and costs nothing else, because containerd still holds every blob and snapshot that is genuinely live. That is a far better trade than the status quo, which is every build on the node failing until someone logs in and does this by hand.
The old store is MOVED, not deleted, so the corruption can still be examined; one previous quarantine is kept and older ones are removed, so a node that keeps tripping this cannot fill its disk with evidence.
In-flight solves against the old controller fail when its gRPC server stops. They were failing anyway — that is why we are here.
func (*Server) SessionManager ¶
SessionManager exposes the session manager so callers (pkg/dind) can hijack incoming POST /session HTTP streams into session gRPC.
func (*Server) Snapshotter ¶ added in v0.1.10
Snapshotter reports the containerd snapshotter the solver's worker uses. Callers repairing the image ↔ snapshot relationship need it to build the `containerd.io/gc.ref.snapshot.<snapshotter>` label key.
type SnapshotKind ¶ added in v0.1.10
type SnapshotKind int
SnapshotKind classifies a dangling snapshot key by who owns it, which decides what repairing it involves.
const ( // SnapshotUnknown is a key we cannot classify. SnapshotUnknown SnapshotKind = iota // SnapshotChainID is an image layer chain ID ("sha256:..."). This key // space is SHARED: containerd's image unpacker and BuildKit's // blob-backed cache records both address layers by chain ID. A dangling // chain ID therefore usually means one or more containerd image records // in the shared namespace are still resolvable while the layers they // name are gone — those records must be evicted too, or the next pull // resolves to the same broken chain. SnapshotChainID // SnapshotCacheRecord is a BuildKit cache record ID — a build-cache // layer produced by a RUN step. No containerd image record can // reference it, so repairing it is purely a BuildKit-side concern. SnapshotCacheRecord )
func (SnapshotKind) String ¶ added in v0.1.10
func (k SnapshotKind) String() string
Source Files