Documentation
¶
Overview ¶
Package firecracker runs each job in its own microVM on bare metal.
THE ISOLATION BOUNDARY IS A KERNEL, which is the whole reason this backend exists and the reason the caching plane sits on top of it: a guest with its own kernel is something a block device can be attached to, and a container is not. Every cache billet will ever mount — a golden image, a per-job root disk, a sticky disk — is a block device, so nothing in that plane can start until this does.
EVERY GUEST RUNS UNDER THE JAILER, never under bare firecracker. The jailer chroots the VMM, drops it to an unprivileged uid, puts it in a cgroup and gives it a pid namespace of its own before the VMM has parsed anything an operator wrote. Running firecracker directly would leave a process with the whole host filesystem in front of a guest whose job is running somebody's CI. (The seccomp filter is not the jailer's: it is compiled into the firecracker binary and on by default, which is worth knowing because it means it is present either way and absent from a debug build either way.)
FOUR THINGS HERE WERE MEASURED ON THE REFERENCE HOST rather than read, and each of them is a way this package could have looked correct and not been:
- The jailer names its chroot after the RESOLVED --exec-file, so the versioned binary behind a stable symlink — which is how the reference host installs firecracker — decides the directory List enumerates. See jail.
- `jailer --daemonize` exits 0 for a VM that died during startup. Its exit code is not evidence, and Launch confirms through the VMM's own API instead.
- The jailer creates a per-VM cgroup only when it is given at least one --cgroup, and MIXING the two forms on one host wedges it outright. See jailerArgs.
- The runner registration travels in the metadata service, placed before the guest's first instruction, so it is never on a disk and never in argv.
Index ¶
- Constants
- Variables
- type HostReport
- type Instance
- type Option
- type Provider
- func (p *Provider) Accepts(trust provider.TrustClass) error
- func (p *Provider) AttachVolume(ctx context.Context, instanceID string, slot int, device string) error
- func (p *Provider) CheckHost(ctx context.Context, needsRootResize bool) (HostReport, error)
- func (p *Provider) Destroy(ctx context.Context, id string) (provider.Teardown, error)
- func (p *Provider) DetachVolume(ctx context.Context, instanceID string, slot int, _ string) error
- func (p *Provider) Find(ctx context.Context, name string) (*Instance, bool, error)
- func (p *Provider) Kind() config.ProviderKind
- func (p *Provider) Launch(ctx context.Context, spec provider.Spec) (*Instance, error)
- func (p *Provider) List(ctx context.Context) ([]*Instance, error)
- type RootDisk
Constants ¶
const DefaultBootWait = 30 * time.Second
DefaultBootWait bounds the wait for a VMM to come up and answer its API.
A microVM boots in tens of milliseconds and the API socket appears before the guest does, so this is a bound on a wedged process rather than a budget for booting. It is far inside the node command timeout, so a VMM that never answers surfaces as a launch failure the listener can hand capacity back for, rather than as a command the control plane gives up on and calls custody.
const GuestContract = "10"
GuestContract is the version of this layout the guest agent must understand.
THE AGENT LIVES IN THE IMAGE, NOT IN THIS BINARY, AND THAT IS THE PROBLEM IT SOLVES. A guest image is published once and booted for months; billet is upgraded independently. So the two can drift, and the drift is silent in the worst direction: a billet that renamed a key here would hand an older agent metadata it does not recognise, and the guest would boot, find nothing it could use, and register no runner — a microVM that starts perfectly and runs nothing, which is the failure this whole backend is built to make impossible.
billet cannot inspect an image to find out what it understands. What it CAN do is say what it is speaking, so the guest can refuse loudly instead of half-working. The agent compares this against what it was built for and stops with a message naming both if they differ.
BUMP IT WHEN THE SHAPE OR REQUIRED EXECUTION SURFACE CHANGES — a renamed key, a new required field, a different encoding, a new host-selected command or a required workflow tool — and republish the image in the same change. Adding an OPTIONAL key that an older agent can ignore is not a contract change.
Republishing is safe precisely because generations are immutable: a job holding a clone of the old generation keeps the agent it booted with, and clone v2 lets the old generation be removed once nothing holds it.
const VerifiedAlias = "verified"
VerifiedAlias is the reference a tier uses to mean "the newest proved one".
SPELLED HERE RATHER THAN IMPORTED, because the provider may not import the store. A test asserts the two agree, since a drift would make every alias look like a pinned generation and quietly disable the re-resolve below.
Variables ¶
var ErrJailExists = errors.New("a previous microVM for this lease was not cleaned up")
build lays out everything the VMM needs inside its chroot, before the jailer runs.
THE JAILER CREATES ONLY WHAT IT KNOWS ABOUT — measured, that is /dev/kvm, /dev/net/tun, /dev/userfaultfd and /dev/urandom. The kernel image and the root disk are billet's to place, and a chroot means placing them INSIDE it: a path that is correct from the host is meaningless to a process whose root has moved. ErrJailExists is returned when a lease's chroot survived a previous run.
A SENTINEL BECAUSE THE UNWIND MUST NOT TOUCH IT. Every other launch failure unwinds what the launch MADE; this one is a refusal to touch what was already there, so tearing it down would destroy the state of a microVM this launch never created — and the error would still be telling the operator it had been left alone.
var ErrNoKVM = errors.New("/dev/kvm is not available, so this host cannot run a microVM")
ErrNoKVM is returned when this machine cannot run a hardware-accelerated guest.
Functions ¶
This section is empty.
Types ¶
type HostReport ¶
type HostReport struct {
// Firecracker and Jailer are the versions the two binaries report.
Firecracker string
Jailer string
// JailDir is the directory the jailer will build chroots in, which is also the
// one List enumerates. It is reported because it is DERIVED — from the resolved
// binary's name — rather than configured, and an operator looking for a running
// guest will not find it under the path they wrote.
JailDir string
// JailUIDMin and JailUIDCount are the range of uids microVMs run as, one per
// guest. Reported because a range that is too small is a host that stops being
// able to launch, and the number is otherwise invisible.
JailUIDMin, JailUIDCount int
// Bridges are the networks guests attach to, trusted first. An empty untrusted
// entry means fork pull-request work is refused.
Bridge string
UntrustedBridge string
}
HostReport is what a host check learned about this machine's ability to run microVMs.
type Instance ¶
Instance is provider.Instance, aliased so this file does not repeat the package name on every line.
type Option ¶
type Option func(*Provider)
Option configures a Provider.
func WithBootWait ¶
WithBootWait bounds how long Launch waits for a new VMM to answer.
func WithLogger ¶
WithLogger sets the logger. The default is slog.Default().
type Provider ¶
type Provider struct {
// contains filtered or unexported fields
}
Provider launches one Firecracker microVM per job.
func New ¶
func New(owner string, cfg config.FirecrackerConfig, disk RootDisk, opts ...Option) (*Provider, error)
New builds a firecracker provider. owner names this billet deployment and is written into every jail it creates.
func (*Provider) Accepts ¶
func (p *Provider) Accepts(trust provider.TrustClass) error
Accepts reports whether this backend may run work of that trust class.
A MICROVM IS A REAL ISOLATION BOUNDARY, so unlike the container backend this one CAN run code billet cannot vouch for: a fork's pull request gets its own kernel, and the machine is destroyed with the job.
BUT THE BOUNDARY IS THE KERNEL, NOT THE NETWORK — the same distinction the ec2 backend draws, and it is sharper here. That backend's guests are in a VPC somebody built; these are on a bridge on a machine that also holds the Ceph cluster, the control-plane database and, on the reference host, an overlay network that reaches production. A guest on the ordinary bridge reaches all of it. So untrusted work runs only once a SEPARATE bridge has been described for it, and its absence is what refuses the job — rather than defaulting onto the trusted one, which is the direction that cannot be undone once a job has run.
UNKNOWN is refused outright, and that is a different judgement: untrusted is a classification billet made, while unknown means it could not classify the job at all, so there is no basis for choosing either network.
func (*Provider) AttachVolume ¶
func (p *Provider) AttachVolume( ctx context.Context, instanceID string, slot int, device string, ) error
AttachVolume replaces one reserved drive slot with a mapped block device.
Firecracker supports this after boot only for a cooperative guest: the guest must not have mounted or accessed the placeholder. The sticky-disk action waits for this method to return before it formats or mounts the device.
func (*Provider) CheckHost ¶
CheckHost proves this machine can act on its firecracker configuration.
THE SAME DISTINCTION checkEC2Credentials AND CheckReachable MAKE, one backend over. Config validation proves the block is coherent; it cannot prove the binaries are installed, that /dev/kvm is there, that the jail account exists, or that the bridge does. A node that is wrong about any of those validates perfectly and then fails on the first job of the day.
IT IS A READ, and the caller says so. Opening /dev/kvm proves the device is there and this process may use it; it proves nothing about the jailer's ability to chroot or to place a cgroup, which needs root and is not something a diagnostic should acquire to find out.
func (*Provider) Destroy ¶
Destroy removes a microVM and everything it owns.
FOUR THINGS OUTLIVE A GUEST and none of them is collected by anything else: the VMM process, its jail, the tap device on the host bridge, and the root disk — which is a mapped kernel block device AND an image holding pool space. Measured: SIGTERM stops the VMM and leaves the other three exactly where they were.
Idempotent, because teardown runs on paths that have already failed once. Every step tolerates its subject being absent, and the errors are joined rather than returned at the first one — stopping early would leave a mapped device behind because a directory was already gone.
CONFIRMING, and the reason is stopVMM below: nothing proceeds until the VMM is stopped, so a Destroy that returns nil has already established the guest is not executing. That is the same promise docker makes and the one EC2 cannot — so a success here may be read as proof, and a failure may not.
func (*Provider) DetachVolume ¶
func (p *Provider) DetachVolume( ctx context.Context, instanceID string, slot int, _ string, ) error
DetachVolume replaces a drive with an empty placeholder before storage unmaps it.
A successful PATCH is the boundary: only then has the VMM closed the RBD device and may the store snapshot or unmap it. A failed patch leaves the device mapped and returns an error so the caller cannot cross that boundary by accident.
func (*Provider) Kind ¶
func (p *Provider) Kind() config.ProviderKind
Kind reports the backend this is.
func (*Provider) Launch ¶
Launch starts one microVM running the job its JIT config names.
IT RETURNS WHEN THE VMM SAYS IT IS RUNNING, not when the jailer returns. That is not belt and braces: `jailer --daemonize` exits 0 for a VM that died during startup — measured, with a pid file and an API socket both present beside a VMM that had exited 1 — which is exactly the shape of the docker default-command bug this repository was bitten by, where every signal reported success and no runner ever started.
func (*Provider) List ¶
List reports every microVM this backend is running for billet.
IT FAILS RATHER THAN REPORTING A SHORTER LIST. The control plane reconciles against this and frees the capacity of any lease ABSENT from it, so an entry silently dropped is capacity handed back for a guest that is still executing a job — the exact failure the inventory exists to prevent, caused by the report meant to prevent it. The ec2 backend fails its own List for the same reason.
A chroot base that does not exist yet is EMPTY rather than an error: a node that has never launched anything has no such directory, and refusing there would make the first sweep on a fresh host a failure.
type RootDisk ¶
type RootDisk interface {
// ResolveGeneration turns a tier's image reference into one exact generation.
//
// SEPARATE FROM THE CLONE so that the concrete generation is known BEFORE
// anything is done with it: it is what the launch logs, and "which image did
// this job actually boot" has to be answerable from that line afterwards. A
// reference that already names a generation comes back unchanged. The guest
// contract keeps a rolling-upgrade node on the newest verified generation its
// own binary can speak.
ResolveGeneration(ctx context.Context, image, guestContract string) (string, error)
// CloneRoot makes a per-job copy-on-write clone of a golden image, grows it to
// at least capacity when capacity is positive and maps it, returning the host
// device path. Zero keeps the generation's size as the backend default.
CloneRoot(ctx context.Context, image, name string, capacity config.ByteSize) (string, error)
// DiscardRoot unmaps and removes a clone. It must be idempotent: teardown runs
// on paths that have already failed once.
DiscardRoot(ctx context.Context, name string) error
// KernelFor reports which kernel file a generation was paired with, if any.
//
// THE PAIRING IS AN INVARIANT, NOT BOOKKEEPING. A guest booted with a different
// kernel than the one its filesystem was verified against does not fail to
// start -- it fails in the middle of somebody's job, which is why the two are
// published together. Recording which kernel a generation needs means nothing
// unless the launch asks.
//
// NOT FOUND IS NORMAL AND NOT AN ERROR: generations published by
// build-guest-image.sh record no kernel, because that script installs none and
// genuinely does not know which will be used.
KernelFor(ctx context.Context, image, generation string) (string, bool, error)
// GenerationGone reports whether a CloneRoot failure means the generation is no
// longer there.
//
// A METHOD RATHER THAN A SENTINEL THIS PACKAGE IMPORTS, because the provider may
// not import the store -- depguard enforces it, and the reason is that either
// direction makes one of them unsubstitutable. The store knows what its own
// errors mean; this package only needs to ask.
GenerationGone(err error) bool
}
RootDisk supplies the block device a guest boots from.
AN INTERFACE BECAUSE STORAGE AND COMPUTE ARE SIBLINGS. A provider may not import the store and the store may not import a provider — depguard enforces it, and the reason is that either direction makes one of them unsubstitutable. So this package declares the two operations it needs and the wiring hands it something that performs them; internal/store/ceph satisfies this today by having the methods, with no adapter and no shared type.
Source Files