ec2

package
v0.6.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 4, 2026 License: Apache-2.0 Imports: 34 Imported by: 0

Documentation

Overview

Package ec2 launches one instance per job in a cloud region.

THIS IS THE AVAILABILITY STORY, NOT A PROOF OF THE ABSTRACTION. A tier that lists `providers: [firecracker, ec2]` may be placed on either, so losing the bare-metal host does not take the `runs-on` label down with it — which is the difference between self-hosted CI you can rely on and self-hosted CI you can rely on until the power goes out.

TWO THINGS ABOUT THIS BACKEND ARE UNLIKE EVERY OTHER ONE, and both follow from the compute being somewhere else:

The node running it is an ORCHESTRATOR. It holds credentials and calls an API; nothing runs on the machine billet is running on. So what that machine has is not what this node can offer, and `node.max_vcpu` / `node.max_memory` are required rather than detected. They are NOT a spending limit, though it is easy to read them as one: the allocator charges a job the selected shape's size while this backend buys the first declared shape that FITS, so shapes larger than their tiers multiply the real spend.

And the instance IS the isolation boundary, so this backend can run fork pull-request code that the docker backend must refuse. That boundary is the kernel, though, and not the network: untrusted work runs only once a separate security group has been described for it.

Index

Constants

View Source 
const (
	IAMRunInstances           = "ec2:RunInstances"
	IAMTerminateInstances     = "ec2:TerminateInstances"
	IAMDescribeInstances      = "ec2:DescribeInstances"
	IAMDescribeImages         = "ec2:DescribeImages"
	IAMCreateTags             = "ec2:CreateTags"
	IAMDescribeSubnets        = "ec2:DescribeSubnets"
	IAMDescribeSecurityGroups = "ec2:DescribeSecurityGroups"
	IAMAttachVolume           = "ec2:AttachVolume"
	IAMDetachVolume           = "ec2:DetachVolume"
	IAMDescribeVolumes        = "ec2:DescribeVolumes"
	IAMCreateImage            = "ec2:CreateImage"
	IAMGetConsoleOutput       = "ec2:GetConsoleOutput"

	IAMSQSReceiveMessage     = "sqs:ReceiveMessage"
	IAMSQSDeleteMessage      = "sqs:DeleteMessage"
	IAMSQSGetQueueAttributes = "sqs:GetQueueAttributes"
)

IAM action names for the operations this backend performs, kept beside the code that performs them so the node's IAM policy grants exactly what this package calls. internal/awspolicy imports these and assembles the policy; a drift test pins the assembled document. A permission with no direct API call site is commented as such — ec2:CreateTags is needed because create-time tagging (a TagSpecification on RunInstances, CreateVolume, CreateSnapshot and CreateImage) requires it. There is now ONE standalone CreateTags call, and only on the builder path: a verified AMI's contract tag is written after the image has been booted and proved, so it cannot be a create-time tag.

View Source 
const AMIContract = 2

AMIContract is what a runner AMI must satisfy for this billet to use it, and goes up whenever the image has to carry a newly required property.

The analogue of firecracker's GuestContract, and it exists for the same reason: an image is built once and used for months, so "which billet made this" has to be a fact recorded ON the artifact rather than something inferred later. A creation date cannot answer it — an image made after a commit is not proof it was built from that commit — and an untagged image is the pre-contract case rather than a passing one.

1: /etc/docker/daemon.json selects the classic image store, so the Docker cache publishes with the images in it rather than an empty filesystem.

2: and the image carries the toolcache GitHub's declaration names, with the variables that make it findable.

THE TAG IS A PROMOTION, NOT A CREATE-TIME CLAIM, and that is the whole content of the number now. CreateImage stamps who owns the image and which billet made it; the contract tag is added afterwards, by CreateTags, once billet has BOOTED the image and proved the properties on the artifact. So an unstamped image is one nothing has verified — which both readers already treat as "no answer, rebuild" — rather than a claim nobody checked.

View Source 
const BuilderOwnerPrefix = "billet-ami-build-"

BuilderOwnerPrefix begins the owner-tag value `billet ami build` stamps on the builder instance it launches — a per-build identity distinct from a deployment id (which is 32 hex characters). A bundled IAM policy scopes the builder's permissions to this prefix so they isolate the builder without being confused with a deployment's job instances.

View Source 
const DefaultBuilderDiskGiB = 80

DefaultBuilderDiskGiB is the root volume a build gets when nobody says.

Exported because `billet ami build` prints it in --builder-disk's help, and a default a flag describes must be the default the code uses.

NOT A ROUND NUMBER SOMEBODY LIKED. Canonical's noble images declare an 8GiB root and GitHub's declared package set is most of it, so the previous behaviour -- writing no Ebs.VolumeSize at all and inheriting whatever the base image said -- left provisioning to die on ENOSPC. Under `set -e` that aborts before the poweroff that signals success, so it produced no image rather than a broken one; what it cost was a paid builder and a failure that reads as an apt problem.

MEASURED ON THE PRODUCED IMAGE, not derived, and re-measured when parity grew. Read by `billet ami verify` from a machine booted off the AMI: 

26.8GiB used of 76.4GiB usable, 49.6GiB free   (root_used_kib=28106956)
/opt/hostedtoolcache: 5.2GiB                   (toolcache_kib=5435176)

THE OLD FIGURE WAS 7GiB AND 30 WAS ENOUGH FOR IT. Parity took the content to 26.8GiB -- the six-runtime toolcache, five JDKs, three .NET SDKs, PowerShell and its four modules, and the Android SDK with three NDKs -- so a 30GiB volume now leaves about a gigabyte free against a floor the provisioning script itself sets at ten, and every build would be refused before it started.

80 BECAUSE 80 IS WHAT WAS MEASURED, AND NOTHING ELSE HAS BEEN.

This was briefly 60, derived as "26.8 measured plus headroom", and that is the derivation this constant has now been wrong about twice. A review put the arithmetic plainly: 60GiB is about 57.3 usable, leaving 30.5 after the image -- and the number that matters is not what the finished image occupies but the PEAK during the build, because the installers unpack archives onto this same filesystem and delete them as they go. That peak is unmeasured. The preflight free-space check cannot see it either: it runs once, before provisioning, so a transient high-water mark fails later with ENOSPC on a volume that passed.

So the default is the size a real build actually completed on. Erring high costs a few cents of EBS for the life of one builder; erring low costs the build, and the failure arrives forty minutes in as an out-of-space error inside whichever install step happened to be running.

View Source 
const OwnerTagKey = ownerTag

OwnerTagKey is the tag billet stamps on every instance and cache volume/snapshot it creates, carrying the deployment identity. A bundled IAM policy conditions the destructive actions on this tag: on its exact VALUE (the deployment id) when that is known, isolating one deployment from another in a shared account, or on PRESENCE otherwise. Either holds because the RUNTIME role only ever tags at create time (a TagSpecification, which the policy restricts to ec2:CreateAction, and in value mode to this deployment's own owner), so it can carry the tag only onto resources it is itself creating. The BUILDER role gets one standalone CreateTags besides — the contract promotion — and that one is scoped to image resources ALREADY carrying the per-build owner tag, so it can still only speak about what it made.

Variables

View Source 
var ErrPromotionUncertain = errors.New("ec2: whether the contract tag was written is unknown")

ErrPromotionUncertain accompanies a failure that happened AFTER CreateTags was accepted, so whether the image carries its contract tag is unknown.

THE THREE-VALUED ANSWER AGAIN, and the third state is the one a caller collapses if you let it. "The verification failed" and "the tag may or may not be there" are different facts, and BuildImage's message told an operator the image was definitely unstamped in both cases — which sends them to re-run a verification against an image that may already be correct, or worse, to go looking for a tag they were told is absent. The same rule the credential paths follow: could-not- tell is never no.

Functions

func BuilderIAMActions 

func BuilderIAMActions() []string

BuilderIAMActions are what `billet ami build` needs beyond the runtime set — it snapshots a stopped builder into an AMI with CreateImage.

func BuilderPromoteIAMActions 

func BuilderPromoteIAMActions() []string

BuilderPromoteIAMActions is the one standalone CreateTags billet makes, and it is standalone for a reason that cannot be worked around: the AMI contract tag records that billet BOOTED the image and proved its properties, which is not knowable at the instant CreateImage creates it. A bundled policy scopes it to image resources carrying the per-build owner tag, which CreateImage stamped.

func BuilderVerifyIAMActions 

func BuilderVerifyIAMActions() []string

BuilderVerifyIAMActions is how a build reads the verdict off the image it just made: the verifier instance prints its report to the serial console and billet reads it back. There is no other channel that needs no key pair, no agent and no inbound access.

func CacheAttachIAMActions 

func CacheAttachIAMActions() []string

CacheAttachIAMActions are attach/detach of the fenced EBS volume, conditioned by a bundled policy on the owner tag so the role touches only billet's volumes.

func CacheDescribeIAMActions 

func CacheDescribeIAMActions() []string

CacheDescribeIAMActions is the volume describe, which acts on "*".

func CheckInterruptionQueue 

func CheckInterruptionQueue(
	ctx context.Context, region string, creds awscreds.Source, queueURL string,
) (string, error)

CheckInterruptionQueue proves the spot interruption queue exists and this identity may read it — via GetQueueAttributes, NEVER ReceiveMessage, which would consume (and hide for the visibility timeout) a real interruption warning some node needed. Returns the queue's ARN.

THE QUEUE URL IS VALIDATED HERE, at the exported boundary, the way ec2.New re-validates it: the request below is signed with the operator's credentials and posted to the URL's own host, so a caller that skipped config.Load must not be able to aim it anywhere. One attempt, no retry ladder: check is interactive and rerunning is cheaper than masking a flap.

func CheckReachable 

func CheckReachable(ctx context.Context, cfg config.EC2Config, opts ...Option) error

CheckReachable proves a set of credentials can reach the EC2 API in a region.

WHAT IT PROVES, EXACTLY: that credentials resolve, that the region and endpoint name something that answers, and that this identity is permitted to call DescribeInstances. It issues one read-only request whose filter deliberately matches nothing.

WHAT IT DOES NOT PROVE: permission to RunInstances, that the subnet and security groups exist, or that the AMI is visible. The subnet, security groups and AMIs are what the cloud preflight's read-only describes add (DescribeSubnet, DescribeSecurityGroups, DescribeImageStates). The launch PERMISSION needs a dry-run, which is a write-SHAPED call — DryRun=true has no side effect, but it is still a launch request — so `billet check` makes it only behind --authorize (DryRunLaunch), an operator opting into AWS's own authorization test rather than a diagnostic doing it uninvited. TerminateInstances cannot be dry-run (it checks the instance id before the permission verdict), so its grant stays advisory.

func OnDemandPriceUSDPerHour 

func OnDemandPriceUSDPerHour(
	ctx context.Context, ec2Region, instanceType string, creds awscreds.Source,
) (config.USDPerHour, error)

OnDemandPriceUSDPerHour fetches the on-demand hourly rate of one shape in one region from the AWS Price List API, so `billet init` can fill the price_usd_per_hour a config must carry rather than leaving the operator to look it up.

THE PRICE IS NOT AN ADMISSION GATE (config records it only to REPORT the maximum configured exposure), so a fetch that cannot answer unambiguously is not fatal to billet — it is fatal to guessing. This returns an error the caller turns into a prompt for an explicit --price rather than writing a number it is unsure of: a single positive hourly USD rate across the returned products is used; zero, or more than one DISTINCT rate, is refused. Distinctness is judged on the exact published rational, BEFORE rounding, so two rates that would round to the same millionth are still seen as two — only the single surviving rate is then rounded into the six decimals the config grammar admits.

func QueueProbeInconclusive 

func QueueProbeInconclusive(err error) bool

QueueProbeInconclusive reports whether a queue-probe failure is a fact about the CHECKING identity rather than the queue: a role provisioned before sqs:GetQueueAttributes joined SpotIAMActions refuses the probe while consuming warnings perfectly well, so AccessDenied must stay advisory or the probe fails every pre-upgrade node.

func RegionMayBeDisabled 

func RegionMayBeDisabled(err error) bool

RegionMayBeDisabled reports whether an error is the shape an AWS region that is NOT ENABLED on the account returns. Measured, twice, with VALID credentials (an IAM user and an SSO role that both work in us-east-1): af-south-1 answers 

<Code>AuthFailure</Code>
<Message>AWS was not able to validate the provided access credentials</Message>

— indistinguishable from a bad key by content. An operator whose key works elsewhere must be told the region is the suspect, or they will rotate credentials that were never wrong.

func RuntimeDescribeIAMActions 

func RuntimeDescribeIAMActions() []string

RuntimeDescribeIAMActions are the read-only describes, including the two the cloud preflight makes on the subnet and security groups — a node whose role cannot describe its own subnet would fail that check.

func RuntimeLaunchIAMActions 

func RuntimeLaunchIAMActions() []string

RuntimeLaunchIAMActions is RunInstances, which a bundled policy leaves on "*": it creates an instance, volumes and a network interface at once, and scoping it tightly enough to be safe is exactly the kind of multi-resource condition that denies a legitimate launch. The instances it creates are tagged, which is what the tag-conditioned teardown below relies on.

func RuntimeTagIAMActions 

func RuntimeTagIAMActions() []string

RuntimeTagIAMActions is CreateTags, restricted by a bundled policy to create-time tagging (ec2:CreateAction) — a node's runtime never calls CreateTags on its own, so this both matches its usage and stops the role from stamping billet's tags onto a resource it did not create.

func RuntimeTerminateIAMActions 

func RuntimeTerminateIAMActions() []string

RuntimeTerminateIAMActions is TerminateInstances, conditioned by a bundled policy on the owner tag's presence.

func SpotIAMActions 

func SpotIAMActions() []string

SpotIAMActions are what a spot node needs to consume its interruption queue, which is how billet learns a spot instance is about to be reclaimed — plus GetQueueAttributes, which is how `billet check` proves the queue answers WITHOUT consuming a warning. A role provisioned before the probe existed lacks it; the probe classifies that refusal as advisory rather than failing a working node.

Types

type BuildSpec 

type BuildSpec struct {
	// BaseImage is the AMI the builder starts from. It must be EBS-backed and
	// Ubuntu 24.04.
	//
	// UBUNTU BECAUSE THE DECLARATION IS UBUNTU'S. GitHub publishes what a hosted
	// runner image contains as an apt package set for noble, and billet's
	// firecracker guest is built from exactly that. Provisioning this backend from
	// a dnf distribution meant hand-translating those names into a second,
	// unpublished list — which is the two-pins problem `runnerrelease` exists to
	// prevent, one backend drifting from the other with nothing to notice.
	BaseImage string
	// InstanceType is the shape of the BUILDER, which has nothing to do with the
	// shapes jobs will later run on. Bigger only makes the build faster.
	InstanceType string
	// BuilderDiskGiB is the root volume the BUILDER launches with. Zero takes
	// DefaultBuilderDiskGiB.
	//
	// THIS WAS UNSET AND THAT IS A DEFECT, not a simplification. Nothing here
	// wrote Ebs.VolumeSize at all, so every build inherited whatever the base
	// image declared -- 8GiB on Canonical's noble images. The declared package
	// set alone is most of that, and the toolcache does not fit in what is left,
	// so provisioning dies on ENOSPC. Under `set -e` that aborts before the
	// poweroff that signals success, so it produces no image rather than a broken
	// one; the cost is a paid builder and a failure that reads as an apt problem.
	//
	// It is the BUILDER's disk and not the runner's. CreateImage snapshots this
	// volume, so it also becomes the size of the root every job launched from the
	// image starts with -- which is why it is sized to what the image needs plus
	// working room, rather than to the largest build anybody might run.
	BuilderDiskGiB int64
	// Arch is the runner build to install: "x64" or "arm64". It must match
	// InstanceType, and nothing here can check that — a mismatch produces an image
	// whose runner will not execute.
	Arch string
	// RunnerVersion is the actions/runner release to install, without the "v".
	RunnerVersion string
	// Name is the name given to the produced AMI. AWS requires it to be unique
	// within the account and region.
	Name string
	// Verify boots the produced AMI and asserts the contract on it before the
	// contract tag is written. It is the caller's default rather than this
	// package's: a zero BuildSpec that skipped verification would make the safe
	// behaviour the one you have to remember.
	Verify bool
	// VerifyInstanceType is the shape the VERIFIER runs on, which has nothing to do
	// with the builder's or with any job's. Empty takes defaultVerifierType for the
	// spec's architecture.
	VerifyInstanceType string
	// CACertPEM, when set, is a PEM bundle of one or more X.509 CAs baked into the
	// image's HOST trust store. The EC2 cache client speaks HTTPS to billet's
	// cache endpoint, whose certificate a private issuer signs; without that
	// issuer in the trust store every cache request fails its TLS handshake and
	// the job falls back to a cold fetch. It is validated before any paid builder
	// launches. The anchor lands in the host trust store only — job CONTAINERS do
	// not inherit it, which is correct: the cache client runs on the host.
	CACertPEM string

	// PayloadBucket is where the shared installers are staged when they will not
	// fit in user data.
	//
	// EMPTY MEANS EMBED, WHICH IS WHAT EVERY BUILD DID UNTIL THE SCRIPT OUTGREW
	// 16384 BYTES. A build whose script still fits needs no bucket and no new
	// permission; one whose script does not is refused with the name of this
	// field rather than by EC2 with a parameter error.
	PayloadBucket string
	// contains filtered or unexported fields
}

BuildSpec describes an AMI to build.

type DryRunOutcome 

type DryRunOutcome int

DryRunOutcome classifies an EC2 DryRun: whether the request would have been authorized, refused for permission, or refused for another reason. 

const (
	// DryRunInconclusive is any code that is not a permission verdict — a bad AMI, a
	// shape not offered in the zone, an invalid parameter. It is the ZERO VALUE on
	// purpose: in a classifier the dangerous mistake is a false "authorized", so an
	// unset or unexpected result must read as "proved nothing", never as a pass.
	DryRunInconclusive DryRunOutcome = iota
	// DryRunAuthorized is DryRunOperation: the request is well-formed and this
	// identity may make it — the launch would proceed.
	DryRunAuthorized
	// DryRunUnauthorized is UnauthorizedOperation: the role lacks the permission.
	// This is the gap the describe-only preflight cannot see.
	DryRunUnauthorized
)

type DryRunResult 

type DryRunResult struct {
	Outcome DryRunOutcome
	Code    string
}

DryRunResult is one dry-run's classification and the AWS code behind it.

type ImageInfo 

type ImageInfo struct {
	ImageID string
	State   string
	Found   bool
	// Contract is the AMIContract the image was stamped with, and BuiltBy the
	// billet that stamped it. Contract is 0 for an image built before billet
	// tagged its output — which is not "contract zero" but "no answer", and is
	// reported as needing a rebuild for the same reason an old contract is.
	Contract int
	BuiltBy  string
}

ImageInfo is one AMI's launch readiness. Found is false when the id resolves to nothing — a not-yet-built placeholder, a typo, or an AMI in another account or region — with State carrying the AWS code (e.g. InvalidAMIID.Malformed) when there was one.

func DescribeImageStates 

func DescribeImageStates(
	ctx context.Context, region, endpoint string, creds awscreds.Source, imageIDs []string,
) ([]ImageInfo, error)

DescribeImageStates reports each AMI's launch readiness, one describe per id so a malformed or not-found placeholder does not fail the lookup of a sibling that resolves. An InvalidAMIID.* code becomes a not-found result rather than an error, because "the AMI is not built yet" is a finding the check reports, not a failure of the check itself; any other API error is returned.

type InstanceTypeInfo 

type InstanceTypeInfo struct {
	Type      string
	VCPU      int
	MemoryMiB int64
}

InstanceTypeInfo is what one EC2 shape holds, as DescribeInstanceTypes reports it. `billet init --provider ec2` reads it so an operator can name a shape and have billet write the vcpu and memory an instance_types entry must DECLARE, rather than looking those up by hand and copying a number that overcommits the host the allocator escrowed against if it is wrong.

func DescribeInstanceTypes 

func DescribeInstanceTypes(
	ctx context.Context, region, endpoint string, creds awscreds.Source, types []string,
) ([]InstanceTypeInfo, error)

DescribeInstanceTypes reports what each named shape holds, following the reply's pagination to the end. It is NOT part of the node's runtime IAM set: it runs at `billet init` time under the operator's own credentials, not under the launched node's least-privilege role, so adding ec2:DescribeInstanceTypes to a node's policy would widen a grant nothing at runtime exercises.

A shape AWS does not offer in the region comes back absent rather than as an error, so the caller checks that every type it asked for is present: billet must not silently write a config missing a shape the operator named, because a tier derived from it would then have nothing to buy.

type Option 

type Option func(*Provider)

Option configures a Provider.

func WithCredentials 

func WithCredentials(src awscreds.Source) Option

WithCredentials sets where AWS credentials come from. The default is the environment, then this instance's own IAM role.

func WithHTTPClient 

func WithHTTPClient(c *http.Client) Option

WithHTTPClient sets the client used for API calls, for a test or for a deployment that needs a proxy.

func WithLogger 

func WithLogger(log *slog.Logger) Option

WithLogger sets the logger. The default is slog.Default().

type ProfileCheck 

type ProfileCheck int

ProfileCheck is the three-valued answer about an instance profile. The values are distinct on purpose: Missing is a misconfiguration a launch will fail on; Unknown means the CHECKING identity may not read IAM, which says nothing about the profile — advisory, never fatal. 

const (
	// Unknown is the ZERO VALUE on purpose: an uninitialized verdict must be
	// the least confident answer, not the most.
	ProfileUnknown ProfileCheck = iota
	ProfileFound
	ProfileMissing
)

func CheckInstanceProfile 

func CheckInstanceProfile(
	ctx context.Context, region, endpoint string, creds awscreds.Source, name string,
) (ProfileCheck, string, error)

CheckInstanceProfile asks IAM whether the named instance profile exists. region is the deployment's EC2 region, from which the partition-global IAM endpoint AND its signing region are derived — IAM is global per partition, and signing a China or GovCloud request as us-east-1 is a SignatureDoesNotMatch that would read as a permanently Unknown verdict. endpoint overrides the host for tests only. reason carries the API's own words for the Unknown verdict.

type Provider 

type Provider struct {
	// contains filtered or unexported fields
}

Provider launches EC2 instances, one per job.

func New 

func New(owner string, cfg config.EC2Config, opts ...Option) (*Provider, error)

New builds an EC2 provider. owner names this billet deployment and is written onto every instance it starts.

func (*Provider) Accepts 

func (p *Provider) Accepts(trust provider.TrustClass) error

Accepts reports whether this backend may run work of that trust class.

UNTRUSTED IS PERMITTED HERE, unlike the container backend, because a whole instance is a real isolation boundary: fork pull-request code gets its own kernel and its own machine, and the machine is destroyed afterwards.

BUT ONLY ONCE ITS NETWORK HAS BEEN DESCRIBED. The boundary an instance provides is the kernel, not the VPC — a fork's job in the same security group as everything else reaches whatever that group reaches, which on a subnet somebody already had is usually a good deal more than they are picturing. Defaulting to the trusted group would be deciding that on their behalf, silently, in the direction that cannot be undone once a job has run.

UNKNOWN is refused outright, and that is not the same judgement. Untrusted is a classification billet made; unknown means it could not classify the job at all, so there is no basis for choosing either group.

func (*Provider) AcknowledgeInterruption 

func (p *Provider) AcknowledgeInterruption(
	ctx context.Context, notice *provider.InterruptionNotice,
) error

AcknowledgeInterruption removes a warning only after the runner has durably recorded and acted on it.

func (*Provider) AttachVolume 

func (p *Provider) AttachVolume(
	ctx context.Context,
	instanceName string,
	slot int,
	volume string,
) error

AttachVolume hot-attaches one owned EBS cache volume and waits for AWS to report the attachment complete before the guest is told to discover it.

func (*Provider) BuildImage 

func (p *Provider) BuildImage(ctx context.Context, spec BuildSpec) (string, error)

BuildImage produces an AMI containing the GitHub Actions runner and Docker.

WHY BILLET DRIVES THIS ITSELF rather than shipping a Packer template: the contract between billet and an image is six lines of shell, and a build tool that has to be kept in sync with those six lines is a second place for them to be wrong. billet already speaks four EC2 actions; this adds one.

HOW IT KNOWS PROVISIONING FINISHED, without SSH, an agent, or any IAM the builder does not already have: the provisioning script ends in `poweroff`. A successful build is an instance that STOPPED ITSELF, which billet can see with the DescribeInstances it already makes. A failed one never reaches that line, so it stays running and the wait below times out against a machine still holding its console log — which is the state you want to debug from, rather than an image made from a half-provisioned disk.

THE BUILDER IS TERMINATED ON EVERY FAILURE PATH THAT HAS AN ID, which is not the same as always, and the difference is worth stating because this is the one thing here that costs money for as long as it exists.

If RunInstances commits and its response is lost — a transport failure, a context expiring mid-reply — launchBuilder returns an error and no id, so there is nothing to terminate and the builder runs until somebody notices. It carries the owner tag, which is how it is found.

A CLIENT TOKEN NARROWS THAT TO A RECOVERY rather than a leak: the token is derived from the image name, which is already unique per account and region, so re-running the same build returns the SAME builder instead of buying a second one. That behaviour is not assumed — a live run measured EC2 refusing a reused token with IdempotentParameterMismatch, which is the same machinery.

func (*Provider) Destroy 

func (p *Provider) Destroy(ctx context.Context, id string) (provider.Teardown, error)

Destroy terminates an instance, whether or not it is still running.

Idempotent: an id that is already gone is success. Teardown runs on paths that have already failed once, and erroring there turns recoverable state into stuck state.

IT DOES NOT CONFIRM THE MACHINE IS GONE. TerminateInstances returns when AWS accepts the request, and an idempotent NotFound may be an eventually consistent miss. The caller transfers the lease to custody, which keeps capacity charged and checks the instance out of band while this serial command queue remains available. `shutting-down` stays in List because the guest may still be executing there. An absence is trusted only after the instance has appeared in inventory or the eventually-consistent stray grace has elapsed.

func (*Provider) DetachVolume 

func (p *Provider) DetachVolume(
	ctx context.Context,
	instanceName string,
	slot int,
	volume string,
) error

DetachVolume uses the volume identity persisted in cache custody. An instance termination can remove the attachment before cleanup runs, so rediscovering a volume from its former instance and slot would lose the only handle to it.

func (*Provider) DryRunLaunch 

func (p *Provider) DryRunLaunch(
	ctx context.Context, image string, trust provider.TrustClass, instanceType config.EC2InstanceType,
	disk config.ByteSize,
) (DryRunResult, error)

DryRunLaunch asks EC2 whether the RunInstances the launch WOULD make — same image, shape, network for the trust class, instance profile, spot, disk and OWNER TAG — is authorized, without launching anything.

DryRun IS SAFE FROM A DIAGNOSTIC, which is why `billet check` gates this behind --authorize rather than running it by default: DryRun=true has no side effect — AWS validates the request and checks IAM, then refuses with DryRunOperation (would have worked) or UnauthorizedOperation (may not) and starts nothing. It is AWS's own authorization test, which is exactly the permission the read-only describes cannot confirm.

THE OWNER TAG MUST BE THE DEPLOYMENT'S, or a per-deployment IAM policy — which conditions ec2:CreateTags on the exact sh.billet.owner value — refuses the launch's TagSpecification and the whole RunInstances fails as UnauthorizedOperation. The real launch tags with the deployment id; so must this, or the diagnostic asks a different question than the launch. The caller constructs the provider with the deployment owner for exactly this reason.

func (*Provider) Find 

func (p *Provider) Find(ctx context.Context, name string) (*provider.Instance, bool, error)

Find reports the instance with that name, including a retained terminal record.

func (*Provider) GuestVolumeDevice 

func (p *Provider) GuestVolumeDevice(_ int, volume string) string

GuestVolumeDevice names the persistent udev link created for an EBS NVMe device. The API's /dev/sdX attachment name is not the name Nitro exposes.

func (*Provider) Kind 

func (p *Provider) Kind() config.ProviderKind

Kind reports the backend this is.

func (*Provider) Launch 

func (p *Provider) Launch(ctx context.Context, spec provider.Spec) (*provider.Instance, error)

Launch starts one instance running the job its JIT config names.

func (*Provider) List 

func (p *Provider) List(ctx context.Context) ([]*provider.Instance, error)

List reports every instance this backend is running for billet.

func (*Provider) NextInterruption 

func (p *Provider) NextInterruption(ctx context.Context) (*provider.InterruptionNotice, error)

NextInterruption blocks for one warning from the configured SQS queue.

func (*Provider) Quotas 

func (p *Provider) Quotas(ctx context.Context) ([]provider.Quota, error)

Quotas reports the account ceiling this node's budget runs against.

ONE LIMIT, DELIBERATELY. An ec2 node may buy several shapes, but the ceiling that binds them is a single vCPU allowance across the standard families rather than a limit per shape — so the useful sentence is "this account will run N vCPUs and you have configured M", which is a comparison `billet check` can make against node.max_vcpu without knowing anything about the catalogue.

WHAT IT DOES NOT COVER, and the report says so: a deployment declaring a shape outside the standard families — a GPU, a metal, a burstable-unlimited spot pool — runs against a different allowance this does not read. Reporting the standard one is still worth more than reporting nothing, and claiming it covers everything would be the overreach ADR-005 warns about.

func (*Provider) VerifyImage 

func (p *Provider) VerifyImage(ctx context.Context, spec VerifySpec) error

VerifyImage boots one instance from an AMI, makes it assert the contract on itself, and stamps the contract tag if it does.

WHY THIS EXISTS AT ALL. Every other claim billet makes about a runner image is checked on the BUILDER, before CreateImage — on a machine that has been apt-installed, part-configured and never rebooted. That is not the machine the image produces, and the difference is not academic: the Docker gate asserted a storage driver against a daemon apt had already started, so it read the answer from before daemon.json was written and failed every build against an image that was correct. Anything a service reads at start, anything cloud-init does at first boot, and anything a job's own `env -i` can or cannot see are all invisible from there.

HOW IT READS THE ANSWER, with no key pair, no agent and no inbound access: the verifier prints a bracketed report to the serial console and billet reads it back with GetConsoleOutput. That is the same shape of signalling BuildImage already relies on to know provisioning finished, and needs no IAM the builder does not already have besides the read itself.

PROVEN ON A REAL BUILD, 2026-08-28, us-west-2. `billet ami build` produced ami-0af6ca1a9ff63a09a, booted it, and read back: 

verdict=ok step=done
docker_driver=overlay2 docker_root=/var/lib/docker docker_server=29.1.3
root_free_kib=18576852 root_total_kib=29378688 root_used_kib=10785452
runner=2.336.0 toolcache_kib=5360256
tc_node=22.23.2 24.20.0
tc_go=1.24.13 1.25.14 1.26.7
tc_python=3.10.21 3.11.16 3.12.14 3.13.15 3.14.7
tc_java_temurin_hotspot_jdk=8.0.504-1 11.0.32-9 17.0.20-1 21.0.12-1 25.0.4-1

then stamped the contract and terminated both machines. `docker_driver=overlay2` is the line the whole issue is about: the in-build gate read `overlayfs` off the builder's own daemon for exactly this image shape.

THE VERIFIER IS TERMINATED ON EVERY FAILURE PATH THAT HAS AN ID, which is not the same as always — the same gap BuildImage documents for the builder, and worth restating because the second bound is weaker here than it looks. If RunInstances commits and its response is lost, launchVerifier returns an error and no id, so there is nothing to terminate. The script's own poweroff covers most of that, and NOT the case this command exists for: an image with broken cloud-init, or one that panics on boot, never runs the script at all. Such an instance carries the per-build owner tag, which is how it is found.

WHAT IT DOES NOT PROVE, said here rather than discovered later: the nonce makes a block THIS run's, not TRUE. An image that prints `verdict=ok` without doing anything passes, exactly as a base image whose own policy powers the machine off would satisfy the build's success signal. This is a check against mistake — against the artifact differing from what the builder measured — and the trust it rests on is the operator's own image, which is the trust `billet ami build` already rests on.

type SecurityGroupInfo 

type SecurityGroupInfo struct {
	GroupID string
	VPCID   string
}

SecurityGroupInfo is what one DescribeSecurityGroups item tells the preflight.

func DescribeSecurityGroups 

func DescribeSecurityGroups(
	ctx context.Context, region, endpoint string, creds awscreds.Source, groupIDs []string,
) ([]SecurityGroupInfo, error)

DescribeSecurityGroups reports each group's VPC. A group id that resolves to nothing is an error, for the same reason as a missing subnet.

type SubnetInfo 

type SubnetInfo struct {
	SubnetID         string
	VPCID            string
	AvailabilityZone string
	State            string
}

SubnetInfo is what one DescribeSubnets item tells the preflight.

func DescribeSubnet 

func DescribeSubnet(
	ctx context.Context, region, endpoint string, creds awscreds.Source, subnetID string,
) (SubnetInfo, error)

DescribeSubnet reports the subnet's VPC, availability zone and state. A subnet id that resolves to nothing is an error rather than an empty result: the check asked about a specific subnet and "there is no such subnet" is the answer it needs, not silence.

type VerifySpec 

type VerifySpec struct {
	// Image is the AMI to verify. It has to be available: this launches from it.
	Image string
	// InstanceType is the shape the VERIFIER runs on, which has nothing to do with
	// the builder's or with any job's. Empty takes defaultVerifierType for the
	// image's own architecture.
	InstanceType string
	// Name prefixes the verifier instance's Name tag. Empty takes the image id,
	// which is what a standalone verification has.
	Name string
}

VerifySpec describes an image to boot and assert on.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.