Documentation
¶
Overview ¶
Package imagesource describes where a published guest image comes from and what has to be true about it before any of its bytes reach the cluster.
WHY A MANIFEST AND NOT JUST A CHECKSUM FILE. A node has to decide whether it can USE an image before it spends four hundred megabytes finding out. Two facts settle that — which guest contract the image speaks, and which architecture it was built for — and both have to be readable from a document small enough to fetch on every check. A SHA256SUMS file carries neither, so a node holding only checksums learns that an image is unusable after downloading it, every time, forever.
WHY THE MANIFEST IS THE ONLY THING THAT NEEDS SIGNING. It carries the digest of every asset, so a signature over the manifest transitively covers the image and the kernel. One signature, one verification, and the large files are checked with a hash rather than public-key arithmetic.
Index ¶
- Constants
- Variables
- func AssembleRootfs(dir string, img Multipart) (string, error)
- func TrustedRoot() (*root.TrustedRoot, error)
- func VerifyFile(path, want string) error
- func VerifySignature(manifest, bundleJSON []byte, p Policy) error
- type Asset
- type Client
- type Manifest
- type Multipart
- type Policy
- type Source
Constants ¶
const ( // SchemaV1 publishes the root filesystem as one release asset. SchemaV1 = 1 // SchemaV2 publishes it as ordered parts, because GitHub caps a single // release asset at 2 GiB and a parity-sized image packs to well past that. SchemaV2 = 2 )
The manifest layouts this build knows about.
Bumped when a field changes meaning. A reader refuses a schema it does not know rather than interpreting unfamiliar fields, because the failure mode of guessing is booting the wrong bytes.
const AgeNotice = 14 * 24 * time.Hour
AgeNotice is when an image is old enough to be worth mentioning.
ABOUT BILLET'S BUILD CADENCE, NOT ABOUT GITHUB, and the distinction is the whole reason this constant exists rather than reusing runnerrelease's window. The guest image is rebuilt weekly, so an image a fortnight old means two builds did not happen or a node stopped refreshing — worth saying, and evidence of nothing else.
const BundleName = "manifest.sigstore.json"
BundleName is the signature over the manifest.
A SIGSTORE BUNDLE, NOT COSIGN'S LEGACY SHAPE. The two are different documents: the legacy one carries base64Signature, cert and rekorBundle, and the library that verifies this on a node parses only the protobuf bundle -- it rejects the other outright. The first release published the legacy form, so nothing could have verified it even though the signature was real. The extension is part of how they are told apart.
The new format also carries the transparency-log inclusion proof, which is what makes verification possible without reaching Rekor -- and a node that may be air gapped cannot reach Rekor.
const DefaultChannelBundleURL = "https://raw.githubusercontent.com/" + DefaultRepo + "/guest-channel/current.sigstore.json"
DefaultChannelBundleURL authenticates DefaultChannelURL.
const DefaultChannelURL = "https://raw.githubusercontent.com/" + DefaultRepo + "/guest-channel/current.json"
DefaultChannelURL discovers the current immutable guest-image release.
RELEASE IMMUTABILITY MAKES A MOVING RELEASE IMPOSSIBLE. A published release's tag and assets cannot be replaced, so the old guest-latest release froze on its first generation. The guest-channel branch carries a signed, expiring pointer; the manifest and large assets still come from an immutable dated release, and every byte is bound to the signed manifest afterward. raw.githubusercontent.com avoids the API's per-address anonymous rate limit for fleets behind one NAT.
const DefaultRepo = "junioryono/billet"
DefaultRepo is the project's own home, and the ONE place its name is written.
EVERY OTHER REFERENCE IS DERIVED. This project expects to move between accounts, and a name spread across a fetch URL, a signing identity and a documentation link is a rename that half-lands: the binary keeps pulling from the old place while the signature check names the new one, and the failure arrives as a signature mismatch that reads like an attack.
It is also only a DEFAULT. The pull path takes a source from configuration, because a deployment that mirrors artifacts internally — or is not on the public internet at all — must not have to patch a constant to do it. Retro- fitting that is the specific thing that hurt other projects who baked one registry in and then needed a second.
const GitHubOIDCIssuer = "https://token.actions.githubusercontent.com"
GitHubOIDCIssuer is the issuer a GitHub Actions workflow's certificate carries.
const ManifestName = "manifest.json"
ManifestName is the index document within a release.
const MaxAssetBytes int64 = 8 << 30
MaxAssetBytes bounds a single published asset at 8 GiB.
The root filesystem is four gigabytes raw and under half a gigabyte packed, so this is generous by an order of magnitude while still refusing a manifest whose size field would have a node write until the disk filled. GitHub caps a release asset at 2 GiB, so anything approaching this bound is a manifest that could not have been published by the pipeline that claims to have made it.
const MaxBundleBytes = 512 << 10
MaxBundleBytes bounds a signature bundle.
A real one is around ten kilobytes -- certificate, signature, and a transparency log entry. The bound is generous against that and exists for the same reason the manifest's does: it is fetched over the network before anything about the far end has been proven, and an unbounded read of an untrusted stream is how a fetch becomes a memory exhaustion.
const MaxChannelBytes = 4 << 10
MaxChannelBytes bounds the first-party release pointer.
const MaxManifestBytes = 64 << 10
MaxManifestBytes bounds what a reader will parse.
A manifest is a few hundred bytes. The bound exists because the document is fetched over the network before anything about the far end has been proven, and an unbounded read of an untrusted stream is how a fetch becomes a memory exhaustion. 64 KiB is roughly two hundred times the real size.
const MaxPartBytes int64 = 2 << 30
MaxPartBytes bounds one part of a multipart asset at GitHub's own limit.
2 GiB IS NOT A CHOICE HERE, it is what a release accepts: "Each file included in a release must be under 2 GiB." A manifest naming a larger part describes something the pipeline it claims to come from could not have uploaded, so it is refused before anything is downloaded rather than after the upload that would have failed.
const MaxParts = 64
MaxParts bounds how many pieces one file may be published in.
A release holds at most a thousand assets, and a parity-sized image needs on the order of ten parts. The bound refuses a manifest that would turn one import into thousands of requests, which is a denial of service against the node rather than a plausible publication.
const MaxRootfsBytes int64 = 128 << 30
MaxRootfsBytes bounds the reassembled root filesystem at 128 GiB.
The bound exists so a manifest cannot ask a node to write until its disk fills. It is well above a parity-sized image — tens of gigabytes packed — and well below anything a node could be expected to stage.
const PublishRef = "refs/heads/main"
PublishRef is the git ref images may be signed from.
PINNED, AND THE FIRST VERSION OF THIS WAS NOT. It ended `@refs/.+`, reasoning that a release cut from another branch should fail review rather than fail verification. That reasoning was wrong in a way that matters: a certificate's ref is whatever ref the workflow RAN on, so a contributor who opens a pull request modifying guest-image.yml gets a certificate reading `.../guest-image.yml@refs/pull/N/head` -- which that pattern accepted. Opening a pull request is a far lower bar than compromising the release process.
const PublishWorkflow = ".github/workflows/guest-image.yml"
PublishWorkflow is the workflow allowed to sign billet's images.
NAMED, NOT JUST THE REPOSITORY. A certificate's SAN identifies the WORKFLOW that requested it, so pinning only the repository would accept a signature from any other workflow in it -- including one added by a pull request, which is a far lower bar to clear than compromising the release process.
const SchemaVersion = SchemaV1
SchemaVersion is the layout this build WRITES.
DELIBERATELY BEHIND WHAT IT READS, and that gap is the whole migration. A reader accepts exactly the schemas it understands and refuses everything else, so publishing a new layout in the same change that teaches the reader about it is a FLAG DAY: the next release becomes unreadable to every already-deployed binary, and the thing that would fix them is the image they can no longer pull. So readers learn v2 and ship; only once the fleet carries them does the writer move. Anything that changes this constant must ask whether every deployment in the wild can already read what it is about to publish.
Variables ¶
var DefaultSigningIdentity = fmt.Sprintf( `^https://github\.com/%s/%s@%s$`, regexp.QuoteMeta(DefaultRepo), regexp.QuoteMeta(PublishWorkflow), regexp.QuoteMeta(PublishRef))
DefaultSigningIdentity is the certificate SAN a manifest from billet must carry.
DERIVED FROM THE ONE CONSTANT NAMING THIS PROJECT, like the fetch URL, because a project that moves between accounts and forgets one of them gets a signature mismatch that reads exactly like an attack.
ANCHORED AT BOTH ENDS AND FULLY ESCAPED. Unanchored, `guest-image\.yml` also matches `guest-image.yml.evil`; unescaped, `github.com` matches `githubXcom`. Every literal here goes through QuoteMeta for that reason rather than being trusted to contain no metacharacters today.
var ErrNotFound = errors.New("imagesource: no such artifact at this source")
ErrNotFound is returned when the source has no such artifact.
DISTINGUISHED FROM EVERY OTHER FAILURE because the callers act on it differently: a deployment that has never published an image is a normal state with an instruction attached, while a network failure is a retry.
Functions ¶
func AssembleRootfs ¶
AssembleRootfs joins a downloaded root filesystem into one verified file and returns the path it landed at.
THIS IS THE ONLY WAY TO GET A USABLE ROOT FILESYSTEM PATH, and that is the design rather than a convenience. Every route ends at one check — the digest of the assembled bytes against what the signed manifest published for the whole file — so there is no shorter path that stops after the individual parts.
WHY THE PER-PART DIGESTS ARE NOT ENOUGH. Download checks each part against the manifest as it arrives, which proves every piece is a piece the publisher signed for. It says nothing about ORDER. A manifest is signed as a document, so an attacker cannot rewrite it — but a reader that concatenated parts and trusted the per-part checks would accept its own mistake: an off-by-one in the loop, a directory listing sorted lexically where part10 precedes part2, a retry that appended a part twice. Every one of those produces a file made entirely of signed bytes and is a different filesystem. The whole-file digest is what refuses it, and it costs one pass over data already on disk.
A SINGLE-PART IMAGE IS VERIFIED IN PLACE rather than copied onto itself. The branch is about avoiding a needless copy of several hundred megabytes; both branches finish at the same verification, so the property this function exists for does not depend on which one ran.
func TrustedRoot ¶
func TrustedRoot() (*root.TrustedRoot, error)
TrustedRoot is the embedded sigstore trust root.
PARSED ONCE. It is a few kilobytes of JSON and parsing it per verification would be wasteful, but the real reason is that a pull verifies two things and both should be judged against the same root rather than two parses that could in principle disagree.
func VerifyFile ¶
VerifyFile checks a file on disk against a published digest.
THE SIDELOAD PATH'S EQUIVALENT OF WHAT Download DOES INLINE. A file that arrived on a USB stick is no more trustworthy than one that arrived over http — less, arguably, since nothing about its journey is even in principle observable — so it is checked against the same manifest with the same digest before anything imports it.
func VerifySignature ¶
VerifySignature checks a manifest against its signature bundle under a policy.
THIS IS WHAT MAKES EVERY OTHER CHECK MEAN ANYTHING. Each asset is verified against a digest the MANIFEST names, so a manifest an attacker serves names digests of bytes the attacker chose and every one of those checks passes. The signature is the only thing binding the manifest to the workflow that produced it; without it the rest is a checksum against itself.
Types ¶
type Asset ¶
type Asset struct {
// Name is the file's name within the release.
//
// A BARE FILENAME, VALIDATED AS ONE. It is joined to a base URL to fetch and
// to a staging directory to write, so a name carrying a separator or a
// parent reference would write outside the directory the caller chose. The
// check lives in Validate rather than at each use, so no future call site
// can forget it.
Name string `json:"name"`
// SHA256 is the digest of the bytes as published, lowercase hex.
//
// OF THE PUBLISHED BYTES, NOT THE DECOMPRESSED ONES. It is checked against
// exactly what came off the network, so verification does not depend on the
// decompressor behaving, and a corrupt download is caught before anything
// tries to interpret it.
SHA256 string `json:"sha256"`
// Size is the published length in bytes.
//
// Carried so a reader can bound the download rather than discovering the
// length by exhausting a disk. A digest alone cannot do that: it is only
// checkable after the last byte.
Size int64 `json:"size"`
// Compression names how the bytes are packed: "" for none, or "zstd".
Compression string `json:"compression,omitempty"`
// Version is what the artifact calls itself, where that means something —
// the kernel's release for a kernel. Empty elsewhere.
Version string `json:"version,omitempty"`
}
Asset is one downloadable file and the digest that proves it arrived intact.
type Client ¶
type Client struct {
// HTTP is the transport. Nil means a bounded default.
HTTP *http.Client
// Source is where artifacts are fetched from.
Source Source
// contains filtered or unexported fields
}
Client fetches artifacts from a source and proves they are what was published before letting anything else see them.
func (*Client) Download ¶
Download retrieves one asset into dir and returns the path it landed at.
THE ORDER IS THE POINT. Bytes are streamed to a temporary name while being hashed, the length and digest are checked against what the manifest promised, and ONLY THEN does the file get the name the caller will use. A verifier that runs after the final rename has already published unverified bytes under a trusted name, and anything reading the directory concurrently — or after a crash between the two steps — cannot tell the difference.
The alternative shape, streaming straight into the cluster and verifying afterwards, is worse in the same way and harder to undo: it imports unverified bytes into shared storage. Staging costs one disk write and makes the failure a deleted temporary file.
func (*Client) Manifest ¶
Manifest fetches the index for the current release, proves it, and validates it.
THE ONLY WAY A Manifest IS PRODUCED FROM THE NETWORK, and it takes a policy for that reason: a caller cannot obtain one without having said what would make it trustworthy. ParseManifest validates before returning, so no caller can hold an unvalidated one either — which is what lets the download path treat the digests and names as constrained.
THE SIGNATURE IS CHECKED OVER THE BYTES THAT ARRIVED, before anything is parsed out of them. Verifying a re-serialised manifest would verify a document this program produced rather than the one that was signed, and the two can differ in whitespace, key order, or any field a future reader drops.
type Manifest ¶
type Manifest struct {
// Schema is the layout version of this document.
Schema int `json:"schema"`
// GuestContract is the protocol the baked agent speaks to the host.
//
// A STRING, NOT AN INTEGER, because it is compared for equality against the
// provider's own constant and never ordered. Making it a number invites a
// reader to accept "greater than or equal", which is exactly wrong: a newer
// contract is not backward compatible by default, and assuming it is turns a
// clean refusal into a guest that boots and never reports.
GuestContract string `json:"guest_contract"`
// Arch is the machine the image was built for, as `uname -m` spells it.
Arch string `json:"arch"`
// RunnerVersion is the Actions runner baked into the image.
//
// THE FIELD THAT MAKES THE THIRTY-DAY RULE ANSWERABLE. GitHub stops handing
// jobs to a runner more than thirty days past the first release newer than it,
// and this is the one fact about an image that lets anything ask: the importer
// resolves it against the release history before downloading, and declines an
// image whose runner is already refused. The build DATE cannot answer that
// question in either direction, which is why nothing here computes from it.
RunnerVersion string `json:"runner_version"`
// BuiltAt is when the image was produced, in UTC.
BuiltAt time.Time `json:"built_at"`
// Rootfs is the guest's root filesystem, published as one asset.
//
// SCHEMA 1 ONLY. A schema 2 manifest carries RootfsMultipart instead and
// leaves this zero; Validate refuses a document that sets both or neither,
// because "which one describes the image" must never be a guess.
// omitzero, NOT omitempty: omitempty has no effect on a struct, so a schema 2
// manifest would publish an empty "rootfs" object beside its parts and invite
// a reader to treat the two as both present.
Rootfs Asset `json:"rootfs,omitzero"`
// RootfsMultipart is the root filesystem published as ordered parts.
//
// SCHEMA 2 ONLY, and it exists because of a hard limit rather than a
// preference: GitHub caps a single release asset at 2 GiB, and an image
// carrying what a github-hosted runner carries packs to well past that. A
// release may hold up to a thousand assets, so the file is split and put
// back together on the way in.
RootfsMultipart *Multipart `json:"rootfs_multipart,omitempty"`
// Kernel is the kernel built to boot it.
//
// SHIPPED AS A PAIR AND VERSIONED TOGETHER. Whether a kernel can boot a
// given root filesystem is a property of the two together — the guest's
// init, its cgroup layout and the options Docker needs are all decided at
// kernel configuration time. A mismatch is not a degradation, it is a VM
// that does not come up, so the two travel in one manifest and are never
// resolved independently.
Kernel Asset `json:"kernel"`
}
Manifest describes one published guest image: the root filesystem, the kernel built to run it, and the facts a node needs in order to refuse it.
func ParseManifest ¶
ParseManifest decodes and validates a manifest document.
REFUSES TRAILING CONTENT. A document with a second JSON value after the first is not a manifest, and accepting one lets a publisher — or anything that can rewrite the response — append a version that a different reader would honour.
func (*Manifest) Aging ¶
Aging reports that the image is old enough to be worth mentioning.
MAINTENANCE INFORMATION, AND NOTHING MORE. There used to be a Stale() beside this that REFUSED an import at built_at + 30 days, on the reasoning that a runner baked N days ago is at least N days old. The arithmetic is true and the conclusion does not follow: GitHub's window opens when the first release NEWER than the baked one appears, so an image built the day a release shipped is still current a year later if nothing else ships, and an image built yesterday around a runner three releases behind is already refused. It rejected images that worked and accepted images that could not, from a number that was never about GitHub at all.
What settles acceptance is the release history, which needs a network call this package deliberately does not make — so the caller asks runnerrelease and is honest when it cannot reach it. Age stays as what it always was: a fact about the artifact.
func (*Manifest) Downloads ¶
Downloads is every asset a node must fetch, in the order it should fetch them.
THE KERNEL LAST, because it is small and the root filesystem is what decides whether the import is worth continuing at all.
func (*Manifest) RootfsImage ¶
RootfsImage describes the root filesystem in ONE shape, whichever schema published it.
THIS IS WHY THE AGGREGATE CANNOT BE SKIPPED. A schema 1 manifest is normalized into a single-part multipart whose whole-file digest is that one asset's digest, so every consumer runs the same download-then-assemble-then-verify path and there is no second, shorter route that stops after the parts. Adding a parts field without this left the old per-asset path in place beside the new one, and the old path is exactly the one that verifies pieces and never checks that they were joined correctly.
The returned value is a copy: Parts aliases nothing the caller can use to change what a later verification hashes.
func (*Manifest) Usable ¶
Usable reports whether this build can import the image the manifest names.
SEPARATE FROM Validate, because the two answer different questions and only one of them means the publisher did something wrong. Validate asks whether the document is well formed; this asks whether THIS billet, on THIS machine, can use what it describes. A well-formed manifest for another architecture is not a defect to report to anyone — it is simply not for this host.
func (*Manifest) Validate ¶
Validate reports everything wrong with a manifest, or nil.
CALLED BEFORE ANY FIELD IS USED, including before the digests are trusted enough to check bytes against. The document arrives over the network from a service nobody here controls, so every field is treated as an assertion by a stranger until it has been through this.
type Multipart ¶
type Multipart struct {
// Name is what the reassembled file is called once the parts are joined.
//
// A BARE FILENAME, validated as one, for the reason Asset.Name is: it is
// joined to a staging directory to write.
Name string `json:"name"`
// SHA256 is the digest of the REASSEMBLED bytes, lowercase hex.
SHA256 string `json:"sha256"`
// Size is the reassembled length in bytes.
Size int64 `json:"size"`
// Compression names how the reassembled file is packed: "" or "zstd".
//
// ON THE WHOLE, NEVER ON A PART. The parts are byte ranges of one
// compressed stream, not independently compressed files, so decompressing a
// part on its own is meaningless and a part that claimed a compression would
// be describing something that does not exist.
Compression string `json:"compression,omitempty"`
// Parts are the published pieces, in the order they concatenate.
Parts []Asset `json:"parts"`
}
Multipart is one logical file published as several release assets.
THE WHOLE-FILE DIGEST IS THE LOAD-BEARING CHECK, and the per-part digests do not replace it. Each part's digest proves that piece arrived as published; only the digest of the reassembled file proves the pieces were put back together in the right order and none was dropped or repeated. A reader that verified parts and skipped the whole would accept a correctly-signed manifest whose parts had been reordered into a different filesystem.
func (*Multipart) Assembled ¶
Assembled reports whether this image arrives as more than one file.
Used only to decide what to tell an operator; nothing branches on it, because both shapes take the same path.
func (*Multipart) UnpackedName ¶
UnpackedName is what the assembled file is called once it is decompressed.
DEFINED HERE RATHER THAN AT THE CALL SITE so validation and the unpacker cannot disagree about it. They did: the name was derived in cmd/billet and checked nowhere, which is what let a decompression land on another asset.
type Policy ¶
type Policy struct {
// Required is whether the manifest's signature must verify.
Required bool
// Identity is the certificate SAN pattern a valid signature must carry.
Identity string
// Issuer is the OIDC issuer that certificate must come from.
Issuer string
// SourceRepositoryURI binds a reusable workflow to the repository it builds.
SourceRepositoryURI string
}
Policy says what a source demands before its manifest may be trusted.
func PolicyFor ¶
PolicyFor decides what verification a source demands.
THE MANIFEST IS THE ONLY THING THAT MAKES THE DIGESTS MEAN ANYTHING. Every asset is checked against a digest the manifest names, so a manifest an attacker serves names digests of bytes the attacker chose and every check passes. The signature is what binds the manifest to the workflow that produced it, and without it the rest of the verification is a checksum against itself.
SO A MISSING POLICY IS AN ERROR RATHER THAN A SKIP. The tempting shape -- "verify if an identity is configured" -- fails open exactly when an operator points at their own mirror and configures nothing, which is the common case and the one where the guarantee silently disappears.
type Source ¶
type Source struct {
// BaseURL is the directory the manifest and its assets sit in, without a
// trailing slash. It is populated by Client.Resolve for the default source.
BaseURL string
// contains filtered or unexported fields
}
Source names where images are fetched from.
func DefaultSource ¶
func DefaultSource() Source
DefaultSource discovers the current immutable guest-image release published by this project.
func ParseSource ¶
ParseSource validates a configured base URL.
HTTPS IS REQUIRED EVEN THOUGH THE CONTENT IS VERIFIED. Digest and signature checks make plaintext transport survivable, not harmless: an observer still learns which version a deployment runs, and — more usefully to them — an active party can hold a node on an old-but-genuine manifest indefinitely, which passes every check this makes and quietly walks the fleet into the thirty-day expiry. Transport security is what makes that expensive.
func (Source) AssetURL ¶
AssetURL is where a named file within the release lives.
The name has already been through Asset.validate, which is what makes simple concatenation safe: it holds no separator, so it cannot climb out of the release it belongs to.
func (Source) IsDefault ¶
IsDefault reports whether this is billet's signed first-party image channel.
func (Source) ManifestURL ¶
ManifestURL is where the index for the current release lives.
Source Files