Documentation
¶
Overview ¶
Package bundle implements harness-source-v1alpha1, the single artifact a "chainctl harness hunt" submission becomes whatever source kind it was built from: a zstd-compressed tar of normalized entries with a manifest first.
chainctl packs a bundle with Writer and the pipeline streams it back through Validate. Both ends link this package so the rules below have one implementation; the server validates every bundle regardless of what the client checked.
The tar ¶
The manifest is the first entry, at ManifestPath; that path is reserved and a source tree containing it is rejected. Every later entry is sorted after it by Entry.Path in byte order, and each path appears once. The ordering key is the entry's path, not the name it takes in the tar: a directory is stored with a trailing slash, and "cmd/" sorts after "cmd.go" where "cmd" sorts before it. Sorting on the tar name would put a directory and its siblings in a different order and reject a valid bundle.
Paths are relative, slash-separated and clean: no leading slash, no ".." or "." segment, no empty segment, no NUL, no backslash, and valid UTF-8. A directory is stored under a name with one trailing slash, which is not part of its Entry path.
Entry paths and link targets reject Windows drive prefixes (including drive-relative forms such as "C:outside"), backslashes, and components ending in a space or dot. Exact "." and ".." components remain valid in relative link targets. These rules apply on every host: Windows drive resolution, backslash separators and trailing-character aliases can otherwise change the tree the validator checks. Intake drops these unrepresentable entries.
Only three entry types exist. A regular file has mode 0644, or 0755 if any execute bit was set on the source; a directory has mode 0755; a symlink has mode 0777 and a relative target that resolves inside the tree. A hardlink, a device, a FIFO, a socket and an escaping symlink are rejected here — intake drops them with a warning before they reach the writer, by the same rule: Normalize is what decides, for one entry, whether the format can carry it. The writer and reader also resolve symlink chains across the complete tree, following each link before applying a subsequent "..". Resolution follows at most 40 links; cycles and longer chains are rejected. No entry may have a regular file or symlink as an ancestor, so entries cannot alias each other through a link. Paths are compared case- and normalization-insensitively here, because Windows and macOS resolve names that way, and a path the bundle uses as a directory cannot also name a file or a link under that comparison. Nor may one directory appear under two spellings, whether an entry declares it or a descendant implies it: such a host keeps one of them, so the paths that land would not be the paths tree_sha256 pins. Streaming extraction must use os.Root as described by Reader.Read, because forward links cannot be checked until the end.
Metadata is zeroed: mtime is the Unix epoch, uid and gid are 0, uname and gname are empty, and no PAX record other than path and linkpath is written or accepted.
After the end-of-archive marker a reader reads on to EOF and requires every remaining byte to be zero, within Limits.MaxTrailingBytes. Refusing padding outright would reject a conforming writer — GNU tar pads its output to a whole 20-block record by default — and leaving it unread would let a bundle carry bytes that no entry declares and tree_sha256 does not cover. What is left of that channel is the padding's length, a few bits that carry nothing on their own; the bundle's own SHA-256, declared at intake and checked after decryption, covers them along with every other byte.
Determinism ¶
The tar layer is reproducible: one tree, packed by any caller in any enumeration order, is the same tar bytes, because entry order, modes, mtimes, ownership and the PAX records written are all pinned here. That holds for a given Go toolchain — the exact framing comes from archive/tar, which is free to change when it writes an extended header, so a future toolchain may frame the same tree differently.
The compressed bundle is not pinned. zstd's output depends on the encoder's version and its level, and the format pins the codec rather than the ratio (see Writer.Write), so two clients on different builds can send different bytes for one tree and both be correct. Nothing in the format compares compressed bundles.
What pins exact bytes end to end is the bundle's own SHA-256, declared at intake and checked after decryption; what pins the tree independently of how it was compressed is tree_sha256.
tree_sha256 ¶
tree_sha256 lets a reader confirm the tar it streamed is the tree the manifest describes, independently of how it was compressed. Both ends and any reimplementation must agree on it byte for byte, so it is defined exactly:
tree_sha256 = lowercase hex of SHA-256 over
field("harness-source-v1alpha1-tree") || record(e₁) || record(e₂) || …
over every entry of the bundle except the manifest, in the one order the
format allows them in the tar — ascending by path, comparing paths as
unsigned bytes — where
record(e) = field(path) || field(mode) || field(type) || field(payload)
field(s) = decimal(len(s)) || ":" || s || "\n"
decimal the byte length of s in ASCII digits, shortest form: no
leading zero, no sign, no padding ("0" for an empty s)
":" one ASCII colon (0x3A), "\n" one ASCII newline (0x0A)
s the field's bytes, verbatim and unescaped, including any
colon or newline of its own
path the entry's path, with no leading or trailing slash (a
directory's trailing "/" in the tar is not part of it)
mode the normalized mode as four octal digits, one of exactly
"0644" | "0755" | "0777"
type one of exactly "file" | "dir" | "symlink"
payload a file's content as 64 lowercase hex digits of SHA-256;
a directory's empty string;
a symlink's target as stored in the tar linkpath, byte for
byte, with no cleaning and no trailing slash
Nothing else is hashed: not the entry's size, not the manifest's own
fields, not any tar framing. A bundle with no entries hashes the domain
field alone.
Every field is length-prefixed, as in chainguard.dev/sdk/skills.HardenJobID, and every record is exactly four fields, so the byte stream parses back to one entry list and no field or record boundary is ambiguous — colons and newlines inside a path or a link target are ordinary bytes the length already accounted for. Two distinct trees therefore cannot produce the same stream, and no payload of one type can be read as another: type is itself a field, so a symlink whose target is 64 hex digits and a file whose content hashes to them differ in the record before the payload. The leading domain tag pins the scheme, so a future change to the record layout yields distinct hashes rather than colliding with a v1 bundle's.
Limits ¶
DefaultLimits is the one policy both ends apply; no call site picks its own. Every limit is enforced before the thing it bounds exists: the reader checks each one against the stream so far, on every read, so a decompression bomb is refused while it inflates rather than once it has; the writer refuses a byte that would carry the tar stream or the compressed bundle past its limit instead of counting it afterwards. The one allowance is Limits.RatioFloorBytes, the decoded size below which the ratio is not judged; that floor is all a bomb gets for free, and no more, because Limits.MaxDecodedBytes still bounds the stream it has to pay for. The writer also streams its compressed output through Reader.Read and waits for validation before returning a manifest, so it enforces the same prefix ratio policy without buffering the complete bundle.
A rejection is an Error carrying a stable Reason the caller can branch on and surface.
Index ¶
Examples ¶
Constants ¶
const ( // Format is the value of a manifest's format field, and the name of this // bundle format. Format = "harness-source-v1alpha1" // ManifestPath is the reserved path of the manifest, the bundle's first // entry. A source tree that contains this path is rejected. ManifestPath = ".chainguard-harness/manifest.json" )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Entry ¶
type Entry struct {
// Path is the entry's slash-separated path relative to the bundle root,
// with no leading or trailing slash.
Path string
// Type is the entry's type.
Type EntryType
// Mode is the entry's permissions. On the way in only the execute bits of a
// regular file are read, to pick 0755 over 0644; everything else is
// normalized away. On the way out it is the normalized mode.
Mode fs.FileMode
// Size is a regular file's content length in bytes, and 0 for other types.
Size int64
// LinkTarget is a symlink's relative target, and empty for other types.
LinkTarget string
// Open opens a regular file's content, and is nil for other types. The
// writer calls it twice — once to digest the content for tree_sha256 and
// once to write it — and rejects the bundle if the two reads disagree, so a
// source that changes underneath the pack is caught rather than packed
// inconsistently.
Open func() (io.ReadCloser, error)
}
Entry is one normalized bundle entry. The writer takes entries from intake and normalizes Mode; the reader yields them with Mode already normalized.
func Normalize ¶
Normalize checks one entry against the format's rules and returns it with its Mode normalized, or an Error naming the Reason it cannot be carried.
It is the seam between intake and the writer. Intake calls it on each candidate and drops what it rejects, with a warning naming the reason; the writer applies the same rule to everything it is handed and refuses it there, because by then an unrepresentable entry is a bug rather than a tree the customer happened to have. Both share these per-entry rules.
Whole-tree rules — ordering, duplicates, entry count, non-directory parents and symlink chains — are checked by the writer and reader. Normalize alone does not establish symlink containment.
Example ¶
Intake walks a source tree and keeps what the format can carry, warning about the rest. It asks Normalize rather than restating the rules, so what it drops and what the writer would refuse stay the same set.
package main
import (
"errors"
"fmt"
"io"
"strings"
"chainguard.dev/sdk/harness/bundle"
)
// content is one source file an entry reads from. Intake opens a file on disk
// or a git blob; the writer only needs it twice, once to digest and once to
// write.
func content(body string) func() (io.ReadCloser, error) {
return func() (io.ReadCloser, error) { return io.NopCloser(strings.NewReader(body)), nil }
}
func main() {
candidates := []bundle.Entry{
{Path: "cmd/run.sh", Type: bundle.TypeFile, Mode: 0o700, Open: content("")},
{Path: "docs/entry.md", Type: bundle.TypeSymlink, LinkTarget: "../README.md"},
{Path: "etc/passwd", Type: bundle.TypeSymlink, LinkTarget: "../../../etc/passwd"},
{Path: "var/run.sock", Type: bundle.EntryType("socket")},
}
keep := make([]bundle.Entry, 0, len(candidates))
for _, c := range candidates {
e, err := bundle.Normalize(c)
if err != nil {
// Intake warns and moves on; the writer, handed the same entry,
// would refuse the whole bundle.
r, _ := errors.AsType[*bundle.Error](err)
fmt.Printf("dropped %s: %s\n", c.Path, r.Reason)
continue
}
keep = append(keep, e)
}
for _, e := range keep {
fmt.Printf("kept %s %04o\n", e.Path, e.Mode.Perm())
}
}
Output: dropped etc/passwd: symlink-escape dropped var/run.sock: unsupported-entry-type kept cmd/run.sh 0755 kept docs/entry.md 0777
type EntryType ¶
type EntryType string
EntryType is one of the three permitted entry types. Its value is the token the tree hash covers, so these strings are part of the format.
type Error ¶
type Error struct {
// Reason is the stable code for the rejection.
Reason Reason
// Path is the offending entry's path, when the rejection has one.
Path string
// contains filtered or unexported fields
}
Error is a bundle rejection.
type Kind ¶
type Kind string
Kind is the source kind a bundle was built from. It records what chainctl found and how it packed it, not the flag the customer passed: "--workspace" packs a git checkout one way and a plain directory another. The bundle never records where the source lived.
The source kinds a client packs today.
const KindSource Kind = "source"
KindSource is reserved. "chainctl harness hunt" does not yet take a repository as a source, so no writer emits it and validation rejects it; the name is spoken for so that taking the work up later adds a kind rather than moving the format onto a name an old bundle already used for something else. It is "source" because that is what the deferred flag is called, and what the sandbox's Session.environment already calls a repository and revision.
type Limits ¶
type Limits struct {
// MaxCompressedBytes bounds the bundle as it arrives, before decompression.
MaxCompressedBytes int64
// MaxDecodedBytes bounds the whole decoded tar stream, tar headers and
// skipped payloads included, not just retained file content.
MaxDecodedBytes int64
// MaxFileBytes bounds one regular file.
MaxFileBytes int64
// MaxEntries bounds the number of tar entries, the manifest included.
MaxEntries int
// MaxPathBytes and MaxPathDepth bound one path's length in bytes and its
// number of slash-separated segments.
MaxPathBytes int
MaxPathDepth int
// MaxRatio bounds decoded bytes divided by compressed bytes read.
MaxRatio int64
// RatioFloorBytes is the decoded size below which MaxRatio does not apply.
// The ratio is measured over the stream so far, and a tar's leading bytes
// are its most compressible: headers are mostly padding, so a tree whose
// first paths are small or empty files inflates far above what the whole
// bundle does. Without a floor those bundles would be refused for what they
// start with rather than for what they are. The floor is also all the
// decoded output MaxRatio gives a bomb for free, so it is sized in
// megabytes; MaxDecodedBytes is what bounds a bomb that pays for its
// inflation.
RatioFloorBytes int64
// MaxTrailingBytes bounds the zero padding a bundle may carry after the
// tar's end-of-archive marker. Every one of those bytes must be zero, so
// what is left of the channel is how many of them there are; the allowance
// caps that at a handful of bits, and the bundle's own SHA-256 covers them.
// It is sized to accept a GNU-style blocked writer: GNU tar pads its output
// to a whole 20-block record by default, and this leaves room for the
// 64-block records a writer may be configured with.
MaxTrailingBytes int64
}
Limits bounds a bundle. One value applies on both sides: the client for an early, specific error, the server because the client's checks are not a control.
Callers do not pick limits — DefaultLimits is the policy. A zero field takes its default, which is what lets a test drive one guard at a small size without restating the rest.
func DefaultLimits ¶
func DefaultLimits() Limits
DefaultLimits is the v1 policy, sized for large monorepos. It is the single place these values change; no call site sets its own.
type Manifest ¶
type Manifest struct {
// Format is always [Format]. The writer sets it.
Format string `json:"format"`
// Kind is the source kind the bundle was built from.
Kind Kind `json:"kind"`
// Commit is the 40-hex base commit the tree was packed against, on
// [KindGit] only, and empty otherwise. A folder and a file list have no
// commit to name.
Commit string `json:"commit,omitempty"`
// Dirty records that the checkout's working tree differed from Commit when
// it was packed. It is false on every other kind.
Dirty bool `json:"dirty"`
// IncludeIgnored records that the checkout's ignore rules were disregarded
// and files .gitignore excludes were packed anyway. Without it a bundle
// holding .env and build output is indistinguishable from one that
// respected those rules: both are [KindGit] at the same commit.
//
// It is false on every other kind, not because they exclude anything but
// because they have no ignore rules to disregard; the kind is already what
// says everything on disk was packed.
IncludeIgnored bool `json:"include_ignored"`
// Files is the number of regular-file entries. The writer sets it.
Files int `json:"files"`
// Bytes is the total content size of those files. The writer sets it.
// Neither Files nor Bytes counts the manifest, directories or symlinks.
Bytes int64 `json:"bytes"`
// TreeSHA256 is the hex tree hash over the entries, defined exactly in the
// package doc. The writer sets it; the reader recomputes it from the
// entries it streamed and rejects a bundle that disagrees.
TreeSHA256 string `json:"tree_sha256"`
// Client identifies the packer, e.g. "chainctl v0.1.2".
Client string `json:"client"`
}
Manifest describes a bundle, and is its first entry at ManifestPath.
These fields are the complete list of metadata a bundle carries, and none of them resolves to a customer: no URL, remote, author or local path, and no customer-supplied free text. Commit and TreeSHA256 pin the tree without them.
func Validate ¶
Validate streams the bundle in src under DefaultLimits, enforcing every format rule, and returns its manifest. It is the server's entry point, run over every bundle regardless of what the client checked.
Example ¶
package main
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"strings"
"chainguard.dev/sdk/harness/bundle"
)
// content is one source file an entry reads from. Intake opens a file on disk
// or a git blob; the writer only needs it twice, once to digest and once to
// write.
func content(body string) func() (io.ReadCloser, error) {
return func() (io.ReadCloser, error) { return io.NopCloser(strings.NewReader(body)), nil }
}
func main() {
ctx := context.Background()
var buf bytes.Buffer
if _, err := (&bundle.Writer{}).Write(ctx, &buf, bundle.Manifest{
Kind: bundle.KindFolder,
Client: "chainctl v0.1.2",
}, []bundle.Entry{{
Path: "main.go",
Type: bundle.TypeFile,
Mode: 0o644,
Size: 29,
Open: content("package main\n\nfunc main() {}\n"),
}}); err != nil {
fmt.Println(err)
return
}
// The server runs Validate over every bundle, whatever the client checked.
m, err := bundle.Validate(ctx, bytes.NewReader(buf.Bytes()))
fmt.Println(m.Kind, m.Files, err)
// A rejection carries a stable reason to surface, not just a message.
_, err = bundle.Validate(ctx, strings.NewReader("not a bundle"))
if e, ok := errors.AsType[*bundle.Error](err); ok {
fmt.Println(e.Reason)
}
}
Output: folder 1 <nil> malformed-bundle
type Reader ¶
type Reader struct {
// Limits bounds the stream. A zero field takes its default, so the zero
// Reader enforces exactly [DefaultLimits]. Only tests set it, to drive a
// guard at a small size.
Limits Limits
}
Reader streams a bundle. The zero value is the one production callers use; its Limits default to DefaultLimits.
func (*Reader) Read ¶
func (r *Reader) Read(ctx context.Context, src io.Reader, fn func(Entry, io.Reader) error) (*Manifest, error)
Read streams the bundle in src, enforcing every format rule and limit as the bytes arrive, and returns its manifest.
If fn is non-nil it is called for each entry after the manifest, in tar order, with a reader over a file's content that is valid only until Read moves on; unread content is drained. A non-nil error from fn stops the stream and is returned unchanged.
Entries reach fn before the stream has been fully checked: tree_sha256, counts and symlink chains can only be confirmed at the end. A caller that materializes a bundle must confine every filesystem operation through an os.Root opened on its staging directory, then commit only once Read returns nil. Staging alone does not contain a link that escapes before validation.
A cancelled ctx stops the stream between entries and is returned as it is, not as an Error: nothing was wrong with the bundle.
Example ¶
package main
import (
"bytes"
"context"
"fmt"
"io"
"strings"
"chainguard.dev/sdk/harness/bundle"
)
// content is one source file an entry reads from. Intake opens a file on disk
// or a git blob; the writer only needs it twice, once to digest and once to
// write.
func content(body string) func() (io.ReadCloser, error) {
return func() (io.ReadCloser, error) { return io.NopCloser(strings.NewReader(body)), nil }
}
func main() {
ctx := context.Background()
var buf bytes.Buffer
if _, err := (&bundle.Writer{}).Write(ctx, &buf, bundle.Manifest{
Kind: bundle.KindFiles,
Client: "chainctl v0.1.2",
}, []bundle.Entry{{
Path: "notes.txt", Type: bundle.TypeFile, Mode: 0o600, Size: 6, Open: content("hello\n"),
}, {
Path: "run.sh", Type: bundle.TypeFile, Mode: 0o755, Size: 12, Open: content("echo hello\n\n"),
}}); err != nil {
fmt.Println(err)
return
}
// Entries arrive sorted, with modes normalized. Nothing read here is
// trustworthy until Read returns: tree_sha256 is only confirmed at the end,
// so a consumer that writes must stage and commit on success.
_, err := (&bundle.Reader{}).Read(ctx, &buf, func(e bundle.Entry, body io.Reader) error {
b, err := io.ReadAll(body)
if err != nil {
return err
}
fmt.Printf("%s %04o %d\n", e.Path, e.Mode.Perm(), len(b))
return nil
})
fmt.Println(err)
}
Output: notes.txt 0644 6 run.sh 0755 12 <nil>
type Reason ¶
type Reason string
Reason is a stable code for why a bundle was rejected. It is what a caller branches on and surfaces; the message is for a human.
const ( ReasonUnsafePath Reason = "unsafe-path" ReasonPathTooLong Reason = "path-too-long" ReasonTooDeep Reason = "too-deep" ReasonReservedPath Reason = "reserved-path" ReasonDuplicatePath Reason = "duplicate-path" ReasonUnsortedEntry Reason = "unsorted-entry" ReasonEntryType Reason = "unsupported-entry-type" ReasonSymlinkEscape Reason = "symlink-escape" ReasonMetadata Reason = "unnormalized-metadata" ReasonFileTooLarge Reason = "file-too-large" ReasonBundleTooLarge Reason = "bundle-too-large" ReasonStreamTooLarge Reason = "stream-too-large" ReasonTooManyEntries Reason = "too-many-entries" ReasonDecompressRatio Reason = "decompress-ratio" ReasonTrailingBytes Reason = "trailing-bytes" ReasonInvalidManifest Reason = "invalid-manifest" ReasonTreeHashMismatch Reason = "tree-hash-mismatch" ReasonMalformed Reason = "malformed-bundle" ReasonSourceChanged Reason = "source-changed" )
The rejection reasons.
type Writer ¶
type Writer struct {
// Limits bounds the bundle being written. A zero field takes its default,
// so the zero Writer enforces exactly [DefaultLimits]. Only tests set it,
// to drive a guard at a small size.
Limits Limits
}
Writer packs a bundle. The zero value is the one production callers use; its Limits default to DefaultLimits.
func (*Writer) Write ¶
func (w *Writer) Write(ctx context.Context, dst io.Writer, m Manifest, entries []Entry) (*Manifest, error)
Write packs entries into a harness-source-v1alpha1 bundle on dst and returns the manifest it wrote.
The caller supplies Kind, Commit, Dirty, IncludeIgnored and Client; Write computes Format, Files, Bytes and TreeSHA256 from the entries so the manifest cannot disagree with what was packed. Entries are sorted by path in byte order and normalized here, so one tree packs to one tar whatever order intake enumerated it in. The compressed bytes are not pinned that way: the encoder's version and level decide them, which is why the manifest carries tree_sha256 and intake declares the bundle's SHA-256. See the package doc.
Write rejects invalid entries and unsafe symlink chains. Normalize lets intake check individual entries; chain containment requires the whole tree.
Write streams, so dst holds a partial bundle whenever Write returns an error — including the ReasonSourceChanged it raises when a file changes under the pack. The caller discards dst on any error; nothing it holds is a bundle.
A cancelled ctx stops the pack between entries and is returned as it is, not as an Error: nothing was wrong with the bundle.
Example ¶
package main
import (
"bytes"
"context"
"fmt"
"io"
"strings"
"chainguard.dev/sdk/harness/bundle"
)
// content is one source file an entry reads from. Intake opens a file on disk
// or a git blob; the writer only needs it twice, once to digest and once to
// write.
func content(body string) func() (io.ReadCloser, error) {
return func() (io.ReadCloser, error) { return io.NopCloser(strings.NewReader(body)), nil }
}
func main() {
ctx := context.Background()
src := map[string]string{
"main.go": "package main\n\nfunc main() {}\n",
"go.mod": "module example.com/widgets\n\ngo 1.27.0\n",
"README.md": "# widgets\n",
}
entries := make([]bundle.Entry, 0, len(src))
for path, body := range src {
entries = append(entries, bundle.Entry{
Path: path,
Type: bundle.TypeFile,
Mode: 0o644,
Size: int64(len(body)),
Open: content(body),
})
}
var buf bytes.Buffer
m, err := (&bundle.Writer{}).Write(ctx, &buf, bundle.Manifest{
Kind: bundle.KindFolder,
Client: "chainctl v0.1.2",
}, entries)
if err != nil {
fmt.Println(err)
return
}
// The manifest is filled in from what was packed, and the bytes do not
// depend on the order the entries were handed over in.
fmt.Println(m.Format, m.Kind, m.Files, m.Bytes)
fmt.Println(m.TreeSHA256)
}
Output: harness-source-v1alpha1 folder 3 77 8535f518bde5831908c614a2e60a6de5dd74cebc0c0eead54f8d745ae55f6813
Source Files