Documentation
¶
Overview ¶
Package build defines the layer build pipeline and recipe contract.
Layers are pre-built, pre-validated binary artifacts. They are never built at profile resolution time — the registry fails loudly if a layer does not exist. Layers are produced by the build pipeline defined here.
Recipe contract ¶
A recipe lives at recipes/<name>/<version>/ and contains two files:
- build.sh — install script; must install into $STRATA_PREFIX only
- meta.yaml — declares provides, build_requires, runtime_requires
The build environment contract:
STRATA_PREFIX — install here and only here STRATA_NCPUS — use for parallel builds (nproc) STRATA_ARCH — target architecture (x86_64, arm64) STRATA_OUT — output directory for the squashfs (same as STRATA_PREFIX)
build.sh must exit non-zero on any failure. Network access is allowed during build but the resulting squashfs must be self-contained.
Build pipeline stages ¶
- Resolve build environment from registry (build_requires)
- Launch clean EC2 instance matching target base
- Mount build environment via Strata overlay
- Execute recipe with STRATA_PREFIX=/strata/out
- Capture /strata/out → squashfs (reproducible options)
- Probe squashfs: what does it actually provide?
- Validate: declared provides ⊆ probed provides
- Generate content manifest (file path → SHA256)
- Sign with cosign → log to Rekor
- Push squashfs + manifest + bundle to S3 registry
- Terminate build instance
Index ¶
- func ArchForEC2(normalizedArch string) string
- func CreateSquashfs(ctx context.Context, srcDir, outPath string) error
- func GenerateModulefile(outputDir, installPrefix string, meta *RecipeMeta) error
- func Run(ctx context.Context, job *Job, recipe *Recipe, reg PushRegistry, ...) (*spec.LayerManifest, error)
- func SquashfsOptions() []string
- func StageSources(ctx context.Context, sources []RecipeSource, destDir, arch string, ...) error
- type ContentManifest
- type DryRunExecutor
- type EC2Config
- type EC2Runner
- type EnvLayer
- type EnvResolver
- type ErrMksquashfsNotFound
- type Executor
- type FakeBuildEnvResolver
- type Job
- type LocalExecutor
- type ModuleEnvVar
- type Plan
- type PushRegistry
- type Recipe
- type RecipeMeta
- type RecipeSource
- type RegistryBuildEnvResolver
- type RegistryClient
- type SpawnLauncher
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ArchForEC2 ¶
ArchForEC2 returns the binary arch suffix for the strata binary filename. x86_64 → "amd64", arm64 → "arm64".
func CreateSquashfs ¶
CreateSquashfs creates a reproducible squashfs image from srcDir at outPath. It uses the flags from SquashfsOptions() to ensure reproducibility. Returns *ErrMksquashfsNotFound if the mksquashfs binary is absent from PATH.
func GenerateModulefile ¶
func GenerateModulefile(outputDir, installPrefix string, meta *RecipeMeta) error
GenerateModulefile writes an Lmod modulefile for the layer rooted at outputDir. The file is written to:
<outputDir>/modulefiles/<name>/<version>.lua
Standard path variables (PATH, LD_LIBRARY_PATH, PKG_CONFIG_PATH, MANPATH, INFOPATH, CPATH) are auto-detected by inspecting the subdirectories present under installPrefix. Declared ModulefileEnv vars are appended as setenv directives. A conflict() directive is emitted for each provided capability name so that only one version can be active in a user session at a time.
GenerateModulefile is a no-op (returns nil) for flat install layouts. The caller should check meta.InstallLayout before calling.
func Run ¶
func Run( ctx context.Context, job *Job, recipe *Recipe, reg PushRegistry, executor Executor, signer trust.Signer, ) (*spec.LayerManifest, error)
Run executes the local build pipeline for a single recipe.
Stages 2 (EC2 launch), 3 (overlay mount), and 11 (terminate) are skipped in v0.9.0 — builds run on the local machine. Pass job.DryRun=true to validate and print a summary without executing.
reg may be nil when job.DryRun is true.
func SquashfsOptions ¶
func SquashfsOptions() []string
SquashfsOptions returns the mksquashfs flags required for reproducible output. Same recipe + same build environment = same SHA256.
func StageSources ¶ added in v0.27.0
func StageSources(ctx context.Context, sources []RecipeSource, destDir, arch string, client *http.Client) error
StageSources fetches every declared source into destDir, verifying each against its pinned SHA256 before the build runs. A mismatch — a source that drifted or was tampered with — fails with both the pinned and the actual digest, so the build never proceeds from bytes the recipe did not pin (#68). The verified files are what build.sh reads from $STRATA_SOURCES; it fetches nothing itself, which is what makes the layer reproducible from the recipe.
client is a parameter so tests drive it against an httptest server; the build pipeline passes nil for a default client with a generous timeout.
Types ¶
type ContentManifest ¶
type ContentManifest struct {
// LayerID is the layer this manifest describes.
LayerID string `yaml:"layer_id"`
// Files maps file path (relative to squashfs root) to SHA256.
Files map[string]string `yaml:"files"`
}
ContentManifest is the per-file content record generated from a built layer. Every file in the squashfs is listed with its path and SHA256. Used by the resolver for file-level conflict detection.
func GenerateContentManifest ¶
func GenerateContentManifest(dir, layerID string) (*ContentManifest, error)
GenerateContentManifest walks dir recursively and computes the SHA256 of each regular file. Paths in Files are relative to dir with a leading "/".
func (*ContentManifest) ConflictsWith ¶
func (m *ContentManifest) ConflictsWith(other *ContentManifest) []string
ConflictsWith reports whether this manifest conflicts with other. Two manifests conflict if any file path appears in both, with different SHA256 values. Identical files (same path, same SHA256) are not conflicts.
type DryRunExecutor ¶
DryRunExecutor validates and prints what would run; never executes the script.
type EC2Config ¶
type EC2Config struct {
// Region is the AWS region, e.g. "us-east-1".
Region string
// AMIID is the base AMI ID for the build instance (must match target OS/arch).
AMIID string
// InstanceType is the EC2 instance type, e.g. "c5.4xlarge" or "c6g.4xlarge".
InstanceType string
// SubnetID is the subnet to launch into; AWS picks a subnet if empty.
SubnetID string
// SecurityGroupID is the security group ID, e.g. "sg-0fca02f58fafcdad1".
SecurityGroupID string
// IAMProfile is the IAM instance profile name, e.g. "strata-builder".
IAMProfile string
// BucketURL is the S3 registry URL, e.g. "s3://strata-registry".
BucketURL string
// BinaryArch is the strata binary architecture suffix: "amd64" or "arm64".
BinaryArch string
// KeyRef is the cosign key reference. If it begins with "s3://", the
// user-data script downloads it to /root/.strata-keys/cosign.key and
// passes that local path to --key. Empty = keyless OIDC.
KeyRef string
// CosignVersion is the cosign release tag to install on the build instance,
// e.g. "v3.0.5". Defaults to "v3.0.5".
CosignVersion string
// RootVolumeGB overrides the root EBS volume size in GiB.
// Default 60. Some AMIs (e.g. DLAMIs) require a larger minimum.
RootVolumeGB int32
// PollInterval is how often to poll for build completion. Default 30s.
PollInterval time.Duration
// UseSpawn launches the build instance through the pinned spore.host spawn
// CLI instead of the AWS SDK, so the instance gets spawn's TTL/auto-terminate
// lifecycle and cannot linger after a failed or hung build (#236). It is
// fire-and-forget: spawn owns the lifecycle, so it pairs with the no-wait
// launch path, not the tag-polling one.
UseSpawn bool
// TTL is the spawn auto-terminate window (e.g. "4h") when UseSpawn is set:
// the instance terminates after it regardless of build outcome — the hard
// cost ceiling. Empty lets spawn apply its default idle timeout.
TTL string
}
EC2Config holds configuration for launching EC2 build instances.
type EC2Runner ¶
type EC2Runner struct {
// contains filtered or unexported fields
}
EC2Runner orchestrates single-layer builds on EC2 instances. It uploads the recipe, launches an instance that runs `strata build` locally, polls for completion via instance tags, and terminates the instance.
func NewEC2Runner ¶
NewEC2Runner creates an EC2Runner backed by real AWS clients using the default credential provider chain.
func (*EC2Runner) LaunchBuildEC2 ¶
func (r *EC2Runner) LaunchBuildEC2(ctx context.Context, jobID string, recipe *Recipe, job *Job) (instanceID string, err error)
LaunchBuildEC2 uploads the recipe to S3 and launches an EC2 build instance, returning the instance ID immediately without waiting for the build to complete. The instance self-terminates on success or self-stops on failure (for log retrieval). Monitor progress via the strata:build-status tag.
When cfg.UseSpawn is set the instance is launched through the pinned spawn CLI with a TTL and on-complete=terminate (#236), so a failed or hung build cannot linger: spawn owns the lifecycle. The failure log is still uploaded to S3 by the user-data before the instance stops, and spawn's TTL reaps it.
type EnvLayer ¶
type EnvLayer struct {
// Manifest is the full layer manifest from the registry.
Manifest *spec.LayerManifest
// SqfsPath is the local path to the downloaded squashfs file.
SqfsPath string
// MountOrder is the layer's position in the OverlayFS lower stack (1 = bottom).
MountOrder int
}
EnvLayer is a resolved build_requires layer with its local .sqfs path, ready for OverlayFS mounting as a build environment.
type EnvResolver ¶
type EnvResolver interface {
// Resolve fetches manifests matching buildRequires (arch + abi) from the
// registry, downloads their squashfs files to cacheDir, and returns the
// resolved layers in mount order (lowest-tier layer first).
Resolve(ctx context.Context, buildRequires []spec.Requirement, arch, abi, cacheDir string) ([]EnvLayer, error)
}
EnvResolver resolves build_requires requirements to local squashfs files, ready for OverlayFS mounting as a build environment. This is Stage 3 of the build pipeline.
type ErrMksquashfsNotFound ¶
type ErrMksquashfsNotFound struct{}
ErrMksquashfsNotFound is returned when mksquashfs is not on PATH.
func (*ErrMksquashfsNotFound) Error ¶
func (e *ErrMksquashfsNotFound) Error() string
type Executor ¶
type Executor interface {
Execute(ctx context.Context, scriptPath string, env []string, outputDir string) error
}
Executor runs a recipe build script in a prepared environment.
type FakeBuildEnvResolver ¶
type FakeBuildEnvResolver struct {
// Layers maps requirement name (e.g. "gcc") to the EnvLayer to return.
// The same layer is returned regardless of version constraint.
Layers map[string]EnvLayer
}
FakeBuildEnvResolver returns pre-configured EnvLayers for testing. It never calls the registry or downloads any files.
func (*FakeBuildEnvResolver) Resolve ¶
func (f *FakeBuildEnvResolver) Resolve(_ context.Context, buildRequires []spec.Requirement, _, _, _ string) ([]EnvLayer, error)
Resolve returns pre-configured layers for each requirement. Returns an error if a requirement has no matching entry in Layers.
type Job ¶
type Job struct {
// RecipeDir is the path to the recipe directory.
RecipeDir string
// Base is the target OS and architecture.
Base spec.BaseRef
// RegistryURL is the S3 URL to push the built layer to.
RegistryURL string
// DryRun skips the actual EC2 launch and squashfs creation.
// Used for recipe validation in CI without cloud access.
DryRun bool
// EnvResolver resolves and downloads build_requires layers for Stage 3
// (OverlayFS build environment). When nil, build_requires are not mounted
// and the layer is marked bootstrap_build=true (Tier 0 only).
EnvResolver EnvResolver
// CacheDir is the local directory for caching downloaded .sqfs files.
// Defaults to os.TempDir()/strata-build-cache when empty.
CacheDir string
// KeyRef is the signing key reference (file path or KMS URI) used to
// sign the layer. Stored as manifest.SignedBy for provenance tracking.
KeyRef string
}
Job describes a single layer build request.
type LocalExecutor ¶
type LocalExecutor struct {
Stdout io.Writer // defaults to os.Stdout
Stderr io.Writer // defaults to os.Stderr
}
LocalExecutor runs build.sh via bash on the local machine.
func (*LocalExecutor) Execute ¶
func (e *LocalExecutor) Execute(ctx context.Context, scriptPath string, env []string, _ string) error
Execute runs scriptPath via bash with the given env vars appended to the process environment. cmd.Dir is set to the parent directory of scriptPath so that relative paths inside the script resolve correctly.
type ModuleEnvVar ¶
type ModuleEnvVar struct {
// Var is the environment variable name, e.g. "GCC_HOME", "CC".
Var string `yaml:"var"`
// Path is the file path relative to the layer's install prefix.
// Empty string means the prefix itself (setenv VAR base).
// A non-empty path is appended: setenv VAR (base .. "/" .. path).
// Example: path "bin/gcc" → setenv("CC", base .. "/bin/gcc")
Path string `yaml:"path"`
}
ModuleEnvVar declares a single environment variable entry for a generated Lmod modulefile.
type Plan ¶
type Plan struct {
// Stages is the ordered list of build stages. Each stage is a set of
// recipe names that can build concurrently.
Stages [][]string
// Recipes maps recipe name to its parsed recipe.
Recipes map[string]*Recipe
}
Plan is a topologically ordered build plan for a catalog of recipes. Recipes in the same Stage can build in parallel on independent EC2 instances. Each stage must complete before the next starts, because later stages may depend on layers produced by earlier stages.
func PlanCatalog ¶
PlanCatalog parses all recipes in recipesDir and returns a BuildPlan with a topologically sorted build order. Recipes that share no dependency relationship appear in the same stage and can build in parallel.
recipesDir is expected to contain subdirectories named after each recipe, each containing version subdirectories (e.g. gcc/13.2.0/). The latest version of each recipe is used.
type PushRegistry ¶
type PushRegistry interface {
PushLayer(ctx context.Context, manifest *spec.LayerManifest,
sqfsPath string, bundleJSON []byte) error
}
PushRegistry is the narrow interface the pipeline needs from the registry. Satisfied by *registry.S3Client. Defined here (consumer side) to enable lightweight mock injection in pipeline tests.
type Recipe ¶
type Recipe struct {
// Meta is the parsed meta.yaml content.
Meta RecipeMeta
// BuildScriptPath is the absolute path to build.sh.
BuildScriptPath string
// Dir is the recipe directory (parent of build.sh and meta.yaml).
Dir string
}
Recipe is a fully parsed recipe: the meta.yaml plus the path to build.sh.
func ParseRecipe ¶
ParseRecipe reads and validates a recipe from dir. dir must contain build.sh and meta.yaml.
type RecipeMeta ¶
type RecipeMeta struct {
// Name is the layer name, e.g. "openmpi".
Name string `yaml:"name"`
// Version is the layer version, e.g. "4.1.6".
Version string `yaml:"version"`
// Description is optional human-readable context.
Description string `yaml:"description,omitempty"`
// Tier classifies where this layer sits in the dependency stack.
// Valid values: "core", "library", "application".
// core: built with OS system compiler; no build_requires (bootstrap).
// library: built with Strata tools; may be dependency-only (user_selectable:false).
// application: end-user software built on top of core/library layers.
// Required; validated by Validate().
Tier string `yaml:"tier"`
// Provides is the list of capabilities this layer makes available.
// Written into LayerManifest.Provides at build time.
Provides []spec.Capability `yaml:"provides"`
// BuildRequires lists capabilities needed during the build but NOT
// included in the output squashfs. The build environment supplies these.
BuildRequires []spec.Requirement `yaml:"build_requires,omitempty"`
// RuntimeRequires lists capabilities required at runtime on the target
// instance. Written into LayerManifest.Requires. The resolver validates
// these against BaseCapabilities before mounting.
RuntimeRequires []spec.Requirement `yaml:"runtime_requires,omitempty"`
// ABI is the C runtime ABI this layer targets: "linux-gnu-2.34" or "linux-gnu-2.35".
// A single ABI covers multiple OS versions (AL2023, Rocky 9/10, RHEL 9).
ABI string `yaml:"abi"`
// UserSelectable is false for dependency-only layers (ucx, hwloc, pmix, libfabric).
// Omit from YAML to default to true. ParseRecipe sets this to true when not
// explicitly set false, because the Go zero value false would hide layers by default.
UserSelectable *bool `yaml:"user_selectable,omitempty"`
// InstallLayout is "versioned" (default) or "flat".
// "flat" layers install directly to $STRATA_PREFIX, not $STRATA_PREFIX/<name>/<version>.
InstallLayout string `yaml:"install_layout,omitempty"`
// ModulefileEnv declares extra environment variables to emit in the
// auto-generated Lmod modulefile beyond the standard auto-detected ones
// (PATH, LD_LIBRARY_PATH, PKG_CONFIG_PATH, MANPATH, INFOPATH, CPATH).
ModulefileEnv []ModuleEnvVar `yaml:"modulefile_env,omitempty"`
// Sources lists the source artifacts the build fetches, each pinned by
// SHA256. The build stages every declared source into $STRATA_SOURCES,
// verifying its digest before build.sh runs, so a build is reproducible from
// its recipe and a tampered source fails the build (#68). A recipe with no
// Sources fetches nothing that is pinned — build.sh reading from the network
// is the reproducibility gap this field closes; see StagedFile.
Sources []RecipeSource `yaml:"sources,omitempty"`
}
RecipeMeta is the parsed content of a recipe's meta.yaml file. It declares what the layer provides and what it needs at build and runtime.
func (*RecipeMeta) ToLayerManifest ¶
func (m *RecipeMeta) ToLayerManifest(arch string) *spec.LayerManifest
ToLayerManifest converts a RecipeMeta to a partial LayerManifest. Fields that are populated at build time (SHA256, Source, RekorEntry, Bundle, ContentManifest, BuiltAt) are left empty.
func (*RecipeMeta) UserSelectableBool ¶
func (m *RecipeMeta) UserSelectableBool() bool
UserSelectableBool returns the effective UserSelectable value. Defaults to true if not explicitly set to false in the recipe YAML.
func (*RecipeMeta) Validate ¶
func (m *RecipeMeta) Validate() error
Validate checks that a RecipeMeta is well-formed.
type RecipeSource ¶ added in v0.27.0
type RecipeSource struct {
// URL is where the artifact is fetched from.
URL string `yaml:"url"`
// SHA256 is the expected content digest (64 lowercase hex). A mismatch
// between this and the fetched bytes fails the build.
SHA256 string `yaml:"sha256"`
// File is the staged filename under $STRATA_SOURCES. Optional; defaults to
// the last path element of URL.
File string `yaml:"file,omitempty"`
// Arch restricts this source to one target architecture ("x86_64" or
// "arm64"). Empty means it applies to every arch — correct for a source
// tarball compiled per arch, where one digest is valid everywhere. An
// arch-specific *binary* distribution declares one source per arch, each with
// the same File, so build.sh reads a consistent $STRATA_SOURCES path while the
// build stages only the source matching the arch it targets.
Arch string `yaml:"arch,omitempty"`
}
RecipeSource is one source artifact a recipe build depends on, pinned by digest. The build fetches URL, verifies its SHA256, and stages it as File under $STRATA_SOURCES; build.sh reads $STRATA_SOURCES/<File> rather than fetching from the network itself, which is what makes the layer reproducible from the recipe (#68).
func (RecipeSource) StagedFile ¶ added in v0.27.0
func (s RecipeSource) StagedFile() string
StagedFile returns the filename this source is staged as under $STRATA_SOURCES: File if set, otherwise the last path element of URL.
type RegistryBuildEnvResolver ¶
type RegistryBuildEnvResolver struct {
Registry RegistryClient
}
RegistryBuildEnvResolver resolves and downloads build environment layers from a Strata registry. Satisfies EnvResolver.
func (*RegistryBuildEnvResolver) Resolve ¶
func (r *RegistryBuildEnvResolver) Resolve(ctx context.Context, buildRequires []spec.Requirement, arch, abi, cacheDir string) ([]EnvLayer, error)
Resolve fetches and downloads each build_requires layer in order. The returned layers are indexed by their position in buildRequires; the first requirement gets MountOrder=1 (lowest in the OverlayFS stack).
type RegistryClient ¶
type RegistryClient interface {
// ResolveLayer returns the highest-versioned layer manifest matching name,
// versionPrefix, arch, and abi.
ResolveLayer(ctx context.Context, name, versionPrefix, arch, abi string) (*spec.LayerManifest, error)
// FetchLayerSqfs downloads the squashfs file for manifest to cacheDir and
// returns the local file path. Cache hit returns the existing file immediately.
FetchLayerSqfs(ctx context.Context, manifest *spec.LayerManifest, cacheDir string) (string, error)
}
RegistryClient is the narrow registry interface needed by RegistryBuildEnvResolver. Satisfied by *registry.S3Client.
type SpawnLauncher ¶ added in v0.29.0
type SpawnLauncher struct {
// contains filtered or unexported fields
}
SpawnLauncher launches strata build instances through the pinned spawn CLI instead of the AWS SDK, so every build instance gets spawn's TTL/auto-terminate lifecycle — no orphaned instance holding EBS after a failed or hung build, which the hand-rolled "stop on failure" path left behind (#236). It shells out via commandRunner, capturing the launched instance ID through spawn's --output-id file rather than parsing stdout.
func NewSpawnLauncher ¶ added in v0.29.0
func NewSpawnLauncher(cfg EC2Config, ttl string) *SpawnLauncher
NewSpawnLauncher builds a launcher that shells out to the real spawn CLI.
func (*SpawnLauncher) Launch ¶ added in v0.29.0
Launch writes the user-data to a temp file, verifies the pinned spawn, invokes it to launch the instance with a TTL and on-complete=terminate, and returns the launched instance ID read from spawn's --output-id file. It is fire-and-forget: spawn owns the instance's lifecycle from here, so there is no stop-on-failure orphan for strata to sweep.