magefiles/

Mage targets

Atmos keeps repository automation in Go-based Mage targets. Run targets from the repository root with the repository-pinned Mage version:

go tool mage -l
go tool mage <target> [arguments]

The target list and per-target usage are generated from the Go declarations:

go tool mage -h <target>

Package layout

magefile.go is a thin //mage:import entrypoint. Targets and their tests live in topic packages: acceptance/ (also coverage), bench/, build/, ci/, lint/, notice/, qualitygate/, release/, s3/, and test/. Shared repository/process helpers, GitHub data types, and test fixtures live under internal/. Command names and the lint:changed default are unchanged. Imported target implementations do not carry a mage build tag: Mage compiles them through normal Go imports. Their tests retain the tag so the dedicated Mage CI job owns their coverage and race instrumentation. Run their full tests with go test -tags=mage ./magefiles/....

Target catalog

Environment variables

Only variables that change target behavior are listed here. Standard Go and GitHub Actions variables continue to work normally.

S3 deployment behavior

s3:deploy stores .cloudposse-deploy-manifest-v1.json in the destination. The manifest records each managed object's SHA-256 digest, size, and content type. On later runs, the target uploads only added or content/metadata-changed files and deletes only removed managed files. Files matched by PROTECTED_PATTERNS are never deleted. Retained protected objects remain in the manifest, including objects discovered during bootstrap, so removing a pattern later makes those objects eligible for cleanup again.

The first deployment uploads every managed file once with explicit metadata, then lists the destination and deletes stale, unprotected remote objects. Browser-significant types are selected by extension; the existing mimetype magic-number library provides the fallback for unknown extensions. Textual types receive explicit UTF-8 metadata, such as text/html; charset=utf-8. The manifest is uploaded only after the deployment succeeds, so a failed run cannot record an incomplete state as current.

aws s3 sync cannot append charset=utf-8 selectively while preserving each file's distinct MIME type: --content-type supplies one value for the entire invocation. The Mage target computes and sends the exact header for each object, so correct browser metadata does not require a second S3-to-S3 copy pass.

All S3 operations use the repository's existing AWS SDK for Go v2 dependency. The target does not shell out to the AWS CLI or create temporary request files. SDK calls preserve typed request metadata, built-in retries, context cancellation, and per-object DeleteObjects error handling.

Callers must serialize deployments that target the same bucket and prefix. The checked-in workflows enforce that contract with one production concurrency group and one concurrency group per preview PR, with in-flight cancellation disabled.

PROTECTED_PATTERNS=$'previews/**\nrobots.txt' \
  go tool mage s3:deploy ./dist s3://example-docs/site

Quality Gate and performance snapshots

Benchmark collection, snapshot retrieval, comparisons, and field tests live in magefiles/bench/. Readiness checks and the composite publisher live separately in magefiles/qualitygate/. The latter contributes bench:report to the benchmark namespace so it can include performance results in the shared comment. Benchmark code never imports or evaluates quality gates.

The Quality Gate maintains one bot-owned PR comment, matching job summaries, and mutually exclusive quality:ready / quality:not-ready labels. Label names, colors, and descriptions live in .github/settings.yml, alongside the repository's existing labels and labels referenced by its automation.

A non-draft PR is ready when all requirements pass:

• All current PR workflows, check runs, and commit statuses have finished successfully (neutral and skipped CI results are allowed). Failures, cancellations, timeouts, pending results, and API errors block readiness. The latest run of each workflow counts, including pending reruns. Checks on the current synthetic merge commit are included as well as head checks.
• Codecov reports success for the current head SHA, using its existing configuration and thresholds. Missing, pending, or neutral coverage results do not pass. Results from previous commits cannot satisfy this requirement.
• CodeRabbit has an active approval on the current head SHA.
• Every PR commit has a GitHub-verified signature.
• Every review thread is resolved or belongs to a dismissed review. Outdated unresolved threads still count. General conversation comments have no GitHub resolution state and are not included.
• Fewer than 150 files changed (149 passes; 150 fails).
• No benchmark report processing error is recorded for the current head.

Performance slowdowns are advisory and do not block this gate. Benchmark execution failures still count as CI failures. Malformed or incomplete benchmark reports refresh the other gates, show “comparison unavailable,” and block readiness. Reconciliation retries failed reports and continues through other PRs before returning aggregated errors. Missing historical baselines do not block readiness. This automation does not modify branch protection or merge PRs.

Adding a quality gate

Add one gateDefinition to defaultQualityRegistry in qualitygate/registry.go, plus the check and its tests. The definition provides a stable ID, display Name, Check, PR-relative EvidencePath, and trusted Remediation instructions. Set Advisory: true only for a check that should warn without blocking readiness. For example:

gateDefinition{
    ID: "example", Name: "Example requirement",
    Check: (*qualityClient).example, EvidencePath: "/checks",
    Remediation: "Inspect the evidence and fix the unmet requirement.",
}

The registry runs checks in registration order and supplies the report links, job summary, label verdict, and dynamic agent instructions. Duplicate IDs or incomplete definitions fail validation; check errors fail closed. Do not copy third-party check text into remediation instructions. Gates are compiled from trusted repository code; artifacts cannot register executable plugins.

Checks should return a concrete Problem and state-specific NextAction for unmet gates. Pending CI calls for waiting; a failed job calls for inspecting its logs; absent Codecov evidence must not be described as failed coverage. Signals name the actual blocking results and are quoted as evidence, never instructions. Remediation is the fallback for simple gates. Keep raw check/review text out of Problem and NextAction; construct those from trusted policy and parsed state.

Run these from the repository root with go tool mage, for example:

go tool mage bench:run build/benchmarks/candidate
go tool mage bench:fetch cloudposse/atmos <main-sha> build/benchmarks/history
go tool mage bench:compare build/benchmarks/history build/benchmarks/candidate build/benchmarks/report

GITHUB_TOKEN authenticates fetch/report operations. ATMOS_BENCH_CANDIDATE_DIR overrides the candidate location for fetch. ATMOS_BENCH_SAMPLES sets 10–100 samples (default 10); ATMOS_BENCH_THRESHOLD sets the positive regression percentage (default 10). ATMOS_BENCH_BASE_SHA records the event's immutable main base. GITHUB_STEP_SUMMARY receives the same Markdown as the sticky comment.

The expandable agent prompt contains instructions for the gates currently failing, with advisory performance investigations listed separately. It uses trusted remediation text rather than copying third-party check output into instructions.

Measurement and comparison contract

The suite includes merging, deep copying, templates, YAML resolution, affected detection, instance listing, repeated locals, and 2,200 deferred store lookups over 11 keys, asserting exactly 11 backend reads. CLI fixtures exercise shared imports, inheritance, locals, templates, and offline YAML functions. describe affected covers no changes, shared catalog changes, and component source changes at 100, 1,000, and 10,000 instances. describe stacks and list instances cover 1,000 instances. Expected output is checked outside the timed region. Fixtures use local Git worktrees and --repo-path without fetching. An additional 1,000-instance catalog-change workload uses --include-dependents with roots, branches, and leaves across ten stacks. It validates the entire nested dependency tree and exercises the resolution path used by Pro uploads without uploading anything.

The FunctionsRepeated/instances=200 affected workload adds 600 references per pass to one static target through !terraform.output, !terraform.state, and atmos.Component. Consumers use one lookup kind each, so a stale value in one path cannot be hidden by another path detecting the change. Scenarios cover unchanged repositories, a changed referenced value, and the same change with --include-dependents. Checks outside timing assert resolved values in HEAD and BASE, exact affected instance names, and all 200 dependents. Masking stays enabled at its default; static backends need neither cloud access nor Terraform. This covers inspection caching and repeated deferred target descriptions. The non-static output/auth-disabled wrapper path and actual OIDC authentication need separate unit tests; static backends return before that wrapper is created.

The StoreCallerStacks/stacks=20/keys=11 workload spreads 200 consumers across 20 caller stacks, changes their shared component, and enables dependents with --process-functions=false --skip=store.get --skip=store. Unlike the default CLI harness, it leaves identity selection implicit and uses fake ambient AWS credentials against a loopback SSM server. It verifies resolved store values, all 200 affected identities, and exactly 22 reads (11 per HEAD/BASE root). This detects caller-stack cache fragmentation for stores without configured auth identities; it does not model a configured default identity or real AWS latency.

Builds and fixture setup are excluded from measurements. Packages run sequentially with GOMAXPROCS=2, CGO disabled, no race/coverage instrumentation, and a one-second calibration for in-process benchmarks. Every CLI sample uses one fresh process with isolated home/configuration/application caches, no cloud credentials, and authentication, telemetry, version checks, and Pro disabled except for the ambient-store scenario described above. CLI results report elapsed time only; harness allocations are not CLI memory. In-process results also include bytes and allocations per operation.

The manifest records source SHA, run/attempt, timestamp, Go/build settings, runner image family, CPU, benchmark inventory, and source/fixture fingerprints. Changes to a package's benchmark definitions invalidate that package's comparisons. Different toolchains, CPUs, runner environments, or build settings are incomparable. Added, removed, changed, and missing metrics are reported explicitly. Raw data must contain the declared sample count for every metric; malformed or incomplete results fail the reporter.

Each baseline-to-PR pair is analyzed independently with pinned golang.org/x/perf benchstat/benchmath. A median increase of at least 10% with p < 0.05 is flagged; a statistically significant increase from zero is also flagged. The newest compatible snapshot determines the headline, while older warnings remain visible individually. These comparisons reuse the same PR samples and are correlated; cross-run timing variability can produce noise. Inspect repeated measurements and full reports before attributing a change to code.

Workflows, trust, and rollout

Benchmarks measures the event's immutable PR merge revision, main pushes, and manual dispatches on ubuntu-24.04 using repository-pinned Go and existing cache restoration. For PRs, SHA identifies the measured merge revision, PRHeadSHA separately identifies the PR head, and BaseSHA records github.event.pull_request.base.sha. The trusted reporter verifies the merge's two parents against the recorded base and workflow's PR head, then selects snapshots reachable from that base. Older snapshots can also reflect intervening main changes. Stale-head and fork association checks continue to use the PR head, not the measured merge SHA. Superseded PR runs cancel; main runs finish. Measurement artifacts use unique run/attempt names and 90-day retention; partial measurements remain available for diagnostics but never qualify as baselines.

Storage uses GitHub Actions artifacts and the built-in GITHUB_TOKEN only; no dedicated credentials or Git object archive are required. Reporters retain read-only repository-content permissions. Expired history is unavailable, not silently rebuilt; use the explicit historical field-test target when needed.

Snapshot selection is frozen at the candidate workflow's creation time: only successful main runs completed before that cutoff and reachable from the recorded main base SHA qualify. Later runs, PR artifacts, expired artifacts, and duplicate source SHAs are excluded. Selection searches up to 1,000 runs, newest first, for ten compatible snapshots. Empty history reports “comparison unavailable”; history bootstraps naturally after merge without rebuilding old commits.

CI Timing Summary is the single automatic completion coordinator: it checks that the latest PR workflows and their jobs have finished, posts its timing comment using the existing TypeScript stopwatch action, then calls the reusable Quality Gate workflow when stopwatch's existing published output is true. The Go reporter publishes its own separate quality/benchmark comment. Only the workflow scheduling is shared; no timing content is passed into Go. Failed or canceled completed runs still produce reports. The standalone Benchmark Report workflow is manual-only for explicit recovery by run ID.

All reporters execute the default branch's trusted code with scoped API permissions. Neither uses pull_request_target. Downloaded PR artifacts are bounded, validated data files, never code or executables. PR association includes forks. The publisher owns only its marked bot comment, rechecks the head before writing, rejects older benchmark runs/attempts, and preserves benchmark results only for the current head. Stopwatch continues to manage its own timing comment.

The read-only Quality Gate Signal forwards PR/review activity to the same completion coordinator. Quality Gate no longer independently subscribes to workflow completions, checks, statuses, or comments. A 15-minute best-effort schedule recovers missed events and late external results, including Codecov and review-thread resolution. It selects open PRs targeting main whose head commit (the later of author and committer date) is within the last 24 hours (inclusive), plus any PR with no gate report for its current head, or whose report for that head has a gate whose evidence was unavailable, at any age. GitHub's updated_at is not used as an activity signal because the gate's own labels and comments, Mergify, CodeRabbit and Codecov advance it for nearly every PR; it only skips the commit lookup for PRs not updated since the cutoff. PRs without a usable report are reconciled first, rotated by schedule slot so a PR that hangs or fails every run cannot keep the PRs behind it from being visited. Event-associated reports are not restricted by age. Commits that exist only in a fork are resolved through the fork's head branch (workflow_run payloads) or a commit SHA search, and a PR matches only while its head is exactly that commit. Manual runs still select all open main PRs. Reconciliation defers while Actions workflows are still pending; completion of Actions workflows does not imply that external checks have finished or passed.

Each PR is reconciled under its own deadline, so one slow or failing PR cannot starve the PRs behind it. Every failure is logged as a workflow warning annotation; the run exits non-zero only when every attempted PR failed (a systemic problem such as credentials or an outage). Isolated failures are retried by the next scheduled run. When the run budget ends, unvisited PRs are left for the next run. If a gate's evidence cannot be read (for example an API error), the existing report for that head, its comment and its label, is kept and the run reports the failure, instead of publishing "not ready" and flipping the label back when the next run succeeds.

Unchanged readiness labels and report comments are not rewritten. Each scheduled run logs its UTC cutoff and selected PR count. Report concurrency coalesces work per PR/head, while scheduled recovery has its own group; publication freshness checks protect overlapping writers. Labels and comments are eventually consistent with GitHub events and are not a replacement for required checks. The schedule bounds activity by head-commit recency, not a fixed request budget, and the completion coordinator still wakes on relevant workflow completions.

These trusted reporters become active after their workflows and Mage code reach the default branch. No compatible main snapshots exist until collection has run there. Benchmark and signal workflow completions participate in the CI timing coordinator.

Historical field tests

When no stored snapshots exist, explicitly rebuild known-good and known-bad versions with the same current CLI harness, rather than comparing whichever benchmarks happened to exist in each old checkout:

go tool mage bench:fieldTest v1.229.0,v1.230.0,v1.230.1,v1.231.0,origin/main .context/field-store

The first comma-separated ref is the reference for every subsequent version. Refs resolve to immutable SHAs before building. All builds finish before measurements begin, using the same local Go toolchain (GOTOOLCHAIN=local), CGO disabled, and GOMAXPROCS=2. The current harness drives the historical binaries; no old test files need to exist or compile. Build worktrees are removed afterward. Binaries, build logs, raw measurements, manifests, and per-version Markdown/JSON/benchstat reports remain under the output directory. The directory must be empty. Use a local toolchain new enough for all refs.

The default field test reproduces #3353: 200 component instances each reference 11 store keys (2,200 template references per processing pass). A loopback SSM endpoint returns fixed values with 1 ms latency per request. This exercises the real CLI, template evaluation, deferred authentication path, SDK client setup, and cache, with synthetic credentials and no AWS access. Both describe affected and describe stacks report elapsed time and backend reads, never harness memory. The fixed CLI may read each key in multiple processing contexts; normal CI bounds those reads at the current main budgets of 44 and 22 respectively. Comparisons still flag increases against older snapshots with fewer reads; the assertion budget is not a claim that current main is optimal. The isolated in-process benchmark separately asserts exactly 11 reads for 2,200 calls in one context. Field tests observe excessive reads instead of aborting, so the known-bad build can produce statistics and demonstrate detection.

ATMOS_BENCH_FIELD_FILTER selects another CLI benchmark expression; for example, ^BenchmarkCLI$/^DescribeAffected$/^instances=1000$ replays the three 1,000-instance scenarios, and ^BenchmarkCLI$ runs the entire CLI suite. For repeated function lookups use ^BenchmarkCLI$/^DescribeAffected$/^FunctionsRepeated$. Append /^instances=200$/^change=none$ to isolate timing on historical releases that fail the changed-value correctness checks; those failures must remain visible and must not be interpreted as a passing performance result. Sample count and regression threshold use the ordinary benchmark environment variables. Historical field-test manifests record the measured revision, source ref, harness revision, and workload fingerprints, and are deliberately incompatible with CI snapshots. This is manual validation, not automatic baseline rebuilding or artifact seeding.

The October 10, 2026 local field test detected the known regression in every bad release. Ten fresh invocations per command/version used Go 1.27.1, macOS on an Apple M4 Max, GOMAXPROCS=2, and the fixed 1 ms loopback delay:

All three bad releases triggered the advisory timing rule against v1.229.0 for both commands (p < 0.00002). Main removed most of the slowdown, but describe affected still triggered a 10.7% increase, p=0.0015 against the good release. Its backend reads increased fourfold; stacks reads doubled even though stacks timing showed no significant increase (p=0.97). These remaining warnings are intentionally preserved. This synthetic local experiment proves detection of #3353; it does not estimate production latency or establish Linux runner noise. A separate unchanged-build check (10 versus 10 CLI samples) showed about 2% timing variation with p=0.91 on this machine.

The remaining read counts are consistent with the fix's deliberate cache key: caller stack and configuration paths are included to preserve authentication isolation. Two stacks explain 22 reads for stacks; two repository contexts explain 44 for affected. The measured increase warrants an advisory, but it does not by itself establish another cache bug or justify removing that isolation.

A second ten-sample field test compared the same revisions without store reads, using ATMOS_BENCH_FIELD_FILTER='^BenchmarkCLI$/^DescribeAffected$/^(instances=1000|Dependents)$'. It found no timing regressions in the additional 1,000-instance paths:

These results narrow the remaining warning to the store-heavy fixture among the paths measured here; they do not prove every affected-detection path is faster.

Coverage checklist from docs/fixes

A workload being present does not prove it detects every regression. Validate a known-good/bad pair before claiming historical detection, and keep provider or runner-specific gaps explicit.