README
¶
Integration Test Harness
Stands up real AerolVM deployments from the released install artifacts and
drives the live /v1 API + Go SDK against them, producing a use-case coverage
matrix. Full design + decisions: ../plans/integration-tests.md.
⚠️ These tests provision real AWS EC2 and cost money. They use the production Terraform module but against isolated state (
integration/<scenario>key + separateTF_DATA_DIR) so production state is never touched. Spot instances + auto-teardown +make integration-reapkeep cost down.
Status
Live scenarios cover single-node (docker / containerd / fc / wasm / isolate),
3× mixed clusters (docker, containerd, gvisor, wasm, fc), hetero, local-mode,
and arm64. Reports and the coverage matrix live in reports/. Isolate is
exercised by make integration-single-isolate (UC-103/104/105) and
make integration-benchmark-isolate (UC-94).
Prerequisites
terraform,go,awscli,jq,yq,curl,openssl,sshzig(brew install zig) — the cross-compiling C toolchain the local artifact build needs;sandboxdlinks CGO sqlite, so a pure-Go cross-compile is not an option.xcaddyonly for--with-caddy.- AWS credentials with permission to create EC2/VPC/etc.
config/secrets.ymlpopulated (PAT, Cloudflare token) — seeconfig/secrets.example.ymlintegration-tests/scenarios/domains.yml— copy fromdomains.example.yml, fill in real test FQDNs you control (distinct from your prod domain)
Run
make integration-single # single-node scenario
make integration-single-isolate # single-node + V8-isolate (workerd); UC-103/104/105
make integration-local # local-mode (--local install on a throwaway box)
integration-tests/run.sh single-node --keep # leave infra up for debugging
integration-tests/run.sh single-node --prod-tls # use real Let's Encrypt instead of staging
make integration-cluster-hetero # 8-node heterogeneous cluster (x86 fc)
make integration-cluster-mixed-fc # 3× mixed cluster, FC on seed c5.metal
make integration-cluster-mixed-gvisor # 3× mixed cluster, gVisor on every node
make integration-cluster-mixed-wasm # 3× mixed cluster, WASM workers
make integration-single-fc # single-node-fc (x86 firecracker, c5.metal)
make integration-arm64 # arm64 firecracker single + cluster scenarios
make integration-arm64-single # single-node-fc-arm64 (Graviton metal)
make integration-arm64-cluster # cluster-arm64 (homogeneous arm64)
make integration-reap # terminate any leaked itest instances past ttl
Reports land in integration-tests/reports/ (<scenario>.md, <scenario>.json,
index.md matrix). Legend: ✅ pass · ❌ fail · ⚪ skip(n/a) · 🟡 pending · 🟤 inconclusive.
Agent eval
agenteval/ checks that Claude can use the aerolvm mcp tools, against a
local sandboxd and the Anthropic API rather than AWS. Run it before changing
tool names, descriptions or notices; see agenteval/README.md.
Where the binaries come from
Scenarios build the daemon locally by default. Every run cross-compiles the
current working tree, uploads it to a per-account artifacts bucket, and hands
the nodes presigned URLs. This is what lets the harness test an unmerged
branch — before this, a scenario could only provision from releases/latest,
so a branch was testable only after it merged and released.
One-time per operator/account:
make itest-artifacts-init # creates s3://aerol-itest-artifacts-<account>
Then the default path needs nothing extra:
make integration-single # builds HEAD, publishes, provisions from it
make integration-single released # old behaviour: releases/latest
make integration-single no-build # reuse the last published build (fast re-provision)
make integration-single FLAGS=--version=v0.7.21 # pin a released tag (A/B vs known-good)
The build id is content-addressed —
AEROL_BUILD_ID = <short-sha>[-dirty-<tree-hash>] — so an unchanged tree skips
both the compile and the upload. That is what makes iterating against a
--keep cluster free. A tree with uncommitted or untracked changes gets a
-dirty- id, and the report says so: those numbers are not reproducible from
git.
Working on the build itself:
make itest-build # build only
make itest-build BUILD_FLAGS="--ref main" # build another commit (detached worktree)
make itest-publish # upload + print presigned tfvars on stdout
Artifacts expire from the bucket after 7 days; presigned URLs default to 12h,
which covers a slow *.metal provision. reports/<scenario>.json carries a
build block naming the tree that produced its numbers.
Container engine
containerd is the default engine for every scenario. run.sh writes
SB_CONTAINER_ENGINE=containerd into the config overlay unless a scenario
advertises the docker-engine capability, in which case it uses dockerd. Only
two scenarios opt out:
local-mode— the local dev install always uses docker.cluster-3-mixed-docker— the docker A/B benchmark baseline, kept on docker so itscluster-3-mixed-docker-*artifacts stay a valid comparison against the containerd engine measured bycluster-3-mixed-containerd.
This encodes the target end-state of the docker→containerd migration: the local
installation uses docker; every real deployment (single-node, mixed cluster,
hetero cluster) uses containerd. The containerd-engine capability no longer
selects the engine — it opts a scenario into the containerd-specific soak UCs
(UC-99..102) and the distinctly-labeled containerd benchmark row.
Because containerd's CNI container networking is still ship-dark
(SB_CONTAINERD_NATIVE_NETNS_POOL_ENABLED, plans/containerd-engine.md §8), the
containerd path has been live-validated on t3 topologies (cluster-3-mixed-containerd);
the metal (*-fc) and arm64 scenarios request containerd on hardware where the
containerd bootstrap has not yet been live-proven — validate those on a --keep
run before relying on them.
Persistent cert store (avoid Let's Encrypt re-issuance)
Every domain scenario provisions a fresh box whose Caddy issues a *.<domain>
wildcard via DNS-01. Because the box is thrown away each run, that cert used to
be re-issued every run — the managed cert bucket is created and destroyed
with each cluster, and single-node disabled sharing entirely. Against
--prod-tls (or if staging limits bite) that burns the issuance budget.
Run this once per operator/account to back the harness with a persistent, cross-run cert store:
make integration-cert-store-init
It idempotently creates a long-lived S3 bucket (aerol-itest-caddy-certs-<acct>,
versioning + AES256 + public-access-block) that lives outside all scenario
Terraform state — so per-scenario terraform destroy never wipes it — and
writes a stable caddy_cert_store block (bucket coords + encryption key) into
config/secrets.yml. From then on run.sh points every domain scenario at
that bucket in BYO mode.
Behaviour after bootstrap: domains are still randomly leased from the pool
(scenarios/domains.yml), but certmagic-s3 keys certs by
<prefix>/certificates/<acme-issuer>/<domain>, so the first run to land on a
domain issues + stores its wildcard; every later run that draws the same domain
reads it back instead of re-issuing. The pool is finite, so each domain issues
exactly once per cert lifetime. Staging and --prod-tls runs never collide
(different issuer directory). The encryption key is the only way to read the
stored certs — save it like any root credential; losing or rotating it orphans
every stored cert. See setup/multi-node-cert-sharing.md.
Egress domain filtering (UC-179..UC-204)
suite/egress_fqdn_test.go, suite/egress_phase2_test.go,
suite/egress_phase3_test.go, suite/egress_firecracker_test.go and
suite/egress_private_cloud_test.go cover hostname egress filtering
(plans/egress-domain-filtering.md P1-10, P1-20, Phases 2-4).
egress-fqdn(advertised bysingle-node,single-node-containerd,cluster-3-mixed-dockerandcluster-3-mixed-gvisor): UC-180..182 run real traffic through the node's egress gateway (allowed name over HTTPS, NXDOMAIN and a fast refusal for others, the 403 naming host and rule, the denial insandbox.audit(),host:portand*.entries). UC-185 is the EF-78 latency gate: withAEROL_BENCH=1it reads thesvc_egress_*stages from Server-Timing and fails over the join (≤2 ms p50) and attach (≤10 ms p99) budgets (AEROL_EGRESS_JOIN_P50_MS/AEROL_EGRESS_ATTACH_P99_MSoverride).- Phase 2, also on
egress-fqdn: UC-190 changes a running sandbox's policy, UC-191 converges a named profile change and refuses deleting it while referenced, and UC-192 (EF-48) learnspip installin learn mode, then locks to the suggested list. UC-193..200 (EF-49) run each built-in profile's real tool with only that profile: pip, npm,git clone, a Hugging Face download,go mod download,cargo fetch, apt andcrane pull(in place ofdocker pull, which would need Docker in Docker). They pull the tool images (python, node, alpine/git, golang, rust, ubuntu, crane) and reach the public registries, so a failure there may be registry drift: the built-in lists are kept by hand and these UCs are how operators catch a stale one (CEO D12). - Phase 3, also on
egress-fqdn: UC-201 (EF-52, EF-53) creates apython:3.12-alpinesandbox allowedapi.github.comwith an inspect rule forGET /repos/**. Python'surllib, trusting the node CA only throughSSL_CERT_FILE, gets a 200 for the ruled GET through the gateway's TLS termination, and the gateway's own 403 (namingnetwork_egress_rules) for a POST and for an unruled path. It calls the public GitHub API unauthenticated, so a 403 on the GET may be GitHub's rate limit; the failure prints the body. UC-202 (EF-54, P3-2) creates apython:3.12-alpinesandbox with a randomTEST_TOKENin its env and an inspect rule onpostman-echo.comfor/headersthat injectsAuthorizationfromenv:TEST_TOKEN. Inside, the env holds onlyaerolvm-placeholder:TEST_TOKEN;urllibsends that placeholder asAuthorization, and the echoed headers must show the real token the gateway put in its place. It depends on the publicpostman-echo.comservice being up. UC-203 (EF-55, P3-3) creates apython:3.12-alpinesandbox, pinned to thedockerruntime (runc under either engine, never runsc, which refusesbinarieswith 501), allowedpypi.organdfiles.pythonhosted.orgwith a binaries-only rule on 443 for each naming/usr/local/bin/pip. Inside,pip install requestsmust pass without inspection (the gateway names pip by its script), and busyboxwgettohttps://pypi.org/simple/must be refused at the TLS handshake. The node needs cgroup v2, which the Ubuntu 22.04 AMIs have; on cgroup v1 every covered connection is refused (binary_unknown). - Phase 4, on
single-node-fc(firecracker+egress-fqdn): UC-204 creates a Firecracker guest allowedpypi.org. Its traffic reaches the egress gateway over its own TAP: the listed name fetches over HTTPS, another is refused fast,egress_statusisactive, and a live switch to block-all shuts it. That scenario also runs the docker egress UCs above. - UC-179 (CIDR allowlist), UC-183 (no
CAP_NET_RAW) and UC-184 (block-all can't reach host services) only needdocker. private-cloud(single-node-private-cloud,make integration-single-private-cloud): the node runs dnsmasq with internal names, an internal HTTP service and squid, wired into the egress operator file. UC-186..189 cover the upstream proxy with synthetic DNS, the internal zone (and its rebinding refusal), the deny floor and the control-port guard. The node keeps its internet gateway; a no-egress VPC with an offline install is the remaining part of P1-20.
Create benchmark (UC-94 / UC-95)
suite/benchmark_test.go is an opt-in benchmark that reuses the live
deployment to measure sandbox creation. It only runs where the scenario
advertises the benchmark capability — cluster-hetero (every runtime),
cluster-3-mixed-docker (docker only), cluster-3-mixed-fc (docker +
firecracker), cluster-3-mixed-gvisor (docker + gvisor),
cluster-3-mixed-wasm (docker + wasm), and single-node-isolate
(isolate) — because it is slow and provisions many sandboxes. Everywhere else
UC-94/UC-95 skip (not-applicable).
- UC-94 — create latency: for each runtime the scenario has (docker,
firecracker, gvisor, wasm, isolate) it creates
AEROL_BENCH_SAMPLESsandboxes serially, reporting p50/p90/p99 + mean for three timings: api (theCreate()client round-trip), server (the API's ownServer-Timingheader for the create — client↔cluster network excluded), and running (create→started).api − serveris that network overhead, so you can tell a slow create from a slow link. For isolate, the bench uploads one JS bundle and pins a sharedtenant_idso samples hit the warm group path. - UC-95 — fleet density: creates docker sandboxes until the fleet rejects on
capacity (HTTP 503
capacity exceeded), reports how many landed, then tears them all down.AEROL_BENCH_MAXbounds cost if admission never trips.
Every result is stamped with the machine configuration parsed from the
scenario's *.tfvars (per-node instance_type, role, with_* flags,
default_instance_type) so a latency number is read against the hardware it ran
on (e.g. firecracker on c5.metal vs docker on t3.medium).
Even on cluster-hetero the benchmark is dormant by default — it's a second
gate, AEROL_BENCH=1, on top of the capability, so a normal hetero run doesn't
pay the cost (UC-94/UC-95 show as ⚪ skip with that reason). Turn it on and it
runs inside the normal orchestrated suite, which provisions and tears down for
you (run.sh passes the parent environment through to go test):
# Full hetero run (docker + firecracker + gvisor + wasm):
make integration-benchmark
# Docker-focused 3× mixed cluster (full suite + docker bench; writes
# cluster-3-mixed-docker.json/.md and cluster-3-mixed-docker-bench.json):
make integration-benchmark-docker
# Firecracker-focused 3× mixed cluster (docker + firecracker only; cheaper):
make integration-benchmark-fc
# WASM-focused 3× mixed cluster (wasm latency benchmark only; no Docker
# sandbox creates). Provisions cluster-3-mixed-wasm, runs UC-94 with
# AEROL_BENCH_RUNTIMES=wasm (UC-95 skips). Full integration coverage
# (docker + wasm UCs) is `make integration-cluster-mixed-wasm`.
make integration-benchmark-wasm
# gVisor-focused 3× mixed cluster (docker + gvisor only; cheap t3 spot boxes):
make integration-benchmark-gvisor
# V8-isolate (workerd) create-latency on single-node-isolate (t3.medium):
# AEROL_BENCH_RUNTIMES=isolate; UC-95 skipped (no CapCluster).
make integration-benchmark-isolate
# Manual env form (hetero):
AEROL_BENCH=1 \
AEROL_BENCH_OUT=integration-tests/reports/cluster-hetero-bench.json \
make integration-cluster-hetero
To iterate without re-provisioning, bring the cluster up once with keep, then
re-run only UC-94/UC-95 with make integration-benchmark-only (hetero) or
make integration-benchmark-fc-only (mixed-fc) — they read the API URL + PAT
from TF state and force AEROL_BENCH=1, so nothing to export:
make integration-cluster-hetero keep # provision, leave it up
make integration-benchmark-only # re-run just the bench against it
make integration-cluster-mixed-docker keep # 3× mixed docker benchmark cluster
make integration-benchmark-docker-only # re-run docker bench against it
make integration-cluster-mixed-fc keep # 3× mixed FC cluster
make integration-benchmark-fc-only # re-run UC-94/UC-95 against it
make integration-cluster-mixed-wasm keep # 3× mixed WASM cluster (full suite)
make integration-benchmark-wasm-only # re-run wasm bench against kept cluster
make integration-cluster-mixed-gvisor keep # 3× mixed gVisor cluster
make integration-benchmark-gvisor-only # re-run UC-94/UC-95 against it
make integration-single-isolate keep # single-node isolate (functional UCs)
make integration-benchmark-isolate-only # re-run UC-94 isolate latency
# Narrow the UC-94 sweep on hetero / mixed-fc / mixed-gvisor benches:
AEROL_BENCH_RUNTIMES=docker,gvisor,wasm make integration-benchmark-only
# …or firecracker-only latency (UC-95 skips when docker is excluded):
AEROL_BENCH_RUNTIMES=firecracker make integration-benchmark-fc-only
# …or gvisor-only latency (UC-95 skips when docker is excluded):
AEROL_BENCH_RUNTIMES=gvisor make integration-benchmark-gvisor-only
# …or add docker density/latency back on the wasm cluster:
AEROL_BENCH_RUNTIMES=docker,wasm make integration-benchmark-wasm-only
make integration-destroy # tear the kept cluster down
make integration-destroy SCENARIO=cluster-3-mixed-docker
make integration-destroy SCENARIO=cluster-3-mixed-fc
make integration-destroy SCENARIO=cluster-3-mixed-gvisor
make integration-destroy SCENARIO=cluster-3-mixed-wasm
make integration-destroy SCENARIO=single-node-isolate
Percentiles + the density ceiling also print as bench[...] log lines (captured
by go test -json); the AEROL_BENCH_OUT JSON adds the machine block.
Docker benchmark path. make integration-benchmark-docker provisions
cluster-3-mixed-docker, runs the full integration suite with
AEROL_BENCH_RUNTIMES=docker (same shape as gvisor/fc benchmarks), and writes:
integration-tests/reports/cluster-3-mixed-docker.json+.md— full UC matrixintegration-tests/reports/cluster-3-mixed-docker-bench.json+-bench.md— UC-94/UC-95 timings
Re-run only the bench against a kept cluster with make integration-benchmark-docker-only.
Use make integration-cluster-mixed-docker when you want the full suite without
the benchmark (UC-94/UC-95 skip unless AEROL_BENCH=1).
Failure logs (cluster-3-mixed-docker-failure-logs.txt) are collected only when
a run fails or infra is inconclusive — not on a green pass (same as every other
scenario). Your bench run passed UC-94/UC-95, so no failure-log file was expected.
To pull daemon logs from a kept cluster any time:
make integration-collect-logs SCENARIO=cluster-3-mixed-docker
WASM warm pool. wasm scenarios boot with the warm-worker pool on so creates
skip the cold module compile — CPython-on-wasm is ~10s cold on a t3.
make integration-benchmark-wasm provisions cluster-3-mixed-wasm and runs
wasm-only benchmarks (--bench-only, AEROL_BENCH_RUNTIMES=wasm): UC-94
times wasm creates, UC-95 skips, and no full-suite Docker sandboxes are created.
Use make integration-cluster-mixed-wasm for the full docker+wasm integration
suite. Pool depth defaults to
2 by default at provision time so UC-94 measures warm hits without exhausting
small nodes; make integration-cluster-mixed-wasm (non-benchmark) also keeps
depth 2.
Depth is baked into node boot env, so a cluster brought up before a pool
depth change shows cold numbers until re-provisioned.
Firecracker benchmark path. UC-94 creates one Firecracker template before
timing and measures sandbox clones with template_id, not plain OCI image cold
boots. Override the template source with AEROL_BENCH_FC_TEMPLATE_IMAGE when
you need a different base image.
| Env var | Default | Purpose |
|---|---|---|
AEROL_BENCH |
(unset) | master switch — must be 1 or the bench skips |
AEROL_BENCH_RUNTIMES |
(all advertised) | comma list narrowing the UC-94 sweep, e.g. docker,gvisor,wasm (skip firecracker) or firecracker (isolate) |
AEROL_BENCH_SAMPLES |
10 |
sandboxes timed per runtime (UC-94) |
AEROL_BENCH_MAX |
200 |
safety cap on the density probe (UC-95) |
AEROL_BENCH_OUT |
(unset) | path to write the JSON artifact; logs only if unset |
AEROL_BENCH_TFVARS |
../scenarios/<scenario>.tfvars |
override the machine-config source |
AEROL_BENCH_FC_TEMPLATE_IMAGE |
docker://alpine:3.20 |
Firecracker template image used for UC-94 clone latency |
AEROL_WASM_MODULE_REF |
(unset) | staged .wasm ref; wasm latency skips without it |
AEROL_WASM_POOL_DEPTH |
2 |
warm wasm workers per module digest (provision-time; 0 when wasm off) |
What runs where
| Layer | Command | Needs AWS? |
|---|---|---|
| Harness self-tests (caps logic, report classifier, prod-safety tripwires) | make test (go test ./...) |
No — offline, in CI |
| Live integration suite | make integration-* / go test -tags=integration ./integration-tests/suite/... |
Yes |
The live suite is behind the integration build tag, so the everyday
make test never reaches AWS.
Safety model
- State:
-backend-config=key=integration/<scenario>/...+ separateTF_DATA_DIR. The prod key (prod/terraform.tfstate) is never opened. - Tripwires (
lib/provision.sh check-safety, unit-tested insafety/): abort before any apply if the state key targets prod, the leased domain collides with the prod domain, orcluster_namelacks theitestmarker. - Config overlay: generated at runtime from
config/cluster.ymlwith AOCR auto-import + fleet disabled and the leased domain substituted — never a committed copy, so it can't drift. - Teardown:
run.shtrap on EXIT/INT/TERM + the standalone reaper.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package agenteval checks that a model can actually use the aerolvm MCP tools (plans/mcp-server-and-agent-cli.md §8, eng review D13).
|
Package agenteval checks that a model can actually use the aerolvm MCP tools (plans/mcp-server-and-agent-cli.md §8, eng review D13). |
|
Package catalogue holds shared helpers for the investor benchmark catalogue artifact: cross-process atomic JSON merge (soak loops) and Pushgateway push.
|
Package catalogue holds shared helpers for the investor benchmark catalogue artifact: cross-process atomic JSON merge (soak loops) and Pushgateway push. |
|
cmd/gen
command
Command gen renders reports/<scenario>-catalogue.md from catalogue JSON.
|
Command gen renders reports/<scenario>-catalogue.md from catalogue JSON. |
|
cmd
|
|
|
audit-receiver
command
audit-receiver is the off-node sink the security scenarios export to (plans/integration-test-security.md §6.4).
|
audit-receiver is the off-node sink the security scenarios export to (plans/integration-test-security.md §6.4). |
|
Command gen turns `go test -json` output into the use-case coverage report: a per-scenario JSON + Markdown file and an aggregate matrix.
|
Command gen turns `go test -json` output into the use-case coverage report: a per-scenario JSON + Markdown file and an aggregate matrix. |
|
harness
Package harness holds the scenario/capability model, the use-case registry, and the thin SDK client wrapper shared by every integration test.
|
Package harness holds the scenario/capability model, the use-case registry, and the thin SDK client wrapper shared by every integration test. |