integration-tests/

Integration Test Harness

Stands up real AerolVM deployments from the released install artifacts and drives the live /v1 API + Go SDK against them, producing a use-case coverage matrix. Full design + decisions: ../plans/integration-tests.md.

⚠️ These tests provision real AWS EC2 and cost money. They use the production Terraform module but against isolated state (integration/<scenario> key + separate TF_DATA_DIR) so production state is never touched. Spot instances + auto-teardown + make integration-reap keep cost down.

Status

Live scenarios cover single-node (docker / containerd / fc / wasm / isolate), 3× mixed clusters (docker, containerd, gvisor, wasm, fc), hetero, local-mode, and arm64. Reports and the coverage matrix live in reports/. Isolate is exercised by make integration-single-isolate (UC-103/104/105) and make integration-benchmark-isolate (UC-94).

Prerequisites

• terraform, go, awscli, jq, yq, curl, openssl, ssh
• AWS credentials with permission to create EC2/VPC/etc.
• config/secrets.yml populated (PAT, Cloudflare token) — see config/secrets.example.yml
• integration-tests/scenarios/domains.yml — copy from domains.example.yml, fill in real test FQDNs you control (distinct from your prod domain)

Run

make integration-single                 # single-node scenario
make integration-single-isolate         # single-node + V8-isolate (workerd); UC-103/104/105
make integration-local                  # local-mode (--local install on a throwaway box)
integration-tests/run.sh single-node --keep        # leave infra up for debugging
integration-tests/run.sh single-node --prod-tls    # use real Let's Encrypt instead of staging
make integration-cluster-hetero           # 8-node heterogeneous cluster (x86 fc)
make integration-cluster-mixed-fc         # 3× mixed cluster, FC on seed c5.metal
make integration-cluster-mixed-gvisor       # 3× mixed cluster, gVisor on every node
make integration-cluster-mixed-wasm         # 3× mixed cluster, WASM workers
make integration-single-fc                # single-node-fc (x86 firecracker, c5.metal)
make integration-arm64                    # arm64 firecracker single + cluster scenarios
make integration-arm64-single             # single-node-fc-arm64 (Graviton metal)
make integration-arm64-cluster            # cluster-arm64 (homogeneous arm64)
make integration-reap                   # terminate any leaked itest instances past ttl

Reports land in integration-tests/reports/ (<scenario>.md, <scenario>.json, index.md matrix). Legend: ✅ pass · ❌ fail · ⚪ skip(n/a) · 🟡 pending · 🟤 inconclusive.

Container engine

containerd is the default engine for every scenario. run.sh writes SB_CONTAINER_ENGINE=containerd into the config overlay unless a scenario advertises the docker-engine capability, in which case it uses dockerd. Only two scenarios opt out:

• local-mode — the local dev install always uses docker.
• cluster-3-mixed-docker — the docker A/B benchmark baseline, kept on docker so its cluster-3-mixed-docker-* artifacts stay a valid comparison against the containerd engine measured by cluster-3-mixed-containerd.

This encodes the target end-state of the docker→containerd migration: the local installation uses docker; every real deployment (single-node, mixed cluster, hetero cluster) uses containerd. The containerd-engine capability no longer selects the engine — it opts a scenario into the containerd-specific soak UCs (UC-99..102) and the distinctly-labeled containerd benchmark row.

Because containerd's CNI container networking is still ship-dark (SB_CONTAINERD_NATIVE_NETNS_POOL_ENABLED, plans/containerd-engine.md §8), the containerd path has been live-validated on t3 topologies (cluster-3-mixed-containerd); the metal (*-fc) and arm64 scenarios request containerd on hardware where the containerd bootstrap has not yet been live-proven — validate those on a --keep run before relying on them.

Persistent cert store (avoid Let's Encrypt re-issuance)

Every domain scenario provisions a fresh box whose Caddy issues a *.<domain> wildcard via DNS-01. Because the box is thrown away each run, that cert used to be re-issued every run — the managed cert bucket is created and destroyed with each cluster, and single-node disabled sharing entirely. Against --prod-tls (or if staging limits bite) that burns the issuance budget.

Run this once per operator/account to back the harness with a persistent, cross-run cert store:

make integration-cert-store-init

It idempotently creates a long-lived S3 bucket (aerol-itest-caddy-certs-<acct>, versioning + AES256 + public-access-block) that lives outside all scenario Terraform state — so per-scenario terraform destroy never wipes it — and writes a stable caddy_cert_store block (bucket coords + encryption key) into config/secrets.yml. From then on run.sh points every domain scenario at that bucket in BYO mode.

Behaviour after bootstrap: domains are still randomly leased from the pool (scenarios/domains.yml), but certmagic-s3 keys certs by <prefix>/certificates/<acme-issuer>/<domain>, so the first run to land on a domain issues + stores its wildcard; every later run that draws the same domain reads it back instead of re-issuing. The pool is finite, so each domain issues exactly once per cert lifetime. Staging and --prod-tls runs never collide (different issuer directory). The encryption key is the only way to read the stored certs — save it like any root credential; losing or rotating it orphans every stored cert. See setup/multi-node-cert-sharing.md.

Create benchmark (UC-94 / UC-95)

suite/benchmark_test.go is an opt-in benchmark that reuses the live deployment to measure sandbox creation. It only runs where the scenario advertises the benchmark capability — cluster-hetero (every runtime), cluster-3-mixed-docker (docker only), cluster-3-mixed-fc (docker + firecracker), cluster-3-mixed-gvisor (docker + gvisor), cluster-3-mixed-wasm (docker + wasm), and single-node-isolate (isolate) — because it is slow and provisions many sandboxes. Everywhere else UC-94/UC-95 skip (not-applicable).

• UC-94 — create latency: for each runtime the scenario has (docker, firecracker, gvisor, wasm, isolate) it creates AEROL_BENCH_SAMPLES sandboxes serially, reporting p50/p90/p99 + mean for three timings: api (the Create() client round-trip), server (the API's own Server-Timing header for the create — client↔cluster network excluded), and running (create→started). api − server is that network overhead, so you can tell a slow create from a slow link. For isolate, the bench uploads one JS bundle and pins a shared tenant_id so samples hit the warm group path.
• UC-95 — fleet density: creates docker sandboxes until the fleet rejects on capacity (HTTP 503 capacity exceeded), reports how many landed, then tears them all down. AEROL_BENCH_MAX bounds cost if admission never trips.

Every result is stamped with the machine configuration parsed from the scenario's *.tfvars (per-node instance_type, role, with_* flags, default_instance_type) so a latency number is read against the hardware it ran on (e.g. firecracker on c5.metal vs docker on t3.medium).

Even on cluster-hetero the benchmark is dormant by default — it's a second gate, AEROL_BENCH=1, on top of the capability, so a normal hetero run doesn't pay the cost (UC-94/UC-95 show as ⚪ skip with that reason). Turn it on and it runs inside the normal orchestrated suite, which provisions and tears down for you (run.sh passes the parent environment through to go test):

# Full hetero run (docker + firecracker + gvisor + wasm):
make integration-benchmark

# Docker-focused 3× mixed cluster (full suite + docker bench; writes
# cluster-3-mixed-docker.json/.md and cluster-3-mixed-docker-bench.json):
make integration-benchmark-docker

# Firecracker-focused 3× mixed cluster (docker + firecracker only; cheaper):
make integration-benchmark-fc

# WASM-focused 3× mixed cluster (wasm latency benchmark only; no Docker
# sandbox creates). Provisions cluster-3-mixed-wasm, runs UC-94 with
# AEROL_BENCH_RUNTIMES=wasm (UC-95 skips). Full integration coverage
# (docker + wasm UCs) is `make integration-cluster-mixed-wasm`.
make integration-benchmark-wasm

# gVisor-focused 3× mixed cluster (docker + gvisor only; cheap t3 spot boxes):
make integration-benchmark-gvisor

# V8-isolate (workerd) create-latency on single-node-isolate (t3.medium):
# AEROL_BENCH_RUNTIMES=isolate; UC-95 skipped (no CapCluster).
make integration-benchmark-isolate

# Manual env form (hetero):
AEROL_BENCH=1 \
AEROL_BENCH_OUT=integration-tests/reports/cluster-hetero-bench.json \
  make integration-cluster-hetero

To iterate without re-provisioning, bring the cluster up once with keep, then re-run only UC-94/UC-95 with make integration-benchmark-only (hetero) or make integration-benchmark-fc-only (mixed-fc) — they read the API URL + PAT from TF state and force AEROL_BENCH=1, so nothing to export:

make integration-cluster-hetero keep         # provision, leave it up
make integration-benchmark-only              # re-run just the bench against it

make integration-cluster-mixed-docker keep   # 3× mixed docker benchmark cluster
make integration-benchmark-docker-only       # re-run docker bench against it

make integration-cluster-mixed-fc keep       # 3× mixed FC cluster
make integration-benchmark-fc-only         # re-run UC-94/UC-95 against it

make integration-cluster-mixed-wasm keep   # 3× mixed WASM cluster (full suite)
make integration-benchmark-wasm-only       # re-run wasm bench against kept cluster

make integration-cluster-mixed-gvisor keep # 3× mixed gVisor cluster
make integration-benchmark-gvisor-only     # re-run UC-94/UC-95 against it

make integration-single-isolate keep       # single-node isolate (functional UCs)
make integration-benchmark-isolate-only    # re-run UC-94 isolate latency

# Narrow the UC-94 sweep on hetero / mixed-fc / mixed-gvisor benches:
AEROL_BENCH_RUNTIMES=docker,gvisor,wasm make integration-benchmark-only
# …or firecracker-only latency (UC-95 skips when docker is excluded):
AEROL_BENCH_RUNTIMES=firecracker make integration-benchmark-fc-only
# …or gvisor-only latency (UC-95 skips when docker is excluded):
AEROL_BENCH_RUNTIMES=gvisor make integration-benchmark-gvisor-only
# …or add docker density/latency back on the wasm cluster:
AEROL_BENCH_RUNTIMES=docker,wasm make integration-benchmark-wasm-only

make integration-destroy                     # tear the kept cluster down
make integration-destroy SCENARIO=cluster-3-mixed-docker
make integration-destroy SCENARIO=cluster-3-mixed-fc
make integration-destroy SCENARIO=cluster-3-mixed-gvisor
make integration-destroy SCENARIO=cluster-3-mixed-wasm
make integration-destroy SCENARIO=single-node-isolate

Percentiles + the density ceiling also print as bench[...] log lines (captured by go test -json); the AEROL_BENCH_OUT JSON adds the machine block.

Docker benchmark path. make integration-benchmark-docker provisions cluster-3-mixed-docker, runs the full integration suite with AEROL_BENCH_RUNTIMES=docker (same shape as gvisor/fc benchmarks), and writes:

• integration-tests/reports/cluster-3-mixed-docker.json + .md — full UC matrix
• integration-tests/reports/cluster-3-mixed-docker-bench.json + -bench.md — UC-94/UC-95 timings

Re-run only the bench against a kept cluster with make integration-benchmark-docker-only. Use make integration-cluster-mixed-docker when you want the full suite without the benchmark (UC-94/UC-95 skip unless AEROL_BENCH=1).

Failure logs (cluster-3-mixed-docker-failure-logs.txt) are collected only when a run fails or infra is inconclusive — not on a green pass (same as every other scenario). Your bench run passed UC-94/UC-95, so no failure-log file was expected. To pull daemon logs from a kept cluster any time:

make integration-collect-logs SCENARIO=cluster-3-mixed-docker

WASM warm pool. wasm scenarios boot with the warm-worker pool on so creates skip the cold module compile — CPython-on-wasm is ~10s cold on a t3. make integration-benchmark-wasm provisions cluster-3-mixed-wasm and runs wasm-only benchmarks (--bench-only, AEROL_BENCH_RUNTIMES=wasm): UC-94 times wasm creates, UC-95 skips, and no full-suite Docker sandboxes are created. Use make integration-cluster-mixed-wasm for the full docker+wasm integration suite. Pool depth defaults to 2 by default at provision time so UC-94 measures warm hits without exhausting small nodes; make integration-cluster-mixed-wasm (non-benchmark) also keeps depth 2. Depth is baked into node boot env, so a cluster brought up before a pool depth change shows cold numbers until re-provisioned.

Firecracker benchmark path. UC-94 creates one Firecracker template before timing and measures sandbox clones with template_id, not plain OCI image cold boots. Override the template source with AEROL_BENCH_FC_TEMPLATE_IMAGE when you need a different base image.

What runs where

The live suite is behind the integration build tag, so the everyday make test never reaches AWS.

Safety model

• State: -backend-config=key=integration/<scenario>/... + separate TF_DATA_DIR. The prod key (prod/terraform.tfstate) is never opened.
• Tripwires (lib/provision.sh check-safety, unit-tested in safety/): abort before any apply if the state key targets prod, the leased domain collides with the prod domain, or cluster_name lacks the itest marker.
• Config overlay: generated at runtime from config/cluster.yml with AOCR auto-import + fleet disabled and the leased domain substituted — never a committed copy, so it can't drift.
• Teardown: run.sh trap on EXIT/INT/TERM + the standalone reaper.