e2e/

e2e scripts

Kind harnesses for the mesh. Each script documents its own legs, knobs and gates in its header. The long-running soak and the collector-pressure test (the on-demand regression test for #662: a collector that refuses telemetry must not block or kill an agent's start) are run by an external harness, maintained outside this repository.

So are six kind reproductions of fixed incidents that no workflow here ran: agent-restart-gap.sh (#1123, proposal 041), drain-propagation.sh (#1103, #1124), eastwest-quic-deadpeer.sh (#1087, #1104), eastwest-quic-hotrestart.sh (#1009, #1054), hotrestart-wedge.sh (#1050) and proxy-concurrency-change.sh (#1136). They left this directory on 2026-10-10 and are maintained with that external harness, where they install the published chart instead of building this tree. The runbook still names each beside the incident it reproduces.

What is here is what something runs: the suites of the nightly e2e workflow and the version pins they source, with their consistency tests. No script here is run by hand only: //e2e:suite_wiring_test fails on a harness that creates a kind cluster and that no job of .github/workflows/e2e.yaml runs.

What runs nightly

.github/workflows/e2e.yaml, at 04:17 UTC. One build job builds the images once; every suite is its own job on its own runner, against those images.

Run one on a branch: gh workflow run e2e.yaml --ref <branch> -f only=<job>. A scheduled run that fails opens or updates one issue (report-failure).

Running a suite locally

Every harness takes the same verbs:

e2e/<suite>.sh up       # build + load the images, create the cluster(s), install, deploy
e2e/<suite>.sh verify   # the assertions (re-runnable)
e2e/<suite>.sh down     # delete the cluster(s) and any etcd container
e2e/<suite>.sh          # up + verify

It needs kind at the release kind-version.sh pins (an older one is refused), docker, kubectl, helm and bazel. up builds the images with Bazel; set the suite's *_SKIP_BUILD=1 (named in its header) when they are already in the local Docker daemon. A harness creates its own cluster and talks only to it (--context kind-<name>), but kind writes that context into your kubeconfig and makes it current, and deleting the cluster leaves no current context: point KUBECONFIG at a scratch file if you care which context you are on afterwards.

A suite's header says what each assertion proves and how to see it fail. Two have a red half that CI does not run:

• MP_MIXED_PORTS=8080 e2e/multiprotocol.sh up && e2e/multiprotocol.sh verify: the mixed workload without its =tcp port. M4 fails, and with it the other legs that need port 9000 of that workload to be raw TCP.
• CONFIG_IMPORT=off e2e/multicluster_config.sh up && e2e/multicluster_config.sh verify: the importing cluster without agent.importConfig. C2 fails.

eastwest-quic.sh has one too (QUIC_DNS_SANS=off).

multiprotocol.sh runs on either registry backend (MP_REGISTRY_BACKEND=etcd, the default, or kubernetes, which starts no etcd; CI runs both) and has three switches for a red run. A failed assertion normally ends the run and prints the cluster's state; MP_KEEP_GOING=1 e2e/multiprotocol.sh verify runs every leg, prints a LEG RESULT DETAIL table and exits non-zero if any failed. MP_LEGS="A5 A7" runs a subset, and MP_TIMEOUT=<seconds> caps how long a leg waits for an answer to converge, which is what each failing assertion costs.

Some of its output is a record and not an assertion, on lines that start with a fixed word so they can be grepped out of a log: W2-OBSERVED (what a client sees on a port a raw-TCP service does not declare: curl's exit code, the total time and the time to connect), P1-OBSERVED (what curl --proxy gets for a URL that spells :80, which curl sends without the port), P5-OBSERVED (the answer to a CONNECT through the proxy port; leg P5 is INFO in the table, never PASS or FAIL) and MP-CELL (below).

Version skew with multiprotocol.sh

The chart is the same on both sides of a skew, so a skew cell is a mix of image tags under one chart. multiprotocol.sh installs two groups of images, each at its own tag (default latest):

To make the older side, build the images from the older commit and give them a tag of their own; <registry> is what scripts/image-registry.sh prefix prints:

git worktree add ../aether-2.6 <the 2.6 commit>
(cd ../aether-2.6 && for t in //agent/cmd/agent //agent/cmd/mesh-dns //agent/cmd/proxy-supervisor \
    //cni/cmd/cni-install //agent/cmd/uds-csi //registrar/cmd/registrar //controller/cmd/controller; do
  bazel run "$t:image_load"; done)
for img in agent mesh-dns proxy-supervisor cni-install uds-csi registrar controller; do
  docker tag <registry>/$img:latest <registry>/$img:2.6
done

Then build the branch (which moves latest to it) and run each cell on a cluster of its own. up loads exactly the tags in use and builds only the images that run at latest; with MP_SKIP_BUILD=1 it builds nothing, which is what a cell wants once both sets are in the Docker daemon:

# the baseline: everything at 2.6
MP_SKIP_BUILD=1 MP_TAG_AGENT=2.6 MP_TAG_REGISTRAR=2.6 e2e/multiprotocol.sh up
MP_KEEP_GOING=1 e2e/multiprotocol.sh verify | tee all-2.6.log; e2e/multiprotocol.sh down

# a cell: new agents under an old registrar and controller
MP_SKIP_BUILD=1 MP_TAG_REGISTRAR=2.6 e2e/multiprotocol.sh up
MP_KEEP_GOING=1 e2e/multiprotocol.sh verify | tee agent-new.log; e2e/multiprotocol.sh down

Run a cell with MP_KEEP_GOING=1 and compare its LEG RESULT DETAIL table with the all-2.6 run, leg by leg: the legs a newer chart adds or changes fail on 2.6, so a cell is read against that baseline and not against a green table. verify reads the tags back from the installed workloads and prints them first, on two lines that start with MP-CELL, so each log names its own cell whatever the environment of that verify was. The workloads and the suite are the branch's in every cell. CI runs no skew cell.

Adding a suite

A harness with the verbs above, its cluster named after the suite, sourcing the pins. Then, in .github/workflows/e2e.yaml: a job whose body is .github/actions/run-e2e-script, with needs: build, the if: its siblings have and a timeout-minutes; its name in the only choices, in report-failure's needs and in its RESULTS; and its images in the build job if it needs one the others do not. //e2e:suite_wiring_test holds all of it, and holds every helm command on the aether chart to an explicit --set clusterName=<name> (no default of the chart can be the name of your cluster; a multi-cluster suite names each one differently).

The pins

Every harness that creates a kind cluster sources kind-version.sh and passes --image "$KIND_NODE_IMAGE", so a local run and CI run the same Kubernetes; bumping it is one file (runbook). //e2e:kind_pin_test enforces both.

Two more pins live beside it, each with its test and its runbook section: gateway-api-version.sh, the Gateway API release whose CRDs a harness installs (source it; never assign GWAPI_VERSION in a script, //e2e:gateway_api_pin_test), and helm-version.sh, the Helm releases CI installs (//e2e:helm_pin_test). A harness runs whatever helm is first on PATH, so write it for Helm 3 and Helm 4 alike: helm list -a exists only in Helm 3, and helm_list_all_flags gives the flags for the one in use.

Rules for script authors

No early-exit reader in a pipeline (SIGPIPE under pipefail)

Every script here runs under set -euo pipefail. A pipeline whose reader exits before its writer is done kills the writer with SIGPIPE, and pipefail turns that into a pipeline status of 141. Under set -e the script then stops dead, with no failed assertion and no message, at a point that depends on timing and on how much the writer had left to print (#1121: eastwest-quic.sh verify died this way at random points). Inside an if/until/&& it is worse: the condition silently reads as false.

Readers that exit early, and their read-to-EOF replacements:

When the writer is a kubectl or curl (or an Envoy admin dump) whose output can be large, capture it first and test the variable — a here-string has no writer process to kill:

grep -q "inbound_${ns}_${pod}" <<<"$(admin /listeners 2>/dev/null)"

grep -q, head and awk … exit are fine on a file argument or a here-string; the rule is only about a process on the left of the |. Scripts that run inside a pod or a kind node (kubectl exec … sh -c '…', docker exec … sh -c '…') run without pipefail and are not affected.

Evidence never fails on "no match" (grep exits 1 under set -e)

grep exits 1 when nothing matches, and under pipefail that is the pipeline's status even when a sed or cut follows it. A forensics, summary or "print the evidence" pipeline without a guard therefore aborts the script exactly when the evidence it looks for is absent (#1143). Such a pipeline ends in || true, or filters with awk (which exits 0 on no match):

sed -n '1,200p' "$f" | grep -aE 'sendmsg|recvmsg' | sed -n '1,40p' || true

Gates are the opposite: a pipeline whose failure should stop the run belongs in an if or ends in || die "…", so it fails with a message, not silently.

A captured gate value never aborts before its check (x="$(…)" under set -e)

An assignment from a command substitution takes the substitution's exit status, and under set -e a non-zero one aborts the script on the assignment — so when a gate captures a value and checks it on the next line, a producer that exits non-zero exactly when the thing is absent (grep with no match, stat/cat of a missing path, curl that cannot connect, a kubectl exec wrapping any of them) stops the run before the die that names the cause can print (#1150). The gate still fails, but with no message, at the wrong line. End the substitution in || true and let the check carry the gate:

# Don't: no `/s` mount → grep exits 1 → silent exit here, the die never runs.
mnt="$(in_pod "$pod" "grep ' /s ' /proc/mounts")"
# Do:
mnt="$(in_pod "$pod" "grep ' /s ' /proc/mounts" || true)"
case "$mnt" in tmpfs\ /s\ tmpfs\ *) ;; *) die "/s in $pod is not a tmpfs mount: '$mnt'" ;; esac

The same holds for a helper function captured as x="$(helper)": guard the call (x="$(helper || true)") or end the helper's pipeline in || true. The check after the capture must then reject the empty string, which an equality or case test against the expected value, or [ -n "$x" ] || die …, already does.