mutationlab/

Mutation-capture lab

A small, separate application that records the exact structures Kubernetes exposes through native watches, audit webhooks, and validating admission webhooks, and commits them as a versioned corpus. It is not a second GitOps Reverser — see the design.

Layout

Scenario index

Each captured row of the Difficult Cases Catalog maps to one driver test and one corpus directory. The "Sources" column is what the corpus commits — a silence (no audit / no admission) is itself the finding, not a gap (see Capturing Intent, Not State).

Row 8a is not a catalog row: it extends row 8 to the case the catalog could not express, because row 8 drives both phases with one identity. A finalized deletion in production has TWO actors — a human asks, a controller cleans up — and the row exists to measure what that does to attribution. The finding is in the last column: the human's delete and the controller's finalizer patch both return a body, and both bodies carry the resourceVersion the DELETION stamped, so the two facts they produce collide on one key. LAB_FINALIZER_HOLD tunes the gap between the phases, which is what decides whether both land in one audit batch. See the attribution spec.

All seventeen catalogued scenarios are now captured. Rows 15a and 15b are not catalog rows: they extend row 15 to the removal verbs, because the create alone could not say whether a proxied delete carries a uid (it does not) or whether a proxied deletecollection returns a response body (it does not). Row 18 is their control: a generateName create has an equally name-less objectRef and joins perfectly well, because it carries a response body to recover the name from, which is what makes the BODY rather than the name the thing the aggregated rows are actually missing. Rows 16 and 17 test the watch transport itself; the driver uses the lab's targeted /watch-probe endpoint so transport-only events can be scenario-attributed — see the watch-first ingestion architecture design notes.

⚠️ The "no audit" rows record this cluster's audit POLICY, not the API server. The lab cluster uses the e2e audit policy (test/e2e/cluster/audit/policy.yaml), which drops pods and every */status at level: None as runtime noise. So rows 5 and 7 show what a cluster configured like this one does not tell us — a DELETE on a pod is an audited request like any other when the policy asks for it. Read them as "excluded here", not as "unknowable". Re-capturing either row against a policy that includes those types would settle it by measurement.

How it integrates: a cluster of its own

The lab runs on its own k3d cluster, gitops-reverser-capture-lab, which runs the lab, the wardle sample aggregated API server, and nothing of the product (design). Every source arrives the way it does on any cluster:

• the API server's audit webhook backend posts to https://127.0.0.1:30444/audit-webhook, a NodePort on its own node, over mutual TLS: it presents a client certificate from the lab CA, and the lab refuses a connection without one;
• the ValidatingWebhookConfiguration mutation-capture-lab sends admission reviews to the lab's Service;
• the scenario Widget CRD's conversion webhook calls the same Service at /convert;
• the lab watches through its own ServiceAccount, with read-only RBAC on what it watches.

lab-certs issues the certificates before the cluster exists, so the audit kubeconfig the API server reads at startup is already the final one and the cluster never restarts. The cluster is created by the same start-cluster.sh as the e2e cluster, with the same audit flags and policy, on one node, and without switching the current kube context.

Nothing in the lab's wiring names a product resource or depends on a product task; selfcontained_test.go enforces that in task test. The lab used to swap its image into the product's Deployment and borrow its webhook registration and certificates, and lost its admission source the day the product removed them, with every check still green.

Row 15 (aggregated API write) is what settled the body-enrichment question: the official audit event for an aggregated-API write carries an empty body, yet the live watch carries the full object. Because the watch supplies the object content natively, the supplementary /audit-webhook-additional body-enrichment proxy was retired — so the lab no longer serves or records that endpoint.

Running it (opt-in, serial)

These tasks are not part of task test-e2e or the per-PR CI lanes; the mutation lab workflow runs lab-capture nightly.

task lab-bootstrap      # certificates, cluster, image, deployment, and a live check of all five sources
task lab-capture        # run the scenarios and compare the capture to the corpus
task lab-corpus-update  # accept a new or changed capture as the corpus, CLUSTER.md included
task lab-status         # the cluster, its k3s version, the lab pod, and how many records it holds
task lab-cleanup        # delete the cluster and its stamps

The lab cluster can run beside the e2e cluster: its own name, stamps (.stamps/cluster/k3d-gitops-reverser-capture-lab/), run lock, and records-API port (18081). It needs no restore afterwards; the e2e cluster is never touched.

The unit tests for the lab packages run in the normal lane (task test); the test/mutationlab/e2e/ driver is behind the mutationlab_e2e build tag and runs only through the lab's tasks.

Validating a new Kubernetes version

task lab-cleanup
task lab-bootstrap LAB_K3S_IMAGE=rancher/k3s:v1.38.0-k3s1
task lab-corpus-update
git diff test/mutationlab/corpus/

The corpus/ diff is the behavioral changelog for the upgrade, scoped to exactly the fine-grained behaviors GitOps Reverser depends on. An empty diff is a positive result. Without LAB_K3S_IMAGE, a new lab cluster runs the k3s image CLUSTER.md records, and lab-capture fails when the cluster serves another version than the corpus was captured on.