kube/

Git proxy and kube guard on Kubernetes, end to end

Two tests, on a real cluster:

• TestGitProxyAndKubeGuardOnKubernetes runs a workload Pod through both monitors on the hub the Helm chart installed — the rest of this page;
• TestKubeShareWithAnExecPlugin shares the cluster with a hub running on the test host, through a kubeconfig that signs in only with an exec credential plugin (Task 20416) — the last section.

kube_test.go runs a workload Pod through the git interception proxy and the Kubernetes access monitor on a hub that deploy/helm/cloop-hub installed, and checks what the Pod, the forge and the audit trail saw (Task 20385). It is the Kubernetes counterpart of the device kit, and it runs in CI on the deploy-artifacts job's kind cluster.

It changes the release it is pointed at — both monitors stay on afterwards — so point it only at a throwaway cluster. CI runs it twice: on the release as installed, one replica, and again once the job has scaled it to a hub cluster of three, where a workload's git and kubectl reach members other than the one that minted their sessions and are forwarded.

What it builds

What it does

1. Builds both images and loads them into kind.
2. Runs the hub's own NetworkPolicy probe (cloop hub doctor --probe-network-policy) and logs the verdict. Egress blocking is asserted only when the probe proved the CNI enforces policies; otherwise the log says it is not asserted.
3. Serves the forge in cloop-e2e-forge-<stamp> with a certificate for github.com from a throwaway CA. acme/granted holds one commit and no .cloop/: the project and its task exist only on the hub, so the Pod can only have them from the seed the hub sends in the run's lease Secret (Task 20402).
4. Creates cloop-e2e-target-<stamp> and cloop-e2e-second-<stamp>, and a ServiceAccount that may edit both, so a refused write — or a refused read of the other grant's namespace — can only have been refused by the monitor.
5. helm upgrade --reuse-values with executor.gitProxy, executor.kubeGuard (with auditAllowed), executor.monitorTLS.auto — the hub issues both certificates from its own CA, and no git_ca_bundle is rendered (checked against the release's config) — the harness image, a per-Pod egress filter opening only the monitors' ports, hostAliases pointing github.com at the forge and extraCACerts trusting its CA.
6. Registers a project whose origin is https://github.com/acme/granted.git, then over the API: the project's goal and its one task (POST /api/init, POST /api/tasks); a github_pat secret granted for acme/granted only, writes limited to cloop/e2e-allowed-*; a kubeconfig grant for the target namespace with no verbs (read-only), and the same kubeconfig granted again for the second namespace with create — one cluster granted twice, which the Pod receives as one merged kubeconfig with a context per grant (Task 20413); a harness credential; the binding to the kubernetes executor. Then POST /api/run.
7. Reads the run's live log and checks:

Then: neither the PAT nor the cluster's ServiceAccount token is anywhere in the Pod object; the forge holds the allowed branch at the probe's commit and not the denied one, and acme/other gained nothing; the audit trail holds gitproxy.push_denied for the denied branch, gitproxy.push_allowed, a refusal naming acme/other, kubeguard.request_denied for configmaps and kubeguard.request_allowed for pods, a kubeguard.request_allowed for the second grant's configmap and a kubeguard.request_denied for the first grant's context reading the second namespace; and the run's Pod, Secrets and NetworkPolicy are deleted. 8. Checks the round trip (Task 20402): the hub's own plan — the one the dashboard renders — shows the task done and attributed to the kubernetes executor, which can only be true if the run's outcome came back out of the Pod's log; the live log says the project state came back, and carries none of the frame it came in. Then it presses Start again and checks the second run did not execute the task: no probe output in its log, and the task still done at the same moment.

Running it

On a kind cluster with the chart installed the way CI installs it (the deploy-artifacts job in .github/workflows/ci.yml):

kind get kubeconfig --name cloop-ci > /tmp/kubeconfig
CLOOP_KUBE_E2E_KUBECONFIG=/tmp/kubeconfig CLOOP_KUBE_E2E_KIND_CLUSTER=cloop-ci \
  go test ./tests/kube/ -run TestGitProxyAndKubeGuardOnKubernetes -count=1 -v -timeout 28m

A kind node image is about 1 GB on disk and the test images another few hundred MB, loaded twice; check free space before running it on a small host.

Sharing a cluster whose kubeconfig runs a plugin

share_test.go needs only the cluster, not the chart: the hub runs on the test host (cloop ui with executors.kube_guard on, tls: auto, the host executor and the stand-in claude), as it would on a developer's machine. It:

1. creates four namespaces — two to share, one for the ServiceAccount, one for a "developer" ServiceAccount holding admin in the other three — and a kubeconfig whose only credential is a fake exec plugin printing the developer's token (client.authentication.k8s.io/v1, a relative command path, provideClusterInfo);
2. runs cloop remote kube share --access write --sa-namespace … --ttl 10m and checks the plugin was told the cluster, the five objects and their RBAC, and that RBAC alone would let the ServiceAccount create its own token;
3. grants the share to the hub's project for writes and dispatches a task running share_probe.sh — which, besides reading and writing, tries every way a write grant could mint a ServiceAccount credential of its own (Task 20420); the test then checks from outside that none of those objects exists, and that a Deployment the cluster administrator made, running as a ServiceAccount of its own, still has its image;
4. waits for the hub to renew the share, then for the first token to expire, and has the same run use the cluster again;
5. revokes the share and checks nothing labelled cloop.dev/kube-share=<id> is left, the secret is gone and the run is cut off;
6. uploads a crafted share: its token is for a ServiceAccount the cluster administrator made, an object named like the share's and a decoy labelled with its id by another owner sit in its namespace, and its metadata lists cluster-admin's binding. Revoked with the administrator's own kubeconfig, it deletes nothing and keeps the hub's secret; --force deletes the secret and still nothing in the cluster. Then the test reads the audit trail, including a kubeguard.request_denied row for each credential refusal's reason.

kind create cluster --name cloop-share
kind get kubeconfig --name cloop-share > /tmp/kubeconfig
CLOOP_KUBE_E2E_KUBECONFIG=/tmp/kubeconfig \
  go test ./tests/kube/ -run TestKubeShareWithAnExecPlugin -count=1 -v -timeout 25m

About thirteen minutes, nearly all of it waiting: ten minutes is the shortest token the TokenRequest API issues, and the API server accepts one for a minute past its expiry. CLOOP_KUBE_E2E_CLOOP_BIN supplies the binary; unset, one is built. CI runs it in the kube-share job on a kind cluster of its own.