aethermesh.dev

Aether

A Kubernetes service mesh data plane built in Go. The Go module is aethermesh.dev (a vanity import path served by the website; versions before the rename remain importable as github.com/bpalermo/aether — see proposal 035). Aether runs a per-node agent (DaemonSet) that drives a custom Envoy build (aether-proxy) via an xDS control plane, plus a CNI plugin that sets up pod network namespaces and registers their endpoints. Config is demand-scoped: each agent generates only the clusters, registry watches, and endpoints its local pods actually depend on (declared via the config.aether.io/upstreams annotation), with on-demand CDS for the cold path. An in-cluster Registrar service proxies all registry operations, caches a versioned endpoint snapshot, and streams changes to agents. Routing is driven by the Gateway API (GAMMA east-west + a north-south edge gateway). It integrates with SPIRE for workload identity and mTLS, supports zero-drop proxy rollouts via Envoy hot restart, and exports OpenTelemetry metrics and traces. Pluggable external registry backends: DynamoDB, etcd, and Kubernetes.

Architecture

Solid arrows are the workload data path; dashed arrows are control plane / telemetry.

graph TD
    subgraph node["Node (DaemonSet)"]
        Pod["Workload Pod"]
        CNI["CNI Plugin<br/><i>netns setup · endpoint registration</i>"]
        Agent["Agent<br/><i>xDS · CNI server · proxy supervisor</i>"]
        Proxy["aether-proxy<br/><i>custom Envoy, hot-restart supervised</i>"]
        MeshDNS["mesh-dns<br/><i>own DaemonSet · snapshot-fed resolver</i>"]
        SPIRE["SPIRE Agent<br/><i>workload identity</i>"]

        Pod == "pod traffic" ==> Proxy
        Pod -. "DNS :53 (CNI DNAT)" .-> MeshDNS
        Agent -. "record snapshot (file)" .-> MeshDNS
        CNI -. "register (gRPC/UDS)" .-> Agent
        Agent -. "xDS, demand-scoped<br/>LDS·CDS·EDS·RDS·SDS·ODCDS" .-> Proxy
        Agent -. "Delegated Identity API" .-> SPIRE
        SPIRE -. "X.509 SVIDs (via SDS)" .-> Proxy
    end

    Peer["Peer node<br/><i>aether-proxy → workload pod</i>"]
    Proxy == "mTLS (SPIFFE)" ==> Peer

    Registrar["Registrar<br/><i>in-cluster Deployment, active/active</i>"]
    Agent -. "register · watch · list" .-> Registrar

    Registry[("External Registry<br/>DynamoDB · etcd · Kubernetes")]
    Registrar -. "sync + persist" .-> Registry

    OTel["OTel Collector<br/><i>metrics · traces</i>"]
    Agent -. "OTLP push" .-> OTel
    Proxy -. "stats sink + aether_stats" .-> OTel

Agent — Runs on each node via controller-runtime. Manages the xDS server, CNI gRPC server, SPIRE bridge, registrar client, and the proxy hot-restart supervisor as runnables. Generates Envoy configuration (listeners, clusters, endpoints, routes) from local pod data and the endpoint cache populated by the Registrar's push stream. Config is demand-scoped to each node's dependency set (see below).

aether-proxy — A custom Envoy build maintained in a separate sibling Bazel workspace under proxy/ (pinned to its own Bazel 8.7.0, built from Envoy source) with a compiled-in C++ aether_stats extension that records source→destination request metrics. The agent supervises it with cross-pod hot restart for hitless rollouts and two-phase connection draining. See proxy/README.md and proposals 010 / 012.

Demand-scoped distribution — Each agent generates only the clusters, registry watches, and endpoints its local pods declare a dependency on via the config.aether.io/upstreams annotation, with on-demand CDS (ODCDS) serving the cold path. This bounds per-node config to the node's actual footprint and replaces fleet-wide CDS and client-side active health checking. Multi-port and FQDN upstreams are demuxed via SNI with per-port EDS. See proposals 004 / 005.

mesh-dns — A slim per-node DaemonSet (its own binary and image) that answers <svc>.<ns>.<meshDomain> from a record snapshot the agent writes to a host path, and forwards everything else upstream. The CNI DNATs each managed pod's :53 to it. It is deliberately decoupled from the agent (#578, #583) so an agent roll never gaps pod DNS.

Gateway API & GAMMA routing — Routing is expressed with the Kubernetes Gateway API. East-west (mesh) traffic uses GAMMA: HTTPRoute/GRPCRoute objects with a parentRef to a Service enrich that service's outbound/capture routes (canary splits, header/method matches, timeouts, redirects). North-south traffic uses the same API against the edge gateway's GatewayClass. Both directions share one projector, common/gammaproject, which turns a route rule into a registryv1.GammaRoute proto; the node agent materializes it locally into Envoy config while the registrar can export it cross-cluster. An HTTPFilter CRD (proposal 025) is the escape hatch for attaching supported Envoy HTTP filters (ext_authz, RBAC, header-to-metadata) at route, service-wide (CHAIN), or destination-side (INBOUND) scope.

Controller — In-cluster Deployment (leader-elected) that serves the admission webhooks (MeshConfig, HTTPFilter, EdgeConfig, EndpointPolicy, HTTPRoute validation + a pod-mutating webhook for mesh-domain ndots and namespace-based mesh injection) and projects each namespace's MeshConfig CR into a ConfigMap the agent and edge mount.

Registrar — In-cluster Deployment that acts as the sole bridge between agents and the external registry. Receives endpoint registrations from agents, persists them externally, maintains a versioned in-memory snapshot via periodic sync, and streams changes to all agents via gRPC server-streaming. Runs as an active/active Deployment (every replica serves gRPC and syncs; peers converge through the external registry), collapsing per-node external connections down to the registrar tier.

CNI Plugin — Implements the CNI spec (Add/Del/Check/GC/Status) to set up each pod's network namespace. Communicates with the agent over a Unix domain socket to register the pod's endpoints on Add and deregister them on Del.

UDS delivery — Workloads that serve on a Unix domain socket instead of a TCP port join the mesh with endpoint.aether.io/uds-socket: <volume>/<file> (or a service-scoped EndpointPolicy CR). The proxy delivers inbound requests to the socket through kubelet's pod-volumes directory; callers are unaffected — the pod is still reached at its pod IP over mTLS. See proposal 034.

SPIRE Bridge — Connects to the SPIRE agent via the Delegated Identity API to obtain X.509 SVIDs and trust bundles. Converts them into Envoy SDS (Secret Discovery Service) resources for automatic mTLS between workloads.

External Registry — Pluggable backend for durable endpoint storage, selected on the Registrar via --registry-backend:

• DynamoDB — single-table design for AWS-native deployments
• etcd — hierarchical key structure with protobuf serialization, native Watch for change streaming
• Kubernetes — registry backed by the cluster API

Observability — Push-first OpenTelemetry. When otel.endpoint is set (chart value; --otlp-endpoint on each binary), the agent, CNI, and registrar export OTLP metrics (--otel-enabled) and optionally traces (--trace-export) to a collector. The proxy ships its Envoy stats over the same sink, and the compiled-in aether_stats extension emits per-source/destination request counters.

Getting Started

Prerequisites

• Bazelisk (Bazel 9.2.0)
• Go 1.26.5
• Docker (or Colima) for container images and integration tests

Setup (macOS with Colima)

If you use Colima for Docker on macOS, run this once to configure the Docker socket for Bazel sandboxed tests:

./bazel/configure_colima.sh

This generates .bazelrc.colima (gitignored) with your socket path. The config is auto-enabled on macOS via --config=colima.

Build

make build-agent           # Build the node agent
make build-registrar       # Build the registrar service
make build-cni-install     # Build the CNI installer

Test

make test                  # Run all tests (requires Docker for integration tests)
make test-unit             # Run unit tests only (no Docker required)
make test-integration      # Run integration tests only (requires Docker)
make test-race             # Run all tests with Go race detector

Code Quality

make format                # Format all code (Go, protobuf, Starlark, shell)
make format-check          # Check formatting (CI-friendly, fails on drift)
make lint                  # Run linters (buf, buildifier, shellcheck)

Formatting uses gofumpt, buildifier, shfmt, and buf via aspect_rules_lint. Linting runs buf (protobuf), buildifier (Starlark), and shellcheck (shell) as Bazel aspects. CI enforces lint violations with --config=ci.

Container Images

make load-all              # Load all images into local Docker
make push-all              # Push all images to registry

Adding Go Dependencies

bazel run @rules_go//go get <package>
bazel run //:gazelle

License

Licensed under the Apache License, Version 2.0. See NOTICE for attribution.