Documentation
¶
Overview ¶
Package drivers orchestrates the docker-side state of `clrk dev`: the shared bridge network the k3d cluster + local registry sit on, and the k3d-library-driven cluster controller that brings them up. The k3d node and the (optional) local registry both attach to the `clrk` bridge so in-cluster pods reach the registry via docker DNS.
Index ¶
- Constants
- Variables
- func EnsureNetwork(ctx context.Context) error
- func HostGatewayIP(ctx context.Context, network string) (string, error)
- func ImageTag() string
- func LocalRegistryHostRef(component string, port int) string
- type ClusterDriver
- func (d *ClusterDriver) ApplyObjects(ctx context.Context, objs ...client.Object) error
- func (d *ClusterDriver) EnsureNamespace(ctx context.Context, ns string) error
- func (d *ClusterDriver) HostKubeconfigPath() string
- func (d *ClusterDriver) KubeClient(ctx context.Context) (client.Client, error)
- func (d *ClusterDriver) KubeconfigPath() string
- func (d *ClusterDriver) NodeContainerName() string
- func (d *ClusterDriver) NodeHostname() string
- func (d *ClusterDriver) RegistryHostPort() int
- func (d *ClusterDriver) Reset(ctx context.Context) error
- func (d *ClusterDriver) Rollout(ctx context.Context, ns, name string) error
- func (d *ClusterDriver) RolloutWorkerPool(ctx context.Context, ns, name string) (int64, error)
- func (d *ClusterDriver) Start(ctx context.Context) error
- func (d *ClusterDriver) Stop(ctx context.Context) error
- func (d *ClusterDriver) WaitDeploymentAvailable(ctx context.Context, ns, name string, timeout time.Duration) error
- func (d *ClusterDriver) WaitRolloutComplete(ctx context.Context, ns, name string, timeout time.Duration, ...) error
- func (d *ClusterDriver) WaitWorkerPoolConverged(ctx context.Context, ns, name string, wantGeneration int64, ...) error
Constants ¶
const ( RegistryDataVolume = "clrk-registry-data" RegistryDataMount = "/var/lib/registry" )
RegistryDataVolume backs the local registry's /var/lib/registry so pushed images survive `clrk dev` stop/start cycles. Attached via k3d's Registry.Volumes field at RegistryRun time.
const ( // ClusterName is the k3d cluster identifier. Distinct from // NetworkName ("clrk") and the in-cluster "clrk" namespace so the // three layers aren't confusable in `docker ps` or `kubectl` output. ClusterName = "clrk-dev" // RegistryName is the docker container name for the local OCI // registry on the shared bridge. In-cluster pods resolve it as // `clrk-registry:5000` via docker DNS. RegistryName = "clrk-registry" // DefaultK3sImage pins a minor version whose k8s.io/api release matches // clrk and apoxy-cloud's module resolution (v0.34.x). Override via // ClusterDriver.K3sImage for CI pinning. DefaultK3sImage = "rancher/k3s:v1.34.1-k3s1" // DefaultRegistryImage is the OCI distribution image k3d boots when // we hand it a Registry with Image=="" — we pin explicitly so a k3d // upgrade can't silently swap the binary. DefaultRegistryImage = "docker.io/library/registry:2" // KubeconfigFileName is written into the shared data dir so other // drivers (and the user) can find the rewritten in-network kubeconfig. KubeconfigFileName = "kubeconfig" )
const ( V4Subnet = "192.168.231.0/24" V6Subnet = "fd00:dead:beef::/64" )
V4Subnet and V6Subnet are the explicit subnets declared on the shared bridge. Both must be present: k3d scans IPAM.Config to decide which address families k3s should enable, and an --ipv6 network with only a v6 subnet declared causes k3s inside k3d to crash-loop with no log output. Declaring v4 as well makes k3d see a dual-stack network (matching how plain `docker run` sees one regardless of IPAM config).
const ( ClrkControllerManagerImagePath = "us-west1-docker.pkg.dev/apoxy-dev/public/clrk-controller-manager" ClrkWorkerImagePath = "us-west1-docker.pkg.dev/apoxy-dev/public/clrk-worker" // LocalRegistryControllerManagerImage is the in-network ref clrk // dev's controller-manager container uses when launched with // --registry-image=controller-manager=<…>. ClusterDriver brings up // a docker registry container at this exact name on the shared // `clrk` network and wires it into the k3d cluster via the // SimpleConfig's Registries.Use field, so docker resolves the host // part via docker DNS and k3s's containerd pulls from it without // a registries.yaml override. The host pushes against // localhost:<RegistryHostPort()> for the same content. LocalRegistryControllerManagerImage = "clrk-registry:5000/clrk/controller-manager:dev" // LocalRegistryWorkerImage is the matching ref for the worker. LocalRegistryWorkerImage = "clrk-registry:5000/clrk/worker:dev" )
Public-GAR repo paths where every clrk main commit lands. The .github/workflows/clrk.yml workflow in apoxy-cloud (triggered by a repository_dispatch from this repo) builds the bazel OCI tarballs and pushes them with the first 8 hex chars of the clrk SHA as the tag. ImageTag() below mirrors that derivation so a `clrk` binary built from a given SHA defaults to the matching pushed image.
const NetworkName = "clrk"
NetworkName is the shared docker network k3d and the local registry attach to. Pods inside the cluster reach the registry as `clrk-registry:5000` over this network.
Variables ¶
var ( DefaultControllerManagerImage = ClrkControllerManagerImagePath + ":" + ImageTag() DefaultWorkerImage = ClrkWorkerImagePath + ":" + ImageTag() )
Default image refs are package vars (not consts) because the tag is derived from the binary's embedded VCS info at process start. Cobra reads these as flag defaults; the drivers fall back to them when no override is supplied.
var ClusterServerContainerName = k3dclient.GenerateNodeName(ClusterName, k3d.ServerRole, 0)
ClusterServerContainerName is the docker container name of the single-server node, sourced from k3d's own naming helper so any future change to k3d's prefix scheme is picked up automatically.
Functions ¶
func EnsureNetwork ¶
EnsureNetwork creates the shared docker network if it does not already exist. Safe to call concurrently; the docker daemon serializes creates.
IPv6 is enabled on the bridge so Envoy's DFP cluster can reach AAAA-only hops. If we find a pre-existing `clrk` network whose IPAM does not declare both subnets (e.g. created by an older clrk dev that only set --subnet for v6, or a v4-only network from before IPv6 landed), tear it down and recreate — the alternative is silent ENETUNREACH on every v6 upstream or k3s crash-looping under k3d.
func HostGatewayIP ¶
HostGatewayIP returns the IP that docker resolves the magic "host-gateway" alias to for a container attached to the given network. On Linux native docker this is the bridge gateway (which is the host). On Docker Desktop for Mac/Windows it's the VPN-kit address that bridges container traffic back to the host process. We discover it by spawning a one-shot container on the same network with `--add-host host.docker.internal:host-gateway` and parsing the /etc/hosts entry docker writes — the only portable way to get the same IP a k8s HostAlias entry needs.
network must be the same docker network k3d's node is attached to — otherwise on Linux the resolved IP is the gateway of a *different* bridge and may not be on the routing path the in-cluster Pod uses. On Docker Desktop the network choice doesn't affect the result.
func ImageTag ¶
func ImageTag() string
ImageTag returns the GAR tag matching this clrk binary, derived from the binary's embedded VCS info. Two cases:
- Local build (`go install ./cmd/clrk` from a clean checkout): debug.BuildInfo.Settings carries `vcs.revision` = full SHA. We truncate to 8 hex chars to match the publish workflow.
- Remote install (`go install github.com/apoxy-dev/clrk/cmd/clrk@<rev>`): debug.BuildInfo.Main.Version is a Go pseudo-version of the form `v0.0.0-<timestamp>-<sha-12>`. We pull the last 12 hex chars and truncate to 8.
Falls back to "latest" if neither is reachable — the publish workflow doesn't push a `:latest` tag, so that fallback surfaces a clear `manifest unknown` from `docker pull` instead of silently running stale code.
func LocalRegistryHostRef ¶
LocalRegistryHostRef returns the host-side image ref for the dev session's local registry on the supplied port. The pods see the same content via LocalRegistry{ControllerManager,Worker}Image (resolved via docker DNS on the shared clrk network); the host has to address the registry as localhost:<port>. Callers like `clrk dev push-image` use this to push bytes the pods will pull.
Types ¶
type ClusterDriver ¶
type ClusterDriver struct {
// DataDir is the host path where rewritten kubeconfigs are written.
// Must be the same path mounted into the cm/worker drivers so they
// can read the in-network kubeconfig.
DataDir string
// K3sImage overrides the k3s image k3d boots. Empty = DefaultK3sImage.
K3sImage string
// EnableRegistry brings up the local OCI registry container alongside
// the cluster and wires it into k3d's containerd registries.yaml. Only
// useful for the inner dev loop where you `clrk dev push-image`
// locally-built images; leave false when pulling published refs.
EnableRegistry bool
// RegistryPort is the host port the local registry publishes. Zero
// means auto-pick a free port; the actual port is available via
// RegistryHostPort() after Start. Ignored when EnableRegistry is false.
RegistryPort int
// contains filtered or unexported fields
}
ClusterDriver brings up a k3d cluster (and optionally a colocated local OCI registry) by calling k3d v5 as a Go library — no `k3d` binary on PATH. Both attach to the shared `clrk` docker network (EnsureNetwork). When EnableRegistry is true, pods reach the registry as `clrk-registry:5000` and the host pushes to `localhost:<RegistryHostPort()>`.
func NewClusterDriver ¶
func NewClusterDriver(dataDir, k3sImage string, registryPort int) *ClusterDriver
NewClusterDriver constructs a driver with sane defaults. dataDir is required because the kubeconfig has to land somewhere other drivers can see. k3sImage may be empty to accept the default.
func (*ClusterDriver) ApplyObjects ¶
ApplyObjects server-side-applies one or more typed objects through the host's controller-runtime client. Force ownership so re-applies from `clrk dev reload` reclaim fields from any previous manager.
func (*ClusterDriver) EnsureNamespace ¶
func (d *ClusterDriver) EnsureNamespace(ctx context.Context, ns string) error
EnsureNamespace creates ns if it doesn't exist. Idempotent. Called before starting the controller-manager so the supervised envoy-gateway certgen can write its TLS Secret into the runtime namespace. The controller binary doesn't install its own Namespace, so whoever brings up the controller has to materialize it.
func (*ClusterDriver) HostKubeconfigPath ¶
func (d *ClusterDriver) HostKubeconfigPath() string
HostKubeconfigPath is the host path to the kubeconfig pointing at the kube API's host-published port. Use this for `kubectl` from the host, integration tests, and `clrk dev wait-ready`.
func (*ClusterDriver) KubeClient ¶
KubeClient returns the lazy controller-runtime client. Useful for callers that need to Get/Watch/Patch outside the SSA-only path ApplyObjects offers.
func (*ClusterDriver) KubeconfigPath ¶
func (d *ClusterDriver) KubeconfigPath() string
KubeconfigPath is the host path to the kubeconfig with the server URL rewritten to the in-network k3d node hostname. Bind-mount into cm / worker containers so they reach k3s via the shared docker network.
func (*ClusterDriver) NodeContainerName ¶
func (d *ClusterDriver) NodeContainerName() string
NodeContainerName is k3d's deterministic name for the server node container. Used by dev_status / dev_logs to tail the right container.
func (*ClusterDriver) NodeHostname ¶
func (d *ClusterDriver) NodeHostname() string
NodeHostname is the in-network DNS name for the apiserver. Same as NodeContainerName because docker resolves container names as DNS labels on the shared bridge network.
func (*ClusterDriver) RegistryHostPort ¶
func (d *ClusterDriver) RegistryHostPort() int
RegistryHostPort returns the host port the local registry is published on. Zero before Start succeeds.
func (*ClusterDriver) Reset ¶
func (d *ClusterDriver) Reset(ctx context.Context) error
Reset is Stop plus the shared docker network. Used by the drift-gate recreate path in `clrk dev`, which has to evict any container still pinning the `clrk` bridge (otherwise the next EnsureNetwork-recreate fails). The network is left in place by Stop because a normal clrk-dev shutdown doesn't need to disturb it.
func (*ClusterDriver) Rollout ¶
func (d *ClusterDriver) Rollout(ctx context.Context, ns, name string) error
Rollout bumps a `clrk.apoxy.dev/restartedAt` annotation on the Deployment's pod template, triggering the same rolling-restart behavior as `kubectl rollout restart`. Uses a strategic-merge patch so concurrent reconciles (e.g. envoy-gateway, the WorkerPool controller) can't lose the rollout to a 409 — no Get + Update race.
func (*ClusterDriver) RolloutWorkerPool ¶
RolloutWorkerPool triggers a rolling restart of a WorkerPool's worker Deployment by bumping RestartedAtAnnotation on the WorkerPool's spec.template.metadata.annotations — NOT on the Deployment. The Deployment is a controller-owned child: WorkerPoolDeploymentReconciler rebuilds its pod template from wp.spec.template on every reconcile (via internal/workerpod, which propagates spec.template.metadata.annotations into the pod template), so an annotation patched straight onto the Deployment is wiped on the next pass and the freshly-created ReplicaSet is scaled back to zero (the rollout silently no-ops). Patching the WorkerPool makes the controller propagate the annotation into the Deployment template itself — it can't revert a change it's the source of. Uses a JSON merge patch: no Get+Update, so it can't lose the rollout to a 409, and on an annotations map its merge semantics match strategic merge anyway.
Returns the WorkerPool's post-patch metadata.generation so the caller can wait for status.observedGeneration to catch up (see WaitWorkerPoolConverged) — the race-free signal that the rollout this patch triggered has completed.
func (*ClusterDriver) Start ¶
func (d *ClusterDriver) Start(ctx context.Context) error
Start brings up the k3d cluster (and the registry when EnableRegistry is set) on the shared `clrk` network. Idempotent: if a cluster/registry with our names already exists (left over from a previous `clrk dev` that wasn't stopped), we reuse it instead of failing.
func (*ClusterDriver) Stop ¶
func (d *ClusterDriver) Stop(ctx context.Context) error
Stop deletes the k3d cluster and the registry. Idempotent: a missing cluster or registry is not an error.
func (*ClusterDriver) WaitDeploymentAvailable ¶
func (d *ClusterDriver) WaitDeploymentAvailable(ctx context.Context, ns, name string, timeout time.Duration) error
WaitDeploymentAvailable blocks until ns/name reports DeploymentAvailable=True or timeout elapses.
func (*ClusterDriver) WaitRolloutComplete ¶
func (d *ClusterDriver) WaitRolloutComplete(ctx context.Context, ns, name string, timeout time.Duration, wantDigests ...string) error
WaitRolloutComplete blocks until ns/name has fully rolled out — observed generation caught up, every replica updated and available, none unavailable — and at least one running, Ready pod reports one of wantDigests as a container image. This turns the fire-and-forget Rollout into a trustworthy gate: instead of returning the instant a rollout is triggered, it returns only once the new pod is actually serving, and (when digests are supplied) fails loudly rather than silently testing stale code when the node served a cached `:dev` tag. Pass no wantDigests to wait for rollout completion only — e.g. `clrk dev reload`, which has no pushed digest to assert against.
func (*ClusterDriver) WaitWorkerPoolConverged ¶
func (d *ClusterDriver) WaitWorkerPoolConverged(ctx context.Context, ns, name string, wantGeneration int64, timeout time.Duration) error
WaitWorkerPoolConverged blocks until the WorkerPool has reconciled at least wantGeneration and reports its workers rolled out and ready — Available=True and Progressing=False. This is the race-free counterpart to triggering a rollout via the WorkerPool: the WorkerPool's status is what the controller derives from the Deployment, so waiting on it (rather than polling the Deployment directly) can't observe the pre-reconcile converged state. The observedGeneration floor ensures we're reading the controller's verdict on THIS rollout, not a stale status from before the trigger.
Source Files