operator

Overview ¶

corednsspread.go — move a CoreDNS replica off a node that holds all of them. The anti-affinity is preferred, so it is only consulted at scheduling time and never rebalances. Eviction, not delete, so a PodDisruptionBudget is honoured.

Kubelet serving-cert CSR approver.

A kubelet that runs serverTLSBootstrap requests a `kubernetes.io/kubelet-serving` CSR — every kubelet under `stratum server --cis`, and also every token-joined worker, which has no local PKI to pin and so bootstraps its serving cert this way. Kubernetes deliberately does NOT auto-approve that CSR (unlike the kubelet CLIENT CSR), because serving-cert SANs (node IPs/DNS) are spoofable and must be validated against the node's real addresses. An unapproved kubelet has no serving cert at all, so the apiserver cannot complete a TLS handshake with it → cluster-wide kubectl logs/exec/top + metrics outage. This reconciler is what makes serverTLSBootstrap safe: it approves a serving CSR ONLY when the requester is the node itself and every SAN it asks for is an address that node legitimately owns.

departednodes.go — remove the Node of a worker that `stratum uninstall` took off the cluster. The worker's own credential cannot delete its Node (the Node authorizer does not let system:node:<name> delete nodes), so it marks the Node as it leaves and this deletes it once it is gone. Only the node itself can write the mark: the stratum-node-uninstall-mark admission policy (internal/addons) refuses it from anyone else.

grants.go — the operator's pass over the stratum-certs grant entries. The Secret carries one ciphertext entry per live control-plane join grant, keyed by bootstrap token ID, beside the legacy certs entry. Owner references, not data entries, are what garbage collection acts on, so an expired or revoked grant's entry stays until something drops it. Every mint drops them inside its own update (ha.UploadCerts); this pass drops them as well, so they go even when no one mints. Which entries are grants, and when a grant is gone, is internal/ha's rule (ha.PruneExpiredGrants), the package that writes the entries. The pass rides the operator's reconcile loop as part of the cleanup step Run binds with cleanupStepFor, beside departed-worker removal.

inventory.go — the prune inventory in cluster state, and the desired bundle generation recorded beside it.

The prune record is a per-node disk file: <manifests dir>/.applied.json, written by whichever operator instance ran the manifest pass. On a multi-controller cluster that record is node-local, so a leader change read whatever the node holding the record had last applied and pruned against a fraction of what the cluster once applied — and an object only another node's record remembered was orphaned forever. The inventory moves the record into cluster state: a ConfigMap in kube-system, stratum-addon-inventory, holds one entry per operator instance under its node's name, each entry the map from stack to object references that instance's manifests directory applied.

Migration: entries are published by union, never rewritten wholesale. Run drives syncInventory on every operator instance — leader or not, on its own background interval beside the CSR approvers — and syncInventory unions the instance's local .applied.json into its node's entry with resourceVersion conflict retry: a conflict re-reads the ConfigMap, re-unions, and retries, so a concurrent writer's entry survives instead of being clobbered and a lost race never fails the pass. The migration is complete when every control-plane node in cluster state has contributed an entry, or when an explicit recovery entry closes it by hand — the escape hatch for a decommissioned node whose contribution can no longer be expected. The recovery entry is a hand edit of the ConfigMap's inventory document (set "recovery": true beside the nodes; there is deliberately no CLI for it), so closing the migration without a missing node is a human decision, never something the operator does on its own. Until then the manifest pass suspends pruning: pruning before the last contribution could delete the objects only that node's record remembers. Once the migration closes, the union of the cluster inventory — not any local file — is the record pruning is driven by, and syncInventory retires the local file, so a stale record can never re-introduce an object the cluster inventory already pruned.

Desired generation: the same ConfigMap also holds the desired bundle generation the staging path (writeBundles in internal/cluster) records whenever it renders a changed bundle set — {generation, digest, provenance}, written through the apiserver, so RBAC is the integrity boundary and the record is separate from the operator binary version. The deployer reads it to tell what it is about to apply apart from what was staged: a staged tree whose digest does not match the record is refused for the pass, and an explicitly recorded empty set — the desired-empty generation — drives pruning of the last stack's objects (see manifests.go).

leaderelect.go — single-active-instance leader election via a Lease.

runLeaderElected campaigns for a coordination.k8s.io Lease (kube-system/stratum-operator) and only runs the reconcile loop while this instance holds it. This lets multiple control-plane hosts run stratum-operator for HA while guaranteeing exactly one actively reconciles (the HelmChart and manifests controllers make mutating cluster changes — two concurrent writers would fight).

On losing leadership we stop the reconcile loop and return an error. The standalone CLI exits nonzero so its supervisor can restart it; importers retain control of their own process and cleanup.

manifests.go — auto-deploy plain Kubernetes YAML from a host directory.

reconcileManifests server-side-applies every *.yaml/*.yml file under a directory (default /etc/stratum/manifests) so operators can drop manifests on the control-plane node and have them applied without a kubectl round-trip — the manifest equivalent of the HelmChart controller. This mirrors the SSA applier in stratum/internal/addons/apply.go: a discovery-backed RESTMapper and a per-object ApplyPatch with FieldManager "stratum-operator", Force.

Stacks and pruning: a first-level subdirectory of the manifests dir is a STACK. Every object applied from a stack is labeled stratum.ciq.com/stack=<name> and recorded as that stack's object references; objects the record remembers but no file of a fully successful pass declares are DELETED on that pass. The keep-set is the whole pass's union, matched by canonical identity (the RESTMapper's GroupResource, the effective namespace and the name), so an object a release moved between stacks, re-versioned or made namespace-explicit survives the pass that applies it under its new shape. Loose files directly in the manifests dir keep the old apply-only contract — their objects join the keep-set but are never recorded, so removing a loose file deletes nothing — the operator drop-a-file-and-go path stays safe.

The record pruning is driven by is the prune inventory in cluster state (see inventory.go): a stratum-addon-inventory ConfigMap in kube-system with one entry per operator instance, whose union every instance of the cluster published from its own manifests directory. Until the migration to that inventory has closed — every control-plane node contributed an entry, or an explicit recovery entry closed it — pruning suspends, and a pass with no inventory ConfigMap at all still prunes against the per-node <dir>/.applied.json earlier releases wrote.

To avoid hot-looping the apiserver, a file is re-applied only when its bytes change OR when the resync window (5m) has elapsed since its last apply.

Leader-only note: this applies the LEADER's local directory. With leader election on and multiple control planes, put identical files on every CP (or just use kubectl) — a follower never applies its copy. The bundled stacks are identical by construction: anything cluster-specific in them is a sentinel filled in here, from cluster state every leader sees the same — Cilium's apiserver endpoint, and the four cluster inputs the staging records in the inventory's cluster-inputs document (the DNS address, the cluster domain, dual-stack and the storage default, resolved through manifestSentinels). A sentinel the cluster state cannot resolve skips its file rather than guess: the file keeps no apply state and retries next pass. The prune inventory is the one piece of pass state that outlives a leader change: the entries live in the cluster, not in the leading node's directory.

Kubelet client-cert CSR approver for bootstrap-token joins: a join token may claim a new node name, never one already registered or claimed by a certificate.

Package operator is the in-binary SKE operator: the HelmChart / HelmChartConfig controller, the manifest deployer — leader-elected so a multi-controller cluster runs one active reconciler — and the kubelet CSR approvers, which run on every controller.

It runs as a SUPERVISED HOST PROCESS (`stratum operator`, a child of `stratum up`), not as an in-cluster Deployment: no operator image to pull keeps the airgap story one-artifact.

Rolling node upgrades are NOT here: `stratum upgrade --cluster` (internal/upgrade) is the one implementation, driven from a control-plane node rather than from a CR.