Documentation
¶
Overview ¶
Package nodeaddr makes an apiserver able to reach kubelets that are only reachable through a tunnel.
WHY THIS EXISTS. The apiserver dials kubelet using the address and port it finds in Node.Status.Addresses + Status.DaemonEndpoints.KubeletEndpoint. When kubelet runs on a machine reachable only through a tunnel, those self-reported defaults (the node's own LAN IP : 10250) do not resolve from the control plane. Every `kubectl logs`, `exec`, `port-forward` and metrics scrape depends on that dial, so without this the cluster looks healthy — nodes go Ready, pods schedule — and the operator commands people actually use are the ones that fail.
This is the peer-hosted counterpart of the same job cloudbox does for the cloud-hosted control plane. It moves here because a control plane running on a user's own machine has no cloudbox in the loop (dhnt/docs/dks-control-plane-on-sphere.md).
Index ¶
- Constants
- func DerivedKubeletPort(nodeName string) (int, bool)
- func IsVirtualNode(n *corev1.Node) bool
- func KubeletPortForNode(name string) int
- func LoopbackForNode(name string, taken map[string]string) string
- func ResetStatus()
- func RoutingName(n *corev1.Node) string
- type KubeletPortFunc
- type NodeAddressDiagnostic
- type Reconciler
- type Status
Constants ¶
const ( RuntimeLabel = "outpost.dhnt.io/runtime" RuntimeVirtual = "virtual" )
Node-identity label vocabulary, matching the cloud-hosted control plane and the peer packages (nodegc, nodecap) exactly. A virtual-kubelet node carries RuntimeLabel=RuntimeVirtual.
Variables ¶
This section is empty.
Functions ¶
func DerivedKubeletPort ¶
DerivedKubeletPort is the KubeletPortFunc that pairs with KubeletPortForNode — the default for a peer-hosted control plane.
func IsVirtualNode ¶
IsVirtualNode reports whether n is a virtual-kubelet node.
Address reconciliation exists to make the apiserver dial a kubelet that is only reachable through a tunnel. A virtual-kubelet node runs NO kubelet of its own — it serves only the Pod lifecycle subset, not the streaming API — so there is nothing at the derived loopback address:port to dial. Patching one would publish an ExternalIP the apiserver then prefers for `kubectl logs`/`exec`/`port-forward`/metrics, turning "unsupported on a virtual node" into a confusing dial timeout, and would consume a slot in k3s's ExternalIP routing trie that a real kubelet could use. So the reconciler skips virtual nodes by LABEL — never by node name.
func KubeletPortForNode ¶
KubeletPortForNode derives the loopback port on which a node's kubelet is published through the tunnel.
DERIVED, NOT ALLOCATED, and that is the point: both ends compute it independently from the node name, so nothing has to distribute it. The control-plane host needs the number to patch DaemonEndpoints; the node needs it to bind kubelet and to name its frpc proxy's remotePort. A registry, a handshake, or a config push would each add a way for the two to disagree — and they must not, because the tunnel publishes remotePort == localPort (see the runtime image's entrypoint).
Cloudbox instead ALLOCATES this per host at pairing time (ClusterConfig.KubeletProxyPort). That is the right design when a central party already mediates every join and can hold the pool. A peer-hosted plane has no such party, so derivation replaces it.
A collision — two node names hashing to one slot — fails LOUD rather than silently misrouting: frp refuses the second proxy's remotePort and that node's kubelet simply is not published, which surfaces as its `kubectl logs` failing rather than as another node's logs being served. At a handful of nodes over 10000 slots the probability is negligible, and the failure mode is the safe one.
func LoopbackForNode ¶
LoopbackForNode maps a node name to a stable, node-UNIQUE address in 127.0.0.0/8.
UNIQUENESS IS LOAD-BEARING, NOT COSMETIC, and it is the whole reason this function is not just "return 127.0.0.1". k3s's tunnel server indexes every node by its InternalIP *and* ExternalIP in a cidranger trie, and that trie's insert is last-writer-wins for an equal network. So if every node advertises ExternalIP=127.0.0.1 they collapse into a SINGLE 127.0.0.1/32 entry, and the backend dialer resolves every kubelet dial to one arbitrary node — tunnelling the request into the wrong node's session, which then fails to dial a kubelet port that is not listening there and drops the session. At most one node ever has working logs/exec, and WHICH one is nondeterministic.
Derived from the node NAME, not the kubelet port: names are stable across restarts and independent of port-allocation state, which has historically contained duplicates.
Collisions resolve by linear probe over `taken`. Callers MUST probe in a deterministic order (sorted names) so a given node set always yields the same mapping — otherwise addresses churn between passes and every pass issues pointless patches.
func ResetStatus ¶
func ResetStatus()
ResetStatus clears the snapshot. Test-only: the state is process-global, so tests that assert on it must not inherit another test's residue.
func RoutingName ¶
RoutingName returns the stable node name used by the worker runtime when it derives its kubelet tunnel endpoint. k3s's --with-node-id appends a persisted random suffix to metadata.name, so hashing metadata.name would disagree with the worker, which only knows the configured --node-name. k3s records that original argument in k3s.io/node-args.
The annotation is used only when it is structurally consistent with the registered name. A malformed or unrelated annotation falls back to the actual Node name instead of letting arbitrary text redirect another node's kubelet route.
Types ¶
type KubeletPortFunc ¶
KubeletPortFunc reports the loopback port on which THIS control-plane host can reach the named node's kubelet — i.e. the local end of whatever tunnel carries it. Returning ok=false skips the node, which is the right behaviour for one whose tunnel is not up yet.
type NodeAddressDiagnostic ¶
type NodeAddressDiagnostic struct {
NodeName string `json:"node_name"`
ExpectedExternalIP string `json:"expected_external_ip,omitempty"`
ObservedExternalIP string `json:"observed_external_ip,omitempty"`
ExpectedKubeletPort int `json:"expected_kubelet_port,omitempty"`
ObservedKubeletPort int `json:"observed_kubelet_port,omitempty"`
Patched bool `json:"patched"`
}
NodeAddressDiagnostic provides deterministic, secret-free diagnostics for node addressing status.
type Reconciler ¶
type Reconciler struct {
Client kubernetes.Interface
PortFor KubeletPortFunc
Interval time.Duration
Log *slog.Logger
}
Reconciler patches node addresses so the apiserver can reach kubelets.
func (*Reconciler) Diagnostics ¶
func (r *Reconciler) Diagnostics(ctx context.Context) ([]NodeAddressDiagnostic, error)
Diagnostics returns deterministic node addressing status for all nodes in the cluster.
func (*Reconciler) Once ¶
func (r *Reconciler) Once(ctx context.Context) error
Once performs a single reconcile pass.
func (*Reconciler) Run ¶
func (r *Reconciler) Run(ctx context.Context)
Run reconciles until ctx is cancelled. Node events trigger immediate reconciliation so kubelet's status writer cannot leave ExternalIP absent until the next polling interval. The informer relists/reconnects watches; the ticker remains as a recovery resync if an event is ever missed.
LOGGING IS PART OF THE CONTRACT HERE, not decoration. Every reconcile failure used to be logged at Debug, which is below the daemon's default level — so a reconciler that was running and failing every single pass emitted NOTHING, and was indistinguishable in the logs from one that had never started. That ambiguity is what made a control plane with no ExternalIP on any node take far longer to diagnose than it should have. So: one Info line on start (the positive evidence "it is running"), and Warn on the transition into failure and on recovery (the negative evidence), with the steady-state repeats kept at Debug so a long outage cannot flood the log.
type Status ¶
type Status struct {
// Started is true once Run has entered its loop. False means the
// reconciler was never started — the failure mode that produced a cluster
// where no Node ever received an ExternalIP.
Started bool
// LastRunAt is when the most recent reconcile pass completed.
LastRunAt time.Time
// LastError is the most recent pass's error, empty when it succeeded.
LastError string
// ConsecutiveFailures counts passes that have failed in a row; zero when
// the last pass succeeded.
ConsecutiveFailures int
}
Process-wide liveness snapshot for the address reconciler.
WHY A PACKAGE-LEVEL SNAPSHOT. The operator's question during the incident this was written for was exactly "is the nodeaddr reconciler running at all?", and the only way to answer it was to grep the daemon's log — which said nothing, because a reconciler that never started and one that started and failed every pass produced identical (empty) output. `outpost status` could not answer it either: nothing in the status surface described the reconcilers.
So the reconciler publishes its own liveness here and the admincore status prober reads it, mirroring runtime.LastServerHealth() — the same shape the control-plane container's health already uses. It is deliberately a process-global rather than plumbed through Deps: there is exactly one control-plane reconciler set per daemon (running one copy per node would have every node racing to patch every other node), so a global is an honest model of a genuinely singleton fact, and it keeps the admin surface from having to thread a handle through boot just to display a boolean.
func LastStatus ¶
LastStatus returns the reconciler's liveness snapshot and whether one has ever been recorded. ok=false means Run was never called in this process.
Source Files