Documentation
¶
Overview ¶
Package nodegc reclaims stale Kubernetes Node objects on a peer-hosted DKS control plane.
WHY. A tunnelled worker that leaves the cluster (or simply dies) cannot delete its own Node object — `outpost cluster leave` holds only a k3s join token, deliberately not an admin credential for the peer apiserver (docs/cluster-peer.md, "Boundary: the Kubernetes Node object is not deleted by the worker"). And k3s agents register with `--with-node-id`, so every rejoin mints a NEW node name: without GC the plane accumulates NotReady ghosts that pollute `kubectl get nodes`, scheduling scores, and the status view forever.
The collector is deliberately narrow and slow. It only ever deletes a node that is self-consistently OURS — exact runtime/backend labels, a nonempty host label, and a name carrying that host's prefix — and that has been NotReady/Unknown for longer than a 24 h grace period. Virtual-kubelet nodes (runtime=virtual), foreign or unlabelled nodes, renamed nodes, Ready nodes, recently-transitioned nodes, and nodes missing readiness evidence are never touched: absence of evidence is unknown, not staleness. Deletes are bounded per tick, re-verified by a fresh GET immediately beforehand, and guarded by a UID precondition so a same-name replacement registered between passes survives. Any API error stops the whole pass — a flaky apiserver is a reason to do nothing, not to guess.
The same evidence rule holds at FLEET level: absence of evidence from ALL nodes at once is the strongest possible signal that the OBSERVER — this host's network, tunnel, or clock — is what broke, not the fleet. So beyond the per-node predicate:
- a pass is refused outright when the stale fraction of the ELIGIBLE outpost-agent population crosses a threshold (a mass partition must never drain the cluster at MaxDeletes/tick). The denominator is the DELETABLE population — structurally-excluded nodes (own host, control-plane role) are left out, so they cannot dilute the fraction and let a full-worker partition slip under it;
- the delete budget is a wall-clock rate persisted in a JSONL ledger, so outpost's frequent self-restarts (any builtin toggle) cannot mint a fresh budget per restart. With NO ledger a real pass fails closed and deletes nothing — an absent durable rate limit is never a licence to delete;
- readiness evidence timestamped in our future refuses the pass, and so does a wall clock that raced FORWARD of the collector's monotonic uptime between passes (VM resume, restored snapshot, delayed NTP step) — the direction the future-stamp check cannot see. Deletions additionally require the collector to have been up — measured monotonically — long enough for a boot-skewed RTC to have been NTP-corrected before we trust wall-clock arithmetic;
- nodes claiming this host's own identity and nodes carrying a control-plane role label are never candidates.
SCOPE DECISION (ownership). The candidate filter trusts node LABELS, which are self-asserted by the joining kubelet — the plane holds no per-node owner record to check against, and this collector's charter is to work offline with no cloudbox feed. Under the dks tenancy model (dhnt/docs/dks-tenancy-model.md) a plane may admit workers joined by other owners; a dead node they labelled as an outpost k3s agent WILL eventually be reaped here. That is accepted and recorded in docs/cluster-peer.md: the plane host is the tenancy authority for its own plane, deletion additionally requires >grace of NotReady (a live tenant node can never be reaped via labels alone), and the Node object is cheap to re-mint by rejoining. What the filter must never do is delete the plane's OWN node or any control-plane node — both are excluded structurally, not by label trust.
This runs ONLY where the plane it manages runs: the control-plane host. It needs no cloudbox peer list or liveness feed — the node's own recorded Ready transition is the staleness clock.
Index ¶
Constants ¶
const ( // EventDeleted records one real node deletion. These entries are // also the restart-safe rate limiter: the delete budget counts // EventDeleted lines inside the trailing window, so a daemon that // restarts ten times in ten minutes shares ONE budget instead of // minting ten. EventDeleted = "deleted" // EventDryRunDeleted records a node dry-run mode WOULD have // deleted. Never counted against the real budget. EventDryRunDeleted = "dry_run_deleted" // EventRefusedMassStale records a whole pass refused by the // cluster-wide unhealthy circuit breaker. EventRefusedMassStale = "refused_mass_stale" // EventRefusedClockSkew records a whole pass refused because a // node's readiness evidence was timestamped in our future. EventRefusedClockSkew = "refused_clock_skew" )
Ledger events. Deletes and refusals both get a durable line: this collector destroys cluster state unattended, so "what did it do and why" must survive the daemon's own logs (which rotate, and which a self-restart truncates from view). Same precedent as the upgrade subsystem's upgrade.log.
const ( // RuntimeLabel / BackendLabel / HostLabel are the node-identity // labels the k3s agent entrypoint stamps at registration // (internal/agent/runtime/image/entrypoint.sh), matching the // cloud-hosted control plane's vocabulary exactly. RuntimeLabel = "outpost.dhnt.io/runtime" BackendLabel = "outpost.dhnt.io/backend" HostLabel = "outpost.dhnt.io/host" // RuntimeAgent / BackendK3s are the only label values GC acts on. // Virtual-kubelet nodes carry runtime=virtual and a vk-* backend; // they have no kubelet lease of their own worth reaping here. RuntimeAgent = "agent" BackendK3s = "k3s" // ControlPlaneRoleLabel / MasterRoleLabel mark apiserver-hosting // nodes (k3s stamps the first on its server node). A node carrying // either is NEVER a GC candidate regardless of its other labels — // reaping the plane's own node is strictly worse than reaping a // worker, and role labels are at least plane-adjacent facts rather // than pure worker self-assertion. ControlPlaneRoleLabel = "node-role.kubernetes.io/control-plane" MasterRoleLabel = "node-role.kubernetes.io/master" // DefaultGrace is how long a node must have been NotReady/Unknown // (per its Ready condition's LastTransitionTime) before it is // stale. A day comfortably clears laptop sleeps, reboots, and // multi-hour network outages — the node of a machine that comes // back inside the window reconnects and goes Ready again. DefaultGrace = 24 * time.Hour // DefaultMaxDeletes bounds deletes per rate window (see // DefaultInterval). GC converges over several windows instead of // mass-deleting in one, so one bad pass can only do bounded damage. DefaultMaxDeletes = 3 // DefaultInterval is the pass cadence AND the delete-rate window: // at most MaxDeletes deletions per Interval of wall clock, counted // from the persisted ledger — not per pass, and not per process // lifetime. Outpost self-restarts on any builtin toggle; a budget // that reset with the process would multiply by restart count. DefaultInterval = time.Hour // DefaultSettle delays the FIRST pass after Run starts. Boot is // exactly when evidence is least trustworthy: the tunnel may not // be up, NTP may not have corrected a wrong RTC yet, and restart // storms (10 toggles == 10 boots) would otherwise each get an // immediate pass. DefaultSettle = 5 * time.Minute // DefaultMinUptime is how long the collector must have been // running — measured monotonically, immune to wall-clock jumps — // before it is allowed to DELETE. Staleness is wall-clock // arithmetic against apiserver-recorded timestamps; a host whose // RTC was wrong at boot (consumer hardware, restored VM snapshot, // dual-boot) computes garbage elapsed times until NTP fixes the // clock, and boot is also when the settle-window pass fires. An // hour of uptime — a meaningful fraction of the 24 h grace — means // NTP had ample time to sync before we trust the math. Passes // before that run read-only. DefaultMinUptime = time.Hour // DefaultMaxStaleFraction is the cluster-wide unhealthy circuit // breaker (upstream's --unhealthy-zone-threshold analog): when // MORE than this fraction of the observed outpost-agent population // is stale at once, the pass deletes nothing. "Everything looks // dead" nearly always means the observer broke — a >24 h partition // of the control-plane host flips every worker to Unknown with the // same LastTransitionTime, and without this guard GC would drain // the cluster at MaxDeletes per window until empty. DefaultMaxStaleFraction = 0.5 )
Variables ¶
This section is empty.
Functions ¶
func StaleSince ¶
StaleSince reports whether node is a stale GC candidate at time now, and if so since when (its Ready condition's LastTransitionTime).
A candidate must be self-consistently an outpost k3s agent node: exact runtime/backend labels, a nonempty host label, and a name prefixed by that host — anything else (virtual, foreign, unlabelled, renamed) is out of scope. It must carry a Ready condition that is False or Unknown with a nonzero LastTransitionTime strictly older than grace; a missing condition or timestamp is unknown, not stale.
This is the pure per-node predicate. The Collector layers its structural exclusions (own host, control-plane role) and the fleet-level guards (mass-stale breaker, clock skew, rate budget) on top — see candidateSince and Once.
Types ¶
type Collector ¶
type Collector struct {
Client kubernetes.Interface
Grace time.Duration // 0 => DefaultGrace
MaxDeletes int // 0 => DefaultMaxDeletes (per Interval window, not per pass)
Interval time.Duration // 0 => DefaultInterval (pass cadence AND rate window)
Log *slog.Logger // nil => slog.Default
Now func() time.Time // nil => time.Now
// DryRun computes, logs, and ledgers exactly what a real pass
// would delete — including budget and breaker decisions — but
// never calls Delete.
DryRun bool
// SelfHost, when set, excludes any node whose host label claims
// this host's own identity (the control-plane host may itself run
// an agent runtime joined to its own plane). Wired from
// FileConfig.ClusterNodeName().
SelfHost string
// MaxStaleFraction overrides DefaultMaxStaleFraction when > 0.
// A value >= 1 disables the breaker (every fraction is <= 1).
MaxStaleFraction float64
// Settle overrides DefaultSettle when > 0: how long Run waits
// before the first pass.
Settle time.Duration
// MinUptime overrides DefaultMinUptime when > 0: monotonic
// collector uptime required before deletions are allowed.
MinUptime time.Duration
// Uptime overrides the monotonic-uptime source (tests). nil =>
// time.Since(first Run/Once), which Go measures monotonically.
Uptime func() time.Duration
// Ledger is the durable JSONL record of deletions and refusals,
// and the restart-safe half of the rate budget. nil => in-memory
// budget only (tests / degraded boot); real wiring always sets it.
Ledger *Ledger
// contains filtered or unexported fields
}
Collector deletes stale agent Node objects from the hosted plane.
func (*Collector) Once ¶
Once performs a single bounded GC pass. Any List/Get/Delete error — and any failure to durably record a deletion — aborts the pass and is returned, never swallowed. Fleet-level refusals (mass-stale breaker, clock skew, insufficient uptime) are DECISIONS, not errors: they log, ledger where durable evidence matters, and return nil.
func (*Collector) Run ¶
Run collects until ctx is cancelled. The first pass waits out the settle window — there is deliberately no immediate pass, so restart storms (outpost restarts on every builtin toggle) never reach a pass at all. Pass errors are logged, never fatal — the next tick retries from scratch.
type Ledger ¶
type Ledger struct {
// contains filtered or unexported fields
}
Ledger is an append-only JSONL writer at a fixed path, mirroring internal/agent/upgrade.Ledger. A nil Ledger (or empty path) is a valid no-op sink — but note Collector treats "no ledger" as "no durable budget", falling back to in-process memory only.
Single-writer by construction: exactly one outpost daemon runs per host (the pidfile gate in `start`), and only the control-plane host's collector writes this file — so O_APPEND line writes are sufficient and no inter-process lock is needed here (contrast the CNI IPAM ledger, which IS written by concurrent processes).
func NewLedger ¶
NewLedger returns a Ledger backed by path. Doesn't touch the filesystem until the first Append.
func (*Ledger) Append ¶
func (l *Ledger) Append(entry LedgerEntry) error
Append writes one entry as a single JSON line. A zero At is filled with the current time. Append errors on REFUSAL entries are logged by the caller and never block anything; a failure to record a DELETE is treated as fatal to the pass by the caller (Collector refuses to keep deleting when it cannot durably count deletions — the record is what bounds the rate across restarts).
func (*Ledger) CountDeletesSince ¶
CountDeletesSince returns how many real deletions (EventDeleted) the ledger records strictly after cutoff. A missing file is zero deletions, not an error. Malformed lines are skipped — a corrupt line can therefore UNDER-count by at most the lines lost, which is bounded by MaxDeletes per window and accepted; any other read error is returned and the caller must fail safe (refuse deletes).
type LedgerEntry ¶
type LedgerEntry struct {
At time.Time `json:"at"`
Event string `json:"event"`
Node string `json:"node,omitempty"`
UID string `json:"uid,omitempty"`
NotReadySince string `json:"not_ready_since,omitempty"`
Stale int `json:"stale,omitempty"`
Scope int `json:"scope,omitempty"`
Detail string `json:"detail,omitempty"`
}
LedgerEntry is one JSON line in the nodegc history file.