nodegc

package
v0.14.29-dev Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 28, 2026 License: MIT Imports: 16 Imported by: 0

Documentation

Overview

Package nodegc reclaims stale Kubernetes Node objects on a peer-hosted DKS control plane.

WHY. A tunnelled worker that leaves the cluster (or simply dies) cannot delete its own Node object — `outpost cluster leave` holds only a k3s join token, deliberately not an admin credential for the peer apiserver (docs/cluster-peer.md, "Boundary: the Kubernetes Node object is not deleted by the worker"). And k3s agents register with `--with-node-id`, so every rejoin mints a NEW node name: without GC the plane accumulates NotReady ghosts that pollute `kubectl get nodes`, scheduling scores, and the status view forever.

The collector is deliberately narrow and slow. It only ever deletes a node that is self-consistently OURS — exact runtime/backend labels, a nonempty host label, and a name carrying that host's prefix — and that has been NotReady/Unknown for longer than a 24 h grace period. Virtual-kubelet nodes (runtime=virtual), foreign or unlabelled nodes, renamed nodes, Ready nodes, recently-transitioned nodes, and nodes missing readiness evidence are never touched: absence of evidence is unknown, not staleness. Deletes are bounded per tick, re-verified by a fresh GET immediately beforehand, and guarded by a UID precondition so a same-name replacement registered between passes survives. Any API error stops the whole pass — a flaky apiserver is a reason to do nothing, not to guess.

The same evidence rule holds at FLEET level: absence of evidence from ALL nodes at once is the strongest possible signal that the OBSERVER — this host's network, tunnel, or clock — is what broke, not the fleet. So beyond the per-node predicate:

  • a pass is refused outright when the stale fraction of the ELIGIBLE outpost-agent population crosses a threshold (a mass partition must never drain the cluster at MaxDeletes/tick). The denominator is the DELETABLE population — structurally-excluded nodes (own host, control-plane role) are left out, so they cannot dilute the fraction and let a full-worker partition slip under it;
  • the delete budget is a wall-clock rate persisted in a JSONL ledger, so outpost's frequent self-restarts (any builtin toggle) cannot mint a fresh budget per restart. With NO ledger a real pass fails closed and deletes nothing — an absent durable rate limit is never a licence to delete;
  • readiness evidence timestamped in our future refuses the pass, and so does a wall clock that raced FORWARD of the collector's monotonic uptime between passes (VM resume, restored snapshot, delayed NTP step) — the direction the future-stamp check cannot see. Deletions additionally require the collector to have been up — measured monotonically — long enough for a boot-skewed RTC to have been NTP-corrected before we trust wall-clock arithmetic;
  • nodes claiming this host's own identity and nodes carrying a control-plane role label are never candidates.

SCOPE DECISION (ownership). The candidate filter trusts node LABELS, which are self-asserted by the joining kubelet — the plane holds no per-node owner record to check against, and this collector's charter is to work offline with no cloudbox feed. Under the dks tenancy model (dhnt/docs/dks-tenancy-model.md) a plane may admit workers joined by other owners; a dead node they labelled as an outpost k3s agent WILL eventually be reaped here. That is accepted and recorded in docs/cluster-peer.md: the plane host is the tenancy authority for its own plane, deletion additionally requires >grace of NotReady (a live tenant node can never be reaped via labels alone), and the Node object is cheap to re-mint by rejoining. What the filter must never do is delete the plane's OWN node or any control-plane node — both are excluded structurally, not by label trust.

This runs ONLY where the plane it manages runs: the control-plane host. It needs no cloudbox peer list or liveness feed — the node's own recorded Ready transition is the staleness clock.

Index

Constants

View Source
const (
	// EventDeleted records one real node deletion. These entries are
	// also the restart-safe rate limiter: the delete budget counts
	// EventDeleted lines inside the trailing window, so a daemon that
	// restarts ten times in ten minutes shares ONE budget instead of
	// minting ten.
	EventDeleted = "deleted"
	// EventDryRunDeleted records a node dry-run mode WOULD have
	// deleted. Never counted against the real budget.
	EventDryRunDeleted = "dry_run_deleted"
	// EventRefusedMassStale records a whole pass refused by the
	// cluster-wide unhealthy circuit breaker.
	EventRefusedMassStale = "refused_mass_stale"
	// EventRefusedClockSkew records a whole pass refused because a
	// node's readiness evidence was timestamped in our future.
	EventRefusedClockSkew = "refused_clock_skew"
)

Ledger events. Deletes and refusals both get a durable line: this collector destroys cluster state unattended, so "what did it do and why" must survive the daemon's own logs (which rotate, and which a self-restart truncates from view). Same precedent as the upgrade subsystem's upgrade.log.

View Source
const (
	// RuntimeLabel / BackendLabel / HostLabel are the node-identity
	// labels the k3s agent entrypoint stamps at registration
	// (internal/agent/runtime/image/entrypoint.sh), matching the
	// cloud-hosted control plane's vocabulary exactly.
	RuntimeLabel = "outpost.dhnt.io/runtime"
	BackendLabel = "outpost.dhnt.io/backend"
	HostLabel    = "outpost.dhnt.io/host"

	// RuntimeAgent / BackendK3s are the only label values GC acts on.
	// Virtual-kubelet nodes carry runtime=virtual and a vk-* backend;
	// they have no kubelet lease of their own worth reaping here.
	RuntimeAgent = "agent"
	BackendK3s   = "k3s"

	// ControlPlaneRoleLabel / MasterRoleLabel mark apiserver-hosting
	// nodes (k3s stamps the first on its server node). A node carrying
	// either is NEVER a GC candidate regardless of its other labels —
	// reaping the plane's own node is strictly worse than reaping a
	// worker, and role labels are at least plane-adjacent facts rather
	// than pure worker self-assertion.
	ControlPlaneRoleLabel = "node-role.kubernetes.io/control-plane"
	MasterRoleLabel       = "node-role.kubernetes.io/master"

	// DefaultGrace is how long a node must have been NotReady/Unknown
	// (per its Ready condition's LastTransitionTime) before it is
	// stale. A day comfortably clears laptop sleeps, reboots, and
	// multi-hour network outages — the node of a machine that comes
	// back inside the window reconnects and goes Ready again.
	DefaultGrace = 24 * time.Hour

	// DefaultMaxDeletes bounds deletes per rate window (see
	// DefaultInterval). GC converges over several windows instead of
	// mass-deleting in one, so one bad pass can only do bounded damage.
	DefaultMaxDeletes = 3

	// DefaultInterval is the pass cadence AND the delete-rate window:
	// at most MaxDeletes deletions per Interval of wall clock, counted
	// from the persisted ledger — not per pass, and not per process
	// lifetime. Outpost self-restarts on any builtin toggle; a budget
	// that reset with the process would multiply by restart count.
	DefaultInterval = time.Hour

	// DefaultSettle delays the FIRST pass after Run starts. Boot is
	// exactly when evidence is least trustworthy: the tunnel may not
	// be up, NTP may not have corrected a wrong RTC yet, and restart
	// storms (10 toggles == 10 boots) would otherwise each get an
	// immediate pass.
	DefaultSettle = 5 * time.Minute

	// DefaultMinUptime is how long the collector must have been
	// running — measured monotonically, immune to wall-clock jumps —
	// before it is allowed to DELETE. Staleness is wall-clock
	// arithmetic against apiserver-recorded timestamps; a host whose
	// RTC was wrong at boot (consumer hardware, restored VM snapshot,
	// dual-boot) computes garbage elapsed times until NTP fixes the
	// clock, and boot is also when the settle-window pass fires. An
	// hour of uptime — a meaningful fraction of the 24 h grace — means
	// NTP had ample time to sync before we trust the math. Passes
	// before that run read-only.
	DefaultMinUptime = time.Hour

	// DefaultMaxStaleFraction is the cluster-wide unhealthy circuit
	// breaker (upstream's --unhealthy-zone-threshold analog): when
	// MORE than this fraction of the observed outpost-agent population
	// is stale at once, the pass deletes nothing. "Everything looks
	// dead" nearly always means the observer broke — a >24 h partition
	// of the control-plane host flips every worker to Unknown with the
	// same LastTransitionTime, and without this guard GC would drain
	// the cluster at MaxDeletes per window until empty.
	DefaultMaxStaleFraction = 0.5
)

Variables

This section is empty.

Functions

func StaleSince

func StaleSince(node *corev1.Node, grace time.Duration, now time.Time) (time.Time, bool)

StaleSince reports whether node is a stale GC candidate at time now, and if so since when (its Ready condition's LastTransitionTime).

A candidate must be self-consistently an outpost k3s agent node: exact runtime/backend labels, a nonempty host label, and a name prefixed by that host — anything else (virtual, foreign, unlabelled, renamed) is out of scope. It must carry a Ready condition that is False or Unknown with a nonzero LastTransitionTime strictly older than grace; a missing condition or timestamp is unknown, not stale.

This is the pure per-node predicate. The Collector layers its structural exclusions (own host, control-plane role) and the fleet-level guards (mass-stale breaker, clock skew, rate budget) on top — see candidateSince and Once.

Types

type Collector

type Collector struct {
	Client     kubernetes.Interface
	Grace      time.Duration    // 0 => DefaultGrace
	MaxDeletes int              // 0 => DefaultMaxDeletes (per Interval window, not per pass)
	Interval   time.Duration    // 0 => DefaultInterval (pass cadence AND rate window)
	Log        *slog.Logger     // nil => slog.Default
	Now        func() time.Time // nil => time.Now

	// DryRun computes, logs, and ledgers exactly what a real pass
	// would delete — including budget and breaker decisions — but
	// never calls Delete.
	DryRun bool

	// SelfHost, when set, excludes any node whose host label claims
	// this host's own identity (the control-plane host may itself run
	// an agent runtime joined to its own plane). Wired from
	// FileConfig.ClusterNodeName().
	SelfHost string

	// MaxStaleFraction overrides DefaultMaxStaleFraction when > 0.
	// A value >= 1 disables the breaker (every fraction is <= 1).
	MaxStaleFraction float64

	// Settle overrides DefaultSettle when > 0: how long Run waits
	// before the first pass.
	Settle time.Duration

	// MinUptime overrides DefaultMinUptime when > 0: monotonic
	// collector uptime required before deletions are allowed.
	MinUptime time.Duration

	// Uptime overrides the monotonic-uptime source (tests). nil =>
	// time.Since(first Run/Once), which Go measures monotonically.
	Uptime func() time.Duration

	// Ledger is the durable JSONL record of deletions and refusals,
	// and the restart-safe half of the rate budget. nil => in-memory
	// budget only (tests / degraded boot); real wiring always sets it.
	Ledger *Ledger
	// contains filtered or unexported fields
}

Collector deletes stale agent Node objects from the hosted plane.

func (*Collector) Once

func (c *Collector) Once(ctx context.Context) error

Once performs a single bounded GC pass. Any List/Get/Delete error — and any failure to durably record a deletion — aborts the pass and is returned, never swallowed. Fleet-level refusals (mass-stale breaker, clock skew, insufficient uptime) are DECISIONS, not errors: they log, ledger where durable evidence matters, and return nil.

func (*Collector) Run

func (c *Collector) Run(ctx context.Context)

Run collects until ctx is cancelled. The first pass waits out the settle window — there is deliberately no immediate pass, so restart storms (outpost restarts on every builtin toggle) never reach a pass at all. Pass errors are logged, never fatal — the next tick retries from scratch.

type Ledger

type Ledger struct {
	// contains filtered or unexported fields
}

Ledger is an append-only JSONL writer at a fixed path, mirroring internal/agent/upgrade.Ledger. A nil Ledger (or empty path) is a valid no-op sink — but note Collector treats "no ledger" as "no durable budget", falling back to in-process memory only.

Single-writer by construction: exactly one outpost daemon runs per host (the pidfile gate in `start`), and only the control-plane host's collector writes this file — so O_APPEND line writes are sufficient and no inter-process lock is needed here (contrast the CNI IPAM ledger, which IS written by concurrent processes).

func NewLedger

func NewLedger(path string) *Ledger

NewLedger returns a Ledger backed by path. Doesn't touch the filesystem until the first Append.

func (*Ledger) Append

func (l *Ledger) Append(entry LedgerEntry) error

Append writes one entry as a single JSON line. A zero At is filled with the current time. Append errors on REFUSAL entries are logged by the caller and never block anything; a failure to record a DELETE is treated as fatal to the pass by the caller (Collector refuses to keep deleting when it cannot durably count deletions — the record is what bounds the rate across restarts).

func (*Ledger) CountDeletesSince

func (l *Ledger) CountDeletesSince(cutoff time.Time) (int, error)

CountDeletesSince returns how many real deletions (EventDeleted) the ledger records strictly after cutoff. A missing file is zero deletions, not an error. Malformed lines are skipped — a corrupt line can therefore UNDER-count by at most the lines lost, which is bounded by MaxDeletes per window and accepted; any other read error is returned and the caller must fail safe (refuse deletes).

func (*Ledger) Path

func (l *Ledger) Path() string

Path is exposed so callers can include it in diagnostics.

func (*Ledger) Tail

func (l *Ledger) Tail(n int) ([]LedgerEntry, error)

Tail returns up to the last n entries, newest last. Missing file returns an empty slice. Mirrors upgrade.Ledger.Tail for a future `outpost cluster nodegc history` surface.

type LedgerEntry

type LedgerEntry struct {
	At            time.Time `json:"at"`
	Event         string    `json:"event"`
	Node          string    `json:"node,omitempty"`
	UID           string    `json:"uid,omitempty"`
	NotReadySince string    `json:"not_ready_since,omitempty"`
	Stale         int       `json:"stale,omitempty"`
	Scope         int       `json:"scope,omitempty"`
	Detail        string    `json:"detail,omitempty"`
}

LedgerEntry is one JSON line in the nodegc history file.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL