infra

package
v1.801.472 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 5, 2026 License: Apache-2.0 Imports: 21 Imported by: 0

Documentation

Overview

Package infra is the platform's DigitalOcean fleet board: the physical inventory (DOKS clusters, droplets, block-storage volumes, load balancers) cross-referenced against what every cluster's Kubernetes actually claims, with the cost of each and an orphan analysis that is safe BY CONSTRUCTION.

THE RULE THIS PACKAGE EXISTS TO ENFORCE. A volume being detached, or carrying a `k8s:<cluster-uuid>` tag for some other cluster, does NOT make it garbage. Those two signals together would have condemned 4.39 TiB of live data belonging to running clusters. The ONLY sound liveness test is a cross-reference against the `spec.csi.volumeHandle` of every PersistentVolume in EVERY cluster — and it is only a valid test when every cluster answered. So:

  • a volume is deletable only when NO PV in ANY cluster names it, and
  • if even one cluster failed to scan, NOTHING is deletable (Complete=false).

Absence of evidence is not evidence of absence: an unreachable cluster is treated as a cluster that might be holding the volume. The analysis fails CLOSED.

"No pod mounts it" is a REVIEW signal, never a delete signal — an idle Bound PVC is an idle database, not garbage. Idle volumes are surfaced as a queue for a human and are never counted as reclaimable.

THE SAME DISCIPLINE GOVERNS EVERY MUTATION, not just volume deletion. A droplet, a load balancer and a node pool each get a (allowed, reason) verdict derived HERE, from the same scan, behind the same completeness gate — see Snapshot.verdict. Handlers carry no policy: they read the verdict this file already reached.

Analyze is a pure function of (DO inventory, cluster scans) so every rule above is unit-testable without a network or a cluster.

Index

Constants

View Source
const (
	// StateAttached — DO reports the volume attached to a droplet. In use, now.
	StateAttached = "attached"
	// StateBound — detached, but a PV in some cluster claims it and that PV is Bound
	// to a PVC. This is live data between mounts; deleting it destroys a database.
	StateBound = "bound"
	// StateReleased — a PV claims it but is no longer Bound (Released/Available/Failed).
	// A genuine cleanup candidate, but the PV still exists, so a human retires the PV.
	StateReleased = "released"
	// StateUnreferenced — no PV in ANY scanned cluster names it. The ONLY deletable state.
	StateUnreferenced = "unreferenced"
)

Volume states. The state machine is total and ordered: attachment beats reference, reference beats absence. Only Unreferenced is ever deletable.

View Source
const (
	SevCritical = "critical"
	SevWarn     = "warn"
	SevInfo     = "info"
)

Finding severities.

Variables

This section is empty.

Functions

func ExpandPVC

func ExpandPVC(ctx context.Context, do *digitalocean.Client, clusterID, ns, name string, gib int) error

ExpandPVC grows a PersistentVolumeClaim.

This is the ONE correct way to grow a volume Kubernetes manages: the resize controller acts on the CLAIM, growing the backing cloud device and then the filesystem on it, so claim, PV, device and filesystem all end up agreeing. Growing the device through the DigitalOcean API instead would leave the claim and the PV declaring the old capacity and the filesystem never grown at all — three sources of truth, two of them wrong.

The StorageClass must set allowVolumeExpansion; when it does not, the apiserver refuses the patch and its message is surfaced verbatim rather than worked around.

func Routes

func Routes(z *zip.App, s *cloud.Service[core.State])

Routes registers the DigitalOcean infrastructure board. SuperAdmin only: this is the whole account's physical inventory and the controls that destroy parts of it.

NOTE ON THE NOUN: this is INFRASTRUCTURE — droplets, volumes, DOKS clusters, load balancers. The pre-existing /v1/fleet surface is compute workers and jobs. Different nouns, deliberately not merged.

func SetSchedulable

func SetSchedulable(ctx context.Context, do *digitalocean.Client, clusterID, node string, schedulable, drain bool) (int, error)

SetSchedulable cordons or uncordons a node, optionally draining it. Returns the number of pods evicted.

Drain uses the Eviction API, not delete: eviction respects PodDisruptionBudgets, so a drain that would break a quorum is REFUSED by the apiserver rather than silently taking a service down. DaemonSet and mirror pods are skipped — they are rescheduled onto the same node by definition and evicting them is a no-op loop.

Types

type Cluster

type Cluster struct {
	ID           string      `json:"id"`
	Name         string      `json:"name"`
	Region       string      `json:"region"`
	Version      string      `json:"version"`
	Status       string      `json:"status"`
	NodePools    int         `json:"nodePools"`
	Pools        []NodePool  `json:"pools"`
	Nodes        int         `json:"nodes"`
	Pods         int         `json:"pods"`
	PVs          int         `json:"pvs"`
	PVCs         int         `json:"pvcs"`
	IdlePVCs     int         `json:"idlePVCs"`
	Scanned      bool        `json:"scanned"`
	ScanError    string      `json:"scanError"`
	MonthlyCents money.Cents `json:"monthlyCents"`
}

Cluster is one DOKS cluster with its scanned Kubernetes rollup.

type ClusterScan

type ClusterScan struct {
	ClusterID string
	Err       error
	Nodes     []NodeState
	PVs       []PVRef
	PVCs      []PVCRef
	Pods      []PodRef
	Services  []ServiceRef
	// Usage is fill, per claim. PARTIAL BY NATURE and never an error: a claim absent here
	// was not measured, which this analysis reports as unknown rather than as zero.
	Usage []VolumeUsage
}

ClusterScan is ONE cluster's Kubernetes truth. Err non-nil means the cluster did not answer — which forces the whole analysis incomplete.

func Scan

func Scan(ctx context.Context, do *digitalocean.Client, clusters []digitalocean.Cluster) []ClusterScan

Scan reads every cluster's Kubernetes state, bounded-parallel. It ALWAYS returns one row per cluster: a cluster that failed comes back with Err set rather than being omitted, because a missing row and a healthy row must never be confusable — that confusion is exactly what would condemn live data.

type CordonIn

type CordonIn struct {
	// ID is the node's droplet id, from the path.
	ID string `json:"id"`
	// Cordon true marks the node unschedulable; false restores it.
	Cordon bool `json:"cordon"`
	// Drain additionally evicts the pods already running there.
	Drain bool `json:"drain"`
}

CordonIn addresses one cluster node by its droplet id.

type Cost

type Cost struct {
	DropletsMonthly      money.Cents `json:"dropletsMonthly"`
	VolumesMonthly       money.Cents `json:"volumesMonthly"`
	LoadBalancersMonthly money.Cents `json:"loadBalancersMonthly"`
	TotalMonthly         money.Cents `json:"totalMonthly"`
	ReclaimableMonthly   money.Cents `json:"reclaimableMonthly"`
	// WastedMonthly is what the fleet pays every month for provisioned-but-empty space on
	// the volumes a kubelet actually measured.
	//
	// It is NOT ReclaimableMonthly and must never be added to it. Reclaimable is money a
	// button on this board collects, by deleting volumes proven to belong to no one.
	// Wasted is money locked inside volumes that are IN USE and holding live data:
	// DigitalOcean can only ever grow a volume, so collecting it means copying a database
	// onto a smaller one. See shrinkRecipe.
	//
	// It is also a LOWER BOUND — unmeasured volumes contribute nothing.
	WastedMonthly money.Cents `json:"wastedMonthly"`
}

Cost is monthly spend in cents. Reclaimable counts ONLY unreferenced volumes — never idle ones, which are live data awaiting a human verdict.

type DropletIn

type DropletIn struct {
	// ID is the DO droplet id, from the path. Numeric.
	ID string `json:"id"`
	// Size is the target DigitalOcean size slug on resize, e.g. "s-4vcpu-8gb".
	Size string `json:"size"`
	// Disk requests a PERMANENT resize that grows the disk. DO can never resize such a
	// droplet down again, so it defaults false — a CPU/RAM-only change, reversible.
	Disk bool `json:"disk"`
}

DropletIn addresses one droplet, optionally with a resize.

type Finding

type Finding struct {
	ID           string      `json:"id"`
	Severity     string      `json:"severity"`
	Kind         string      `json:"kind"`
	Title        string      `json:"title"`
	Detail       string      `json:"detail"`
	Resource     string      `json:"resource"`
	Cluster      string      `json:"cluster"`
	MonthlyCents money.Cents `json:"monthlyCents"`
}

Finding is one audit result — the "is anything bad" surface.

type Inventory

type Inventory struct {
	Clusters      []digitalocean.Cluster
	Droplets      []digitalocean.Droplet
	Volumes       []digitalocean.Volume
	LoadBalancers []digitalocean.LoadBalancer
}

Inventory is the DigitalOcean account read — the half of the analysis input that needs no cluster.

type LoadBalancer

type LoadBalancer struct {
	ID           string      `json:"id"`
	Name         string      `json:"name"`
	Region       string      `json:"region"`
	Status       string      `json:"status"`
	IP           string      `json:"ip"`
	SizeUnit     int         `json:"sizeUnit"`
	MonthlyCents money.Cents `json:"monthlyCents"`
	Droplets     int         `json:"droplets"`
	Cluster      string      `json:"cluster"`
	// Service is the `namespace/name` of the live type=LoadBalancer Service that claims
	// this load balancer, proven from the cluster scan. Non-empty means IN USE.
	Service       string `json:"service"`
	Deletable     bool   `json:"deletable"`
	BlockedReason string `json:"blockedReason"`
}

LoadBalancer is one DO load balancer, attributed to a cluster via the Service that claims it, or failing that via its member droplets.

type LoadBalancerIn

type LoadBalancerIn struct {
	// ID is the DO load balancer id, from the path.
	ID string `json:"id"`
}

LoadBalancerIn addresses one load balancer.

type MutationOut

type MutationOut struct {
	Status string `json:"status"`
	Msg    string `json:"msg"`
	Data   any    `json:"data"`
}

MutationOut is the envelope EVERY infra change answers with. One type, because there is one mutation discipline (run) behind all of them: re-scan, check the fresh verdict, apply, audit.

`data` is the per-action result and is declared opaque — its keys differ by action and each handler's doc comment names them. On a refusal or a failure it is null and msg says why; a refusal reads "refusing: <reason>" and means the board proved the change unsafe, which is different from the change being attempted and failing.

type Node

type Node struct {
	ID           int         `json:"id"`
	Name         string      `json:"name"`
	Cluster      string      `json:"cluster"`
	ClusterID    string      `json:"clusterId"`
	Region       string      `json:"region"`
	Status       string      `json:"status"`
	SizeSlug     string      `json:"sizeSlug"`
	VCPUs        int         `json:"vcpus"`
	MemoryMiB    int         `json:"memoryMiB"`
	LocalDiskGiB int         `json:"localDiskGiB"`
	MonthlyCents money.Cents `json:"monthlyCents"`
	CreatedAt    string      `json:"createdAt"`
	PrivateIP    string      `json:"privateIp"`
	PublicIP     string      `json:"publicIp"`
	Tags         []string    `json:"tags"`
	Ready        bool        `json:"ready"`
	Schedulable  bool        `json:"schedulable"`
	Pods         int         `json:"pods"`
	Volumes      int         `json:"volumes"`
	// Mutable reports whether this droplet may be changed DIRECTLY — deleted or resized.
	// One predicate covers both because one fact decides both: a DOKS node belongs to a
	// node pool, and the pool is the only thing allowed to change it.
	Mutable       bool   `json:"mutable"`
	BlockedReason string `json:"blockedReason"`
}

Node is one droplet, joined to the Kubernetes node of the same name.

type NodePool

type NodePool struct {
	ID                 string `json:"id"`
	Name               string `json:"name"`
	Size               string `json:"size"`
	Count              int    `json:"count"`
	ClusterID          string `json:"clusterId"`
	Cluster            string `json:"cluster"`
	ClusterSchedulable int    `json:"clusterSchedulable"`
	Scalable           bool   `json:"scalable"`
	BlockedReason      string `json:"blockedReason"`
}

NodePool is one DOKS node pool — the only correct place to change a cluster's node count. ClusterSchedulable is the whole cluster's schedulable-and-ready node count, carried on the row so the shrink verdict is a pure method of it.

func (NodePool) ScaleTo

func (p NodePool) ScaleTo(count int) (bool, string)

ScaleTo answers whether this pool may be set to count.

PROVEN HERE: a pool keeps at least one node, and a shrink never leaves the cluster with zero schedulable nodes — with none, every pod the removed nodes carried is unschedulable, guaranteed.

NOT PROVEN, and deliberately not pretended: DOKS picks WHICH nodes it removes, so no particular pod can be shown to survive a shrink that leaves capacity behind. Node affinity, taints and resource requests decide that, and PodDisruptionBudgets are enforced by the cluster during DOKS's own drain — not by this board. A shrink that merely MIGHT not fit is allowed, and the response says so.

type NodeState

type NodeState struct {
	Name        string
	Ready       bool
	Schedulable bool
}

NodeState is one Kubernetes node's own view of itself.

type PVCRef

type PVCRef struct {
	Namespace string
	Name      string
	Phase     string
	Volume    string
}

PVCRef is one PersistentVolumeClaim.

type PVRef

type PVRef struct {
	Name         string
	Phase        string
	VolumeHandle string
	ClaimNS      string
	ClaimName    string
}

PVRef is one PersistentVolume's identity: which DO volume it claims, and whether that claim is still live.

type PodRef

type PodRef struct {
	Namespace string
	Name      string
	Phase     string
	Reason    string
	Node      string
	Claims    []string
	Images    []string
	// Controller is the controlling owner as `Kind/Name` ("StatefulSet/luxd"), or "" for
	// a bare pod. Its kind decides how this pod's volumes can be right-sized.
	Controller string
}

PodRef is one pod, reduced to what the board needs: where it runs, whether it is healthy, which PVCs it mounts, what images it runs, and the workload that controls it.

type ReadIn

type ReadIn struct {
	// Refresh, when present, forces a full re-scan instead of serving the cached
	// snapshot. Every MUTATION re-scans regardless — this is only for the reader.
	Refresh string `json:"refresh"`
}

ReadIn is the GET /v1/admin/infra query.

type ReadOut

type ReadOut struct {
	Status string    `json:"status"`
	Msg    string    `json:"msg"`
	Data   *Snapshot `json:"data"`
}

ReadOut is the GET /v1/admin/infra envelope.

type ScaleIn

type ScaleIn struct {
	// ID is the DOKS cluster id, from the path.
	ID string `json:"id"`
	// Pool is the node pool, from the path. Its DO id or its name — both are unique
	// within a cluster, and an operator reads the name off the board.
	Pool string `json:"pool"`
	// Count is the node count to set.
	Count int `json:"count"`
}

ScaleIn addresses one node pool and the count to set.

type ServiceRef

type ServiceRef struct {
	Namespace string
	Name      string
	LBID      string
	IPs       []string
}

ServiceRef is one Service of type LoadBalancer: the identities by which it claims a DO load balancer. It is to load balancers exactly what PVRef is to volumes — the only sound liveness test, because a DO load balancer carries no back-reference of its own.

type Snapshot

type Snapshot struct {
	At               string              `json:"at"`
	Complete         bool                `json:"complete"`
	IncompleteReason string              `json:"incompleteReason"`
	Sources          []core.SourceStatus `json:"sources"`
	Totals           Totals              `json:"totals"`
	Cost             Cost                `json:"cost"`
	Clusters         []Cluster           `json:"clusters"`
	Nodes            []Node              `json:"nodes"`
	Volumes          []Volume            `json:"volumes"`
	LoadBalancers    []LoadBalancer      `json:"loadBalancers"`
	Findings         []Finding           `json:"findings"`
}

Snapshot is the whole board in one value.

func Analyze

func Analyze(inv Inventory, scans []ClusterScan, sources []core.SourceStatus, at time.Time) Snapshot

Analyze folds the DO inventory and the per-cluster Kubernetes scans into the board. PURE: no clock, no network, no cluster — `at` is passed in so the result is byte-reproducible in tests.

type Totals

type Totals struct {
	Clusters            int `json:"clusters"`
	Nodes               int `json:"nodes"`
	Volumes             int `json:"volumes"`
	LoadBalancers       int `json:"loadBalancers"`
	VolumeGiB           int `json:"volumeGiB"`
	AttachedVolumes     int `json:"attachedVolumes"`
	AttachedGiB         int `json:"attachedGiB"`
	DetachedVolumes     int `json:"detachedVolumes"`
	DetachedGiB         int `json:"detachedGiB"`
	UnreferencedVolumes int `json:"unreferencedVolumes"`
	UnreferencedGiB     int `json:"unreferencedGiB"`
	IdlePVCs            int `json:"idlePVCs"`
	LocalDiskGiB        int `json:"localDiskGiB"`

	// Fill. MeasuredVolumes/UnmeasuredVolumes are the honesty denominator: UsedGiB and
	// WastedGiB describe the measured set ONLY, so a board showing waste must show how
	// much of the fleet the figure was computed from. Unmeasured capacity contributes
	// nothing to either — it is not assumed empty, and it is not assumed full.
	MeasuredVolumes   int `json:"measuredVolumes"`
	MeasuredGiB       int `json:"measuredGiB"`
	UnmeasuredVolumes int `json:"unmeasuredVolumes"`
	UnmeasuredGiB     int `json:"unmeasuredGiB"`
	UsedGiB           int `json:"usedGiB"`
	WastedGiB         int `json:"wastedGiB"`
}

Totals are fleet counts. LocalDiskGiB is broken out precisely so it can be shown as NOT separately billed.

type Volume

type Volume struct {
	ID           string      `json:"id"`
	Name         string      `json:"name"`
	Region       string      `json:"region"`
	SizeGiB      int         `json:"sizeGiB"`
	MonthlyCents money.Cents `json:"monthlyCents"`
	State        string      `json:"state"`
	DropletIDs   []int       `json:"dropletIds"`
	NodeName     string      `json:"nodeName"`
	// Cluster/ClusterID are the PROVEN owner — resolved through a PV that names this
	// volume, never through the tag.
	Cluster   string `json:"cluster"`
	ClusterID string `json:"clusterId"`
	// TagCluster is the `k8s:<uuid>` tag. ADVISORY ONLY: it outlives the cluster that
	// set it. Shown so the operator can see tag-vs-truth disagree, never acted on.
	TagCluster   string   `json:"tagCluster"`
	PV           string   `json:"pv"`
	PVPhase      string   `json:"pvPhase"`
	PVCNamespace string   `json:"pvcNamespace"`
	PVCName      string   `json:"pvcName"`
	MountedBy    []string `json:"mountedBy"`
	Idle         bool     `json:"idle"`
	CreatedAt    string   `json:"createdAt"`
	// Controller is the workload owning the pod that mounts this volume
	// ("StatefulSet/luxd"), or "" when nothing mounts it. It names who has to act.
	Controller string `json:"controller"`

	// HasUsage reports whether a kubelet actually MEASURED this volume's filesystem.
	//
	// False means NOT MEASURED. It does NOT mean empty, and the three fields below are
	// meaningless — not zero — when it is false. A reading exists only while a running pod
	// has the volume mounted on a node that answered; a detached, idle or unreferenced
	// volume has none. Rendering an unmeasured volume as "0 used / 100% wasted" would
	// invent the single most expensive lie this board could tell, so every consumer must
	// branch on this flag and show unknown.
	HasUsage bool `json:"hasUsage"`
	// UsedBytes is the measured filesystem usage. BYTES, not GiB: the volumes this exists
	// to catch hold a fraction of a GiB in 200, and rounding that to an integer GiB would
	// print the very 0 the flag above exists to prevent.
	UsedBytes int64 `json:"usedBytes"`
	// WastedGiB is provisioned minus measured, in the unit DigitalOcean BILLS: whole GiB
	// of the volume's own size, never the filesystem's capacity — a 200 GiB volume carries
	// a 196 GiB filesystem after format overhead, and the invoice says 200.
	WastedGiB          int         `json:"wastedGiB"`
	WastedMonthlyCents money.Cents `json:"wastedMonthlyCents"`

	Deletable     bool   `json:"deletable"`
	BlockedReason string `json:"blockedReason"`
	// Expandable/ExpandBlockedReason are the GROW verdict, kept separate from Deletable
	// because the two ask opposite questions: a volume is deletable when nothing uses it,
	// and expandable when something uses it in a way this board can grow completely.
	Expandable          bool   `json:"expandable"`
	ExpandBlockedReason string `json:"expandBlockedReason"`
}

Volume is one block-storage volume with its PROVEN cluster ownership.

func (Volume) ExpandTo

func (v Volume) ExpandTo(gib int) (bool, string)

ExpandTo answers whether this volume may be grown to gib, mirroring NodePool.ScaleTo: the row carries the standing verdict, and the method judges the number asked for.

GROW ONLY. DigitalOcean can never shrink a block-storage volume, so a smaller target is not a slow operation — it is an impossible one, and pretending otherwise behind a button is how a chain node loses its data. Shrinking is a migration; see shrinkRecipe.

type VolumeIn

type VolumeIn struct {
	// ID is the DO volume id, from the path.
	ID string `json:"id"`
	// Snapshot is the snapshot-first switch on DELETE. Anything other than the literal
	// "false" snapshots before destroying — the snapshot IS the undo, so waiving it is
	// deliberate and explicit.
	Snapshot string `json:"snapshot"`
	// Name is the snapshot name on the snapshot action. Blank gets a deterministic
	// "<volume>-predelete-<unix>" so the undo is findable in the DO console.
	Name string `json:"name"`
	// SizeGiB is the target size on the resize action. A volume only ever grows —
	// ExpandTo is the verdict that refuses a shrink, so this is not validated here.
	SizeGiB int `json:"sizeGiB"`
}

VolumeIn addresses one DigitalOcean volume.

type VolumeSnapshotOut

type VolumeSnapshotOut struct {
	Status string                 `json:"status"`
	Msg    string                 `json:"msg"`
	Data   *digitalocean.Snapshot `json:"data"`
}

VolumeSnapshotOut is the POST /v1/admin/infra/volumes/:id/snapshot envelope. It is the one infra change with a typed result, because DO returns a real snapshot object.

type VolumeUsage

type VolumeUsage struct {
	Namespace string
	Name      string
	UsedBytes int64
}

VolumeUsage is one PersistentVolumeClaim's REAL filesystem usage, as measured by the kubelet that has it mounted.

A reading exists ONLY for a volume a running pod has mounted on a node that answered. Everything else has NO reading — a different fact from "empty", carried as such all the way to the screen. See Volume.HasUsage.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL