Documentation
¶
Overview ¶
Package infra is the platform's DigitalOcean fleet board: the physical inventory (DOKS clusters, droplets, block-storage volumes, load balancers) cross-referenced against what every cluster's Kubernetes actually claims, with the cost of each and an orphan analysis that is safe BY CONSTRUCTION.
THE RULE THIS PACKAGE EXISTS TO ENFORCE. A volume being detached, or carrying a `k8s:<cluster-uuid>` tag for some other cluster, does NOT make it garbage. Those two signals together would have condemned 4.39 TiB of live data belonging to running clusters. The ONLY sound liveness test is a cross-reference against the `spec.csi.volumeHandle` of every PersistentVolume in EVERY cluster — and it is only a valid test when every cluster answered. So:
- a volume is deletable only when NO PV in ANY cluster names it, and
- if even one cluster failed to scan, NOTHING is deletable (Complete=false).
Absence of evidence is not evidence of absence: an unreachable cluster is treated as a cluster that might be holding the volume. The analysis fails CLOSED.
"No pod mounts it" is a REVIEW signal, never a delete signal — an idle Bound PVC is an idle database, not garbage. Idle volumes are surfaced as a queue for a human and are never counted as reclaimable.
THE SAME DISCIPLINE GOVERNS EVERY MUTATION, not just volume deletion. A droplet, a load balancer and a node pool each get a (allowed, reason) verdict derived HERE, from the same scan, behind the same completeness gate — see Snapshot.verdict. Handlers carry no policy: they read the verdict this file already reached.
Analyze is a pure function of (DO inventory, cluster scans) so every rule above is unit-testable without a network or a cluster.
Index ¶
- Constants
- func ExpandPVC(ctx context.Context, do *digitalocean.Client, clusterID, ns, name string, ...) error
- func Routes(z *zip.App, s *cloud.Service[core.State])
- func SetSchedulable(ctx context.Context, do *digitalocean.Client, clusterID, node string, ...) (int, error)
- type Cluster
- type ClusterScan
- type CordonIn
- type Cost
- type DropletIn
- type Finding
- type Inventory
- type LoadBalancer
- type LoadBalancerIn
- type MutationOut
- type Node
- type NodePool
- type NodeState
- type PVCRef
- type PVRef
- type PodRef
- type ReadIn
- type ReadOut
- type ScaleIn
- type ServiceRef
- type Snapshot
- type Totals
- type Volume
- type VolumeIn
- type VolumeSnapshotOut
- type VolumeUsage
Constants ¶
const ( // StateAttached — DO reports the volume attached to a droplet. In use, now. StateAttached = "attached" // StateBound — detached, but a PV in some cluster claims it and that PV is Bound // to a PVC. This is live data between mounts; deleting it destroys a database. StateBound = "bound" // StateReleased — a PV claims it but is no longer Bound (Released/Available/Failed). // A genuine cleanup candidate, but the PV still exists, so a human retires the PV. StateReleased = "released" // StateUnreferenced — no PV in ANY scanned cluster names it. The ONLY deletable state. StateUnreferenced = "unreferenced" )
Volume states. The state machine is total and ordered: attachment beats reference, reference beats absence. Only Unreferenced is ever deletable.
const ( SevCritical = "critical" SevWarn = "warn" SevInfo = "info" )
Finding severities.
Variables ¶
This section is empty.
Functions ¶
func ExpandPVC ¶
func ExpandPVC(ctx context.Context, do *digitalocean.Client, clusterID, ns, name string, gib int) error
ExpandPVC grows a PersistentVolumeClaim.
This is the ONE correct way to grow a volume Kubernetes manages: the resize controller acts on the CLAIM, growing the backing cloud device and then the filesystem on it, so claim, PV, device and filesystem all end up agreeing. Growing the device through the DigitalOcean API instead would leave the claim and the PV declaring the old capacity and the filesystem never grown at all — three sources of truth, two of them wrong.
The StorageClass must set allowVolumeExpansion; when it does not, the apiserver refuses the patch and its message is surfaced verbatim rather than worked around.
func Routes ¶
Routes registers the DigitalOcean infrastructure board. SuperAdmin only: this is the whole account's physical inventory and the controls that destroy parts of it.
NOTE ON THE NOUN: this is INFRASTRUCTURE — droplets, volumes, DOKS clusters, load balancers. The pre-existing /v1/fleet surface is compute workers and jobs. Different nouns, deliberately not merged.
func SetSchedulable ¶
func SetSchedulable(ctx context.Context, do *digitalocean.Client, clusterID, node string, schedulable, drain bool) (int, error)
SetSchedulable cordons or uncordons a node, optionally draining it. Returns the number of pods evicted.
Drain uses the Eviction API, not delete: eviction respects PodDisruptionBudgets, so a drain that would break a quorum is REFUSED by the apiserver rather than silently taking a service down. DaemonSet and mirror pods are skipped — they are rescheduled onto the same node by definition and evicting them is a no-op loop.
Types ¶
type Cluster ¶
type Cluster struct {
ID string `json:"id"`
Name string `json:"name"`
Region string `json:"region"`
Version string `json:"version"`
Status string `json:"status"`
NodePools int `json:"nodePools"`
Pools []NodePool `json:"pools"`
Nodes int `json:"nodes"`
Pods int `json:"pods"`
PVs int `json:"pvs"`
PVCs int `json:"pvcs"`
IdlePVCs int `json:"idlePVCs"`
Scanned bool `json:"scanned"`
ScanError string `json:"scanError"`
MonthlyCents money.Cents `json:"monthlyCents"`
}
Cluster is one DOKS cluster with its scanned Kubernetes rollup.
type ClusterScan ¶
type ClusterScan struct {
ClusterID string
Err error
Nodes []NodeState
PVs []PVRef
PVCs []PVCRef
Pods []PodRef
Services []ServiceRef
// Usage is fill, per claim. PARTIAL BY NATURE and never an error: a claim absent here
// was not measured, which this analysis reports as unknown rather than as zero.
Usage []VolumeUsage
}
ClusterScan is ONE cluster's Kubernetes truth. Err non-nil means the cluster did not answer — which forces the whole analysis incomplete.
func Scan ¶
func Scan(ctx context.Context, do *digitalocean.Client, clusters []digitalocean.Cluster) []ClusterScan
Scan reads every cluster's Kubernetes state, bounded-parallel. It ALWAYS returns one row per cluster: a cluster that failed comes back with Err set rather than being omitted, because a missing row and a healthy row must never be confusable — that confusion is exactly what would condemn live data.
type CordonIn ¶
type CordonIn struct {
// ID is the node's droplet id, from the path.
ID string `json:"id"`
// Cordon true marks the node unschedulable; false restores it.
Cordon bool `json:"cordon"`
// Drain additionally evicts the pods already running there.
Drain bool `json:"drain"`
}
CordonIn addresses one cluster node by its droplet id.
type Cost ¶
type Cost struct {
DropletsMonthly money.Cents `json:"dropletsMonthly"`
VolumesMonthly money.Cents `json:"volumesMonthly"`
LoadBalancersMonthly money.Cents `json:"loadBalancersMonthly"`
TotalMonthly money.Cents `json:"totalMonthly"`
ReclaimableMonthly money.Cents `json:"reclaimableMonthly"`
// WastedMonthly is what the fleet pays every month for provisioned-but-empty space on
// the volumes a kubelet actually measured.
//
// It is NOT ReclaimableMonthly and must never be added to it. Reclaimable is money a
// button on this board collects, by deleting volumes proven to belong to no one.
// Wasted is money locked inside volumes that are IN USE and holding live data:
// DigitalOcean can only ever grow a volume, so collecting it means copying a database
// onto a smaller one. See shrinkRecipe.
//
// It is also a LOWER BOUND — unmeasured volumes contribute nothing.
WastedMonthly money.Cents `json:"wastedMonthly"`
}
Cost is monthly spend in cents. Reclaimable counts ONLY unreferenced volumes — never idle ones, which are live data awaiting a human verdict.
type DropletIn ¶
type DropletIn struct {
// ID is the DO droplet id, from the path. Numeric.
ID string `json:"id"`
// Size is the target DigitalOcean size slug on resize, e.g. "s-4vcpu-8gb".
Size string `json:"size"`
// Disk requests a PERMANENT resize that grows the disk. DO can never resize such a
// droplet down again, so it defaults false — a CPU/RAM-only change, reversible.
Disk bool `json:"disk"`
}
DropletIn addresses one droplet, optionally with a resize.
type Finding ¶
type Finding struct {
ID string `json:"id"`
Severity string `json:"severity"`
Kind string `json:"kind"`
Title string `json:"title"`
Detail string `json:"detail"`
Resource string `json:"resource"`
Cluster string `json:"cluster"`
MonthlyCents money.Cents `json:"monthlyCents"`
}
Finding is one audit result — the "is anything bad" surface.
type Inventory ¶
type Inventory struct {
Clusters []digitalocean.Cluster
Droplets []digitalocean.Droplet
Volumes []digitalocean.Volume
LoadBalancers []digitalocean.LoadBalancer
}
Inventory is the DigitalOcean account read — the half of the analysis input that needs no cluster.
type LoadBalancer ¶
type LoadBalancer struct {
ID string `json:"id"`
Name string `json:"name"`
Region string `json:"region"`
Status string `json:"status"`
IP string `json:"ip"`
SizeUnit int `json:"sizeUnit"`
MonthlyCents money.Cents `json:"monthlyCents"`
Droplets int `json:"droplets"`
Cluster string `json:"cluster"`
// Service is the `namespace/name` of the live type=LoadBalancer Service that claims
// this load balancer, proven from the cluster scan. Non-empty means IN USE.
Service string `json:"service"`
Deletable bool `json:"deletable"`
BlockedReason string `json:"blockedReason"`
}
LoadBalancer is one DO load balancer, attributed to a cluster via the Service that claims it, or failing that via its member droplets.
type LoadBalancerIn ¶
type LoadBalancerIn struct {
// ID is the DO load balancer id, from the path.
ID string `json:"id"`
}
LoadBalancerIn addresses one load balancer.
type MutationOut ¶
type MutationOut struct {
Status string `json:"status"`
Msg string `json:"msg"`
Data any `json:"data"`
}
MutationOut is the envelope EVERY infra change answers with. One type, because there is one mutation discipline (run) behind all of them: re-scan, check the fresh verdict, apply, audit.
`data` is the per-action result and is declared opaque — its keys differ by action and each handler's doc comment names them. On a refusal or a failure it is null and msg says why; a refusal reads "refusing: <reason>" and means the board proved the change unsafe, which is different from the change being attempted and failing.
type Node ¶
type Node struct {
ID int `json:"id"`
Name string `json:"name"`
Cluster string `json:"cluster"`
ClusterID string `json:"clusterId"`
Region string `json:"region"`
Status string `json:"status"`
SizeSlug string `json:"sizeSlug"`
VCPUs int `json:"vcpus"`
MemoryMiB int `json:"memoryMiB"`
LocalDiskGiB int `json:"localDiskGiB"`
MonthlyCents money.Cents `json:"monthlyCents"`
CreatedAt string `json:"createdAt"`
PrivateIP string `json:"privateIp"`
PublicIP string `json:"publicIp"`
Tags []string `json:"tags"`
Ready bool `json:"ready"`
Schedulable bool `json:"schedulable"`
Pods int `json:"pods"`
Volumes int `json:"volumes"`
// Mutable reports whether this droplet may be changed DIRECTLY — deleted or resized.
// One predicate covers both because one fact decides both: a DOKS node belongs to a
// node pool, and the pool is the only thing allowed to change it.
Mutable bool `json:"mutable"`
BlockedReason string `json:"blockedReason"`
}
Node is one droplet, joined to the Kubernetes node of the same name.
type NodePool ¶
type NodePool struct {
ID string `json:"id"`
Name string `json:"name"`
Size string `json:"size"`
Count int `json:"count"`
ClusterID string `json:"clusterId"`
Cluster string `json:"cluster"`
ClusterSchedulable int `json:"clusterSchedulable"`
Scalable bool `json:"scalable"`
BlockedReason string `json:"blockedReason"`
}
NodePool is one DOKS node pool — the only correct place to change a cluster's node count. ClusterSchedulable is the whole cluster's schedulable-and-ready node count, carried on the row so the shrink verdict is a pure method of it.
func (NodePool) ScaleTo ¶
ScaleTo answers whether this pool may be set to count.
PROVEN HERE: a pool keeps at least one node, and a shrink never leaves the cluster with zero schedulable nodes — with none, every pod the removed nodes carried is unschedulable, guaranteed.
NOT PROVEN, and deliberately not pretended: DOKS picks WHICH nodes it removes, so no particular pod can be shown to survive a shrink that leaves capacity behind. Node affinity, taints and resource requests decide that, and PodDisruptionBudgets are enforced by the cluster during DOKS's own drain — not by this board. A shrink that merely MIGHT not fit is allowed, and the response says so.
type PVRef ¶
PVRef is one PersistentVolume's identity: which DO volume it claims, and whether that claim is still live.
type PodRef ¶
type PodRef struct {
Namespace string
Name string
Phase string
Reason string
Node string
Claims []string
Images []string
// Controller is the controlling owner as `Kind/Name` ("StatefulSet/luxd"), or "" for
// a bare pod. Its kind decides how this pod's volumes can be right-sized.
Controller string
}
PodRef is one pod, reduced to what the board needs: where it runs, whether it is healthy, which PVCs it mounts, what images it runs, and the workload that controls it.
type ReadIn ¶
type ReadIn struct {
// Refresh, when present, forces a full re-scan instead of serving the cached
// snapshot. Every MUTATION re-scans regardless — this is only for the reader.
Refresh string `json:"refresh"`
}
ReadIn is the GET /v1/admin/infra query.
type ReadOut ¶
type ReadOut struct {
Status string `json:"status"`
Msg string `json:"msg"`
Data *Snapshot `json:"data"`
}
ReadOut is the GET /v1/admin/infra envelope.
type ScaleIn ¶
type ScaleIn struct {
// ID is the DOKS cluster id, from the path.
ID string `json:"id"`
// Pool is the node pool, from the path. Its DO id or its name — both are unique
// within a cluster, and an operator reads the name off the board.
Pool string `json:"pool"`
// Count is the node count to set.
Count int `json:"count"`
}
ScaleIn addresses one node pool and the count to set.
type ServiceRef ¶
ServiceRef is one Service of type LoadBalancer: the identities by which it claims a DO load balancer. It is to load balancers exactly what PVRef is to volumes — the only sound liveness test, because a DO load balancer carries no back-reference of its own.
type Snapshot ¶
type Snapshot struct {
At string `json:"at"`
Complete bool `json:"complete"`
IncompleteReason string `json:"incompleteReason"`
Sources []core.SourceStatus `json:"sources"`
Totals Totals `json:"totals"`
Cost Cost `json:"cost"`
Clusters []Cluster `json:"clusters"`
Nodes []Node `json:"nodes"`
Volumes []Volume `json:"volumes"`
LoadBalancers []LoadBalancer `json:"loadBalancers"`
Findings []Finding `json:"findings"`
}
Snapshot is the whole board in one value.
func Analyze ¶
func Analyze(inv Inventory, scans []ClusterScan, sources []core.SourceStatus, at time.Time) Snapshot
Analyze folds the DO inventory and the per-cluster Kubernetes scans into the board. PURE: no clock, no network, no cluster — `at` is passed in so the result is byte-reproducible in tests.
type Totals ¶
type Totals struct {
Clusters int `json:"clusters"`
Nodes int `json:"nodes"`
Volumes int `json:"volumes"`
LoadBalancers int `json:"loadBalancers"`
VolumeGiB int `json:"volumeGiB"`
AttachedVolumes int `json:"attachedVolumes"`
AttachedGiB int `json:"attachedGiB"`
DetachedVolumes int `json:"detachedVolumes"`
DetachedGiB int `json:"detachedGiB"`
UnreferencedVolumes int `json:"unreferencedVolumes"`
UnreferencedGiB int `json:"unreferencedGiB"`
IdlePVCs int `json:"idlePVCs"`
LocalDiskGiB int `json:"localDiskGiB"`
// Fill. MeasuredVolumes/UnmeasuredVolumes are the honesty denominator: UsedGiB and
// WastedGiB describe the measured set ONLY, so a board showing waste must show how
// much of the fleet the figure was computed from. Unmeasured capacity contributes
// nothing to either — it is not assumed empty, and it is not assumed full.
MeasuredVolumes int `json:"measuredVolumes"`
MeasuredGiB int `json:"measuredGiB"`
UnmeasuredVolumes int `json:"unmeasuredVolumes"`
UnmeasuredGiB int `json:"unmeasuredGiB"`
UsedGiB int `json:"usedGiB"`
WastedGiB int `json:"wastedGiB"`
}
Totals are fleet counts. LocalDiskGiB is broken out precisely so it can be shown as NOT separately billed.
type Volume ¶
type Volume struct {
ID string `json:"id"`
Name string `json:"name"`
Region string `json:"region"`
SizeGiB int `json:"sizeGiB"`
MonthlyCents money.Cents `json:"monthlyCents"`
State string `json:"state"`
DropletIDs []int `json:"dropletIds"`
NodeName string `json:"nodeName"`
// Cluster/ClusterID are the PROVEN owner — resolved through a PV that names this
// volume, never through the tag.
Cluster string `json:"cluster"`
ClusterID string `json:"clusterId"`
// TagCluster is the `k8s:<uuid>` tag. ADVISORY ONLY: it outlives the cluster that
// set it. Shown so the operator can see tag-vs-truth disagree, never acted on.
TagCluster string `json:"tagCluster"`
PV string `json:"pv"`
PVPhase string `json:"pvPhase"`
PVCNamespace string `json:"pvcNamespace"`
PVCName string `json:"pvcName"`
MountedBy []string `json:"mountedBy"`
Idle bool `json:"idle"`
CreatedAt string `json:"createdAt"`
// Controller is the workload owning the pod that mounts this volume
// ("StatefulSet/luxd"), or "" when nothing mounts it. It names who has to act.
Controller string `json:"controller"`
// HasUsage reports whether a kubelet actually MEASURED this volume's filesystem.
//
// False means NOT MEASURED. It does NOT mean empty, and the three fields below are
// meaningless — not zero — when it is false. A reading exists only while a running pod
// has the volume mounted on a node that answered; a detached, idle or unreferenced
// volume has none. Rendering an unmeasured volume as "0 used / 100% wasted" would
// invent the single most expensive lie this board could tell, so every consumer must
// branch on this flag and show unknown.
HasUsage bool `json:"hasUsage"`
// UsedBytes is the measured filesystem usage. BYTES, not GiB: the volumes this exists
// to catch hold a fraction of a GiB in 200, and rounding that to an integer GiB would
// print the very 0 the flag above exists to prevent.
UsedBytes int64 `json:"usedBytes"`
// WastedGiB is provisioned minus measured, in the unit DigitalOcean BILLS: whole GiB
// of the volume's own size, never the filesystem's capacity — a 200 GiB volume carries
// a 196 GiB filesystem after format overhead, and the invoice says 200.
WastedGiB int `json:"wastedGiB"`
WastedMonthlyCents money.Cents `json:"wastedMonthlyCents"`
Deletable bool `json:"deletable"`
BlockedReason string `json:"blockedReason"`
// Expandable/ExpandBlockedReason are the GROW verdict, kept separate from Deletable
// because the two ask opposite questions: a volume is deletable when nothing uses it,
// and expandable when something uses it in a way this board can grow completely.
Expandable bool `json:"expandable"`
ExpandBlockedReason string `json:"expandBlockedReason"`
}
Volume is one block-storage volume with its PROVEN cluster ownership.
func (Volume) ExpandTo ¶
ExpandTo answers whether this volume may be grown to gib, mirroring NodePool.ScaleTo: the row carries the standing verdict, and the method judges the number asked for.
GROW ONLY. DigitalOcean can never shrink a block-storage volume, so a smaller target is not a slow operation — it is an impossible one, and pretending otherwise behind a button is how a chain node loses its data. Shrinking is a migration; see shrinkRecipe.
type VolumeIn ¶
type VolumeIn struct {
// ID is the DO volume id, from the path.
ID string `json:"id"`
// Snapshot is the snapshot-first switch on DELETE. Anything other than the literal
// "false" snapshots before destroying — the snapshot IS the undo, so waiving it is
// deliberate and explicit.
Snapshot string `json:"snapshot"`
// Name is the snapshot name on the snapshot action. Blank gets a deterministic
// "<volume>-predelete-<unix>" so the undo is findable in the DO console.
Name string `json:"name"`
// SizeGiB is the target size on the resize action. A volume only ever grows —
// ExpandTo is the verdict that refuses a shrink, so this is not validated here.
SizeGiB int `json:"sizeGiB"`
}
VolumeIn addresses one DigitalOcean volume.
type VolumeSnapshotOut ¶
type VolumeSnapshotOut struct {
Status string `json:"status"`
Msg string `json:"msg"`
Data *digitalocean.Snapshot `json:"data"`
}
VolumeSnapshotOut is the POST /v1/admin/infra/volumes/:id/snapshot envelope. It is the one infra change with a typed result, because DO returns a real snapshot object.
type VolumeUsage ¶
VolumeUsage is one PersistentVolumeClaim's REAL filesystem usage, as measured by the kubelet that has it mounted.
A reading exists ONLY for a volume a running pod has mounted on a node that answered. Everything else has NO reading — a different fact from "empty", carried as such all the way to the screen. See Volume.HasUsage.