diskusage

package
v0.0.5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 12, 2026 License: MIT Imports: 9 Imported by: 0

Documentation

Overview

Package diskusage measures what a project's .cloop directory actually costs on disk, broken down by the thing that produced it (Task 20229).

It exists because "the hub filled its disk" is an availability defect that nothing in cloop could previously see coming. `cloop doctor` reported one aggregate number for .cloop and a separate one for state.db, which is not enough to act on: an operator looking at "4.9 GB" cannot tell whether to vacuum the database, prune plan history, or ship audit archives to cold storage, and those have very different costs. The per-entry breakdown names the culprit.

The database gets a second number the filesystem cannot provide. A SQLite file does not shrink when rows are deleted; the pages move to a freelist and are reused for future writes. On this repository's own hub, state.db was 2.3 GB on disk of which 87% was freelist — reclaimable by a VACUUM, and invisible to du. ReclaimableBytes reports that, so the difference between "this database holds 2.3 GB of data" and "this database holds 300 MB of data in a 2.3 GB file" is legible before the disk fills rather than after.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func FreeBytes

func FreeBytes(path string) (int64, error)

FreeBytes reports the space this process could actually write on the filesystem holding path, or an error if it cannot be determined.

This exists because VACUUM is not a pure reclaim: SQLite rebuilds the database into a temporary copy and then writes it back through the journal, so it needs free space *before* it returns any. Running it on a nearly-full disk is therefore the one case where the operation meant to save you is the one that finishes the job — and "nearly full" is exactly when a janitor decides to run. The machine this was written on was at 100% with zero bytes available, which is how the check came to exist.

Bavail or Bfree depending on who we are, and the distinction matters here more than it usually does. Bavail excludes the blocks reserved for root — typically 5%, which on a 150 GB volume is 7.5 GB. A hub running as root reads Bavail as zero on exactly the full disk it was deployed to rescue, and would then decline to vacuum forever. Asking for the number that describes *this* process is the only reading that does not either overstate the space for an unprivileged hub or understate it for a privileged one.

func HumanBytes

func HumanBytes(n int64) string

HumanBytes formats a byte count for operator-facing output.

Types

type Entry

type Entry struct {
	Name  string `json:"name"`
	Bytes int64  `json:"bytes"`
	// Files counts regular files beneath this entry (1 for a plain file).
	// It is what distinguishes "one enormous archive" from "four thousand
	// small snapshots", which have different remedies.
	Files int  `json:"files"`
	IsDir bool `json:"is_dir"`
}

Entry is one top-level member of .cloop — a subdirectory such as plan-history, or a file such as state.db.

Top-level files are reported alongside directories rather than lumped into an "other" bucket because the two largest single files in a busy project, state.db and its write-ahead log, are exactly the ones an operator needs to see. On the hub this was measured at, state.db-wal alone was 63 MB.

type FloorStanding added in v0.0.5

type FloorStanding int

FloorStanding is where a volume's free space stands against a free-space floor (orchestrator.min_free_disk_mb).

const (
	// FloorOff: the floor is 0, which turns the check off.
	FloorOff FloorStanding = iota
	// FloorClear: at least twice the floor is free.
	FloorClear
	// FloorNear: above the floor but under twice it — the band an operator
	// is warned in before anything pauses.
	FloorNear
	// FloorBelow: under the floor, where a run pauses before its next task.
	FloorBelow
)

type Usage

type Usage struct {
	Dir        string `json:"dir"`
	TotalBytes int64  `json:"total_bytes"`
	// Entries is sorted largest-first so the caller can render the top N
	// without re-sorting and an operator reads the culprit on line one.
	Entries []Entry `json:"entries"`

	// DBBytes is state.db's own size, repeated out of Entries so callers
	// that only care about the database need not search the slice.
	DBBytes int64 `json:"db_bytes"`
	// ReclaimableBytes is what a VACUUM would return to the filesystem:
	// freelist_count × page_size. Zero when the database could not be read.
	ReclaimableBytes int64 `json:"reclaimable_bytes"`
	// FreeRatio is ReclaimableBytes / DBBytes, in [0,1]. This is the number
	// the janitor's VACUUM threshold is compared against — a ratio rather
	// than an absolute size, because a 2 GB freelist is urgent in a 2.3 GB
	// file and unremarkable in a 200 GB one.
	FreeRatio float64 `json:"free_ratio"`
	// DBError explains why the database numbers are absent, if they are.
	// A project with no state.db yet is not an error; it leaves this empty
	// and the DB fields zero.
	DBError string `json:"db_error,omitempty"`
}

Usage is the full picture for one .cloop directory.

func Measure

func Measure(workDir string) (*Usage, error)

Measure walks workDir/.cloop and reads the database's page statistics.

The database read is best-effort: a state.db that is missing, locked by a peer, or newer than this binary (statedb.Open refuses those, by design) leaves the DB fields zero and DBError set. Disk usage is still reported, because the reason an operator is running this is usually that something is wrong with the database.

func MeasureFiles

func MeasureFiles(workDir string) (*Usage, error)

MeasureFiles reports the on-disk breakdown without touching the database. Measure is the usual entry point; this exists for callers that already have the page statistics, and for tests that have no database at all.

func (*Usage) ApplySizeStats

func (u *Usage) ApplySizeStats(stats statedb.SizeStats)

ApplySizeStats fills the database fields from an already-open connection. Callers inside the hub, which holds the control-plane database open, use this to avoid a second connection to a multi-gigabyte file.

func (*Usage) Bytes

func (u *Usage) Bytes(name string) int64

Bytes returns the size of the named entry, or zero if absent.

func (*Usage) Entry

func (u *Usage) Entry(name string) (Entry, bool)

Entry returns the named entry and whether it was present.

func (*Usage) VacuumHeadroom

func (u *Usage) VacuumHeadroom() int64

VacuumHeadroom estimates the free space a VACUUM of this database needs.

SQLite copies the live pages into a temporary database and then writes that back into the original through the journal, so the peak requirement is about twice the *compacted* size — not twice the file size, which is the number that would stop a mostly-empty database from ever being reclaimed. The fixed addend covers the schema, the WAL's own overhead, and the fact that the freelist figure is a snapshot of a file a writer may still be extending.

type Volume added in v0.0.5

type Volume struct {
	// Mount names the volume the way an operator does: the directory it is
	// mounted on, as far as walking up from Path can tell — the highest
	// ancestor still on the same device. "/" on a single-disk host.
	Mount string `json:"mount"`
	// Path is the first of the probed paths that lives on this volume, as
	// given (before symlinks were resolved).
	Path string `json:"path"`
	// Device is the st_dev the volume was told apart by.
	Device uint64 `json:"device"`
	// FreeBytes is what this process could write there; see FreeBytes for
	// why that depends on whether it runs as root.
	FreeBytes int64 `json:"free_bytes"`
}

Volume is one filesystem that a set of paths lives on, and the space this process could still write there (Task 20381).

func Volumes added in v0.0.5

func Volumes(paths ...string) ([]Volume, error)

Volumes resolves each path to the filesystem holding it and returns one Volume per distinct device, in the order the paths first name them.

It exists because the question "is there room to run" is per filesystem, not per path: a project's .cloop and its working tree are normally one volume and measuring it twice would print the same warning twice, but .cloop can be a symlink onto another disk, and then both have to be watched. Symlinks are followed, so a path counts against the volume its data lands on.

A path that does not exist yet is measured at its nearest existing ancestor — that is where it will be created. An empty path is skipped. An error comes back only when a path's volume cannot be determined at all.

func (Volume) AgainstFloor added in v0.0.5

func (v Volume) AgainstFloor(floor int64) FloorStanding

AgainstFloor places v's free space against a floor of floor bytes. The hub's admin banner and `cloop hub doctor` both read it, so the band one warns in is the band the other reports (Task 20387); a run pauses in FloorBelow, by the same strict comparison.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL