storage

package
v1.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 29, 2026 License: Apache-2.0 Imports: 32 Imported by: 0

Documentation

Overview

Package storage owns host storage budgets and the lifecycle of explicitly registered local development caches. Persistent application data is never GC'd.

Index

Constants

View Source
const (
	HoldLocked    = "locked"
	HoldActive    = "recently active"
	HoldInUse     = "in use"
	HoldDirty     = "dirty"
	HoldData      = "unrecognized ignored data"
	HoldUnpushed  = "unpushed"
	HoldRefused   = "git refused removal"
	HoldUnchecked = "could not be checked"
	HoldHidden    = "hidden-changes"
	HoldNested    = "nested-repository"
	HoldOrphans   = "unreachable-commits"
	// HoldQuarantined is a worktree a previous pass moved aside and could not
	// move back. It is still a valid worktree and is never deleted by forge.
	HoldQuarantined = "quarantined"
	// HoldNotLinked is anything that is not plainly a linked worktree of the
	// repository: its main checkout, a directory containing it or the home
	// directory, or a path whose .git is not a gitdir file naming one of the
	// repository's worktrees (a stale registration now holding a clone).
	HoldNotLinked = "not a linked worktree"
	// HoldManaged is a worktree under Reliant's worktree root. Reliant knows
	// which chat each is bound to and reclaims them itself.
	HoldManaged = "managed by reliant"
	// HoldDeferred is a worktree whose removal would overrun what is left of
	// the pass's delete budget; a later pass takes it.
	HoldDeferred = "deferred"
)

Hold reasons: why the reaper deliberately left a worktree alone.

View Source
const FullGCStaleAfter = 48 * time.Hour

FullGCStaleAfter is when a successful full pass stops being evidence that retention is running: the schedule is daily, so two days means it is not firing.

View Source
const GiB = uint64(1 << 30)

GiB is one gibibyte in bytes.

Variables

View Source
var ErrMachinePolicyUnderTest = errors.New("storage: refusing to write the machine storage policy under test " +
	"(set FORGE_STORAGE_POLICY to a t.TempDir() path to exercise policy writes)")

ErrMachinePolicyUnderTest refuses a write to the developer's real machine policy from inside `go test`. Callers on a warn-never-fail path (the zero-touch converge hooks, builder registration, release pinning) treat it as "skip": a test has nothing real to register.

View Source
var LockWaitPoll = 200 * time.Millisecond

LockWaitPoll is how often WithLockWait retries a held lock.

View Source
var PolicyWarnings io.Writer = os.Stderr

PolicyWarnings receives the notice that storage.json holds keys this build does not know (written by a newer or older forge, or misspelled). Per-machine state is shared across forge versions, so an unknown key must never fail an unrelated command; strictness belongs to forge.yaml. Unknown keys fall back to defaults here but are preserved on Save for other versions. Warned once per path.

Functions

func AnonymousVolumeRecordPath

func AnonymousVolumeRecordPath(policyPath string) string

AnonymousVolumeRecordPath is where forge records, beside the policy, when it first saw each anonymous volume unused.

func AutoGCRecordPath

func AutoGCRecordPath(policyPath string) string

AutoGCRecordPath is the opportunistic pass's attempt record.

func Converge

func Converge(policyPath string, f Facts) error

Converge upserts Facts into the machine storage policy, under the same lock every other policy writer takes.

Three properties make this safe to call from a hot path, on every run:

  • IDEMPOTENT. Converging the same facts twice leaves the file untouched (it is not even rewritten), so warm runs cost one read.
  • ADDITIVE ONLY. Nothing is ever removed here. Several projects on one machine converge into the same policy, and a project that stops using a context must not silently un-protect it for everybody else. Entries that are GONE — a cluster no longer in kubeconfig or docker, a deleted project directory — are removed by Prune at the start of GC, never here. A project under the temp dir is never added at all.
  • COMPLETE-OR-NOTHING for a registry. A registry becomes a cleanup target only once the container, its aliases, the protecting contexts and at least one repository are all known. Recording a registry whose protecting contexts were still unknown would authorize deletion against an unknown protected set, which is the one failure mode retention must never have. Partial facts still converge their project/context/pin half.

func DefaultPath

func DefaultPath() (string, error)

DefaultPath resolves the machine policy location.

func DefaultWorktreeRebuildable

func DefaultWorktreeRebuildable() []string

DefaultWorktreeRebuildable returns a copy of the built-in allowlist.

func Exec

func Exec(ctx context.Context, name string, args ...string) ([]byte, error)

Exec runs a command with a bounded lifetime and returns its STDOUT.

Stderr is captured separately and only ever reported, in the error of a command that failed. Callers parse what Exec returns — kubectl and docker JSON — and a tool that succeeds while warning on stderr (kubectl's "Warning: v1 ComponentStatus is deprecated") would otherwise put the warning in front of the JSON. That made every protected-set scan fail to parse, so registry GC refused on every pass.

func FullGCProblem

func FullGCProblem(p Policy, last GCResult, ok bool, now time.Time) string

FullGCProblem reports, in one sentence, why registry retention is not known to be working, or "" if it is. It applies only when registries are registered: a machine with none has nothing that only the full pass can reclaim. Doctor and `forge env up` both read it, so they cannot disagree.

func FullGCRecordPath

func FullGCRecordPath(policyPath string) string

FullGCRecordPath is the full pass's record, beside the policy.

func GoCacheGCRecordPath added in v1.0.0

func GoCacheGCRecordPath(policyPath string) string

GoCacheGCRecordPath is the Go-cache pass's record (`forge storage gocache --apply`): its outcome and each cache's GoCacheTrim. It is that pass's own record, apart from the opportunistic one, so a scheduler running both on different cadences can rate-limit each by its own last run.

func LedgerPins

func LedgerPins(dir string) ([]string, error)

LedgerPins imports historical local releases before registration enables GC. A missing ledger is normal; malformed entries abort rather than omit pins.

func NodeConfigPath

func NodeConfigPath(policyPath string, p Policy) (string, error)

NodeConfigPath is stable across command exits; k3d bind mounts must never refer to a temporary file removed at the end of cluster creation.

func ProductionGoCacheRoots

func ProductionGoCacheRoots(ctx context.Context, command func(context.Context, string, ...string) ([]byte, error)) (gocache, golangci string)

ProductionGoCacheRoots resolves the machine's Go-format cache locations for a Runner: the shared GOCACHE (from `go env`) and the golangci-lint cache. Anything that cannot be located is left empty, which the layers skip rather than guess.

func RealFailure

func RealFailure(err error) error

RealFailure returns err only when the pass really failed. A pass whose every layer finished or was cut off after making progress is a success, so the command that ran it (and the scheduler watching its exit status) sees nil.

func RecordAutoGC

func RecordAutoGC(policyPath string, r GCResult) error

RecordAutoGC records an opportunistic pass's attempt.

func RecordFullGC

func RecordFullGC(policyPath string, r GCResult) error

RecordFullGC records an applied full pass.

func RecordGoCacheGC added in v1.0.0

func RecordGoCacheGC(policyPath string, r GCResult) error

RecordGoCacheGC records an applied Go-cache pass.

func Register

func Register(ctx context.Context, path string, contexts, repositories, pins []string) error

Register is the MANUAL escape hatch (`forge storage register`): it DISCOVERS the registry facts from live k3d/docker metadata instead of being told them, then applies them through exactly the same merge rule Converge uses (upsertFacts) — one definition of "upsert, never remove, complete-or-nothing" rather than two that can drift.

Prefer Converge from inside forge's own commands: they already know these facts declaratively, and discovery requires a reachable Docker daemon. Register exists for a registry forge did not create, or to repair a policy by hand. It never makes an arbitrary remote registry a cleanup target.

func RegisterProject

func RegisterProject(path, project string) error

RegisterProject adds a project whose rotated logs Forge can maintain.

func RegistryAliases

func RegistryAliases(container, hostPort string) []string

RegistryAliases is every host name one local registry answers to: the host ports a developer pushes to, the in-network `<container>:5000` that pods pull by, and the `registry.localhost` names forge's containerd mirror config templates. Retention attributes a repository to a registry by matching this set, so a name missing here means images pushed under it look unowned.

func Save

func Save(path string, p Policy) error

Save atomically replaces a validated policy, preserving unknown settings. Read-modify-write callers must hold WithLock across their Load and Save.

func WithLock

func WithLock(policyPath string, fn func() error) error

WithLock serializes policy updates and maintenance on this machine. It does not wait: a held lock is an immediate error, which is right for a pass that would only duplicate the holder's work.

func WithLockWait

func WithLockWait(policyPath string, wait time.Duration, fn func() error) error

WithLockWait is WithLock for a caller that must not be turned away by maintenance. It retries a held lock until wait elapses, then fails with the same "already running" error. The bound is the point: an unbounded wait would let a wedged maintenance pass hang a release cut forever, and no wait at all failed a cut — after a six-minute build — because a background pass happened to hold a lock for the few milliseconds the cut needs.

Types

type AnonymousVolume

type AnonymousVolume struct {
	Name        string
	Created     time.Time
	UnusedSince time.Time
}

AnonymousVolume is one anonymous volume a pass may remove.

type BackedOffError

type BackedOffError struct {
	// Path is where the distress was observed.
	Path string
	Err  error
}

BackedOffError marks a pass that stopped because the filesystem reported distress. Like a cut-off it is not a failure: everything not yet removed is retained, and the next pass continues. It is reported separately because it says something about the machine, not about the time budget.

func (*BackedOffError) Error

func (e *BackedOffError) Error() string

func (*BackedOffError) Unwrap

func (e *BackedOffError) Unwrap() error

type BudgetUnreachableError added in v1.0.0

type BudgetUnreachableError struct {
	// Cache names the cache ("shared go build cache").
	Cache string
	// Estimated is the cache's estimated size, Budget its size budget, and
	// Young the estimated bytes last used inside Floor (the safety floor, or the
	// age of the oldest running build plus its margin when that is longer), in bytes.
	Estimated, Budget, Young uint64
	Floor                    time.Duration
}

BudgetUnreachableError marks a cache no trim can bring under its size budget, because more than the budget was used inside the safety floor (Floor, goCacheFloor). Entries that young may belong to a running build — cmd/go restamps a used entry only once it is an hour stale — so they are never deleted, whatever the pressure: the pass removed everything older and the cache still sits at roughly Young. Only a slower fill, or a larger budget, fixes it.

It is not a failure — NewGCResult records it apart, in OverBudget — but it is never silent either. Before it existed a 20 GiB budget held a 152 GiB cache on a machine filling at ~70 GiB/h, and every pass reported itself finished.

func (*BudgetUnreachableError) Error added in v1.0.0

func (e *BudgetUnreachableError) Error() string

type CutOffError

type CutOffError struct{ Err error }

CutOffError marks a layer that stopped at its deadline having made whatever progress it could. NewGCResult records it as a cut-off, not a failure.

func (*CutOffError) Error

func (e *CutOffError) Error() string

func (*CutOffError) Unwrap

func (e *CutOffError) Unwrap() error

type DanglingImage

type DanglingImage struct {
	ID string
	// Repository is the repository the image was pushed to, from its digest
	// reference, or "". It is for the person reading the plan; no decision
	// reads it.
	Repository string
	Created    time.Time
	Size       uint64
}

DanglingImage is one untagged image a pass may remove.

type Disk

type Disk struct {
	Path      string `json:"path"`
	Available uint64 `json:"available_bytes"`
	Capacity  uint64 `json:"capacity_bytes"`
}

Disk describes physical filesystem capacity.

func CheckBuildSpace

func CheckBuildSpace(p Policy, paths ...string) ([]Disk, error)

CheckBuildSpace includes build scratch and explicitly configured Go caches, which can live on different filesystems from the project and home directory.

func CheckSpace

func CheckSpace(p Policy, paths ...string) ([]Disk, error)

CheckSpace refuses builds that would start below the physical host reserve. A shortfall is returned as a *ReserveError; any other error means a disk could not be measured at all.

func DiskSpace

func DiskSpace(path string) (Disk, error)

DiskSpace reports capacity on the physical host filesystem.

type Facts

type Facts struct {
	// Project is the project directory whose rotated logs forge may expire.
	// Stored absolute.
	Project string
	// Contexts are kubectl contexts whose workloads protect images in the
	// registry. Only local `k3d-` contexts are accepted: retention reads the
	// protected set from them, and a remote cluster forge cannot list would
	// make deletion unsafe.
	Contexts []string
	// Registry is the local registry's CONTAINER name (`k3d-…`), which is
	// also how `forge storage gc` addresses it.
	Registry string
	// Aliases are every host name under which that one registry is pushed to
	// or pulled from — the containerd mirror keys plus the in-network
	// `<container>:5000`. A repository is attributed to this registry only
	// when its host is one of these.
	Aliases []string
	// Repositories are full image references, registry host included. The
	// host is matched against Aliases and then stripped: retention works in
	// repository paths, as the registry API does.
	Repositories []string
	// Pins are image digests that must survive retention regardless of age.
	Pins []string
	// Repos are git repositories (any path inside one) whose worktrees the
	// worktree layer may reclaim: the project's own and its build siblings.
	Repos []string
}

Facts are what a forge command ALREADY knows about the local caches it just used: the project it ran in, the k3d contexts the env declares, the local registry container those contexts pull from, that registry's host aliases, the repositories this build pushes, and the project's release pins.

Every field is optional. Converge records what it is given and leaves the rest of the policy alone — a cluster phase knows contexts and aliases but no repositories, a build knows all five. That is why activation does not need a command of its own: the facts arrive from whichever touch point happens to run, and accumulate.

type GCResult

type GCResult struct {
	At time.Time `json:"at"`
	OK bool      `json:"ok"`
	// FailedLayers names each layer that failed: "logs", "temp sweep",
	// "source cache", "builder <name>", "registry <container>", "docker".
	FailedLayers []string `json:"failed_layers,omitempty"`
	// Error is the combined error text, for a human reading the record.
	Error string `json:"error,omitempty"`
	// CutOffLayers names each layer that ran out of its time slice (or the
	// pass out of its budget) after doing what it could. That is not a
	// failure: the layer reclaimed what it reached and resumes next pass. A
	// pass whose only problems are cut-offs is OK.
	CutOffLayers []string `json:"cut_off_layers,omitempty"`
	// BackedOffLayers names each layer that stopped, or never started,
	// because the filesystem reported distress (EMFILE, ENFILE, ENOTCONN,
	// EIO). Like a cut-off it retained the rest for the next pass and is not
	// a failure, but it says the MACHINE is unwell, so it is reported apart.
	BackedOffLayers []string `json:"backed_off_layers,omitempty"`
	// OverBudget describes each cache the pass could not bring under its size
	// budget because more than the budget is younger than the trim's safety
	// floor (BudgetUnreachableError). Not a failure — the pass removed what it
	// safely could — but a budget that is not being held, so it is reported
	// on every pass until the fill slows.
	OverBudget []string `json:"over_budget,omitempty"`
	// GoCaches is what each Go-format cache trim in the pass measured and
	// did (Runner.OnGoCacheTrim). Set by the caller that ran the pass.
	GoCaches []GoCacheTrim `json:"go_caches,omitempty"`
}

GCResult is the outcome of one maintenance pass.

func LastAutoGC

func LastAutoGC(policyPath string) (GCResult, bool)

LastAutoGC reads the opportunistic pass's attempt record.

func LastFullGC

func LastFullGC(policyPath string) (GCResult, bool)

LastFullGC reads the full pass's record. ok is false when there is none, or it cannot be read: "never ran" and "cannot tell" lead to the same advice.

func LastGoCacheGC added in v1.0.0

func LastGoCacheGC(policyPath string) (GCResult, bool)

LastGoCacheGC reads the Go-cache pass's record.

func NewGCResult

func NewGCResult(at time.Time, err error) GCResult

NewGCResult summarises a pass's error into a record.

func (GCResult) Summary

func (r GCResult) Summary() string

Summary is one line naming what failed, for `env up` and `doctor`.

type GoCacheTrim added in v1.0.0

type GoCacheTrim struct {
	// Cache names the cache ("shared go build cache"); Root is its directory.
	Cache string `json:"cache"`
	Root  string `json:"root"`
	// Applied is false for a dry run, which removes nothing.
	Applied bool `json:"applied"`
	// EstimatedBytes is the cache's estimated size before the trim,
	// BudgetBytes its size budget, and YoungBytes the estimated bytes last
	// used inside the safety floor, which no trim deletes.
	EstimatedBytes uint64 `json:"estimated_bytes"`
	BudgetBytes    uint64 `json:"budget_bytes"`
	YoungBytes     uint64 `json:"young_bytes"`
	// Cutoff is the last-use time below which entries were removed.
	Cutoff time.Time `json:"cutoff"`
	// Removed and FreedBytes are what the trim removed (or, dry, would have).
	Removed    int    `json:"removed"`
	FreedBytes uint64 `json:"freed_bytes"`
	// ShardsCovered of ShardsTotal were walked to the end; Finished is true
	// when that is all of them. A trim that did not finish resumes at the
	// first uncovered shard next pass.
	ShardsCovered int  `json:"shards_covered"`
	ShardsTotal   int  `json:"shards_total"`
	Finished      bool `json:"finished"`
	// OldestLiveBuild is the age of the oldest running Go build that held the
	// cutoff back past the floor (gocache_livebuild.go); zero when none did.
	OldestLiveBuild time.Duration `json:"oldest_live_build_ns,omitempty"`
}

GoCacheTrim is what one trim of one Go-format build cache measured and did. It is recorded in the pass's GCResult (GoCaches), so whoever schedules the pass can see the cache's size, the part no trim may touch, and whether the walk covered every shard, without parsing human output.

type LayerError

type LayerError struct {
	Layer string
	Err   error
}

LayerError attributes a failure to the maintenance layer it came from, so a result can name the layers that failed without parsing error text.

func (*LayerError) Error

func (e *LayerError) Error() string

func (*LayerError) Unwrap

func (e *LayerError) Unwrap() error

type PlanEntry

type PlanEntry struct {
	Version
	// Untagged marks a manifest no tag points at: an orphan left behind by a
	// re-push of a reused tag, unreachable from anything retained.
	Untagged bool
	// Tags are the aliases that disappear with this manifest.
	Tags  []string
	Bytes uint64
	Sized bool
}

PlanEntry is one manifest selected for deletion, with the reason it became eligible and a size estimate when the manifest could be read.

type Policy

type Policy struct {
	Projects         []string `json:"projects,omitempty"`
	LogBudgetGiB     uint64   `json:"log_budget_gib"`
	HostReserveGiB   uint64   `json:"host_reserve_gib"`
	HostPaths        []string `json:"host_paths,omitempty"`
	BuildCacheGiB    uint64   `json:"build_cache_gib"`
	BuildCacheUnused string   `json:"build_cache_unused"`
	ImageUnused      string   `json:"image_unused"`
	// GoCacheUnused is how long a shared Go build cache entry (and the
	// golangci-lint cache) may sit unused before it is trimmed. Minimum 2h:
	// Go refreshes an entry's mtime on use once it is an hour stale, so that
	// floor is what keeps the trim safe under running builds.
	GoCacheUnused string `json:"go_cache_unused"`
	// GoCacheGiB is the size the shared Go caches are trimmed down to,
	// oldest first, when age alone leaves them larger.
	GoCacheGiB uint64 `json:"go_cache_gib"`
	// GoCacheVolumePercent caps that budget at this share of the volume the
	// cache lives on, so the default works on a 40 GiB build volume as well as
	// on a 1 TB laptop: the budget is the smaller of go_cache_gib and this
	// share. Zero means the default (15); 100 leaves go_cache_gib alone.
	GoCacheVolumePercent int `json:"go_cache_volume_percent,omitempty"`
	// GolangciCacheGiB is the size the golangci-lint cache is trimmed down
	// to, least recently used first, like go_cache_gib for the build cache
	// and capped the same way. Its own budget because its entries are tiny
	// and recomputed in seconds: under go_cache_gib it was never trimmed.
	GolangciCacheGiB uint64 `json:"golangci_cache_gib"`
	// SourceCacheUnused is how long a cross-repo source clone may go
	// unresolved before it becomes an eviction candidate. Longer than the
	// image and build-cache windows because the cost of being wrong is a
	// full re-clone of a repository, not a re-pull of a layer.
	SourceCacheUnused string `json:"source_cache_unused"`
	// SourceCacheKeep is how many of the most recently used clones of one
	// repository are retained regardless of age. This is the floor that
	// keeps an idle project's current pin warm, so it is validated at 1 or
	// more: a policy that can empty a repository's only pin is not a policy
	// anyone wants by accident.
	SourceCacheKeep int `json:"source_cache_keep"`
	// SourceCacheGiB is the size the source cache is evicted down to, least
	// recently used clone first, never touching the newest source_cache_keep
	// clones of a repository. The age rule alone does not bound it: a project
	// that bumps its pin every few hours produces more clones in
	// source_cache_unused than any age can hold.
	SourceCacheGiB uint64 `json:"source_cache_gib"`
	// SourceCacheSlimAfter is how long a clone beyond the newest
	// source_cache_keep of its repository may sit unused before its installed
	// dependencies (node_modules) are removed. Those are 80% of a clone's
	// bytes and a build reinstalls them (frontenddeps.Ensure); the pristine
	// checkout stays.
	SourceCacheSlimAfter string     `json:"source_cache_slim_after"`
	RegistryDays         int        `json:"registry_days"`
	RegistryKeep         int        `json:"registry_keep"`
	DockerContext        string     `json:"docker_context,omitempty"`
	Builders             []string   `json:"builders"`
	Registries           []Registry `json:"registries,omitempty"`
	Clusters             []string   `json:"clusters,omitempty"`
	Pins                 []string   `json:"pins,omitempty"`

	// Repos are git repositories whose linked worktrees the worktree layer
	// reclaims, beyond the repository of each registered project: sibling
	// repos a project builds from (docker.build_contexts, go.mod replaces).
	// Filled by Converge; git itself enumerates the worktrees, wherever they live.
	Repos []string `json:"repos,omitempty"`
	// WorktreeRebuildable replaces the built-in allowlist of ignored paths a
	// removable worktree may contain. Nil means the defaults.
	WorktreeRebuildable []string `json:"worktree_rebuildable,omitempty"`
	// WorktreeReap lets an explicit `forge storage gc --apply` remove
	// worktrees. Off, the layer only reports. No automatic pass ever removes
	// one, whatever this says (Runner.Explicit).
	WorktreeReap bool `json:"worktree_reap,omitempty"`
	// WorktreeReapReliantManaged extends worktree_reap to worktrees under
	// Reliant's worktree root (~/.reliant/worktrees). Reliant owns those: it
	// knows which chat each one is bound to, and reclaims them itself. Off,
	// they are held like a locked worktree.
	WorktreeReapReliantManaged bool `json:"worktree_reap_reliant_managed,omitempty"`

	// MaxDeletesPerPass bounds how many filesystem entries one maintenance
	// pass may unlink, across every layer. What is left is retained and the
	// next pass continues, so a budget is reached over several passes rather
	// than by one burst of six-figure unlinks. Zero means the default.
	MaxDeletesPerPass int `json:"max_deletes_per_pass"`
	// DeleteRatePerSec paces those unlinks. Zero means the default.
	DeleteRatePerSec int `json:"delete_rate_per_sec"`

	// AnonymousVolumeUnused is how long forge must have seen an anonymous
	// Docker volume referenced by no container, running or stopped, before
	// an explicit `forge storage gc --apply` may remove it (anonvolumes.go).
	// Minimum 24h; "off" disables the layer. Named volumes are never removed,
	// whatever this says.
	AnonymousVolumeUnused string `json:"anonymous_volume_unused"`
	// MaxDockerRemovalsPerPass bounds how many Docker images and volumes one
	// pass asks the daemon to remove. Zero means the default.
	MaxDockerRemovalsPerPass int `json:"max_docker_removals_per_pass"`
}

Policy defines one machine’s cache budgets and protected references.

func DefaultPolicy

func DefaultPolicy() Policy

DefaultPolicy returns conservative local development budgets.

func Load

func Load(path string) (Policy, error)

Load overlays a JSON document on the default policy, tolerating unknown keys.

func (Policy) KubeletConfig

func (p Policy) KubeletConfig() []byte

KubeletConfig relies on kubelet's own usage tracking and GC, never on deleting containerd files underneath it. Unused age is independent of VM disk fullness.

func (Policy) Validate

func (p Policy) Validate() error

Validate rejects unbounded budgets and unscoped registry targets.

type Registry

type Registry struct {
	Container    string   `json:"container"`
	Repositories []string `json:"repositories"`
	Aliases      []string `json:"aliases"`
	Contexts     []string `json:"contexts"`
}

Registry explicitly identifies a local registry and all of its consumers.

type ReserveError

type ReserveError struct {
	Path       string
	Available  uint64
	ReserveGiB uint64
}

ReserveError is a MEASURED shortfall: the filesystem under Path was read and holds less than the policy's host reserve. It is a distinct type because it is the one CheckSpace outcome a caller may knowingly accept — an ephemeral CI runner, whose disk is discarded with the job — while an unreadable disk or path is an unknown that no caller should wave through.

func (*ReserveError) Error

func (e *ReserveError) Error() string

type Revision

type Revision struct {
	Version
	Modified time.Time
}

Revision is one manifest link on the registry filesystem and the moment it was written. Untagged manifests have no push timestamp anywhere else, so the link mtime is the only available age for them.

type Runner

type Runner struct {
	Policy  Policy
	Out     io.Writer
	Command func(context.Context, string, ...string) ([]byte, error)
	// SourceCacheRoot overrides the cross-repo source cache the Sources
	// layer reclaims from. Empty means the real machine-local cache in
	// production, and is REFUSED under `go test` — see sources.go.
	SourceCacheRoot string
	// TempRoot overrides the system temp directory the TempSweep layer
	// reclaims from. Empty means os.TempDir() in production, and is
	// REFUSED under `go test` — see tempsweep.go.
	TempRoot string
	// PolicyPath is the policy file this pass was loaded from. Maintenance
	// state that must outlive the process — the marker recording that a
	// registry was stopped for GC — is kept beside it, so a later pass can
	// find and repair what an interrupted one left behind.
	PolicyPath string
	// Ctx bounds the host-filesystem layers (logs, temp sweep, source cache),
	// which take no context of their own. GC and NonDisruptiveGC set it from
	// theirs, so a pass's deadline bounds the whole pass — its lsof snapshot
	// and its walk over entries, not only its docker calls. Nil means
	// unbounded.
	Ctx context.Context

	// Explicit marks the one pass a person or agent invoked: `forge storage
	// gc` without --scheduled. Only it may remove what no observation can
	// prove abandoned — worktrees (and then only under the policy's
	// worktree_reap) and anonymous Docker volumes. Every automatic pass leaves
	// it false: the background auto-gc, the installed schedule and `forge
	// storage daemon`. An agent between two commands holds no process in its
	// worktree, and an orphaned volume may be the only copy of a database, so
	// to a pass nobody invoked either looks exactly like garbage.
	Explicit bool

	// Go-cache layer roots (gocache.go). Empty means the real machine
	// location in production and is REFUSED under `go test`.
	GoCacheRoot       string // shared GOCACHE
	GolangciCacheRoot string
	// OpenPaths is the lsof snapshot. REFUSED under test when unset.
	OpenPaths func(context.Context) (openfiles.Snapshot, error)
	// OnGoCacheTrim, when set, receives what each Go-format cache trim
	// measured and did (gocache.go). The CLI collects them into the pass's
	// record. It may be called from the layer's goroutine; it is never called
	// concurrently.
	OnGoCacheTrim func(GoCacheTrim)
	// LiveToolchainAges lists how long each running Go toolchain process has
	// lived (gocache_livebuild.go). Nil reads the process table in
	// production and means "no builds" under test.
	LiveToolchainAges func(context.Context) ([]time.Duration, error)
	// contains filtered or unexported fields
}

Runner executes maintenance using injectable process operations.

func (Runner) AnonymousVolumes

func (r Runner) AnonymousVolumes(ctx context.Context, apply bool) error

AnonymousVolumes previews or applies the anonymous-volume rules. Only an explicit pass (Runner.Explicit) that applies removes anything; every other applying pass records what it observed and retains.

func (Runner) ConfigureNodes

func (r Runner) ConfigureNodes(ctx context.Context, policyPath string, apply bool) error

ConfigureNodes migrates existing k3d nodes without deleting their volumes. --apply is explicit because restarting a single-server cluster interrupts it.

func (Runner) DanglingImages

func (r Runner) DanglingImages(ctx context.Context, apply bool) error

DanglingImages removes every planned dangling image, or only lists them when apply is false. Oldest go first, so a pass stopped by its caps or deadline has reclaimed the longest-superseded builds.

func (Runner) GC

func (r Runner) GC(ctx context.Context, apply bool) error

GC previews or applies the registered cache and registry policy.

Every layer runs even when an earlier one failed: a layer's error is recorded and the combined error is returned at the end. One unreadable registered project used to end the pass at the first layer, so the temp sweep, source eviction, BuildKit and registry retention silently never ran. Two checks still stop everything, because nothing after them is safe without them: an invalid policy, and a Docker endpoint that is not local (which ends the Docker layers, not the host ones already done).

func (Runner) GoCacheGC added in v1.0.0

func (r Runner) GoCacheGC(ctx context.Context, apply bool) error

GoCacheGC runs only the Go-toolchain caches (GoCaches), with the whole of ctx's budget rather than the slice a full pass gives them.

It exists because the shared build cache fills on a different clock from everything else: under agent load ~25 GiB an hour crosses the trim's 2h safety floor, so a trim that runs hourly — or one whose share of a larger pass runs out under load — leaves tens of GiB that could already go on the disk. This pass is cheap enough (one walk of the cache, paced unlinks, no Docker, no registry) to run every few minutes, and it is non-disruptive for the same reason the Go-cache layer of NonDisruptiveGC is.

func (Runner) GoCaches

func (r Runner) GoCaches(apply bool) error

GoCaches trims every Go-toolchain cache. Dry-run unless apply is set. Layers run independently; one failure does not stop the rest.

The shared build cache is the layer that matters most and it grows ~40 GB/h under agent load, so it runs FIRST and each step gets a guaranteed slice of what remains: a slow step cannot starve the next. A step cut off by its slice has still reclaimed what it covered (see trimGoBuildCache); that is a partial success, never a failure. Only real errors are returned.

The delete budget is shared the same way: each step may spend a share of what the steps before it left (deletes.go), so the shared cache cannot spend the whole pass on itself. A step that reaches its share stops and is reported as cut off; a back-off ends every step after it.

func (Runner) GolangciLintCache

func (r Runner) GolangciLintCache(apply bool) error

GolangciLintCache trims golangci-lint's fork of the Go cache format.

func (Runner) Local

func (r Runner) Local(ctx context.Context) error

Local refuses remote Docker endpoints.

func (Runner) LocalImages

func (r Runner) LocalImages(ctx context.Context, apply bool) error

LocalImages untags every StaleLocalImage, or only lists them when apply is false. `docker rmi <ref>` removes the layers once the last tag is gone, and fails rather than force-removing an image a container has since started using, which is reported and does not stop the rest.

func (Runner) Logs

func (r Runner) Logs(apply bool) error

Logs only expires rotated Forge logs. Current streams, releases, promotion history and other files in .forge are never candidates.

Each registered project is maintained independently: one that cannot be read (a project under ~/Documents the scheduled job has no TCC access to, a permission error, a log removed underneath the pass) is reported in the combined error and the remaining projects are still maintained.

func (Runner) NonDisruptiveGC

func (r Runner) NonDisruptiveGC(ctx context.Context, apply bool) error

NonDisruptiveGC runs only the layers that cannot interrupt anything a developer is currently using: expiring rotated logs, trimming the Go caches, removing dangling images, evicting unused builder cache, the temp sweep and the source cache.

Registry GC and node reconfiguration are deliberately NOT here. Registry cleanup takes the registry offline for the duration (pushes and pulls fail), and node reconfiguration restarts kubelet — both are fine for a scheduled 03:30 pass and unacceptable as a side effect of `forge env up`, which is typically the command that is about to push to that registry.

Worktrees are not here either, not even as a preview. This pass runs behind the user's back — at the end of `forge env up`, and hourly from the installed schedule — and nothing it can observe tells an abandoned worktree from one an agent is between two commands in: no process holds it open, and its files can sit untouched for a day while the agent reads. Removing one is a decision only an explicit `forge storage gc --apply` may take. Anonymous volumes are the same decision about data: this pass records which ones are unused, so their clocks start, and never removes one (anonvolumes.go).

As in GC, each layer's failure is recorded and the rest still run — unless the filesystem reported distress, which stops the pass (deletes.go).

func (Runner) PrintHeldWorktrees

func (r Runner) PrintHeldWorktrees(ctx context.Context) error

PrintHeldWorktrees reports the worktrees the reaper will not touch, by reason, for `forge storage status`.

func (Runner) Prune

func (r Runner) Prune(ctx context.Context, policyPath string) error

Prune removes stale clusters and projects from the policy at policyPath and saves it. The caller must hold the maintenance lock (WithLock), as GC does.

func (Runner) RegisterBuilder

func (r Runner) RegisterBuilder(ctx context.Context, path, builder string) error

RegisterBuilder adopts a named local builder without requiring a registry or cluster. Its endpoint is checked again before every maintenance pass.

func (Runner) RegistryGC

func (r Runner) RegistryGC(ctx context.Context, reg Registry, apply bool) (err error)

RegistryGC deletes eligible manifests and collects blobs with all writers stopped.

func (Runner) SharedGoCache

func (r Runner) SharedGoCache(apply bool) error

SharedGoCache trims the shared `go env GOCACHE`.

func (Runner) Sources

func (r Runner) Sources(apply bool) error

Sources reclaims cross-repo source clones that no project has resolved recently. One full clone exists per repo+ref pin and nothing else ever removes one, so a project that bumps its pin weekly accumulates a complete checkout per bump — 9.2 GB of one repository on the machine this layer was written for, most of it an installed node_modules.

The eviction MECHANISM and its safety rules live in internal/gitsource, next to the code that writes the cache; the BUDGETS come from this machine's storage policy, the way every other layer's do, so a developer tunes one file rather than learning where each reclaimer hides its defaults. The newest-K floor means an idle project's current pin survives, so this is safe to run unattended.

An unparseable SourceCacheUnused cannot happen on a loaded policy — Validate rejects it and GC calls Validate first — so it is treated the same as an unset value: the gitsource default, never "evict everything".

A cache that cannot be located is not a failure: os.UserCacheDir fails on a host with no HOME, where there is also nothing to reclaim.

func (Runner) StaleLocalImages

func (r Runner) StaleLocalImages(ctx context.Context) ([]StaleLocalImage, error)

StaleLocalImages lists host image tags whose repository host is a local registry alias (`localhost:<port>`, `registry.localhost:<port>`, `k3d-<name>:<port>`) that no running container backs, and whose image no container, running or stopped, uses.

An alias is dead only when NO container, running or stopped, backs it: a `localhost` or `registry.localhost` alias is backed by any container whose port bindings include that host port, a `k3d-<name>` alias by any container of that name, and a registry registered in the policy by its container name. A stopped registry (the full pass stops it for garbage-collect; Docker Desktop restarts leave k3d registries stopped) therefore keeps its images. Anything unparseable is kept; a failed docker call fails the layer rather than guessing.

func (Runner) Status

func (r Runner) Status(ctx context.Context) error

Status reports host, Docker, builder and kubelet storage usage.

func (Runner) TempSweep

func (r Runner) TempSweep(apply bool) error

TempSweep reclaims abandoned scratch entries in the system temp directory. Dry-run by default: it prints each candidate and its size, and removes nothing unless apply is set.

func (Runner) Worktrees

func (r Runner) Worktrees(ctx context.Context, repo, base string, idle time.Duration, apply, reliantManaged bool) error

Worktrees is the explicit `forge storage worktrees` entry point. It runs the same classifier the GC layer does, over one repository. reliantManaged includes worktrees under Reliant's worktree root, which are otherwise held.

type StaleLocalImage

type StaleLocalImage struct {
	Ref string
	ID  string
}

StaleLocalImage is one host image reference to be untagged.

type Tag

type Tag struct {
	Repository string
	Name       string
	Digest     string
	Pushed     time.Time
}

Tag is a registry alias and its last push timestamp.

type Version

type Version struct {
	Repository string
	Digest     string
}

Version identifies one content-addressed registry manifest.

func RegistryCandidates

func RegistryCandidates(tags []Tag, p Policy, reg Registry, protected map[string]map[string]bool, now time.Time) []Version

RegistryCandidates protects aliases at digest granularity: deleting one manifest also removes every tag that points to it.

type WorktreeHold

type WorktreeHold struct {
	Path, Repo, Reason, Detail string
}

WorktreeHold is one worktree the reaper left alone, and why.

type WorktreeReport

type WorktreeReport struct {
	// Removable are worktrees that passed every check (removed when applied).
	Removable []string
	// Removed is the subset actually removed.
	Removed []string
	Held    []WorktreeHold
}

WorktreeReport is the outcome of one classification pass.

func (WorktreeReport) HeldByReason

func (w WorktreeReport) HeldByReason() map[string]int

HeldByReason counts held worktrees per reason.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL