Documentation
¶
Overview ¶
Package storage owns host storage budgets and the lifecycle of explicitly registered local development caches. Persistent application data is never GC'd.
Index ¶
- Constants
- Variables
- func AnonymousVolumeRecordPath(policyPath string) string
- func AutoGCRecordPath(policyPath string) string
- func Converge(policyPath string, f Facts) error
- func DefaultPath() (string, error)
- func DefaultWorktreeRebuildable() []string
- func Exec(ctx context.Context, name string, args ...string) ([]byte, error)
- func FullGCProblem(p Policy, last GCResult, ok bool, now time.Time) string
- func FullGCRecordPath(policyPath string) string
- func GoCacheGCRecordPath(policyPath string) string
- func LedgerPins(dir string) ([]string, error)
- func NodeConfigPath(policyPath string, p Policy) (string, error)
- func ProductionGoCacheRoots(ctx context.Context, ...) (gocache, golangci string)
- func RealFailure(err error) error
- func RecordAutoGC(policyPath string, r GCResult) error
- func RecordFullGC(policyPath string, r GCResult) error
- func RecordGoCacheGC(policyPath string, r GCResult) error
- func Register(ctx context.Context, path string, contexts, repositories, pins []string) error
- func RegisterProject(path, project string) error
- func RegistryAliases(container, hostPort string) []string
- func Save(path string, p Policy) error
- func WithLock(policyPath string, fn func() error) error
- func WithLockWait(policyPath string, wait time.Duration, fn func() error) error
- type AnonymousVolume
- type BackedOffError
- type BudgetUnreachableError
- type CutOffError
- type DanglingImage
- type Disk
- type Facts
- type GCResult
- type GoCacheTrim
- type LayerError
- type PlanEntry
- type Policy
- type Registry
- type ReserveError
- type Revision
- type Runner
- func (r Runner) AnonymousVolumes(ctx context.Context, apply bool) error
- func (r Runner) ConfigureNodes(ctx context.Context, policyPath string, apply bool) error
- func (r Runner) DanglingImages(ctx context.Context, apply bool) error
- func (r Runner) GC(ctx context.Context, apply bool) error
- func (r Runner) GoCacheGC(ctx context.Context, apply bool) error
- func (r Runner) GoCaches(apply bool) error
- func (r Runner) GolangciLintCache(apply bool) error
- func (r Runner) Local(ctx context.Context) error
- func (r Runner) LocalImages(ctx context.Context, apply bool) error
- func (r Runner) Logs(apply bool) error
- func (r Runner) NonDisruptiveGC(ctx context.Context, apply bool) error
- func (r Runner) PrintHeldWorktrees(ctx context.Context) error
- func (r Runner) Prune(ctx context.Context, policyPath string) error
- func (r Runner) RegisterBuilder(ctx context.Context, path, builder string) error
- func (r Runner) RegistryGC(ctx context.Context, reg Registry, apply bool) (err error)
- func (r Runner) SharedGoCache(apply bool) error
- func (r Runner) Sources(apply bool) error
- func (r Runner) StaleLocalImages(ctx context.Context) ([]StaleLocalImage, error)
- func (r Runner) Status(ctx context.Context) error
- func (r Runner) TempSweep(apply bool) error
- func (r Runner) Worktrees(ctx context.Context, repo, base string, idle time.Duration, ...) error
- type StaleLocalImage
- type Tag
- type Version
- type WorktreeHold
- type WorktreeReport
Constants ¶
const ( HoldLocked = "locked" HoldActive = "recently active" HoldInUse = "in use" HoldDirty = "dirty" HoldData = "unrecognized ignored data" HoldUnpushed = "unpushed" HoldRefused = "git refused removal" HoldUnchecked = "could not be checked" HoldHidden = "hidden-changes" HoldNested = "nested-repository" HoldOrphans = "unreachable-commits" // HoldQuarantined is a worktree a previous pass moved aside and could not // move back. It is still a valid worktree and is never deleted by forge. HoldQuarantined = "quarantined" // HoldNotLinked is anything that is not plainly a linked worktree of the // repository: its main checkout, a directory containing it or the home // directory, or a path whose .git is not a gitdir file naming one of the // repository's worktrees (a stale registration now holding a clone). HoldNotLinked = "not a linked worktree" // HoldManaged is a worktree under Reliant's worktree root. Reliant knows // which chat each is bound to and reclaims them itself. HoldManaged = "managed by reliant" // HoldDeferred is a worktree whose removal would overrun what is left of // the pass's delete budget; a later pass takes it. HoldDeferred = "deferred" )
Hold reasons: why the reaper deliberately left a worktree alone.
const FullGCStaleAfter = 48 * time.Hour
FullGCStaleAfter is when a successful full pass stops being evidence that retention is running: the schedule is daily, so two days means it is not firing.
const GiB = uint64(1 << 30)
GiB is one gibibyte in bytes.
Variables ¶
var ErrMachinePolicyUnderTest = errors.New("storage: refusing to write the machine storage policy under test " +
"(set FORGE_STORAGE_POLICY to a t.TempDir() path to exercise policy writes)")
ErrMachinePolicyUnderTest refuses a write to the developer's real machine policy from inside `go test`. Callers on a warn-never-fail path (the zero-touch converge hooks, builder registration, release pinning) treat it as "skip": a test has nothing real to register.
var LockWaitPoll = 200 * time.Millisecond
LockWaitPoll is how often WithLockWait retries a held lock.
var PolicyWarnings io.Writer = os.Stderr
PolicyWarnings receives the notice that storage.json holds keys this build does not know (written by a newer or older forge, or misspelled). Per-machine state is shared across forge versions, so an unknown key must never fail an unrelated command; strictness belongs to forge.yaml. Unknown keys fall back to defaults here but are preserved on Save for other versions. Warned once per path.
Functions ¶
func AnonymousVolumeRecordPath ¶
AnonymousVolumeRecordPath is where forge records, beside the policy, when it first saw each anonymous volume unused.
func AutoGCRecordPath ¶
AutoGCRecordPath is the opportunistic pass's attempt record.
func Converge ¶
Converge upserts Facts into the machine storage policy, under the same lock every other policy writer takes.
Three properties make this safe to call from a hot path, on every run:
- IDEMPOTENT. Converging the same facts twice leaves the file untouched (it is not even rewritten), so warm runs cost one read.
- ADDITIVE ONLY. Nothing is ever removed here. Several projects on one machine converge into the same policy, and a project that stops using a context must not silently un-protect it for everybody else. Entries that are GONE — a cluster no longer in kubeconfig or docker, a deleted project directory — are removed by Prune at the start of GC, never here. A project under the temp dir is never added at all.
- COMPLETE-OR-NOTHING for a registry. A registry becomes a cleanup target only once the container, its aliases, the protecting contexts and at least one repository are all known. Recording a registry whose protecting contexts were still unknown would authorize deletion against an unknown protected set, which is the one failure mode retention must never have. Partial facts still converge their project/context/pin half.
func DefaultPath ¶
DefaultPath resolves the machine policy location.
func DefaultWorktreeRebuildable ¶
func DefaultWorktreeRebuildable() []string
DefaultWorktreeRebuildable returns a copy of the built-in allowlist.
func Exec ¶
Exec runs a command with a bounded lifetime and returns its STDOUT.
Stderr is captured separately and only ever reported, in the error of a command that failed. Callers parse what Exec returns — kubectl and docker JSON — and a tool that succeeds while warning on stderr (kubectl's "Warning: v1 ComponentStatus is deprecated") would otherwise put the warning in front of the JSON. That made every protected-set scan fail to parse, so registry GC refused on every pass.
func FullGCProblem ¶
FullGCProblem reports, in one sentence, why registry retention is not known to be working, or "" if it is. It applies only when registries are registered: a machine with none has nothing that only the full pass can reclaim. Doctor and `forge env up` both read it, so they cannot disagree.
func FullGCRecordPath ¶
FullGCRecordPath is the full pass's record, beside the policy.
func GoCacheGCRecordPath ¶ added in v1.0.0
GoCacheGCRecordPath is the Go-cache pass's record (`forge storage gocache --apply`): its outcome and each cache's GoCacheTrim. It is that pass's own record, apart from the opportunistic one, so a scheduler running both on different cadences can rate-limit each by its own last run.
func LedgerPins ¶
LedgerPins imports historical local releases before registration enables GC. A missing ledger is normal; malformed entries abort rather than omit pins.
func NodeConfigPath ¶
NodeConfigPath is stable across command exits; k3d bind mounts must never refer to a temporary file removed at the end of cluster creation.
func ProductionGoCacheRoots ¶
func ProductionGoCacheRoots(ctx context.Context, command func(context.Context, string, ...string) ([]byte, error)) (gocache, golangci string)
ProductionGoCacheRoots resolves the machine's Go-format cache locations for a Runner: the shared GOCACHE (from `go env`) and the golangci-lint cache. Anything that cannot be located is left empty, which the layers skip rather than guess.
func RealFailure ¶
RealFailure returns err only when the pass really failed. A pass whose every layer finished or was cut off after making progress is a success, so the command that ran it (and the scheduler watching its exit status) sees nil.
func RecordAutoGC ¶
RecordAutoGC records an opportunistic pass's attempt.
func RecordFullGC ¶
RecordFullGC records an applied full pass.
func RecordGoCacheGC ¶ added in v1.0.0
RecordGoCacheGC records an applied Go-cache pass.
func Register ¶
Register is the MANUAL escape hatch (`forge storage register`): it DISCOVERS the registry facts from live k3d/docker metadata instead of being told them, then applies them through exactly the same merge rule Converge uses (upsertFacts) — one definition of "upsert, never remove, complete-or-nothing" rather than two that can drift.
Prefer Converge from inside forge's own commands: they already know these facts declaratively, and discovery requires a reachable Docker daemon. Register exists for a registry forge did not create, or to repair a policy by hand. It never makes an arbitrary remote registry a cleanup target.
func RegisterProject ¶
RegisterProject adds a project whose rotated logs Forge can maintain.
func RegistryAliases ¶
RegistryAliases is every host name one local registry answers to: the host ports a developer pushes to, the in-network `<container>:5000` that pods pull by, and the `registry.localhost` names forge's containerd mirror config templates. Retention attributes a repository to a registry by matching this set, so a name missing here means images pushed under it look unowned.
func Save ¶
Save atomically replaces a validated policy, preserving unknown settings. Read-modify-write callers must hold WithLock across their Load and Save.
func WithLock ¶
WithLock serializes policy updates and maintenance on this machine. It does not wait: a held lock is an immediate error, which is right for a pass that would only duplicate the holder's work.
func WithLockWait ¶
WithLockWait is WithLock for a caller that must not be turned away by maintenance. It retries a held lock until wait elapses, then fails with the same "already running" error. The bound is the point: an unbounded wait would let a wedged maintenance pass hang a release cut forever, and no wait at all failed a cut — after a six-minute build — because a background pass happened to hold a lock for the few milliseconds the cut needs.
Types ¶
type AnonymousVolume ¶
AnonymousVolume is one anonymous volume a pass may remove.
type BackedOffError ¶
BackedOffError marks a pass that stopped because the filesystem reported distress. Like a cut-off it is not a failure: everything not yet removed is retained, and the next pass continues. It is reported separately because it says something about the machine, not about the time budget.
func (*BackedOffError) Error ¶
func (e *BackedOffError) Error() string
func (*BackedOffError) Unwrap ¶
func (e *BackedOffError) Unwrap() error
type BudgetUnreachableError ¶ added in v1.0.0
type BudgetUnreachableError struct {
// Cache names the cache ("shared go build cache").
Cache string
// Estimated is the cache's estimated size, Budget its size budget, and
// Young the estimated bytes last used inside Floor (the safety floor, or the
// age of the oldest running build plus its margin when that is longer), in bytes.
Estimated, Budget, Young uint64
Floor time.Duration
}
BudgetUnreachableError marks a cache no trim can bring under its size budget, because more than the budget was used inside the safety floor (Floor, goCacheFloor). Entries that young may belong to a running build — cmd/go restamps a used entry only once it is an hour stale — so they are never deleted, whatever the pressure: the pass removed everything older and the cache still sits at roughly Young. Only a slower fill, or a larger budget, fixes it.
It is not a failure — NewGCResult records it apart, in OverBudget — but it is never silent either. Before it existed a 20 GiB budget held a 152 GiB cache on a machine filling at ~70 GiB/h, and every pass reported itself finished.
func (*BudgetUnreachableError) Error ¶ added in v1.0.0
func (e *BudgetUnreachableError) Error() string
type CutOffError ¶
type CutOffError struct{ Err error }
CutOffError marks a layer that stopped at its deadline having made whatever progress it could. NewGCResult records it as a cut-off, not a failure.
func (*CutOffError) Error ¶
func (e *CutOffError) Error() string
func (*CutOffError) Unwrap ¶
func (e *CutOffError) Unwrap() error
type DanglingImage ¶
type DanglingImage struct {
ID string
// Repository is the repository the image was pushed to, from its digest
// reference, or "". It is for the person reading the plan; no decision
// reads it.
Repository string
Created time.Time
Size uint64
}
DanglingImage is one untagged image a pass may remove.
type Disk ¶
type Disk struct {
Path string `json:"path"`
Available uint64 `json:"available_bytes"`
Capacity uint64 `json:"capacity_bytes"`
}
Disk describes physical filesystem capacity.
func CheckBuildSpace ¶
CheckBuildSpace includes build scratch and explicitly configured Go caches, which can live on different filesystems from the project and home directory.
func CheckSpace ¶
CheckSpace refuses builds that would start below the physical host reserve. A shortfall is returned as a *ReserveError; any other error means a disk could not be measured at all.
type Facts ¶
type Facts struct {
// Project is the project directory whose rotated logs forge may expire.
// Stored absolute.
Project string
// Contexts are kubectl contexts whose workloads protect images in the
// registry. Only local `k3d-` contexts are accepted: retention reads the
// protected set from them, and a remote cluster forge cannot list would
// make deletion unsafe.
Contexts []string
// Registry is the local registry's CONTAINER name (`k3d-…`), which is
// also how `forge storage gc` addresses it.
Registry string
// Aliases are every host name under which that one registry is pushed to
// or pulled from — the containerd mirror keys plus the in-network
// `<container>:5000`. A repository is attributed to this registry only
// when its host is one of these.
Aliases []string
// Repositories are full image references, registry host included. The
// host is matched against Aliases and then stripped: retention works in
// repository paths, as the registry API does.
Repositories []string
// Pins are image digests that must survive retention regardless of age.
Pins []string
// Repos are git repositories (any path inside one) whose worktrees the
// worktree layer may reclaim: the project's own and its build siblings.
Repos []string
}
Facts are what a forge command ALREADY knows about the local caches it just used: the project it ran in, the k3d contexts the env declares, the local registry container those contexts pull from, that registry's host aliases, the repositories this build pushes, and the project's release pins.
Every field is optional. Converge records what it is given and leaves the rest of the policy alone — a cluster phase knows contexts and aliases but no repositories, a build knows all five. That is why activation does not need a command of its own: the facts arrive from whichever touch point happens to run, and accumulate.
type GCResult ¶
type GCResult struct {
At time.Time `json:"at"`
OK bool `json:"ok"`
// FailedLayers names each layer that failed: "logs", "temp sweep",
// "source cache", "builder <name>", "registry <container>", "docker".
FailedLayers []string `json:"failed_layers,omitempty"`
// Error is the combined error text, for a human reading the record.
Error string `json:"error,omitempty"`
// CutOffLayers names each layer that ran out of its time slice (or the
// pass out of its budget) after doing what it could. That is not a
// failure: the layer reclaimed what it reached and resumes next pass. A
// pass whose only problems are cut-offs is OK.
CutOffLayers []string `json:"cut_off_layers,omitempty"`
// BackedOffLayers names each layer that stopped, or never started,
// because the filesystem reported distress (EMFILE, ENFILE, ENOTCONN,
// EIO). Like a cut-off it retained the rest for the next pass and is not
// a failure, but it says the MACHINE is unwell, so it is reported apart.
BackedOffLayers []string `json:"backed_off_layers,omitempty"`
// OverBudget describes each cache the pass could not bring under its size
// budget because more than the budget is younger than the trim's safety
// floor (BudgetUnreachableError). Not a failure — the pass removed what it
// safely could — but a budget that is not being held, so it is reported
// on every pass until the fill slows.
OverBudget []string `json:"over_budget,omitempty"`
// GoCaches is what each Go-format cache trim in the pass measured and
// did (Runner.OnGoCacheTrim). Set by the caller that ran the pass.
GoCaches []GoCacheTrim `json:"go_caches,omitempty"`
}
GCResult is the outcome of one maintenance pass.
func LastAutoGC ¶
LastAutoGC reads the opportunistic pass's attempt record.
func LastFullGC ¶
LastFullGC reads the full pass's record. ok is false when there is none, or it cannot be read: "never ran" and "cannot tell" lead to the same advice.
func LastGoCacheGC ¶ added in v1.0.0
LastGoCacheGC reads the Go-cache pass's record.
func NewGCResult ¶
NewGCResult summarises a pass's error into a record.
type GoCacheTrim ¶ added in v1.0.0
type GoCacheTrim struct {
// Cache names the cache ("shared go build cache"); Root is its directory.
Cache string `json:"cache"`
Root string `json:"root"`
// Applied is false for a dry run, which removes nothing.
Applied bool `json:"applied"`
// EstimatedBytes is the cache's estimated size before the trim,
// BudgetBytes its size budget, and YoungBytes the estimated bytes last
// used inside the safety floor, which no trim deletes.
EstimatedBytes uint64 `json:"estimated_bytes"`
BudgetBytes uint64 `json:"budget_bytes"`
YoungBytes uint64 `json:"young_bytes"`
// Cutoff is the last-use time below which entries were removed.
Cutoff time.Time `json:"cutoff"`
// Removed and FreedBytes are what the trim removed (or, dry, would have).
Removed int `json:"removed"`
FreedBytes uint64 `json:"freed_bytes"`
// ShardsCovered of ShardsTotal were walked to the end; Finished is true
// when that is all of them. A trim that did not finish resumes at the
// first uncovered shard next pass.
ShardsCovered int `json:"shards_covered"`
ShardsTotal int `json:"shards_total"`
Finished bool `json:"finished"`
// OldestLiveBuild is the age of the oldest running Go build that held the
// cutoff back past the floor (gocache_livebuild.go); zero when none did.
OldestLiveBuild time.Duration `json:"oldest_live_build_ns,omitempty"`
}
GoCacheTrim is what one trim of one Go-format build cache measured and did. It is recorded in the pass's GCResult (GoCaches), so whoever schedules the pass can see the cache's size, the part no trim may touch, and whether the walk covered every shard, without parsing human output.
type LayerError ¶
LayerError attributes a failure to the maintenance layer it came from, so a result can name the layers that failed without parsing error text.
func (*LayerError) Error ¶
func (e *LayerError) Error() string
func (*LayerError) Unwrap ¶
func (e *LayerError) Unwrap() error
type PlanEntry ¶
type PlanEntry struct {
Version
// Untagged marks a manifest no tag points at: an orphan left behind by a
// re-push of a reused tag, unreachable from anything retained.
Untagged bool
// Tags are the aliases that disappear with this manifest.
Tags []string
Bytes uint64
Sized bool
}
PlanEntry is one manifest selected for deletion, with the reason it became eligible and a size estimate when the manifest could be read.
type Policy ¶
type Policy struct {
Projects []string `json:"projects,omitempty"`
LogBudgetGiB uint64 `json:"log_budget_gib"`
HostReserveGiB uint64 `json:"host_reserve_gib"`
HostPaths []string `json:"host_paths,omitempty"`
BuildCacheGiB uint64 `json:"build_cache_gib"`
BuildCacheUnused string `json:"build_cache_unused"`
ImageUnused string `json:"image_unused"`
// GoCacheUnused is how long a shared Go build cache entry (and the
// golangci-lint cache) may sit unused before it is trimmed. Minimum 2h:
// Go refreshes an entry's mtime on use once it is an hour stale, so that
// floor is what keeps the trim safe under running builds.
GoCacheUnused string `json:"go_cache_unused"`
// GoCacheGiB is the size the shared Go caches are trimmed down to,
// oldest first, when age alone leaves them larger.
GoCacheGiB uint64 `json:"go_cache_gib"`
// GoCacheVolumePercent caps that budget at this share of the volume the
// cache lives on, so the default works on a 40 GiB build volume as well as
// on a 1 TB laptop: the budget is the smaller of go_cache_gib and this
// share. Zero means the default (15); 100 leaves go_cache_gib alone.
GoCacheVolumePercent int `json:"go_cache_volume_percent,omitempty"`
// GolangciCacheGiB is the size the golangci-lint cache is trimmed down
// to, least recently used first, like go_cache_gib for the build cache
// and capped the same way. Its own budget because its entries are tiny
// and recomputed in seconds: under go_cache_gib it was never trimmed.
GolangciCacheGiB uint64 `json:"golangci_cache_gib"`
// SourceCacheUnused is how long a cross-repo source clone may go
// unresolved before it becomes an eviction candidate. Longer than the
// image and build-cache windows because the cost of being wrong is a
// full re-clone of a repository, not a re-pull of a layer.
SourceCacheUnused string `json:"source_cache_unused"`
// SourceCacheKeep is how many of the most recently used clones of one
// repository are retained regardless of age. This is the floor that
// keeps an idle project's current pin warm, so it is validated at 1 or
// more: a policy that can empty a repository's only pin is not a policy
// anyone wants by accident.
SourceCacheKeep int `json:"source_cache_keep"`
// SourceCacheGiB is the size the source cache is evicted down to, least
// recently used clone first, never touching the newest source_cache_keep
// clones of a repository. The age rule alone does not bound it: a project
// that bumps its pin every few hours produces more clones in
// source_cache_unused than any age can hold.
SourceCacheGiB uint64 `json:"source_cache_gib"`
// SourceCacheSlimAfter is how long a clone beyond the newest
// source_cache_keep of its repository may sit unused before its installed
// dependencies (node_modules) are removed. Those are 80% of a clone's
// bytes and a build reinstalls them (frontenddeps.Ensure); the pristine
// checkout stays.
SourceCacheSlimAfter string `json:"source_cache_slim_after"`
RegistryDays int `json:"registry_days"`
RegistryKeep int `json:"registry_keep"`
DockerContext string `json:"docker_context,omitempty"`
Builders []string `json:"builders"`
Registries []Registry `json:"registries,omitempty"`
Clusters []string `json:"clusters,omitempty"`
Pins []string `json:"pins,omitempty"`
// Repos are git repositories whose linked worktrees the worktree layer
// reclaims, beyond the repository of each registered project: sibling
// repos a project builds from (docker.build_contexts, go.mod replaces).
// Filled by Converge; git itself enumerates the worktrees, wherever they live.
Repos []string `json:"repos,omitempty"`
// WorktreeRebuildable replaces the built-in allowlist of ignored paths a
// removable worktree may contain. Nil means the defaults.
WorktreeRebuildable []string `json:"worktree_rebuildable,omitempty"`
// WorktreeReap lets an explicit `forge storage gc --apply` remove
// worktrees. Off, the layer only reports. No automatic pass ever removes
// one, whatever this says (Runner.Explicit).
WorktreeReap bool `json:"worktree_reap,omitempty"`
// WorktreeReapReliantManaged extends worktree_reap to worktrees under
// Reliant's worktree root (~/.reliant/worktrees). Reliant owns those: it
// knows which chat each one is bound to, and reclaims them itself. Off,
// they are held like a locked worktree.
WorktreeReapReliantManaged bool `json:"worktree_reap_reliant_managed,omitempty"`
// MaxDeletesPerPass bounds how many filesystem entries one maintenance
// pass may unlink, across every layer. What is left is retained and the
// next pass continues, so a budget is reached over several passes rather
// than by one burst of six-figure unlinks. Zero means the default.
MaxDeletesPerPass int `json:"max_deletes_per_pass"`
// DeleteRatePerSec paces those unlinks. Zero means the default.
DeleteRatePerSec int `json:"delete_rate_per_sec"`
// AnonymousVolumeUnused is how long forge must have seen an anonymous
// Docker volume referenced by no container, running or stopped, before
// an explicit `forge storage gc --apply` may remove it (anonvolumes.go).
// Minimum 24h; "off" disables the layer. Named volumes are never removed,
// whatever this says.
AnonymousVolumeUnused string `json:"anonymous_volume_unused"`
// MaxDockerRemovalsPerPass bounds how many Docker images and volumes one
// pass asks the daemon to remove. Zero means the default.
MaxDockerRemovalsPerPass int `json:"max_docker_removals_per_pass"`
}
Policy defines one machine’s cache budgets and protected references.
func DefaultPolicy ¶
func DefaultPolicy() Policy
DefaultPolicy returns conservative local development budgets.
func (Policy) KubeletConfig ¶
KubeletConfig relies on kubelet's own usage tracking and GC, never on deleting containerd files underneath it. Unused age is independent of VM disk fullness.
type Registry ¶
type Registry struct {
Container string `json:"container"`
Repositories []string `json:"repositories"`
Aliases []string `json:"aliases"`
Contexts []string `json:"contexts"`
}
Registry explicitly identifies a local registry and all of its consumers.
type ReserveError ¶
ReserveError is a MEASURED shortfall: the filesystem under Path was read and holds less than the policy's host reserve. It is a distinct type because it is the one CheckSpace outcome a caller may knowingly accept — an ephemeral CI runner, whose disk is discarded with the job — while an unreadable disk or path is an unknown that no caller should wave through.
func (*ReserveError) Error ¶
func (e *ReserveError) Error() string
type Revision ¶
Revision is one manifest link on the registry filesystem and the moment it was written. Untagged manifests have no push timestamp anywhere else, so the link mtime is the only available age for them.
type Runner ¶
type Runner struct {
Policy Policy
Out io.Writer
Command func(context.Context, string, ...string) ([]byte, error)
// SourceCacheRoot overrides the cross-repo source cache the Sources
// layer reclaims from. Empty means the real machine-local cache in
// production, and is REFUSED under `go test` — see sources.go.
SourceCacheRoot string
// TempRoot overrides the system temp directory the TempSweep layer
// reclaims from. Empty means os.TempDir() in production, and is
// REFUSED under `go test` — see tempsweep.go.
TempRoot string
// PolicyPath is the policy file this pass was loaded from. Maintenance
// state that must outlive the process — the marker recording that a
// registry was stopped for GC — is kept beside it, so a later pass can
// find and repair what an interrupted one left behind.
PolicyPath string
// Ctx bounds the host-filesystem layers (logs, temp sweep, source cache),
// which take no context of their own. GC and NonDisruptiveGC set it from
// theirs, so a pass's deadline bounds the whole pass — its lsof snapshot
// and its walk over entries, not only its docker calls. Nil means
// unbounded.
Ctx context.Context
// Explicit marks the one pass a person or agent invoked: `forge storage
// gc` without --scheduled. Only it may remove what no observation can
// prove abandoned — worktrees (and then only under the policy's
// worktree_reap) and anonymous Docker volumes. Every automatic pass leaves
// it false: the background auto-gc, the installed schedule and `forge
// storage daemon`. An agent between two commands holds no process in its
// worktree, and an orphaned volume may be the only copy of a database, so
// to a pass nobody invoked either looks exactly like garbage.
Explicit bool
// Go-cache layer roots (gocache.go). Empty means the real machine
// location in production and is REFUSED under `go test`.
GoCacheRoot string // shared GOCACHE
GolangciCacheRoot string
// OpenPaths is the lsof snapshot. REFUSED under test when unset.
OpenPaths func(context.Context) (openfiles.Snapshot, error)
// OnGoCacheTrim, when set, receives what each Go-format cache trim
// measured and did (gocache.go). The CLI collects them into the pass's
// record. It may be called from the layer's goroutine; it is never called
// concurrently.
OnGoCacheTrim func(GoCacheTrim)
// LiveToolchainAges lists how long each running Go toolchain process has
// lived (gocache_livebuild.go). Nil reads the process table in
// production and means "no builds" under test.
LiveToolchainAges func(context.Context) ([]time.Duration, error)
// contains filtered or unexported fields
}
Runner executes maintenance using injectable process operations.
func (Runner) AnonymousVolumes ¶
AnonymousVolumes previews or applies the anonymous-volume rules. Only an explicit pass (Runner.Explicit) that applies removes anything; every other applying pass records what it observed and retains.
func (Runner) ConfigureNodes ¶
ConfigureNodes migrates existing k3d nodes without deleting their volumes. --apply is explicit because restarting a single-server cluster interrupts it.
func (Runner) DanglingImages ¶
DanglingImages removes every planned dangling image, or only lists them when apply is false. Oldest go first, so a pass stopped by its caps or deadline has reclaimed the longest-superseded builds.
func (Runner) GC ¶
GC previews or applies the registered cache and registry policy.
Every layer runs even when an earlier one failed: a layer's error is recorded and the combined error is returned at the end. One unreadable registered project used to end the pass at the first layer, so the temp sweep, source eviction, BuildKit and registry retention silently never ran. Two checks still stop everything, because nothing after them is safe without them: an invalid policy, and a Docker endpoint that is not local (which ends the Docker layers, not the host ones already done).
func (Runner) GoCacheGC ¶ added in v1.0.0
GoCacheGC runs only the Go-toolchain caches (GoCaches), with the whole of ctx's budget rather than the slice a full pass gives them.
It exists because the shared build cache fills on a different clock from everything else: under agent load ~25 GiB an hour crosses the trim's 2h safety floor, so a trim that runs hourly — or one whose share of a larger pass runs out under load — leaves tens of GiB that could already go on the disk. This pass is cheap enough (one walk of the cache, paced unlinks, no Docker, no registry) to run every few minutes, and it is non-disruptive for the same reason the Go-cache layer of NonDisruptiveGC is.
func (Runner) GoCaches ¶
GoCaches trims every Go-toolchain cache. Dry-run unless apply is set. Layers run independently; one failure does not stop the rest.
The shared build cache is the layer that matters most and it grows ~40 GB/h under agent load, so it runs FIRST and each step gets a guaranteed slice of what remains: a slow step cannot starve the next. A step cut off by its slice has still reclaimed what it covered (see trimGoBuildCache); that is a partial success, never a failure. Only real errors are returned.
The delete budget is shared the same way: each step may spend a share of what the steps before it left (deletes.go), so the shared cache cannot spend the whole pass on itself. A step that reaches its share stops and is reported as cut off; a back-off ends every step after it.
func (Runner) GolangciLintCache ¶
GolangciLintCache trims golangci-lint's fork of the Go cache format.
func (Runner) LocalImages ¶
LocalImages untags every StaleLocalImage, or only lists them when apply is false. `docker rmi <ref>` removes the layers once the last tag is gone, and fails rather than force-removing an image a container has since started using, which is reported and does not stop the rest.
func (Runner) Logs ¶
Logs only expires rotated Forge logs. Current streams, releases, promotion history and other files in .forge are never candidates.
Each registered project is maintained independently: one that cannot be read (a project under ~/Documents the scheduled job has no TCC access to, a permission error, a log removed underneath the pass) is reported in the combined error and the remaining projects are still maintained.
func (Runner) NonDisruptiveGC ¶
NonDisruptiveGC runs only the layers that cannot interrupt anything a developer is currently using: expiring rotated logs, trimming the Go caches, removing dangling images, evicting unused builder cache, the temp sweep and the source cache.
Registry GC and node reconfiguration are deliberately NOT here. Registry cleanup takes the registry offline for the duration (pushes and pulls fail), and node reconfiguration restarts kubelet — both are fine for a scheduled 03:30 pass and unacceptable as a side effect of `forge env up`, which is typically the command that is about to push to that registry.
Worktrees are not here either, not even as a preview. This pass runs behind the user's back — at the end of `forge env up`, and hourly from the installed schedule — and nothing it can observe tells an abandoned worktree from one an agent is between two commands in: no process holds it open, and its files can sit untouched for a day while the agent reads. Removing one is a decision only an explicit `forge storage gc --apply` may take. Anonymous volumes are the same decision about data: this pass records which ones are unused, so their clocks start, and never removes one (anonvolumes.go).
As in GC, each layer's failure is recorded and the rest still run — unless the filesystem reported distress, which stops the pass (deletes.go).
func (Runner) PrintHeldWorktrees ¶
PrintHeldWorktrees reports the worktrees the reaper will not touch, by reason, for `forge storage status`.
func (Runner) Prune ¶
Prune removes stale clusters and projects from the policy at policyPath and saves it. The caller must hold the maintenance lock (WithLock), as GC does.
func (Runner) RegisterBuilder ¶
RegisterBuilder adopts a named local builder without requiring a registry or cluster. Its endpoint is checked again before every maintenance pass.
func (Runner) RegistryGC ¶
RegistryGC deletes eligible manifests and collects blobs with all writers stopped.
func (Runner) SharedGoCache ¶
SharedGoCache trims the shared `go env GOCACHE`.
func (Runner) Sources ¶
Sources reclaims cross-repo source clones that no project has resolved recently. One full clone exists per repo+ref pin and nothing else ever removes one, so a project that bumps its pin weekly accumulates a complete checkout per bump — 9.2 GB of one repository on the machine this layer was written for, most of it an installed node_modules.
The eviction MECHANISM and its safety rules live in internal/gitsource, next to the code that writes the cache; the BUDGETS come from this machine's storage policy, the way every other layer's do, so a developer tunes one file rather than learning where each reclaimer hides its defaults. The newest-K floor means an idle project's current pin survives, so this is safe to run unattended.
An unparseable SourceCacheUnused cannot happen on a loaded policy — Validate rejects it and GC calls Validate first — so it is treated the same as an unset value: the gitsource default, never "evict everything".
A cache that cannot be located is not a failure: os.UserCacheDir fails on a host with no HOME, where there is also nothing to reclaim.
func (Runner) StaleLocalImages ¶
func (r Runner) StaleLocalImages(ctx context.Context) ([]StaleLocalImage, error)
StaleLocalImages lists host image tags whose repository host is a local registry alias (`localhost:<port>`, `registry.localhost:<port>`, `k3d-<name>:<port>`) that no running container backs, and whose image no container, running or stopped, uses.
An alias is dead only when NO container, running or stopped, backs it: a `localhost` or `registry.localhost` alias is backed by any container whose port bindings include that host port, a `k3d-<name>` alias by any container of that name, and a registry registered in the policy by its container name. A stopped registry (the full pass stops it for garbage-collect; Docker Desktop restarts leave k3d registries stopped) therefore keeps its images. Anything unparseable is kept; a failed docker call fails the layer rather than guessing.
func (Runner) TempSweep ¶
TempSweep reclaims abandoned scratch entries in the system temp directory. Dry-run by default: it prints each candidate and its size, and removes nothing unless apply is set.
func (Runner) Worktrees ¶
func (r Runner) Worktrees(ctx context.Context, repo, base string, idle time.Duration, apply, reliantManaged bool) error
Worktrees is the explicit `forge storage worktrees` entry point. It runs the same classifier the GC layer does, over one repository. reliantManaged includes worktrees under Reliant's worktree root, which are otherwise held.
type StaleLocalImage ¶
StaleLocalImage is one host image reference to be untagged.
type WorktreeHold ¶
type WorktreeHold struct {
Path, Repo, Reason, Detail string
}
WorktreeHold is one worktree the reaper left alone, and why.
type WorktreeReport ¶
type WorktreeReport struct {
// Removable are worktrees that passed every check (removed when applied).
Removable []string
// Removed is the subset actually removed.
Removed []string
Held []WorktreeHold
}
WorktreeReport is the outcome of one classification pass.
func (WorktreeReport) HeldByReason ¶
func (w WorktreeReport) HeldByReason() map[string]int
HeldByReason counts held worktrees per reason.
Source Files
¶
- allocated_unix.go
- anonvolumes.go
- builders.go
- converge.go
- danglingimages.go
- deletes.go
- disk_unix.go
- dockerobjects.go
- gcrecord.go
- gocache.go
- gocache_livebuild.go
- local.go
- localimages.go
- lock_unix.go
- logs.go
- nodes.go
- passctx.go
- policy.go
- prune.go
- register.go
- registry.go
- registry_refs.go
- sources.go
- tempsweep.go
- worktrees.go
- worktrees_data.go