Documentation
¶
Overview ¶
Package janitor bounds a project's .cloop directory automatically (Task 20229).
cloop had three retention mechanisms before this package and ran none of them on its own: `cloop compact` prunes snapshots and artifacts, `cloop db maintain` vacuums the database, and `cloop hub audit prune` seals and trims the audit trail. All three are operator-invoked. A hub nobody is babysitting therefore grows until the disk fills — measured on this repository's own control plane at 4.9 GB, of which roughly 4.3 GB was reclaimable: 2 GB of SQLite freelist that no `du` reports, and 2 GB of plan history in 3,983 files. That is an availability defect, not housekeeping, so the fix belongs inside the hub process rather than in a runbook.
What a pass does, in order:
- Bound .cloop/plan-history to the configured keep-count. This is a catch-up pass: pm.SaveSnapshot enforces the same bound at write time, so steady-state history never exceeds it between passes. The janitor exists to reclaim installs that predate the write-time bound, and to apply a keep-count an operator has since lowered.
- Apply size-and-age retention to .cloop/audit-archive.
- VACUUM the database, but only when the freelist is large enough in proportion to the file to be worth it.
Every step is individually skippable and none of them aborts the others: a pass that cannot read the archive directory still vacuums.
Why the VACUUM is conditional ¶
VACUUM rewrites the entire database file and needs free space to do it — SQLite copies the live pages out and writes them back through the journal, so it consumes space before it returns any. That is the exact thing you cannot afford on a disk that is nearly full, which is when a janitor is most likely to want it. Running it daily regardless of need would be a worse defect than the one it fixes. So the janitor reads freelist_count and page_count (both O(1) header reads, unlike dbstat, which scans every page) and vacuums only when the free fraction crosses a threshold, only when the filesystem has room for the rebuild, and only after the file-level steps above have released what they can.
Why it takes the lease, and why it still declines ¶
VACUUM rebuilds the file underneath every open connection. Task 20214's hub lease already answers "is another hub live against this database" exactly, so the janitor refuses to vacuum unless the live lease holder is the process it is running inside. Two hubs sharing a directory — which this deployment has had — cannot both decide to rewrite the file.
Owning the lease is necessary but not sufficient. The lease *lives in the database being rewritten*, so a VACUUM long enough to outlast its TTL starves the heartbeat and the hub stands down believing a peer took over. DefaultVacuumMaxInlineBytes is the ceiling that prevents it; past that the rewrite belongs to `cloop hub retention --apply` with the hub stopped, which has no heartbeat to lose.
Index ¶
- Constants
- func ApplySnapshotRetention(cfg *config.Config)
- func Due(workDir string, interval time.Duration, now time.Time) bool
- func LastRun(workDir string) (t time.Time, ok bool)
- func MarkRun(workDir string, now time.Time) error
- func SizeStatsFor(workDir string) (statedb.SizeStats, error)
- func StampPath(workDir string) string
- type Options
- type Policy
- type Report
- type StepResult
Constants ¶
const ( // DefaultInterval is how often the hub runs a pass. Daily: the growth // this bounds is measured in days, and a shorter interval would spend // I/O re-reading directories that have not changed. DefaultInterval = 24 * time.Hour // DefaultVacuumFreeRatio is the freelist fraction at which a VACUUM // becomes worth its cost. At 0.30, a database is rewritten only once // nearly a third of it is dead space — well above the 0.25 that `cloop // doctor` has always used to *warn*, so the janitor acts strictly after // the point an operator was already being told to act. DefaultVacuumFreeRatio = 0.30 // DefaultVacuumMinFreeBytes stops the ratio triggering pointless work on // a small database. A fresh project can be 90% freelist and still only // waste a few hundred kilobytes; rewriting it every day to reclaim that // is cost without benefit. DefaultVacuumMinFreeBytes int64 = 64 << 20 // 64 MiB // DefaultVacuumMaxInlineBytes bounds how much *live* data the in-hub // janitor will rewrite without stopping the hub, and it exists because // the obvious implementation takes the hub down. // // VACUUM holds a write lock for its whole duration, and the control-plane // database is where the hub's own lease lives. So a VACUUM that outlasts // the lease TTL (hublease.DefaultTTL, 60s) starves the heartbeat, the hub // concludes a peer took the fence, and Run returns "standing down" — a // self-inflicted outage misreported as a second instance. Worse, it would // repeat: the pass runs 30s after startup, so a hub whose database is // large enough never stays up. // // The bound is on live bytes rather than file size because live bytes are // what VACUUM copies — a 2.3 GB file holding 290 MB of real data is a // sub-second rewrite, and gating on 2.3 GB would refuse exactly the // mostly-empty databases that most need reclaiming. 1 GiB at a // conservative 100 MB/s is ~10s: longer than one 20s renewal interval can // absorb without a retry, comfortably inside the 60s that would actually // lose the fence. // // Above the bound the janitor declines and says so; `cloop hub retention // --apply` with the hub stopped has no lease to lose and no ceiling. DefaultVacuumMaxInlineBytes int64 = 1 << 30 // 1 GiB of live data )
Defaults for a policy an operator has not configured. Plan-history and VACUUM default on because both grow on every install; audit-archive retention defaults off — see Policy.ArchiveMaxBytes.
Variables ¶
This section is empty.
Functions ¶
func ApplySnapshotRetention ¶
ApplySnapshotRetention pushes the configured plan-history keep-count into pkg/pm, which enforces it inside SaveSnapshot.
This is what stops history outrunning the janitor between passes: a busy project writes a full plan copy on every mutation, so thousands of snapshots can accumulate inside a single daily interval. Binding the value at process start means every writer — the hub's UI mutations and the orchestrator subprocess alike — honours the operator's policy, without threading it through SaveSnapshot's dozen call sites.
A disabled janitor disables the write-time bound too: an operator who turned retention off asked for their history to be kept.
func Due ¶
Due reports whether interval has elapsed since the last recorded pass. No stamp means due, so a fresh project is cleaned shortly after the hub starts.
func LastRun ¶
LastRun reports when the last pass completed. ok is false when no pass has been recorded, which is not an error: it is what a hub that has just started for the first time looks like.
func MarkRun ¶
MarkRun records that a pass just completed. Best-effort by contract: a failure here means the next sweep repeats the pass, which is harmless, and must not be mistaken for the pass itself having failed.
func SizeStatsFor ¶
SizeStatsFor reads the database's page statistics without a full disk walk. Exposed for callers that want the VACUUM threshold inputs alone.
func StampPath ¶
StampPath is where the hub records the completion time of the last pass, so the daily cadence survives a restart and `cloop hub doctor` can report it.
A file rather than a row in state.db, and deliberately: the janitor has to work on a project whose database is the thing that is broken. The stamp is advisory — losing it costs one extra pass, which is idempotent and cheap.
Types ¶
type Options ¶
type Options struct {
// WorkDir is the project root; the janitor operates on WorkDir/.cloop.
WorkDir string
Policy Policy
// DryRun reports what a pass would reclaim without deleting or vacuuming.
DryRun bool
// InstanceID is the hub lease identity of the process running this pass.
// When empty, the VACUUM step refuses if *any* hub holds the lease — the
// correct behaviour for an out-of-process caller such as the CLI. When
// set, it refuses only if the holder is somebody else.
InstanceID string
// SkipVacuum suppresses the VACUUM regardless of the freelist, and
// SkipVacuumReason says why. The hub sets this for a project that is
// currently executing a task: SQLite would serialise the rewrite against
// the orchestrator's writes rather than corrupt anything, but stalling a
// live run for the minutes a multi-gigabyte rewrite takes is not a
// trade the janitor gets to make unprompted. The file-level steps still
// run — pruning old snapshots is safe under a live writer.
SkipVacuum bool
SkipVacuumReason string
// Now is the clock, injectable for tests. Nil means time.Now.
Now func() time.Time
}
Options configures a single pass.
type Policy ¶
type Policy struct {
// Enabled gates the whole janitor. False means the hub starts no
// background pass at all — the documented opt-out.
Enabled bool
// Interval is how often the hub runs a pass. Zero selects
// DefaultInterval.
Interval time.Duration
// KeepSnapshots bounds .cloop/plan-history. Zero or less disables
// plan-history pruning and keeps everything.
KeepSnapshots int
// ArchiveMaxBytes caps the total size of .cloop/audit-archive; the
// oldest seals are deleted until the directory fits. Zero disables the
// size cap.
//
// Both archive limits default to off, and deliberately so. A seal is the
// only copy of audit rows that have already been deleted from the
// database — deleting one destroys the record rather than relocating it,
// which is the opposite of what the rest of the audit path does. It is
// also not a default-install problem: archives exist only if the operator
// opted into audit retention in the first place (config audit.retention_days
// is itself zero by default), so an operator who has seals has already
// made a retention decision and is the right person to make this one too.
// `cloop doctor` reports the directory's size so that decision is
// prompted by data rather than by a full disk.
ArchiveMaxBytes int64
// ArchiveMaxAgeDays deletes seals older than this many days. Zero
// disables the age limit. See ArchiveMaxBytes for why this is opt-in.
ArchiveMaxAgeDays int
// VacuumFreeRatio is the freelist fraction at or above which the janitor
// vacuums. Zero selects DefaultVacuumFreeRatio; a value >= 1 disables
// vacuuming, since no database can be more than entirely free.
VacuumFreeRatio float64
// VacuumMinFreeBytes is an absolute floor beneath which the ratio is
// ignored. Zero selects DefaultVacuumMinFreeBytes; negative disables the
// floor.
VacuumMinFreeBytes int64
// VacuumMaxInlineBytes caps the live data an *in-process* VACUUM will
// rewrite, protecting the hub's own lease heartbeat from a rewrite that
// outlasts it. Zero selects DefaultVacuumMaxInlineBytes; negative removes
// the cap. It applies only when Options.InstanceID is set — a caller with
// no lease at stake has nothing to protect.
VacuumMaxInlineBytes int64
// ArchiveDir is the directory holding sealed audit exports. Empty means
// <workDir>/.cloop/audit-archive, which is auditretention's default — but
// a real deployment is advised to point audit.export_dir at storage the
// hub does not otherwise own, and retention has to follow the seals there
// or it silently bounds an empty directory while the real one grows.
ArchiveDir string
}
Policy is the retention configuration for one pass.
func DefaultPolicy ¶
func DefaultPolicy() Policy
DefaultPolicy returns the policy a hub uses when config says nothing.
func PolicyFromConfig ¶
PolicyFromConfig translates the operator-facing config section into a policy. A nil config yields the defaults, so a caller that failed to load config.yaml still gets a hub that bounds its disk.
Config expresses the two byte-valued limits in megabytes because that is what an operator writes; the conversion lives here rather than in pkg/config so the config struct stays a faithful record of the file.
type Report ¶
type Report struct {
WorkDir string `json:"work_dir"`
StartedAt time.Time `json:"started_at"`
Duration time.Duration `json:"duration"`
DryRun bool `json:"dry_run"`
PlanHistory StepResult `json:"plan_history"`
Archive StepResult `json:"audit_archive"`
Vacuum StepResult `json:"vacuum"`
// Before and After are the .cloop breakdowns either side of the pass.
// After is nil for a dry run, which changes nothing.
Before *diskusage.Usage `json:"before,omitempty"`
After *diskusage.Usage `json:"after,omitempty"`
}
Report describes a completed pass.
func RunOnce ¶
RunOnce performs a single retention pass. It returns a Report even when individual steps fail; the error return is reserved for a failure that prevented the pass from being attempted at all.
func (*Report) BytesFreed ¶
BytesFreed totals what the pass reclaimed.
type StepResult ¶
type StepResult struct {
// Ran is false when the step was disabled or had nothing to do; Reason
// then says which.
Ran bool
// Skipped marks a step that was applicable but deliberately not run —
// principally the VACUUM below its threshold. Distinguished from "ran and
// found nothing" so an operator can tell a working threshold from a
// broken step.
Skipped bool
Reason string
Deleted int
BytesFreed int64
Err error
}
StepResult is the outcome of one retention step.