janitor

package
v0.0.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 27, 2026 License: MIT Imports: 13 Imported by: 0

Documentation

Overview

Package janitor bounds a project's .cloop directory automatically (Task 20229).

cloop had three retention mechanisms before this package and ran none of them on its own: `cloop compact` prunes snapshots and artifacts, `cloop db maintain` vacuums the database, and `cloop hub audit prune` seals and trims the audit trail. All three are operator-invoked. A hub nobody is babysitting therefore grows until the disk fills — measured on this repository's own control plane at 4.9 GB, of which roughly 4.3 GB was reclaimable: 2 GB of SQLite freelist that no `du` reports, and 2 GB of plan history in 3,983 files. That is an availability defect, not housekeeping, so the fix belongs inside the hub process rather than in a runbook.

What a pass does, in order:

  1. Bound .cloop/plan-history to the configured keep-count. This is a catch-up pass: pm.SaveSnapshot enforces the same bound at write time, so steady-state history never exceeds it between passes. The janitor exists to reclaim installs that predate the write-time bound, and to apply a keep-count an operator has since lowered.
  2. Apply size-and-age retention to .cloop/audit-archive.
  3. VACUUM the database, but only when the freelist is large enough in proportion to the file to be worth it.

Every step is individually skippable and none of them aborts the others: a pass that cannot read the archive directory still vacuums.

Why the VACUUM is conditional

VACUUM rewrites the entire database file and needs free space to do it — SQLite copies the live pages out and writes them back through the journal, so it consumes space before it returns any. That is the exact thing you cannot afford on a disk that is nearly full, which is when a janitor is most likely to want it. Running it daily regardless of need would be a worse defect than the one it fixes. So the janitor reads freelist_count and page_count (both O(1) header reads, unlike dbstat, which scans every page) and vacuums only when the free fraction crosses a threshold, only when the filesystem has room for the rebuild, and only after the file-level steps above have released what they can.

Why it takes the lease, and why it still declines

VACUUM rebuilds the file underneath every open connection. Task 20214's hub lease already answers "is another hub live against this database" exactly, so the janitor refuses to vacuum unless the live lease holder is the process it is running inside. Two hubs sharing a directory — which this deployment has had — cannot both decide to rewrite the file.

Owning the lease is necessary but not sufficient. The lease *lives in the database being rewritten*, so a VACUUM long enough to outlast its TTL starves the heartbeat and the hub stands down believing a peer took over. DefaultVacuumMaxInlineBytes is the ceiling that prevents it; past that the rewrite belongs to `cloop hub retention --apply` with the hub stopped, which has no heartbeat to lose.

Index

Constants

View Source
const (
	// DefaultInterval is how often the hub runs a pass. Daily: the growth
	// this bounds is measured in days, and a shorter interval would spend
	// I/O re-reading directories that have not changed.
	DefaultInterval = 24 * time.Hour

	// DefaultVacuumFreeRatio is the freelist fraction at which a VACUUM
	// becomes worth its cost. At 0.30, a database is rewritten only once
	// nearly a third of it is dead space — well above the 0.25 that `cloop
	// doctor` has always used to *warn*, so the janitor acts strictly after
	// the point an operator was already being told to act.
	DefaultVacuumFreeRatio = 0.30

	// DefaultVacuumMinFreeBytes stops the ratio triggering pointless work on
	// a small database. A fresh project can be 90% freelist and still only
	// waste a few hundred kilobytes; rewriting it every day to reclaim that
	// is cost without benefit.
	DefaultVacuumMinFreeBytes int64 = 64 << 20 // 64 MiB

	// DefaultVacuumMaxInlineBytes bounds how much *live* data the in-hub
	// janitor will rewrite without stopping the hub, and it exists because
	// the obvious implementation takes the hub down.
	//
	// VACUUM holds a write lock for its whole duration, and the control-plane
	// database is where the hub's own lease lives. So a VACUUM that outlasts
	// the lease TTL (hublease.DefaultTTL, 60s) starves the heartbeat, the hub
	// concludes a peer took the fence, and Run returns "standing down" — a
	// self-inflicted outage misreported as a second instance. Worse, it would
	// repeat: the pass runs 30s after startup, so a hub whose database is
	// large enough never stays up.
	//
	// The bound is on live bytes rather than file size because live bytes are
	// what VACUUM copies — a 2.3 GB file holding 290 MB of real data is a
	// sub-second rewrite, and gating on 2.3 GB would refuse exactly the
	// mostly-empty databases that most need reclaiming. 1 GiB at a
	// conservative 100 MB/s is ~10s: longer than one 20s renewal interval can
	// absorb without a retry, comfortably inside the 60s that would actually
	// lose the fence.
	//
	// Above the bound the janitor declines and says so; `cloop hub retention
	// --apply` with the hub stopped has no lease to lose and no ceiling.
	DefaultVacuumMaxInlineBytes int64 = 1 << 30 // 1 GiB of live data
)

Defaults for a policy an operator has not configured. Plan-history and VACUUM default on because both grow on every install; audit-archive retention defaults off — see Policy.ArchiveMaxBytes.

View Source
const (
	// DefaultProviderCallBodyMaxAge is when a recorded prompt and response are
	// cleared, leaving the row's metadata behind.
	//
	// Thirty days is chosen from what the bodies are for. The inspector's
	// per-call modal and its replay button are debugging tools aimed at a call
	// you are currently reasoning about; their value decays in hours, while the
	// row's timestamp, model, token counts and latency keep feeding the list
	// and every cost-shaped report indefinitely. At the measured 178 KB per row
	// this is the single largest reclaim in the whole janitor.
	DefaultProviderCallBodyMaxAge = 30 * 24 * time.Hour

	// DefaultProviderCallMaxRows bounds the table itself. A stripped row costs
	// roughly 200 bytes, so ten thousand of them is about 2 MB — small enough
	// to keep a long history, finite enough that the table is bounded.
	DefaultProviderCallMaxRows = 10000

	// DefaultStepMaxRows bounds the step history. Steps are loaded in full by
	// state.Load, so this ceiling is also a bound on a run's resident memory;
	// at the measured average of roughly 450 bytes of output per step, twenty
	// thousand is about 9 MB.
	DefaultStepMaxRows = 20000

	// DefaultEventMaxRows bounds the unified event journal, which the dashboard
	// pages through newest-first and never reads whole.
	DefaultEventMaxRows = 20000

	// DefaultCostMaxRows bounds the cost ledger, and is deliberately the
	// loosest of these: a cost row is tiny, `cloop cost report` aggregates over
	// the entire table, and the hub's spend enforcement reads forward from a
	// stored cursor. At the measured rate this ceiling is decades away, which
	// is the intent — it exists so the table is bounded, not so it is pruned.
	DefaultCostMaxRows = 50000

	// DefaultTelemetryMaxAge ages out browser diagnostic trails. The write path
	// already caps the table at statedb.TelemetryMaxRows — a public ingest
	// endpoint must not be able to fill a disk between two passes — so this is
	// the half no write-time cap can express: a trail nobody came back for.
	DefaultTelemetryMaxAge = 30 * 24 * time.Hour
)

Defaults for the row tables (Task 20291). Every one of these is on by default, because unlike the audit archive these tables hold derived observability data with a second copy or no lasting value, and unlike plan history nothing bounded them at write time.

The numbers come from measuring this repository's control plane, where provider_calls held 47.5 MB in 265 rows — 14% of everything live in a 2.4 GB file — and steps held 5,152 rows after four and a half months.

Variables

This section is empty.

Functions

func ApplySnapshotRetention

func ApplySnapshotRetention(cfg *config.Config)

ApplySnapshotRetention pushes the configured plan-history keep-count into pkg/pm, which enforces it inside SaveSnapshot.

This is what stops history outrunning the janitor between passes: a busy project writes a full plan copy on every mutation, so thousands of snapshots can accumulate inside a single daily interval. Binding the value at process start means every writer — the hub's UI mutations and the orchestrator subprocess alike — honours the operator's policy, without threading it through SaveSnapshot's dozen call sites.

A disabled janitor disables the write-time bound too: an operator who turned retention off asked for their history to be kept.

func Due

func Due(workDir string, interval time.Duration, now time.Time) bool

Due reports whether interval has elapsed since the last recorded pass. No stamp means due, so a fresh project is cleaned shortly after the hub starts.

func LastRun

func LastRun(workDir string) (t time.Time, ok bool)

LastRun reports when the last pass completed. ok is false when no pass has been recorded, which is not an error: it is what a hub that has just started for the first time looks like.

func MarkRun

func MarkRun(workDir string, now time.Time) error

MarkRun records that a pass just completed. Best-effort by contract: a failure here means the next sweep repeats the pass, which is harmless, and must not be mistaken for the pass itself having failed.

func RowTableStats added in v0.0.2

func RowTableStats(workDir string) ([]statedb.TableStat, error)

RowTableStats measures the row tables without pruning anything.

Returns an error wrapping os.ErrNotExist when the project has no database, which lets a caller distinguish "nothing to report" from "could not read it".

func SizeStatsFor

func SizeStatsFor(workDir string) (statedb.SizeStats, error)

SizeStatsFor reads the database's page statistics without a full disk walk. Exposed for callers that want the VACUUM threshold inputs alone.

func StampPath

func StampPath(workDir string) string

StampPath is where the hub records the completion time of the last pass, so the daily cadence survives a restart and `cloop hub doctor` can report it.

A file rather than a row in state.db, and deliberately: the janitor has to work on a project whose database is the thing that is broken. The stamp is advisory — losing it costs one extra pass, which is idempotent and cheap.

func TableBounded added in v0.0.2

func TableBounded(pol Policy, table string) bool

TableBounded reports whether pol puts a ceiling on the named table's size.

A ceiling, specifically: an age limit is not one. A table pruned only by age still grows without bound under an unbounded write rate, and the question the hub doctor is asking — "will this hub still have a disk" — is answered by the ceiling or not at all. provider_calls is the exception worth stating: its body strip is an age limit and is not what makes it bounded; its row ceiling is.

An unknown table name is reported as unbounded, so a table added to statedb.GrowthTables without a matching policy knob shows up in the doctor rather than passing quietly.

Types

type Options

type Options struct {
	// WorkDir is the project root; the janitor operates on WorkDir/.cloop.
	WorkDir string
	Policy  Policy

	// DryRun reports what a pass would reclaim without deleting or vacuuming.
	DryRun bool

	// InstanceID is the hub lease identity of the process running this pass.
	// When empty, the VACUUM step refuses if *any* hub holds the lease — the
	// correct behaviour for an out-of-process caller such as the CLI. When
	// set, it refuses only if the holder is somebody else.
	InstanceID string

	// SkipVacuum suppresses the VACUUM regardless of the freelist, and
	// SkipVacuumReason says why. The hub sets this for a project that is
	// currently executing a task: SQLite would serialise the rewrite against
	// the orchestrator's writes rather than corrupt anything, but stalling a
	// live run for the minutes a multi-gigabyte rewrite takes is not a
	// trade the janitor gets to make unprompted. The file-level steps still
	// run — pruning old snapshots is safe under a live writer.
	SkipVacuum       bool
	SkipVacuumReason string

	// RunActive tells the pass that this project is currently executing a
	// task, and RunActiveReason says how the caller knows. It suppresses the
	// steps prune only.
	//
	// Separate from SkipVacuum because it protects against a different thing.
	// SkipVacuum is about latency: a rewrite would stall a live run's writes.
	// This is about correctness: a run keeps the whole step history in memory
	// and upserts all of it on every save, so rows deleted underneath it come
	// back. Every other row table here is append-only and safe to prune while a
	// task runs.
	RunActive       bool
	RunActiveReason string

	// Now is the clock, injectable for tests. Nil means time.Now.
	Now func() time.Time
}

Options configures a single pass.

type Policy

type Policy struct {
	// Enabled gates the whole janitor. False means the hub starts no
	// background pass at all — the documented opt-out.
	Enabled bool

	// Interval is how often the hub runs a pass. Zero selects
	// DefaultInterval.
	Interval time.Duration

	// KeepSnapshots bounds .cloop/plan-history. Zero or less disables
	// plan-history pruning and keeps everything.
	KeepSnapshots int

	// ArchiveMaxBytes caps the total size of .cloop/audit-archive; the
	// oldest seals are deleted until the directory fits. Zero disables the
	// size cap.
	//
	// Both archive limits default to off, and deliberately so. A seal is the
	// only copy of audit rows that have already been deleted from the
	// database — deleting one destroys the record rather than relocating it,
	// which is the opposite of what the rest of the audit path does. It is
	// also not a default-install problem: archives exist only if the operator
	// opted into audit retention in the first place (config audit.retention_days
	// is itself zero by default), so an operator who has seals has already
	// made a retention decision and is the right person to make this one too.
	// `cloop doctor` reports the directory's size so that decision is
	// prompted by data rather than by a full disk.
	ArchiveMaxBytes int64

	// ArchiveMaxAgeDays deletes seals older than this many days. Zero
	// disables the age limit. See ArchiveMaxBytes for why this is opt-in.
	ArchiveMaxAgeDays int

	// VacuumFreeRatio is the freelist fraction at or above which the janitor
	// vacuums. Zero selects DefaultVacuumFreeRatio; a value >= 1 disables
	// vacuuming, since no database can be more than entirely free.
	VacuumFreeRatio float64

	// VacuumMinFreeBytes is an absolute floor beneath which the ratio is
	// ignored. Zero selects DefaultVacuumMinFreeBytes; negative disables the
	// floor.
	VacuumMinFreeBytes int64

	// VacuumMaxInlineBytes caps the live data an *in-process* VACUUM will
	// rewrite, protecting the hub's own lease heartbeat from a rewrite that
	// outlasts it. Zero selects DefaultVacuumMaxInlineBytes; negative removes
	// the cap. It applies only when Options.InstanceID is set — a caller with
	// no lease at stake has nothing to protect.
	VacuumMaxInlineBytes int64

	// ProviderCallBodyMaxAge clears the prompt, system prompt and response of
	// provider_calls rows older than this, keeping the row. See
	// DefaultProviderCallBodyMaxAge for why the bodies and the row have
	// different lifetimes.
	ProviderCallBodyMaxAge time.Duration

	// ProviderCallMaxRows, StepMaxRows, EventMaxRows and CostMaxRows are the
	// per-table row ceilings; the newest rows survive.
	ProviderCallMaxRows int
	StepMaxRows         int
	EventMaxRows        int
	CostMaxRows         int

	// TelemetryMaxAge deletes browser telemetry older than this. Age rather
	// than count because the table already has a write-time row cap; what it
	// lacks is a way for a quiet trail to expire.
	TelemetryMaxAge time.Duration

	// ArchiveDir is the directory holding sealed audit exports. Empty means
	// <workDir>/.cloop/audit-archive, which is auditretention's default — but
	// a real deployment is advised to point audit.export_dir at storage the
	// hub does not otherwise own, and retention has to follow the seals there
	// or it silently bounds an empty directory while the real one grows.
	ArchiveDir string
}

Policy is the retention configuration for one pass.

func DefaultPolicy

func DefaultPolicy() Policy

DefaultPolicy returns the policy a hub uses when config says nothing.

func PolicyFromConfig

func PolicyFromConfig(cfg *config.Config) Policy

PolicyFromConfig translates the operator-facing config section into a policy. A nil config yields the defaults, so a caller that failed to load config.yaml still gets a hub that bounds its disk.

Config expresses the two byte-valued limits in megabytes because that is what an operator writes; the conversion lives here rather than in pkg/config so the config struct stays a faithful record of the file.

type Report

type Report struct {
	WorkDir     string        `json:"work_dir"`
	StartedAt   time.Time     `json:"started_at"`
	Duration    time.Duration `json:"duration"`
	DryRun      bool          `json:"dry_run"`
	PlanHistory StepResult    `json:"plan_history"`
	Archive     StepResult    `json:"audit_archive"`
	// The row tables (Task 20291), pruned before the VACUUM so it has
	// something to reclaim.
	ProviderCalls StepResult `json:"provider_calls"`
	Steps         StepResult `json:"steps"`
	Events        StepResult `json:"events"`
	Costs         StepResult `json:"costs"`
	Telemetry     StepResult `json:"telemetry"`
	Vacuum        StepResult `json:"vacuum"`
	// Before and After are the .cloop breakdowns either side of the pass.
	// After is nil for a dry run, which changes nothing.
	Before *diskusage.Usage `json:"before,omitempty"`
	After  *diskusage.Usage `json:"after,omitempty"`
}

Report describes a completed pass.

func RunOnce

func RunOnce(opts Options) (*Report, error)

RunOnce performs a single retention pass. It returns a Report even when individual steps fail; the error return is reserved for a failure that prevented the pass from being attempted at all.

func (*Report) BytesFreed

func (r *Report) BytesFreed() int64

BytesFreed totals what the pass returned to the filesystem.

Row pruning is not in this sum: deleting a row frees a database page, not a byte of disk, and the VACUUM step below already reports the shrink those pages made possible. See BytesReleased.

func (*Report) BytesReleased added in v0.0.2

func (r *Report) BytesReleased() int64

BytesReleased totals what row pruning returned to the database freelist, which the VACUUM step reclaims if it runs.

func (*Report) Errs

func (r *Report) Errs() []error

Errs collects the per-step failures, if any.

func (*Report) RowsDeleted added in v0.0.2

func (r *Report) RowsDeleted() int

RowsDeleted totals the rows removed across every table.

func (*Report) Summary

func (r *Report) Summary() string

Summary renders a one-line operator-facing description of the pass.

type StepResult

type StepResult struct {
	// Ran is false when the step was disabled or had nothing to do; Reason
	// then says which.
	Ran bool
	// Skipped marks a step that was applicable but deliberately not run —
	// principally the VACUUM below its threshold. Distinguished from "ran and
	// found nothing" so an operator can tell a working threshold from a
	// broken step.
	Skipped bool
	Reason  string
	Deleted int
	// BytesFreed is space returned to the filesystem — files removed, or the
	// shrink a VACUUM achieved.
	BytesFreed int64
	// BytesReleased is space returned to the *database freelist* by deleting
	// rows. The file does not shrink until the VACUUM step runs, which is why
	// this is reported separately and is deliberately excluded from
	// Report.BytesFreed: adding both would count the same bytes twice, once
	// when the rows went and again when the rewrite handed the pages back.
	BytesReleased int64
	Err           error
}

StepResult is the outcome of one retention step.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL