jobs

package
v0.16.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 12, 2026 License: Apache-2.0 Imports: 34 Imported by: 0

Documentation

Overview

Package jobs is the phase-5 background-work seam (plan Decision 1): one Queue interface with two implementations sharing the same worker code — River (durable Postgres jobs, at-least-once with backoff retries and unique-by-repo sync jobs) for postgres deployments, and an in-process goroutine queue for local/sqlite mode, whose queued work is lost on process restart (the documented ADR trade-off; local mode keeps working, no silent breakage).

The queue deliberately reuses the payload types its two producers and workers already share: events.Delivery (webhook fan-out, delivered via events.Deliverer) and sync.Job (upstream sync runs, executed by sync.(*Worker).Run).

purge.go implements repository reclamation for soft-deleted repositories (issue #52): soft-delete burns the name and keeps every registry row and blob (docs/explanation/domain-model.md), so without a follow-up story a deleted repository's storage lives forever. The RepoPurger permanently reclaims it — on the sweep cadence once OPENSTORAGE_REPO_RETENTION expires, or synchronously via `openstorage purge-repo`.

The purge is TOMBSTONE-THEN-DELETE, the inverse of publish-before-CAS: registry rows are dropped (packs rows becoming pack_retained tombstones) strictly BEFORE their blobs are deleted, and ref rows die first of all, so no crash point can leave a committed ref pointing at deleted bytes — only reapable tombstones and orphan-class blobs. The soft-deleted repos row is the durable marker that drives resumption: every phase is idempotent, a crash anywhere resumes on the next pass, and stamping the row's purged_at tombstone is the completion commit. The row itself is kept forever — the burned (org, url) name is never released, because URL-scoped JWTs can outlive any retention window (see meta/purge.go) — and restore after purge is impossible.

Blob disposal reuses the sweeper's quarantine handshake verbatim (safeDelete: quarantine-copy → delete → registry recheck, GuardedLFSDelete for lfs/ keys), so purge deletions inherit every crash/race repair layer, and the aged quarantine purge collects the backups afterwards.

repack.go implements the insert-triggered background repack (issue #1): per-repo pack rows otherwise grow monotonically with every push and API write (cache repos pin gc.auto=0 / receive.unpackLimit=0), inflating hydration cost and blob-request amplification without bound. The repack job folds a repo's registered packs into one consolidated pack:

  1. snapshot the registered pack rows (the fold input),
  2. select the fold input under per-fold object/byte budgets — sizes and exact object counts come from per-candidate point reads, so selection is decided BEFORE anything is downloaded (repack runs automatically on writes, so one fold's work must be bounded even against adversarial pack shapes; see selectFoldInput),
  3. materialize only the selected packs into the cache repo,
  4. write ONE self-contained pack holding the union of the selected packs' objects,
  5. upload the new pack blob (publish-before-flip: bytes must be durable before any row references them),
  6. install the fold with ref.RefStore.ReplacePacks — one transaction that deletes exactly the selected rows and inserts the new one.

A lost install (another node folded first) stops cleanly; there is no eager retry — the next write past the threshold re-triggers. NOTHING is ever deleted from blob storage on this path: superseded pack blobs and a lost fold's own upload alike become unregistered, age past the orphan sweeper's threshold, and are collected by its crash-safe quarantine protocol (issue #5) — synchronous deletion raced concurrent identical folds on content-addressed keys and could not be made crash-safe without duplicating that protocol. Concurrent hydrators that raced the swap re-list and retry via the bounded loop in githttp.HydratePacks (issue #6).

Fold contents — why the object UNION and not `git pack-objects --all` (reachable objects): reachability at fold time is NOT a safe upper bound for what the deleted rows must keep serving. A push registers its pack in pre-receive and CASes refs in the update hook moments later — a fold snapshotting between the two would drop the (momentarily unreachable) pushed objects, then the ref CAS lands and points refs at objects no registered pack contains: cold-hydrate corruption no swap-time CAS can prevent. Packing the union of the snapshotted packs' objects instead preserves the invariant that the registered object set never shrinks, so any interleaving of registrations and ref CASes stays consistent. The cost is that unreachable objects (crashed pushes, deleted branches) are carried forward; reclaiming them safely needs a reachability grace protocol and is out of scope here.

sweep.go implements the orphan-pack sweeper (issue #5): the publish-before-CAS ordering means a crash between pack blob upload and RegisterPacks strands the blob forever — committed refs never point at it, so it is invisible except as storage cost. The sweeper diffs the blob store's packs/ listing against the authoritative pack registry and reports (and, only when explicitly enabled, deletes) unregistered blobs older than an age threshold. The threshold is the only concurrency guard needed: in-flight pushes register their packs within a request timeout, far below any sane threshold.

Repos are discovered from the blob listing itself (the repo segment of each packs/<repo>/... key) rather than from repo metadata: namespaced repos (`<repo>+import`/`<repo>+ephemeral`) keep fully separate pack spaces registered under their full name, and packs of deleted repos must still be sweepable — both fall out of keying the authoritative lookup by the listed segment.

Forward-compat (issue #1): a background repack will later deregister superseded packs and rely on this same age-threshold diff to collect their blobs after a grace period.

Index

Constants

View Source
const (

	// DefaultAbsenceReconcileBudget bounds the work units one pass may
	// charge (OPENSTORAGE_ABSENCE_RECONCILE_BUDGET overrides): one unit
	// per row examined plus one per extra twin-spelling probe. Sized for
	// point Stats — remote HEAD requests, not row reads — so a default
	// pass stays minutes, not hours.
	DefaultAbsenceReconcileBudget = int64(100_000)
	// MinAbsenceReconcileBudget is the configuration floor (Codex round
	// 1, P2): one row can cost up to three units (registration probe +
	// stat + twin probe), so a degenerate budget cannot guarantee the
	// renewal/rotation split makes progress on both registries. The
	// floor keeps every configurable budget far above single-row
	// overdraw territory.
	MinAbsenceReconcileBudget = int64(100)
)
View Source
const (

	// DefaultLFSReclaimObjectBudget bounds the git objects one run may
	// list across all reachability walks (OPENSTORAGE_LFS_GC_OBJECT_BUDGET
	// overrides).
	DefaultLFSReclaimObjectBudget = int64(1_000_000)
	// DefaultLFSReclaimGrace is how long a row must stay continuously
	// unreferenced before retirement (OPENSTORAGE_LFS_GC_GRACE overrides).
	DefaultLFSReclaimGrace = 24 * time.Hour
)
View Source
const MaxDeliveryAttempts = 10

MaxDeliveryAttempts bounds webhook delivery retries on both backends (docs/reference/webhooks.md: capped exponential backoff, then discard).

View Source
const MinRepoPurgeAge = time.Hour

MinRepoPurgeAge is the minimum time a repository must have been soft-deleted before any purge path (background or CLI) will touch it: a push or import that resolved the repo just before its soft-delete can still be in flight, and its post-tombstone registrations would make the purge churn (spare, defer, retry). One hour comfortably exceeds bounded request lifetimes; the spare-and-defer verification (see purgeRepo) remains the correctness backstop for anything longer.

View Source
const QuarantinePurgeFloor = 4 * sweepRecoveryTimeout

QuarantinePurgeFloor is the minimum age before a quarantine backup may be purged, independent of the configured classification threshold: a backup is safe to purge only once its handshake is provably dead, and safeDelete bounds a handshake's lifetime at 2×sweepRecoveryTimeout from the backup's write. 2× that bound doubles the margin. Exported because the LFS reclamation refund floor must cover it (Codex security round 14): every reap strands one quarantine copy for at least this long, so cap slots must refund no faster than quarantine purging frees them.

View Source
const ReconcileTickInterval = time.Hour

ReconcileTickInterval is the always-live twin janitor's cadence when periodic sweeping is disabled (Codex round 88): one cheap point read per tick when no twins exist, hourly like the default sweep.

Variables

View Source
var (
	// ErrRepoRecentlyDeleted rejects purging a repo soft-deleted less than
	// MinRepoPurgeAge ago.
	ErrRepoRecentlyDeleted = errors.New("repository was soft-deleted too recently to purge")
	// ErrPurgeDeferred reports that registry rows reappeared after
	// tombstoning (an in-flight pre-delete writer): the run made progress
	// but left the repos row unstamped so the next pass re-tombstones and
	// retries.
	ErrPurgeDeferred = errors.New("purge deferred: registry rows were re-registered mid-purge")
)

Purge outcome errors, matched by the CLI and tests via errors.Is.

Functions

func StartAbsenceReconcileTicker

func StartAbsenceReconcileTicker(schedule *AbsenceReconcileSchedule) (stop func())

StartAbsenceReconcileTicker runs the schedule every Interval on a background goroutine — local mode's analogue of the River periodic job, with the same run-on-start behavior. The returned stop function cancels the loop and waits for an in-flight run.

func StartLFSReclaimTicker

func StartLFSReclaimTicker(schedule *LFSReclaimSchedule) (stop func())

StartLFSReclaimTicker runs the schedule every Interval on a background goroutine — local mode's analogue of the River periodic job, with the same run-on-start behavior. The returned stop function cancels the loop and waits for an in-flight run.

func StartReconcileTicker

func StartReconcileTicker(sweeper *Sweeper, interval time.Duration, drainOnly bool) (stop func())

StartReconcileTicker runs the legacy/canonical LFS row-twin janitor on a background goroutine (Codex round 88 + security twin): with OPENSTORAGE_SWEEP_INTERVAL=0 the startup Heal was the ONLY reconciliation, but during a mixed-version rollout an old replica can insert a legacy row beside an existing canonical one at any time after startup — registered fast paths never re-register, so the duplicate object-count and usage charge would persist indefinitely. The returned stop function cancels the loop and waits for an in-flight pass. drainOnly (Codex round 129) stops the ticker once a pass completes below the page cap: the residual-backlog mode for post-roll compat-off deployments, where no new twins can appear and the hourly unindexed scan must not outlive the drain.

func StartSweepTicker

func StartSweepTicker(schedule *SweepSchedule) (stop func())

StartSweepTicker runs the schedule every Interval on a background goroutine — local mode's analogue of the River periodic job, with the same run-on-start behavior. The returned stop function cancels the loop and waits for an in-flight run to finish.

Types

type AbsenceReconcileSchedule

type AbsenceReconcileSchedule struct {
	Reconciler *AbsenceReconciler
	Interval   time.Duration
}

AbsenceReconcileSchedule pairs the reconciler with its cadence — its own schedule, independent of the sweeper's (absence detection is useful even where blob sweeping stays disabled).

type AbsenceReconciler

type AbsenceReconciler struct {
	// contains filtered or unexported fields
}

AbsenceReconciler runs the reconciliation scan. Construct with NewAbsenceReconciler; Run executes one bounded pass per schedule tick. All cross-pass state is durable (job_scan_state cursors + the cover rows themselves); the struct carries only configuration and per-pass scratch.

func NewAbsenceReconciler

func NewAbsenceReconciler(registry AbsenceRegistry, blobs BlobStatter, quotaEnforced bool, budget int64, coverTTL, interval time.Duration, logger *slog.Logger) *AbsenceReconciler

NewAbsenceReconciler wires the reconciliation job. interval is the scheduling cadence between passes: the admission fence's lapse safety bound must cover the time until the NEXT renewal attempt (Codex round 27 on PR #132), not just the current pass.

func (*AbsenceReconciler) Run

Run executes one bounded reconciliation pass. Order (Codex round 1 on PR #123): cover RENEWAL first — keeping already-discovered losses covered is the correctness core and must not wait on the rotation — then the two registry rotations split the remaining budget, leading class alternating between passes.

type AbsenceRegistry

type AbsenceRegistry interface {
	// ListFleetLFSObjects pages lfs_objects rows fleet-wide, keyset on
	// the (repo, key) primary key ascending.
	ListFleetLFSObjects(ctx context.Context, afterRepo, afterKey string, limit int) ([]ref.AbsenceLFSRow, error)
	// ListFleetPacks pages packs rows fleet-wide, keyset on the
	// (repo, id) primary key ascending.
	ListFleetPacks(ctx context.Context, afterRepo, afterID string, limit int) ([]ref.AbsencePackRow, error)
	// RenewAbsenceCovers re-stamps the TTL of every LIVE deterministic
	// cover row AND every un-folded legacy nonce row in DB-clock
	// statements — the pass-start bulk touch that makes cover protection
	// independent of the keyset listing completing within the TTL (Codex
	// rounds 9+15). Strict TTL: expired rows stay dead; nonce rows
	// marked folded decay.
	RenewAbsenceCovers(ctx context.Context, ttl time.Duration) (int64, error)
	// CASAbsenceCoverOwner rewrites one legacy nonce row's owner guarded
	// on the listed (owner, bytes) — the seam the folded sentinel goes
	// through (Codex rounds 15+18+19): a CAS miss proves a concurrent
	// re-charge, which must win.
	CASAbsenceCoverOwner(ctx context.Context, rawKey, expectedOwner string, expectedBytes int64, newOwner string) (bool, error)
	// LapsedAbsenceCoverExists reports whether any physically present,
	// un-folded cover row aged past the ttl — the positive lapse
	// evidence required to arm the admission fence when the
	// renewal-health watermark is absent (Codex round 32, security):
	// an absent watermark alone is the upgrade/first-enable signature,
	// not proof of a fleet-wide lapse.
	LapsedAbsenceCoverExists(ctx context.Context, ttl time.Duration) (bool, error)
	// ListAbsenceCovers pages the live absence-cover reservation rows —
	// the durable tracking state the renewal phase re-derives its
	// obligations from every pass (issue #125). ttl is the liveness
	// window admission consults: expired rows are excluded (renewing or
	// migrating one would resurrect a stale reservation, Codex round 6).
	ListAbsenceCovers(ctx context.Context, afterRaw string, limit int, ttl time.Duration) ([]ref.AbsenceCoverRow, error)
	// AbsenceCoverHead point-reads the highest LIVE incarnation under a
	// synthetic-key prefix — the charge path's fallback when the renewal
	// listing did not cover the base this pass. Same ttl semantics.
	AbsenceCoverHead(ctx context.Context, prefix string, ttl time.Duration) (ref.AbsenceCoverRow, bool, error)
	// DeleteAbsenceCover releases one cover row (healed loss, superseded
	// incarnation, or retired legacy spelling).
	DeleteAbsenceCover(ctx context.Context, key string) error
	// IsRepoPurged probes the repository purge fence (issue #52): fenced
	// repos' rows are on their way out and are never covered.
	IsRepoPurged(ctx context.Context, base string) (bool, error)
	// BumpLFSAdoptionGen invalidates cached usage floors after a cover
	// made absent bytes visible to admission.
	BumpLFSAdoptionGen(ctx context.Context, repo string) error
	// LFSAdoptionGen reads one repo's shared adoption generation — the
	// mutation fence resumed ranges verify across park gaps (Codex
	// round 20 on PR #132): summed over the base group, any bump while
	// the cursor was parked taints the pending range's observation.
	LFSAdoptionGen(ctx context.Context, repo string) (int64, error)
	// ReservePackBytes is the ref.RestoreCoverStore seam
	// ref.ReserveRestoreCover charges covers through.
	ReservePackBytes(ctx context.Context, group []string, packs []ref.Pack, owner string, usageFloor, floorGen, quotaBytes int64, ttl time.Duration) (ref.ReserveDecision, error)
	// ArmAbsenceAdmissionFence upserts the admission-fence marker
	// serialized against in-flight finite quota admissions (Codex round
	// 35, security): a plain scan-state upsert could land between an
	// admission's fence check and its commit on PostgreSQL, letting it
	// spend the deflated headroom the fence closes.
	ArmAbsenceAdmissionFence(ctx context.Context, payload string) error
	// ScanStateStore persists the rotation cursors and traversal
	// bookkeeping (issue #125).
	ref.ScanStateStore
}

AbsenceRegistry is the ref-store surface the absence reconciler drives. Both concrete backends (ref.PG, ref.SQLite) satisfy it.

type AbsenceReport

type AbsenceReport struct {
	LFSRowsExamined  int64 `json:"lfs_rows_examined"`
	PackRowsExamined int64 `json:"pack_rows_examined"`
	// LFSRowsAbsent counts rows whose bytes are missing under BOTH ADR-41
	// spellings — genuine storage loss.
	LFSRowsAbsent int64 `json:"lfs_rows_absent"`
	// LFSRowsTruncated counts rows physically present (under either
	// spelling) but SHORTER than their registered size — a partial loss
	// whose deficit deflates the floor exactly like full absence (Codex
	// round 4, security).
	LFSRowsTruncated int64 `json:"lfs_rows_truncated"`
	// LFSRowsTwinOnly counts rows absent under their registered spelling
	// but fully present under the canonical/legacy twin: physically
	// listed (no floor deflation), healed by download-time restore (PR
	// #104).
	LFSRowsTwinOnly   int64 `json:"lfs_rows_twin_only"`
	PackRowsAbsent    int64 `json:"pack_rows_absent"`
	PackRowsTruncated int64 `json:"pack_rows_truncated"`
	// PackRowsUnsized counts absent pack rows registered before the
	// packs.bytes migration: the loss is reported but uncoverable until a
	// repack fold rewrites the row with its size.
	PackRowsUnsized   int64 `json:"pack_rows_unsized"`
	CoversCharged     int64 `json:"covers_charged"`
	BytesCovered      int64 `json:"bytes_covered"`
	CoverErrors       int64 `json:"cover_errors"`
	StatErrors        int64 `json:"stat_errors"`
	RowsSkippedPurged int64 `json:"rows_skipped_purged"`
	// CoversRenewed counts live cover rows the renewal phase re-charged
	// this pass (fresh TTL, same key and bytes) — the fleet-derived
	// replacement for the per-process known-absent refresh (issue #125).
	CoversRenewed int64 `json:"covers_renewed"`
	// CoversRotated counts shrink re-mints: a base's exact re-derived
	// total came in below its live row, so a fresh incarnation charged
	// the exact total and the stale row was deleted.
	CoversRotated int64 `json:"covers_rotated"`
	// CoversReleased counts cover rows DELETED: fully healed bases and
	// superseded incarnations left by crashes.
	CoversReleased int64 `json:"covers_released"`
	// CoversDecaying counts listed legacy (pre-#125 nonce-keyed) covers
	// left to expire by TTL once both classes proved their base within
	// one pass (foldRedundantLegacy): never deleted — a not-yet-upgraded
	// replica may still depend on them mid-roll.
	CoversDecaying int64 `json:"covers_decaying"`
	// RowsVerified counts registry rows re-observed by verifyAndJudge —
	// the fresh storage-visible evidence behind every verdict whose
	// trigger spanned a pass boundary (Codex round 21).
	RowsVerified int64 `json:"rows_verified"`
	// CoversTouched counts live deterministic cover rows whose TTL the
	// pass-start bulk touch re-stamped (Codex round 9): protection never
	// depends on the keyset listing completing within the TTL, so a
	// multi-pass listing round robin can delay lifecycle work
	// (fold/shrink/release) but cannot lapse a live cover.
	CoversTouched int64 `json:"covers_touched"`
	// RenewalIncomplete reports the renewal listing was cut short by the
	// pass deadline (never by budget): un-renewed rows resume first on
	// the durable cursor next pass. Investigate the blob store or ledger
	// if persistent — covers lapse if renewals cannot land within the
	// TTL.
	RenewalIncomplete bool `json:"renewal_incomplete"`
	// RenewalOverBudget reports renewal charged more than its soft
	// allocation (half the budget): every live cover still renewed —
	// renewal is the correctness core and never lapses on budget — while
	// discovery keeps its guaranteed half-budget allowance (Codex round
	// 14: a zero-allowance clamp would be an absorbing state under the
	// bulk touch). Raise the budget above twice the live cover
	// population.
	RenewalOverBudget bool `json:"renewal_over_budget"`
	// LFSTraversals/PacksTraversals are the classes' completed rotation
	// counts (durable bookkeeping, useful for tracking convergence).
	LFSTraversals  int64 `json:"lfs_traversals"`
	PackTraversals int64 `json:"pack_traversals"`
	QuotaEnforced  bool  `json:"quota_enforced"`
}

AbsenceReport summarizes one reconciliation pass.

type BlobStatter

type BlobStatter interface {
	Stat(ctx context.Context, key string) (int64, error)
}

BlobStatter is the single blob-store probe the reconciler needs.

type DefaultBranchFunc

type DefaultBranchFunc func(ctx context.Context, base string) (branch string, ok bool, err error)

DefaultBranchFunc resolves a base repo's default branch from the metadata store; ok=false skips the repo (deleted or foreign — never an error, and never a retirement).

type HydrateFunc

type HydrateFunc func(ctx context.Context, repo, defaultBranch string) (repoPath string, err error)

HydrateFunc materializes one namespace repo's git cache from the stores (gitread.Materialize in production: hydrate packs, sync refs from the DB authority) and returns its bare-repo path.

type LFSReclaimRegistry

type LFSReclaimRegistry interface {
	// LFSReposWithObjects pages base repos holding registered LFS objects
	// (keyset on repo, ascending) — the scan's work queue.
	LFSReposWithObjects(ctx context.Context, after string, limit int) ([]string, error)
	// ListRepoLFSObjects pages repo's registered rows (keyset on key).
	ListRepoLFSObjects(ctx context.Context, repo, after string, limit int) ([]ref.LFSObjectRow, error)
	// MarkLFSObjectsUnreferenced stamps unreferenced_since=now() on
	// unmarked rows; an existing mark keeps its original clock.
	MarkLFSObjectsUnreferenced(ctx context.Context, repo string, keys []string) (int64, error)
	// ClearLFSObjectsUnreferenced removes marks from re-referenced rows.
	ClearLFSObjectsUnreferenced(ctx context.Context, repo string, keys []string) (int64, error)
	// RetireUnreferencedLFSObject is the guarded verdict — see the ref
	// store contract (generation fence, purge fence, in-transaction grace
	// recheck, reap-watermark freshness, deferred refund bucketing).
	RetireUnreferencedLFSObject(ctx context.Context, repo, key string, pinnedGens map[string]int64, grace, reapMaxAge time.Duration) (bool, error)
	// SettleLFSRefunds releases a base's pending retirement refunds
	// covered by a completed destructive sweep (Codex round 33 on PR
	// #132) — see the ref store contract.
	SettleLFSRefunds(ctx context.Context, repo string) (ref.SettledLFSRefunds, error)
	// ScanStateStore persists the rotation cursor, the per-repo
	// row-settlement cursors, and serves the sweeper's destructive-reap
	// watermark (issues #120/#121): River may run each pass on any
	// replica, so scan progress must not live in process memory.
	ref.ScanStateStore
	// IsRepoPurged skips purge-fenced repos (their rows die with the purge).
	IsRepoPurged(ctx context.Context, base string) (bool, error)
	// GetGeneration pins each namespace-group member's ref generation
	// before hydration; retirement re-checks it.
	GetGeneration(ctx context.Context, repo string) (int64, error)
	// CountRefs gates hydration — a member with no refs has nothing
	// reachable and needs no cache — and prices ref fanout WITHOUT
	// materializing the refs (Codex security round 7): the reclaimer
	// never needs the ref values, and a counting query keeps a
	// million-ref repo from ballooning the worker's heap before the
	// budget can reject it. The count stops at limit (Codex round 12) so
	// even the index walk stays budget-bounded.
	CountRefs(ctx context.Context, repo string, limit int64) (int64, error)
}

LFSReclaimRegistry is the ref-store surface the reclaimer drives — carried by both concrete backends (like RepoPurgeRegistry), deliberately not on ref.RefStore: only this job may mint retirement verdicts.

type LFSReclaimReport

type LFSReclaimReport struct {
	ReposScanned    int   `json:"repos_scanned"`
	ReposSkipped    int   `json:"repos_skipped"`
	ReposOverBudget int   `json:"repos_over_budget"`
	ObjectsWalked   int64 `json:"objects_walked"`
	RowsExamined    int64 `json:"rows_examined"`
	RowsReferenced  int64 `json:"rows_referenced"`
	RowsMarked      int64 `json:"rows_marked"`
	RowsCleared     int64 `json:"rows_cleared"`
	RowsInGrace     int64 `json:"rows_in_grace"`
	// RowsEligible counts grace-expired marked rows the run would retire;
	// with deletes disabled (dry run) nothing else happens to them.
	RowsEligible int64 `json:"rows_eligible"`
	RowsRetired  int64 `json:"rows_retired"`
	// RowsSpared counts guarded verdicts that declined at commit time
	// (generation drift, concurrent touch/re-registration, purge fence).
	RowsSpared   int64 `json:"rows_spared"`
	BytesRetired int64 `json:"bytes_retired"`
	DryRun       bool  `json:"dry_run"`
	// ReapWatermarkStale reports the pass ran report-only because the
	// sweeper's last completed destructive LFS pass is older than the
	// grace window (or has never happened) — refunds gate on OBSERVED
	// reap completion, not configured cadence (issue #120).
	ReapWatermarkStale bool `json:"reap_watermark_stale"`
	// RowsDeferredReapStale counts grace-expired eligible rows whose
	// retirement was deferred by the stale watermark; they retire on a
	// later pass once destructive sweeps demonstrably complete again.
	RowsDeferredReapStale int64 `json:"rows_deferred_reap_stale"`
	// RowsDeferredLegacyUnarmed counts grace-expired eligible
	// LEGACY-SPELLED rows deferred because legacy reaping is not armed
	// end-to-end — this replica's OPENSTORAGE_SWEEP_DELETE_LFS_LEGACY is
	// off, or the fresh watermark does not certify a legacy-armed sweep
	// (Codex round 17 on PR #132). Their blobs would never be physically
	// reaped, so refunding their slots would grow storage past the cap.
	RowsDeferredLegacyUnarmed int64 `json:"rows_deferred_legacy_unarmed"`
	// RefundRowsSettled/RefundBytesSettled count deferred retirement
	// refunds released this pass: a destructive sweep with a strictly
	// greater epoch completed since the rows were retired, proving their
	// orphaned blobs were visible to a full clean reap (Codex round 33
	// on PR #132).
	RefundRowsSettled  int64 `json:"refund_rows_settled"`
	RefundBytesSettled int64 `json:"refund_bytes_settled"`
	// RefundBucketsPending counts buckets still awaiting a covering
	// completion at their settlement attempts this pass.
	RefundBucketsPending int `json:"refund_buckets_pending"`
}

LFSReclaimReport is one run's outcome, logged as structured fields (the dogfooding/observability record, mirroring SweepReport).

type LFSReclaimSchedule

type LFSReclaimSchedule struct {
	Reclaimer *LFSReclaimer
	Interval  time.Duration
}

LFSReclaimSchedule pairs the reclaimer with its cadence — its own schedule, independent of the sweeper's (count reclamation is useful even where blob sweeping stays disabled).

type LFSReclaimer

type LFSReclaimer struct {
	// contains filtered or unexported fields
}

LFSReclaimer runs the reclamation scan. Construct with NewLFSReclaimer; Run executes one bounded pass on the schedule's cadence.

func NewLFSReclaimer

func NewLFSReclaimer(registry LFSReclaimRegistry, hydrate HydrateFunc, walk LFSWalkFunc, defaultBranch DefaultBranchFunc, grace time.Duration, objectBudget int64, deleteEnabled, legacyDelete bool, logger *slog.Logger) *LFSReclaimer

NewLFSReclaimer wires the reclamation job. legacyDelete mirrors the sweeper's OPENSTORAGE_SWEEP_DELETE_LFS_LEGACY arming — see the field.

func (*LFSReclaimer) Run

Run executes one bounded reclamation pass: page LFS-holding repos from the rotating cursor, scan each within the remaining object budget, and mark/clear/retire its rows. Per-repo failures defer that repo to a later run without stopping the pass; they are joined into the returned error so the job backend records them.

type LFSWalkFunc

type LFSWalkFunc func(ctx context.Context, repoPath string, budget int64) (oids map[string]struct{}, walked int64, err error)

LFSWalkFunc computes the reachable pointer OIDs of the cache repo at repoPath (lfsscan.ReachableLFSOIDs in production). It must return an error wrapping lfsscan.ErrLFSScanBudget when the walk exceeds budget.

type Local

type Local struct {
	// contains filtered or unexported fields
}

Local is the in-process queue for local/sqlite mode. Deliveries run on their own goroutines through events.Deliverer.Deliver (which owns the re-sign-per-attempt retry loop); sync runs go through sync.InProcessQueue (per-repo coalescing). Queued work is lost on process restart — the documented local-mode ADR trade-off.

func NewLocal

func NewLocal(deliverer *events.Deliverer, runSync, runPush SyncRunner, repacker *Repacker, logger *slog.Logger) *Local

NewLocal builds the in-process queue. deliverer must be configured with the webhook lookup; runSync is normally sync.(*Worker).Run and runPush sync.(*Worker).RunPush (issue #90). repacker may be nil (repack disabled or not wired, e.g. hook-subprocess queues that must never fold); EnqueueRepack is then refused.

func NewLocalScheduled

func NewLocalScheduled(deliverer *events.Deliverer, sched *sync.LocalScheduler, repacker *Repacker, logger *slog.Logger) *Local

NewLocalScheduled is NewLocal with sync and push runs driven by the durable SQLite schedule (issue #266). Close closes sched.

func (*Local) Close

func (q *Local) Close()

Close stops accepting work and waits for in-flight deliveries and syncs.

func (*Local) CompleteSyncCreate

func (q *Local) CompleteSyncCreate(ctx context.Context, org, repoID string) (meta.CreateOutcome, error)

CompleteSyncCreate delegates to the SQLite schedule (issue #388).

func (*Local) DrainSyncRuns

func (q *Local) DrainSyncRuns(ctx context.Context, wait time.Duration)

DrainSyncRuns runs the scheduled sync queue's shutdown drain (issue #342, sync.LocalScheduler.Drain) while serve's listener still takes loopback pushes. Deliveries and repacks keep running until Close. It does nothing for the in-process queue.

func (*Local) EnqueuePush

func (q *Local) EnqueuePush(ctx context.Context, job sync.Job) error

EnqueuePush delegates to the in-process queue's outbound lane (issue #90).

func (*Local) EnqueueRepack

func (q *Local) EnqueueRepack(_ context.Context, repo string) error

EnqueueRepack runs one fold attempt on a background goroutine with a fresh context (it must outlive the HTTP request that triggered it). At most one repack per repo is in flight; a trigger landing while one runs marks ONE pending rerun that starts when the current fold finishes — mirroring River's unique-by-repo jobs, whose uniqueness spans only the queued states so a mid-run trigger queues exactly one follow-up. Queued work is lost on process restart (the documented local-mode trade-off).

func (*Local) EnqueueSync

func (q *Local) EnqueueSync(ctx context.Context, job sync.Job) error

EnqueueSync delegates to the coalescing in-process sync queue.

func (*Local) EnqueueWebhookDelivery

func (q *Local) EnqueueWebhookDelivery(_ context.Context, d events.Delivery) error

EnqueueWebhookDelivery starts the delivery's retry loop on a background goroutine with a fresh context: it must outlive the HTTP request (or CAS call) that enqueued it. Terminal failures are logged by the Deliverer.

func (*Local) OfferSync

func (q *Local) OfferSync(ctx context.Context, job sync.Job) error

OfferSync offers a pull run for a request the caller already recorded (API pull-upstream). On the SQLite schedule it does not record the request again; the in-process queue has no such distinction and enqueues.

func (*Local) TracksCreateState

func (q *Local) TracksCreateState() bool

TracksCreateState is true only on the SQLite schedule; the in-process queue serves S3-WAL, which stores no create state (issue #335).

type MaterializeFunc

type MaterializeFunc func(ctx context.Context, repo string, packs []ref.Pack) (repoPath string, err error)

MaterializeFunc materializes EXACTLY the given registered packs into repo's cache repo and returns its path. It is injected — wiring composes githttp.EnsureCacheRepo + githttp.MaterializePack over the serve cache dir — so this package never imports githttp, whose own tests import this package's sweeper. The fold selects its input under byte/object budgets BEFORE materializing, so a full-repo hydrate here would defeat them; the implementation must honor ctx (the fold's wall-clock budget) in lock waits, downloads, and git subprocesses, and must propagate blob.ErrNotFound (a selected pack's blob vanishing means a concurrent fold already replaced the snapshotted rows).

type OrphanPack

type OrphanPack struct {
	Key  string
	Size int64
	Age  time.Duration
}

OrphanPack is one orphan candidate found by a sweep.

type PackRegistry

type PackRegistry interface {
	HasPack(ctx context.Context, repo, key string) (bool, error)
	// HasLFSObject is HasPack's twin for the LFS registry (issue #51):
	// point membership for lfs/<repo>/<oid> keys, written by verified
	// uploads (ref.RegisterLFSObject).
	HasLFSObject(ctx context.Context, repo, key string) (bool, error)
	// LFSObjectRegisteredAt reads a registered object's registration time
	// (see ref.RefStore): the legacy reap grace's clock (Codex round 89)
	// — legacy presigned GETs are minted right up to the registration
	// moment, and neither blob timestamp bounds that. bytes is the
	// registered size, the reap gate's completeness yardstick (#127).
	LFSObjectRegisteredAt(ctx context.Context, repo, key string) (at time.Time, bytes int64, ok bool, err error)
	// PackRegisteredBytes reads a registered pack's recorded size (see
	// ref.RefStore): the quarantine heal's completeness yardstick for
	// pack keys (issue #129). bytes=0 means the row predates the size
	// column (issue #101) — size UNKNOWN, never "complete at any length".
	PackRegisteredBytes(ctx context.Context, repo, key string) (bytes int64, ok bool, err error)
	// TryAdoptionClaim / RenewAdoptionClaim / ReleaseAdoptionClaim (see
	// ref.RefStore) fence the heal's truncated-original overwrite (issue
	// #129): rewriting a PRESENT canonical LFS key is a canonical
	// materialization, so it must hold the same per-key fleet claim as
	// every other one (PUT publish/register/verify region, adoption
	// copy) — the adopter's corrupt-canonical check→delete is only safe
	// under that mutual exclusion. The into-absence restore stays
	// deliberately unfenced: it deletes nothing and writes
	// digest-verified true bytes with a fresh fingerprint.
	TryAdoptionClaim(ctx context.Context, repo, key string) (claimed bool, owner string, err error)
	RenewAdoptionClaim(ctx context.Context, key, owner string) (renewed bool, err error)
	ReleaseAdoptionClaim(ctx context.Context, key, owner string) error
	// GuardedLFSDelete runs del serialized against LFS registration (see
	// ref.RefStore): safeDelete's LFS blob delete must go through it, or
	// a verified upload's register+verify can complete between the
	// quarantine copy and the delete — the sweep would then delete the
	// verified replacement and (correctly) refuse to restore a
	// digest-mismatched backup, leaving a registered row blobless with no
	// remaining writer check (Codex round 20).
	GuardedLFSDelete(ctx context.Context, repo, key string, del func(context.Context) error) (deleted bool, err error)
	// BumpLFSAdoptionGen advances the repo's shared FLOOR generation
	// (see ref.RefStore; despite the name it invalidates every cached
	// uncounted floor, not just LFS state). A quarantine restore makes
	// a counter-accounted blob physically visible again without touching
	// the counter (Codex round 54, security): a floor scanned while the
	// blob was absent under-measures — listing minus counter can hide
	// equally sized rowless legacy bytes — so the repair must invalidate
	// it, or a stale floor admits past quota until expiry.
	BumpLFSAdoptionGen(ctx context.Context, repo string) error
	// ReservePackBytes backs ref.ReserveRestoreCover for quarantine
	// restores (Codex round 58): the generation bump above only stops
	// FUTURE floor serves, so the returning bytes must be accounted as a
	// reservation BEFORE they become physical — an admission already
	// holding a floor scanned during the absence then sees them in
	// active_reservations instead of committing into hidden headroom.
	ReservePackBytes(ctx context.Context, group []string, packs []ref.Pack, owner string, usageFloor, floorGen, quotaBytes int64, ttl time.Duration) (ref.ReserveDecision, error)
	// IsRetainedPack classifies (repo, key) as superseded-by-repack: a
	// blob whose registry row a ReplacePacks fold removed. Provably
	// DB-absent by that CAS — a narrower, safer deletion class than
	// general orphans (issue #31).
	IsRetainedPack(ctx context.Context, repo, key string) (bool, error)
	// ReclaimRetainedPack debits the retained bytes from the repo's quota
	// usage once the blob actually left the pack prefix; idempotent.
	ReclaimRetainedPack(ctx context.Context, repo, key string) (int64, error)
	// SweepExpiredPackReservations prunes reservation rows abandoned by
	// crashed pushes (issue #29); expired rows already stopped gating
	// admission, so this is pure garbage collection.
	SweepExpiredPackReservations(ctx context.Context, olderThan time.Duration) (int64, error)
	// ReconcileLegacyLFSObjects deletes legacy-spelled LFS registry rows
	// whose canonical twin exists, refunding what they double-charged
	// (ADR-41, Codex round 79) — rolling-upgrade races can strand such
	// twins where no registration ever runs again.
	ReconcileLegacyLFSObjects(ctx context.Context) (reconciled int64, partial bool, err error)
}

PackRegistry is the slice of ref.RefStore the sweeper needs: point membership in the authoritative registered-pack set, plus the quota ledger's sweep-side operations (issues #29/#31). Point reads, not ListPacks: a repo's registered-pack count is attacker-growable (pre-receive registers packs before ref policy), so the sweeper must never materialize a repo's full pack set.

type PurgeRunReport

type PurgeRunReport struct {
	Eligible int // repos past retention this pass picked up
	Purged   int // completed purges (repos row deleted)
	Failed   int // deferred or errored repos (retried next pass)
}

PurgeRunReport summarizes one PurgeExpired pass.

type Queue

type Queue interface {
	// EnqueueWebhookDelivery schedules one signed webhook POST. Retries and
	// re-signing per attempt are the worker's job (events.Deliverer).
	EnqueueWebhookDelivery(ctx context.Context, d events.Delivery) error
	// EnqueueSync schedules one upstream sync run for job.RepoID. Runs for
	// the same repo are deduplicated while queued (River unique jobs /
	// in-process coalescing); overlap with an in-flight run is handled by
	// the store's TryBeginSync claim (invariant 11). Deduplication against
	// a run that already claimed can drop the enqueue (issue #79): callers
	// that promise the sync (pull-upstream 202) mark the request first via
	// UpstreamStore.RequestSync, and the in-flight run's finisher schedules
	// the coalesced follow-up run.
	EnqueueSync(ctx context.Context, job sync.Job) error
	// EnqueuePush schedules one outbound push-upstream run (issue #90).
	// Deduplication and the mark-before-enqueue protocol mirror
	// EnqueueSync on the push lane (UpstreamStore.RequestPush).
	EnqueuePush(ctx context.Context, job sync.Job) error
	// EnqueueRepack schedules one background repack of repo's registered
	// packs (issue #1). Jobs for the same repo are deduplicated while
	// active (River unique-by-args jobs / in-process in-flight skip); a
	// fold that loses its install race is not retried eagerly — the next
	// write past the threshold re-triggers.
	EnqueueRepack(ctx context.Context, repo string) error
	Close()
}

Queue schedules background work. Enqueue methods are safe for concurrent use; Close stops accepting work and, for the local queue, waits for in-flight jobs.

type RepackTrigger

type RepackTrigger struct {
	// contains filtered or unexported fields
}

RepackTrigger runs the post-write threshold check (issue #1): callers invoke MaybeEnqueue after a write that registered packs — the git surface after each receive-pack, the API after each RegisterPacks — and one unique-by-repo repack job is enqueued when the registered pack count reached the threshold. Construct with NewRepackTrigger; a nil trigger at the call sites means repacking is disabled.

func NewRepackTrigger

func NewRepackTrigger(store ref.RefStore, threshold int, enqueue func(ctx context.Context, repo string) error, logger *slog.Logger) *RepackTrigger

NewRepackTrigger builds the trigger. threshold is the registered-pack count at which a fold is scheduled; enqueue is normally the jobs queue's EnqueueRepack.

func (*RepackTrigger) MaybeEnqueue

func (t *RepackTrigger) MaybeEnqueue(ctx context.Context, repo string)

MaybeEnqueue schedules one repack when repo's registered pack count reached the threshold. Failures are logged, never propagated: the write that prompted the check already succeeded, and the count only keeps growing until a fold lands, so the next write retries the trigger. Execution is time-bounded regardless of the caller's context — the trigger runs synchronously inside write handlers (before the ref CAS in the API commit pipeline) on a cancel-detached context, and a stalled store must not pin them. (ListPacks materializes the row set — acceptable here because the trigger is what keeps that set near one row in steady state, and the full-hydrate path already lists it on every cache rebuild.)

type Repacker

type Repacker struct {

	// FollowUpThreshold, when >= 2, is the registered-pack count at which
	// a just-finished fold is followed by another one (queue backends
	// consult NeedsFollowUp): packs registered after the fold's row
	// snapshot survive it, and a burst could otherwise finish above the
	// trigger threshold with nothing scheduled until the next write.
	// Wiring sets it to the trigger threshold; 0 disables follow-ups.
	FollowUpThreshold int

	// MaxFoldObjects/MaxFoldBytes bound ONE fold's input (0 = default):
	// repack runs automatically on writes, and a git:write caller must not
	// be able to make the service enumerate, recompress, and upload an
	// unbounded object set on every threshold crossing. Packs are selected
	// oldest-first under the cumulative budgets (see selectFoldInput);
	// oversized leftovers stay registered as-is, so a huge repo converges
	// to O(size/budget) consolidated rows instead of one — bounded work
	// per fold beats one unbounded fold.
	MaxFoldObjects int
	MaxFoldBytes   int64
	// MaxFoldDuration bounds ONE fold's wall-clock runtime (0 = default).
	// The input budgets above measure compressed pack bytes, which
	// undercount CPU: a few-hundred-KiB pack can inflate to hundreds of
	// MiB of logical content that pack-objects re-delta-compresses. The
	// runtime budget caps the actual work directly — the whole fold runs
	// under a deadline and every git subprocess it spawns is killed on
	// expiry. A timed-out fold is crash-equivalent at every point by
	// design (at worst one unregistered uploaded blob, sweeper class).
	MaxFoldDuration time.Duration

	// BeforeSwap is a test seam: when set, it runs after the consolidated
	// pack's blob upload and before ReplacePacks — the fold-install race
	// window — receiving the pending pack (its content-addressed blob key
	// in particular; a listing diff cannot identify it when an identical
	// orphan already exists). Tests and the DST harness use it to
	// interleave competing folds deterministically. Never set in
	// production wiring.
	BeforeSwap func(ctx context.Context, pending ref.Pack) error
	// contains filtered or unexported fields
}

Repacker folds one repo's registered packs into a consolidated pack. Construct with NewRepacker; Repack is safe to run concurrently with pushes, API writes, hydrators, sweeps, and other repackers (ReplacePacks is the only mutation and it is a CAS).

func NewRepacker

func NewRepacker(store ref.RefStore, blobs blob.BlobStore, materialize MaterializeFunc, logger *slog.Logger) *Repacker

NewRepacker builds a Repacker over the raw ref/blob stores and the injected materialization seam (normally githttp cache-repo helpers closed over the serve cache dir, so the fold packs in the same cache the git surface uses and a successful fold leaves this node warm).

func (*Repacker) FollowUpDue

func (r *Repacker) FollowUpDue(ctx context.Context, repo string) (count int, due bool, err error)

FollowUpDue reports repo's registered pack count and whether it has reached FollowUpThreshold — the recheck queue backends use to schedule one more fold for registrations that landed mid-fold. The count lets the river worker distinguish growth from a wedged plateau (a repo stuck at/above the threshold with nothing pairable must not chain no-op successors forever). Read failures propagate: inside the river worker's completion transaction a silently-false answer would commit the job away without a successor, stranding coalesced triggers — the caller decides whether to snooze (river) or log and defer (local).

func (*Repacker) Repack

func (r *Repacker) Repack(ctx context.Context, repo string) (folded bool, err error)

Repack runs one fold attempt for repo. folded reports whether a fold WAS OR MAY HAVE BEEN installed — true on a won swap even when its post-swap verification fails, and true on an ambiguous swap error (committed with the response lost): queue backends key their follow-up threshold recheck on folded, and packs registered after the fold's snapshot must not strand when the swap actually landed. A no-op (fewer than two registered packs, or budgets selecting fewer than two) and a cleanly LOST race both return (false, nil) — a loss proves a concurrent fold won, and in every deployment shape that winner runs the same follow-up recheck.

type RepoPurgeMeta

type RepoPurgeMeta interface {
	// PurgeableRepos lists up to limit purge candidates soft-deleted at or
	// before deletedBefore and not yet purged, (deleted_at, repo_id)
	// ascending, strictly after the cursor (zero = start).
	PurgeableRepos(ctx context.Context, deletedBefore time.Time, after meta.PurgeCandidate, limit int) ([]meta.PurgeCandidate, error)
	// RepoPurgeState returns the repo row plus its purge tombstone
	// (found=false when absent; Repo.DeletedAt==nil when live; PurgedAt
	// non-nil once a purge completed).
	RepoPurgeState(ctx context.Context, repoID string) (state meta.PurgeState, found bool, err error)
	// PurgeRepoMeta deletes credentials and the upstream row and stamps
	// the repos row's purged_at tombstone in one guarded transaction
	// (phase 7, the completion commit; the row is kept as the permanent
	// name burn).
	PurgeRepoMeta(ctx context.Context, repoID string) error
}

RepoPurgeMeta is the slice of the concrete meta stores the purge needs: eligibility reads and the final metadata deletion. Same concrete-backend discipline as RepoPurgeRegistry.

type RepoPurgeRegistry

type RepoPurgeRegistry interface {
	// FenceRepoPurge durably write-fences the base repo (phase 0, before
	// any tombstoning): pack/LFS registrations for the group fail with
	// ref.ErrRepoPurged from then on, so the later verification and
	// completion cannot be raced by an in-flight pre-delete writer.
	// Permanent, like the name burn; idempotent.
	FenceRepoPurge(ctx context.Context, base string) error
	// IsRepoPurged reports whether the base repo is purge-fenced — the
	// sweeper's straggler-blob probe (SweepConfig.PurgedRepoProbe).
	IsRepoPurged(ctx context.Context, base string) (bool, error)
	// DropRepoRefs deletes the group's ref rows (purge phase 1 — refs die
	// first, so no later phase can strand a committed ref).
	DropRepoRefs(ctx context.Context, group []string) (int64, error)
	// RetireRepoPacks tombstones the group's packs rows into pack_retained
	// (phase 2): a crash after this leaves the blobs in the sweeper's
	// provably-DB-absent retained class (reaped under DeleteSuperseded,
	// on by default) even if retention is later disabled.
	RetireRepoPacks(ctx context.Context, group []string) (int64, error)
	// DropRepoLFSObjects deletes the group's lfs_objects/count rows
	// (phase 3).
	DropRepoLFSObjects(ctx context.Context, group []string) (int64, error)
	// CountRepoRows counts the group's refs+packs+lfs_objects rows — the
	// pre-completion verification read (phase 5).
	CountRepoRows(ctx context.Context, group []string) (int64, error)
	// PurgeRepoLedger removes the group's remaining accounting/state rows
	// (phase 6).
	PurgeRepoLedger(ctx context.Context, group []string) (int64, error)
}

RepoPurgeRegistry is the slice of the concrete ref stores the purge needs beyond PackRegistry: row tombstoning and final ledger cleanup. Deliberately NOT part of ref.RefStore — that contract has many test/simulation mirrors that must not grow purge-only surface — so the wiring type-asserts the concrete backend (see the compile-time assertions below).

type RepoPurgeReport

type RepoPurgeReport struct {
	RepoID       string
	RefsDropped  int64 // ref rows deleted (phase 1)
	PacksRetired int64 // packs rows tombstoned into pack_retained (phase 2)
	LFSDropped   int64 // lfs_objects rows deleted (phase 3)
	BlobsReaped  int   // blobs quarantined-then-deleted (phase 4)
	BytesReaped  int64 // their sizes (bytes move to quarantine, freed by the aged purge)
	BlobsSpared  int   // still-registered blobs spared (in-flight writer; forces deferral)
	RowsPurged   int64 // ledger/state rows removed (phase 6)
}

RepoPurgeReport summarizes one repository's purge.

type RepoPurger

type RepoPurger struct {
	// contains filtered or unexported fields
}

RepoPurger reclaims soft-deleted repositories. Construct with NewRepoPurger; PurgeExpired runs on the sweep cadence, PurgeRepo backs the operator CLI. Both funnel through the same purge primitive.

func NewRepoPurger

func NewRepoPurger(sweeper *Sweeper, registry RepoPurgeRegistry, metaStore RepoPurgeMeta, retention time.Duration, logger *slog.Logger) *RepoPurger

NewRepoPurger builds a RepoPurger over the sweeper (blob disposal machinery) and the concrete stores' purge seams.

func (*RepoPurger) PurgeExpired

func (p *RepoPurger) PurgeExpired(ctx context.Context) (PurgeRunReport, error)

PurgeExpired reclaims one batch of repositories soft-deleted longer than the retention. Per-repo failures defer that repo to the next pass without stopping the batch; they are joined into the returned error so the job backend records them.

func (*RepoPurger) PurgeRepo

func (p *RepoPurger) PurgeRepo(ctx context.Context, repoID string) (RepoPurgeReport, error)

PurgeRepo purges one soft-deleted repository synchronously (the `openstorage purge-repo` primitive). The repo must be a BASE repo ID and must have been soft-deleted at least MinRepoPurgeAge ago; the retention window does NOT apply — invoking the command is the operator's consent.

type River

type River struct {
	// contains filtered or unexported fields
}

River is the durable Postgres queue (plan Decision 1). The serve process runs it in full mode (migrations + workers); hook subprocesses run insert-only clients against the same database, which is how git-surface events reach the server's workers across process boundaries.

func NewRiver

func NewRiver(ctx context.Context, pgURL string, deliverer *events.Deliverer, runSync, runPush SyncRunner, upstreams sync.UpstreamStore, sweep *SweepSchedule, lfsReclaim *LFSReclaimSchedule, absence *AbsenceReconcileSchedule, repack *Repacker, logger *slog.Logger) (*River, error)

NewRiver opens a dedicated pool on pgURL, applies the river migrations (serve owns DDL, mirroring meta/ref InitSchema), and starts workers that share the delivery/sync code with the local queue. upstreams is the durable upstream-row store the sync worker's successor gate reads (syncWorker.Work, issue #79). A non-nil sweep schedules the periodic orphan-pack sweep (run once on start, then every Interval); nil disables it (hook subprocesses, tests without storage). A non-nil repack registers the background-repack worker; nil rejects EnqueueRepack (repacking disabled or not wired, e.g. insert-only hook clients that must never fold).

func NewRiverInsertOnly

func NewRiverInsertOnly(ctx context.Context, pgURL string) (*River, error)

NewRiverInsertOnly connects an insert-only client (no migrations — hooks never run DDL — and no workers): the hook subprocess enqueues into the serve process's durable queue.

func (*River) Close

func (q *River) Close()

Close stops the workers (waiting for in-flight jobs) and closes the pool. Unfinished durable jobs remain in Postgres and resume on next start.

func (*River) CompleteSyncCreate

func (q *River) CompleteSyncCreate(ctx context.Context, org, repoID string) (meta.CreateOutcome, error)

CompleteSyncCreate inserts the first-sync job and marks the create complete in one transaction on the shared database (issue #388).

func (*River) DrainSyncRuns

func (q *River) DrainSyncRuns(ctx context.Context, wait time.Duration)

DrainSyncRuns is serve's shutdown drain (issue #343), run while the listener still takes the runs' loopback pushes. River stops fetching jobs and waits up to wait for running ones. Then sync and push runs are cancelled with sync.ErrShutdown, so they release their claims and re-mark their requests without counting a run, and River keeps waiting until ctx ends; whatever still runs then (other job kinds) is cancelled as a plain stop. The worker keeps a cancelled run's job (shutdownSnooze), so the job, its Retry, and any request deduplicated against it pass to the next process or another replica. Close still runs afterwards.

func (*River) EnqueuePush

func (q *River) EnqueuePush(ctx context.Context, job sync.Job) error

EnqueuePush inserts one unique-by-repo outbound push job (issue #90). Insert works on insert-only hook clients too — workers are only needed to execute, not to enqueue.

func (*River) EnqueueRepack

func (q *River) EnqueueRepack(ctx context.Context, repo string) error

EnqueueRepack inserts one unique-by-repo repack job. Refused when this client has no repack worker registered: a durable job nobody works would sit queued forever AND its uniqueness would swallow every later insert.

func (*River) EnqueueSync

func (q *River) EnqueueSync(ctx context.Context, job sync.Job) error

func (*River) EnqueueWebhookDelivery

func (q *River) EnqueueWebhookDelivery(ctx context.Context, d events.Delivery) error

func (*River) TracksCreateState

func (q *River) TracksCreateState() bool

TracksCreateState: River runs only on PostgreSQL.

type SweepConfig

type SweepConfig struct {
	// Threshold is the minimum age (since blob upload) before an
	// unregistered pack blob counts as an orphan candidate. It must
	// comfortably exceed the longest possible push/import duration so
	// in-flight uploads are never classified.
	Threshold time.Duration
	// Delete reaps candidates instead of only reporting them. Off by
	// default everywhere: enable only after list-only reports have been
	// boring for a while (issue #5 ships list-only first).
	Delete bool
	// DeleteSuperseded reaps aged candidates in the RETAINED class —
	// blobs a ReplacePacks fold provably removed from the registry
	// (issue #31) — even when Delete is off. ON by default (the serve
	// config's OPENSTORAGE_SWEEP_DELETE_SUPERSEDED=false disables it):
	// unlike general orphans, whose registration status the sweeper infers
	// by diffing, this class carries a durable DB record written in the
	// fold's own transaction, and without deletion a drip-writer turns the
	// repack threshold into a storage multiplier bounded only by the
	// quota. Disposal still runs the full revalidate + quarantine
	// handshake and the same age threshold.
	DeleteSuperseded bool
	// DeleteLFS reaps aged candidates in the LFS class — lfs/ keys with no
	// lfs_objects registry row (issue #51). A SEPARATE opt-in from Delete
	// on purpose, and off by default: LFS objects uploaded before the
	// registry existed have no rows, so a deployment already running
	// Delete=true must not start reaping its entire pre-existing LFS
	// population on upgrade. Write-scoped touches (upload batches, re-PUTs)
	// adopt stored objects into the registry; enable this only after
	// list-only reports stop naming objects that should live. Disposal
	// runs the same revalidate + quarantine handshake and age threshold.
	DeleteLFS bool
	// DeleteLFSLegacy additionally admits LEGACY-SPELLED LFS keys
	// (lfs/<repo>/<oid>, ADR-41) to DeleteLFS's reap — the explicit
	// post-roll cleanup opt-in of the two-phase key migration (Codex
	// round 81). Off, legacy keys are never deleted regardless of every
	// other flag: during a rolling upgrade old replicas derive only that
	// spelling, and adoption's retained sources become aged rowless
	// orphans immediately, so an undiscriminating DeleteLFS would 404
	// their downloads mid-roll. Enable only once the fleet runs the new
	// binary (plus presign expiry); requires DeleteLFS.
	DeleteLFSLegacy bool
	// DeleteGuard, when set, is consulted before each sweep's destructive
	// work (Delete and DeleteSuperseded alike). A non-nil error downgrades
	// that run to list-only (heals still run) and is reported. It
	// re-checks preconditions that startup validated once but that can
	// drift while the process runs — e.g. S3 bucket versioning being
	// enabled mid-flight, which would turn deletes into billed delete
	// markers.
	DeleteGuard func(context.Context) error
	// PurgedRepoProbe, when set, makes the sweep PURGE-AWARE (issue #52):
	// aged unregistered blobs of purge-FENCED base repos classify as
	// reapable in every class (packs and LFS alike, no per-class opt-in
	// needed), because an in-flight pre-delete writer publishes its blob
	// BEFORE its fenced registration fails — the blob can land after a
	// purge's own prefix scan, and fenced repos never register again, so
	// such blobs are provably permanent garbage. Wired whenever the ref
	// store carries the purge seams, independent of the retention window:
	// `openstorage purge-repo` mints fences on retention-less deployments
	// too (PR #105 round 8). One extra point read per aged candidate not
	// already covered by another mode, mirroring IsRetainedPack.
	//
	// Probe presence also joins the destructive-mode set for quarantine
	// aging — purge-minted backups must be collected even with every
	// delete flag off, or reclamation would only move bytes into
	// quarantine — and for the per-run DeleteGuard consult, whose failure
	// downgrades purge-aware reaping alongside the delete modes.
	PurgedRepoProbe func(ctx context.Context, base string) (bool, error)
	// QuotaEnforced reports that pack-quota admission is enabled
	// (OPENSTORAGE_PACK_QUOTA_BYTES > 0). Quarantine restores take cover
	// reservations and hand off floor-generation bumps ONLY then (Codex
	// round 73): with enforcement off no admission consults floors or
	// covers, and the rows would only burn the per-repo reservation cap —
	// enough restores in one TTL could wedge later restores outright.
	QuotaEnforced bool
	// ReservationTTL, when positive, prunes pack-quota reservation rows
	// older than it each sweep (issue #29's crash-swept leaks). Use the
	// same TTL the write gates charge so pruning never races a live
	// reservation.
	ReservationTTL time.Duration
	// RecordLFSReapComplete, when set, persists the fleet's "last
	// completed destructive LFS pass" watermark (issue #120) — the
	// observed evidence LFS reclamation's slot refunds gate on
	// (RetireUnreferencedLFSObject refuses refunds when it goes stale).
	// Called only when this pass ran with the LFS delete class ARMED at
	// pass end (DeleteLFS configured and not demoted by the delete
	// guard) and finished BOTH keyspace listings with zero recorded
	// errors. Whole-pass cleanliness is deliberately stricter than
	// LFS-class cleanliness: a heal or pack-class failure may mean
	// quarantine backups are not draining or the listing machinery is
	// degraded, and the cost of a false freeze is only deferred refunds
	// — fail closed. A recording failure is reported like any sweep
	// error; the watermark simply does not advance. legacyArmed reports
	// whether the pass also reaped LEGACY-SPELLED LFS keys
	// (DeleteLFSLegacy): legacy registry rows may only be retired under
	// a legacy-armed watermark (Codex round 17 on PR #132) — an unarmed
	// pass deliberately spares legacy orphan blobs, so its watermark
	// must not certify their refunds. epoch is the pass's
	// BeginLFSReapEpoch value: refund settlement releases only buckets
	// whose retirement epoch is STRICTLY below a completion's (Codex
	// round 33 on PR #132) — a completion proves reaping only for
	// orphans that existed when its pass STARTED.
	RecordLFSReapComplete func(ctx context.Context, epoch int64, legacyArmed bool) error
	// BeginLFSReapEpoch, when set alongside RecordLFSReapComplete,
	// increments the fleet's destructive-sweep epoch at pass START,
	// before any blob listing (Codex round 33): retirements pin the
	// epoch they run under, and the increment's row lock serializes
	// against in-flight retire commits. A stamp failure withholds this
	// pass's completion watermark (refunds defer — the conservative
	// direction) without stopping the sweep itself.
	BeginLFSReapEpoch func(ctx context.Context) (int64, error)
}

SweepConfig controls one sweeper's classification and disposal.

type SweepReport

type SweepReport struct {
	Live  int // blobs present in the pack registry
	Fresh int // unregistered but younger than the threshold
	// Orphans is a bounded sample (maxOrphanSample) of the candidates:
	// an orphan backlog after an outage can be arbitrarily large, and the
	// report must not scale worker memory with it. OrphanCount and
	// OrphanBytes stay exact; every candidate is also logged as it is
	// classified.
	Orphans     []OrphanPack
	OrphanCount int
	OrphanBytes int64
	// Deleted counts orphans reaped this run. Their bytes move to a
	// quarantine backup rather than disappearing, so reaping frees nothing
	// by itself; BytesReclaimed counts only quarantine backups purged this
	// run — the moment storage consumption actually decreases.
	Deleted        int
	BytesReclaimed int64
	// SupersededDeleted counts retained-class reaps (issue #31, provably
	// superseded by a ReplacePacks fold), a subset of Deleted;
	// RetainedBytesReclaimed is the quota-usage headroom their reaps
	// returned via ReclaimRetainedPack.
	SupersededDeleted      int
	RetainedBytesReclaimed int64
	// ReservationsExpired counts crash-leaked pack-quota reservation rows
	// pruned this run (issue #29).
	ReservationsExpired int64
}

SweepReport summarizes one sweep run.

type SweepSchedule

type SweepSchedule struct {
	Sweeper  *Sweeper
	Interval time.Duration
	// Purger, when non-nil, runs a repo-retention purge pass after each
	// sweep on the same cadence (issue #52; nil when
	// OPENSTORAGE_REPO_RETENTION is unset).
	Purger *RepoPurger
}

SweepSchedule configures the periodic orphan-pack sweep on a queue backend: a River periodic job in postgres mode, a background ticker in local mode. Nil disables sweeping (hook subprocesses, tests).

type Sweeper

type Sweeper struct {
	// contains filtered or unexported fields
}

Sweeper finds (and optionally deletes) orphan pack blobs. Construct with NewSweeper; Sweep is safe to run concurrently with pushes and repacks (the age threshold is the guard) and is idempotent.

func NewSweeper

func NewSweeper(blobs blob.BlobStore, packs PackRegistry, cfg SweepConfig, logger *slog.Logger) *Sweeper

NewSweeper builds a Sweeper over the raw blob store and pack registry.

func (*Sweeper) Heal

func (s *Sweeper) Heal(ctx context.Context) (residual bool, err error)

Heal runs one standalone, strictly restorative quarantine pass. Serve calls it at startup when periodic sweeping is disabled (OPENSTORAGE_SWEEP_INTERVAL=0): a delete-enabled process that crashed mid-handshake may have left a registered pack's only copy in quarantine, and disabling the schedule must not strand it. Purging is forced off regardless of SweepConfig.Delete — interval zero documents "no sweeping", so an emergency stop must never destroy anything.

func (*Sweeper) Sweep

func (s *Sweeper) Sweep(ctx context.Context) (SweepReport, error)

Sweep lists every blob under packs/, diffs each repo group against the registered pack set, and reports orphan candidates (deleting them only when SweepConfig.Delete is set). Per-key delete failures do not stop the sweep; they are joined into the returned error so the job backend records the failure while the report stays complete.

Deletes never trust the initial listing snapshot: a retried push can overwrite an orphaned key (pack keys are content-addressed, so the retry reproduces the same key — and therefore identical bytes) at any point during the sweep, and no portable blob backend offers conditional deletes. Four layers make deletion safe anyway:

  1. revalidateOrphan re-reads the registry and the object's ModTime just before acting, sparing candidates registered or refreshed mid-sweep.
  2. safeDelete copies the bytes to a quarantine key before deleting, then re-reads the registry AFTER the delete: a registration that landed mid-delete is repaired on the spot by re-uploading the identical bytes. Pack restores are always correct by construction (content-addressed keys over server-built packs); LFS restores are digest-verified first, because a pre-registry legacy blob's bytes may not hash to its key's oid (see restoreFromQuarantine), and the LFS delete itself runs inside ref.GuardedLFSDelete so no verified upload's register→verify pair can complete between the quarantine copy and the delete (Codex round 20).
  3. healQuarantine, at the start of EVERY sweep (list-only included, so a rollback to list-only still repairs a delete-enabled crash), restores any quarantined pack whose key is registered but missing (covering a sweeper crash mid-handshake); in delete mode it also purges quarantine backups once they age past the threshold.
  4. Writers verify after registering (blob.EnsurePublished at every RegisterPacks call site): a registration that lands after safeDelete's recheck necessarily verifies after the delete, sees the missing key, and re-uploads it on the spot — so the recheck and the writer verify jointly cover every interleaving with no timing assumptions (see EnsurePublished's ordering proof).

Any residual exposure (e.g. a reader racing the restore itself) is transient, not data loss — hydration's bounded missing-pack retry absorbs it — and the next sweep's heal pass remains the crash backstop.

type SyncRunner

type SyncRunner func(ctx context.Context, job sync.Job) error

SyncRunner executes one sync run; production wiring passes sync.(*Worker).Run. Its error return is for logging only — documented outcomes (detach, overlapping run, failed sync with an emitted repo.sync.failed event) are fully handled inside the run.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL