ref

package
v0.19.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 30, 2026 License: Apache-2.0 Imports: 26 Imported by: 0

README

internal/ref

Overview

The ref authority. RefStore holds a repo's refs and registered packs; refs change only through compare-and-swap (CASRef), and packs register only after their bytes are durable. Three backends implement it: Postgres (PG), SQLite (SQLite), and S3WAL, which keeps each base repo's whole state in one manifest object advanced by conditional PUT (see the s3wal child package). Around that core the package holds the pack-quota ledger (reservations, usage, retained bytes, adoption claims, restore covers), the LFS object registry and reclamation rows, repo purge phases, absence-cover rows, job scan state, and the IdempotentCAS decorator that recovers ambiguous CAS outcomes.

Interface

  • ZeroSHA, ValidSHA, ErrNotFound, ErrRepoPurged, ErrRefNameNotUTF8 [ref.go:ZeroSHA] [purge.go:ErrRepoPurged]
  • Types: Pack, RefInfo, RefEntry, HydrateSnapshot [ref.go:Pack]
  • Contracts: RefStore, Store (adds InitSchema, Close), HydrateSnapshotter, ScanStateStore, RestoreCoverStore [ref.go:RefStore] [scanstate.go:ScanStateStore]
  • Open — dispatch postgres/postgresql, sqlite, or s3wal URLs; never runs InitSchema [ref.go:Open]
  • Backends: PG/OpenPG, SQLite/OpenSQLite, S3WAL/NewS3WAL (plus S3WAL.ConditionalStore) [ref.go:OpenPG] [sqlite.go:OpenSQLite] [s3walstore.go:NewS3WAL]
  • Ref and pack methods: ListRefs, ListRefsWithMeta, ListRefsAfter, GetRef, CASRef, CASRefGen, RegisterPacks, ReplacePacks, ListPacks, HasPack, PackRegisteredBytes, GetGeneration, HydrateSnapshot, DFConflicts, RefValues, CountRefs [ref.go:CASRef] [dfconflicts.go:DFConflicts] [refvalues.go:RefValues]
  • WithSyncClaimFence — attach a sync-claim holder fence to a CAS context [ref.go:WithSyncClaimFence]
  • IdempotentCAS — decorator that re-reads the ref after an ambiguous or lost CAS [idempotent.go:IdempotentCAS]
  • Quota ledger: NewReservationOwner, ReserveNoFloorGen, ReserveDecision, ReservePackBytes, RetirePackAttempts, SweepExpiredPackReservations, IsRetainedPack, ReclaimRetainedPack, QuotaLedgerState, PackQuotaLedgerState, IncomingBytes, Admitted, AddSat [ledger.go:ReservePackBytes]
  • Adoption and covers: AdoptionClaimSuffix, AdoptionClaimTTL, TryAdoptionClaim, RenewAdoptionClaim, ReleaseAdoptionClaim, RepairCoverSuffix, ReserveRestoreCover [ledger.go:TryAdoptionClaim]
  • LFS registry: MaxLFSObjectsPerRepo, ErrLFSObjectCapExceeded, RegisterLFSObject, RegisterAdoptedLFSObject, ReconcileLegacyLFSObjects, GuardedLFSDelete, BumpLFSAdoptionGen, LFSAdoptionGen, RecordLFSMismatch, LFSMismatchFingerprint, ClearLFSMismatch, CountLFSObjects, HasLFSObject, TouchLFSObject, LFSObjectRegisteredAt [ledger.go:RegisterLFSObject]
  • LFS reclamation: LFSObjectRow, LFSReposWithObjects, ListRepoLFSObjects, MarkLFSObjectsUnreferenced, ClearLFSObjectsUnreferenced, RetireUnreferencedLFSObject, SettleLFSRefunds [reclaim.go:RetireUnreferencedLFSObject]
  • Purge: FenceRepoPurge, IsRepoPurged, DropRepoRefs, RetireRepoPacks, DropRepoLFSObjects, CountRepoRows, PurgeRepoLedger [purge.go:FenceRepoPurge]
  • Absence covers: AbsenceCoverPrefix, AbsenceFoldedOwner, AbsenceCoverRow, AbsenceLFSRow, AbsencePackRow, ListFleetLFSObjects, ListFleetPacks, ListAbsenceCovers, AbsenceCoverHead, RenewAbsenceCovers, LapsedAbsenceCoverExists, CASAbsenceCoverOwner, DeleteAbsenceCover [absence.go:ListAbsenceCovers]
  • Scan state: JobScanState, ReapWatermark, DecodeReapWatermark, ErrReapWatermarkStale, ErrLegacyReapUnarmed, LFSRefundBucket, SettledLFSRefunds, ScanJobLFSReclaim, ScanJobAbsence, ScanJobSweep, ScanScope* builders, plus GetJobScanState, PutJobScanState, DeleteJobScanState, ParkRepoJobScanState, ListJobScanStateScopes, JobScanStateAge, BeginLFSReapEpoch, RecordLFSReapComplete, ArmAbsenceAdmissionFence [scanstate.go:JobScanState]
  • S3WAL sync-claim fence: S3WAL.SetSyncClaim, S3WAL.ClearSyncClaim [s3walstore_syncclaim.go:SetSyncClaim]
  • SQL tables refs, packs, repo_generations, pack_reservations, pack_retained, pack_usage, lfs_objects, lfs_object_counts, lfs_adoption_gens, lfs_mismatches, purged_repos, job_scan_state [ref.go:initSchemaTx] [sqlite.go:InitSchema]
  • Child package: s3wal — conditional-object CAS client, instance lease, v1 manifest schema, MemCAS fake, and boot probe used by the S3WAL backend

Invariants

  • assertf panics with ref assertion failed on bad arguments [ref.go:assertf]
  • casRef asserts valid SHAs, rejects zero-to-zero, returns ErrRepoPurged for a fenced repo before the swap, and reports won only when one row changed [ref.go:casRef] [sqlite.go:casRef]
  • Fenced CAS requires the upstream's sync_claim_holder to match the fence holder [sqlite.go:casRef]
  • RegisterPacks asserts non-empty packs and fails for purge-fenced repos; ReplacePacks returns won=false when the old set is not fully present [ref.go:RegisterPacks] [ref.go:ReplacePacks]
  • Stored SHAs are asserted valid on every read; ListRefsAfter orders by byte order [ref.go:ListRefsAfter]
  • IdempotentCAS never recovers a delete, a definitive error, or a clean create conflict; PG SQLSTATE 40003 and 08007 count as ambiguous [idempotent.go:reconcileLoss] [idempotent.go:ambiguousError]
  • ReservePackBytes denies with FloorStale on a floor-generation mismatch, KeyCapExceeded past 10,000 active rows, RenewalDegraded while the absence fence is armed, and on quota overflow [ledger.go:ReservePackBytes] [ledger_sqlite.go:ReservePackBytes]
  • addSat saturates at MaxInt64, and admitted denies a saturated total [ledger.go:admitted]
  • LFS registration rolls back above MaxLFSObjectsPerRepo and refuses purge-fenced repos [ledger.go:registerLFSObjectTx]
  • RetireUnreferencedLFSObject refuses with ErrReapWatermarkStale or ErrLegacyReapUnarmed, and skips moved generations, live adoption claims, and rows inside grace [reclaim.go:RetireUnreferencedLFSObject] [s3walstore_purge.go:RetireUnreferencedLFSObject]
  • Refunds settle only under a strictly newer sweep epoch [reclaim.go:SettleLFSRefunds]
  • DFConflicts counts only strict prefix or descendant refs as conflicts [dfconflicts.go:dfConflictsFromSorted]
  • S3WAL mutate maps exhausted contention to ErrCASContention, resolves ambiguous writes by writer nonce, and refuses growing manifests above the hard size cap [s3walstore.go:mutate] [s3walstore.go:checkManifestSize]
  • S3WAL InitSchema refuses to serve without a fresh store Date witness after the probe; with a nil probe (every store built by NewS3WAL) it returns nil without checking [s3walstore.go:InitSchema] [s3walstore.go:NewS3WAL]
  • S3WAL reservation expiry needs elapsed time plus a skew guard, and leans live when drift or Date checks fail [s3walstore_quota.go:s3walReservationLive]
  • S3WAL single-writer scan rows treat a 412 lost race as dual-serve evidence; the shared fleet-global lapse marker treats it as contention and retries with jitter [s3walstore_scanstate.go:scanMutateWith] [s3walstore_scanstate.go:scanMutateShared] [s3walstore_quota.go:s3walRecordLapse]
  • S3WAL scan rows stamped more than s3walScanFutureTolerance (2 s) ahead read as stale; a stamp inside that tolerance clamps to age 0 [s3walstore_scanstate.go:scanRowAge] [s3walstore_scanstate.go:s3walScanFutureTolerance]
  • GuardedLFSDelete never reports a delete when its S3WAL lease was lost [s3walstore_lfs.go:GuardedLFSDelete]

Side Effects

  • PG.InitSchema runs DDL under advisory locks 0x5350494458/0x5350494B45, installs generation triggers, and builds indexes concurrently [ref.go:InitSchema]
  • SQLite.InitSchema creates tables and generation triggers; writes use BEGIN IMMEDIATE [sqlite.go:InitSchema]
  • Ledger and purge paths take the per-base advisory lock and write the ledger tables listed above [ledger.go:ReservePackBytes] [purge.go:FenceRepoPurge]
  • GuardedLFSDelete invokes the caller's delete callback inside its guard [ledger.go:GuardedLFSDelete]
  • IdempotentCAS issues a 5 s point read on a detached context [idempotent.go:reconcileLoss]
  • S3WAL reads and writes manifests under _wal/repo/ and scan rows under _wal/scan/ through conditional GET and PUT, with per-base in-process locks and jittered retries [s3walstore.go:mutate] [s3walstore_scanstate.go:s3walScanPrefix]
  • S3WAL fleet walks list manifests 512 keys per page [s3walstore_quota.go:s3walEachBase]
  • S3WAL increments metrics.S3WALManifestSizeGuard and metrics.S3WALContentionExhausted, and logs size and skew warnings [s3walstore.go:countContention]
  • NewReservationOwner reads crypto/rand [ledger.go:NewReservationOwner]

Verification

  • Run: go test ./internal/ref/... -tags=openstorage_viewcheck -count=1 -race -timeout=30m
  • Needs Docker: the backend fan-out's /pg subtests start Postgres and its /s3wal subtests start SeaweedFS through s3waltest

Documentation

Overview

absence_sqlite.go mirrors absence.go's fleet-wide registry paging for the SQLite backend (issue #101) — see absence.go for the rationale.

purge.go implements the repository-purge row seams (issue #52). The purge orchestrator (jobs.RepoPurger) reclaims a soft-deleted repository by tombstoning its registry rows BEFORE deleting its blobs — the inverse of publish-before-CAS: a crash mid-purge must leave reapable tombstones (pack_retained rows, orphan-class blobs), never a committed ref pointing at deleted bytes.

Every seam takes the repository's full accounting group (base repo plus its namespaces, blob.QuotaRepoGroup order — ref stays namespace-agnostic like ReservePackBytes) and is idempotent: a resumed purge re-runs each phase as a no-op. These methods live on the concrete stores rather than RefStore on purpose — the purge is wired against the raw backends, and RefStore's many test/simulation mirrors must not grow with it.

purge_sqlite.go mirrors purge.go's repository-purge row seams for the SQLite backend (issue #52) — statement-for-statement twins; see purge.go for the phase rationale and invariants.

Package ref provides the ref authority + pack registry: the RefStore contract plus the Postgres implementation (pgx).

S3WAL implements Store over the s3wal manifest-CAS substrate (issue #133, docs/plans/s3wal-plan.md): the ENTIRE authority state of one base repo group {base, +ephemeral, +import} lives in one JSON manifest object advanced by conditional PUT, so every multi-entity transaction the SQL backends run is a single-object CAS here.

The implementation lives in package ref (not package s3wal) on purpose: it needs ref's unexported machinery — casFence, notExecuted, the ledger knobs (maxActiveReservations, reconcileLegacyLFSBatch), baseRepoOf, lfsClaimTwin — and the internal conformance tests must inspect it, while ref.Open must dispatch to it. A store.go inside package s3wal would be an import cycle in both directions; s3wal stays a leaf substrate (manifest codec, conditional clients, fakes).

Concurrency: every manifest operation — reads included — runs under a per-base in-process mutex. This mirrors the serialization the SQL backends already impose (PG's lockBaseLedger advisory lock, SQLite's single-writer BEGIN IMMEDIATE) under the single-node decree, and it is what keeps the casobj session-scoped seq watermark sound: casobj.go documents that goroutines sharing one Client can process responses out of store-serialization order on a hot key, so unserialized same-key reads could trip the stale-read assert on benign reordering. Cross-PROCESS writers remain serialized by the manifest CAS itself.

S3WAL absence-cover surface (issues #101/#125, PR #132; plan §1/§2.10): covers are ordinary reservation-map entries in each base group's manifest (ReserveRestoreCover charges them like any admission), so the fleet listing/head/renew/CAS/delete surface is a `_wal/repo/` prefix walk plus per-manifest CAS — never a separate table.

Raw-key spelling: the SQL backends expose RawKey as the reservation-row spelling (synthetic key + RepairCoverSuffix + reservKeySuffix). s3wal manifests key reservations on PLAIN blob keys (no era suffix — the documented divergence in s3walstore_quota.go), but this surface still EMITS the fleet-uniform decorated spelling and strips it back at the boundary: RawKey is a persisted cursor and fold-CAS identity (issue #121 stores it durably), so its observable shape must not depend on which backend minted it.

Listing order: rows are emitted per base group in decorated-raw byte order, base groups in manifest byte order — a total order that matches the SQL backends' global raw-key byte order except when one base ID is a byte-prefix of another (the §2.9 family caveat: '/' vs '-' sort around the base boundary). Completeness and duplicate-freedom under cursor resume hold regardless, which is the property the renewal listing consumes.

S3WAL LFS registry: verified-object registration with legacy-spelling migration and twin reconciliation (ADR-41), the object-count cap, adoption claims and generations, mismatch markers, and the §2.5 W6 delete-lease GuardedLFSDelete (docs/plans/s3wal-plan.md).

LFS state is keyed by the BASE repository (ADR-35): the manifest's single LFS block is the group's registry.

S3WAL repository-purge, LFS-reclamation, and absence-scan seams (issues #52/#73/#101). Purge phases are each ONE manifest CAS in the SQL pipeline's order; the fence is a field of the same object every write CASes, so the fence-first total order is inherent. The manifest object is NEVER deleted — it persists as the burned-name tombstone.

S3WAL quota ledger: reservations, admission arithmetic, retained-byte accounting, and the reservation janitor over manifest CAS (issues #29/#31 semantics, s3wal mapping per docs/plans/s3wal-plan.md §1/§2).

Reservations key on PLAIN blob keys: s3wal has no v0.1.0 legacy binaries, so the era-spelling fence (reservKeySuffix) does not exist here — an intentional, documented divergence from the SQL backends.

Every TTL comparison follows the §2.4 manifest-clock rules: stamps are written with the manifest logical clock, elapsed time is min(localNow, clk_candidate) − stamp, and destructive expiry (pruning, steals) demotes to report-only under the fail-closed skew guard.

S3WAL fleet-persistent scan state (PR #132, plan §2.10): one CAS-advanced side object per (job, scope) under _wal/scan/<job>/<hex(scope)>.

Mapping decisions, concrete and documented:

  • Keys are extension-free (the §2.1 ordering argument: a ".json" suffix corrupts listing order between prefix-related scopes) and the scope is LOWERCASE-HEX encoded — the plan's "path-escaped" sketch is replaced deliberately: %XX escaping reorders '%' (0x25) against literals, while lowercase hex is a strictly monotone, fixed-width (2:1), reversible injection, so bucket listing order over encoded keys IS byte order over the unencoded scopes — exactly the ORDER BY scope COLLATE "C" contract ListJobScanStateScopes promises on the SQL backends.
  • The clock story: the SQL backends stamp updated_at with the DATABASE clock so freshness verdicts never mix application clocks. s3wal side objects have no manifest logical clock, but under the single-process decree every scan-state reader and writer shares ONE process clock, so the "database clock at both ends" contract collapses to the local clock honestly. Stamps are ClkNext-monotone per object (a backward local-clock jump can never regress updated_at); ages inside a small future tolerance clamp to zero, and a stamp further ahead of the local clock is UNVERIFIABLE — it reads as stale (fail closed), never fresh (see scanRowAge).
  • Single-writer discipline (§2.10): River's unique periodic jobs — and here, the single-process decree — make each (job, scope) row single-writer. In-process writers of one key serialize on a per-key mutex, so a conditional-write conflict that still reaches the store is evidence of a SECOND serving process: it fails loudly and is never retried. The §2.11 instance lease (Phase 3) upgrades the loud error to a process fail-stop. ONE deliberate exception: fleet-global markers upserted by concurrent receive-hook processes (the lapse-evidence marker) are multi-writer by design and retry conflicts instead — see scanMutateShared.
  • There is no cross-delete seq watermark for side objects: unlike manifests, scan rows are legitimately deleted and re-created (bucket drains, watermark resets), which resets Seq to 1. The per-key lock plus the CAS conflict policy are the staleness defense instead; the shared casobj DecodeSeq probe deliberately does not match this codec (distinct field names), so the manifest watermark stays unarmed here.

Phase 3 sync-claim write-through seam (plan §2.8, issue #133): the S3 meta backend (internal/meta) mirrors every inbound sync claim into the base repo group's manifest so a fenced CASRef checks the holder in the SAME object it CASes — steal-vs-write serialization is the manifest CAS itself, strictly stronger than the SQL FOR-UPDATE argument. These two exported methods are the entire seam: the meta backend calls them through a small interface (meta.SyncClaimSink) so ref never imports meta; wiring happens at serve time.

Like the SQL fence (`... AND sync_claim_holder = $4`), the manifest fence checks holder identity only — expiry arbitration stays in the meta store's claim lease, whose CAS decides steals before the thief rewrites this token. CreatedAt is the manifest write clock, recorded for observability, never compared.

Index

Constants

View Source
const (
	// ScanJobLFSReclaim owns the LFS reclamation scan's rotation cursor
	// and per-repo row-settlement cursors (issue #121).
	ScanJobLFSReclaim = "lfs_reclaim"
	// ScanScopeReclaimRotation is the reclaimer's rotating repo cursor.
	ScanScopeReclaimRotation = "rotation"
	// ScanJobAbsence owns the absence reconciler's registry rotation
	// cursors and cover-renewal cursor (issue #125).
	ScanJobAbsence = "absence_reconcile"
	// ScanScopeAbsenceLFS/Packs are the two registry rotation cursors;
	// ScanScopeAbsenceRenewal is the cover-renewal listing cursor;
	// ScanScopeAbsenceProofs carries per-base fenced class proofs
	// awaiting their counterpart class (Codex round 21 on PR #132).
	ScanScopeAbsenceLFS     = "lfs"
	ScanScopeAbsencePacks   = "packs"
	ScanScopeAbsenceRenewal = "renewal"
	ScanScopeAbsenceProofs  = "proofs"
	// ScanScopeAbsenceRenewalHealth is a DB-clock watermark stamped
	// after every SUCCESSFUL bulk cover renewal; its age is the proof
	// that TTL'd absence covers cannot have lapsed un-renewed.
	// ScanScopeAbsenceAdmissionFence, while present, fails finite
	// quota-gated admissions closed (Codex round 26 on PR #132,
	// security): it is armed when a renewal failure finds the health
	// watermark absent or older than the cover-lapse safety bound, and
	// cleared only after a healthy pass completes a full recovery
	// rotation of both registry classes.
	ScanScopeAbsenceRenewalHealth  = "renewal-health"
	ScanScopeAbsenceAdmissionFence = "admission-fence"
	// ScanScopeAbsenceLapse is the durable lapse-evidence marker (Codex
	// round 33 on PR #132): ReservePackBytes' expired-row GC records it
	// in the same transaction whenever it deletes an expired UN-FOLDED
	// absence-cover row — physical evidence the reconciler's
	// first-health-stamp probe would otherwise never see, because the
	// very writers the admission fence should block prune it first.
	// Consumed (deleted) when the fence settles after a full recovery
	// rotation.
	ScanScopeAbsenceLapse = "lapse-evidence"
	// ScanJobSweep owns the sweeper's watermarks. The destructive-reap
	// scope's updated_at is the "last completed destructive LFS pass"
	// watermark (issue #120): written only when a sweep ran with the LFS
	// delete class armed (not demoted by the delete guard) and finished
	// its keyspace with zero recorded errors; consulted transactionally
	// by RetireUnreferencedLFSObject before any slot refund.
	ScanJobSweep                = "sweep"
	ScanScopeLFSDestructiveReap = "lfs-destructive-reap"
	// ScanScopeLFSDestructiveReapLegacy mirrors the completion watermark
	// but is written ONLY by legacy-armed sweeps (Codex round 33 on PR
	// #132): a later canonical-only completion overwrites the main
	// scope's payload, and legacy refund settlement must not lose the
	// proof that a legacy-armed sweep completed at some epoch.
	ScanScopeLFSDestructiveReapLegacy = "lfs-destructive-reap-legacy"
	// ScanScopeLFSReapEpoch holds the monotone destructive-sweep epoch
	// in its GENERATION column (Codex round 33 on PR #132): every
	// destructive LFS sweep increments it at pass START, committed
	// before the blob listing begins. Retirement records the epoch it
	// ran under; a completion watermark whose epoch is STRICTLY GREATER
	// proves a full clean sweep began after the retirement committed —
	// the only sound "the orphan was visible to that sweep" relation
	// (wall-clock comparison is not: PG now() is transaction-start
	// time, and a sweep can start between a retire's now() and its
	// commit).
	ScanScopeLFSReapEpoch = "lfs-reap-epoch"
)

Scan-state row coordinates. Scopes are per-job namespaces; helpers below build the parameterized ones.

View Source
const AbsenceCoverPrefix = "absence-lost/"

AbsenceCoverPrefix is the synthetic blob-key namespace the absence reconciler's aggregate covers reserve under (jobs.AbsenceReconciler). No real blob key can collide: pack keys live under "packs/", LFS keys under "lfs/". The prefix is load-bearing for ListAbsenceCovers' range scan and must not contain "\n" (reservRowKey rejects it).

View Source
const AbsenceFoldedOwner = "absence-folded"

AbsenceFoldedOwner is the sentinel owner the reconciler CASes onto a legacy nonce cover once both registry rotations proved its base's true deficits within one pass (the recomputed ground truth subsumes whatever the nonce aggregated): marked rows leave the bulk touch and decay by TTL. Reservation owners are 32 hex chars (NewReservationOwner), so the sentinel cannot collide, and every ReserveRestoreCover re-charge overwrites owner — an old replica re-charging its nonce un-marks it.

View Source
const AdoptionClaimSuffix = "/adopt-claim"

AdoptionClaimSuffix distinguishes an adoption claim row (see RefStore.TryAdoptionClaim) from real reservations and covers: LFS keys end in the 64-hex oid, so the suffix never collides, and neither settlement (blob-key spellings), retirement (era rows), nor the admission GC (which excludes this suffix — Codex round 74: the GC runs with the ADMISSION ttl, as short as ~61s, while a legitimate hash runs 5m + 1s/MiB) ever touches it. A live claim contributes one byte to admission sums (deny-side, negligible); the sweeper prune reaps claims older than AdoptionClaimTTL.

View Source
const AdoptionClaimTTL = 30 * time.Minute

AdoptionClaimTTL is the lease length of one adoption claim: generously longer than the renewal cadence the hashing winner maintains (renewals restart the clock chunk-by-chunk), yet finite so a crashed winner's claim expires and the next write-scoped touch takes over.

View Source
const RepairCoverSuffix = "/repair-cover"

RepairCoverSuffix distinguishes a restore's cover reservation key from the restored blob's settle key (Codex rounds 57+58): settlement consumes the blob key's own spellings, and the cover must instead outlive the request until the TTL collects it. Pack keys end in ".pack" and LFS keys in the 64-hex oid, while the ledger reserves "\n" for its era spellings, so this suffix never collides with a real reservation.

View Source
const ReserveNoFloorGen int64 = -1

ReserveNoFloorGen skips ReservePackBytes' transactional floor-generation recheck (Codex round 60). Legitimate only where the reservation is not a floor-based admission: restore covers (unbounded quota — the floor input is a constant 0) and quota-disabled maintenance paths. Every real admission must pass the generation its floor was validated against.

View Source
const ScanScopeReclaimRefundRoot = "refund/"

ScanScopeReclaimRefundRoot prefixes every pending-refund bucket scope — the reclaimer's rotation pages it to settle bases that vanished from LFSReposWithObjects (all rows retired).

View Source
const ScanScopeReclaimRefundRotation = "refund-rotation"

ScanScopeReclaimRefundRotation is the reclaimer's durable cursor over the refund bucket keyspace ('-' sorts below '/': the cursor row never collides with the ScanScopeReclaimRefundRoot listing range).

View Source
const ZeroSHA = "0000000000000000000000000000000000000000"

ZeroSHA is git's null object id (40 zeros) marking ref create/delete.

Variables

View Source
var ErrLFSObjectCapExceeded = errors.New("repository LFS object count limit reached")

ErrLFSObjectCapExceeded is RegisterLFSObject's typed denial: inserting this NEW object would take the repository past MaxLFSObjectsPerRepo. Re-registration of an existing key never returns it (zero growth).

View Source
var ErrLegacyReapUnarmed = errors.New("legacy LFS reaping is not armed: legacy row retirement deferred")

ErrLegacyReapUnarmed reports that a retirement verdict refused to retire a LEGACY-SPELLED registry row because the freshest destructive-reap watermark does not certify legacy reaping (see ReapWatermark). The caller defers the row; it retires once a legacy-armed sweep completes.

View Source
var ErrNotFound = errors.New("ref not found")

ErrNotFound reports a GetRef miss: the ref does not currently exist.

View Source
var ErrReapWatermarkStale = errors.New("destructive LFS sweep watermark is stale: slot refund deferred")

ErrReapWatermarkStale reports that a retirement verdict refused to refund an object-cap slot because the sweeper's destructive-reap watermark is absent or older than the caller's bound — destructive LFS sweeps are not demonstrably completing, so refunds must wait (issue #120). The caller reports the deferral distinctly and retries on a later pass; nothing was written.

View Source
var ErrRefNameNotUTF8 = errors.New("s3wal: ref name is not valid UTF-8")

ErrRefNameNotUTF8 rejects ref names carrying invalid UTF-8 (git accepts arbitrary non-NUL bytes in ref names, e.g. a trailing 0xff). The s3wal JSON codec cannot represent such keys losslessly — encoding/json rewrites invalid bytes to U+FFFD, so a raw-0xff name that PASSED policy (the update hook compares raw bytes, and raw-0xff ≠ the U+FFFD-spelled protected ref) would encode onto the SAME manifest key as the protected ref and overwrite its SHA on decode (duplicate JSON keys keep the last). s3wal therefore fails closed BEFORE any manifest mutation. Pre-existing cross-backend divergence, documented rather than unified: PostgreSQL TEXT rejects invalid UTF-8 at the wire with its own encoding error, while SQLite stores the raw bytes verbatim. Reads need no guard: no manifest can contain such a key, so GetRef/hydration comparisons with raw-byte arguments are clean misses.

View Source
var ErrRepoPurged = errors.New("repository is being purged")

ErrRepoPurged rejects a pack or LFS registration for a repository whose purge has begun (fence row present, see FenceRepoPurge): the write pipeline resolved the repository before its soft-delete, and letting the registration commit could strand rows — or worse, re-point storage — behind a completed purge. Surfaces to the writer as a failed push.

View Source
var MaxLFSObjectsPerRepo = int64(100_000)

MaxLFSObjectsPerRepo caps a repository's REGISTERED LFS object count (issue #51, Codex security round 2): registered objects are permanent rows plus bucket keys, and the byte quota bounds their population at nothing (16M three-byte objects cost 48 MiB). Enforced ATOMICALLY inside RegisterLFSObject's serialized transaction (Codex round 6: an HTTP-layer read-then-register check admits concurrent overshoot); the LFS surface also pre-checks it for fast-fail UX. A var for tests.

Functions

func AddSat

func AddSat(a, b int64) int64

AddSat is the exported saturating sum, for callers combining the usage counter with the uncounted-bytes floor outside the ledger (the LFS batch's advisory check) exactly as ReservePackBytes does (round 44).

func Admitted

func Admitted(used, reserved, incoming, quota int64) bool

Admitted is the exported, side-effect-free admission arithmetic — the same saturating comparison ReservePackBytes applies transactionally. The LFS batch surface uses it for its ADVISORY fast-fail check (issue #51, Codex round 17): batch-time durable holds proved rotatable (a fresh OID per TTL re-acquires the whole headroom without ever transferring), so batches insert nothing and only real transfer progress reserves.

func IncomingBytes

func IncomingBytes(packs []Pack) int64

IncomingBytes is the exported saturating declared-size sum for admission callers. The LFS batch surface sums CLIENT-DECLARED sizes before its advisory Admitted check; a naive sum of two MaxInt64 entries wraps negative and panics Admitted's non-negative assertion instead of denying (Codex round 21), so the sum must use the ledger's own arithmetic.

func NewReservationOwner

func NewReservationOwner() string

NewReservationOwner mints the random owner token stamped on one writer's reservation rows. The owner is pure provenance (Codex round 22): rows end by settlement (RegisterPacks/RegisterLFSObject consume by key), token-bound retirement (RetirePackAttempts), or TTL expiry — same-key rows are shared by every concurrent writer via refresh, so no owner-scoped delete can ever be sound. The mixed-fleet fence against v0.1.0's owner-scoped release lives in the row SPELLING, not in this column's value — see reservKeySuffix.

func ReserveRestoreCover

func ReserveRestoreCover(ctx context.Context, st RestoreCoverStore, group []string, key string, size int64, ttl time.Duration) error

ReserveRestoreCover accounts bytes a writer-side restore is about to re-materialize under an already-registered key (packs and LFS alike, Codex rounds 57+58). The shared floor-generation bump after a restore only stops FUTURE floor serves: an admission already holding a floor scanned during the blob's absence (deflated by the missing object's size, hiding equally sized rowless legacy bytes) can commit after the bytes return and durably exceed quota. The cover makes those admissions see the returning bytes in active_reservations instead. It reserves under an unbounded quota — restores must never be quota-denied, or a registered row stays blobless — but the row cap still binds and any denial fails the restore (the retry converges); the TTL collects the row after the last stale floor can commit.

func ScanScopeReclaimRefund

func ScanScopeReclaimRefund(base string, epoch int64) string

ScanScopeReclaimRefund is the pending-refund bucket scope for one base and retirement epoch. Zero-padded so bucket scopes list in epoch order.

func ScanScopeReclaimRefundPrefix

func ScanScopeReclaimRefundPrefix(base string) string

ScanScopeReclaimRefundPrefix prefixes one base's pending-refund bucket scopes (Codex round 33 on PR #132): retirement no longer refunds count/usage in its own transaction — a completion watermark only proves a sweep that ran BEFORE the deletion, which necessarily spared the blob. Refunds park in per-epoch buckets and settle once a clean sweep with a STRICTLY GREATER epoch completes.

func ScanScopeReclaimRepo

func ScanScopeReclaimRepo(base string) string

ScanScopeReclaimRepo is the reclaimer's per-repo row-settlement cursor scope for one base repository.

func ValidSHA

func ValidSHA(s string) bool

ValidSHA reports whether s is a well-formed 40-hex git object id.

func WithSyncClaimFence

func WithSyncClaimFence(ctx context.Context, repo, holder string) context.Context

WithSyncClaimFence fences every CASRef executed under the returned context: the swap applies only while repo's upstream-sync claim is held by holder, evaluated ATOMICALLY inside the CAS statement itself against the repo_upstreams table (deployment invariant: meta and ref share one database — a single --data-dir sqlite file or the single serve PG URL). A fenced CAS whose holder no longer matches (its claim was stolen, or the upstream was detached) reports won=false, exactly like losing the SHA race: the stale sync push's update is refused before publication. Only the update hook sets this, and only for loopback sync pushes (EnvSyncClaim).

Types

type AbsenceCoverRow

type AbsenceCoverRow struct {
	// RawKey is the reservation-row spelling — the opaque keyset cursor
	// for the next page.
	RawKey string
	// Key is the synthetic cover key ReserveRestoreCover was charged
	// with (reservation spellings stripped), the spelling a renewal
	// re-charges.
	Key string
	// Base is the base repository the reservation charges.
	Base string
	// Bytes is the row's reserved size (GREATEST-pinned across charges).
	Bytes int64
	// Owner is the row's current owner column verbatim: a random
	// reservation token or the AbsenceFoldedOwner sentinel (Codex
	// rounds 18-19 on PR #132). Every re-charge overwrites it, so CAS
	// on the listed value detects concurrent re-charges.
	Owner string
	// Folded reports the row carries the AbsenceFoldedOwner sentinel:
	// both rotations proved the row redundant and no re-charge has
	// arrived since (every re-charge overwrites owner).
	// Meaningful for legacy nonce rows; deterministic rows read false.
	Folded bool
}

AbsenceCoverRow is one absence-cover reservation row as the reconciler's renewal phase sees it (issue #125): the durable record of a charged loss aggregate, re-derived at pass start instead of persisting the tracking set separately.

type AbsenceLFSRow

type AbsenceLFSRow struct {
	Repo  string
	Key   string
	Bytes int64
}

AbsenceLFSRow is one registered LFS object as the absence reconciler sees it: the row key (canonical or v0.2.0 legacy spelling) and its counted bytes.

type AbsencePackRow

type AbsencePackRow struct {
	Repo  string
	ID    string
	Key   string
	Bytes int64
}

AbsencePackRow is one registered pack as the absence reconciler sees it. ID is the keyset cursor component (the PK is (repo, id)); Key is the blob key the row's bytes live under. Bytes is 0 for rows registered before the packs.bytes migration (size unknown — reported, never covered).

type HydrateSnapshot

type HydrateSnapshot struct {
	Generation int64
	Packs      []Pack
	Refs       []RefEntry
}

HydrateSnapshot is one coherent point-in-time view of a repo's hydration inputs: its generation, its registered packs (oldest first, the ListPacks order), and its FULL ref set (ascending byte order of name, the ListRefsAfter order).

type HydrateSnapshotter

type HydrateSnapshotter interface {
	HydrateSnapshot(ctx context.Context, repo string) (HydrateSnapshot, error)
}

HydrateSnapshotter is an optional RefStore capability (issue #182) for backends whose GetGeneration/ListPacks/ListRefsAfter reads each cost a full round trip over the SAME underlying object: s3wal re-fetches the whole base manifest per call, so a hydrate pass that consumed them separately paid three-plus round trips for one object's state. Backends implement this ONLY when (a) the three views can be served from one atomic read and (b) the full ref set is inherently bounded (the s3wal manifest hard cap) — SQL backends keep the issue-#119 paged scan, whose per-page cost is a cheap query, and never materialize unbounded ref tables.

type IdempotentCAS

type IdempotentCAS struct {
	RefStore
}

IdempotentCAS decorates a RefStore so a CASRef whose outcome is ambiguous (or, for updates, cleanly lost) is recovered as a win when the ref already holds the target sha — see CASRef for the exact matrix. It is applied at the two write seams (events.NewRefStore wraps its inner store; the update hook's eventless fallback wraps directly), so every CAS caller — git hook path, API branch/tag/notes handlers, commit publish, merge fast-forward, fork — sees reconciled outcomes.

func (IdempotentCAS) CASRef

func (s IdempotentCAS) CASRef(ctx context.Context, repo, name, old, new string) (bool, error)

CASRef delegates to the inner store and reconciles non-win outcomes by operation shape (the same create/update/delete split CASRef itself switches on):

  • ambiguous error (see ambiguousError), create or update: recheck; current == new is a win — our statement may well have committed, and reporting the win emits the push webhook that would otherwise be dropped.
  • definitive error (the backend rejected the statement, or the request provably never reached it): NEVER recovered — the statement did not commit, so current == new only means someone else put the target there; recovering would fabricate a success (and a push event) for a transition this call never committed. The error propagates; a client's retry then converges through the clean-loss rules below.
  • clean loss (won=false, err=nil), update (old and new both non-zero): recheck; current == new is a win. This heals the client that retries an update whose first attempt committed but failed to report and would otherwise see a spurious conflict against its own sha.
  • clean loss, create (old == ZeroSHA): NEVER recovered. A clean create loss is authoritative — our INSERT affirmatively wrote nothing and the ref exists. "Already exists" is the correct answer even when the existing sha equals the target: recovering would turn every same-sha duplicate create into a fabricated success and emit a fabricated zero->sha push event for a transition this call never committed.
  • delete (new == ZeroSHA): NEVER recovered, clean or ambiguous. The row is absent after a committed delete, and absence cannot distinguish "our DELETE committed" from "the ref never existed" or "someone else deleted it" — unlike current == new, it is not positive evidence. Accepted cost: a client retrying a delete whose first attempt committed but failed to report sees a spurious CAS conflict; the end state (ref absent) is the one it asked for.

Callers observing a recovered win may emit duplicate webhooks for one committed update (first attempt + recovered retry); delivery is at-least-once, so duplicates are acceptable.

Generation interaction (issue #4): a recovered win needs no generation bump. The bump commits atomically with the swap itself (schema-level AFTER-row triggers fire inside the swap statement's own transaction), so an ambiguous-but-actually-committed CAS already moved the generation, and the recheck below is a pure point-read. Recovering here can therefore never leave a won CAS without its bump, nor bump twice.

func (IdempotentCAS) CASRefGen

func (s IdempotentCAS) CASRefGen(ctx context.Context, repo, name, old, new string) (bool, int64, error)

CASRefGen applies the same recovery matrix as CASRef. A recovered win forwards gen == 0 — unattributed by contract: the swap committed and bumped the generation, but WHICH value it produced is unknowable after the fact (the point-read recheck observes current state, not the commit that made it), and guessing via GetGeneration could blend in a concurrent writer's bump. Consumers coalescing generation ranges treat 0 as an unclaimed mutation — the fail-safe rescan direction.

type JobScanState

type JobScanState struct {
	Payload    string
	Generation int64
	// UpdatedAt is the row's last write time on the DATABASE clock
	// (PG now(); SQLite unix-ms), the clock freshness bounds compare on.
	UpdatedAt time.Time
}

JobScanState is one durable scan-state row.

type LFSObjectRow

type LFSObjectRow struct {
	Key               string
	Bytes             int64
	UnreferencedSince time.Time
}

LFSObjectRow is one registered LFS object as the reclamation scan sees it: the row key (canonical or v0.2.0 legacy spelling), its counted bytes, and the reclamation mark (zero while unmarked).

type LFSRefundBucket

type LFSRefundBucket struct {
	Rows        int64 `json:"rows,omitempty"`
	Bytes       int64 `json:"bytes,omitempty"`
	LegacyRows  int64 `json:"legacy_rows,omitempty"`
	LegacyBytes int64 `json:"legacy_bytes,omitempty"`
}

LFSRefundBucket is one pending-refund bucket's JSON payload: the canonical- and legacy-spelled row counts and byte totals retired under the bucket's epoch (the row's generation column). Legacy parts settle only under a legacy-armed completion — canonical-only sweeps spare legacy orphans.

type PG

type PG struct {
	// contains filtered or unexported fields
}

PG implements Store on Postgres.

func OpenPG

func OpenPG(ctx context.Context, pgURL string) (*PG, error)

OpenPG connects to pgURL and pings the database.

func (*PG) AbsenceCoverHead

func (s *PG) AbsenceCoverHead(ctx context.Context, prefix string, ttl time.Duration) (AbsenceCoverRow, bool, error)

AbsenceCoverHead returns the HIGHEST live cover row whose synthetic key starts with prefix (byte order; the reconciler's zero-padded incarnation spelling makes byte order numeric order). ok=false when none exists. This is the reconciler's point lookup for a base's current aggregate incarnation before charging into it (issue #125). ttl filters expired rows exactly like ListAbsenceCovers: charging into an expired higher incarnation would resurrect its stale GREATEST-pinned total.

func (*PG) ArmAbsenceAdmissionFence

func (s *PG) ArmAbsenceAdmissionFence(ctx context.Context, payload string) error

ArmAbsenceAdmissionFence upserts the admission-fence marker under the EXCLUSIVE side of the fleet-global fence lock (Codex round 35 on PR #132, security): every finite quota admission holds the SHARED side across its fence check and reservation sums, so arming waits out in-flight admissions — whose covers were still inside the sums the arming pass saw — and every admission starting afterwards observes the row and fails closed. A plain PutJobScanState upsert had no such ordering on PostgreSQL.

func (*PG) BeginLFSReapEpoch

func (s *PG) BeginLFSReapEpoch(ctx context.Context) (int64, error)

BeginLFSReapEpoch — see ScanStateStore.

func (*PG) BumpLFSAdoptionGen

func (s *PG) BumpLFSAdoptionGen(ctx context.Context, repo string) error

BumpLFSAdoptionGen — see RefStore (Codex round 49).

func (*PG) CASAbsenceCoverOwner

func (s *PG) CASAbsenceCoverOwner(ctx context.Context, rawKey, expectedOwner string, expectedBytes int64, newOwner string) (bool, error)

CASAbsenceCoverOwner rewrites one legacy cover row's owner by its raw reservation-row key, guarded on the (owner, bytes) the caller listed (Codex rounds 18-19 on PR #132): every ReserveRestoreCover re-charge overwrites owner and GREATEST-grows bytes, so a matched CAS proves no re-charge landed between the listing and this write — the folded sentinel must never clobber a newer reliance. ok=false when the row vanished or drifted; nothing is written.

func (*PG) CASRef

func (s *PG) CASRef(ctx context.Context, repo, name, old, new string) (bool, error)

CASRef performs the compare-and-swap ref update. Returns won=false (no error) when the precondition failed, i.e. someone else won the race. Refs only ever change inside these single statements, so a broken/partial ref state is impossible by construction; the statement runs inside an explicit transaction whose only other work happens strictly BEFORE it (the base-ledger lock and the purge-fence read below), so the statement remains the single mutation unit and the IdempotentCAS ambiguity classification (idempotent.go) is unchanged: pre-statement failures are tagged notExecuted, statement failures classify exactly as before, and a lost COMMIT response stays ambiguous (recovered by the recheck). A won swap also bumps the repo generation via the InitSchema-installed deferred constraint trigger at commit.

The transaction takes the base-ledger advisory lock first and rejects purge-fenced repos with ErrRepoPurged (issue #52, PR #105 security round): ref writers do not all pass through pack registration — REST branch/tag creation CASes an existing sha with no new pack — so without this, a create landing between the purge's zero-row verification and its completion would strand a ref row forever. The lock totally orders every CAS against FenceRepoPurge's insert; registration paths already take it, so per-base writer serialization is the established cost of the fence.

Under WithSyncClaimFence the statement additionally requires the repo's sync claim to be held by the fencing holder, with FOR UPDATE on the repo_upstreams row: a concurrent claim steal serializes against the CAS instead of racing its snapshot, so a stolen run's update can never commit after the thief's claim (issue #86).

func (*PG) CASRefGen

func (s *PG) CASRefGen(ctx context.Context, repo, name, old, new string) (bool, int64, error)

CASRefGen implements the RefStore contract: exact win attribution. See casRef for how the read composes with the deferred bump trigger.

func (*PG) ClearLFSMismatch

func (s *PG) ClearLFSMismatch(ctx context.Context, repo, key string) error

ClearLFSMismatch — see RefStore (#104, Codex round 132).

func (*PG) ClearLFSObjectsUnreferenced

func (s *PG) ClearLFSObjectsUnreferenced(ctx context.Context, repo string, keys []string) (int64, error)

ClearLFSObjectsUnreferenced removes the reclamation mark from the given rows (their OIDs were observed referenced by a reachable pointer file).

func (*PG) Close

func (s *PG) Close()

func (*PG) CountLFSObjects

func (s *PG) CountLFSObjects(ctx context.Context, repo string) (int64, error)

CountLFSObjects returns repo's registered LFS object count — an O(1) counter-row read (issue #51; the LFS surface's fast-fail pre-checks call it per upload, so it must not scan the attacker-growable object table).

func (*PG) CountRefs

func (s *PG) CountRefs(ctx context.Context, repo string, limit int64) (int64, error)

CountRefs prices a namespace member's ref fanout for the reclamation budget without materializing the refs (an attacker can mint millions of same-tip refs; loading them just to count would balloon the worker's heap before the budget could reject the repo). The count stops at limit (Codex round 12): even an index-only COUNT walks every matching entry, so an uncapped count on a fanout repo would do budget-scale index work before the budget could reject it — callers pass one past their remaining budget, and a saturated count yields the same over-budget verdict.

func (*PG) CountRepoRows

func (s *PG) CountRepoRows(ctx context.Context, group []string) (int64, error)

CountRepoRows counts the group's refs + packs + lfs_objects rows — the purge's pre-completion verification read: a non-zero count means a writer that resolved the repo before its soft-delete registered rows after the tombstoning phases, and the purge must defer (the next run re-tombstones) instead of deleting the repo's meta row over them.

func (*PG) DFConflicts

func (s *PG) DFConflicts(ctx context.Context, repo string, names []string) (map[string]string, error)

DFConflicts implements the batched authority D/F probe for PostgreSQL: one indexed membership read over the candidates' deduped strict prefixes plus one LATERAL single-row range seek per candidate — two round trips, index work proportional to the candidate set, never to the repository.

func (*PG) DeleteAbsenceCover

func (s *PG) DeleteAbsenceCover(ctx context.Context, key string) error

DeleteAbsenceCover removes one absence-cover reservation row by its synthetic key; deleting an absent row is a no-op. Only the absence reconciler mints and owns rows under AbsenceCoverPrefix, and River's unique periodic jobs serialize its passes fleet-wide, so the delete cannot race another writer of the row; an admission racing the delete sees the same outcome as the row's TTL expiry.

func (*PG) DeleteJobScanState

func (s *PG) DeleteJobScanState(ctx context.Context, job, scope string) error

func (*PG) DropRepoLFSObjects

func (s *PG) DropRepoLFSObjects(ctx context.Context, group []string) (int64, error)

DropRepoLFSObjects deletes the group's LFS registry rows and their object-count rows (LFS rows key by the BASE repo, group[0]; the extra group keys match nothing and cost nothing). The blobs become unregistered and are reaped by the purge's own blob pass — GuardedLFSDelete recreates the count row it locks on, so dropping it here is safe. Returns the lfs_objects rows deleted.

func (*PG) DropRepoRefs

func (s *PG) DropRepoRefs(ctx context.Context, group []string) (int64, error)

DropRepoRefs deletes every ref row of the group's repos (purge phase: refs die FIRST, so no later phase can strand a committed ref). Returns the rows deleted. The InitSchema triggers bump the repo generations as usual; the generation rows themselves fall to PurgeRepoLedger.

func (*PG) FenceRepoPurge

func (s *PG) FenceRepoPurge(ctx context.Context, base string) error

FenceRepoPurge durably marks the BASE repo as purge-fenced — the purge's first destructive-side step, BEFORE any row is tombstoned. Registrations serialize against the fence through lockBaseLedger (PG) or the database single-writer lock (SQLite): once this commits, every later RegisterPacks / RegisterLFSObject for the group fails with ErrRepoPurged, so the purge's later verification-then-completion cannot be raced by a write that settles in between (PR #105 security round) — any straggler row either landed before the fence (re-tombstoned by the resumed purge) or can never land at all. Fences are permanent, like the name burn; idempotent.

func (*PG) GetGeneration

func (s *PG) GetGeneration(ctx context.Context, repo string) (int64, error)

GetGeneration returns repo's mutation generation (0 = never mutated).

func (*PG) GetJobScanState

func (s *PG) GetJobScanState(ctx context.Context, job, scope string) (JobScanState, bool, error)

func (*PG) GetRef

func (s *PG) GetRef(ctx context.Context, repo, name string) (string, error)

GetRef returns the current sha of one ref, or ErrNotFound.

func (*PG) GuardedLFSDelete

func (s *PG) GuardedLFSDelete(ctx context.Context, repo, key string, del func(context.Context) error) (bool, error)

GuardedLFSDelete — see RefStore. The counter row lock is the serialization point: RegisterLFSObject's first-registration path upserts that row inside its transaction, so a registration either committed before this lock (the registry re-read sees it → abort) or blocks until this transaction commits (its writer-verify then republishes the bytes).

func (*PG) HasLFSObject

func (s *PG) HasLFSObject(ctx context.Context, repo, key string) (bool, error)

HasLFSObject reports whether (repo, key) is a registered LFS object.

func (*PG) HasPack

func (s *PG) HasPack(ctx context.Context, repo, key string) (bool, error)

HasPack reports whether repo has a registered pack with blob key.

func (*PG) InitSchema

func (s *PG) InitSchema(ctx context.Context) error

InitSchema creates the ref/pack tables (idempotent). The concurrentIndexes entries (packs_repo_s3_key, refs_repo_name_c) are deliberately NOT part of the transactional DDL: they are built out-of-transaction with CREATE INDEX CONCURRENTLY (issue #24, rationale on buildIndexesLocked). Once every index is valid — every boot after the first — only the transactional DDL runs. While an index is missing or invalid, a SESSION advisory lock is held across BOTH the transactional DDL and the index builds: the DDL takes ACCESS EXCLUSIVE locks on refs/packs (DROP TRIGGER), and a concurrent replica's CONCURRENTLY build both conflicts with them and waits out every older transaction snapshot, so letting the two phases interleave across replicas deadlocks (SQLSTATE 40P01, found via the concurrent-boot test). Skipping the lock when every index is valid is sound because builders re-check state under the lock and never build over a valid index — a valid read means no build can be running.

func (*PG) IsRepoPurged

func (s *PG) IsRepoPurged(ctx context.Context, base string) (bool, error)

IsRepoPurged reports whether the BASE repo is purge-fenced — the sweeper's point read for classifying straggler blobs (a fenced repo's unregistered blobs are provably garbage: registrations reject forever), mirroring HasPack's per-candidate shape.

func (*PG) IsRetainedPack

func (s *PG) IsRetainedPack(ctx context.Context, repo, key string) (bool, error)

IsRetainedPack reports whether (repo, key) is a recorded superseded blob.

func (*PG) JobScanStateAge

func (s *PG) JobScanStateAge(ctx context.Context, job, scope string) (time.Duration, bool, error)

JobScanStateAge — see ScanStateStore.

func (*PG) LFSAdoptionGen

func (s *PG) LFSAdoptionGen(ctx context.Context, repo string) (int64, error)

LFSAdoptionGen — see RefStore (Codex round 49).

func (*PG) LFSMismatchFingerprint

func (s *PG) LFSMismatchFingerprint(ctx context.Context, repo, key string) (string, bool, error)

LFSMismatchFingerprint — see RefStore (#104, Codex round 132).

func (*PG) LFSObjectRegisteredAt

func (s *PG) LFSObjectRegisteredAt(ctx context.Context, repo, key string) (time.Time, int64, bool, error)

LFSObjectRegisteredAt — see RefStore (Codex round 89; bytes since #127).

func (*PG) LFSReposWithObjects

func (s *PG) LFSReposWithObjects(ctx context.Context, after string, limit int) ([]string, error)

LFSReposWithObjects pages base repos that hold at least one registered LFS object (keyset on repo, ascending) — the reclamation scan's work queue.

func (*PG) LapsedAbsenceCoverExists

func (s *PG) LapsedAbsenceCoverExists(ctx context.Context, ttl time.Duration) (bool, error)

LapsedAbsenceCoverExists reports whether any physically present absence-cover row — deterministic incarnation or un-folded legacy nonce — has aged past the ttl: positive evidence that cover protection lapsed while no renewal was running (Codex round 32 on PR #132, security). Folded nonces are excluded: their decay is deliberate retirement, not a lapse. Consulted when the renewal-health watermark is ABSENT — the upgrade/first-enable signature — where arming the fleet-global admission fence without evidence would turn every first pass under this feature into a cross-tenant write outage. Rows GC'd before the probe (admission's per-base expired-row pruning) are a best-effort residual: those bases' writers re-derive their own floors at admission time.

func (*PG) ListAbsenceCovers

func (s *PG) ListAbsenceCovers(ctx context.Context, afterRaw string, limit int, ttl time.Duration) ([]AbsenceCoverRow, error)

ListAbsenceCovers pages the live absence-cover reservation rows, keyset on the raw reservation key ascending (afterRaw "" starts the scan). The rows are the DURABLE tracking state of discovered losses: the renewal phase re-derives its obligations from this listing every pass, so any replica keeps every charged aggregate alive (issue #125). ttl is the liveness window admission consults (PackReservationTTL): rows older than it are EXPIRED — admission no longer counts them, so renewing or migrating one would resurrect a stale reservation (Codex round 6 on PR #132) — and are left to the ledger's expired-row cleanup.

func (*PG) ListFleetLFSObjects

func (s *PG) ListFleetLFSObjects(ctx context.Context, afterRepo, afterKey string, limit int) ([]AbsenceLFSRow, error)

ListFleetLFSObjects pages registered LFS object rows fleet-wide, keyset on the (repo, key) primary key ascending — the absence reconciler's LFS work queue.

func (*PG) ListFleetPacks

func (s *PG) ListFleetPacks(ctx context.Context, afterRepo, afterID string, limit int) ([]AbsencePackRow, error)

ListFleetPacks pages registered pack rows fleet-wide, keyset on the (repo, id) primary key ascending — the absence reconciler's pack work queue.

func (*PG) ListJobScanStateScopes

func (s *PG) ListJobScanStateScopes(ctx context.Context, job, prefix, after string, limit int) ([]string, error)

ListJobScanStateScopes — see ScanStateStore.

func (*PG) ListPacks

func (s *PG) ListPacks(ctx context.Context, repo string) ([]Pack, error)

ListPacks returns all registered packs for repo, oldest first.

func (*PG) ListRefs

func (s *PG) ListRefs(ctx context.Context, repo string) (map[string]string, error)

ListRefs returns all refs for repo as name -> sha.

func (*PG) ListRefsAfter

func (s *PG) ListRefsAfter(ctx context.Context, repo, after string, limit int) ([]RefEntry, error)

ListRefsAfter returns up to limit refs for repo above the after cursor in ascending byte order (issue #119). COLLATE "C" pins byte-wise comparison regardless of the database's default collation — the order git's ref iteration uses and the only one hydration's merge-join can trust; the refs_repo_name_c index (see concurrentIndexes) serves the keyset scan so a page never sorts the repo's whole remaining ref set.

func (*PG) ListRefsWithMeta

func (s *PG) ListRefsWithMeta(ctx context.Context, repo string) ([]RefInfo, error)

ListRefsWithMeta returns all refs for repo sorted by name ascending.

func (*PG) ListRepoLFSObjects

func (s *PG) ListRepoLFSObjects(ctx context.Context, repo, after string, limit int) ([]LFSObjectRow, error)

ListRepoLFSObjects pages repo's registered LFS object rows (keyset on key, ascending).

func (*PG) MarkLFSObjectsUnreferenced

func (s *PG) MarkLFSObjectsUnreferenced(ctx context.Context, repo string, keys []string) (int64, error)

MarkLFSObjectsUnreferenced stamps unreferenced_since = now() on the given rows where no mark exists yet, and reports how many rows it stamped. An existing mark keeps its original clock — the grace window measures how long the object has CONTINUOUSLY been observed unreferenced.

func (*PG) PackRegisteredBytes

func (s *PG) PackRegisteredBytes(ctx context.Context, repo, key string) (int64, bool, error)

PackRegisteredBytes returns the registered pack's recorded size — see RefStore (0 = pre-#101 row, size unknown).

func (*PG) ParkRepoJobScanState

func (s *PG) ParkRepoJobScanState(ctx context.Context, job, scope, payload string, generation int64, base string) error

func (*PG) PurgeRepoLedger

func (s *PG) PurgeRepoLedger(ctx context.Context, group []string) (int64, error)

PurgeRepoLedger removes the group's remaining accounting and state rows (reservations, retained tombstones whose blobs are already gone, usage counters, LFS counts, adoption generations, repo generations). Runs strictly AFTER the blob pass verified the group's registry rows are gone — the retained rows must outlive their blobs, and the usage counter row is the accounting cleanup (no per-row debit: the whole unit dies). Returns the total rows deleted.

func (*PG) PutJobScanState

func (s *PG) PutJobScanState(ctx context.Context, job, scope, payload string, generation int64) error

func (*PG) QuotaLedgerState

func (s *PG) QuotaLedgerState(ctx context.Context, group []string, ttl time.Duration, excludeKeys ...string) (PackQuotaLedgerState, error)

QuotaLedgerState reads the group's ledger state (see PackQuotaLedgerState).

func (*PG) ReclaimRetainedPack

func (s *PG) ReclaimRetainedPack(ctx context.Context, repo, key string) (int64, error)

ReclaimRetainedPack removes the retained row and debits the usage counter.

func (*PG) ReconcileLegacyLFSObjects

func (s *PG) ReconcileLegacyLFSObjects(ctx context.Context) (int64, bool, error)

ReconcileLegacyLFSObjects — see RefStore.

func (*PG) RecordLFSMismatch

func (s *PG) RecordLFSMismatch(ctx context.Context, repo, key, fingerprint string) error

RecordLFSMismatch — see RefStore (#104, Codex round 132).

func (*PG) RecordLFSReapComplete

func (s *PG) RecordLFSReapComplete(ctx context.Context, epoch int64, legacyArmed bool) error

RecordLFSReapComplete — see ScanStateStore.

func (*PG) RefValues

func (s *PG) RefValues(ctx context.Context, repo string, names []string) (map[string]string, error)

RefValues implements the batched exact-value probe for PostgreSQL: indexed membership reads in bounded dfProbeChunk-sized batches (Codex round-21 P2 on #173: the push cap admits pushes of hundreds of thousands of commands, and encoding ALL their names as one array parameter sends one correspondingly oversized query to the shared database while the push holds those refs — the same reason the D/F probe chunks).

func (*PG) RegisterAdoptedLFSObject

func (s *PG) RegisterAdoptedLFSObject(ctx context.Context, repo, key string, bytes int64) error

RegisterAdoptedLFSObject — see RefStore (Codex round 51): the adoption generation bump commits atomically with the registration.

func (*PG) RegisterLFSObject

func (s *PG) RegisterLFSObject(ctx context.Context, repo, key string, bytes int64) error

RegisterLFSObject records one verified LFS object and settles its quota reservation (issue #51) — see RefStore. Mirrors RegisterPacks' ledger discipline: the one-transaction insert + reservation consumption + credit is what lets ReservePackBytes read a single snapshot without locking against LFS settlement. The object-count cap is enforced HERE (Codex round 6) via the maintained counter row (Codex round 10: a COUNT(*) under lock made building to the cap quadratic): the counter upsert's row lock serializes same-repo registrations, so the RETURNING value is the authoritative post-insert population. Returns ErrLFSObjectCapExceeded (and commits nothing — insert and increment roll back together) when the NEW row would exceed the cap.

func (*PG) RegisterPacks

func (s *PG) RegisterPacks(ctx context.Context, repo string, packs []Pack) error

RegisterPacks inserts pack rows in ONE transaction (crash-ordering: this runs strictly after the blob upload succeeded). Idempotent on retry: a re-pushed identical pack has the same id.

The same transaction settles the quota ledger (issues #29/#31): each pack's admission reservation is consumed (deleted by KEY, any owner — the bytes become counter-accounted below, and identical keys are identical bytes, so a same-key reservation another push re-acquired would double-count if kept), and each NEWLY inserted row credits its size to the repo's usage counter unless the key was retained (superseded bytes were counted at their original registration and never debited — re-registering just makes them live again). Atomicity here is what lets ReservePackBytes read one snapshot without locking against registration.

func (*PG) ReleaseAdoptionClaim

func (s *PG) ReleaseAdoptionClaim(ctx context.Context, key, owner string) error

ReleaseAdoptionClaim — see RefStore.

func (*PG) RenewAbsenceCovers

func (s *PG) RenewAbsenceCovers(ctx context.Context, ttl time.Duration) (int64, error)

RenewAbsenceCovers re-stamps created_at on every LIVE absence-cover row the reconciler is responsible for keeping alive, in DB-clock statements (Codex round 9 on PR #132): cover protection must never depend on the keyset renewal listing completing within the TTL — a large population or degraded ledger latency turns that listing into a multi-pass round robin, and a row aging past the TTL mid-cycle would be excluded as expired and silently never renewed. The strict TTL predicate is the round-6 invariant: already-expired rows stay dead. Two row classes are touched: deterministic incarnation rows (`/<class>-e<19 digits>` tail), and legacy nonce rows NOT yet marked folded (Codex round 15: an un-folded nonce minted just behind the listing cursor must not lapse before its first visit). A proven fold marks the nonce (MarkAbsenceCoverFolded) which removes it from the touch so it decays by TTL — and any old replica re-charging its nonce overwrites owner, un-marking it, exactly matching "a re-charge is a new reliance". Returns the number of rows re-stamped.

func (*PG) RenewAdoptionClaim

func (s *PG) RenewAdoptionClaim(ctx context.Context, key, owner string) (bool, error)

RenewAdoptionClaim — see RefStore.

func (*PG) ReplacePacks

func (s *PG) ReplacePacks(ctx context.Context, repo string, old []Pack, newPack Pack) (won bool, err error)

ReplacePacks installs a repack fold in one transaction (see RefStore). The insert tolerates an existing (repo, id) row — pack ids derive from pack content, so a same-id row registered concurrently is byte-identical (RegisterPacks invariant) and keeping it is equivalent to inserting.

func (*PG) ReservePackBytes

func (s *PG) ReservePackBytes(ctx context.Context, group []string, packs []Pack, owner string, usageFloor, floorGen, quotaBytes int64, ttl time.Duration) (ReserveDecision, error)

ReservePackBytes implements the admission check documented atop this file on Postgres.

func (*PG) RetirePackAttempts

func (s *PG) RetirePackAttempts(ctx context.Context, group []string, gens map[string]string) error

RetirePackAttempts — see RefStore. Serialized with ReservePackBytes per base repo (same advisory lock class): a retire must not interleave with a concurrent same-key refresh or the decrement could race the increment.

func (*PG) RetireRepoPacks

func (s *PG) RetireRepoPacks(ctx context.Context, group []string) (int64, error)

RetireRepoPacks tombstones every registered pack of the group's repos: in one transaction each packs row becomes a pack_retained row and is deleted. bytes is 0 for purge tombstones (packs rows store no size) — these rows are the sweeper-reapable CLASSIFICATION tombstone (issue #31 retained class, provably DB-absent), not accounting: the usage counter stays credited until PurgeRepoLedger drops it, over-counting at worst (the safe direction; the group is unreachable anyway). An existing retained row (a real repack fold's, carrying real bytes) is kept as-is so its eventual reclaim still debits correctly. Returns the packs rows retired.

func (*PG) RetireUnreferencedLFSObject

func (s *PG) RetireUnreferencedLFSObject(ctx context.Context, repo, key string, pinnedGens map[string]int64, grace, reapMaxAge time.Duration) (bool, error)

RetireUnreferencedLFSObject deletes one registered LFS object row whose reclamation mark has survived a full grace window — the issue #73 GC verdict. The object-count and usage-byte refund is DEFERRED (Codex round 33 on PR #132): it parks in a per-epoch refund bucket and settles through SettleLFSRefunds only after a destructive sweep whose epoch strictly exceeds this verdict's — a sweep that provably listed blobs after this deletion, so the orphan was reaped (or the key re-registered with fresh accounting) before the capacity is handed back. It is the same guard class as GuardedLFSDelete: the whole verdict runs under the base-ledger lock RegisterLFSObject serializes on, and the delete predicate re-checks the mark IN the transaction, so an object re-uploaded or touched mid-scan (mark cleared, or a fresh mark younger than grace) is spared.

pinnedGens fences the verdict against refs that moved since the reachability scan pinned them (repo generation per namespace-group member, 0 for never-mutated): each current generation is re-read under FOR UPDATE and any drift skips the retirement — a pointer file pushed mid-scan lives in a bumped generation by definition. The FOR UPDATE also forces a concurrent push's deferred generation-bump trigger to serialize against the verdict commit. Residual (documented in ADR-42): a push whose commit began before the recheck locked the rows can still land its bump after the verdict commits; only a raw-git pointer push referencing a grace-expired marked object inside that window is exposed, because git-lfs clients always touch via the batch API first.

reapMaxAge is the observed-coupling bound (issue #120): the verdict re-reads the sweeper's "last completed destructive LFS pass" watermark (job_scan_state, ScanJobSweep/ScanScopeLFSDestructiveReap) INSIDE the transaction and refuses with ErrReapWatermarkStale when the watermark is absent or older than the bound. Refunding an object-cap slot is only sound while the physical reap is demonstrably completing — a configured sweep cadence says nothing when deletes fail, a pass outruns the reclamation cycle, or the versioning guard demotes sweeps to list-only mid-run. The comparison runs on the database clock at both ends (the watermark is stamped with now()), so replica clock skew cannot widen the window.

retired=false means the verdict lost — fence drift, purge fence, cleared or young mark, or a vanished row — and the caller moves on; the next scan re-derives everything from durable state.

func (*PG) SettleLFSRefunds

func (s *PG) SettleLFSRefunds(ctx context.Context, repo string) (SettledLFSRefunds, error)

SettleLFSRefunds refunds this base's pending retirement buckets that a completed destructive sweep now covers (Codex round 33 on PR #132): a bucket's canonical part settles under any completion whose epoch strictly exceeds the bucket's, its legacy part only under a legacy-armed completion's epoch (canonical-only sweeps spare legacy orphans). The count/usage decrements and the adoption-generation bump (floor-cache invalidation, the ADR-41 refund pattern) run atomically with the bucket rewrite under the base-ledger lock. Purge-fenced repos settle nothing: the purge drops the whole ledger, buckets included.

func (*PG) SweepExpiredPackReservations

func (s *PG) SweepExpiredPackReservations(ctx context.Context, olderThan time.Duration) (int64, error)

SweepExpiredPackReservations prunes reservation rows at least olderThan old. Deleting an expired UN-FOLDED absence-cover row destroys the physical lapse evidence the reconciler's first-health-stamp probe arms on — the scheduled sweep is a second consumer of that evidence beside ReservePackBytes' admission-time GC (Codex rounds 33+34 on PR #132) — so the statement records the durable lapse-evidence marker atomically. The sweeper passes the admission TTL as olderThan, so every doomed cover row is exactly a lapsed cover; folded rows decay deliberately and never mark.

func (*PG) TouchLFSObject

func (s *PG) TouchLFSObject(ctx context.Context, repo, key string) (bool, error)

TouchLFSObject — see RefStore.

func (*PG) TryAdoptionClaim

func (s *PG) TryAdoptionClaim(ctx context.Context, repo, key string) (bool, string, error)

TryAdoptionClaim — see RefStore.

type Pack

type Pack struct {
	ID  string
	Key string
	// Size is the pack's byte size, feeding the quota ledger (issues
	// #29/#31): ReservePackBytes admits Size against the quota,
	// RegisterPacks credits it to the repo's usage counter, and
	// ReplacePacks records superseded rows' sizes as retained bytes. It is
	// accounting input, not a persisted row column — ListPacks returns it
	// zero. Writers that know the pack file must set it; a zero Size
	// registers fine but contributes nothing to the ledger (the blob
	// listing floor still covers such bytes conservatively).
	Size int64
	// Adopted marks a pack whose BLOB already existed before this
	// registration (a crash-window retry re-registering a published-but-
	// rowless pack — the pack analog of LFS adoption, Codex round 53):
	// physical listings may already contain its bytes, so RegisterPacks
	// bumps the shared floor generation IN ITS REGISTRATION TRANSACTION
	// when it credits a newly inserted Adopted pack, invalidating every
	// replica's cached uncounted floor. Fresh uploads leave it false —
	// their bytes are in no cached scan, and bumping per push would
	// defeat the floor cache under write load. Accounting input like
	// Size, never persisted.
	Adopted bool
}

Pack is one registered pack for a repo. Key is the blob-store object key (blob.PackKey); the backing SQL column stays named s3_key in every backend so the pack SQL is byte-comparable across them (no PG migration).

type PackQuotaLedgerState

type PackQuotaLedgerState struct {
	UsageBytes    int64 // SUM(pack_usage) over the group
	ReservedBytes int64 // SUM(active reservations) for the base repo
	RetainedBytes int64 // SUM(pack_retained) over the group
	// RenewalDegraded mirrors ReservePackBytes' cover-renewal degraded
	// fence (the absence reconciler's admission-fence marker): while
	// set, every finite quota-gated admission WILL deny, so advisory
	// preflights can fail hopeless work before spending on it (Codex
	// round 36 on PR #132, security — the adoption preflight hashed
	// whole legacy objects only to be denied post-hash). Advisory
	// only: the authoritative check stays inside ReservePackBytes
	// under the fence's shared advisory lock.
	RenewalDegraded bool
}

PackQuotaLedgerState is a point-in-time read of one accounting group's ledger, for tests, oracles, and operator inspection — NOT part of the admission path (ReservePackBytes reads its own single snapshot).

type ReapWatermark

type ReapWatermark struct {
	Epoch  int64 `json:"epoch"`
	Legacy bool  `json:"legacy,omitempty"`
}

ReapWatermark is the destructive-reap completion watermark's JSON payload. Epoch is the pass's ScanScopeLFSReapEpoch value, stamped at sweep START — refund settlement requires a completion whose epoch strictly exceeds a retirement's recorded epoch (Codex round 33 on PR #132). Legacy certifies the sweep also had the LEGACY LFS delete class armed (OPENSTORAGE_SWEEP_DELETE_LFS_LEGACY): legacy-spelled registry rows may only be retired, and their refunds settled, under legacy-armed certification (Codex round 17) — with the legacy class off, the sweeper deliberately spares legacy-spelled orphan blobs, so refunding their slots would grow storage past the cap.

func DecodeReapWatermark

func DecodeReapWatermark(payload string) ReapWatermark

DecodeReapWatermark parses a completion watermark payload. Undecodable payloads (never written by this binary) fail toward zero values — canonical-only certification at epoch 0, which defers every settlement and legacy retirement: strictly the conservative direction.

type RefEntry

type RefEntry struct {
	Name string
	SHA  string
}

RefEntry is one (name, sha) ref row — the bounded-enumeration unit (issue #119): pages of these replace whole-repo ref materialization on the hydration path.

type RefInfo

type RefInfo struct {
	Name      string
	SHA       string
	CreatedAt time.Time // UTC
}

RefInfo is one ref row with its metadata. CreatedAt is the row's creation time (docs/reference/http-api.md branches[].created_at): set on first insert, preserved across SHA updates, and reset when a ref is deleted and recreated.

type RefStore

type RefStore interface {
	// ListRefs returns all refs for repo as name -> sha.
	ListRefs(ctx context.Context, repo string) (map[string]string, error)
	// ListRefsWithMeta returns all refs for repo sorted by name ascending,
	// including creation timestamps (the read API's branches/tags source).
	ListRefsWithMeta(ctx context.Context, repo string) ([]RefInfo, error)
	// ListRefsAfter returns up to limit refs for repo whose name is
	// byte-wise greater than after, in ascending BYTE order of name (the
	// order git's own ref iteration uses, so hydration can merge-join
	// pages against for-each-ref output — issue #119). A page shorter
	// than limit means the scan is complete. Each call is one consistent
	// snapshot; cross-page consistency is the CALLER's contract:
	// githttp.Hydrate binds a paged scan to the repo generation read
	// before it, so any concurrent ref mutation bumps the generation,
	// invalidates the marker the sync publishes, and forces the next
	// hydrate down the full path — healing whatever a torn scan
	// mis-synced.
	ListRefsAfter(ctx context.Context, repo, after string, limit int) ([]RefEntry, error)
	// GetRef returns the current sha of one ref (a point-read: the
	// IdempotentCAS recovery recheck must not scan the repo's whole ref
	// table). Returns ErrNotFound when the ref does not exist.
	GetRef(ctx context.Context, repo, name string) (string, error)
	// CASRef performs a compare-and-swap ref update (old/new = ZeroSHA for
	// create/delete). Returns won=false (no error) when the precondition
	// failed, i.e. someone else won the race.
	CASRef(ctx context.Context, repo, name, old, new string) (won bool, err error)

	// CASRefGen is CASRef additionally reporting the repository's
	// post-CAS mutation generation ATOMICALLY with the win: gen is the
	// exact value GetGeneration observes once this swap's commit — and
	// no later mutation — is visible (issue #186 finding 5: push-hold
	// generation-range coalescing needs each won hook CAS attributed to
	// the exact generation it produced; a post-hoc GetGeneration could
	// blend in a concurrent writer's bump and over-claim). gen > 0 only
	// on won; a won outcome with gen == 0 means the attribution is
	// unknown (a decorator recovered an ambiguous outcome by point-read,
	// where the committed generation is unknowable after the fact) and
	// callers must treat the mutation as unattributed — the fail-safe
	// direction for range coalescing.
	CASRefGen(ctx context.Context, repo, name, old, new string) (won bool, gen int64, err error)
	// RegisterPacks records packs for repo atomically; idempotent on retry.
	RegisterPacks(ctx context.Context, repo string, packs []Pack) error
	// ListPacks returns all registered packs for repo, oldest first.
	ListPacks(ctx context.Context, repo string) ([]Pack, error)
	// HasPack reports whether repo has a registered pack with the given
	// blob key (a point read: the orphan-pack sweeper classifies each
	// listed blob individually so its memory never scales with a repo's
	// registered-pack count, which pre-policy pushes can grow).
	HasPack(ctx context.Context, repo, key string) (bool, error)
	// ReplacePacks atomically installs a repack fold (issue #1): in ONE
	// transaction it deletes exactly the old rows and inserts newPack.
	// The delete doubles as the CAS precondition — every old row must
	// still exist — so a fold racing another fold loses cleanly:
	// won=false, no error, zero DB changes. Rows registered concurrently
	// by new pushes are NOT part of the precondition and survive the
	// swap; the caller guarantees newPack covers every object of the old
	// rows' packs (never a reachability-trimmed subset), so surviving
	// rows and refs stay consistent regardless of interleaving. Callers
	// must have published newPack's bytes to blob storage BEFORE calling
	// (publish-before-flip); on won=false they own deleting that blob.
	// A won swap bumps the repo generation via the InitSchema triggers.
	ReplacePacks(ctx context.Context, repo string, old []Pack, newPack Pack) (won bool, err error)
	// ReservePackBytes atomically admits the given incoming packs against
	// the per-repo pack-bytes quota (issue #29), inserting one durable
	// reservation row per pack key on success. See ledger.go for the
	// admission formula and serialization discipline.
	// The floorGen argument is the shared floor generation the caller's
	// usageFloor was validated against (ReserveNoFloorGen skips the
	// transactional recheck — cover reservations only, Codex round 60).
	ReservePackBytes(ctx context.Context, group []string, packs []Pack, owner string, usageFloor, floorGen, quotaBytes int64, ttl time.Duration) (ReserveDecision, error)
	// QuotaLedgerState reads one accounting group's quota ledger in one
	// snapshot (usage counter sum, active reservations, retained bytes).
	// The LFS batch surface reads it for its ADVISORY fast-fail admission
	// (issue #51 — batches insert no rows; the PUT's transactional
	// admission is authoritative); also tests, oracles, and operator
	// inspection.
	// excludeKeys omits those keys' active reservations from the
	// ReservedBytes sum (Codex round 24), mirroring ReservePackBytes's
	// same-key exclusion: an admission check for specific keys must not
	// double-count a reservation those keys already hold (a failed
	// verified upload's settle-or-TTL row would otherwise 507 its own
	// retry until the TTL).
	QuotaLedgerState(ctx context.Context, group []string, ttl time.Duration, excludeKeys ...string) (PackQuotaLedgerState, error)
	// RetirePackAttempts retires one same-key reservation attempt per
	// entry — the PROVEN-clean-failure cleanup (Codex rounds 30+33 on
	// #51). Every reserve/refresh increments the row's attempt count, so
	// a failed writer retires exactly its own attempt: the row is deleted
	// only when no attempts remain, never out from under a live same-key
	// writer (the round-22 unsoundness of owner-scoped deletes). gens
	// maps each key to the row-incarnation gen ReserveDecision.Gens
	// returned: a replacement row minted after this caller's expired and
	// was pruned carries a different gen and is untouched (round 33).
	// Callers retire only with EVIDENCE of non-publication (the blob
	// provably absent) — ambiguous outcomes leave the row to settlement
	// or TTL, and crashes never retire. Absent or re-minted keys are
	// no-ops. Mixed fleets are fenced structurally: this binary's rows
	// live under an era-spelled key (reservKeySuffix) no v0.1.0
	// statement addresses, and retirement touches only that spelling —
	// a legacy same-key twin row is left to its own binary's lifecycle.
	RetirePackAttempts(ctx context.Context, group []string, gens map[string]string) error
	// SweepExpiredPackReservations deletes reservation rows at least
	// olderThan old (crash-swept leak reclamation, issue #29; the orphan
	// sweeper calls it on its cadence). olderThan=0 expires everything —
	// a test/recovery escape hatch. Returns the number of rows deleted.
	SweepExpiredPackReservations(ctx context.Context, olderThan time.Duration) (int64, error)
	// ReconcileLegacyLFSObjects deletes legacy-spelled lfs_objects rows
	// whose canonical twin exists, refunding the count and bytes each
	// double-charged, and returns how many it reconciled (ADR-41, Codex
	// round 79): a rolling upgrade can commit a v0.2.0 legacy
	// registration alongside the canonical row, after which the fast
	// paths never call registration for the object again — without this
	// janitor the twin would overstate usage and the object cap forever.
	ReconcileLegacyLFSObjects(ctx context.Context) (reconciled int64, partial bool, err error)
	// IsRetainedPack reports whether (repo, key) is recorded as a
	// superseded-by-repack blob (issue #31): its row was folded away by
	// ReplacePacks but its bytes still occupy the pack prefix. A point
	// read, mirroring HasPack, for the sweeper's per-object classification.
	IsRetainedPack(ctx context.Context, repo, key string) (bool, error)
	// ReclaimRetainedPack removes (repo, key) from the retained set and
	// subtracts its recorded bytes from the repo's usage counter, returning
	// the bytes reclaimed (0 when no retained row exists — idempotent).
	// Callers invoke it exactly when the superseded blob's bytes leave the
	// pack prefix (the sweeper's safeDelete, or its heal pass discovering
	// the blob already gone).
	ReclaimRetainedPack(ctx context.Context, repo, key string) (int64, error)
	// RegisterLFSObject records one verified LFS object for repo (always
	// the BASE RepoID — namespaces share the LFS space, ADR-35) strictly
	// after its bytes are durable in blob storage, mirroring RegisterPacks
	// (issue #51). In ONE transaction it inserts the row (idempotent on
	// retry), consumes any admission reservation for key (deleted by KEY,
	// any owner — the bytes become counter-accounted), and credits a NEWLY
	// inserted row's bytes to the repo's usage counter. bytes may be zero
	// (the empty object registers without ledger effect).
	RegisterLFSObject(ctx context.Context, repo, key string, bytes int64) error
	// RegisterAdoptedLFSObject is RegisterLFSObject for ADOPTED objects
	// (pre-registry blobs whose bytes physical listings already contain):
	// the same transaction also bumps the adoption generation (Codex
	// round 51 — a separate best-effort bump could time out after the
	// registration committed, leaving remote floors double-counting the
	// adopted bytes with no retry, because adoption short-circuits once
	// the row exists). A re-registration of an existing row commits
	// nothing and bumps nothing.
	RegisterAdoptedLFSObject(ctx context.Context, repo, key string, bytes int64) error
	// RecordLFSMismatch persists that a full content verification proved
	// the legacy blob at key (fingerprint at hash time) wrong for its
	// OID (#104, Codex round 132): the marker must survive restarts and
	// be visible fleet-wide, or its loss makes a registered object's
	// corrupt legacy spelling implicitly clean. Upsert; purge-fenced
	// repos silently no-op like every row-creating write.
	RecordLFSMismatch(ctx context.Context, repo, key, fingerprint string) error
	// LFSMismatchFingerprint returns key's pending mismatch marker.
	LFSMismatchFingerprint(ctx context.Context, repo, key string) (string, bool, error)
	// ClearLFSMismatch removes key's marker after a decisive verdict.
	ClearLFSMismatch(ctx context.Context, repo, key string) error
	// HasLFSObject reports whether (repo, key) is a registered LFS object
	// (a point read mirroring HasPack, for the sweeper's per-object
	// classification of the lfs/ prefix).
	HasLFSObject(ctx context.Context, repo, key string) (bool, error)
	// TouchLFSObject clears (repo, key)'s reclamation mark (issue #73)
	// and reports whether the row still exists. It is the liveness
	// handshake behind every zero-growth "already have it" answer: a
	// client that just learned the object is stored may reference it
	// from a pointer file it pushes NEXT, so the touch guarantees at
	// least one full reclamation grace window before retirement can
	// reconsider the row. touched=false means the row is gone — the
	// caller must fall through to its upload path instead of promising
	// an object the reclaimer just retired. The single-statement UPDATE
	// serializes against RetireUnreferencedLFSObject's guarded delete on
	// the row itself: whichever commits first wins, and the loser
	// observes it (the retire recheck sees the cleared mark; the touch
	// sees zero rows).
	TouchLFSObject(ctx context.Context, repo, key string) (touched bool, err error)
	// LFSObjectRegisteredAt returns (repo, key)'s registration time — the
	// row's created_at, which the in-place legacy→canonical migration
	// REFRESHES (Codex round 89): downloads presign the still-rowless
	// LEGACY spelling right up to the moment the canonical row exists, so
	// the row's clock is the only trustworthy "no more legacy URLs are
	// being minted" anchor. Blob timestamps are not: a rowless canonical
	// copy can be hours old (publication or adoption copied it and only
	// the registration failed) while its late registration retry
	// verifies it in place without rewriting it. bytes is the row's
	// registered size, the sweeper's completeness yardstick (issue
	// #127): the reap gate must spare a retained legacy source whose
	// canonical copy is SHORTER than registered, not merely present. ok
	// is false when no row exists.
	LFSObjectRegisteredAt(ctx context.Context, repo, key string) (at time.Time, bytes int64, ok bool, err error)
	// PackRegisteredBytes is LFSObjectRegisteredAt's size half for the
	// pack registry: the packs.bytes column for (repo, key), the
	// quarantine heal's completeness yardstick (issue #129). bytes is 0
	// for rows registered before the column existed (issue #101) — the
	// caller must treat 0 as size-UNKNOWN, never as "complete at any
	// length". ok is false when no row exists.
	PackRegisteredBytes(ctx context.Context, repo, key string) (bytes int64, ok bool, err error)
	// BumpLFSAdoptionGen increments repo's shared adoption generation
	// (Codex round 49 on #51): adopting a pre-registry LFS object moves
	// bytes that physical listings ALREADY contain into the usage
	// counter, so every replica's cached uncounted floor scanned before
	// the adoption would double-count them (floor + counter) until
	// expiry. Floor caches snapshot this generation at scan start and
	// reject entries whose generation moved — the ref store is the only
	// state all replicas and hook processes share.
	BumpLFSAdoptionGen(ctx context.Context, repo string) error
	// LFSAdoptionGen returns repo's adoption generation (0 before any
	// adoption) — a point read on the floor-serve path.
	LFSAdoptionGen(ctx context.Context, repo string) (int64, error)
	// CountLFSObjects returns repo's registered LFS object count — the
	// admission-time read behind the per-repo object-count cap (issue
	// #51): bytes bound row count at nothing, and registered objects are
	// permanent, so their population needs its own bound.
	CountLFSObjects(ctx context.Context, repo string) (int64, error)
	// GuardedLFSDelete runs del (a blob deletion for a stored key whose
	// bytes failed content verification) serialized against concurrent
	// LFS registrations of the same repo (issue #51, Codex round 18):
	// within one transaction it takes the lock RegisterLFSObject's
	// counter update takes, re-reads the registry, and only if key is
	// still unregistered runs del. A concurrent verified upload therefore
	// either registered first (deleted=false, del never runs) or blocks
	// until this commits — its post-registration blob.EnsurePublished
	// then republishes its bytes, so no interleaving can leave a
	// registered row pointing at deleted bytes.
	GuardedLFSDelete(ctx context.Context, repo, key string, del func(context.Context) error) (deleted bool, err error)
	// TryAdoptionClaim atomically claims exclusive fleet-wide ownership
	// of one legacy-object adoption (Codex security round 73): the
	// verification hash downloads the whole object, and without a shared
	// claim every replica races the same hash — tiny write-scoped
	// batches fanned across the fleet multiply backend reads by fleet
	// size. claimed=false means a live claim exists (another replica is
	// hashing; the caller denies transiently and the client retries —
	// usually landing on the winner's committed registration). A claim
	// older than AdoptionClaimTTL is expired and taken over. The
	// returned owner token fences every later mutation (Codex round 74):
	// a worker whose claim was taken over must not be able to release or
	// renew the successor's lease. Claims live in pack_reservations
	// under a suffixed key that no settlement, retirement, or
	// admission-GC path touches; only expiry-takeover, owner-fenced
	// release, and the claim-aware sweeper prune remove them.
	TryAdoptionClaim(ctx context.Context, repo, key string) (claimed bool, owner string, err error)
	// RenewAdoptionClaim restarts the claim's TTL clock when key's live
	// claim still belongs to owner (Codex round 74 — verification is
	// size-scaled and can outlive any fixed TTL). renewed=false means
	// the lease was lost (expired and taken over, or released): the
	// caller must stop hashing.
	RenewAdoptionClaim(ctx context.Context, key, owner string) (renewed bool, err error)
	// ReleaseAdoptionClaim frees key's adoption claim if it still
	// belongs to owner (best-effort: a crashed winner's claim expires
	// via AdoptionClaimTTL instead).
	ReleaseAdoptionClaim(ctx context.Context, key, owner string) error
	// GetGeneration returns repo's mutation generation: 0 for a repo no
	// mutation ever touched, strictly positive and monotonically increasing
	// afterwards. It is an invalidation token over committed row changes,
	// not a call counter: schema-level triggers installed by InitSchema
	// bump it inside the same transaction as ANY committed change to the
	// repo's refs or packs rows — won CAS swaps, pack registrations
	// (per changed row; a no-op idempotent retry changes nothing and bumps
	// nothing), future pack rewrites (issue #1's ReplacePacks), and even
	// writers running binaries that predate the counter. An unchanged value
	// therefore proves the repo's packs and refs are unchanged; the hydrate
	// short-circuit relies on exactly that (issue #4).
	GetGeneration(ctx context.Context, repo string) (int64, error)
}

RefStore is the ref-authority contract: refs only ever change through CASRef, and packs are registered strictly after their bytes are durable in blob storage.

type ReserveDecision

type ReserveDecision struct {
	// OK reports whether the reservation was admitted and inserted.
	OK bool
	// FloorStale denies because the caller's usage floor was validated
	// against a shared floor generation that moved before this
	// transaction (Codex round 60, security): a restore, adoption, or
	// retained-bytes debit changed what the floor measures between the
	// caller's generation check and this reserve — the classic
	// check-then-act window no post-hoc invalidation can reach. The
	// caller must re-derive a fresh floor (with its generation) and
	// retry; admitting against the stale integer could durably exceed
	// quota.
	FloorStale bool
	// UsedBytes is the usage the decision charged: the max of the ledger
	// counter and the caller's listing floor.
	UsedBytes int64
	// ReservedBytes is the active in-flight reservations charged on top of
	// UsedBytes (other pushes' keys only).
	ReservedBytes int64
	// Gens maps each reserved key to its row-incarnation gen (Codex round
	// 33): minted at INSERT, preserved across same-key refreshes.
	// RetirePackAttempts requires it, so a stale writer whose row expired
	// and was replaced can never retire the replacement. Within era rows
	// (reservKeySuffix) every refresh increments attempts, so gen plus the
	// attempts refcount fully identify a retirable row — round 33's extra
	// created_at comparison was dropped with the era fence (Codex round
	// 35: it retained reverse-order zero-attempt rows to the TTL, letting
	// failed tiny same-key upload pairs wedge the row cap). Set only when
	// OK.
	Gens map[string]string
	// RenewalDegraded denies because the absence reconciler's cover
	// renewal is in a degraded window (Codex round 26 on PR #132,
	// security): the TTL'd compensation covers this admission counts in
	// active_reservations may have lapsed un-renewed, deflating the
	// quota floor, so finite quota-gated admissions fail closed until a
	// complete recovery rotation re-covers every registered-but-missing
	// byte. Unbounded internal reservations (restores, covers — quota
	// math.MaxInt64) are exempt so recovery itself can proceed.
	// Transient from the client's perspective: retry later.
	RenewalDegraded bool
	// KeyCapExceeded reports that the denial was the per-repo cap on
	// ACTIVE reservation ROWS, not byte arithmetic (Codex security round
	// on #51): declared bytes bound row count at nothing — 1-byte LFS
	// batch entries could mint durable rows ~quota times over — so the
	// row population needs its own bound. Transient: rows expire with the
	// TTL, so callers should tell the client to retry later.
	KeyCapExceeded bool
}

ReserveDecision reports one admission decision. UsedBytes and ReservedBytes carry the arithmetic the caller folds into its denial message (the client-facing sideband line / 403 detail).

type RestoreCoverStore

type RestoreCoverStore interface {
	ReservePackBytes(ctx context.Context, group []string, packs []Pack, owner string, usageFloor, floorGen, quotaBytes int64, ttl time.Duration) (ReserveDecision, error)
}

RestoreCoverStore is the single-method ledger slice ReserveRestoreCover needs, so every restore path — server handlers, the pre-receive hook, and the sweeper's narrow PackRegistry — can share the one cover implementation.

type S3WAL

type S3WAL struct {
	// contains filtered or unexported fields
}

S3WAL implements Store (plus the purge/reclaim/absence job seams) over a conditional object store.

func NewS3WAL

func NewS3WAL(cs s3wal.ConditionalStore, now func() time.Time) *S3WAL

NewS3WAL wraps an injectable conditional store, probe-less: the constructor the DST harness (internal/simtest) builds drivers from. Production serving goes through ref.Open, which wires the real probe. now overrides the driver's local clock source (nil keeps time.Now); simulations pass their logical clock so manifest stamps and TTL verdicts are a pure function of the seed, never wall time.

func (*S3WAL) AbsenceCoverHead

func (s *S3WAL) AbsenceCoverHead(ctx context.Context, prefix string, ttl time.Duration) (AbsenceCoverRow, bool, error)

AbsenceCoverHead — see the PG twin. The class prefix always spells <AbsenceCoverPrefix><base>/<class-tail>, so the head lookup is one manifest read, never a fleet walk.

func (*S3WAL) ArmAbsenceAdmissionFence

func (s *S3WAL) ArmAbsenceAdmissionFence(ctx context.Context, payload string) error

ArmAbsenceAdmissionFence — see the PG twin for the ordering contract. The write side of absenceFenceMu mirrors PG's exclusive advisory lock: finite admissions hold the read side across their fence check and admission CAS (sound in-process under the single-process decree), so arming waits out in-flight admissions and every admission starting afterwards observes the marker and fails closed.

func (*S3WAL) BeginLFSReapEpoch

func (s *S3WAL) BeginLFSReapEpoch(ctx context.Context) (int64, error)

BeginLFSReapEpoch — see ScanStateStore. The write side of reapEpochMu mirrors PG's row-lock serialization: retirements hold the read side from their epoch read through the retire CAS and refund park, so every retirement recorded under an older epoch committed before this sweep's listing began.

func (*S3WAL) BumpLFSAdoptionGen

func (s *S3WAL) BumpLFSAdoptionGen(ctx context.Context, repo string) error

BumpLFSAdoptionGen — see RefStore: purge-fenced repos silently no-op (nothing left to invalidate — PR #105 round 10).

func (*S3WAL) CASAbsenceCoverOwner

func (s *S3WAL) CASAbsenceCoverOwner(ctx context.Context, rawKey, expectedOwner string, expectedBytes int64, newOwner string) (bool, error)

CASAbsenceCoverOwner — see the PG twin: guarded on the LISTED (owner, bytes) pair so a concurrent re-charge always wins. Recovery class OPS (an owner rewrite is not idempotent against a re-charge): the op-ID on the touched entry resolves ambiguous PUTs exactly.

func (*S3WAL) CASRef

func (s *S3WAL) CASRef(ctx context.Context, repo, name, old, new string) (bool, error)

CASRef performs the compare-and-swap ref update (see RefStore). The purge fence and the sync-claim fence are fields of the SAME object the CAS advances, so both checks are same-atom with the swap — the total order PG buys with lockBaseLedger is inherent here (plan §1).

Recovery class CAS (plan Appendix A): a lost race re-reads and re-evaluates — a fresh snapshot no longer matching old is won=false, never retried. An ambiguous PUT whose writer nonce is not ours and whose fresh snapshot no longer satisfies the precondition surfaces the ambiguity as an error: exactly like a lost-commit-response on SQL, the IdempotentCAS decorator's recheck (current == new) recovers the win.

func (*S3WAL) CASRefGen

func (s *S3WAL) CASRefGen(ctx context.Context, repo, name, old, new string) (bool, int64, error)

CASRefGen implements the RefStore contract: exact win attribution. The mutation closure bumps mem.Gen itself, so the post-CAS value is known by construction on the attempt whose manifest swap actually lands (mutate re-runs the closure per retry; the captured value is the last — winning — attempt's).

func (*S3WAL) ClearLFSMismatch

func (s *S3WAL) ClearLFSMismatch(ctx context.Context, repo, key string) error

ClearLFSMismatch — see RefStore.

func (*S3WAL) ClearLFSObjectsUnreferenced

func (s *S3WAL) ClearLFSObjectsUnreferenced(ctx context.Context, repo string, keys []string) (int64, error)

ClearLFSObjectsUnreferenced removes the reclamation mark from the given rows, returning the rows cleared.

func (*S3WAL) ClearSyncClaim

func (s *S3WAL) ClearSyncClaim(ctx context.Context, repoID, holder string) error

ClearSyncClaim removes repoID's sync-claim fence token iff it still belongs to holder (owner-guarded: a thief's token is never touched — the late finisher of a stolen claim must leave the thief's fence intact, issue #86).

Recovery class PRED on the semantic effect: the postcondition is "holder's token is not in the manifest", so a fresh snapshot with no claim or a foreign holder proves it regardless of whether our PUT or a competing overwrite produced it.

func (*S3WAL) Close

func (s *S3WAL) Close()

func (*S3WAL) ConditionalStore

func (s *S3WAL) ConditionalStore() s3wal.ConditionalStore

ConditionalStore exposes the underlying CAS substrate for the serve-time instance lease (§2.11) — one lease object per bucket, managed by s3wal.AcquireInstanceLease outside the driver's manifest protocol. Not for state access: every manifest/scan mutation must go through the driver's locking and recovery machinery.

func (*S3WAL) CountLFSObjects

func (s *S3WAL) CountLFSObjects(ctx context.Context, repo string) (int64, error)

CountLFSObjects returns repo's registered LFS object count — an O(1) maintained-counter read.

func (*S3WAL) CountRefs

func (s *S3WAL) CountRefs(ctx context.Context, repo string, limit int64) (int64, error)

CountRefs prices repo's ref fanout, capped at limit (the count is one in-hand map, so the cap only bounds the REPORTED value).

func (*S3WAL) CountRepoRows

func (s *S3WAL) CountRepoRows(ctx context.Context, group []string) (int64, error)

CountRepoRows counts the group's refs + packs + LFS rows — the purge's pre-completion verification read.

func (*S3WAL) DFConflicts

func (s *S3WAL) DFConflicts(ctx context.Context, repo string, names []string) (map[string]string, error)

DFConflicts implements the batched authority D/F probe for s3wal from ONE manifest read (the price every access on this backend pays), with both arms joined in memory against the sorted name set.

func (*S3WAL) DeleteAbsenceCover

func (s *S3WAL) DeleteAbsenceCover(ctx context.Context, key string) error

DeleteAbsenceCover — see the PG twin. Recovery class PRED/ID: re-applying on a fresh snapshot finds the row gone and no-ops.

func (*S3WAL) DeleteJobScanState

func (s *S3WAL) DeleteJobScanState(ctx context.Context, job, scope string) error

DeleteJobScanState — see ScanStateStore.

func (*S3WAL) DropRepoLFSObjects

func (s *S3WAL) DropRepoLFSObjects(ctx context.Context, group []string) (int64, error)

DropRepoLFSObjects deletes the group's LFS registry rows and the maintained count (LFS rows key by the BASE repo). Returns the object rows deleted. Idempotent.

func (*S3WAL) DropRepoRefs

func (s *S3WAL) DropRepoRefs(ctx context.Context, group []string) (int64, error)

DropRepoRefs deletes every ref row of the group's repos (refs die FIRST) and returns the rows deleted. Member generations bump per dropped row, like the SQL triggers; the generation state itself falls to PurgeRepoLedger. Idempotent.

func (*S3WAL) FenceRepoPurge

func (s *S3WAL) FenceRepoPurge(ctx context.Context, base string) error

FenceRepoPurge durably marks the BASE group purge-fenced (idempotent, permanent): every later row-creating write for the group rejects, so the purge's zero-row verification cannot be raced by a settling writer.

func (*S3WAL) GetGeneration

func (s *S3WAL) GetGeneration(ctx context.Context, repo string) (int64, error)

GetGeneration returns repo's mutation generation (0 = never mutated). s3wal has no schema triggers; the driver bumps member generations inside every CAS that changes the member's refs or packs rows — same invalidation-token contract, same per-changed-row granularity.

func (*S3WAL) GetJobScanState

func (s *S3WAL) GetJobScanState(ctx context.Context, job, scope string) (JobScanState, bool, error)

GetJobScanState — see ScanStateStore.

func (*S3WAL) GetRef

func (s *S3WAL) GetRef(ctx context.Context, repo, name string) (string, error)

GetRef returns the current sha of one ref, or ErrNotFound.

func (*S3WAL) GuardedLFSDelete

func (s *S3WAL) GuardedLFSDelete(ctx context.Context, repo, key string, del func(context.Context) error) (bool, error)

GuardedLFSDelete — see RefStore. SQL serializes delete-vs-register under the ledger lock; an object store cannot join a check and a blob DELETE in one atom, so s3wal PREVENTS the race with the §2.5 W6 delete lease: (i) CAS-insert the lock only while the key is unregistered and no live lock exists; (ii) run del under a hard deleteIOTimeout with a TTL/3 renewal loop; (iii) CAS-clear the lock. Registrations block on live locks for TTL + deleteIOTimeout + safety, so even a lease that missed its final renewal fences the worst-case in-flight DELETE. Recovery class LEASE.

func (*S3WAL) HasLFSObject

func (s *S3WAL) HasLFSObject(ctx context.Context, repo, key string) (bool, error)

HasLFSObject reports whether (repo, key) is a registered LFS object.

func (*S3WAL) HasPack

func (s *S3WAL) HasPack(ctx context.Context, repo, key string) (bool, error)

HasPack reports whether repo has a registered pack with blob key.

func (*S3WAL) HydrateSnapshot

func (s *S3WAL) HydrateSnapshot(ctx context.Context, repo string) (HydrateSnapshot, error)

HydrateSnapshot serves the ref.HydrateSnapshotter capability from ONE manifest read (issue #182): generation, packs, and the full ref set are views of the same atom here, and a hydrate pass consuming them through the portable RefStore methods paid one whole-manifest round trip per call. Materializing the refs is bounded by the manifest hard cap — exactly the eligibility condition the interface documents.

func (*S3WAL) InitSchema

func (s *S3WAL) InitSchema(ctx context.Context) error

InitSchema proves the store's conditional-write fidelity before serving (plan §2.6): bucket reachability and header behavior are both covered by the 5-step probe, which refuses to serve on any deviation. Idempotent.

func (*S3WAL) IsRepoPurged

func (s *S3WAL) IsRepoPurged(ctx context.Context, base string) (bool, error)

IsRepoPurged reports whether the BASE repo is purge-fenced.

func (*S3WAL) IsRetainedPack

func (s *S3WAL) IsRetainedPack(ctx context.Context, repo, key string) (bool, error)

IsRetainedPack reports whether (repo, key) is a recorded superseded blob.

func (*S3WAL) JobScanStateAge

func (s *S3WAL) JobScanStateAge(ctx context.Context, job, scope string) (time.Duration, bool, error)

JobScanStateAge — see ScanStateStore and the header's clock story.

func (*S3WAL) LFSAdoptionGen

func (s *S3WAL) LFSAdoptionGen(ctx context.Context, repo string) (int64, error)

LFSAdoptionGen — see RefStore. The generation is accounted at group scope (the floor caches' quota accounting unit is the base repository).

func (*S3WAL) LFSMismatchFingerprint

func (s *S3WAL) LFSMismatchFingerprint(ctx context.Context, repo, key string) (string, bool, error)

LFSMismatchFingerprint — see RefStore.

func (*S3WAL) LFSObjectRegisteredAt

func (s *S3WAL) LFSObjectRegisteredAt(ctx context.Context, repo, key string) (time.Time, int64, bool, error)

LFSObjectRegisteredAt — see RefStore.

func (*S3WAL) LFSReposWithObjects

func (s *S3WAL) LFSReposWithObjects(ctx context.Context, after string, limit int) ([]string, error)

LFSReposWithObjects pages base repos holding at least one registered LFS object (keyset on repo ascending): a fleet prefix walk reading each group's maintained count (§2.9 — bucket listing order IS base byte order).

func (*S3WAL) LapsedAbsenceCoverExists

func (s *S3WAL) LapsedAbsenceCoverExists(ctx context.Context, ttl time.Duration) (bool, error)

LapsedAbsenceCoverExists — see the PG twin: any physically present un-folded cover row aged past ttl is lapse evidence. Liveness leans LIVE under the §2.4 skew guard (s3walReservationLive): a clock jump must not mass-manufacture false lapse evidence, and rows persist, so genuine evidence resurfaces once clocks agree.

func (*S3WAL) ListAbsenceCovers

func (s *S3WAL) ListAbsenceCovers(ctx context.Context, afterRaw string, limit int, ttl time.Duration) ([]AbsenceCoverRow, error)

ListAbsenceCovers — see the PG twin for the contract: rows ascend by RAW key across pages (the consumer's rotating renewal cursor wraps on that order). The manifest walk therefore runs in decorated cover-key order (s3walEachBaseCoverOrder), and cursor resume derives the cursor row's base group from its own spelling, re-emits that group's rows past the cursor, seeds the held prefix chain, then continues strictly after it. Every emitted row is asserted strictly above its predecessor — the advertised order made a crash-and-burn invariant.

func (*S3WAL) ListFleetLFSObjects

func (s *S3WAL) ListFleetLFSObjects(ctx context.Context, afterRepo, afterKey string, limit int) ([]AbsenceLFSRow, error)

ListFleetLFSObjects pages registered LFS rows fleet-wide in (repo, key) byte order. LFS rows key by the base repo, so each group contributes one member; cursor resume strips the member suffix, re-reads that group, and continues the listing after it.

func (*S3WAL) ListFleetPacks

func (s *S3WAL) ListFleetPacks(ctx context.Context, afterRepo, afterID string, limit int) ([]AbsencePackRow, error)

ListFleetPacks pages registered pack rows fleet-wide in (repo, id) byte order across every group member (§2.9: a base's member keys are contiguous in byte order — the key-build alphabet assert makes the lemma checked, not assumed).

func (*S3WAL) ListJobScanStateScopes

func (s *S3WAL) ListJobScanStateScopes(ctx context.Context, job, prefix, after string, limit int) ([]string, error)

ListJobScanStateScopes — see ScanStateStore. Listing order over the hex-encoded keys IS byte order over the unencoded scopes (the header's monotone-injection argument), and both range ends are index bounds exactly like the SQL twins: keyset from max(after, prefix), exclusive upper bound coverRangeEnd(prefix).

func (*S3WAL) ListPacks

func (s *S3WAL) ListPacks(ctx context.Context, repo string) ([]Pack, error)

ListPacks returns all registered packs for repo, oldest first (byte order of id breaks CreatedAt ties, mirroring ORDER BY created_at, id).

func (*S3WAL) ListRefs

func (s *S3WAL) ListRefs(ctx context.Context, repo string) (map[string]string, error)

ListRefs returns all refs for repo as name -> sha.

func (*S3WAL) ListRefsAfter

func (s *S3WAL) ListRefsAfter(ctx context.Context, repo, after string, limit int) ([]RefEntry, error)

ListRefsAfter pages repo's refs in ascending byte order above the exclusive cursor (issue #119). The whole set is already in hand (one manifest read), so paging is in-memory — the contract (byte order, exclusive cursor, short page ends the scan) is preserved.

func (*S3WAL) ListRefsWithMeta

func (s *S3WAL) ListRefsWithMeta(ctx context.Context, repo string) ([]RefInfo, error)

ListRefsWithMeta returns all refs for repo sorted by name ascending.

func (*S3WAL) ListRepoLFSObjects

func (s *S3WAL) ListRepoLFSObjects(ctx context.Context, repo, after string, limit int) ([]LFSObjectRow, error)

ListRepoLFSObjects pages repo's registered LFS rows (keyset on key, ascending byte order).

func (*S3WAL) MarkLFSObjectsUnreferenced

func (s *S3WAL) MarkLFSObjectsUnreferenced(ctx context.Context, repo string, keys []string) (int64, error)

MarkLFSObjectsUnreferenced stamps the reclamation mark on unmarked rows, preserving existing stamps (the grace window measures CONTINUOUS unreferenced observation). Returns the rows stamped.

func (*S3WAL) PackRegisteredBytes

func (s *S3WAL) PackRegisteredBytes(ctx context.Context, repo, key string) (int64, bool, error)

PackRegisteredBytes returns the registered pack's recorded size — see RefStore (0 = size unknown).

func (*S3WAL) ParkRepoJobScanState

func (s *S3WAL) ParkRepoJobScanState(ctx context.Context, job, scope, payload string, generation int64, base string) error

ParkRepoJobScanState — see ScanStateStore. The purge fence and the park cannot share one atom across a manifest and a side object; the honest s3wal substitute is the per-base in-process lock BOTH this check and FenceRepoPurge's manifest CAS run under (the same serialization PG buys with the base-ledger advisory lock, sound under the single-process decree): a park ordered before the fence lands before the purge's ledger cleanup sweeps the row; a park ordered after sees the fence and skips.

func (*S3WAL) PurgeRepoLedger

func (s *S3WAL) PurgeRepoLedger(ctx context.Context, group []string) (int64, error)

PurgeRepoLedger removes the group's remaining accounting and state rows (reservations incl. claim leases, retained tombstones, usage counter, LFS counts/mismatches/adoption gens, member generations), returning the rows deleted — the SQL row-per-row counting mapped onto manifest entries/fields. Runs strictly AFTER the blob pass; idempotent (rerun returns 0). The manifest itself persists as the fence + tombstone.

func (*S3WAL) PutJobScanState

func (s *S3WAL) PutJobScanState(ctx context.Context, job, scope, payload string, generation int64) error

PutJobScanState — see ScanStateStore.

func (*S3WAL) QuotaLedgerState

func (s *S3WAL) QuotaLedgerState(ctx context.Context, group []string, ttl time.Duration, excludeKeys ...string) (PackQuotaLedgerState, error)

QuotaLedgerState reads the group's ledger state in one snapshot (see PackQuotaLedgerState). Usage is accounted at group scope in the manifest (namespace usage rolls up to the base — the quota accounting unit), so UsageBytes reports the group counter.

func (*S3WAL) ReclaimRetainedPack

func (s *S3WAL) ReclaimRetainedPack(ctx context.Context, repo, key string) (int64, error)

ReclaimRetainedPack removes the retained row, debits the usage counter (clamped at zero), and bumps the shared floor generation when bytes actually left (Codex round 59). Recovery class PRED/ID: re-applying on a fresh snapshot finds the row gone and reports 0 — no double debit.

func (*S3WAL) ReconcileLegacyLFSObjects

func (s *S3WAL) ReconcileLegacyLFSObjects(ctx context.Context) (int64, bool, error)

ReconcileLegacyLFSObjects — see RefStore: fleet walk, twin-drop + refund per group CAS (L+M, idempotent). Pages honor reconcileLegacyLFSBatch and the reconcileLegacyLFSMaxPages invocation cap (partial=true when capped).

func (*S3WAL) RecordLFSMismatch

func (s *S3WAL) RecordLFSMismatch(ctx context.Context, repo, key, fingerprint string) error

RecordLFSMismatch — see RefStore (#104): fleet-persistent upsert; purge-fenced repos silently no-op like every row-creating write.

func (*S3WAL) RecordLFSReapComplete

func (s *S3WAL) RecordLFSReapComplete(ctx context.Context, epoch int64, legacyArmed bool) error

RecordLFSReapComplete — see the SQL twins.

func (*S3WAL) RefValues

func (s *S3WAL) RefValues(ctx context.Context, repo string, names []string) (map[string]string, error)

RefValues implements the batched exact-value probe for s3wal from ONE manifest read (the price every access on this backend pays), joined in memory against the member's ref map.

func (*S3WAL) RegisterAdoptedLFSObject

func (s *S3WAL) RegisterAdoptedLFSObject(ctx context.Context, repo, key string, bytes int64) error

RegisterAdoptedLFSObject — see RefStore: the adoption-generation bump commits atomically with the registration (Codex round 51).

func (*S3WAL) RegisterLFSObject

func (s *S3WAL) RegisterLFSObject(ctx context.Context, repo, key string, bytes int64) error

RegisterLFSObject — see RefStore. One CAS: legacy migration / twin reconcile, idempotent insert, atomic cap check, reservation settlement, mark clearing, usage credit. Recovery class PRED/ID: re-applying on a fresh snapshot is zero-growth by construction.

func (*S3WAL) RegisterPacks

func (s *S3WAL) RegisterPacks(ctx context.Context, repo string, packs []Pack) error

RegisterPacks records packs atomically (see RefStore): pack-set insert, reservation consumption, retained-reregistration credit rule, usage credit, adoption-gen bump, and per-row generation bumps are ONE CAS. Recovery class PRED/ID: re-applying on a fresh snapshot is the SQL upsert contract (existing rows skip, reservations already gone consume nothing, no double credit).

func (*S3WAL) ReleaseAdoptionClaim

func (s *S3WAL) ReleaseAdoptionClaim(ctx context.Context, key, owner string) error

ReleaseAdoptionClaim — see RefStore: owner-fenced delete (best-effort; a crashed winner's claim expires via AdoptionClaimTTL instead).

func (*S3WAL) RenewAbsenceCovers

func (s *S3WAL) RenewAbsenceCovers(ctx context.Context, ttl time.Duration) (int64, error)

RenewAbsenceCovers — see the PG twin: LIVE deterministic incarnation rows and LIVE un-folded legacy nonce rows are re-stamped with the write-time manifest clock; expired rows stay dead (the strict-TTL round-6 invariant) and folded nonces decay deliberately. One CAS per base holding matching rows; re-stamping again on a lost-race re-run is harmless (ID class).

func (*S3WAL) RenewAdoptionClaim

func (s *S3WAL) RenewAdoptionClaim(ctx context.Context, key, owner string) (bool, error)

RenewAdoptionClaim — see RefStore: restarts the TTL clock while the live claim still belongs to owner.

func (*S3WAL) ReplacePacks

func (s *S3WAL) ReplacePacks(ctx context.Context, repo string, old []Pack, newPack Pack) (won bool, err error)

ReplacePacks installs a repack fold in one CAS (see RefStore): the precondition — every old row still present — is evaluated on the read snapshot and enforced by the CAS itself. Recovery class PRED: fold present on the fresh snapshot ⇒ applied; old set fully intact ⇒ our write never landed (re-run); anything else is inconclusive and fails closed (a false loss would make the caller delete a REGISTERED fold blob).

func (*S3WAL) ReservePackBytes

func (s *S3WAL) ReservePackBytes(ctx context.Context, group []string, packs []Pack, owner string, usageFloor, floorGen, quotaBytes int64, ttl time.Duration) (ReserveDecision, error)

ReservePackBytes implements the admission check documented atop ledger.go over one manifest CAS. Recovery class OPS (attempt increments are non-idempotent): ambiguity resolves via the writer nonce, then the op-ID's membership in the ops ring or on a touched reservation entry; both missing fails closed with a retryable error — never a blind re-apply.

func (*S3WAL) RetirePackAttempts

func (s *S3WAL) RetirePackAttempts(ctx context.Context, group []string, gens map[string]string) error

RetirePackAttempts retires one same-key attempt per entry (see RefStore): gen-fenced decrement, delete at zero attempts. Recovery class OPS: nonce, then op-ID membership, else fail closed.

func (*S3WAL) RetireRepoPacks

func (s *S3WAL) RetireRepoPacks(ctx context.Context, group []string) (int64, error)

RetireRepoPacks tombstones every registered pack of the group's repos: each packs row becomes a retained row (bytes 0 — classification tombstones, not accounting; the usage counter stays credited until PurgeRepoLedger) and is deleted. An EXISTING retained row (a real fold's, carrying real bytes) is kept as-is so its eventual reclaim still debits correctly. Returns the pack rows retired. Idempotent.

func (*S3WAL) RetireUnreferencedLFSObject

func (s *S3WAL) RetireUnreferencedLFSObject(ctx context.Context, repo, key string, pinnedGens map[string]int64, grace, reapMaxAge time.Duration) (bool, error)

RetireUnreferencedLFSObject deletes one grace-expired marked row with the fences evaluated on the retire CAS's snapshot: purge fence, pinned member generations (a pointer push mid-scan lives in a bumped generation by definition), and the live adoption-claim spare. The count/usage refund is DEFERRED (Codex round 33 on PR #132): the deletion parks its refund in the per-epoch bucket and only SettleLFSRefunds hands capacity back, under a completion whose epoch strictly exceeds the pin.

§2.10 substitute rule for the SQL twins' in-transaction reads: the destructive-reap watermark, the legacy certification, and the epoch pin are side-object reads immediately before the retire CAS — sweeper and reclaimer share one process (the single-process decree), reapEpochMu mirrors PG's FOR SHARE pin against BeginLFSReapEpoch, and the clocks at both ends are the one process clock. The refund park is a second, non-atomic write ORDERED AFTER the row deletion: a crash in between leaks the parked slot (count/usage stay charged — strictly the conservative direction, mirroring "orphan packs are tolerable"); the reverse order could double-refund and is never used. Recovery class OPS on the delete: the op-ID keeps an ambiguously-landed deletion's retired=true so its refund still parks.

func (*S3WAL) SetSyncClaim

func (s *S3WAL) SetSyncClaim(ctx context.Context, repoID, holder string) error

SetSyncClaim installs holder as repoID's sync-claim fence token, overwriting any previous token: the caller won the meta claim lease (possibly by stealing an expired one), so the previous token — live or stale — is superseded by construction.

Recovery class PRED: holder tokens are unique per sync run, so no same-holder entry can predate this call — a fresh snapshot carrying our holder proves OUR write landed (the Phase 1b renewal lesson about same-field rewrites does not bite: there is no pre-existing same-holder entry whose old stamp could alias a dropped PUT).

func (*S3WAL) SettleLFSRefunds

func (s *S3WAL) SettleLFSRefunds(ctx context.Context, repo string) (SettledLFSRefunds, error)

SettleLFSRefunds — see the PG twin for the settlement rules (epoch strictly exceeded; legacy halves only under legacy-armed completions; keyset-paged past stuck buckets). The s3wal split, ordered fail-closed: bucket rewrites/drains happen FIRST as side-object CASes, then ONE manifest CAS applies the accumulated count/usage refund and the adoption-generation bump — a crash in between loses the drained refund (capacity stays charged: conservative), while the reverse order could refund twice. Watermark reads are §2.10 side-object reads; purge-fenced repos settle nothing (checked before draining AND re-checked inside the refund CAS — the purge drops buckets and counters wholesale).

func (*S3WAL) SweepExpiredPackReservations

func (s *S3WAL) SweepExpiredPackReservations(ctx context.Context, olderThan time.Duration) (int64, error)

SweepExpiredPackReservations prunes reservation rows fleet-wide: one prefix walk, one CAS per group holding expired rows (L+M, idempotent). Non-claim rows expire at olderThan (0 = everything — the test/recovery hatch); claim leases expire only at AdoptionClaimTTL. Like the SQL twins (Codex round 34 on PR #132), consuming an expired UN-FOLDED absence-cover row records the durable lapse-evidence marker before the pruning CAS commits.

func (*S3WAL) TouchLFSObject

func (s *S3WAL) TouchLFSObject(ctx context.Context, repo, key string) (bool, error)

TouchLFSObject — see RefStore: clears the reclamation mark and reports whether the row exists (the zero-growth liveness handshake, issue #73).

func (*S3WAL) TryAdoptionClaim

func (s *S3WAL) TryAdoptionClaim(ctx context.Context, repo, key string) (bool, string, error)

TryAdoptionClaim — see RefStore: claims are reservation entries under the /adopt-claim key suffix (exactly the SQL spelling), stolen only past AdoptionClaimTTL under the §2.4 rules. Purge-fenced repos report claimed=false. Recovery class PRED: owner token == ours ⇒ applied.

type SQLite

type SQLite struct {
	// contains filtered or unexported fields
}

SQLite implements Store on one local SQLite file via the pure-Go modernc.org/sqlite driver (cgo-free cross-compiled single binaries).

func OpenSQLite

func OpenSQLite(ctx context.Context, sqliteURL string) (*SQLite, error)

OpenSQLite opens sqlite:///abs/path/db.sqlite. The path is everything after "sqlite://" (relative paths are tolerated; --data-dir always derives absolute ones). Parent directories are created; the DB file itself is created by the driver on first open.

func (*SQLite) AbsenceCoverHead

func (s *SQLite) AbsenceCoverHead(ctx context.Context, prefix string, ttl time.Duration) (AbsenceCoverRow, bool, error)

AbsenceCoverHead — see the PG twin.

func (*SQLite) ArmAbsenceAdmissionFence

func (s *SQLite) ArmAbsenceAdmissionFence(ctx context.Context, payload string) error

ArmAbsenceAdmissionFence — see the PG twin for the ordering contract. SQLite needs no explicit lock: the upsert is one IMMEDIATE-mode write, and ReservePackBytes' whole check-and-reserve runs in a single-writer BEGIN IMMEDIATE transaction, so the pair cannot interleave.

func (*SQLite) BeginLFSReapEpoch

func (s *SQLite) BeginLFSReapEpoch(ctx context.Context) (int64, error)

BeginLFSReapEpoch — see ScanStateStore. Single-writer statements make the increment atomic; retirements' epoch reads serialize behind it.

func (*SQLite) BumpLFSAdoptionGen

func (s *SQLite) BumpLFSAdoptionGen(ctx context.Context, repo string) error

BumpLFSAdoptionGen — see RefStore (Codex round 49). The purge-fence guard is inside the single upsert statement (issue #52, PR #105 round 10; see the PG twin's rationale): no row proposed for a fenced repo means no conflict and no update, and SQLite's single writer totally orders the statement against FenceRepoPurge's insert.

func (*SQLite) CASAbsenceCoverOwner

func (s *SQLite) CASAbsenceCoverOwner(ctx context.Context, rawKey, expectedOwner string, expectedBytes int64, newOwner string) (bool, error)

CASAbsenceCoverOwner — see the PG twin.

func (*SQLite) CASRef

func (s *SQLite) CASRef(ctx context.Context, repo, name, old, new string) (bool, error)

CASRef performs the compare-and-swap ref update. Returns won=false (no error) when the precondition failed, i.e. someone else won the race. Refs only ever change inside these single statements, so a broken/partial ref state is impossible by construction; the statement runs inside an explicit BEGIN IMMEDIATE transaction whose only other work — the purge- fence read below — happens strictly BEFORE it, so the statement remains the single mutation unit and the IdempotentCAS ambiguity classification (idempotent.go) is unchanged: pre-statement failures are tagged notExecuted, and a lost COMMIT stays ambiguous (recovered by the recheck). A won swap also bumps the repo generation via the InitSchema-installed triggers inside the same transaction.

The transaction rejects purge-fenced repos with ErrRepoPurged (issue #52, PR #105 security round; see the PG twin's rationale): BEGIN IMMEDIATE takes the database write lock at begin, totally ordering the fence read + CAS pair against FenceRepoPurge's insert.

Under WithSyncClaimFence (ref.go) the statement additionally requires the repo's sync claim to be held by the fencing holder: SQLite's single-writer lock serializes the fenced CAS against a concurrent claim steal, so a stolen run's update can never commit after the thief's claim (issue #86). Requires the shared local database (meta + ref in one --data-dir file).

func (*SQLite) CASRefGen

func (s *SQLite) CASRefGen(ctx context.Context, repo, name, old, new string) (bool, int64, error)

CASRefGen implements the RefStore contract: exact win attribution. The schema triggers fire immediately (unlike PG's deferred constraint triggers), so the post-bump value is readable inside the transaction, and BEGIN IMMEDIATE's single-writer property makes it exact.

func (*SQLite) ClearLFSMismatch

func (s *SQLite) ClearLFSMismatch(ctx context.Context, repo, key string) error

ClearLFSMismatch — see RefStore (#104, Codex round 132).

func (*SQLite) ClearLFSObjectsUnreferenced

func (s *SQLite) ClearLFSObjectsUnreferenced(ctx context.Context, repo string, keys []string) (int64, error)

ClearLFSObjectsUnreferenced — see the PG twin.

func (*SQLite) Close

func (s *SQLite) Close()

func (*SQLite) CountLFSObjects

func (s *SQLite) CountLFSObjects(ctx context.Context, repo string) (int64, error)

CountLFSObjects returns repo's registered LFS object count — an O(1) counter-row read (see the PG twin).

func (*SQLite) CountRefs

func (s *SQLite) CountRefs(ctx context.Context, repo string, limit int64) (int64, error)

CountRefs — see the PG twin.

func (*SQLite) CountRepoRows

func (s *SQLite) CountRepoRows(ctx context.Context, group []string) (int64, error)

CountRepoRows — see the PG twin.

func (*SQLite) DFConflicts

func (s *SQLite) DFConflicts(ctx context.Context, repo string, names []string) (map[string]string, error)

DFConflicts implements the batched authority D/F probe for SQLite: one chunked indexed membership read over the candidates' deduped strict prefixes plus one local single-row indexed range seek per candidate — index work proportional to the candidate set, never to the repository (SQLite's default BINARY collation already compares by byte order).

func (*SQLite) DeleteAbsenceCover

func (s *SQLite) DeleteAbsenceCover(ctx context.Context, key string) error

DeleteAbsenceCover — see the PG twin.

func (*SQLite) DeleteJobScanState

func (s *SQLite) DeleteJobScanState(ctx context.Context, job, scope string) error

func (*SQLite) DropRepoLFSObjects

func (s *SQLite) DropRepoLFSObjects(ctx context.Context, group []string) (int64, error)

DropRepoLFSObjects — see the PG twin.

func (*SQLite) DropRepoRefs

func (s *SQLite) DropRepoRefs(ctx context.Context, group []string) (int64, error)

DropRepoRefs — see the PG twin.

func (*SQLite) FenceRepoPurge

func (s *SQLite) FenceRepoPurge(ctx context.Context, base string) error

FenceRepoPurge — see the PG twin. SQLite's single-writer transactions (BEGIN IMMEDIATE) already totally order the fence insert against every registration transaction, so no extra lock is needed.

func (*SQLite) GetGeneration

func (s *SQLite) GetGeneration(ctx context.Context, repo string) (int64, error)

GetGeneration returns repo's mutation generation (0 = never mutated).

func (*SQLite) GetJobScanState

func (s *SQLite) GetJobScanState(ctx context.Context, job, scope string) (JobScanState, bool, error)

func (*SQLite) GetRef

func (s *SQLite) GetRef(ctx context.Context, repo, name string) (string, error)

GetRef returns the current sha of one ref, or ErrNotFound.

func (*SQLite) GuardedLFSDelete

func (s *SQLite) GuardedLFSDelete(ctx context.Context, repo, key string, del func(context.Context) error) (bool, error)

func (*SQLite) HasLFSObject

func (s *SQLite) HasLFSObject(ctx context.Context, repo, key string) (bool, error)

HasLFSObject reports whether (repo, key) is a registered LFS object.

func (*SQLite) HasPack

func (s *SQLite) HasPack(ctx context.Context, repo, key string) (bool, error)

HasPack reports whether repo has a registered pack with blob key.

func (*SQLite) InitSchema

func (s *SQLite) InitSchema(ctx context.Context) error

InitSchema creates the ref/pack tables (idempotent). No advisory lock (unlike PG): SQLite's file lock + busy_timeout serialize concurrent startups, and CREATE TABLE IF NOT EXISTS is race-free under that lock.

func (*SQLite) IsRepoPurged

func (s *SQLite) IsRepoPurged(ctx context.Context, base string) (bool, error)

IsRepoPurged — see the PG twin.

func (*SQLite) IsRetainedPack

func (s *SQLite) IsRetainedPack(ctx context.Context, repo, key string) (bool, error)

IsRetainedPack reports whether (repo, key) is a recorded superseded blob.

func (*SQLite) JobScanStateAge

func (s *SQLite) JobScanStateAge(ctx context.Context, job, scope string) (time.Duration, bool, error)

JobScanStateAge — see ScanStateStore. updated_at is epoch milliseconds at SECOND precision (strftime %s × 1000), so the age can over-read by up to a second; every freshness bound comparing it has a ≥1 minute floor (jobs.reapWatermarkMinFreshness).

func (*SQLite) LFSAdoptionGen

func (s *SQLite) LFSAdoptionGen(ctx context.Context, repo string) (int64, error)

LFSAdoptionGen — see RefStore (Codex round 49).

func (*SQLite) LFSMismatchFingerprint

func (s *SQLite) LFSMismatchFingerprint(ctx context.Context, repo, key string) (string, bool, error)

LFSMismatchFingerprint — see RefStore (#104, Codex round 132).

func (*SQLite) LFSObjectRegisteredAt

func (s *SQLite) LFSObjectRegisteredAt(ctx context.Context, repo, key string) (time.Time, int64, bool, error)

LFSObjectRegisteredAt — see RefStore (Codex round 89; bytes since #127). created_at is unixepoch() seconds in the SQLite schema.

func (*SQLite) LFSReposWithObjects

func (s *SQLite) LFSReposWithObjects(ctx context.Context, after string, limit int) ([]string, error)

LFSReposWithObjects — see the PG twin.

func (*SQLite) LapsedAbsenceCoverExists

func (s *SQLite) LapsedAbsenceCoverExists(ctx context.Context, ttl time.Duration) (bool, error)

LapsedAbsenceCoverExists — see the PG twin (Codex round 32, security).

func (*SQLite) ListAbsenceCovers

func (s *SQLite) ListAbsenceCovers(ctx context.Context, afterRaw string, limit int, ttl time.Duration) ([]AbsenceCoverRow, error)

ListAbsenceCovers — see the PG twin. SQLite's default BINARY collation is already byte order, so the plain range scan matches the PG COLLATE "C" semantics.

func (*SQLite) ListFleetLFSObjects

func (s *SQLite) ListFleetLFSObjects(ctx context.Context, afterRepo, afterKey string, limit int) ([]AbsenceLFSRow, error)

ListFleetLFSObjects — see the PG twin.

func (*SQLite) ListFleetPacks

func (s *SQLite) ListFleetPacks(ctx context.Context, afterRepo, afterID string, limit int) ([]AbsencePackRow, error)

ListFleetPacks — see the PG twin.

func (*SQLite) ListJobScanStateScopes

func (s *SQLite) ListJobScanStateScopes(ctx context.Context, job, prefix, after string, limit int) ([]string, error)

ListJobScanStateScopes — see ScanStateStore. SQLite TEXT compares by byte order natively.

func (*SQLite) ListPacks

func (s *SQLite) ListPacks(ctx context.Context, repo string) ([]Pack, error)

ListPacks returns all registered packs for repo, oldest first.

func (*SQLite) ListRefs

func (s *SQLite) ListRefs(ctx context.Context, repo string) (map[string]string, error)

ListRefs returns all refs for repo as name -> sha.

func (*SQLite) ListRefsAfter

func (s *SQLite) ListRefsAfter(ctx context.Context, repo, after string, limit int) ([]RefEntry, error)

ListRefsAfter returns up to limit refs for repo above the after cursor in ascending byte order (issue #119). The name column's default BINARY collation is byte-wise comparison — matching PG's COLLATE "C" page order and git's own ref iteration order — and the (repo, name) primary key serves the keyset scan directly.

func (*SQLite) ListRefsWithMeta

func (s *SQLite) ListRefsWithMeta(ctx context.Context, repo string) ([]RefInfo, error)

ListRefsWithMeta returns all refs for repo sorted by name ascending.

func (*SQLite) ListRepoLFSObjects

func (s *SQLite) ListRepoLFSObjects(ctx context.Context, repo, after string, limit int) ([]LFSObjectRow, error)

ListRepoLFSObjects — see the PG twin.

func (*SQLite) MarkLFSObjectsUnreferenced

func (s *SQLite) MarkLFSObjectsUnreferenced(ctx context.Context, repo string, keys []string) (int64, error)

MarkLFSObjectsUnreferenced — see the PG twin.

func (*SQLite) PackRegisteredBytes

func (s *SQLite) PackRegisteredBytes(ctx context.Context, repo, key string) (int64, bool, error)

PackRegisteredBytes returns the registered pack's recorded size — see RefStore (0 = pre-#101 row, size unknown).

func (*SQLite) ParkRepoJobScanState

func (s *SQLite) ParkRepoJobScanState(ctx context.Context, job, scope, payload string, generation int64, base string) error

func (*SQLite) PurgeRepoLedger

func (s *SQLite) PurgeRepoLedger(ctx context.Context, group []string) (int64, error)

PurgeRepoLedger — see the PG twin.

func (*SQLite) PutJobScanState

func (s *SQLite) PutJobScanState(ctx context.Context, job, scope, payload string, generation int64) error

func (*SQLite) QuotaLedgerState

func (s *SQLite) QuotaLedgerState(ctx context.Context, group []string, ttl time.Duration, excludeKeys ...string) (PackQuotaLedgerState, error)

QuotaLedgerState reads the group's ledger state (see the PG twin).

func (*SQLite) ReclaimRetainedPack

func (s *SQLite) ReclaimRetainedPack(ctx context.Context, repo, key string) (int64, error)

ReclaimRetainedPack removes the retained row and debits the usage counter.

func (*SQLite) ReconcileLegacyLFSObjects

func (s *SQLite) ReconcileLegacyLFSObjects(ctx context.Context) (int64, bool, error)

ReconcileLegacyLFSObjects — see RefStore. Paged like the PG twin (Codex round 91): one bounded candidate page in memory at a time; each reconcile deletes its row, so re-querying converges without offsets.

func (*SQLite) RecordLFSMismatch

func (s *SQLite) RecordLFSMismatch(ctx context.Context, repo, key, fingerprint string) error

RecordLFSMismatch — see RefStore (#104, Codex round 132). The purge fence rides the INSERT like BumpLFSAdoptionGen's; BEGIN IMMEDIATE's single writer is the serialization.

func (*SQLite) RecordLFSReapComplete

func (s *SQLite) RecordLFSReapComplete(ctx context.Context, epoch int64, legacyArmed bool) error

RecordLFSReapComplete — see the PG twin.

func (*SQLite) RefValues

func (s *SQLite) RefValues(ctx context.Context, repo string, names []string) (map[string]string, error)

RefValues implements the batched exact-value probe for SQLite: chunked indexed membership reads over the candidate names (see refvalues.go for the contract; the chunking mirrors DFConflicts' parameter-limit bound).

func (*SQLite) RegisterAdoptedLFSObject

func (s *SQLite) RegisterAdoptedLFSObject(ctx context.Context, repo, key string, bytes int64) error

RegisterAdoptedLFSObject — see RefStore (Codex round 51): the adoption generation bump commits atomically with the registration.

func (*SQLite) RegisterLFSObject

func (s *SQLite) RegisterLFSObject(ctx context.Context, repo, key string, bytes int64) error

RegisterLFSObject records one verified LFS object and settles its quota reservation (issue #51) — statement-for-statement mirror of the PG twin.

func (*SQLite) RegisterPacks

func (s *SQLite) RegisterPacks(ctx context.Context, repo string, packs []Pack) error

RegisterPacks inserts pack rows in ONE transaction (crash-ordering: this runs strictly after the blob upload succeeded). Idempotent on retry: a re-pushed identical pack has the same id. The transaction settles the quota ledger exactly like the PG twin (see its doc): reservations are consumed by key, and newly inserted non-retained rows credit the usage counter.

func (*SQLite) ReleaseAdoptionClaim

func (s *SQLite) ReleaseAdoptionClaim(ctx context.Context, key, owner string) error

ReleaseAdoptionClaim — see RefStore.

func (*SQLite) RenewAbsenceCovers

func (s *SQLite) RenewAbsenceCovers(ctx context.Context, ttl time.Duration) (int64, error)

RenewAbsenceCovers — see the PG twin. SQLite lacks regexp by default; GLOB is case-sensitive and supports [0-9] classes, so one pattern per class replaces the PG alternation (the suffix contains no GLOB metacharacters).

func (*SQLite) RenewAdoptionClaim

func (s *SQLite) RenewAdoptionClaim(ctx context.Context, key, owner string) (bool, error)

RenewAdoptionClaim — see RefStore.

func (*SQLite) ReplacePacks

func (s *SQLite) ReplacePacks(ctx context.Context, repo string, old []Pack, newPack Pack) (won bool, err error)

ReplacePacks installs a repack fold in one transaction (see RefStore and the PG implementation, whose semantics this mirrors byte-for-byte: the delete-rowcount is the CAS precondition — summed across bounded delete chunks here — the insert tolerates an existing identical (repo, id) row, and a lost race rolls back with won=false).

func (*SQLite) ReservePackBytes

func (s *SQLite) ReservePackBytes(ctx context.Context, group []string, packs []Pack, owner string, usageFloor, floorGen, quotaBytes int64, ttl time.Duration) (ReserveDecision, error)

ReservePackBytes implements the ledger.go admission check on SQLite.

func (*SQLite) RetirePackAttempts

func (s *SQLite) RetirePackAttempts(ctx context.Context, group []string, gens map[string]string) error

RetirePackAttempts — see the PG twin; BEGIN IMMEDIATE's single writer is the serialization.

func (*SQLite) RetireRepoPacks

func (s *SQLite) RetireRepoPacks(ctx context.Context, group []string) (int64, error)

RetireRepoPacks — see the PG twin.

func (*SQLite) RetireUnreferencedLFSObject

func (s *SQLite) RetireUnreferencedLFSObject(ctx context.Context, repo, key string, pinnedGens map[string]int64, grace, reapMaxAge time.Duration) (bool, error)

RetireUnreferencedLFSObject — see the PG twin for the full contract.

func (*SQLite) SettleLFSRefunds

func (s *SQLite) SettleLFSRefunds(ctx context.Context, repo string) (SettledLFSRefunds, error)

SettleLFSRefunds — see the PG twin (Codex round 33 on PR #132).

func (*SQLite) SweepExpiredPackReservations

func (s *SQLite) SweepExpiredPackReservations(ctx context.Context, olderThan time.Duration) (int64, error)

SweepExpiredPackReservations prunes reservation rows at least olderThan old. The marker insert runs BEFORE the delete inside one IMMEDIATE transaction: deleting an expired un-folded absence-cover row destroys the physical lapse evidence the reconciler's first-health-stamp probe arms on, and the scheduled sweep is a second consumer of that evidence beside ReservePackBytes' admission-time GC (Codex rounds 33+34 on PR #132). The sweeper passes the admission TTL as olderThan, so every doomed cover row is exactly a lapsed cover; folded rows never mark.

func (*SQLite) TouchLFSObject

func (s *SQLite) TouchLFSObject(ctx context.Context, repo, key string) (bool, error)

TouchLFSObject — see RefStore.

func (*SQLite) TryAdoptionClaim

func (s *SQLite) TryAdoptionClaim(ctx context.Context, repo, key string) (bool, string, error)

GuardedLFSDelete — see the PG twin; BEGIN IMMEDIATE's single writer is the serialization here. TryAdoptionClaim — see RefStore (Codex security round 73). Local mode is single-process, but the shared-table claim keeps semantics identical to PG (and dedupes across the serve process and hook subprocesses).

type ScanStateStore

type ScanStateStore interface {
	// GetJobScanState reads one row; ok=false when absent.
	GetJobScanState(ctx context.Context, job, scope string) (JobScanState, bool, error)
	// JobScanStateAge reports how long ago (job, scope) was last
	// written, measured entirely on the DATABASE clock — freshness
	// verdicts must never compare a DB-stamped updated_at against an
	// application clock (Codex round 10 on PR #132: a replica skewed
	// ahead by more than the bound would permanently misjudge a
	// DB-fresh watermark as stale). ok=false when the row is absent.
	JobScanStateAge(ctx context.Context, job, scope string) (time.Duration, bool, error)
	// PutJobScanState atomically upserts one row, stamping updated_at
	// with the database clock.
	PutJobScanState(ctx context.Context, job, scope, payload string, generation int64) error
	// DeleteJobScanState removes one row; deleting an absent row is a
	// no-op.
	DeleteJobScanState(ctx context.Context, job, scope string) error
	// ParkRepoJobScanState upserts one PER-REPO row exactly like
	// PutJobScanState, but consults base's purge fence in the same
	// transaction and silently skips the write when fenced (Codex
	// round 26 on PR #132): the purge's ledger cleanup deletes the
	// repo's scan state, and an unfenced park racing the purge would
	// recreate a row nothing ever removes — the purged repo is never
	// listed again. On PG the check holds the base-ledger advisory
	// lock, totally ordering it against FenceRepoPurge: a park ordered
	// before the fence commits before it, so the later ledger cleanup
	// still sweeps the row; a park ordered after sees the fence and
	// no-ops. On SQLite the conditional upsert is one single-writer
	// statement.
	ParkRepoJobScanState(ctx context.Context, job, scope, payload string, generation int64, base string) error
	// ListJobScanStateScopes pages one job's scopes in byte order,
	// keyset from max(after, prefix), bounded to scopes carrying the
	// prefix (Codex round 33 on PR #132): the seam refund-bucket and
	// candidacy rotations page durable populations through.
	ListJobScanStateScopes(ctx context.Context, job, prefix, after string, limit int) ([]string, error)
	// BeginLFSReapEpoch atomically increments and returns the monotone
	// destructive-sweep epoch (see ScanScopeLFSReapEpoch). Called at
	// destructive LFS sweep START, before any blob listing; the write's
	// row lock serializes against in-flight retirements' epoch reads,
	// so every retirement recorded under an older epoch committed
	// before this sweep's listing began.
	BeginLFSReapEpoch(ctx context.Context) (int64, error)
	// RecordLFSReapComplete writes the destructive-reap completion
	// watermark (see ReapWatermark) — and, when legacy-armed, the
	// separate legacy completion scope — after a clean destructive LFS
	// pass.
	RecordLFSReapComplete(ctx context.Context, epoch int64, legacyArmed bool) error
}

ScanStateStore is the surface background jobs persist scan state through. Both concrete backends (PG, SQLite) satisfy it.

type SettledLFSRefunds

type SettledLFSRefunds struct {
	Rows  int64 // object-cap slots refunded
	Bytes int64 // usage bytes refunded
	// Pending counts buckets left unsettled (no covering completion yet).
	Pending int
}

SettledLFSRefunds is one SettleLFSRefunds call's outcome.

type Store

type Store interface {
	RefStore
	// InitSchema creates the ref/pack tables (idempotent, safe under
	// concurrent startups).
	InitSchema(ctx context.Context) error
	Close()
}

Store extends RefStore with the lifecycle the wire seam needs: serve runs InitSchema once at startup and Close on shutdown. Consumers keep coding against RefStore; hooks never run DDL.

func Open

func Open(ctx context.Context, dbURL string) (Store, error)

Open dispatches on the URL scheme to the matching Store backend:

postgres:// or postgresql://  → OpenPG
sqlite:///abs/path/db.sqlite  → pure-Go SQLite store
s3wal://bucket?endpoint=…     → manifest-CAS store over conditional S3

Open never runs InitSchema: hooks reconstruct stores from env URLs and must not run DDL — the serve/wire seam owns schema init (for s3wal that includes the §2.6 conditional-write probe: Open performs no network I/O on that scheme — ADR-12).

Directories

Path Synopsis
Package s3wal is the S3-manifest-WAL RefStore experiment (issue #133, docs/plans/s3wal-plan.md): one JSON manifest object per base repo group, advanced by conditional PUT (If-Match / If-None-Match: *).
Package s3wal is the S3-manifest-WAL RefStore experiment (issue #133, docs/plans/s3wal-plan.md): one JSON manifest object per base repo group, advanced by conditional PUT (If-Match / If-None-Match: *).
s3waltest
Package s3waltest carries the shared s3wal test fixtures: the pinned SeaweedFS/MinIO testcontainers, raw path-style S3 client construction, bucket bootstrap/cleanup helpers, and the Phase 1a {seq, nonce} payload codec.
Package s3waltest carries the shared s3wal test fixtures: the pinned SeaweedFS/MinIO testcontainers, raw path-style S3 client construction, bucket bootstrap/cleanup helpers, and the Phase 1a {seq, nonce} payload codec.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL