Documentation
¶
Overview ¶
Package simtest is the deterministic simulation testing harness (issue #3, ported from waltier's SimStore + DST design): fault-injection decorators for blob.BlobStore and ref.RefStore plus a seeded, serialized multi-node simulation with invariant oracles.
The package is test-only by convention: nothing under cmd/openstorage or the serve wiring imports it. It ships as regular (non _test.go) files so the seed-reproduction CLI (cmd/dst) and the go-test entry points share one implementation.
The fault model distinguishes the two classes that matter for storage correctness:
- clean fault: the operation did NOT reach the backing store; the caller sees an error. The injected error satisfies pgconn's SafeToRetry contract, exactly like the real stores' acquire-stage failures, so ref.IdempotentCAS classifies it as definitive non-execution.
- ambiguous fault: the operation DID execute against the backing store, but its result is discarded and the caller sees an error — the "committed, then the connection dropped" class behind issue #2.
Blob reads additionally support a transient not-found fault (wrapping blob.ErrNotFound) modeling a listed pack vanishing mid-hydration — the class behind issue #6's bounded retry.
Outbound push-lane claim/marker scenario ops and oracle (issue #136): the push_* edition of sync_ops.go over the lane PR #137 added. A minimal forwarder actor drives RequestPush/TryBeginPush/FinishPush cycles against the real meta store (no real git — the transfer is a seeded number of sim steps) with seeded crash injection between claim and finish, seeded transfer outcomes mirroring the worker's failure taxonomy (success / transient re-mark / terminal non-FF rejection). Node crash ops kill the node's in-flight push runs too. The store's clock is the simulation's logical clock, so lease expiry is a pure function of the seed.
Oracles, all against a ground-truth model maintained by the serialized scheduler (reads through the clean oracle meta handle), keyed to the four #136 invariants:
- convergence: refs are abstracted to a per-repo version counter — every push-request bumps the local version, a claim snapshots it (the loopback fetch postdates the claim), and a successful finish covers through its snapshot. At quiescence every accepted request must be covered by a postdating successful run, terminally rejected (non-FF, needs a human), or still CARRIED (pending marker set or an orphaned claim a later steal takes over) — and the final claim+finish per repo converges covered == local (upstream refs equal local refs);
- force monotonicity: the model pending level only coalesces upward until a claim consumes it whole, the force TryBeginPush reports must equal the model mandate (pending == force OR inherited from a dead claim's persisted push_claim_force), and the full-row mirror turns any unmodeled marker downgrade into a violation at the very next observation. A dead claim's force mandate survives on its push_claim_force alone, and the steal inherits it;
- lease safety: at most one unexpired holder per repo (fresh claims only against idle rows, steals only of expired claims naming the exact previous holder, refusals only against live ones), and a stolen holder's FinishPush fails (ErrClaimLost while the thief holds, ErrNotFound after it finished) leaving row and counters untouched;
- lane isolation: the mirror both lanes share (checkUpstreamMirror, sync_ops.go) compares every sync_* AND push_* column after every mutation point in either lane, so a pull-lane transition touching push-lane columns (or vice versa) violates immediately.
Quota-ledger scenario ops and oracle (issues #29/#31): leaked admission reservations (reserve → crash) with sweep-side reclamation, and a per-step consistency check tying the maintained usage counter to ground truth — registered pack blobs plus recorded retained bytes.
repack_ops.go adds the issue-#1 background repack to the simulation (the "pack rows only grow, modulo repack" oracle the harness deferred until repack existed):
- opRepack folds through the node's FAULT-INJECTED stores, exercising clean/ambiguous faults on the fold pipeline (list, upload, replace);
- opRepackRace interleaves two folds deterministically on the CLEAN oracle stores — a mid-race push changes the winner's input set — and asserts the loser installed nothing and deleted exactly its own blob;
- the pack-row oracle (checkPackRows) asserts after every step that a repo's row set only ever GROWS except across a repack, whose exact before→after transition is pinned in-op (atomicity: the observable state is the pre-set or the folded set, never anything between).
Upstream-sync claim-lease scenario ops and oracle (issues #86/#88): a minimal sync actor drives TryBeginSync/FinishSync cycles against the real meta store (no real git — the transfer is a seeded number of sim steps), with seeded crash injection between claim and finish. Node crash ops kill every in-flight run on the node too. The store's clock is the simulation's logical clock (meta.WithNowFunc), so lease expiry is a pure function of the seed.
Oracles, all against a ground-truth model maintained by the serialized scheduler (reads through the clean oracle meta handle):
- lease liveness (the #86 invariant this file exists for): after any claim attempt at sim time T, no row may keep syncing=true with sync_claim_expires_at <= T — an expired claim must be stolen by the very next attempt. Exact (no slack): store and oracle share one injected clock, so there is no skew to absorb;
- steal safety: refusals only against unexpired claims, steals only of expired ones, stolenFrom naming the exact previous holder;
- late-finish choreography: a stolen run's FinishSync must fail (ErrClaimLost while the thief holds, ErrNotFound after the thief finished) and leave the row and counters untouched; an unstolen run's FinishSync must succeed;
- counter/pending integrity: RunCount/SuccessCount/sync_pending mirror the model at every claim/finish boundary (only successful FinishSync calls count; crashed and stolen runs never do);
- quiescence: past every lease, one final claim per repo must succeed (steal or fresh) and finish cleanly — no wedged row survives.
Index ¶
- Constants
- Variables
- type Config
- type FaultClass
- type FaultEvent
- type Injector
- func (in *Injector) Count(op string) int
- func (in *Injector) Counts() map[string]int
- func (in *Injector) FailNext(op string, class FaultClass)
- func (in *Injector) FailNth(op string, n int, class FaultClass)
- func (in *Injector) Fired() []FaultEvent
- func (in *Injector) FiredSince(seq int) ([]FaultEvent, int)
- func (in *Injector) Seq() int
- func (in *Injector) SetProbs(op string, p Probs)
- type Probs
- type RefBackend
- type Result
- type SimBlobStore
- func (s *SimBlobStore) Close() error
- func (s *SimBlobStore) Copy(ctx context.Context, dstKey, srcKey string) error
- func (s *SimBlobStore) Delete(ctx context.Context, key string) error
- func (s *SimBlobStore) DownloadFile(ctx context.Context, key, destPath string) error
- func (s *SimBlobStore) DownloadFileCapped(ctx context.Context, key, destPath string, capBytes int64) error
- func (s *SimBlobStore) EnsureBucket(ctx context.Context) error
- func (s *SimBlobStore) List(ctx context.Context, prefix string) ([]blob.ListedObject, error)
- func (s *SimBlobStore) ListPages(ctx context.Context, prefix string, pageSize int, ...) error
- func (s *SimBlobStore) ListPagesAfter(ctx context.Context, prefix, after string, pageSize int, ...) error
- func (s *SimBlobStore) ReadRange(ctx context.Context, key string, offset, length int64) ([]byte, error)
- func (s *SimBlobStore) SignedURL(ctx context.Context, key, method string, expiry time.Duration) (string, error)
- func (s *SimBlobStore) Stat(ctx context.Context, key string) (int64, error)
- func (s *SimBlobStore) StatFingerprint(ctx context.Context, key string) (int64, string, error)
- func (s *SimBlobStore) UploadFile(ctx context.Context, key, path string) error
- type SimCAS
- func (c *SimCAS) Advance(ctx context.Context, key string, body []byte, etag string) (string, error)
- func (c *SimCAS) Create(ctx context.Context, key string, body []byte) (string, error)
- func (c *SimCAS) Delete(ctx context.Context, key string) error
- func (c *SimCAS) Get(ctx context.Context, key string) (s3wal.Object, error)
- func (c *SimCAS) GetIfChanged(ctx context.Context, key, etag string) (s3wal.Object, error)
- func (c *SimCAS) List(ctx context.Context, prefix, startAfter string, limit int) ([]string, error)
- func (c *SimCAS) StoreDate() (date, observedAt time.Time, ok bool)
- type SimRefStore
- func (s *SimRefStore) BumpLFSAdoptionGen(ctx context.Context, repo string) error
- func (s *SimRefStore) CASRef(ctx context.Context, repo, name, old, new string) (bool, error)
- func (s *SimRefStore) CASRefGen(ctx context.Context, repo, name, old, new string) (bool, int64, error)
- func (s *SimRefStore) ClearLFSMismatch(ctx context.Context, repo, key string) error
- func (s *SimRefStore) CountLFSObjects(ctx context.Context, repo string) (int64, error)
- func (s *SimRefStore) GetGeneration(ctx context.Context, repo string) (int64, error)
- func (s *SimRefStore) GetRef(ctx context.Context, repo, name string) (string, error)
- func (s *SimRefStore) GuardedLFSDelete(ctx context.Context, repo, key string, del func(context.Context) error) (bool, error)
- func (s *SimRefStore) HasLFSObject(ctx context.Context, repo, key string) (bool, error)
- func (s *SimRefStore) HasPack(ctx context.Context, repo, key string) (bool, error)
- func (s *SimRefStore) IsRetainedPack(ctx context.Context, repo, key string) (bool, error)
- func (s *SimRefStore) LFSAdoptionGen(ctx context.Context, repo string) (int64, error)
- func (s *SimRefStore) LFSMismatchFingerprint(ctx context.Context, repo, key string) (string, bool, error)
- func (s *SimRefStore) LFSObjectRegisteredAt(ctx context.Context, repo, key string) (time.Time, int64, bool, error)
- func (s *SimRefStore) ListPacks(ctx context.Context, repo string) ([]ref.Pack, error)
- func (s *SimRefStore) ListRefs(ctx context.Context, repo string) (map[string]string, error)
- func (s *SimRefStore) ListRefsAfter(ctx context.Context, repo, after string, limit int) ([]ref.RefEntry, error)
- func (s *SimRefStore) ListRefsWithMeta(ctx context.Context, repo string) ([]ref.RefInfo, error)
- func (s *SimRefStore) PackRegisteredBytes(ctx context.Context, repo, key string) (int64, bool, error)
- func (s *SimRefStore) QuotaLedgerState(ctx context.Context, group []string, ttl time.Duration, excludeKeys ...string) (ref.PackQuotaLedgerState, error)
- func (s *SimRefStore) ReclaimRetainedPack(ctx context.Context, repo, key string) (int64, error)
- func (s *SimRefStore) ReconcileLegacyLFSObjects(ctx context.Context) (int64, bool, error)
- func (s *SimRefStore) RecordLFSMismatch(ctx context.Context, repo, key, fingerprint string) error
- func (s *SimRefStore) RegisterAdoptedLFSObject(ctx context.Context, repo, key string, bytes int64) error
- func (s *SimRefStore) RegisterLFSObject(ctx context.Context, repo, key string, bytes int64) error
- func (s *SimRefStore) RegisterPacks(ctx context.Context, repo string, packs []ref.Pack) error
- func (s *SimRefStore) ReleaseAdoptionClaim(ctx context.Context, key, owner string) error
- func (s *SimRefStore) RenewAdoptionClaim(ctx context.Context, key, owner string) (bool, error)
- func (s *SimRefStore) ReplacePacks(ctx context.Context, repo string, old []ref.Pack, newPack ref.Pack) (bool, error)
- func (s *SimRefStore) ReservePackBytes(ctx context.Context, group []string, packs []ref.Pack, owner string, ...) (ref.ReserveDecision, error)
- func (s *SimRefStore) RetirePackAttempts(ctx context.Context, group []string, gens map[string]string) error
- func (s *SimRefStore) SweepExpiredPackReservations(ctx context.Context, olderThan time.Duration) (int64, error)
- func (s *SimRefStore) TouchLFSObject(ctx context.Context, repo, key string) (bool, error)
- func (s *SimRefStore) TryAdoptionClaim(ctx context.Context, repo, key string) (bool, string, error)
Constants ¶
const ( OpRefListRefs = "ref.ListRefs" OpRefListRefsWithMeta = "ref.ListRefsWithMeta" OpRefGetRef = "ref.GetRef" OpRefCASRef = "ref.CASRef" OpRefRegisterPacks = "ref.RegisterPacks" OpRefReplacePacks = "ref.ReplacePacks" OpRefListPacks = "ref.ListPacks" OpRefGetGeneration = "ref.GetGeneration" OpRefReservePacks = "ref.ReservePackBytes" OpBlobUploadFile = "blob.UploadFile" OpBlobDownloadFile = "blob.DownloadFile" OpBlobList = "blob.List" OpBlobSignedURL = "blob.SignedURL" OpBlobStat = "blob.Stat" OpBlobEnsureBucket = "blob.EnsureBucket" OpBlobDelete = "blob.Delete" OpBlobCopy = "blob.Copy" // s3wal conditional-store transport ops (SimCAS, RefBackendS3WAL): // these sit BELOW the driver, so their faults exercise the driver's // own §2.3 recovery machinery rather than ref.IdempotentCAS. OpCASGet = "cas.Get" OpCASCreate = "cas.Create" OpCASAdvance = "cas.Advance" OpCASList = "cas.List" )
Operation names the decorators inject on. The ref/blob prefixes keep one flat namespace so a single Injector can serve both decorators.
Variables ¶
var ( // ErrInjectedClean marks a clean fault: the op did not execute. ErrInjectedClean = errors.New("simtest: injected clean fault (op did not execute)") // ErrInjectedAmbiguous marks an ambiguous fault: the op executed but // the result was lost. ErrInjectedAmbiguous = errors.New("simtest: injected ambiguous fault (op executed, result lost)") )
Sentinel errors for injected faults; match with errors.Is. The concrete returned errors carry the op name.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
Seed uint64
Steps int // scheduler steps (default 150)
Nodes int // simulated server nodes sharing one SQLite+fileblob (default 3)
Repos int // simulated repos (default 2)
Branches int // branches per repo (default 3)
// RefBackend selects the authority store under test (default
// RefBackendSQLite). The meta store (sync/push claim lanes) and the
// fileblob store are backend-independent and stay SQLite/file.
RefBackend RefBackend
// FullCheckEvery runs the expensive cold-hydrate oracle every N steps
// (default 50). It always also runs once after the last step.
FullCheckEvery int
// Dir is the run's scratch root; empty means a fresh os.MkdirTemp that
// is removed afterwards unless KeepDir is set.
Dir string
KeepDir bool
// Trace, when non-nil, receives the live trace lines (the same bytes
// that feed TraceHash).
Trace io.Writer
}
Config parameterizes one deterministic simulation run. The zero value of every field defaults sensibly (see withDefaults); Seed 0 is a valid seed.
type FaultClass ¶
type FaultClass int
FaultClass selects what an injected fault does to the decorated call.
const ( // FaultNone means the call proceeds to the backing store untouched. FaultNone FaultClass = iota // FaultClean fails the call WITHOUT executing it. FaultClean // FaultAmbiguous executes the call, discards its result, and fails it. FaultAmbiguous // FaultNotFound fails a blob op with blob.ErrNotFound without // executing it (transient missing-object; only valid on // OpBlobDownloadFile, OpBlobStat, OpBlobDelete, and OpBlobCopy). FaultNotFound )
func (FaultClass) String ¶
func (c FaultClass) String() string
String returns the fault class name for traces.
type FaultEvent ¶
type FaultEvent struct {
Seq int // 1-based injector-wide sequence number of the faulted call
Op string
Class FaultClass
// InnerOutcome describes the backing store's real result for ambiguous
// faults (e.g. "won=true err=<nil>"); empty for non-executed classes.
InnerOutcome string
}
FaultEvent records one injected fault, for traces and oracle bookkeeping.
type Injector ¶
type Injector struct {
// contains filtered or unexported fields
}
Injector decides, per decorated call, whether to inject a fault. One injector is shared by all decorators of a scenario: a single seeded PRNG stream plus FIFO one-shot plans keep decisions deterministic as long as calls are serialized (the DST scheduler runs one op at a time; the decorators still lock so misuse cannot corrupt state).
func NewInjector ¶
NewInjector returns an Injector seeded with seed. The zero probability state injects nothing until FailNext/FailNth/SetProbs are called.
func (*Injector) Count ¶
Count returns how many calls of op the decorators have seen (including faulted ones).
func (*Injector) FailNext ¶
func (in *Injector) FailNext(op string, class FaultClass)
FailNext queues a one-shot fault on the next call of op.
func (*Injector) FailNth ¶
func (in *Injector) FailNth(op string, n int, class FaultClass)
FailNth queues a one-shot fault on the n-th call of op counted from now (n=1 is the next call). Queued plans fire in FIFO order: each plan's countdown starts only after the previous plan for the same op has fired.
func (*Injector) Fired ¶
func (in *Injector) Fired() []FaultEvent
Fired returns a copy of every injected fault so far, in injection order.
func (*Injector) FiredSince ¶
func (in *Injector) FiredSince(seq int) ([]FaultEvent, int)
FiredSince returns injected faults with Seq > seq, plus the current high-water mark to pass back next time (per-step fault attribution). It does not mutate injector state, so overlapping windows are fine.
type Probs ¶
Probs are per-op fault probabilities; the remainder is FaultNone. Draws happen once per matching call in call order, so a fixed seed yields a fixed decision sequence under serialized execution.
type RefBackend ¶
type RefBackend string
RefBackend names the authority-store lane a simulation runs on.
const ( // RefBackendSQLite is the default lane: every node opens the one // shared SQLite database file. RefBackendSQLite RefBackend = "sqlite" // RefBackendS3WAL runs the s3wal manifest-CAS driver over one SHARED // in-memory conditional store (plan §4 Phase 2's multi-node // shared-fake topology). This is stricter than the backend's // single-node deployment decree on purpose: it exercises every // PROTOCOL race between manifest writers, not deployment races. RefBackendS3WAL RefBackend = "s3wal" )
type Result ¶
type Result struct {
Seed uint64
Backend RefBackend // authority-store lane the run executed on
Steps int // steps executed (< Config.Steps when a violation stopped the run)
TraceHash string
Violations []string
Transitions int // distinct committed ref transitions observed
Deliveries int // webhook deliveries recorded
Faults int // faults injected
// CASFaults is the subset of Faults DELIVERED at the s3wal
// conditional-store transport layer (cas.* ops; zero on the SQLite
// backend; store-rejected attempts excluded) — the vacuity guard for
// the Phase 2 fault plan. CASFaultOps breaks it down per op so every
// advertised surface can be required to FIRE, not merely be called.
CASFaults int
CASFaultOps map[string]int
Steals int // expired sync claims stolen (issues #86/#88)
// CrashSteals is the subset of Steals whose previous holder actually
// crashed mid-run (dead-holder class) rather than outliving its lease.
CrashSteals int
// PushSteals/PushCrashSteals are the outbound-lane counterparts
// (issue #136); PushForcedRescues counts steals that inherited a force
// mandate carried only by the dead claim's push_claim_force.
PushSteals int
PushCrashSteals int
PushForcedRescues int
OpCounts map[string]int
}
Result summarizes one run. Violations non-empty means an invariant broke; TraceHash is a digest of the full deterministic trace (two runs with the same Config must produce identical hashes).
type SimBlobStore ¶
type SimBlobStore struct {
// contains filtered or unexported fields
}
SimBlobStore decorates a blob.BlobStore with seeded fault injection. Uploads support the clean/ambiguous split (an ambiguous upload publishes the object, then loses the result — the "S3 PUT succeeded, response dropped" class); downloads and stats additionally support a transient FaultNotFound wrapping blob.ErrNotFound, the class issue #6's bounded hydration retry absorbs.
func NewSimBlobStore ¶
func NewSimBlobStore(inner blob.BlobStore, inj *Injector) *SimBlobStore
NewSimBlobStore wraps inner with fault injection driven by inj.
func (*SimBlobStore) Close ¶
func (s *SimBlobStore) Close() error
Close always delegates: lifecycle teardown is never fault-injected (a crash is simulated by dropping the handle, not by failing Close).
func (*SimBlobStore) Copy ¶
func (s *SimBlobStore) Copy(ctx context.Context, dstKey, srcKey string) error
Copy delegates unless a fault is injected; an ambiguous fault performs the copy and then reports failure (a quarantine copy that landed but whose result was lost); FaultNotFound reports a missing source.
func (*SimBlobStore) Delete ¶
func (s *SimBlobStore) Delete(ctx context.Context, key string) error
Delete delegates unless a fault is injected; an ambiguous fault deletes the object and then reports failure (the sweeper's delete outcome lost in transit); FaultNotFound reports the object as already missing.
func (*SimBlobStore) DownloadFile ¶
func (s *SimBlobStore) DownloadFile(ctx context.Context, key, destPath string) error
DownloadFile delegates unless a fault is injected:
- clean: no download happens, transport-shaped error.
- ambiguous: the file IS materialized at destPath, then an error is reported (callers must tolerate the atomic download landing).
- notfound: blob.ErrNotFound without executing — a listed pack transiently missing, which hydration must absorb by re-listing.
func (*SimBlobStore) DownloadFileCapped ¶
func (s *SimBlobStore) DownloadFileCapped(ctx context.Context, key, destPath string, capBytes int64) error
DownloadFileCapped shares DownloadFile's fault point: the cap is a local disk-budget concern, not a distinct backend operation.
func (*SimBlobStore) EnsureBucket ¶
func (s *SimBlobStore) EnsureBucket(ctx context.Context) error
EnsureBucket delegates to the inner store unless a fault is injected.
func (*SimBlobStore) List ¶
func (s *SimBlobStore) List(ctx context.Context, prefix string) ([]blob.ListedObject, error)
List delegates to the inner store unless a fault is injected.
func (*SimBlobStore) ListPages ¶
func (s *SimBlobStore) ListPages(ctx context.Context, prefix string, pageSize int, fn func([]blob.ListedObject) error) error
ListPages delegates to the inner store unless a fault is injected; it shares OpBlobList with List (one listing surface, one fault point).
func (*SimBlobStore) ListPagesAfter ¶ added in v0.20.0
func (s *SimBlobStore) ListPagesAfter(ctx context.Context, prefix, after string, pageSize int, fn func([]blob.ListedObject) error) error
ListPagesAfter is ListPages from the first key after after, with the same fault point: the helper uses the inner store's own ListPagesAfter if it has one.
func (*SimBlobStore) ReadRange ¶
func (s *SimBlobStore) ReadRange(ctx context.Context, key string, offset, length int64) ([]byte, error)
ReadRange delegates unless a fault is injected. It deliberately shares OpBlobStat with Stat rather than adding a fault point: both are the repack selection's per-candidate point reads (size, pack-header object count) with identical failure handling — one metadata surface, one fault point (the List/ListPages precedent). FaultNotFound reports the object as missing via blob.ErrNotFound.
func (*SimBlobStore) SignedURL ¶
func (s *SimBlobStore) SignedURL(ctx context.Context, key, method string, expiry time.Duration) (string, error)
SignedURL delegates to the inner store unless a fault is injected.
func (*SimBlobStore) Stat ¶
Stat delegates unless a fault is injected; FaultNotFound reports the object as missing via blob.ErrNotFound.
func (*SimBlobStore) StatFingerprint ¶
StatFingerprint delegates unless a fault is injected; it shares Stat's fault op — it IS a stat with a version fingerprint attached.
func (*SimBlobStore) UploadFile ¶
func (s *SimBlobStore) UploadFile(ctx context.Context, key, path string) error
UploadFile delegates unless a fault is injected; an ambiguous fault uploads the object and then reports failure.
type SimCAS ¶
type SimCAS struct {
// contains filtered or unexported fields
}
SimCAS is one node's fault-injecting transport view of the SHARED in-memory conditional store (s3wal.MemCAS) — the s3wal DST fault plan (plan §4 Phase 2). It sits BELOW the manifest driver, so its faults exercise the driver's own mutate/recover machinery (§2.3), while the SimRefStore decorator above the driver keeps exercising ref.IdempotentCAS — both recovery layers run in one simulation. Rebuilt on node crash, like the process it stands for.
Fault classes per op (drawn from the shared Injector, so decisions are deterministic under the serialized scheduler):
- cas.Create / cas.Advance: FaultClean returns s3wal.ErrNotExecuted without executing (the provably-never-sent class); FaultAmbiguous splits deterministically into applied (execute, discard the result, report *s3wal.AmbiguousError — the §2.3 committed-response-lost window) and dropped (never execute, same error).
- cas.Get / cas.List: FaultClean fails the read without executing. Reads are idempotent, so a lost response is indistinguishable from a failed request — one class covers both.
Stale reads are NOT injected (issue #216). The s3wal store contract requires strong read-after-write consistency (plan §1/§2.6), the driver trusts every unconditional read by construction, and its seq watermark only catches a read that regresses below this process's own last observation — so a served-stale manifest yields a silently wrong read-only verdict (a sweep that "finds nothing to prune", a CAS that "lost") with no conditional PUT to expose it. Modelling that class makes the exact-state oracles unsatisfiable by any read-only path; the retired one-shot arm (PR #159) drove precisely that violation on seed 833534285 (pinned in dstRegressions). The 412 lost-race recovery the arm was meant to trigger is exercised dozens of times per run through the multi-node seed cache (issue #182): every cross-node write leaves the other nodes' cached snapshots stale, and their next seeded PUT 412s.
409-exhaustion (s3wal.ErrCASContention) is deliberately not injected: genuine CAS contention arises naturally from the multi-node shared-fake topology, and the exhaustion taxonomy is unit-tested in internal/ref. Deletes pass through unfaulted: the protocol never guards DELETEs (§2.7), and the W6 delete-lease races are unit-tested in internal/ref.
func (*SimCAS) GetIfChanged ¶
GetIfChanged fetches key unless etag is current, possibly failing clean.
type SimRefStore ¶
type SimRefStore struct {
// contains filtered or unexported fields
}
SimRefStore decorates a ref.RefStore with seeded clean/ambiguous fault injection. It sits BELOW ref.IdempotentCAS in the decorator chain (wrap SimRefStore in IdempotentCAS, or hand it to events.NewRefStore, which applies IdempotentCAS itself), so injected ambiguous CAS errors exercise the issue-#2 recovery path exactly where real transport errors surface.
func NewSimRefStore ¶
func NewSimRefStore(inner ref.RefStore, inj *Injector) *SimRefStore
NewSimRefStore wraps inner with fault injection driven by inj.
func (*SimRefStore) BumpLFSAdoptionGen ¶
func (s *SimRefStore) BumpLFSAdoptionGen(ctx context.Context, repo string) error
BumpLFSAdoptionGen delegates to the inner store.
func (*SimRefStore) CASRef ¶
CASRef injects the fault classes that matter for issue #2:
- clean: the statement never executes; the returned error satisfies SafeToRetry, so IdempotentCAS must propagate it un-recovered.
- ambiguous: the statement executes (the ref may really move) but the caller sees a transport-shaped error; the real outcome is recorded on the FaultEvent for oracle bookkeeping.
func (*SimRefStore) CASRefGen ¶
func (s *SimRefStore) CASRefGen(ctx context.Context, repo, name, old, new string) (bool, int64, error)
CASRefGen mirrors CASRef's fault split; the generation attribution is lost with the outcome on ambiguous faults (gen == 0, the unattributed contract), exactly as a real lost response loses it.
func (*SimRefStore) ClearLFSMismatch ¶
func (s *SimRefStore) ClearLFSMismatch(ctx context.Context, repo, key string) error
func (*SimRefStore) CountLFSObjects ¶
CountLFSObjects delegates to the inner store.
func (*SimRefStore) GetGeneration ¶
GetGeneration delegates to the inner store unless a fault is injected. Faulting it starves the issue-#4 hydrate short-circuit, which must then surface the error rather than serve from a stale marker.
func (*SimRefStore) GetRef ¶
GetRef delegates to the inner store unless a fault is injected. Faulting it starves ref.IdempotentCAS's recovery recheck, which must then fall back to the original ambiguous outcome.
func (*SimRefStore) GuardedLFSDelete ¶
func (s *SimRefStore) GuardedLFSDelete(ctx context.Context, repo, key string, del func(context.Context) error) (bool, error)
GuardedLFSDelete delegates to the inner store.
func (*SimRefStore) HasLFSObject ¶
HasLFSObject delegates to the inner store.
func (*SimRefStore) HasPack ¶
HasPack delegates to the inner store unless a fault is injected; it shares OpRefListPacks with ListPacks (one registry-read fault point).
func (*SimRefStore) IsRetainedPack ¶
IsRetainedPack delegates to the inner store.
func (*SimRefStore) LFSAdoptionGen ¶
LFSAdoptionGen delegates to the inner store.
func (*SimRefStore) LFSMismatchFingerprint ¶
func (*SimRefStore) LFSObjectRegisteredAt ¶
func (s *SimRefStore) LFSObjectRegisteredAt(ctx context.Context, repo, key string) (time.Time, int64, bool, error)
LFSObjectRegisteredAt delegates to the inner store.
func (*SimRefStore) ListRefsAfter ¶
func (s *SimRefStore) ListRefsAfter(ctx context.Context, repo, after string, limit int) ([]ref.RefEntry, error)
ListRefsAfter delegates to the inner store unless a fault is injected. It shares OpRefListRefs with ListRefs: both are the same ref-enumeration read surface, just bounded vs whole-repo (issue #119).
func (*SimRefStore) ListRefsWithMeta ¶
ListRefsWithMeta delegates to the inner store unless a fault is injected.
func (*SimRefStore) PackRegisteredBytes ¶
func (s *SimRefStore) PackRegisteredBytes(ctx context.Context, repo, key string) (int64, bool, error)
PackRegisteredBytes delegates to the inner store.
func (*SimRefStore) QuotaLedgerState ¶
func (s *SimRefStore) QuotaLedgerState(ctx context.Context, group []string, ttl time.Duration, excludeKeys ...string) (ref.PackQuotaLedgerState, error)
QuotaLedgerState delegates to the inner store (oracle reads).
func (*SimRefStore) ReclaimRetainedPack ¶
ReclaimRetainedPack delegates to the inner store.
func (*SimRefStore) ReconcileLegacyLFSObjects ¶
func (*SimRefStore) RecordLFSMismatch ¶
func (s *SimRefStore) RecordLFSMismatch(ctx context.Context, repo, key, fingerprint string) error
func (*SimRefStore) RegisterAdoptedLFSObject ¶
func (s *SimRefStore) RegisterAdoptedLFSObject(ctx context.Context, repo, key string, bytes int64) error
RegisterAdoptedLFSObject delegates to the inner store.
func (*SimRefStore) RegisterLFSObject ¶
RegisterLFSObject delegates to the inner store (the DST does not model LFS transfers yet; delegation keeps the interface honest without adding an unexercised fault point).
func (*SimRefStore) RegisterPacks ¶
RegisterPacks supports the same clean/ambiguous split as CASRef: an ambiguous fault registers the packs (idempotently) and then loses the result.
func (*SimRefStore) ReleaseAdoptionClaim ¶
func (s *SimRefStore) ReleaseAdoptionClaim(ctx context.Context, key, owner string) error
func (*SimRefStore) RenewAdoptionClaim ¶
func (*SimRefStore) ReplacePacks ¶
func (s *SimRefStore) ReplacePacks(ctx context.Context, repo string, old []ref.Pack, newPack ref.Pack) (bool, error)
ReplacePacks supports the same clean/ambiguous split as CASRef: an ambiguous fault executes the swap (which may really commit) and then loses the outcome, exercising the repack job's leave-the-blob handling.
func (*SimRefStore) ReservePackBytes ¶
func (s *SimRefStore) ReservePackBytes(ctx context.Context, group []string, packs []ref.Pack, owner string, usageFloor, floorGen, quotaBytes int64, ttl time.Duration) (ref.ReserveDecision, error)
ReservePackBytes supports the same clean/ambiguous split as CASRef: an ambiguous fault runs the admission (a reservation row may really be inserted — the leaked-reservation class TTL expiry and the sweeper reclaim) and then loses the outcome.
func (*SimRefStore) RetirePackAttempts ¶
func (s *SimRefStore) RetirePackAttempts(ctx context.Context, group []string, gens map[string]string) error
RetirePackAttempts delegates to the inner store (no fault point: a failed retire only leaves a TTL-expiring row, exactly what the crash ops already exercise).
func (*SimRefStore) SweepExpiredPackReservations ¶
func (s *SimRefStore) SweepExpiredPackReservations(ctx context.Context, olderThan time.Duration) (int64, error)
SweepExpiredPackReservations delegates to the inner store.
func (*SimRefStore) TouchLFSObject ¶
TouchLFSObject delegates to the inner store.
func (*SimRefStore) TryAdoptionClaim ¶
TryAdoptionClaim / ReleaseAdoptionClaim pass through without fault injection: adoption claims are an advisory cross-replica dedup (Codex security round 73), not lifecycle state the oracle models.