simtest

package
v0.19.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 30, 2026 License: Apache-2.0 Imports: 28 Imported by: 0

README

simtest — deterministic simulation testing (DST)

Overview

Deterministic, seeded simulation of a multi-node open.storage cluster with fault injection, ported from waltier's SimStore + DST design (issue #3). Everything runs in one process — zero containers — on two authority-store lanes (Config.RefBackend): the shared SQLite file (default) and the s3wal manifest-CAS driver over one shared in-memory conditional store (s3wal.MemCAS), the plan §4 Phase 2 lane. The s3wal lane's multi-node shared-fake topology is deliberately stricter than that backend's single-node deployment decree: every PROTOCOL race between manifest writers is simulated. The meta store (sync/push claim lanes) and fileblob are backend-independent.

What it exercises

A seeded scheduler serializes logically-concurrent ops against N simulated nodes that share one SQLite ref DB, one SQLite meta DB, and one fileblob directory (each node has its own handles and cache dir, like separate processes on a host):

  • pushes / API writes — deterministic commits minted with git plumbing, driven through the real pipeline: blob upload → RegisterPacks → CASRef, with the CAS flowing through the production events.RefStore → ref.IdempotentCAS chain;
  • ref deletes, node hydrations (githttp.Hydrate), mid-push crashes (after upload / after register), node crash-restarts (warm and cold);
  • background repacks (issue #1, jobs.Repacker): plain folds through a node's fault-injected stores, plus a deterministic two-folder race (repack-race) where a mid-race push forces the competing folds apart and the loser must delete exactly its own uploaded blob;
  • upstream-sync claim leases (issues #86/#88, sync-claim / sync-request): a minimal sync actor drives TryBeginSync/FinishSync cycles against the real meta store (the transfer is a seeded number of sim steps, no real git), with seeded crash injection between claim and finish; node crashes kill the node's in-flight runs too. The meta store runs on the simulation's logical clock (meta.WithNowFunc), so lease expiry is a pure function of the seed;
  • outbound push-forward claim leases and pending markers (issue #136, push-request / push-claim): the push_* lane PR #137 added, driven the same way (RequestPush/TryBeginPush/FinishPush against the real meta store on the logical clock) with seeded transfer outcomes mirroring the worker's failure taxonomy — success, transient failure (the actor re-marks the durable pending level with the run's force, like schedulePushRetry), terminal non-FF rejection (no retry);
  • quota reservations (issues #29/#31, reserve-crash / reserve-sweep): a ReservePackBytes admission whose pusher then crashes, which leaks the reservation, and a sweep that prunes every reservation through the node's fault-injected stores.

Storage calls pass through SimRefStore/SimBlobStore, which inject:

  • clean faults — the op never executes; the error satisfies pgconn's SafeToRetry, like the real stores' acquire-stage failures;
  • ambiguous faults — the op executes, the result is lost (the class behind issue #2). This holds for mutating ops and for blob downloads (DownloadFile, DownloadFileCapped). Ref-store reads such as ListRefs, GetRef, and ListPacks, and blob List, skip the inner call under an ambiguous fault, exactly like a clean fault;
  • transient not-found faults — a pack download reports blob.ErrNotFound once (the vanished-pack class behind issue #6).

On the s3wal lane a second injection layer sits BELOW the driver: each node's SimCAS transport view of the shared conditional store injects clean read/list failures, never-executed and ambiguous(applied|dropped) conditional PUTs — exercising the driver's own §2.3 mutate/recover machinery under the same run that exercises ref.IdempotentCAS above it.

Stale reads are deliberately not in the fault plan. The s3wal driver is written against a store with strong read-after-write consistency (plan §1/§2.6; a documented deployment prerequisite — AWS S3 guarantees it since December 2020), so it trusts every unconditional read; its seq watermark only catches a read that regresses below the process's own last observation. A stale manifest therefore yields a silently wrong read-only verdict — a sweep that "finds nothing to prune", a CAS that "lost" — with no conditional PUT to expose it, and under such a fault the exact-state oracles are unsatisfiable by any read-only path. PR #159's one-shot stale first read on half the node reboots modelled exactly that store class and drove the issue-#216 violation (seed 833534285, pinned in TestDSTRegressions); it was retired. The 412 lost-race recovery it was meant to trigger needs no injection: the multi-node seed cache (issue #182) leaves every other node's cached snapshot stale after each cross-node write, so their next seeded PUT 412s and re-reads — measured at 34–59 real 412s per 300-step seed with the arm gone, versus 7–69 with it (the arm added a handful of unseeded 412s that walk the same mutate path).

Invariant oracles

Checked from clean (never fault-injected) handles:

  1. (a) every committed ref resolves through registered, downloadable packs: periodic + final cold hydrate on a fresh node must succeed and pass git fsck --connectivity-only;
  2. (b) a cold node reproduces the exact ref state: hydrated for-each-ref must mirror the ref authority;
  3. (c) no packs row references a missing blob (checked every step);
  4. step-level CAS atomicity, the issue-#2 recovery contract (ambiguous-but-committed create/update with an unstarved recheck must be reported as a win; a clean conflict must never hide a committed own-write), and hydration absorbing transient not-founds (issue #6);
  5. webhook soundness/completeness: every delivery matches a committed transition; every reported win emitted at least one delivery;
  6. pack rows only grow, modulo repack (issue #1, checked every step): outside a repack a registered pack row never disappears or changes key; a repack's before→after transition is pinned in-op — the observable row set is exactly the pre-set or exactly the folded set (replacement atomicity), a lost race changes nothing and leaks no blob, and superseded blobs outlive the fold (the sweeper owns them);
  7. no sync claim outlives its lease without progress (issues #86/#88, sync_ops.go): after any claim attempt at sim time T, no repo_upstreams row keeps syncing=true with an expiry ≤ T — an expired claim must be stolen by the very next attempt; steals only take expired claims (never live ones) and name the exact previous holder; a stolen run's late FinishSync fails (ErrClaimLost while the thief holds, ErrNotFound after it finished); after every mutation point (claim attempt, request, finish, late finish, quiescence) the row is mirrored field-for-field against a ground-truth model — claim state, pending marker, and both counters — so any unmodeled change violates at the next observation; and after quiescence (clock jumped past every lease) one final claim per repo must succeed and finish cleanly — a refusal is exactly the pre-#86 wedged row;
  8. sync-forward oracle over the outbound lane (issue #136, push_ops.go), the four #136 invariants: convergence — refs are abstracted to a per-repo version counter (requests bump it, claims snapshot it, successful finishes cover through their snapshot); at quiescence every accepted request must be covered by a postdating successful run, terminally rejected, or still carried by the pending marker or an orphaned claim, and the final claim+finish converges covered == local; force monotonicity — the pending level only coalesces upward until a claim consumes it whole, the force TryBeginPush reports must equal the model mandate (marker force OR inherited from a dead claim's persisted push_claim_force), so a dead claim's mandate survives on the claim row alone; lease safety — the #88 liveness/steal/late-finish choreography, push edition; lane isolation — one shared full-row mirror (checkUpstreamMirror) compares every sync_* AND push_* column after every mutation point in either lane, so a pull-lane transition touching push-lane columns (or vice versa) violates at the next observation.

Entry points

What How
PR gate (fast) TestDSTFast, TestDSTDeterminism, TestDSTRegressions (the pinned-seed corpus) — run in the normal go test ./... pipeline, one subtest per backend lane
Reproduce a seed DST_SEED=<seed> [DST_STEPS=<n>] [DST_REF_BACKEND=s3wal] go test ./internal/simtest -run 'TestDSTSeed' -v -count=1
CLI repro go run ./cmd/dst -seed <seed> [-steps N] [-backend s3wal] [-v] [-keep]
Nightly sweep .github/workflows/dst-nightly.yml → TestDSTSweep (DST_SWEEP_SEEDS, DST_SWEEP_BASE, DST_STEPS, DST_REF_BACKEND — one job per backend lane)

Triaging a failing seed

  1. A failing run prints the seed, the step count, and the violation. Replay it exactly:

    go run ./cmd/dst -seed <seed> -steps <steps> -v
    

    -v streams the trace: one line per op with the injected faults, the op's outcome, and the ground-truth tip after it. The violation names the step and invariant.

  2. Determinism means the replay is exact — same commits, same faults, same outcome (the trace hash is stable). Narrow with -steps (runs stop at the first violation anyway) and inspect surviving state with -keep, which prints the run directory (SQLite DB, blob dir, node caches).

  3. The trace's faults=[...] field attributes every injected fault to the op that consumed it; ref.CASRef/ambiguous(won=true err=<nil>) means the CAS committed under an ambiguous fault — the #2 recovery path.

Validating the oracle (mutation test)

The harness must catch re-broken variants of the fixes it guards. Locally (never commit these):

  • revert #2: make ambiguousError in internal/ref/idempotent.go return false — the sweep reports an issue-#2 class violation;
  • revert #6: make HydratePacks in internal/githttp/hydrate.go fail on blob.ErrNotFound instead of re-listing — the sweep reports an issue #6 class violation;
  • revert #86: make TryBeginSync in internal/meta/sqlite_eventing.go refuse every syncing=true row regardless of expiry (change the refusal guard to if prev.Syncing {) — the sweep reports an issue-#88 lease liveness violation on the first claim attempt after a crashed holder's lease expires (and an issue-#88 quiescence violation on seeds whose crashes go unprobed in-run);
  • break the #136 rescue semantics: make TryBeginPush in internal/meta/sqlite_eventing.go not inherit the dead claim's force (change the mandate to force = prev.PushPending == 2) — the sweep reports an issue-#136 force monotonicity violation on the first steal of a crashed forced claim;
  • break #136 lane isolation: make FinishSync also reset a push-lane column (e.g. add push_claim_force = 0 to its UPDATE) — the shared full-row mirror reports a push-claim-state divergence at the next observation after a sync finish overlapping a forced push claim;
  • break #136 marker monotonicity: make RequestPush overwrite instead of coalesce (SET push_pending = ? instead of MAX(push_pending, ?)) — the mirror reports a pending-level divergence at the first plain request or rescue re-mark that lands on an unconsumed force marker;
  • break the s3wal ambiguous-PUT recovery: in mutate (internal/ref/s3walstore.go) replace the fresh.Writer == ourNonce identity check with true (and drop its seq assert) so every ambiguous re-read is treated as an own-write — the s3wal lane reports a reported won but tip= violation on the first ambiguous-DROPPED conditional PUT under a CAS (detecting run: DST_SEED=3 DST_STEPS=200 DST_REF_BACKEND=s3wal, violation at step 131).

The first two were validated when this harness landed, the #86 one when the sync oracle landed, and the #136 ones when the push-lane oracle landed; the landing PRs document the detecting seeds.

Determinism rules

Same seed ⇒ same trace hash. Preserved by: PCG-seeded scheduler/injector streams (draw order fixed), serialized op execution, pinned git identity and logical-clock dates (commit SHAs are pure functions of the seed), sequential pack ids, sorted iteration in checks, and path-free trace lines. When extending the harness, never iterate a Go map into scheduler decisions or trace lines, never read wall-clock time, and route every random choice through the two seeded streams.

Interface

  • Run — runs one deterministic simulation; infrastructure errors return as error, invariant breaches as Result.Violations [scenario.go:Run]
  • Config, Result — run inputs (seed, steps, nodes, repos, branches, RefBackend, FullCheckEvery, Dir, KeepDir, Trace) and outputs (trace hash, violations, counters) [scenario.go:Config]
  • RefBackend, RefBackendSQLite, RefBackendS3WAL — authority-store lane ("sqlite" or "s3wal") [scenario.go:RefBackend]
  • Injector, NewInjector — seeded fault injector; FailNext, FailNth, SetProbs, Count, Counts, Fired, FiredSince, Seq [fault.go:NewInjector]
  • FaultClass (FaultNone, FaultClean, FaultAmbiguous, FaultNotFound), FaultEvent, Probs, ErrInjectedClean, ErrInjectedAmbiguous [fault.go:FaultClass]
  • FaultClass.String — the fault class name for traces [fault.go:FaultClass.String]
  • Op names OpRef*, OpBlob*, OpCASGet, OpCASCreate, OpCASAdvance, OpCASList — the injection points [fault.go:OpRefListRefs]
  • SimRefStore, NewSimRefStore — fault-injecting ref.RefStore decorator [simref.go:SimRefStore]
  • SimBlobStore, NewSimBlobStore — fault-injecting blob.BlobStore decorator [simblob.go:SimBlobStore]
  • SimCAS, NewSimCAS — per-node fault-injecting view of a shared s3wal.MemCAS [simcas.go:SimCAS]

Invariants

  • Config.withDefaults asserts positive dimensions and a known RefBackend [scenario.go:withDefaults]
  • FailNth asserts a non-empty op, n >= 1, and a clean, ambiguous, or not-found class; SetProbs asserts non-negative probabilities that sum to at most 1 [fault.go:SetProbs]
  • One-shot plans fire before probabilistic faults [fault.go:Injector.begin]
  • A clean fault satisfies SafeToRetry, so ref.IdempotentCAS treats it as non-execution [fault.go:cleanFaultError]
  • On a clean fault the inner call never runs; on an ambiguous fault a mutating inner call runs and the caller gets an error [simref.go:CASRef] [simblob.go:UploadFile]. Ref-store reads and blob List skip the inner call under an ambiguous fault [simref.go:refReadFault] [simblob.go:List]. Blob DownloadFile and DownloadFileCapped still run the inner call [simblob.go:DownloadFile] [simblob.go:DownloadFileCapped]
  • SimCAS returns a definitive store rejection as a real verdict and never injects stale reads, 409 exhaustion, or Delete faults [simcas.go:put]
  • violate records the breach and stops the run at the first violation [scenario.go:violate]
  • casAttempt flags a ref moved to anything but the target, a win that does not match the tip, and an ambiguous committed write not reported as a win [scenario.go:casAttempt]
  • The quota ledger oracle requires usage == registered pack bytes + retained bytes after every step [quota_ops.go:checkQuotaLedger]
  • After a completed full reservation sweep, every repo must hold ReservedBytes == 0 [quota_ops.go:doReserveSweep]
  • Repack oracles require the post-op pack set to equal the pre-set or the folded set [repack_ops.go:doRepack]
  • Sync and push oracles check lease liveness, steal safety, late-finish errors, and a full-row mirror of the upstream row [sync_ops.go:checkUpstreamMirror] [push_ops.go:doPushClaim]

Side Effects

  • Run creates a temp run dir (removed unless KeepDir), a shared meta.sqlite file, a fileblob dir, and per-node cache dirs. The ref authority is a shared refs.sqlite file on the sqlite lane and a shared in-memory s3wal.MemCAS on the s3wal lane [scenario.go:Run] [cluster.go:newCluster]
  • Runs hermetic git subprocesses (init --bare, config gc.auto 0, hash-object, mktree, commit-tree, pack-objects, for-each-ref, fsck) [gitenv.go:simGit] [gitenv.go:initScratchRepo]
  • Drives the real stores: RegisterPacks, CASRef, ReservePackBytes, TryBeginSync, TryBeginPush, and githttp.Hydrate [scenario.go:doPush] [quota_ops.go:doReserveCrash] [sync_ops.go:doSyncClaim] [push_ops.go:doPushClaim] [oracle.go:doHydrate]
  • initSync inserts orgs and repos rows with raw SQL [sync_ops.go:initSync]
  • Cold crashes remove a node's cache dir [cluster.go:crashNode]
  • tracef hashes every trace line and writes it to Config.Trace when set [scenario.go:tracef]

Verification

  • Run: go test ./internal/simtest/... -tags=openstorage_viewcheck -count=1 -race -timeout=30m
  • Nightly sweep (.github/workflows/dst-nightly.yml, one job per backend): DST_SWEEP_BASE=<base> DST_SWEEP_SEEDS=48 DST_STEPS=300 DST_REF_BACKEND=<sqlite|s3wal> go test ./internal/simtest -count=1 -race -run 'TestDSTSweep' -v -timeout=110m (the workflow defaults; the nightly job runs once per backend and takes <base> from its base input, else github.run_id % 1000000000)
  • Reproduce a seed: DST_SEED=<seed> DST_STEPS=<steps> go test ./internal/simtest -run 'TestDSTSeed' -v -count=1

Documentation

Overview

Package simtest is the deterministic simulation testing harness (issue #3, ported from waltier's SimStore + DST design): fault-injection decorators for blob.BlobStore and ref.RefStore plus a seeded, serialized multi-node simulation with invariant oracles.

The package is test-only by convention: nothing under cmd/openstorage or the serve wiring imports it. It ships as regular (non _test.go) files so the seed-reproduction CLI (cmd/dst) and the go-test entry points share one implementation.

The fault model distinguishes the two classes that matter for storage correctness:

  • clean fault: the operation did NOT reach the backing store; the caller sees an error. The injected error satisfies pgconn's SafeToRetry contract, exactly like the real stores' acquire-stage failures, so ref.IdempotentCAS classifies it as definitive non-execution.
  • ambiguous fault: the operation DID execute against the backing store, but its result is discarded and the caller sees an error — the "committed, then the connection dropped" class behind issue #2.

Blob reads additionally support a transient not-found fault (wrapping blob.ErrNotFound) modeling a listed pack vanishing mid-hydration — the class behind issue #6's bounded retry.

Outbound push-lane claim/marker scenario ops and oracle (issue #136): the push_* edition of sync_ops.go over the lane PR #137 added. A minimal forwarder actor drives RequestPush/TryBeginPush/FinishPush cycles against the real meta store (no real git — the transfer is a seeded number of sim steps) with seeded crash injection between claim and finish, seeded transfer outcomes mirroring the worker's failure taxonomy (success / transient re-mark / terminal non-FF rejection). Node crash ops kill the node's in-flight push runs too. The store's clock is the simulation's logical clock, so lease expiry is a pure function of the seed.

Oracles, all against a ground-truth model maintained by the serialized scheduler (reads through the clean oracle meta handle), keyed to the four #136 invariants:

  1. convergence: refs are abstracted to a per-repo version counter — every push-request bumps the local version, a claim snapshots it (the loopback fetch postdates the claim), and a successful finish covers through its snapshot. At quiescence every accepted request must be covered by a postdating successful run, terminally rejected (non-FF, needs a human), or still CARRIED (pending marker set or an orphaned claim a later steal takes over) — and the final claim+finish per repo converges covered == local (upstream refs equal local refs);
  2. force monotonicity: the model pending level only coalesces upward until a claim consumes it whole, the force TryBeginPush reports must equal the model mandate (pending == force OR inherited from a dead claim's persisted push_claim_force), and the full-row mirror turns any unmodeled marker downgrade into a violation at the very next observation. A dead claim's force mandate survives on its push_claim_force alone, and the steal inherits it;
  3. lease safety: at most one unexpired holder per repo (fresh claims only against idle rows, steals only of expired claims naming the exact previous holder, refusals only against live ones), and a stolen holder's FinishPush fails (ErrClaimLost while the thief holds, ErrNotFound after it finished) leaving row and counters untouched;
  4. lane isolation: the mirror both lanes share (checkUpstreamMirror, sync_ops.go) compares every sync_* AND push_* column after every mutation point in either lane, so a pull-lane transition touching push-lane columns (or vice versa) violates immediately.

Quota-ledger scenario ops and oracle (issues #29/#31): leaked admission reservations (reserve → crash) with sweep-side reclamation, and a per-step consistency check tying the maintained usage counter to ground truth — registered pack blobs plus recorded retained bytes.

repack_ops.go adds the issue-#1 background repack to the simulation (the "pack rows only grow, modulo repack" oracle the harness deferred until repack existed):

  • opRepack folds through the node's FAULT-INJECTED stores, exercising clean/ambiguous faults on the fold pipeline (list, upload, replace);
  • opRepackRace interleaves two folds deterministically on the CLEAN oracle stores — a mid-race push changes the winner's input set — and asserts the loser installed nothing and deleted exactly its own blob;
  • the pack-row oracle (checkPackRows) asserts after every step that a repo's row set only ever GROWS except across a repack, whose exact before→after transition is pinned in-op (atomicity: the observable state is the pre-set or the folded set, never anything between).

Upstream-sync claim-lease scenario ops and oracle (issues #86/#88): a minimal sync actor drives TryBeginSync/FinishSync cycles against the real meta store (no real git — the transfer is a seeded number of sim steps), with seeded crash injection between claim and finish. Node crash ops kill every in-flight run on the node too. The store's clock is the simulation's logical clock (meta.WithNowFunc), so lease expiry is a pure function of the seed.

Oracles, all against a ground-truth model maintained by the serialized scheduler (reads through the clean oracle meta handle):

  1. lease liveness (the #86 invariant this file exists for): after any claim attempt at sim time T, no row may keep syncing=true with sync_claim_expires_at <= T — an expired claim must be stolen by the very next attempt. Exact (no slack): store and oracle share one injected clock, so there is no skew to absorb;
  2. steal safety: refusals only against unexpired claims, steals only of expired ones, stolenFrom naming the exact previous holder;
  3. late-finish choreography: a stolen run's FinishSync must fail (ErrClaimLost while the thief holds, ErrNotFound after the thief finished) and leave the row and counters untouched; an unstolen run's FinishSync must succeed;
  4. counter/pending integrity: RunCount/SuccessCount/sync_pending mirror the model at every claim/finish boundary (only successful FinishSync calls count; crashed and stolen runs never do);
  5. quiescence: past every lease, one final claim per repo must succeed (steal or fresh) and finish cleanly — no wedged row survives.

Index

Constants

View Source
const (
	OpRefListRefs         = "ref.ListRefs"
	OpRefListRefsWithMeta = "ref.ListRefsWithMeta"
	OpRefGetRef           = "ref.GetRef"
	OpRefCASRef           = "ref.CASRef"
	OpRefRegisterPacks    = "ref.RegisterPacks"
	OpRefReplacePacks     = "ref.ReplacePacks"
	OpRefListPacks        = "ref.ListPacks"
	OpRefGetGeneration    = "ref.GetGeneration"
	OpRefReservePacks     = "ref.ReservePackBytes"

	OpBlobUploadFile   = "blob.UploadFile"
	OpBlobDownloadFile = "blob.DownloadFile"
	OpBlobList         = "blob.List"
	OpBlobSignedURL    = "blob.SignedURL"
	OpBlobStat         = "blob.Stat"
	OpBlobEnsureBucket = "blob.EnsureBucket"
	OpBlobDelete       = "blob.Delete"
	OpBlobCopy         = "blob.Copy"

	// s3wal conditional-store transport ops (SimCAS, RefBackendS3WAL):
	// these sit BELOW the driver, so their faults exercise the driver's
	// own §2.3 recovery machinery rather than ref.IdempotentCAS.
	OpCASGet     = "cas.Get"
	OpCASCreate  = "cas.Create"
	OpCASAdvance = "cas.Advance"
	OpCASList    = "cas.List"
)

Operation names the decorators inject on. The ref/blob prefixes keep one flat namespace so a single Injector can serve both decorators.

Variables

View Source
var (
	// ErrInjectedClean marks a clean fault: the op did not execute.
	ErrInjectedClean = errors.New("simtest: injected clean fault (op did not execute)")
	// ErrInjectedAmbiguous marks an ambiguous fault: the op executed but
	// the result was lost.
	ErrInjectedAmbiguous = errors.New("simtest: injected ambiguous fault (op executed, result lost)")
)

Sentinel errors for injected faults; match with errors.Is. The concrete returned errors carry the op name.

Functions

This section is empty.

Types

type Config

type Config struct {
	Seed     uint64
	Steps    int // scheduler steps (default 150)
	Nodes    int // simulated server nodes sharing one SQLite+fileblob (default 3)
	Repos    int // simulated repos (default 2)
	Branches int // branches per repo (default 3)
	// RefBackend selects the authority store under test (default
	// RefBackendSQLite). The meta store (sync/push claim lanes) and the
	// fileblob store are backend-independent and stay SQLite/file.
	RefBackend RefBackend
	// FullCheckEvery runs the expensive cold-hydrate oracle every N steps
	// (default 50). It always also runs once after the last step.
	FullCheckEvery int
	// Dir is the run's scratch root; empty means a fresh os.MkdirTemp that
	// is removed afterwards unless KeepDir is set.
	Dir     string
	KeepDir bool
	// Trace, when non-nil, receives the live trace lines (the same bytes
	// that feed TraceHash).
	Trace io.Writer
}

Config parameterizes one deterministic simulation run. The zero value of every field defaults sensibly (see withDefaults); Seed 0 is a valid seed.

type FaultClass

type FaultClass int

FaultClass selects what an injected fault does to the decorated call.

const (
	// FaultNone means the call proceeds to the backing store untouched.
	FaultNone FaultClass = iota
	// FaultClean fails the call WITHOUT executing it.
	FaultClean
	// FaultAmbiguous executes the call, discards its result, and fails it.
	FaultAmbiguous
	// FaultNotFound fails a blob op with blob.ErrNotFound without
	// executing it (transient missing-object; only valid on
	// OpBlobDownloadFile, OpBlobStat, OpBlobDelete, and OpBlobCopy).
	FaultNotFound
)

func (FaultClass) String

func (c FaultClass) String() string

String returns the fault class name for traces.

type FaultEvent

type FaultEvent struct {
	Seq   int // 1-based injector-wide sequence number of the faulted call
	Op    string
	Class FaultClass
	// InnerOutcome describes the backing store's real result for ambiguous
	// faults (e.g. "won=true err=<nil>"); empty for non-executed classes.
	InnerOutcome string
}

FaultEvent records one injected fault, for traces and oracle bookkeeping.

type Injector

type Injector struct {
	// contains filtered or unexported fields
}

Injector decides, per decorated call, whether to inject a fault. One injector is shared by all decorators of a scenario: a single seeded PRNG stream plus FIFO one-shot plans keep decisions deterministic as long as calls are serialized (the DST scheduler runs one op at a time; the decorators still lock so misuse cannot corrupt state).

func NewInjector

func NewInjector(seed uint64) *Injector

NewInjector returns an Injector seeded with seed. The zero probability state injects nothing until FailNext/FailNth/SetProbs are called.

func (*Injector) Count

func (in *Injector) Count(op string) int

Count returns how many calls of op the decorators have seen (including faulted ones).

func (*Injector) Counts

func (in *Injector) Counts() map[string]int

Counts returns a copy of all op counters.

func (*Injector) FailNext

func (in *Injector) FailNext(op string, class FaultClass)

FailNext queues a one-shot fault on the next call of op.

func (*Injector) FailNth

func (in *Injector) FailNth(op string, n int, class FaultClass)

FailNth queues a one-shot fault on the n-th call of op counted from now (n=1 is the next call). Queued plans fire in FIFO order: each plan's countdown starts only after the previous plan for the same op has fired.

func (*Injector) Fired

func (in *Injector) Fired() []FaultEvent

Fired returns a copy of every injected fault so far, in injection order.

func (*Injector) FiredSince

func (in *Injector) FiredSince(seq int) ([]FaultEvent, int)

FiredSince returns injected faults with Seq > seq, plus the current high-water mark to pass back next time (per-step fault attribution). It does not mutate injector state, so overlapping windows are fine.

func (*Injector) Seq

func (in *Injector) Seq() int

Seq returns the current call sequence number (a mark for FiredSince).

func (*Injector) SetProbs

func (in *Injector) SetProbs(op string, p Probs)

SetProbs configures probabilistic fault injection for op. Passing all zeros clears it. One-shot plans take precedence over probabilities.

type Probs

type Probs struct {
	Clean     float64
	Ambiguous float64
	NotFound  float64
}

Probs are per-op fault probabilities; the remainder is FaultNone. Draws happen once per matching call in call order, so a fixed seed yields a fixed decision sequence under serialized execution.

type RefBackend

type RefBackend string

RefBackend names the authority-store lane a simulation runs on.

const (
	// RefBackendSQLite is the default lane: every node opens the one
	// shared SQLite database file.
	RefBackendSQLite RefBackend = "sqlite"
	// RefBackendS3WAL runs the s3wal manifest-CAS driver over one SHARED
	// in-memory conditional store (plan §4 Phase 2's multi-node
	// shared-fake topology). This is stricter than the backend's
	// single-node deployment decree on purpose: it exercises every
	// PROTOCOL race between manifest writers, not deployment races.
	RefBackendS3WAL RefBackend = "s3wal"
)

type Result

type Result struct {
	Seed        uint64
	Backend     RefBackend // authority-store lane the run executed on
	Steps       int        // steps executed (< Config.Steps when a violation stopped the run)
	TraceHash   string
	Violations  []string
	Transitions int // distinct committed ref transitions observed
	Deliveries  int // webhook deliveries recorded
	Faults      int // faults injected
	// CASFaults is the subset of Faults DELIVERED at the s3wal
	// conditional-store transport layer (cas.* ops; zero on the SQLite
	// backend; store-rejected attempts excluded) — the vacuity guard for
	// the Phase 2 fault plan. CASFaultOps breaks it down per op so every
	// advertised surface can be required to FIRE, not merely be called.
	CASFaults   int
	CASFaultOps map[string]int
	Steals      int // expired sync claims stolen (issues #86/#88)
	// CrashSteals is the subset of Steals whose previous holder actually
	// crashed mid-run (dead-holder class) rather than outliving its lease.
	CrashSteals int
	// PushSteals/PushCrashSteals are the outbound-lane counterparts
	// (issue #136); PushForcedRescues counts steals that inherited a force
	// mandate carried only by the dead claim's push_claim_force.
	PushSteals        int
	PushCrashSteals   int
	PushForcedRescues int
	OpCounts          map[string]int
}

Result summarizes one run. Violations non-empty means an invariant broke; TraceHash is a digest of the full deterministic trace (two runs with the same Config must produce identical hashes).

func Run

func Run(cfg Config) (Result, error)

Run executes one deterministic simulation and reports the outcome. The returned error is an infrastructure failure (harness bug, git or filesystem trouble) — invariant breaches land in Result.Violations.

type SimBlobStore

type SimBlobStore struct {
	// contains filtered or unexported fields
}

SimBlobStore decorates a blob.BlobStore with seeded fault injection. Uploads support the clean/ambiguous split (an ambiguous upload publishes the object, then loses the result — the "S3 PUT succeeded, response dropped" class); downloads and stats additionally support a transient FaultNotFound wrapping blob.ErrNotFound, the class issue #6's bounded hydration retry absorbs.

func NewSimBlobStore

func NewSimBlobStore(inner blob.BlobStore, inj *Injector) *SimBlobStore

NewSimBlobStore wraps inner with fault injection driven by inj.

func (*SimBlobStore) Close

func (s *SimBlobStore) Close() error

Close always delegates: lifecycle teardown is never fault-injected (a crash is simulated by dropping the handle, not by failing Close).

func (*SimBlobStore) Copy

func (s *SimBlobStore) Copy(ctx context.Context, dstKey, srcKey string) error

Copy delegates unless a fault is injected; an ambiguous fault performs the copy and then reports failure (a quarantine copy that landed but whose result was lost); FaultNotFound reports a missing source.

func (*SimBlobStore) Delete

func (s *SimBlobStore) Delete(ctx context.Context, key string) error

Delete delegates unless a fault is injected; an ambiguous fault deletes the object and then reports failure (the sweeper's delete outcome lost in transit); FaultNotFound reports the object as already missing.

func (*SimBlobStore) DownloadFile

func (s *SimBlobStore) DownloadFile(ctx context.Context, key, destPath string) error

DownloadFile delegates unless a fault is injected:

  • clean: no download happens, transport-shaped error.
  • ambiguous: the file IS materialized at destPath, then an error is reported (callers must tolerate the atomic download landing).
  • notfound: blob.ErrNotFound without executing — a listed pack transiently missing, which hydration must absorb by re-listing.

func (*SimBlobStore) DownloadFileCapped

func (s *SimBlobStore) DownloadFileCapped(ctx context.Context, key, destPath string, capBytes int64) error

DownloadFileCapped shares DownloadFile's fault point: the cap is a local disk-budget concern, not a distinct backend operation.

func (*SimBlobStore) EnsureBucket

func (s *SimBlobStore) EnsureBucket(ctx context.Context) error

EnsureBucket delegates to the inner store unless a fault is injected.

func (*SimBlobStore) List

func (s *SimBlobStore) List(ctx context.Context, prefix string) ([]blob.ListedObject, error)

List delegates to the inner store unless a fault is injected.

func (*SimBlobStore) ListPages

func (s *SimBlobStore) ListPages(ctx context.Context, prefix string, pageSize int, fn func([]blob.ListedObject) error) error

ListPages delegates to the inner store unless a fault is injected; it shares OpBlobList with List (one listing surface, one fault point).

func (*SimBlobStore) ReadRange

func (s *SimBlobStore) ReadRange(ctx context.Context, key string, offset, length int64) ([]byte, error)

ReadRange delegates unless a fault is injected. It deliberately shares OpBlobStat with Stat rather than adding a fault point: both are the repack selection's per-candidate point reads (size, pack-header object count) with identical failure handling — one metadata surface, one fault point (the List/ListPages precedent). FaultNotFound reports the object as missing via blob.ErrNotFound.

func (*SimBlobStore) SignedURL

func (s *SimBlobStore) SignedURL(ctx context.Context, key, method string, expiry time.Duration) (string, error)

SignedURL delegates to the inner store unless a fault is injected.

func (*SimBlobStore) Stat

func (s *SimBlobStore) Stat(ctx context.Context, key string) (int64, error)

Stat delegates unless a fault is injected; FaultNotFound reports the object as missing via blob.ErrNotFound.

func (*SimBlobStore) StatFingerprint

func (s *SimBlobStore) StatFingerprint(ctx context.Context, key string) (int64, string, error)

StatFingerprint delegates unless a fault is injected; it shares Stat's fault op — it IS a stat with a version fingerprint attached.

func (*SimBlobStore) UploadFile

func (s *SimBlobStore) UploadFile(ctx context.Context, key, path string) error

UploadFile delegates unless a fault is injected; an ambiguous fault uploads the object and then reports failure.

type SimCAS

type SimCAS struct {
	// contains filtered or unexported fields
}

SimCAS is one node's fault-injecting transport view of the SHARED in-memory conditional store (s3wal.MemCAS) — the s3wal DST fault plan (plan §4 Phase 2). It sits BELOW the manifest driver, so its faults exercise the driver's own mutate/recover machinery (§2.3), while the SimRefStore decorator above the driver keeps exercising ref.IdempotentCAS — both recovery layers run in one simulation. Rebuilt on node crash, like the process it stands for.

Fault classes per op (drawn from the shared Injector, so decisions are deterministic under the serialized scheduler):

  • cas.Create / cas.Advance: FaultClean returns s3wal.ErrNotExecuted without executing (the provably-never-sent class); FaultAmbiguous splits deterministically into applied (execute, discard the result, report *s3wal.AmbiguousError — the §2.3 committed-response-lost window) and dropped (never execute, same error).
  • cas.Get / cas.List: FaultClean fails the read without executing. Reads are idempotent, so a lost response is indistinguishable from a failed request — one class covers both.

Stale reads are NOT injected (issue #216). The s3wal store contract requires strong read-after-write consistency (plan §1/§2.6), the driver trusts every unconditional read by construction, and its seq watermark only catches a read that regresses below this process's own last observation — so a served-stale manifest yields a silently wrong read-only verdict (a sweep that "finds nothing to prune", a CAS that "lost") with no conditional PUT to expose it. Modelling that class makes the exact-state oracles unsatisfiable by any read-only path; the retired one-shot arm (PR #159) drove precisely that violation on seed 833534285 (pinned in dstRegressions). The 412 lost-race recovery the arm was meant to trigger is exercised dozens of times per run through the multi-node seed cache (issue #182): every cross-node write leaves the other nodes' cached snapshots stale, and their next seeded PUT 412s.

409-exhaustion (s3wal.ErrCASContention) is deliberately not injected: genuine CAS contention arises naturally from the multi-node shared-fake topology, and the exhaustion taxonomy is unit-tested in internal/ref. Deletes pass through unfaulted: the protocol never guards DELETEs (§2.7), and the W6 delete-lease races are unit-tested in internal/ref.

func NewSimCAS

func NewSimCAS(mem *s3wal.MemCAS, inj *Injector) *SimCAS

NewSimCAS wraps the shared in-memory conditional store for one node.

func (*SimCAS) Advance

func (c *SimCAS) Advance(ctx context.Context, key string, body []byte, etag string) (string, error)

Advance overwrites key iff etag is current, under the injector's verdict.

func (*SimCAS) Create

func (c *SimCAS) Create(ctx context.Context, key string, body []byte) (string, error)

Create writes key iff absent, under the injector's verdict.

func (*SimCAS) Delete

func (c *SimCAS) Delete(ctx context.Context, key string) error

Delete removes key unconditionally (never faulted; see package comment).

func (*SimCAS) Get

func (c *SimCAS) Get(ctx context.Context, key string) (s3wal.Object, error)

Get fetches key, possibly failing clean.

func (*SimCAS) GetIfChanged

func (c *SimCAS) GetIfChanged(ctx context.Context, key, etag string) (s3wal.Object, error)

GetIfChanged fetches key unless etag is current, possibly failing clean.

func (*SimCAS) List

func (c *SimCAS) List(ctx context.Context, prefix, startAfter string, limit int) ([]string, error)

List pages keys under prefix, possibly failing clean.

func (*SimCAS) StoreDate

func (c *SimCAS) StoreDate() (date, observedAt time.Time, ok bool)

StoreDate delegates to the shared store's Date witness.

type SimRefStore

type SimRefStore struct {
	// contains filtered or unexported fields
}

SimRefStore decorates a ref.RefStore with seeded clean/ambiguous fault injection. It sits BELOW ref.IdempotentCAS in the decorator chain (wrap SimRefStore in IdempotentCAS, or hand it to events.NewRefStore, which applies IdempotentCAS itself), so injected ambiguous CAS errors exercise the issue-#2 recovery path exactly where real transport errors surface.

func NewSimRefStore

func NewSimRefStore(inner ref.RefStore, inj *Injector) *SimRefStore

NewSimRefStore wraps inner with fault injection driven by inj.

func (*SimRefStore) BumpLFSAdoptionGen

func (s *SimRefStore) BumpLFSAdoptionGen(ctx context.Context, repo string) error

BumpLFSAdoptionGen delegates to the inner store.

func (*SimRefStore) CASRef

func (s *SimRefStore) CASRef(ctx context.Context, repo, name, old, new string) (bool, error)

CASRef injects the fault classes that matter for issue #2:

  • clean: the statement never executes; the returned error satisfies SafeToRetry, so IdempotentCAS must propagate it un-recovered.
  • ambiguous: the statement executes (the ref may really move) but the caller sees a transport-shaped error; the real outcome is recorded on the FaultEvent for oracle bookkeeping.

func (*SimRefStore) CASRefGen

func (s *SimRefStore) CASRefGen(ctx context.Context, repo, name, old, new string) (bool, int64, error)

CASRefGen mirrors CASRef's fault split; the generation attribution is lost with the outcome on ambiguous faults (gen == 0, the unattributed contract), exactly as a real lost response loses it.

func (*SimRefStore) ClearLFSMismatch

func (s *SimRefStore) ClearLFSMismatch(ctx context.Context, repo, key string) error

func (*SimRefStore) CountLFSObjects

func (s *SimRefStore) CountLFSObjects(ctx context.Context, repo string) (int64, error)

CountLFSObjects delegates to the inner store.

func (*SimRefStore) GetGeneration

func (s *SimRefStore) GetGeneration(ctx context.Context, repo string) (int64, error)

GetGeneration delegates to the inner store unless a fault is injected. Faulting it starves the issue-#4 hydrate short-circuit, which must then surface the error rather than serve from a stale marker.

func (*SimRefStore) GetRef

func (s *SimRefStore) GetRef(ctx context.Context, repo, name string) (string, error)

GetRef delegates to the inner store unless a fault is injected. Faulting it starves ref.IdempotentCAS's recovery recheck, which must then fall back to the original ambiguous outcome.

func (*SimRefStore) GuardedLFSDelete

func (s *SimRefStore) GuardedLFSDelete(ctx context.Context, repo, key string, del func(context.Context) error) (bool, error)

GuardedLFSDelete delegates to the inner store.

func (*SimRefStore) HasLFSObject

func (s *SimRefStore) HasLFSObject(ctx context.Context, repo, key string) (bool, error)

HasLFSObject delegates to the inner store.

func (*SimRefStore) HasPack

func (s *SimRefStore) HasPack(ctx context.Context, repo, key string) (bool, error)

HasPack delegates to the inner store unless a fault is injected; it shares OpRefListPacks with ListPacks (one registry-read fault point).

func (*SimRefStore) IsRetainedPack

func (s *SimRefStore) IsRetainedPack(ctx context.Context, repo, key string) (bool, error)

IsRetainedPack delegates to the inner store.

func (*SimRefStore) LFSAdoptionGen

func (s *SimRefStore) LFSAdoptionGen(ctx context.Context, repo string) (int64, error)

LFSAdoptionGen delegates to the inner store.

func (*SimRefStore) LFSMismatchFingerprint

func (s *SimRefStore) LFSMismatchFingerprint(ctx context.Context, repo, key string) (string, bool, error)

func (*SimRefStore) LFSObjectRegisteredAt

func (s *SimRefStore) LFSObjectRegisteredAt(ctx context.Context, repo, key string) (time.Time, int64, bool, error)

LFSObjectRegisteredAt delegates to the inner store.

func (*SimRefStore) ListPacks

func (s *SimRefStore) ListPacks(ctx context.Context, repo string) ([]ref.Pack, error)

ListPacks delegates to the inner store unless a fault is injected.

func (*SimRefStore) ListRefs

func (s *SimRefStore) ListRefs(ctx context.Context, repo string) (map[string]string, error)

ListRefs delegates to the inner store unless a fault is injected.

func (*SimRefStore) ListRefsAfter

func (s *SimRefStore) ListRefsAfter(ctx context.Context, repo, after string, limit int) ([]ref.RefEntry, error)

ListRefsAfter delegates to the inner store unless a fault is injected. It shares OpRefListRefs with ListRefs: both are the same ref-enumeration read surface, just bounded vs whole-repo (issue #119).

func (*SimRefStore) ListRefsWithMeta

func (s *SimRefStore) ListRefsWithMeta(ctx context.Context, repo string) ([]ref.RefInfo, error)

ListRefsWithMeta delegates to the inner store unless a fault is injected.

func (*SimRefStore) PackRegisteredBytes

func (s *SimRefStore) PackRegisteredBytes(ctx context.Context, repo, key string) (int64, bool, error)

PackRegisteredBytes delegates to the inner store.

func (*SimRefStore) QuotaLedgerState

func (s *SimRefStore) QuotaLedgerState(ctx context.Context, group []string, ttl time.Duration, excludeKeys ...string) (ref.PackQuotaLedgerState, error)

QuotaLedgerState delegates to the inner store (oracle reads).

func (*SimRefStore) ReclaimRetainedPack

func (s *SimRefStore) ReclaimRetainedPack(ctx context.Context, repo, key string) (int64, error)

ReclaimRetainedPack delegates to the inner store.

func (*SimRefStore) ReconcileLegacyLFSObjects

func (s *SimRefStore) ReconcileLegacyLFSObjects(ctx context.Context) (int64, bool, error)

func (*SimRefStore) RecordLFSMismatch

func (s *SimRefStore) RecordLFSMismatch(ctx context.Context, repo, key, fingerprint string) error

func (*SimRefStore) RegisterAdoptedLFSObject

func (s *SimRefStore) RegisterAdoptedLFSObject(ctx context.Context, repo, key string, bytes int64) error

RegisterAdoptedLFSObject delegates to the inner store.

func (*SimRefStore) RegisterLFSObject

func (s *SimRefStore) RegisterLFSObject(ctx context.Context, repo, key string, bytes int64) error

RegisterLFSObject delegates to the inner store (the DST does not model LFS transfers yet; delegation keeps the interface honest without adding an unexercised fault point).

func (*SimRefStore) RegisterPacks

func (s *SimRefStore) RegisterPacks(ctx context.Context, repo string, packs []ref.Pack) error

RegisterPacks supports the same clean/ambiguous split as CASRef: an ambiguous fault registers the packs (idempotently) and then loses the result.

func (*SimRefStore) ReleaseAdoptionClaim

func (s *SimRefStore) ReleaseAdoptionClaim(ctx context.Context, key, owner string) error

func (*SimRefStore) RenewAdoptionClaim

func (s *SimRefStore) RenewAdoptionClaim(ctx context.Context, key, owner string) (bool, error)

func (*SimRefStore) ReplacePacks

func (s *SimRefStore) ReplacePacks(ctx context.Context, repo string, old []ref.Pack, newPack ref.Pack) (bool, error)

ReplacePacks supports the same clean/ambiguous split as CASRef: an ambiguous fault executes the swap (which may really commit) and then loses the outcome, exercising the repack job's leave-the-blob handling.

func (*SimRefStore) ReservePackBytes

func (s *SimRefStore) ReservePackBytes(ctx context.Context, group []string, packs []ref.Pack, owner string, usageFloor, floorGen, quotaBytes int64, ttl time.Duration) (ref.ReserveDecision, error)

ReservePackBytes supports the same clean/ambiguous split as CASRef: an ambiguous fault runs the admission (a reservation row may really be inserted — the leaked-reservation class TTL expiry and the sweeper reclaim) and then loses the outcome.

func (*SimRefStore) RetirePackAttempts

func (s *SimRefStore) RetirePackAttempts(ctx context.Context, group []string, gens map[string]string) error

RetirePackAttempts delegates to the inner store (no fault point: a failed retire only leaves a TTL-expiring row, exactly what the crash ops already exercise).

func (*SimRefStore) SweepExpiredPackReservations

func (s *SimRefStore) SweepExpiredPackReservations(ctx context.Context, olderThan time.Duration) (int64, error)

SweepExpiredPackReservations delegates to the inner store.

func (*SimRefStore) TouchLFSObject

func (s *SimRefStore) TouchLFSObject(ctx context.Context, repo, key string) (bool, error)

TouchLFSObject delegates to the inner store.

func (*SimRefStore) TryAdoptionClaim

func (s *SimRefStore) TryAdoptionClaim(ctx context.Context, repo, key string) (bool, string, error)

TryAdoptionClaim / ReleaseAdoptionClaim pass through without fault injection: adoption claims are an advisory cross-replica dedup (Codex security round 73), not lifecycle state the oracle models.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL