chaos

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 26, 2026 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Overview

Package chaos provides model-based chaos testing for the FDB Record Layer.

The framework maintains an in-memory model (a simple map) that shadows the real FDB store. Operations are applied to both; after each operation, the framework verifies they agree. Disagreement = bug.

Faults (commit-unknown, conflicts, timeouts) are injected via ChaosTransactor, which wraps fdb.Transactor. Seeded PRNG ensures reproducibility.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// FaultsNone disables all fault injection (pure stress test).
	FaultsNone = &FaultConfig{}

	// FaultsRetryHeavy injects commit-unknown at 5% rate.
	FaultsRetryHeavy = &FaultConfig{Rates: map[FaultType]float64{
		FaultCommitUnknown: 0.05,
	}}

	// FaultsRetryVeryHeavy injects commit-unknown at 20% rate.
	FaultsRetryVeryHeavy = &FaultConfig{Rates: map[FaultType]float64{
		FaultCommitUnknown: 0.20,
	}}

	// FaultsAll injects all fault types at moderate rates.
	FaultsAll = &FaultConfig{Rates: map[FaultType]float64{
		FaultCommitUnknown:     0.03,
		FaultConflict:          0.03,
		FaultTransactionTooOld: 0.02,
	}}
)

Preset fault profiles.

Functions

func RunConcurrent 

func RunConcurrent(t testing.TB, realDB fdb.Database, metadata *recordlayer.RecordMetaData, cfg ConcurrentConfig)

RunConcurrent runs concurrent chaos testing against a real FDB store. Multiple worker goroutines hammer the same subspace with random Save/Delete operations. A validator goroutine periodically takes a snapshot-consistent read and verifies that derived state (indexes, counts) matches the records.

No ChaosTransactor — real FDB transaction conflicts provide the chaos. FDB conflict errors are expected and silently retried by Run().

Types

type ChaosTransactor 

type ChaosTransactor struct {

	// Log records all injected faults for post-mortem analysis.
	Log []FaultLogEntry
	// contains filtered or unexported fields
}

ChaosTransactor wraps an fdb.Transactor to inject faults at the transaction boundary. It implements fdb.Transactor so it can be used with NewFDBDatabaseWithTransactor.

func NewChaosTransactor 

func NewChaosTransactor(inner fdb.Transactor, faults *FaultConfig, seed uint64) *ChaosTransactor

NewChaosTransactor wraps an existing transactor with fault injection.

func (*ChaosTransactor) InjectOnce 

func (c *ChaosTransactor) InjectOnce(fault FaultType)

InjectOnce schedules a specific fault to fire on the next Transact call. The fault fires exactly once and then clears. Use for targeted tests.

func (*ChaosTransactor) InjectReadErrorOnce 

func (c *ChaosTransactor) InjectReadErrorOnce(keyPrefix []byte)

InjectReadErrorOnce schedules FaultReadError for the next Transact call: every read of a key starting with keyPrefix fails with a non-retryable FDB error. A nil prefix fails every read. The fault fires exactly once and then clears.

func (*ChaosTransactor) ReadTransact 

func (c *ChaosTransactor) ReadTransact(fn func(fdb.ReadTransaction) (any, error)) (any, error)

ReadTransact implements fdb.ReadTransactor. No fault injection on reads.

func (*ChaosTransactor) ReadTransactCtx 

func (c *ChaosTransactor) ReadTransactCtx(ctx context.Context, fn func(fdb.ReadTransaction) (any, error)) (any, error)

ReadTransactCtx implements fdb.CtxReadTransactor (threads ctx to the inner read path).

func (*ChaosTransactor) Transact 

func (c *ChaosTransactor) Transact(fn func(fdb.WritableTransaction) (any, error)) (any, error)

Transact implements fdb.Transactor. Wraps the inner Transact with fault injection.

func (*ChaosTransactor) TransactCtx 

func (c *ChaosTransactor) TransactCtx(ctx context.Context, fn func(fdb.WritableTransaction) (any, error)) (any, error)

TransactCtx implements fdb.CtxTransactor — same fault injection, threading ctx to the inner transactor's ctx-aware path when present (RFC-090).

type ConcurrentConfig 

type ConcurrentConfig struct {
	// Seed for the PRNG. Each worker gets seed + workerID.
	Seed uint64

	// Workers is the number of concurrent goroutines (default 4).
	Workers int

	// Duration is how long to run (default 5s).
	Duration time.Duration

	// MaxPKs bounds the primary key range [0, MaxPKs) (default 50).
	MaxPKs int64

	// ValidateEvery controls validation interval (default 1s).
	ValidateEvery time.Duration
}

ConcurrentConfig controls concurrent chaos testing parameters.

type FaultConfig 

type FaultConfig struct {
	// Rates maps each fault type to its injection probability (0.0–1.0).
	Rates map[FaultType]float64
}

FaultConfig controls fault injection rates.

type FaultLogEntry 

type FaultLogEntry struct {
	OpIndex int
	Fault   FaultType
}

FaultLogEntry records a single injected fault for reproducibility.

type FaultType 

type FaultType int

FaultType identifies a specific fault that can be injected. 

const (
	// FaultCommitUnknown simulates FDB error 1021 (commit_unknown_result).
	// The transaction commits successfully, but the ChaosTransactor re-executes
	// the function in a new transaction (simulating a client retry after
	// ambiguous commit). This tests idempotency — does a retry corrupt state?
	//
	// Critical for: COUNT/SUM indexes (atomic ADD is not idempotent),
	// record counting, any mutation that isn't naturally idempotent.
	FaultCommitUnknown FaultType = iota

	// FaultConflict simulates FDB error 1020 (not_committed / transaction conflict).
	// Implemented identically to FaultCommitUnknown at the Transactor level:
	// both commit, then re-execute. In real FDB, the first attempt's writes
	// would be rolled back, but we can't simulate true rollback at this
	// abstraction level. The double-commit is a superset test: if the code
	// is correct under double-commit, it's correct under rollback+retry too.
	FaultConflict

	// FaultTransactionTooOld simulates FDB error 1007 (transaction_too_old).
	// Same implementation as FaultConflict — see comment above.
	FaultTransactionTooOld

	// FaultReadError simulates FDB error 1510 (io_error) surfacing from a READ
	// inside the transaction, rather than from the commit at its boundary. The
	// other fault types all commit and re-execute, so none of them can reach a
	// caller that only reads — and read paths are exactly where "an error means
	// the data is absent" is an easy and silent mistake to make.
	//
	// Scoped to a key prefix (see InjectReadErrorOnce) so a test can fail the one
	// read it is reasoning about and leave the surrounding open/scan reads alone;
	// a blanket failure would be satisfied by whichever read happens to come
	// first, which is not a property worth pinning.
	//
	// 1510 is deliberately NOT retryable (fdb.IsRetryable), so the fault surfaces
	// to the caller instead of spinning the transactor's retry loop against a
	// wrapper that would re-arm on every attempt.
	FaultReadError
)

type ModelRecord 

type ModelRecord struct {
	PrimaryKey tuple.Tuple
	TypeName   string
	Message    proto.Message
}

ModelRecord tracks a single record in the model.

type OpWeights 

type OpWeights struct {
	SaveNew        int // Save with a PK not in the model
	SaveOverwrite  int // Save with a PK already in the model
	DeleteExisting int // Delete a PK that exists in the model
	DeleteMissing  int // Delete a PK that does NOT exist in the model
	DeleteAll      int // Delete all records
}

OpWeights controls the relative probability of each operation type.

type Option 

type Option func(*scenarioConfig)

Option configures a Scenario.

func WithFaults 

func WithFaults(faults *FaultConfig) Option

WithFaults sets the fault injection configuration.

func WithSeed 

func WithSeed(seed uint64) Option

WithSeed sets the random seed for deterministic replay.

type RandomConfig 

type RandomConfig struct {
	// Seed for the PRNG. Same seed = same operations = same result.
	Seed uint64

	// NumOps is the total number of operations to execute.
	NumOps int

	// Faults controls fault injection (nil = no faults).
	Faults *FaultConfig

	// VerifyEvery controls how often Verify() is called.
	// Default: 50 (every 50 operations).
	VerifyEvery int

	// MaxPKs bounds the primary key range [0, MaxPKs) for higher overwrite rates.
	// Default: 50.
	MaxPKs int64

	// Weights overrides the default operation weights. Nil uses defaults.
	Weights *OpWeights
}

RandomConfig controls the random operation generator.

type Scenario 

type Scenario struct {
	Rng *rand.Rand
	// contains filtered or unexported fields
}

Scenario is the primary chaos testing primitive. It wraps a real FDB store with a model and optional fault injection, providing operations that update both and verification that they agree.

func NewScenario 

func NewScenario(t testing.TB, realDB fdb.Database, metadata *recordlayer.RecordMetaData, opts ...Option) *Scenario

NewScenario creates a new chaos testing scenario. Each scenario gets its own FDB subspace for isolation. By default, no faults are injected — use WithFaults() or InjectOnce().

func RunRandom 

func RunRandom(t testing.TB, realDB fdb.Database, metadata *recordlayer.RecordMetaData, cfg RandomConfig) *Scenario

RunRandom executes a random sequence of operations against the store, comparing against the model periodically. Same seed = same sequence = same result.

Returns the final Scenario so callers can inspect model state, fault logs, etc.

func (*Scenario) ChaosDB 

func (s *Scenario) ChaosDB() *recordlayer.FDBDatabase

ChaosDB exposes the fault-injecting database for tests that drive concurrent raw operations (where the single-threaded model would race). Pair with VerifySnapshot, which rebuilds the model from store state.

func (*Scenario) CleanDB 

func (s *Scenario) CleanDB() *recordlayer.FDBDatabase

CleanDB exposes the no-fault database for draining and verification.

func (*Scenario) DeleteAllRecords 

func (s *Scenario) DeleteAllRecords()

DeleteAllRecords deletes all records and resets the model.

func (*Scenario) DeleteRecord 

func (s *Scenario) DeleteRecord(pk tuple.Tuple)

DeleteRecord deletes a record by primary key and updates the model.

func (*Scenario) DrainSPFresh 

func (s *Scenario) DrainSPFresh(indexName string)

DrainSPFresh drains the maintenance queue to quiescence through the CLEAN transactor (no faults), failing the test on error. Call before Verify: the structural-integrity invariant is strict (every membership target ACTIVE), which holds only once the lifecycle has settled. Draining clean also proves the post-fault state is *recoverable* — whatever a fault left mid-flight, a clean pass completes it.

func (*Scenario) FaultLog 

func (s *Scenario) FaultLog() []FaultLogEntry

FaultLog returns the list of injected faults so far.

func (*Scenario) InjectOnce 

func (s *Scenario) InjectOnce(fault FaultType)

InjectOnce schedules a fault for the next operation's transaction. The fault fires exactly once, then clears.

func (*Scenario) OpenStore 

func (s *Scenario) OpenStore(rtx *recordlayer.FDBRecordContext) (*recordlayer.FDBRecordStore, error)

OpenStore opens the scenario's store within a transaction — a storeBuilder usable with the exported SPFresh maintenance/search entry points.

func (*Scenario) RebalanceSPFresh 

func (s *Scenario) RebalanceSPFresh(indexName string) (int, error)

RebalanceSPFresh drains the index's maintenance queue THROUGH the fault-injecting transactor. Returns the lifecycle actions taken and any error (an undrained queue or a poisoned task surfaces here). Maintenance changes layout/recall, not record membership, so it does not touch the model.

func (*Scenario) RefineSPFresh 

func (s *Scenario) RefineSPFresh(indexName string, budget int) (int, bool, error)

RefineSPFresh runs one budgeted RFC-104 refinement pass through the fault-injecting transactor. Returns (moved, cycleConverged, error).

func (*Scenario) SaveRecord 

func (s *Scenario) SaveRecord(msg proto.Message)

SaveRecord saves a record to the store and updates the model. The transaction goes through the ChaosTransactor (fault injection). On success, the model is updated. On failure, the test fails.

func (*Scenario) Seed 

func (s *Scenario) Seed() uint64

Seed returns the scenario's random seed.

func (*Scenario) SweepSPFresh 

func (s *Scenario) SweepSPFresh(indexName string, timer *recordlayer.StoreTimer) (recordlayer.SPFreshSweepResult, error)

SweepSPFresh runs one bounded multi-tenant sweep pass through the fault-injecting transactor, accumulating per-kind lifecycle counts into the timer. Generous per-pass budgets so a sweep drains most pending work in one call; tests read timer.GetCount(CountSPFreshSplits/Merges/...) to PROVE the lifecycle fired under faults (not a fake checkbox).

func (*Scenario) TrySaveRecord 

func (s *Scenario) TrySaveRecord(msg proto.Message) error

TrySaveRecord attempts to save a record and returns the error (if any). Unlike SaveRecord, it does NOT call t.Fatal on error — the caller handles it. Model is only updated on success.

func (*Scenario) Verify 

func (s *Scenario) Verify()

Verify compares the model against the actual store state. Uses the clean DB (no fault injection) to avoid spurious failures. Fails the test if any violations are found.

type StoreModel 

type StoreModel struct {
	Records map[string]*ModelRecord // pk.Pack() -> record

	// CountUpdates tracks cumulative insert+update events per grouping key
	// for COUNT_UPDATES indexes. Key: indexName + ":" + packedGroupingKey.
	// COUNT_UPDATES ignores deletes and never decrements.
	CountUpdates map[string]int64

	// MaxEver tracks the maximum value ever seen per (indexName, groupingKey).
	// Key: indexName + ":" + packedGroupingKey. EVER semantics: only ratchets up,
	// individual deletes are no-ops. Reset on DeleteAll (store clears index data).
	MaxEver map[string]int64

	// MinEver tracks the minimum value ever seen per (indexName, groupingKey).
	// Key: indexName + ":" + packedGroupingKey. EVER semantics: only ratchets down,
	// individual deletes are no-ops. Reset on DeleteAll (store clears index data).
	MinEver map[string]int64
	// contains filtered or unexported fields
}

StoreModel is a trivially simple in-memory shadow of the record store. It tracks which records exist and their content. It IS the specification: if the real store disagrees with the model, the store has a bug.

func NewStoreModel 

func NewStoreModel(metadata *recordlayer.RecordMetaData) *StoreModel

NewStoreModel creates a new empty model.

func (*StoreModel) Count 

func (m *StoreModel) Count() int64

Count returns the total number of records in the model.

func (*StoreModel) Delete 

func (m *StoreModel) Delete(pk tuple.Tuple)

Delete removes a record from the model. No-op if not found.

func (*StoreModel) DeleteAll 

func (m *StoreModel) DeleteAll()

DeleteAll removes all records from the model. Resets EVER tracking too — the store's DeleteAllRecords clears index data.

func (*StoreModel) Has 

func (m *StoreModel) Has(pk tuple.Tuple) bool

Has returns true if a record with the given PK exists in the model.

func (*StoreModel) Save 

func (m *StoreModel) Save(msg proto.Message)

Save adds or overwrites a record in the model. Extracts the record type name and primary key from the proto message using the metadata's record type definitions.

type Violation 

type Violation struct {
	Invariant  string      // e.g., "record_count", "record_missing", "record_orphan"
	PrimaryKey tuple.Tuple // relevant PK (if applicable)
	Expected   any         // what the model says
	Actual     any         // what the store has
}

Violation represents a single inconsistency between the model and the store.

func Verify 

func Verify(store *recordlayer.FDBRecordStore, model *StoreModel) []Violation

Verify compares the store's actual state against the model's expected state. Returns all violations found. An empty slice means the store is consistent.

Checks performed:

  1. Record count (store.GetRecordCount() vs model.Count())
  2. Record existence (every model record exists in store)
  3. No orphans (every store record exists in model)
  4. VALUE index entries
  5. Atomic index values (COUNT, SUM, COUNT_UPDATES)
  6. MIN/MAX_EVER index values
  7. RANK index entries + ranked set consistency
  8. PERMUTED_MIN/MAX index entries (primary + permuted subspace)
  9. VERSION index entries (PK matching + versionstamp consistency)

10. MULTIDIMENSIONAL index entries (R-tree scan vs model, set-based) 11. VECTOR index entries (HNSW self-search + count + orphan check) 12. BITMAP_VALUE index entries (bitmap bits vs model records) 13. TEXT index entries (token→PK set vs model tokenization)

func VerifySnapshot 

func VerifySnapshot(store *recordlayer.FDBRecordStore, metadata *recordlayer.RecordMetaData) []Violation

VerifySnapshot builds a model from the store's current records and verifies that derived state (indexes, counts) matches. Unlike Verify(), this does NOT require a pre-built model — it reconstructs one from the actual store data.

Checks performed (snapshot-derivable only):

  • Record count (GetRecordCount vs scanned records)
  • VALUE index entries
  • COUNT index values (recomputable from current records)
  • SUM index values (recomputable from current records)
  • RANK index entries
  • PERMUTED_MIN/MAX index entries
  • VERSION index entries
  • Covering index value verification

NOT checked (requires history tracking):

  • COUNT_UPDATES (cumulative across all saves)
  • MAX_EVER / MIN_EVER (needs full mutation history)
  • MAX_EVER_VERSION (needs full mutation history)

func (Violation) String 

func (v Violation) String() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.