chaos

type BusyHolder ¶

type BusyHolder struct {
	// contains filtered or unexported fields
}

BusyHolder injects sqlite-busy contention by opening a sibling connection to the project's state database and parking a write transaction inside it. While the transaction is open, every other writer that exceeds the project's busy_timeout returns SQLITE_BUSY — the realistic SQLITE_BUSY scenario the rest of cloop's code is designed to survive.

The holder uses an *exclusive* transaction (BEGIN IMMEDIATE) so reads from other connections still proceed under WAL — only writers contend, which matches the production failure mode we want to reproduce.

Concurrency: a holder is single-shot. Start opens the transaction, Stop rolls it back. Stop is safe to call multiple times and returns the wall- clock duration the transaction was actually held.

func NewBusyHolder(dbPath string) *BusyHolder

func (h *BusyHolder) Start(ctx context.Context) error

func (h *BusyHolder) Stop() time.Duration

type Controller ¶

type Controller struct {
	// contains filtered or unexported fields
}

Controller is the in-process owner of currently-active faults. A single instance is shared across the orchestrator, provider HTTP clients, and any other subsystem that wants to participate in chaos.

Concurrency: the active set is guarded by a sync.RWMutex, with reads dominating in the hot HTTP path. The refresher goroutine is the sole writer outside the public Inject/Clear methods.

Lifecycle: NewController returns an idle controller with an empty active set; call Start(ctx) to begin file-watching. The controller is safe to instantiate even when no chaos is intended — every ShouldInject fast-path returns false in zero time when the active set is empty (atomic counter).

func Global() *Controller

func NewController(workDir string) *Controller

func SetGlobal(c *Controller) *Controller

func (c *Controller) Active() []Fault

func (c *Controller) Clear() error

func (c *Controller) FaultsOfType(t FaultType) []Fault

func (c *Controller) Inject(f Fault) error

func (c *Controller) LastError() error

func (c *Controller) Refresh() error

func (c *Controller) ShouldInject(t FaultType) bool

func (c *Controller) Start(ctx context.Context)

func (c *Controller) Stop()

type DiskFullSimulator ¶

type DiskFullSimulator struct {
	// contains filtered or unexported fields
}

DiskFullSimulator creates a large ballast file inside .cloop/chaos/ to occupy disk space, simulating an ENOSPC environment for code paths that write into the project tree. The simulator targets a fixed byte size; if the filesystem cannot accommodate it, we surface that directly to the caller (which, for chaos purposes, is the desired outcome).

The simulator never deletes files outside .cloop/chaos/ballast, so it is safe to run alongside other chaos faults. Stop() removes the ballast file regardless of how Start exited.

func NewDiskFullSimulator(workDir string, sizeBytes int64) *DiskFullSimulator

func (d *DiskFullSimulator) Start() error

func (d *DiskFullSimulator) Stop() error

type Fault ¶

type Fault struct {
	// ID is the chaos_runs row primary key. Zero before persistence; set by
	// (*Store).Insert and copied back into the file mirror so the daemon-side
	// controller knows which row to stamp with the observed outcome.
	ID int64 `json:"id,omitempty"`

	// Type is the fault category. Unknown types are tolerated by readers.
	Type FaultType `json:"type"`

	// StartedAt is when the fault was injected. Mostly diagnostic — the active
	// window is governed by Until.
	StartedAt time.Time `json:"started_at"`

	// Until is the wall-clock time at which the fault stops applying. The
	// controller skips faults with Until.Before(time.Now()) without removing
	// them from disk; the next persistence cycle prunes them.
	Until time.Time `json:"until"`

	// Probability (0..1) of injecting the fault on each candidate event. Most
	// faults default to 1.0 (always inject) but network-flap and provider-429
	// are typically partial so retry paths actually exercise their happy path.
	Probability float64 `json:"probability,omitempty"`

	// Note is free-form text shown in `cloop chaos report` and useful when
	// running scripted chaos campaigns.
	Note string `json:"note,omitempty"`
}

Fault describes a single active fault injection.

All times are wall-clock UTC. Until == zero means "no expiry", but every public injection path requires a positive Duration so this is a defensive guard rather than a documented mode.

func (f Fault) Active(now time.Time) bool

func (f Fault) Duration() time.Duration

type FaultType ¶

type FaultType string

FaultType identifies a kind of injected failure.

New values must be added to AllFaultTypes (used by validators) and to the switch in Apply* helpers. Readers must treat unknown values as no-ops so downgrading the binary while a fault file lingers on disk does not panic.

const (
	FaultProviderTimeout FaultType = "provider-timeout"
	FaultProvider429     FaultType = "provider-429"
	FaultProvider500     FaultType = "provider-500"
	FaultSQLiteBusy      FaultType = "sqlite-busy"
	FaultNetworkFlap     FaultType = "network-flap"
	FaultDiskFull        FaultType = "disk-full-simulation"
	FaultSlowDisk        FaultType = "slow-disk"
)

func ParseFaultType(s string) (FaultType, error)

type Outcome ¶

type Outcome string

Outcome captures how the system reacted to a fault. Recorded back to chaos_runs at the end of an injection window. Values are deliberately coarse-grained — fine analysis comes from the steps/events tables.

const (
	OutcomeRecovered Outcome = "recovered" // system continued operating
	OutcomeDegraded  Outcome = "degraded"  // continued but with errors/retries
	OutcomeCrashed   Outcome = "crashed"   // a process exited or panicked
	OutcomeUnknown   Outcome = "unknown"   // no observation collected
)

type Run ¶

type Run struct {
	ID                 int64
	FaultType          FaultType
	Probability        float64
	StartedAt          time.Time
	StoppedAt          time.Time
	DurationMS         int64
	Outcome            Outcome
	OutcomeDetail      string
	ObservedErrors     int
	ObservedRetries    int
	ObservedRecoveries int
	Note               string
}

Run is one row of the chaos_runs table — a single fault injection together with the observed outcome.

type Store ¶

type Store struct {
	// contains filtered or unexported fields
}

Store is a thin SQLite-backed log of chaos injections. Uses its own *sql.DB rather than statedb.DB so chaos can run even when statedb itself is the subject of the fault (for sqlite-busy testing).

The handle is opened under statedb's connection policy (Task 20374): a lock is not what the store is for, so it waits one out like every other writer. Its own writes never meet its own fault — the suite and the CLI insert a run's row before the sqlite-busy holder takes the lock and finalise it after the holder has let go. The holder itself (sqlite_busy.go) opens bare, because there the lock is the point.

func OpenStore(dbPath string) (*Store, error)

func (s *Store) Close() error

func (s *Store) Insert(r Run) (int64, error)

func (s *Store) List(offset, limit int) ([]Run, error)

func (s *Store) Update(r Run) error

type Suite ¶

type Suite struct {
	WorkDir    string
	StatePath  string
	Controller *Controller
	Store      *Store
	Cases      []SuiteCase
}

Suite runs each case sequentially: inject → probe → clear → record. Designed to be invoked by `cloop chaos suite` against either:

• a fresh in-process probe (the default — exercises Transport, the SQLite busy holder, and the disk simulator without needing an external cloop process), or
• a probe that reaches into a running cloop ui via HTTP (callers can provide a custom Probe that hits /healthz, runs a task, etc).

func (s *Suite) Run(ctx context.Context) ([]Run, error)

type SuiteCase ¶

type SuiteCase struct {
	// Fault is injected when the case runs. Until-StartedAt is its window,
	// which Run measures from the start of the case rather than from when
	// the case was built; a Fault with no StartedAt keeps Until as given.
	Fault       Fault
	Description string
	// Probe is the function the suite runs against the system under test
	// while the fault is active. It returns the outcome and a short detail
	// string. Probes must complete inside the fault's window or the suite
	// considers the run "degraded".
	Probe func(ctx context.Context) (Outcome, string)
}

SuiteCase describes one chaos test case. The default suite covers every FaultType in turn; callers can extend or subset it via WithCases.

func DefaultCases(workDir string) []SuiteCase

type SummaryRow ¶

type SummaryRow struct {
	FaultType   FaultType
	Total       int
	Recovered   int
	Degraded    int
	Crashed     int
	Unknown     int
	LastRunAt   time.Time
	AvgDuration time.Duration
}

SummaryRow aggregates chaos_runs entries by fault type for the headline table that `cloop chaos report` prints.

func SummaryRows(runs []Run) []SummaryRow

type Transport ¶

type Transport struct {
	// Base is the underlying transport that handles non-faulted requests. nil
	// means use http.DefaultTransport, matching the convention for net/http
	// middleware. Tests typically inject an httptest.NewServer's transport.
	Base http.RoundTripper

	// Controller is consulted on every request. nil means "use the global
	// controller" — see chaos.Global. If Global() is also nil, the transport
	// is fully transparent.
	Controller *Controller
}

Transport is an http.RoundTripper that consults a Controller before dispatching each request, and replaces the response with a fault when one is active. When no controller is installed (or no faults match), it forwards the request to Base unchanged.

The hot path adds one atomic load + one type comparison per request.

Why a wrapper rather than a Client option: Anthropic, OpenAI and Ollama all build their *http.Client via provider.NewHTTPClient(), which is the single integration point. Wiring chaos into NewHTTPTransport via this wrapper means every provider — present and future — gets fault injection for free.

func (t *Transport) RoundTrip(req *http.Request) (*http.Response, error)