Documentation
¶
Overview ¶
Package faulttest injects deterministic faults into Gombit's own tests.
A fault is expressed as "fail exactly the second call", never as a probability: an Injector walks a fixed list of steps, one per call, and every call it sees is counted, so a test can reproduce a failure path on every run and assert what the code under test did about it.
inj := faulttest.FailOnCall(2, errConnLost) // the 2nd call fails
err := inj.Do(ctx, func(ctx context.Context) error { return dep.Call(ctx) })
Injectors wrap a dependency at a narrow seam: Do around a function, or the database/sql wrapper in this package (OpenDB, WrapConnector) for GORM and transactions. Nothing here is global and nothing changes production code paths: a test opts in by building the wrapped dependency itself.
A test that sets up over the same wrapped dependency (migrations, fixtures) disarms the injector first and arms it for the calls under test.
Delays and blocks wait on timers, channels, and the call's context, never on sleeps the test has to guess at: a test that needs a call to be in flight waits on Reached(n), and releases it by closing the channel it handed to BlockUntil (or Block).
Index ¶
- Variables
- func CheckRetryPolicy(t testing.TB, c RetryContract)
- func ForEachDB(t *testing.T, dbs []TestDB, ...)
- func Idle(t testing.TB, db *database.DB)
- func Inserts(table string) func(query string) bool
- func OpenDB(kind database.Driver, dsn string, faults *DBFaults) (*database.DB, error)
- func PostgresDSNVia(dsn, addr string) (string, error)
- func PostgresHostPort(dsn string) (string, error)
- func WrapConnector(c driver.Connector, faults *DBFaults) driver.Connector
- type BodyTracker
- type DBFaults
- type FakeSleeper
- type HTTPDependency
- func (d *HTTPDependency) Abandoned() int
- func (d *HTTPDependency) Calls() int
- func (d *HTTPDependency) Client() *http.Client
- func (d *HTTPDependency) Close()
- func (d *HTTPDependency) InFlight() int
- func (d *HTTPDependency) Otherwise(step HTTPStep) *HTTPDependency
- func (d *HTTPDependency) Reached(n int) <-chan struct{}
- func (d *HTTPDependency) URL() string
- func (d *HTTPDependency) WaitIdle(t testing.TB)
- type HTTPStep
- type Injector
- func BlockUntil(release <-chan struct{}) *Injector
- func Delay(d time.Duration) *Injector
- func FailAlways(err error) *Injector
- func FailNTimes(n int, err error) *Injector
- func FailOnCall(n int, err error) *Injector
- func FailOnce(err error) *Injector
- func Sequence(steps ...Step) *Injector
- func SequenceThen(rest Step, steps ...Step) *Injector
- func (i *Injector) Arm()
- func (i *Injector) Calls() int
- func (i *Injector) Disarm() *Injector
- func (i *Injector) Do(ctx context.Context, fn func(context.Context) error) error
- func (i *Injector) Failures() int
- func (i *Injector) Hit(ctx context.Context) error
- func (i *Injector) Reached(n int) <-chan struct{}
- func (i *Injector) Reset()
- type RealSleeper
- type RetryContract
- type Sleeper
- type Step
- type TCPProxy
- type TestDB
Constants ¶
This section is empty.
Variables ¶
var ErrInjected = errors.New("faulttest: injected fault")
ErrInjected is a ready-made fault for tests that do not need a specific error.
Functions ¶
func CheckRetryPolicy ¶
func CheckRetryPolicy(t testing.TB, c RetryContract)
CheckRetryPolicy runs c.Do through the retry contract and reports every violated property (by name) through t.Errorf:
- bounded: an op that always fails retryably is tried no more than MaxAttempts times;
- retryable: ...and no fewer (a retryable failure is retried while the budget lasts), and an op that fails once and then succeeds succeeds on the retry;
- cancellation: a context that has already ended makes Do return the context's error without starting an attempt; a context canceled mid-backoff ends Do promptly with the context's error, and no further attempt runs;
- permanent: a non-retryable error is not retried;
- backoff: the waits asked for follow Delay (exactly, when Deterministic) and lie within [0, MaxDelay]; and Delay itself, called 20 times per attempt (for jitter) at every attempt 1..max(MaxAttempts, 128) and at 1000, is within [0, MaxDelay]. That catches a shift or doubling that overflows: any base of 1ns or more wraps by attempt 64. It does not catch an attempt multiplied by a unit, which wraps at an attempt that depends on the unit, usually far past 1000: compute a delay so it cannot overflow (cap before the arithmetic, not after);
- success: a success ends the retries at once;
- observable failure: the final error wraps the last attempt's error (and reports the attempt count, when Attempts is set).
Each run of the policy is bounded by a guard: a policy that never returns fails the check, and its goroutine is left running for the rest of the test binary (Go cannot stop it), so a broken policy should be fixed, not kept failing. The contract describes attempt-bounded policies with a Delay per attempt; a policy bounded by a total deadline states the attempts that deadline allows.
func Idle ¶
Idle waits until no connection of db is in use, and fails t if one stays in use: a transaction a failure left open (never committed or rolled back) holds its connection, and row counts on another connection cannot see it. database/sql releases a context-canceled transaction's connection on its own goroutine, so this waits (up to 5s) rather than sampling once.
func Inserts ¶
Inserts matches INSERT statements into table, however the dialect quotes it (a DBFaults.Match).
func OpenDB ¶
OpenDB opens kind's database at dsn through a connection wrapped with faults, as a *database.DB (database.OpenConn): code under test gets the same GORM setup as from database.Open. Opening is not part of any fault sequence: every injector is suspended while GORM opens (its ping, a dialector's version query), then restored to how it was armed and reset, and the connections the open dialed are closed. So call 1 is the test's own first call on every driver: Statement FailOnce fails the test's first statement, and a Connect fault meets the test's first dial. The pool gets database.Open's defaults for the driver (ConfigurePool). Close the DB when done.
func PostgresDSNVia ¶
PostgresDSNVia points a postgres:// URL DSN at addr, keeping everything else.
func PostgresHostPort ¶
PostgresHostPort returns a postgres:// URL DSN's host:port: the upstream to proxy.
Types ¶
type BodyTracker ¶
type BodyTracker struct {
// contains filtered or unexported fields
}
BodyTracker is an http.RoundTripper that counts response bodies opened and not yet closed, to prove a client closes every body, failures included.
func TrackBodies ¶
func TrackBodies(base http.RoundTripper) *BodyTracker
TrackBodies wraps base (http.DefaultTransport when nil).
func (*BodyTracker) Open ¶
func (b *BodyTracker) Open() int
Open reports the bodies returned and not closed.
type DBFaults ¶
type DBFaults struct {
// Connect is hit when the pool opens a new connection.
Connect *Injector
// Begin is hit when a transaction starts.
Begin *Injector
// Statement is hit by every Exec and Query, inside a transaction or not,
// before the statement reaches the database: a failed statement has no
// effect. With Match set, only matching statements are hit (and counted).
Statement *Injector
// Match limits Statement to the statements it reports true for (nil:
// all). Dialects quote identifiers differently, so match on the table
// name rather than a quoted form.
Match func(query string) bool
// Commit is hit before a transaction commits. A failed commit rolls the
// transaction back and returns the injected error: nothing it did
// persists, and its connection goes back to the pool clean. database/sql
// gives Commit and Rollback no context, so a Delay or BlockUntil here
// ends only by its timer or channel, never by the caller's context.
Commit *Injector
// Rollback is hit before a transaction rolls back. The rollback still
// happens (the connection must not stay inside the transaction); the
// injected error is what the caller sees. When the real rollback fails
// too (after either fault), the error also carries it and
// driver.ErrBadConn, so the pool discards the connection.
Rollback *Injector
}
DBFaults are the faults a wrapped database/sql driver injects, one Injector per boundary. A nil Injector never fails. Set them before the connection is opened; an Injector's own state (counts, Reset) can be used at any time.
func (*DBFaults) Arm ¶
func (f *DBFaults) Arm()
Arm arms every injector: the next call at each boundary is its call 1.
type FakeSleeper ¶
type FakeSleeper struct {
// Blocking makes Sleep wait until ctx ends.
Blocking bool
// contains filtered or unexported fields
}
FakeSleeper records every delay it is asked for and returns at once (a context that has already ended still returns its error). With Blocking set, it instead waits for the context to end: a retry that is mid-backoff when it is canceled.
func (*FakeSleeper) Delays ¶
func (s *FakeSleeper) Delays() []time.Duration
Delays returns the delays asked for, in order.
func (*FakeSleeper) Sleeping ¶
func (s *FakeSleeper) Sleeping() <-chan struct{}
Sleeping returns a channel closed once the first Sleep has begun.
type HTTPDependency ¶
type HTTPDependency struct {
// contains filtered or unexported fields
}
HTTPDependency is a loopback HTTP server standing in for a dependency. Each call gets the next scripted step; calls after the script get the fallback (200 with an empty JSON object unless set with Otherwise). It counts calls, reports when a call arrives (Reached), and when a handler returns (InFlight), so a test can prove none is left running.
func NewHTTPDependency ¶
func NewHTTPDependency(t testing.TB, steps ...HTTPStep) *HTTPDependency
NewHTTPDependency starts a dependency answering with steps, one per call, and closes it when t ends.
func (*HTTPDependency) Abandoned ¶
func (d *HTTPDependency) Abandoned() int
Abandoned reports how many calls the client gave up on (canceled, timed out, disconnected) before they were answered; calls ended by Close do not count.
func (*HTTPDependency) Calls ¶
func (d *HTTPDependency) Calls() int
Calls reports how many calls reached the dependency.
func (*HTTPDependency) Client ¶
func (d *HTTPDependency) Client() *http.Client
Client is an HTTP client for the dependency (loopback only).
func (*HTTPDependency) Close ¶
func (d *HTTPDependency) Close()
Close releases any hanging call and shuts the dependency down, waiting for its handlers to return.
func (*HTTPDependency) InFlight ¶
func (d *HTTPDependency) InFlight() int
InFlight reports how many calls are being answered now.
func (*HTTPDependency) Otherwise ¶
func (d *HTTPDependency) Otherwise(step HTTPStep) *HTTPDependency
Otherwise sets the answer for calls past the script.
func (*HTTPDependency) Reached ¶
func (d *HTTPDependency) Reached(n int) <-chan struct{}
Reached returns a channel closed once call n has arrived.
func (*HTTPDependency) URL ¶
func (d *HTTPDependency) URL() string
URL is the dependency's base URL.
func (*HTTPDependency) WaitIdle ¶
func (d *HTTPDependency) WaitIdle(t testing.TB)
WaitIdle waits until no call is being answered (the server notices a client that went away on its own goroutine), and fails t if one still is after 5s: a handler left running is a leak.
type HTTPStep ¶
type HTTPStep struct {
// contains filtered or unexported fields
}
HTTPStep is how the dependency answers one call.
func CutBody ¶
CutBody sends the status line, headers promising a longer body, and partial of it, then closes the connection: the client's read fails mid-body.
func Hang ¶
func Hang(release <-chan struct{}) HTTPStep
Hang holds the call without answering until release is closed (then answers 200 with an empty JSON object), the client goes away, or the dependency is closed: a dependency slower than any deadline, driven by a channel rather than a sleep.
func ResetConnection ¶
func ResetConnection() HTTPStep
ResetConnection closes the connection with a TCP reset before answering: the client sees "connection reset by peer" (or an EOF), never a response.
func TooManyRequests ¶
TooManyRequests answers 429 with Retry-After, which HTTP gives in whole seconds: retryAfter is rounded up, so a sub-second wait is 1, never 0 ("retry now").
type Injector ¶
type Injector struct {
// contains filtered or unexported fields
}
Injector decides, call by call, whether a call fails. It is safe for concurrent use: calls are numbered in the order they reach it. A nil *Injector lets every call through, so an optional fault can be left unset.
func BlockUntil ¶
func BlockUntil(release <-chan struct{}) *Injector
BlockUntil holds every call until release is closed (see Block).
func FailNTimes ¶
FailNTimes fails the first n calls with err and lets the rest through.
func FailOnCall ¶
FailOnCall fails only the nth call (1-based) with err.
func Sequence ¶
Sequence applies steps to the first calls, one each, and lets every call after them through.
func SequenceThen ¶
SequenceThen applies steps to the first calls, one each, and rest to every call after them (Sequence is SequenceThen(Success(), ...)).
func (*Injector) Arm ¶
func (i *Injector) Arm()
Arm ends Disarm and starts over (Reset): the next call is call 1.
func (*Injector) Disarm ¶
Disarm lets every call through without counting it, until Arm: a test disarms an injector while it sets up (migrations, fixtures, a login) over the same wrapped dependency, so only the calls under test are numbered.
func (*Injector) Failures ¶
Failures reports how many calls it failed (a context that ended during a wait or block included).
func (*Injector) Hit ¶
Hit counts a call and applies its step: nil to let it through, or the error it fails with.
func (*Injector) Reached ¶
Reached returns a channel closed once call n (1-based) has reached the injector, before its step runs: a test waits on it to know a blocked call is in flight.
func (*Injector) Reset ¶
func (i *Injector) Reset()
Reset starts the injector over: counts go to zero and the next call is call 1 again. A call still in flight from before (a blocked one, say) finishes as it would have, but no longer counts. Channels from Reached for calls not yet reached stay pending, numbered from the new start.
type RetryContract ¶
type RetryContract struct {
// Do runs op under the policy, waiting between attempts through
// sleeper. It is the policy under test.
Do func(ctx context.Context, sleeper Sleeper, op func(context.Context) error) error
// Retryable is an error the policy retries; Permanent one it must not.
Retryable, Permanent error
// MaxAttempts is the policy's attempt budget, the first attempt
// included: an op that keeps failing retryably runs exactly MaxAttempts
// times (never more, which is the bound; never fewer, since a retryable
// failure is retried while budget remains).
MaxAttempts int
// Delay is the policy's backoff: the wait after the attempt-th failure
// (1-based). Every delay it produces must lie within [0, MaxDelay] (no
// negative or overflowed durations, jitter included). The checker samples
// that (see CheckRetryPolicy's backoff bullet, which says what the sample
// does not catch): cap a delay before the arithmetic can overflow, not
// after.
Delay func(attempt int) time.Duration
// MaxDelay caps Delay.
MaxDelay time.Duration
// Deterministic says Delay is exact (no jitter): the waits the policy
// asks the sleeper for must equal Delay(1), Delay(2), ...
Deterministic bool
// Attempts, when set, reads the attempt count from the policy's final
// error (a diagnostic the policy promises to expose).
Attempts func(err error) (int, bool)
}
RetryContract describes a retry policy for CheckRetryPolicy (INV-4: every retry is bounded, honors cancellation, and ends in success or an observable terminal error).
type Sleeper ¶
type Sleeper interface {
// Sleep waits for d, or until ctx ends (returning ctx's error).
Sleep(ctx context.Context, d time.Duration) error
}
Sleeper is the time seam a retry policy waits through between attempts, so its backoff can be tested without sleeping.
type Step ¶
type Step struct {
// contains filtered or unexported fields
}
Step is what one call does: succeed, fail with an error, wait, or block.
func Block ¶
func Block(release <-chan struct{}) Step
Block holds the call until release is closed, then lets it through. A context that ends first fails the call with the context's error.
func BlockThenFail ¶
BlockThenFail holds the call until release is closed, then fails it with err: a call that stalls, and then its dependency gives up (the second writer waits on the first's lock, which then rolls back). A context that ends first fails the call with the context's error.
type TCPProxy ¶
type TCPProxy struct {
// contains filtered or unexported fields
}
TCPProxy sits between the code under test and a real dependency (a test database, a test Redis) on loopback and faults the transport on command: a test toggles each fault at an explicit point, so the scenario is the same on every run. It is in-process: no proxy service to run, and it can only reach the upstream the test names.
proxy := faulttest.NewTCPProxy(t, "127.0.0.1:5432") dsn := faulttest.PostgresDSNVia(dsn, proxy.Addr()) proxy.Hold() // the dependency stops answering (latency past any deadline) proxy.Cut() // every open connection drops proxy.Refuse() // new connections are refused proxy.Heal() // all of it lifted: the next operation must succeed
func NewTCPProxy ¶
NewTCPProxy starts a proxy to upstream on a loopback port, and closes it when t ends.
func (*TCPProxy) Addr ¶
Addr is the proxy's address: point the code under test here instead of at the upstream.
func (*TCPProxy) Close ¶
func (p *TCPProxy) Close()
Close stops the proxy and drops every connection.
func (*TCPProxy) Connections ¶
Connections reports how many connections the proxy has accepted.
func (*TCPProxy) Cut ¶
func (p *TCPProxy) Cut()
Cut closes every open connection, both sides (a connection lost mid-query).
func (*TCPProxy) Heal ¶
Heal lifts every fault: held traffic flows, and a refusing proxy listens again on the same address. Bytes held on a connection that is still open are delivered, as a recovering network would, even if their caller gave up meanwhile; those on a connection that closed are dropped with it. Connections that were cut stay cut; the next operation has to reconnect, which is the recovery a test asserts.
func (*TCPProxy) Held ¶
func (p *TCPProxy) Held() <-chan struct{}
Held returns a channel closed once traffic is waiting on the current Hold: the proxy has the call's bytes and has not forwarded them (the dependency has not seen the call yet).
func (*TCPProxy) Hold ¶
func (p *TCPProxy) Hold()
Hold stops forwarding in both directions: connections stay open, and every byte waits until Heal. To the client the dependency is slower than any deadline.
Source Files