Documentation
¶
Overview ¶
Package testkit is shared test infrastructure for the agent: deterministic fault injection and invariant checks that make rigorous tests cheap to write.
Chaos engineering falls out of the ports architecture - wrap a unit of work or any port (dispatch work, dispatch.EventSink, ...) with a FaultPlan and assert the system degrades and recovers cleanly. Combined with clock.Manual and seeded inputs, the faults are deterministic, so a failing run reproduces exactly.
Index ¶
- func ActionGen() *rapid.Generator[dispatch.Action]
- func AppendInputGen(stream string) *rapid.Generator[spine.AppendInput]
- func CountByType(events []dispatch.Event) map[string]int
- func Diff[T any](t TB, want, got T)
- func DiffEvents(t TB, want, got []dispatch.Event)
- func FaultyBus(inner bus.Bus, plan *FaultPlan) bus.Bus
- func FaultyDoer(inner request.Doer, plan *FaultPlan) request.Doer
- func FaultyExecutor(inner goal.StepExecutor, plan *FaultPlan) goal.StepExecutor
- func FaultyLog(inner spine.Log, plan *FaultPlan) spine.Log
- func FaultyModel(inner llm.Model, plan *FaultPlan) llm.Model
- func FaultyQueue(inner jobs.Queue, faults QueueFaults) jobs.Queue
- func FaultyReader(inner io.Reader, plan *FaultPlan) io.Reader
- func FaultySink(inner dispatch.EventSink, plan *FaultPlan) dispatch.EventSink
- func FaultySource(inner secret.Source, plan *FaultPlan) secret.Source
- func FaultyWork(inner func(context.Context) (dispatch.Metering, error), plan *FaultPlan) func(context.Context) (dispatch.Metering, error)
- func FaultyWriter(inner io.Writer, plan *FaultPlan) io.Writer
- func Golden[T any](t TB, name string, got T)
- func MemoryItemGen() *rapid.Generator[state.MemoryItem]
- func RequireLifecycle(t TB, events []dispatch.Event)
- func ScopeGen() *rapid.Generator[state.Scope]
- func SkillGen() *rapid.Generator[state.Skill]
- type FaultPlan
- type QueueFaults
- type TB
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func AppendInputGen ¶
func AppendInputGen(stream string) *rapid.Generator[spine.AppendInput]
AppendInputGen generates a spine.AppendInput for a fixed stream.
func CountByType ¶
CountByType tallies events by their Type, for assertions on what was emitted.
func Diff ¶
Diff fails the test with a readable diff when want != got. It is the generic comparison primitive for any type, so packages don't write a bespoke DiffXxx; nil and empty slices/maps compare equal.
func DiffEvents ¶
DiffEvents is Diff specialised to a dispatch-event stream, kept for readability at call sites. Two runs of the same scenario under a clock.Manual produce byte-identical streams, so this is also the determinism/replay check.
func FaultyBus ¶
FaultyBus wraps a bus.Bus, injecting plan's faults on Publish before delegating; Subscribe and Close pass through. It proves a publisher tolerates a flaky or dead signal bus: a failed or dropped publish must never corrupt the durable record a subscriber eventually reads. A nil plan never faults. A nil inner bus models a fully dead bus: Publish is a silent no-op and Subscribe returns an inert subscription that never delivers, so a consumer must make progress some other way (a poll floor).
func FaultyDoer ¶
FaultyDoer wraps a request.Doer, injecting plan's faults before delegating, so a flaky network (a few transient failures then recovery, or a hard outage) is modelled deterministically against the shared HTTP transport. The injected error flows through the transport's classification, so a fault.Transient drives the retry path and a fault.Terminal proves a non-retryable failure is not replayed. A nil inner doer returns a minimal 200 response when not faulting, so a plan alone is enough to drive the transport through its retry and recovery path.
func FaultyExecutor ¶
func FaultyExecutor(inner goal.StepExecutor, plan *FaultPlan) goal.StepExecutor
FaultyExecutor wraps a goal.StepExecutor, injecting plan's faults before delegating, so a flaky model or tool call (a step that fails a few times then succeeds) is modelled deterministically. A nil inner executor performs no work when not failing, so a plan alone is enough to drive a goal through its retry and recovery path. The checkpoint of a faulting call is dropped: a failed step makes no progress.
func FaultyLog ¶
FaultyLog wraps a spine.Log, injecting plan's faults on Append before delegating; Read passes through. It models a flaky durable log: an append that fails records nothing and assigns no Seq, so the stream stays gap-free (a dropped event simply never existed) rather than leaving a hole. A nil plan never faults; a nil inner log returns a zero Event on a non-faulting append and reads back nothing.
func FaultyModel ¶
FaultyModel wraps an llm.Model, injecting plan's faults on Generate before delegating. It models a flaky provider (a transient API error) so a test can drive the conversation loop's retry path, where the fault lands AFTER a turn has already been announced. That is the case the well-formedness invariant guards: a retried turn must not duplicate its turn.started on the event stream.
func FaultyQueue ¶ added in v0.1.3
func FaultyQueue(inner jobs.Queue, faults QueueFaults) jobs.Queue
FaultyQueue wraps a jobs.Queue, injecting plan's faults before delegating, so a durable queue whose backing store is failing (a disk that is full, a database that is down, a write that is lost mid-commit) is modelled deterministically against the worker loop that drives it.
The faults land where at-least-once delivery is actually decided: a Complete or Fail that never reaches the store leaves the job running, and it is the lapse of its lease, not any bookkeeping the worker did, that returns the job to the queue. A wrapped queue therefore proves the loop cannot lose a job by failing to record its outcome, and cannot retry one past its attempt ceiling either, since the reclaim spends an attempt exactly as a crash would.
Get is never faulted: tests assert against it, so it stays a truthful oracle.
func FaultyReader ¶
FaultyReader wraps an io.Reader, injecting plan's faults before delegating, so an input stream that fails mid-read (a hung console, a torn multiplexer session) is modelled deterministically against anything that consumes it. A firing fault consumes nothing and returns the error; a nil inner reader reports io.EOF when not faulting.
func FaultySink ¶
FaultySink wraps a dispatch.EventSink, injecting plan's faults before delegating - used to prove that event-sink failures never break a dispatch. A nil inner sink discards events when not failing.
func FaultySource ¶
FaultySource wraps a secret.Source, injecting plan's faults on Lookup before delegating, so a flaky or unreachable credential store (a locked OS keychain, an unreachable vault or broker) is modelled deterministically. It proves a credential consumer fails closed on a resolve failure: a step that needs a secret must stop, never proceed without it or act on a partial value. A nil inner source resolves every reference to an empty value when not faulting, so a plan alone drives the consumer's failure path.
func FaultyWork ¶
func FaultyWork(inner func(context.Context) (dispatch.Metering, error), plan *FaultPlan) func(context.Context) (dispatch.Metering, error)
FaultyWork wraps a unit of dispatch work, injecting plan's faults before delegating. A nil inner returns zero Metering when not failing, so a plan alone is enough to drive a Dispatcher.Govern call through its retry and recovery path.
func FaultyWriter ¶
FaultyWriter wraps an io.Writer, injecting plan's faults before delegating, so a terminal that dies mid-session (a closed pty, a dropped SSH connection) is modelled deterministically against anything that renders. A firing fault writes nothing and returns the error; a nil inner writer discards writes when not faulting, so a plan alone can drive a renderer's error path.
func Golden ¶
Golden compares got against testdata/<name>.golden (pretty JSON). Run the tests with -update to (re)write the file. It lets an entire output - a full mission replay, a rendered spec, a whole event stream - be a single snapshot with no hand-written expected value, which is what keeps large tests small.
func MemoryItemGen ¶
func MemoryItemGen() *rapid.Generator[state.MemoryItem]
MemoryItemGen generates a state.MemoryItem.
func RequireLifecycle ¶
RequireLifecycle asserts a coherent event lifecycle on a recorded stream: every dispatched action is either a start paired with an end, or a single rejection. An imbalance means an action was lost or double-recorded.
Types ¶
type FaultPlan ¶
type FaultPlan struct {
// contains filtered or unexported fields
}
FaultPlan decides, deterministically, when to inject a fault. Each wrapped call advances a counter; the plan returns an error to inject on that call, or nil to pass through. Safe for concurrent use.
func FailFirst ¶
FailFirst injects err on the first k calls, then passes through. Models a flaky dependency that recovers - exercises retry and degrade paths.
func FailOnCall ¶
FailOnCall injects err only on the call with the given 1-based index.
type QueueFaults ¶ added in v0.1.3
type QueueFaults struct {
Enqueue *FaultPlan
Claim *FaultPlan
Complete *FaultPlan
Fail *FaultPlan
Recover *FaultPlan
}
QueueFaults holds one FaultPlan per jobs.Queue method a durable store can fail at. Each plan counts only its own method's calls, so a case can fail Complete on its second call without the intervening Claims advancing that counter. A nil plan never faults, so a case names only the method it is injecting at.
Source Files