README
¶
Fault Injection
Fault injection is a chaos engineering utility that allows you to test error scenarios with real components by adding fault points.
The package has strictly tested.
Getting Started
To allow your code to be injected with faults, you must follow the context-checking idiom from the stdlib. This way, your code does a common practice with context verification, and you gain the ability to inject fault as well with the existing code base.
if err := ctx.Err(); err != nil {
return err
}
// or
if ctx.Err() != nil {
return ctx.Err()
}
// or
select {
case <-ctx.Done():
return ctx.Err()
}
If you feel adventurous and want to do chaos engineering, then you can use Context#Value to look for injected errors for a given FaultName key. E.g., in a round-trip middleware, you inject a timeout error and test retries logic with it.
type NamedFault struct{}
// optionally you can hard code an error that makes the most sense in your code as a return value
if ctx.Value(NamedFault{}) != nil {
return ErrMyDomainErr
}
// optionally you can also accept an injected error value
if err, ok := ctx.Value(NamedFault{}).(error); ok {
return err
}
To utilise these existing conventions, the testcase/faultinject package provides you with the following tools:
- faultinject.Inject
- Allows you to inject consumable faults into a context. Consumable faults are removed upon retrieval, thus allowing testing retry mechanism.
- CallerFault
- Allows you to define what package/function/receiver should trigger an error in Context#Err.
Features
- You can add fault points to specific points
- This simplifies your expectations in your integration tests; you can trigger these fault points instead of analyzing the underlying error handling of a given component when you don't want the test of the implementation details of a given dependent component
- The ability to simulate temporary errors
- This allows testing retry logic with ease without the need to build and maintain intelligent mocks that try to mimic real components
- Global Enable switch to allow fault injection on demand
- By default, fault injection ignores all calls, doesn't check, doesn't inject unless Fault Injection is explicitly allowed
Example
The Fault injection package doesn't depend on the testing package and should be safe to use in production code.
Description
This approach enables you to test out small error cases or event the cascading effects in a microservice setup. One of the instant benefits of fault injection is that your clients can test with your actual errors and don't need to maintain their mocks/stubs arrangements manually. If fuel injection is exposed on your API, then It also enables your clients to write integration tests against error scenarios with your system's API. Last but not least, it allows you to remove forced indirections from your codebase, where you have to use a header interface for the sake of testing error handling in a component.
One often mentioned argument about fault injection is the need to add something to the production codebase for testing, but in practice, if you have many header interfaces in your codebase, then you are already actively altering your production codebase for testing purposes, In the end, you need to judge if header interface-based indirections or fault injection makes more sense for your use-cases, as this is not a silver bullet.
By this time, I believe you might feel reticent to put fault injection into your non-test code. Engineering controlled chaos into your application is not a standard testing strategy. It has its pros and cons. For example, you can simplify your code through using less header interface based indirection to test error cases in your code. It allows you to trigger faults without the need to understand the internal logic of that concrete implementation's error cases. It also allows you to specify expectations about fault injection in your Role Interface's interface testing suite. You can do simulation of temporary outages and test retry mechanisms.
But on the grand scale, the real value with fault injection is the ability to test error cases at the system level in a micro-service setup, where errors can have unexpected cascading effects. I loved to see how easy to find bugs with fault injection when I was working with a mobile team in one of my previous job. Our biggest issue was that after you released a mobile client, it was a pain point to make our users upgrade to the latest client version when we identified rainy cases during production use.
Limitations / Known Issues
When context.cancelCtx is wrapping the faultinject.injectContext,
the Err() method is not propagated from context.cancelCtx down to the parent context,
instead it swallow the error checking until a Done closing is triggered.
To work around this limitation, call the context Value() method with any value to ensure fault injection checking.
// manual fault injection checking,
// the key for the value doesn't matter.
_ = ctx.Value(42)
return ctx.Err()
Or alternatively you can use the faultinject.Check method for a specific fault tag.
type FaultName struct{}
if err := faultinject.Check(ctx, FaultName{}); err != nil {
return // err
}
Documentation
¶
Overview ¶
Example ¶
package main
import (
"context"
"fmt"
"go.llib.dev/testcase/faultinject"
)
type FaultTag struct{}
func main() {
defer faultinject.Enable()()
ctx := context.Background()
fmt.Println(ctx.Err()) // no error as expected
// arrange one fault injection for FaultTag
ctx = faultinject.Inject(ctx, FaultTag{}, fmt.Errorf("example error to inject"))
if err, ok := ctx.Value(FaultTag{}).(error); ok {
fmt.Println(err) // prints the injected error
}
if err, ok := ctx.Value(FaultTag{}).(error); ok {
fmt.Println(err) // code not reached as injectedFault is already consumed
}
}
Example (ChaosEngineeringWithExplicitFaultPoints) ¶
package main
import (
"context"
"errors"
"fmt"
"go.llib.dev/testcase/faultinject"
)
type FaultTag struct{}
func main() {
defer faultinject.Enable()()
ctx := context.Background()
fmt.Println(MyFuncWithChaosEngineeringFaultPoints(ctx)) // no error
ctx = faultinject.Inject(ctx, FaultTag{}, errors.New("boom")) // arrange fault injection for FaultTag
fmt.Println(MyFuncWithChaosEngineeringFaultPoints(ctx)) // "boom" is returned
}
func MyFuncWithChaosEngineeringFaultPoints(ctx context.Context) error {
if err, ok := ctx.Value(FaultTag{}).(error); ok {
return err
}
if err := ctx.Err(); err != nil {
return err
}
return nil
}
Example (FaultInjectWithFixErrorReplyFromTheFaultPoint) ¶
package main
import (
"context"
"errors"
"fmt"
"go.llib.dev/testcase/faultinject"
)
type FaultTag struct{}
func main() {
defer faultinject.Enable()()
ctx := context.Background()
ctx = faultinject.Inject(ctx, FaultTag{}, errors.New("ignored"))
fmt.Println(MyFuncWithFixErrorReplyFromTheFaultPoin(ctx)) // error is returned
}
func MyFuncWithFixErrorReplyFromTheFaultPoin(ctx context.Context) error {
if _, ok := ctx.Value(FaultTag{}).(error); ok {
return errors.New("my error value")
}
return nil
}
Index ¶
- Constants
- Variables
- func After(returnErr *error, ctx context.Context, faults ...any)
- func Check(ctx context.Context, faults ...any) error
- func Enable() (Disable func())
- func EnableForTest(tb testingTB)
- func Enabled() bool
- func Finish(returnErr *error, ctx context.Context, faults ...any)deprecated
- func Inject(ctx context.Context, fault any, err error) context.Context
- type CallerFault
Examples ¶
Constants ¶
const DefaultErr errT = "fault injected"
Variables ¶
var WaitForContextDoneTimeout = time.Second / 2
Functions ¶
func After ¶
After is function that can be called from a deferred context, and will inject fault after the function finished its execution. The error pointer should point to the function's named return error variable. If the function encountered an actual error, fault injection is skipped. It is safe to use from production code.
Example ¶
package main
import (
"context"
"fmt"
"go.llib.dev/testcase/faultinject"
)
func main() {
type fault struct{}
ctx := faultinject.Inject(context.Background(), fault{}, fmt.Errorf("boom"))
_ = func(ctx context.Context) (returnErr error) {
defer faultinject.After(&returnErr, ctx, fault{})
return nil
}(ctx)
}
func Check ¶
Check is a fault-injection helper method which check if there is an injected fault(s) in the given context. It checks for errors injected as context value, or ensures to trigger a CallerFault. It is safe to use from production code.
Example ¶
package main
import (
"context"
"go.llib.dev/testcase/faultinject"
)
func main() {
type FaultName struct{}
ctx := context.Background()
if err := faultinject.Check(ctx, FaultName{}); err != nil {
return // err
}
}
func EnableForTest ¶
func EnableForTest(tb testingTB)
func Finish
deprecated
Finish is an alias for After
Deprecated: use After instead
Example ¶
package main
import (
"context"
"go.llib.dev/testcase/faultinject"
)
func main() {
type FaultName struct{}
ctx := context.Background()
_ = func(ctx context.Context) (rErr error) {
defer faultinject.After(&rErr, ctx, FaultName{})
return nil
}(ctx)
}
func Inject ¶
Inject will arrange context to trigger fault injection for the provided fault.
Example (ByTargetingCaller) ¶
package main
import (
"context"
"errors"
"fmt"
"go.llib.dev/testcase/faultinject"
)
func main() {
ctx := faultinject.Inject(
context.Background(),
faultinject.CallerFault{
Package: "", // empty will match everything
Receiver: "", //
Function: "", //
},
errors.New("boom"),
)
fmt.Println(ctx.Err()) // "boom"
}
Types ¶
type CallerFault ¶
CallerFault allows you to inject Fault by Caller stack position.
Example ¶
package main
import (
"context"
"fmt"
"go.llib.dev/testcase/faultinject"
"go.llib.dev/testcase/random"
)
func main() {
defer faultinject.Enable()()
ctx := context.Background()
fmt.Println(MyFuncWithStandardContextErrCheck(ctx)) // no error
fault := faultinject.CallerFault{ // inject Fault that targets a specific context Err check
Function: "MyFuncWithStandardContextErrCheck", // selector to tell where to inject
}
err := random.New(random.CryptoSeed{}).Error()
ctx = faultinject.Inject(ctx, fault, err) // some random error)
fmt.Println(MyFuncWithStandardContextErrCheck(ctx)) // Fault.Error injected and returned
}
func MyFuncWithStandardContextErrCheck(ctx context.Context) error {
if err := ctx.Err(); err != nil {
return err
}
return nil
}
Source Files
Directories