README
¶
Gremlins
A mutation testing tool for Go
Gremlins is a mutation testing tool for Go. It has been made to work well on smallish Go modules, for example microservices, on which it helps validate the tests, aids the TDD process and can be used as a CI quality gate. As of now, Gremlins doesn't work very well on very big Go modules, mainly because a run can take hours to complete.
Gremlins version
Gremlins is still in its 0.x.x release, which, as per SemVer, doesn't guarantee backward compatibility. What this means for Gremlins is that configuration flags and/or configuration files can change among minor releases. This may be important if you use Gremlins in automated builds, for example in continuous integration pipelines.
Also, while in 0.x.x releases, only the current minor release will be maintained; if we are in v0.2.0, no patch releases will be provided for the v0.1.0 line. Once Gremlins will reach it's 1.x.x release, it will start to be backward compatible, and we'll decide how many releases to maintain at the same time (ex. up to two minors before, as Go itself does).
What is Mutation Testing
Code coverage is unreliable as a measure of test quality. It is too easy to have tests that exercise a piece of code but don't test anything at all.
Mutation testing works by mutating the code exercised by the tests and verifying if the mutation is caught by the test suite. Imagine gremlins going into your code and messing around: will your test suit catch their damage?
Here is a nice intro to mutation testing.
How to use Gremlins
Please refer to the documentation for instructions on how to obtain, configure and use Gremlins.
Quick start
This is just to get you started, do not forget to check the complete documentation.
Download the pre-built binary for your OS/ARCH from
the release page
and put it somewhere in the PATH, then:
gremlins unleash
Gremlins will report each mutation as:
RUNNABLE: In dry-run mode, a mutation that can be tested.NOT COVERED: A mutation not covered by tests; it will not be tested.KILLED: The mutation has been caught by the test suite.LIVED: The mutation hasn't been caught by the test suite.TIMED OUT: The tests timed out while testing the mutation: the mutation actually made the tests fail, but not explicitly.NOT VIABLE: The mutation makes the build fail.
What inspired Gremlins
Mutation testing exists since the early days of computer science, so loads of papers and articles do exists. Gremlins is inspired from those.
Among the existing mutation testing tools, Gremlins is inspired especially by PITest.
Other Mutation Testing tools for Go
There is not much around, except from:
Contributing
See contributing.
License
Gremlins is licensed under the Apache 2.0 License. The Gremlins documentation is licensed under CC BY-SA 4.0.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package cmd implements the Gremlins command-line interface.
|
Package cmd implements the Gremlins command-line interface. |
|
gremlins
command
Gremlins is a mutation testing tool for Go.
|
Gremlins is a mutation testing tool for Go. |
|
internal/flags
Package flags provides common command-line flag definitions for Gremlins commands.
|
Package flags provides common command-line flag definitions for Gremlins commands. |
|
internal
|
|
|
configuration
Package configuration manages application settings from flags, config files, and environment variables.
|
Package configuration manages application settings from flags, config files, and environment variables. |
|
coverage
Package coverage analyzes Go test coverage to identify which code is tested.
|
Package coverage analyzes Go test coverage to identify which code is tested. |
|
diff
Package diff parses git diff output to identify changed lines for incremental mutation testing.
|
Package diff parses git diff output to identify changed lines for incremental mutation testing. |
|
engine
Package engine orchestrates mutation testing by discovering, applying, and testing mutations.
|
Package engine orchestrates mutation testing by discovering, applying, and testing mutations. |
|
engine/workdir
Package workdir manages temporary working directories for isolated mutation testing.
|
Package workdir manages temporary working directories for isolated mutation testing. |
|
engine/workerpool
Package workerpool provides a worker pool for concurrent mutation testing execution.
|
Package workerpool provides a worker pool for concurrent mutation testing execution. |
|
exclusion
Package exclusion provides file exclusion rules based on regex patterns.
|
Package exclusion provides file exclusion rules based on regex patterns. |
|
execution
Package execution provides error types and status codes for command execution.
|
Package execution provides error types and status codes for command execution. |
|
gomodule
Package gomodule detects and provides information about Go modules.
|
Package gomodule detects and provides information about Go modules. |
|
log
Package log provides logging utilities with verbosity control.
|
Package log provides logging utilities with verbosity control. |
|
mutator
Package mutator provides mutation types, statuses, and interfaces for mutation testing.
|
Package mutator provides mutation types, statuses, and interfaces for mutation testing. |
|
report
Package report formats and outputs mutation testing results.
|
Package report formats and outputs mutation testing results. |
|
report/internal
Package internal provides data structures for the Gremlins JSON output format.
|
Package internal provides data structures for the Gremlins JSON output format. |