README
¶
Testy
Docs incoming :)
We couldn't find a framework that addressed our API acceptance testing desires, so we're making one ourselves. This project is still a rapid work in progress, so full details will be added once things start to stabilize a bit and we're actually using this for tests.
This requires go 1.18 due to using type parameters.
Documentation
¶
Overview ¶
Package testy is a Go test running framework.
We couldn't find a framework that addressed our API acceptance testing desires, so we're making one ourselves. This project is still a rapid work in progress, so full details will be added once things start to stabilize a bit and we're actually using this for tests.
Index ¶
- Variables
- func AddEchoRoutes(router *echo.Group)
- func AfterPackage(f Tester) any
- func AfterTest(f Tester) any
- func BeforePackage(f Tester) any
- func BeforeTest(f Tester) any
- func EchoRenderer() (echo.Renderer, error)
- func RunAsTest(t *testing.T)
- func SaveResult(ctx context.Context, tr TestResult) (string, error)
- func SetDB(db DB)
- func Test(name string, tester Tester) any
- func TestEach[V any](t TestingT, values []V, tester func(TestingT, V))
- type DB
- type Level
- type Msg
- type Result
- type Summary
- type TestResult
- type Tester
- type TestingT
Constants ¶
This section is empty.
Variables ¶
var ErrNoDB = errors.New("no DB set")
ErrNoDB indicates no DB has been set via SetDB.
var ErrNotFound = errors.New("not found")
ErrNotFound indicates the provided result ID was not found in the datastore.
Functions ¶
func AddEchoRoutes ¶
AddEchoRoutes adds routes to an Echo router that can run tests and retrieve tests results.
func AfterPackage ¶
AfterPackage registers a function to be run once after all tests in the given package have finished. A package may only have one AfterPackage function.
If AfterPackage panics, the reporting behavior depends on if RunAsTest or Run was called. If AfterPackage panics and BeforePackage had already panicked, then the panic for BeforePackage takes priority.
If RunAsTest was called, the panic will be reported as the top-level bootstrap test for the package. If Run was called, every test in the package will be marked as failed, and the panic's message will be appended to every test's own messages or the BeforePackage panic that replaced every test.
When run via Run, any logging output to the provided Tester will only be visible if AfterPackage panics or the Tester is marked as failed. When run via RunAsTest, the standard `go test` output rules apply. Notably, if a test fails, the output is not visible via Run but is via RunAsTest.
NOTE: Do not call TestingT.Fail, TestingT.FailNow, TestingT.Fatal, or TestingT.Fatalf in AfterPackage to report errors; always panic. It will work as expected in RunAsTest mode, but not Run mode. TODO parity here.
The return value may be discarded (and is always nil); it is provided to simplify writing test code, like so:
var _ = testy.AfterPackage(func(){})
func AfterTest ¶
AfterTest registers a function to be run once after every top level registered test in the given package has finished. It is not run after subtests created by `t.Run`. A package may only have one AfterTest function.
If AfterTest panics, the specific test that was just run will be marked as failed, and the panic's message will be appended to the test's own messages. Any subtests of the test will not be modified.
When run via Run, any logging output to the provided Tester will only be visible if AfterTest panics or the Tester is marked as failed. When run via RunAsTest, the standard `go test` output rules apply. Notably, if a test fails, the output is not visible via Run but is via RunAsTest.
NOTE: Do not call TestingT.Fail, TestingT.FailNow, TestingT.Fatal, or TestingT.Fatalf in AfterTest to report errors; always panic. It will work as expected in RunAsTest mode, but not Run mode. TODO parity here.
The return value may be discarded (and is always nil); it is provided to simplify writing test code, like so:
var _ = testy.AfterTest(func(){})
func BeforePackage ¶
BeforePackage registers a function to be run once before any tests in the given package are run. A package may only have one BeforePackage function.
If BeforePackage panics, no tests in the package will be run and the reporting behavior depends on if RunAsTest or Run was called. AfterPackage will still be run.
If RunAsTest was called, the panic will be reported as the top-level bootstrap test for the package. If Run was called, every registered test in the package will be marked as failed with the panic's message.
When run via Run, any logging output to the provided Tester will only be visible if BeforePackage panics or the Tester is marked as failed. When run via RunAsTest, the standard `go test` output rules apply. Notably, if a test fails, the output is not visible via Run but is via RunAsTest.
NOTE: Do not call TestingT.Fail, TestingT.FailNow, TestingT.Fatal, or TestingT.Fatalf in BeforePackage to report errors; always panic. It will work as expected in RunAsTest mode, but not Run mode. TODO parity here.
The return value may be discarded (and is always nil); it is provided to simplify writing test code, like so:
var _ = testy.BeforePackage(func(){})
func BeforeTest ¶
BeforeTest registers a function to be run before every top level registered test in the given package is run. It is not run before subtests created by `t.Run`. A package may only have one BeforeTest function.
If BeforeTest panics, the specific registered test that was about to be invoked will be marked as failed with the panic's message. AfterTest will still be run.
When run via Run, any logging output to the provided Tester will only be visible if BeforeTest panics or the Tester is marked as failed. When run via RunAsTest, the standard `go test` output rules apply. Notably, if a test fails, the output is not visible via Run but is via RunAsTest.
NOTE: Do not call TestingT.Fail, TestingT.FailNow, TestingT.Fatal, or TestingT.Fatalf in BeforeTest to report errors; always panic. It will work as expected in RunAsTest mode, but not Run mode. TODO parity here.
The return value may be discarded (and is always nil); it is provided to simplify writing test code, like so:
var _ = testy.BeforeTest(func(){})
func EchoRenderer ¶ added in v0.2.0
EchoRenderer loads the HTML templates and returns an echo.Renderer for the routes provided by this package. Assign this to the Renderer field of your Echo app (or wrap it with your own)
func RunAsTest ¶
RunAsTest runs all registered tests under Go's testing framework. To run tests on a per-package basis, put a test file in each package containing a single test that calls this function. This is recommended so accurate per-package execution times are reported, as well as using the test cache. Do not import a test package into another test package as that will cause the tests in the second package to get executed with the first package. If code or resources need shared between test packages, put them in their own package which does not contain any test definitions.
Individual tests in a package may still be run using the standard -run test flag. See `go help testflag` for more information.
TODO: shuffle test execution order (see -shuffle in `go help testflag`)
func SaveResult ¶ added in v0.2.0
func SaveResult(ctx context.Context, tr TestResult) (string, error)
SaveResult saves the provided result to the registered datastore. If no datastore has been registered, an error wrapping ErrNoDB is returned.
func SetDB ¶ added in v0.2.0
func SetDB(db DB)
SetDB sets the datastore to use for test reports. This must be called during application startup.
func Test ¶
Test registers a new test to be run. Tests are run in lexicographical order within a package. TODO: support the -shuffle testflag. though maybe that only has to be in the runner?
The return value may be discarded (and is always nil); it is provided to simplify writing test code, like so:
var _ = testy.Test("my test", func(t testy.TestingT){})
Types ¶
type DB ¶ added in v0.2.0
type DB interface {
// Enumerate lists the test results for the given page. The datastore determines the page size.
Enumerate(ctx context.Context, page int) (results []Summary, more bool, err error)
// Load retrieves the specified test result from the datastore.
// If the ID is invalid, ErrNotFound should be returned.
Load(ctx context.Context, id string) (TestResult, error)
// Save stores the provided TestResult in the data store and returns its unique ID.
Save(context.Context, TestResult) (string, error)
}
DB is the interface for something which can save and retrieve test reports.
type Summary ¶ added in v0.2.0
type Summary struct {
// ID is an opaque unique identifier for a test result. The specific format is defined by the datastore.
ID string
Started time.Time
Dur time.Duration
Total int
Passed int
Failed int
}
Summary is an overview of a TestResult, used to populate the list of past results.
type TestResult ¶
type TestResult struct {
Package string
Name string
Msgs []Msg
Result Result
Started time.Time
Dur time.Duration
DurHuman string
Subtests []TestResult
}
func LoadResult ¶ added in v0.2.0
func LoadResult(ctx context.Context, id string) (TestResult, error)
LoadResult loads the specified result from the registered datastore. If no datastore has been registered, an error wrapping ErrNoDB is returned. If the ID is invalid, an error wrapping ErrNotFound is returned.
func Run ¶
func Run() TestResult
Run runs all registered tests and returns result information about them.
TODO: ability to filter for specific packages and tests TODO: shuffle test execution order (see -shuffle in `go help testflag`) TODO: channel for results to support progressive result loading?
func (TestResult) FailedSubtests ¶ added in v0.2.0
func (tr TestResult) FailedSubtests() int
FailedSubtests returns the number of leaf subtests that failed. Prefer to use SumTestStats, as that returns more information for the same recursion cost; this is intended for Go templates, which are more limited in what you can do.
func (TestResult) FindFailingTests ¶ added in v0.1.0
func (tr TestResult) FindFailingTests() []TestResult
FindFailingTests finds the least deeply nested subtests that have sibling tests that passed. These subtests may be in different branches of subtests. This implies that this test failed; if it did not, then a nil slice is returned. If every subtest of test failed or if test has no subtests, then test itself is returned.
func (TestResult) PassedSubtests ¶ added in v0.2.0
func (tr TestResult) PassedSubtests() int
PassedSubtests returns the number of leaf subtests that passed. Prefer to use SumTestStats, as that returns more information for the same recursion cost; this is intended for Go templates, which are more limited in what you can do.
func (TestResult) SumTestStats ¶ added in v0.1.0
func (tr TestResult) SumTestStats() (total, passed, failed int)
SumTestStats returns the total number of leaf subtests, as well as the number of those that passed and failed.
func (TestResult) TotalSubtests ¶ added in v0.2.0
func (tr TestResult) TotalSubtests() int
TotalSubtests returns the total number of leaf subtests. Prefer to use SumTestStats, as that returns more information for the same recursion cost; this is intended for Go templates, which are more limited in what you can do.
func (TestResult) TruncatedTimestamp ¶ added in v0.2.0
func (tr TestResult) TruncatedTimestamp() time.Time
TruncatedTimestamp returns the started timestamp truncated to second precision.
type TestingT ¶
type TestingT interface {
// Fail marks the function as having failed but continues execution.
Fail()
// FailNow marks the function as having failed and stops its execution
// by calling runtime.Goexit (which then runs all deferred calls in the
// current goroutine).
// Execution will continue at the next test or benchmark.
// FailNow must be called from the goroutine running the
// test or benchmark function, not from other goroutines
// created during the test. Calling FailNow does not stop
// those other goroutines.
FailNow()
// Fatal is equivalent to Log followed by FailNow.
Fatal(args ...interface{})
// Fatalf is equivalent to Logf followed by FailNow.
Fatalf(format string, args ...interface{})
// Errorf is equivalent to Logf followed by Fail.
Errorf(format string, args ...interface{})
// Helper does not do anything useful since the call stack when passed to the actual implementation has an extra
// level in it.
Helper()
// Log formats its arguments using default formatting, analogous to Println,
// and records the text in the error log. For tests, the text will be printed only if
// the test fails or the -test.v flag is set.
Log(args ...interface{})
// Logf formats its arguments according to the format, analogous to Printf, and
// records the text in the error log. A final newline is added if not provided. For
// tests, the text will be printed only if the test fails or the -test.v flag is
// set.
Logf(format string, args ...interface{})
// Run runs f as a subtest of t called name. It runs f in a separate goroutine
// and blocks until f returns (or, if running via go test, calls t.Parallel to become a parallel test).
// Run reports whether f succeeded (or, if running via go test, at least did not fail before calling t.Parallel).
//
// Run may be called simultaneously from multiple goroutines, but all such calls
// must return before the outer test function for t returns.
Run(string, Tester) bool
// Parallel signals that this test is to be run in parallel with (and only with)
// other parallel tests. When a test is run multiple times due to use of
// -test.count or -test.cpu, multiple instances of a single test never run in
// parallel with each other.
//
// Parallel only affects RunAsTest as it relies on testing.T's implementation.
Parallel()
}
TestingT is a subset of testing.T that we have to implement for non-`go test` runs.
TODO flesh this out with more useful stuff from testing.T -- Parallel would be nice but tricky
Source Files
Directories