Documentation
¶
Overview ¶
Package testy is a Go test running framework.
Index ¶
- Variables
- func AddEchoRoutes(router *echo.Group)
- func AfterPackage(f Tester) any
- func AfterTest(f Tester) any
- func BeforePackage(f Tester) any
- func BeforeTest(f Tester) any
- func Cleanup(t TestingT, f func())
- func ConcurrentTest(name string, tester Tester) any
- func Context(t TestingT) context.Context
- func EchoRenderer() (echo.Renderer, error)
- func RunAsTest(t *testing.T)
- func SaveResult(ctx context.Context, tr TestResult) (string, error)
- func SetDB(db DB)
- func Skipf(t TestingT, format string, args ...interface{})
- func Skipped(t TestingT) bool
- func Test(name string, tester Tester) any
- func TestEach[V any](t TestingT, values []V, tester func(TestingT, V))
- type CaseExecutor
- type CaseSpec
- type DB
- type InMemoryDB
- type Level
- type Msg
- type PackageSpec
- type Result
- type RunOptions
- type Summary
- type TestResult
- func BuildSuiteResult(start time.Time, packageResults []TestResult) TestResult
- func LoadResult(ctx context.Context, id string) (TestResult, error)
- func Run() TestResult
- func RunPackage(pkg string) (TestResult, error)
- func RunPackageWithOptions(ctx context.Context, pkg string, opts RunOptions) (TestResult, error)
- func RunWithContext(ctx context.Context, opts RunOptions) TestResult
- func RunWithOptions(opts RunOptions) TestResult
- func (tr TestResult) FailedSubtests() int
- func (tr TestResult) FindFailingTests() []TestResult
- func (tr TestResult) PassedSubtests() int
- func (tr TestResult) SkippedSubtests() int
- func (tr TestResult) SumTestStats() (total, passed, failed int)
- func (tr TestResult) SumTestStatsWithSkipped() (total, passed, failed, skipped int)
- func (tr TestResult) TotalSubtests() int
- func (tr TestResult) TruncatedTimestamp() time.Time
- type Tester
- type TestingT
Constants ¶
This section is empty.
Variables ¶
var ErrNoDB = errors.New("no DB set")
ErrNoDB indicates no DB has been set via SetDB.
var ErrNotFound = errors.New("not found")
ErrNotFound indicates the provided result ID was not found in the datastore.
var ErrPackageNotFound = errors.New("package not found")
ErrPackageNotFound indicates a requested registered package does not exist.
Functions ¶
func AddEchoRoutes ¶
AddEchoRoutes adds routes to an Echo router that can run tests and retrieve tests results.
func AfterPackage ¶
AfterPackage registers a function to be run once after all tests in the given package have finished. A package may only have one AfterPackage function.
If AfterPackage panics, the reporting behavior depends on if RunAsTest or Run was called. If AfterPackage panics and BeforePackage had already panicked, then the panic for BeforePackage takes priority.
If RunAsTest was called, the panic will be reported as the top-level bootstrap test for the package. If Run was called, every test in the package will be marked as failed, and the panic's message will be appended to every test's own messages or the BeforePackage panic that replaced every test.
When run via Run, any logging output to the provided Tester will only be visible if AfterPackage panics or the Tester is marked as failed. When run via RunAsTest, the standard `go test` output rules apply. Notably, if a test fails, the output is not visible via Run but is via RunAsTest.
NOTE: Do not call TestingT.Fail, TestingT.FailNow, TestingT.Fatal, or TestingT.Fatalf in AfterPackage to report errors; always panic. It will work as expected in RunAsTest mode, but not Run mode. TODO parity here.
The return value may be discarded (and is always nil); it is provided to simplify writing test code, like so:
var _ = testy.AfterPackage(func(){})
func AfterTest ¶
AfterTest registers a function to be run once after every top level registered test in the given package has finished. It is not run after subtests created by `t.Run`. A package may only have one AfterTest function.
If AfterTest panics, the specific test that was just run will be marked as failed, and the panic's message will be appended to the test's own messages. Any subtests of the test will not be modified.
When run via Run, any logging output to the provided Tester will only be visible if AfterTest panics or the Tester is marked as failed. When run via RunAsTest, the standard `go test` output rules apply. Notably, if a test fails, the output is not visible via Run but is via RunAsTest.
NOTE: Do not call TestingT.Fail, TestingT.FailNow, TestingT.Fatal, or TestingT.Fatalf in AfterTest to report errors; always panic. It will work as expected in RunAsTest mode, but not Run mode. TODO parity here.
The return value may be discarded (and is always nil); it is provided to simplify writing test code, like so:
var _ = testy.AfterTest(func(){})
func BeforePackage ¶
BeforePackage registers a function to be run once before any tests in the given package are run. A package may only have one BeforePackage function.
If BeforePackage panics, no tests in the package will be run and the reporting behavior depends on if RunAsTest or Run was called. AfterPackage will still be run.
If RunAsTest was called, the panic will be reported as the top-level bootstrap test for the package. If Run was called, every registered test in the package will be marked as failed with the panic's message.
When run via Run, any logging output to the provided Tester will only be visible if BeforePackage panics or the Tester is marked as failed. When run via RunAsTest, the standard `go test` output rules apply. Notably, if a test fails, the output is not visible via Run but is via RunAsTest.
NOTE: Do not call TestingT.Fail, TestingT.FailNow, TestingT.Fatal, or TestingT.Fatalf in BeforePackage to report errors; always panic. It will work as expected in RunAsTest mode, but not Run mode. TODO parity here.
The return value may be discarded (and is always nil); it is provided to simplify writing test code, like so:
var _ = testy.BeforePackage(func(){})
func BeforeTest ¶
BeforeTest registers a function to be run before every top level registered test in the given package is run. It is not run before subtests created by `t.Run`. A package may only have one BeforeTest function.
If BeforeTest panics, the specific registered test that was about to be invoked will be marked as failed with the panic's message. AfterTest will still be run.
When run via Run, any logging output to the provided Tester will only be visible if BeforeTest panics or the Tester is marked as failed. When run via RunAsTest, the standard `go test` output rules apply. Notably, if a test fails, the output is not visible via Run but is via RunAsTest.
NOTE: Do not call TestingT.Fail, TestingT.FailNow, TestingT.Fatal, or TestingT.Fatalf in BeforeTest to report errors; always panic. It will work as expected in RunAsTest mode, but not Run mode. TODO parity here.
The return value may be discarded (and is always nil); it is provided to simplify writing test code, like so:
var _ = testy.BeforeTest(func(){})
func Cleanup ¶ added in v0.4.0
func Cleanup(t TestingT, f func())
Cleanup registers f to run after this test and its subtests, in last-in, first-out order. It runs on return, Fatal/FailNow, and panic. Register cleanup immediately after acquiring a resource. Cleanup is supported in Test and Run callbacks, not the legacy Before/After hooks.
This optional capability leaves TestingT source-compatible with custom implementations. Custom runners must implement Cleanup(func()) to use it; unsupported implementations panic rather than silently leak resources.
func ConcurrentTest ¶ added in v0.4.0
ConcurrentTest opts an independent top-level case into bounded concurrency. Its inputs/resources must not race other concurrent cases. Ordered Run children remain sequential in hosted mode; this does not activate nested Parallel calls. Native RunAsTest uses Go's -parallel limit. Hosted runners must explicitly supply a shared CaseExecutor; without one this registration remains serial.
func Context ¶ added in v0.4.0
Context returns this test's context, canceled immediately before its cleanup callbacks run. Subtest contexts inherit their parent's cancellation. Cleanup that performs I/O must use its own bounded context. Like Cleanup, this is an optional capability for custom TestingT implementations and is unavailable in legacy Before/After hooks.
func EchoRenderer ¶ added in v0.2.0
EchoRenderer loads the HTML templates and returns an echo.Renderer for the routes provided by this package. Assign this to the Renderer field of your Echo app (or wrap it with your own)
func RunAsTest ¶
RunAsTest runs all registered tests under Go's testing framework.
To run tests on a per-package basis, put a test file in each package containing a single test that calls this function. This is recommended so accurate per-package execution times are reported, as well as using the test cache. Do not import a test package into another test package as that will cause the tests in the second package to get executed with the first package. If code or resources need shared between test packages, put them in their own package which does not contain any test definitions.
Individual tests in a package may still be run using the standard -run test flag. See `go help testflag` for more information.
TODO: shuffle test execution order (see -shuffle in `go help testflag`)
func SaveResult ¶ added in v0.2.0
func SaveResult(ctx context.Context, tr TestResult) (string, error)
SaveResult saves the provided result to the registered datastore. If no datastore has been registered, an error wrapping ErrNoDB is returned.
func SetDB ¶ added in v0.2.0
func SetDB(db DB)
SetDB sets the datastore to use for test reports. This must be called during application startup.
func Skipf ¶ added in v0.4.0
Skipf records why this test cannot execute and stops it with runtime.Goexit, running its defers and registered cleanup. A prior or later failure always overrides the skipped outcome. Call only from the test's goroutine.
Skipping is an optional capability, preserving existing TestingT implementations. Custom runners must implement Skipf(string, ...interface{}); unsupported runners panic rather than silently report unexecuted assertions as passed. Legacy Before/After hooks are not test scopes and cannot skip.
func Skipped ¶ added in v0.4.0
Skipped reports whether the test requested a skip, including during cleanup. This does not mean it passed or cannot also have failed. Custom runners must implement Skipped() bool to use this optional capability.
func Test ¶
Test registers a new test to be run. Tests are run in lexicographical order within a package.
Test cases should *not* be defined in `_test.go` files if they are to be run via Run. If they are, they will not be compiled into the binary. This also means that you need to ensure that your test packages are eventually imported by your main package. You may need to do this with a side effects import (`import _ "my/package"`).
The return value may be discarded (and is always nil); it is provided to simplify writing test code, like so:
var _ = testy.Test("my test", func(t testy.TestingT){})
Types ¶
type CaseExecutor ¶ added in v0.4.0
type CaseExecutor struct {
// contains filtered or unexported fields
}
CaseExecutor bounds admitted case lifecycles across all packages/runs sharing this instance. Share one per hosted worker, not one per package. It is not a cross-process limit. Non-ConcurrentTest cases and package hooks are exclusive. A permit covers setup, body, ordered children, cleanup and AfterTest.
func NewCaseExecutor ¶ added in v0.4.0
func NewCaseExecutor(limit int) *CaseExecutor
type CaseSpec ¶ added in v0.4.0
CaseSpec describes canonical registered case identity, not a nested step.
type DB ¶ added in v0.2.0
type DB interface {
// Enumerate lists the test results for the given page. The datastore determines the page size.
Enumerate(ctx context.Context, page int) (results []Summary, more bool, err error)
// Load retrieves the specified test result from the datastore.
// If the ID is invalid, ErrNotFound should be returned.
Load(ctx context.Context, id string) (TestResult, error)
// Save stores the provided TestResult in the data store and returns its unique ID.
Save(context.Context, TestResult) (string, error)
}
DB is the interface for something which can save and retrieve test reports.
type InMemoryDB ¶ added in v0.2.1
type InMemoryDB struct {
// contains filtered or unexported fields
}
InMemoryDB is an implementation of DB that is stored in memory, with no persistent storage. It should be used for demonstration purposes only.
func (*InMemoryDB) Load ¶ added in v0.2.1
func (db *InMemoryDB) Load(_ context.Context, id string) (TestResult, error)
func (*InMemoryDB) Save ¶ added in v0.2.1
func (db *InMemoryDB) Save(_ context.Context, result TestResult) (string, error)
type PackageSpec ¶ added in v0.3.1
type PackageSpec struct {
// Name is the package import path.
Name string
// Index is the package's deterministic registration-order index.
Index int
}
PackageSpec describes a package registered with testy.
func ListPackages ¶ added in v0.3.1
func ListPackages() []PackageSpec
ListPackages returns the registered packages in deterministic execution order.
The returned package list is intentionally independent from any execution backend. Callers that want to orchestrate package execution outside this process, such as a durable workflow engine, can use the package names with RunPackage and then combine the package results with BuildSuiteResult.
type Result ¶ added in v0.1.0
type Result string
Result indicates the result of a test.
const ( // ResultPassed indicates that this test (and all of its subtests) passed. ResultPassed Result = "passed" // ResultFailed indicates that this test or at least one of its subtests failed. ResultFailed Result = "failed" // ResultSkipped indicates this test explicitly skipped execution. It is not a pass. ResultSkipped Result = "skipped" // ResultIncomplete indicates a nonfailed container with skipped coverage. ResultIncomplete Result = "incomplete" )
type RunOptions ¶ added in v0.3.1
type RunOptions struct {
// PackageConcurrency is the maximum number of packages executed at once.
// Values <= 1 preserve the historical serial package execution behavior.
PackageConcurrency int
// CaseExecutor is shared across package activities in one worker. Nil keeps
// case execution serial and preserves existing package concurrency behavior.
CaseExecutor *CaseExecutor
// CaseOrder is an optional immutable complete permutation of case names per
// package. It changes admission order only, never result identity or steps.
// Missing/invalid hints retain canonical order and invalid hints are reported.
CaseOrder map[string][]string
}
RunOptions controls how the non-go-test runner executes registered tests.
type Summary ¶ added in v0.2.0
type Summary struct {
// ID is an opaque unique identifier for a test result. The specific format is defined by the datastore.
ID string
// Started indicates when a test run was started.
Started time.Time
// Dur is how long the test run took to complete.
Dur time.Duration
// Total is the total number of tests that were run.
Total int
// Passed is the number of tests that passed.
Passed int
// Failed is the number of tests that failed.
Failed int
// Skipped is unexecuted coverage, not included in Passed.
Skipped int
}
Summary is an overview of a TestResult, used to populate the list of past results.
func (Summary) TruncatedTimestamp ¶ added in v0.2.1
TruncatedTimestamp returns the started timestamp truncated to second precision.
type TestResult ¶
type TestResult struct {
// Package is the Go package that contains the test.
Package string
// Name is the name of the test as provided to Test (for top-level tests), Run (for subtests),
// or the string representation of each value (for TestEach).
Name string
// Msgs contains each message that was emitted during the test via the methods on TestingT that emit messages.
Msgs []Msg
// Result is the result of the test.
Result Result
// SkipReason records an explicit skip request, even if a later failure wins.
SkipReason string `json:",omitempty"`
// Started is when the test was started.
Started time.Time
// Dur is how long the test took.
Dur time.Duration
// DurHuman is how long the test took in human-readable form.
DurHuman string
// QueueDur measures case admission delay after package setup, separate from
// Dur, which includes setup/body/children/cleanup and AfterTest.
QueueDur time.Duration
// Subtests contains the test result of every test this test started via Run or TestEach.
Subtests []TestResult
}
TestResult is the result of a specific test.
func BuildSuiteResult ¶ added in v0.3.1
func BuildSuiteResult(start time.Time, packageResults []TestResult) TestResult
BuildSuiteResult combines package-level results into a suite-level result.
func LoadResult ¶ added in v0.2.0
func LoadResult(ctx context.Context, id string) (TestResult, error)
LoadResult loads the specified result from the registered datastore. If no datastore has been registered, an error wrapping ErrNoDB is returned. If the ID is invalid, an error wrapping ErrNotFound is returned.
func Run ¶
func Run() TestResult
Run runs all registered tests and returns result information about them.
TODO: ability to filter for specific packages and tests
TODO: shuffle test execution order (see -shuffle in `go help testflag`)
TODO: channel for results to support progressive result loading?
func RunPackage ¶ added in v0.3.1
func RunPackage(pkg string) (TestResult, error)
RunPackage runs a single registered package by import path.
func RunPackageWithOptions ¶ added in v0.4.0
func RunPackageWithOptions(ctx context.Context, pkg string, opts RunOptions) (TestResult, error)
RunPackageWithOptions shares the worker executor and propagates cancellation. Even on cancellation it waits for admitted callbacks, cleanup and hooks before returning a result and ctx.Err; orchestration must not start a replacement early.
func RunWithContext ¶ added in v0.4.0
func RunWithContext(ctx context.Context, opts RunOptions) TestResult
RunWithContext cancels case contexts and drains their lifecycle before returning. Callbacks must cooperate with cancellation; arbitrary Go code is not preempted.
func RunWithOptions ¶ added in v0.3.1
func RunWithOptions(opts RunOptions) TestResult
RunWithOptions runs all registered tests and returns result information about them.
func (TestResult) FailedSubtests ¶ added in v0.2.0
func (tr TestResult) FailedSubtests() int
FailedSubtests returns the number of leaf subtests that failed. Prefer to use SumTestStats, as that returns more information for the same recursion cost; this is intended for Go templates, which are more limited in what you can do.
func (TestResult) FindFailingTests ¶ added in v0.1.0
func (tr TestResult) FindFailingTests() []TestResult
FindFailingTests finds the least deeply nested subtests that have sibling tests that passed. These subtests may be in different branches of subtests. This implies that this test failed; if it did not, then a nil slice is returned. If every subtest of test failed or if test has no subtests, then test itself is returned.
func (TestResult) PassedSubtests ¶ added in v0.2.0
func (tr TestResult) PassedSubtests() int
PassedSubtests returns the number of leaf subtests that passed. Prefer to use SumTestStats, as that returns more information for the same recursion cost; this is intended for Go templates, which are more limited in what you can do.
func (TestResult) SkippedSubtests ¶ added in v0.4.0
func (tr TestResult) SkippedSubtests() int
SkippedSubtests returns skipped coverage, including a parent-only skip.
func (TestResult) SumTestStats ¶ added in v0.1.0
func (tr TestResult) SumTestStats() (total, passed, failed int)
SumTestStats returns total, passed and failed counts. Skipped tests are in total but NOT passed; use SumTestStatsWithSkipped for complete coverage counts.
func (TestResult) SumTestStatsWithSkipped ¶ added in v0.4.0
func (tr TestResult) SumTestStatsWithSkipped() (total, passed, failed, skipped int)
SumTestStatsWithSkipped counts leaf outcomes and parent-only failures/skips. Unknown leaf outcomes are conservatively counted as failed, never passed.
func (TestResult) TotalSubtests ¶ added in v0.2.0
func (tr TestResult) TotalSubtests() int
TotalSubtests returns the total number of leaf subtests. Prefer to use SumTestStats, as that returns more information for the same recursion cost; this is intended for Go templates, which are more limited in what you can do.
func (TestResult) TruncatedTimestamp ¶ added in v0.2.0
func (tr TestResult) TruncatedTimestamp() time.Time
TruncatedTimestamp returns the started timestamp truncated to second precision.
type TestingT ¶
type TestingT interface {
// Fail marks the function as having failed but continues execution.
Fail()
// FailNow marks the function as having failed and stops its execution
// by calling runtime.Goexit (which then runs all deferred calls in the
// current goroutine).
// Execution will continue at the next test or benchmark.
// FailNow must be called from the goroutine running the
// test or benchmark function, not from other goroutines
// created during the test. Calling FailNow does not stop
// those other goroutines.
FailNow()
// Fatal is equivalent to Log followed by FailNow.
Fatal(args ...interface{})
// Fatalf is equivalent to Logf followed by FailNow.
Fatalf(format string, args ...interface{})
// Errorf is equivalent to Logf followed by Fail.
Errorf(format string, args ...interface{})
// Helper does not do anything useful since the call stack when passed to the actual implementation has an extra
// level in it.
Helper()
// Log formats its arguments using default formatting, analogous to Println,
// and records the text in the error log. For tests, the text will be printed only if
// the test fails or the -test.v flag is set.
Log(args ...interface{})
// Logf formats its arguments according to the format, analogous to Printf, and
// records the text in the error log. A final newline is added if not provided. For
// tests, the text will be printed only if the test fails or the -test.v flag is
// set.
Logf(format string, args ...interface{})
// Run runs f as a subtest of t called name. It runs f in a separate goroutine
// and blocks until f returns (or, if running via go test, calls t.Parallel to become a parallel test).
// Run reports whether f succeeded (or, if running via go test, at least did not fail before calling t.Parallel).
//
// Run may be called simultaneously from multiple goroutines, but all such calls
// must return before the outer test function for t returns.
Run(string, Tester) bool
// Parallel signals that this test is to be run in parallel with (and only with)
// other parallel tests. When a test is run multiple times due to use of
// -test.count or -test.cpu, multiple instances of a single test never run in
// parallel with each other.
//
// Parallel only affects RunAsTest as it relies on testing.T's implementation.
Parallel()
}
TestingT is a subset of testing.T that we have to implement for non-`go test` runs.
TODO flesh this out with more useful stuff from testing.T -- Parallel would be nice but tricky