Documentation
¶
Overview ¶
Package evaltest integrates eval reports with Go's testing package. It presents a report through a *testing.T as subtests and provides report-level assertions (RequirePass, RequireVerified) for use in ordinary Go tests. All orchestration stays in github.com/looprig/eval: evaltest only runs the suite through eval.Run and presents or asserts on the resulting report — it never re-implements the runner.
The TB subset ¶
testing.TB carries an unexported marker method, so no type outside the testing package can implement it. evaltest therefore accepts a minimal exported subset, TB, containing only the methods it uses. Both *testing.T and testing.TB satisfy TB, so callers pass a *testing.T exactly as the design intends; a test can also supply a fake recorder that implements TB to capture the presented output without failing a real test.
Subtest presentation vs. flat fallback ¶
Subtests require Run(string, func(*testing.T)) bool, which is a *testing.T method and is not part of TB. Run type-asserts the passed value to that method set: when present (a real *testing.T) it emits a subtest per scenario and per evaluator; when absent (a fake recorder, or a *testing.B, whose Run signature differs) it falls back to flat, line-per-assessment rendering. Presentation is informational only — it renders the verdicts but does not itself fail the test on a failing verdict. Use RequirePass or RequireVerified to gate on the report.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func RequirePass ¶
RequirePass asserts that every evaluation in report reached a passing verdict and that the report actually verified something. It fails (via Errorf) when:
- the report has no samples (nothing was verified — fail secure: an empty report is not a pass);
- any sample failed at the target stage (its evaluators never ran);
- any sample succeeded at the target stage but carries zero assessments (no evaluator ran, so nothing about that sample was verified);
- any assessment has a status other than pass or skipped; or
- the report contains no passing assessment at all (an all-skipped or otherwise pass-free report verified nothing and is not a pass).
StatusSkipped is permitted on an individual assessment: a deliberately skipped evaluator is an intentional non-run, not a failure. But at least one passing assessment must be present for the report as a whole to pass. Every other non-pass status — fail, unverified, error — fails the assertion.
func RequireVerified ¶
RequireVerified asserts that every evaluation in report reached a definite disposition — that nothing was left unverified or errored, and that every sample was actually covered by at least one evaluator. It fails (via Errorf) when:
- the report has no samples;
- any sample failed at the target stage;
- any sample succeeded at the target stage but carries zero assessments (no evaluator ran, so the sample was not verified); or
- any assessment has status unverified or error.
It ACCEPTS pass, fail, and skipped: a fail is a definite quality verdict and a skipped evaluator was intentionally not run (and, being an assessment, still counts as coverage), so neither is "unverified". Unlike RequirePass it does not require a passing assessment — it asserts coverage (everything ran and decided), not that the subject passed.
func Run ¶
Run executes suite against target with evaluators via eval.Run under a default RunConfig (one trial, sequential), presents the result through tb, and returns the complete report. It never fails tb on a failing or errored verdict — presentation is informational; gate the report with RequirePass or RequireVerified. It calls tb.Helper so any failure it does signal is attributed to the caller.
When eval.Run returns a non-nil error — a preflight rejection (an ill-formed suite or config, a nil target or evaluator) or a context cancellation — Run surfaces it through tb.Errorf with a safe message and still returns whatever report exists (the zero report for a preflight failure, or the partial report for a cancellation). The eval package guarantees its error strings never echo untrusted content, so the surfaced message is safe.
func RunScenario ¶
func RunScenario(tb TB, scenario eval.Scenario, target eval.Target, evaluators ...eval.Evaluator) eval.Report
RunScenario is the single-scenario convenience: it wraps scenario in a one-scenario suite (reusing the scenario's own Name and Revision for suite identity) and delegates to Run. An ill-formed scenario is rejected by eval.Run's preflight and surfaced through tb exactly as in Run.
Types ¶
type TB ¶
type TB interface {
// Helper marks the calling function as a test helper so failures are
// attributed to the caller's line.
Helper()
// Logf records informational, non-failing output.
Logf(format string, args ...any)
// Errorf records a failure and continues (it never calls runtime.Goexit), so
// the caller can still return the complete report.
Errorf(format string, args ...any)
}
TB is the minimal subset of testing.TB that evaltest uses. Both *testing.T and testing.TB satisfy it, so a caller passes a *testing.T unchanged; a test may supply a fake implementation to capture presented output. The variadic any is the standard printf logging boundary and is narrowed immediately by the format verbs.
Source Files