Documentation
¶
Overview ¶
Package runner implements go test wrappers, iteration orchestration, and result analysis.
Index ¶
- func Analyze(iterations []io.Reader, slowThreshold time.Duration) (*Report, LogMap, error)
- func AnalyzeResults(resultsDir string, slowThreshold time.Duration) (*Report, LogMap, error)
- func DiagnoseGracefulStopChan(ctx context.Context) <-chan struct{}
- func DiagnoseGracefulStopRequested(ctx context.Context) bool
- func EffectiveParallelIterations(conf *config.App) int
- func FinishDiagnoseAnalysis(ctx context.Context, conf *config.App, out *output.Printer, ...) error
- func GoTest(ctx context.Context, conf *config.App, args []string, env []string) error
- func Gotestsum(ctx context.Context, conf *config.App, args []string, env []string) error
- func NewDiagnoseRunContext(parent context.Context) (context.Context, func())
- func NewDiagnoseRunContextForTest(parent context.Context) (context.Context, func())
- func PrintSummary(w io.Writer, rep *Report)
- func RequestDiagnoseGracefulStop(ctx context.Context)
- func RequestDiagnoseHardCancel(ctx context.Context)
- func ServeTrace(ctx context.Context, resultsDir string, files []string, out *output.Printer, ...) error
- func WarnDiagnoseGoTestCount(goTestArgs []string) error
- func WarnDiagnoseGoTestTrace(goTestArgs []string) error
- func WilsonScoreInterval(k, n int, z float64) (lower float64, upper float64)
- func WriteCSV(resultsDir string, rep *Report) error
- func WriteLogFiles(resultsDir string, rep *Report, logs LogMap) error
- func WriteReport(resultsDir string, rep *Report) error
- func WriteTrace(out *output.Printer, resultsDir string) ([]string, error)
- type DiagnoseRunState
- type IterationDigest
- type IterationSummary
- type LogMap
- type ProblemLog
- type Report
- type ReportSummary
- type RunMeta
- type RunStateSnapshot
- type TestEntry
- type TestEvent
- type TestGroup
- type TraceEvent
- type TraceServeOptions
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Analyze ¶
Analyze reads per-iteration test2json streams and classifies tests. Malformed lines are silently skipped (go test can interleave non-JSON).
func AnalyzeResults ¶
AnalyzeResults opens every `iteration-*.log.jsonl` file in resultsDir, in numeric-iteration order, and delegates to Analyze.
func DiagnoseGracefulStopChan ¶ added in v0.0.9
DiagnoseGracefulStopChan returns a channel closed on the first graceful stop request, or nil when ctx is not from NewDiagnoseRunContext.
func DiagnoseGracefulStopRequested ¶ added in v0.0.9
DiagnoseGracefulStopRequested reports whether the user requested a graceful stop (first Ctrl+C) on a context from NewDiagnoseRunContext.
func EffectiveParallelIterations ¶
EffectiveParallelIterations returns the bounded diagnose worker count.
func FinishDiagnoseAnalysis ¶ added in v0.0.8
func FinishDiagnoseAnalysis( ctx context.Context, conf *config.App, out *output.Printer, goTestArgs []string, state *DiagnoseRunState, start time.Time, resultsDir string, ) error
FinishDiagnoseAnalysis completes the analysis phase of a diagnose run. It is intended to be called after RunIterations and after resources have been cleaned up.
func GoTest ¶
GoTest runs `go test` with the given args (repo root as working directory). env is appended to the child process environment (e.g. resource Env).
func Gotestsum ¶
Gotestsum runs `gotestsum` with the given args (repo root as working directory). env is appended to the child process environment (e.g. resource Env).
func NewDiagnoseRunContext ¶ added in v0.0.9
NewDiagnoseRunContext returns a context for diagnose iteration runs with two-stage cancellation. The first SIGINT/SIGTERM requests a graceful stop (finish in-flight iterations, do not enqueue new ones). The second signal hard-cancels the context. Pass context.WithoutCancel(cmd.Context()) as parent so the root CLI signal handler does not cancel iteration execution on the first press.
func NewDiagnoseRunContextForTest ¶ added in v0.0.9
NewDiagnoseRunContextForTest is like NewDiagnoseRunContext but without OS signal handling. Use in unit and synctest tests with RequestDiagnoseGracefulStop and RequestDiagnoseHardCancel.
func PrintSummary ¶
PrintSummary writes a human-readable diagnose summary: scope line and rates table first, then detail sections (Broken, Flaky, Timeout, Slow) as flat lines. Broken and Timeout entries are sorted alphabetically by package then test. Flaky entries are sorted by fails/runs (desc), then fails (desc), then name. Slow entries are sorted by max runtime (desc), then name.
func RequestDiagnoseGracefulStop ¶ added in v0.0.9
RequestDiagnoseGracefulStop simulates the first interrupt for tests.
func RequestDiagnoseHardCancel ¶ added in v0.0.9
RequestDiagnoseHardCancel simulates the second interrupt for tests.
func ServeTrace ¶ added in v0.0.5
func ServeTrace( ctx context.Context, resultsDir string, files []string, out *output.Printer, opts TraceServeOptions, ) error
ServeTrace serves trace files at traceListenAddr, opens the Perfetto UI in browser tabs, and blocks until all traces are fetched or ctx is cancelled.
func WarnDiagnoseGoTestCount ¶
WarnDiagnoseGoTestCount returns an error if the user sets -count on go test.
func WarnDiagnoseGoTestTrace ¶ added in v0.0.5
WarnDiagnoseGoTestTrace returns an error if the user sets -trace on go test.
func WilsonScoreInterval ¶
WilsonScoreInterval calculates our confidence interval for how (non) flaky a test is. We use a 95% confidence interval, so z = 1.96. https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval
func WriteCSV ¶
WriteCSV writes a human-readable CSV of every flagged test (Flakes ∪ Failures ∪ Timeouts ∪ Slow) to <resultsDir>/report.csv. Rows sort worst-first: (timeouts+fails) desc, then package, then test.
func WriteLogFiles ¶
WriteLogFiles writes per-test per-iteration log files under <resultsDir>/logs/ for flagged tests and populates each flagged TestEntry's Logs slice with a compact problem-kind, iteration-range, and path pattern.
func WriteReport ¶
WriteReport writes the report as pretty JSON to <resultsDir>/report.json.
Types ¶
type DiagnoseRunState ¶ added in v0.0.8
type DiagnoseRunState struct {
GracefulStop bool
// contains filtered or unexported fields
}
DiagnoseRunState captures the runtime progress and limits of a diagnose session.
func RunIterations ¶ added in v0.0.8
func RunIterations( ctx context.Context, conf *config.App, out *output.Printer, goTestArgs []string, resources []hooks.Resource, iterSetup, iterTeardown func(context.Context) error, ) (*DiagnoseRunState, time.Time, string, error)
RunIterations runs go test -json once per iteration, writing each stream to iteration-<n>.log.jsonl under a new results directory and run-state.json on return. Test iteration failures do not stop later runs (unless --fail-fast); compile/build failures stop immediately. Returns a non-nil error for setup failures (e.g. mkdir, database reset) or ctx errors from dependencies — not for failing tests alone. iterSetup and iterTeardown run before/after each iteration. Either may be nil. Teardown runs even when the iteration's go test invocation fails; its error is reported only when the iteration itself succeeded. Call FinishDiagnoseAnalysis separately to analyze results and write report.json.
type IterationDigest ¶
type IterationDigest struct {
Result string // pass, fail, timeout
RanTests int // distinct named tests (package.test) that executed (pass/fail/timeout), excluding skip-only
FailTests int // len(IterationSummaries[0].FailingTests)
TimeoutTests int // len(Timeouts) for this iteration
SkipTests int // distinct named tests skipped in this iteration
SlowTests int // tests over slow threshold
BuildFailure bool // compile/build failed or heuristic package-level fail with no named tests run
}
IterationDigest summarizes one iteration JSONL log for per-iteration CLI output. Counts match a single-iteration Analyze (same rules as the final report).
func DigestIterationJSONL ¶
DigestIterationJSONL parses one `go test -json` stream and returns counts for progress UI. It uses the same scan + report pipeline as Analyze for one iteration (no redundant Analyze wrapper).
type IterationSummary ¶
type IterationSummary struct {
Index int `json:"index"`
Duration time.Duration `json:"duration,omitempty"`
Result string `json:"result"` // "pass", "fail", "timeout"
FailingTests []string `json:"failing_tests,omitempty"`
ShuffleSeed int64 `json:"shuffle_seed,omitempty"`
}
IterationSummary captures high-level stats for a single diagnose iteration. Duration and ShuffleSeed are populated by the runner after analysis.
type LogMap ¶
LogMap maps (package,test) → iteration → raw interleaved output. Returned alongside Report so callers can write per-test log files without coupling the parser to the filesystem.
type ProblemLog ¶
type ProblemLog struct {
Type string `json:"type"`
Iters string `json:"iters"`
Path string `json:"path"`
}
ProblemLog points to log files for iterations where this entry actually had the reported problem. Path uses "{iter}" as the iteration placeholder.
type Report ¶
type Report struct {
Run *RunMeta `json:"run,omitempty"`
Iterations int `json:"iterations"`
SlowThreshold time.Duration `json:"slow_threshold"`
Summary *ReportSummary `json:"summary,omitempty"`
IterationSummaries []IterationSummary `json:"iteration_summaries,omitempty"`
Flakes []TestEntry `json:"flakes,omitempty"`
Failures []TestEntry `json:"failures,omitempty"`
Timeouts []TestEntry `json:"timeouts,omitempty"`
Slow []TestEntry `json:"slow,omitempty"`
SlowestPackages []TestEntry `json:"slowest_packages,omitempty"`
}
Report classifies tests across iterations of a diagnose run.
func (*Report) TestGroups ¶
TestGroups returns all flagged test categories in precedence order (Timeout > Failure > Flake > Slow) for deduplication and output generation.
type ReportSummary ¶
type ReportSummary struct {
DistinctNamedTests int `json:"distinct_named_tests"`
FlakeNamedCount int `json:"flake_named_count"`
FlakePrevalence *float64 `json:"flake_prevalence,omitempty"`
FlakeFailRuns int `json:"flake_fail_runs,omitempty"`
FlakeTotalRuns int `json:"flake_total_runs,omitempty"`
FlakeExecutionFailRate *float64 `json:"flake_execution_fail_rate,omitempty"`
FlakeExecutionFailRateLower *float64 `json:"flake_execution_fail_rate_lower,omitempty"`
FlakeExecutionFailRateUpper *float64 `json:"flake_execution_fail_rate_upper,omitempty"`
// FlakeFailingIterations is how many diagnose iterations had at least one
// flake failure; FlakeIterationTotal is rep.Iterations (not summed per-test runs).
FlakeFailingIterations int `json:"flake_failing_iterations,omitempty"`
FlakeIterationTotal int `json:"flake_iteration_total,omitempty"`
FlakeIterationFailRate *float64 `json:"flake_iteration_fail_rate,omitempty"`
FlakeIterationFailRateLower *float64 `json:"flake_iteration_fail_rate_lower,omitempty"`
FlakeIterationFailRateUpper *float64 `json:"flake_iteration_fail_rate_upper,omitempty"`
SlowCount int `json:"slow_count,omitempty"`
SlowPrevalence *float64 `json:"slow_prevalence,omitempty"`
// IterationDurationMin/Max/P50 summarize wall-clock runtimes (IterationSummary.Duration) across all completed iterations.
IterationDurationMin time.Duration `json:"iteration_duration_min,omitempty"`
IterationDurationMax time.Duration `json:"iteration_duration_max,omitempty"`
IterationDurationP50 time.Duration `json:"iteration_duration_p50,omitempty"`
}
ReportSummary holds aggregate flake and slow rates for the full diagnose run. FlakePrevalence uses distinct named tests (package.test keys). Per-execution flake_fail_runs / flake_total_runs sum across flaky entries; flake_failing_iterations / flake_iteration_total count diagnose iterations (union of flake failures vs rep.Iterations).
type RunMeta ¶
type RunMeta struct {
ResultsDirBasename string `json:"results_dir_basename"`
StartedAt time.Time `json:"started_at"`
FinishedAt *time.Time `json:"finished_at,omitempty"`
GoTestArgs []string `json:"go_test_args"`
TargetSlug string `json:"target_slug"`
DiagnoseIterations int `json:"diagnose_iterations"`
ParallelIterations int `json:"parallel_iterations,omitempty"`
SlowThreshold time.Duration `json:"slow_threshold"`
FailFast bool `json:"fail_fast,omitempty"`
FailFastOn []string `json:"fail_fast_on,omitempty"`
Shuffle bool `json:"shuffle,omitempty"`
}
RunMeta records how the diagnose harness was invoked and where output lives. Use this for full argv and flags; the directory name only carries a short target slug and timestamp.
type RunStateSnapshot ¶ added in v0.0.8
type RunStateSnapshot struct {
Conf *config.App `json:"conf"`
GoTestArgs []string `json:"go_test_args"`
State *DiagnoseRunState `json:"state"`
Start time.Time `json:"start"`
}
RunStateSnapshot is saved to disk so the diagnose analysis phase can be run separately from the execution phase.
func ReadRunState ¶ added in v0.0.8
func ReadRunState(resultsDir string) (*RunStateSnapshot, error)
ReadRunState loads the snapshot from a diagnose results directory.
type TestEntry ¶
type TestEntry struct {
Package string `json:"package"`
Test string `json:"test,omitempty"`
Runs int `json:"runs"`
Successes int `json:"successes"`
Fails int `json:"fails"`
Skips int `json:"skips"`
Timeouts int `json:"timeouts"`
FailRateLower *float64 `json:"fail_rate_lower,omitempty"`
FailRateUpper *float64 `json:"fail_rate_upper,omitempty"`
MinElapsed time.Duration `json:"min_elapsed"`
MaxElapsed time.Duration `json:"max_elapsed"`
P50Elapsed time.Duration `json:"p50_elapsed"`
Logs []ProblemLog `json:"logs,omitempty"`
FailIters []int `json:"-"`
TimeoutIters []int `json:"-"`
SlowIters []int `json:"-"`
}
TestEntry is a single row in the analysis report.
type TestEvent ¶
type TestEvent struct {
Time time.Time
Action string
Package string
Test string
Elapsed float64
Output string
FailedBuild string
}
TestEvent represents a parsed test event from the JSONL log stream.
type TestGroup ¶
type TestGroup struct {
Entries *[]TestEntry
LogKind string
CSVCategory string
Iters func(TestEntry) []int
}
TestGroup defines a category of flagged tests within a Report, exposing references for mutation and metadata for presentation.
type TraceEvent ¶ added in v0.0.5
type TraceEvent struct {
Name string `json:"name"`
Cat string `json:"cat"`
Ph string `json:"ph"` // "X" for Complete event, "M" for Metadata
Ts int64 `json:"ts"` // Microseconds
Dur int64 `json:"dur"` // Microseconds
Pid int `json:"pid"`
Tid int `json:"tid"`
Cname string `json:"cname,omitempty"`
Args map[string]any `json:"args,omitempty"`
}
TraceEvent represents a single event in Chrome's Trace Event Format.
type TraceServeOptions ¶ added in v0.0.5
type TraceServeOptions struct {
// Addr overrides traceListenAddr. Production leaves this empty; tests set "127.0.0.1:0".
Addr string
OpenBrowser func(string) error
}
TraceServeOptions groups parameters for ServeTrace.