runner

package
v0.0.8 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 16, 2026 License: MIT Imports: 34 Imported by: 0

Documentation

Overview

Package runner implements go test wrappers, iteration orchestration, and result analysis.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Analyze

func Analyze(iterations []io.Reader, slowThreshold time.Duration) (*Report, LogMap, error)

Analyze reads per-iteration test2json streams and classifies tests. Malformed lines are silently skipped (go test can interleave non-JSON).

func AnalyzeResults

func AnalyzeResults(resultsDir string, slowThreshold time.Duration) (*Report, LogMap, error)

AnalyzeResults opens every `iteration-*.log.jsonl` file in resultsDir, in numeric-iteration order, and delegates to Analyze.

func EffectiveParallelIterations

func EffectiveParallelIterations(conf *config.App) int

EffectiveParallelIterations returns the bounded diagnose worker count.

func FinishDiagnoseAnalysis added in v0.0.8

func FinishDiagnoseAnalysis(
	ctx context.Context,
	conf *config.App,
	out *output.Printer,
	goTestArgs []string,
	state *DiagnoseRunState,
	start time.Time,
	resultsDir string,
) error

FinishDiagnoseAnalysis completes the analysis phase of a diagnose run. It is intended to be called after RunIterations and after resources have been cleaned up.

func GoTest

func GoTest(ctx context.Context, conf *config.App, args []string, env []string) error

GoTest runs `go test` with the given args (repo root as working directory). env is appended to the child process environment (e.g. resource Env).

func Gotestsum

func Gotestsum(ctx context.Context, conf *config.App, args []string, env []string) error

Gotestsum runs `gotestsum` with the given args (repo root as working directory). env is appended to the child process environment (e.g. resource Env).

func PrintSummary

func PrintSummary(w io.Writer, rep *Report)

PrintSummary writes a human-readable diagnose summary: scope line and rates table first, then detail sections (Broken, Flaky, Timeout, Slow) as flat lines. Broken and Timeout entries are sorted alphabetically by package then test. Flaky entries are sorted by fails/runs (desc), then fails (desc), then name. Slow entries are sorted by max runtime (desc), then name.

func ServeTrace added in v0.0.5

func ServeTrace(
	ctx context.Context,
	resultsDir string,
	files []string,
	out *output.Printer,
	opts TraceServeOptions,
) error

ServeTrace serves trace files at traceListenAddr, opens the Perfetto UI in browser tabs, and blocks until all traces are fetched or ctx is cancelled.

func WarnDiagnoseGoTestCount

func WarnDiagnoseGoTestCount(goTestArgs []string) error

WarnDiagnoseGoTestCount returns an error if the user sets -count on go test.

func WarnDiagnoseGoTestTrace added in v0.0.5

func WarnDiagnoseGoTestTrace(goTestArgs []string) error

WarnDiagnoseGoTestTrace returns an error if the user sets -trace on go test.

func WilsonScoreInterval

func WilsonScoreInterval(k, n int, z float64) (lower float64, upper float64)

WilsonScoreInterval calculates our confidence interval for how (non) flaky a test is. We use a 95% confidence interval, so z = 1.96. https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval

func WriteCSV

func WriteCSV(resultsDir string, rep *Report) error

WriteCSV writes a human-readable CSV of every flagged test (Flakes ∪ Failures ∪ Timeouts ∪ Slow) to <resultsDir>/report.csv. Rows sort worst-first: (timeouts+fails) desc, then package, then test.

func WriteLogFiles

func WriteLogFiles(resultsDir string, rep *Report, logs LogMap) error

WriteLogFiles writes per-test per-iteration log files under <resultsDir>/logs/ for flagged tests and populates each flagged TestEntry's Logs slice with a compact problem-kind, iteration-range, and path pattern.

func WriteReport

func WriteReport(resultsDir string, rep *Report) error

WriteReport writes the report as pretty JSON to <resultsDir>/report.json.

func WriteTrace added in v0.0.5

func WriteTrace(out *output.Printer, resultsDir string) ([]string, error)

WriteTrace parses iteration logs under resultsDir and writes a separate trace-<N>.json file per iteration. It returns a list of the generated filenames.

Types

type DiagnoseRunState added in v0.0.8

type DiagnoseRunState struct {
	// contains filtered or unexported fields
}

DiagnoseRunState captures the runtime progress and limits of a diagnose session.

func RunIterations added in v0.0.8

func RunIterations(
	ctx context.Context,
	conf *config.App,
	out *output.Printer,
	goTestArgs []string,
	resources []hooks.Resource,
	iterSetup, iterTeardown func(context.Context) error,
) (*DiagnoseRunState, time.Time, string, error)

RunIterations runs go test -json once per iteration, writing each stream to iteration-<n>.log.jsonl under a new results directory and run-state.json on return. Test iteration failures do not stop later runs (unless --fail-fast); compile/build failures stop immediately. Returns a non-nil error for setup failures (e.g. mkdir, database reset) or ctx errors from dependencies — not for failing tests alone. iterSetup and iterTeardown run before/after each iteration. Either may be nil. Teardown runs even when the iteration's go test invocation fails; its error is reported only when the iteration itself succeeded. Call FinishDiagnoseAnalysis separately to analyze results and write report.json.

type IterationDigest

type IterationDigest struct {
	Result       string // pass, fail, timeout
	RanTests     int    // distinct named tests (package.test) that executed (pass/fail/timeout), excluding skip-only
	FailTests    int    // len(IterationSummaries[0].FailingTests)
	TimeoutTests int    // len(Timeouts) for this iteration
	SkipTests    int    // distinct named tests skipped in this iteration
	SlowTests    int    // tests over slow threshold
	BuildFailure bool   // compile/build failed or heuristic package-level fail with no named tests run
}

IterationDigest summarizes one iteration JSONL log for per-iteration CLI output. Counts match a single-iteration Analyze (same rules as the final report).

func DigestIterationJSONL

func DigestIterationJSONL(r io.Reader, slowThreshold time.Duration) (IterationDigest, error)

DigestIterationJSONL parses one `go test -json` stream and returns counts for progress UI. It uses the same scan + report pipeline as Analyze for one iteration (no redundant Analyze wrapper).

type IterationSummary

type IterationSummary struct {
	Index        int           `json:"index"`
	Duration     time.Duration `json:"duration,omitempty"`
	Result       string        `json:"result"` // "pass", "fail", "timeout"
	FailingTests []string      `json:"failing_tests,omitempty"`
	ShuffleSeed  int64         `json:"shuffle_seed,omitempty"`
}

IterationSummary captures high-level stats for a single diagnose iteration. Duration and ShuffleSeed are populated by the runner after analysis.

type LogMap

type LogMap map[testKey]map[int]string

LogMap maps (package,test) → iteration → raw interleaved output. Returned alongside Report so callers can write per-test log files without coupling the parser to the filesystem.

type ProblemLog

type ProblemLog struct {
	Type  string `json:"type"`
	Iters string `json:"iters"`
	Path  string `json:"path"`
}

ProblemLog points to log files for iterations where this entry actually had the reported problem. Path uses "{iter}" as the iteration placeholder.

type Report

type Report struct {
	Run                *RunMeta           `json:"run,omitempty"`
	Iterations         int                `json:"iterations"`
	SlowThreshold      time.Duration      `json:"slow_threshold"`
	Summary            *ReportSummary     `json:"summary,omitempty"`
	IterationSummaries []IterationSummary `json:"iteration_summaries,omitempty"`
	Flakes             []TestEntry        `json:"flakes,omitempty"`
	Failures           []TestEntry        `json:"failures,omitempty"`
	Timeouts           []TestEntry        `json:"timeouts,omitempty"`
	Slow               []TestEntry        `json:"slow,omitempty"`
	SlowestPackages    []TestEntry        `json:"slowest_packages,omitempty"`
}

Report classifies tests across iterations of a diagnose run.

func (*Report) TestGroups

func (rep *Report) TestGroups() []TestGroup

TestGroups returns all flagged test categories in precedence order (Timeout > Failure > Flake > Slow) for deduplication and output generation.

type ReportSummary

type ReportSummary struct {
	DistinctNamedTests          int      `json:"distinct_named_tests"`
	FlakeNamedCount             int      `json:"flake_named_count"`
	FlakePrevalence             *float64 `json:"flake_prevalence,omitempty"`
	FlakeFailRuns               int      `json:"flake_fail_runs,omitempty"`
	FlakeTotalRuns              int      `json:"flake_total_runs,omitempty"`
	FlakeExecutionFailRate      *float64 `json:"flake_execution_fail_rate,omitempty"`
	FlakeExecutionFailRateLower *float64 `json:"flake_execution_fail_rate_lower,omitempty"`
	FlakeExecutionFailRateUpper *float64 `json:"flake_execution_fail_rate_upper,omitempty"`
	// FlakeFailingIterations is how many diagnose iterations had at least one
	// flake failure; FlakeIterationTotal is rep.Iterations (not summed per-test runs).
	FlakeFailingIterations      int      `json:"flake_failing_iterations,omitempty"`
	FlakeIterationTotal         int      `json:"flake_iteration_total,omitempty"`
	FlakeIterationFailRate      *float64 `json:"flake_iteration_fail_rate,omitempty"`
	FlakeIterationFailRateLower *float64 `json:"flake_iteration_fail_rate_lower,omitempty"`
	FlakeIterationFailRateUpper *float64 `json:"flake_iteration_fail_rate_upper,omitempty"`
	SlowCount                   int      `json:"slow_count,omitempty"`
	SlowPrevalence              *float64 `json:"slow_prevalence,omitempty"`
	// IterationDurationMin/Max/P50 summarize wall-clock runtimes (IterationSummary.Duration) across all completed iterations.
	IterationDurationMin time.Duration `json:"iteration_duration_min,omitempty"`
	IterationDurationMax time.Duration `json:"iteration_duration_max,omitempty"`
	IterationDurationP50 time.Duration `json:"iteration_duration_p50,omitempty"`
}

ReportSummary holds aggregate flake and slow rates for the full diagnose run. FlakePrevalence uses distinct named tests (package.test keys). Per-execution flake_fail_runs / flake_total_runs sum across flaky entries; flake_failing_iterations / flake_iteration_total count diagnose iterations (union of flake failures vs rep.Iterations).

type RunMeta

type RunMeta struct {
	ResultsDirBasename string        `json:"results_dir_basename"`
	StartedAt          time.Time     `json:"started_at"`
	FinishedAt         *time.Time    `json:"finished_at,omitempty"`
	GoTestArgs         []string      `json:"go_test_args"`
	TargetSlug         string        `json:"target_slug"`
	DiagnoseIterations int           `json:"diagnose_iterations"`
	ParallelIterations int           `json:"parallel_iterations,omitempty"`
	SlowThreshold      time.Duration `json:"slow_threshold"`
	FailFast           bool          `json:"fail_fast,omitempty"`
	FailFastOn         []string      `json:"fail_fast_on,omitempty"`
	Shuffle            bool          `json:"shuffle,omitempty"`
}

RunMeta records how the diagnose harness was invoked and where output lives. Use this for full argv and flags; the directory name only carries a short target slug and timestamp.

type RunStateSnapshot added in v0.0.8

type RunStateSnapshot struct {
	Conf       *config.App       `json:"conf"`
	GoTestArgs []string          `json:"go_test_args"`
	State      *DiagnoseRunState `json:"state"`
	Start      time.Time         `json:"start"`
}

RunStateSnapshot is saved to disk so the diagnose analysis phase can be run separately from the execution phase.

func ReadRunState added in v0.0.8

func ReadRunState(resultsDir string) (*RunStateSnapshot, error)

ReadRunState loads the snapshot from a diagnose results directory.

type TestEntry

type TestEntry struct {
	Package       string        `json:"package"`
	Test          string        `json:"test,omitempty"`
	Runs          int           `json:"runs"`
	Successes     int           `json:"successes"`
	Fails         int           `json:"fails"`
	Skips         int           `json:"skips"`
	Timeouts      int           `json:"timeouts"`
	FailRateLower *float64      `json:"fail_rate_lower,omitempty"`
	FailRateUpper *float64      `json:"fail_rate_upper,omitempty"`
	MinElapsed    time.Duration `json:"min_elapsed"`
	MaxElapsed    time.Duration `json:"max_elapsed"`
	P50Elapsed    time.Duration `json:"p50_elapsed"`
	Logs          []ProblemLog  `json:"logs,omitempty"`
	FailIters     []int         `json:"-"`
	TimeoutIters  []int         `json:"-"`
	SlowIters     []int         `json:"-"`
}

TestEntry is a single row in the analysis report.

type TestEvent

type TestEvent struct {
	Time        time.Time
	Action      string
	Package     string
	Test        string
	Elapsed     float64
	Output      string
	FailedBuild string
}

TestEvent represents a parsed test event from the JSONL log stream.

type TestGroup

type TestGroup struct {
	Entries     *[]TestEntry
	LogKind     string
	CSVCategory string
	Iters       func(TestEntry) []int
}

TestGroup defines a category of flagged tests within a Report, exposing references for mutation and metadata for presentation.

type TraceEvent added in v0.0.5

type TraceEvent struct {
	Name  string         `json:"name"`
	Cat   string         `json:"cat"`
	Ph    string         `json:"ph"`  // "X" for Complete event, "M" for Metadata
	Ts    int64          `json:"ts"`  // Microseconds
	Dur   int64          `json:"dur"` // Microseconds
	Pid   int            `json:"pid"`
	Tid   int            `json:"tid"`
	Cname string         `json:"cname,omitempty"`
	Args  map[string]any `json:"args,omitempty"`
}

TraceEvent represents a single event in Chrome's Trace Event Format.

type TraceServeOptions added in v0.0.5

type TraceServeOptions struct {
	// Addr overrides traceListenAddr. Production leaves this empty; tests set "127.0.0.1:0".
	Addr        string
	OpenBrowser func(string) error
}

TraceServeOptions groups parameters for ServeTrace.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL