Documentation
¶
Index ¶
- Constants
- func BuildDigest(testOutcomes []models.TestOutcome, durationMs int64, runsPerTest int) models.OutcomeDigest
- func ComputeTestStats(runs []models.RunResult) *models.TestStats
- func FilterTestCases(testCases []*models.TestCase, taskPatterns []string, tagPatterns []string) ([]*models.TestCase, error)
- func RegradeOutcome(original *models.EvaluationOutcome, gradedOutcomes []models.TestOutcome, ...) *models.EvaluationOutcome
- func ResolveWorkers(requested, jobs int, phase string, out io.Writer) int
- func ResolveWorkersStderr(requested, jobs int, phase string) int
- type EvalRunner
- type EventType
- type ProgressEvent
- type ProgressListener
- type RunnerOption
- func WithCache(c *cache.Cache) RunnerOption
- func WithRedactionPolicy(p *snapshot.Policy) RunnerOption
- func WithSkipGraders() RunnerOption
- func WithSnapshotEnvAllow(keys []string) RunnerOption
- func WithSnapshotWriter(w *snapshot.Writer) RunnerOption
- func WithTagFilters(patterns ...string) RunnerOption
- func WithTaskFilters(patterns ...string) RunnerOption
- func WithTelemetry(p *telemetry.Provider) RunnerOption
- func WithUpdateSnapshots(enabled bool) RunnerOption
- func WithWazaVersion(v string) RunnerOption
- type TestRunnerdeprecated
Constants ¶
const DefaultWorkerCap = 8
DefaultWorkerCap is the conservative upper bound applied when --workers is not set explicitly. It protects high-CPU CI runners from spawning a session per core (memory pressure, Copilot session limits) until R4 introduces a separate --sessions knob. See docs/design/135-improve-concurrency.md.
Variables ¶
This section is empty.
Functions ¶
func BuildDigest ¶ added in v0.22.0
func BuildDigest(testOutcomes []models.TestOutcome, durationMs int64, runsPerTest int) models.OutcomeDigest
BuildDigest computes an OutcomeDigest from test outcomes. durationMs is the total wall-clock duration to store in the digest. runsPerTest controls whether digest-level bootstrap CI is computed (requires > 1).
func ComputeTestStats ¶ added in v0.22.0
ComputeTestStats computes aggregate statistics for a set of run results.
func FilterTestCases ¶
func FilterTestCases(testCases []*models.TestCase, taskPatterns []string, tagPatterns []string) ([]*models.TestCase, error)
FilterTestCases returns the subset of testCases based on whether it matches tags or task display name, or task id glob patterns. - taskPatterns - matches either the task display name or the task ID. - tagPatterns - matches tags.
If taskPatterns and tagPatterns are specified the result is the intersection of the matches between them. If both taskPatterns and tagPatterns are empty, all test cases are returned.
func RegradeOutcome ¶ added in v0.22.0
func RegradeOutcome(original *models.EvaluationOutcome, gradedOutcomes []models.TestOutcome, judgeModel string) *models.EvaluationOutcome
RegradeOutcome produces a new EvaluationOutcome by replacing test outcomes in the original with the graded ones and recomputing stats and digest.
func ResolveWorkers ¶ added in v0.37.0
ResolveWorkers picks the effective worker count for a phase that is about to dispatch `jobs` units of work.
- When `requested <= 0`, defaults to min(runtime.NumCPU(), jobs, DefaultWorkerCap).
- When `requested > 0`, clamps to `jobs` (no point spawning more workers than there is work) and emits a one-line notice to `out` so users understand actual parallelism.
`phase` is a short label such as "tasks" or "trigger tests" used in the notice. Pass `nil` for `out` to silence the message (tests, etc.).
func ResolveWorkersStderr ¶ added in v0.37.0
ResolveWorkersStderr is a convenience wrapper that logs to os.Stderr.
Types ¶
type EvalRunner ¶ added in v0.31.0
type EvalRunner struct {
// contains filtered or unexported fields
}
EvalRunner orchestrates the execution of tests.
Deprecated alias: TestRunner is provided for backward compatibility.
func NewEvalRunner ¶ added in v0.31.0
func NewEvalRunner(cfg *config.EvalConfig, engine execution.AgentEngine, opts ...RunnerOption) *EvalRunner
NewEvalRunner creates a new test runner. The caller owns the engine and is responsible for initializing and shutting it down as needed.
func NewTestRunner
deprecated
func NewTestRunner(cfg *config.EvalConfig, engine execution.AgentEngine, opts ...RunnerOption) *EvalRunner
Deprecated: Use NewEvalRunner instead.
func (*EvalRunner) OnProgress ¶ added in v0.31.0
func (r *EvalRunner) OnProgress(listener ProgressListener)
OnProgress registers a progress listener
func (*EvalRunner) RunBenchmark ¶ added in v0.31.0
func (r *EvalRunner) RunBenchmark(ctx context.Context) (*models.EvaluationOutcome, error)
RunBenchmark executes the entire benchmark If Baseline is enabled, runs twice: skills-enabled and skills-disabled
type EventType ¶
type EventType string
EventType represents the type of progress event
const ( EventBenchmarkStart EventType = "benchmark_start" EventBenchmarkComplete EventType = "benchmark_complete" EventBenchmarkStopped EventType = "benchmark_stopped" EventTestStart EventType = "test_start" EventTestComplete EventType = "test_complete" EventTestCached EventType = "test_cached" EventRunStart EventType = "run_start" EventRunComplete EventType = "run_complete" EventAgentPrompt EventType = "agent_prompt" EventAgentResponse EventType = "agent_response" EventGraderResult EventType = "grader_result" )
EventType constants
type ProgressEvent ¶
type ProgressEvent struct {
EventType EventType
TestName string
TestNum int
TotalTests int
RunNum int
TotalRuns int
Status models.Status
DurationMs int64
Details map[string]any
}
ProgressEvent represents a progress update
type ProgressListener ¶
type ProgressListener func(event ProgressEvent)
ProgressListener receives progress updates
type RunnerOption ¶
type RunnerOption func(*EvalRunner)
RunnerOption configures a EvalRunner.
func WithRedactionPolicy ¶ added in v0.38.0
func WithRedactionPolicy(p *snapshot.Policy) RunnerOption
WithRedactionPolicy overrides the default snapshot redaction policy. Pass nil to use snapshot.DefaultPolicy().
func WithSkipGraders ¶ added in v0.22.0
func WithSkipGraders() RunnerOption
WithSkipGraders disables grading so only execution occurs.
func WithSnapshotEnvAllow ¶ added in v0.38.0
func WithSnapshotEnvAllow(keys []string) RunnerOption
WithSnapshotEnvAllow sets the env-var allow-list captured in each snapshot's env block. Default is empty (default-deny).
func WithSnapshotWriter ¶ added in v0.38.0
func WithSnapshotWriter(w *snapshot.Writer) RunnerOption
WithSnapshotWriter enables snapshot capture for every completed run. The resulting snapshot.json file path is recorded on RunResult.SnapshotPath so consumers of results.json can correlate runs to their snapshots.
Snapshot capture is best-effort: write failures are logged but never fail the run.
func WithTagFilters ¶
func WithTagFilters(patterns ...string) RunnerOption
func WithTaskFilters ¶
func WithTaskFilters(patterns ...string) RunnerOption
WithTaskFilters sets glob patterns used to filter test cases by DisplayName or TestID.
func WithTelemetry ¶ added in v0.38.0
func WithTelemetry(p *telemetry.Provider) RunnerOption
WithTelemetry attaches an OpenTelemetry provider so the runner emits eval/task/turn/tool_call/model_call spans. Pass nil (or omit) to disable tracing; in that case the runner's internal calls fall through the no-op tracer and the runner behaves exactly as it did before.
func WithUpdateSnapshots ¶
func WithUpdateSnapshots(enabled bool) RunnerOption
WithUpdateSnapshots enables snapshot file updates in diff graders.
func WithWazaVersion ¶ added in v0.38.0
func WithWazaVersion(v string) RunnerOption
WithWazaVersion records the waza binary version on each snapshot for diagnostics. Defaults to empty (no version stamping).
type TestRunner
deprecated
type TestRunner = EvalRunner
Deprecated: Use EvalRunner instead.
Source Files