Help us shape the future of Go! Take the 2026 Go Developer Survey

eval

package
v0.10.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 30, 2026 License: MIT Imports: 15 Imported by: 0

Documentation

Overview

Package eval runs a suite of coding tasks against tinyagent and reports how often each model/profile configuration passes the task's own checks. See docs/eval-framework.md for the design.

Index

Constants

View Source 
const (
	OutcomePassed      = "passed"
	OutcomeFailedCheck = "failed_check"
	OutcomeLimit       = "limit"
	OutcomeNeedsInput  = "NeedsInput"
	OutcomeError       = "error"
	OutcomeSkipped     = "skipped"
)

Outcomes recorded per trial.

View Source 
const DefaultCheckTimeout = 300

DefaultCheckTimeout bounds a check that does not set timeout_seconds.

View Source 
const MaxCheckOutput = 4096

MaxCheckOutput caps the check output kept in results.jsonl (the tail is kept).

View Source 
const SchemaVersion = 1

SchemaVersion of results.jsonl records and report.json.

Variables

This section is empty.

Functions

func DryRun 

func DryRun(w io.Writer, s *Suite, defaults Limits, est Estimator)

DryRun prints the expanded matrix and a cost upper bound without running anything. defaults are the forwarded run limits a task may override.

func ExecAgent 

func ExecAgent(exe string) func(context.Context, []string, io.Writer) error

ExecAgent returns a RunAgent that runs exe as a child process. On context cancellation the child receives an interrupt so it can save its trajectory.

func PassAtK 

func PassAtK(n, c, k int) float64

PassAtK is the unbiased estimator 1 - C(n-c, k)/C(n, k) of the chance that at least one of k samples passes, given c passes in n trials.

func PassHatK 

func PassHatK(n, c, k int) float64

PassHatK is C(c, k)/C(n, k), the chance that all k samples pass.

func Run 

func Run(ctx context.Context, o Options) int

Run executes the suite sequentially and writes results.jsonl, report.json and report.md to OutDir. It returns the exit code: 0 on success, 1 when the pass-rate gate fails or the harness fails, 130 when interrupted.

Types

type Check 

type Check struct {
	Argv           []string `yaml:"argv"`
	TimeoutSeconds int      `yaml:"timeout_seconds"`
}

Check is one command run in the trial's worktree; exit 0 means pass.

type CheckResult 

type CheckResult struct {
	Argv       []string `json:"argv"`
	ExitCode   int      `json:"exit_code"`
	DurationMS int64    `json:"duration_ms"`
	TimedOut   bool     `json:"timed_out,omitempty"`
	Output     string   `json:"output,omitempty"`
}

CheckResult is one check's outcome in a result record.

type ConfigReport 

type ConfigReport struct {
	Model         string             `json:"model"`
	Profile       string             `json:"profile"`
	Trials        int                `json:"trials"`
	Attempted     int                `json:"attempted"`
	Passed        int                `json:"passed"`
	PassRate      float64            `json:"pass_rate"`
	PassAtK       map[string]float64 `json:"pass_at_k"`
	PassHatK      map[string]float64 `json:"pass_hat_k"`
	Outcomes      map[string]int     `json:"outcomes"`
	CostUSD       float64            `json:"cost_usd"`
	MeanCostUSD   float64            `json:"mean_cost_usd"`
	CostPerPass   *float64           `json:"cost_per_pass_usd"`
	InputTokens   int                `json:"input_tokens"`
	OutputTokens  int                `json:"output_tokens"`
	MeanSteps     float64            `json:"mean_steps"`
	MeanWallTimeS float64            `json:"mean_wall_time_s"`
	Tasks         []TaskReport       `json:"tasks"`
}

ConfigReport aggregates one (model, profile) configuration.

type Estimator 

type Estimator func(model string, l Limits) (usd float64, ok bool, reason string)

Estimator returns the per-trial upper-bound cost in USD for a model under the given limits, or ok=false with a reason when no bound exists.

type Limits 

type Limits struct {
	StepLimit  int     `yaml:"step_limit"`
	CostLimit  float64 `yaml:"cost_limit"`
	TokenLimit int     `yaml:"token_limit"` // input+output tokens
	TimeLimit  int     `yaml:"time_limit"`  // seconds
}

Limits are per-trial run limits passed to the child tinyagent.

type Options 

type Options struct {
	Suite       *Suite
	SuitePath   string
	OutDir      string
	Keep        bool    // keep worktrees after each trial
	CostLimit   float64 // suite-wide USD budget; 0 = none
	TimeLimit   time.Duration
	MinPassRate float64  // < 0 = no gate
	ExtraArgs   []string // flags forwarded to every child run
	Version     string
	Stderr      io.Writer
	// RunAgent runs one child tinyagent with args, writing its output to log.
	RunAgent func(ctx context.Context, args []string, log io.Writer) error
}

Options configures one eval run.

type Record 

type Record struct {
	SchemaVersion    int           `json:"schema_version"`
	TrialID          string        `json:"trial_id"`
	TaskID           string        `json:"task_id"`
	Trial            int           `json:"trial"`
	Model            string        `json:"model"`
	Provider         string        `json:"provider,omitempty"`
	ActiveModel      string        `json:"active_model,omitempty"`
	FallbackModel    string        `json:"fallback_model,omitempty"`
	Profile          string        `json:"profile"`
	TinyagentVersion string        `json:"tinyagent_version"`
	BaseCommit       string        `json:"base_commit"`
	Outcome          string        `json:"outcome"`
	ExitStatus       string        `json:"exit_status,omitempty"`
	Checks           []CheckResult `json:"checks"`
	CostUSD          float64       `json:"cost_usd"`
	InputTokens      int           `json:"input_tokens"`
	OutputTokens     int           `json:"output_tokens"`
	Steps            int           `json:"steps"`
	WallTimeMS       int64         `json:"wall_time_ms"`
	Trajectory       string        `json:"trajectory,omitempty"`
	Log              string        `json:"log,omitempty"`
	Worktree         string        `json:"worktree,omitempty"`
	Error            string        `json:"error,omitempty"`
	SkipReason       string        `json:"skip_reason,omitempty"`
}

Record is one line of results.jsonl.

type Report 

type Report struct {
	SchemaVersion    int            `json:"schema_version"`
	Suite            string         `json:"suite"`
	TinyagentVersion string         `json:"tinyagent_version"`
	Trials           int            `json:"trials_per_cell"`
	MinPassRate      *float64       `json:"min_pass_rate,omitempty"`
	Configs          []ConfigReport `json:"configs"`
	BelowThreshold   []string       `json:"below_threshold,omitempty"`
}

Report is the aggregate written to report.json.

func BuildReport 

func BuildReport(o Options, recs []Record) Report

BuildReport aggregates records by configuration in first-seen order.

type Suite 

type Suite struct {
	Version  int      `yaml:"version"`
	Models   []string `yaml:"models"`
	Profiles []string `yaml:"profiles"`
	Trials   int      `yaml:"trials"`
	Tasks    []Task   `yaml:"tasks"`
	// contains filtered or unexported fields
}

Suite is a parsed, validated suite file.

func Load 

func Load(path string, knownProfile func(string) bool) (*Suite, error)

Load reads and validates a suite file. knownProfile reports whether a profile name exists in the user's config; "default" is always accepted and means the base config.

func (*Suite) Expand 

func (s *Suite) Expand() []Trial

Expand returns the trial matrix in a stable order: task, model, profile, trial number. A task's own model/profile replaces the suite list.

type Task 

type Task struct {
	ID            string   `yaml:"id"`
	Repo          string   `yaml:"repo"`
	Base          string   `yaml:"base"`
	Task          string   `yaml:"task"`
	TaskFile      string   `yaml:"task_file"`
	Model         string   `yaml:"model"`
	Profile       string   `yaml:"profile"`
	Checks        []Check  `yaml:"checks"`
	ChecksOnLimit *bool    `yaml:"checks_on_limit"`
	Limits        Limits   `yaml:"limits"`
	Tags          []string `yaml:"tags"`
	// contains filtered or unexported fields
}

Task is one coding task: a repo at a base commit, a prompt, and checks.

type TaskReport 

type TaskReport struct {
	TaskID    string `json:"task_id"`
	Attempted int    `json:"attempted"`
	Passed    int    `json:"passed"`
}

TaskReport is one task's results within a configuration.

type Trial 

type Trial struct {
	ID      string
	Task    *Task
	Model   string
	Profile string
	N       int // 1-based trial number
}

Trial is one expanded (task, model, profile, n) cell of the matrix.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.