eval

package
v0.0.66 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 10, 2026 License: MIT Imports: 10 Imported by: 0

Documentation

Overview

Package eval measures a real /run end-to-end: the ATIF trajectories it wrote, the subagent concurrency it reached, and how efficiently its tools were used. It is the metric layer behind the manually-run eval harness (make eval-run) — see eval.md. All computation is pure so the driving test (package tui, which has access to the unexported run handlers) stays thin.

Index

Constants

This section is empty.

Variables

View Source
var JudgeDimensions = []string{
	"outcome_correctness",
	"trajectory_quality",
	"concurrency_use",
	"tools_efficiency",
}

JudgeDimensions are the axes the judge is asked to grade, in prompt order.

Functions

func BuildJudgePrompt

func BuildJudgePrompt(r *RunReport, digest string) string

BuildJudgePrompt renders the evidence the judge grades: the run's outcome and metrics, then the condensed tool-call timeline. It deliberately hands over the computed metrics rather than raw trajectories — the numbers are what the harness is confident about, and they fit in a prompt.

func RenderMarkdown

func RenderMarkdown(r *RunReport) string

RenderMarkdown renders a RunReport as human-readable Markdown for stdout.

func TrajectoryDigest

func TrajectoryDigest(loaded []*LoadedTrajectory, maxCallsPerSession int) string

TrajectoryDigest condenses the loaded trajectories into the timeline the judge reads: one line per tool call, with truncated arguments and a marker on results that look like failures. Sessions are ordered as loaded (parent first), and each session's calls are capped so one runaway worker cannot crowd the others out of the prompt.

func WriteReport

func WriteReport(report *RunReport, outDir string) (jsonPath, mdPath, md string, err error)

WriteReport writes the run report as JSON and Markdown into outDir and returns both paths. The Markdown rendering is also returned so the caller can print it to stdout without re-reading the file.

Types

type CompleteFunc

type CompleteFunc func(ctx context.Context, system, user string) (string, error)

CompleteFunc is a single-shot LLM call: system prompt plus user prompt in, assistant text out. The eval package stays free of provider dependencies by taking one of these from the caller.

type ConcurrencyMetrics

type ConcurrencyMetrics struct {
	PoolBudget      int                 `json:"pool_budget"`
	WorkerBudget    int                 `json:"worker_budget"`
	MaxRunning      int                 `json:"max_running"`
	MeanRunning     float64             `json:"mean_running"`
	AgentsSeen      int                 `json:"agents_seen"`
	ParallelOverlap float64             `json:"parallel_overlap"`
	Samples         []ConcurrencySample `json:"samples"`
}

ConcurrencyMetrics summarizes the top-level orchestrator pool over time. Nested fan-out inside the coordinator process is not visible to the poller — it lives in other processes — so those numbers come from trajectories (see TrajectoryMetrics.NestedAgentCalls / MaxDepth).

func ComputeConcurrencyMetrics

func ComputeConcurrencyMetrics(poolBudget int, samples []ConcurrencySample) ConcurrencyMetrics

ComputeConcurrencyMetrics summarizes the polled orchestrator state.

type ConcurrencySample

type ConcurrencySample struct {
	Timestamp time.Time `json:"ts"`
	Running   int       `json:"running"`
	AgentIDs  []string  `json:"agent_ids,omitempty"`
}

ConcurrencySample is one poll of the orchestrator's live agent set.

type GateResult

type GateResult struct {
	Name    string `json:"name"`
	Command string `json:"command"`
	Passed  bool   `json:"passed"`
	Output  string `json:"output,omitempty"`
}

GateResult is the result of a single gate command.

type GoldenFile

type GoldenFile struct {
	Name  string `json:"name"`
	Match bool   `json:"match"`
	Diff  string `json:"diff,omitempty"`
	Error string `json:"error,omitempty"`
}

GoldenFile is a per-file comparison between the produced artifacts and the expected golden copy.

func DiffGolden

func DiffGolden(producedDir, goldenDir string, files []string) ([]GoldenFile, bool)

DiffGolden compares each named file between the produced artifacts directory and the golden copy. Returns the per-file match report and the overall pass. A produced file that is missing counts as a mismatch.

type JudgeScore

type JudgeScore struct {
	Dimension string `json:"dimension"`
	Score     int    `json:"score"` // 1 (bad) .. 5 (excellent)
	Rationale string `json:"rationale"`
}

JudgeScore is one graded dimension of the run.

type JudgeVerdict

type JudgeVerdict struct {
	Model   string       `json:"model"`
	Scores  []JudgeScore `json:"scores"`
	Overall float64      `json:"overall"` // mean of Scores, 0 when none
	Verdict string       `json:"verdict"` // "pass" | "fail"
	Summary string       `json:"summary"`
	Issues  []string     `json:"issues,omitempty"`
	// Error is set when the judge could not be run or its reply could not be
	// parsed. The rest of the verdict is then zero.
	Error string `json:"error,omitempty"`
}

JudgeVerdict is an LLM's assessment of a run. It grades the same three axes the harness measures numerically — trajectory, concurrency, tools — plus the run's outcome, because a run can score well on efficiency and still produce the wrong artifacts.

The verdict is advisory: it is recorded in the report, never asserted on. An LLM grader is not a stable enough signal to fail a build, but it catches the qualitative regressions the counters miss (a worker that thrashed, retried the same failing command, or reached the answer by accident).

func Judge

func Judge(ctx context.Context, complete CompleteFunc, model string, r *RunReport, digest string) JudgeVerdict

Judge asks an LLM to grade a run and returns its verdict. Any failure — transport, empty reply, unparseable JSON — comes back as a verdict carrying Error rather than as a Go error, so an unavailable judge degrades the report instead of failing the eval.

func ParseJudgeVerdict

func ParseJudgeVerdict(reply string) (JudgeVerdict, error)

ParseJudgeVerdict reads the model's reply into a verdict. Models routinely wrap JSON in prose or a code fence despite instructions, so the outermost JSON object is extracted rather than requiring a clean reply.

type LoadedTrajectory

type LoadedTrajectory struct {
	Path       string
	SessionDir string
	SessionID  string
	Traj       *atif.Trajectory
}

LoadedTrajectory is a parsed ATIF trajectory with its source path and the session directory it lives in.

func LoadTrajectories

func LoadTrajectories(sessionsDir string) ([]*LoadedTrajectory, error)

LoadTrajectories reads and parses every trajectory.atif.json under the sessions directory. Malformed files are skipped (their parse error is reported via the returned error if nothing at all parsed).

type ReportMetadata

type ReportMetadata struct {
	Spec      string    `json:"spec"`
	Mode      string    `json:"mode"` // "single" | "parallel"
	Model     string    `json:"model"`
	Binary    string    `json:"binary"`
	GitHead   string    `json:"git_head"`
	Timestamp time.Time `json:"timestamp"`
	Duration  string    `json:"duration"`
	// BaseRef is the pinned starting point the run's worktree was created
	// from. Runs started from the same BaseRef are comparable; a run against
	// a moving HEAD is not comparable with anything.
	BaseRef string `json:"base_ref,omitempty"`
	// BaseCommit is BaseRef resolved to a commit, so a report stays meaningful
	// after the ref moves.
	BaseCommit string `json:"base_commit,omitempty"`
}

ReportMetadata describes what was run and when.

type RunOutcome

type RunOutcome struct {
	FinalPhase string `json:"final_phase"` // "done", "failed", ...
	Retries    int    `json:"retries"`
	// Reason is the terminal failure message from the run flow ("Merge
	// failed…", "Verification failed…", …), empty when the run did not fail.
	Reason        string       `json:"reason,omitempty"`
	GateResults   []GateResult `json:"gate_results"`
	GoldenCheck   []GoldenFile `json:"golden_check"`
	GoldenPass    bool         `json:"golden_pass"`
	BaselineRef   string       `json:"baseline_ref,omitempty"`
	BaselineCheck []GoldenFile `json:"baseline_check,omitempty"`
	// BaselinePass is only meaningful when BaselineRef is non-empty.
	BaselinePass bool `json:"baseline_pass"`
}

RunOutcome captures how the run itself ended and whether the produced artifacts matched the golden copy.

type RunReport

type RunReport struct {
	Metadata    ReportMetadata     `json:"metadata"`
	Outcome     RunOutcome         `json:"outcome"`
	Trajectory  TrajectoryMetrics  `json:"trajectory"`
	Concurrency ConcurrencyMetrics `json:"concurrency"`
	Tools       ToolsMetrics       `json:"tools"`
	Tokens      TokenMetrics       `json:"tokens"`
	// Judge is the LLM grader's advisory verdict, absent when no judge ran.
	Judge *JudgeVerdict `json:"judge,omitempty"`
}

RunReport is the complete machine-readable output of one eval run.

type SessionSummary

type SessionSummary struct {
	SessionID     string `json:"session_id"`
	AgentName     string `json:"agent_name"`
	Model         string `json:"model"`
	Steps         int    `json:"steps"`
	ToolCalls     int    `json:"tool_calls"`
	SubagentRefs  int    `json:"subagent_refs"`
	Depth         int    `json:"depth"`
	StartedAt     string `json:"started_at,omitempty"`
	Duration      string `json:"duration,omitempty"`
	ContinuedFrom string `json:"continued_from,omitempty"`
}

SessionSummary is a one-line summary of a single trajectory.

type SessionTokenUsage

type SessionTokenUsage struct {
	SessionID        string `json:"session_id"`
	PromptTokens     int    `json:"prompt_tokens"`
	CompletionTokens int    `json:"completion_tokens"`
	CachedTokens     int    `json:"cached_tokens"`
}

SessionTokenUsage is the token usage attributed to one session.

type TokenMetrics

type TokenMetrics struct {
	PromptTokens     int                 `json:"prompt_tokens"`
	CompletionTokens int                 `json:"completion_tokens"`
	CachedTokens     int                 `json:"cached_tokens"`
	TotalTokens      int                 `json:"total_tokens"`
	CostUSD          float64             `json:"cost_usd,omitempty"`
	Sessions         []SessionTokenUsage `json:"sessions,omitempty"`
}

TokenMetrics aggregates provider-reported token usage across every session in the run. Usage is read from each session's events.jsonl because the ATIF trajectory does not carry it.

func ComputeTokenMetrics

func ComputeTokenMetrics(loaded []*LoadedTrajectory) TokenMetrics

ComputeTokenMetrics sums provider-reported token usage across every session in the run, attributing each session's usage to its trajectory.

type ToolStats

type ToolStats struct {
	Calls          int `json:"calls"`
	Results        int `json:"results"`
	Errors         int `json:"errors"`
	Wasted         int `json:"wasted"`
	Duplicates     int `json:"duplicates"`
	AvgResultBytes int `json:"avg_result_bytes"`
	AvgLatencyMs   int `json:"avg_latency_ms"`
}

ToolStats is the per-tool efficiency summary.

  • wasted: a tool_call with no matching observation result. The model asked for a result that never arrived — either the call was interrupted or the provider stream ended before the observation was recorded.
  • duplicates: occurrences of an identical (function_name, arguments) pair after the first. Repeating the exact same call suggests the model was not paying attention to the result it already had.
  • errors: results whose content looks like a failure (heuristic — see errorMarkerRE). A "failed" bash call still produced a result, so it counts as a result AND an error.

type ToolsMetrics

type ToolsMetrics struct {
	TotalCalls       int                  `json:"total_calls"`
	TotalResults     int                  `json:"total_results"`
	Wasted           int                  `json:"wasted"`
	Duplicates       int                  `json:"duplicates"`
	NestedAgentCalls int                  `json:"nested_agent_calls"`
	ByTool           map[string]ToolStats `json:"by_tool"`
}

ToolsMetrics aggregates tool usage across every trajectory in the run.

func ComputeToolsMetrics

func ComputeToolsMetrics(loaded []*LoadedTrajectory) ToolsMetrics

ComputeToolsMetrics derives per-tool efficiency from the run's trajectories.

type TrajectoryMetrics

type TrajectoryMetrics struct {
	Sessions         []SessionSummary `json:"sessions"`
	TotalSteps       int              `json:"total_steps"`
	TotalToolCalls   int              `json:"total_tool_calls"`
	NestedAgentCalls int              `json:"nested_agent_calls"`
	MaxDepth         int              `json:"max_depth"`
}

TrajectoryMetrics summarizes the trajectories written during the run, including nested subagent sessions and their nesting depth.

func ComputeTrajectoryMetrics

func ComputeTrajectoryMetrics(loaded []*LoadedTrajectory) TrajectoryMetrics

ComputeTrajectoryMetrics builds the trajectory summary, computing nesting depth by following subagent_trajectory_ref links.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL