Documentation
¶
Overview ¶
Package eval measures a real /run end-to-end: the ATIF trajectories it wrote, the subagent concurrency it reached, and how efficiently its tools were used. It is the metric layer behind the manually-run eval harness (make eval-run) — see eval.md. All computation is pure so the driving test (package tui, which has access to the unexported run handlers) stays thin.
Index ¶
- Variables
- func BuildJudgePrompt(r *RunReport, digest string) string
- func RenderMarkdown(r *RunReport) string
- func TrajectoryDigest(loaded []*LoadedTrajectory, maxCallsPerSession int) string
- func WriteReport(report *RunReport, outDir string) (jsonPath, mdPath, md string, err error)
- type CompleteFunc
- type ConcurrencyMetrics
- type ConcurrencySample
- type GateResult
- type GoldenFile
- type JudgeScore
- type JudgeVerdict
- type LoadedTrajectory
- type ReportMetadata
- type RunOutcome
- type RunReport
- type SessionSummary
- type SessionTokenUsage
- type TokenMetrics
- type ToolStats
- type ToolsMetrics
- type TrajectoryMetrics
Constants ¶
This section is empty.
Variables ¶
var JudgeDimensions = []string{
"outcome_correctness",
"trajectory_quality",
"concurrency_use",
"tools_efficiency",
}
JudgeDimensions are the axes the judge is asked to grade, in prompt order.
Functions ¶
func BuildJudgePrompt ¶
BuildJudgePrompt renders the evidence the judge grades: the run's outcome and metrics, then the condensed tool-call timeline. It deliberately hands over the computed metrics rather than raw trajectories — the numbers are what the harness is confident about, and they fit in a prompt.
func RenderMarkdown ¶
RenderMarkdown renders a RunReport as human-readable Markdown for stdout.
func TrajectoryDigest ¶
func TrajectoryDigest(loaded []*LoadedTrajectory, maxCallsPerSession int) string
TrajectoryDigest condenses the loaded trajectories into the timeline the judge reads: one line per tool call, with truncated arguments and a marker on results that look like failures. Sessions are ordered as loaded (parent first), and each session's calls are capped so one runaway worker cannot crowd the others out of the prompt.
func WriteReport ¶
WriteReport writes the run report as JSON and Markdown into outDir and returns both paths. The Markdown rendering is also returned so the caller can print it to stdout without re-reading the file.
Types ¶
type CompleteFunc ¶
CompleteFunc is a single-shot LLM call: system prompt plus user prompt in, assistant text out. The eval package stays free of provider dependencies by taking one of these from the caller.
type ConcurrencyMetrics ¶
type ConcurrencyMetrics struct {
PoolBudget int `json:"pool_budget"`
WorkerBudget int `json:"worker_budget"`
MaxRunning int `json:"max_running"`
MeanRunning float64 `json:"mean_running"`
AgentsSeen int `json:"agents_seen"`
ParallelOverlap float64 `json:"parallel_overlap"`
Samples []ConcurrencySample `json:"samples"`
}
ConcurrencyMetrics summarizes the top-level orchestrator pool over time. Nested fan-out inside the coordinator process is not visible to the poller — it lives in other processes — so those numbers come from trajectories (see TrajectoryMetrics.NestedAgentCalls / MaxDepth).
func ComputeConcurrencyMetrics ¶
func ComputeConcurrencyMetrics(poolBudget int, samples []ConcurrencySample) ConcurrencyMetrics
ComputeConcurrencyMetrics summarizes the polled orchestrator state.
type ConcurrencySample ¶
type ConcurrencySample struct {
Timestamp time.Time `json:"ts"`
Running int `json:"running"`
AgentIDs []string `json:"agent_ids,omitempty"`
}
ConcurrencySample is one poll of the orchestrator's live agent set.
type GateResult ¶
type GateResult struct {
Name string `json:"name"`
Command string `json:"command"`
Passed bool `json:"passed"`
Output string `json:"output,omitempty"`
}
GateResult is the result of a single gate command.
type GoldenFile ¶
type GoldenFile struct {
Name string `json:"name"`
Match bool `json:"match"`
Diff string `json:"diff,omitempty"`
Error string `json:"error,omitempty"`
}
GoldenFile is a per-file comparison between the produced artifacts and the expected golden copy.
func DiffGolden ¶
func DiffGolden(producedDir, goldenDir string, files []string) ([]GoldenFile, bool)
DiffGolden compares each named file between the produced artifacts directory and the golden copy. Returns the per-file match report and the overall pass. A produced file that is missing counts as a mismatch.
type JudgeScore ¶
type JudgeScore struct {
Dimension string `json:"dimension"`
Score int `json:"score"` // 1 (bad) .. 5 (excellent)
Rationale string `json:"rationale"`
}
JudgeScore is one graded dimension of the run.
type JudgeVerdict ¶
type JudgeVerdict struct {
Model string `json:"model"`
Scores []JudgeScore `json:"scores"`
Overall float64 `json:"overall"` // mean of Scores, 0 when none
Verdict string `json:"verdict"` // "pass" | "fail"
Summary string `json:"summary"`
Issues []string `json:"issues,omitempty"`
// Error is set when the judge could not be run or its reply could not be
// parsed. The rest of the verdict is then zero.
Error string `json:"error,omitempty"`
}
JudgeVerdict is an LLM's assessment of a run. It grades the same three axes the harness measures numerically — trajectory, concurrency, tools — plus the run's outcome, because a run can score well on efficiency and still produce the wrong artifacts.
The verdict is advisory: it is recorded in the report, never asserted on. An LLM grader is not a stable enough signal to fail a build, but it catches the qualitative regressions the counters miss (a worker that thrashed, retried the same failing command, or reached the answer by accident).
func Judge ¶
func Judge(ctx context.Context, complete CompleteFunc, model string, r *RunReport, digest string) JudgeVerdict
Judge asks an LLM to grade a run and returns its verdict. Any failure — transport, empty reply, unparseable JSON — comes back as a verdict carrying Error rather than as a Go error, so an unavailable judge degrades the report instead of failing the eval.
func ParseJudgeVerdict ¶
func ParseJudgeVerdict(reply string) (JudgeVerdict, error)
ParseJudgeVerdict reads the model's reply into a verdict. Models routinely wrap JSON in prose or a code fence despite instructions, so the outermost JSON object is extracted rather than requiring a clean reply.
type LoadedTrajectory ¶
type LoadedTrajectory struct {
Path string
SessionDir string
SessionID string
Traj *atif.Trajectory
}
LoadedTrajectory is a parsed ATIF trajectory with its source path and the session directory it lives in.
func LoadTrajectories ¶
func LoadTrajectories(sessionsDir string) ([]*LoadedTrajectory, error)
LoadTrajectories reads and parses every trajectory.atif.json under the sessions directory. Malformed files are skipped (their parse error is reported via the returned error if nothing at all parsed).
type ReportMetadata ¶
type ReportMetadata struct {
Spec string `json:"spec"`
Mode string `json:"mode"` // "single" | "parallel"
Model string `json:"model"`
Binary string `json:"binary"`
GitHead string `json:"git_head"`
Timestamp time.Time `json:"timestamp"`
Duration string `json:"duration"`
// BaseRef is the pinned starting point the run's worktree was created
// from. Runs started from the same BaseRef are comparable; a run against
// a moving HEAD is not comparable with anything.
BaseRef string `json:"base_ref,omitempty"`
// BaseCommit is BaseRef resolved to a commit, so a report stays meaningful
// after the ref moves.
BaseCommit string `json:"base_commit,omitempty"`
}
ReportMetadata describes what was run and when.
type RunOutcome ¶
type RunOutcome struct {
FinalPhase string `json:"final_phase"` // "done", "failed", ...
Retries int `json:"retries"`
// Reason is the terminal failure message from the run flow ("Merge
// failed…", "Verification failed…", …), empty when the run did not fail.
Reason string `json:"reason,omitempty"`
GateResults []GateResult `json:"gate_results"`
GoldenCheck []GoldenFile `json:"golden_check"`
GoldenPass bool `json:"golden_pass"`
BaselineRef string `json:"baseline_ref,omitempty"`
BaselineCheck []GoldenFile `json:"baseline_check,omitempty"`
// BaselinePass is only meaningful when BaselineRef is non-empty.
BaselinePass bool `json:"baseline_pass"`
}
RunOutcome captures how the run itself ended and whether the produced artifacts matched the golden copy.
type RunReport ¶
type RunReport struct {
Metadata ReportMetadata `json:"metadata"`
Outcome RunOutcome `json:"outcome"`
Trajectory TrajectoryMetrics `json:"trajectory"`
Concurrency ConcurrencyMetrics `json:"concurrency"`
Tools ToolsMetrics `json:"tools"`
Tokens TokenMetrics `json:"tokens"`
// Judge is the LLM grader's advisory verdict, absent when no judge ran.
Judge *JudgeVerdict `json:"judge,omitempty"`
}
RunReport is the complete machine-readable output of one eval run.
type SessionSummary ¶
type SessionSummary struct {
SessionID string `json:"session_id"`
AgentName string `json:"agent_name"`
Model string `json:"model"`
Steps int `json:"steps"`
ToolCalls int `json:"tool_calls"`
SubagentRefs int `json:"subagent_refs"`
Depth int `json:"depth"`
StartedAt string `json:"started_at,omitempty"`
Duration string `json:"duration,omitempty"`
ContinuedFrom string `json:"continued_from,omitempty"`
}
SessionSummary is a one-line summary of a single trajectory.
type SessionTokenUsage ¶
type SessionTokenUsage struct {
SessionID string `json:"session_id"`
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
CachedTokens int `json:"cached_tokens"`
}
SessionTokenUsage is the token usage attributed to one session.
type TokenMetrics ¶
type TokenMetrics struct {
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
CachedTokens int `json:"cached_tokens"`
TotalTokens int `json:"total_tokens"`
CostUSD float64 `json:"cost_usd,omitempty"`
Sessions []SessionTokenUsage `json:"sessions,omitempty"`
}
TokenMetrics aggregates provider-reported token usage across every session in the run. Usage is read from each session's events.jsonl because the ATIF trajectory does not carry it.
func ComputeTokenMetrics ¶
func ComputeTokenMetrics(loaded []*LoadedTrajectory) TokenMetrics
ComputeTokenMetrics sums provider-reported token usage across every session in the run, attributing each session's usage to its trajectory.
type ToolStats ¶
type ToolStats struct {
Calls int `json:"calls"`
Results int `json:"results"`
Errors int `json:"errors"`
Wasted int `json:"wasted"`
Duplicates int `json:"duplicates"`
AvgResultBytes int `json:"avg_result_bytes"`
AvgLatencyMs int `json:"avg_latency_ms"`
}
ToolStats is the per-tool efficiency summary.
- wasted: a tool_call with no matching observation result. The model asked for a result that never arrived — either the call was interrupted or the provider stream ended before the observation was recorded.
- duplicates: occurrences of an identical (function_name, arguments) pair after the first. Repeating the exact same call suggests the model was not paying attention to the result it already had.
- errors: results whose content looks like a failure (heuristic — see errorMarkerRE). A "failed" bash call still produced a result, so it counts as a result AND an error.
type ToolsMetrics ¶
type ToolsMetrics struct {
TotalCalls int `json:"total_calls"`
TotalResults int `json:"total_results"`
Wasted int `json:"wasted"`
Duplicates int `json:"duplicates"`
NestedAgentCalls int `json:"nested_agent_calls"`
ByTool map[string]ToolStats `json:"by_tool"`
}
ToolsMetrics aggregates tool usage across every trajectory in the run.
func ComputeToolsMetrics ¶
func ComputeToolsMetrics(loaded []*LoadedTrajectory) ToolsMetrics
ComputeToolsMetrics derives per-tool efficiency from the run's trajectories.
type TrajectoryMetrics ¶
type TrajectoryMetrics struct {
Sessions []SessionSummary `json:"sessions"`
TotalSteps int `json:"total_steps"`
TotalToolCalls int `json:"total_tool_calls"`
NestedAgentCalls int `json:"nested_agent_calls"`
MaxDepth int `json:"max_depth"`
}
TrajectoryMetrics summarizes the trajectories written during the run, including nested subagent sessions and their nesting depth.
func ComputeTrajectoryMetrics ¶
func ComputeTrajectoryMetrics(loaded []*LoadedTrajectory) TrajectoryMetrics
ComputeTrajectoryMetrics builds the trajectory summary, computing nesting depth by following subagent_trajectory_ref links.