Documentation
¶
Overview ¶
Package eval measures a real /run end-to-end: the ATIF trajectories it wrote, the subagent concurrency it reached, and how efficiently its tools were used. It is the metric layer behind the manually-run eval harness (make eval-run) — see eval.md. All computation is pure so the driving test (package tui, which has access to the unexported run handlers) stays thin.
Index ¶
- Constants
- Variables
- func BuildJudgePrompt(r *RunReport, digest string) string
- func BuildToolsJudgePrompt(r *ToolsReport, digest string) string
- func HomeEnv(home string) []string
- func RenderMarkdown(r *RunReport) string
- func RenderToolsMarkdown(r *ToolsReport) string
- func ScenarioConfig(s Scenario, model, thinking string) config.Config
- func SeedMemory(ctx context.Context, home, project string, seeds []MemorySeed) error
- func SeedWorkspace(s Scenario, workDir, home string) error
- func ToolNames(inv []ToolInfo) []string
- func TrajectoryDigest(loaded []*LoadedTrajectory, maxCallsPerSession int) string
- func UnknownTargets(inv []ToolInfo, scenarios []Scenario) []string
- func UnmappedTools(inv []ToolInfo, scenarios []Scenario, exclusions []Exclusion) []string
- func WriteReport(report *RunReport, outDir string) (jsonPath, mdPath, md string, err error)
- func WriteToolsReport(report *ToolsReport, outDir string) (jsonPath, mdPath, md string, err error)
- type Check
- type CheckKind
- type CheckOutcome
- type CompleteFunc
- type ConcurrencyMetrics
- type ConcurrencySample
- type Coverage
- type Exclusion
- type GateResult
- type GoldenFile
- type JudgeScore
- type JudgeVerdict
- type LoadedTrajectory
- type MemorySeed
- type ReportMetadata
- type RunOutcome
- type RunReport
- type Scenario
- type ScenarioResult
- type SessionSummary
- type SessionTokenUsage
- type TokenMetrics
- type ToolCoverage
- type ToolInfo
- type ToolOutcome
- type ToolStats
- type ToolsMetrics
- type ToolsReport
- type ToolsReportMetadata
- type TrajectoryMetrics
Constants ¶
const ( // CoverageOK: the tool was called and at least one call produced a // result that does not look like an error. CoverageOK = "ok" // CoverageErrorsOnly: every call to the tool produced an error-looking // result (or no result at all). CoverageErrorsOnly = "errors-only" // CoverageNotCalled: the tool has a scenario but no trajectory called it. CoverageNotCalled = "not-called" // CoverageExcluded: the tool is documented as excluded from the suite. CoverageExcluded = "excluded" // CoverageUnmapped: the tool has neither a scenario nor an exclusion. // This is the state the mapping test exists to prevent. CoverageUnmapped = "unmapped" )
Coverage status values, from best to worst.
const ( StatusPass = "pass" StatusFail = "fail" StatusSkip = "skip" StatusError = "error" )
Scenario result statuses.
Variables ¶
var JudgeDimensions = []string{
"outcome_correctness",
"trajectory_quality",
"concurrency_use",
"tools_efficiency",
}
JudgeDimensions are the axes the judge is asked to grade, in prompt order.
var ToolsJudgeDimensions = []string{
"outcome_correctness",
"tool_selection",
"tools_efficiency",
"trajectory_quality",
}
ToolsJudgeDimensions are the axes the tool-coverage judge grades.
Functions ¶
func BuildJudgePrompt ¶
BuildJudgePrompt renders the evidence the judge grades: the run's outcome and metrics, then the condensed tool-call timeline. It deliberately hands over the computed metrics rather than raw trajectories — the numbers are what the harness is confident about, and they fit in a prompt.
func BuildToolsJudgePrompt ¶ added in v0.0.77
func BuildToolsJudgePrompt(r *ToolsReport, digest string) string
BuildToolsJudgePrompt renders the evidence the tools judge grades: the per-scenario verdicts, the coverage summary, then the per-scenario tool-call timelines.
func HomeEnv ¶ added in v0.0.77
HomeEnv returns the environment entries that point the home directory at home for a child process. os.UserHomeDir reads HOME on Unix but USERPROFILE on Windows, so both are set there; with HOME alone a pi process on Windows would silently read and write the runner's real ~/.pi-go (the same trap internal/testenv.SetHome guards against in tests).
func RenderMarkdown ¶
RenderMarkdown renders a RunReport as human-readable Markdown for stdout.
func RenderToolsMarkdown ¶ added in v0.0.77
func RenderToolsMarkdown(r *ToolsReport) string
RenderToolsMarkdown renders a ToolsReport as human-readable Markdown.
func ScenarioConfig ¶ added in v0.0.77
ScenarioConfig derives the config.json a scenario's run is seeded with: the eval model on every role, the given thinking level ("none" when empty — cheapest, and the only level every model accepts), memory off unless the scenario seeds it, llms sources when it asks for them, and no hooks, MCP servers or A2A agents.
func SeedMemory ¶ added in v0.0.77
func SeedMemory(ctx context.Context, home, project string, seeds []MemorySeed) error
SeedMemory creates the observation store the run will open (the default path under home) and inserts the seeds under a synthetic session, so the mem-* tools have something to return.
func SeedWorkspace ¶ added in v0.0.77
SeedWorkspace writes the scenario's fixture files into workDir, creating the git history and uncommitted modifications it asks for. home is the isolated HOME the run uses, so git never reads the user's global config.
func TrajectoryDigest ¶
func TrajectoryDigest(loaded []*LoadedTrajectory, maxCallsPerSession int) string
TrajectoryDigest condenses the loaded trajectories into the timeline the judge reads: one line per tool call, with truncated arguments and a marker on results that look like failures. Sessions are ordered as loaded (parent first), and each session's calls are capped so one runaway worker cannot crowd the others out of the prompt.
func UnknownTargets ¶ added in v0.0.77
UnknownTargets returns scenario tool targets that match nothing in the inventory — a typo, a renamed tool, or a tool that was removed. For an alternatives target ("grep|ripgrep") at least one alternative must exist.
func UnmappedTools ¶ added in v0.0.77
UnmappedTools returns the inventory tools that have neither a scenario targeting them nor an exclusion. A non-empty result means the suite has a blind spot; the mapping test fails on it.
func WriteReport ¶
WriteReport writes the run report as JSON and Markdown into outDir and returns both paths. The Markdown rendering is also returned so the caller can print it to stdout without re-reading the file.
func WriteToolsReport ¶ added in v0.0.77
func WriteToolsReport(report *ToolsReport, outDir string) (jsonPath, mdPath, md string, err error)
WriteToolsReport writes the report as JSON and Markdown into outDir (eval-tools-<ts>.json / .md) and returns both paths plus the Markdown.
Types ¶
type Check ¶ added in v0.0.77
type Check struct {
Kind CheckKind `json:"kind"`
Path string `json:"path,omitempty"`
Text string `json:"text,omitempty"`
Tool string `json:"tool,omitempty"`
Arg string `json:"arg,omitempty"`
N int `json:"n,omitempty"`
}
Check is one deterministic assertion. Which fields matter depends on Kind.
type CheckKind ¶ added in v0.0.77
type CheckKind string
CheckKind enumerates the assertion vocabulary.
const ( // CheckFileExists: Path exists in the workdir. CheckFileExists CheckKind = "file_exists" // CheckFileAbsent: Path does not exist in the workdir. CheckFileAbsent CheckKind = "file_absent" // CheckFileContains: the file at Path contains Text. CheckFileContains CheckKind = "file_contains" // CheckFileNotContains: the file at Path does not contain Text. CheckFileNotContains CheckKind = "file_not_contains" // CheckToolArgContains: some call to Tool has an argument Arg whose // string form contains Text. Arg empty = any argument. CheckToolArgContains CheckKind = "tool_arg_contains" // CheckToolResultContains: some result of Tool contains Text (the // result is matched on its JSON/text rendering). CheckToolResultContains CheckKind = "tool_result_contains" // CheckSubagentSpawned: at least one observation links a nested // subagent trajectory, i.e. a child pi session really ran. CheckSubagentSpawned CheckKind = "subagent_spawned" // CheckToolCalledAtLeast: Tool was called at least N times. CheckToolCalledAtLeast CheckKind = "tool_called_at_least" )
type CheckOutcome ¶ added in v0.0.77
type CheckOutcome struct {
Check string `json:"check"`
Passed bool `json:"passed"`
Detail string `json:"detail,omitempty"`
}
CheckOutcome is the result of one Check.
type CompleteFunc ¶
CompleteFunc is a single-shot LLM call: system prompt plus user prompt in, assistant text out. The eval package stays free of provider dependencies by taking one of these from the caller.
func ProviderComplete ¶ added in v0.0.77
func ProviderComplete(model string) (CompleteFunc, string)
ProviderComplete builds the single-shot LLM call a judge uses, resolving the model the same way the CLI resolves any model. It returns nil and a reason when the model is unresolvable or has no API key, so the caller can record "judge unavailable" and still produce its measured report.
type ConcurrencyMetrics ¶
type ConcurrencyMetrics struct {
PoolBudget int `json:"pool_budget"`
WorkerBudget int `json:"worker_budget"`
MaxRunning int `json:"max_running"`
MeanRunning float64 `json:"mean_running"`
AgentsSeen int `json:"agents_seen"`
ParallelOverlap float64 `json:"parallel_overlap"`
Samples []ConcurrencySample `json:"samples"`
}
ConcurrencyMetrics summarizes the top-level orchestrator pool over time. Nested fan-out inside the coordinator process is not visible to the poller — it lives in other processes — so those numbers come from trajectories (see TrajectoryMetrics.NestedAgentCalls / MaxDepth).
func ComputeConcurrencyMetrics ¶
func ComputeConcurrencyMetrics(poolBudget int, samples []ConcurrencySample) ConcurrencyMetrics
ComputeConcurrencyMetrics summarizes the polled orchestrator state.
type ConcurrencySample ¶
type ConcurrencySample struct {
Timestamp time.Time `json:"ts"`
Running int `json:"running"`
AgentIDs []string `json:"agent_ids,omitempty"`
}
ConcurrencySample is one poll of the orchestrator's live agent set.
type Coverage ¶ added in v0.0.77
type Coverage struct {
Tools []ToolCoverage `json:"tools"`
// Summary counts over Tools.
Total int `json:"total"`
OK int `json:"ok"`
NotCalled int `json:"not_called"`
Errors int `json:"errors_only"`
Excluded int `json:"excluded"`
Unmapped int `json:"unmapped"`
// Unknown lists tools that were called but are not in the inventory —
// MCP tools, or a tool the inventory does not know how to construct.
Unknown []string `json:"unknown,omitempty"`
// Gap is the list of non-excluded tools that were never successfully
// exercised (status not-called, errors-only or unmapped).
Gap []string `json:"gap,omitempty"`
}
Coverage is the tool coverage matrix for a set of trajectories against the registered tool inventory.
func ComputeCoverage ¶ added in v0.0.77
func ComputeCoverage(inv []ToolInfo, scenarios []Scenario, exclusions []Exclusion, loaded []*LoadedTrajectory) Coverage
ComputeCoverage builds the coverage matrix: for each inventoried tool, the scenarios targeting it, its exclusion (if any), and what the trajectories show — calls, results, error-looking results and wasted calls.
type Exclusion ¶ added in v0.0.77
type Exclusion struct {
// Tool is the exact tool name, or a "prefix*" pattern for a whole family
// (e.g. "palace-*").
Tool string `json:"tool"`
Reason string `json:"reason"`
}
Exclusion documents a registered tool that deliberately has no eval scenario. Every tool in the inventory must be either targeted by a scenario or excluded here with a reason — TestScenarios_CoverInventory enforces it — so a new tool cannot silently go unmeasured.
type GateResult ¶
type GateResult struct {
Name string `json:"name"`
Command string `json:"command"`
Passed bool `json:"passed"`
Output string `json:"output,omitempty"`
}
GateResult is the result of a single gate command.
type GoldenFile ¶
type GoldenFile struct {
Name string `json:"name"`
Match bool `json:"match"`
Diff string `json:"diff,omitempty"`
Error string `json:"error,omitempty"`
}
GoldenFile is a per-file comparison between the produced artifacts and the expected golden copy.
func DiffGolden ¶
func DiffGolden(producedDir, goldenDir string, files []string) ([]GoldenFile, bool)
DiffGolden compares each named file between the produced artifacts directory and the golden copy. Returns the per-file match report and the overall pass. A produced file that is missing counts as a mismatch.
type JudgeScore ¶
type JudgeScore struct {
Dimension string `json:"dimension"`
Score int `json:"score"` // 1 (bad) .. 5 (excellent)
Rationale string `json:"rationale"`
}
JudgeScore is one graded dimension of the run.
type JudgeVerdict ¶
type JudgeVerdict struct {
Model string `json:"model"`
Scores []JudgeScore `json:"scores"`
Overall float64 `json:"overall"` // mean of Scores, 0 when none
Verdict string `json:"verdict"` // "pass" | "fail"
Summary string `json:"summary"`
Issues []string `json:"issues,omitempty"`
// Error is set when the judge could not be run or its reply could not be
// parsed. The rest of the verdict is then zero.
Error string `json:"error,omitempty"`
}
JudgeVerdict is an LLM's assessment of a run. It grades the same three axes the harness measures numerically — trajectory, concurrency, tools — plus the run's outcome, because a run can score well on efficiency and still produce the wrong artifacts.
The verdict is advisory: it is recorded in the report, never asserted on. An LLM grader is not a stable enough signal to fail a build, but it catches the qualitative regressions the counters miss (a worker that thrashed, retried the same failing command, or reached the answer by accident).
func Judge ¶
func Judge(ctx context.Context, complete CompleteFunc, model string, r *RunReport, digest string) JudgeVerdict
Judge asks an LLM to grade a run and returns its verdict. Any failure — transport, empty reply, unparseable JSON — comes back as a verdict carrying Error rather than as a Go error, so an unavailable judge degrades the report instead of failing the eval.
func JudgeTools ¶ added in v0.0.77
func JudgeTools(ctx context.Context, complete CompleteFunc, model string, r *ToolsReport, digest string) JudgeVerdict
JudgeTools asks an LLM to grade a tool-coverage run. Like Judge, every failure comes back as a verdict carrying Error rather than a Go error, so an unavailable judge degrades the report instead of failing the eval.
func ParseJudgeVerdict ¶
func ParseJudgeVerdict(reply string) (JudgeVerdict, error)
ParseJudgeVerdict reads the model's reply into a verdict. Models routinely wrap JSON in prose or a code fence despite instructions, so the outermost JSON object is extracted rather than requiring a clean reply.
type LoadedTrajectory ¶
type LoadedTrajectory struct {
Path string
SessionDir string
SessionID string
Traj *atif.Trajectory
}
LoadedTrajectory is a parsed ATIF trajectory with its source path and the session directory it lives in.
func LoadTrajectories ¶
func LoadTrajectories(sessionsDir string) ([]*LoadedTrajectory, error)
LoadTrajectories reads and parses every trajectory.atif.json under the sessions directory. Malformed files are skipped (their parse error is reported via the returned error if nothing at all parsed).
type MemorySeed ¶ added in v0.0.77
type MemorySeed struct {
Title string `json:"title"`
Text string `json:"text"`
Type string `json:"type,omitempty"` // memory.ObservationType; default "discovery"
}
MemorySeed is one observation written into the memory store before the run.
type ReportMetadata ¶
type ReportMetadata struct {
Spec string `json:"spec"`
Mode string `json:"mode"` // "single" | "parallel"
Model string `json:"model"`
Binary string `json:"binary"`
GitHead string `json:"git_head"`
Timestamp time.Time `json:"timestamp"`
Duration string `json:"duration"`
// BaseRef is the pinned starting point the run's worktree was created
// from. Runs started from the same BaseRef are comparable; a run against
// a moving HEAD is not comparable with anything.
BaseRef string `json:"base_ref,omitempty"`
// BaseCommit is BaseRef resolved to a commit, so a report stays meaningful
// after the ref moves.
BaseCommit string `json:"base_commit,omitempty"`
}
ReportMetadata describes what was run and when.
type RunOutcome ¶
type RunOutcome struct {
FinalPhase string `json:"final_phase"` // "done", "failed", ...
Retries int `json:"retries"`
// Reason is the terminal failure message from the run flow ("Merge
// failed…", "Verification failed…", …), empty when the run did not fail.
Reason string `json:"reason,omitempty"`
GateResults []GateResult `json:"gate_results"`
GoldenCheck []GoldenFile `json:"golden_check"`
GoldenPass bool `json:"golden_pass"`
BaselineRef string `json:"baseline_ref,omitempty"`
BaselineCheck []GoldenFile `json:"baseline_check,omitempty"`
// BaselinePass is only meaningful when BaselineRef is non-empty.
BaselinePass bool `json:"baseline_pass"`
}
RunOutcome captures how the run itself ended and whether the produced artifacts matched the golden copy.
type RunReport ¶
type RunReport struct {
Metadata ReportMetadata `json:"metadata"`
Outcome RunOutcome `json:"outcome"`
Trajectory TrajectoryMetrics `json:"trajectory"`
Concurrency ConcurrencyMetrics `json:"concurrency"`
Tools ToolsMetrics `json:"tools"`
Tokens TokenMetrics `json:"tokens"`
// Judge is the LLM grader's advisory verdict, absent when no judge ran.
Judge *JudgeVerdict `json:"judge,omitempty"`
}
RunReport is the complete machine-readable output of one eval run.
type Scenario ¶ added in v0.0.77
type Scenario struct {
// Name identifies the scenario in reports and selects it with
// PI_EVAL_SCENARIO / -run. Lowercase, dash-separated.
Name string `json:"name"`
// Description says what the scenario exercises, for the report.
Description string `json:"description,omitempty"`
// Tools the agent must call at least once, each with at least one
// non-error result. An entry may list alternatives separated by "|"
// ("grep|ripgrep"): any one of them satisfies the requirement, and the
// scenario counts as covering all of them.
Tools []string `json:"tools"`
// Requires lists capabilities the host must provide: "lsp" (a language
// server on PATH), "network" (outbound HTTPS), "vision" (a model that
// accepts images). A scenario whose requirement is unmet is skipped, not
// failed.
Requires []string `json:"requires,omitempty"`
// Files are fixture files written into the workdir before the run, keyed
// by relative path.
Files map[string]string `json:"files,omitempty"`
// Git initializes a repository in the workdir and commits Files as the
// initial commit, so git-* tools have history to look at.
Git bool `json:"git,omitempty"`
// Modified files are written after the initial commit (uncommitted
// changes), so diffs are non-empty. Implies Git.
Modified map[string]string `json:"modified,omitempty"`
// Memory enables the observation memory system for the run and seeds
// the store with the given observations, so mem-* tools have something
// to find. Nil disables memory for the run (the default, because the
// memory worker spawns compressor subagents after every tool call).
Memory []MemorySeed `json:"memory,omitempty"`
// LLMS sources to configure, which registers the fetch_docs tool.
LLMS []config.LLMSSource `json:"llms,omitempty"`
// Args are extra flags passed to the pi binary (e.g. --lsp full).
Args []string `json:"args,omitempty"`
// Prompt is the single user turn the agent runs.
Prompt string `json:"prompt"`
// Checks are the deterministic assertions evaluated after the run, on
// top of the implicit "every target tool was called successfully".
Checks []Check `json:"checks,omitempty"`
// Timeout bounds one run of the scenario. Zero uses the runner default.
Timeout time.Duration `json:"-"`
}
Scenario is one tool-coverage eval case: a seeded workspace, a prompt that should make a competent agent reach for specific tools, and deterministic assertions over the resulting trajectory and filesystem.
Scenarios are declarative data so the table (internal/eval/scenarios) reads as a spec and the runner (the e2e test in the same package) stays generic.
type ScenarioResult ¶ added in v0.0.77
type ScenarioResult struct {
Name string `json:"name"`
Description string `json:"description,omitempty"`
// Status is "pass", "fail", "skip" or "error" (the run itself broke).
Status string `json:"status"`
Reason string `json:"reason,omitempty"`
Duration string `json:"duration,omitempty"`
Tools []ToolOutcome `json:"tools,omitempty"`
Checks []CheckOutcome `json:"checks,omitempty"`
Sessions int `json:"sessions"`
// ToolCalls is the total across every session the run produced,
// including subagents.
ToolCalls int `json:"tool_calls"`
// ExitCode of the pi process, -1 when it did not exit cleanly.
ExitCode int `json:"exit_code"`
Output string `json:"output,omitempty"`
}
ScenarioResult is the graded outcome of one scenario run.
func EvaluateScenario ¶ added in v0.0.77
func EvaluateScenario(s Scenario, workDir string, loaded []*LoadedTrajectory) ScenarioResult
EvaluateScenario grades a finished run: every target tool called with at least one non-error result, and every Check satisfied. workDir is the scenario's workspace after the run; loaded are the trajectories the run wrote (root session plus any subagents).
type SessionSummary ¶
type SessionSummary struct {
SessionID string `json:"session_id"`
AgentName string `json:"agent_name"`
Model string `json:"model"`
Steps int `json:"steps"`
ToolCalls int `json:"tool_calls"`
SubagentRefs int `json:"subagent_refs"`
Depth int `json:"depth"`
StartedAt string `json:"started_at,omitempty"`
Duration string `json:"duration,omitempty"`
ContinuedFrom string `json:"continued_from,omitempty"`
}
SessionSummary is a one-line summary of a single trajectory.
type SessionTokenUsage ¶
type SessionTokenUsage struct {
SessionID string `json:"session_id"`
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
CachedTokens int `json:"cached_tokens"`
}
SessionTokenUsage is the token usage attributed to one session.
type TokenMetrics ¶
type TokenMetrics struct {
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
CachedTokens int `json:"cached_tokens"`
TotalTokens int `json:"total_tokens"`
CostUSD float64 `json:"cost_usd,omitempty"`
Sessions []SessionTokenUsage `json:"sessions,omitempty"`
}
TokenMetrics aggregates provider-reported token usage across every session in the run. Usage is read from each session's events.jsonl because the ATIF trajectory does not carry it.
func ComputeTokenMetrics ¶
func ComputeTokenMetrics(loaded []*LoadedTrajectory) TokenMetrics
ComputeTokenMetrics sums provider-reported token usage across every session in the run, attributing each session's usage to its trajectory.
type ToolCoverage ¶ added in v0.0.77
type ToolCoverage struct {
Name string `json:"name"`
Group string `json:"group"`
Requires string `json:"requires,omitempty"`
// Scenarios that target this tool (may be empty for excluded tools).
Scenarios []string `json:"scenarios,omitempty"`
// Excluded carries the exclusion reason when the tool is excluded.
Excluded string `json:"excluded,omitempty"`
Calls int `json:"calls"`
Results int `json:"results"`
Errors int `json:"errors"`
Wasted int `json:"wasted"`
Status string `json:"status"`
}
ToolCoverage is one row of the coverage matrix.
type ToolInfo ¶ added in v0.0.77
type ToolInfo struct {
// Name is the function name the model calls — exactly what lands in
// ATIF tool_calls[].function_name.
Name string `json:"name"`
// Group is the constructor family the tool comes from (core, bash-control,
// subagent, memory, lsp, llms, a2a, palace, provider).
Group string `json:"group"`
// Requires names the capability the tool is gated on in a real session:
// "" when it is always registered in print mode, otherwise the config or
// environment a session needs before the tool is advertised at all.
Requires string `json:"requires,omitempty"`
}
ToolInfo describes one tool the pi agent can expose to the model.
func Inventory ¶ added in v0.0.77
Inventory enumerates every tool the pi agent can register, by constructing them through the same constructors the CLI uses (tools.CoreTools, tools.BashControlTools, tools.SubagentTools, tools.MemoryTools, tools.LSPToolsFor, tools.LLMSTools, tools.A2ATools, palace.PalaceTools and the Gemini grounding tool). It is the source of truth for "which tools exist": a tool added to any of those constructors shows up here without anyone editing a list, which is what lets the coverage check catch a new tool that has no eval scenario.
Tools are constructed against a throwaway sandbox rooted at dir and never run, so the call is cheap and side-effect free. Optional groups are built with an empty configuration (no LSP server, no llms sources, no A2A agents, an in-memory observation store, a zero palace): the tool objects exist regardless of whether the capability behind them would work, which is exactly what is needed to learn their names.
type ToolOutcome ¶ added in v0.0.77
type ToolOutcome struct {
Tool string `json:"tool"`
Calls int `json:"calls"`
Results int `json:"results"`
Errors int `json:"errors"`
Wasted int `json:"wasted"`
OK bool `json:"ok"`
}
ToolOutcome is what the trajectories show for one target tool.
type ToolStats ¶
type ToolStats struct {
Calls int `json:"calls"`
Results int `json:"results"`
Errors int `json:"errors"`
Wasted int `json:"wasted"`
Duplicates int `json:"duplicates"`
AvgResultBytes int `json:"avg_result_bytes"`
AvgLatencyMs int `json:"avg_latency_ms"`
}
ToolStats is the per-tool efficiency summary.
- wasted: a tool_call with no matching observation result. The model asked for a result that never arrived — either the call was interrupted or the provider stream ended before the observation was recorded.
- duplicates: occurrences of an identical (function_name, arguments) pair after the first. Repeating the exact same call suggests the model was not paying attention to the result it already had.
- errors: results whose content looks like a failure (heuristic — see errorMarkerRE). A "failed" bash call still produced a result, so it counts as a result AND an error.
type ToolsMetrics ¶
type ToolsMetrics struct {
TotalCalls int `json:"total_calls"`
TotalResults int `json:"total_results"`
Wasted int `json:"wasted"`
Duplicates int `json:"duplicates"`
NestedAgentCalls int `json:"nested_agent_calls"`
ByTool map[string]ToolStats `json:"by_tool"`
}
ToolsMetrics aggregates tool usage across every trajectory in the run.
func ComputeToolsMetrics ¶
func ComputeToolsMetrics(loaded []*LoadedTrajectory) ToolsMetrics
ComputeToolsMetrics derives per-tool efficiency from the run's trajectories.
type ToolsReport ¶ added in v0.0.77
type ToolsReport struct {
Metadata ToolsReportMetadata `json:"metadata"`
Scenarios []ScenarioResult `json:"scenarios"`
Coverage Coverage `json:"coverage"`
Tools ToolsMetrics `json:"tools"`
Tokens TokenMetrics `json:"tokens"`
// Judge is the LLM grader's advisory verdict, absent when no judge ran.
Judge *JudgeVerdict `json:"judge,omitempty"`
}
ToolsReport is the machine-readable output of one tool-coverage eval run (make eval-tools): per-scenario verdicts, the coverage matrix against the registered tool inventory, and the usual tool/token metrics over every trajectory the scenarios produced.
func (*ToolsReport) Tally ¶ added in v0.0.77
func (r *ToolsReport) Tally()
Tally counts the scenario statuses into the metadata.
type ToolsReportMetadata ¶ added in v0.0.77
type ToolsReportMetadata struct {
Model string `json:"model"`
Binary string `json:"binary"`
GitHead string `json:"git_head"`
Timestamp time.Time `json:"timestamp"`
Duration string `json:"duration"`
// Selected is the scenario filter in effect (PI_EVAL_SCENARIO), empty for
// the whole suite. A filtered run's coverage gap is not meaningful.
Selected string `json:"selected,omitempty"`
Passed int `json:"passed"`
Failed int `json:"failed"`
Skipped int `json:"skipped"`
Errored int `json:"errored"`
}
ToolsReportMetadata describes what was run and when.
type TrajectoryMetrics ¶
type TrajectoryMetrics struct {
Sessions []SessionSummary `json:"sessions"`
TotalSteps int `json:"total_steps"`
TotalToolCalls int `json:"total_tool_calls"`
NestedAgentCalls int `json:"nested_agent_calls"`
MaxDepth int `json:"max_depth"`
}
TrajectoryMetrics summarizes the trajectories written during the run, including nested subagent sessions and their nesting depth.
func ComputeTrajectoryMetrics ¶
func ComputeTrajectoryMetrics(loaded []*LoadedTrajectory) TrajectoryMetrics
ComputeTrajectoryMetrics builds the trajectory summary, computing nesting depth by following subagent_trajectory_ref links.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package scenarios is the tool-coverage eval suite for the pi coding agent: one declarative scenario per tool family, each seeding a workspace and asking for something a competent agent does with specific tools, plus deterministic assertions over the trajectory and filesystem.
|
Package scenarios is the tool-coverage eval suite for the pi coding agent: one declarative scenario per tool family, each seeding a workspace and asking for something a competent agent does with specific tools, plus deterministic assertions over the trajectory and filesystem. |