eval

package
v0.4.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 17, 2026 License: MIT Imports: 23 Imported by: 0

Documentation

Overview

Package eval measures a real /run end-to-end: the ATIF trajectories it wrote, the subagent concurrency it reached, and how efficiently its tools were used. It is the metric layer behind the manually-run eval harness (make eval-run) — see eval.md. All computation is pure so the driving test (package tui, which has access to the unexported run handlers) stays thin.

Index

Constants

View Source
const (
	// CoverageOK: the tool was called and at least one call produced a
	// result that does not look like an error.
	CoverageOK = "ok"
	// CoverageErrorsOnly: every call to the tool produced an error-looking
	// result (or no result at all).
	CoverageErrorsOnly = "errors-only"
	// CoverageNotCalled: the tool has a scenario but no trajectory called it.
	CoverageNotCalled = "not-called"
	// CoverageExcluded: the tool is documented as excluded from the suite.
	CoverageExcluded = "excluded"
	// CoverageUnmapped: the tool has neither a scenario nor an exclusion.
	// This is the state the mapping test exists to prevent.
	CoverageUnmapped = "unmapped"
)

Coverage status values, from best to worst.

View Source
const (
	StatusPass  = "pass"
	StatusFail  = "fail"
	StatusSkip  = "skip"
	StatusError = "error"
)

Scenario result statuses.

Variables

View Source
var JudgeDimensions = []string{
	"outcome_correctness",
	"trajectory_quality",
	"concurrency_use",
	"tools_efficiency",
}

JudgeDimensions are the axes the judge is asked to grade, in prompt order.

View Source
var ToolsJudgeDimensions = []string{
	"outcome_correctness",
	"tool_selection",
	"tools_efficiency",
	"trajectory_quality",
}

ToolsJudgeDimensions are the axes the tool-coverage judge grades.

Functions

func BuildJudgePrompt

func BuildJudgePrompt(r *RunReport, digest string) string

BuildJudgePrompt renders the evidence the judge grades: the run's outcome and metrics, then the condensed tool-call timeline. It deliberately hands over the computed metrics rather than raw trajectories — the numbers are what the harness is confident about, and they fit in a prompt.

func BuildToolsJudgePrompt added in v0.0.77

func BuildToolsJudgePrompt(r *ToolsReport, digest string) string

BuildToolsJudgePrompt renders the evidence the tools judge grades: the per-scenario verdicts, the coverage summary, then the per-scenario tool-call timelines.

func HomeEnv added in v0.0.77

func HomeEnv(home string) []string

HomeEnv returns the environment entries that point the home directory at home for a child process. os.UserHomeDir reads HOME on Unix but USERPROFILE on Windows, so both are set there; with HOME alone a pi process on Windows would silently read and write the runner's real ~/.pi-go (the same trap internal/testenv.SetHome guards against in tests).

func RenderMarkdown

func RenderMarkdown(r *RunReport) string

RenderMarkdown renders a RunReport as human-readable Markdown for stdout.

func RenderToolsMarkdown added in v0.0.77

func RenderToolsMarkdown(r *ToolsReport) string

RenderToolsMarkdown renders a ToolsReport as human-readable Markdown.

func ScenarioConfig added in v0.0.77

func ScenarioConfig(s Scenario, model, thinking string) config.Config

ScenarioConfig derives the config.json a scenario's run is seeded with: the eval model on every role, the given thinking level ("none" when empty — cheapest, and the only level every model accepts), memory off unless the scenario seeds it, llms sources when it asks for them, and no hooks, MCP servers or A2A agents.

func SeedMemory added in v0.0.77

func SeedMemory(ctx context.Context, home, project string, seeds []MemorySeed) error

SeedMemory creates the observation store the run will open (the default path under home) and inserts the seeds under a synthetic session, so the mem-* tools have something to return.

func SeedWorkspace added in v0.0.77

func SeedWorkspace(s Scenario, workDir, home string) error

SeedWorkspace writes the scenario's fixture files into workDir, creating the git history and uncommitted modifications it asks for. home is the isolated HOME the run uses, so git never reads the user's global config.

func ToolNames added in v0.0.77

func ToolNames(inv []ToolInfo) []string

ToolNames returns the names of an inventory in order.

func TrajectoryDigest

func TrajectoryDigest(loaded []*LoadedTrajectory, maxCallsPerSession int) string

TrajectoryDigest condenses the loaded trajectories into the timeline the judge reads: one line per tool call, with truncated arguments and a marker on results that look like failures. Sessions are ordered as loaded (parent first), and each session's calls are capped so one runaway worker cannot crowd the others out of the prompt.

func UnknownTargets added in v0.0.77

func UnknownTargets(inv []ToolInfo, scenarios []Scenario) []string

UnknownTargets returns scenario tool targets that match nothing in the inventory — a typo, a renamed tool, or a tool that was removed. For an alternatives target ("grep|ripgrep") at least one alternative must exist.

func UnmappedTools added in v0.0.77

func UnmappedTools(inv []ToolInfo, scenarios []Scenario, exclusions []Exclusion) []string

UnmappedTools returns the inventory tools that have neither a scenario targeting them nor an exclusion. A non-empty result means the suite has a blind spot; the mapping test fails on it.

func WriteReport

func WriteReport(report *RunReport, outDir string) (jsonPath, mdPath, md string, err error)

WriteReport writes the run report as JSON and Markdown into outDir and returns both paths. The Markdown rendering is also returned so the caller can print it to stdout without re-reading the file.

func WriteToolsReport added in v0.0.77

func WriteToolsReport(report *ToolsReport, outDir string) (jsonPath, mdPath, md string, err error)

WriteToolsReport writes the report as JSON and Markdown into outDir (eval-tools-<ts>.json / .md) and returns both paths plus the Markdown.

Types

type Check added in v0.0.77

type Check struct {
	Kind CheckKind `json:"kind"`
	Path string    `json:"path,omitempty"`
	Text string    `json:"text,omitempty"`
	Tool string    `json:"tool,omitempty"`
	Arg  string    `json:"arg,omitempty"`
	N    int       `json:"n,omitempty"`
}

Check is one deterministic assertion. Which fields matter depends on Kind.

func (Check) String added in v0.0.77

func (c Check) String() string

String renders the check for reports.

type CheckKind added in v0.0.77

type CheckKind string

CheckKind enumerates the assertion vocabulary.

const (
	// CheckFileExists: Path exists in the workdir.
	CheckFileExists CheckKind = "file_exists"
	// CheckFileAbsent: Path does not exist in the workdir.
	CheckFileAbsent CheckKind = "file_absent"
	// CheckFileContains: the file at Path contains Text.
	CheckFileContains CheckKind = "file_contains"
	// CheckFileNotContains: the file at Path does not contain Text.
	CheckFileNotContains CheckKind = "file_not_contains"
	// CheckToolArgContains: some call to Tool has an argument Arg whose
	// string form contains Text. Arg empty = any argument.
	CheckToolArgContains CheckKind = "tool_arg_contains"
	// CheckToolResultContains: some result of Tool contains Text (the
	// result is matched on its JSON/text rendering).
	CheckToolResultContains CheckKind = "tool_result_contains"
	// CheckSubagentSpawned: at least one observation links a nested
	// subagent trajectory, i.e. a child pi session really ran.
	CheckSubagentSpawned CheckKind = "subagent_spawned"
	// CheckToolCalledAtLeast: Tool was called at least N times.
	CheckToolCalledAtLeast CheckKind = "tool_called_at_least"
)

type CheckOutcome added in v0.0.77

type CheckOutcome struct {
	Check  string `json:"check"`
	Passed bool   `json:"passed"`
	Detail string `json:"detail,omitempty"`
}

CheckOutcome is the result of one Check.

type CompleteFunc

type CompleteFunc func(ctx context.Context, system, user string) (string, error)

CompleteFunc is a single-shot LLM call: system prompt plus user prompt in, assistant text out. The eval package stays free of provider dependencies by taking one of these from the caller.

func ProviderComplete added in v0.0.77

func ProviderComplete(model string) (CompleteFunc, string)

ProviderComplete builds the single-shot LLM call a judge uses, resolving the model the same way the CLI resolves any model. It returns nil and a reason when the model is unresolvable or has no API key, so the caller can record "judge unavailable" and still produce its measured report.

type ConcurrencyMetrics

type ConcurrencyMetrics struct {
	PoolBudget      int                 `json:"pool_budget"`
	WorkerBudget    int                 `json:"worker_budget"`
	MaxRunning      int                 `json:"max_running"`
	MeanRunning     float64             `json:"mean_running"`
	AgentsSeen      int                 `json:"agents_seen"`
	ParallelOverlap float64             `json:"parallel_overlap"`
	Samples         []ConcurrencySample `json:"samples"`
}

ConcurrencyMetrics summarizes the top-level orchestrator pool over time. Nested fan-out inside the coordinator process is not visible to the poller — it lives in other processes — so those numbers come from trajectories (see TrajectoryMetrics.NestedAgentCalls / MaxDepth).

func ComputeConcurrencyMetrics

func ComputeConcurrencyMetrics(poolBudget int, samples []ConcurrencySample) ConcurrencyMetrics

ComputeConcurrencyMetrics summarizes the polled orchestrator state.

type ConcurrencySample

type ConcurrencySample struct {
	Timestamp time.Time `json:"ts"`
	Running   int       `json:"running"`
	AgentIDs  []string  `json:"agent_ids,omitempty"`
}

ConcurrencySample is one poll of the orchestrator's live agent set.

type Coverage added in v0.0.77

type Coverage struct {
	Tools []ToolCoverage `json:"tools"`
	// Summary counts over Tools.
	Total     int `json:"total"`
	OK        int `json:"ok"`
	NotCalled int `json:"not_called"`
	Errors    int `json:"errors_only"`
	Excluded  int `json:"excluded"`
	Unmapped  int `json:"unmapped"`
	// Unknown lists tools that were called but are not in the inventory —
	// MCP tools, or a tool the inventory does not know how to construct.
	Unknown []string `json:"unknown,omitempty"`
	// Gap is the list of non-excluded tools that were never successfully
	// exercised (status not-called, errors-only or unmapped).
	Gap []string `json:"gap,omitempty"`
}

Coverage is the tool coverage matrix for a set of trajectories against the registered tool inventory.

func ComputeCoverage added in v0.0.77

func ComputeCoverage(inv []ToolInfo, scenarios []Scenario, exclusions []Exclusion, loaded []*LoadedTrajectory) Coverage

ComputeCoverage builds the coverage matrix: for each inventoried tool, the scenarios targeting it, its exclusion (if any), and what the trajectories show — calls, results, error-looking results and wasted calls.

type Exclusion added in v0.0.77

type Exclusion struct {
	// Tool is the exact tool name, or a "prefix*" pattern for a whole family
	// (e.g. "palace-*").
	Tool   string `json:"tool"`
	Reason string `json:"reason"`
}

Exclusion documents a registered tool that deliberately has no eval scenario. Every tool in the inventory must be either targeted by a scenario or excluded here with a reason — TestScenarios_CoverInventory enforces it — so a new tool cannot silently go unmeasured.

type GateResult

type GateResult struct {
	Name    string `json:"name"`
	Command string `json:"command"`
	Passed  bool   `json:"passed"`
	Output  string `json:"output,omitempty"`
}

GateResult is the result of a single gate command.

type GoldenFile

type GoldenFile struct {
	Name  string `json:"name"`
	Match bool   `json:"match"`
	Diff  string `json:"diff,omitempty"`
	Error string `json:"error,omitempty"`
}

GoldenFile is a per-file comparison between the produced artifacts and the expected golden copy.

func DiffGolden

func DiffGolden(producedDir, goldenDir string, files []string) ([]GoldenFile, bool)

DiffGolden compares each named file between the produced artifacts directory and the golden copy. Returns the per-file match report and the overall pass. A produced file that is missing counts as a mismatch.

type JudgeScore

type JudgeScore struct {
	Dimension string `json:"dimension"`
	Score     int    `json:"score"` // 1 (bad) .. 5 (excellent)
	Rationale string `json:"rationale"`
}

JudgeScore is one graded dimension of the run.

type JudgeVerdict

type JudgeVerdict struct {
	Model   string       `json:"model"`
	Scores  []JudgeScore `json:"scores"`
	Overall float64      `json:"overall"` // mean of Scores, 0 when none
	Verdict string       `json:"verdict"` // "pass" | "fail"
	Summary string       `json:"summary"`
	Issues  []string     `json:"issues,omitempty"`
	// Error is set when the judge could not be run or its reply could not be
	// parsed. The rest of the verdict is then zero.
	Error string `json:"error,omitempty"`
}

JudgeVerdict is an LLM's assessment of a run. It grades the same three axes the harness measures numerically — trajectory, concurrency, tools — plus the run's outcome, because a run can score well on efficiency and still produce the wrong artifacts.

The verdict is advisory: it is recorded in the report, never asserted on. An LLM grader is not a stable enough signal to fail a build, but it catches the qualitative regressions the counters miss (a worker that thrashed, retried the same failing command, or reached the answer by accident).

func Judge

func Judge(ctx context.Context, complete CompleteFunc, model string, r *RunReport, digest string) JudgeVerdict

Judge asks an LLM to grade a run and returns its verdict. Any failure — transport, empty reply, unparseable JSON — comes back as a verdict carrying Error rather than as a Go error, so an unavailable judge degrades the report instead of failing the eval.

func JudgeTools added in v0.0.77

func JudgeTools(ctx context.Context, complete CompleteFunc, model string, r *ToolsReport, digest string) JudgeVerdict

JudgeTools asks an LLM to grade a tool-coverage run. Like Judge, every failure comes back as a verdict carrying Error rather than a Go error, so an unavailable judge degrades the report instead of failing the eval.

func ParseJudgeVerdict

func ParseJudgeVerdict(reply string) (JudgeVerdict, error)

ParseJudgeVerdict reads the model's reply into a verdict. Models routinely wrap JSON in prose or a code fence despite instructions, so the outermost JSON object is extracted rather than requiring a clean reply.

type LoadedTrajectory

type LoadedTrajectory struct {
	Path       string
	SessionDir string
	SessionID  string
	Traj       *atif.Trajectory
}

LoadedTrajectory is a parsed ATIF trajectory with its source path and the session directory it lives in.

func LoadTrajectories

func LoadTrajectories(sessionsDir string) ([]*LoadedTrajectory, error)

LoadTrajectories reads and parses every trajectory.atif.json under the sessions directory. Malformed files are skipped (their parse error is reported via the returned error if nothing at all parsed).

type MemorySeed added in v0.0.77

type MemorySeed struct {
	Title string `json:"title"`
	Text  string `json:"text"`
	Type  string `json:"type,omitempty"` // memory.ObservationType; default "discovery"
}

MemorySeed is one observation written into the memory store before the run.

type ReportMetadata

type ReportMetadata struct {
	Spec      string    `json:"spec"`
	Mode      string    `json:"mode"` // "single" | "parallel"
	Model     string    `json:"model"`
	Binary    string    `json:"binary"`
	GitHead   string    `json:"git_head"`
	Timestamp time.Time `json:"timestamp"`
	Duration  string    `json:"duration"`
	// BaseRef is the pinned starting point the run's worktree was created
	// from. Runs started from the same BaseRef are comparable; a run against
	// a moving HEAD is not comparable with anything.
	BaseRef string `json:"base_ref,omitempty"`
	// BaseCommit is BaseRef resolved to a commit, so a report stays meaningful
	// after the ref moves.
	BaseCommit string `json:"base_commit,omitempty"`
}

ReportMetadata describes what was run and when.

type RunOutcome

type RunOutcome struct {
	FinalPhase string `json:"final_phase"` // "done", "failed", ...
	Retries    int    `json:"retries"`
	// Reason is the terminal failure message from the run flow ("Merge
	// failed…", "Verification failed…", …), empty when the run did not fail.
	Reason        string       `json:"reason,omitempty"`
	GateResults   []GateResult `json:"gate_results"`
	GoldenCheck   []GoldenFile `json:"golden_check"`
	GoldenPass    bool         `json:"golden_pass"`
	BaselineRef   string       `json:"baseline_ref,omitempty"`
	BaselineCheck []GoldenFile `json:"baseline_check,omitempty"`
	// BaselinePass is only meaningful when BaselineRef is non-empty.
	BaselinePass bool `json:"baseline_pass"`
}

RunOutcome captures how the run itself ended and whether the produced artifacts matched the golden copy.

type RunReport

type RunReport struct {
	Metadata    ReportMetadata     `json:"metadata"`
	Outcome     RunOutcome         `json:"outcome"`
	Trajectory  TrajectoryMetrics  `json:"trajectory"`
	Concurrency ConcurrencyMetrics `json:"concurrency"`
	Tools       ToolsMetrics       `json:"tools"`
	Tokens      TokenMetrics       `json:"tokens"`
	// Judge is the LLM grader's advisory verdict, absent when no judge ran.
	Judge *JudgeVerdict `json:"judge,omitempty"`
}

RunReport is the complete machine-readable output of one eval run.

type Scenario added in v0.0.77

type Scenario struct {
	// Name identifies the scenario in reports and selects it with
	// PI_EVAL_SCENARIO / -run. Lowercase, dash-separated.
	Name string `json:"name"`
	// Description says what the scenario exercises, for the report.
	Description string `json:"description,omitempty"`
	// Tools the agent must call at least once, each with at least one
	// non-error result. An entry may list alternatives separated by "|"
	// ("grep|ripgrep"): any one of them satisfies the requirement, and the
	// scenario counts as covering all of them.
	Tools []string `json:"tools"`
	// Requires lists capabilities the host must provide: "lsp" (a language
	// server on PATH), "network" (outbound HTTPS), "vision" (a model that
	// accepts images). A scenario whose requirement is unmet is skipped, not
	// failed.
	Requires []string `json:"requires,omitempty"`
	// Files are fixture files written into the workdir before the run, keyed
	// by relative path.
	Files map[string]string `json:"files,omitempty"`
	// Git initializes a repository in the workdir and commits Files as the
	// initial commit, so git-* tools have history to look at.
	Git bool `json:"git,omitempty"`
	// Modified files are written after the initial commit (uncommitted
	// changes), so diffs are non-empty. Implies Git.
	Modified map[string]string `json:"modified,omitempty"`
	// Memory enables the observation memory system for the run and seeds
	// the store with the given observations, so mem-* tools have something
	// to find. Nil disables memory for the run (the default, because the
	// memory worker spawns compressor subagents after every tool call).
	Memory []MemorySeed `json:"memory,omitempty"`
	// LLMS sources to configure, which registers the fetch_docs tool.
	LLMS []config.LLMSSource `json:"llms,omitempty"`
	// Args are extra flags passed to the pi binary (e.g. --lsp full).
	Args []string `json:"args,omitempty"`
	// Prompt is the single user turn the agent runs.
	Prompt string `json:"prompt"`
	// Checks are the deterministic assertions evaluated after the run, on
	// top of the implicit "every target tool was called successfully".
	Checks []Check `json:"checks,omitempty"`
	// Timeout bounds one run of the scenario. Zero uses the runner default.
	Timeout time.Duration `json:"-"`
}

Scenario is one tool-coverage eval case: a seeded workspace, a prompt that should make a competent agent reach for specific tools, and deterministic assertions over the resulting trajectory and filesystem.

Scenarios are declarative data so the table (internal/eval/scenarios) reads as a spec and the runner (the e2e test in the same package) stays generic.

type ScenarioResult added in v0.0.77

type ScenarioResult struct {
	Name        string `json:"name"`
	Description string `json:"description,omitempty"`
	// Status is "pass", "fail", "skip" or "error" (the run itself broke).
	Status   string         `json:"status"`
	Reason   string         `json:"reason,omitempty"`
	Duration string         `json:"duration,omitempty"`
	Tools    []ToolOutcome  `json:"tools,omitempty"`
	Checks   []CheckOutcome `json:"checks,omitempty"`
	Sessions int            `json:"sessions"`
	// ToolCalls is the total across every session the run produced,
	// including subagents.
	ToolCalls int `json:"tool_calls"`
	// ExitCode of the pi process, -1 when it did not exit cleanly.
	ExitCode int    `json:"exit_code"`
	Output   string `json:"output,omitempty"`
}

ScenarioResult is the graded outcome of one scenario run.

func EvaluateScenario added in v0.0.77

func EvaluateScenario(s Scenario, workDir string, loaded []*LoadedTrajectory) ScenarioResult

EvaluateScenario grades a finished run: every target tool called with at least one non-error result, and every Check satisfied. workDir is the scenario's workspace after the run; loaded are the trajectories the run wrote (root session plus any subagents).

type SessionSummary

type SessionSummary struct {
	SessionID     string `json:"session_id"`
	AgentName     string `json:"agent_name"`
	Model         string `json:"model"`
	Steps         int    `json:"steps"`
	ToolCalls     int    `json:"tool_calls"`
	SubagentRefs  int    `json:"subagent_refs"`
	Depth         int    `json:"depth"`
	StartedAt     string `json:"started_at,omitempty"`
	Duration      string `json:"duration,omitempty"`
	ContinuedFrom string `json:"continued_from,omitempty"`
}

SessionSummary is a one-line summary of a single trajectory.

type SessionTokenUsage

type SessionTokenUsage struct {
	SessionID        string `json:"session_id"`
	PromptTokens     int    `json:"prompt_tokens"`
	CompletionTokens int    `json:"completion_tokens"`
	CachedTokens     int    `json:"cached_tokens"`
}

SessionTokenUsage is the token usage attributed to one session.

type TokenMetrics

type TokenMetrics struct {
	PromptTokens     int                 `json:"prompt_tokens"`
	CompletionTokens int                 `json:"completion_tokens"`
	CachedTokens     int                 `json:"cached_tokens"`
	TotalTokens      int                 `json:"total_tokens"`
	CostUSD          float64             `json:"cost_usd,omitempty"`
	Sessions         []SessionTokenUsage `json:"sessions,omitempty"`
}

TokenMetrics aggregates provider-reported token usage across every session in the run. Usage is read from each session's events.jsonl because the ATIF trajectory does not carry it.

func ComputeTokenMetrics

func ComputeTokenMetrics(loaded []*LoadedTrajectory) TokenMetrics

ComputeTokenMetrics sums provider-reported token usage across every session in the run, attributing each session's usage to its trajectory.

type ToolCoverage added in v0.0.77

type ToolCoverage struct {
	Name     string `json:"name"`
	Group    string `json:"group"`
	Requires string `json:"requires,omitempty"`
	// Scenarios that target this tool (may be empty for excluded tools).
	Scenarios []string `json:"scenarios,omitempty"`
	// Excluded carries the exclusion reason when the tool is excluded.
	Excluded string `json:"excluded,omitempty"`
	Calls    int    `json:"calls"`
	Results  int    `json:"results"`
	Errors   int    `json:"errors"`
	Wasted   int    `json:"wasted"`
	Status   string `json:"status"`
}

ToolCoverage is one row of the coverage matrix.

type ToolInfo added in v0.0.77

type ToolInfo struct {
	// Name is the function name the model calls — exactly what lands in
	// ATIF tool_calls[].function_name.
	Name string `json:"name"`
	// Group is the constructor family the tool comes from (core, bash-control,
	// subagent, memory, lsp, llms, a2a, palace, provider).
	Group string `json:"group"`
	// Requires names the capability the tool is gated on in a real session:
	// "" when it is always registered in print mode, otherwise the config or
	// environment a session needs before the tool is advertised at all.
	Requires string `json:"requires,omitempty"`
}

ToolInfo describes one tool the pi agent can expose to the model.

func Inventory added in v0.0.77

func Inventory(dir string) ([]ToolInfo, error)

Inventory enumerates every tool the pi agent can register, by constructing them through the same constructors the CLI uses (tools.CoreTools, tools.BashControlTools, tools.SubagentTools, tools.MemoryTools, tools.LSPToolsFor, tools.LLMSTools, tools.A2ATools, palace.PalaceTools and the Gemini grounding tool). It is the source of truth for "which tools exist": a tool added to any of those constructors shows up here without anyone editing a list, which is what lets the coverage check catch a new tool that has no eval scenario.

Tools are constructed against a throwaway sandbox rooted at dir and never run, so the call is cheap and side-effect free. Optional groups are built with an empty configuration (no LSP server, no llms sources, no A2A agents, an in-memory observation store, a zero palace): the tool objects exist regardless of whether the capability behind them would work, which is exactly what is needed to learn their names.

type ToolOutcome added in v0.0.77

type ToolOutcome struct {
	Tool    string `json:"tool"`
	Calls   int    `json:"calls"`
	Results int    `json:"results"`
	Errors  int    `json:"errors"`
	Wasted  int    `json:"wasted"`
	OK      bool   `json:"ok"`
}

ToolOutcome is what the trajectories show for one target tool.

type ToolStats

type ToolStats struct {
	Calls          int `json:"calls"`
	Results        int `json:"results"`
	Errors         int `json:"errors"`
	Wasted         int `json:"wasted"`
	Duplicates     int `json:"duplicates"`
	AvgResultBytes int `json:"avg_result_bytes"`
	AvgLatencyMs   int `json:"avg_latency_ms"`
}

ToolStats is the per-tool efficiency summary.

  • wasted: a tool_call with no matching observation result. The model asked for a result that never arrived — either the call was interrupted or the provider stream ended before the observation was recorded.
  • duplicates: occurrences of an identical (function_name, arguments) pair after the first. Repeating the exact same call suggests the model was not paying attention to the result it already had.
  • errors: results whose content looks like a failure (heuristic — see errorMarkerRE). A "failed" bash call still produced a result, so it counts as a result AND an error.

type ToolsMetrics

type ToolsMetrics struct {
	TotalCalls       int                  `json:"total_calls"`
	TotalResults     int                  `json:"total_results"`
	Wasted           int                  `json:"wasted"`
	Duplicates       int                  `json:"duplicates"`
	NestedAgentCalls int                  `json:"nested_agent_calls"`
	ByTool           map[string]ToolStats `json:"by_tool"`
}

ToolsMetrics aggregates tool usage across every trajectory in the run.

func ComputeToolsMetrics

func ComputeToolsMetrics(loaded []*LoadedTrajectory) ToolsMetrics

ComputeToolsMetrics derives per-tool efficiency from the run's trajectories.

type ToolsReport added in v0.0.77

type ToolsReport struct {
	Metadata  ToolsReportMetadata `json:"metadata"`
	Scenarios []ScenarioResult    `json:"scenarios"`
	Coverage  Coverage            `json:"coverage"`
	Tools     ToolsMetrics        `json:"tools"`
	Tokens    TokenMetrics        `json:"tokens"`
	// Judge is the LLM grader's advisory verdict, absent when no judge ran.
	Judge *JudgeVerdict `json:"judge,omitempty"`
}

ToolsReport is the machine-readable output of one tool-coverage eval run (make eval-tools): per-scenario verdicts, the coverage matrix against the registered tool inventory, and the usual tool/token metrics over every trajectory the scenarios produced.

func (*ToolsReport) Tally added in v0.0.77

func (r *ToolsReport) Tally()

Tally counts the scenario statuses into the metadata.

type ToolsReportMetadata added in v0.0.77

type ToolsReportMetadata struct {
	Model     string    `json:"model"`
	Binary    string    `json:"binary"`
	GitHead   string    `json:"git_head"`
	Timestamp time.Time `json:"timestamp"`
	Duration  string    `json:"duration"`
	// Selected is the scenario filter in effect (PI_EVAL_SCENARIO), empty for
	// the whole suite. A filtered run's coverage gap is not meaningful.
	Selected string `json:"selected,omitempty"`
	Passed   int    `json:"passed"`
	Failed   int    `json:"failed"`
	Skipped  int    `json:"skipped"`
	Errored  int    `json:"errored"`
}

ToolsReportMetadata describes what was run and when.

type TrajectoryMetrics

type TrajectoryMetrics struct {
	Sessions         []SessionSummary `json:"sessions"`
	TotalSteps       int              `json:"total_steps"`
	TotalToolCalls   int              `json:"total_tool_calls"`
	NestedAgentCalls int              `json:"nested_agent_calls"`
	MaxDepth         int              `json:"max_depth"`
}

TrajectoryMetrics summarizes the trajectories written during the run, including nested subagent sessions and their nesting depth.

func ComputeTrajectoryMetrics

func ComputeTrajectoryMetrics(loaded []*LoadedTrajectory) TrajectoryMetrics

ComputeTrajectoryMetrics builds the trajectory summary, computing nesting depth by following subagent_trajectory_ref links.

Directories

Path Synopsis
Package scenarios is the tool-coverage eval suite for the pi coding agent: one declarative scenario per tool family, each seeding a workspace and asking for something a competent agent does with specific tools, plus deterministic assertions over the trajectory and filesystem.
Package scenarios is the tool-coverage eval suite for the pi coding agent: one declarative scenario per tool family, each seeding a workspace and asking for something a competent agent does with specific tools, plus deterministic assertions over the trajectory and filesystem.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL