Documentation
¶
Overview ¶
Package eval is the entry point of the offline eval harness: it loads scenario files, validates them (dry-run), runs each through its workflow driver, judges the produced artifact against the scenario checklist, dumps diagnostics for failing scenarios, and renders the aggregated report.
The final OK/NG decision is left to a human reviewer: the harness never gates on check verdicts. The process exits non-zero only on execution errors.
Index ¶
Constants ¶
View Source
const ( ExitOK = 0 ExitError = 2 )
Exit codes.
Variables ¶
This section is empty.
Functions ¶
func Run ¶
Run executes the harness over the given paths and writes the summary to stdout. It returns the process exit code.
func ToolCatalog ¶
func ToolCatalog() []toolCatalogEntry
ToolCatalog is the single source of truth for valid tool names (FR-16). The scenario validator and `--list-tools` both read from here.
Types ¶
type Config ¶
type Config struct {
// LLM is shared by all roles (agent, judge, simulators) — FR-7.
LLM gollem.LLMClient
Concurrency int
DryRun bool
DumpDir string
DumpAll bool
Language string // output language for judge reasons / analysis.md (FR-14)
ReportPath string
Quiet bool
Verbose bool
// Live tool clients, used only for tools marked live=true.
LiveSlackSearch slacktool.SearchService
LiveNotion notiontool.Client
GitHub *githubtool.Client
// WebFetch carries the HTTP-side settings for the live-only webfetch tool;
// the eval env injects the eval LLM client before the tool is built.
WebFetch *webfetch.ClientConfig
// JiraTools carries the already-expanded Jira read tools (see
// pkg/agent/tool/jira). Live-only, like GitHub: nil/empty means jira_search
// scenarios cannot run live.
JiraTools []gollem.Tool
}
Config carries the runner inputs assembled by the CLI.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package driver runs a workflow type for one scenario against a prepared env and returns the artifact to judge.
|
Package driver runs a workflow type for one scenario against a prepared env and returns the artifact to judge. |
|
Package env builds the in-memory environment a scenario runs against: a memory repository (seeded with the scenario's prior cases), in-memory history and trace repositories, the system-under-test wired via usecase.New, a recording fake Slack service, and per-tool clients that are either simulated (ToolSimulator) or live (recorded).
|
Package env builds the in-memory environment a scenario runs against: a memory repository (seeded with the scenario's prior cases), in-memory history and trace repositories, the system-under-test wired via usecase.New, a recording fake Slack service, and per-tool clients that are either simulated (ToolSimulator) or live (recorded). |
|
Package evaltype holds the shared value types and small interfaces of the eval harness.
|
Package evaltype holds the shared value types and small interfaces of the eval harness. |
|
Package judge evaluates a produced artifact against a scenario's checklist.
|
Package judge evaluates a produced artifact against a scenario's checklist. |
|
Package llmrun provides the gollem-backed implementation of evaltype.Completer.
|
Package llmrun provides the gollem-backed implementation of evaltype.Completer. |
|
Package report aggregates scenario results and renders them: a human-readable stdout summary, a machine-readable JSON file, and per-scenario diagnostic dumps (see dump.go).
|
Package report aggregates scenario results and renders them: a human-readable stdout summary, a machine-readable JSON file, and per-scenario diagnostic dumps (see dump.go). |
|
Package scenario defines the TOML schema for eval scenarios and loads / validates them.
|
Package scenario defines the TOML schema for eval scenarios and loads / validates them. |
|
Package toolsim provides simulated implementations of the agent's tool client interfaces.
|
Package toolsim provides simulated implementations of the agent's tool client interfaces. |
|
Package usersim implements evaltype.Simulator: a simulated end-user that answers the agent's clarification questions, drawing on the scenario persona (who they are + what they know).
|
Package usersim implements evaltype.Simulator: a simulated end-user that answers the agent's clarification questions, drawing on the scenario persona (who they are + what they know). |
Click to show internal directories.
Click to hide internal directories.