Documentation
¶
Overview ¶
Diagnostic track (-track diagnostic): v5 task set with dimensions, prompt levels, expected behaviors, and counterfactual oracles. See docs/design/diagnostic-evaluation.md.
Command runner executes the Agent Eval regression track: a frozen task set (eval/regression/tasks.yaml) against a deterministic fixture, driven by any OpenAI-compatible chat-completions endpoint with tool calling. It scores tasks mechanically (see eval/README.md) and prints a JSON report to stdout.
Configuration (environment):
EVAL_API_KEY required; bearer token for the model endpoint EVAL_MODEL required; model name EVAL_BASE_URL optional; defaults to https://api.openai.com/v1
The workload track (-track workload) runs the fixtures/v4 realistic business workload (docs/design/business-workload-model.md): complex multi-module tasks graded on two channels — a mechanical answer check whose key values come from the fixture generator (single source of truth), and an evidence channel that scans tool results for the expected result rows and attributes failures to one of the roadmap's ten failure categories.
Additional environment (dogfooding mode):
EVAL_DSN optional; external database DSN — skips the testcontainer
and fixture seeding (bring your own data)
EVAL_CONFIG optional; server config path (default the v4 profile)
EVAL_TASKS optional; task file path (default fixtures/v4/tasks/tasks.yaml)
Source Files