Documentation
¶
Overview ¶
Command wbt-eval is the standalone, provider-backed eval CLI (Phase 6.5). It runs the same deterministic graders internal/evals/*_test.go already exercises through `go test` / `task check`, but through internal/evals.ProviderEvalSuite instead — kept as a separate binary and deliberately NOT wired into build/Taskfile.yml's lint/test/test-integration targets, because future categories will call out to a real LLM provider (TODO Phase 6+ in internal/evals/provider_eval.go), and provider-backed evals MUST NOT run on every `task check` invocation: they cost money and depend on network availability.
Usage:
wbt-eval [--provider=claude] [--model=claude-haiku-4-5] [--category=all]
Exit codes: 0 when every graded case passes, or when ANTHROPIC_API_KEY is unset (see below); 1 when any case fails or a flag/fixture error occurs.
stdout/stderr contract: stdout carries ONLY the JSON result array on a successful (configured) run — nothing else is ever written there. All diagnostics, including the "no API key, skipping" notice, go to stderr so a caller piping stdout through `jq`/`json.Decode` never has to special-case a non-JSON line mixed into the payload.
Source Files