evals

Overview ¶

The evals are behind a build tag so an ordinary `go build ./...` never reaches for a model. tasks.go carries no tag, so `go test ./scripts/evals` walks every prompt without one.

Command evals scores whether a model can do a task through this server's tools alone, reading only the in-memory instance of internal/gapi/gitlabtest (CLAUDE.md rule 1, docs/architecture.md §13).

The live driver proves the tools work. This asks whether a model reading nothing but the tool descriptions gets the right answer out of them, and whether content written to steer it does. It is run by hand: it spends money and is not deterministic. -self-check runs everything but the model, with no key, and is in `make check`: every task must FAIL on a run that did nothing, or its scorer scores nothing.

The task table lives in this file and carries no build tag, so `go test ./scripts/evals` walks every prompt without a model, a key or an instance.