Documentation
¶
Overview ¶
The evals are behind a build tag so an ordinary `go build ./...` never reaches for a model. tasks.go carries no tag, so `go test ./scripts/evals` walks every prompt without one.
Command evals scores whether a model can do a task through this server's tools alone, reading only the in-memory instance of internal/gapi/gitlabtest, never a real one.
The live driver proves the tools work. This asks whether a model reading nothing but the tool descriptions gets the right answer out of them, and whether content written to steer it does. It is run by hand: it spends money and is not deterministic. -self-check runs everything but the model, with no key, and is in `make check`: every task must FAIL on a run that did nothing, or its scorer scores nothing.
The task table lives in this file and carries no build tag, so `go test ./scripts/evals` walks every prompt without a model, a key or an instance.
Source Files