evalrun

command
v3.1.0-dev.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 3, 2026 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

evalrun runs one eval case at both tiers and reports (#652).

It is the thin part. Everything worth arguing with lives in internal/evals; this binary parses flags, decides an exit code, and prints something a person can read at 2am.

Exit codes:

0  the tools tier passed and the no-access baseline held
1  a check failed, or the baseline scored, or the harness errored
2  the report is indeterminate — some check observed nothing

2 is separate from 1 on purpose. A failing check says the agent did the wrong thing; an indeterminate one says the harness did not find out what the agent did, and the two send an operator to different places. Both are non-zero: an unanswered question is not a pass.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL