Documentation
¶
Overview ¶
Package eval scores completed task output against a configurable rubric by calling the AI provider once per criterion and aggregating weighted scores. Results are stored as JSON under .cloop/evals/<task-id>.json.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Criterion ¶
type Criterion struct {
Name string `yaml:"name" json:"name"`
Weight float64 `yaml:"weight" json:"weight"`
Description string `yaml:"description" json:"description"`
}
Criterion is a single evaluation dimension with a weight and description.
type EvalResult ¶
type EvalResult struct {
TaskID int `json:"task_id"`
TaskTitle string `json:"task_title"`
Scores []Score `json:"scores"`
// Weighted is the overall weighted average (1-10).
Weighted float64 `json:"weighted_average"`
EvaluatedAt time.Time `json:"evaluated_at"`
}
EvalResult is the full evaluation output for one task.
func Evaluate ¶
func Evaluate(ctx context.Context, prov provider.Provider, model string, timeout time.Duration, workDir string, task *pm.Task, output string, rubric Rubric) (*EvalResult, error)
Evaluate calls the AI provider to score each criterion in the rubric for the given task and its output. Returns a populated EvalResult and saves it to .cloop/evals/<task-id>.json under workDir.
Two stability bounds protect long-running eval runs:
- ctx.Err() is checked between criteria so a cancelled parent (Ctrl+C translated to cancel, or an enclosing watchdog firing) bails before the next per-criterion call instead of running the full rubric.
- Each scoreOneCriterion call is wrapped in panic recovery so a panic inside a provider implementation (e.g. nil-pointer in a third-party SDK) becomes a returned error instead of crashing `cloop eval` and losing every score that completed earlier in the loop.
func Load ¶
func Load(workDir string, taskID int) (*EvalResult, error)
Load reads a previously saved EvalResult for the given task ID. Returns nil, nil if no result exists yet.
A torn or hand-edited eval file used to abort `cloop eval show`, the auto- promote heuristic, and any UI handler that lists per-task scores. The bad bytes are now quarantined to a sibling .corrupt-* file and Load returns (nil, nil) so the caller treats the task as un-evaluated. The next eval run re-creates the file cleanly.
Source Files