Help us shape the future of Go! Take the 2026 Go Developer Survey

eval

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 27, 2026 License: Apache-2.0 Imports: 6 Imported by: 0

Documentation

Overview

Package eval implements the agent eval harness (design 009): "unit tests for agents." A fixture describes a task and structural assertions; a Result is what running the agent produced; Evaluate checks the Result against the assertions. The format is open and in-repo.

This package is the fixture format + the pure, deterministic assertion engine (no token spend). Live execution that PRODUCES a Result by running the agent (`rysh run`) is the integration half; the structural mode here runs against a recorded/produced Result artifact.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Passed

func Passed(as []Assertion) bool

Passed reports whether all assertions passed.

Types

type Assertion

type Assertion struct {
	Name   string
	Pass   bool
	Detail string
}

Assertion is one checked rule.

func Evaluate

func Evaluate(e Expect, r Result) []Assertion

Evaluate checks a Result against an Expect, returning one Assertion per rule.

type Case

type Case struct {
	Name   string
	Prompt string
	Until  string
	Expect Expect
	// Judge is the optional LLM-judge rubric (evals/<case>/judge.md, design
	// 009 §3.2). Empty when the case has no judge.md: assertions alone grade
	// the case. When present, the judge runs AFTER assertions and both must
	// pass — the judge is a signal, structural assertions stay the hard gate.
	Judge string
}

Case is a loaded fixture.

func LoadCase

func LoadCase(dir string) (*Case, error)

LoadCase loads evals/<case>/task.md (+ optional frontmatter) and expect.yaml.

type Expect

type Expect struct {
	// FilesChanged: every changed file must match at least one of these globs
	// (scope containment — catches out-of-scope edits). Empty ⇒ not checked.
	FilesChanged []string `yaml:"files_changed"`
	// CommandsAllowed: if non-empty, every executed command must match one.
	CommandsAllowed []string `yaml:"commands_allowed"`
	// CommandsForbidden: no executed command may match any of these.
	CommandsForbidden []string `yaml:"commands_forbidden"`
	// OutputMatches: every regex must match somewhere in the output.
	OutputMatches []string `yaml:"output_matches"`
	// MaxTokens: tokens_used must be ≤ this (0 ⇒ not checked).
	MaxTokens int `yaml:"max_tokens"`
}

Expect is the assertion set for a case (evals/<case>/expect.yaml).

type Result

type Result struct {
	FilesChanged []string `json:"files_changed"`
	Commands     []string `json:"commands"`
	Output       string   `json:"output"`
	TokensUsed   int      `json:"tokens_used"`
}

Result is what an agent run produced (structural facts, not a transcript).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL