eval

package
v0.97.8 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 20, 2026 License: GPL-3.0 Imports: 7 Imported by: 0

Documentation

Overview

Package eval provides FakeModel-scripted harness evaluation scenarios.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func TextScript

func TextScript(content string) agentllm.ResponseScript

TextScript builds a FakeModel script that returns plain assistant text.

func ToolCallScript

func ToolCallScript(id, name, argsJSON string) agentllm.ResponseScript

ToolCallScript builds a FakeModel script that requests one tool call.

Types

type Expectation

type Expectation struct {
	// ExpectedTools lists tool names that must appear in order among executed tools.
	ExpectedTools []string
	// RequiredArgs maps tool name to required argument keys that must be non-empty.
	RequiredArgs map[string][]string
	// MaxSteps fails the scenario when the loop exceeds this step count (0 disables).
	MaxSteps int
	// RequireCompletion requires a non-error final assistant text response.
	RequireCompletion bool
}

Expectation describes success criteria for one eval scenario.

type Metrics

type Metrics struct {
	// ToolSelectionCorrect is true when expected tools were selected in order.
	ToolSelectionCorrect bool
	// ArgsValid is true when required tool arguments were present and non-empty.
	ArgsValid bool
	// StepCount is the number of assistant turns observed.
	StepCount int
	// Completed is true when the run finished with a final assistant message and no error.
	Completed bool
	// ToolsCalled lists tool names executed during the run.
	ToolsCalled []string
}

Metrics captures scored outcomes for one scenario run.

func Score

func Score(messages []msg.AgentMessage, expect Expectation, runErr error) Metrics

Score derives metrics from a completed agent run.

type Scenario

type Scenario struct {
	// Name identifies the scenario in table tests.
	Name string
	// Prompt is the user message.
	Prompt string
	// Scripts are FakeModel responses in order.
	Scripts []agentllm.ResponseScript
	// Tools are registered for the run.
	Tools []tool.Tool
	// Expect defines scoring criteria.
	Expect Expectation
}

Scenario is one FakeModel-driven harness evaluation case.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL