Documentation
¶
Overview ¶
Package eval provides FakeModel-scripted harness evaluation scenarios.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func TextScript ¶
func TextScript(content string) agentllm.ResponseScript
TextScript builds a FakeModel script that returns plain assistant text.
func ToolCallScript ¶
func ToolCallScript(id, name, argsJSON string) agentllm.ResponseScript
ToolCallScript builds a FakeModel script that requests one tool call.
Types ¶
type Expectation ¶
type Expectation struct {
// ExpectedTools lists tool names that must appear in order among executed tools.
ExpectedTools []string
// RequiredArgs maps tool name to required argument keys that must be non-empty.
RequiredArgs map[string][]string
// MaxSteps fails the scenario when the loop exceeds this step count (0 disables).
MaxSteps int
// RequireCompletion requires a non-error final assistant text response.
RequireCompletion bool
}
Expectation describes success criteria for one eval scenario.
type Metrics ¶
type Metrics struct {
// ToolSelectionCorrect is true when expected tools were selected in order.
ToolSelectionCorrect bool
// ArgsValid is true when required tool arguments were present and non-empty.
ArgsValid bool
// StepCount is the number of assistant turns observed.
StepCount int
// Completed is true when the run finished with a final assistant message and no error.
Completed bool
// ToolsCalled lists tool names executed during the run.
ToolsCalled []string
}
Metrics captures scored outcomes for one scenario run.
func Score ¶
func Score(messages []msg.AgentMessage, expect Expectation, runErr error) Metrics
Score derives metrics from a completed agent run.
type Scenario ¶
type Scenario struct {
// Name identifies the scenario in table tests.
Name string
// Prompt is the user message.
Prompt string
// Scripts are FakeModel responses in order.
Scripts []agentllm.ResponseScript
// Tools are registered for the run.
Tools []tool.Tool
// Expect defines scoring criteria.
Expect Expectation
}
Scenario is one FakeModel-driven harness evaluation case.
Click to show internal directories.
Click to hide internal directories.