agenteval

package
v0.6.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 10, 2026 License: MIT Imports: 10 Imported by: 0

README

Agent eval for the aerolvm MCP tools

Checks that Claude can actually use the tools aerolvm mcp offers. A unit test proves a tool works when called correctly; this proves a model calls it correctly from its name, description and notices alone.

Run it before you change a tool name, a tool description or a notice's wording, and before a release. It calls the Anthropic API, so it is never part of make test or CI.

Tasks

Each task checks the sandbox afterwards, not only what the model answered.

  1. Create and exec: create a named sandbox, run a command, report the output.
  2. Write then read: write a file in a pinned sandbox and read it back.
  3. Dev server: start a server that keeps running with start_process, publish it with expose_port, and fetch it. Without a reachable URL (a local install with no domain), the harness checks the server from inside the sandbox.
  4. Recover after a recreate: the harness destroys the pinned sandbox between two turns. The next tool result must carry the recreate notice, and the model must say so and restore the file.
  5. Exec on WASM: WASM has no streaming exec. Skipped unless AEROLVM_EVAL_WASM_IMAGE names a module the sandboxd can run.

Run

Point it at a local single-node sandboxd (Local Setup):

export SB_API_URL=http://127.0.0.1:21212 SB_PAT_TOKEN=<token> ANTHROPIC_API_KEY=<key>
go test -tags=agenteval -count=1 -v ./integration-tests/agenteval/
Variable Default Meaning
AEROLVM_EVAL_MODEL claude-opus-5-5 Model to evaluate
AEROLVM_EVAL_IMAGE python:3.12-alpine Image for tasks 1-4
AEROLVM_EVAL_WASM_IMAGE unset Module ref for task 5
AEROLVM_EVAL_WASM_COMMAND echo hello-wasm Command for task 5
AEROLVM_EVAL_WASM_EXPECT hello-wasm Text task 5's answer must contain

The harness builds aerolvm from this checkout, so it grades the tool text you are about to ship. -v prints every tool call and answer. Read the transcripts as well as the verdict: a task that passed after five failed calls means a description needs work.

loop_test.go checks the agent loop itself offline, against a scripted model and the fake sandboxd, and does run in make test.

Documentation

Overview

Package agenteval checks that a model can actually use the aerolvm MCP tools (plans/mcp-server-and-agent-cli.md §8, eng review D13). Claude gets the tools exactly as `aerolvm mcp` lists them and works through a task; the harness then checks the sandbox, not just the answer.

The live tasks are in agenteval_test.go behind -tags=agenteval. An operator runs them against a local single-node sandboxd before changing a tool name, description or notice text, and before a release. They are never part of `make test` or CI: they call the Anthropic API. This file is the agent loop they share, kept free of the tag so loop_test.go checks it offline.

Index

Constants

View Source
const DefaultModel = "claude-opus-5-5"

DefaultModel is the model the eval runs against unless AEROLVM_EVAL_MODEL says otherwise.

Variables

This section is empty.

Functions

This section is empty.

Types

type Anthropic

type Anthropic struct {
	APIKey     string
	Model      string
	URL        string // defaults to the public Messages API
	HTTPClient *http.Client
	MaxTokens  int
	// Backoff is the first retry delay after a 429 or 5xx; it doubles.
	Backoff time.Duration
}

Anthropic is a minimal Messages API client: the eval needs one endpoint, and a raw request keeps the SDK out of go.mod.

type Conversation

type Conversation struct {

	// MaxTurns bounds the model calls per Ask, so a model that loops on a
	// failing tool ends the task instead of spending without limit.
	MaxTurns int
	// Steps is every tool call, across all Ask calls, in order.
	Steps []Step
	// contains filtered or unexported fields
}

Conversation is one agent session: a model, the MCP tools, and the message history across Ask calls.

func NewConversation

func NewConversation(ctx context.Context, model *Anthropic, session *mcp.ClientSession, system string) (*Conversation, error)

NewConversation lists the session's tools and hands them to the model as they are: same names, descriptions and schemas a real MCP client sends.

func (*Conversation) Ask

func (c *Conversation) Ask(ctx context.Context, prompt string) (string, error)

Ask sends prompt as the user and runs the model's tool calls until it answers without one. It returns that answer.

func (*Conversation) Called

func (c *Conversation) Called(tool string) bool

Called reports whether the model called tool at least once.

func (*Conversation) Summary

func (c *Conversation) Summary() string

Summary is a one-line-per-call log of the session, for the test output.

type Step

type Step struct {
	Tool       string
	Args       map[string]any
	IsError    bool
	Text       string         // the result as the model saw it
	Structured map[string]any // the tool's structured output, for checks
}

Step is one tool call the model made and what it got back.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL