Documentation
¶
Overview ¶
Package agenteval checks that a model can actually use the aerolvm MCP tools (plans/mcp-server-and-agent-cli.md §8, eng review D13). Claude gets the tools exactly as `aerolvm mcp` lists them and works through a task; the harness then checks the sandbox, not just the answer.
The live tasks are in agenteval_test.go behind -tags=agenteval. An operator runs them against a local single-node sandboxd before changing a tool name, description or notice text, and before a release. They are never part of `make test` or CI: they call the Anthropic API. This file is the agent loop they share, kept free of the tag so loop_test.go checks it offline.
Index ¶
Constants ¶
const DefaultModel = "claude-opus-5-5"
DefaultModel is the model the eval runs against unless AEROLVM_EVAL_MODEL says otherwise.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Anthropic ¶
type Anthropic struct {
APIKey string
Model string
URL string // defaults to the public Messages API
HTTPClient *http.Client
MaxTokens int
// Backoff is the first retry delay after a 429 or 5xx; it doubles.
Backoff time.Duration
}
Anthropic is a minimal Messages API client: the eval needs one endpoint, and a raw request keeps the SDK out of go.mod.
type Conversation ¶
type Conversation struct {
// MaxTurns bounds the model calls per Ask, so a model that loops on a
// failing tool ends the task instead of spending without limit.
MaxTurns int
// Steps is every tool call, across all Ask calls, in order.
Steps []Step
// contains filtered or unexported fields
}
Conversation is one agent session: a model, the MCP tools, and the message history across Ask calls.
func NewConversation ¶
func NewConversation(ctx context.Context, model *Anthropic, session *mcp.ClientSession, system string) (*Conversation, error)
NewConversation lists the session's tools and hands them to the model as they are: same names, descriptions and schemas a real MCP client sends.
func (*Conversation) Ask ¶
Ask sends prompt as the user and runs the model's tool calls until it answers without one. It returns that answer.
func (*Conversation) Called ¶
func (c *Conversation) Called(tool string) bool
Called reports whether the model called tool at least once.
func (*Conversation) Summary ¶
func (c *Conversation) Summary() string
Summary is a one-line-per-call log of the session, for the test output.