README
¶
Agent Arena
Agent Arena is the evaluation layer for go-agent. It separates agent execution from evaluation so the same tasks can benchmark different agents, models, swarms, or remote runtimes.
Quick start
package main
import (
"context"
"fmt"
"github.com/Protocol-Lattice/go-agent/arena"
)
func main() {
runner := myRunner{}
tasks := []arena.Task{
{
Name: "capital",
Input: "What is the capital of France?",
Evaluator: arena.ContainsEvaluator{Required: []string{"Paris"}},
},
}
results := (&arena.Arena{Runner: runner}).RunAll(context.Background(), tasks, 4)
fmt.Printf("%+v\n", arena.Summarize(results))
}
Compare agents
suite := arena.RunSuite(ctx, tasks, []arena.Competitor{
{Name: "agent-a", Runner: runnerA},
{Name: "agent-b", Runner: runnerB},
}, 4)
for _, entry := range arena.RankSuite(suite) {
fmt.Printf("%s: %.2f\n", entry.Name, entry.Summary.AverageScore)
}
Native go-agent
Use arena.AgentRunner to benchmark a normal *agent.Agent:
runner := arena.AgentRunner{Agent: myAgent}
Each task gets an isolated arena:<task-name> session unless SessionID is explicitly supplied.
Built-in evaluators
ExactEvaluator— normalized exact output match.ContainsEvaluator— all required fragments must be present; partial scores are supported.FuncEvaluator— custom boolean/score evaluation.ScoreEvaluator— custom score function with automatic clamping to[0, 1].
The result model also tracks duration, tokens, tool calls, retries, cost, feedback, and metadata when the runner provides those metrics.
Documentation
¶
Overview ¶
Package arena provides a small, deterministic evaluation harness for go-agent.
Arena separates task execution from evaluation so the same task suite can be used with different agents, models, or runners. It also records enough execution metadata to compare correctness, latency, failures, retries, tokens, and cost.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type AgentRunner ¶
AgentRunner adapts the native go-agent Agent to the Arena Runner interface. SessionID is used to isolate memory between benchmark tasks.
type Arena ¶
type Arena struct {
Runner Runner
// CostPerInputToken and CostPerOutputToken are optional USD rates.
CostPerInputToken float64
CostPerOutputToken float64
}
Arena executes tasks and aggregates their results.
type Competitor ¶
Competitor names a runner participating in the same task suite.
type ContainsEvaluator ¶
type ContainsEvaluator struct {
Required []string
}
ContainsEvaluator passes when all required fragments occur in the output.
func (ContainsEvaluator) Evaluate ¶
func (e ContainsEvaluator) Evaluate(_ context.Context, _ Task, output RunOutput) Evaluation
type Evaluation ¶
Evaluation is the result returned by an Evaluator.
type Evaluator ¶
type Evaluator interface {
Evaluate(context.Context, Task, RunOutput) Evaluation
}
Evaluator evaluates a completed task run. Score must be in [0, 1].
type ExactEvaluator ¶
type ExactEvaluator struct {
Expected string
}
ExactEvaluator passes when the normalized output exactly matches Expected.
func (ExactEvaluator) Evaluate ¶
func (e ExactEvaluator) Evaluate(_ context.Context, _ Task, output RunOutput) Evaluation
type FuncEvaluator ¶
type FuncEvaluator func(context.Context, Task, RunOutput) Evaluation
FuncEvaluator adapts a function into an Evaluator.
func (FuncEvaluator) Evaluate ¶
func (f FuncEvaluator) Evaluate(ctx context.Context, task Task, output RunOutput) Evaluation
type LeaderboardEntry ¶
LeaderboardEntry is a comparable aggregate for one named runner.
func Rank ¶
func Rank(entries []LeaderboardEntry) []LeaderboardEntry
Rank returns entries sorted by score descending, then success rate, duration, and name. Sorting is deterministic.
func RankSuite ¶
func RankSuite(results []SuiteResult) []LeaderboardEntry
RankSuite converts suite results into the same deterministic ordering used by Rank.
type Result ¶
type Result struct {
TaskName string
Output string
Success bool
Score float64
Duration time.Duration
InputTokens int
OutputTokens int
ToolCalls int
Retries int
Cost float64
Error error
Feedback []string
Metadata map[string]string
}
Result contains execution and evaluation information for one task.
type RunOutput ¶
type RunOutput struct {
Output string
InputTokens int
OutputTokens int
ToolCalls int
Retries int
Metadata map[string]string
}
RunOutput is the observable result of executing a task.
type Runner ¶
Runner executes an arena task. Implementations may wrap agent.Agent, an HTTP-hosted agent, a swarm, or any other runtime.
type ScoreEvaluator ¶
ScoreEvaluator converts a scoring function into an Evaluator. The function returns a score in [0, 1] and optional feedback.
func (ScoreEvaluator) Evaluate ¶
func (f ScoreEvaluator) Evaluate(ctx context.Context, task Task, output RunOutput) Evaluation
type SuiteResult ¶
SuiteResult contains per-task results and an aggregate leaderboard entry.
func RunSuite ¶
func RunSuite(ctx context.Context, tasks []Task, competitors []Competitor, concurrency int) []SuiteResult
RunSuite runs the same task set against multiple competitors and returns deterministic leaderboard-ready results. Each competitor gets an isolated Arena instance, while task definitions remain shared.
Source Files