benchmarks

Massive Concurrency & Scale Benchmark

This directory contains a standalone performance utility for the Go Agent SDK. It runs real pkg/agent execution loops under configurable load—mock LLM and tools by default—so you can measure latency, memory, CPU, token counts, and success rate without external API keys.

Use it to stress-test orchestration behavior (multi-turn runs, tool batches, sub-agents, local vs Temporal runtime) before pointing the same harness at real LLMs and tools.

The Core Purpose

Most agent benchmarks focus on token throughput alone. Production workloads also depend on how well the orchestration layer scales: many concurrent runs, multi-turn tool loops, sub-agent delegation, durable Temporal workflows, and stable memory use over hundreds of executions.

This benchmark exercises the SDK’s actual agent engine (agent.NewAgent, Run) with:

• Configurable run count and concurrency
• Mock or (later) real LLM + tool backends
• Optional sub-agent trees
• Local in-process runtime or Temporal with optional external workers
• Structured metrics and reports for comparison across config changes

How It Works

Each run calls agent.Run() once on a shared root agent instance. The mock LLM follows a fixed multi-turn script:

1. Turn 1 — returns tool calls for all registered tools (benchmark tools and sub-agent tools when configured).
2. Turn 2 — returns a final text response after tool results are applied.

Mock components apply configurable latency and jitter so results reflect realistic timing, not instant stubs:

Concurrency: one root agent is reused for all runs. When concurrent: true, runs execute in batches of concurrent_count (goroutines), each batch waiting for the previous batch to finish.

Metrics collected per benchmark session:

• Latency p50 / p95 / p99 / avg (wall-clock per Run())
• Heap and total allocation delta
• Process CPU time
• Total input/output tokens (from mock LLM stats; includes sub-agent LLM calls)
• Success rate (Run() completed without error)
• Long-term memory recalls/stores (when memory.enabled: true; from run telemetry)
• est_cost_usd — placeholder 0 until pricing is configured

Reports are written to benchmarks/reports/ (JSON or text). SDK logs (optional) go to benchmarks/logs/.

Running the Benchmark

Run from the repository root (agent-sdk-go/):

Quick run (default config)

Uses benchmarks/config.yaml (100 sequential runs, local runtime, 3 tools, 2 sub-agents):

go run ./benchmarks/

Custom config file

go run ./benchmarks/ -config benchmarks/config.yaml
go run ./benchmarks/ -config /path/to/my-benchmark.yaml

Command-line flags

All other settings are controlled via the YAML file. Edit benchmarks/config.yaml (or copy it) and re-run with -config.

Example scenarios

Fast local smoke test — reduce runs and latency in a copy of the config:

runtime: local
llm:
  latency_ms: 5
  jitter_ms: 0
tool:
  latency_ms: 2
  jitter_ms: 0
agent:
  runs: 10
  concurrent: false
  tools:
    count: 2
    execution: parallel
  subagents:
    count: 0
    levels: 0

go run ./benchmarks/ -config /tmp/fast-benchmark.yaml

Concurrent batch runs:

agent:
  runs: 100
  concurrent: true
  concurrent_count: 10   # 10 runs in parallel per batch

Temporal runtime — requires a running Temporal server (localhost:7233 by default):

runtime: temporal
temporal:
  host: localhost
  port: 7233
  namespace: default
  task_queue: agent-sdk-go
  workers_count: 0   # embedded worker in agent process only

go run ./benchmarks/ -config benchmarks/config.yaml

External root workers (workers_count: 1+) — benchmark spawns separate worker processes and enables EnableRemoteWorkers() on the root agent. Embedded local workers still run for the root agent and all sub-agents (sub-agents always use embedded workers on their own task queues).

runtime: temporal
temporal:
  workers_count: 2

Workers are started automatically and stopped when the benchmark finishes. You can also run a worker manually:

go run ./benchmarks/worker -config benchmarks/config.yaml -worker-id 1

Debug logging — SDK logs to timestamped files under benchmarks/logs/:

logger:
  enabled: true
  dir: benchmarks/logs
  level: debug    # debug | info | warn | error

Log files: agent_<timestamp>.log, worker_1_<timestamp>.log, …

Configuration reference

All paths in config (dir fields) are relative to the repository root unless absolute.

runtime

temporal

llm

tool

agent

memory

Long-term memory (agent.WithMemory) using an in-process inmem backend (no Docker). Disabled by default.

When memory.enabled: true, agent.tools.count may be 0 (memory-only runs). The mock LLM handles save_memory tool args and memory-extract JSON like the eval harness.

logger

output

Sample output

Text (output.format: text)

=== Benchmark Report ===
Runtime          : local
Concurrent       : false
Total runs       : 100
Tools            : 3 (sequential)
Sub-agents       : 2 (levels 1)
---
Latency p50 (ms) : 245.00
Latency p95 (ms) : 312.00
Latency p99 (ms) : 389.00
Latency avg (ms) : 250.00
Heap alloc (B)   : 12345678
Total alloc (B)  : 98765432
CPU time (ms)    : 1500.00
Input tokens     : 50000
Output tokens    : 33333
Est. cost (USD)  : 0.0000  # pricing placeholder
Success rate (%) : 100.00

JSON (output.format: json)

Written to benchmarks/reports/benchmark_<timestamp>.json:

{
  "runtime": "local",
  "generated_at": "2026-06-06T03:23:33Z",
  "config": {
    "Runtime": "local",
    "Temporal": {
      "Host": "localhost",
      "Port": 7233,
      "Namespace": "default",
      "TaskQueue": "agent-sdk-go",
      "WorkersCount": 0
    },
    "LLM": {
      "LatencyMs": 200,
      "JitterMs": 50,
      "MockTokens": 500
    },
    "Tool": {
      "LatencyMs": 50,
      "JitterMs": 10
    },
    "Agent": {
      "Runs": 100,
      "Concurrent": false,
      "ConcurrentCount": 10,
      "Tools": {
        "Count": 3,
        "Execution": "sequential"
      },
      "Subagents": {
        "Count": 2,
        "Levels": 1
      }
    },
    "Logger": {
      "Enabled": false,
      "Dir": "benchmarks/logs",
      "Level": "info"
    },
    "Output": {
      "Console": true,
      "File": true,
      "Dir": "benchmarks/reports",
      "Format": "json"
    }
  },
  "metrics": {
    "p50_ms": 245,
    "p95_ms": 312,
    "p99_ms": 389,
    "avg_ms": 250,
    "heap_alloc_bytes": 12345678,
    "total_alloc_bytes": 98765432,
    "cpu_time_ms": 1500,
    "total_input_tokens": 50000,
    "total_output_tokens": 33333,
    "est_cost_usd": 0,
    "total_runs": 100,
    "success_rate": 100
  }
}

Note

LLM and tool calls are mocked by default with configurable latency and fixed token counts to keep results reproducible and free of API cost. Latency percentiles, memory, and CPU reflect real SDK orchestration overhead under that simulation.

When you swap in real LLMs and tools, metrics will change: latency follows network and model speed, token counts come from provider usage, and cost requires your own pricing model (the benchmark leaves est_cost_usd at 0 until configured). The harness structure—runs, concurrency, reporting, Temporal workers—stays the same; only the LLM client and tool registry need to be replaced in benchmarks/setup/.