eval-harness/

Eval harness

Runs a single agent execution with mock LLM and mock tools. Prints JSON to stdout with content, llm_usage, and telemetry for evaluation assertions.

Runner

From the repo root:

go run ./eval-harness/runner
go run ./eval-harness/runner -prompt "custom prompt"
go run ./eval-harness/runner -runtime temporal
go run ./eval-harness/runner -tools 2
go run ./eval-harness/runner -config eval-harness/runner/config.yaml

Arguments

config.yaml

Default path: eval-harness/runner/config.yaml

Memory (same file; enabled: false for default tool tests):

./eval-harness/run_agent_memory.sh ondemand   # same config.yaml, -memory flag
./eval-harness/run_agent_memory.sh always

Temporal mode uses an embedded local worker.

Output

Stdout is always JSON:

{
  "content": "eval complete",
  "llm_usage": { "prompt_tokens": 600, "completion_tokens": 400, "total_tokens": 1000 },
  "telemetry": { "run": { ... }, "tools": { ... }, "storage": { ... } },
  "memory_scenario": {
    "store": { "content": "...", "telemetry": { ... } },
    "recall": { "content": "...", "telemetry": { ... } }
  }
}

memory_scenario is present only for memory.scenario: store_recall.

PromptFoo

Config: eval-harness/promptfoo/config.yaml

PromptFoo runs the eval harness as an exec provider. Each test invokes the runner once, parses the JSON stdout, and asserts on content, llm_usage, and telemetry.

Run

cd eval-harness/promptfoo
npx promptfoo eval -c config.yaml

View results in the web UI:

npx promptfoo view

Requires Node.js. PromptFoo is installed on demand via npx; no local install is required.

How it works

The runner accepts PromptFoo’s prompt as a positional argument when -prompt is not set. Agent settings (tool_count, runtime, etc.) still come from eval-harness/runner/config.yaml.

Tests

Six test cases in config.yaml (each scoped to one provider — Promptfoo reports 6 runs, not test×provider duplicates):

Customizing

• Change the prompt — edit prompts in promptfoo/config.yaml, or add vars and use {{var}} in the prompt string.
• Change agent behavior — edit eval-harness/runner/config.yaml (tool count, runtime, system prompt), or adjust eval-harness/run_agent.sh.
• Add tests — append cases under tests: with type: javascript and value: returning a boolean.
• Filter providers — set providers: [label] on each test so agent checks do not run on memory providers (and vice versa).

DeepEval

Python tests in eval-harness/deepeval/. The suite runs the Go eval harness, parses the JSON stdout, and asserts on content, llm_usage, and telemetry — the same output contract as the runner and PromptFoo.

Run

cd eval-harness/deepeval
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
pytest test_agent.py -v

Requires Python 3.10+ and Go. No API key is required for the default tests.

How it works

1. harness.run_agent() calls eval-harness/run_agent.sh and parses JSON.
2. Tests read telemetry from the agent SDK run output.
3. assert_test() runs DeepEval metrics where useful; plain pytest asserts cover the rest.

Example — extract tools from telemetry:

agent_res = run_agent()
tools = list(agent_res["telemetry"]["tools"]["breakdown"].keys())
finish_reason = agent_res["telemetry"]["run"]["finish_reason"]

Tests

Four pytest tests in test_agent.py:

Customizing

• Change the prompt — pass a different string to run_agent(prompt=...).
• Change agent behavior — edit eval-harness/runner/config.yaml or eval-harness/run_agent.sh.
• Add tests — extend test_agent.py with more telemetry asserts or DeepEval LLMTestCase fields.

Note: CI runs both PromptFoo and DeepEval on PRs — see .github/workflows/ci.yml (eval-harness job). Locally: make eval-harness from the repo root.