Documentation
¶
Overview ¶
Command benchmark is a standalone binary for running Tier 1 external benchmarks against a running Synapses daemon. It is purely external — it calls the daemon via the REST HTTP transport (/v1/tools/{tool}?project=...) and does not import any internal Synapses packages.
Usage:
# Local modes (no daemon needed):
benchmark --benchmark=repobench --retrieval=hybrid-rrf --no-synapses
# Full synapses-embed mode (all repos cloned + indexed automatically):
benchmark --benchmark=repobench --retrieval=synapses-embed \
--repos-dir=/tmp/repobench_repos --cache-file=/tmp/index_cache.json
# Index only (pre-flight step):
benchmark --benchmark=repobench --index-only \
--repos-dir=/tmp/repobench_repos --cache-file=/tmp/index_cache.json
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
claude_agent.go implements a minimal agent loop that calls the Anthropic Messages API with tool_use.
|
claude_agent.go implements a minimal agent loop that calls the Anthropic Messages API with tool_use. |
|
ContextBench runner — measures context retrieval quality on the ContextBench dataset (1,136 tasks, 66 repos, 8 languages).
|
ContextBench runner — measures context retrieval quality on the ContextBench dataset (1,136 tasks, 66 repos, 8 languages). |
|
Package indexer clones and indexes GitHub repositories using the Synapses CLI, caching results to disk so re-runs skip already-indexed repos.
|
Package indexer clones and indexes GitHub repositories using the Synapses CLI, caching results to disk so re-runs skip already-indexed repos. |
|
Package reporter writes benchmark results to disk as JSON and Markdown.
|
Package reporter writes benchmark results to disk as JSON and Markdown. |
Click to show internal directories.
Click to hide internal directories.