Documentation
¶
Overview ¶
Command exploreprobe scores the served retrieval against SWE-Explore, the first external benchmark that grades the WHOLE pipeline rather than its entrance.
CORE-Bench already covers the seed stage — see BENCHMARK.md — but it feeds its own chunked corpus, so PageRank, the intent ranker, the evidence windows and the packer are downstream of anything it can see. Its labels also come from git diffs: what a developer CHANGED. SWE-Explore labels what strong agents READ before they succeeded, intersected across runs and audited by hand, which is nearer to what this tool is for — we do not predict the patch, we hand the agent what it has to read first.
The benchmark's own framing is that "sparse retrievers, interactive agents, and long-context selectors are all compared as producers of the same ranked region list", so a standalone search tool plugs in directly: every result is a (file, start, end) tuple, which is exactly a find_context symbol.
Data (CC-BY-NC-ND: run it and publish numbers, do not redistribute):
huggingface.co/datasets/SWE-Explore-Bench/SWE-Explore-Bench
Usage:
exploreprobe -manifest manifest.jsonl -repos repos [-n 30] [-dataset verified]
Source Files