Documentation
¶
Overview ¶
Package bench measures the project's core bet — token efficiency — the same headline the upstream codebase-memory-mcp reports (10x fewer tokens, 2.1x fewer tool calls on structural questions; see docs/UPSTREAM.md).
For each structural question ("who calls X?") it compares three strategies:
graph — one callers/callees query, returns compact refs (1 tool call) baseline-win — grep X, then read a ±window around each match (1+N calls) baseline-file — grep X, then read each whole matched file (1+N calls)
The graph wins because it has already resolved the enclosing caller of every match and filtered out definitions/imports/homonyms — exactly the work grep dumps onto the agent. The reported number is a RATIO (baseline/graph), so the rough token estimate cancels out. We do NOT measure answer quality here: that needs an LLM-as-judge harness and would be a romantic number, not an engineered one. This file measures only what is deterministic.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func EstimateTokens ¶
EstimateTokens approximates LLM token count as bytes/4 (the classic ~4 chars/token rule). The absolute value is rough, but the same function meters both sides of every comparison, so the RATIO — the only number we report — is stable under the approximation.
Types ¶
type Corpus ¶
type Corpus struct {
// contains filtered or unexported fields
}
Corpus is the repo's source loaded once, reused across every question's grep.
func LoadCorpus ¶
LoadCorpus reads every discoverable source file into memory (the same files the indexer sees), so the baseline's grep is apples-to-apples with the graph.
type Outcome ¶
type Outcome struct {
Question Question
Graph Cost
BaselineWin Cost // grep + read ±window around each match
BaselineFile Cost // grep + read whole file per match
MatchFiles int // distinct files the grep baseline had to open
GraphResults int // nodes the graph returned
}
Outcome is the full three-way comparison for one Question.
type Question ¶
type Question struct {
Kind string // "callers" | "callees"
QN string // qualified name to query
Name string // short symbol name the grep baseline searches for
}
Question is one structural query to benchmark.
func QuestionsFromHubs ¶
QuestionsFromHubs turns call hubs into "who calls X" questions.
type Summary ¶
type Summary struct {
N int
MedianRatioWin float64
MedianRatioFile float64
TotalRatioWin float64
TotalRatioFile float64
CallRatioWin float64 // baseline-win tool-calls / graph tool-calls (total)
GraphTokens int
BaselineWinTokens int
BaselineFileTokens int
GraphCalls int
BaselineWinCalls int
BaselineFileCalls int
}
Summary aggregates outcomes. Two aggregate ratios are reported because they answer different questions: the MEDIAN per-query ratio (typical case, robust to one common-named outlier) and the TOTAL/TOTAL ratio (the "10x" headline: total tokens an agent spends across the whole question set).
Source Files