Documentation
¶
Overview ¶
Package bench measures what it costs an agent to answer a question with jade's tools versus with ordinary shell and file tools.
What this measures, and what it does not ¶
docs/scope.md §26 asks whether an agent using jade "completes real tasks more efficiently and reliably than shell/file tooling", and §42-43 lists tokens, turns and success rate as the metrics. This harness measures the first of those three, deterministically and without an LLM in the loop: for each scenario it runs both arms and counts the bytes each one puts in front of the agent — a "tokens to answer" metric rather than a search-precision score.
It deliberately does NOT measure turns or success rate. Both require a real agent making real decisions, and faking them with a scripted call sequence would produce a number that looks like evidence while measuring only the script's author. Turn counts are reported as the number of tool calls each arm needs, which is a floor on turns, not a measurement of them.
The honest reading of a result table is therefore: "answering this question costs N tokens of context through jade and M through shell." That is a real, reproducible comparison, and it is the part of docs/scope.md's question that can be answered without a benchmark agent.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func EstimateTokens ¶
EstimateTokens approximates tokens from bytes at the widely used ~4 characters per token. It is an estimate, not a tokenizer: the comparison between two arms is the meaningful output, not either absolute number, and both arms are estimated identically so the ratio holds regardless of the constant's exact value.
Types ¶
type API ¶
type API interface {
Outline(protocol.OutlineRequest) (protocol.InspectResponse, error)
ReadSymbol(protocol.ReadSymbolRequest) (protocol.InspectResponse, error)
ReadRange(protocol.ReadRangeRequest) (protocol.InspectResponse, error)
Search(protocol.SearchRequest) (protocol.SearchResponse, error)
References(protocol.ReferencesRequest) (protocol.ReferencesResponse, error)
WorkspaceTree(protocol.WorkspaceTreeRequest) (protocol.WorkspaceTreeResponse, error)
Changes() protocol.ChangesResponse
}
API is the subset of the internal server the scenarios exercise. Declared as an interface so the benchmark cannot quietly grow a dependency on server internals.
type Arm ¶
Arm is one way of answering a scenario's question. It returns everything the agent would have to read, concatenated, plus how many tool calls it took.
type Result ¶
Result is one scenario measured on both arms.
func Run ¶
Run measures every scenario. A failing arm is recorded rather than aborting the suite: one broken scenario should not hide the rest, and a tool that errors is itself a result worth seeing.
func (Result) TokenRatio ¶
TokenRatio is jade's token cost as a multiple of shell's. Below 1.0 means jade is cheaper. Returns 0 when shell produced nothing to compare against.
type Scenario ¶
type Scenario struct {
Name string
// Question is the agent-level task, phrased as an agent would think of
// it rather than as a tool invocation.
Question string
Jade Arm
Shell Arm
}
Scenario is one question asked of a repository, answered two ways.
func DefaultScenarios ¶
DefaultScenarios are questions an agent actually asks while working in a repository, each answered both ways.
Fairness rules, because a benchmark whose arms answer different questions measures nothing:
- both arms must produce enough for the agent to answer the question;
- the shell arm uses the command a competent agent would actually reach for, not a deliberately clumsy one;
- jade's arm is measured as its response is actually serialized to the agent over MCP (JSON), not as some hypothetical trimmed form.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package agent runs the release plan's external benchmark: a real agent (Claude Code headless) solving real tasks in a repository Jade was not built in, once per arm — shell tools only, Jade only, and Jade plus shell.
|
Package agent runs the release plan's external benchmark: a real agent (Claude Code headless) solving real tasks in a repository Jade was not built in, once per arm — shell tools only, Jade only, and Jade plus shell. |