token-bench

Overview ¶

Command token-bench measures mache's token-efficiency claim: how many bytes reach an agent's context to answer a construct-level question, comparing file-granular retrieval (arm A, the built-in Read tool) against construct-granular retrieval (arm B, mache find_definition / read_file).

Bead mache-544659. This is the BASELINE half of the falsification design — it must run BEFORE any projection change lands (mache-qzsk), or there is nothing to compare a post-change number against.

What this tool measures deterministically:

F3  tokens-to-answer  — arm A vs arm B bytes for the same question
F4  generality        — the same ratio broken out per language/extension

What it deliberately does NOT measure, because both need a live agent loop and this tool must stay deterministic and CI-runnable:

F1  escalation rate   — how often a construct-granular answer is followed
                        by a full Read of the same file. Measured instead
                        from session transcripts; see the bead.
F2  completion rate   — held as a CONTROL. A token reduction is meaningless
                        unless task success is unchanged; entire-graph's
                        published comparison holds exactly this constant.

Reporting an arm-B win without F1 and F2 would be the failure this bench exists to prevent: returning less is trivially achievable by returning less.