corpus-mutation-proof

Overview ¶

Command corpus-mutation-proof measures what a corpus actually DETECTS, rather than what it covers.

It exists to settle one question that a coverage number cannot: if a subset of the committed factory corpus is kept, does it still catch every bug the whole corpus caught? Line coverage answers "was this code executed", which is not the same claim — a scenario can execute a line and assert nothing that depends on it.

The keep list is any newline-separated set of scenario names, so the tool is independent of how that set was chosen. That independence is the finding it produced: a greedy set cover over coverage tokens detected no more than a same-size RANDOM sample on any mutation tried, including one reached by only 2.6% of scenarios. The selection tool was deleted; the measurement stands.

METHOD. For each mutation: patch one engine source file, PROVE the patch landed, build, run the FULL corpus once, and record which scenarios failed. The pruned corpus is a strict SUBSET of the full one, so "does the pruned corpus catch this mutation" is answered by intersecting that failure set with the keep list — no second run needed, and the answer is exact rather than an approximation of one.

WHY THE PATCH IS VERIFIED. A mutation that silently fails to apply produces a green run, and a green run is exactly what "the corpus did not catch it" also looks like. Worse, an un-applied mutation makes the corpus look PERFECT — every scenario passes — so the failure mode flatters the thing under test. Every mutation therefore reports its occurrence count before and after, and a mutation whose count did not move is a hard error, never a datum. The build result is read separately from the test result for the same reason: a mutation that does not compile makes the target report "Executed 0 out of 1", which a grep for failures renders as silence.