judgescan

Overview ¶

judgescan reports every repository whose tests look for an external tool that its CI never installs — a foreign judge that exists in the source and has never once run.

judgescan                  # the whole fleet, under the GitHub root
judgescan -root DIR        # somewhere else
judgescan -repo o/r        # one repository, verbosely

It exists because the same defect turned up twice in one afternoon, in two unrelated repositories, and neither showed any symptom.

go-fde/luks had TestInterop_LiveCryptsetup, which skipped unless it was root AND cryptsetup was present. The CI is unprivileged and installed neither, so it had never run. The committed fixtures were all LUKS2, so LUKS1 was read back only by the code that writes it — and a misreading of the LUKS anti-forensic split lived on both sides of that fence for months, agreed with itself, and passed.

go-filesystems/btrfs had seven kernel oracles, each gated on `os.Geteuid() != 0` as a single block. But `btrfs check` — upstream's own reader, judging OUR WRITES, the most valuable direction there is — reads an image FILE and needs no privilege at all. Only the loop-mount half did. The check half was locked behind a requirement it never had, in a CI that also did not install btrfs-progs.

Both look identical from outside: a green lane, a test file full of assertions about a third-party tool, and nothing whatsoever being judged.

What it cannot tell you ¶

This reports a MISSING TOOL, not a useless test. A macOS-only tool on a Linux lane is correct. A tool the repository deliberately treats as optional is a decision, not a defect. The output is a list of questions to ask, and the interesting column is the last one: what a test is prepared to check and no machine has ever checked.

It reads the working tree only. No API calls, no network — which also means it cannot starve Renovate's hourly budget, and can be run as often as you like.