fleettools

fleet — sweeping ~1800 repositories across ~320 organisations

Written 2026-08-30 while getting every default branch green. They lived in a scratchpad, which is purged; they are here because each one encodes something that cost a wrong answer to learn.

What is built into them, and why

Every one of them says when it is not the code in this repository

fleet.WarnIfStale runs first in every main, costs one compare call, and writes at most one line to stderr.

It is here because of a measured failure. A guard landed on 2026-09-27 that sets aside a red belonging to a workflow every push skips. Nothing rebuilt the installed binaries, so for four days redscan reported the same three repositories as RED default branches — go-fsctl/go-fsctl.github.io, go-ruby-hanami/go-ruby-hanami.github.io, nano-container-linux/dnsctl — and the fix for exactly that was sitting on main, 32 commits ahead. Rebuilt: 1998 of 1998, 0 red. Nothing was broken, every number was wrong, and no output said so.

It reads the VCS stamps, not the module pseudo-version, because only the stamps carry whether the tree was clean, and a reading from a modified tree is not one anybody can reproduce.

⛔ It is silent in exactly one case: built from the head of the default branch, from a clean tree. It says something in every other case including "I could not tell" — a check that falls silent when it cannot answer teaches you to trust a silence it never earned.

⛔ It asks whether those commits could reach THIS command, and this is where it nearly destroyed itself. Until 2026-10-18 it compared the module, so a binary installed the way Go installs tools — go install …/cmd/vulngate@v0.18.1 — was called STALE because some other command had moved. Measured by a peer session and verified here: git diff --name-only v0.18.1..v0.20.1 -- cmd/vulngate internal/fleet is empty, those four commits touching only README.md, cmd/buildscan, cmd/docscan and cmd/modrename; built with -buildvcs=false the two vulngate binaries have the same sha256. The weaker case is the better evidence: a one-line README fix — the commonest commit here — made every tool print "Those commits may change what this pass reports", all afternoon, until it was being read past. A warning that fires on the canonical install is how a true warning gets ignored.

The rule is provable, not a heuristic: package main cannot be imported, so a change under cmd/Y can never reach a binary built from cmd/X. Everything else — internal/, a root package, go.mod, go.sum — keeps the warning. The quieter line still names the distance, because "nothing reached you" is a fact worth printing, not a silence.

⛔ Three things make that safe rather than clever. GitHub's compare endpoint returns at most 300 files and does not announce that it truncated, so at the cap the answer is unknown and the warning stands — unknown is never quiet. An empty debug.BuildInfo.Path reaches everything: not knowing which command is running is not evidence that the commits miss it. And the list of paths that "cannot end up in a binary" (*.md, .github/, LICENSE, .gitignore, renovate.json) is held to the real tree by a test that reads every go:embed directive in the module and fails if one names such a path — it also fails if it reads no directive at all, because a walk that finds nothing asserts nothing.

The tests that keep this honest enumerate cmd/ on disk rather than naming the commands: a list written in a test has to be remembered, a directory listing cannot be forgotten, so a ninth command fails the suite until it is wired up. Both sabotage directions are checked — for the per-command rule that means a row for every "still warns" case too, since a function that always went quiet would pass a one-directional test and take away a true warning.

To rebuild them all:

GOWORK=off GOBIN=~/.local/bin go install ./cmd/...

Every one of them answers -h with 0

Measured 2026-10-18 by building all 25 flag-parsing commands and asking each one: 13 of 25 answered -h with exit 2. Now 0 of 25 do.

The cause is one the standard library hands you and then hides. flag reports a help request as an error like any other — flag.ErrHelp — and the two-line shape every command here was written with cannot tell it from a typo:

if err := fs.Parse(args); err != nil {
	return 2
}

flag.ExitOnError does tell them apart: it calls os.Exit(0) for ErrHelp and os.Exit(2) for the rest. Every command here uses flag.ContinueOnError instead, so that run() can be tested rather than exit the process — and that choice silently gave the distinction up. It is decided in one place now, fleet.ParseFlags, which is also the only place the help branch has to be covered; thirteen hand-written copies would be thirteen branches nobody exercises, and the twelve commands that did get it right prove the shape is as easy to forget as it is to type.

⛔ The text gate missed one, and the behavioural measure caught it. blobpurge dispatches on a subcommand, so -h was neither scan nor strip and fell through to "I do not know that subcommand" → 2, with both subcommands' own flag sets already correct. Running the binaries is what found it; a check reading sources was satisfied. fleet.IsHelpRequest answers that half, and the three commands that asked it by hand each knew a different three of the four spellings people type — prmerge missed -help, pagespublish missed help, blobpurge asked nothing at all.

⛔ And fixing it broke the control next door, which is the lesson worth more than the fix: TestEveryCommandReadsItsArguments recognises the spelling .Parse( or "-h", so moving the idiom into two helpers made it fail for 13 commands it had just been made right about. A check that names one spelling of the answer fails at the refactor instead of at the defect. Both helpers are named in it now, because what they guarantee by construction — the arguments are read before anything happens — is the intent it was always testing.

Why -h matters more here than in most repositories: these tools sweep thousands of repositories, -h is how somebody finds out what one does before running it, and a non-zero status there is read as "this tool is broken" by every wrapper that checks statuses. The sibling failure is worse and has happened — on 2026-10-11 redscan -h, run to read its flags, started a sweep of every organisation and had to be killed.

prmerge never merges a Go toolchain bump. The org presets set automerge: false on it, and a sweep that merges whatever is green does not know that. Mine took two before the guard existed. It now skips them with the reason go-toolchain-left-to-a-person; one pass skipped 18.

prmerge reads agentsync's leases and stays out of claimed repositories. A fleet sweep cannot claim a hundred repos without blocking every other session, so it does the opposite: it looks at what someone else claimed and leaves it alone. See ../agentsync.

prmerge treats "no checks at all" as not-green. A PR with nothing run against it cannot be merged on evidence. This is also how repositories with no CI get found.

tidyall.sh names its work directory per PULL REQUEST, because two PRs in one repository otherwise collide and the second clone fails — which reads as an auth error and is not one. It tidies every go.mod in the tree: a sub-module keeps its own, and a lane that vets it fails while the root module is spotless (go-widgets/toolkit has rougelex/). It tolerates modules that cannot resolve standalone, but requires the ROOT one to tidy.

What quietscan is, and the seven rules it will not soften

Nothing else in the fleet queries run recency. redscan fetches created_at and never compares it to anything: it branches on conclusion alone, and status=completed&per_page=1 on a repository with zero runs returns {} -- so a runner that has NEVER run is invisible to it.

⛔ Activity is not health, and this tool learned that the expensive way. On 2026-10-31 it reported sixteen organisations as healthy whose Renovate runner failed in about ten seconds every single night: RENOVATE_TOKEN had never been set, and an unset secret expands to the empty string. None of those sixteen had ever received a dependency update. go-approx's entire history was 11 runs, 11 failures, zero successes. The sweep asked WHEN the last run started and never what became of it, so timely was the only way to be fine — and a runner that fails fast is the most active-looking failure there is. redscan caught them only incidentally, because a runner's workflow happens to sit on a default branch; the specialised sweep, the one whose whole job is Renovate runners, was the one that said nothing.

So there is a broken verdict, it leads the report, and each line carries the conclusion and the failure streak:

go-autodiff/.github   last run 2026-10-31 19:13Z ENDED FAILURE
                      9 of the 9 scheduled runs read, NONE successful

The streak is the diagnosis, not the failure: "it broke last night" and "it has never once worked" are the same red tick and different fixes. It costs no extra request — the page comes back from the call already being made.

⛔ And it now names the secret instead of pointing at a log. The workflow says which secrets it needs — token: ${{ secrets.RENOVATE_TOKEN }} — and the API says which exist, so missing_secret reports the name of the fix. The reference costs nothing to read: it is in the file already fetched for the cron.

Three things about it are deliberate, and two of them are refusals:

• secrets.GITHUB_TOKEN is excluded. GitHub injects it and it appears in no listing, so counting it would report all 156 runners as missing a secret — and a check that fires on everything is read as firing on nothing.
• The organisation listing is not read, on purpose. GET /orgs/{org}/actions/secrets answers 403 — "You must be an org admin or have the actions secrets fine-grained permission", "needs the admin:org scope" — measured across this fleet on 2026-11-01. Widening the fleet's token to admin:org so a read-only health check could enumerate every organisation's credentials is a bad trade, and the measurement showed it is unnecessary.
• So a missing name is a verdict only on a runner that is failing or has never run. A scheduled run that succeeded has already proved its secret resolves, wherever it lives; on a working runner the fact is printed on the line and never paged on. The first version of this check asked for the organisation listing and would have been dead fleet-wide.

Nothing here reads a secret's value, and nothing can: GitHub's own words for the endpoint are that it lists secrets "without revealing their encrypted values". A name is a property, and a secret is verified by its properties.

⛔ And the evidence is the scheduled path only, which cuts both ways. Runs are read with event=schedule, because a manual dispatch is not evidence that the cron still fires. A runner whose dispatches succeed while its nightly runs fail is therefore reported broken. Three of those sixteen were called proven on 2026-10-31 on the strength of a successful dispatch; this tool disagreed, and it was right — their scheduled runs had still never worked.

Twelve hours of slack, and the observed lag printed every pass. GitHub's scheduler was measured on this fleet running 1h59m to 7h56m behind its crons. A one-hour tolerance would have paged on 113 runners that were working perfectly, and the eight hours this shipped with left four minutes of headroom against the worst lag observed on the second day. A pager that goes off on a fleet with nothing wrong is the same failure as an informational section nobody reads, and it costs more. Because the number drifts, quietscan prints the distribution on every pass and says so out loud when the worst lag passes three quarters of the slack -- the threshold is a measurement to re-take, not a constant to trust.

Age against each runner's OWN cron, never a constant. The fleet's runners are staggered across 24 hours -- 0 4, 44 16, 11 15, 33 12 -- so the period is read from each workflow's own cron: line.

"Not yet due" is a different verdict from "never fired." Conflating them produced a false alarm about 26 dead runners that were merely queued behind a schedule they had not reached. The birth date is keyed on the workflow's created_at, and the classifier starts its clock at the last run, falling back to the birth date ONLY when there has never been one. Never on the last commit to the workflow file: a healthy Renovate rewrites that file every time it bumps its own action pin, so a monitor keyed on the file's mtime resets its own clock whenever Renovate works, and goes blind precisely when things are fine. Taking the LATER of birth and last run reintroduces the same blindness by another door, and did, until a test caught it.

Four verdicts, four different fixes, never one total. archived (no pull request can fix it), disabled_inactivity (re-enable -- and it expires again in 60 days), disabled_manually (may be entirely legitimate), and the runner that simply stopped. Each section names its own fix.

Driven from the ORGANISATION list, not the runner list. A recency check over runners structurally cannot see an organisation that has no runner to be quiet.

Attribution: the property belongs to whatever COVERS the repository

The first version got its headline right and its informational output wrong, in two ways that were the same mistake twice.

It reported no_runner for 199 organisations. They were not uncovered: a Renovate runner does not watch the repository it lives in, it watches whatever its autodiscoverFilter names, and go-ruby-stdlib/renovate-runner names go-ruby-*/** -- 204 organisations from one repository. And it warned that some twenty go-ruby-*/.github repositories were 53 days from a 60-day latch, when those repositories hold no workflow at all: there is no schedule there for GitHub to disable. The repository that actually carries that risk for all of them is the shared runner, and that is not where the warning pointed.

So quietscan reads every runner's autodiscoverFilter and resolves coverage fleet-wide:

• An organisation reached by a live runner's filter reads covered, naming the runner -- a count by default, a list under -all. What is left in no_runner is what the verdict was built for, and it is printed even at zero, because "no organisation in this fleet is unwatched" is a strong statement and a section that vanishes when empty cannot make it.
• The idle/latch warning applies only to a repository that holds a scheduled workflow, names the runner, and says how many organisations its silence would take with it.
• A retired runner covers nothing. go-attest/renovate-runner is disabled_manually, so the organisations it used to watch are not covered by it any more; the line says what it used to watch.
• A filter that could not be read is reported as unread, never as zero. go-attest/renovate-runner builds its filter with mine.map((o) => ...) over a list sliced by an env var, and no string scan can know what that resolves to. Treating it as no coverage would be the same mistake in a smaller coat.

The Renovate App is the third form of coverage, asked about only where it can change an answer -- an organisation with no runner and no filter reaching it. openweft has the App installed and 16 open pull requests from it, and no census of workflows can see that: there is no workflow, no cron and no repository to be quiet. A refused installations read is coverage unknown (App visibility) on a line of its own. An unread installation is unread, not absent, and that rule now runs the other way too: if a LIVE runner's filter could not be read, an organisation "no filter reaches" might in fact be reached by it, so it reads coverage_unknown rather than becoming a gap.

Both candidate repositories -- .github and renovate-runner -- are probed in every organisation, not just until one answers: go-attest holds an active own runner beside the retired shared one, and stopping at the first would have hidden the retirement.

Coverage is a snapshot with a time on it. go-pdfkit/renovate-runner was retired between two passes an evening apart, and nothing says it could not be retired between two calls of one pass. Every runner that others are counted as covered by is re-read at the end of the pass; if one moved, coverage is re-resolved from scratch and the move is printed. The header carries the window the runners were read in, not a single instant.

A watcher that prints 199 non-problems and a latch date that cannot arrive gets scrolled past, and then the one real page is scrolled past with it. That is the failure mode for an alerting tool -- not being wrong, but being ignored.

The trap quietscan exists to catch before it latches

All runner repositories here are public, so GitHub's 60-day inactivity rule applies, and a .github repository whose only content is the runner receives no push except Renovate's own. If Renovate stops, the repository goes quiet; at day 60 GitHub disables the schedule; and the disablement makes the silence permanent. Re-enabling expires again. The durable answer is a working Renovate, which is why quietscan warns at 45 days -- fifteen days of margin -- and prints the date each repository latches.

A failed call must never become a verdict

The core budget was exhausted mid-evening with three sessions on the same fleet, and /rate_limit reported core 5000/5000 while a real request returned 403 with X-RateLimit-Remaining: 0. So the budget is judged by what the API did, never by what it says about itself.

The worst possible outcome for a watcher is a verdict derived from a call that did not answer -- an outage wearing the mask of a stopped runner. Every refused read becomes unreadable, which pages, and never never_fired, overdue or no_runner. Three specific holes were closed:

• isNotFound requires the 404 itself, not the words around it. A 403 that happens to say "Not Found" would otherwise have read as "this organisation has no runner".
• A failed config.js read used to leave only a note, silently removing a runner's coverage and turning its 204 organisations into gaps. It now makes them coverage_unknown, naming the runner that could not be read.
• A core rate-limit refusal fails fast instead of backing off. It refills on the hour, so it will not clear inside a pass, and 1300 calls x 5 sleeps is not a pass but a hang. A secondary limit still backs off, because that one is genuinely transient. A rate-limited pass prints a banner saying its verdicts are incomplete.

What one pass costs, and when not to run it

About 1300 REST calls, roughly 26% of an hour's core budget: two repository probes per organisation, then the workflow, its cron, its config.js and its last scheduled run per runner, plus one installations read per otherwise-uncovered organisation. An archived runner costs one call and a disabled one three -- their cron and run history decide nothing, though their filter still does.

Twice a day is comfortable; a tighter loop is not. Do not run it beside another fleet sweep -- redscan and prmerge walk 1800 repositories, and two of those in one hour will starve one or the other. Three sessions working this fleet concurrently exhausted the budget once already, and eight were holding leases on it the same evening.

-orgs a,b,c checks a handful for about four calls each, which is how to re-check one organisation without spending a quarter of the hour. A narrowed pass cannot establish a gap, because coverage is decided by other organisations' runners: -orgs go-ruby-aasm alone cannot see the filter that covers it. So no_runner is unreachable under -orgs -- those organisations read coverage_unknown naming the narrowing. Safe by construction, not safe if you read the banner. What a narrowed pass CAN establish -- a runner's own health, an App installed on that organisation -- it still reports.

Two things quietscan is not

It is not a scheduled GitHub Actions workflow. A watcher that shares the failure mode it watches for is not a watcher. Run it from this machine like its three siblings, or from launchd if it is to be unattended. It exits 1 when anything pages.

It never dispatches a run. A dispatch opens real pull requests across an organisation, and actions/workflows/{id}/dispatches is one path segment from the runs listing it reads all day. So every call passes through readOnly, which allows gh api and nothing else -- an allowlist of one, because the first version of that gate was a denylist of shapes I had thought of, and its own test caught gh workflow run walking straight past it.

The runs listing is filtered to event=schedule, and that filter is the whole tool. On the pass that found them, twelve runners that had never once started on their own carried successful workflow_dispatch runs from rollout day. Any check that counts "a run" rather than "a scheduled run" calls all twelve healthy.

quietscan                                    # the live fleet
quietscan -all                               # every runner, and every covered org
quietscan -fixture cmd/quietscan/testdata/broken.json -all
quietscan -fixture cmd/quietscan/testdata/unread-filter.json -all

The fixtures are how the watcher is proved: doctored copies of one real runner, each changed in exactly one way, so the verdict names what was changed. A watcher that has never fired in anger is not known to work, and this one's whole purpose is to speak when everything looks quiet.

broken.json walks the verdict set. unread-filter.json is separate because it would swallow the first: a LIVE runner whose filter could not be read casts doubt on every organisation no readable filter reaches, so no_runner correctly drops to zero there and coverage_unknown takes its place. That is the rule working, and it is easier to see on its own.

The failure these were built for

Renovate writes the new go.sum lines and leaves the superseded ones, so any lane asserting go mod tidy produces no diff fails on go.sum alone — with the module itself perfectly fine. The message names no file. It hit 11 PRs at once across four organisations. tidyall.sh is the pass that fixes it.

The workflow-scope blocker

gh pr merge cannot merge a pull request that edits .github/workflows/*: the merge API refuses an OAuth app without the workflow scope, and gh's keyring token carries repo but not workflow. On 2026-08-31 this was the single largest blocker in the fleet -- 64 green pull requests waiting on it in one pass, after 69 had already been landed by hand.

wfmerge.sh was the workaround: squash locally, push to the default branch. That is exactly what git-pre-push-guard was installed to stop -- "a fix went straight onto main by habit, green, tested and reviewed by nobody" -- so the script now refuses unless someone says WFMERGE_I_MEAN_IT=1 for a specific case.

The fix is one command, and it is not automatable: gh auth refresh -s workflow opens a browser. Until it is run, these pull requests wait.

The three vulnerability lanes, and why two of them run on a clock

.github/workflows/vuln.yml holds all three, on push, pull_request, a daily cron and workflow_dispatch.

⛔ Two lists because they answer two different questions — blast radius and exposure — and either can be dropped without touching the other. And ⛔ two JOBS rather than two steps, because a step after a failing one is skipped: in one job, every night the tools sweep went red the second list would go unswept, and silently.

⛔ Choosing that second population is what found something. On 2026-10-31, the first sweep anyone had run over the 58 Go repositories of go-webengine, go-compressions, go-filesystems and go-images: 56 clean and 2 reaching GO-2026-6860 (golang.org/x/net, fixed in v0.62.0) — in go-webengine/browserproxy through engineRender → Engine.RenderWithLinks → html.Parse, which is the actual job of that program rather than an init artefact. x/net was // indirect there, so no dependency manager had a version string to compare or rewrite and nothing was watching it at all. The repositories that reach an advisory are not the ones with the most dependencies; they are the ones that read bytes they did not write.

⛔ A ceiling measured in the wrong place cost this lane a day, and the mistake is worth more than the number. GitHub rate limits the git transport, not only the REST API, so a clone-based sweep has a ceiling even though it spends none of the REST budget the Renovate runners draw on. Measured from this workstation on 2026-10-31 — and acted on:

So the nightly list was cut to 22 and the 49 local-byte codecs were relegated to workflow_dispatch. The first real run from a runner, the next morning, answered differently:

72 clones in one run, nothing unread. The ceiling belonged to the workstation's unauthenticated requests, not to the machine that executes the lane — a limit measured from the wrong client is a property of the instrument, not of the subject. All three lists are nightly now.

⛔ One run of 72 clones is not proof of a steady state, and that is acceptable for one reason: the failure is visible. UNREAD names the repositories and vulnscan exits 2, so a night that does meet the limit says so instead of going quiet. The in-tool retry is #150.

⛔ Slicing across the week stays rejected, and that was never about the limit: it keeps everything in view while reintroducing exactly the latency a clock exists to remove, which is how those go-pkgx advisories went 2 and 4 days unnoticed. The two lists remain separate because they answer different questions, and the evidence still supports the distinction — both repositories that reached an advisory were network-facing.

⛔ A gate that only fires on a push has the coverage of the repository's activity, and the quietest repository needs it most. Measured on go-pkgx, 2026-10-27 to 10-30: GO-2026-6860 and GO-2026-6866 (x/net/html, fixed in v0.62.0) sat green for days in two repositories and were found 2 and 4 days late — in one by a pull request adding a subcommand that touched no dependency, in the other by a pull request editing one workflow file. Every run of the scan was right; it simply only ran when somebody pushed. This repository had that exact shape, and it is where the tooling that finds the defect lives.

Daily rather than weekly, because the database is updated on business days and a weekly cadence would have left those two sitting for six days. ⛔ And a disabled cron reports nothing rather than failing — GitHub switches a scheduled workflow off after 60 days without repository activity — so silence on this lane is not green, and workflow_dispatch is there for a person to ask.

⛔ Renovate cannot do the second half. An indirect dependency is not a floor the module declares for itself, so there is no version string for a manager to compare or rewrite — which is how x/net reached both go-pkgx repositories. Nothing watched indirect dependencies anywhere in this fleet, although cmd/vulnscan was written for that question and was scheduled nowhere. It needs no credential: it clones at depth 1 and makes no REST call per repository, so it cannot starve the hourly budget the Renovate runners spend from, and the workflow runs with contents: read.

What a sweep of that list has already found, by hand: on 2026-10-22, GO-2026-6809 (CVE-2026-111943, crypto/x509 Certificate.Verify, published the day before, fixed in go1.26.10 and go1.27.3) was reachable — not merely present — in four of them: gitsafe, fleettools, modfloor and landingkit. Nothing would have reported it; a person happened to run a sweep.

Two details that are easy to get wrong:

• The full list runs only on the schedule and on dispatch, because an advisory in landingkit must not redden an unrelated pull request here. But a command line exercised only at 06:17 is one nobody has seen work, so every event runs the identical binary, flags and -list mechanism over .github/vulnscan-smoke.txt, which names this repository.
• vulnscan exits 0 clean, 1 reachable, 2 unread, and 2 is not a lesser 1: a repository govulncheck could not load prints its config and no finding, which reads like a clean one. Both fail the lane. -selftest runs first and judges three recorded streams, so a binary that cannot see a known positive says not to trust the sweep behind it.

Layout and gates

quiet/           the cron reader and the verdict, with its tests
cmd/<name>/      one sweep each

go build ./... && go vet ./...
go test ./...

quiet is held at 100% of statements, the standard gitsafe/protect and gitsafe/redact hold: it is the part that can be wrong in a way nobody notices. The command around it is at 62%, with read() exercised against a stub gh on PATH -- a wrong field name there would make every runner in the fleet look healthy, silently.

See "What one pass costs" above before scheduling it beside another sweep.

Flags: one dash became two

Every command here is a cobra command. Until v0.20.1 they used the standard flag package, and the two disagree about one spelling — the one this fleet typed everywhere:

flag   accepts -releases, --releases and -j   (one dash or two, any length)
pflag  accepts --releases and -j              (one dash means ONE letter)

So vulngate -go go1.27.2 is no longer the long flag. It still works, and says so once on stderr: internal/cli's Compat rewrites a single-dash long flag into its double-dash spelling when the long flag is registered, warns, and will not outlive the next major. It leaves an unknown token alone so pflag still rejects it, copies everything after -- verbatim, and refuses the ambiguous case — -ab when ab is a long flag and a and b are both shorthands, where pflag reads a cluster and flag read the long name.

The scope was measured before the migration began, with a Go tool rather than a guess: 16 call sites in workflows across 13 repositories, all of them vulngate -go or brandcheck -org, plus 71 more in this repository's scripts, READMEs and Go source. There was no flag day, because those lanes install a pinned version: a lane on v0.20.1 keeps working unchanged, and only a lane that bumps has to spell its flags with two dashes.

One-letter flags did not move. -q, -n, -j, -v are registered as shorthands of a named flag (--quiet, --dry-run, --jobs, --verbose), so every existing call site reaches them.

internal/cli carries the three things all of them shared, so that they are decided once instead of thirty-three times: