refusal-probe

Overview ¶

Command refusal-probe measures the corpus's refusals without regenerating it, and diffs two such measurements.

It exists because every agent working the language wall rebuilt this same throwaway program, one per branch, costing five to ten minutes each before any of the actual work started. Worse than the time: each rebuild made its own choices about what to count, so two agents reporting "sites" were not always reporting the same number, and reconciling them cost more time again. This is that program, written once.

Why not just run the generator ¶

`just corpus` acquires provider schemas across ~75 plugin subprocesses and takes long enough that six agents in one session backgrounded it, lost track of it, and stalled. It is also the wrong tool for a with/without comparison: it writes live/corpus-refusals.json, so two agents running it concurrently in the same tree clobber each other.

This runs the same check.Dir the generator runs, over the same manifest, writes wherever you point it, and finishes in about twenty seconds.

The bound you must carry ¶

Without provider schemas this UNDERCOUNTS refusals and OVERCOUNTS what a fix clears. A type absent from the generated admission table is refused as unadmitted here, where the provider's own identity schema would have settled it - see check.Context.Schemas. More important for measuring a fix: clearing a refusal often reveals another refusal underneath it that only a schema-backed run can see.

That is not hypothetical. One fix measured 11 sites cleared with this instrument and delivered 10046 -> 10046 on the schema-backed regeneration, because fourteen sites had merely moved from one refusal ID to another. Another measured 60 and delivered exactly 60. The difference was not luck: the second agent checked, per entry, that no other refusal count had changed.

So: sweep with this, then verify the entries you care about against real schemas before reporting a number as anything but an upper bound. Every output carries schemas:false to make that hard to forget.

-schemas, for the questions the default mode cannot reach ¶

The bound above is not a caveat to recite, it is a blind spot with a shape, and three separate agents each hand-wrote their own schema-backed probe after hitting it. What the default mode cannot see:

• The whole stamp layer. check.Analyze runs it, but every resource reads as "schema not available" without schemas, which stamp.SkipReason.Unknown correctly reports as unknown rather than refused - so the layer contributes nothing at all and the 110 stamp sites in the committed corpus are invisible here.
• Any rule that returns false when schemas are nil. Its zero is not evidence of anything; it is the absence of evidence, and counting it as agreement is how a fix "clears" sites that were never measured.
• Non-AWS estates. A google_* configuration measured against no google schema reports unadmitted-type for every resource in it, which is a property of the run and not of the configuration.

-schemas acquires each entry's OWN providers (see schemas.go) and runs the identical analysis with them. It costs about two and a half minutes warm against the default mode's twenty seconds, which is why it is opt-in, and it reports per entry which providers it got and which it did not - a partial acquisition that goes unreported is how a gate reading len(schemas) > 0 came to fabricate hard refusals for types whose own provider had simply failed to install. 35 of the 250 manifest entries are missing at least one provider, so that is the common case.

It also reports the per-site CAUSE, not only the count: see cause.go. That is what the hand-built probes were for.

What the two modes actually differ by, measured ¶

Both modes over all 250 manifest entries at commit 7d66fa0968, schema-less then schema-backed:

sites      8767 -> 8461      blocked configurations   193 -> 206
instances  3587 -> 3921

Unmarked apply of a marker-only resource      0 -> 110   (the stamp layer)
Two resources with the same identity          0 ->  34
unadmitted-type                            1835 -> 1299
Not an identity attribute                   111 ->   36
Unresolvable identity                       174 ->  131

Note the two directions. Sites fall, which is the documented bound - the schema-less mode over-reports refusals a provider's own identity schema would have settled. But BLOCKED CONFIGURATIONS RISE, by thirteen: the schema-less mode reads thirteen configurations as unblocked that a real run refuses, because a rule that needs a schema to fire returns false without one and a false there is not evidence of anything. So "upper bound" is true of sites and false of the verdict, and a fix validated only against the default mode can look like it unblocked something it did not.

What it refuses to measure, and what it refuses to compare ¶

Two silent failures of this tool were found by the agents who hit them, and both are the class the tool exists to prevent: an instrument reporting a number about something other than what the reader assumes.

A fresh worktree has no .corpus - it is gitignored, and agents symlink one in rather than refetch 250 repositories - and a sweep there reported "entries 31" with exit 0 and no other output. So a sweep now refuses unless every manifest source expands to at least one configuration on this disk, and every fetched source sits at the commit the manifest pins. The manifest holds 21 globs rather than 250 entries, so the expectation is derived per source rather than hard-coded (see corpus.go). -allow-partial-corpus measures anyway and records what was wrong.

-diff compared two sweeps of two different trees without a word. It could not have done otherwise: -root defaults to "." and both sweeps recorded ".", so the field that should have caught it compared equal. Every sweep now records its root resolved absolutely (symlinks and all), its commit, the manifest's content digest and its per-source corpus accounting, and -diff refuses any pair differing in an input that is not the change under test: another tree, another manifest, one side schema-backed, another provider version, a partial corpus on one side, or two different sets of entries.

Three inputs are allowed to differ and are reported rather than refused, because each is normal and none of them is the change: module install state per entry (one corpus entry moved 59 -> 394 refused sites on that alone), the var files each entry was measured with, and per-provider acquisition.

Use ¶

refusal-probe -out before.json                 # sweep the whole manifest
  ... make your change ...
refusal-probe -out after.json
refusal-probe -diff before.json,after.json     # what moved
refusal-probe -entry .corpus/vpc -v            # one entry, per-site detail

refusal-probe -schemas -out before.json        # ~2.5 min warm, real schemas
refusal-probe -schemas -entry .corpus/vpc -v   # one entry, per-site cause

A schema-less run and a schema-backed one measure different things, so -diff refuses to compare them.

-onboarded, for the question none of the above asks ¶

Everything above measures the ADOPTION question: can a stranger's published configuration be taken over exactly as it stands. That is not the product's primary goal, which is a fully migrated estate - ordinary Terraform, plus a live block, applied, and managed from then on with no state file. Nothing measured that, because every corpus entry is somebody else's published configuration and (verified over all 250) not one declares a live block, a record_store or the sidecar file. So the classes a record_store answers were counted as language wall in every figure this project has produced.

refusal-probe -schemas -onboarded -out both.json    # ~3 min warm

analyzes each entry TWICE: once as published, and once after internal/live/onboard's computed edit. See onboarded.go for the design and for what it still does not claim. The published-form fields are produced by the identical call in both modes and do not move; -diff compares the published form and refuses a mixed pair.