Documentation
¶
Overview ¶
Package corroborate holds the confirm-only index used to decide whether a string recovered from pixels was really something the site shipped.
Why this is confirm-only, and why that is not a limitation ¶
The text on a WebGL brand site almost always exists as text before it becomes pixels: a hydration blob, a CMS payload, a string table in the bundle. That makes those payloads an enormously tempting source of content, and using them as one would be a mistake.
A hydration blob routinely carries draft copy, other locales, unpublished fields, internal identifiers, and adjacent records that were never on the page. Emitting any of it stops being extraction and becomes republishing someone's back office -- which the project's ethics position rules out, and which no amount of filtering makes safe, because the filter would have to know which records the site intended to publish and it cannot.
There is a second, quieter reason. Fidelity is defined as the share of artifact statements verifiable in the source. Content that only ever existed in a data feed fails that check while being technically true, so a distiller that mines feeds scores worse on its own headline metric.
So the rule is absolute and it is enforced by this package's shape rather than by discipline: an Index answers one question -- "does this string appear in what the site shipped?" -- and offers no way to enumerate its contents. Confirmation promotes a guess from speculative to verified. Absence leaves it speculative. Nothing here can become a block.
Index ¶
- Constants
- type Index
- func (ix *Index) AddScript(kind, src string)
- func (ix *Index) AddText(kind, s string)
- func (ix *Index) Contains(candidate string) bool
- func (ix *Index) ContainsAny(candidate string) (string, bool)
- func (ix *Index) Saturated() bool
- func (ix *Index) Size() int
- func (ix *Index) Sources() map[string]int
Constants ¶
const DefaultCap = 8 << 20
DefaultCap bounds the corpus. Large enough for a hydration blob and the string tables of a typical bundle; small enough that a hostile response cannot exhaust memory.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Index ¶
type Index struct {
// contains filtered or unexported fields
}
Index answers membership questions about the text a site shipped.
It deliberately exposes no iteration, no listing, and no way to read a stored string back out. The only thing a caller can learn is whether a string it already has appears somewhere in the payload.
func (*Index) AddScript ¶
AddScript folds in a JavaScript bundle by extracting its string literals.
Storing whole bundles would be tens of megabytes of minified code, almost all of it operators and identifiers. The literals are the part that can contain a headline, and they are a small fraction of the bytes.
func (*Index) AddText ¶
AddText folds in plain text: inline JSON, hydration blobs, the page's own rendered text.
func (*Index) Contains ¶
Contains reports whether a candidate string appears in the shipped payload.
The comparison is on a normalised form -- lowercased, whitespace collapsed, punctuation folded -- because a headline set in a 3D scene and the same headline in a JSON field routinely differ in casing, curly quotes, and non-breaking spaces while being unmistakably the same sentence.
func (*Index) ContainsAny ¶
ContainsAny reports whether any substantial fragment of the candidate appears. A vision model rarely reproduces a headline verbatim; it commonly gets a clause right and the rest approximate. Matching on the longest clause is the difference between confirming most real recoveries and confirming none.
func (*Index) Saturated ¶
Saturated reports that the cap was reached, which means a negative result is weaker evidence than it would otherwise be.
Source Files