Documentation
¶
Overview ¶
Package escalate decides how hard to work on a page.
The decision, and the cost of getting it wrong ¶
Escalation is what makes sieve correct at the appropriate cost across the whole web rather than a heavy tool that is overkill for a blog. It is also the change most capable of damaging the project, because a tool that judges the same page differently on different days has forfeited the claim that the same input yields the same output -- and pages near a threshold are exactly where that happens.
Three things keep the decision stable, and all three are load-bearing:
- The thresholds are pinned constants, not values computed per run. A threshold derived from the page being judged can move under it.
- Once a domain escalates, it stays escalated. Memory is consulted before scoring, so a site that needed a browser last week needs one today regardless of which A/B variant happened to be served.
- Every input to the decision, and the decision itself, is recorded in the artifact. A bug report starts with "which tier answered and why", and that question has to be answerable without re-running anything.
The stability check asserts that the *tier* is stable separately from asserting that the content is stable, because those are different failures with different causes.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Decision ¶
type Decision struct {
Tier Tier
Score float64
// Reason is a human-readable account of the scoring, recorded in the
// artifact so a caller never has to guess why a page was judged cheap.
Reason string
// Pinned reports that the tier came from per-domain memory rather than from
// this page's score.
Pinned bool
// Factors is the itemised scoring, for `sieve doctor` and for tuning.
Factors []Factor
}
Decision is the outcome, with everything that produced it.
type Factor ¶
type Factor struct {
Name string `json:"name"`
Value float64 `json:"value"`
Weight float64 `json:"weight"`
Note string `json:"note,omitempty"`
}
Factor is one contribution to the score.
type Memory ¶
type Memory struct {
// contains filtered or unexported fields
}
Memory records which tier a domain has needed before.
This is the hysteresis that stops a page near the threshold from being judged differently on different days. A domain that has ever needed a browser keeps needing one: the cost of occasionally over-working a page is a few seconds, and the cost of a tool that wavers is the project's central claim.
It only ever ratchets upward. There is no decay, because a site that used to need rendering and now does not is a rare event, and getting it wrong in that direction silently loses content.
func (*Memory) Apply ¶
Apply consults the memory and raises the decision if the domain has needed more before. It then records the outcome.
func (*Memory) Note ¶
Note records that a domain turned out to need a given tier, whether or not the scorer predicted it. The sweep calls this when it discovers a scroll hijacker or a canvas that static extraction could not have known about.
func (*Memory) Predicted ¶
Predicted reports the tier this domain has needed before, or the empty tier when it has never been seen.
It exists so work that only pays off on an escalated page can be started before the page has been scored. The memory is the only evidence available at that point, and it is evidence the ladder already trusts enough to override a fresh score with.
type Thresholds ¶
type Thresholds struct {
// EscalateAbove is the score at or above which a browser is used.
EscalateAbove float64
// SweepAbove is the score at or above which the full sweep is used rather
// than a single post-settle capture.
SweepAbove float64
// RecoverAbove is the score at or above which canvas recovery runs.
RecoverAbove float64
// MinStaticChars is the amount of readable text below which static
// extraction is treated as having failed outright, regardless of anything
// else. A page with 200 characters is a shell.
MinStaticChars int
// ThinTextRatio is the text-to-markup ratio below which the served document
// looks like an application shell.
ThinTextRatio float64
// for recovery to be worth attempting.
CanvasViewportShare float64
// ShatteredTextRatio is the share of text runs that may be one- or
// two-character fragments before the served HTML is treated as split text
// that only a browser can reassemble.
ShatteredTextRatio float64
// MinShatteredRuns stops the ratio firing on tiny documents, where three
// fragments out of eight runs is noise rather than a pattern.
MinShatteredRuns int
}
Thresholds are pinned, not computed.
Every number here is a constant that a maintainer can change deliberately and a user can read. Deriving any of them from the page under judgement would make the decision depend on the thing being decided, which is how a scorer starts oscillating on pages near the line.
func DefaultThresholds ¶
func DefaultThresholds() Thresholds
DefaultThresholds is the pinned configuration.
type Tier ¶
type Tier string
Tier is a rung of the ladder.
const ( // TierFetch is a plain HTTP GET with static extraction. Sub-second. TierFetch Tier = "fetch" // TierRender loads the page in a browser and captures once after settle. TierRender Tier = "render" // TierSweep is the full checkpoint sweep. TierSweep Tier = "sweep" // TierRecover adds canvas recovery on top of the sweep. TierRecover Tier = "recover" )