static

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 17, 2026 License: Apache-2.0 Imports: 10 Imported by: 0

Documentation

Overview

Package static extracts content from served HTML without a browser.

This is tier 0 of the escalation ladder, and it is what stops sieve from being "an expensive tool that is overkill for a blog". A plain GET plus this package answers a documentation page in under a second. The browser is reserved for pages that actually need it, and the decision is recorded in the artifact so a caller can always see which tier answered.

The extraction here is deliberately conventional: readability-style scoring over the DOM, boilerplate removal, and the same block vocabulary the rendered path produces. What matters is not that it is clever but that its output slots into the same graph, so the two tiers cannot drift into producing different shapes for the same page.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func HydrationText

func HydrationText(root *html.Node) []string

HydrationText returns the prose-looking strings a framework shipped as server-rendered state, in document order, deduplicated.

Types

type HydrationLink struct {
	Text string
	Href string
}

HydrationLink is a labelled destination found in a hydration payload.

func HydrationLinks(root *html.Node) []HydrationLink

HydrationLinks returns the labelled destinations a payload carries.

Frameworks serialise a link as its label and its href, and in a flat payload array those land next to each other: "Ethereum" followed by "https://ethereum.foundation/". That adjacency is the whole rule, and it is deliberately the only one -- resolving Nuxt's index-reference format properly would mean reimplementing devalue, and guessing at structure any less literally than this is how a config value becomes a link in an artifact.

A label that fails this pairing is simply not harvested. It is not turned into prose, because a bare capitalised word with no sentence around it is exactly what a component name, a route name and an icon name all look like.

type Result

type Result struct {
	Merged *capture.Merged
	// Signals feed the escalation scorer.
	Signals Signals
	// RawHTML is retained for the change-detection ladder and for the token
	// comparison that justifies the project.
	RawHTML string
	// Hydration is the prose a framework shipped as server-rendered state in a
	// typed JSON container. It is not content until something decides the page
	// produced none of its own; see HydrationText.
	Hydration []string
	// HydrationLinks are the labelled destinations from the same payload.
	HydrationLinks []HydrationLink
}

Result is a static extraction, shaped so the graph builder can consume it exactly as it consumes a rendered capture.

func Extract

func Extract(pageURL string, body io.Reader, sizeHint int) (*Result, error)

Extract parses served HTML into the same shape a rendered sweep produces.

type Signals

type Signals struct {
	// HTMLBytes is the size of the served document.
	HTMLBytes int
	// TextChars is how much readable text static extraction found.
	TextChars int
	// MarkupChars is how much text a naive tag-strip finds in the same bytes.
	//
	// The two together say whether static extraction is getting the page or
	// only part of it. TextChars alone cannot: three thousand characters reads
	// as a substantial page and the escalator was scoring it that way, while a
	// tag-strip of the same document found nine thousand. cuberto.com lost
	// every one of its service descriptions that way -- the escalator saw
	// "substantial text served statically", stayed at tier 0, and returned 25
	// of 34 ground-truth facts where the browser returns all 34.
	//
	// This is a crude count on purpose. It includes text a visitor never sees,
	// which is exactly why it is a floor for suspicion rather than a target:
	// the question it answers is "is there plainly more here than we extracted",
	// and only a large shortfall is worth a browser.
	MarkupChars int
	// TextRatio is TextChars over HTMLBytes. A documentation page is text with
	// a little markup around it; an application shell is markup with no text.
	TextRatio float64
	// Headings, Paragraphs and Landmarks measure structural richness. A page
	// that arrives with a real outline probably does not need rendering.
	Headings   int
	Paragraphs int
	Landmarks  int
	// Links and Images are counted because a page with many links and no prose
	// is usually an index, which static extraction handles fine.
	Links  int
	Images int
	// CanvasElements and CanvasWithSize hint at a WebGL-driven page before any
	// browser has run.
	CanvasElements int
	// ScriptBytes is how much JavaScript the page ships. A megabyte of bundle
	// with two kilobytes of text is the signature of a client-rendered site.
	ScriptBytes int
	// NoScriptWarning is set when the page ships a "you need JavaScript"
	// message, which is an explicit statement that static extraction will fail.
	NoScriptWarning bool
	// HydrationBlob is set when a framework payload is present, which means the
	// served HTML may be a shell.
	HydrationBlob bool
	// Title and Description come from the head.
	Title       string
	Description string

	// ExternalStylesheets and ComplexHidingRules record how much of the page's
	// visibility logic the static tier could not evaluate.
	//
	// This is the honest counterpart to the CSS scan: an external stylesheet is
	// not fetched, and a selector with a combinator is not resolved. Reporting
	// the counts lets the escalation scorer treat a page whose hiding logic is
	// out of reach as one worth rendering, instead of quietly assuming
	// everything it could not analyse was visible.
	ExternalStylesheets int
	ComplexHidingRules  int

	// FoldedControls and FoldedChars count the disclosures the served document
	// ships shut, and how much text they hold.
	//
	// This is the one shortfall tier 0 can see coming and can never repair. A
	// closed <details>, a panel with aria-hidden="true", an unselected tab: the
	// text is in the bytes, so MarkupChars counts it and no shortfall is
	// detected, but a reader gets it by pressing something and tier 0 has
	// nothing to press with. The page then scores as a well-served static
	// document while a third of what it says stays folded.
	//
	// Without this signal the disclosure prober was mostly unreachable in
	// practice. It only ran on pages that had already escalated for some other
	// reason, so a small, fast, entirely static page whose pricing sits behind
	// a tab -- which describes a great many real pages -- was answered from
	// tier 0 with the pricing missing and no indication anything was.
	FoldedControls int
	FoldedChars    int

	// TextRuns and ShortRuns detect split text: markup where an animation
	// library has shattered a heading into one element per character or per
	// word so each can be tweened separately.
	//
	// This is the one thing tier 0 cannot repair. Putting the pieces back
	// together needs the rendered line box and the computed style -- which
	// fragments shared a line, in what order they sat on it -- and the served
	// HTML has neither, so tier 0 emits the fragments in document order. When
	// the library also reorders the source for its effect, the result is not
	// merely ugly, it is wrong: organimo.com yields "Liitless m", "Te real h"
	// and a heading spelled out as "e / v / er / n / eed."
	//
	// Every other escalation signal here asks whether the text is present.
	// This one asks whether it is legible, which is a different question and
	// the only one that catches a page that serves all of its copy and none of
	// it readably.
	TextRuns  int
	ShortRuns int
}

Signals are the measurements the escalation scorer reads.

They are all cheap and all computed from the served bytes, which is the whole point: the decision about whether to spend thirty seconds on a browser has to cost far less than thirty seconds.

func (Signals) IsShell

func (s Signals) IsShell() bool

IsShell reports that the served document is an application shell: markup with a mount point and no content in it.

This is the condition behind the most deceptive failure in the category. A React or Next.js page that has not hydrated answers with a valid 200 and valid HTML carrying no text, so nothing in the response says the read failed, and an agent handed it either reports the site as empty or invents something. Naming the condition here keeps the threshold beside the signals it reads rather than copied into every caller that wants to ask.

A hydration payload is corroborating rather than sufficient: plenty of server-rendered pages ship one alongside their content, so it only counts when the text is missing too.

func (Signals) SplitTextRatio

func (s Signals) SplitTextRatio() float64

SplitTextRatio is the share of text runs short enough to be fragments.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL