capture

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 17, 2026 License: Apache-2.0 Imports: 7 Imported by: 0

Documentation

Overview

Package capture defines the wire format produced by the in-page extraction script and the merge logic that folds a sequence of scroll checkpoints into a single deduplicated set of nodes.

The JSON field names are deliberately one or two characters. A single checkpoint on a heavy site can carry tens of thousands of nodes, and the payload crosses the CDP boundary as a string; short keys cut that transfer by roughly a third with no loss of meaning.

Index

Constants

View Source
const MinVisibleOpacity = 0.12

MinVisibleOpacity is the opacity a run must reach at some point in the sweep to count as content.

It is defined here and consumed by the graph rather than duplicated, because the retention audit divides what the graph emitted by what the capture observed. Two thresholds that drifted apart would make that ratio compare different populations and quietly stop meaning anything.

The value is not 1.0 because a great deal of legitimate body copy is set at 0.7 or 0.8 opacity as a design choice.

Variables

View Source
var Bootstrap string

Bootstrap is installed with Page.addScriptToEvaluateOnNewDocument so it runs before any page script, in every frame, on every navigation. It only hooks what cannot be recovered after the fact.

View Source
var Script string

Script defines window.__sieve in the page. It is evaluated once per document after load and then called once per checkpoint.

Functions

func ScriptHash

func ScriptHash() string

ScriptHash identifies the extraction script that produced a capture.

It belongs in every trace. The script is the single largest determinant of what a render sees, and an artifact recorded under one version of it is not comparable with one recorded under another. Without this, a golden-file corpus silently drifts the first time the walk changes.

Types

type Accumulator

type Accumulator struct {
	// contains filtered or unexported fields
}

Accumulator folds snapshots into a Merged as they arrive, so a sweep never holds more than one checkpoint's raw nodes in memory at a time. On a large site the raw capture is 30-50x the size of the deduplicated result.

func NewAccumulator

func NewAccumulator() *Accumulator

func (*Accumulator) Add

func (a *Accumulator) Add(s *Snapshot) int

Add folds one checkpoint in and reports how many nodes were new.

func (*Accumulator) NodeCount

func (a *Accumulator) NodeCount() int

func (*Accumulator) Result

func (a *Accumulator) Result() *Merged

Result finalises the accumulation. Nodes come out in document order by first-sighting checkpoint, which gives the ordering pass a deterministic starting permutation regardless of map iteration.

func (*Accumulator) StableFor

func (a *Accumulator) StableFor(k int) bool

StableFor reports whether the last k checkpoints all added nothing new. This is the sweep's termination condition.

type Action

type Action struct {
	Path     string  `json:"p"`
	Kind     string  `json:"k"`           // link | button | form
	Label    string  `json:"l"`           // accessible name
	Href     string  `json:"h,omitempty"` // absolute URL for links, form action for forms
	Method   string  `json:"m,omitempty"` // GET | POST
	Fields   []Field `json:"f,omitempty"` // form controls
	BBox     Box     `json:"bb"`
	Landmark string  `json:"lm,omitempty"`
	Disabled bool    `json:"dis,omitempty"`
}

Action is a link, button or form discovered in the page. The PRD treats this as first-class output: an agent that can read a page but cannot see that a contact form exists has only done half the job.

type Box

type Box [4]float64

Box is a rectangle in document space: x, y, width, height. Document space means the page origin, not the viewport, so boxes are directly comparable across checkpoints.

func (Box) Area

func (b Box) Area() float64

func (Box) Bottom

func (b Box) Bottom() float64

func (Box) CenterY

func (b Box) CenterY() float64

CenterY is the vertical midpoint, used by the ordering pass so that two runs on the same visual line sort together even when their heights differ.

func (Box) H

func (b Box) H() float64

func (Box) Right

func (b Box) Right() float64

func (Box) W

func (b Box) W() float64

func (Box) X

func (b Box) X() float64

func (Box) Y

func (b Box) Y() float64

type Canvas

type Canvas struct {
	Path string `json:"p"`
	BBox Box    `json:"bb"`
	// ViewBox is the same rectangle in viewport coordinates, which is what a
	// clipped screenshot needs.
	ViewBox Box `json:"vb"`
	// ViewportShare is the fraction of the viewport this canvas covered at the
	// checkpoint where it was largest.
	ViewportShare float64 `json:"vs"`
	// Context is the rendering context the page acquired, when detectable:
	// webgl, webgl2, 2d, or empty.
	Context string `json:"cx,omitempty"`
	Label   string `json:"l,omitempty"` // aria-label or title, free text if present
	// Fallback is the canvas element's child content, which is what a screen
	// reader is given. A well-built WebGL site puts a real description there,
	// authored rather than inferred, and it is the cheapest recovery available.
	Fallback string `json:"fb,omitempty"`
	// Blank is set when the canvas rasterised to a single flat colour, which
	// means there is nothing for vision to describe.
	Blank bool `json:"bl,omitempty"`
}

Canvas is a canvas element and the evidence needed to decide whether it is worth spending a vision call on.

type Disclosure

type Disclosure struct {
	Label string `json:"l"`
	Kind  string `json:"k"` // tab | disclosure | details | control
	// Expanded is nil when the control does not declare aria-expanded.
	Expanded *bool `json:"e,omitempty"`
	Selected bool  `json:"s,omitempty"`
}

Disclosure is a widget that reveals hidden content.

type Field

type Field struct {
	Name     string   `json:"n"`
	Type     string   `json:"t"`
	Label    string   `json:"l,omitempty"`
	Required bool     `json:"r,omitempty"`
	Options  []string `json:"o,omitempty"`
	Pattern  string   `json:"pt,omitempty"`
}

Field is one form control.

type LatentNode

type LatentNode struct {
	Path  string `json:"p"`
	Block string `json:"bp"`
	Tag   string `json:"t"`
	Text  string `json:"x"`

	Role     string `json:"r,omitempty"`
	Landmark string `json:"lm,omitempty"`
	Href     string `json:"h,omitempty"`

	// Reason is why this text was not rendered: display-none today, with room
	// for further mechanisms.
	Reason string `json:"why"`
	// ControlLabel is the accessible name of the widget that would reveal this
	// content -- the tab, the accordion header, the <summary>. It is what lets
	// an artifact say "there is a section behind a tab labelled Pricing"
	// instead of silently omitting it.
	ControlLabel string `json:"cl,omitempty"`
	// ControlKind is tab, disclosure, details or control.
	ControlKind string `json:"ck,omitempty"`

	Depth int `json:"d"`

	// Checkpoint is filled in by the sweep.
	Checkpoint int `json:"-"`
}

LatentNode is text that exists in the document but was never rendered.

This is the quarantine. It holds exactly the material the visibility filter exists to exclude, because that material is not one thing: a collapsed accordion body is content a reader can reach with one click, and an off-screen instruction aimed at an AI agent is an attack. Both look identical to a walker, so neither is discarded and neither is mixed into the content tier. Nothing here ever reaches a default payload.

type Media

type Media struct {
	Path      string `json:"p"`
	Kind      string `json:"k"` // image | video | model
	Src       string `json:"s"`
	Alt       string `json:"a,omitempty"`
	AltCapped bool   `json:"ac,omitempty"` // alt text was truncated at the metadata cap
	Title     string `json:"ti,omitempty"`
	Caption   string `json:"cp,omitempty"`
	BBox      Box    `json:"bb"`
	// Decorative marks images the page itself flags as presentational
	// (role="presentation" or an explicitly empty alt).
	Decorative bool `json:"dec,omitempty"`
}

Media is an image, video or 3D model reference.

type Merged

type Merged struct {
	Nodes       []Node
	Latent      []LatentNode
	Actions     []Action
	Media       []Media
	Canvases    []Canvas
	Disclosures []Disclosure
	Meta        Meta

	Checkpoints   int
	DocHeight     float64
	ViewportW     float64
	ViewportH     float64
	FramesRead    int
	FramesBlocked int
	Truncated     bool
	// LatentTruncated reports that the hidden-content budget was exhausted, so
	// the latent tier is itself incomplete.
	LatentTruncated bool

	// ObservedVisibleChars is the largest number of readable characters the
	// browser had on screen at any single checkpoint, summed across the sweep
	// as new content appeared. It is the denominator for graph retention.
	ObservedVisibleChars int

	// NewPerCheckpoint records how many previously unseen nodes each checkpoint
	// contributed. The sweep uses it to decide when to stop, and the artifact
	// reports it because a long tail of zeros is evidence the sweep was
	// complete rather than merely budget-limited.
	NewPerCheckpoint []int
}

Merged is the union of every checkpoint, with each distinct piece of content present exactly once.

func (*Merged) RevealsOnScroll

func (m *Merged) RevealsOnScroll() bool

RevealsOnScroll reports whether this page was seen animating text into view.

It counts runs that were observed below the visible-opacity floor at one checkpoint and above it at another. Such a run was revealed while sieve was watching, which makes it evidence about the page's mechanism rather than an inference about its markup -- and it is the only evidence available on a site that drives its reveals from JavaScript, where the computed style declares no transition and no animation because there is none: a script is simply writing a new opacity every frame.

The threshold is small but not one. A single run crossing the floor could be a cross-fade or a hover; several is how the page works.

type Meta

type Meta struct {
	Title       string            `json:"ti"`
	Lang        string            `json:"lg,omitempty"`
	Description string            `json:"de,omitempty"`
	Canonical   string            `json:"ca,omitempty"`
	URL         string            `json:"u"`
	OpenGraph   map[string]string `json:"og,omitempty"`
	// JSONLD is raw structured-data text. It never renders, so it is a pure
	// metadata channel and is never emitted as-is: only a whitelisted set of
	// schema.org fields is read out of it.
	JSONLD []string `json:"ld,omitempty"`
}

Meta carries page-level facts that only the page can report.

type Node

type Node struct {
	Path  string `json:"p"`  // stable structural path, e.g. "html/body/div[2]/p[0]"
	Block string `json:"bp"` // path of the nearest block-level ancestor
	Tag   string `json:"t"`  // lowercase tag name
	Text  string `json:"x"`  // whitespace-normalised own text

	Role string `json:"r,omitempty"`  // explicit ARIA role
	Aria string `json:"al,omitempty"` // aria-label, when it differs from text
	// Landmark is the nearest landmark ancestor: nav, header, footer, main,
	// aside, form, dialog.
	//
	// <header> and <footer> are resolved during the walk rather than reported
	// raw. A document has at most one banner and one contentinfo, so only the
	// first page-level header and the last page-level footer are landmarks; the
	// rest are section furniture, which is content. See countPageLandmarks.
	Landmark string `json:"lm,omitempty"`
	Href     string `json:"h,omitempty"` // resolved href when the node is inside a link

	FontSize   float64 `json:"fs"`           // px
	Weight     int     `json:"fw"`           // 100..900, normalised from keywords
	Tracking   float64 `json:"ls"`           // letter-spacing in px, 0 for "normal"
	LineHeight float64 `json:"lh,omitempty"` // px
	Family     string  `json:"ff,omitempty"` // first font family only
	Transform  string  `json:"tt,omitempty"` // text-transform, empty when "none"
	Color      string  `json:"c,omitempty"`  // rgb/rgba as serialised by the engine
	Italic     bool    `json:"it,omitempty"`

	// Opacity is the product of the element's own opacity and every ancestor's,
	// computed on the way down the tree. Scroll-reveal animations run this from
	// 0 to 1, so the maximum observed across checkpoints is the signal that
	// matters, not the value at any one checkpoint.
	Opacity float64 `json:"o"`
	// Visible reports whether the element was inside the viewport with a
	// non-degenerate box, a visible `visibility` chain and no `display:none`
	// ancestor at this checkpoint.
	Visible bool `json:"v"`
	// InvisibleColor marks text whose colour is indistinguishable from what is
	// behind it. Opacity and visibility are the two hiding techniques the
	// capture already sees; matching the text colour to the background is the
	// third, and it defeats both.
	InvisibleColor bool `json:"iv,omitempty"`

	// Pad records whitespace that surrounded this fragment in the source before
	// normalisation: bit 1 for leading, bit 2 for trailing. It matters only for
	// fragments of mixed content, where an inline link abuts the text beside it
	// and geometry alone cannot say whether a space was written between them.
	Pad int `json:"pd,omitempty"`

	// Revealable marks a run that is not currently legible but whose element or
	// an ancestor declares a transition or animation that would make it so.
	//
	// It is the page's own statement of intent, read from computed style: an
	// author writes `transition: opacity .8s` on a section because the section
	// is meant to appear. Text hidden in order to stay hidden carries no such
	// declaration. The flag never promotes anything by itself; it lets the graph
	// tell "waiting to be revealed" apart from "hidden", which are the same
	// thing to an opacity threshold and very different things to a reader.
	Revealable bool `json:"rv,omitempty"`

	// Fixed marks a node inside a position:fixed or position:sticky subtree.
	// Its BBox is in viewport coordinates, not document coordinates, because a
	// pinned element has no single document position. It is also the strongest
	// single signal of chrome that the page gives us.
	Fixed bool `json:"fx,omitempty"`

	BBox Box `json:"bb"`
	// LineTop is the rounded top edge of the element's first client rect. Runs
	// that share a block ancestor and a LineTop sat on the same rendered line,
	// which is how a heading shattered into per-character spans is put back
	// together.
	LineTop float64 `json:"lt"`
	// LineLeft is the left edge of that same first client rect: where the run
	// starts reading, as opposed to how far it extends.
	//
	// The two differ whenever a run wraps. BBox is the union of every line, so
	// its left edge is the left margin as soon as any line but the first begins
	// there -- which makes a wrapped run sort as though it started at the
	// margin, ahead of everything that really precedes it on its own first
	// line. Ordering by this instead is what keeps an inline link inside the
	// sentence it interrupts.
	LineLeft float64 `json:"lx"`

	// Depth is the element's depth in the composed tree, used as a tiebreaker
	// when two runs occupy the same geometry.
	Depth int `json:"d"`

	// Checkpoint records the first checkpoint at which this node was seen.
	Checkpoint int `json:"cp,omitempty"`
	// MaxOpacity, MinOpacity and EverVisible accumulate across checkpoints
	// during merge.
	//
	// MinOpacity exists to catch a reveal in the act. A run that was seen at
	// zero and later at one was animated into view while sieve was watching,
	// and that is direct evidence -- not a declaration, not a guess -- that this
	// page brings its text in by animation. It is the only signal that works on
	// a page whose reveals are driven from JavaScript, where nothing in the
	// computed style says anything is going to happen.
	MaxOpacity  float64 `json:"mo,omitempty"`
	MinOpacity  float64 `json:"mio,omitempty"`
	EverVisible bool    `json:"ev,omitempty"`
	// Seen counts the checkpoints this node appeared in.
	Seen int `json:"sn,omitempty"`
	// CountedVisible records that this run has already been added to the
	// observed-text total, so a run visible across ten checkpoints is counted
	// once.
	CountedVisible bool `json:"cv,omitempty"`
}

Node is one text-bearing element: an element that owns at least one non-empty direct child text node. Elements that only contain other elements are not captured, which is what keeps the node count proportional to visible text rather than to DOM size.

type SceneIntrospection

type SceneIntrospection struct {
	Names []string `json:"n"`
	Texts []string `json:"t"`
	// Runs are the text objects found in the scene, each with the words it was
	// built from, in the order the scene was assembled.
	//
	// Texts flattens the same material into a summary for the canvas recovery
	// tier. This keeps them separate, because a site that draws its whole body
	// copy into WebGL -- igloo.inc draws every paragraph as glyph geometry --
	// deserves its paragraphs as paragraphs rather than one welded blob.
	Runs []SceneRun `json:"r,omitempty"`

	// Observed counts how many times three.js has touched the devtools hook
	// sieve installs. Non-zero means three.js is on the page and running, which
	// is knowable long before it has finished building anything -- so an empty
	// scene with a non-zero count is a scene that is coming, and one with a
	// zero count is a page that has no three.js on it at all. Waiting is worth
	// it in the first case and is pure cost in the second.
	Observed int `json:"o,omitempty"`
}

SceneIntrospection is what walking the live 3D scene produced. It catches procedurally built scenes that never loaded an asset file.

type SceneRun

type SceneRun struct {
	Text string `json:"x"`
	Name string `json:"n,omitempty"`
}

SceneRun is one text object from a 3D scene.

type Snapshot

type Snapshot struct {
	Checkpoint  int          `json:"n"`
	ScrollY     float64      `json:"sy"`
	DocHeight   float64      `json:"dh"`
	ViewportW   float64      `json:"vw"`
	ViewportH   float64      `json:"vh"`
	Nodes       []Node       `json:"nodes"`
	Latent      []LatentNode `json:"latent"`
	Actions     []Action     `json:"actions"`
	MediaItems  []Media      `json:"media"`
	Canvases    []Canvas     `json:"canvases"`
	Disclosures []Disclosure `json:"disc"`
	Meta        *Meta        `json:"meta,omitempty"`
	// VisibleChars is how many characters of readable text the browser had on
	// screen at this checkpoint. It is the denominator for the retention audit.
	VisibleChars int `json:"vc"`
	// Frames counts same-origin iframes that were descended into, and
	// FramesBlocked counts cross-origin ones that could not be.
	Frames        int `json:"fr"`
	FramesBlocked int `json:"frx"`
	// Truncated is set when the node budget was hit and the walk stopped early.
	Truncated bool `json:"tr,omitempty"`
	// LatentTruncated is the same for the latent budget.
	LatentTruncated bool `json:"ltr,omitempty"`
}

Snapshot is one checkpoint's worth of observation.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL