Documentation
¶
Overview ¶
Package htmlextract picks values out of an HTML document with CSS selectors, within fixed bounds. It does no networking: callers hand it a body they fetched (txco://html/extract uses chassis/fetch).
The document is untrusted, and selector matching can't be interrupted once it starts, so the work is bounded up front instead: the selector grammar is limited to features whose cost per element is constant or one walk up the tree, and the document's tokens, nodes, depth and attributes are capped, with an estimate of the total matching work checked before any matching runs.
Index ¶
Constants ¶
const ( CodeInvalidSelectors = "invalid_selectors" CodeParseFailed = "parse_failed" CodeDocumentTooComplex = "document_too_complex" CodeOutputLimit = "output_limit_exceeded" CodeTimeout = "timeout" )
Error codes.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Error ¶
Error is a refused spec or document. Field names the selector field a spec error is about.
type Limits ¶
type Limits struct {
MaxFields int // selector fields per call
MaxSelectorBytes int // one selector's length
MaxMatches int // a field's `limit` ceiling
DefaultMatches int // a field's `limit` when `all` is set without one
MaxValueBytes int // one value; longer ones are cut (truncated)
MaxOutputBytes int // the whole data object; more fails
MaxTokens int // tokens in the document
MaxNodes int // nodes in the parsed tree
MaxBuildNodes int // the prescan's estimate of nodes and attributes the parser will create
MaxDepth int // nesting of the parsed tree
MaxAttrs int // attributes on one element
MaxWork int64 // estimated selector work: nodes × per-field weight
MaxTextVisits int // nodes walked collecting text, across all fields
// MaxTime is the CPU budget for parsing and matching; a document that
// takes longer is document_too_complex. 0 means only ctx bounds it.
MaxTime time.Duration
}
Limits bound one extraction. The zero value is not usable; start from DefaultLimits.
func DefaultLimits ¶
func DefaultLimits() Limits
DefaultLimits are the chassis ceilings, sized for the 3 MiB fetch cap: a dense 3 MiB page of ordinary markup (about 530k tokens, 380k nodes) fits, and costs about 80 MB and a quarter of a second to parse and match (bench_test.go). Callers may lower them; the handler never raises them.
type Result ¶
type Result struct {
Data json.RawMessage
Truncated bool // a value was cut at MaxValueBytes, or text collection hit MaxTextVisits
Nodes int
Depth int
Charset string
}
Result is one extraction: the data object, fields in spec order.
func Extract ¶
func Extract(ctx context.Context, body []byte, contentType string, specs []Spec, lim Limits) (*Result, error)
Extract parses body (decoded to UTF-8 from the charset the header or the document declares) and evaluates every spec against it, within lim.MaxTime of CPU and ctx's deadline. Every phase checks the budget as it goes — the parser through the reader it pulls from, the walks every few thousand nodes — so nothing outlives it by more than a moment.
Source Files