Documentation
¶
Overview ¶
Package htmlextract picks values out of an HTML document with CSS selectors, within fixed bounds. It does no networking: callers hand it a body they fetched (txco://html/extract uses chassis/fetch).
The document is untrusted, and selector matching can't be interrupted once it starts, so the work is bounded up front instead: the selector grammar is limited to features whose cost per element is constant or one walk up the tree, and the document's tokens, nodes, depth and attributes are capped, with an estimate of the total matching work checked before any matching runs.
Index ¶
Constants ¶
const ( CodeInvalidSelectors = "invalid_selectors" CodeParseFailed = "parse_failed" CodeDocumentTooComplex = "document_too_complex" CodeOutputLimit = "output_limit_exceeded" CodeTimeout = "timeout" // Read's own (read.go): the root selector matched nothing; the part // asked for is past the page's last; a request field is malformed. CodeRootNotFound = "root_not_found" CodeNoSuchPart = "no_such_part" CodeInvalidRequest = "invalid_request" )
Error codes.
const DefaultPartChars = 3000
DefaultPartChars is the part size when the caller names none: what one turn of a model's loop comfortably takes.
const MaxPartChars = 20000
MaxPartChars is the largest part a caller may ask for.
const MinPartChars = 200
MinPartChars is the smallest.
Variables ¶
This section is empty.
Functions ¶
func ParseRootSelector ¶ added in v0.2.49
func ParseRootSelector(sel string, lim Limits) (cascadia.SelectorGroup, error)
ParseRootSelector reads Read's optional root selector under the same grammar as a field's (one descendant space, no :has(), …). An empty selector is nil: the reader picks main, article or body itself.
Types ¶
type Document ¶ added in v0.2.49
Document is a parsed page: the tree, every element in document order, and the counts the caps were checked against. One bounded parse serves Extract and Read.
type Error ¶
Error is a refused spec or document. Field names the selector field a spec error is about.
type Limits ¶
type Limits struct {
MaxFields int // selector fields per call
MaxSelectorBytes int // one selector's length
MaxMatches int // a field's `limit` ceiling
DefaultMatches int // a field's `limit` when `all` is set without one
MaxValueBytes int // one value; longer ones are cut (truncated)
MaxOutputBytes int // the whole data object; more fails
MaxTokens int // tokens in the document
MaxNodes int // nodes in the parsed tree
MaxBuildNodes int // the prescan's estimate of nodes and attributes the parser will create
MaxDepth int // nesting of the parsed tree
MaxAttrs int // attributes on one element
MaxWork int64 // estimated selector work: nodes × per-field weight
MaxTextVisits int // nodes walked collecting text, across all fields
// MaxReadBytes caps the Markdown Read renders from a page; past it the
// rendering stops and the result is truncated.
MaxReadBytes int
// MaxTime is the CPU budget for parsing and matching; a document that
// takes longer is document_too_complex. 0 means only ctx bounds it.
MaxTime time.Duration
}
Limits bound one extraction. The zero value is not usable; start from DefaultLimits.
func DefaultLimits ¶
func DefaultLimits() Limits
DefaultLimits are the chassis ceilings, sized for the 3 MiB fetch cap: a dense 3 MiB page of ordinary markup (about 530k tokens, 380k nodes) fits, and costs about 80 MB and a quarter of a second to parse and match (bench_test.go). Callers may lower them; the handler never raises them.
type ReadOptions ¶ added in v0.2.49
type ReadOptions struct {
// Root is a CSS selector naming the element to read (the first match
// in document order). Empty: <main>, else <article>, else <body>.
Root string
// Part is which part of the text to answer, from 1 (0 is 1); Last
// asks for the last part instead.
Part int
Last bool
// PartChars is the most characters in one part. 0 is DefaultPartChars.
PartChars int
// BaseURL is the page's own address (after redirects), for resolving
// the links the text keeps. A <base href> in the page refines it.
BaseURL string
}
ReadOptions say what to read and which part to answer.
type ReadResult ¶ added in v0.2.49
type ReadResult struct {
Title string // the page's <title>, whitespace collapsed
Root string // what was read: "main", "article", "body", "document", or the selector
Part int // the part answered (0 when the page has no text)
Parts int // how many parts the text has
Text string // the part
Chars int // characters in the part
TotalChars int // characters in the whole text
Truncated bool // the rendering stopped at a cap: the parts are of what was rendered
Nodes int
Depth int
Charset string
}
ReadResult is one read: the part asked for and what the page is.
func Read ¶ added in v0.2.49
func Read(ctx context.Context, body []byte, contentType string, opts ReadOptions, lim Limits) (*ReadResult, error)
Read parses body and renders the root as Markdown within lim, then answers the part opts asks for. A part past the last is CodeNoSuchPart, with the result's Parts filled in so the caller can say how many there are. A root selector that matches nothing is CodeRootNotFound.
type Result ¶
type Result struct {
Data json.RawMessage
Truncated bool // a value was cut at MaxValueBytes, or text collection hit MaxTextVisits
Nodes int
Depth int
Charset string
}
Result is one extraction: the data object, fields in spec order.
func Extract ¶
func Extract(ctx context.Context, body []byte, contentType string, specs []Spec, lim Limits) (*Result, error)
Extract parses body (decoded to UTF-8 from the charset the header or the document declares) and evaluates every spec against it, within lim.MaxTime of CPU and ctx's deadline. Every phase checks the budget as it goes — the parser through the reader it pulls from, the walks every few thousand nodes — so nothing outlives it by more than a moment.
Source Files