Documentation
¶
Overview ¶
Package html reads HTML payloads using github.com/PuerkitoBio/goquery.
The reader extracts visible text from an HTML document. Two modes:
- Whole-document mode (default): a single *document.Document containing the body text with title / description / canonical URL stamped as metadata.
- Selector mode (opt in via ReaderConfig.Selector): emits one document per element matched by the CSS selector — useful for scraping blog post lists, search results, etc.
Example:
r, _ := html.NewReader(strings.NewReader(htmlSrc), html.ReaderConfig{})
docs, _ := r.Read(ctx)
r, _ := html.NewReader(strings.NewReader(htmlSrc),
html.ReaderConfig{Selector: "article"})
docs, _ := r.Read(ctx) // one doc per <article>
Index ¶
Constants ¶
const ( MetadataTitle = "html.title" MetadataDescription = "html.description" MetadataCanonical = "html.canonical" MetadataSelector = "html.selector" MetadataSourceName = "html.source" )
These keys name the reader-derived metadata attached to every emitted document, so a downstream splitter or retriever can rely on them being present rather than re-deriving them from content.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader extracts documents from HTML.
func NewReader ¶
func NewReader(source io.Reader, config ReaderConfig) (*Reader, error)
NewReader compiles the selector at construction so an invalid one fails where it is configured rather than on every document. Unlike PDF, the budget is enforced while reading because an io.Reader cannot report its length in advance.
type ReaderConfig ¶
type ReaderConfig struct {
Selector string
SourceName string
PreserveWhitespace bool
Metadata coremetadata.Map
SourceBudget etl.SourceBudget
}
ReaderConfig controls HTML extraction. By default whitespace runs are collapsed; PreserveWhitespace retains the source spacing instead. Metadata is cloned by NewReader, and reader-derived html.* keys take precedence on conflict. A zero SourceBudget uses etl.DefaultMaxSourceBytes.
Source Files