Documentation
¶
Overview ¶
Package html reads HTML payloads with github.com/PuerkitoBio/goquery.
By default the reader emits one document with static body text and the page title, description, and canonical URL as metadata. Hidden subtrees and non-content elements are omitted; CSS and JavaScript are not evaluated. With ReaderConfig.Selector it emits one document per element matched by the CSS selector, subject to the same exclusions, including hidden ancestors.
Index ¶
Examples ¶
Constants ¶
const ( MetadataTitle = "html.title" MetadataDescription = "html.description" MetadataCanonical = "html.canonical" MetadataSelector = "html.selector" )
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader parses the sequential input consumed by etltext.Reader. Reopen the original source for a new attempt; cancellation before reading leaves it intact.
Example ¶
package main
import (
"context"
"fmt"
"strings"
"github.com/Tangerg/scope/etl/html"
)
func main() {
const page = `<html><head><title>Scope</title></head>` +
`<body><h1>Readers</h1><p>Scope turns HTML into documents.</p></body></html>`
reader, err := html.NewReader(strings.NewReader(page), html.ReaderConfig{})
if err != nil {
panic(err)
}
docs, err := reader.Read(context.Background())
if err != nil {
panic(err)
}
values, err := docs[0].Metadata.Values()
if err != nil {
panic(err)
}
fmt.Println(docs[0].Text)
fmt.Println(values[html.MetadataTitle])
}
Output: Readers Scope turns HTML into documents. Scope
type ReaderConfig ¶
type ReaderConfig struct {
Selector string
SourceName string
PreserveWhitespace bool
Metadata coremetadata.Map
SourceBudget etl.SourceBudget
}
ReaderConfig controls HTML extraction. By default whitespace runs are collapsed; PreserveWhitespace retains the source spacing instead. Metadata is cloned by NewReader and cannot contain the reader's title, description, canonical, or selector keys. A zero SourceBudget uses etl.DefaultMaxSourceBytes.
Source Files