Documentation
¶
Index ¶
- Variables
- func ApplyEntityExtraction(rawHTML string, rules *schema.ExtractionRules) map[string]any
- func ApplyNextDataPaths(pageProps map[string]any, paths map[string]string) map[string]any
- func ApplyTableRowsExtraction(rawHTML string, cfg *schema.TableRowsConfig) []map[string]any
- func ExtractEmbeddedScripts(rawHTML string) map[string]any
- func ExtractSinglePattern(rawHTML string, name string) any
- func ExtractStructured(rawHTML string, baseURL string) map[string]any
- func FetchHTML(ctx context.Context, targetURL string, proxyURL string, insecure bool) (string, error)
- func FetchHTMLWithClient(ctx context.Context, client httpclient.Doer, targetURL string) (string, error)
- func MarshalStructured(data map[string]any) ([]byte, error)
- func ValidateSelectors(rules *schema.ExtractionRules) error
- type CleanContent
- type ExtractionResult
- type Form
- type FormField
- type Heading
- type Image
- type Link
- type PageContent
- type PatternInfo
Constants ¶
This section is empty.
Variables ¶
var ErrNoContentMatch = strings.NewReader("") // sentinel, compared by identity
ErrNoContentMatch signals that the content selector matched nothing, indicating the cached rules are stale (site redesign).
Functions ¶
func ApplyEntityExtraction ¶
func ApplyEntityExtraction(rawHTML string, rules *schema.ExtractionRules) map[string]any
ApplyEntityExtraction applies entity field selectors to HTML and returns a compact map[string]any with the extracted data. Routes to the appropriate strategy (css_selector or table_rows).
func ApplyNextDataPaths ¶
ApplyNextDataPaths extracts named sub-trees from a __NEXT_DATA__ pageProps map. Each key in paths is a human-readable alias, each value is a dot-path (e.g. ".initialZustandState.navigationData"). Returns a compact map keyed by alias. Paths that don't resolve are silently omitted. If ALL paths fail, returns nil so the caller can fall back to the full payload.
func ApplyTableRowsExtraction ¶
func ApplyTableRowsExtraction(rawHTML string, cfg *schema.TableRowsConfig) []map[string]any
ApplyTableRowsExtraction extracts structured data from table-based layouts where one logical item spans multiple consecutive <tr> rows.
For example, Hacker News uses 3 <tr> rows per story:
- Row 0: rank, title, URL
- Row 1: score, author, time, comments
- Row 2: spacer
The config specifies group_size=3 and maps each field to its row index.
func ExtractEmbeddedScripts ¶ added in v0.1.5
ExtractEmbeddedScripts scans all <script> tags in the HTML for known embedded data patterns (SSR state, hydration data, etc.). Returns a map of pattern name -> parsed JSON data.
func ExtractSinglePattern ¶ added in v0.1.5
ExtractSinglePattern extracts one named pattern from the HTML. Stops walking the DOM as soon as the pattern is found.
func ExtractStructured ¶
ExtractStructured produces compact, agent-friendly output from HTML. It combines deterministic parsing (meta, OG, JSON-LD) with main content extraction, preferring JSON-LD as the richest structured data source.
func FetchHTML ¶
func FetchHTML(ctx context.Context, targetURL string, proxyURL string, insecure bool) (string, error)
FetchHTML performs an HTTP GET and returns the response body as a string. It uses a browser-like User-Agent to avoid bot-detection blocks.
func FetchHTMLWithClient ¶
func FetchHTMLWithClient(ctx context.Context, client httpclient.Doer, targetURL string) (string, error)
FetchHTMLWithClient is like FetchHTML but accepts a caller-provided HTTP client.
func MarshalStructured ¶
MarshalStructured converts a map[string]any to compact JSON bytes.
func ValidateSelectors ¶
func ValidateSelectors(rules *schema.ExtractionRules) error
ValidateSelectors checks that all selectors in the rules compile with cascadia.
Types ¶
type CleanContent ¶
type CleanContent struct {
Title string `json:"title"`
Author string `json:"author,omitempty"`
Date string `json:"date,omitempty"`
Content string `json:"content"`
Fields map[string]string `json:"fields,omitempty"`
}
CleanContent is the structured output from CSS selector-based extraction.
func ExtractWithRules ¶
func ExtractWithRules(rawHTML string, baseURL string, rules *schema.ExtractionRules) (*CleanContent, error)
ExtractWithRules applies cached CSS selectors to extract clean content. Returns nil CleanContent if the content_selector matches nothing.
type ExtractionResult ¶
type ExtractionResult struct {
Data map[string]any
MissingRequired []string // names of required fields that returned empty
}
ExtractionResult holds the extracted data and any missing required fields.
func ApplyEntityExtractionWithValidation ¶
func ApplyEntityExtractionWithValidation(rawHTML string, rules *schema.ExtractionRules) ExtractionResult
ApplyEntityExtractionWithValidation applies extraction rules and also reports which required fields are missing (for triggering re-discovery). Routes to the appropriate strategy based on rules.Strategy.
type Form ¶
type Form struct {
Name string `json:"name,omitempty"`
Method string `json:"method"`
Action string `json:"action"`
Fields []FormField `json:"fields,omitempty"`
}
Form represents an HTML form that may be executable without a browser.
type FormField ¶
type FormField struct {
Name string `json:"name"`
Type string `json:"type,omitempty"`
Value string `json:"value,omitempty"`
Required bool `json:"required,omitempty"`
Options []string `json:"options,omitempty"`
}
FormField represents one input within a form.
type PageContent ¶
type PageContent struct {
Title string `json:"title"`
Description string `json:"description,omitempty"`
Language string `json:"language,omitempty"`
Canonical string `json:"canonical,omitempty"`
OpenGraph map[string]string `json:"open_graph,omitempty"`
Meta map[string]string `json:"meta,omitempty"`
JSONLD []any `json:"json_ld,omitempty"`
NextData any `json:"next_data,omitempty"` // __NEXT_DATA__ from Next.js SSR
EmbeddedScripts map[string]any `json:"embedded_scripts,omitempty"` // All detected embedded data patterns
Headings []Heading `json:"headings,omitempty"`
Links []Link `json:"links,omitempty"`
Images []Image `json:"images,omitempty"`
Forms []Form `json:"forms,omitempty"`
BodyText string `json:"body_text"`
HasArticle bool `json:"has_article,omitempty"`
}
PageContent is the structured data extracted from an HTML page.
func Extract ¶
func Extract(rawHTML string, baseURL string) PageContent
Extract parses an HTML string and returns structured page content. It never returns an error — partial results are returned on malformed HTML.
type PatternInfo ¶ added in v0.1.5
PatternInfo describes a known pattern for listing.
func ListPatterns ¶ added in v0.1.5
func ListPatterns() []PatternInfo
ListPatterns returns deduplicated metadata for all known embedded data patterns.
Source Files