parse

package
v0.0.27 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 28, 2026 License: MIT Imports: 20 Imported by: 0

Documentation

Overview

Package parse provides HTML, JSON, and other response parsing utilities for the Foxhound scraping framework.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func DecodeCFEmail 

func DecodeCFEmail(encoded string) string

DecodeCFEmail decodes a CloudFlare-obfuscated email string. CloudFlare XOR-encodes emails: first 2 hex chars are the key, remaining hex pairs are XOR'd with the key.

func DetectFramework 

func DetectFramework(resp *foxhound.Response) string

DetectFramework inspects the response for framework-specific markers and returns the framework name: "nextjs", "nuxt", "react", "vue", "angular", or "unknown".

func ExtractEmails 

func ExtractEmails(resp *foxhound.Response) []string

ExtractEmails finds all email addresses in the response, including CloudFlare cfemail-obfuscated ones, mailto: links, and plaintext.

func ExtractInlineJSON 

func ExtractInlineJSON(resp *foxhound.Response, varPattern string) (any, error)

ExtractInlineJSON finds var {varPattern} = {...} or {varPattern} = {...} in <script> blocks and extracts the JSON object using balanced-brace matching. Returns nil, nil if not found.

func ExtractJSONLD 

func ExtractJSONLD(resp *foxhound.Response) ([]map[string]any, error)

ExtractJSONLD extracts and unmarshals all <script type="application/ld+json"> tags.

func ExtractJSONLDFromDoc 

func ExtractJSONLDFromDoc(doc *goquery.Document) ([]map[string]any, error)

ExtractJSONLDFromDoc is like ExtractJSONLD but operates on an already-parsed goquery.Document, avoiding a redundant HTML parse when the caller has already parsed the response body.

func ExtractMainContent 

func ExtractMainContent(resp *foxhound.Response) string

ExtractMainContent attempts to extract the main content from a page, excluding navigation, sidebars, footers, and other boilerplate.

func ExtractMainContentMarkdown 

func ExtractMainContentMarkdown(resp *foxhound.Response) string

ExtractMainContentMarkdown is like ExtractMainContent but returns Markdown.

func ExtractMeta 

func ExtractMeta(resp *foxhound.Response) map[string]string

ExtractMeta extracts all <meta> name/content pairs.

func ExtractMetaFromDoc 

func ExtractMetaFromDoc(doc *goquery.Document) map[string]string

ExtractMetaFromDoc is like ExtractMeta but accepts an already-parsed document.

func ExtractNextData 

func ExtractNextData(resp *foxhound.Response) (map[string]any, error)

ExtractNextData extracts and unmarshals __NEXT_DATA__ from Next.js pages.

func ExtractNuxtData 

func ExtractNuxtData(resp *foxhound.Response) (map[string]any, error)

ExtractNuxtData extracts window.__NUXT__ data from Nuxt.js pages. Falls back to <script id="__NUXT_DATA__"> if present.

func ExtractOpenGraph 

func ExtractOpenGraph(resp *foxhound.Response) map[string]string

ExtractOpenGraph extracts all og: meta properties from the page.

func ExtractOpenGraphFromDoc 

func ExtractOpenGraphFromDoc(doc *goquery.Document) map[string]string

ExtractOpenGraphFromDoc is like ExtractOpenGraph but accepts an already-parsed document.

func ExtractPhones 

func ExtractPhones(resp *foxhound.Response) []string

ExtractPhones finds phone numbers in the response, including tel: links and plaintext. Numbers are returned as-is (not normalized).

func ExtractWindowVar 

func ExtractWindowVar(resp *foxhound.Response, varName string) (any, error)

ExtractWindowVar extracts a named window variable from the page. Strategy 1: look for <script id="__VARNAME__" type="application/json">. Strategy 2: regex for window.{varName} = {...} with balanced-brace extraction. Returns nil, nil if not found.

func JSON 

func JSON(resp *foxhound.Response, target any) error

JSON parses the response body as JSON into target. target must be a pointer (e.g. *map[string]any or a pointer to a struct).

func JSONPath 

func JSONPath(resp *foxhound.Response, path string) (any, error)

JSONPath extracts a value from the response body using a dot-separated path. For example, "data.items" navigates {"data":{"items":[...]}}. Only map traversal is supported; the final value may be of any JSON type.

func NormalizeAddress 

func NormalizeAddress(raw string) (street, city, state, zip, country string)

NormalizeAddress parses a raw address string into components. It first tries a US-pattern match, then falls back to comma-split heuristics.

func NormalizeRating 

func NormalizeRating(text string) (rating float64, reviewCount int)

NormalizeRating parses a rating string and extracts the numeric rating and optional review count. Handles formats like "4.5 (123 reviews)", "4.5/5", "4.5 stars", and Unicode star characters.

func ParseSitemapIndex 

func ParseSitemapIndex(resp *foxhound.Response) ([]string, error)

ParseSitemapIndex parses a sitemap index file, returning child sitemap URLs.

func RegexExtract 

func RegexExtract(resp *foxhound.Response, pattern string) (string, error)

RegexExtract returns the text of the first match of pattern in the response body. Returns an empty string when there is no match. Returns an error only when pattern is syntactically invalid.

func RegexExtractAll 

func RegexExtractAll(resp *foxhound.Response, pattern string) ([]string, error)

RegexExtractAll returns the text of every non-overlapping match of pattern in the response body. Returns an empty slice when there are no matches. Returns an error only when pattern is syntactically invalid.

func RegexExtractNamed 

func RegexExtractNamed(resp *foxhound.Response, pattern string) (map[string]string, error)

RegexExtractNamed extracts named capture groups from the first match of pattern in the response body. Returns a map of group name → captured text. Returns an empty map when there is no match. Returns an error only when pattern is syntactically invalid.

Example: 

groups, _ := RegexExtractNamed(resp, `Price: (?P<price>\d+\.\d+)`)
fmt.Println(groups["price"]) // "29.99"

func RegexExtractSubmatch 

func RegexExtractSubmatch(resp *foxhound.Response, pattern string) ([]string, error)

RegexExtractSubmatch returns all submatch groups (including the full match at index 0) for the first match of pattern in the response body. Returns an empty slice when there is no match. Returns an error only when pattern is syntactically invalid.

func Similarity 

func Similarity(a, b *ElementSignature) float64

Similarity calculates how similar two element signatures are. The returned value is in the range [0.0, 1.0].

Each signal contributes a maximum weight: 

Tag match:            0.15
ID match:             0.25  (only counted when at least one sig has an ID)
Class Jaccard:        0.15
Text similarity:      0.15
Parent tag match:     0.10
Depth match:          0.10
Position proximity:   0.10

The raw score is normalised by the maximum achievable score given the signals present in both signatures, so that identical elements always return 1.0 regardless of which optional fields are populated.

func ToMarkdown 

func ToMarkdown(resp *foxhound.Response) string

ToMarkdown converts the HTML body of a Response to Markdown format. It handles common HTML elements: headings, paragraphs, links, lists, bold, italic, code blocks, images, blockquotes, and horizontal rules.

Example: 

md := parse.ToMarkdown(resp)

func ToText 

func ToText(resp *foxhound.Response) string

ToText converts the HTML body of a Response to plain text. All HTML tags are stripped and whitespace is normalized.

Example: 

text := parse.ToText(resp)

Types

type AdaptiveExtractor 

type AdaptiveExtractor struct {
	// contains filtered or unexported fields
}

AdaptiveExtractor manages multiple adaptive selectors with optional persistence. It is safe for concurrent use.

func NewAdaptiveExtractor 

func NewAdaptiveExtractor(savePath string) *AdaptiveExtractor

NewAdaptiveExtractor creates an extractor. When savePath is non-empty the constructor attempts to load previously saved signatures from that file; any load error is silently ignored (the file may not exist yet on the first run).

func NewAdaptiveExtractorWithOptions 

func NewAdaptiveExtractorWithOptions(opts ...AdaptiveOption) *AdaptiveExtractor

NewAdaptiveExtractorWithOptions constructs an AdaptiveExtractor with the supplied options. With no options, the returned extractor is in-memory only and Save is a no-op.

This is the preferred constructor for new code. The legacy NewAdaptiveExtractor(savePath string) entry point is retained for backward compatibility and is equivalent to calling this function with WithJSONStorage(savePath).

func (*AdaptiveExtractor) Extract 

func (ae *AdaptiveExtractor) Extract(doc *Document, name string) *Element

Extract tries the CSS selector first. If no element is found and a saved signature exists, it falls back to similarity matching (using MinScore as the threshold, picking the best match). On a successful extraction the element signature is updated for future runs.

Returns nil when neither strategy finds an element.

func (*AdaptiveExtractor) ExtractAll 

func (ae *AdaptiveExtractor) ExtractAll(doc *Document, name string) []*Element

ExtractAll extracts all elements matching the selector. When the selector matches nothing and a signature is available, it returns the single best similarity match wrapped in a slice.

func (*AdaptiveExtractor) ExtractText 

func (ae *AdaptiveExtractor) ExtractText(doc *Document, name string) string

ExtractText is a convenience wrapper that returns the text of the first matched element, or an empty string when nothing is found.

func (*AdaptiveExtractor) Load 

func (ae *AdaptiveExtractor) Load(path string) error

Load reads selector signatures from the given file path and merges them into the extractor. Selectors not already registered are added; existing registrations have their signatures updated without overwriting their current CSS selector.

func (*AdaptiveExtractor) Register 

func (ae *AdaptiveExtractor) Register(name, selector string) *AdaptiveExtractor

Register adds an adaptive selector by name with its primary CSS selector. Returns the extractor for method chaining.

func (*AdaptiveExtractor) Save 

func (ae *AdaptiveExtractor) Save() error

Save persists all selector signatures to the configured savePath as JSON. When savePath is empty, Save is a no-op and returns nil.

type AdaptiveOption 

type AdaptiveOption func(*adaptiveConfig)

AdaptiveOption configures an AdaptiveExtractor created via NewAdaptiveExtractorWithOptions. Options are applied in order; later options override earlier ones when they touch the same field.

func WithJSONStorage 

func WithJSONStorage(path string) AdaptiveOption

WithJSONStorage configures the extractor to persist learned signatures as a JSON file at the given path. The file is read on construction (if it exists) and written on Save. An empty path disables persistence.

func WithSQLiteStorage 

func WithSQLiteStorage(path string) AdaptiveOption

WithSQLiteStorage configures the extractor to persist learned signatures in a SQLite database at the given path. The database and schema are created on first use.

Returns an option that opens the SQLite store lazily; any error from opening the database is reported when NewAdaptiveExtractorWithOptions returns. When the store cannot be opened, the option is silently dropped and the extractor falls back to in-memory only.

type AdaptiveSelector 

type AdaptiveSelector struct {
	Name      string            `json:"name"`
	Selector  string            `json:"selector"`
	Signature *ElementSignature `json:"signature,omitempty"`
	MinScore  float64           `json:"min_score"`
}

AdaptiveSelector tracks an element across page changes. When the primary CSS selector fails, it falls back to similarity matching using a saved signature from the last successful extraction.

type AdaptiveStore 

type AdaptiveStore interface {
	// Save persists an element signature for a domain and selector name.
	Save(domain, name string, sig *ElementSignature) error
	// Load retrieves a previously saved signature.
	Load(domain, name string) (*ElementSignature, error)
	// Close releases storage resources.
	Close() error
}

AdaptiveStore is the interface for persisting element signatures. Implementations handle the actual storage mechanism (file, SQLite, etc.).

type Article 

type Article struct {
	Title           string
	Author          string
	PublishedDate   string
	Content         string // cleaned HTML of main content
	ContentText     string // plain text
	ContentMarkdown string // markdown
	Summary         string // first ~200 chars of text
	Images          []string
	Tags            []string
	WordCount       int
	ReadingTime     time.Duration // based on 200 wpm
	Score           float64       // readability confidence score
}

Article represents extracted article content with metadata.

func ExtractArticle 

func ExtractArticle(resp *foxhound.Response) (*Article, error)

ExtractArticle applies a readability-style scoring algorithm to extract the primary article content, metadata, and computed statistics from a response.

type AutoResult 

type AutoResult struct {
	Type     ContentType
	Article  *Article         // populated when Type == ContentArticle
	Listings []Listing        // populated when Type == ContentListing
	Items    []*foxhound.Item // generic extraction for other types
}

AutoResult contains the auto-extracted content based on detected page type.

func AutoExtract 

func AutoExtract(resp *foxhound.Response) (*AutoResult, error)

AutoExtract detects the page content type and extracts content accordingly. For articles it populates AutoResult.Article, for listings AutoResult.Listings, and for other types it returns a generic item with the main content.

type ContentType 

type ContentType int

ContentType classifies the type of content on a web page. 

const (
	ContentUnknown ContentType = iota
	ContentArticle
	ContentListing
	ContentProduct
	ContentSearch
	ContentFeed
)

func DetectContentType 

func DetectContentType(resp *foxhound.Response) ContentType

DetectContentType analyses the HTML in resp and returns the most likely content type using structural signals, JSON-LD types, and text analysis.

func (ContentType) String 

func (ct ContentType) String() string

String returns a human-readable name for the content type.

type Document 

type Document struct {
	// contains filtered or unexported fields
}

Document wraps goquery.Document for convenient CSS-selector-based extraction.

func NewDocument 

func NewDocument(resp *foxhound.Response) (*Document, error)

NewDocument creates a Document from a foxhound Response. The response body is parsed as HTML.

func (*Document) Adaptive 

func (d *Document) Adaptive() *AdaptiveExtractor

Adaptive returns the AdaptiveExtractor associated with this document, if any. Returns nil when no extractor was attached via SetAdaptive.

func (*Document) Attr 

func (d *Document) Attr(selector, attr string) string

Attr extracts an attribute value from the first element matching selector. Returns an empty string if no element matches or the attribute is absent.

func (*Document) Attrs 

func (d *Document) Attrs(selector, attr string) []string

Attrs extracts attribute values from all elements matching selector. Elements where the attribute is absent contribute an empty string.

func (*Document) CSS 

func (d *Document) CSS(selector string) *Results

CSS selects elements with comparable Python framework-compatible pseudo-selector support. 

doc.CSS("h1")             — elements matched by h1
doc.CSS("h1::text")       — trimmed text content of h1 elements
doc.CSS("a::attr(href)")  — href attribute values of a elements

func (*Document) Each 

func (d *Document) Each(selector string, fn func(i int, s *goquery.Selection))

Each iterates over all elements matching selector, calling fn with the zero-based index and the goquery.Selection for each element.

func (*Document) Find 

func (d *Document) Find(selector string) *goquery.Selection

Find returns the goquery Selection for a CSS selector.

func (*Document) FindAll 

func (d *Document) FindAll(selector string) []*Element

FindAll returns all elements matching the CSS selector as []*Element.

func (*Document) FindAllFunc 

func (d *Document) FindAllFunc(fn func(*Element) bool) []*Element

FindAllFunc returns every element in the document for which fn returns true. The document is traversed in DOM order.

func (*Document) FindByAttr 

func (d *Document) FindByAttr(attr, value string) []*Element

FindByAttr returns all elements where the named attribute exactly equals value. Uses CSS attribute selector for O(matched) instead of O(all nodes).

func (*Document) FindByAttrContains 

func (d *Document) FindByAttrContains(attr, substring string) []*Element

FindByAttrContains returns all elements where the named attribute contains the given substring. Uses CSS attribute selector [attr*='substring'].

func (*Document) FindByTag 

func (d *Document) FindByTag(tag string) []*Element

FindByTag returns all elements with the given tag name in document order.

func (*Document) FindByTagAndAttr 

func (d *Document) FindByTagAndAttr(tag string, attrs map[string]string) []*Element

FindByTagAndAttr returns all elements that match both the given tag name and every key-value pair in attrs. An empty attrs map matches any element with the given tag.

func (*Document) FindByText 

func (d *Document) FindByText(text string) []*Element

FindByText returns all elements whose trimmed text content exactly equals the given string.

func (*Document) FindByTextContains 

func (d *Document) FindByTextContains(substring string) []*Element

FindByTextContains returns all elements whose text content contains the given substring.

func (*Document) FindByTextRegex 

func (d *Document) FindByTextRegex(pattern string) []*Element

FindByTextRegex returns all elements whose text content matches the regular expression pattern. An invalid pattern returns nil without panicking.

func (*Document) FindSimilar 

func (d *Document) FindSimilar(sig *ElementSignature, minScore float64) []SimilarMatch

FindSimilar searches the document for elements most similar to the given signature. Returns matches sorted by score descending, filtered by minScore.

func (*Document) First 

func (d *Document) First(selector string) *Element

First returns the first element matching the CSS selector, or nil when there is no match.

func (*Document) HTML 

func (d *Document) HTML(selector string) string

HTML returns the inner HTML of the first element matching selector. Returns an empty string if no element matches.

func (*Document) Response 

func (d *Document) Response() *foxhound.Response

Response returns the original foxhound Response that produced this document.

func (*Document) SetAdaptive 

func (d *Document) SetAdaptive(ae *AdaptiveExtractor)

SetAdaptive attaches an AdaptiveExtractor to this document so callers can retrieve it later via Adaptive() — useful for wiring a Hunt-scoped extractor through to processors that build their own Document.

func (*Document) Text 

func (d *Document) Text(selector string) string

Text extracts trimmed text from the first element matching selector. Returns an empty string if no element matches.

func (*Document) Texts 

func (d *Document) Texts(selector string) []string

Texts extracts trimmed text from all elements matching selector.

func (*Document) URLJoin 

func (d *Document) URLJoin(relative string) string

URLJoin resolves a relative URL against the document's page URL. When the document has no URL, the relative string is returned unchanged.

type Element 

type Element struct {
	// contains filtered or unexported fields
}

Element wraps a goquery.Selection with convenience navigation and extraction methods.

func (*Element) Ancestors 

func (e *Element) Ancestors() []*Element

Ancestors returns all ancestor elements from the direct parent up to the root, in nearest-first order.

func (*Element) Attr 

func (e *Element) Attr(name string) string

Attr returns the value of the named attribute. Returns an empty string when the attribute is absent.

func (*Element) Attrs 

func (e *Element) Attrs() map[string]string

Attrs returns a map of all attribute names to their values.

func (*Element) BelowElements 

func (e *Element) BelowElements() []*Element

BelowElements returns all descendant elements (direct and indirect children) in document order. Text nodes and non-element nodes are excluded.

func (*Element) CSS 

func (e *Element) CSS(selector string) *Element

CSS returns the first descendant matching the CSS selector, or nil when there is no match.

func (*Element) Children 

func (e *Element) Children() []*Element

Children returns the direct child elements.

func (*Element) Clean 

func (e *Element) Clean() string

Clean returns the element's text content with leading/trailing whitespace trimmed and internal runs of whitespace collapsed to a single space.

func (*Element) Find 

func (e *Element) Find(selector string) []*Element

Find returns all descendant elements matching the CSS selector.

func (*Element) FindAncestor 

func (e *Element) FindAncestor(fn func(*Element) bool) *Element

FindAncestor walks up the DOM tree and returns the first ancestor element where fn returns true. Returns nil if no ancestor satisfies fn.

func (*Element) FindSimilar 

func (e *Element) FindSimilar(opts ...SimilarOption) []*Element

FindSimilar finds elements in the document that are structurally similar to this element. It matches elements at the same tree depth with the same tag and parent tag hierarchy, then ranks by attribute similarity.

This is useful for cases where you found one element (e.g. a product card) and want to find all similar elements on the page.

The returned slice is sorted by similarity score (highest first) and excludes the element itself.

func (*Element) GenerateCSSSelector 

func (e *Element) GenerateCSSSelector() string

GenerateCSSSelector generates a CSS selector that uniquely identifies this element within its document. When the element has an id attribute, "#id" is returned directly. Otherwise, a path of "tag:nth-of-type(n)" segments separated by " > " is built from the nearest ancestor with an id (or the root) down to this element.

func (*Element) GenerateXPathSelector 

func (e *Element) GenerateXPathSelector() string

GenerateXPathSelector generates a unique XPath expression for this element, using positional predicates at each level. Example: /html[1]/body[1]/div[2]/h1[1]

func (*Element) GetAllText 

func (e *Element) GetAllText() string

GetAllText returns all text content of the element including all nested descendants, concatenated via goquery's Text() method.

func (*Element) HTML 

func (e *Element) HTML() (string, error)

HTML returns the inner HTML of the element.

func (*Element) HasClass 

func (e *Element) HasClass(class string) bool

HasClass reports whether the element has the named CSS class.

func (*Element) InnerHTML 

func (e *Element) InnerHTML() string

InnerHTML returns the HTML content inside the element (the element's children serialised to HTML, excluding the element's own opening/closing tags). Equivalent to Element.HTML().

func (*Element) JSON 

func (e *Element) JSON() (map[string]any, error)

JSON parses the element's text content as JSON and returns a map[string]any. Returns an error when the text is not valid JSON.

func (*Element) Next 

func (e *Element) Next() *Element

Next returns the immediately following sibling element, or nil when there is none.

func (*Element) OuterHTML 

func (e *Element) OuterHTML() string

OuterHTML returns the full HTML of the element including its own tags.

func (*Element) Parent 

func (e *Element) Parent() *Element

Parent returns the parent element, or nil when the element has no parent.

func (*Element) Path 

func (e *Element) Path() []string

Path returns the tag names from the document root down to this element, e.g. ["html", "body", "div", "h1"].

func (*Element) Prettify 

func (e *Element) Prettify() string

Prettify returns the outer HTML representation of the element. For a full indented rendering the outer HTML is returned as-is; full pretty-printing requires a dedicated HTML formatter not included here.

func (*Element) Prev 

func (e *Element) Prev() *Element

Prev returns the immediately preceding sibling element, or nil when there is none.

func (*Element) Re 

func (e *Element) Re(pattern string) []string

Re applies the regular-expression pattern to the element's text content and returns all matches. Returns nil for an invalid pattern.

func (*Element) ReFirst 

func (e *Element) ReFirst(pattern string) string

ReFirst returns the first regex match on the element's text content, or "" if there is no match. Returns "" for an invalid pattern.

func (*Element) Siblings 

func (e *Element) Siblings() []*Element

Siblings returns all sibling elements (excluding the element itself).

func (*Element) Tag 

func (e *Element) Tag() string

Tag returns the lowercase tag name of the element (e.g. "div", "span").

func (*Element) Text 

func (e *Element) Text() string

Text returns the combined text content of the element, with leading and trailing whitespace stripped.

type ElementSignature 

type ElementSignature struct {
	Tag       string            `json:"tag"`
	ID        string            `json:"id,omitempty"`
	Classes   []string          `json:"classes,omitempty"`
	Attrs     map[string]string `json:"attrs,omitempty"`
	Text      string            `json:"text,omitempty"` // truncated to 200 chars
	ParentTag string            `json:"parent_tag,omitempty"`
	Position  int               `json:"position"` // nth-of-type (1-based)
	Depth     int               `json:"depth"`    // nesting depth from <html>
}

ElementSignature captures the identity of an element for similarity matching. It is designed to be serialised as JSON so that it can be persisted alongside an AdaptiveSelector.

func CaptureSignature 

func CaptureSignature(el *Element) *ElementSignature

CaptureSignature creates a signature from an element for later matching.

type FeedEntry 

type FeedEntry struct {
	Title       string    `json:"title"`
	Link        string    `json:"link"`
	Description string    `json:"description,omitempty"`
	Published   time.Time `json:"published,omitempty"`
	GUID        string    `json:"guid,omitempty"`
}

FeedEntry represents a single entry from an RSS or Atom feed.

func ParseAtom 

func ParseAtom(resp *foxhound.Response) ([]FeedEntry, error)

ParseAtom parses an Atom feed response.

func ParseRSS 

func ParseRSS(resp *foxhound.Response) ([]FeedEntry, error)

ParseRSS parses an RSS feed response.

type FieldDef 

type FieldDef struct {
	// Name is the key used in the resulting Item.Fields map.
	Name string
	// Selector is a CSS selector targeting the element to extract from.
	Selector string
	// Attr is the HTML attribute to extract.  When empty, the text content is
	// used instead.
	Attr string
	// Required causes Extract / ExtractAll to return an error when the selector
	// matches no element.
	Required bool
}

FieldDef describes how to extract a single field from an HTML element.

type FileAdaptiveStore 

type FileAdaptiveStore struct {
	// contains filtered or unexported fields
}

FileAdaptiveStore wraps the existing JSON file-based storage to implement the AdaptiveStore interface.

func NewFileAdaptiveStore 

func NewFileAdaptiveStore(path string) *FileAdaptiveStore

NewFileAdaptiveStore creates a file-based adaptive store using the existing JSON persistence mechanism.

func (*FileAdaptiveStore) Close 

func (fs *FileAdaptiveStore) Close() error

Close saves the file and releases resources.

func (*FileAdaptiveStore) Load 

func (fs *FileAdaptiveStore) Load(domain, name string) (*ElementSignature, error)

Load retrieves a signature from the file-based JSON storage.

func (*FileAdaptiveStore) Save 

func (fs *FileAdaptiveStore) Save(domain, name string, sig *ElementSignature) error

Save persists a signature using the file-based JSON storage.

type Listing 

type Listing struct {
	Name        string
	Address     string
	Phone       string
	Email       string
	Website     string
	Categories  []string
	Rating      float64
	ReviewCount int
	Hours       map[string]string
	Latitude    float64
	Longitude   float64
	Image       string
	RawFields   map[string]string
}

Listing represents a business or place extracted from a directory page.

func ExtractListings 

func ExtractListings(resp *foxhound.Response) ([]Listing, error)

ExtractListings tries multiple strategies to extract business listings from a response. It attempts JSON-LD first, then microdata, then repeating DOM patterns. Returns nil, nil if no listings are detected.

func ExtractListingsWithSchema 

func ExtractListingsWithSchema(resp *foxhound.Response, schema ListingSchema) ([]Listing, error)

ExtractListingsWithSchema extracts listings using a user-defined CSS selector mapping. It finds all elements matching schema.Root and extracts fields from each using the Fields and Attrs maps.

func (*Listing) AsItem 

func (l *Listing) AsItem() *foxhound.Item

AsItem converts a Listing to a foxhound.Item with all fields mapped.

type ListingSchema 

type ListingSchema struct {
	Root   string            // CSS selector for each listing container
	Fields map[string]string // field name -> CSS selector (relative to root)
	Attrs  map[string]string // field name -> attribute to extract (default: text)
}

ListingSchema defines a custom CSS selector mapping for listing extraction.

type PageContent 

type PageContent struct {
	URL      string
	HTML     string
	Text     string
	Markdown string
	PageNum  int
}

PageContent represents a single page of paginated content.

type PaginatedContent 

type PaginatedContent struct {
	Title        string
	Author       string
	URL          string // first page URL
	Pages        []PageContent
	FullText     string
	FullMarkdown string
	TotalPages   int
}

PaginatedContent represents content assembled from multiple pages.

func AssemblePages 

func AssemblePages(pages []*foxhound.Response, contentSelector string) *PaginatedContent

AssemblePages combines multiple page responses into a single PaginatedContent. If contentSelector is non-empty, it is used to extract content from each page; otherwise findMainContent is used as a fallback.

func ExtractArticleFromPageBreaks 

func ExtractArticleFromPageBreaks(resp *foxhound.Response, contentSelector string) *PaginatedContent

ExtractArticleFromPageBreaks splits a single response on `<!-- foxhound:page-break -->` markers and assembles the segments into a PaginatedContent. 

type PaginationLink struct {
	URL       string
	Text      string
	Score     int
	Direction string // "next" or "prev"
}

PaginationLink represents a detected pagination link with confidence score.

func DetectPagination 

func DetectPagination(resp *foxhound.Response) []PaginationLink

DetectPagination analyses the response HTML and returns scored pagination links. Links with a score below 50 are discarded. Results are sorted by score descending.

type PreloadedData 

type PreloadedData struct {
	Framework string         // "nextjs", "nuxt", "react", "vue", "angular", "unknown"
	Variables map[string]any // all detected window.__VAR__ values
	NextData  map[string]any // shortcut to __NEXT_DATA__.props.pageProps (nil if not Next.js)
}

PreloadedData contains all detected JavaScript preloaded data from a page.

func ExtractPreloadedData 

func ExtractPreloadedData(resp *foxhound.Response) (*PreloadedData, error)

ExtractPreloadedData auto-detects all preloaded data on the page by checking well-known window variables. It also detects the framework and provides a shortcut to Next.js pageProps.

type Results 

type Results struct {
	// contains filtered or unexported fields
}

Results is a chainable collection of extraction results. Supports comparable Python framework-style CSS pseudo-selectors: ::text and ::attr(name).

func (*Results) Each 

func (r *Results) Each(fn func(int, *Element))

Each iterates over element results, calling fn with the zero-based index and the element. Text-only results are skipped.

func (*Results) Filter 

func (r *Results) Filter(fn func(*Element) bool) *Results

Filter returns a new Results containing only element results where fn returns true. Text-only results (from ::text/::attr pseudo-selectors) are always excluded because they carry no Element.

func (*Results) First 

func (r *Results) First() *Element

First returns the first element result, or nil when the result set is empty or was produced by a ::text/::attr pseudo-selector (which has no elements).

func (*Results) Get 

func (r *Results) Get() string

Get returns the first result as text, or "" when empty. For plain-element results the trimmed text of the first element is returned.

func (*Results) GetAll 

func (r *Results) GetAll() []string

GetAll returns all results as text strings.

func (*Results) Last 

func (r *Results) Last() *Element

Last returns the last element result, or nil when none exist.

func (*Results) Len 

func (r *Results) Len() int

Len returns the number of results.

func (*Results) Re 

func (r *Results) Re(pattern string) []string

Re applies the regular-expression pattern to every result's text and returns all matches flattened into a single slice. Returns nil for an invalid pattern.

func (*Results) ReFirst 

func (r *Results) ReFirst(pattern string) string

ReFirst returns the first regex match across all results, or "" if none. Returns "" for an invalid pattern.

type SQLiteAdaptiveStore 

type SQLiteAdaptiveStore struct {
	// contains filtered or unexported fields
}

SQLiteAdaptiveStore persists adaptive selector signatures in a SQLite database, keyed by domain and selector name. This allows element tracking across scraping runs and supports concurrent access.

The storage is thread-safe via an RWMutex and SQLite WAL mode for concurrent reads.

func NewSQLiteAdaptiveStore 

func NewSQLiteAdaptiveStore(dbPath string) (*SQLiteAdaptiveStore, error)

NewSQLiteAdaptiveStore creates a new SQLite-backed adaptive storage at the given database path. The database and table are created if they don't exist.

func (*SQLiteAdaptiveStore) Close 

func (s *SQLiteAdaptiveStore) Close() error

Close releases the database connection.

func (*SQLiteAdaptiveStore) Load 

func (s *SQLiteAdaptiveStore) Load(domain, name string) (*ElementSignature, error)

Load retrieves a previously saved signature. Returns nil, nil when no signature is found for the given domain and name.

func (*SQLiteAdaptiveStore) Save 

func (s *SQLiteAdaptiveStore) Save(domain, name string, sig *ElementSignature) error

Save persists an element signature. If a signature already exists for the given domain and name, it is replaced.

type Schema 

type Schema struct {
	Fields []FieldDef
}

Schema defines a set of fields to extract from a page.

func (*Schema) Extract 

func (s *Schema) Extract(resp *foxhound.Response) (*foxhound.Item, error)

Extract applies the schema to the entire response document and returns a single Item containing all extracted fields. Required fields that produce no match cause an error to be returned.

func (*Schema) ExtractAll 

func (s *Schema) ExtractAll(resp *foxhound.Response, rootSelector string) ([]*foxhound.Item, error)

ExtractAll applies the schema to every element matching rootSelector and returns one Item per element. Required fields inside each element follow the same rules as Extract.

type SimilarMatch 

type SimilarMatch struct {
	Element *Element
	Score   float64 // 0.0 (no match) to 1.0 (exact match)
}

SimilarMatch pairs an element with its similarity score.

type SimilarOption 

type SimilarOption func(*similarConfig)

SimilarOption configures how FindSimilar matches elements.

func WithSimilarIgnoreAttrs 

func WithSimilarIgnoreAttrs(attrs ...string) SimilarOption

WithSimilarIgnoreAttrs sets attribute names to exclude from the similarity comparison. Default: ["href", "src"].

func WithSimilarMatchText 

func WithSimilarMatchText(match bool) SimilarOption

WithSimilarMatchText includes text content in the similarity calculation. Not recommended for most cases as text varies between similar elements.

func WithSimilarThreshold 

func WithSimilarThreshold(t float64) SimilarOption

WithSimilarThreshold sets the minimum attribute similarity score (0.0-1.0). Elements below this threshold are excluded. Default: 0.2.

type SitemapEntry 

type SitemapEntry struct {
	URL        string    `xml:"loc" json:"url"`
	LastMod    time.Time `json:"last_mod,omitempty"`
	ChangeFreq string    `xml:"changefreq" json:"change_freq,omitempty"`
	Priority   float64   `xml:"priority" json:"priority,omitempty"`
}

SitemapEntry represents a single URL entry in a sitemap.xml.

func ParseSitemap 

func ParseSitemap(resp *foxhound.Response) ([]SitemapEntry, error)

ParseSitemap parses a sitemap.xml response into entries.

type Table 

type Table struct {
	Headers []string   // column names from <th> or first row
	Rows    [][]string // data rows (rectangular grid)
	Caption string     // <caption> text if present
	// contains filtered or unexported fields
}

Table represents a parsed HTML table with resolved colspan/rowspan.

func ExtractTable 

func ExtractTable(resp *foxhound.Response, selector string) (*Table, error)

ExtractTable parses HTML from resp, finds the table matching selector, and returns a *Table with resolved colspan/rowspan. Returns nil, nil when no table matches.

func ExtractTables 

func ExtractTables(resp *foxhound.Response) ([]*Table, error)

ExtractTables finds ALL <table> elements on the page and returns each as a *Table.

func (*Table) AsItems 

func (t *Table) AsItems() []*foxhound.Item

AsItems converts rows to Items using headers as field keys. Each row becomes one Item with Fields[header] = cellValue. URL and Timestamp are set on each Item.

func (*Table) Cell 

func (t *Table) Cell(row int, col string) string

Cell returns the value at the given row index and named column. Returns "" if the row or column is out of bounds.

func (*Table) Column 

func (t *Table) Column(name string) []string

Column returns all values for a named column. Returns nil if the column name is not found in Headers.

func (*Table) Row 

func (t *Table) Row(i int) []string

Row returns the row at index i (zero-based). Returns nil if out of bounds.

type XPath 

type XPath struct {
	// contains filtered or unexported fields
}

XPath provides simplified XPath-like queries on HTML responses.

Only a commonly-needed subset of XPath is supported. Expressions are converted to CSS selectors and delegated to the existing goquery Document, so no additional dependencies are required.

Supported syntax: 

//tag                  → tag
//tag[@attr]           → tag[attr]
//tag[@attr='value']   → tag[attr='value']  (id shorthand: tag#value)
//tag/subtag           → tag > subtag        (direct child)
//tag//subtag          → tag subtag          (any descendant)

func NewXPath 

func NewXPath(resp *foxhound.Response) (*XPath, error)

NewXPath creates an XPath from a foxhound Response.

func (*XPath) Select 

func (x *XPath) Select(expr string) string

Select evaluates a simplified XPath expression and returns the trimmed text content of the first matching element. Returns an empty string when no element matches.

func (*XPath) SelectAll 

func (x *XPath) SelectAll(expr string) []string

SelectAll evaluates and returns the trimmed text content of all matching elements.

func (*XPath) SelectAttr 

func (x *XPath) SelectAttr(expr, attr string) string

SelectAttr evaluates the expression and returns the value of attr on the first matching element. Returns an empty string when no element matches.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.