engine

package
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 31, 2026 License: MIT Imports: 52 Imported by: 0

Documentation

Overview

Package portal provides content extraction with SPA detection

Package engine — eval corpus loader.

CorpusEntry mirrors the schema in tests/eval/corpus.yaml. LoadCorpus parses the YAML file and returns the list of entries; no extraction is performed here.

Package portal provides content extraction with SPA detection

Package portal provides content extraction with SPA detection

Package portal provides content extraction with SPA detection

Package portal provides content extraction with SPA detection

Index

Constants

View Source 
const (
	CDNCloudflare  = "cloudflare"
	CDNCloudFront  = "cloudfront"
	CDNFastly      = "fastly"
	CDNAkamai      = "akamai"
	CDNVarnish     = "varnish"
	CDNNetlify     = "netlify"
	CDNVercel      = "vercel"
	CDNAzure       = "azure"
	CDNGCP         = "gcp"
	CDNFly         = "fly"
	CDNDeno        = "deno"
	CDNHeroku      = "heroku"
	CDNRender      = "render"
	CDNRailway     = "railway"
	CDNShopify     = "shopify"
	CDNSquarespace = "squarespace"
	CDNWix         = "wix"
	CDNWebflow     = "webflow"
)

CDN provider constants

View Source 
const DefaultAccept = "text/markdown, text/html;q=0.9, application/xhtml+xml;q=0.8, application/xml;q=0.7, */*;q=0.1"
View Source 
const DefaultAcceptEncoding = "gzip, deflate, br, zstd"
View Source 
const DefaultUserAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"

Must match a real browser exactly — Cloudflare blocks truncated/incomplete UAs.

Variables

View Source 
var (
	ErrSecurityScheme     = errors.New("scheme not allowed by security policy")
	ErrSecurityDomain     = errors.New("host blocked by security policy")
	ErrPrivateIPBlocked   = errors.New("target resolves to a private/internal IP")
	ErrSecurityResolve    = errors.New("could not resolve target host")
	ErrResponseTooLarge   = errors.New("response exceeds the configured size cap")
	ErrDecompressTooLarge = errors.New("decompressed body exceeds the configured size cap")
)

Security sentinel errors. All are wrapped with target context at the call site and surface on Result.Error / Result.SecurityBlock.

Functions

func ApplySchema 

func ApplySchema(htmlStr string, schema Schema) (map[string]interface{}, error)

ApplySchema parses htmlStr and walks the supplied Schema, returning a map[string]interface{} keyed by FieldSpec name. Schema is intended to run on RAW html (before preprocess/sanitize) so the caller-supplied selectors can target chrome elements like nav/sidebar/footer that the main extraction pipeline strips.

func CanonicalizeURL 

func CanonicalizeURL(rawURL string) (string, error)

CanonicalizeURL normalises a URL by lowercasing the host, dropping the fragment, removing default ports, stripping tracking parameters, sorting remaining params, and collapsing duplicate path slashes. Malformed input is returned unchanged alongside the parse error — never crashes.

func ChangeSignificance 

func ChangeSignificance(oldContent, newContent string) int

ChangeSignificance returns a score 0-100 indicating how significant the content change is (0 = noise only, 100 = completely different)

func CleanupMarkdown 

func CleanupMarkdown(md string) string

func ComputeConfidence 

func ComputeConfidence(length, headingCount, paragraphCount, spaSignalCount int, isBlocked bool) int

func ContentChanged 

func ContentChanged(oldContent, newContent string) bool

ContentChanged returns true if the semantic fingerprint differs This is smarter than raw byte comparison - ignores timestamps, counters, etc.

func ConvertLinksToCitations 

func ConvertLinksToCitations(md string) string

ConvertLinksToCitations rewrites every inline Markdown link `[text](url)` into `text ⟨N⟩` and appends a `## References` section listing each unique URL once (in document order, 1-based). Identical URLs reuse the same number. Fenced code blocks and inline code spans are preserved verbatim, so any `[text](url)` inside code is left alone. Image syntax `![alt](url)` is also untouched — the leading `!` distinguishes it.

Out of scope for V1:

If the input contains no convertible inline links, the input is returned unchanged (no empty `## References` section is appended).

func CountMarkdownHeadings 

func CountMarkdownHeadings(content string) int

CountMarkdownHeadings counts lines starting with # in markdown content. 

func CountMarkdownLinks(content string) int

CountMarkdownLinks counts Markdown-style links [text](url) in content. Useful for React/SPA pages where links appear in converted markdown but not raw HTML.

func CountPattern 

func CountPattern(html string, pattern string) int

func DedupeLines 

func DedupeLines(content string) string

DedupeLines removes exact duplicate lines from content. This is a lighter-weight deduplication for line-based content.

func DetectBlocked 

func DetectBlocked(html string) bool

DetectBlocked: triggers on challenge pages (Cloudflare, captcha, etc.) Strategy: check title/head indicators first (reliable), then body text if short

func DetectJSChallenge 

func DetectJSChallenge(html string, contentType string, length int) bool

DetectJSChallenge heuristically identifies "site returned 200 but shipped a JS challenge instead of real content" — the missing complement to DetectBlocked, which keys off head-level patterns (titles, well-known challenge JS variables, etc.). Strategy: a small HTML response that matches one of the cross-cutting CDN/anti-bot SIGNATURES (cf-mitigated, challenge-platform, datadome, perimeterx, akamai-bm) is almost certainly a challenge page. The 1500-byte cap is the key gatekeeper — real CDN- served pages (which all mention Cloudflare in headers/scripts) are far larger, so the length floor avoids "page mentions cloudflare in a footer" false positives. No hostname-specific logic; signatures only.

func DetectLanguage 

func DetectLanguage(markdown string) string

DetectLanguage returns a BCP-47-ish language code (e.g. "en", "es", "ja") by sampling the input. Returns "" when no language scores above the confidence threshold. The script-block fast path handles CJK + Arabic deterministically; for Latin-script languages it counts stopword hits.

func DetectSPA 

func DetectSPA(html string) (signals []string, isSPA bool)

func ExtractFromHTML 

func ExtractFromHTML(html string, targetURL string) (string, error)

func ExtractMetaAuthors 

func ExtractMetaAuthors(rawHTML string) []string

func ExtractPDFText 

func ExtractPDFText(body []byte) (out string, err error)

ExtractPDFText opens the PDF body bytes and returns plain text with per-page separators. Returns an error on encrypted, malformed, or empty PDF input.

regression: pdf-malformed-xref-panic — `pdfReader.NewReader` and `NumPage` both panic on malformed startxref offsets (upstream `ledongthuc/pdf` returns no error). Top-level recover converts those panics into Go errors so a single bad PDF can't kill the caller.

func FetchBytes 

func FetchBytes(ctx context.Context, rawURL string, opts FetchBytesOptions) ([]byte, http.Header, int, error)

FetchBytes fetches raw bytes for rawURL, applying SecurityPolicy redirect, private-IP, and body/decompression limits when configured.

func LDJSONToMarkdown 

func LDJSONToMarkdown(blocks []LDJSONBlock) string

LDJSONToMarkdown converts LD+JSON blocks into supplementary markdown content. Returns empty string if no useful content found.

func NearDuplicateScore 

func NearDuplicateScore(a, b string) int

NearDuplicateScore returns a similarity score (0-100) between two blocks 100 = identical, 0 = completely different

func PickCanonical 

func PickCanonical(rawURL, html string) string

PickCanonical orchestrates canonical-URL selection. A <link rel="canonical"> in the HTML wins; otherwise we fall back to algorithmic canonicalisation. Returns "" when the canonical is empty or equal to the raw URL.

func PreprocessHTML 

func PreprocessHTML(htmlStr string) string

PreprocessHTML applies a host-agnostic preprocessing pass: replace twoslash buttons, strip common chrome (nav/aside/footer/sidebar/cookie banners), then scope to the most likely main-content container.

func PreprocessHTMLWithURL 

func PreprocessHTMLWithURL(htmlStr string, _ *url.URL) string

PreprocessHTMLWithURL keeps the URL parameter for API compatibility but the preprocessing pipeline is now entirely host-agnostic.

func PruneToContent 

func PruneToContent(htmlStr string) string

PruneToContent runs a tag-density + position heuristic over the body and returns HTML containing only the highest-scoring contiguous content region wrapped in <article>. Returns the original input unchanged when no candidate scores above a minimum threshold (textLen >= 200).

Score: textLen / (1 + linkTextLen) - 100 * chromeChildCount, multiplied by a position bias that peaks at mid-document (1.0) and decays toward the document edges (>= 0).

func QuickNeedsBrowser 

func QuickNeedsBrowser(html string) (needsBrowser bool, reason string)
func ResolveCanonicalLink(htmlStr string, baseURL string) string

ResolveCanonicalLink finds <link rel="canonical" href="…"> in the first 4 KB of HTML and resolves the href against baseURL. Returns "" if absent or if the href uses a non-http(s) scheme.

func ResolveUserAgent 

func ResolveUserAgent(s string) string

ResolveUserAgent returns the UA string for a preset name (case-insensitive), or the input itself when it doesn't match a known preset (treated as a literal UA string). Empty input returns DefaultUserAgent.

func ResultToTEIXML 

func ResultToTEIXML(r Result) ([]byte, error)

ResultToTEIXML wraps a Result into a TEI-Lite XML document.

func SanitizeHTML 

func SanitizeHTML(html string) string

SanitizeHTML removes hidden elements, junk tags, and invisible content from HTML before passing it to the readability extractor. This significantly improves extraction quality on complex pages like StackOverflow and NYTimes.

func SemanticFingerprint 

func SemanticFingerprint(content string) string

func ShouldUseIndexFallback 

func ShouldUseIndexFallback(readabilityResult Result, indexResult IndexPageResult) bool

func StripInvisibleUnicode 

func StripInvisibleUnicode(text string) string

StripInvisibleUnicode removes zero-width and invisible Unicode characters from extracted text content.

func TruncateMarkdownAtParagraph 

func TruncateMarkdownAtParagraph(md string, maxTokens int) (string, bool)

TruncateMarkdownAtParagraph returns md truncated to fit within an approximate maxTokens budget (using ~4 chars/token heuristic), cut at the latest paragraph boundary (blank-line separator) that fits. Falls back to single-newline boundary, then hard cut. Appends a "*[truncated]*" marker. Returns the truncated string and a bool indicating whether truncation occurred.

Note: 4 chars/token is the cl100k-base approximation; we deliberately do not import a real tokenizer (heavy dep). For exact budgeting, callers should tokenize themselves.

V1 limitation: paragraph boundaries can straddle fenced code blocks, so truncation inside a fence may leave an unterminated ```. Documented; V2 could honour fence balance.

Types

type BrowserDecision 

type BrowserDecision string

BrowserDecision is the single routing category a caller (e.g. PinchTab) reads to decide whether to fall through to a real browser. It refines Outcome with transport/status context; BrowserRecommended is the one boolean to branch on. 

const (
	DecisionStaticHighConfidence BrowserDecision = "static-high-confidence"
	DecisionStaticOK             BrowserDecision = "static-ok"
	DecisionStaticCaution        BrowserDecision = "static-caution"
	DecisionBrowserNeeded        BrowserDecision = "browser-needed"
	DecisionBlocked              BrowserDecision = "blocked"
	DecisionUnreachable          BrowserDecision = "unreachable"
	DecisionNotFound             BrowserDecision = "not-found"
	DecisionUnsupported          BrowserDecision = "unsupported"
)

type CardItem 

type CardItem struct {
	Title   string
	URL     string
	Teaser  string
	Section string
}

type Chunk 

type Chunk struct {
	Index   int    `json:"index"`
	Heading string `json:"heading,omitempty"`
	Text    string `json:"text"`
	Tokens  int    `json:"tokens"`
}

Chunk is a single piece of a chunked Markdown body.

func ChunkMarkdown 

func ChunkMarkdown(md string, cfg ChunkConfig) []Chunk

ChunkMarkdown applies cfg to md and returns the resulting Chunks. Returns nil when chunking is off, content is empty, or content is shorter than 100 chars (not worth chunking).

Code regions are masked before splitting and unmasked per emitted chunk so fenced code blocks are never split by heading/sentence boundaries inside.

type ChunkConfig 

type ChunkConfig struct {
	Strategy ChunkStrategy
	// Size meaning depends on Strategy:
	//   sentence: target tokens per group
	//   window:   chars per window
	Size int
	// Overlap is only used by window strategy (chars of overlap).
	Overlap int
}

ChunkConfig controls how Markdown is split into Chunks.

func ParseChunkConfig 

func ParseChunkConfig(s string) (ChunkConfig, error)

ParseChunkConfig parses the colon-form CLI argument. 

""                          -> {ChunkOff, 0, 0}
"heading"                   -> {ChunkHeading, 0, 0}
"sentence" / "sentence:N"   -> {ChunkSentence, N|512, 0}
"window" / "window:N[:O]"   -> {ChunkWindow, N|2000, O|200}

Validation:

  • overlap must be < size (window strategy)
  • unknown names error out

func (ChunkConfig) String 

func (c ChunkConfig) String() string

String renders the canonical CLI form of the config (or "" for off).

type ChunkStrategy 

type ChunkStrategy int

ChunkStrategy selects a chunking algorithm. Default is off. 

const (
	// ChunkOff disables chunking (default — Result.Chunks stays nil).
	ChunkOff ChunkStrategy = iota
	// ChunkHeading splits at H2-H6 boundaries, preserving the heading.
	ChunkHeading
	// ChunkSentence groups sentences until a ~Size-token threshold.
	ChunkSentence
	// ChunkWindow slides a Size-char window with Overlap chars of backstep.
	ChunkWindow
)

type CommentRef 

type CommentRef struct {
	Author    string `json:"author,omitempty"`
	Text      string `json:"text"`
	Timestamp string `json:"timestamp,omitempty"`
}

CommentRef is a single user-generated comment harvested from a recognised comment-container element (Disqus, native comments, "Replies" widgets). Surfaced on Result.Comments when the caller opts in via Options.WithComments.

func ExtractComments 

func ExtractComments(htmlStr string, _ string) []CommentRef

ExtractComments walks htmlStr for comment containers and harvests one CommentRef per detected comment block, best-effort. baseURL is reserved for future use (resolving relative author profile links). Returns nil when no containers are found.

type CorpusEntry 

type CorpusEntry struct {
	Path        string   `yaml:"path"`
	MustInclude []string `yaml:"must_include"`
	MustExclude []string `yaml:"must_exclude"`
	ExpectClass string   `yaml:"expect_class"`
	ExpectLang  string   `yaml:"expect_lang"`
}

CorpusEntry describes a single fixture's extraction-quality expectations.

  • Path: repo-relative path to the HTML fixture.
  • MustInclude: substrings that MUST appear in the extracted Markdown.
  • MustExclude: substrings that MUST NOT survive extraction (chrome, signup CTAs, layout debris, etc.).
  • ExpectClass: canonical PageClass value (static|ssr|hydrated|spa|dynamic|blocked).
  • ExpectLang: expected BCP-47 language tag; empty when undefined.

func LoadCorpus 

func LoadCorpus(path string) ([]CorpusEntry, error)

LoadCorpus reads and YAML-decodes a corpus file. Any read or parse error is wrapped for caller context.

type CrawlDelayCache 

type CrawlDelayCache struct {
	// contains filtered or unexported fields
}

CrawlDelayCache stores per-domain crawl-delay values + Allow/Disallow rules from robots.txt. Keeps its historical name even though it now caches rules too — the storage and fetch path is shared.

func NewCrawlDelayCache 

func NewCrawlDelayCache() *CrawlDelayCache

NewCrawlDelayCache creates a new crawl-delay + robots-rule cache.

func (*CrawlDelayCache) GetDelay 

func (c *CrawlDelayCache) GetDelay(domain string, userAgent string) time.Duration

GetDelay returns the crawl-delay for a domain over HTTPS.

func (*CrawlDelayCache) GetDelayWithScheme 

func (c *CrawlDelayCache) GetDelayWithScheme(domain string, userAgent string, scheme string) time.Duration

GetDelayWithScheme returns the crawl-delay for a domain using the given scheme.

func (*CrawlDelayCache) IsAllowed 

func (c *CrawlDelayCache) IsAllowed(domain string, userAgent string, scheme string, path string) bool

IsAllowed returns true if the given path is permitted for userAgent under the domain's robots.txt rules. Fail-open: when robots.txt is unreachable (network error, 4xx, 5xx, parse failure) the path is treated as allowed.

type DedupeOptions 

type DedupeOptions struct {
	// MinBlockLen is the minimum length for a block to be tracked for deduplication
	// Shorter blocks (like single words) are always kept to avoid over-aggressive removal
	MinBlockLen int
	// NormalizeWhitespace collapses all whitespace to single spaces before comparison
	NormalizeWhitespace bool
	// CaseSensitive controls whether duplicate detection is case-sensitive
	CaseSensitive bool
	// NearDup enables simhash-based near-duplicate detection after the exact-hash
	// check misses. Only blocks whose normalised length is >= nearDupMinLength are
	// compared; matches within nearDupHammingThreshold bits are dropped.
	NearDup bool
}

DedupeOptions configures deduplication behavior

func DefaultDedupeOptions 

func DefaultDedupeOptions() DedupeOptions

DefaultDedupeOptions returns sensible defaults for content deduplication

type DedupeResult 

type DedupeResult struct {
	Content              string   `json:"content,omitempty"`
	OriginalBlocks       int      `json:"originalBlocks,omitempty"`
	UniqueBlocks         int      `json:"uniqueBlocks,omitempty"`
	DuplicatesFound      int      `json:"duplicatesFound,omitempty"` // exact-hash matches
	DuplicateSignals     []string `json:"duplicateSignals,omitempty"`
	NearDuplicatesFound  int      `json:"nearDuplicatesFound,omitempty"` // simhash matches
	NearDuplicateSignals []string `json:"nearDuplicateSignals,omitempty"`
}

DedupeResult holds deduplication metrics and output

func Dedupe 

func Dedupe(content string) DedupeResult

Dedupe removes duplicate blocks from markdown content. It splits content into logical blocks (paragraphs, headings, list items, etc.) and removes exact duplicates while preserving structure and order.

func DedupeWithOptions 

func DedupeWithOptions(content string, opts DedupeOptions) DedupeResult

type DiskCache 

type DiskCache struct {
	// contains filtered or unexported fields
}

DiskCache is a simple on-disk content cache keyed by URL + representation headers. Safe for concurrent use across processes (POSIX rename is atomic).

func NewDiskCache 

func NewDiskCache(dir string, ttl time.Duration) (*DiskCache, error)

NewDiskCache creates a cache rooted at dir with the given TTL. A zero TTL defaults to 24h. The directory is created if it does not exist.

func (*DiskCache) Get 

func (c *DiskCache) Get(url string, req *http.Request) (*cachedResponse, []byte, bool)

Get returns the cached response for (url, req) if present and still fresh. A missing headers file (whether the entry was never written or only the body half landed) is reported as a clean miss.

func (*DiskCache) GetStale 

func (c *DiskCache) GetStale(url string, req *http.Request) (*cachedResponse, []byte, bool, bool)

GetStale returns the cached response for (url, req) when present, either as fresh (within TTL) or as stale-but-validatable (past TTL with at least one of ETag/Last-Modified). Callers use the fresh bool to short-circuit the network, or use the stale bool plus ConditionalHeaders to send a conditional GET. A miss with neither validator returns (nil, nil, false, false).

Equivalent to GetStaleWithTolerance(url, req, 0) where the SWR band has zero width and any past-TTL entry must either revalidate or miss.

func (*DiskCache) GetStaleWithTolerance 

func (c *DiskCache) GetStaleWithTolerance(url string, req *http.Request, tolerance time.Duration) (*cachedResponse, []byte, bool, bool, bool)

GetStaleWithTolerance classifies a cached entry into one of three bands:

  • fresh: age <= TTL → serve directly.
  • stale (SWR): TTL < age <= TTL + tolerance → serve as stale, caller is expected to fire a background revalidate. Validators are NOT required for the SWR band — within tolerance we trust the cached body unconditionally.
  • beyondTolerance: age > TTL + tolerance AND has validators (ETag / Last-Modified) → caller runs a synchronous conditional GET.

A past-TTL+tolerance entry without validators is a clean miss (nil, nil, false, false, false). A zero tolerance collapses the SWR band to zero width and the function behaves identically to GetStale.

func (*DiskCache) Put 

func (c *DiskCache) Put(url string, req *http.Request, status int, headers http.Header, body []byte) error

Put writes a cache entry atomically. Body lands first; headers second. A crash between the two writes leaves a body-only orphan which Get correctly treats as a miss. Callers must only Put for status == 200.

func (*DiskCache) TouchByKey 

func (c *DiskCache) TouchByKey(key string) error

TouchByKey refreshes the FetchedAt timestamp on the headers sidecar for the given cache key. Called after a 304 confirms the cached body is still valid so that subsequent Gets see it as fresh. The body file is untouched.

type DomainRetry 

type DomainRetry struct {
	MaxRetries   int
	MaxRetryWait time.Duration
}

type ExtractionOutcome 

type ExtractionOutcome string
const (
	OutcomeExtract        ExtractionOutcome = "extract"
	OutcomeExtractWarning ExtractionOutcome = "extract-with-warning"
	OutcomeFailFast       ExtractionOutcome = "fail-fast"
	OutcomeNeedsBrowser   ExtractionOutcome = "needs-browser"
)

type FeedItem 

type FeedItem struct {
	Title     string `json:"title,omitempty"`
	Link      string `json:"link,omitempty"`
	Published string `json:"published,omitempty"`
	Summary   string `json:"summary,omitempty"`
	Author    string `json:"author,omitempty"`
	GUID      string `json:"guid,omitempty"`
}

FeedItem is a normalised feed entry covering RSS 2.0, Atom 1.0, and JSON Feed 1.x sources.

func ParseFeed 

func ParseFeed(ctx context.Context, feedURL string, opts ParseFeedOptions) ([]FeedItem, error)

ParseFeed fetches feedURL and parses it as RSS 2.0, Atom 1.0, or JSON Feed 1.x, returning a unified slice of FeedItem in feed order, capped to opts.MaxItems.

type FetchBytesOptions 

type FetchBytesOptions struct {
	Client    *http.Client
	Timeout   time.Duration
	Security  *SecurityPolicy
	Accept    string
	UserAgent string
}

FetchBytesOptions controls a raw network fetch with optional security enforcement and size caps.

type FieldSpec 

type FieldSpec struct {
	Selector string               `json:"selector,omitempty" yaml:"selector,omitempty"`
	Attr     string               `json:"attr,omitempty" yaml:"attr,omitempty"`
	Multiple bool                 `json:"multiple,omitempty" yaml:"multiple,omitempty"`
	Fields   map[string]FieldSpec `json:"fields,omitempty" yaml:"fields,omitempty"`
}

FieldSpec describes how to extract one field from the DOM.

  • Selector: CSS selector (cascadia syntax). Required.
  • Attr: when set, the value is the named attribute of the matched element rather than its joined text content.
  • Multiple: when true (and Fields empty), emit an array of values.
  • Fields: when non-empty, treat this as a nested-object collector; query QueryAll on Selector and recursively apply Fields scoped to each match, emitting an array of maps.

type FlattenSitemapOptions 

type FlattenSitemapOptions struct {
	MaxDepth int             // default 5
	MaxURLs  int             // default 50_000
	Timeout  time.Duration   // per-fetch timeout (used when Client is nil)
	Client   *http.Client    // optional; falls back to engine getClient()
	Security *SecurityPolicy // optional fetch guard (SSRF, redirects, size caps)
}

FlattenSitemapOptions controls FlattenSitemap behaviour.

type HostRateLimiter 

type HostRateLimiter struct {
	// contains filtered or unexported fields
}

HostRateLimiter enforces a minimum interval between requests to the same host. Mutex-guarded map; thread-safe. Callers wanting cross-call throttling must share one instance via Options.RateLimiter.

func NewHostRateLimiter 

func NewHostRateLimiter() *HostRateLimiter

NewHostRateLimiter returns a fresh limiter with an empty last-request map.

func (*HostRateLimiter) Wait 

func (l *HostRateLimiter) Wait(host string, minInterval time.Duration)

Wait sleeps if the current time is within minInterval of the last recorded request for host. No-op when minInterval <= 0 or host is empty.

type ImageRef 

type ImageRef struct {
	Src    string `json:"src"`
	Alt    string `json:"alt,omitempty"`
	Srcset string `json:"srcset,omitempty"`
	Title  string `json:"title,omitempty"`
}

ImageRef is a structured representation of a discovered <img> element in the raw page HTML. Surfaced on Result.Images when the caller opts in via Options.WithImages so agents can pick a cover image, fetch alt-text for accessibility, or download referenced assets without re-parsing the HTML.

func ExtractImages 

func ExtractImages(htmlStr string, baseURL string) []ImageRef

ExtractImages walks htmlStr and returns every <img src="…"> as an ImageRef in document order. Relative src values are resolved against baseURL; entries are deduplicated by src (first occurrence wins — alt/srcset from the first hit are kept). Images inside <script>/<style> subtrees are skipped, as are empty src and data: URLs (typically noisy inline-encoded SVGs/PNGs).

V1 scope: only the <img> element's own src/srcset/alt/title are read. Known limitations (deferred to V2):

  • <picture><source srcset=…> siblings are NOT flattened; only the <img> fallback is captured (its own src/srcset is the agreed default).
  • data-src / data-srcset lazy-load attributes are NOT consulted; sites that put real URLs only in data-src will surface placeholder src values.
  • <img> inside <noscript> is invisible because the HTML5 parser treats <noscript> content as text.

type IndexPageResult 

type IndexPageResult struct {
	IsIndexPage   bool
	Items         []CardItem
	Markdown      string
	Confidence    int
	ArticleCount  int
	HeadlineCount int
}

func DetectIndexPage 

func DetectIndexPage(htmlStr string) IndexPageResult

type LDJSONBlock 

type LDJSONBlock struct {
	Type        string `json:"type,omitempty"`
	Headline    string `json:"headline,omitempty"`
	Description string `json:"description,omitempty"`
	Author      string `json:"author,omitempty"`
	DatePub     string `json:"datePublished,omitempty"`
	Publisher   string `json:"publisher,omitempty"`
	URL         string `json:"url,omitempty"`
	Keywords    string `json:"keywords,omitempty"`
	Language    string `json:"inLanguage,omitempty"` // BCP-47 language tag
	Section     string `json:"articleSection,omitempty"`
	Body        string `json:"articleBody,omitempty"` // may be HTML or plain text
}

LDJSONBlock represents a single LD+JSON structured data block.

func ExtractLDJSON 

func ExtractLDJSON(html string) []LDJSONBlock

ExtractLDJSON extracts and parses all LD+JSON blocks from HTML.

type LinkRef 

type LinkRef struct {
	Href string `json:"href"`
	Text string `json:"text,omitempty"`
	Rel  string `json:"rel,omitempty"`
}

LinkRef is a structured representation of a discovered <a> link in the raw page HTML. Surfaced on Result.Links when the caller opts in via Options.WithLinks so agents can pick the next page without re-parsing the extracted Markdown. 

func ExtractLinks(htmlStr string, baseURL string) []LinkRef

ExtractLinks walks htmlStr and returns every <a href="…"> as a LinkRef in document order. Relative hrefs are resolved against baseURL; entries are deduplicated by the (href, text) pair (first occurrence wins). Anchors inside <script>/<style> subtrees, fragment-only hrefs (#…), empty hrefs, and javascript/mailto/tel/data schemes are skipped.

type LinkRetention 

type LinkRetention int

LinkRetention controls how inline Markdown links `[text](url)` are rendered after extraction. The default (LinkRetentionAll) leaves links untouched. 

const (
	// LinkRetentionAll keeps inline `[text](url)` as-is (default).
	LinkRetentionAll LinkRetention = iota
	// LinkRetentionNone strips both link text and URL.
	LinkRetentionNone
	// LinkRetentionText keeps the link text, drops the URL.
	LinkRetentionText
	// LinkRetentionFooter delegates to ConvertLinksToCitations:
	// inline links become `text ⟨N⟩` plus a numbered `## References` section.
	LinkRetentionFooter
)

func ParseLinkRetention 

func ParseLinkRetention(s string) (LinkRetention, error)

ParseLinkRetention parses a mode name. Accepts lowercase "none", "text", "all", "footer". Returns an error otherwise.

func (LinkRetention) String 

func (m LinkRetention) String() string

String returns the canonical lowercase name of the mode.

type Metadata 

type Metadata struct {
	Author        string
	PublishedDate string
	Language      string
	Section       string
	Description   string
	ImageURL      string
	OGType        string
	Keywords      string
}

Metadata holds the resolved per-field winners from a single HTML pass.

func ExtractMetadata 

func ExtractMetadata(rawHTML string) Metadata

ExtractMetadata walks every <meta> tag once and resolves the per-field priority chains documented in the user story.

type Options 

type Options struct {
	FailFast             bool
	FastMode             bool
	ProbeSearch          bool
	NoPooling            bool
	MaxRetries           int
	MaxRetryWait         time.Duration
	TotalRetryTimeout    time.Duration
	HeadPreflight        bool
	ContentTypePreflight bool
	RetryLogger          func(event RetryEvent)
	DomainRetryConfig    map[string]DomainRetry
	UserAgent            string
	DomainUserAgent      map[string]string
	DomainTimeout        map[string]time.Duration
	RespectCrawlDelay    bool
	RespectRobots        bool
	CrawlDelayCache      *CrawlDelayCache
	RateLimit            time.Duration
	RateLimiter          *HostRateLimiter
	RequestID            string
	SendRequestID        bool
	Dedupe               bool
	NoNearDedupe         bool
	WithLinks            bool
	WithImages           bool
	WithTables           bool
	WithComments         bool
	Citations            bool
	LinkRetention        LinkRetention
	Chunk                ChunkConfig
	SelectCSS            string
	StripCSS             string
	MaxTokens            int
	HeadOnly             bool
	NoPruneFallback      bool
	Proxy                string
	CacheDir             string
	CacheTTL             time.Duration
	CacheStaleTolerance  time.Duration
	NoCache              bool
	NoPDF                bool
	SchemaPath           string
	Schema               *Schema
	Query                string
	TopN                 int
	FilterByQuery        bool
	SplitOut             string
	SplitBytes           int

	// Transport overrides the default utls Chrome-fingerprint transport when
	// non-nil. Primary use: tests injecting a record/replay RoundTripper from
	// internal/engine/mock so HTTP-touching tests stay hermetic. Production
	// callers should leave this nil; opts.Proxy is independently honoured.
	Transport http.RoundTripper

	// Security, when non-nil, enforces an SSRF / private-IP / redirect /
	// decompression policy across the whole fetch path. Nil (the zero value)
	// keeps the historical unguarded behaviour. Build a safe default with
	// DefaultSecurityPolicy. Safe to share across concurrent calls.
	Security *SecurityPolicy
}

type PageClass 

type PageClass string
const (
	PageStatic   PageClass = "static"
	PageSSR      PageClass = "ssr"
	PageHydrated PageClass = "hydrated"
	PageSPA      PageClass = "spa"
	PageDynamic  PageClass = "dynamic"
	PageBlocked  PageClass = "blocked"
)

type PageProfile 

type PageProfile struct {
	Class              PageClass         `json:"class"`
	Outcome            ExtractionOutcome `json:"outcome"`
	Decision           BrowserDecision   `json:"decision"`
	BrowserRecommended bool              `json:"browserRecommended"`
	Reasons            []string          `json:"reasons"`
	Confidence         int               `json:"confidence"`
	Trustworthy        bool              `json:"trustworthy"`
}

func ClassifyPage 

func ClassifyPage(result Result) PageProfile

ClassifyPage determines the page profile and the browser-routing decision.

func (PageProfile) String 

func (p PageProfile) String() string

type ParseFeedOptions 

type ParseFeedOptions struct {
	MaxItems int             // default 200
	Timeout  time.Duration   // per-fetch timeout (used when Client is nil)
	Client   *http.Client    // optional; falls back to engine getClient()
	Security *SecurityPolicy // optional fetch guard (SSRF, redirects, size caps)
}

ParseFeedOptions controls ParseFeed behaviour.

type QualityMetrics 

type QualityMetrics = quality.Metrics

QualityMetrics is an alias for quality.Metrics to preserve API compatibility

func ComputeQuality 

func ComputeQuality(markdown string) QualityMetrics

type RankedSection 

type RankedSection struct {
	Score   float64 `json:"score"`
	Heading string  `json:"heading,omitempty"`
	Text    string  `json:"text"`
	Tokens  int     `json:"tokens"`
}

RankedSection is a heading-bounded slice of Markdown scored against a query via BM25. Higher Score = more relevant. Heading is the H2/H3 line (may be empty for the prologue or a heading-less document).

func RankSections 

func RankSections(content, query string, k1, b float64, topN int) []RankedSection

RankSections splits content by H2/H3 boundaries (reusing chunkByHeading) and scores each section by BM25 against the query. Returns sections in descending score order. When topN > 0, truncates to topN. Returns nil for empty query.

BM25 formula (per query term q): 

idf(q)  = log((N - df + 0.5) / (df + 0.5) + 1)
tfNorm  = tf * (k1 + 1) / (tf + k1 * (1 - b + b * dl/avgdl))
score  += idf(q) * tfNorm

Defaults applied when caller passes 0: k1 = 1.5, b = 0.75.

type Result 

type Result struct {
	URL              string   `json:"url"`
	CanonicalURL     string   `json:"canonicalUrl,omitempty"`
	Title            string   `json:"title"`
	Content          string   `json:"content"`
	Byline           string   `json:"byline"`
	PublishedDate    string   `json:"publishedDate,omitempty"`
	Language         string   `json:"language,omitempty"`
	Charset          string   `json:"charset,omitempty"`
	Section          string   `json:"section,omitempty"`
	Description      string   `json:"description,omitempty"`
	ImageURL         string   `json:"imageUrl,omitempty"`
	Excerpt          string   `json:"excerpt"`
	SiteName         string   `json:"sitename"`
	Length           int      `json:"length"`
	TimeMs           int64    `json:"timeMs"`
	Confidence       int      `json:"confidence"`
	IsSPA            bool     `json:"isSpa"`
	IsBlocked        bool     `json:"isBlocked"`
	SPASignals       []string `json:"spaSignals,omitempty"`
	Error            string   `json:"error,omitempty"`
	FromCache        bool     `json:"fromCache,omitempty"`
	CacheHit         bool     `json:"cacheHit,omitempty"`
	CacheRevalidated bool     `json:"cacheRevalidated,omitempty"`
	CacheStale       bool     `json:"cacheStale,omitempty"`
	StatusCode       int      `json:"statusCode,omitempty"`
	HeadOnly         bool     `json:"headOnly,omitempty"`
	Protocol         string   `json:"protocol,omitempty"`

	BlockedByRobots bool `json:"blockedByRobots,omitempty"`

	HeadingCount   int `json:"headingCount"`
	LinkCount      int `json:"linkCount"`
	ParagraphCount int `json:"paragraphCount"`

	Quality     float64        `json:"quality"`
	QualityInfo QualityMetrics `json:"qualityInfo"`
	Profile     PageProfile    `json:"profile"`
	PageClass   PageClass      `json:"pageClass"`
	Fingerprint string         `json:"fingerprint"`
	Validation  Validation     `json:"validation"`

	RetryCount          int           `json:"retryCount,omitempty"`
	TotalRetryWait      time.Duration `json:"totalRetryWait,omitempty"`
	HeadPreflightStatus int           `json:"headPreflightStatus,omitempty"`

	FetchTimeMs   int64 `json:"fetchTimeMs,omitempty"`
	ParseTimeMs   int64 `json:"parseTimeMs,omitempty"`
	ConvertTimeMs int64 `json:"convertTimeMs,omitempty"`

	ContentLength       int64  `json:"contentLength,omitempty"`
	ResponseContentType string `json:"responseContentType,omitempty"`

	RedirectCount int      `json:"redirectCount,omitempty"`
	RedirectChain []string `json:"redirectChain,omitempty"`
	FinalURL      string   `json:"finalUrl,omitempty"`

	IsSoft404    bool     `json:"isSoft404,omitempty"`
	Soft404Hints []string `json:"soft404Hints,omitempty"`

	ResponseETag         string `json:"responseEtag,omitempty"`
	ResponseLastModified string `json:"responseLastModified,omitempty"`

	ResponseContentEncoding string `json:"responseContentEncoding,omitempty"`
	ResponseServer          string `json:"responseServer,omitempty"`
	ResponseXForwardedFor   string `json:"responseXForwardedFor,omitempty"`

	RequestAcceptEncoding string `json:"requestAcceptEncoding,omitempty"`

	TTFBMs     int64 `json:"ttfbMs,omitempty"`
	DownloadMs int64 `json:"downloadMs,omitempty"`

	ResponseVia        string `json:"responseVia,omitempty"`
	ResponseConnection string `json:"responseConnection,omitempty"`
	ResponseAge        string `json:"responseAge,omitempty"`

	ResponseCacheControl string `json:"responseCacheControl,omitempty"`
	ResponseXCache       string `json:"responseXCache,omitempty"`
	ResponseVary         string `json:"responseVary,omitempty"`

	ResponseXCacheHits       string `json:"responseXCacheHits,omitempty"`
	ResponseSurrogateControl string `json:"responseSurrogateControl,omitempty"`
	ResponseCFCacheStatus    string `json:"responseCfCacheStatus,omitempty"`

	ResponseXServedBy        string `json:"responseXServedBy,omitempty"`
	ResponseXFastlyRequestID string `json:"responseXFastlyRequestId,omitempty"`

	ResponseXAkamaiTransformed string `json:"responseXAkamaiTransformed,omitempty"`
	ResponseXAkamaiSessionInfo string `json:"responseXAkamaiSessionInfo,omitempty"`
	ResponseXAkamaiRequestID   string `json:"responseXAkamaiRequestId,omitempty"`

	ResponseXRequestId     string `json:"responseXRequestId,omitempty"`
	ResponseXCorrelationId string `json:"responseXCorrelationId,omitempty"`

	ResponseXVarnish string `json:"responseXVarnish,omitempty"`

	ResponseXCDN string `json:"responseXCdn,omitempty"`

	ResponseXTraceId        string `json:"responseXTraceId,omitempty"`
	ResponseXB3TraceId      string `json:"responseXB3TraceId,omitempty"`
	ResponseXB3SpanId       string `json:"responseXB3SpanId,omitempty"`
	ResponseXB3ParentSpanId string `json:"responseXB3ParentSpanId,omitempty"`
	ResponseXB3Sampled      string `json:"responseXB3Sampled,omitempty"`
	ResponseB3              string `json:"responseB3,omitempty"`
	ResponseTraceparent     string `json:"responseTraceparent,omitempty"`
	ResponseTracestate      string `json:"responseTracestate,omitempty"`
	ResponseXAmznTraceId    string `json:"responseXAmznTraceId,omitempty"`

	TraceFormats     []string `json:"traceFormats,omitempty"`
	TraceCorrelation string   `json:"traceCorrelation,omitempty"`

	ResponseNEL string `json:"responseNel,omitempty"`

	ResponseReportTo string `json:"responseReportTo,omitempty"`

	ResponsePermissionsPolicy  string `json:"responsePermissionsPolicy,omitempty"`
	ResponseExpectCT           string `json:"responseExpectCt,omitempty"`
	ResponseFeaturePolicy      string `json:"responseFeaturePolicy,omitempty"`
	ResponseReportingEndpoints string `json:"responseReportingEndpoints,omitempty"`
	ResponseCSP                string `json:"responseCsp,omitempty"`
	ResponseCSPReportOnly      string `json:"responseCspReportOnly,omitempty"`

	ResponseCORP string `json:"responseCorp,omitempty"`
	ResponseCOEP string `json:"responseCoep,omitempty"`
	ResponseCOOP string `json:"responseCoop,omitempty"`

	ResponseHSTS string `json:"responseHsts,omitempty"`

	ResponseXContentTypeOptions string `json:"responseXContentTypeOptions,omitempty"`
	ResponseXFrameOptions       string `json:"responseXFrameOptions,omitempty"`

	ResponseReferrerPolicy string `json:"responseReferrerPolicy,omitempty"`

	ResponseXXSSProtection                string `json:"responseXXssProtection,omitempty"`
	ResponseXPermittedCrossDomainPolicies string `json:"responseXPermittedCrossDomainPolicies,omitempty"`
	ResponseXDownloadOptions              string `json:"responseXDownloadOptions,omitempty"`

	ResponseClearSiteData string `json:"responseClearSiteData,omitempty"`

	ResponseTimingAllowOrigin string `json:"responseTimingAllowOrigin,omitempty"`

	ResponseOriginAgentCluster string `json:"responseOriginAgentCluster,omitempty"`

	ResponseDocumentPolicy string `json:"responseDocumentPolicy,omitempty"`

	ResponseAcceptCH string `json:"responseAcceptCH,omitempty"`

	ResponseSecCHUA                 string `json:"responseSecChUa,omitempty"`
	ResponseSecCHUAMobile           string `json:"responseSecChUaMobile,omitempty"`
	ResponseSecCHUAPlatform         string `json:"responseSecChUaPlatform,omitempty"`
	ResponseSecCHUAFullVersionList  string `json:"responseSecChUaFullVersionList,omitempty"`
	ResponseSecCHPrefersColorScheme string `json:"responseSecChPrefersColorScheme,omitempty"`

	ResponseCriticalCH string `json:"responseCriticalCh,omitempty"`

	ResponseCOEPReportOnly string `json:"responseCoepReportOnly,omitempty"`
	ResponseCOOPReportOnly string `json:"responseCoopReportOnly,omitempty"`

	ResponseDocumentPolicyReportOnly string `json:"responseDocumentPolicyReportOnly,omitempty"`

	ResponseSourceMap string `json:"responseSourceMap,omitempty"`

	ResponseAccessControlAllowOrigin      string `json:"responseAccessControlAllowOrigin,omitempty"`
	ResponseAccessControlAllowMethods     string `json:"responseAccessControlAllowMethods,omitempty"`
	ResponseAccessControlAllowHeaders     string `json:"responseAccessControlAllowHeaders,omitempty"`
	ResponseAccessControlAllowCredentials string `json:"responseAccessControlAllowCredentials,omitempty"`
	ResponseAccessControlExposeHeaders    string `json:"responseAccessControlExposeHeaders,omitempty"`
	ResponseAccessControlMaxAge           string `json:"responseAccessControlMaxAge,omitempty"`

	ResponseLink string `json:"responseLink,omitempty"`

	LLMsTxtURL string `json:"llmsTxtUrl,omitempty"`

	LDJSONBlocks []LDJSONBlock `json:"ldJsonBlocks,omitempty"`

	HasLLMContent bool `json:"hasLlmContent,omitempty"`

	ResponseXRobotsTag string `json:"responseXRobotsTag,omitempty"`

	ResponseContentDisposition string `json:"responseContentDisposition,omitempty"`

	ResponseXContentDuration string `json:"responseXContentDuration,omitempty"`

	ResponseRefresh string `json:"responseRefresh,omitempty"` // Refresh header for HTTP-level redirect/meta refresh (e.g., "5; url=https://example.com")

	ResponseContentLanguage string `json:"responseContentLanguage,omitempty"`

	ResponseXUACompatible string `json:"responseXUaCompatible,omitempty"`

	ResponseAcceptRanges string `json:"responseAcceptRanges,omitempty"`

	ResponseTransferEncoding string `json:"responseTransferEncoding,omitempty"`

	ResponseContentRange string `json:"responseContentRange,omitempty"`

	ResponsePragma string `json:"responsePragma,omitempty"`

	ResponseXPoweredBy string `json:"responseXPoweredBy,omitempty"`

	ResponseXAspNetVersion string `json:"responseXAspNetVersion,omitempty"`

	ResponseXAspNetMvcVersion string `json:"responseXAspNetMvcVersion,omitempty"`

	ResponseServerTiming string `json:"responseServerTiming,omitempty"`

	ResponseXGenerator string `json:"responseXGenerator,omitempty"`

	ResponseXRuntime string `json:"responseXRuntime,omitempty"`

	ResponseXDrupalCache string `json:"responseXDrupalCache,omitempty"`

	ResponseXMagentoCacheControl string `json:"responseXMagentoCacheControl,omitempty"`

	ResponseXDrupalDynamicCache string `json:"responseXDrupalDynamicCache,omitempty"`

	ResponseXMagentoTags string `json:"responseXMagentoTags,omitempty"`

	ResponseXShopifyStage string `json:"responseXShopifyStage,omitempty"`

	ResponseXShopifyRequestID string `json:"responseXShopifyRequestId,omitempty"`

	ResponseXWPTotal      string `json:"responseXWpTotal,omitempty"`
	ResponseXWPTotalPages string `json:"responseXWpTotalPages,omitempty"`

	ResponseXCraftCache string `json:"responseXCraftCache,omitempty"`

	ResponseXDiscourseRoute string `json:"responseXDiscourseRoute,omitempty"`

	ResponseXGhostCacheStatus string `json:"responseXGhostCacheStatus,omitempty"`

	ResponseXJoomlaCache string `json:"responseXJoomlaCache,omitempty"`

	ResponseXDiscourseMediaType string `json:"responseXDiscourseMediaType,omitempty"`

	ResponseXPrestaShopCache string `json:"responseXPrestaShopCache,omitempty"`

	ResponseXMagentoCacheDebug string `json:"responseXMagentoCacheDebug,omitempty"`

	ResponseXTypo3Cache string `json:"responseXTypo3Cache,omitempty"`

	ResponseXWixRequestId string `json:"responseXWixRequestId,omitempty"`

	ResponseXSquarespaceRequestId string `json:"responseXSquarespaceRequestId,omitempty"`

	ResponseXWebflowRequestId string `json:"responseXWebflowRequestId,omitempty"`

	ResponseXContentfulRequestId string `json:"responseXContentfulRequestId,omitempty"`

	ResponseXNetlifyRequestId string `json:"responseXNetlifyRequestId,omitempty"`

	ResponseXVercelId string `json:"responseXVercelId,omitempty"`

	ResponseXHerokuRequestId string `json:"responseXHerokuRequestId,omitempty"`

	ResponseXRenderRequestId string `json:"responseXRenderRequestId,omitempty"`

	ResponseXRailwayRequestId string `json:"responseXRailwayRequestId,omitempty"`

	ResponseXFlyRequestId string `json:"responseXFlyRequestId,omitempty"`

	ResponseXDenoRegion string `json:"responseXDenoRegion,omitempty"`

	ResponseXCloudflareWorkersRequestId string `json:"responseXCloudflareWorkersRequestId,omitempty"`

	ResponseXAzureRef string `json:"responseXAzureRef,omitempty"`

	ResponseXGCPRegion string `json:"responseXGcpRegion,omitempty"`

	ResponseXAmzCfId string `json:"responseXAmzCfId,omitempty"`

	NormalizedCacheStatus string `json:"normalizedCacheStatus,omitempty"`
	CacheStatusSource     string `json:"cacheStatusSource,omitempty"`

	CDNProvider string   `json:"cdnProvider,omitempty"`
	CDNSignals  []string `json:"cdnSignals,omitempty"`

	ViaHops     []ViaHop `json:"viaHops,omitempty"`
	ProxyLayers int      `json:"proxyLayers,omitempty"`

	CacheAge       int  `json:"cacheAge,omitempty"`
	CacheMaxAge    int  `json:"cacheMaxAge,omitempty"`
	CacheSMaxAge   int  `json:"cacheSMaxAge,omitempty"`
	CacheFreshness int  `json:"cacheFreshness,omitempty"`
	CacheIsStale   bool `json:"cacheIsStale,omitempty"`

	CacheStaleWhileRevalidate int  `json:"cacheStaleWhileRevalidate,omitempty"`
	CacheStaleIfError         int  `json:"cacheStaleIfError,omitempty"`
	CacheHasSWR               bool `json:"cacheHasSwr,omitempty"`

	CDNEdgeLocation string `json:"cdnEdgeLocation,omitempty"`

	CacheMustRevalidate  bool     `json:"cacheMustRevalidate,omitempty"`
	CacheProxyRevalidate bool     `json:"cacheProxyRevalidate,omitempty"`
	CacheNoCache         bool     `json:"cacheNoCache,omitempty"`
	CacheNoCacheFields   []string `json:"cacheNoCacheFields,omitempty"`
	CacheNoStore         bool     `json:"cacheNoStore,omitempty"`
	CacheImmutable       bool     `json:"cacheImmutable,omitempty"`
	CachePrivate         bool     `json:"cachePrivate,omitempty"`
	CachePublic          bool     `json:"cachePublic,omitempty"`

	CachePolicySummary string `json:"cachePolicySummary,omitempty"`
	CachePolicyScore   int    `json:"cachePolicyScore,omitempty"`

	EffectiveCDNTTL       int      `json:"effectiveCdnTtl,omitempty"`
	CacheRecommendations  []string `json:"cacheRecommendations,omitempty"`
	CDNOptimizationIssues []string `json:"cdnOptimizationIssues,omitempty"`

	SurrogateMaxAge               int  `json:"surrogateMaxAge,omitempty"`
	SurrogateStaleWhileRevalidate int  `json:"surrogateStaleWhileRevalidate,omitempty"`
	SurrogateStaleIfError         int  `json:"surrogateStaleIfError,omitempty"`
	SurrogateNoStore              bool `json:"surrogateNoStore,omitempty"`
	SurrogateNoStoreRemote        bool `json:"surrogateNoStoreRemote,omitempty"`

	RequestID string `json:"requestId,omitempty"`

	CacheHitRateEstimate  string `json:"cacheHitRateEstimate,omitempty"`
	BandwidthSavingsLevel string `json:"bandwidthSavingsLevel,omitempty"`
	CacheCostAnalysis     string `json:"cacheCostAnalysis,omitempty"`

	DedupeApplied         bool     `json:"dedupeApplied,omitempty"`
	DuplicatesRemoved     int      `json:"duplicatesRemoved,omitempty"`
	DuplicateSignals      []string `json:"duplicateSignals,omitempty"`
	NearDuplicatesRemoved int      `json:"nearDuplicatesRemoved,omitempty"`
	NearDuplicateSignals  []string `json:"nearDuplicateSignals,omitempty"`
	OriginalBlockCount    int      `json:"originalBlockCount,omitempty"`
	UniqueBlockCount      int      `json:"uniqueBlockCount,omitempty"`

	Links    []LinkRef    `json:"links,omitempty"`
	Images   []ImageRef   `json:"images,omitempty"`
	Tables   []TableRef   `json:"tables,omitempty"`
	Comments []CommentRef `json:"comments,omitempty"`
	Chunks   []Chunk      `json:"chunks,omitempty"`

	SplitFiles []SplitFile `json:"splitFiles,omitempty"`

	RankedSections []RankedSection `json:"rankedSections,omitempty"`

	Schema map[string]interface{} `json:"schema,omitempty"`

	Warnings []string `json:"warnings,omitempty"`

	Truncated bool `json:"truncated,omitempty"`

	PruneFallbackUsed bool `json:"pruneFallbackUsed,omitempty"`

	ExtractionMethod string `json:"extractionMethod,omitempty"`

	// SecurityBlock carries the reason a fetch was refused by the SecurityPolicy
	// (SSRF / private-IP / blocked-scheme / blocked-domain / size-cap). Empty
	// when no policy is active or the fetch passed every check.
	SecurityBlock string `json:"securityBlock,omitempty"`
}

func FromHTML 

func FromHTML(html string, targetURL string) Result

func FromHTMLWithOptions 

func FromHTMLWithOptions(html string, targetURL string, opts Options) Result

FromHTMLWithOptions runs the full extraction pipeline on pre-fetched HTML with custom Options. Same as FromHTML, but flag-aware — use this when the caller has already obtained the body (e.g. piped from a browser fetcher) and wants links/citations/strip/etc. honoured.

func FromResponse 

func FromResponse(resp *http.Response, targetURL string, start time.Time) (result Result)

func FromURL 

func FromURL(targetURL string) Result

func FromURLWithDedupe 

func FromURLWithDedupe(targetURL string) Result

func FromURLWithOptions 

func FromURLWithOptions(targetURL string, opts Options) (result Result)

type RetryEvent 

type RetryEvent struct {
	Attempt    int
	StatusCode int
	WaitTime   time.Duration
	Error      error
	Outcome    string
}

type Schema 

type Schema struct {
	Fields map[string]FieldSpec `json:"fields" yaml:"fields"`
}

Schema is a declarative CSS-selector mapping from output field names to extraction specs. Top-level fields capture single values; nested Fields (on a FieldSpec) produce arrays of objects (one per matching element); Multiple=true produces a flat array of strings.

func LoadSchema 

func LoadSchema(path string) (Schema, error)

LoadSchema reads a schema file from disk and decodes it as YAML or JSON. The format is picked from the file extension (.yaml/.yml → YAML, .json → JSON). When the extension is ambiguous or unknown we try YAML first and fall back to JSON.

type SecurityPolicy 

type SecurityPolicy struct {
	// BlockPrivateIPs rejects targets that resolve to RFC1918 / loopback /
	// link-local / ULA / unspecified / multicast / carrier-grade-NAT / cloud
	// metadata (169.254.169.254) addresses. Enforced both pre-fetch and at the
	// dial Control hook (post-DNS) to close the DNS-rebinding window.
	BlockPrivateIPs bool

	// TrustedResolveCIDRs are CIDRs (or bare IPs) allowed to resolve to an
	// otherwise-blocked non-public address — the escape hatch for a known
	// internal host or a loopback proxy. Mirrors navguard's TrustedResolveCIDRs.
	TrustedResolveCIDRs []string

	// AllowedDomains, when non-empty, is an allowlist: a target host must equal
	// or be a sub-domain of one entry. DeniedDomains is a blocklist checked
	// first (deny wins). Both match on registrable-suffix (host == d or
	// host endsWith "."+d).
	AllowedDomains []string
	DeniedDomains  []string

	// AllowedSchemes restricts the URL scheme. Empty means "no scheme
	// restriction"; the default is {"http","https"} which rejects file:, ftp:,
	// gopher:, ws:, etc. (data: never reaches the network and is exempt.)
	AllowedSchemes []string

	// MaxRedirects caps redirect hops: >0 = cap at N, 0 = no redirects,
	// -1 = unlimited.
	MaxRedirects int

	// RevalidateRedirects re-runs scheme + domain + resolved-IP validation on
	// every redirect hop, blocking a public→internal redirect SSRF.
	RevalidateRedirects bool

	// MaxResponseBytes caps the raw (still-encoded) response body read off the
	// wire; 0 = unbounded. MaxDecompressedBytes caps decompressor output to
	// stop gzip/br/deflate/zstd decompression bombs; 0 = unbounded.
	MaxResponseBytes     int64
	MaxDecompressedBytes int64
}

SecurityPolicy is the opt-in, sane-by-default fetch policy threaded through the whole FromURL* path. It guards an extraction against SSRF, redirect-to- internal, DNS-rebinding, oversized-body, and decompression-bomb attacks when SeaPortal is pointed at attacker-controlled URLs.

Construct a secure default with DefaultSecurityPolicy. A nil *SecurityPolicy (the zero value of Options.Security) disables every check and preserves the historical "fetch anything" behaviour, so existing callers are unaffected.

The struct is pure configuration with no hidden mutable state, so a single policy value may be shared across concurrent FromURL* calls.

func DefaultSecurityPolicy 

func DefaultSecurityPolicy() *SecurityPolicy

DefaultSecurityPolicy returns the recommended secure-by-default posture: block private/internal IPs, http/https only, a 10-redirect cap with per-hop revalidation, and 50 MiB raw / 200 MiB decompressed body caps. This is what the CLI and the MCP server apply by default; library callers opt in by setting Options.Security.

func (*SecurityPolicy) ValidateURL 

func (p *SecurityPolicy) ValidateURL(ctx context.Context, rawURL string) error

ValidateURL enforces scheme + domain + resolved-IP rules on a single URL. It is the pre-fetch gate (and the per-hop redirect gate). A nil policy or an unparseable URL is a no-op — the normal fetch path surfaces parse errors so the security layer never masks them.

type SitemapEntry 

type SitemapEntry struct {
	Loc        string `json:"loc"`
	LastMod    string `json:"lastmod,omitempty"`
	ChangeFreq string `json:"changefreq,omitempty"`
	Priority   string `json:"priority,omitempty"`
}

SitemapEntry is a single `<url>` entry flattened from a sitemap.

func FlattenSitemap 

func FlattenSitemap(ctx context.Context, sitemapURL string, opts FlattenSitemapOptions) ([]SitemapEntry, error)

FlattenSitemap fetches sitemapURL and recursively flattens sitemap-index references into a single slice of SitemapEntry. Bounded by opts.MaxDepth and opts.MaxURLs. Deduplicated by Loc. Handles `.gz` URLs and `Content-Encoding: gzip`.

type SnapshotNode 

type SnapshotNode struct {
	Role        string         `json:"role"`
	Name        string         `json:"name,omitempty"`
	Tag         string         `json:"tag,omitempty"`
	Ref         string         `json:"ref,omitempty"`
	Selector    string         `json:"selector,omitempty"`
	Depth       int            `json:"depth,omitempty"`
	Interactive bool           `json:"interactive,omitempty"`
	Level       int            `json:"level,omitempty"`
	Value       string         `json:"value,omitempty"`
	Href        string         `json:"href,omitempty"`
	Checked     *bool          `json:"checked,omitempty"`
	Disabled    bool           `json:"disabled,omitempty"`
	Children    []SnapshotNode `json:"children,omitempty"`
}

SnapshotNode represents a node in the accessibility tree

func BuildSnapshot 

func BuildSnapshot(htmlStr string) (*SnapshotNode, error)

BuildSnapshot parses HTML and returns an accessibility tree

func BuildSnapshotWithOptions 

func BuildSnapshotWithOptions(htmlStr string, opts SnapshotOptions) (*SnapshotNode, error)

BuildSnapshotWithOptions parses HTML with configurable options

func (*SnapshotNode) ToCompact 

func (n *SnapshotNode) ToCompact() string

ToCompact returns a compact text representation of the tree

type SnapshotOptions 

type SnapshotOptions struct {
	FilterInteractive bool // Only include interactive elements
	MaxTokens         int  // Approximate token limit (0 = unlimited)
}

SnapshotOptions configures snapshot generation

type SplitConfig 

type SplitConfig struct {
	Dir      string
	MaxBytes int    // soft cap per file; 0 = use default (32 KB)
	BaseName string // optional; defaults to URL-slug-derived name
	Format   string // "md" | "json" (default "md")
}

SplitConfig controls SplitResultToFiles.

type SplitFile 

type SplitFile struct {
	Path  string `json:"path"`
	Index int    `json:"index"`
	Of    int    `json:"of"`
	Bytes int    `json:"bytes"`
}

SplitFile is one entry in the manifest returned by SplitResultToFiles.

func SplitResultToFiles 

func SplitResultToFiles(r Result, cfg SplitConfig) ([]SplitFile, error)

SplitResultToFiles writes r's content into one or more files under cfg.Dir. Prefers r.Chunks when present; otherwise paragraph-splits r.Content. Returns the manifest of written files with absolute paths. Empty Content + no chunks returns (nil, nil) and writes nothing.

type TableKind 

type TableKind int

TableKind classifies whether a <table> carries semantic tabular data or is being used purely for visual layout. Layout tables get flattened during preprocess; data tables are preserved (and optionally extracted as structured TableRef values when Options.WithTables is set). 

const (
	TableData TableKind = iota
	TableLayout
)

type TableRef 

type TableRef struct {
	Caption string     `json:"caption,omitempty"`
	Headers []string   `json:"headers,omitempty"`
	Rows    [][]string `json:"rows"`
}

TableRef is a structured, host-agnostic representation of an HTML <table> classified as data. Surfaced on Result.Tables when the caller opts in via Options.WithTables.

Cells with colspan/rowspan expand into a regular grid; rows in TableRef.Rows have the same column count as headers when present. Cell-internal HTML (<a>, <strong>, etc.) is flattened to text; HTML entities are decoded; internal whitespace is collapsed.

func ExtractTables 

func ExtractTables(htmlStr string, _ string) []TableRef

ExtractTables walks htmlStr and returns one TableRef per <table> classified as data. baseURL is reserved for future use (e.g. resolving in-cell hrefs); V1 only extracts text.

type TextFallbackResult 

type TextFallbackResult struct {
	Content  string
	Length   int
	Headings int
	Links    int
}

TextFallbackResult holds results from text-based fallback extraction

func TextFallback 

func TextFallback(htmlStr string) TextFallbackResult

TextFallback extracts visible text content directly from HTML when readability fails. This is a last-resort extraction that finds all text nodes in the body, skipping script/style/nav/footer elements, and formats them as markdown.

type Validation 

type Validation struct {
	IsValid       bool     `json:"isValid"`
	NeedsBrowser  bool     `json:"needsBrowser"`
	Confidence    float64  `json:"confidence"`
	Issues        []string `json:"issues"`
	LinkDensity   float64  `json:"linkDensity"`
	SkeletonScore float64  `json:"skeletonScore"`
	ContentRatio  float64  `json:"contentRatio"`
}

func ValidateExtraction 

func ValidateExtraction(r *Result) Validation

type ViaHop 

type ViaHop struct {
	Protocol string `json:"protocol,omitempty"` // HTTP protocol version (e.g., "1.1", "HTTP/1.1", "2")
	Host     string `json:"host,omitempty"`     // Proxy/CDN hostname or identifier
	Comment  string `json:"comment,omitempty"`  // Optional comment (e.g., "(squid)", "(Varnish)")
}

ViaHop represents a single hop in the Via header chain. Format: [protocol] host [comment] Example: "1.1 varnish", "HTTP/1.1 cache.example.com (squid)", "1.1 google"

Source Files

View all Source files

Directories

Path Synopsis
Package leakcheck provides a test helper that fails a Go test when it finishes with more live goroutines than it started with.
 Package leakcheck provides a test helper that fails a Go test when it finishes with more live goroutines than it started with.
Package mock provides a record/replay HTTP RoundTripper layer for tests that exercise the engine's HTTP-fetch paths without paying real-network latency or flakiness costs.
 Package mock provides a record/replay HTTP RoundTripper layer for tests that exercise the engine's HTTP-fetch paths without paying real-network latency or flakiness costs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.