scrape

package
v0.13.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 25, 2026 License: MIT Imports: 27 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// SourceHTTP — page is not a JS shell; plain HTTP fetch is authoritative.
	SourceHTTP = "http"
	// SourceHTTPShell — page is a JS shell but rendering wasn't possible
	// (no browser configured, or browser fetch failed). Always invalid as a
	// cache hit when a browser is now available, so we retry rendering.
	SourceHTTPShell = "http_shell"
	// SourceBrowser — page was rendered via the headless browser.
	SourceBrowser = "browser"
)

Source identifies how a Page was fetched. Stored in the cache so we can invalidate stale entries (notably JS-shell extractions cached before the user configured a browser) without churning entries for plain HTTP pages that don't need rendering.

View Source 
const MaxBodyBytes = 20 << 20 // 20 MiB

MaxBodyBytes caps how much of an HTTP response body we will read. Prevents a malicious or misconfigured server from OOMing the process.

Variables

View Source 
var (
	// ErrBadSelector wraps selector-extraction failures (typically an invalid
	// CSS selector) — a caller-input problem, not an upstream fault.
	ErrBadSelector = errors.New("selector extraction failed")
	// ErrSelectorNoMatch reports that the selector matched no elements.
	ErrSelectorNoMatch = errors.New("no elements matched selector")
	// ErrPDFSelectorUnsupported reports that CSS selectors only apply to HTML.
	ErrPDFSelectorUnsupported = errors.New("CSS selector extraction is not supported for PDF documents")
	// ErrPDFRawUnsupported reports that returning PDF binary data is forbidden.
	ErrPDFRawUnsupported = errors.New("raw output is not supported for PDF documents")
)

Sentinel errors for selector scrapes so callers can classify failures (CLI exit codes, MCP error kinds) without string matching.

View Source 
var ErrNoBrowser = errors.New("no browser configured (set with: ketch config set browser chrome)")

ErrNoBrowser is returned when browser rendering is needed but not configured.

View Source 
var Version = "dev"

Version is the ketch version string embedded into DefaultUserAgent. cmd sets this from build-time version info at process start; library callers that leave it untouched get "dev".

Functions

func CacheStaleForBrowser 

func CacheStaleForBrowser(source string, hasBrowser bool) bool

CacheStaleForBrowser reports whether a cache entry with the given source should be bypassed because rendering via the browser would do better. True when the cached entry is a known unrendered JS shell, or when it predates source tracking and a browser is now available (one-time migration for pre-source caches where the entry might be unrendered shell garbage).

func ContentHash 

func ContentHash(s string) string

ContentHash returns the first 16 hex chars of the sha256 of s.

func DefaultUserAgent added in v0.13.0 

func DefaultUserAgent() string

DefaultUserAgent is the honest HTTP User-Agent ketch sends when no operator override is configured. It identifies ketch and points at the project; it does not impersonate a browser.

func FetchLLMSTxt 

func FetchLLMSTxt(ctx context.Context, baseURL string) (string, bool)

FetchLLMSTxt attempts to fetch /llms.txt anonymously. It is kept for package compatibility; callers with a Scraper should use its method so the configured cookie jar and redirect re-scoping apply.

func InstallBrowser 

func InstallBrowser() (string, error)

InstallBrowser downloads Chromium to the ketch cache directory.

The revision directory is removed before downloading. Extraction is not atomic: an interrupted download leaves a partial tree behind, and the next attempt fails on it rather than replacing it — either because the unzip hits existing symlinks ("file exists") or because the leftover confuses the single-directory check during extraction. Both surface as errors that look unrelated to the real cause, so retries appear to fail differently each time. Clearing first makes every install start from a known state.

func ResolveBrowserBin 

func ResolveBrowserBin(configured string) (string, error)

ResolveBrowserBin resolves a browser configuration value to an absolute path. Accepts "chrome", "chromium" (searched in PATH), or an absolute path.

Types

type BrowserConn 

type BrowserConn interface {
	// Fetch navigates to a URL and returns the rendered HTML. The context
	// bounds navigation and JS settling; cancellation unblocks the caller.
	Fetch(ctx context.Context, url string) (html string, err error)
	// Close releases browser resources.
	Close()
}

BrowserConn represents a connection to a headless browser for JS rendering.

func NewBrowserConn 

func NewBrowserConn(binPath string) (BrowserConn, error)

NewBrowserConn launches a headless browser without cookie injection. This legacy signature is preserved for external package callers.

func NewBrowserConnWithCookies added in v0.12.0 

func NewBrowserConnWithCookies(binPath string, jar *cookies.Jar) (BrowserConn, error)

NewBrowserConnWithCookies launches a headless browser and injects cookies from jar (which may be nil) before each navigation.

type FetchResult 

type FetchResult struct {
	Page        *Page
	RawHTML     string
	ContentType string
	NotModified bool
	JSDetection string // "static", "likely_shell", "ambiguous"

	// Doc is the parsed document that ScrapeConditional used for JS-shell
	// detection. Downstream callers (e.g. link extraction during a crawl)
	// can reuse it to avoid re-parsing the same HTML. Nil when the page
	// was re-fetched via the browser — in that case RawHTML is the
	// rendered HTML and needs a fresh parse.
	Doc *goquery.Document

	// Source is the fetch path that produced Page (SourceHTTP or SourceBrowser).
	Source string
}

FetchResult holds the output of a conditional scrape.

type FetchedContent added in v0.12.0 

type FetchedContent struct {
	Body        []byte
	ContentType string
}

FetchedContent preserves the response body bytes and declared content type.

type Page 

type Page struct {
	URL          string `json:"url"`
	FetchedURL   string `json:"fetched_url,omitempty"`
	Title        string `json:"title"`
	Markdown     string `json:"markdown"`
	ETag         string `json:"etag,omitempty"`
	LastModified string `json:"last_modified,omitempty"`
	ContentHash  string `json:"content_hash,omitempty"`
}

Page represents a scraped web page.

type PageCache 

type PageCache interface {
	Get(url string) (*Page, string)
	Put(url string, page *Page, source string)
	GetRaw(url string) (rawHTML, source string, page *Page)
	PutRaw(url string, page *Page, source, rawHTML string)
}

PageCache is the subset of the cache API the scrape pipeline needs. *cache.Cache implements it (the cache package imports scrape, so the dependency points this way via an interface). Callers may pass nil to bypass caching entirely.

type Scraper 

type Scraper struct {
	// contains filtered or unexported fields
}

Scraper fetches web pages and extracts content as markdown.

func New 

func New() *Scraper

New creates a Scraper with defaults.

func NewFromConfig 

func NewFromConfig(cfg *config.Config) (*Scraper, error)

NewFromConfig builds a Scraper from cfg: compiled URL rewriter, optional browser binary, and operator-configured SPA markers. The rewriter's regexes are compiled once here — callers should construct one Scraper and reuse it. The returned Scraper is safe for concurrent use and must be Closed by the caller.

func NewWithBrowser 

func NewWithBrowser(browserBin string) *Scraper

NewWithBrowser creates a Scraper with browser fallback for JS-rendered pages.

func NewWithBrowserConn 

func NewWithBrowserConn(conn BrowserConn, rw *urlrewrite.Rewriter) *Scraper

NewWithBrowserConn creates a Scraper backed by a pre-supplied BrowserConn, bypassing binary resolution. HasBrowser reports true and getBrowser returns conn directly. Used to drive browser code paths (e.g. --force-browser) without a real Chrome install.

func NewWithConfig 

func NewWithConfig(browserBin string, rw *urlrewrite.Rewriter, spaMarkers []string) *Scraper

NewWithConfig creates a Scraper like NewWithRewriter but folds operator- configured SPA markers (config spa_markers) into the JS-shell detector in addition to the built-in markers. Pass nil/empty spaMarkers for built-ins only.

func NewWithRewriter 

func NewWithRewriter(browserBin string, rw *urlrewrite.Rewriter) *Scraper

NewWithRewriter creates a Scraper with an optional browser binary and optional URL rewriter. Pass "" for browserBin to disable browser fallback; pass nil rewriter to disable URL rewriting. The detector uses only the built-in SPA markers; use NewWithConfig to add operator-configured markers.

func (*Scraper) BrowserScrape 

func (s *Scraper) BrowserScrape(ctx context.Context, rawURL string) (*Page, string, error)

BrowserScrape fetches a URL using the browser directly. Used when a host is known to require browser rendering.

func (*Scraper) CacheKey added in v0.12.0 

func (s *Scraper) CacheKey(fetchURL string) string

CacheKey returns the page-cache key for an already-rewritten fetch URL. A configured jar with live cookies gets a jar-specific namespace even when no cookie matches the initial URL: a redirect may land on another host or path where a cookie does match. This prevents redirected authenticated content from colliding with an anonymous cache entry.

func (*Scraper) CachedScrape 

func (s *Scraper) CachedScrape(ctx context.Context, pc PageCache, url string) (*Page, error)

CachedScrape checks the cache first, falls back to fetch+extract. Hits are bypassed when the entry was extracted from an unrendered JS shell and a browser is now available to do better, or when the entry predates source tracking (a one-time migration once a browser is configured). The cache is keyed by the rewritten URL so original and rewritten URLs share one cache entry.

func (*Scraper) CachedScrapeForce 

func (s *Scraper) CachedScrapeForce(ctx context.Context, pc PageCache, url string) (*Page, error)

CachedScrapeForce is the forced-browser markdown path. It classifies a plain HTTP response before any browser render so PDFs bypass Chromium's PDF viewer and use the configured PDF extractor. For HTML, a cache entry is reused only when that entry is itself a browser render.

func (*Scraper) CachedScrapeRaw 

func (s *Scraper) CachedScrapeRaw(ctx context.Context, pc PageCache, url string) (*Page, string, string, error)

CachedScrapeRaw is the raw-HTML path. It routes through ScrapeConditional so one fetch yields Page + RawHTML + Source (the markdown path's Scrape discards the body). Raw lookup is a hit only when RawHTML is non-empty — a markdown-only entry does not poison a raw request. On a raw miss against an existing markdown entry, the refetch back-fills RawHTML while preserving the cached Page (one fetch, both representations cached). A nil pc skips cache read/write and returns the fresh fetch result directly.

func (*Scraper) CachedScrapeRawForce 

func (s *Scraper) CachedScrapeRawForce(ctx context.Context, pc PageCache, url string) (*Page, string, string, error)

CachedScrapeRawForce is the forced-browser raw path. It classifies the HTTP response before consulting the browser cache or rendering, rejecting PDFs rather than returning Chromium's PDF-viewer HTML.

func (*Scraper) Close 

func (s *Scraper) Close()

Close releases browser resources if any.

func (*Scraper) Fetch 

func (s *Scraper) Fetch(ctx context.Context, rawURL string) (string, error)

Fetch is the compatibility wrapper for callers that expect a string body.

func (*Scraper) FetchContent added in v0.12.0 

func (s *Scraper) FetchContent(ctx context.Context, rawURL string) (*FetchedContent, error)

FetchContent fetches a URL without extraction or browser fallback while preserving response bytes and the server-declared content type.

func (*Scraper) FetchLLMSTxt added in v0.12.0 

func (s *Scraper) FetchLLMSTxt(ctx context.Context, baseURL string) (string, bool)

FetchLLMSTxt attempts to fetch /llms.txt from a bare-domain URL through the scraper's authenticated HTTP path. All errors are swallowed because this is only a best-effort shortcut before the requested page is scraped.

func (*Scraper) HasBrowser 

func (s *Scraper) HasBrowser() bool

HasBrowser reports whether this scraper has browser fallback configured. Guarded by browserMu because getBrowser clears browserBin on failed resolution — concurrent callers (MCP tool calls, multi-URL scrapes) must not race that write.

func (*Scraper) MaybeBrowserFetch 

func (s *Scraper) MaybeBrowserFetch(ctx context.Context, rawURL, html string) (string, string)

MaybeBrowserFetch re-fetches rawURL via the browser if html looks JS-rendered. Returns the (possibly rendered) html and the fetch source actually used.

func (*Scraper) Rewrite 

func (s *Scraper) Rewrite(url string) string

Rewrite returns the URL after applying configured rewrite rules, or the original URL if no rule matches or no rewriter is configured. Safe to call on a Scraper with no rewriter (nil-safe via urlrewrite.Rewriter).

func (*Scraper) Scrape 

func (s *Scraper) Scrape(ctx context.Context, rawURL string) (*Page, string, error)

Scrape fetches a URL and returns extracted markdown content along with the fetch source (SourceHTTP or SourceBrowser). If the page appears JS-rendered and a browser is configured, automatically retries with the browser for full content extraction.

func (*Scraper) ScrapeConditional 

func (s *Scraper) ScrapeConditional(ctx context.Context, rawURL, etag, lastModified string) (*FetchResult, error)

ScrapeConditional fetches a URL with conditional headers and JS detection.

func (*Scraper) ScrapeMarkdown 

func (s *Scraper) ScrapeMarkdown(ctx context.Context, pc PageCache, url string, forceBrowser bool) (*Page, error)

ScrapeMarkdown picks the markdown fetch path. Forced-browser HTML renders; forced-browser PDFs still use text extraction after content classification.

func (*Scraper) ScrapeRaw 

func (s *Scraper) ScrapeRaw(ctx context.Context, pc PageCache, url string, forceBrowser bool) (*Page, string, string, error)

ScrapeRaw picks the raw-HTML fetch path, rejecting PDFs before any forced browser render.

func (*Scraper) ScrapeSelector 

func (s *Scraper) ScrapeSelector(ctx context.Context, rawURL, selector string, forceBrowser bool) (*Page, error)

ScrapeSelector fetches rawURL and returns only elements matching the CSS selector, converted to markdown — bypassing readability extraction and the page cache. Under forceBrowser it first classifies the response, rejects PDFs, then renders HTML and selects against the rendered DOM; otherwise it fetches plain HTTP with JS-shell auto-detection. The URL is rewritten before fetch so selector scrapes share the canonical URL-rewrite path with Scrape/ScrapeConditional.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.