Documentation
¶
Overview ¶
Package hybrid implements the functionality for a hybrid-headless crawler. It uses both headless browser and net/http for making requests, and goquery for processing rawand dom-rendered web page HTML.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func FetchContinueRequest ¶
func FetchContinueRequest(page *rod.Page, e *proto.FetchRequestPaused) error
FetchContinueRequest continue request
func FetchGetResponseBody ¶
FetchGetResponseBody get request body.
Types ¶
type Crawler ¶
type Crawler struct {
// contains filtered or unexported fields
}
Crawler is a standard crawler instance
func New ¶
func New(options *types.CrawlerOptions) (*Crawler, error)
New returns a new standard crawler instance
func (*Crawler) BrowserContextID ¶ added in v1.8.0
BrowserContextID returns the id of the browser context this crawler created in New, or "" if it has no browser. It lets an embedding program dispose the context out of band: when a crawl dies without running Close, the context otherwise stays resident in a shared, long-lived browser.
func (*Crawler) Do ¶ added in v1.4.0
func (c *Crawler) Do(crawlSession *common.CrawlSession, doRequest common.DoRequestFunc) error
Do executes the crawling loop with browser-safe concurrency. Unlike the base implementation, this uses sequential processing (concurrency=1) because Chrome DevTools Protocol operations cannot safely run concurrently on the same browser instance. Multiple concurrent page operations cause race conditions, navigation conflicts, and network interception issues.
func (*Crawler) SetHooks ¶ added in v1.8.0
SetHooks installs per-page lifecycle callbacks on the hybrid crawler. The supplied struct is copied, so mutating it after SetHooks returns has no effect; call SetHooks again to change the installed hooks. Passing nil clears any previously installed hooks. SetHooks is not safe to call concurrently with Crawl on the same crawler.
type Hijack ¶
type Hijack struct {
// contains filtered or unexported fields
}
Hijack is a hijack handler
func (*Hijack) SetPattern ¶
func (h *Hijack) SetPattern(pattern *proto.FetchRequestPattern)
SetPattern set pattern directly
type HijackHandler ¶
type HijackHandler = func(e *proto.FetchRequestPaused) error
HijackHandler type
type Hooks ¶ added in v1.8.0
type Hooks struct {
// headers/user agent are applied, and before request interception starts
// and the page navigates to req.URL. It is the place for per-page setup
// such as page.SetCookies or page.EvalOnNewDocument.
BeforeNavigate func(ctx context.Context, page *rod.Page, req *navigation.Request) error
// AfterLoad is invoked once the page has loaded according to the
// page-load strategy, its rendered HTML has been captured into resp, and
// before the page is closed. It is only called when resp is non-nil and
// the request has not already failed. The callback may read the live page
// (evaluate scripts, take a screenshot) and may attach data to the result
// via resp.Extra, which is delivered to OnResult and the output writers.
AfterLoad func(ctx context.Context, page *rod.Page, req *navigation.Request, resp *navigation.Response) error
}
Hooks bundles optional per-page lifecycle callbacks invoked by the hybrid crawler for every navigated page. All fields are optional; nil callbacks are skipped, so the zero value changes nothing.
Both callbacks run synchronously on the goroutine that navigates the page and block it for their duration. If Crawl is invoked concurrently on the same crawler, the callbacks must themselves be safe for concurrent use.
page is the crawler's own tab, bound to the crawl session's context (it is cancelled with the crawl). ctx is cancelled with the crawl and carries a -timeout deadline: for BeforeNavigate it is the navigation's own context; AfterLoad gets a fresh -timeout budget, so a page that loaded slowly does not leave it an almost-expired context. Use page.Context(ctx) or page.Timeout(d) to bound any CDP call made from a callback. The page is closed by the crawler as soon as the callback returns: callbacks must not retain it, close it, or navigate it.
A non-nil error from a callback aborts that request: it is reported exactly like any other navigation error (an output.Result carrying the error, plus the error writer), and the crawl continues with the next request. Return nil for failures that should not cost the page. Callbacks must not panic.
Source Files