fetch

package
v0.0.27 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 28, 2026 License: MIT Imports: 26 Imported by: 0

Documentation

Overview

Package fetch provides the dual-mode fetching layer for Foxhound.

stealth_default.go is a FALLBACK with NO TLS fingerprint impersonation. Production code should build with -tags tls to use stealth_tls.go. Without -tags tls, the TLS ClientHello is Go's crypto/tls default — trivially detected by every modern anti-bot system. Headers will look like Firefox but the TLS layer will not. This is almost always the wrong choice for scraping production targets that have any bot defence.

To verify which build you have: 

fetcher := fetch.NewStealth(...)
if !fetcher.IsImpersonating() { log.Fatal("built without -tags tls") }

Or check the binary: 

go tool nm /path/to/binary | grep -q azuretls && echo OK || echo MISSING

Build the impersonating variant with: 

go build -tags tls ./...
go test  -tags tls ./fetch/...

Index

Constants

View Source 
const CaptureXHRMetaKey = "_foxhound_capture_xhr"

CaptureXHRMetaKey is the Job.Meta key under which Trail.CaptureXHR stores per-job XHR capture URL patterns. The walker passes the job through to the camoufox fetcher unchanged; the fetcher merges these patterns with any fetcher-global patterns configured via WithCaptureXHR.

Variables

This section is empty.

Functions

func ContentOnlyResources 

func ContentOnlyResources() map[ResourceType]bool

ContentOnlyResources returns a minimal set of resources to block that preserves page layout while eliminating heavy binary assets. Stylesheets are NOT blocked so the page renders correctly for layout-dependent extraction.

func DefaultBlockedResources 

func DefaultBlockedResources() map[ResourceType]bool

DefaultBlockedResources returns the standard set of resource types to block for speed optimization. Blocking these typically cuts page load time by 30-70% for content-only scraping.

Types

type BlockDetector 

type BlockDetector interface {
	// IsBlocked returns true when the response should trigger escalation to
	// the browser fetcher.
	IsBlocked(resp *foxhound.Response) bool
}

BlockDetector determines whether an HTTP response indicates that the server has blocked or rate-limited the scraper.

type BrowserCookie 

type BrowserCookie struct {
	Name     string
	Value    string
	Domain   string
	Path     string
	Secure   bool
	HttpOnly bool
}

BrowserCookie represents a cookie to inject into the browser context.

type CamoufoxFetcher 

type CamoufoxFetcher struct {
	// contains filtered or unexported fields
}

CamoufoxFetcher drives a Camoufox (Firefox fork) browser instance via the Juggler protocol using playwright-go.

Stub build (!playwright tag): Fetch always returns errPlaywrightNotConfigured. Real build ( playwright tag): see camoufox_playwright.go.

func NewCamoufox 

func NewCamoufox(opts ...CamoufoxOption) (*CamoufoxFetcher, error)

NewCamoufox creates a CamoufoxFetcher. In the stub build no browser is launched; the constructor stores the configuration and returns immediately.

func (*CamoufoxFetcher) Close 

func (f *CamoufoxFetcher) Close() error

Close is a no-op in the stub build; no resources were allocated.

func (*CamoufoxFetcher) ExportStorageState 

func (f *CamoufoxFetcher) ExportStorageState() (*StorageState, error)

ExportStorageState is a stub; always returns an error in the non-playwright build.

func (*CamoufoxFetcher) Fetch 

func (f *CamoufoxFetcher) Fetch(_ context.Context, _ *foxhound.Job) (*foxhound.Response, error)

Fetch always returns errPlaywrightNotConfigured in the stub build. Rebuild with -tags playwright to enable real browser navigation.

func (*CamoufoxFetcher) LoadStorageState 

func (f *CamoufoxFetcher) LoadStorageState() (*StorageState, error)

LoadStorageState reads saved state from the configured file path. Returns (nil, nil) when no path is set or the file does not exist yet.

func (*CamoufoxFetcher) SaveStorageState 

func (f *CamoufoxFetcher) SaveStorageState() error

SaveStorageState exports and saves the session state to the configured file path. In the stub build, this falls back to saving the in-memory cookie list.

type CamoufoxOption 

type CamoufoxOption func(*CamoufoxFetcher)

CamoufoxOption is a functional option for configuring a CamoufoxFetcher.

func WithBehaviorProfile 

func WithBehaviorProfile(p *behavior.BehaviorProfile) CamoufoxOption

WithBehaviorProfile sets the behavior profile for scroll and keyboard configs. In the stub build this stores the value but has no effect.

func WithBlockImages 

func WithBlockImages(block bool) CamoufoxOption

WithBlockImages controls whether the browser route handler intercepts and aborts image/media/font requests. Reduces bandwidth and speeds up navigation for content-only scraping.

func WithBrowserCookies 

func WithBrowserCookies(cookies []BrowserCookie) CamoufoxOption

WithBrowserCookies sets cookies to inject into the browser context before page navigation. Useful for pre-authenticated sessions.

func WithBrowserIdentity 

func WithBrowserIdentity(p *identity.Profile) CamoufoxOption

WithBrowserIdentity sets the identity profile used to configure the Camoufox browser context. All CAMOU_CONFIG environment variables are derived from p.

func WithBrowserProxy 

func WithBrowserProxy(proxyURL string) CamoufoxOption

WithBrowserProxy sets the proxy URL for all browser requests. Supports SOCKS5 (socks5://user:pass@host:port) and HTTP (http://user:pass@host:port).

func WithBrowserTimeout 

func WithBrowserTimeout(d time.Duration) CamoufoxOption

WithBrowserTimeout overrides the default per-navigation timeout.

func WithCDPURL 

func WithCDPURL(url string) CamoufoxOption

WithCDPURL sets a Chrome DevTools Protocol endpoint URL (e.g. "http://localhost:9222"). When non-empty the real build connects to that existing browser instance via pw.Chromium.ConnectOverCDP instead of launching a new browser process. In the stub build this stores the value but has no effect.

func WithCaptureXHR 

func WithCaptureXHR(patterns ...*regexp.Regexp) CamoufoxOption

WithCaptureXHR configures URL patterns for XHR/fetch response capture. Captured responses are available in Response.CapturedXHR after fetch.

func WithExtensionPath 

func WithExtensionPath(path string) CamoufoxOption

WithExtensionPath sets the path to a Firefox extension directory to load. By default, NopeCHA is auto-downloaded and loaded. Set to "none" to disable. In the stub build this stores the value but has no effect — the extension is only loaded when compiled with the playwright build tag.

func WithHeadless 

func WithHeadless(mode string) CamoufoxOption

WithHeadless sets the display mode for the Camoufox browser:

  • "virtual": Xvfb virtual display (recommended on servers without a GPU)
  • "true": native headless mode
  • "false": full visible browser (useful for local debugging)

func WithInitScript 

func WithInitScript(script string) CamoufoxOption

WithInitScript sets a JavaScript snippet that is injected into every new page before the page's own scripts execute. This is useful for overriding navigator properties (e.g. navigator.webdriver) or installing global hooks. In the stub build this stores the value but has no effect.

func WithInterceptConfig 

func WithInterceptConfig(ic *InterceptConfig) CamoufoxOption

WithInterceptConfig wires a route-level blocking configuration into the browser. Resources matching BlockedResourceTypes are aborted before they reach the network, and requests to BlockedDomains (or their subdomains) are also aborted. Use NewInterceptConfig to construct the value, then pass it here. In the stub build this stores the value but has no effect.

func WithMaxBrowserRequests 

func WithMaxBrowserRequests(n int) CamoufoxOption

WithMaxBrowserRequests configures the browser instance to be restarted after serving n requests. This clears accumulated state (cookies, cache, memory leaks) and rotates the browser fingerprint.

Set n=0 to disable automatic restarts (not recommended for long-running hunts). The default is 300.

func WithPageReuseLimit 

func WithPageReuseLimit(n int) CamoufoxOption

WithPageReuseLimit sets the maximum number of requests a pooled page handles before being destroyed and replaced. This prevents long-lived pages from accumulating state that increases detection risk. Default 0 means unlimited. Recommended: 50-200 for anti-detection scraping. In the stub build this stores the value but has no effect.

func WithPersistSession 

func WithPersistSession(persist bool) CamoufoxOption

WithPersistSession controls whether the same BrowserContext is reused across requests for the same walker session (cookies and localStorage are preserved between fetches). When false (default) a fresh context is created per fetch for full isolation between requests.

Use true when scraping sites that require login state to be maintained across multiple page visits in a single session.

func WithPoolSize 

func WithPoolSize(n int) CamoufoxOption

WithPoolSize enables page pooling with the given max size. When enabled, browser contexts and pages are reused instead of created fresh for each request, significantly reducing per-request overhead (~3s context creation cost is eliminated after warmup). Only applies when PersistSession is false (non-persistent mode). Set to 0 (default) to disable pooling and use per-request contexts. In the stub build this stores the value but has no effect.

func WithRealChrome 

func WithRealChrome(use bool) CamoufoxOption

WithRealChrome switches the real build from Firefox/Camoufox to pw.Chromium.Launch with channel="chrome". Use this when you need Chrome's rendering behaviour or have a Chrome installation but not Camoufox. Falls back to Chromium if the Chrome channel is not installed. In the stub build this stores the value but has no effect.

func WithSkipExtension 

func WithSkipExtension() CamoufoxOption

WithSkipExtension prevents the NopeCHA addon from being auto-loaded. Use when the NopeCHA Token API solver is active — the API and addon should not run simultaneously. In the stub build this stores the value but has no effect.

func WithSolveCloudflare 

func WithSolveCloudflare(timeout time.Duration) CamoufoxOption

WithSolveCloudflare enables verified Cloudflare challenge resolution. After the existing handler runs (extension addon or manual click), the fetcher polls for up to timeout for three success signals: the cf_clearance cookie is present, Turnstile DOM markers are gone, and the cf-turnstile-response token is set. When all three pass within the budget the resulting Response has CloudflareSolved=true. On timeout the fetch is NOT failed; the response is still returned with CloudflareSolved=false so callers can decide whether to retry, escalate, or accept a partial page.

Pass 0 to disable verified-solve mode (default behaviour).

In the stub build this stores the value but has no effect.

func WithStorageState 

func WithStorageState(path string) CamoufoxOption

WithStorageState sets a file path for session state persistence. In the stub build this stores the value but has no effect.

func WithUserDataDir 

func WithUserDataDir(dir string) CamoufoxOption

WithUserDataDir sets a persistent profile directory. When non-empty the real build uses LaunchPersistentContext so cookies, localStorage, and cached resources survive across browser restarts. In the stub build this stores the value but has no effect.

type CapturePattern 

type CapturePattern struct {
	Pattern *regexp.Regexp
}

CapturePattern defines a URL pattern to capture.

type CapturedExchange 

type CapturedExchange struct {
	RequestURL    string            `json:"request_url"`
	RequestMethod string            `json:"request_method"`
	Status        int               `json:"status"`
	Headers       map[string]string `json:"headers,omitempty"`
	Body          []byte            `json:"body,omitempty"`
}

CapturedExchange holds a captured XHR/fetch request-response pair.

type ContentBlockDetector 

type ContentBlockDetector struct{}

ContentBlockDetector extends status-code-only detection with body-based CAPTCHA and soft-block detection using the captcha package. This is the default detector used by NewSmart.

func (*ContentBlockDetector) IsBlocked 

func (d *ContentBlockDetector) IsBlocked(resp *foxhound.Response) bool

IsBlocked returns true when the response indicates a block — either via status code (same as DefaultBlockDetector) or via body content (CAPTCHA pages, soft blocks that return 200).

type DefaultBlockDetector 

type DefaultBlockDetector struct{}

DefaultBlockDetector treats common anti-scraping status codes as blocks: 401 Unauthorised, 403 Forbidden, 407 Proxy Auth Required, 429 Too Many Requests, 503 Service Unavailable.

func (*DefaultBlockDetector) IsBlocked 

func (d *DefaultBlockDetector) IsBlocked(resp *foxhound.Response) bool

IsBlocked returns true for status codes that commonly indicate blocking or rate-limiting by anti-bot systems.

type DisplayManager 

type DisplayManager struct{}

DisplayManager is a no-op stub when playwright is not enabled.

func NewDisplayManager 

func NewDisplayManager(opts ...DisplayOption) (*DisplayManager, error)

NewDisplayManager returns nil in the stub build — Xvfb is never needed without playwright.

func (*DisplayManager) Close 

func (d *DisplayManager) Close() error

Close is a no-op in the stub build.

func (*DisplayManager) Display 

func (d *DisplayManager) Display() string

Display returns an empty string in the stub build.

type DisplayOption 

type DisplayOption func(*DisplayManager)

DisplayOption is a functional option (no-op in stub build).

func WithDisplayNumber 

func WithDisplayNumber(n int) DisplayOption

WithDisplayNumber is a no-op in the stub build.

func WithScreenResolution 

func WithScreenResolution(res string) DisplayOption

WithScreenResolution is a no-op in the stub build.

type DomainScore 

type DomainScore struct {
	// contains filtered or unexported fields
}

DomainScore tracks static vs browser fetch outcomes for a single domain using a Bayesian Beta distribution model.

Risk score formula (Beta posterior mean): 

risk = (blocked + priorAlpha) / (blocked + success + priorAlpha + priorBeta)

With prior Beta(1, 3): assumes static usually works (75% prior success rate). As blocked events accumulate, risk rises toward 1.0.

Time decay ensures old outcomes fade: 

effective_count = count * exp(-age / halflife)

type DomainScoreAction 

type DomainScoreAction int

DomainScoreAction indicates what the SmartFetcher should do. 

const (
	// ActionStaticNormal means use static with normal timeout.
	ActionStaticNormal DomainScoreAction = iota
	// ActionStaticCautious means use static but with a shorter timeout (fail-fast).
	ActionStaticCautious
	// ActionBrowserDirect means skip static and go directly to browser.
	ActionBrowserDirect
)

type DomainScoreConfig 

type DomainScoreConfig struct {
	// PriorAlpha is the Beta distribution prior for blocked (default 1.0).
	PriorAlpha float64
	// PriorBeta is the Beta distribution prior for success (default 3.0).
	PriorBeta float64
	// EscalationThreshold: if risk > this, skip static entirely (default 0.6).
	EscalationThreshold float64
	// CautionThreshold: if risk > this, use static with shorter timeout (default 0.3).
	CautionThreshold float64
	// DecayHalflife: old outcomes decay with this half-life (default 1h).
	DecayHalflife time.Duration
}

DomainScoreConfig controls the learning behavior.

func DefaultDomainScoreConfig 

func DefaultDomainScoreConfig() DomainScoreConfig

DefaultDomainScoreConfig returns sensible defaults.

func SocialMediaScoreConfig 

func SocialMediaScoreConfig() DomainScoreConfig

SocialMediaScoreConfig returns a configuration tuned for social media targets where static fetches are almost always blocked. Prior Beta(3,1) = 75% prior block rate — escalates after just 1 blocked attempt.

type DomainScorer 

type DomainScorer struct {
	// contains filtered or unexported fields
}

DomainScorer manages per-domain risk scores.

func NewDomainScorer 

func NewDomainScorer(cfg DomainScoreConfig) *DomainScorer

NewDomainScorer creates a scorer with the given configuration.

func (*DomainScorer) Recommend 

func (ds *DomainScorer) Recommend(domain string) DomainScoreAction

Recommend returns the recommended fetch action for a domain.

func (*DomainScorer) RecordBrowser 

func (ds *DomainScorer) RecordBrowser(domain string, blocked bool)

RecordBrowser records the outcome of a browser fetch for the given domain.

func (*DomainScorer) RecordStatic 

func (ds *DomainScorer) RecordStatic(domain string, blocked bool)

RecordStatic records the outcome of a static fetch for the given domain.

func (*DomainScorer) Risk 

func (ds *DomainScorer) Risk(domain string) float64

Risk returns the current risk score for a domain (probability that static will be blocked). Uses Bayesian Beta posterior mean with time decay.

type InterceptConfig 

type InterceptConfig struct {
	// BlockedResourceTypes maps resource types (e.g. "font", "stylesheet")
	// to true for those that should be aborted.
	BlockedResourceTypes map[ResourceType]bool
	// BlockedDomains maps domain names to true. Subdomains are also matched:
	// "example.com" blocks "sub.example.com".
	BlockedDomains map[string]bool
}

InterceptConfig configures what to block during browser navigation. It combines resource-type filtering and domain-level blocking into a single check that is called for every network request via the browser's route handler.

func NewInterceptConfig 

func NewInterceptConfig(resourceTypes map[ResourceType]bool, domains map[string]bool) *InterceptConfig

NewInterceptConfig creates an InterceptConfig with the given resource types and domains to block.

func (*InterceptConfig) IsActive 

func (ic *InterceptConfig) IsActive() bool

IsActive returns true if any blocking rules are configured.

func (*InterceptConfig) ShouldBlock 

func (ic *InterceptConfig) ShouldBlock(resourceType string, requestURL string) bool

ShouldBlock returns true if the given resource type and URL should be blocked based on this configuration.

type OriginStorage 

type OriginStorage struct {
	Origin       string            `json:"origin"`
	LocalStorage map[string]string `json:"local_storage"`
}

OriginStorage represents localStorage/sessionStorage for a single origin.

type PagePool 

type PagePool struct {
	// contains filtered or unexported fields
}

PagePool manages a pool of reusable browser pages/tabs. Instead of creating a fresh browser context + page for every fetch, the pool maintains a set of warm pages that are checked out, used, and returned. This eliminates the per-request overhead of context creation (~200ms) and reduces memory churn.

Pages are reset between uses (cookies cleared, navigated to about:blank) to prevent session bleed. The pool is safe for concurrent use.

Usage: 

pool := NewPagePool(8, createPageFunc, destroyPageFunc)
defer pool.Close()

page, err := pool.Acquire(ctx)
if err != nil { ... }
defer pool.Release(page)

func NewPagePool 

func NewPagePool(maxSize int, create func() (any, error), destroy func(any) error, opts ...PagePoolOption) *PagePool

NewPagePool creates a pool of reusable browser pages with the given capacity. create is called to instantiate a new page; destroy is called when a page is evicted or the pool is closed.

func (*PagePool) Acquire 

func (p *PagePool) Acquire(ctx context.Context) (any, error)

Acquire checks out a page from the pool. If no pages are available and the pool hasn't reached maxSize, a new page is created. If maxSize is reached, Acquire blocks until a page is returned or ctx is cancelled.

func (*PagePool) AcquireWithTimeout 

func (p *PagePool) AcquireWithTimeout(timeout time.Duration) (any, error)

AcquireWithTimeout is a convenience wrapper around Acquire with a timeout.

func (*PagePool) Busy 

func (p *PagePool) Busy() int64

Busy returns the number of pages currently checked out.

func (*PagePool) Close 

func (p *PagePool) Close() error

Close destroys all pooled pages and prevents further acquisitions.

func (*PagePool) Free 

func (p *PagePool) Free() int64

Free returns the number of idle pages available for immediate acquisition.

func (*PagePool) Release 

func (p *PagePool) Release(page any)

Release returns a page to the pool. If a reset function is configured, the page is reset before being made available. If reset fails, the page is destroyed and a new slot opens.

Release is a no-op when called on a nil receiver, so callers may safely hold a snapshotted pool pointer that was nil-ed concurrently (e.g. during browser restart).

func (*PagePool) Stats 

func (p *PagePool) Stats() PagePoolStats

Stats returns pool usage statistics.

func (*PagePool) Total 

func (p *PagePool) Total() int64

Total returns the total number of pages that have been created and not yet destroyed.

func (*PagePool) WarmUp 

func (p *PagePool) WarmUp(n int) int

WarmUp pre-creates n pages in the pool. This reduces latency for the first n Acquire calls. Returns the number of pages successfully created.

type PagePoolOption 

type PagePoolOption func(*PagePool)

PagePoolOption configures a PagePool.

func WithPageReset 

func WithPageReset(fn func(any) error) PagePoolOption

WithPageReset sets a function called when a page is returned to the pool. The reset function should clear cookies, navigate to about:blank, etc.

func WithPoolReuseLimit 

func WithPoolReuseLimit(n int) PagePoolOption

WithPoolReuseLimit sets the maximum number of times a page can be reused before it is destroyed and replaced. Default 0 means unlimited.

type PagePoolStats 

type PagePoolStats struct {
	MaxSize  int
	Created  int64
	Acquired int64
	Released int64
	Recycled int64 // pages destroyed due to reuse limit
	Idle     int64
	Busy     int64 // currently checked out (Acquired - Released)
	Total    int64 // created and not destroyed
}

PagePoolStats holds pool usage metrics.

type ResourceType 

type ResourceType string

ResourceType represents a browser resource type that can be blocked during page navigation. These correspond to the resource types reported by playwright's route.request.resource_type. 

const (
	// ResourceFont matches font file requests (woff, woff2, ttf, otf, eot).
	ResourceFont ResourceType = "font"
	// ResourceImage matches image requests (png, jpg, gif, svg, webp, etc.).
	ResourceImage ResourceType = "image"
	// ResourceMedia matches audio/video requests (mp4, webm, ogg, mp3, etc.).
	ResourceMedia ResourceType = "media"
	// ResourceBeacon matches navigator.sendBeacon() requests.
	ResourceBeacon ResourceType = "beacon"
	// ResourceObject matches plugin requests (Flash, Java applets).
	ResourceObject ResourceType = "object"
	// ResourceImageSet matches <picture> source requests.
	ResourceImageSet ResourceType = "imageset"
	// ResourceTextTrack matches video subtitle/caption track requests.
	ResourceTextTrack ResourceType = "texttrack"
	// ResourceWebSocket matches WebSocket connection requests.
	ResourceWebSocket ResourceType = "websocket"
	// ResourceCSPReport matches Content-Security-Policy violation reports.
	ResourceCSPReport ResourceType = "csp_report"
	// ResourceStylesheet matches CSS stylesheet requests.
	ResourceStylesheet ResourceType = "stylesheet"
)

type SmartFetcher 

type SmartFetcher struct {
	// contains filtered or unexported fields
}

SmartFetcher routes each job to the appropriate fetcher. It implements foxhound.Fetcher and can be used transparently wherever a Fetcher is expected.

func NewSmart 

func NewSmart(static, browser foxhound.Fetcher, opts ...SmartOption) *SmartFetcher

NewSmart creates a SmartFetcher. static must not be nil; browser may be nil (in which case FetchAuto will never escalate beyond the static result).

Additional SmartOption values may be used to override the detector or either fetcher after construction.

func (*SmartFetcher) Close 

func (f *SmartFetcher) Close() error

Close releases resources held by both the static and browser fetchers. Errors from both fetchers are collected; if both fail the errors are joined.

func (*SmartFetcher) Fetch 

func (f *SmartFetcher) Fetch(ctx context.Context, job *foxhound.Job) (*foxhound.Response, error)

Fetch dispatches the job according to its FetchMode:

  • FetchBrowser: calls the browser fetcher directly; returns an error if no browser fetcher is configured.
  • FetchStatic: calls the static fetcher; no escalation regardless of the response status.
  • FetchAuto: calls the static fetcher first. If the response is flagged as blocked AND a browser fetcher is available, escalates to the browser. If no browser fetcher is available, returns the static result with a warning log.

type SmartOption 

type SmartOption func(*SmartFetcher)

SmartOption is a functional option for configuring a SmartFetcher after construction. Options override the fetchers and detector set in NewSmart.

func WithBlockDetector 

func WithBlockDetector(d BlockDetector) SmartOption

WithBlockDetector replaces the default block detector with a custom one.

func WithBrowserFetcher 

func WithBrowserFetcher(fetcher foxhound.Fetcher) SmartOption

WithBrowserFetcher replaces the browser (Camoufox) fetcher.

func WithCautiousTimeout 

func WithCautiousTimeout(d time.Duration) SmartOption

WithCautiousTimeout sets the timeout for static fetches when the domain scorer recommends caution (moderate risk). Default is 5 seconds.

func WithDomainScorer 

func WithDomainScorer(scorer *DomainScorer) SmartOption

WithDomainScorer enables adaptive domain learning. When a scorer is provided, FetchAuto uses Bayesian risk scores to decide whether to attempt static fetching or skip directly to browser.

func WithStaticFetcher 

func WithStaticFetcher(fetcher foxhound.Fetcher) SmartOption

WithStaticFetcher replaces the static (HTTP) fetcher.

type StealthFetcher 

type StealthFetcher struct {
	// contains filtered or unexported fields
}

StealthFetcher is the non-impersonating fallback used when the binary is built without -tags tls. The TLS ClientHello is Go's crypto/tls default (well-known JA3) — header ordering from the identity profile still applies but the TLS layer will not match a real browser.

func NewStealth 

func NewStealth(opts ...StealthOption) *StealthFetcher

NewStealth creates a StealthFetcher. Call with any number of StealthOption functional options to configure identity, timeout, and proxy.

IMPORTANT: this build does NOT perform TLS impersonation. See the package doc and IsImpersonating().

func (*StealthFetcher) Close 

func (f *StealthFetcher) Close() error

Close is a no-op for StealthFetcher; the underlying http.Client manages its own idle connections via the transport.

func (*StealthFetcher) Fetch 

func (f *StealthFetcher) Fetch(ctx context.Context, job *foxhound.Job) (*foxhound.Response, error)

Fetch performs an HTTP request using the stealth client and returns a foxhound.Response. The response FetchMode is always FetchStatic.

Header ordering follows identity.HeaderOrder to match browser fingerprints. If no identity is set a bare request is sent with minimal headers.

func (*StealthFetcher) IsImpersonating added in v0.0.18 

func (f *StealthFetcher) IsImpersonating() bool

IsImpersonating reports whether this fetcher performs real JA3/JA4 TLS fingerprint impersonation. In the default (non-tls) build it always returns false; build with -tags tls to enable real impersonation.

type StealthOption 

type StealthOption func(*StealthFetcher)

StealthOption is a functional option for configuring a StealthFetcher.

func WithHTTP2Fingerprint added in v0.0.18 

func WithHTTP2Fingerprint(_ string) StealthOption

WithHTTP2Fingerprint is a no-op in the default build. Build with -tags tls.

func WithHTTP3Fingerprint added in v0.0.18 

func WithHTTP3Fingerprint(_ string) StealthOption

WithHTTP3Fingerprint is a no-op in the default build. Build with -tags tls.

func WithIdentity 

func WithIdentity(p *identity.Profile) StealthOption

WithIdentity sets the identity profile used to populate request headers. All headers (User-Agent, Accept-Language, Sec-Fetch-* etc.) are derived from this profile, ensuring internal consistency across UA, header order, and locale.

func WithJA3 added in v0.0.18 

func WithJA3(_ string) StealthOption

WithJA3 is a no-op in the default (non-tls) build — the underlying net/http transport cannot customise the TLS ClientHello. Build with -tags tls for the real implementation.

func WithJA3Pool added in v0.0.18 

func WithJA3Pool(pool []string) StealthOption

WithJA3Pool is a no-op in the default build. Build with -tags tls.

func WithProxy 

func WithProxy(proxyURL string) StealthOption

WithProxy sets a proxy URL on the underlying HTTP transport. The proxyURL must be a fully-qualified URL, e.g. "http://user:pass@host:port".

func WithTimeout 

func WithTimeout(d time.Duration) StealthOption

WithTimeout overrides the HTTP client timeout.

type StorageState 

type StorageState struct {
	Cookies    []BrowserCookie `json:"cookies"`
	Origins    []OriginStorage `json:"origins"`
	ExportedAt time.Time       `json:"exported_at"`
}

StorageState represents the full browser session state (cookies + localStorage + sessionStorage).

Source Files

View all Source files

Directories

Path Synopsis
Package presets ships a single curated Firefox JA3 fingerprint that the stealth fetcher applies automatically when an identity profile is configured.
 Package presets ships a single curated Firefox JA3 fingerprint that the stealth fetcher applies automatically when an identity profile is configured.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.