middleware

func Chain ¶

func Chain(middlewares ...foxhound.Middleware) foxhound.Middleware

Chain composes multiple Middleware values into one. Middlewares are applied outermost-first: the first argument's Wrap runs before subsequent ones, so request processing follows the slice order.

func NewAutoThrottle ¶

func NewAutoThrottle(cfg AutoThrottleConfig) foxhound.Middleware

NewAutoThrottle creates an adaptive throttle middleware.

After each response the delay for that domain is recomputed:

• Normal response: delay = EMA(latency) / TargetConcurrency
• 429 or 503: delay spikes to MaxDelay immediately.
• Delay is clamped to [MinDelay, MaxDelay].

The computed delay is slept before the *next* request to that domain.

func NewBlockDetector ¶

func NewBlockDetector(maxRetries int, baseDelay time.Duration, patterns ...BlockPattern) foxhound.Middleware

NewBlockDetector creates a Middleware that detects soft blocks in responses based on HTTP status codes and body content heuristics. When a block is detected it retries with exponential backoff up to maxRetries additional times.

If no patterns are provided DefaultBlockPatterns() is used.

func NewCircuitBreaker ¶

func NewCircuitBreaker(cfg CircuitBreakerConfig) foxhound.Middleware

NewCircuitBreaker creates a circuit breaker middleware.

The circuit breaker monitors the failure rate per domain using a sliding window. When failures exceed the threshold, the circuit opens and all requests fail immediately for an exponentially increasing duration:

open_duration = min(BaseTimeout * 2^(trips-1), MaxTimeout) * U(0.5, 1.5)

After the open duration, one probe request is allowed (half-open state). If it succeeds, the circuit closes. If it fails, the circuit re-opens with an incremented trip count.

func NewConcurrency ¶

func NewConcurrency(perDomain int) foxhound.Middleware

NewConcurrency returns a Middleware that limits the number of concurrent in-flight requests per target domain. A separate semaphore (buffered channel of size perDomain) is created lazily for each unique domain.

The middleware is intended to sit as the outermost layer in the chain so it caps parallelism before any rate-limit or dedup checks are performed.

func NewCookies ¶

func NewCookies() foxhound.Middleware

NewCookies returns a Middleware that automatically persists and replays HTTP cookies across requests within the same session.

func NewDedup ¶

func NewDedup(opts ...DedupOption) foxhound.Middleware

NewDedup creates a Middleware that skips URLs that have already been fetched.

Canonicalisation rules applied before storing/checking:

• Only scheme + host + path + query are compared (fragment dropped).
• Query parameters are sorted alphabetically.

Duplicate requests return a zero-value Response (StatusCode 0, empty body) without calling the underlying Fetcher.

func NewDeltaFetch ¶

func NewDeltaFetch(strategy DeltaStrategy, store DeltaStore, ttl time.Duration) foxhound.Middleware

NewDeltaFetch returns a Middleware that avoids re-fetching known URLs.

• strategy=DeltaSkipSeen: skip if ever seen.
• strategy=DeltaSkipRecent: skip only if seen within ttl.

Skipped requests return a zero-value Response (StatusCode 0) without calling the underlying Fetcher, matching the behaviour of NewDedup.

func NewDepthLimit ¶

func NewDepthLimit(maxDepth int) foxhound.Middleware

NewDepthLimit creates a Middleware that returns an error for any job with Depth > maxDepth, preventing the underlying Fetcher from being called.

func NewDomainDelay ¶

func NewDomainDelay(cfg DomainDelayConfig) foxhound.Middleware

NewDomainDelay creates a Middleware that enforces per-domain download delays. The delay is enforced as a minimum interval between consecutive requests to the same domain.

Example:

middleware.NewDomainDelay(middleware.DomainDelayConfig{
    DefaultDelay: 2 * time.Second,
    PerDomain: map[string]time.Duration{
        "api.example.com": 500 * time.Millisecond,
    },
    Randomize: true,
})

func NewMetrics ¶

func NewMetrics(namespace string) foxhound.Middleware

NewMetrics returns a Middleware that records Prometheus metrics.

Three instruments are registered under namespace:

• <namespace>_requests_total{domain, status} — request counter
• <namespace>_request_duration_seconds{domain} — latency histogram
• <namespace>_errors_total{domain, error_type} — error counter

All instruments are registered with the default Prometheus registry via promauto, so they are automatically included in the default /metrics handler.

func NewRateLimit ¶

func NewRateLimit(requestsPerSec float64, burstSize int) foxhound.Middleware

NewRateLimit creates a Middleware that allows requestsPerSec requests per second per domain, with a burst of burstSize tokens. A separate rate.Limiter is created lazily for each unique domain.

func NewRedirect ¶

func NewRedirect(maxRedirects int) foxhound.Middleware

NewRedirect returns a Middleware that follows HTTP redirects.

Up to maxRedirects hops are followed. If the chain exceeds maxRedirects an error is returned. A value of 0 disables redirect following entirely.

func NewReferer ¶

func NewReferer() foxhound.Middleware

NewReferer returns a Middleware that automatically sets the Referer header.

Behaviour:

• If a Referer header is already present on the job, it is left unchanged.
• On the first request to a domain, Referer is set to a Google search URL for that domain, mimicking organic search traffic.
• On subsequent requests to the same domain, Referer is set to the URL of the previous request to that domain.

func NewRetry ¶

func NewRetry(maxRetries int, baseDelay time.Duration) foxhound.Middleware

NewRetry creates a Middleware that retries failed or blocked requests up to maxRetries additional times (i.e. 1 + maxRetries total attempts).

Backoff between attempts is baseDelay * 2^attempt, with ±25 % uniform jitter to spread retry storms. Context cancellation stops retries immediately.

func NewRobotsTxt ¶

func NewRobotsTxt(userAgent string) foxhound.Middleware

NewRobotsTxt returns a Middleware that optionally respects robots.txt.

On the first request to a domain the middleware fetches /robots.txt using a plain http.Client and caches the result. Subsequent requests to the same domain use the cached rules.

If robots.txt cannot be fetched (network error, non-2xx status) all URLs are allowed — conservative fail-open behaviour is safer for scraping than incorrectly blocking pages.

Disallowed URLs return a Response with StatusCode 0 without calling the underlying Fetcher, matching the pattern used by NewDedup and NewDeltaFetch.