httpclient

HTTP Client Package

This package provides a unified HTTP client factory for all scrapers, eliminating code duplication and providing consistent proxy, retry, and header handling.

Overview

The package implements three patterns:

• Builder Pattern: For custom HTTP client configuration
• Factory Functions: Quick client creation for common use cases
• FlareSolverr Integration: For sites requiring browser challenge bypass

Usage Examples

Simple Scrapers (No ProxyProfile needed)

For scrapers that don't need browser capabilities:

import (
    "github.com/go-resty/resty/v2"
    "github.com/javinizer/javinizer-go/internal/httpclient"
)

func NewHTTPClient(cfg *config.ScraperSettings, globalProxy *config.ProxyConfig, globalFlareSolverr config.FlareSolverrConfig) (*resty.Client, error) {
    return httpclient.FromScraperSettings(cfg, globalProxy, globalFlareSolverr,
        httpclient.WithHeaders(httpclient.CombineHeaders(
            httpclient.StandardHTMLHeaders(),
            httpclient.UserAgentHeader(cfg.UserAgent),
        )),
    ).BuildClient()
}

Returns: (*resty.Client, error)

FlareSolverr Scrapers (Browser bypass)

For scrapers that need FlareSolverr to bypass Cloudflare/browser challenges:

import (
    "github.com/go-resty/resty/v2"
    "github.com/javinizer/javinizer-go/internal/httpclient"
)

func NewHTTPClient(cfg *config.ScraperSettings, globalProxy *config.ProxyConfig, globalFlareSolverr config.FlareSolverrConfig) (*resty.Client, *httpclient.FlareSolverr, error) {
    builder := httpclient.FromScraperSettings(cfg, globalProxy, globalFlareSolverr,
        httpclient.WithHeaders(httpclient.StandardHTMLHeaders()),
        httpclient.WithHeaders(httpclient.UserAgentHeader(cfg.UserAgent)),
    )
    return builder.BuildWithFlareSolverr()
}

Returns: (*resty.Client, *httpclient.FlareSolverr, error)

Browser Scrapers (Need ProxyProfile)

For scrapers that need to pass proxy configuration to browser automation:

import (
    "github.com/go-resty/resty/v2"
    "github.com/javinizer/javinizer-go/internal/httpclient"
    "github.com/javinizer/javinizer-go/internal/config"
)

func NewHTTPClient(cfg *config.ScraperSettings, globalProxy *config.ProxyConfig, globalFlareSolverr config.FlareSolverrConfig) (*resty.Client, *config.ProxyProfile, error) {
    return httpclient.FromScraperSettings(cfg, globalProxy, globalFlareSolverr,
        httpclient.WithHeaders(httpclient.CombineHeaders(
            httpclient.DMMHeaders(),
            httpclient.UserAgentHeader(cfg.UserAgent),
        )),
    ).BuildWithProxy()
}

Returns: (*resty.Client, *config.ProxyProfile, error)

Header Presets

The package provides pre-built header presets:

Combining Headers

headers := httpclient.CombineHeaders(
    httpclient.StandardHTMLHeaders(),
    httpclient.UserAgentHeader("custom-agent"),
    httpclient.RefererHeader("https://example.com"),
)

Merging Cookies

existingCookies := map[string]string{"session": "abc123"}
newCookies := map[string]string{"token": "xyz789"}
cookieHeader := httpclient.MergeCookieHeader(existingCookies, newCookies)
// Result: "session=abc123; token=xyz789"

Functional Options

The builder supports these options:

Builder Methods

FlareSolverr Usage

// Create FlareSolverr-enabled client
client, flare, err := httpclient.FromScraperSettings(cfg, globalProxy, globalFlareSolverr,
    httpclient.WithHeaders(httpclient.StandardHTMLHeaders()),
    httpclient.WithFlareSolverr(true),
).BuildWithFlareSolverr()

// Make request through FlareSolverr
resp, err := flare.RequestGet(ctx, "https://example.com", nil)

// The HTML content is in resp.Solution.Response
html := resp.Solution.Response

Proxy Support

The package automatically handles:

• HTTP/HTTPS proxies
• SOCKS5 proxies
• Proxy authentication
• Per-scraper proxy overrides

Proxy priority: ScraperSettings.Proxy > Global Proxy > None

Migration Guide

Before (Duplicated Code)

func NewHTTPClient(cfg *config.Config) (*resty.Client, error) {
    client := resty.New()
    client.SetTimeout(30 * time.Second)
    client.SetRetryCount(3)
    
    // Handle proxy
    if cfg.Proxy.Enabled {
        // 40+ lines of proxy setup...
    }
    
    // Set headers
    client.SetHeaders(map[string]string{
        "Accept": "text/html...",
        // More headers...
    })
    
    return client, nil
}

After (Using Factory)

func NewHTTPClient(cfg *config.ScraperSettings, globalProxy *config.ProxyConfig, globalFlareSolverr config.FlareSolverrConfig) (*resty.Client, error) {
    return httpclient.FromScraperSettings(cfg, globalProxy, globalFlareSolverr,
        httpclient.WithHeaders(httpclient.StandardHTMLHeaders()),
    ).BuildClient()
}

Architecture

┌─────────────────────────────────────────────────────────────┐
│                      Scraper Packages                        │
│  ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐           │
│  │   DMM   │ │ JavBus  │ │ JavDB   │ │  FC2    │ ...       │
│  └────┬────┘ └────┬────┘ └────┬────┘ └────┬────┘           │
│       │           │           │           │                 │
│       └───────────┴───────────┴───────────┘                 │
│                       │                                     │
│                       ▼                                     │
│  ┌─────────────────────────────────────────────────────┐   │
│  │           internal/httpclient                         │   │
│  │  ┌──────────────┐  ┌──────────────┐  ┌───────────┐  │   │
│  │  │   Builder    │  │   Factory    │  │ FlareSol. │  │   │
│  │  │   builder.go │  │  factory.go  │  │(embedded) │  │   │
│  │  └──────────────┘  └──────────────┘  └───────────┘  │   │
│  │  ┌──────────────┐  ┌──────────────┐                  │   │
│  │  │   Presets    │  │   Client     │                  │   │
│  │  │  presets.go  │  │  client.go   │                  │   │
│  │  └──────────────┘  └──────────────┘                  │   │
│  └─────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘

Testing

# Run package tests
go test ./internal/httpclient/...

# Run with race detector
go test -race ./internal/httpclient/...