fetch

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 19, 2026 License: Apache-2.0 Imports: 12 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func DefaultBackoff added in v0.2.0

func DefaultBackoff(retry int) time.Duration

DefaultBackoff 默认退避策略

func IsHTMLContentType

func IsHTMLContentType(ct string) bool

IsHTMLContentType 检查是否为 HTML 内容类型

func ReadBodyWithLimit

func ReadBodyWithLimit(body io.Reader, maxBytes int64) ([]byte, error)

ReadBodyWithLimit 读取响应体,限制大小

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client HTTP 客户端封装

func NewClient

func NewClient(cfg ClientConfig) (*Client, error)

NewClient 创建 HTTP 客户端(带 TLS 指纹伪装)

func (*Client) Get

func (c *Client) Get(ctx context.Context, rawURL string) (*fhttp.Response, error)

Get 发起 GET 请求(带重试)

func (*Client) GetBytes

func (c *Client) GetBytes(ctx context.Context, rawURL string) ([]byte, error)

GetBytes 下载内容为字节流(用于 PDF 等)

type ClientConfig

type ClientConfig struct {
	UserAgent       string
	ExtraHeaders    map[string]string
	Timeout         time.Duration
	MaxRedirects    int
	ProxyURL        string        // 显式代理地址,优先级最高
	UseSystemProxy  bool          // 是否自动使用系统代理(环境变量 HTTP_PROXY/HTTPS_PROXY)
	MaxRetries      int           // 最大重试次数,默认 3
	RetryBackoffMin time.Duration // 重试最小退避,默认 1s
	RetryBackoffMax time.Duration // 重试最大退避,默认 5s
}

ClientConfig HTTP 客户端配置

type Detector

type Detector struct{}

Detector 反爬/WAF 检测器

func NewDetector

func NewDetector() *Detector

NewDetector 创建检测器

func (*Detector) Detect

func (d *Detector) Detect(resp *fhttp.Response, body io.Reader) (bool, string)

Detect 检测响应是否被 WAF/反爬拦截 返回 (是否拦截, 原因) 仅扫描响应头 + body 前 8KB,< 5ms 开销

type Fetcher

type Fetcher interface {
	Get(ctx context.Context, url string) (*fhttp.Response, error)
}

Fetcher 抓取器接口

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL