fetch

package

v0.1.0 Latest Latest Go to latest Published: May 28, 2026 License: Apache-2.0 Imports: 8 Imported by: 0

Details

This section is empty.

This section is empty.

func IsHTMLContentType(ct string) bool

IsHTMLContentType 检查是否为 HTML 内容类型

func ReadBodyWithLimit(resp *http.Response, maxBytes int64) ([]byte, error)

ReadBodyWithLimit 读取响应体，限制大小

type Client struct {
	// contains filtered or unexported fields
}

Client HTTP 客户端封装

func NewClient(cfg ClientConfig) (*Client, error)

NewClient 创建 HTTP 客户端

func (c *Client) Get(ctx context.Context, rawURL string) (*http.Response, error)

Get 发起 GET 请求

func (c *Client) GetBytes(ctx context.Context, rawURL string) ([]byte, error)

GetBytes 下载内容为字节流（用于 PDF 等）

type ClientConfig struct {
	UserAgent    string
	ExtraHeaders map[string]string
	Timeout      time.Duration
	MaxRedirects int
	ProxyURL     string
}

ClientConfig HTTP 客户端配置

type Detector struct{}

Detector 反爬/WAF 检测器

func NewDetector() *Detector

NewDetector 创建检测器

func (d *Detector) Detect(resp *http.Response, body io.Reader) (bool, string)

Detect 检测响应是否被 WAF/反爬拦截返回 (是否拦截, 原因) 仅扫描响应头 + body 前 8KB，< 5ms 开销

type Fetcher interface {
	Get(ctx context.Context, url string) (*http.Response, error)
}

Fetcher 抓取器接口