fetch

package
v1.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 15, 2026 License: MIT Imports: 28 Imported by: 0

Documentation

Index

Constants

View Source
const (
	DefaultMaxBodySize int64 = 5 * 1024 * 1024
	DefaultMaxIconSize int64 = 512 * 1024
)
View Source
const JSExecuteTemplate = `` /* 813-byte string literal not displayed */

Variables

View Source
var (
	ErrFetch      = errors.New("fetch failed")
	ErrInvalidURL = errors.New("invalid URL")
)
View Source
var DefaultStackDepthLimit = 300

默认JavaScript执行的堆栈深度限制

Functions

func ExtractCharset deprecated

func ExtractCharset(htmlContent []byte) string

ExtractCharset extracts charset from an HTML document.

Deprecated: this helper is exported for compatibility and may become internal in a future release.

func ExtractContentTypeCharset deprecated

func ExtractContentTypeCharset(contentType string) (charset string)

ExtractContentTypeCharset extracts charset from a Content-Type header.

Deprecated: this helper is exported for compatibility and may become internal in a future release.

func InsertInto deprecated

func InsertInto(s string, interval int, sep rune) string

InsertInto inserts sep every interval runes.

Deprecated: this helper is exported for compatibility and may become internal in a future release.

func ResponseDecoding deprecated

func ResponseDecoding(body []byte, label string) string

ResponseDecoding decodes response bytes with a charset label.

Deprecated: this helper is exported for compatibility and may become internal in a future release.

Types

type Banner struct {
	Uri         string               `json:"uri"`
	BodyHash    int32                `json:"body_hash"`
	Body        string               `json:"body"`
	Header      string               `json:"header"`
	Headers     map[string]string    `json:"-"`
	Title       string               `json:"title"`
	StatusCode  int                  `json:"status_code"`
	Response    string               `json:"_"`
	SSL         bool                 `json:"ssl"`
	Certificate string               `json:"certificate"`
	IconHash    int32                `json:"icon_hash"`
	IconType    string               `json:"icon_type"`
	Charset     string               `json:"-"`
	Cert        *tls.ConnectionState `json:"-"`
	IconURI     string               `json:"icon_uri"`
	IconBytes   []byte               `json:"-"`
	Compliance  map[string]string    `json:"-"`
}

Banner 表示爬取的网站信息

func (*Banner) CacheLower

func (b *Banner) CacheLower()

缓存小写内容、避免匹配时进行大小写转换出现的性能损耗、虽然会增加内存开销、但是可以显著提高匹配速度

type Fetcher

type Fetcher struct {
	// contains filtered or unexported fields
}

Fetcher 定义HTTP探测和Banner采集的核心结构。

func NewFetcher

func NewFetcher(options *Options) (*Fetcher, error)

func (*Fetcher) GetBanner

func (c *Fetcher) GetBanner(ctx context.Context, uri string) (*Banner, error)

func (*Fetcher) GetBanners

func (c *Fetcher) GetBanners(ctx context.Context, uri string) ([]*Banner, error)

GetBanners 实现BannerProvider接口

type Options

type Options struct {
	DisableIcon       bool
	Proxy             string
	DebugReq          bool
	DebugResp         bool
	DisableJavaScript bool
	MaxBodySize       int64
	MaxIconSize       int64
	Timeout           time.Duration
	Retries           int // 失败后的额外重试次数,默认 0
	// HTTPClient allows SDK users to provide a fully configured client.
	// When set, appfinger uses it as-is and does not mutate transport settings.
	HTTPClient *http.Client
	// Transport allows SDK users to provide custom transport-level behavior.
	// When set, appfinger wraps it in an http.Client and does not mutate it.
	Transport *http.Transport
}

func DefaultOption

func DefaultOption() *Options

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL