web_crawler

package
v0.0.0-...-8b05944 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 8, 2026 License: MIT Imports: 31 Imported by: 1

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var ChromeRemoteURL string

ChromeRemoteURL Chrome远程调试URL ChromeRemoteURL is the Chrome remote debugging URL

View Source 
var ClosePageLimit = 0
View Source 
var DefaultClient = &Client{
	Policies: GlobalPolicies,
	Config:   DefaultAntiBotConfig(),
}

DefaultClient 默认客户端实例 DefaultClient default client instance

View Source 
var (

	// GlobalPolicies 全局策略列表
	// GlobalPolicies global policy list
	GlobalPolicies []ChallengePolicy
)
View Source 
var WebChrome *core.Contain

WebChrome Chrome浏览器实例 WebChrome is a Chrome browser instance

Functions

func DiyConvertHtmlToArticle 

func DiyConvertHtmlToArticle(code string) string

DiyConvertHtmlToArticle 处理HTML字符串以提取和清理文章内容 DiyConvertHtmlToArticle processes an HTML string to extract and clean the article content

func GenerateRodPageSetupSnippet 

func GenerateRodPageSetupSnippet(rawHeaders, rawCookies, targetURL string) (string, error)

GenerateRodPageSetupSnippet converts raw headers/cookies into rod page setup code.

func GenerateRodPageSetupSnippetFromCURL 

func GenerateRodPageSetupSnippetFromCURL(curl string) (string, error)

GenerateRodPageSetupSnippetFromCURL parses a curl command and returns a rod snippet.

func GenerateRodPageSetupSnippetFromCURLWithOptions 

func GenerateRodPageSetupSnippetFromCURLWithOptions(curl string, options RodPasteOptions) (string, error)

GenerateRodPageSetupSnippetFromCURLWithOptions parses a curl command and returns a rod snippet.

func GenerateRodPageSetupSnippetFromHAR 

func GenerateRodPageSetupSnippetFromHAR(har string, entryIndex int) (string, error)

GenerateRodPageSetupSnippetFromHAR parses a HAR JSON string and returns a rod snippet.

func GenerateRodPageSetupSnippetFromHARWithOptions 

func GenerateRodPageSetupSnippetFromHARWithOptions(har string, entryIndex int, options RodPasteOptions) (string, error)

GenerateRodPageSetupSnippetFromHARWithOptions parses a HAR JSON string and returns a rod snippet.

func GenerateRodPageSetupSnippetWithOptions 

func GenerateRodPageSetupSnippetWithOptions(rawHeaders, rawCookies string, options RodPasteOptions) (string, error)

GenerateRodPageSetupSnippetWithOptions converts raw headers/cookies into rod page setup code.

func MustWaitPageReady 

func MustWaitPageReady(page *rod.Page, options PageWaitOptions)

MustWaitPageReady panics on wait error.

func NewAntiBotClient 

func NewAntiBotClient(chrome *core.Contain, timeout time.Duration, policies ...ChallengePolicy) *http.Client

NewAntiBotClient 创建一个支持反爬虫Bypass的HTTP Client NewAntiBotClient creates an HTTP Client supporting anti-bot bypass

func NewAntiBotClientWithConfig 

func NewAntiBotClientWithConfig(chrome *core.Contain, timeout time.Duration, config *AntiBotConfig, policies ...ChallengePolicy) *http.Client

NewAntiBotClientWithConfig 创建一个支持反爬虫Bypass的HTTP Client（带配置） NewAntiBotClientWithConfig creates an HTTP Client supporting anti-bot bypass (with config)

func NewCloudscraperClient 

func NewCloudscraperClient(chrome *core.Contain, timeout time.Duration) *http.Client

NewCloudscraperClient 创建一个支持反爬虫Bypass的HTTP Client NewCloudscraperClient creates an HTTP Client supporting anti-bot bypass Deprecated: use NewAntiBotClient instead

func ParsePath 

func ParsePath(url, path string) string

func ReadFeed 

func ReadFeed(xmlURL string) rss.RSS

ReadFeed 读取Atom订阅源并转换为RSS格式 ReadFeed reads Atom feed source and converts to RSS format

func ReadRSS 

func ReadRSS(xmlURL string) rss.RSS

ReadRSS 读取RSS订阅源 ReadRSS reads RSS feed source

func RenderWithWait 

func RenderWithWait(options PageWaitOptions) func(*rod.Page)

RenderWithWait returns a renderFunc using WaitPageReady.

func SolveChallenge 

func SolveChallenge(chrome *core.Contain, url string) ([]*http.Cookie, string, error)

SolveChallenge 使用WebChrome解决验证（通用逻辑，保留给CF使用）

func StripHTMLTags 

func StripHTMLTags(html string) string

StripHTMLTags 使用正则表达式移除HTML标签 StripHTMLTags removes HTML tags using regular expressions

func WaitPageReady 

func WaitPageReady(page *rod.Page, options PageWaitOptions) error

WaitPageReady waits for ready state and optional selector conditions.

Types

type AnchorArticle 

type AnchorArticle struct {
	Title       string `json:"title,omitempty"`
	Link        string `json:"link,omitempty"`
	Content     string `json:"content,omitempty"`
	TextContent string `json:"text_content,omitempty"`
}

AnchorArticle holds article content fetched from an anchor link.

type AnchorArticleValue 

type AnchorArticleValue struct {
	AnchorArticles
	WebReaderError
}

AnchorArticleValue wraps the anchor articles with error information.

func ReadAllAnchorArticles 

func ReadAllAnchorArticles(url string, renderFunc func(*rod.Page), requestModifiers ...RequestWith) AnchorArticleValue

ReadAllAnchorArticles reads all anchor links and fetches title/content.

func ReadAllAnchorArticlesWithChrome 

func ReadAllAnchorArticlesWithChrome(url string, listRenderFunc func(*rod.Page), articleRenderFunc func(*rod.Page), restartCondition func(Article) bool, i ...int) AnchorArticleValue

ReadAllAnchorArticlesWithChrome reads all anchor links and fetches title/content with Chrome.

func ReadAnchorArticlesWithCURL 

func ReadAnchorArticlesWithCURL(curl string, renderFunc func(*rod.Page), requestModifiers ...RequestWith) AnchorArticleValue

ReadAnchorArticlesWithCURL 使用 curl 命令读取锚点链接并获取标题/内容。 ReadAnchorArticlesWithCURL reads anchor links and fetches title/content using a curl command.

func ReadAnchorArticlesWithCURLOptions 

func ReadAnchorArticlesWithCURLOptions(curl string, renderFunc func(*rod.Page), options LinkExtractOptions, requestModifiers ...RequestWith) AnchorArticleValue

ReadAnchorArticlesWithCURLOptions 使用 curl 命令读取锚点链接并获取标题/内容（带提取选项）。 ReadAnchorArticlesWithCURLOptions reads anchor links and fetches title/content using a curl command with options.

func ReadAnchorArticlesWithChromeOptions 

func ReadAnchorArticlesWithChromeOptions(url string, listRenderFunc func(*rod.Page), articleRenderFunc func(*rod.Page), restartCondition func(Article) bool, options LinkExtractOptions, i ...int) AnchorArticleValue

ReadAnchorArticlesWithChromeOptions reads anchor links and fetches title/content with Chrome and options.

func ReadAnchorArticlesWithOptions 

func ReadAnchorArticlesWithOptions(url string, renderFunc func(*rod.Page), options LinkExtractOptions, requestModifiers ...RequestWith) AnchorArticleValue

ReadAnchorArticlesWithOptions reads anchor links and fetches title/content with options.

type AnchorArticles 

type AnchorArticles []AnchorArticle

AnchorArticles is a collection of anchor articles.

func (AnchorArticles) Stream 

func (a AnchorArticles) Stream() *stream.Stream[AnchorArticle, AnchorArticles]

Stream converts the collection to a stream.

type AntiBotConfig 

type AntiBotConfig struct {
	// EnableCloudflare 是否启用 Cloudflare 绕过
	// EnableCloudflare whether to enable Cloudflare bypass
	EnableCloudflare bool
}

AntiBotConfig 反爬虫配置 AntiBotConfig anti-bot configuration

func DefaultAntiBotConfig 

func DefaultAntiBotConfig() *AntiBotConfig

DefaultAntiBotConfig 返回默认配置 DefaultAntiBotConfig returns default configuration

type AntiBotTransport 

type AntiBotTransport struct {
	Transport http.RoundTripper
	Chrome    *core.Contain
	Policies  []ChallengePolicy
	Config    *AntiBotConfig
}

AntiBotTransport 自定义Transport，处理各种反爬虫验证 AntiBotTransport custom Transport to handle various anti-bot verifications

func (*AntiBotTransport) RoundTrip 

func (t *AntiBotTransport) RoundTrip(req *http.Request) (*http.Response, error)

RoundTrip 实现RoundTripper接口 RoundTrip implements RoundTripper interface

type Article 

type Article struct {
	readability.Article
	WebReaderError
}

Article 文章结构体 Article represents an article structure

func GetArticleWithChrome 

func GetArticleWithChrome(url string, rendorFunc func(*rod.Page), restartCondition func(Article) bool, i ...int) Article

GetArticleWithChrome 使用Chrome浏览器获取文章内容（支持JavaScript渲染） GetArticleWithChrome gets article content using Chrome browser (supports JavaScript rendering)

func GetArticleWithSourceCode 

func GetArticleWithSourceCode(code SourceCode, url string) Article

GetArticleWithSourceCode 从源代码获取文章内容 GetArticleWithSourceCode gets article content from source code

func GetArticleWithURL 

func GetArticleWithURL(url string, requestModifiers ...RequestWith) Article

GetArticleWithURL 通过URL获取文章内容 GetArticleWithURL gets article content by URL 

type ArticleLink struct {
	Title string `json:"title,omitempty"`
	Link  string `json:"link,omitempty"`
}

ArticleLink holds a single article title and link.

type ArticleLinkValue 

type ArticleLinkValue struct {
	ArticleLinks
	WebReaderError
}

ArticleLinkValue wraps the article links with error information. 

func AllLink(url string, renderFunc func(*rod.Page), requestModifiers ...RequestWith) ArticleLinkValue

AllLink reads all anchor links on a page (titles + links only).

func AllLinkWithCURL 

func AllLinkWithCURL(curl string, renderFunc func(*rod.Page), requestModifiers ...RequestWith) ArticleLinkValue

AllLinkWithCURL 使用 curl 命令读取所有锚点链接（仅标题与链接）。 AllLinkWithCURL reads all anchor links (titles + links only) using a curl command. 

func ReadArticleLinks(url, js string, renderFunc func(*rod.Page), requestModifiers ...RequestWith) ArticleLinkValue

ReadArticleLinks reads article links and titles from a list page.

func ReadArticleLinksAuto 

func ReadArticleLinksAuto(url string, renderFunc func(*rod.Page), requestModifiers ...RequestWith) ArticleLinkValue

ReadArticleLinksAuto reads article links and titles without custom JS selectors.

func ReadArticleLinksAutoWithCURL 

func ReadArticleLinksAutoWithCURL(curl string, renderFunc func(*rod.Page), requestModifiers ...RequestWith) ArticleLinkValue

ReadArticleLinksAutoWithCURL 使用 curl 命令读取文章链接和标题（自动解析，无 JS 选择器）。 ReadArticleLinksAutoWithCURL reads article links and titles using a curl command without custom JS selectors.

func ReadArticleLinksAutoWithCURLOptions 

func ReadArticleLinksAutoWithCURLOptions(curl string, renderFunc func(*rod.Page), options LinkExtractOptions, requestModifiers ...RequestWith) ArticleLinkValue

ReadArticleLinksAutoWithCURLOptions 使用 curl 命令读取文章链接和标题（带提取选项）。 ReadArticleLinksAutoWithCURLOptions reads article links and titles using a curl command with extraction options.

func ReadArticleLinksAutoWithChrome 

func ReadArticleLinksAutoWithChrome(url string, listRenderFunc func(*rod.Page), articleRenderFunc func(*rod.Page), restartCondition func(Article) bool, i ...int) ArticleLinkValue

ReadArticleLinksAutoWithChrome reads article links and titles without custom JS selectors.

func ReadArticleLinksAutoWithChromeOptions 

func ReadArticleLinksAutoWithChromeOptions(url string, listRenderFunc func(*rod.Page), articleRenderFunc func(*rod.Page), restartCondition func(Article) bool, options LinkExtractOptions, i ...int) ArticleLinkValue

ReadArticleLinksAutoWithChromeOptions reads article links and titles with extraction options.

func ReadArticleLinksAutoWithOptions 

func ReadArticleLinksAutoWithOptions(url string, renderFunc func(*rod.Page), options LinkExtractOptions, requestModifiers ...RequestWith) ArticleLinkValue

ReadArticleLinksAutoWithOptions reads article links and titles with extraction options.

func ReadArticleLinksWithCURL 

func ReadArticleLinksWithCURL(curl, js string, renderFunc func(*rod.Page), requestModifiers ...RequestWith) ArticleLinkValue

ReadArticleLinksWithCURL 使用 curl 命令读取文章链接和标题。 ReadArticleLinksWithCURL reads article links and titles using a curl command.

func ReadArticleLinksWithChrome 

func ReadArticleLinksWithChrome(url, js string, listRenderFunc func(*rod.Page), articleRenderFunc func(*rod.Page), restartCondition func(Article) bool, i ...int) ArticleLinkValue

ReadArticleLinksWithChrome reads article links and titles using Chrome for article pages. 

type ArticleLinks []ArticleLink

ArticleLinks is a collection of article links.

func (ArticleLinks) Stream 

func (a ArticleLinks) Stream() *stream.Stream[ArticleLink, ArticleLinks]

Stream converts the collection to a stream.

type AttrValue 

type AttrValue struct {
	WebReaderValue
	WebReaderError
}

AttrValue 属性值包装器 AttrValue is an attribute value wrapper

func ReadArray 

func ReadArray(url, js string, renderFunc func(*rod.Page)) AttrValue

ReadArray 读取数组数据 ReadArray reads array data

type CFSession 

type CFSession struct {
	Cookies   []*http.Cookie
	UserAgent string
	Expires   time.Time
}

CFSession Cloudflare会话缓存 CFSession Cloudflare session cache

type CaptchaSolver 

type CaptchaSolver struct {
	Page *rod.Page
}

CaptchaSolver 验证码解决器 CaptchaSolver captcha solver

func NewCaptchaSolver 

func NewCaptchaSolver(page *rod.Page) *CaptchaSolver

NewCaptchaSolver 创建验证码解决器 NewCaptchaSolver creates a captcha solver

func (*CaptchaSolver) FindSliderGap 

func (cs *CaptchaSolver) FindSliderGap(bgElem *rod.Element, sliderElem *rod.Element) (int, error)

FindSliderGap 简单的滑块缺口识别（基于像素差异） FindSliderGap simple slider gap detection (based on pixel difference) bgElem: 背景图片元素 / background image element sliderElem: 滑块元素（用于确定起始搜索位置，避免识别到滑块本身） / slider element (to determine start search position, avoid detecting slider itself)

func (*CaptchaSolver) HumanDrag 

func (cs *CaptchaSolver) HumanDrag(box *rod.Element, x, y float64) error

HumanDrag 模拟人类拖拽行为 HumanDrag simulates human drag behavior

func (*CaptchaSolver) SolveSlideToVerify 

func (cs *CaptchaSolver) SolveSlideToVerify(sliderSelector, containerSelector string, bgSelector ...string) error

SolveSlideToVerify 解决“滑动验证”类型（拖到最右侧或指定缺口） SolveSlideToVerify solves "slide to verify" type (drag to rightmost or specific gap) sliderSelector: 滑块按钮选择器 / slider button selector bgSelector: 背景图选择器（可选，如果是缺口验证则需要） / background image selector (optional, needed if gap verification) containerSelector: 滑块容器选择器（用于计算拖拽距离） / slider container selector (used to calculate drag distance)

type ChallengePolicy 

type ChallengePolicy interface {
	// ShouldHandle 检查是否应该处理该响应
	// ShouldHandle checks if the response should be handled
	ShouldHandle(resp *http.Response) bool

	// Solve 解决挑战，返回Cookies, UserAgent和错误
	// Solve solves the challenge, returning Cookies, UserAgent, and error
	Solve(chrome *core.Contain, url string) ([]*http.Cookie, string, error)
}

ChallengePolicy 挑战策略接口 ChallengePolicy challenge policy interface

type Client 

type Client struct {
	Chrome   *core.Contain
	Policies []ChallengePolicy
	Config   *AntiBotConfig

	ClosePageLimit int
	// contains filtered or unexported fields
}

Client 网络爬虫客户端 Client Web crawler client

func NewClient 

func NewClient(chrome *core.Contain, closePage int, policies ...ChallengePolicy) *Client

NewClient 创建新的客户端 NewClient creates a new client

func NewClientWithConfig 

func NewClientWithConfig(chrome *core.Contain, config *AntiBotConfig, policies ...ChallengePolicy) *Client

NewClientWithConfig 创建新的客户端（带配置） NewClientWithConfig creates a new client (with config)

func (*Client) AddPolicy 

func (c *Client) AddPolicy(policy ChallengePolicy)

AddPolicy 添加验证策略 AddPolicy adds challenge policy 

func (c *Client) AllLink(url string, renderFunc func(*rod.Page), requestModifiers ...RequestWith) ArticleLinkValue

AllLink reads all anchor links on a page (titles + links only).

func (*Client) AllLinkWithCURL 

func (c *Client) AllLinkWithCURL(curl string, renderFunc func(*rod.Page), requestModifiers ...RequestWith) ArticleLinkValue

AllLinkWithCURL 使用 curl 命令读取所有锚点链接（仅标题与链接）。 AllLinkWithCURL reads all anchor links (titles + links only) using a curl command.

func (*Client) GetArticleWithChrome 

func (c *Client) GetArticleWithChrome(url string, rendorFunc func(*rod.Page), restartCondition func(Article) bool, i ...int) Article

GetArticleWithChrome 使用Chrome浏览器获取文章内容（支持JavaScript渲染） GetArticleWithChrome gets article content using Chrome browser (supports JavaScript rendering)

func (*Client) GetArticleWithURL 

func (c *Client) GetArticleWithURL(url string, requestModifiers ...RequestWith) Article

GetArticleWithURL 通过URL获取文章内容 GetArticleWithURL gets article content by URL

func (*Client) GetChrome 

func (c *Client) GetChrome() *core.Contain

GetChrome 获取Chrome实例（公开方法） GetChrome gets the Chrome instance (public method)

func (*Client) GetConfig 

func (c *Client) GetConfig() *AntiBotConfig

GetConfig 获取配置（公开方法） GetConfig gets the config (public method)

func (*Client) GetPolicies 

func (c *Client) GetPolicies() []ChallengePolicy

GetPolicies 获取策略列表（公开方法） GetPolicies gets the policies list (public method)

func (*Client) ReadAllAnchorArticles 

func (c *Client) ReadAllAnchorArticles(url string, renderFunc func(*rod.Page), requestModifiers ...RequestWith) AnchorArticleValue

ReadAllAnchorArticles reads all anchor links and fetches title/content.

func (*Client) ReadAllAnchorArticlesWithChrome 

func (c *Client) ReadAllAnchorArticlesWithChrome(url string, listRenderFunc func(*rod.Page), articleRenderFunc func(*rod.Page), restartCondition func(Article) bool, i ...int) AnchorArticleValue

ReadAllAnchorArticlesWithChrome reads all anchor links and fetches title/content with Chrome.

func (*Client) ReadAnchorArticlesWithCURL 

func (c *Client) ReadAnchorArticlesWithCURL(curl string, renderFunc func(*rod.Page), requestModifiers ...RequestWith) AnchorArticleValue

ReadAnchorArticlesWithCURL 使用 curl 命令读取锚点链接并获取标题/内容。 ReadAnchorArticlesWithCURL reads anchor links and fetches title/content using a curl command.

func (*Client) ReadAnchorArticlesWithCURLOptions 

func (c *Client) ReadAnchorArticlesWithCURLOptions(curl string, renderFunc func(*rod.Page), options LinkExtractOptions, requestModifiers ...RequestWith) AnchorArticleValue

ReadAnchorArticlesWithCURLOptions 使用 curl 命令读取锚点链接并获取标题/内容（带提取选项）。 ReadAnchorArticlesWithCURLOptions reads anchor links and fetches title/content using a curl command with options.

func (*Client) ReadAnchorArticlesWithChromeOptions 

func (c *Client) ReadAnchorArticlesWithChromeOptions(url string, listRenderFunc func(*rod.Page), articleRenderFunc func(*rod.Page), restartCondition func(Article) bool, options LinkExtractOptions, i ...int) AnchorArticleValue

ReadAnchorArticlesWithChromeOptions reads anchor links and fetches title/content with Chrome and options.

func (*Client) ReadAnchorArticlesWithOptions 

func (c *Client) ReadAnchorArticlesWithOptions(url string, renderFunc func(*rod.Page), options LinkExtractOptions, requestModifiers ...RequestWith) AnchorArticleValue

ReadAnchorArticlesWithOptions reads anchor links and fetches title/content with options.

func (*Client) ReadArray 

func (c *Client) ReadArray(url, js string, renderFunc func(*rod.Page)) AttrValue

ReadArray 读取数组数据 ReadArray reads array data 

func (c *Client) ReadArticleLinks(url, js string, renderFunc func(*rod.Page), requestModifiers ...RequestWith) ArticleLinkValue

ReadArticleLinks reads article links and titles from a list page.

func (*Client) ReadArticleLinksAuto 

func (c *Client) ReadArticleLinksAuto(url string, renderFunc func(*rod.Page), requestModifiers ...RequestWith) ArticleLinkValue

ReadArticleLinksAuto reads article links and titles without custom JS selectors.

func (*Client) ReadArticleLinksAutoWithCURL 

func (c *Client) ReadArticleLinksAutoWithCURL(curl string, renderFunc func(*rod.Page), requestModifiers ...RequestWith) ArticleLinkValue

ReadArticleLinksAutoWithCURL 使用 curl 命令读取文章链接和标题（自动解析，无 JS 选择器）。 ReadArticleLinksAutoWithCURL reads article links and titles using a curl command without custom JS selectors.

func (*Client) ReadArticleLinksAutoWithCURLOptions 

func (c *Client) ReadArticleLinksAutoWithCURLOptions(curl string, renderFunc func(*rod.Page), options LinkExtractOptions, requestModifiers ...RequestWith) ArticleLinkValue

ReadArticleLinksAutoWithCURLOptions 使用 curl 命令读取文章链接和标题（带提取选项）。 ReadArticleLinksAutoWithCURLOptions reads article links and titles using a curl command with extraction options.

func (*Client) ReadArticleLinksAutoWithChrome 

func (c *Client) ReadArticleLinksAutoWithChrome(url string, listRenderFunc func(*rod.Page), articleRenderFunc func(*rod.Page), restartCondition func(Article) bool, i ...int) ArticleLinkValue

ReadArticleLinksAutoWithChrome reads article links and titles without custom JS selectors.

func (*Client) ReadArticleLinksAutoWithChromeOptions 

func (c *Client) ReadArticleLinksAutoWithChromeOptions(url string, listRenderFunc func(*rod.Page), articleRenderFunc func(*rod.Page), restartCondition func(Article) bool, options LinkExtractOptions, i ...int) ArticleLinkValue

ReadArticleLinksAutoWithChromeOptions reads article links and titles with extraction options.

func (*Client) ReadArticleLinksAutoWithOptions 

func (c *Client) ReadArticleLinksAutoWithOptions(url string, renderFunc func(*rod.Page), options LinkExtractOptions, requestModifiers ...RequestWith) ArticleLinkValue

ReadArticleLinksAutoWithOptions reads article links and titles with extraction options.

func (*Client) ReadArticleLinksWithCURL 

func (c *Client) ReadArticleLinksWithCURL(curl, js string, renderFunc func(*rod.Page), requestModifiers ...RequestWith) ArticleLinkValue

ReadArticleLinksWithCURL 使用 curl 命令读取文章链接和标题。 ReadArticleLinksWithCURL reads article links and titles using a curl command.

func (*Client) ReadArticleLinksWithChrome 

func (c *Client) ReadArticleLinksWithChrome(url, js string, listRenderFunc func(*rod.Page), articleRenderFunc func(*rod.Page), restartCondition func(Article) bool, i ...int) ArticleLinkValue

ReadArticleLinksWithChrome reads article links and titles using Chrome for article pages.

func (*Client) ReadFeed 

func (c *Client) ReadFeed(xmlURL string) rss.RSS

ReadFeed 读取Atom订阅源并转换为RSS格式 ReadFeed reads Atom feed source and converts to RSS format

func (*Client) ReadRSS 

func (c *Client) ReadRSS(xmlURL string) rss.RSS

ReadRSS 读取RSS订阅源 ReadRSS reads RSS feed source

func (*Client) ReadSourceCode 

func (c *Client) ReadSourceCode(url, execJsFunc string, rendorFunc func(*rod.Page)) SourceCode

ReadSourceCode 读取网页源代码（带重试机制） ReadSourceCode reads web page source code (with retry mechanism)

func (*Client) ReadSourceCodeWithCURL 

func (c *Client) ReadSourceCodeWithCURL(curl string, renderFunc func(*rod.Page)) SourceCode

ReadSourceCodeWithCURL reads source code using headers/cookies from a curl command.

func (*Client) ReadSourceCodeWithHAR 

func (c *Client) ReadSourceCodeWithHAR(har string, entryIndex int, renderFunc func(*rod.Page)) SourceCode

ReadSourceCodeWithHAR reads source code using headers/cookies from a HAR entry.

func (*Client) ReadToJson 

func (c *Client) ReadToJson(url string, obj any, js string, renderFunc func(*rod.Page)) JsonValue[any]

ReadToJson 读取JSON数据并反序列化为指定类型 ReadToJson reads JSON data and deserializes to specified type

func (*Client) SolvePuzzleSlider 

func (c *Client) SolvePuzzleSlider(sliderSelector, containerSelector, bgSelector string) error

SolvePuzzleSlider 通用缺口滑块解决入口 SolvePuzzleSlider generic puzzle slider solver entry point

func (*Client) SolveSimpleSlider 

func (c *Client) SolveSimpleSlider(sliderSelector, containerSelector string) error

SolveSimpleSlider 通用简单滑块解决入口 SolveSimpleSlider generic simple slider solver entry point

type CloudflarePolicy 

type CloudflarePolicy struct{}

CloudflarePolicy Cloudflare策略实现

func (*CloudflarePolicy) ShouldHandle 

func (p *CloudflarePolicy) ShouldHandle(resp *http.Response) bool

func (*CloudflarePolicy) Solve 

func (p *CloudflarePolicy) Solve(chrome *core.Contain, url string) ([]*http.Cookie, string, error)

type JsonValue 

type JsonValue[T any] struct {
	Value *T
	WebReaderError
}

JsonValue JSON值包装器 JsonValue is a JSON value wrapper

func ReadToJson 

func ReadToJson[T any](url string, obj T, js string, renderFunc func(*rod.Page)) JsonValue[T]

ReadToJson 读取JSON数据并反序列化为指定类型 ReadToJson reads JSON data and deserializes to specified type

type LinkExtractOptions 

type LinkExtractOptions struct {
	Selector      string
	IncludeHref   []string
	ExcludeHref   []string
	IncludeText   []string
	ExcludeText   []string
	AllowExternal bool
	MaxLinks      int
}

LinkExtractOptions controls how links are selected from a list page.

func AllAnchorOptions 

func AllAnchorOptions() LinkExtractOptions

AllAnchorOptions returns options that read all anchors, including external links.

func DefaultLinkExtractOptions 

func DefaultLinkExtractOptions() LinkExtractOptions

DefaultLinkExtractOptions returns the default extraction options.

type PageWaitOptions 

type PageWaitOptions struct {
	Selector    string
	MinCount    int
	ReadyState  string
	Timeout     time.Duration
	IdleDelay   time.Duration
	Scroll      bool
	MaxScroll   int
	ScrollDelay time.Duration
}

PageWaitOptions controls how to wait for a page to finish async loading.

func DefaultPageWaitOptions 

func DefaultPageWaitOptions() PageWaitOptions

DefaultPageWaitOptions returns default async wait options.

type RequestWith 

type RequestWith func(r *http.Request)

RequestWith 请求修改函数类型 RequestWith is a function type for request modification

type RodPasteOptions 

type RodPasteOptions struct {
	TargetURL        string
	ExcludeHeaders   []string
	IncludeUserAgent bool
	IncludeHeaders   bool
	IncludeCookies   bool
}

RodPasteOptions controls how to generate rod page setup code.

func DefaultRodPasteOptions 

func DefaultRodPasteOptions(targetURL string) RodPasteOptions

DefaultRodPasteOptions returns default options for generating rod page setup code.

type RodPasteResult 

type RodPasteResult struct {
	Headers        map[string]string
	Cookies        []*proto.NetworkCookieParam
	UserAgent      string
	AcceptLanguage string
}

RodPasteResult holds parsed headers/cookies and can emit code snippets.

func ParseRodPaste 

func ParseRodPaste(rawHeaders, rawCookies string, options RodPasteOptions) (*RodPasteResult, error)

ParseRodPaste parses raw headers/cookies into a reusable structure.

func ParseRodPasteFromCURL 

func ParseRodPasteFromCURL(curl string, options RodPasteOptions) (*RodPasteResult, error)

ParseRodPasteFromCURL parses a curl command into a RodPasteResult.

func ParseRodPasteFromCURLWithURL 

func ParseRodPasteFromCURLWithURL(curl string, options RodPasteOptions) (*RodPasteResult, string, error)

ParseRodPasteFromCURLWithURL parses a curl command into a RodPasteResult and returns target URL.

func ParseRodPasteFromHAR 

func ParseRodPasteFromHAR(har string, entryIndex int, options RodPasteOptions) (*RodPasteResult, error)

ParseRodPasteFromHAR parses a HAR JSON string into a RodPasteResult.

func ParseRodPasteFromHARWithURL 

func ParseRodPasteFromHARWithURL(har string, entryIndex int, options RodPasteOptions) (*RodPasteResult, string, error)

ParseRodPasteFromHARWithURL parses a HAR JSON string into a RodPasteResult and returns target URL.

func (*RodPasteResult) ApplyToPage 

func (r *RodPasteResult) ApplyToPage(page *rod.Page) (cleanup func(), err error)

ApplyToPage applies headers/cookies/user-agent to a rod page.

func (*RodPasteResult) MustApplyToPage 

func (r *RodPasteResult) MustApplyToPage(page *rod.Page) (cleanup func())

MustApplyToPage applies headers/cookies/user-agent and panics on error.

func (*RodPasteResult) RenderFunc 

func (r *RodPasteResult) RenderFunc() func(*rod.Page)

RenderFunc returns a render function that applies the rod paste settings.

func (*RodPasteResult) RequestModifier 

func (r *RodPasteResult) RequestModifier() RequestWith

RequestModifier 构建一个用于 HTTP 请求的 RequestWith，注入 headers/cookies。 RequestModifier builds a RequestWith that applies headers/cookies to http requests.

func (*RodPasteResult) ToSnippet 

func (r *RodPasteResult) ToSnippet(pageVar string) string

ToSnippet renders a Go snippet that sets headers/cookies on a rod.Page.

type SourceCode 

type SourceCode struct {
	WebReaderString
	WebReaderError
}

SourceCode 源代码结构体，包含网页源码和错误信息 SourceCode represents source code structure containing web page source and error information

func ReadSourceCode 

func ReadSourceCode(url, execJsFunc string, rendorFunc func(*rod.Page)) SourceCode

ReadSourceCode 读取网页源代码（带重试机制） ReadSourceCode reads web page source code (with retry mechanism)

func ReadSourceCodeWithCURL 

func ReadSourceCodeWithCURL(curl string, renderFunc func(*rod.Page)) SourceCode

ReadSourceCodeWithCURL reads source code using headers/cookies from a curl command.

func ReadSourceCodeWithHAR 

func ReadSourceCodeWithHAR(har string, entryIndex int, renderFunc func(*rod.Page)) SourceCode

ReadSourceCodeWithHAR reads source code using headers/cookies from a HAR entry.

type WebReaderError 

type WebReaderError struct {
	// contains filtered or unexported fields
}

WebReaderError Web读取错误包装器 WebReaderError is a web reading error wrapper

func (WebReaderError) IsErr 

func (w WebReaderError) IsErr() bool

IsErr 检查是否有错误 IsErr checks if there is an error

func (WebReaderError) SolveErrors 

func (w WebReaderError) SolveErrors(errs error) error

SolveErrors 处理错误 SolveErrors handles errors

type WebReaderString 

type WebReaderString string

WebReaderString Web读取字符串类型 WebReaderString is a web reading string type

func (WebReaderString) String 

func (w WebReaderString) String() string

String 转换为字符串 String converts to string

type WebReaderValue 

type WebReaderValue []WebReaderString

WebReaderValue Web读取值集合 WebReaderValue is a collection of web reading values

func NewWebReaderValue 

func NewWebReaderValue(strs ...string) WebReaderValue

NewWebReaderValue 创建新的Web读取值 NewWebReaderValue creates a new web reading value

func (WebReaderValue) Stream 

func (w WebReaderValue) Stream() *stream.Stream[WebReaderString, WebReaderValue]

Stream 转换为流 Stream converts to stream

Source Files

View all Source files

Directories

Path Synopsis
test
example command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.