spider

package
v1.0.7-alpha Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 15, 2022 License: Apache-2.0 Imports: 5 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	RegisteredNotSpider       = errors.New("the registered type must be SpiderInfo")
	RegisterSpiderUidConflict = errors.New("uid already exists in the crawler collection, try replacing spider uid")

	RefCallFuncBackErr = errors.New("an unreliable reflection function was called")
)

Functions

This section is empty.

Types

type Manage 

type Manage struct {
	// contains filtered or unexported fields
}

func SpiderManage 

func SpiderManage() *Manage

func (*Manage) MustRegister 

func (s *Manage) MustRegister(sp Spider)

register Spider

func (*Manage) ParseResp 

func (s *Manage) ParseResp(resp *stream.HttpResponse) (stream.Stream, error)

爬虫解析方案

func (*Manage) RegisterCallbackFunc 

func (s *Manage) RegisterCallbackFunc(sp Spider, callback stream.CallbackFunc)

type Spider 

type Spider interface {
	stream.Stream

	Parse(response *stream.HttpResponse) (stream.Stream, error)
	StartRequest() stream.Stream
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.