crawl

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 14, 2026 License: Apache-2.0 Imports: 9 Imported by: 0

Documentation

Overview

Package crawl runs the Baidu mirror: it drains the queue, fetches each item (a Baike lemma or a web search page) through the baidu client, stores the raw bytes and a parsed record, expands the related lemmas it discovers, and marks every row's outcome. The engine is resumable (in-flight rows stay pending) and polite (the client paces itself), and it never silently caps; it logs what it blocks or skips.

Index

Constants

View Source 
const (
	EntityLemma = "lemma"
	EntitySERP  = "serp"
)

Entity type names used as queue keys.

Variables

This section is empty.

Functions

func SERPRef 

func SERPRef(query string, page int) string

SERPRef encodes a (query, page) pair as a queue ref.

Types

type Config 

type Config struct {
	Concurrency int // worker pool size (default 4)
	Limit       int // max items to process this run (0 = drain the queue)
	Logf        func(format string, args ...any)
}

Config tunes a crawl run.

type Engine 

type Engine struct {
	// contains filtered or unexported fields
}

Engine ties the store and the baidu client together.

func New 

func New(st *store.Store, client *baidu.Client, cfg Config) *Engine

New builds an Engine.

func (*Engine) Run 

func (e *Engine) Run(ctx context.Context) (Stats, error)

Run drains pending queue rows until none remain, the limit is hit, or the context is cancelled. Newly discovered related lemmas are enqueued as it goes, so a single Run expands the mirror outward from its seeds.

type Stats 

type Stats struct {
	Processed  int
	Done       int
	Failed     int
	Blocked    int
	Discovered int
}

Stats summarizes a run.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.