baidu

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 14, 2026 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Overview

Package baidu is the library behind the baidu command: the HTTP clients, request shaping, anti-bot handling, parsers, and typed data models for Baidu.

Baidu is several products behind one brand, and each is served by its own host: top.baidu.com for the hot search board, www.baidu.com for the typeahead (the open sugrec endpoint) and web search, and baike.baidu.com for the encyclopedia. The hot board and suggest serve open JSON from any IP. The web SERP is walled (CAPTCHA) and best effort. The encyclopedia (card API, article HTML, category API) is anti-bot/geo-walled: it answers from China IPs but returns 403 / {"errno":2} / empty lists from at least non-China and anonymous IPs, which the client surfaces as a clean block rather than a hard failure. No API key is needed anywhere.

Package baidu provides typed data models for the baidu command.

Index

Constants

View Source 
const (

	// DefaultUserAgent identifies the client to Baidu when the operator does
	// not pin one with --user-agent. The search and baike fetchers rotate the
	// pool instead, but this stays the baseline for the hot board and suggest.
	DefaultUserAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"

	// DefaultRate is the spacing for the open hot board and suggest surfaces.
	DefaultRate = 200 * time.Millisecond
	// DefaultTimeout is the per-request HTTP timeout.
	DefaultTimeout = 30 * time.Second
	// DefaultRetries is how many times a transient failure is retried.
	DefaultRetries = 3
)
View Source 
const (
	// BaikeBaseURL is the encyclopedia host.
	BaikeBaseURL = "https://baike.baidu.com"
	// SearchBaseURL is the web search host (also serves the suggest sugrec path).
	SearchBaseURL = "https://www.baidu.com"

	// EntityLemma is a Baike article frontier row.
	EntityLemma = "lemma"
	// EntitySERP is a web search results frontier row.
	EntitySERP = "serp"
	// EntitySuggest is a suggest query record source.
	EntitySuggest = "suggest"
)

Variables

View Source 
var BaikeCategoryTags = []string{
	"历史", "科学", "地理", "人物", "文化", "技术", "艺术", "体育",
	"经济", "政治", "教育", "医学", "自然", "哲学", "法律", "军事",
}

BaikeCategoryTags are the 16 top-level Baike category tags walked by the categories op and the mirror seeder.

View Source 
var BaikeSeedTopics = []string{

	"Python", "Java", "Go语言", "C++", "JavaScript", "人工智能", "机器学习",
	"深度学习", "神经网络", "大数据", "区块链", "云计算", "物联网",
	"操作系统", "Linux", "互联网", "数据库", "算法", "计算机",

	"量子力学", "相对论", "DNA", "基因组", "宇宙", "黑洞", "进化论",
	"光合作用", "元素周期表", "引力波", "化学元素", "物理学", "数学",

	"中国历史", "中华人民共和国", "唐朝", "宋朝", "明朝", "清朝",
	"秦始皇", "汉朝", "丝绸之路", "改革开放", "抗日战争", "长征",

	"北京", "上海", "广州", "深圳", "长江", "黄河", "西藏",
	"新疆", "台湾", "喜马拉雅山", "珠穆朗玛峰", "黄土高原",

	"春节", "中秋节", "汉字", "普通话", "儒家思想", "道教", "佛教",
	"中医", "武术", "围棋", "书法", "中国画", "京剧",

	"毛泽东", "邓小平", "孔子", "老子", "李白", "杜甫",
	"鲁迅", "袁隆平", "屠呦呦",

	"新冠病毒", "流感", "癌症", "高血压", "糖尿病", "疫苗",

	"GDP", "人民币", "通货膨胀", "一带一路", "股票市场",

	"电影", "音乐", "网络小说", "动漫",
}

BaikeSeedTopics is the bootstrap list of seed topics for breadth-first discovery. Each is resolved through the card API and, if it resolves, the article HTML is enqueued for the full body.

View Source 
var ErrBlocked = errors.New("blocked by Baidu (CAPTCHA or rate limit)")

ErrBlocked is the sentinel for a walled SERP (CAPTCHA or block page) so the kit layer can map it to the rate-limited exit and surface the --baiduid hint.

View Source 
var ErrNotFound = errors.New("not found")

ErrNotFound is the sentinel for a missing lemma so the kit layer can map it to the not-found / no-results exit codes.

Functions

func Defaults added in v0.2.0 

func Defaults(c *kit.Config)

Defaults seeds the framework baseline with baidu's own values, so an unset --rate or --timeout uses the baidu default rather than the generic kit one. It is passed to kit.New via kit.WithDefaults.

func EncodeTitle added in v0.2.0 

func EncodeTitle(title string) string

EncodeTitle percent-encodes non-ASCII characters in a Baike title for URLs.

func ExtractRelatedLemmaIDs added in v0.2.0 

func ExtractRelatedLemmaIDs(body []byte) []int

ExtractRelatedLemmaIDs parses all /item/xxx/{id} links from an HTML body, a fast regex path used independently of a full parse.

func Identity added in v0.2.0 

func Identity() kit.Identity

Identity is the fixed description of the baidu CLI, shared by the domain and the standalone composition root so help and version read the same everywhere.

func IsCaptchaPage added in v0.2.0 

func IsCaptchaPage(body []byte) bool

IsCaptchaPage reports whether the body looks like a Baidu CAPTCHA or block page.

func LemmaURL added in v0.2.0 

func LemmaURL(title string, lemmaID int) string

LemmaURL builds the canonical Baike URL for a given lemma id and title.

func LemmaURLByID added in v0.2.0 

func LemmaURLByID(lemmaID int) string

LemmaURLByID builds the Baike URL using only the lemma id; Baike redirects to the full canonical URL.

func MergeAPIData added in v0.2.0 

func MergeAPIData(art *BaikeArticle, api *baikeAPIResp)

MergeAPIData overlays the Baike card API onto an HTML article, filling any gap.

Types

type BaikeArticle added in v0.2.0 

type BaikeArticle struct {
	LemmaID         int           `json:"lemma_id"`
	URL             string        `json:"url"`
	Title           string        `json:"title"`
	Subtitle        string        `json:"subtitle"`
	Abstract        string        `json:"abstract" table:",truncate"`
	BodyMarkdown    string        `json:"body_markdown" table:"-"`
	Infobox         []InfoboxItem `json:"infobox" table:"-"`
	Sections        []Section     `json:"sections" table:"-"`
	Categories      []string      `json:"categories"`
	Tags            []string      `json:"tags"`
	Synonyms        []string      `json:"synonyms"`
	RelatedIDs      []int         `json:"related_ids" table:"-"`
	Images          []ImageItem   `json:"images" table:"-"`
	ImageURL        string        `json:"image_url"`
	ReferencesCount int           `json:"references_count"`
	// Metadata from window.PAGE_DATA
	Creator      string    `json:"creator"`
	LastEditor   string    `json:"last_editor"`
	LastEditTime time.Time `json:"last_edit_time"`
	VersionID    int64     `json:"version_id"`
	FetchedAt    time.Time `json:"fetched_at"`
}

BaikeArticle is a Baidu Baike encyclopedia article. It is the centerpiece record, filled from the card API and the article HTML.

func ParseBaikeHTML added in v0.2.0 

func ParseBaikeHTML(body []byte) (*BaikeArticle, error)

ParseBaikeHTML extracts the full article content from a Baike HTML page. It uses J-* stable selectors rather than hashed CSS class names, plus the embedded window.PAGE_DATA JSON. It is best effort and never panics; the raw artifact is the lossless copy.

type CategoryLemma added in v0.2.0 

type CategoryLemma struct {
	LemmaID  int    `json:"lemma_id"`
	Title    string `json:"title"`
	Category string `json:"category"`
}

CategoryLemma is a lightweight discovery record: one lemma found under a Baike category tag.

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client talks to all four Baidu surfaces. It holds two HTTP clients: one follows redirects (the hot board, suggest, and Baike, whose article pages legitimately 302 to a canonical URL) and one blocks them (the SERP, where a 302 is a CAPTCHA bounce). A single mutex paces requests across both.

func ClientFromConfig added in v0.2.0 

func ClientFromConfig(cfg kit.Config) *Client

ClientFromConfig maps the framework config onto a baidu.Config and returns a client. The mirror commands reuse it so the CLI surface and the crawler share one notion of rate, timeout, retries, User-Agent, and BAIDUID.

func NewClient 

func NewClient(cfg Config) *Client

NewClient returns a Client with the given config.

func (*Client) Article added in v0.2.0 

func (c *Client) Article(ctx context.Context, ref string) (*BaikeArticle, error)

Article resolves and returns a single Baidu Baike encyclopedia article.

ref is one of:

  • a numeric lemma id ("123456"),
  • a topic title ("人工智能"),
  • a title/id pair ("人工智能/123456"), the canonical Baike path shape.

Resolution: a bare title is first looked up through the card API to find its numeric lemma id and abstract; then the article HTML is fetched and parsed for the full body, infobox, sections, and related ids; finally the card API data is merged in to fill any gaps. A ref that resolves to nothing returns ErrNotFound.

func (*Client) Categories added in v0.2.0 

func (c *Client) Categories(ctx context.Context, tag string, limit int) ([]CategoryLemma, error)

Categories walks Baidu Baike category tags and returns the lemma stubs found under them. With a tag, only that tag is walked. With an empty tag, all 16 built-in top-level tags are walked in turn.

limit caps the total number of lemmas returned across all walked tags; a non-positive limit means no cap. The wikitag API is open and paginated by an opaque "after" cursor; this follows the cursor until it is exhausted or the limit is reached.

func (*Client) FetchBaiduSearch added in v0.2.0 

func (c *Client) FetchBaiduSearch(ctx context.Context, query string, page int) ([]byte, bool, error)

FetchBaiduSearch fetches a Baidu Search SERP page. It returns (body, true, nil) on success and (nil, false, nil) on a CAPTCHA block, so the caller can degrade cleanly rather than treat a challenge page as data.

func (*Client) FetchBaikeAPI added in v0.2.0 

func (c *Client) FetchBaikeAPI(ctx context.Context, key string) (*baikeAPIResp, error)

FetchBaikeAPI fetches the Baike JSON card API for a given search key. It is the cheap probe that resolves a topic name to a numeric lemma id.

func (*Client) FetchBaikeHTML added in v0.2.0 

func (c *Client) FetchBaikeHTML(ctx context.Context, lemmaURL string) ([]byte, int, error)

FetchBaikeHTML fetches a Baike article HTML page by lemma URL.

func (*Client) FetchCategoryLemmas added in v0.2.0 

func (c *Client) FetchCategoryLemmas(ctx context.Context, tagName, after string) ([]wikitagLemma, string, error)

FetchCategoryLemmas fetches one page of lemmas for a Baike tag/category name. It returns the items and the cursor for the next page (empty = last page).

func (*Client) Hot 

func (c *Client) Hot(ctx context.Context, tab string, limit int) ([]HotItem, error)

Hot fetches the hot search board for the given tab and returns up to limit items. tab is one of: realtime, novel, movie, teleplay, car.

func (*Client) LemmaURLByID added in v0.2.0 

func (c *Client) LemmaURLByID(lemmaID int) string

LemmaURLByID is the exported, base-aware form used by the mirror crawler so the queued URL points at the client's configured Baike host.

func (*Client) Search added in v0.2.0 

func (c *Client) Search(ctx context.Context, query string, pages int) ([]SearchResult, error)

Search runs a Baidu web search for query across the given number of result pages (page 1..pages) and returns the merged result list in rank order.

The web SERP is the only walled Baidu surface. When Baidu serves a CAPTCHA or block page instead of results, Search returns ErrBlocked so the caller can map it to a rate-limited exit and surface the --baiduid hint.

func (*Client) Suggest 

func (c *Client) Suggest(ctx context.Context, query string) ([]Suggestion, error)

Suggest fetches query suggestions for the given search term. It uses the open sugrec endpoint, which returns clean UTF-8 JSON ({"q":..,"g":[{"q":..}]}) from any IP — unlike the legacy /su path, which serves GBK-encoded, unquoted-key JSONP that a JSON decoder cannot parse.

type Config 

type Config struct {
	BaseURL        string // top.baidu.com override
	SuggestBaseURL string // suggestion.baidu.com override
	SearchBaseURL  string // www.baidu.com override
	BaikeBaseURL   string // baike.baidu.com override
	UserAgent      string // "" rotates the pool for search/baike
	BaiduID        string // "" synthesizes a BAIDUID per SERP request
	Rate           time.Duration
	Timeout        time.Duration
	Retries        int
}

Config holds constructor parameters. When BaseURL or one of the per-surface base URLs is set, the client resolves every matching host to it, which is the injectable-base pattern the tests use to point all four surfaces at one httptest mux. An empty value means the real Baidu host.

func DefaultConfig 

func DefaultConfig() Config

DefaultConfig returns sensible defaults that hit the real Baidu hosts.

type Domain added in v0.2.0 

type Domain struct{}

Domain is the baidu driver for the any-cli/kit framework. It declares the lookup operations once, and kit derives the CLI subcommands, the HTTP API routes, and the MCP tools from that single registry. The same Domain powers the standalone baidu binary and a multi-domain host that mounts it.

func (Domain) Info added in v0.2.0 

func (Domain) Info() kit.DomainInfo

Info describes the domain: its URI scheme, the hosts it owns, and the identity the standalone binary reuses for help and version.

func (Domain) Register added in v0.2.0 

func (Domain) Register(app *kit.App)

Register installs the client factory and every lookup operation on the app. The mirror subsystem is wired as escape-hatch commands by the cli package, since it is CLI-only (stateful crawl, not the emit-records shape).

type HotItem 

type HotItem struct {
	Rank int    `json:"rank"`
	Word string `json:"word"`
	Tag  string `json:"tag"`
	URL  string `json:"url"`
}

HotItem is one entry from the Baidu hot search board.

type ImageItem added in v0.2.0 

type ImageItem struct {
	URL     string `json:"url"`
	Caption string `json:"caption"`
}

ImageItem is an image found in a Baike article body.

type InfoboxItem added in v0.2.0 

type InfoboxItem struct {
	Name  string `json:"name"`
	Value string `json:"value"`
}

InfoboxItem is one name/value pair of a Baike article infobox.

type SearchResult added in v0.2.0 

type SearchResult struct {
	ID         string    `json:"id"`
	Query      string    `json:"query"`
	Page       int       `json:"page"`
	Position   int       `json:"position"`
	URL        string    `json:"url"`
	DisplayURL string    `json:"display_url"`
	Title      string    `json:"title"`
	Snippet    string    `json:"snippet" table:",truncate"`
	Tpl        string    `json:"tpl"`
	IsAd       bool      `json:"is_ad"`
	FetchedAt  time.Time `json:"fetched_at"`
}

SearchResult is a single Baidu Search SERP entry.

func ParseSERP added in v0.2.0 

func ParseSERP(body []byte, query string, page int) ([]SearchResult, error)

ParseSERP parses a Baidu Search HTML page for a given query and page number.

type Section added in v0.2.0 

type Section struct {
	Level int    `json:"level"`
	Name  string `json:"name"`
}

Section is one heading in a Baike article body.

type Suggestion 

type Suggestion struct {
	Rank int    `json:"rank"`
	Word string `json:"word"`
}

Suggestion is one entry from the Baidu suggest API.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.