Directories
¶
| Path | Synopsis |
|---|---|
|
Package baidu is the library behind the baidu command: the HTTP clients, request shaping, anti-bot handling, parsers, and typed data models for Baidu.
|
Package baidu is the library behind the baidu command: the HTTP clients, request shaping, anti-bot handling, parsers, and typed data models for Baidu. |
|
Package cli builds the baidu command tree on top of the baidu library and the any-cli/kit framework.
|
Package cli builds the baidu command tree on top of the baidu library and the any-cli/kit framework. |
|
cmd
|
|
|
baidu
command
Command baidu is the single-binary command line for baidu-cli.
|
Command baidu is the single-binary command line for baidu-cli. |
|
mirror
|
|
|
crawl
Package crawl runs the Baidu mirror: it drains the queue, fetches each item (a Baike lemma or a web search page) through the baidu client, stores the raw bytes and a parsed record, expands the related lemmas it discovers, and marks every row's outcome.
|
Package crawl runs the Baidu mirror: it drains the queue, fetches each item (a Baike lemma or a web search page) through the baidu client, stores the raw bytes and a parsed record, expands the related lemmas it discovers, and marks every row's outcome. |
|
store
Package store is the persistence layer for the Baidu mirror: a crawl queue, a visited set, the harvested records (Baike articles, search results, suggest queries), the per-run job log, and the raw artifact files captured verbatim.
|
Package store is the persistence layer for the Baidu mirror: a crawl queue, a visited set, the harvested records (Baike articles, search results, suggest queries), the per-run job log, and the raw artifact files captured verbatim. |
Click to show internal directories.
Click to hide internal directories.