Documentation
¶
Overview ¶
Package sitemap enumerates Douban's crawlable URL space from its sitemap backbone. There is exactly one backbone for the whole site (www.douban.com/sitemap_index.xml); every subdomain's robots.txt points at it. The index lists 10,706 gzipped child sitemaps of 50,000 URLs each, partitioned into contiguous bands by entity type, plus a daily updated index for freshness.
Index ¶
- Constants
- Variables
- func Children(ctx context.Context, f Fetcher, childURL string) ([]string, error)
- func Gunzip(data []byte) ([]byte, error)
- func Index(ctx context.Context, f Fetcher, indexURL string) ([]string, error)
- func ParseIndex(data []byte) ([]string, error)
- func ParseURLSet(data []byte) ([]string, error)
- func SelectBand(children []string, name string) []string
- type Band
- type Fetcher
Constants ¶
const ( IndexURL = "https://www.douban.com/sitemap_index.xml" UpdatedIndexURL = "https://www.douban.com/sitemap_updated_index.xml" )
The site-wide sitemap entry points.
Variables ¶
var Bands = []Band{
{"subject", 0, 88},
{"review", 88, 285},
{"group-topic", 285, 4890},
{"event", 4890, 4925},
{"people", 4925, 9995},
{"note", 9995, 10635},
{"celebrity", 10640, 10704},
{"musician", 10705, 10706},
}
Bands is the approximate entity partition of the full index (probed 2026-06-14).
Functions ¶
func Gunzip ¶
Gunzip decompresses gzip data, passing already-plain XML through unchanged so callers need not know whether a child arrived compressed.
func ParseIndex ¶
ParseIndex parses a sitemap index document into its ordered child loc URLs.
func ParseURLSet ¶
ParseURLSet parses a (decompressed) child sitemap into its loc URLs.
func SelectBand ¶
SelectBand returns the children of the named band from an ordered index list, clamped to the list bounds. An unknown band name returns nil.
Types ¶
type Band ¶
Band names a contiguous run of child sitemaps holding one entity type. The range is [Start, End) over the ordered child list. Boundaries drift as the catalog grows, so a crawl should sniff edge files before trusting them.
func BandByName ¶
BandByName returns the named band, or ok=false if unknown.
Source Files