Documentation
¶
Overview ¶
Package mirror holds the shared routing logic for the Douban mirror crawler: turning a URL into the entity it represents and the source best suited to fetch it. The store, sitemap, and crawl packages build on this classification.
Index ¶
Constants ¶
const ( SourceFrodo = "frodo" SourceHTML = "html" SourceMobile = "mobile" )
Source values mirror store.Source* and name the fetch surface a URL is best reached through. Kept as plain strings to avoid importing the store here.
Variables ¶
This section is empty.
Functions ¶
func CanonicalURL ¶
CanonicalURL returns the single representative URL for an entity, the one a sub-page collapses onto. It is the source of truth for both seeding an id range and normalizing discovered links. ok is false for unknown types.
func NormalizeFrodo ¶
func NormalizeFrodo(entityType, id, url string, body []byte) (json.RawMessage, error)
NormalizeFrodo maps a Frodo JSON document to the uniform record, hoisting the common fields and keeping the full object under Data. It returns an error only when the body is not a JSON object, so the caller can fall back to raw bytes.
func NormalizeHTML ¶
func NormalizeHTML(entityType, id, url string, body []byte) json.RawMessage
NormalizeHTML maps an HTML page to the uniform record on a best-effort basis, capturing the title, Open Graph and named meta, every JSON-LD block, the microdata rating, and the subject #info label/value pairs. It never errors; the raw artifact remains the lossless copy.
Types ¶
type Class ¶
Class is the result of routing a URL: what entity it is, its id, and which source should fetch it. Canonical is the one URL that represents the entity, so sub-pages (/comments, /reviews, /new_review) collapse onto a single frontier row instead of each becoming a separate, often login-gated, fetch.
type Normalized ¶
type Normalized struct {
EntityType string `json:"entity_type"`
EntityID string `json:"entity_id"`
Source string `json:"source"`
URL string `json:"url,omitempty"`
Title string `json:"title,omitempty"`
Year string `json:"year,omitempty"`
Cover string `json:"cover,omitempty"`
Intro string `json:"intro,omitempty"`
Rating *Rating `json:"rating,omitempty"`
Fields map[string]string `json:"fields,omitempty"`
Meta map[string]string `json:"meta,omitempty"`
JSONLD []json.RawMessage `json:"jsonld,omitempty"`
Data json.RawMessage `json:"data,omitempty"`
}
Normalized is the uniform record every entity maps to, whatever its source. It hoists the fields worth querying to the top level and keeps the complete source alongside them so nothing is lost: Frodo documents land in Data verbatim, and HTML structured-data blobs land in Meta, JSONLD, and Fields.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package crawl runs the Douban mirror: it drains the frontier, fetches each URL through the source that serves it, stores the raw bytes and a record, expands the links it discovers, and marks every row's outcome.
|
Package crawl runs the Douban mirror: it drains the frontier, fetches each URL through the source that serves it, stores the raw bytes and a record, expands the links it discovers, and marks every row's outcome. |
|
Package sitemap enumerates Douban's crawlable URL space from its sitemap backbone.
|
Package sitemap enumerates Douban's crawlable URL space from its sitemap backbone. |
|
Package store is the persistence layer for the Douban mirror: a frontier of URLs to crawl, the normalized records harvested from them, crawl cursors, and the raw artifact files captured verbatim.
|
Package store is the persistence layer for the Douban mirror: a frontier of URLs to crawl, the normalized records harvested from them, crawl cursors, and the raw artifact files captured verbatim. |