Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Crawl ¶
func Crawl(ctx context.Context, seed string, s *scrape.Scraper, opts Options, pc *cache.Cache, sitemap bool, fn func(Result)) error
Crawl performs a BFS crawl from the seed URL, calling fn for each page. The context bounds the entire crawl: cancelling it stops workers promptly and aborts in-flight HTTP/browser requests.
URL rewrite rules on the Scraper apply twice along the crawl path: once in enqueue (so the visited-set and cache key are canonical) and again inside ScrapeConditional (which doesn't know its input was already rewritten). Rules must therefore be idempotent on their own output — a rule whose replacement re-matches its own pattern (e.g. `^/(.*)$ → /v2/$1`) will double-apply when crawled. The documented use cases (host swaps like www.reddit.com → old.reddit.com, path appends like /uk → /uk/rss) are idempotent and safe.
func GenerateCrawlID ¶
func GenerateCrawlID() string
GenerateCrawlID returns a unique crawl identifier like "c_a1b2c3d4".
func StatusPath ¶
StatusPath returns the file path for a crawl status by ID.
func WriteStatus ¶
func WriteStatus(s *CrawlStatus) error
WriteStatus atomically writes a crawl status file.
Types ¶
type CrawlStatus ¶
type CrawlStatus struct {
ID string `json:"id"`
PID int `json:"pid"`
Seed string `json:"seed"`
Status string `json:"status"` // "starting", "running", "completed", "failed", "stopped"
Pages int `json:"pages"`
New int `json:"new"`
Changed int `json:"changed"`
Unchanged int `json:"unchanged"`
Errors int `json:"errors"`
StartedAt time.Time `json:"started_at"`
UpdatedAt time.Time `json:"updated_at"`
Error string `json:"error,omitempty"`
}
CrawlStatus tracks the progress of a background crawl.
func ListStatuses ¶
func ListStatuses() ([]*CrawlStatus, error)
ListStatuses returns all crawl statuses, newest first.
func ReadStatus ¶
func ReadStatus(id string) (*CrawlStatus, error)
ReadStatus reads a crawl status file by ID.
type Options ¶
type Options struct {
Depth int // max BFS depth, default 3
Concurrency int // worker pool size, default 8
Allow []string // path substrings that URLs must contain (any match passes)
Deny []string // regex patterns to reject URLs
}
Options configures the crawl behavior.
type Result ¶
type Result struct {
Page *scrape.Page `json:"page,omitempty"`
Depth int `json:"depth"`
Status string `json:"status"` // "new", "changed", "unchanged"
Source string `json:"source"` // "seed", "link", "sitemap"
Error string `json:"error,omitempty"`
URL string `json:"url"`
}
Result represents a single crawled page.
Source Files