Documentation
¶
Overview ¶
Package crawler implements a concurrent documentation crawler that fetches, parses, and indexes entities from a source adapter into a store. The crawler uses a fixed worker pool with a shared rate-limited fetcher, a SQLite-backed store, and a source-specific adapter to discover and parse entities.
Package crawler implements the concurrent crawl pipeline: a rate-limited HTTP fetcher and a worker pool that discovers, fetches, parses, and stores WordPress documentation entities.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrFetch404 signals that the target URL returned HTTP 404 Not Found. ErrFetch404 = errors.New("http 404 not found") // ErrFetch5xx signals that the target URL returned an HTTP 5xx server error. ErrFetch5xx = errors.New("http 5xx server error") // ErrFetchNetwork signals a transient network-layer failure (connection // refused, DNS error, timeout, etc.). ErrFetchNetwork = errors.New("network fetch failed") )
Sentinel errors for fetch failure classification. Fetcher implementations should wrap their errors with these sentinels using fmt.Errorf("...: %w", ErrFetch404) so that classifyError can use errors.Is instead of string matching.
Functions ¶
This section is empty.
Types ¶
type Crawler ¶
type Crawler struct {
// contains filtered or unexported fields
}
Crawler orchestrates concurrent fetching and indexing of documentation entities. Each call to Run discovers all entity URLs from the configured source adapter, then processes them in parallel using a worker pool, persisting results to the store and rebuilding the FTS5 search index.
func New ¶
New constructs a Crawler backed by the given fetcher, store, and source adapter. workers controls the number of concurrent fetch goroutines; if workers is <= 0 it defaults to 10.
func (*Crawler) Run ¶
func (c *Crawler) Run(ctx context.Context, opts RunOptions) error
Run executes the full crawl pipeline: register the library, discover all entity URLs, optionally resume or retry a previous session, process entities concurrently with a worker pool, update snippet counts, and rebuild the FTS5 search index. The context controls graceful cancellation; on cancellation the interrupted session state is persisted and the search index is rebuilt from whatever was indexed so far.
type Fetcher ¶
Fetcher is the interface satisfied by both HTTPFetcher and LocalFetcher. Fetch retrieves the content at url, respecting ctx cancellation. Close releases any resources held by the fetcher (e.g. the rate-limit ticker in HTTPFetcher). Close is a no-op on LocalFetcher and always returns nil.
type HTTPFetcher ¶ added in v0.1.3
type HTTPFetcher struct {
// contains filtered or unexported fields
}
HTTPFetcher fetches URLs over HTTP with ticker-based rate limiting and exponential-backoff retry. It is the production fetch implementation used by the crawl worker pool.
Rate limiting is implemented with a time.Ticker whose channel (rateLimiter) is consumed once per Fetch call. This is goroutine-safe without any mutex: the channel itself serialises access to the rate budget.
Call Close when the HTTPFetcher is no longer needed to stop the ticker goroutine.
func NewHTTPFetcher ¶ added in v0.1.3
func NewHTTPFetcher(requestsPerSecond int, maxRetry int, userAgent string) *HTTPFetcher
NewHTTPFetcher creates an HTTPFetcher that issues at most requestsPerSecond requests per second. Each request is retried up to maxRetry times on transient failures (429, 5xx, network error) with exponential backoff. userAgent is sent in the User-Agent request header.
Call Close when the HTTPFetcher is no longer needed to stop the ticker goroutine.
func (*HTTPFetcher) Close ¶ added in v0.1.3
func (f *HTTPFetcher) Close() error
Close stops the rate-limiting ticker, releasing its goroutine. Safe to call multiple times. Implements Fetcher.
func (*HTTPFetcher) Fetch ¶ added in v0.1.3
Fetch waits for a rate-limiter tick, then issues a GET request with exponential-backoff retry on transient errors (429, 5xx, network failures). 404 responses are returned immediately without retrying.
Fetch respects ctx cancellation at the rate-limiter wait and at each backoff sleep. Implements Fetcher.
type LocalFetcher ¶ added in v0.1.3
type LocalFetcher struct {
// contains filtered or unexported fields
}
LocalFetcher reads bytes from the local filesystem instead of the network. Fragment identifiers in URLs (the "#…" suffix) are stripped before the path is read. File contents are cached in memory so that multiple calls with different fragments of the same file incur only one os.ReadFile call.
LocalFetcher has no rate limiting; workers read at maximum speed and the sqlite.mu is the sole bottleneck during local crawls.
Close is a no-op; it always returns nil.
func NewLocalFetcher ¶ added in v0.1.3
func NewLocalFetcher() *LocalFetcher
NewLocalFetcher creates a LocalFetcher that reads from the local filesystem. rootDir is informational only; paths in Fetch calls are used as-is.
Intended for unit tests and local PHP-repository sources.
func (*LocalFetcher) Close ¶ added in v0.1.3
func (f *LocalFetcher) Close() error
Close is a no-op on LocalFetcher. Implements Fetcher.
type RunOptions ¶
type RunOptions struct {
// Resume resumes the most recent interrupted session, skipping URLs that
// were already successfully processed.
Resume bool
// RetryFailed re-queues URLs that failed with transient errors in the most
// recent session. Permanent errors (http_404, parse_error, fk_error) are
// not retried.
RetryFailed bool
}
RunOptions controls resume and retry behaviour for a crawl run.
Source Files