Documentation
¶
Overview ¶
Package crawler implements a bounded streaming pipeline for crawling cloud object storage.
The crawler coordinates three stages:
- Lister: Fetches object listings from provider (parallelized by prefix)
- Matcher: Filters objects by glob patterns
- Writer: Emits matched objects as JSONL records
Bounded channels between stages provide backpressure to prevent memory exhaustion on large buckets.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
// Concurrency is the number of parallel list operations.
// Each prefix from Matcher.Prefixes() can be listed concurrently.
// Default: 4
Concurrency int
// ChannelBuffer is the size of bounded channels between pipeline stages.
// Larger buffers reduce blocking but increase memory usage.
// Default: 1000
ChannelBuffer int
// RateLimit is the maximum requests per second to the provider.
// Zero means unlimited (provider handles its own throttling).
// Default: 0
RateLimit float64
// ProgressEvery controls how often progress records are emitted.
// A progress record is written every N matched objects.
// Default: 1000
ProgressEvery int
}
Config configures crawler behavior.
func DefaultConfig ¶
func DefaultConfig() Config
DefaultConfig returns the default crawler configuration.
type Crawler ¶
type Crawler struct {
// contains filtered or unexported fields
}
Crawler executes a crawl job against a cloud storage provider.
Crawler is safe for single use only. Create a new Crawler for each job.
func New ¶
New creates a new crawler.
Parameters:
- p: Provider for listing objects
- m: Matcher for filtering objects by pattern
- w: Writer for JSONL output
- jobID: Correlation ID for this crawl job
- cfg: Crawler configuration (use DefaultConfig() as base)
Use WithFilter() to add metadata filters after creation.
func (*Crawler) Run ¶
Run executes the crawl and returns summary statistics.
Run blocks until the crawl completes, is cancelled via context, or encounters a fatal error. Non-fatal errors (e.g., permission denied on a single object) are written as error records and counted in the summary.
The crawl can be cancelled by cancelling the context. Cancellation is graceful: in-flight operations complete, channels are drained, and a partial summary is returned.
func (*Crawler) WithFilter ¶
func (c *Crawler) WithFilter(f *match.CompositeFilter) *Crawler
WithFilter sets an optional metadata filter for the crawler. Filters are applied after glob pattern matching with AND semantics. Returns the crawler for method chaining.
func (*Crawler) WithPrefixes ¶
WithPrefixes overrides the prefixes to crawl.
When set, the crawler uses these prefixes instead of matcher-derived prefixes.
func (*Crawler) WithRequestBudget ¶
func (c *Crawler) WithRequestBudget(b *RequestBudget) *Crawler
WithRequestBudget shares a run-global request budget with this crawler, replacing its own Config.Concurrency and Config.RateLimit ceilings.
Use it when several crawlers make up one logical run: without a shared budget each crawler enforces the configured ceilings independently, so N crawlers issue N times the requested listing concurrency and N times the requested request rate against the provider. Returns the crawler for method chaining.
This crawler's share of the budget is reserved here, at construction, rather than at Run. A permit is held for a whole paginated listing, so a crawler that waited until Run to reserve could find its peers already holding every permit. Construct every crawler in the run before starting any of them.
Calling this more than once is safe: the crawler's previous share is returned before a new one is taken, so rebinding cannot strand a permit that nothing can reach again.
type RequestBudget ¶
type RequestBudget struct {
// contains filtered or unexported fields
}
RequestBudget bounds provider request work across every Crawler it is injected into, so a run that fans out over several Crawlers stays inside one ceiling instead of multiplying an operator-set one per Crawler.
It owns both ceilings deliberately. Config.Concurrency bounds how many prefix listings run at once and Config.RateLimit bounds how often requests start; carrying only the first would let an injected budget hold concurrency global while the request rate silently scaled with the number of Crawlers. A caller reading "shared request budget" would reasonably expect both, so both live here, behind an opaque surface that exposes neither the permit channel nor the rate limiter.
When a budget is injected, Config.Concurrency and Config.RateLimit on the individual Crawler no longer apply: both ceilings are run-global and come from the budget. A Crawler with no injected budget keeps a private one built from its own Config, which is exactly today's per-Crawler behavior.
A RequestBudget is safe for concurrent use and may be injected into any number of Crawlers.
func NewRequestBudget ¶
func NewRequestBudget(concurrency int, rateLimit float64) *RequestBudget
NewRequestBudget returns a budget carrying a run-global listing-concurrency ceiling and request-rate ceiling.
concurrency below 1 resolves to the package default. rateLimit of zero or less means unlimited, matching Config.RateLimit.
func (*RequestBudget) Concurrency ¶
func (b *RequestBudget) Concurrency() int
Concurrency reports the run-global listing-concurrency ceiling.
type Summary ¶
type Summary struct {
// ObjectsListed is the total number of objects seen from the provider.
ObjectsListed int64
// ObjectsMatched is the number of objects that matched the patterns.
ObjectsMatched int64
// BytesTotal is the cumulative size of matched objects in bytes.
BytesTotal int64
// Duration is the total time spent crawling.
Duration time.Duration
// Errors is the count of non-fatal errors encountered.
Errors int64
// Prefixes lists the prefixes that were crawled.
Prefixes []string
}
Summary contains aggregate statistics from a completed crawl.
Source Files