Documentation
¶
Overview ¶
Package seed defines what ami crawls: a stream of URLs, each optionally carrying a content digest and a free-form metadata map. A seed is just a list of URLs to ami; where the list comes from is the job of a SeedSource, and the engine never assumes any particular origin.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type JSONL ¶
type JSONL struct {
Path string
}
JSONL reads newline-delimited JSON objects. Each object must carry a "url" field; "digest" is optional, and any remaining string-valued fields become Meta. Non-string fields are ignored so the format stays forgiving.
type Lines ¶
type Lines struct {
Path string
}
Lines reads one URL per line from a file (or stdin when path is "-"). Blank lines and lines beginning with # are skipped. It is the simplest source and the default.
type Parquet ¶
type Parquet struct {
Path string
}
Parquet reads seeds from a Parquet file with a "url" column. An optional "digest" column carries a prior body's SHA-1 for change detection; every other string-typed leaf column is carried into Meta under its column name. The schema is read dynamically so any producer can hand ami a Parquet file without a shared Go type.
type Seed ¶
type Seed struct {
URL string
Digest string
// ETag and ModTime are response validators from a prior capture. When set,
// ami issues a conditional request (If-None-Match / If-Modified-Since) so an
// unchanged page returns a bodiless 304. They make a prior run's capture
// index usable directly as a recrawl seed.
ETag string
ModTime string
Meta map[string]string
}
Seed is one unit of work: a URL plus optional hints.
Digest, when present, is the content digest of a previously fetched version of this URL (any opaque string the producer chose, commonly a sha1). ami uses it to decide whether a re-fetch changed, by comparing it against the sha1 of the freshly fetched body; it never parses it otherwise.
Meta is arbitrary key/value context the producer wants carried through to the capture index unchanged (for example a source partition, a language guess, or a fetch timestamp). ami stores it verbatim and attaches no meaning to it.
type Sitemap ¶
Sitemap reads <loc> URLs from an XML sitemap or sitemap index at a URL. A sitemap index is followed one level deep so a single entry point expands into every child sitemap's URLs. Gzip-encoded sitemaps (.xml.gz) are decoded transparently.
type Source ¶
type Source interface {
// Name identifies the adapter for logs and the run manifest.
Name() string
// Iterate calls yield once per seed until the input is exhausted, ctx is
// cancelled, or yield returns an error.
Iterate(ctx context.Context, yield func(Seed) error) error
}
Source is a one-shot stream of seeds. Implementations push each seed to yield and return the first error they hit; returning a non-nil error from yield means the consumer is done and the source should stop and return that error.
A Source must be cheap to start and must stream rather than buffer: seed files can hold tens of millions of rows, far more than fits comfortably in memory.
Source Files