ccrawl

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 17, 2026 License: Apache-2.0 Imports: 37 Imported by: 0

Documentation

Overview

Package ccrawl is a Go library for working with Common Crawl data: the collection list, the CDX URL index, WARC/WAT/WET archive files, the columnar Parquet index, CC-NEWS, and the host/domain ranks. It is the engine behind the ccrawl command line tool but is usable on its own.

Index

Constants

View Source 
const (
	CollInfoURL = "https://index.commoncrawl.org/collinfo.json"
	DataBaseURL = "https://data.commoncrawl.org/"
	CDXBaseURL  = "https://index.commoncrawl.org/"
	S3BaseURL   = "s3://commoncrawl/"

	// ColumnarPrefix is the root of the columnar (Parquet) index.
	ColumnarPrefix = "cc-index/table/cc-main/warc/"

	// UserAgent identifies the client politely to Common Crawl's CDN.
	UserAgent = "ccrawl/1.0 (+https://github.com/tamnd/ccrawl-cli)"
)

Common Crawl endpoints.

View Source 
const (
	DefaultTimeout = 120 * time.Second
	DefaultRetries = 5
	DefaultDelay   = 200 * time.Millisecond
)

Defaults for the client and downloader.

View Source 
const DuckDBPrelude = "INSTALL httpfs; LOAD httpfs; SET enable_progress_bar=false; SET allow_asterisks_in_http_paths=true;"

DuckDBPrelude is prepended to every statement ccrawl sends to the duckdb binary. httpfs reads remote Parquet over HTTPS or S3; the progress bar is noise on a pipe; and allow_asterisks_in_http_paths is required because the columnar index is addressed with a glob (subset=warc/*.parquet) over HTTP, which duckdb refuses by default.

Variables

View Source 
var DefaultBM25Params = BM25Params{K1: 1.2, B: 0.75}

DefaultBM25Params are the standard Okapi BM25 defaults.

View Source 
var DefaultColumnarColumns = []string{
	"url", "url_host_registered_domain", "fetch_status",
	"content_mime_detected", "content_languages",
	"warc_filename", "warc_record_offset", "warc_record_length",
}

DefaultColumnarColumns are the columns selected when none are given.

View Source 
var DefaultCrawlConfig = CrawlConfig{
	UserAgent:   "CCrawl/2.0 (+https://ccrawl.tamnd.com/bot)",
	MaxRedirect: 5,
	Timeout:     120 * time.Second,
}

DefaultCrawlConfig returns sensible defaults for the crawler.

View Source 
var LocationColumns = []string{"url", "warc_filename", "warc_record_offset", "warc_record_length"}

LocationColumns return just the fields needed to range-fetch a record.

View Source 
var PathKinds = []string{"warc", "wat", "wet", "robotstxt", "non200responses", "cc-index", "cc-index-table", "segment"}

PathKinds are the file manifests published per crawl.

Functions

func BM25IDF added in v0.2.0 

func BM25IDF(N, df int) float64

BM25IDF computes the IDF component for a term given N total docs and df doc frequency.

func BM25TF added in v0.2.0 

func BM25TF(tf, dl int, avgDL float64, p BM25Params) float64

BM25TF computes the BM25 TF component.

func CDXNumPages 

func CDXNumPages(ctx context.Context, h *HTTPClient, crawlID string, q CDXQuery) (int, error)

CDXNumPages returns the number of result pages for a query.

func CDXStream 

func CDXStream(ctx context.Context, h *HTTPClient, crawlID string, q CDXQuery, fn func(CDXRecord) error) error

CDXStream runs a query and calls fn for each matching record, paginating through the server's pages and stopping at q.Limit.

func CanonicalURL 

func CanonicalURL(raw string) string

CanonicalURL applies light canonicalization: ensure a scheme, lower-case the host, and drop a fragment. It does not reorder query parameters.

func ColumnarParquetURLs 

func ColumnarParquetURLs(ctx context.Context, h *HTTPClient, cache *Cache, crawlID, subset string, src Source) ([]string, error)

ColumnarParquetURLs resolves the columnar index glob into the explicit list of parquet file URLs for one crawl and subset. Common Crawl's bucket does not allow anonymous listing, so a duckdb run cannot expand the `*.parquet` glob over HTTPS (or anonymous S3) on its own. The crawl publishes the full file list in cc-index-table.paths.gz, so we read that manifest (cached) and turn each entry into a fetchable URL for the chosen source.

func ColumnarSource 

func ColumnarSource(crawlID, subset string, src Source) string

ColumnarSource returns the parquet glob for one crawl's columnar index subset (subset is warc, crawldiagnostics, or robotstxt).

func ComputeEdgeDegrees added in v0.2.0 

func ComputeEdgeDegrees(ctx context.Context, h *HTTPClient, manifestURL string, nodeCount int64) (inDeg, outDeg []int32, err error)

ComputeEdgeDegrees streams all edge part files and computes in-degree and out-degree for every node. It requires knowing nodeCount (the total number of vertices) to allocate the degree arrays. Degrees are returned as two arrays indexed by node ID; the caller should join with vertex IDs.

func ConfigDir 

func ConfigDir() string

ConfigDir returns the directory holding the config file.

func ContentSHA1 added in v0.2.0 

func ContentSHA1(content []byte) string

ContentSHA1 returns the hex SHA-1 of raw content bytes (matches CC's digest field in CDX records).

func CrawlScore added in v0.2.0 

func CrawlScore(harmonicVal, changeRate float64) float64

CrawlScore computes a composite score combining link-graph importance and empirical change rate. Higher = higher crawl priority.

func CrawlTier added in v0.2.0 

func CrawlTier(harmonicPos int64, changeRate float64) int

CrawlTier assigns a 1–5 crawl tier to a host based on its harmonic rank position and estimated change rate. Lower tier = more frequent crawling. 

Tier 1: > 0.8 change rate + top 100 K rank  → 24 h interval
Tier 2: 0.5–0.8 + top 1 M                  → 3 days
Tier 3: 0.2–0.5 + top 5 M                  → 7 days
Tier 4: < 0.2  + top 10 M                  → 30 days
Tier 5: everything else                     → on-demand

func DecodeVarInt added in v0.2.0 

func DecodeVarInt(b []byte) (uint64, int)

DecodeVarInt decodes a VByte-encoded integer from b, returning the value and the number of bytes consumed.

func DiffCDX added in v0.2.0 

func DiffCDX(ctx context.Context, urlsA, urlsB []string, crawlA, crawlB string, fn func(HostDiffEntry) error) error

DiffCDX runs the differential CDX analysis via DuckDB and calls fn for each HostDiffEntry. Requires DuckDB on PATH.

func DiffCDXSQL added in v0.2.0 

func DiffCDXSQL(urlsA, urlsB []string, crawlA, crawlB string) string

DiffCDXSQL returns the DuckDB SQL that computes per-host change rates between two crawl snapshots by joining on url and comparing content digests.

func DocumentID added in v0.2.0 

func DocumentID(canonicalURL string) uint64

DocumentID returns a stable 64-bit document ID for a canonical URL.

func DownloadFiles 

func DownloadFiles(ctx context.Context, h *HTTPClient, src Source, paths []string, localDir string, workers int, flat bool, progress func(DownloadResult)) error

DownloadFiles fetches a list of Common Crawl relative paths into localDir, concurrently and resumably. progress is called once per file when non-nil.

func DuckDBAvailable 

func DuckDBAvailable() bool

DuckDBAvailable reports whether a duckdb binary is on PATH.

func EncodeVarInt added in v0.2.0 

func EncodeVarInt(v uint64) []byte

EncodeVarInt encodes v as a VByte-encoded byte slice.

func ExtractMarkdown 

func ExtractMarkdown(body []byte) (string, error)

ExtractMarkdown converts an HTML document to a compact Markdown approximation. It is intentionally light: headings, paragraphs, list items, links, and emphasis, which covers the bulk of crawled article content. 

func ExtractOutLinks(htmlBytes []byte, baseURL string) []string

ExtractOutLinks extracts absolute URLs from HTML anchor hrefs, resolving relative URLs against baseURL.

func ExtractText 

func ExtractText(body []byte) string

ExtractText returns readable plain text from an HTML document, dropping the contents of script and style elements and collapsing whitespace.

func ExtractTitle 

func ExtractTitle(body []byte) string

ExtractTitle returns the <title> text of an HTML document.

func FetchPaths 

func FetchPaths(ctx context.Context, h *HTTPClient, cache *Cache, crawlID, kind string) ([]string, error)

FetchPaths downloads and decompresses a crawl's path manifest.

func FileURL 

func FileURL(path string, src Source) string

FileURL turns a Common Crawl relative path into a fetchable URL for the given source. HTTPS uses the CloudFront mirror; S3 uses the bucket URI.

func HTMLCanonicalURL added in v0.2.0 

func HTMLCanonicalURL(htmlBytes []byte) string

HTMLCanonicalURL returns the best canonical URL from HTML headers (in priority order): link[rel=canonical], og:url. Returns "" if none found.

func HTTPBody 

func HTTPBody(block []byte) []byte

HTTPBody splits a response block at the header/body boundary and returns the body. It returns the whole block when no boundary is found.

func HTTPHeaders 

func HTTPHeaders(block []byte) []byte

HTTPHeaders returns the header section (status line + headers) of a response block, without the body.

func HTTPSURL 

func HTTPSURL(path string) string

HTTPSURL always returns the HTTPS mirror URL (used for control-plane fetches like manifests and collinfo regardless of the bulk source).

func HarmonicTier added in v0.2.0 

func HarmonicTier(harmonicPos int64, totalHosts int64) int

HarmonicTier returns a 1–10 tier for a host's harmonic rank position (1 = most important, 10 = long tail). Used for multiplicative ranking boosts.

func HostCDXAgg added in v0.2.0 

func HostCDXAgg(ctx context.Context, parquetURLs []string, crawlID, host string, fn func(HostCDXStats) error) error

HostCDXAgg runs the CDX aggregation for one or all hosts via DuckDB and calls fn for each result row. If host is non-empty, only that host is aggregated.

func HostCDXAggSQL added in v0.2.0 

func HostCDXAggSQL(parquetURLs []string, crawlID, host string) string

HostCDXAggSQL returns the DuckDB SQL to aggregate per-host CDX statistics from the columnar Parquet index for a given crawl. If host is non-empty the query filters to that host only; otherwise it aggregates all hosts (full table scan).

func HostCDXSingleSQL added in v0.2.0 

func HostCDXSingleSQL(parquetURLs []string, crawlID, host string) string

HostCDXSingleSQL returns the DuckDB SQL to aggregate CDX stats for one host. This is faster than the full aggregation when only one host is needed.

func HostOf 

func HostOf(raw string) string

HostOf returns the lower-case host of a URL, or "" if it has none.

func InferMatchType 

func InferMatchType(pattern string) (cleanURL, matchType string)

InferMatchType guesses the CDX matchType from a user-supplied URL pattern. "*.example.com" -> domain, "example.com/*" -> prefix, otherwise exact unless the caller already chose host/domain/prefix.

func IterateWARC 

func IterateWARC(r io.Reader, fn func(WARCRecord) error) error

IterateWARC reads a WARC file (a multi-member gzip stream where each member is one record) and calls fn for every record. The parser lives in pkg/warc.

func IterateWAT 

func IterateWAT(r io.Reader, crawlID string, fn func(WATRecord) error) error

IterateWAT reads a WAT file and calls fn for each parsed record. The parser lives in pkg/wat.

func IterateWET 

func IterateWET(r io.Reader, crawlID string, fn func(WETRecord) error) error

IterateWET reads a WET file (WARC conversion records holding plain text) and calls fn for each record. The parser lives in pkg/wet.

func LibraryDir 

func LibraryDir() string

LibraryDir is the root of the structured dataset library that the --library flag downloads into and processes from. It is deliberately separate from the data dir: the data dir (see Config) holds ad-hoc downloads, the cache, and the local DuckDB file, while the library is a curated, browsable corpus you build up over time. CCRAWL_LIBRARY overrides the default of ~/notes/ccrawl.

func LinkBoost added in v0.2.0 

func LinkBoost(bm25Score, harmonicVal, alpha float64) float64

LinkBoost blends a BM25 score with a host's harmonic centrality value. alpha controls the weight of the link signal (spec recommends 0.3).

func NormalizeURL added in v0.2.0 

func NormalizeURL(raw string) string

NormalizeURL applies light URL normalization: lowercase scheme+host, remove default port, strip fragment, strip known tracking parameters.

func ParquetListLiteral 

func ParquetListLiteral(urls []string) string

ParquetListLiteral renders parquet URLs as a duckdb list literal, e.g. ['https://a', 'https://b'], suitable as the argument to read_parquet.

func RankStream added in v0.2.0 

func RankStream(ctx context.Context, h *HTTPClient, url, tld string, fn func(Rank) error) error

RankStream streams every row of a gzipped rank table, calling fn for each entry. If tld is non-empty only hosts under that TLD are emitted. The caller controls early-exit by returning an error from fn.

func ResolveCrawl 

func ResolveCrawl(ctx context.Context, h *HTTPClient, cache *Cache, ref string) (string, error)

ResolveCrawl turns a loose reference into a canonical crawl ID. 

"latest"           -> newest crawl
"CC-MAIN-2024-51"  -> itself
"2024-51"          -> "CC-MAIN-2024-51"
"2024"             -> newest crawl whose ID starts with CC-MAIN-2024

func RunColumnarDuckDB 

func RunColumnarDuckDB(ctx context.Context, sql string, emit func(map[string]any) error) error

RunColumnarDuckDB executes sql with the local duckdb binary, installing the httpfs extension for S3/HTTPS parquet access, and streams JSON rows to emit.

func RunDuckDBJSON 

func RunDuckDBJSON(ctx context.Context, dbPath, sql string, emit func(map[string]any) error) error

RunDuckDBJSON runs sql with the local duckdb binary and streams JSON rows to emit. An empty dbPath runs against an in-memory database; a path opens (and creates) a persistent database file. httpfs is loaded so remote parquet over HTTPS or S3 works either way.

func SURT 

func SURT(raw string) string

SURT converts a URL into a Sort-friendly URI Reordering Transform key, the canonical form Common Crawl uses to sort and group its index. For example "https://www.example.com/a/b?q=1" becomes "com,example,www)/a/b?q=1".

The transform lower-cases the scheme and host, reverses the host labels, drops a leading "www.", strips the default port, and keeps the path and query.

func StreamPaths 

func StreamPaths(ctx context.Context, h *HTTPClient, crawlID, kind string, fn func(string) error) error

StreamPaths streams a crawl's path manifest one path at a time.

func StripTrackingParams added in v0.2.0 

func StripTrackingParams(rawQuery string) string

StripTrackingParams removes known tracking query parameters from a raw URL query string. Returns the cleaned query string (without leading "?").

func TierBoost added in v0.2.0 

func TierBoost(tier int) float64

TierBoost returns the multiplicative ranking boost for a harmonic tier. Tier 1 = 2.0×, tier 10 = 1.0×.

func TierInterval added in v0.2.0 

func TierInterval(tier int) int

TierInterval returns the recommended re-crawl interval in hours for a tier.

func Tokenize added in v0.2.0 

func Tokenize(text string) []string

Tokenize splits text into lowercase, stopword-filtered, min-2-char tokens.

func VertexStream added in v0.2.0 

func VertexStream(ctx context.Context, h *HTTPClient, manifestURL string, fn func(VertexRecord) error) error

VertexStream downloads and streams all vertex part files listed in the manifest at manifestURL, calling fn for each record. Parts are fetched sequentially; use multiple goroutines externally for parallelism.

func WriteWARCResponse added in v0.2.0 

func WriteWARCResponse(w io.Writer, rec NewWARCRecord) error

WriteWARCResponse writes a WARC/1.0 response record to w.

Types

type APIServer added in v0.2.0 

type APIServer struct {
	// contains filtered or unexported fields
}

APIServer is the ccrawl v2 HTTP API server.

func NewAPIServer added in v0.2.0 

func NewAPIServer(cfg ServeConfig, hosts HostStore, search SearchStore) *APIServer

NewAPIServer creates an API server with the given stores. Either store may be nil; endpoints that require the missing store return 503.

func (*APIServer) Addr added in v0.2.0 

func (s *APIServer) Addr() string

Addr returns the address the server is listening on. Valid after ListenAndServe returns its listener address (useful in tests with ":0").

func (*APIServer) ListenAndServe added in v0.2.0 

func (s *APIServer) ListenAndServe(ctx context.Context) error

ListenAndServe starts the HTTP server. It blocks until the context is done or an unrecoverable error occurs.

type BM25Params added in v0.2.0 

type BM25Params struct {
	K1 float64 // term frequency saturation (default 1.2)
	B  float64 // length normalization (default 0.75)
}

BM25Params holds the BM25 hyper-parameters.

type CDXQuery 

type CDXQuery struct {
	URL    string // URL or pattern
	Match  string // exact|prefix|host|domain (empty -> inferred from URL)
	From   string // 14-digit (or loose) lower time bound
	To     string // 14-digit (or loose) upper time bound
	Status string // HTTP status filter (e.g. "200")
	MIME   string // mime-detected filter
	Lang   string // languages filter (ISO-639-3)
	Filter []string
	Limit  int
}

CDXQuery describes a query against the CDX URL index.

type CDXRecord 

type CDXRecord struct {
	CrawlID      string `json:"crawl,omitempty" table:"crawl"`
	URLKey       string `json:"urlkey" table:"-"`
	Timestamp    string `json:"timestamp" table:"timestamp"` // 14-digit YYYYMMDDHHmmss
	URL          string `json:"url" table:"url,url"`
	MIME         string `json:"mime" table:"-"`
	MIMEDetected string `json:"mime-detected" table:"mime"`
	Status       string `json:"status" table:"status"`
	Digest       string `json:"digest" table:"digest"`
	Length       string `json:"length" table:"length"`
	Offset       string `json:"offset" table:"offset"`
	Filename     string `json:"filename" table:"filename"`
	Charset      string `json:"charset,omitempty" table:"-"`
	Languages    string `json:"languages,omitempty" table:"languages"`
	Truncated    string `json:"truncated,omitempty" table:"-"`
	Redirect     string `json:"redirect,omitempty" table:"-"`
}

CDXRecord is one capture from the URL index. Numeric fields stay as strings because that is how the CDX server returns them; helpers convert on demand.

func CDXSearch 

func CDXSearch(ctx context.Context, h *HTTPClient, crawlID string, q CDXQuery) ([]CDXRecord, error)

CDXSearch runs a query and collects matching records (bounded by q.Limit).

func (CDXRecord) Location 

func (r CDXRecord) Location() Location

Location returns the byte span of this capture within its WARC file.

func (CDXRecord) Time 

func (r CDXRecord) Time() time.Time

Time parses the 14-digit timestamp. The zero time is returned on failure.

type Cache 

type Cache struct {
	// contains filtered or unexported fields
}

Cache is a tiny on-disk blob cache keyed by an arbitrary string, with a TTL per entry. It is safe for the simple single-process use the CLI makes of it.

func NewCache 

func NewCache(dir string, enabled bool) *Cache

NewCache returns a cache rooted under dir. If dir is empty or enabled is false, all operations are no-ops (cache miss on every Get).

func (*Cache) Clear 

func (c *Cache) Clear() (int, error)

Clear removes every cached entry. It returns the number of files removed.

func (*Cache) Dir 

func (c *Cache) Dir() string

Dir returns the cache directory.

func (*Cache) Get 

func (c *Cache) Get(key string, ttl time.Duration) ([]byte, bool)

Get returns cached bytes for key if present and younger than ttl.

func (*Cache) Put 

func (c *Cache) Put(key string, data []byte)

Put stores data under key.

type ColumnarQuery 

type ColumnarQuery struct {
	Crawl      string
	Subset     string // warc (default) | crawldiagnostics | robotstxt
	Domain     string // url_host_registered_domain
	Host       string // url_host_name
	TLD        string // url_host_tld
	MIME       string // content_mime_detected
	Lang       string // content_languages (substring match)
	PathPrefix string // url_path prefix
	Status     int    // fetch_status (0 = any)
	Select     []string
	Limit      int
}

ColumnarQuery builds SQL against the columnar (Parquet) index. The zero value selects everything; set fields to add WHERE clauses.

func (ColumnarQuery) SQL 

func (q ColumnarQuery) SQL(src Source) string

SQL renders the query as a runnable DuckDB statement reading parquet over the given source. The same text runs in Athena or Spark after swapping read_parquet for the engine's table reference.

type Config 

type Config struct {
	DataDir   string
	CacheDir  string
	DBPath    string
	Source    Source
	Workers   int
	Timeout   time.Duration
	Delay     time.Duration
	Retries   int
	UserAgent string
	CrawlID   string
}

Config controls library behaviour. The zero value is not usable; call DefaultConfig and adjust.

func DefaultConfig 

func DefaultConfig() Config

DefaultConfig returns a Config rooted at the XDG data/cache directories, with the most recent crawl resolved lazily (CrawlID == "latest").

func (Config) ParquetDir 

func (c Config) ParquetDir() string

ParquetDir is where converted Parquet files land.

func (Config) RawDir 

func (c Config) RawDir() string

RawDir is where downloaded archive files land.

type Crawl 

type Crawl struct {
	ID     string `json:"id" kit:"id" table:"id"`
	Name   string `json:"name" table:"name"`
	CDXAPI string `json:"cdx-api" table:"-"`
	From   string `json:"from,omitempty" table:"from"`
	To     string `json:"to,omitempty" table:"to"`
}

Crawl is one Common Crawl collection as published in collinfo.json.

func ListCrawls 

func ListCrawls(ctx context.Context, h *HTTPClient, cache *Cache) ([]Crawl, error)

ListCrawls fetches and parses collinfo.json. Results are cached when a cache is supplied (pass nil to skip).

type CrawlConfig added in v0.2.0 

type CrawlConfig struct {
	UserAgent   string
	MaxRedirect int
	Timeout     time.Duration
}

CrawlConfig holds configuration for the crawler.

type CrawlResult added in v0.2.0 

type CrawlResult struct {
	URL         string
	FinalURL    string // after redirects
	Status      int
	ContentType string
	Body        []byte
	Digest      string // SHA-1 of body
	FetchedAt   time.Time
	// Links extracted from HTML (relative links resolved to FinalURL)
	Links []string
}

CrawlResult is the output of fetching a single URL.

func CrawlURL added in v0.2.0 

func CrawlURL(ctx context.Context, rawURL string, cfg CrawlConfig) (*CrawlResult, error)

CrawlURL fetches a single URL and returns a CrawlResult. It does not consult the robots.txt cache; the caller must do that before calling CrawlURL.

type DownloadResult 

type DownloadResult struct {
	Path      string
	LocalPath string
	Bytes     int64
	Skipped   bool
	Err       error
}

DownloadResult is the outcome of fetching one file.

type EdgeDegrees added in v0.2.0 

type EdgeDegrees struct {
	NodeID    int64 `json:"node_id"`
	InDegree  int32 `json:"in_degree"`
	OutDegree int32 `json:"out_degree"`
}

EdgeDegrees holds the in-degree and out-degree for one node.

type ForwardDoc added in v0.2.0 

type ForwardDoc struct {
	DocID       uint64  `json:"doc_id" table:"doc_id"`
	URL         string  `json:"url" table:"url"`
	CanonURL    string  `json:"canon_url" table:"canon_url"`
	Host        string  `json:"host" table:"host"`
	Title       string  `json:"title" table:"title"`
	Description string  `json:"description" table:"description"`
	Language    string  `json:"language" table:"language"`
	WordCount   int     `json:"word_count" table:"word_count"`
	LinkScore   float32 `json:"link_score" table:"link_score"`
	Snippet     string  `json:"snippet" table:"snippet"`
}

ForwardDoc is one row in the forward index.

type ForwardIndexWriter added in v0.2.0 

type ForwardIndexWriter struct {
	// contains filtered or unexported fields
}

ForwardIndexWriter appends ForwardDoc rows to a JSONL file.

func NewForwardIndexWriter added in v0.2.0 

func NewForwardIndexWriter(path string) (*ForwardIndexWriter, error)

NewForwardIndexWriter opens (or creates) a JSONL forward index file.

func (*ForwardIndexWriter) Close added in v0.2.0 

func (fw *ForwardIndexWriter) Close() error

Close flushes and closes the writer.

func (*ForwardIndexWriter) Write added in v0.2.0 

func (fw *ForwardIndexWriter) Write(d ForwardDoc) error

Write appends one ForwardDoc to the index.

type Frontier added in v0.2.0 

type Frontier struct {
	// contains filtered or unexported fields
}

Frontier is an in-memory URL frontier with a priority heap and per-host politeness (minimum delay between requests to the same host).

func NewFrontier added in v0.2.0 

func NewFrontier(delay time.Duration) *Frontier

NewFrontier creates a Frontier with the given per-host politeness delay.

func (*Frontier) Add added in v0.2.0 

func (f *Frontier) Add(e FrontierEntry) bool

Add enqueues a URL if it has not been seen before. Returns true if added.

func (*Frontier) Len added in v0.2.0 

func (f *Frontier) Len() int

Len returns the number of queued URLs.

func (*Frontier) Pop added in v0.2.0 

func (f *Frontier) Pop(now int64) (FrontierEntry, bool)

Pop removes and returns the highest-priority URL that is eligible to crawl now (host delay has elapsed). Returns false if no eligible URL exists.

To avoid starvation when the top-priority host is in its politeness window, Pop scans up to scanLimit candidates before giving up. Skipped (ineligible) entries are temporarily held and re-pushed so the heap remains consistent.

type FrontierEntry added in v0.2.0 

type FrontierEntry struct {
	URL      string  // normalized URL
	Host     string  // hostname for politeness grouping
	Priority float32 // harmonic centrality (higher = crawl sooner)
	NextAt   int64   // earliest Unix timestamp to fetch
	Depth    uint8   // BFS depth from seed
	Retries  uint8
}

FrontierEntry is one URL in the crawl frontier.

type HTTPClient 

type HTTPClient struct {
	// contains filtered or unexported fields
}

HTTPClient is a polite, retrying HTTP client for Common Crawl. It rate-limits requests, retries on 429/5xx with linear backoff, and supports byte-range requests for single-record retrieval.

func NewHTTPClient 

func NewHTTPClient(cfg Config) *HTTPClient

NewHTTPClient builds an HTTPClient from cfg.

func (*HTTPClient) FetchBytes 

func (h *HTTPClient) FetchBytes(ctx context.Context, url string) ([]byte, error)

FetchBytes fetches url and returns the whole body.

func (*HTTPClient) Get 

func (h *HTTPClient) Get(ctx context.Context, url string) (*http.Response, error)

Get fetches url with retries.

func (*HTTPClient) GetDownload 

func (h *HTTPClient) GetDownload(ctx context.Context, url string) (*http.Response, error)

GetDownload fetches url with no client timeout (relies on ctx cancellation), for large archive bodies.

func (*HTTPClient) GetRange 

func (h *HTTPClient) GetRange(ctx context.Context, url string, offset, length int64) (*http.Response, error)

GetRange fetches the [offset, offset+length) byte span of url.

type HostCDXStats added in v0.2.0 

type HostCDXStats struct {
	Host             string `json:"url_host_name"`
	RegisteredDomain string `json:"registered_domain"`
	URLCount         int64  `json:"url_count"`
	Status2xx        int64  `json:"status_2xx"`
	Status3xx        int64  `json:"status_3xx"`
	Status4xx        int64  `json:"status_4xx"`
	Status5xx        int64  `json:"status_5xx"`
	TopMIME          string `json:"top_mime"`
	Language         string `json:"language"`
	FirstSeen        string `json:"first_seen"`
	LastSeen         string `json:"last_seen"`
	TotalBytes       int64  `json:"total_bytes"`
}

HostCDXStats is the CDX aggregation result for one host, produced by a DuckDB GROUP BY query over the columnar Parquet index.

type HostDiffEntry added in v0.2.0 

type HostDiffEntry struct {
	Host        string  `json:"host" table:"host"`
	TotalURLs   int64   `json:"total_urls" table:"total_urls"`
	ChangedURLs int64   `json:"changed_urls" table:"changed_urls"`
	ChangeRate  float64 `json:"change_rate" table:"change_rate"`
	Tier        int     `json:"tier" table:"tier"`
}

HostDiffEntry is one row of the per-host change report between two CDX snapshots. It shows how many URLs were seen in both crawls, how many had their content digest change, and the derived change rate.

type HostRecord added in v0.2.0 

type HostRecord struct {
	// Identity
	Host             string `json:"host" kit:"id" table:"host"`
	HostRev          string `json:"host_rev,omitempty" table:"-"`
	TLD              string `json:"tld" table:"tld"`
	RegisteredDomain string `json:"registered_domain,omitempty" table:"registered_domain"`

	// Rank signals (from rank table)
	HarmonicPos int64   `json:"harmonic_pos" table:"harmonic_pos"`
	HarmonicVal float64 `json:"harmonic_val" table:"harmonic_val"`
	PageRankPos int64   `json:"pagerank_pos" table:"pagerank_pos"`
	PageRankVal float64 `json:"pagerank_val" table:"pagerank_val"`

	// Graph topology (from edge files)
	InDegree  int64 `json:"in_degree" table:"in_degree"`
	OutDegree int64 `json:"out_degree" table:"out_degree"`

	// CDX statistics (from columnar Parquet index)
	URLCount   int64  `json:"url_count" table:"url_count"`
	Status2xx  int64  `json:"status_2xx" table:"status_2xx"`
	Status3xx  int64  `json:"status_3xx" table:"status_3xx"`
	Status4xx  int64  `json:"status_4xx" table:"status_4xx"`
	Status5xx  int64  `json:"status_5xx" table:"status_5xx"`
	TopMIME    string `json:"top_mime,omitempty" table:"top_mime"`
	Language   string `json:"language,omitempty" table:"language"`
	FirstSeen  string `json:"first_seen,omitempty" table:"first_seen"`
	LastSeen   string `json:"last_seen,omitempty" table:"last_seen"`
	TotalBytes int64  `json:"total_bytes" table:"total_bytes"`
}

HostRecord is the fully enriched profile for one host, combining signals from the web-graph rank table, graph topology (in/out-degree), and the CDX URL index.

func HostFromRank added in v0.2.0 

func HostFromRank(r Rank) HostRecord

HostFromRank builds a minimal HostRecord from a Rank entry.

func HostLookupRank added in v0.2.0 

func HostLookupRank(ctx context.Context, h *HTTPClient, rankURL, host string) (HostRecord, error)

HostLookupRank returns the rank-only HostRecord for a host by streaming the rank table. This is O(rank_table_size) — for a single-host lookup it is best to use HostLookupCDX for CDX stats and join separately.

type HostSchedule added in v0.2.0 

type HostSchedule struct {
	Host        string  `json:"host" table:"host"`
	HarmonicPos int64   `json:"harmonic_pos" table:"harmonic_pos"`
	HarmonicVal float64 `json:"harmonic_val" table:"harmonic_val"`
	ChangeRate  float64 `json:"change_rate" table:"change_rate"`
	Tier        int     `json:"tier" table:"tier"`
	IntervalH   int     `json:"interval_h" table:"interval_h"`
	Score       float64 `json:"score" table:"score"`
}

HostSchedule holds the crawl schedule for one host: its tier and derived priority score.

func HostScheduleFrom added in v0.2.0 

func HostScheduleFrom(r Rank, changeRate float64) HostSchedule

HostScheduleFrom builds a HostSchedule from a Rank entry and a change rate.

type HostStore added in v0.2.0 

type HostStore interface {
	GetHost(ctx context.Context, host string) (*HostRecord, error)
	TopHosts(ctx context.Context, n int, tld string) ([]HostRecord, error)
}

HostStore is the interface for host metadata lookups used by the API.

type IndexReader added in v0.2.0 

type IndexReader struct {
	N     int
	AvgDL float64
	// contains filtered or unexported fields
}

IndexReader reads a flushed inverted index from disk.

func OpenIndex added in v0.2.0 

func OpenIndex(dir string) (*IndexReader, error)

OpenIndex opens a flushed inverted index directory.

func (*IndexReader) Close added in v0.2.0 

func (r *IndexReader) Close() error

Close releases the index file handles.

func (*IndexReader) Lookup added in v0.2.0 

func (r *IndexReader) Lookup(term string) ([]PostingEntry, float64, bool)

Lookup returns the posting list for a term.

func (*IndexReader) Search added in v0.2.0 

func (r *IndexReader) Search(tokens []string, k int) []ScoredDoc

Search returns the top-k documents for query tokens using BM25.

type IndexSearchStore added in v0.2.0 

type IndexSearchStore struct {
	// contains filtered or unexported fields
}

IndexSearchStore wraps an IndexReader to implement the SearchStore interface. The forward index is looked up from a map of docID → ForwardDoc.

func NewIndexSearchStore added in v0.2.0 

func NewIndexSearchStore(idx *IndexReader, forward map[uint64]ForwardDoc) *IndexSearchStore

NewIndexSearchStore wraps an IndexReader.

func (*IndexSearchStore) Search added in v0.2.0 

func (s *IndexSearchStore) Search(_ context.Context, query string, k int) ([]SearchResult, error)

Search tokenizes the query and returns top-k search results with snippets.

type InvertedIndexBuilder added in v0.2.0 

type InvertedIndexBuilder struct {
	N int // total docs indexed

	TermCount int // unique term count (set by Flush)
	// contains filtered or unexported fields
}

InvertedIndexBuilder accumulates (term → postings) in memory and writes the index to disk when Flush() is called.

func NewInvertedIndexBuilder added in v0.2.0 

func NewInvertedIndexBuilder(dir string) (*InvertedIndexBuilder, error)

NewInvertedIndexBuilder creates a builder that writes to dir.

func (*InvertedIndexBuilder) Add added in v0.2.0 

func (b *InvertedIndexBuilder) Add(docID uint64, tokens []string)

Add indexes a single document. DL (the document token length) is stored in each PostingEntry so BM25 scoring can use per-document length normalization.

func (*InvertedIndexBuilder) Flush added in v0.2.0 

func (b *InvertedIndexBuilder) Flush() error

Flush writes the inverted index to disk in shard_NNN/ directories.

type Library 

type Library struct {
	Root  string
	Crawl string
}

Library is a structured corpus of Common Crawl archive files for one crawl, rooted at Root. The layout is predictable so a directory listing tells you exactly what you have: 

<root>/<crawl>/<kind>/<file>.gz               raw downloaded archives
<root>/<crawl>/<format>/<kind>/<file>.<ext>   processed output (parquet|jsonl)

Files are stored flat under each kind by their base name. A Common Crawl file name already encodes its segment and timestamp and is unique within a crawl, so the base name alone is a safe, stable key with no risk of collision.

func NewLibrary 

func NewLibrary(root, crawl string) Library

NewLibrary returns a Library rooted at root (or LibraryDir() when root is empty) for the given crawl ID.

func (Library) CrawlDir 

func (l Library) CrawlDir() string

CrawlDir is the per-crawl root, the parent of every kind and format directory.

func (Library) ProcessedDir 

func (l Library) ProcessedDir(format, kind string) string

ProcessedDir is where processed output of a kind lives, grouped by format (parquet or jsonl) so the same archives can be materialised more than one way side by side.

func (Library) RawDir 

func (l Library) RawDir(kind string) string

RawDir is where downloaded archives of a kind live.

func (Library) RawPath 

func (l Library) RawPath(kind, ccPath string) string

RawPath is the local path a given Common Crawl path maps to under the library.

type Location 

type Location struct {
	Filename string `json:"filename" table:"filename"`
	Offset   int64  `json:"offset" table:"offset"`
	Length   int64  `json:"length" table:"length"`
	URL      string `json:"url,omitempty" table:"url,url"`
}

Location is the WARC file plus byte span needed to range-fetch this capture.

type MemHostStore added in v0.2.0 

type MemHostStore struct {
	// contains filtered or unexported fields
}

MemHostStore is a simple in-memory HostStore backed by a slice of HostRecord.

func NewMemHostStore added in v0.2.0 

func NewMemHostStore(recs []HostRecord) *MemHostStore

NewMemHostStore builds a MemHostStore from a slice of HostRecord.

func (*MemHostStore) GetHost added in v0.2.0 

func (s *MemHostStore) GetHost(_ context.Context, host string) (*HostRecord, error)

GetHost returns the HostRecord for a given hostname.

func (*MemHostStore) TopHosts added in v0.2.0 

func (s *MemHostStore) TopHosts(_ context.Context, n int, tld string) ([]HostRecord, error)

TopHosts returns up to n hosts filtered optionally by TLD.

type NewWARCRecord added in v0.2.0 

type NewWARCRecord struct {
	TargetURI string
	Date      string
	RecordID  string
	Block     []byte // raw HTTP response bytes
}

NewWARCRecord holds the fields to write a fresh WARC response record.

type NewsFile 

type NewsFile struct {
	Path string `json:"path" table:"path"`
	Year int    `json:"year" table:"year"`
	Mon  int    `json:"month" table:"month"`
}

NewsFile describes one CC-NEWS WARC file.

func ListNewsFiles 

func ListNewsFiles(ctx context.Context, h *HTTPClient, year, month int) ([]NewsFile, error)

ListNewsFiles returns the CC-NEWS WARC files for a year and month. Pass month 0 to list every month of a year; pass year 0 to list everything found via the index page.

type ParquetWriter 

type ParquetWriter[T any] struct {
	// contains filtered or unexported fields
}

ParquetWriter writes rows of type T to a zstd-compressed Parquet file.

func NewParquetWriter 

func NewParquetWriter[T any](path string) (*ParquetWriter[T], error)

NewParquetWriter creates a Parquet writer for path.

func (*ParquetWriter[T]) Close 

func (p *ParquetWriter[T]) Close() error

Close flushes and closes the file.

func (*ParquetWriter[T]) Rows 

func (p *ParquetWriter[T]) Rows() int64

Rows returns the number of rows written.

func (*ParquetWriter[T]) Write 

func (p *ParquetWriter[T]) Write(row T) error

Write appends one row.

type PostingEntry added in v0.2.0 

type PostingEntry struct {
	DocID uint64
	TF    uint32
	DL    uint32 // document token length at index time
}

PostingEntry is one (doc_id, term_freq, doc_len) triple in a posting list. DL is the document token length, stored per-posting so BM25 length normalization uses the actual document length rather than the corpus average.

type QualityResult added in v0.2.0 

type QualityResult struct {
	WordCount      int
	TitleLength    int
	HasMainContent bool
	SpamScore      float64 // 0–1
	IsParked       bool
	IsShortContent bool // word_count < 50
}

QualityResult holds per-document quality signals computed from a TextResult.

func QualitySignals added in v0.2.0 

func QualitySignals(tr TextResult) QualityResult

QualitySignals computes per-document quality signals from a TextResult.

type Rank 

type Rank struct {
	Key         string  `json:"key" kit:"id" table:"key"` // host or domain (forward form)
	HarmonicPos int64   `json:"harmonic_pos" table:"harmonic_pos"`
	HarmonicVal float64 `json:"harmonic_val" table:"harmonic_val"`
	PageRankPos int64   `json:"pagerank_pos" table:"pagerank_pos"`
	PageRankVal float64 `json:"pagerank_val" table:"pagerank_val"`
}

Rank is a host/domain entry from the web-graph rank tables.

func RankLookup 

func RankLookup(ctx context.Context, h *HTTPClient, url, hostOrDomain string) (Rank, error)

RankLookup streams a gzipped rank table from url and returns the entry whose reversed key matches the given host or domain, or a not-found error.

func RankTop 

func RankTop(ctx context.Context, h *HTTPClient, url, tld string, n int) ([]Rank, error)

RankTop streams a rank table and returns the first n rows (the table is sorted by harmonic centrality, most central first).

type RobotsCache added in v0.2.0 

type RobotsCache struct {
	// contains filtered or unexported fields
}

RobotsCache caches parsed robots.txt per host with a TTL.

func NewRobotsCache added in v0.2.0 

func NewRobotsCache(ttl time.Duration, userAgent string) *RobotsCache

NewRobotsCache creates a cache with the given TTL and user-agent string.

func (*RobotsCache) Get added in v0.2.0 

func (rc *RobotsCache) Get(host string) *RobotsEntry

Get returns the cached robots entry for host, or nil if not cached or expired.

func (*RobotsCache) Put added in v0.2.0 

func (rc *RobotsCache) Put(host string, e *RobotsEntry)

Put stores a robots entry for host.

type RobotsEntry added in v0.2.0 

type RobotsEntry struct {
	Rules      []RobotsRule
	CrawlDelay time.Duration
	ExpiresAt  int64 // Unix timestamp
}

RobotsEntry is a cached robots.txt for one host.

func FetchRobots added in v0.2.0 

func FetchRobots(ctx context.Context, h *HTTPClient, host, scheme string) *RobotsEntry

FetchRobots fetches and parses robots.txt for the given host. On timeout or 5xx, returns a permissive entry (allow all). On 404, returns allow all permanently. The caller should Put the result into the cache.

func (*RobotsEntry) IsAllowed added in v0.2.0 

func (e *RobotsEntry) IsAllowed(path string) bool

IsAllowed reports whether the given path is allowed. The most specific (longest) matching rule wins, following the standard robots.txt precedence.

type RobotsRule added in v0.2.0 

type RobotsRule struct {
	Allow   bool
	Pattern string
}

RobotsRule is one allow/disallow rule from robots.txt.

type ScoredDoc added in v0.2.0 

type ScoredDoc struct {
	DocID uint64
	Score float64
}

ScoredDoc is a document with its BM25 score.

type SearchResult added in v0.2.0 

type SearchResult struct {
	DocID    uint64  `json:"doc_id"`
	URL      string  `json:"url"`
	Host     string  `json:"host"`
	Title    string  `json:"title"`
	Snippet  string  `json:"snippet"`
	Score    float64 `json:"score"`
	Language string  `json:"language,omitempty"`
}

SearchResult is one search hit returned by the API.

type SearchStore added in v0.2.0 

type SearchStore interface {
	Search(ctx context.Context, query string, k int) ([]SearchResult, error)
}

SearchStore is the interface for full-text search used by the API.

type ServeConfig added in v0.2.0 

type ServeConfig struct {
	Addr     string // e.g. ":8080"
	DBPath   string // path to DuckDB/SQLite host database (optional)
	IndexDir string // path to the inverted index directory (optional)
}

ServeConfig holds configuration for the HTTP API server.

type Source 

type Source string

Source selects the transport used for bulk data files. 

const (
	SourceHTTPS Source = "https"
	SourceS3    Source = "s3"
)

type TextResult added in v0.2.0 

type TextResult struct {
	Title       string
	Description string // <meta name="description">
	CanonURL    string // canonical URL from meta or link[rel=canonical]
	Body        string // extracted clean body text
	WordCount   int
	Language    string // BCP-47 code inferred from lang attribute
}

TextResult holds the output of HTML-to-text extraction.

func ExtractContent added in v0.2.0 

func ExtractContent(htmlBytes []byte) TextResult

ExtractContent parses HTML bytes and returns a TextResult with clean text, title, description, canonical URL, and inferred language.

type VertexRecord added in v0.2.0 

type VertexRecord struct {
	ID      int64  `json:"id" kit:"id" table:"id"`
	HostRev string `json:"host_rev" table:"host_rev"`
	Host    string `json:"host" table:"host"`
}

VertexRecord is one row from the host vertices files: a numeric graph ID and the host name in reversed form (com.example.www).

type WARCHeader 

type WARCHeader = warc.Header

WARCHeader holds parsed WARC record headers.

type WARCParquetRow 

type WARCParquetRow struct {
	RecordID      string    `parquet:"record_id,dict"`
	CrawlID       string    `parquet:"crawl_id,dict"`
	WARCType      string    `parquet:"warc_type,dict"`
	TargetURI     string    `parquet:"target_uri"`
	Date          time.Time `parquet:"date,timestamp(microsecond)"`
	IPAddress     string    `parquet:"ip_address,dict"`
	PayloadDigest string    `parquet:"payload_digest"`
	ContentType   string    `parquet:"content_type,dict"`
	ContentLength int64     `parquet:"content_length"`
	Truncated     string    `parquet:"truncated,dict"`
	HTTPStatus    int32     `parquet:"http_status"`
	HTTPMIME      string    `parquet:"http_mime,dict"`
	WARCFilename  string    `parquet:"warc_filename,dict"`
	WARCOffset    int64     `parquet:"warc_offset"`
	WARCLength    int64     `parquet:"warc_length"`
	Title         string    `parquet:"title"`
	Language      string    `parquet:"language,dict"`
	Markdown      string    `parquet:"markdown"`
	Text          string    `parquet:"text"`
}

WARCParquetRow is the columnar schema for parsed WARC record metadata. When a response body is converted, the content fields are populated too.

type WARCRecord 

type WARCRecord = warc.Record

WARCRecord is a parsed WARC record: its header and the raw block bytes. For a response record the block is the full HTTP message (status line, headers, body).

func FetchWARCRecord 

func FetchWARCRecord(ctx context.Context, h *HTTPClient, filename string, offset, length int64) (WARCRecord, error)

FetchWARCRecord retrieves a single WARC record from the given file using a byte-range request. This is how a capture's content is pulled without downloading the whole multi-gigabyte WARC. 

type WATLink = wat.Link

WATLink is a hyperlink extracted from page HTML. 

func ExtractLinks(base string, body []byte) []WATLink

ExtractLinks returns the outbound hyperlinks of an HTML document, resolved against base when possible.

type WATMeta 

type WATMeta = wat.Meta

WATMeta is a <meta> tag extracted from page HTML.

type WATParquetRow 

type WATParquetRow struct {
	RecordID    string    `parquet:"record_id,dict"`
	CrawlID     string    `parquet:"crawl_id,dict"`
	URL         string    `parquet:"url"`
	Date        time.Time `parquet:"date,timestamp(microsecond)"`
	HTTPStatus  int32     `parquet:"http_status"`
	ContentType string    `parquet:"content_type,dict"`
	Title       string    `parquet:"title"`
	LinksCount  int32     `parquet:"links_count"`
	Links       string    `parquet:"links"` // JSON
	Metas       string    `parquet:"metas"` // JSON
}

WATParquetRow is the columnar schema for WAT link and metadata records.

type WATRecord 

type WATRecord = wat.Record

WATRecord is metadata extracted by Common Crawl from a single page.

type WETParquetRow 

type WETParquetRow struct {
	RecordID        string    `parquet:"record_id,dict"`
	CrawlID         string    `parquet:"crawl_id,dict"`
	URL             string    `parquet:"url"`
	Date            time.Time `parquet:"date,timestamp(microsecond)"`
	ContentLanguage string    `parquet:"content_language,dict"`
	TextLength      int32     `parquet:"text_length"`
	Text            string    `parquet:"text"`
}

WETParquetRow is the columnar schema for WET plain-text records.

type WETRecord 

type WETRecord = wet.Record

WETRecord is extracted plain text for one page.

type WebGraph added in v0.2.0 

type WebGraph struct {
	ID        string // e.g. cc-main-2026-mar-apr-may
	BaseURL   string // https://data.commoncrawl.org/projects/hyperlinkgraph/{ID}/
	HostNodes int64
	HostArcs  int64
	Published string
}

WebGraph describes one Common Crawl web-graph release.

func LatestWebGraph added in v0.2.0 

func LatestWebGraph(ctx context.Context, h *HTTPClient, cache *Cache) (WebGraph, error)

LatestWebGraph fetches the CC web-graphs index page and returns the most recent host-level graph release. Results are cached for 24 hours.

func (WebGraph) HostEdgesManifestURL added in v0.2.0 

func (g WebGraph) HostEdgesManifestURL() string

HostEdgesManifestURL is the manifest listing edge part files.

func (WebGraph) HostRankURL added in v0.2.0 

func (g WebGraph) HostRankURL() string

HostRankURL is the single gzipped rank table for hosts.

func (WebGraph) HostVerticesManifestURL added in v0.2.0 

func (g WebGraph) HostVerticesManifestURL() string

HostVerticesManifestURL is the manifest listing vertex part files.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.