pack

type Capture ¶

type Capture struct {
	URL         string `parquet:"url"`
	Host        string `parquet:"host"`
	Status      int32  `parquet:"status"`
	FetchedAt   int64  `parquet:"fetched_at"` // unix millis
	ContentType string `parquet:"content_type"`
	BodyLength  int64  `parquet:"body_length"`
	Digest      string `parquet:"digest"`
	Unchanged   bool   `parquet:"unchanged"`

	// Response validators, so this output doubles as a recrawl seed: a later run
	// reads etag/last_modified back and issues conditional requests.
	ETag         string `parquet:"etag"`
	LastModified string `parquet:"last_modified"`

	// Pointer into the WARC (warc format only; zero in parquet format).
	WARCFile   string `parquet:"warc_file"`
	WARCOffset int64  `parquet:"warc_offset"`
	WARCLength int64  `parquet:"warc_length"`

	// Error text for failed fetches (empty on success).
	Error string `parquet:"error"`

	// MetaJSON carries the seed's Meta map verbatim as a JSON object string, so
	// arbitrary producer context survives without a fixed schema.
	MetaJSON string `parquet:"meta_json"`

	// The captured exchange, stored inline in the parquet body-store format and
	// left empty in warc format (where the bytes live in the WARC instead). The
	// header fields hold the reconstructed HTTP head text, so a reader can rebuild
	// the full request/response without a WARC.
	RespHeaders string `parquet:"resp_headers"`
	ReqHeaders  string `parquet:"req_headers"`
	Body        []byte `parquet:"body"`
}

Capture is one row of the columnar output: enough to find and describe a stored response, the response validators so the file doubles as a recrawl seed, and (in the parquet body-store format) the captured exchange itself.

type IndexWriter ¶

type IndexWriter struct {
	// contains filtered or unexported fields
}

IndexWriter writes Capture rows to zstd-compressed Parquet, rotating to a new file once a target amount of uncompressed payload has accumulated. Rows are buffered and handed to the encoder in batches, so the hot path is one append rather than a one-element slice allocation and an encoder call per capture. The underlying file is wrapped in a large buffered writer, and the row-group size is bounded so a multi-million-row run keeps a steady memory footprint instead of buffering the whole file before the first flush.

Each rotated file is finalized with its own footer, so it is independently readable, can be offloaded and deleted mid-run, and a crash loses only the open file. The body and header columns compress columnar with zstd, which packs thousands of similar pages together far more tightly than per-record gzip; the crawl is network-bound, so the heavier compression is effectively free.

func NewIndexWriter(dir, name string, batchRows int, targetSize int64) (*IndexWriter, error)

func (iw *IndexWriter) Close() error

func (iw *IndexWriter) Write(c Capture) error

type Location ¶

type Location struct {
	File   string
	Offset int64
	Length int64
}

Location is where a record landed: which file and the byte offset of its gzip member, so the index can point straight at it.

type WARCWriter ¶

type WARCWriter struct {
	// contains filtered or unexported fields
}

WARCWriter appends WARC records and rotates to a new file once one passes the target size. Each record is its own gzip member (a "WARC record gzip"), so a reader can seek to any record's offset and inflate just that member.

func NewWARCWriter(dir, prefix string, targetSize int64) (*WARCWriter, error)

func (w *WARCWriter) Close() error

func (w *WARCWriter) WriteResponse(targetURI string, status int, reqHeader, respHeader http.Header, body []byte, revisit bool, refDigest string) (Location, error)