Documentation
¶
Overview ¶
Package pack writes a run's results to durable output: WARC/1.1 files holding the raw HTTP exchanges, and a columnar Parquet index describing every capture for fast analytics without reopening the WARCs.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func MetaToJSON ¶
MetaToJSON encodes a meta map to a compact JSON object string, returning "" for an empty map so the column stays cheap.
func RequestHead ¶
RequestHead reconstructs the HTTP request head ami sent, as text.
func ResponseHead ¶
ResponseHead reconstructs the HTTP response head (status line plus headers) as text, ending with the blank line that separates head from body. The parquet body store keeps this alongside the body so a reader can rebuild the full response without a WARC.
Types ¶
type Capture ¶
type Capture struct {
URL string `parquet:"url"`
Host string `parquet:"host"`
Status int32 `parquet:"status"`
FetchedAt int64 `parquet:"fetched_at"` // unix millis
ContentType string `parquet:"content_type"`
BodyLength int64 `parquet:"body_length"`
Digest string `parquet:"digest"`
Unchanged bool `parquet:"unchanged"`
// Response validators, so this output doubles as a recrawl seed: a later run
// reads etag/last_modified back and issues conditional requests.
ETag string `parquet:"etag"`
LastModified string `parquet:"last_modified"`
// Pointer into the WARC (warc format only; zero in parquet format).
WARCFile string `parquet:"warc_file"`
WARCOffset int64 `parquet:"warc_offset"`
WARCLength int64 `parquet:"warc_length"`
// Error text for failed fetches (empty on success).
Error string `parquet:"error"`
// MetaJSON carries the seed's Meta map verbatim as a JSON object string, so
// arbitrary producer context survives without a fixed schema.
MetaJSON string `parquet:"meta_json"`
// Markdown is the body rendered to Markdown, populated only when the crawl ran
// with --markdown and the response was HTML that yielded an article. It is
// empty otherwise. MarkdownLength is its byte length, so a reader can size the
// text column without decompressing it.
Markdown string `parquet:"markdown"`
MarkdownLength int64 `parquet:"markdown_length"`
// Timing and network metadata captured during the fetch.
TTFBMS int64 `parquet:"ttfb_ms"` // time to first byte in milliseconds
FetchDurMS int64 `parquet:"fetch_duration_ms"` // total fetch wall-clock in milliseconds
FinalURL string `parquet:"final_url"` // URL after following redirects; empty if same as URL
IPAddress string `parquet:"ip_address"` // IP that served the response
// The captured exchange, stored inline in the parquet body-store format and
// left empty in warc format (where the bytes live in the WARC instead). The
// header fields hold the reconstructed HTTP head text, so a reader can rebuild
// the full request/response without a WARC.
RespHeaders string `parquet:"resp_headers"`
ReqHeaders string `parquet:"req_headers"`
Body []byte `parquet:"body"`
}
Capture is one row of the columnar output: enough to find and describe a stored response, the response validators so the file doubles as a recrawl seed, and (in the parquet body-store format) the captured exchange itself.
type IndexWriter ¶
type IndexWriter struct {
// contains filtered or unexported fields
}
IndexWriter writes Capture rows to zstd-compressed Parquet, rotating to a new file once a target amount of uncompressed payload has accumulated. Rows are buffered and handed to the encoder in batches, so the hot path is one append rather than a one-element slice allocation and an encoder call per capture. The underlying file is wrapped in a large buffered writer, and the row-group size is bounded so a multi-million-row run keeps a steady memory footprint instead of buffering the whole file before the first flush.
Each rotated file is finalized with its own footer, so it is independently readable, can be offloaded and deleted mid-run, and a crash loses only the open file. The body and header columns compress columnar with zstd, which packs thousands of similar pages together far more tightly than per-record gzip; the crawl is network-bound, so the heavier compression is effectively free.
func NewIndexWriter ¶
func NewIndexWriter(dir, name string, batchRows int, targetSize int64) (*IndexWriter, error)
NewIndexWriter creates a rotating capture writer under dir. name is the base file name; its .parquet extension is replaced by a -NNNNN.parquet sequence. batchRows is how many rows are buffered before a write to the encoder; a non-positive value falls back to a default. targetSize rotates the file once that many uncompressed payload bytes have been written, so a long run produces a series of bounded files; zero disables rotation (a single file).
func (*IndexWriter) Close ¶
func (iw *IndexWriter) Close() error
Close flushes the pending batch and finalizes the open file.
func (*IndexWriter) Write ¶
func (iw *IndexWriter) Write(c Capture) error
Write appends one capture row, flushing the batch to the encoder once it is full and rotating to a new file once the payload target is reached.
type Location ¶
Location is where a record landed: which file and the byte offset of its gzip member, so the index can point straight at it.
type WARCWriter ¶
type WARCWriter struct {
// contains filtered or unexported fields
}
WARCWriter appends WARC records and rotates to a new file once one passes the target size. Each record is its own gzip member (a "WARC record gzip"), so a reader can seek to any record's offset and inflate just that member.
func NewWARCWriter ¶
func NewWARCWriter(dir, prefix string, targetSize int64) (*WARCWriter, error)
NewWARCWriter creates a writer that rotates files of about targetSize bytes under dir, naming them prefix-00000.warc.gz and so on.
func (*WARCWriter) WriteResponse ¶
func (w *WARCWriter) WriteResponse(targetURI string, status int, reqHeader, respHeader http.Header, body []byte, revisit bool, refDigest string) (Location, error)
WriteResponse writes a paired request and response (or a revisit) record for one capture and returns the response record's location.
Source Files