exporter

func WriteZip ¶

func WriteZip(ctx context.Context, w io.Writer, store *sqlite.Store, dataDir string, filters sqlite.ListFilters, byteHardCap int64, coord *storage.Coordinator) error

WriteZip streams a zip archive of every row matched by filters into w.

Two-phase pipeline (v0.1.1):

1. Cursor (one SQLite conn borrowed via StreamMatching) walks rows, builds the per-file groups (jsonl_offset → trace ID) and the slim mediaRef slice. Incremental minTs / maxTs / matched count. No zip bytes are written here — if the SQLite query errors, the caller can still return a clean HTTP status.
2. After the cursor closes (conn released back to the pool), the zip is built. zip.NewWriter is registered with a level-1 Deflate compressor — adopters trade ~5% size for ~3× wall-clock on 100k+ row exports (the workload that motivated streaming in the first place). Each (date, keyhash) group acquires a storage lease in an inner-func scope, writes its entry, and releases on iteration exit; the naive top-of-loop defer would pin every bucket.

dataDir is the storage root that JSONL paths in the SQLite rows are relative to (or absolute; we accept either — the row's JSONLPath is the authoritative reader-side path).

coord (v0.1.1) is the storage coordinator. nil disables lease arbitration: the exporter reads JSONL files without notifying retention — fine for tests / no-retention deployments. Non-nil: groups whose bucket is mid-eviction (ErrFileBeingDeleted) are skipped with a WARN log; the rest of the zip still streams cleanly.

ctx is propagated into StreamMatching (cursor abort on cancel) and honored at every group + media iteration. Cancel mid-export leaves the zip well-formed up to the cancel point — adopters see a truncated body, never a corrupted central directory.

Row cap: callers gate oversize exports via CountMatching BEFORE calling WriteZip. The cursor walks every matching row; pre-flight is the contract.

byteHardCap (v0.1.2) is a post-Phase-1 safety net measured against the sum of source JSONL file sizes for matched groups (one os.Stat per group, not per row). Zero disables the check (`?bytes_all=1` bypass). The cap is independent of the row cap and its `?all=1` bypass — opting out of one does not opt out of the other.

When the cap is exceeded, WriteZip returns *ByteCapExceededError BEFORE constructing the zip writer, so the handler can emit a 413 without any zip bytes hitting the wire. The estimate is an over-count by design (it sums entire source files even when the matched subset is small, and counts gzipped days at compressed size — see the 413 hint / note copy).

The returned error means either the SQLite read failed (zip not yet started), the byte cap was exceeded (also pre-zip), or a disk / zip write failed mid-stream (zip may be partial — zw.Close() is still called via defer to write the central directory of whatever was emitted, but downstream callers should treat the archive as suspect).