Documentation
¶
Overview ¶
Package exporter builds the /api/export zip. It ships JSONL bytes straight from disk into a streaming zip with no transformation; SQLite only selects which JSONL lines to include.
The zip layout follows docs/specs/phase-i-export-contract.md and docs/specs/phase-k-media-contract.md:
data/<YYYY-MM-DD>/<keyhash8>.jsonl (all rows on disk matched) data/<YYYY-MM-DD>/<keyhash8>.partial.jsonl (subset matched) media/<trace_id>/<idx>.<ext> (extracted attachments, if any) agent/CLAUDE.md agent/jq-cheatsheet.md README.md
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func WriteZip ¶
func WriteZip(ctx context.Context, w io.Writer, store *sqlite.Store, dataDir string, filters sqlite.ListFilters, coord *storage.Coordinator) error
WriteZip streams a zip archive of every row matched by filters into w.
Two-phase pipeline (v0.1.1):
- Cursor (one SQLite conn borrowed via StreamMatching) walks rows, builds the per-file groups (jsonl_offset → trace ID) and the slim mediaRef slice. Incremental minTs / maxTs / matched count. No zip bytes are written here — if the SQLite query errors, the caller can still return a clean HTTP status.
- After the cursor closes (conn released back to the pool), the zip is built. zip.NewWriter is registered with a level-1 Deflate compressor — adopters trade ~5% size for ~3× wall-clock on 100k+ row exports (the workload that motivated streaming in the first place). Each (date, keyhash) group acquires a storage lease in an inner-func scope, writes its entry, and releases on iteration exit; the naive top-of-loop defer would pin every bucket.
dataDir is the storage root that JSONL paths in the SQLite rows are relative to (or absolute; we accept either — the row's JSONLPath is the authoritative reader-side path).
coord (v0.1.1) is the storage coordinator. nil disables lease arbitration: the exporter reads JSONL files without notifying retention — fine for tests / no-retention deployments. Non-nil: groups whose bucket is mid-eviction (ErrFileBeingDeleted) are skipped with a WARN log; the rest of the zip still streams cleanly.
ctx is propagated into StreamMatching (cursor abort on cancel) and honored at every group + media iteration. Cancel mid-export leaves the zip well-formed up to the cancel point — adopters see a truncated body, never a corrupted central directory.
No internal hardCap: callers gate oversize exports via CountMatching BEFORE calling WriteZip. The cursor walks every matching row; pre-flight is the contract.
The returned error means either the SQLite read failed (zip not yet started) or a disk / zip write failed mid-stream (zip may be partial — zw.Close() is still called via defer to write the central directory of whatever was emitted, but downstream callers should treat the archive as suspect).
Types ¶
This section is empty.
Source Files