Documentation
¶
Overview ¶
Command payload-migrate backfills a JSONL file-backend payload dump (relay-payloads.jsonl) into the ClickHouse payload_logs table, so historical request/response bodies become queryable through the same reader the relay uses — after which the multi-GB local file can be deleted.
It reuses payload.Record, so the file's base64-encoded bodies decode to raw bytes and the RFC3339 timestamp parses natively; rows are appended in the exact column order the live ClickHouse sink and reader use, guaranteeing the migrated rows are readable via the relay's payload Get/List.
One-shot and resumable: every request_id already present in payload_logs is loaded up front and skipped, so re-running after an interruption only inserts the remainder (and never double-inserts). Bodies are MB-scale, so inserts flush on a byte budget as well as a row count.
go run ./cmd/payload-migrate -file relay-payloads.jsonl # DSN from $RELAY_CH_DSN
Source Files