Documentation
¶
Overview ¶
Package s3 provides a replay.Driver that reads JSON-Lines event archives from S3 (or any S3-compatible store like MinIO) and feeds the records into the pipeline.
This is the standard "kappa replay" path for Kinesis-backed pipelines whose live retention is shorter than the desired backfill window. Events are typically archived to S3 by Kinesis Data Firehose with date-partitioning (e.g. year=/month=/day=/), or by Kafka Connect's S3 sink for Kafka topics.
Phase 1 supports JSON-Lines (one JSON object per line). Parquet support is a Phase 2 addition gated on real demand.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config[T any] struct { Client *s3.Client Bucket string // Prefix narrows the object scan (e.g. "events/year=2026/month=05/"). Empty means // the entire bucket. Prefix string // Decode converts a single JSON-Lines line to T. Decode Decoder[T] // OnDecodeError, if non-nil, is called for every line whose Decode returned an // error. Default behavior is to drop silently. OnDecodeError func(key string, lineNum int, raw []byte, err error) }
Config configures an S3 replay driver.
type Driver ¶
type Driver[T any] struct { // contains filtered or unexported fields }
Driver implements replay.Driver for S3-archived JSON-Lines events.
Source Files
Click to show internal directories.
Click to hide internal directories.