Documentation
¶
Overview ¶
Package source defines the live-event Source contract that Murmur pipelines read from. Concrete implementations live in subpackages (source/kinesis, source/kafka).
Sources here are streaming/live. Snapshot bootstraps live in source/snapshot, and replay drivers (which feed historical events through the same pipeline DSL) live in pkg/replay.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Record ¶
type Record[T any] struct { // EventID is a stable identifier used for at-least-once dedup. Sources should derive // this from the underlying message (Kinesis sequence number, Kafka topic+partition+ // offset) so duplicates are detectable. EventID string // EventTime is the time the upstream system attaches to this record. Used by windowed // aggregations for bucket assignment. Falls back to processing-time when missing. EventTime time.Time // PartitionKey, when nonempty, is the upstream partitioning key (Kinesis partition // key, Kafka message key). Useful for diagnostics and for sources that want to // surface partition affinity to the runtime. PartitionKey string // Value is the decoded payload. Value T // Ack must be called once the record has been fully processed (state-update committed, // dedup entry written). Sources use Ack to advance their checkpoint position. Ack func() error }
Record is the unit of work the streaming runtime hands to the pipeline. Implementations of Source decode their wire format into Records before yielding them.
type Source ¶
type Source[T any] interface { // Read yields records to the consumer. The consumer must call Ack on each record after // committing its effects. Errors are returned to the caller; a Source may also surface // retriable errors as a sentinel and continue. Read(ctx context.Context, out chan<- Record[T]) error // Name returns a stable identifier for logging and metrics (e.g. "kinesis:events-stream"). Name() string // Close releases any underlying resources. Close() error }
Source is a live event reader. Implementations must support graceful shutdown via the supplied context and yield records until the context is canceled.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package kafka provides a Kafka-backed implementation of source.Source via the franz-go client.
|
Package kafka provides a Kafka-backed implementation of source.Source via the franz-go client. |
|
Package kinesis provides a Kinesis Data Streams source via aws-sdk-go-v2.
|
Package kinesis provides a Kinesis Data Streams source via aws-sdk-go-v2. |
|
Package snapshot defines the SnapshotSource contract used by Murmur's Bootstrap execution mode.
|
Package snapshot defines the SnapshotSource contract used by Murmur's Bootstrap execution mode. |
|
dynamodb
Package dynamodb provides a snapshot.Source backed by DynamoDB ParallelScan, suitable for bootstrapping a Murmur pipeline from a DDB-resident OLTP table.
|
Package dynamodb provides a snapshot.Source backed by DynamoDB ParallelScan, suitable for bootstrapping a Murmur pipeline from a DDB-resident OLTP table. |
|
jsonl
Package jsonl provides a snapshot.Source that reads JSON Lines from any io.Reader — local files, S3 objects, gzip streams, etc.
|
Package jsonl provides a snapshot.Source that reads JSON Lines from any io.Reader — local files, S3 objects, gzip streams, etc. |
|
mongo
Package mongo provides a Mongo-backed implementation of snapshot.Source for Murmur's Bootstrap execution mode.
|
Package mongo provides a Mongo-backed implementation of snapshot.Source for Murmur's Bootstrap execution mode. |
|
parquet
Package parquet provides a snapshot.Source that reads Apache Parquet from any io.ReaderAt — local files, S3 objects fetched into memory, in-memory bytes from tests, etc.
|
Package parquet provides a snapshot.Source that reads Apache Parquet from any io.ReaderAt — local files, S3 objects fetched into memory, in-memory bytes from tests, etc. |
|
s3
Package s3 provides a snapshot.Source that scans every JSON-Lines object under an S3 prefix and emits records for bootstrap.
|
Package s3 provides a snapshot.Source that scans every JSON-Lines object under an S3 prefix and emits records for bootstrap. |