source

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 12, 2026 License: Apache-2.0 Imports: 2 Imported by: 0

Documentation

Overview

Package source defines the live-event Source contract that Murmur pipelines read from. Concrete implementations live in subpackages (source/kinesis, source/kafka).

Sources here are streaming/live. Snapshot bootstraps live in source/snapshot, and replay drivers (which feed historical events through the same pipeline DSL) live in pkg/replay.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Record

type Record[T any] struct {
	// EventID is a stable identifier used for at-least-once dedup. Sources should derive
	// this from the underlying message (Kinesis sequence number, Kafka topic+partition+
	// offset) so duplicates are detectable.
	EventID string

	// EventTime is the time the upstream system attaches to this record. Used by windowed
	// aggregations for bucket assignment. Falls back to processing-time when missing.
	EventTime time.Time

	// PartitionKey, when nonempty, is the upstream partitioning key (Kinesis partition
	// key, Kafka message key). Useful for diagnostics and for sources that want to
	// surface partition affinity to the runtime.
	PartitionKey string

	// Value is the decoded payload.
	Value T

	// Ack must be called once the record has been fully processed (state-update committed,
	// dedup entry written). Sources use Ack to advance their checkpoint position.
	Ack func() error
}

Record is the unit of work the streaming runtime hands to the pipeline. Implementations of Source decode their wire format into Records before yielding them.

type Source

type Source[T any] interface {
	// Read yields records to the consumer. The consumer must call Ack on each record after
	// committing its effects. Errors are returned to the caller; a Source may also surface
	// retriable errors as a sentinel and continue.
	Read(ctx context.Context, out chan<- Record[T]) error

	// Name returns a stable identifier for logging and metrics (e.g. "kinesis:events-stream").
	Name() string

	// Close releases any underlying resources.
	Close() error
}

Source is a live event reader. Implementations must support graceful shutdown via the supplied context and yield records until the context is canceled.

Directories

Path Synopsis
Package kafka provides a Kafka-backed implementation of source.Source via the franz-go client.
Package kafka provides a Kafka-backed implementation of source.Source via the franz-go client.
Package kinesis provides a Kinesis Data Streams source via aws-sdk-go-v2.
Package kinesis provides a Kinesis Data Streams source via aws-sdk-go-v2.
Package snapshot defines the SnapshotSource contract used by Murmur's Bootstrap execution mode.
Package snapshot defines the SnapshotSource contract used by Murmur's Bootstrap execution mode.
dynamodb
Package dynamodb provides a snapshot.Source backed by DynamoDB ParallelScan, suitable for bootstrapping a Murmur pipeline from a DDB-resident OLTP table.
Package dynamodb provides a snapshot.Source backed by DynamoDB ParallelScan, suitable for bootstrapping a Murmur pipeline from a DDB-resident OLTP table.
jsonl
Package jsonl provides a snapshot.Source that reads JSON Lines from any io.Reader — local files, S3 objects, gzip streams, etc.
Package jsonl provides a snapshot.Source that reads JSON Lines from any io.Reader — local files, S3 objects, gzip streams, etc.
mongo
Package mongo provides a Mongo-backed implementation of snapshot.Source for Murmur's Bootstrap execution mode.
Package mongo provides a Mongo-backed implementation of snapshot.Source for Murmur's Bootstrap execution mode.
parquet
Package parquet provides a snapshot.Source that reads Apache Parquet from any io.ReaderAt — local files, S3 objects fetched into memory, in-memory bytes from tests, etc.
Package parquet provides a snapshot.Source that reads Apache Parquet from any io.ReaderAt — local files, S3 objects fetched into memory, in-memory bytes from tests, etc.
s3
Package s3 provides a snapshot.Source that scans every JSON-Lines object under an S3 prefix and emits records for bootstrap.
Package s3 provides a snapshot.Source that scans every JSON-Lines object under an S3 prefix and emits records for bootstrap.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL