Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Record ¶
type Record struct {
// Key is used for partitioning after KeyBy. Nil means unkeyed.
Key []byte
// Value is the raw data payload.
Value []byte
// Timestamp is the event time — when the event actually happened.
// Used for windowing and watermark generation.
Timestamp time.Time
// Offset is the source offset (e.g. Kafka partition offset).
// Used for checkpointing — when we restore, we rewind to this offset.
Offset int64
// Partition is the source partition this record came from.
// Together with Offset it lets the engine track barrier-aligned
// per-partition positions in-band, immune to channel buffering.
Partition int
// Headers carry optional metadata (e.g. Kafka headers, trace IDs).
Headers map[string][]byte
// Parsed holds an optionally-deserialized view of Value.
// It is nil unless a Deserializer is configured on the source.
// Value always carries the raw bytes; Parsed is a convenience for
// operators that want to work with a typed payload.
Parsed any
// IsWatermark marks this record as a watermark marker instead of data.
// Watermarks have no Key or Value — they only carry a Timestamp.
// Operators use watermarks to know when windows can close.
IsWatermark bool
// IsBarrier marks this record as a checkpoint barrier.
// Barriers flow in-band through the pipeline (like watermarks).
// When a stateful operator sees a barrier, it snapshots its state,
// then forwards the barrier downstream. When the barrier reaches
// the sink, the checkpoint is complete.
IsBarrier bool
// CheckpointID identifies which checkpoint this barrier belongs to.
// Only valid when IsBarrier == true.
CheckpointID string
}
Record is the fundamental data unit flowing through a stream pipeline. Every source emits Records, every operator transforms them, every sink receives them.
A Record can be one of three types:
- Data record: IsWatermark == false, IsBarrier == false — carries key/value/timestamp
- Watermark: IsWatermark == true — carries a Timestamp that says "no records with event time < this timestamp will arrive after this point"
- Barrier: IsBarrier == true — carries a CheckpointID that says "snapshot state for this checkpoint when the barrier passes"
Watermarks drive window firing. Barriers drive checkpointing.
func NewBarrier ¶
NewBarrier creates a checkpoint barrier Record with the given checkpoint ID. Barriers flow through the pipeline and trigger stateful operators to snapshot their state. When the barrier reaches the sink, the checkpoint is complete.
func NewWatermark ¶
NewWatermark creates a watermark Record with the given timestamp. Watermarks signal that no more records with event time < timestamp will arrive.
func (Record) WithHeader ¶
WithHeader returns a copy of the record with the given header added.
func (Record) WithOffset ¶
WithOffset returns a copy of the record with the given source offset.