exactly-once

Overview ¶

Exactly-once pipeline: Kafka → keyed aggregation → transactional Kafka.

Source offsets, operator state, and sink output commit as ONE coordinated checkpoint. After any crash, the pipeline restores the latest completed checkpoint and replays — committed output is never duplicated, uncommitted output was never visible.

Requires a running Kafka broker (e.g. docker):

docker run -p 9092:9092 apache/kafka:latest
go run ./examples/exactly-once/

IMPORTANT: consumers of the output topic must set isolation.level=read_committed, or they will see records from aborted transactions. Try:

kafka-console-consumer --bootstrap-server localhost:9092 \
    --topic order-totals --isolation-level read_committed