tipoca-stream

tipoca-stream

Near realtime cloud native data pipeline. Just another data pipeline.

Install

The pipeline is a combination of services deployed independently. This repo holds the code for the redshiftsink only.

• RedshiftSink Please follow REDSHIFTSINK.md to install the RedshiftSink Kubernetes Operator. Creating the RedshiftSink resource installs Batcher and Loader pods in the cluster. These pods sinks the data from Kafka topics to Redshift, it takes care of the database migration when required. Redshiftsink has a rich masking support. It also supports table reloads in Redshift when masking configurations are modified in Github.

      kubectl get redshiftsink

• Kafka Install Kafka using Strimzi CRDs or self hosted or managed kafka.

      kubectl get kafka

• Producer Install Producer using Strimzi CRDs and Debezium. Creating the kafkaconnect and kafkaconnector creates a kafkaconnect pod in the cluster which start streaming the data from the source(MYSQL, RDS, etc..) to Kafka.

      kubectl get kafkaconnect
      kubectl get kafkaconnector

The project has pluggable libraries which can be composed to solve any other data pipeline use case.

Contribute

Please follow this to bring a change.

Thanks

• Debezium.
• Strimzi.io for the Kafka CRDs.
• Yelp for open-sourcing the the blog on the redshift connector.
• Linkedin for open-sourcing goavro.
• Linkedin for donating Kafka.
• Shopify for open-sourcing sarama.
• Thockin for open-sourcing go-build-template.
• Clever for open-sourcing s3-to-redshift library.
• danielqsj for kafka-exporter.