konsume

konsume is a powerful and flexible tool designed to consume messages from various message queues like RabbitMQ and Kafka and perform HTTP requests based on configurations.

Table of Contents

• Overview
• Features
• Installation
• Usage
• Configuration
• FAQ
• Contributing

Overview

TLDR; If you want to consume messages from RabbitMQ or Kafka and perform HTTP requests based on configurations, konsume is for you.

komsume is a tool that easily connects message queues like RabbitMQ and Kafka with web services, automating data-driven HTTP requests. It bridges complex messaging systems and web APIs, enabling you to create workflows where queue messages automatically trigger web requests. Its flexible setup, including various retry options and customizable request formats, suits a range of uses, from basic data transfers to intricate processing tasks.

Features

• Message Consumption: Efficiently consumes messages from specified queues.
• Dynamic HTTP Requests: Sends HTTP requests based on message content and predefined configurations.
• Retry Strategies: Supports fixed, exponential, and random retry strategies for handling request failures.
• Request Body Templating: Dynamically constructs request bodies using templates with values extracted from incoming messages.
• Custom HTTP Headers: Allows setting custom HTTP headers for outgoing requests.
• Configurable via YAML: Easy configuration using a YAML file for defining queues, routes, and behaviors.
• Monitoring: Provides a Prometheus endpoint for monitoring metrics.

Installation

Easiest way to install konsume is to run via Docker. konsume will look for a configuration file named config.yaml in the /config directory. Or you can set the path of the configuration file using the KONSUME_CONFIG_PATH environment variable.

docker run -d --name konsume -v /path/to/config.yaml:/config/config.yaml bugrakocabay/konsume:latest

Alternatively, you can download the latest binary with the go installer:

go install github.com/bugrakocabay/konsume@latest

Usage

konsume depends on a YAML configuration file for defining queues, routes, and behaviors. There are two main sections in the configuration file: providers and queues. In the providers section, you can define the message queue providers that konsume will use to consume messages. In the queues section, you can define the queues that konsume will consume messages from and the routes that konsume will use to send HTTP requests. You can also find detailed usage examples in the examples directory.

A simple usage for konsume with RabbitMQ:

providers:
  - name: 'rabbit-queue'
    type: 'rabbitmq'
    amqp-config:
      host: 'localhost'
      port: 5672
      username: 'user'
      password: 'password'
queues:
  - name: 'queue-for-rabbit'
    provider: 'rabbit-queue'
    routes:
      - name: 'ServiceA_Queue2'
        type: 'REST'
        method: 'POST'
        url: 'https://someurl.com'

A simple usage for konsume with Kafka:

providers:
  - name: 'kafka-queue'
    type: 'kafka'
    kafka-config:
      brokers:
        - 'localhost:9092'
      topic: 'your_topic_name'
      group: 'group1'
queues:
  - name: 'queue-for-kafka'
    provider: 'kafka-queue'
    routes:
      - name: 'ServiceA_Queue2'
        type: 'REST'
        method: 'POST'
        url: 'https://someurl.com'

Configuration

FAQ

{
	"name": "John",
	"email": "john@doe.com"
}

routes:
  - name: 'test-route'
    method: 'POST'
    type: 'REST'
    headers:
      Content-Type: 'application/json'
    body:
      userName: '{{name}}'
      eMail: '{{email}}'
    url: 'http://someurl.com'

routes:
  - name: 'test-route'
    method: 'POST'
    type: 'graphql'
    headers:
      Content-Type: 'application/json'
    body:
      mutation: |
        mutation {
          addUser(name: {{name1}}, email: {{email1}}) {
            id
            name
            email
          }
        }
    url: 'http://someurl:4000/graphql'

queues:
  - name: 'queue-for-rabbit'
    provider: 'rabbit-queue'
    retry:
      enabled: true
      strategy: 'fixed'
      max_retries: 5
      interval: 5s
      threshold_status: 500
    routes:
      - name: 'ServiceA_Queue2'
        type: 'REST'
        method: 'POST'
        url: 'https://someurl.com'

Contributing

🌟 Your contributions are welcome!

Whether you're looking to fix bugs, add new features, or improve documentation, your help is greatly appreciated. Contributing to konsume is not only a great way to enhance this tool, but it's also an excellent opportunity to get involved with a community of like-minded individuals.

Here's how you can contribute:

Report Issues: Found a bug or have a suggestion? Open an issue and let us know! Submit Pull Requests: Have a fix or a new feature? Submit a pull request! Feedback and Ideas: Share your thoughts and ideas on how we can improve konsume. Documentation: Help us improve or translate the documentation. Before contributing, please read our Contributing Guidelines for more information on how to get started.

💡 No contribution is too small – whether it's fixing typos, improving code readability, or updating documentation, all contributions are valuable and appreciated!

Join us in making konsume better for everyone! 🚀