sandglass

Sandglass

Sandglass is a distributed, horizontally scalable, persistent, delayed message queue. It was developped to support asynchronous tasks. It supports synchronous tasks as well. It supports the competing consumers pattern.

Features

• Horizontal scalability
• Highly available
• Persistent storage
• Roughly strong ordering with a single consumer in a consumer group
• Round robin consumption between multiple consumers in a consumer group (looses ordering)
• Produce message to be consumed in the future
• Acknowledge each message individualy
• Automatic consumer offset tracking

Project status

EXPERIMENTAL: This is a prototype. This should not be used in production in its current form.

See TODO section below for more information

Installation

As of now there is no binaries available, you can only install from source using:

$ go get -u github.com/celrenheit/sandglass/cmd/sandglass

Usage

All data will be stored in /tmp/node1 for the first and /tmp/node2 for the second.

Open a first terminal window:

$ sandglass --config https://raw.githubusercontent.com/celrenheit/sandglass/master/demo/node1.yaml

On a second terminal window:

$ sandglass --config https://raw.githubusercontent.com/celrenheit/sandglass/master/demo/node2.yaml

Now you can create a topic using:

$ curl -XPOST http://localhost:2108/topics -d '{"name": "payments", "kind": 0, "replicationFactor": 2, "numPartitions": 6 }'

...and public a message:

$ curl -XPOST http://localhost:2108/topics/payments -d '{"value": "eyJoZWxsbyI6ICJ3b3JsZCIgfQo="}' -H 'Content-Type: application/json'

Architecture

General

                                                                  +-----------------+
                                                                  |                 |
                          +--------------------------+     +------>  Consumer       |
                          |                          |     |      |                 |
                          |    Sandglass Cluster     |     |      +-----------------+
                          |                          |     |
                          |                          +-----+  Round robin consumption
+-----------------+       |   +------------------+   |     |
|                 |       |   |                  |   |     |      +-----------------+
|  Producer       +------->   |                  |   |     |      |                 |
|                 |       |   |    Broker 1      |   |     +------>  Consumer       |
+-----------------+       |   |                  |   |            |                 |
                          |   |                  |   |            +-----------------+
                          |   +------------------+   |
+-----------------+       |                          |
|                 |       |   +------------------+   |
|  Producer       +------->   |                  |   |            +-----------------+
|                 |       |   |                  |   |            |                 |
+-----------------+       |   |    Broker 2      |   +-----+------>  Consumer       |
                          |   |                  |   |     |      |                 |
                          |   |                  |   |     |      +-----------------+
+-----------------+       |   +------------------+   |     |
|                 |       |                          |     |  Failover consumption
|  Producer       +------->   +------------------+   |     |     (NOT DONE YET)
|                 |       |   |                  |   |     |
+-----------------+       |   |                  |   |     |      +-----------------+
                          |   |    Broker 3      |   |     |      |                 |
                          |   |                  |   |     +------+  Consumer       |
                          |   |                  |   |            |                 |
                          |   +------------------+   |            +-----------------+
                          |                          |
                          |                          |
                          +--------------------------+

Topics

There is two kinds of topics:

• Timer:

  • Fixed number of partitions (set up-front, could change)
  • Time ordered using sandflake IDs
  • Can produce messages in the future

• Compacted (might change the name for this):

  • Fixed number of partitions (set up-front, cannot change)
  • Behaves like a distributed key value store

A topic has a number of partitions. Data is written into a single partition. Either the destination partition is specified by the producer. Otherwise, we fallback to choosing the destination partition using a consistent hashing algorithm.

Each produced message to a partition writes a message to a Write Ahead Log (WAL) and to the message log. The WAL is used for the replication logic, it is sorted in the order each message was produced. The message log is used for message comsumption, it is mainly sorted by time (please refer to sandflake ids for the exact composition)

The content of the message is stored in the message log and not in the WAL (only the keys are important). This way the message log is used for fast consumption avoiding random reads.

This will probably change in order to have the WAL as the only source of truth instead of storing the content in the message log. This of course will have an impact because we are transfering random reads to the consumption path. Utlimately, we are going to have to store the content of the message in both logs for better performance at the cost of disk space.

A message is composed of the following fields:

    topic
    partition

    index   <- position in the WAL

    offset  <- position in the message log for timer topics
    key     <- position in the message log for key for compacted topics

    value   <- your payload

TODO

• Clean up all the mess
• Fix replication and re assign partitions correctly when a node goes down
• Save all the registered nodes and not rely on gossip to allow topic creation even if there is not enough nodes
• More TODOs in TODO section (#inception)
• Make everything more robust...