go-graphql-subscription-example

go-graphql-subscription-example

Project that demonstrates GraphQL subscriptions (over Websocket) to consume Apache Kafka messages.

Technical stack

This application mainly uses:

• GraphQL

  ↳ graph-gophers/graphql-go ┇ graph-gophers/graphql-transport-ws ┇ graphql/graphiql

• Kafka

  ↳ segment-integrations/connect-kafka

• Expression language

  ↳ antonmedv/expr

• CLI

  ↳ spf13/cobra

• Log

  ↳ rs/zerolog

Pre-requisites

Requires Go 1.11.x or above, which support Go modules. Read more about them here.

Build

The project comes with a Makefile, so all the main activities can be performed by make.

⚠ The source code provided is incomplete: it does not contain generated code:

• generated code for embedding the static resources

To build the project, simply invoke the build targets:

make build  

How to use

1. Start Kafka server

At first, kafka must be started. See official documentation for more.

Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you don't already have one.

> bin/zookeeper-server-start.sh config/zookeeper.properties

Now start the Kafka server:

> bin/kafka-server-start.sh config/server.properties

2. Create topics

For the purpose of the demo, some topics shall be created. So, let's create 2 topics named topic-a and topic-b, with a single partition and only one replica:

> bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic topic-a
> bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic topic-b

3. Start the GraphQL server

The configuration is pretty straightforward:

> ./go-graphql-subscription-example --help
Usage:
  go-graphql-subscription-example [flags]

Flags:
      --brokers strings   The list of broker addresses used to connect to the kafka cluster (default [localhost:9092])
  -h, --help              help for go-graphql-subscription-example
      --port uint16       The listening port (default 8000)
      --topics strings    The list of kafka topics that subscribers can consume (default [foo])

Run the application which exposes the 2 previously created topics to subscribers:

> ./go-graphql-subscription-example --topics topic-a,topic-b 

4. Subscribe

The application exposes a graphql endpoint through which clients can receive messages coming from a kafka topic.

Navigate to http://localhost:8000/graphiql URL and submit the subscription to the topic topic-a.

subscription {
  event(on: "topic-a")
}

The offset id to consume from can also be specified. Negative values have a special meaning:

• -1: the most recent offset available for a partition (end)
• -2: the least recent offset available for a partition (beginning)

subscription {
  event(on: "topic-a", at: -1)
}

Additionally, a filter expression can be specified. The events consumed are then only ones matching the given predicate.

subscription {
  event(
    on: "topic-a",
    at: -1,
    matching: "value > 8"
  )
}

5. Push messages

Run the producer and then type a few messages into the console to send to Kafka. Note that messages shall be JSON objects.

> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic topic-a
{ "message": "hello world !", "value": 14 }

The message should be displayed on the browser.