scraphook

Scraphook

Centralized webhook management system

Scraphook is a fast, reliable and powerful webhook management. It enables a capability of manage your webhook messages at one place, forward them to appropriate endpoints based on your rules.

Table of contents

• How it works?
• Features
• Get started

How it works?

The core of Scraphook is a simple API route - the /webhook/:webhook_id. Whenever data is posted to that route, we store the message down to the database, then publish an event to the message bus. Your task is simple, just defined a consumer that subscribes to the message bus and handle the webhook message in your consumer.

Based on that concept, we chose RabbitMQ with Publish/Subscribe model (Documentation), and defined a forwarding http request consumer that supports rule based routing.

Features

• A just works webhook endpoint - received message and save it to database
• Publish/subscribed model and predefined forwarding http request consumer
• Re-publish a webhook message whenever you want
• A simple APIs set to manage your events, your forwarding logs
• Anonymous (for testing and demo) and Restricted (for production) mode (with token authentication)

Get started

Set up the application

Create a config file (configs.props) and a docker compose file (docker-compose.yaml)

SCRAP_APP_ENV=dev
SCRAP_APP_VERSION=v1.0.0
SCRAP_APP_PORT=3000

SCRAP_DB_URI_MASTER=postgres://postgres:postgres@postgres:5432/postgres?application_name=scraphook

SCRAP_MQ_URI=amqp://rabbitmq:changemenow@rabbitmq:5672
SCRAP_MQ_EXCHANGE_NAME=scraphook.webhook
SCRAP_MQ_QUEUE_NAME=consumer.forwarding.http

SCRAP_CACHE_URI=redis:6379
SCRAP_CACHE_DB=1
SCRAP_CACHE_EXPIRY_SECONDS=1
SCRAP_REDLOCK_EXPIRY_SECONDS=900

SCRAP_QUERY_LIMIT=20
SCRAP_QUERY_LIMIT_MAX=100

SCRAP_WEBHOOK_MODE=anonymous

version: "3.9"

services:
  scraphook.api:
    image: "scrapnode/scraphook:latest"
    restart: always
    depends_on:
      - redis
      - postgres
      - rabbitmq
    ports:
      - "3000:3000"
    volumes:
      - ./configs.props:/app/configs.props
        
  scraphook.consumer:
    image: "scrapnode/scraphook:latest"
    restart: always
    depends_on:
      - redis
      - postgres
      - rabbitmq
    volumes:
      - ./configs.props:/app/configs.props
    environment:
      - SCRAP_SERVICE=CONSUMER.FORWARDING_HTTP

  redis:
    image: "redis:5"
    restart: always
    ports:
      - "6379:6379"

  postgres:
    image: postgres:13
    restart: always
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_NAME=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres

  rabbitmq:
    image: rabbitmq:3-management-alpine
    restart: always
    ports:
      - "5672:5672"
      - "15672:15672"
    environment:
        RABBITMQ_DEFAULT_USER: rabbitmq
        RABBITMQ_DEFAULT_PASS: changemenow

So your folder will be structured like this

- your-folder
    |- configs.props
    |- docker-compose.yaml

To start your application, use command docker compose up -d && sleep 10. You have to wait for 10s because initialize a database is take time. After that, open http://localhost:3000/info to make sure the API is up.

Test the application

In this tutorial, we will use choose the webhook id is wh_test, so your webhook endpoint is http://localhost:3000/webhook/wh_test

1. Post an event to your webhook with cURL and get the response

   curl -X POST http://localhost:3000/webhook/wh_test -d '{"id":1,"from":{"name": "Tuan Nguyen"}}' 
   

   Response

   {
     "id": "msg_25h9ej6g8OKUTV6D9t3pRxsQTfB",
     "webhook_id": "wh_test",
     "message": "{\"id\":1,\"from\":{\"name\": \"Tuan Nguyen\"}}",
     "signature": "",
     "created_at": "2022-02-27T13:29:07.405297Z"
   }
   

2. Get the list of your webhook messages that was stored in your database

   curl http://localhost:3000/webhook/wh_test/messages
   

   Response

   {
     "after": "",
     "before": "2022-02-27T00:00:00Z",
     "data": [
       {
         "id": "msg_25h9ej6g8OKUTV6D9t3pRxsQTfB",
         "webhook_id": "wh_test",
         "message": "{\"id\": 1, \"from\": {\"name\": \"Tuan Nguyen\"}}",
         "signature": "",
         "created_at": "2022-02-27T13:29:07.405297Z"
       }
     ]
   }
   

3. Here is the most interesting part - create a forwarding rule for your webhook message. The rule is based on regex, if the message body is matched by a regex rule, it will be forwarded to your configured endpoint. At this step, we will create a rule that forward all message that contains the word Tuan (my name :D) to a testing HTTP RESTful API (httpbin.org)

    curl -X POST http://localhost:3000/webhook/wh_test/endpoints -d '{"description":"Testingforwardingendpoint","endpoint_url":"https://httpbin.org/post","forwarding_rule_regex":"Tuan"}' 
   

   Response

   {
     "id": "whe_25lcjHZB30ALG55MINBy4OKQ6Mj",
     "webhook_id": "wh_test",
     "description": "Testingforwardingendpoint",
     "endpoint_url": "https://httpbin.org/post",
     "forwarding_rule_regex": "Tuan",
     "forwarding_rule_regex_negative": "",
     "created_at": "2022-03-01T03:27:26Z",
     "updated_at": "2022-03-01T10:27:26.298828759+07:00",
     "deleted_at": null
   }
   

4. Post an event to your webhook again

   curl -X POST http://localhost:3000/webhook/wh_test -d '{"id":2,"from":{"name": "Tuan Nguyen"}}' 
   

   Response

   {
     "id": "msg_25hAwcIQ4GEgg1ACvZdE3DapKEo",
     "webhook_id": "wh_test",
     "message": "{\"id\":2,\"from\":{\"name\": \"Tuan Nguyen\"}}",
     "signature": "",
     "created_at": "2022-02-27T13:39:43.852509Z"
   }
   

5. Check your log with the webhook message id in the response above

   curl http://localhost:3000/webhook/wh_test/logs?message_id=msg_25hAwcIQ4GEgg1ACvZdE3DapKEo
   

   Response

   {
     "after": "",
     "before": "2022-02-27T00:00:00Z",
     "data": [
       {
         "id": "log_25hAwdofryY8P3nQMpiCXGyHg47",
         "webhook_id": "wh_test",
         "message_id": "msg_25hAwcIQ4GEgg1ACvZdE3DapKEo",
         "message": null,
         "endpoint_id": "whe_25lcjHZB30ALG55MINBy4OKQ6Mj",
         "endpoint": null,
         "request_url": "https://httpbin.org/post",
         "request_headers": "{\"Accept\":[\"application/json\"],\"Content-Type\":[\"application/json\"],\"Scraphook-Auth-Token\":[\"\"],\"Scraphook-Message-Id\":[\"msg_25hAwcIQ4GEgg1ACvZdE3DapKEo\"],\"Scraphook-Message-Signature\":[\"\"],\"User-Agent\":[\"go-resty/2.7.0 (https://github.com/go-resty/resty)\"]}",
         "request_body": "{\"id\":2,\"from\":{\"name\": \"Tuan Nguyen\"}}",
         "response_status": 200,
         "response_headers": "{\"Access-Control-Allow-Credentials\":[\"true\"],\"Access-Control-Allow-Origin\":[\"*\"],\"Content-Length\":[\"721\"],\"Content-Type\":[\"application/json\"],\"Date\":[\"Sun, 27 Feb 2022 13:39:45 GMT\"],\"Server\":[\"gunicorn/19.9.0\"]}",
         "response_body": "{\n  \"args\": {}, \n  \"data\": \"{\\\"id\\\":2,\\\"from\\\":{\\\"name\\\": \\\"Tuan Nguyen\\\"}}\", \n  \"files\": {}, \n  \"form\": {}, \n  \"headers\": {\n    \"Accept\": \"application/json\", \n    \"Accept-Encoding\": \"gzip\", \n    \"Content-Length\": \"39\", \n    \"Content-Type\": \"application/json\", \n    \"Host\": \"httpbin.org\", \n    \"Scraphook-Auth-Token\": \"\", \n    \"Scraphook-Message-Id\": \"msg_25hAwcIQ4GEgg1ACvZdE3DapKEo\", \n    \"Scraphook-Message-Signature\": \"\", \n    \"User-Agent\": \"go-resty/2.7.0 (https://github.com/go-resty/resty)\", \n    \"X-Amzn-Trace-Id\": \"Root=1-621b7f21-35cb969c439856a825bcfe33\"\n  }, \n  \"json\": {\n    \"from\": {\n      \"name\": \"Tuan Nguyen\"\n    }, \n    \"id\": 2\n  }, \n  \"origin\": \"113.161.39.115\", \n  \"url\": \"https://httpbin.org/post\"\n}\n",
         "created_at": "2022-02-27T13:39:45.394489Z"
       }
     ]
   }
   

You can see, our consumer logged all the properties of forwarding request (request headers and url, response status, and headers) so that you can trace which request is success or which is failed with response data.

Summary

The tutorial used a anonymous configuration set that allow us using API without authentication credentials. On production environment, you may want to have a different configuration to protect your webhook. Please check our documentation for more details.