heartbeats

Heartbeats

Small helper service to monitor heartbeats (repeating "pings" from other systems). If a "ping" does not arrive in the given interval & grace period, Heartbeats will send notifications.

Flags

-c, --config string      Path to notifications config file (default "./config.yaml")
-d, --debug              Verbose logging.
-t, --trace              More verbose logging.
-j, --json               Output logging as json.
-v, --version            Print the current version and exit.
    --host string        Host of Heartbeat service. (default "127.0.0.1")
-p, --port int           Port to listen on (default 8090)
-s, --site-root string   Site root for the heartbeat service (default "http://host:port")
-h, --help               help for heartbeat

Endpoints

Send a heartbeat

GET|POST https://heartbeats.example.com/ping/heartbeat1

Sends a "alive" message.

Query parameters

output=json|yaml|yml|text|txt

Format response in one of the passed format. If no specific format is passed the response will be text.

Response Codes

Send a failed heartbeat

GET|POST https://heartbeats.example.com/ping/heartbeat1/fail

Send a direct failure to not wait until the heartbeat grace period is expired.

Query parameters

output=json|yaml|yml|text|txt

Format response in one of the passed format. If no specific format is passed the response will be text.

Response Codes

Show heartbeat status

GET https://heartbeats.example.com/status

Shows current status of all heartbeats.

GET https://heartbeats.example.com/status/heartbeat

Show current status of given heartbeat.

Query parameters

output=json|yaml|yml|text|txt

Format response in one of the passed format. If no specific format is passed the response will be text.

Response Codes

Show configuration

GET https://heartbeats.example.com/config

Shows current configuration.

Query parameters

output=json|yaml|yml|text|txt

Format response in one of the passed format. If no specific format is passed the response will be text.

Response Codes

Show metrics

GET https://heartbeats.example.com/metrics

Shows metrics for Prometheus.

Response Codes

Show Heartbeats server status

GET https://heartbeats.example.com/metrics

Shows Heartbeats server status.

Response Codes

Config file

Heartbeats and notifications must be configured in a file. Config files can be yaml, json or toml. The config file should be loaded automatically if changed. Please check the log output to control if the automatic config reload works in your environment. If interval and grace where changed, they will be reset to the corresponding new value!

Avoid using "secrets" directly in your config file by using environment variables. Set the prefix env: followed by the environment variable to load the corresponding environment variable.

Examples:

./config.yaml

---
heartbeats:
  - name: watchdog-prometheus-prd
    description: test prometheus -> alertmanager workflow
    interval: 5m
    grace: 30s
    notifications: # must match with notifications.services[*].name
      - slack
      - mail_provider_x
  - name: watchdog-prometheus-int
    description: test prometheus -> alertmanager workflow
    interval: 60m
    grace: 5m
    notifications:
      - msTeams
notifications:
  defaults: # uses this subject & message if not overwritten in a service
    subject: Heartbeat {{ .Name }} «{{ .Status }}»
    message: "*Description:*\n{{.Description}}.\n\nLast ping: {{ .TimeAgo .LastPing }}"
    sendResolved: true
  services:
    - name: slack
      enabled: false
      type: slack
      sendResolved: true
      oauthToken: env:ENV_VARIABLE_YOU_DEFINE
      channels:
      - test
    - name: mail_provider_x
      enabled: true
      type: mail
      sendResolved: false
      subject: "[Heartbeat]: {{ .Name }}"
      message: "Heartbeat is missing.\n\n{{.Description}}\n interval: {{.Interval}}, grace: {{.Grace}}\nPlease check your sending service!"
      senderAddress: heartbeat@example.com
      smtpHostAddr: smtp.example.com
      smtpHostPort: 587
      smtpAuthUser: heartbeat@example.com
      smtpAuthPassword: env:ENV_VARIABLE_YOU_DEFINE
      receiverAddresses:
        - heartbeat@example.com
    - name: msTeams
      enabled: true
      type: msteams
      message: "Heartbeat is missing.\n\n{{.Description}}\n interval: {{.Interval}}, grace: {{.Grace}}\nPlease check your sending service!"
      webhooks:
        - http://example.webhook.office.com/webhook2/...
        - env:WHY_NOT_A_SECRET_WEBHOOK

Heartbeat

Each Heartbeat must have following parameters:

Notifications

Heartbeat uses the library https://github.com/nikoksr/notify for notification.

For the moment only mail, slack and msteams are implemented. Feel free to create a Pull Request.

Defaults (notification.defaults) set the general subject, message and/or sendResolved for each service. Each service can override these settings by adding the corresponding key (subject, message and/or sendResolved).

You can use all properties from heartbeats in subject and/or message. The variables must start with a dot, a capital letter and be surrounded by double curly brackets. Example: {{ .Status }}

There is a function (TimeAgo) that calculates the time of the last ping to now. (borrowed from here)

Example:

message: "Last ping was: {{ .TimeAgo .LastPing }}"

Slack

Mail

MS Teams

Metrics

Prometheus metrics can be scraped at the endpoint /metrics. Prometheus metrics starts with heartbeats_.

Do not forget to update your Prometheus scrape config.

Example:

scrape_configs:
- job_name: heartbeats
  scrape_interval: 30s
  static_configs:
  - targets:
    - heardbeats:8090