half-life

HalfLife

HalfLife is a Cosmos-based blockchain validator monitoring and alerting utility

Monitors and alerts for scenarios such as:

• Slashing period uptime
• Recent missed blocks (is the validator signing currently)
• Jailed status
• Tombstoned status
• Individual sentry nodes unreachable/out of sync
• Chain halted

Discord messages are created in the configured webhook channel for:

• Current validator status
• Detected alerts

Quick start

Download

Binaries are available on the releases page.

or you can deploy the docker file.

Setup config.yaml

Copy config.yaml.example to config.yaml and populate with your discord and validator information. You can optionally provide the sentries array to also monitor the sentries via grpc. rpc-retries can optionally be provided to override the default of 5 RPC retries before alerting, useful for congested RPC servers. fullnode can be set to true to only monitor reachable and out of sync for the provided sentries. address is not required when fullnode is true. sentry-grpc-error-threshold can be provided for each validator to tune how many grpc errors are detected (roughtly 30 seconds between checks) before issuing a notification.

See here for how to create a webhook for a discord channel.

Once you've created the webhook, copy the URL. It'll look something like this: https://discord.com/api/webhooks/978129125394247720/cwM4Ks-kWcK3Jsg4I_cboauYjOa48ngI2VKaS76afsMwuY7-U4Frw3BGcYXCJvZJ2kWD

This will be used later to be put into the config.yaml. The webhook id is 978129125394247720 (from the URL), and webhook token is cwM4Ks-kWcK3Jsg4I_cboauYjOa48ngI2VKaS76afsMwuY7-U4Frw3BGcYXCJvZJ2kWD

Save the values as follows (note these values are from the URL):

webhook:
  id: 978129125394247720
  token: cwM4Ks-kWcK3Jsg4I_cboauYjOa48ngI2VKaS76afsMwuY7-U4Frw3BGcYXCJvZJ2kWD

Start monitoring

Begin monitoring with:

halflife monitor

By default, half-life monitor will look for config.yaml and status.yaml in the current working directory. To specify a different config file path, use the --file/-f or --status/-s flag:

halflife monitor -f ~/config.yaml -s ~/status.yaml

When a validator is first added to config.yaml and halflife is started, a status message will be created in the discord channel and the ID of that message will be added to status.yaml. Pin this message so that the channel's pinned messages can act as a dashboard to see the realtime status of the validators.

Alerts will be posted when any error conditions are detected, and follow up messages will be posted when those errors are cleared.

For high and critical errors, the configured discord user IDs will be tagged

Build from source

Install Go

Go is necessary in order to build half-life from source.

# install
wget https://golang.org/dl/go1.18.2.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.18.2.linux-amd64.tar.gz

# source go
cat <<EOF >> ~/.profile
export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export GO111MODULE=on
export PATH=$PATH:/usr/local/go/bin:$HOME/go/bin
EOF
source ~/.profile
go version

Install halflife binary

cd ~/half-life
go install

Running with docker

I usually have multiple snippets of the yaml file and put them in the config/ directory.

The docker file can be run via.

docker run -it -v/tmp/config:/config -v/tmp/status:/status --rm ghcr.io/pfc-developer/half-life:latest