README
¶
Manager
The Manager service is responsible for ensuring all the data is processed within a timeframe. By default, the timeframe is a single block. If we are left with unprocessed data by the time a new block arrives, the manager would scale up by deploying more instances of the service that is currently lagging. Conversely, if we have too many instances of a service, it will be scaled down by removing deployments.
This service also collects runtime performance data of all the parts of the system and will make the information available to other third-party tools so that performance dashboards can be constructed.
Building
Requirements
- Go
- Docker (for Redis instance)
The service is built using a Makefile, use make help for more information.
Basic build and run
# Install dependencies
make dependencies
# Build
make
# Build and run
make run
# Build Docker image
make docker_build
# Run Docker image
make docker_run
# Run tests
make test
Configuration
The service is configured via environment variables, they can be found and changed in the Makefile.
| VARIABLE | TYPE | DESCRIPTION |
|---|---|---|
| LOG_LEVEL | string | Logging level - debug, info, warn, error, fatal |
| LOG_FORMAT | string | Format of the log - text or json |
| SERVICE_NAME | string | The name of the service in the structured log |
| CHAIN_ID | string | Name of the chain for logging purposes |
| HIVE_ENDPOINT | string | The Hive endpoint for all services in a deployment to use |
| RPC_ENDPOINT | string | The JSON-RPC endpoint for all services in a deployment to use |
| RPC_WEBSOCKET_ENDPOINT | string | The websocket endpoint for the chain to read new blocks from |
| REDIS_DATABASE | int | The Redis database number for the queues |
| REDIS_ENDPOINT | string | The Redis endpoint - 127.0.0.1:6379 |
| COLLECTOR_QUEUE_NAME | string | The Redis key to write work items to |
| HEALTH_CHECK_QUEUE_NAME | string | The Redis key to monitor for health checker scaling |
| EXECUTOR_QUEUE_NAME | string | The Redis key to monitor for executor scaling |
| DEPLOYER_TYPE | string | The type of deployer to use, docker or aws-ecs |
| COLLECTOR_IMAGE | string | The Docker image for the Collector service |
| HEALTH_CHECKER_IMAGE | string | The Docker image for the Health Checker service |
| EXECUTOR_IMAGE | string | The Docker image for the executor service |
| SCALING_TYPE | string | The method to use for determining scaling, only watermark is available |
| COLLECTOR_CONTRACT | string | The contract size to monitor for scaling the collector |
AWS ECS specific variables
| VARIABLE | TYPE | DESCRIPTION |
|---|---|---|
| AWS_ACCESS_KEY_ID | string | The AWS access key ID |
| AWS_SECRET_ACCESS_KEY | string | The AWS secret access key |
| AWS_CLUSTER_ARN | string | The ARN for the cluster to deploy to |
| AWS_SERVICE_CPU_UNITS | int | The CPU units for each service deployment |
| AWS_SERVICE_MEMORY_MB | int | The amount of memory in MB to assign to each instance |
| AWS_SERVICE_SUBNETS | string array | The comma-delimited network subnets to deploy to, ex "subnet-123,subnet-124" |
| AWS_SERVICE_SECURITY_GROUPS | string array | The comma-delimited security groups to deploy to, ex "sg-123,sg124" |
Monitoring
Enable metrics export to DataDog by setting the environment variable
DATADOG_ENABLED to true.
The service will collect a bunch of metrics at the start of a new block and push it to DataDog, these include:
General
- Current height (Chain)
- Last processed height (Bot)
- Lag between current and last processed (should be zero)
Manager
- Count of running collectors
- Count of running health checkers
- Count of running executors
Collector
- Total amount of contract items (collateral + debts) (per block)
- Total amount of contract items scanned (collateral + debts) (per block)
- Total amount of contract items missed (collateral + debts) (per block)
Health checker
- Total amount of health checks to perform (per block)
- Total amount of health checks performed (per block)
- Total amount of health checks missed (per block)
- Total healthy positions (since last check)
- Total unhealthy positions (since last check)
Executor
- Total liquidations
- Total liquidations (per block)
- Total liquidations missed
DataDog required variables
| VARIABLE | TYPE | DESCRIPTION |
|---|---|---|
| DD_SITE | string | The DataDog domain to use, ex. us1.datadoghq.com |
| DD_API_KEY | string | Your DataDog API key |
Directories
¶
| Path | Synopsis |
|---|---|
|
deployer
Package deployer implements various deployment mechanisms of services
|
Package deployer implements various deployment mechanisms of services |
|
interfaces
Package interfaces defines the behaviour of service components
|
Package interfaces defines the behaviour of service components |
|
manager
Package manager implements the management of the liquidation bot services including automatic deployment and scaling of the collector, health-checker and liquidator
|
Package manager implements the management of the liquidation bot services including automatic deployment and scaling of the collector, health-checker and liquidator |
|
scaler
Package scaler implements the scaling of services based on externally defines parameters
|
Package scaler implements the scaling of services based on externally defines parameters |