leifdb

LeifDB

LeifDb a clustered K-V store application that implements Raft for consistency, in Go, based on the short Raft paper. It has an OpenAPIv2.0-compatible HTTP interface for client interaction, and serves the schema for the client interface at the root HTTP endpoint to allow clients to discover and use endpoints programatically.

The aim of this project is to build a distributed, consistent, fault-tolerant database along the lines of etcd, which backs Kubernetes; Consul, which backs Vault and other HashiCorp tools; or ZooKeeper, which backs most Hadoop-related projects. (etcd and Consul use Raft, ZooKeeper uses a similar algorithm called Zab, and there are others that use other algorithms such as Paxos)

Contributions are welcome! Check out the Contributing Guide for more info on how to make feature requests, submit bug reports, or create pull requests.

Install

This project requires Go 1.14.x and the swaggo/swag cli tool, and modifying some elements requires protobuf. If you do not have these installed, see the instructions in the Contributing Guide.

Build and run

The simplest way to build and test the application is to enter:

make

This will clean, build, and test the code (make tasks may not currently work on Windows without Windows Subsystem for Linux). To automatically run (after clean and build), use:

make run

To configure options (see Configuration):

env LEIFDB_RAFT_PORT=16991 make run

To build and run the app manually on Linux/Unix:

go clean
go build -tags=unit,mgmttest -o leifdb

On Windows:

go clean
go build -tags=unit,mgmttest -o leifdb.exe

Responses to client endpoints are string-formatted.

To manually run the test suite:

go test -tags=unit ./...
go test -tags=mgmttest ./...

To run the server with default parameters, do:

./leifdb

Or on Windows:

./leifdb.exe

UI

There's a very basic front end! It's capable to connecting to a server, and doing read/write/delete actions. Check out the readme in that directory for directions on installing and running it.

Endpoints

Database requests

After starting the server, you'll be able to interact with the database via the HTTP client interface. The Swagger/OpenAPIv2.0 schema describes the endpoints in the HTTP interface. First, start the server (assuming default HTTP port of 8080 for examples) with make run or by invoking the binary directly. Then, to get a copy of the Swagger JSON schema:

curl -i localhost:8080/

You can also view and interact with endpoints via the auto-generated Swagger API page. This has a section for each endpoint with descriptions, parameters, and an interactive query-runner to make it easy to manually test the server.

Adding or changing endpoints will probably require regenerating the Swagger schema file and related code. Make sure that the endpoint declarative comments are updated (see the swaggo/swag documentation for reference info on what options are available and their parameters). Running make after updating the comments and code will generate the new "swagger.[json|yaml]" files, or you can run swag init manually.

Once any changes are solidified, you will probably also need to update the UI subproject, since it uses code autogenerated from the Swagger schema.

CORS

CORS is enabled, and you can double-check to make sure that preflight requests are handled correctly by doing:

curl -X OPTIONS -D - -H 'Origin: http://foo.com' -H 'Access-Control-Request-Method: PUT' localhost:8080/db/testKey?value=something

Server should respond with roughly:

HTTP/1.1 200 OK
Access-Control-Allow-Methods: PUT
Access-Control-Allow-Origin: *
Vary: Origin
Vary: Access-Control-Request-Method
Vary: Access-Control-Request-Headers
Date: Fri, 05 Jun 2020 20:47:51 GMT
Content-Length: 0

Raft requests

Messages used for managing Raft state use protobuf. See test cases for examples of how to construct message bodies. For more info on creating valid values for fields, see the short Raft paper.

Configuration

HTTP interface

The HTTP interface is used for client interactions with the database. It can be specified using the LEIFDB_HTTP_PORT environment variable with an integer value. If no value is provided, port 8080 is used.

gPRC interface

The gRPC interface is used for interactions between members of the Raft cluster. It can be specified using the LEIFDB_RAFT_PORT environment variable with an integer value. If no value is provided, port 16990 is used.

Data directory

The persistent data directory is used for storing configuration files and non-volatile server state, and can be specified using the LEIFDB_DATA_DIR environment variable with a path. The path may point to a non-existent location, but cannot exactly match an existing file (an existing directory is fine). If no value is provided, "$HOME/.leifdb/<addr_hash>" is used, where "<addr_hash>" is a non-cryptographic hash of the gRPC interface for the server (such that configuration is consistent for a server as long as it is deployed with the same hostname or IP address and same port specified by LEIFDB_DATA_DIR)

Cluster configuration

In order to interact with other members of a raft cluster, each node must know the addresses for other members. Currently, this is not determined dynamically. In order to create a multi-node deployment, there must be two environment variables set:

• LEIFDB_MODE: must be "multi" (default is "single")
• LEIFDB_MEMBER_NODES: must be a comma-separated list of addresses for other nodes, such as "10.10.0.2:16990,10.10.0.3:16990,10.10.0.4:16990"

To run a cluster on one machine, make 3 directories named "/data/a", "/data/b", and "/data/c". Replace "10.10.0.x" with either "localhost" or your computer's preferred IP (can get it from ifconfig on Unix/Linux or ipconfig on Windows, or from an error message by running a server with the config file as written--better methods forthcoming). Then open three terminal windows and execute these in each:

env LEIFDB_DATA_DIR=/data/a \
  LEIFDB_MODE=multi \
  LEIFDB_MEMBER_NODES="localhost:16990,localhost:16991,localhost:16992" \
  LEIFDB_HOST=localhost \
  LEIFDB_HTTP_PORT=8080 \
  LEIFDB_RAFT_PORT=16990 \
  ./leifdb

env LEIFDB_DATA_DIR=/data/b \
  LEIFDB_MODE=multi \
  LEIFDB_MEMBER_NODES="localhost:16990,localhost:16991,localhost:16992" \
  LEIFDB_HOST=localhost \
  LEIFDB_HTTP_PORT=8081 \
  LEIFDB_RAFT_PORT=16991 \
  ./leifdb

env LEIFDB_DATA_DIR=/data/c \
  LEIFDB_MODE=multi \
  LEIFDB_MEMBER_NODES="localhost:16990,localhost:16991,localhost:16992" \
  LEIFDB_HOST=localhost \
  LEIFDB_HTTP_PORT=8082 \
  LEIFDB_RAFT_PORT=16992 \
  ./leifdb

(keep track of which window is which, since you'll need to figure out what the ports are for the one that becomes the leader, but you generally can't control which one it will be)

The output will have a lot of chatter (unless you change the log level in the init function in "main.go"), but should reach a steady state where one of the nodes is the leader. The leader node will be logging a stream of messages like:

2020-06-04T07:40:16-04:00 DBG Number needed for append: 2
2020-06-04T07:40:16-04:00 DBG Appended to 3 nodes
2020-06-04T07:40:16-04:00 DBG Need to apply message to 2 nodes
2020-06-04T07:40:16-04:00 DBG Checking for update to commit index commitIndex=1 lastIndex=1
2020-06-04T07:40:16-04:00 DBG Applying records to database lastApplied=1

Follower nodes will be streaming messages like:

2020-06-04T07:40:16-04:00 DBG apply commits current=1 leader=1
2020-06-04T07:40:16-04:00 DBG Received append request: term:105 leaderId:"192.168.1.21:16991" prevLogIndex:1 prevLogTerm:97 leaderCommit:1

Determine which ports correspond to the leader (let's say it's the one with an HTTP service bound to port 8080), then you can issue writes to the leader node, followed by reads to any node. See Database requests for writing read/write requests. [Note that this is not a final restriction--once redirect responses are implemented, the servers will handle redirecting writes to the leader and the user will be able to initiate a write with any node]

Todo

Raft basics (everything from the short Raft paper):

• add log comparison check to vote handler (election restriction)

Raft complete (additional functionality in the full Raft paper):

• log compaction
• changes in cluster membership

General application:

• Add scripts for starting a cluster / changing membership (probably something to the tune of Docker + Kubernetes + Terraform)
• Performance benchmarking (see the "Measurement" section of Paxos Made Live for a couple of ways to set up benchmarks) (also, compare performance with differing levels of debug logging turned on)

Prior art

Aside from the Raft papers themselves, here are some related resources:

• The Secret Lives of Data
• Eli Bendersky's blog post
• A talk on Raft from the Consul team
• Jay Kreps' article on Logs
• Distributed systems for fun and profit