sharkstore

module
v0.6.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 1, 2018 License: Apache-2.0

README

SHARKSTORE

中文版本

SharkStore is a distributed, persistent key-value store, whose database layer is based on the rocksdb and the replication works are based on raft.

Modules

  • master-server(golang) -- metadata server
  • data-server(c++) -- database server
  • gateway-server(golang) -- proxy
  • console(golang) -- web administration

Directory

|-- README.md
|-- console                 web administration tool
|-- data-server             business data storage service, where data is stored
|-- glide.yaml
|-- master-server           metadata service, where you get metadata
|-- model
|-- pkg-go                  common module
|-- proxy
    |-- gateway-server      sql/restful proxy

Architecture Design

See arch.md .

Install

See INSTALL.md .

Performance

We created a table named 'metric' witch 12 columns on 3 data servers with NVMe disk, and each of the columns is integer type. The first four columns(named salt, key, host, ts) are made a unified index together as the key. On the other hand, the table is pre-splitted to 100 ranges with 3 replicas running on raft.

  • Batch Insert

    100-row records inserted each time, 61,000,000,000 records inserted totally.

    average response time: 54ms, best tps: 800,000

  • Select

    • select one record by salt+key+host+ts

      average response time: 5ms

    • select 10-100 records by salt+key+host

      average response time: 50ms(before caching) VS 8ms(after caching)

Features

  • SQL syntax compatible and restful api supported
    Users can access by sql client directly, and also by restful api.

  • Dynamic table scheme
    Table columns is allowed to be added or renamed dynamically.

  • Pre-sharding
    Pre-sharding is supported when creating a new table.

  • Globally sorted data
    User can scan tables by primary key globally.

  • Strong consistency.
    Strong consistency is guaranteed by the data replication works on raft group.

  • Online scalability | Auto failover | Auto rebalance | Auto schedule

  • NVMe+SPDK Supported
    Under testing...

License

Under the Apache 2.0 license. See the LICENSE file for details.

Directories

Path Synopsis
console
auth
cmd command
common
config
controllers
models
*
 *
right
routers
*
 *
service
*
 *
master-server
cmd command
engine/boltstore
engine/errors
engine/example command
engine/model
http_reply
metric
mock
raft
raft/logger
raft/playground command
raft/proto
raft/storage
raft/storage/wal
raft/storage/wal/bench command
raft/test
raft/util
raftgroup
server
pkg/alarmpb
Package alarmpb is a generated protocol buffer package.
 Package alarmpb is a generated protocol buffer package.
pkg/aspb
Package aspb is a generated protocol buffer package.
 Package aspb is a generated protocol buffer package.
pkg/errorpb
Package errorpb is a generated protocol buffer package.
 Package errorpb is a generated protocol buffer package.
pkg/eventpb
Package eventpb is a generated protocol buffer package.
 Package eventpb is a generated protocol buffer package.
pkg/funcpb
Package funcpb is a generated protocol buffer package.
 Package funcpb is a generated protocol buffer package.
pkg/kvrpcpb
Package kvrpcpb is a generated protocol buffer package.
 Package kvrpcpb is a generated protocol buffer package.
pkg/lockpb
Package lockrpcpb is a generated protocol buffer package.
 Package lockrpcpb is a generated protocol buffer package.
pkg/metapb
Package metapb is a generated protocol buffer package.
 Package metapb is a generated protocol buffer package.
pkg/ms_raftcmdpb
Package ms_raftcmdpb is a generated protocol buffer package.
 Package ms_raftcmdpb is a generated protocol buffer package.
pkg/mspb
Package mspb is a generated protocol buffer package.
 Package mspb is a generated protocol buffer package.
pkg/raft_cmdpb
Package raft_cmdpb is a generated protocol buffer package.
 Package raft_cmdpb is a generated protocol buffer package.
pkg/redispb
Package redispb is a generated protocol buffer package.
 Package redispb is a generated protocol buffer package.
pkg/schpb
Package schpb is a generated protocol buffer package.
 Package schpb is a generated protocol buffer package.
pkg/statspb
Package statspb is a generated protocol buffer package.
 Package statspb is a generated protocol buffer package.
pkg/taskpb
Package taskpb is a generated protocol buffer package.
 Package taskpb is a generated protocol buffer package.
pkg/timestamp
Package timestamp is a generated protocol buffer package.
 Package timestamp is a generated protocol buffer package.
pkg/util
pkg-go
ds_client
gs_client
ms_client
util
proxy
benchmark/blob_store
benchmark/cmd command
gateway-server/cmd command
gateway-server/errors
gateway-server/mysql
Package query provides functions to transform queries.
 Package query provides functions to transform queries.
gateway-server/server
gateway-server/sqlparser
gateway-server/sqltypes
Package sqltypes implements interfaces and types that represent SQL values.
 Package sqltypes implements interfaces and types that represent SQL values.
gateway-server/yacc command
metric
store/dskv
store/dskv/mock_ds
store/dskv/mock_ms
store/localstore/boltdb
store/localstore/engine
store/localstore/goleveldb
boom command
boom/boomer
Package boomer provides commands to run load tests and display results.
 Package boomer provides commands to run load tests and display results.
client-ds command
Command hey is an HTTP load generator.
 Command hey is an HTTP load generator.
client-ds/requester
Package requester provides commands to run load tests and display results.
 Package requester provides commands to run load tests and display results.
cluster-repair command
alarm
apd
Package apd implements arbitrary-precision decimals.
 Package apd implements arbitrary-precision decimals.
assert
Package assert contains functions for making assertions in unit tests From github.com/docker/docker/pkg/testutil/assert
 Package assert contains functions for making assertions in unit tests From github.com/docker/docker/pkg/testutil/assert
bufalloc
bufio2
bytesize
compare
config
context
deepcopy
deepcopy deep copies maps, slices, etc.
 deepcopy deep copies maps, slices, etc.
duration
elastic_search
encoding
gogc
hack
hlc
Package hlc implements the Hybrid Logical Clock outlined in "Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases", available online at http://www.cse.buffalo.edu/tech-reports/2014-04.pdf.
 Package hlc implements the Hybrid Logical Clock outlined in "Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases", available online at http://www.cse.buffalo.edu/tech-reports/2014-04.pdf.
inet
log
lrucache
metric
metrics
randutil
safemap
server
structure/maps/treemap
Implements a map backed by llrb tree.
 Implements a map backed by llrb tree.
structure/trees/llrbtree
Implements Left-Leaning Red Black trees.
 Implements Left-Leaning Red Black trees.
sync2
sync2/atomic2
timesize
ttlcache
unsafe2

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.