db/

Introduction

Hydra is a modular peer-to-peer data store with block-dag data structures:

• Advanced Structures: block-graph SQL, Git, Graph DB, Files, k/v...
• Cross-Platform: supports web-browser, native process, mobile, embedded...
• Data-exchange: leverage optimized data transfer paths between volumes.
• Encryption: transformations can encrypt or compress data at rest.
• Identity: each volume has an identity independent from the host.
• Replication: bucket policies implement data replication behaviors.

Stores peer-to-peer data structures (similar to IPLD) on pluggable storage backends like bbolt, Valkey, browser storage and more. Any K/V store can be used as a storage backend.

Overview

Hydra is built on the ControllerBus framework, which defines the Config, Controller, Directive primitives.

It uses the Bifrost network engine for communication between peers.

The core storage system is implemented as:

• Volume: common storage volume interface for persistence of data.
• Block: a chunk of data hashed and identified by content-ID (see: IPFS).
• Bucket: collection of blocks with attached data management policies.
• Object: a pointer to a Block DAG located in a Bucket with transforms.

The block provides a Cursor for reading and modifying block DAGs. Most data structures have block-DAG / block-cursor implementations:

• Bitset: bitset backed with a uint64 array.
• Blob: split a large piece of data into deterministic chunks.
• Bloom: bloom filter for efficient presence checking.
• Fibheap: efficient min() queries on a k/v heap.
• Kvtx: transactional key/value store (i.e. AVL tree).
• Msgpack: blob encoded with the Msgpack protocol.

The following high-level data structures are implemented:

• File: collection of written Ranges composed of Blobs of data.
• Git: code revision tracking engine with go-git.
• Graph: graph database w/ quads: <subject, predicate, object, value>
• Sql: SQL data store backed by go-mysql-server.
• UnixFS: directories, files, permissions, FUSE mounts.
• World: key/value store coupled with a graph database + changelog.

For more details, see the design overview.

Volumes

Hydra assigns a persistent public/private keypair and peer ID to volumes.

The following volume types are currently implemented in this repository:

• Block: nested volume backed by a peer-to-peer block graph.
• BoltDB: embedded key/value data store, using bbolt.
• Browser: append-only payload log and index on OPFS or IndexedDB in the web browser.
• In-memory: in-memory key/value store for temporary data.
• RPC: access a Volume on a remote Bus via a RPC service.
• Redis: key/value storage with a remote Redis database.
• s4db: single-file on-disk key/value store, the native default.
• World: nested volume backed by Object in a shared World.

The volume controller accepts any implementation of the Store interface.

Any key/value store can be used as a Volume by implementing the kvtx interface and constructing the volume controller with the kvtx-backed Store. In-memory key-value volume is an example of this approach.

Volumes can be nested: the Block volume uses the kvtx/block key/value store to create a shared Volume, which can be encrypted or compressed by adding a transform config. The World volume stores data in a world.

OPFS Chrome Tests

The OPFS Chrome tests run the WASM storage stack inside Chromium workers. They are disabled by default because they require a browser runtime.

# Full Chrome OPFS suite.
bun run test:go:db:opfs:chrome

# Small CI smoke profile.
bun run test:go:db:opfs:chrome:smoke

# Heavier manual stress profile.
bun run test:go:db:opfs:chrome:stress

The smoke profile is intended for routine automation. The stress profile covers Web Lock behavior, worker termination, the browser volume across worker restarts, and world crash recovery.

Examples

Hydra can be used as either a Go library or a command-line / daemon.

GO111MODULE=on go install -v github.com/s4wave/spacewave/db/cmd/hydra

Access help by adding the "-h" tag or running "hydra help."

Launch the daemon (uses hydra_daemon.yaml on default):

hydra daemon

The full list of available daemon CLI flags is currently:

OPTIONS:
   --s4db value [ --s4db value ]                        set a path to an s4db file to load on startup [$HYDRA_S4DB]
   --s4db-verbose                                       if set, mark s4db databases as verbose (default: false) [$HYDRA_S4DB_VERBOSE]
   --redis-url value                                    set a url to a redis instance to connect to on startup [$HYDRA_REDIS_URL]
   --inmem-db                                           if set, start a in-memory volume on startup (default: false) [$HYDRA_INMEM_DB]
   --inmem-db-verbose                                   if set, mark inmem database as verbose. implies --inmem-db (default: false) [$HYDRA_INMEM_DB_VERBOSE]
   --node-priv value                                    path to node private key, will be generated if doesn't exist (default: "daemon_node_priv.pem")
   --api-listen value                                   if set, will listen on address for API connections, ex :5110 (default: ":5110")
   --prof-listen value                                  if set, debug profiler will be hosted on the port, ex :8080
   --config value, -c value                             path to configuration yaml file (default: "hydra_daemon.yaml") [$HYDRA_CONFIG]
   --write-config                                       write the daemon config file on startup (default: false) [$HYDRA_WRITE_CONFIG]
   --hold-open-links                                    if set, hold open links without an inactivity timeout (default: false) [$BIFROST_HOLD_OPEN_LINKS]
   --websocket-listen value                             if set, will listen on address for websocket connections, ex :5111 [$BIFROST_WS_LISTEN]
   --udp-listen value                                   if set, will listen on address for udp connections, ex :5112 [$BIFROST_UDP_LISTEN]
   --establish-peers value [ --establish-peers value ]  if set, request establish links to list of peer ids [$BIFROST_ESTABLISH_PEERS]
   --udp-peers value [ --udp-peers value ]              list of peer-id@address known UDP peers [$BIFROST_UDP_PEERS]
   --websocket-peers value [ --websocket-peers value ]  list of peer-id@address known WebSocket peers [$BIFROST_WS_PEERS]
   --pubsub value                                       if set, will configure pubsub from options: [floodsub] [$BIFROST_PUBSUB]

If --write-config is set, the options configured on the CLI will be written to the config YAML file: hydra_daemon.yaml on default.

The CLI arguments are provided for convenience: the YAML configuration format allows an infinite number of concurrent controllers to be configured: for example, several backing volumes, network transports, and app controllers.

APIs and Client CLI

The client CLI has the following help output:

USAGE:
   hydra client command [command options] [arguments...]

COMMANDS:
   block                 volume bucket handle block sub-commands
   bucket                bucket store sub-commands
   object                object store sub-commands
   volume                volume sub-commands
   controller-bus, cbus  ControllerBus system sub-commands.
   bifrost               Bifrost network-router sub-commands.

This is an example of configuring a bucket and storing a block:

  cat > /tmp/basic-bucket.json <<'JSON'
  {"id":"bucket-basic-1","version":1}
  JSON

  ./hydra client bucket config -f /tmp/basic-bucket.json --volume-regex ".*"

  echo "hello world" | ./hydra client block \
    --bucket-id bucket-basic-1 \
    --volume-id default \
    put -f "-"

  # The data hash is printed:
  # 2W1M3RQW6kLcw6kLCNWw9mA1pWRqGGFv9NxmjXNjjWjj6iLVLJM4

  # Now we can lookup the block 
  ./hydra client block \
    --bucket-id bucket-basic-1 \
    get --ref 2W1M3RQW6kLcw6kLCNWw9mA1pWRqGGFv9NxmjXNjjWjj6iLVLJM4

To store data into the key/value store:

  ./hydra client object \
    --store-id store-basic-1 \
    --volume-id default \
    put --key "test" -f cmd_client.go
  ./hydra client object \
    --store-id store-basic-1 \
    --volume-id default \
    get --key test

Demonstration of data exchange between two peers:

  # configure the lookup via pubsub
  hydra client bucket config -f ../../examples/bucket-configs/psecho-1.json  --volume-regex ".*"
  
  # put a block on one peer
  echo "hello world 123" | hydra client block \
    --bucket-id bucket-psecho-1 \
    --volume-id default \
    put -f "-"

  # on the other peer
  hydra client block \
    --bucket-id bucket-psecho-1 \
    get --ref 2W1M3RQWBWZxSFDV91oXXsVay12Nho1K4dvnVNZjkoCzR8Gix5xr

See the Bifrost docs for how to configure the peers to connect to each other.

For a daemon status output, use hydra client cbus bus-info:

✓ controller-bus running
Controllers:
        controllerbus/loader 0.0.1
        controllerbus/resolver/static 0.0.1
        controllerbus/configset 0.0.1
        hydra/daemon/api 0.0.1
        bifrost/transport/udp 0.0.1
        hydra/volume/s4db 0.0.1
        hydra/world/block/engine 0.0.1
        bifrost/floodsub 0.0.1
        hydra/dex/psecho 0.0.1
        hydra/lookup/concurrent 0.0.1
[...]

YAML Configuration File

The ConfigSet YAML format is defined by ControllerBus for specifying controllers to load and run concurrently with associated configurations.

For example:

# In the below example, "my-volume" is the unique ConfigSet controller ID.
# If multiple ConfigSet are applied with the same ID, the config with the highest revision will be used.

# Starts an s4db database at a path.
my-volume:
  id: hydra/volume/s4db
  config:
    path: data.s4wave
    volumeConfig:
      volumeIdAlias: ["default"]
    verbose: true

# Starts the floodsub implementation of pub-sub.
pubsub:
  id: bifrost/floodsub
  config: {}

# Listen for incoming UDP connections (w/ Quic) on port 5112
udp:
  id: bifrost/udp
  config:
    dialers: {}
    listenAddr: :5112

# Create a simple storage bucket on startup.
# Add it to all loaded volumes.
create-mybucket:
  id: hydra/bucket/setup
  config:
    applyBucketConfigs:
    - volumeIdList:
      - "default"
      config:
        id: example-bucket-1
        version: 1

# Configure the "psecho" data-exchange controller.
# Serves & fetches data lookup searches over a pub-sub channel.
# Nodes will connect directly to each other to transfer data.
dex:
  id: hydra/dex/psecho
  config:
    bucketId: example-bucket-1
    pubsubChannel: example-psecho-1-ch

# Create an example data structure: load a Hydra "World Engine"
# The world stores a k/v tree of Objects with a Graph DB.
world-example:
  id: hydra/world/block/engine
  config:
    engineId: example-1
    bucketId: example-bucket-1

Related Projects

The following Aperture Robotics components are dependencies, and their clients are included in the client bundle:

• ControllerBus: similar to microservices - communicating controllers.
• Bifrost: networking components and engine built with ControllerBus.

Testing

The "testbed" package provides a standard cross-platform ephemeral test setup with in-memory storage and a reasonable set of default controllers loaded.

It is used across the Hydra project to write end-to-end unit tests:

// TestWorld performs a simple test of operations against world.
func TestWorld(t *testing.T) {
	ctx := context.Background()
	log := logrus.New()
	log.SetLevel(logrus.DebugLevel)
	le := logrus.NewEntry(log)

	// Construct a new testbed
	tb, err := testbed.NewTestbed(ctx, le)

	// Construct a new storage cursor
	ocs, err := tb.BuildEmptyCursor(ctx)
	defer ocs.Release()

	// Construct a block transaction
	btx, bcs := ocs.BuildTransaction(nil)
	bcs.SetBlock(NewExampleBlock())
	// can use bcs.SetRef ....

	// Write the block structure to storage.
	rootRef, bcs, err = obtx.Write(ctx, true)
}

You can run the tests with go test ./...

Build Targets

Hydra compiles for the native platform and for WebAssembly. Go build constraints select the code for each target:

• js (GOOS=js GOARCH=wasm): WebAssembly in the browser, using syscall/js for browser APIs. The browser volume and the rest of the browser storage stack build here. The daemon, CLI, and bolt-backed stores are marked !js && !wasip1.
• tinygo: defined by the TinyGo compiler, which produces a smaller browser WASM binary. Files marked js && tinygo under opfs and util/jsbuf carry the TinyGo variants of the browser bindings, with js && !tinygo counterparts for the standard Go compiler.
• goscript: defined by the GoScript compiler, which translates Go to TypeScript and loads packages as GOOS=js GOARCH=wasm. The -goscript.go files, for example in store/kvtx and kvtx/block/okra, substitute implementations for that runtime.
• wasip1 (GOOS=wasip1): WASI runtimes. Files marked !js && !wasip1 are excluded from WASI builds, js || wasip1 files share the browser fallbacks, and the world-engine lease provider falls back to its in-memory implementation (world-engine-lease-fallback.go).

go test ./... builds and tests only the native target. In CI, bun run test:go:db runs the native tests, GOOS=js GOARCH=wasm go vet ./db/block/gc/... type-checks the block garbage collector for the browser, the OPFS Chrome suites run the js/wasm storage stack in Chromium, and the goscript CI member compiles and tests core/space/world/blocktype and object/peer with GoScript. The nightly release-WASM matrix builds the full browser bundle with the standard Go compiler, TinyGo, and GoScript. No CI job builds the wasip1 target.