gomap

TreeDB

TreeDB is a pre-alpha persistent storage engine built around a copy-on-write B+Tree, a persistent value log, and command-WAL recovery. It includes a native wire server, a Mongo-compatible gateway, a collection/document layer, secondary indexes, and benchmark tooling for comparing TreeDB against MongoDB and other engines.

TreeDB is the main focus of this repository. The repo also contains HashDB, an older mmap-backed hash engine used for experiments and comparison; see HashDB/README.md.

Status

TreeDB is pre-alpha:

• APIs and on-disk formats may change without backward-compatibility guarantees.
• New binaries may intentionally reject old DB directories.
• Benchmark DB directories should be rebuilt from scratch unless a report says otherwise.

Benchmark Highlights

These checked-in reports use different workloads, profiles, and caveats. Treat each workload as scoped evidence from its linked benchmark, not as one combined benchmark suite.

YCSB Server Workload

External go-ycsb, local loopback TCP, recordcount=100000, operationcount=10000, threadcount=16, BSON document format, and zero YCSB operation errors. Run rows use the median total-throughput repeat from the latest-main June 3 HST / June 4 UTC report.

Full report, commands, host context, run repeats, and artifact paths: June 3 latest-main YCSB report.

Indexed Collection Insert Workload

Two secondary indexes, June 17 UTC full benchmark refresh, 100000 documents, batch size 16000, and command_wal_relaxed for TreeDB. docs/sec is the timed insert measurement for the value-log outer-leaf layout. Compacted B/doc uses the byte-minimized exhaustive_compact row for TreeDB template-v1 and SQLite after VACUUM; TreeDB JSON is omitted from the README compacted-size headline until the canonical exhaustive fixture covers that format.

Source: June 17 full benchmark refresh.

Collection Read And Lookup Workload

Two secondary indexes, April 27 collection/SQLite matrix.

Source: April 27 collection/SQLite matrix.

Collection Concurrency Workload

The collection insert/read/lookup rows above are single benchmark-driver rows. For concurrent collection-layer reads and mixed read/write evidence, use the separate concurrency report. In the June 4 run, TreeDB template-v1 primary reads into a caller buffer measured 332.3 ns/op at GOMAXPROCS=12; mixed primary reads with one writer measured 2.96M reader ops/sec and 44.3k writer docs/sec.

Source: June 4 collection concurrency report.

Vector Search Serving Workload

Dated Tier S exact-FP32 no-document snapshot: Apple M3 (darwin/arm64), 2026-06-05, commit 2feb1f0e35459d1b3d044008203d0c8afcf5630f, 10000 documents, 64 dimensions, M=16, efConstruction=128, efSearch=128, topK=10, query stream length 16, BENCHTIME=1000x, and COUNT=3. TreeDB rows use warmed persisted column_graph / hnsw_search_pack_v1 no-document routes. USearch rows are pure in-memory external ANN baselines, not persistence-equivalent storage rows.

Source and reproduction workflow: TreeDB vs USearch vector benchmark workflow. API chooser, route guardrails, and runnable exact-only demo: high-QPS collection vector-search guide and cmd/treedb_vector_highqps_demo.

TreeDB Mongo Gateway Client-Shape Workload

Gateway-shaped BSON documents, 200000 documents, batch size 1000, 16 insert producers, two secondary indexes (email, city), TreeDB command_wal_relaxed, settled read state, and 16 concurrent readers for the read phases. This benchmark keeps TreeDB storage constant while changing the client/protocol boundary.

Direct collection API is the TreeDB collection/storage ceiling, not a Mongo-compatible protocol row. Mongo driver unacknowledged writes is not a durability-equivalent default. Use this table to compare TreeDB access-path overhead; compare against MongoDB only when client mode and acknowledgement semantics match.

Source: June 4 fast Mongo/native client-shape matrix.

Mongo API Full-Sweep Workload

June 17 UTC full benchmark refresh, Docker MongoDB 8 baseline, 100000 BSON documents, TreeDB Mongo gateway driver-command-raw, and command_wal_relaxed. The full-sweep load rows include the additional age_1 range index even when the displayed secondary-index count is 0.

Source: June 17 full benchmark refresh.

Vector Search External Snapshot

Local Apple M3 snapshot for 10k x 1536 vectors, topK=10, M=16, efConstruction=128, and efSearch=128. TreeDB rows use DB-demo no-document search; USearch is an in-memory library comparator; pgvector is a PostgreSQL server HNSW comparator.

USearch c=1/c=8 averages are from batch searches with threads=1/8, while TreeDB and pgvector report per-query latency samples. Source and caveats: June 8 vector external comparison.

TreeDB is also wired into snissn/vectordbbench through the document service. Those rows measure Python/client/HTTP/service overhead and are separate from native Go 0 B/op no-document evidence; see the TreeDB VectorDBBench runbook.

application.db Offline Density Workload

Offline compacted-size comparison from the June 2 Celestia application.db rerun.

Source: June 2 density rerun.

Indexed Text/Vector/Hybrid Insert And Search Workload

The #2589 optimized context rows use the #2564 fixture on an active Apple M3 laptop: 256 JSON documents, scalar indexes on tenant/region, a lexical title/body text index (new text indexes default to text-v2; v1 is explicit compatibility), and an exact cosine column graph (dims=16, M=8). The insert row times InsertBatch + Flush + RebuildVectorIndex; search rows build/index the fixture before timing and then time the search API call only. Original #2564 rows remain in the linked runbook for context; treat these as same-host/context evidence, not universal throughput claims.

The strict same-host M4 paired insert/readiness delta was B/op -28.2%, allocs/op -71.8%, and ns/op +0.8%. Remaining allocation floors include DB open/commitlog setup in profiles plus value-log append buffers, typed-column image builders, text analysis/posting construction, and vector rebuild state. Source, commands, artifact paths, guardrails, and caveats: TreeDB indexed insertion/search benchmark.

What TreeDB Provides

• Persistent B+Tree index with copy-on-write root publishing.
• Persistent value log for large values and leaf/value placement experiments.
• Command WAL for collection and raw-key redo/recovery.
• Snapshot-isolated readers and exclusive process-level DB directory locking.
• Collection/document APIs with BSON, template-v1, secondary indexes, and vector search experiments.
• Native wire protocol through cmd/treedb-native-server.
• Mongo-compatible gateway through TreeDB/mongo_gateway.
• Benchmark and profiling scripts for YCSB, collections, vector search, and storage-engine comparison.

Quickstart

Build the primary TreeDB servers:

mkdir -p bin
go build -o bin/treedb-native-server ./cmd/treedb-native-server
go build -o bin/treedb-mongo-gateway ./TreeDB/mongo_gateway/server.go

Run the native server:

./bin/treedb-native-server \
  -dir /tmp/treedb-native \
  -profile command_wal_durable \
  -addr 127.0.0.1:17130

Run the Mongo-compatible gateway:

./bin/treedb-mongo-gateway \
  -dir /tmp/treedb-mongo \
  -profile command_wal_durable \
  -document-format bson \
  -addr 127.0.0.1:27130

Minimal Go usage:

package main

import (
	"fmt"
	"log"

	treedb "github.com/snissn/gomap/TreeDB"
)

func main() {
	opts := treedb.OptionsFor(treedb.ProfileCommandWALDurable, "./my-db")
	db, err := treedb.Open(opts)
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()

	if err := db.Set([]byte("key"), []byte("value")); err != nil {
		log.Fatal(err)
	}
	value, err := db.Get([]byte("key"))
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(string(value))
}

Profiles

The current public TreeDB profile surface is intentionally small:

• command_wal_durable: recommended server default; command WAL enabled with durable sync/checksum settings.
• command_wal_relaxed: command WAL enabled with relaxed sync/read-integrity settings for high-throughput ingest and comparative benchmarks.
• bench: explicit no-WAL benchmark-only ceiling.

Legacy/raw profile names are retained only for compatibility and focused low-level tests. They should not be used as public server defaults.

More detail: docs/TREEDB_PROFILES.md and docs/TREEDB_WRITE_PATHS.md.

Benchmarking

Current YCSB status and rerun commands:

• docs/benchmarks/ycsb_mongodb_treedb_current.md
• docs/benchmarks/ycsb_latest_main_2026-06-03.md
• scripts/ycsb_compare_mongodb_treedb.sh

Collection, vector-search, and engine benchmark runbooks:

• docs/benchmarks/full_benchmark_report_2026-06-17.md
• docs/benchmarks/collections_insert_two_index_exhaustive_main_2026-06-04.md
• docs/benchmarks/mongo_gateway_fast_client_matrix_2026-06-04.md
• TreeDB/docs/guides/vector-search-high-qps-collection-api.md
• TreeDB/docs/guides/vector-search-benchmark-workflow.md
• cmd/treedb_vector_highqps_demo/README.md
• docs/benchmarks/treedb_canonical_benchmark_runbook.md
• docs/benchmarks/collections_canonical_benchmark.md
• cmd/unified_bench/README.md
• cmd/benchprof/README.md

Profile capture workflow:

OUT=$(mktemp -d /tmp/gomap_profiles_XXXXXX)
./bin/unified-bench ... -profile-dir "$OUT"
./bin/benchprof -profiles-dir "$OUT"

Documentation

• TreeDB canonical spec: TreeDB/docs/spec/README.md
• TreeDB guides: TreeDB/docs/guides/README.md
• TreeDB concepts: docs/TREEDB_CONCEPTS.md
• TreeDB storage format: docs/TREEDB_STORAGE_FORMAT.md
• TreeDB recovery: docs/TREEDB_RECOVERY.md
• TreeDB collection quickstart: docs/TREEDB_COLLECTION_QUICKSTART.md
• Contracts: docs/contracts/README.md
• Full docs index: docs/README.md

Repo Contents

• TreeDB/: TreeDB storage engine, collection layer, native APIs, command WAL, and Mongo gateway.
• cmd/treedb-native-server/: native wire server.
• TreeDB/mongo_gateway/: Mongo-compatible TreeDB gateway.
• cmd/unified_bench/: cross-engine benchmark harness.
• cmd/benchprof/: profile/result summarizer.
• HashDB/: mmap-backed hash-index engine used for experiments and comparison.

Testing

go test ./...
go test ./TreeDB/... ./cmd/treedb-native-server ./TreeDB/mongo_gateway

For large benchmark runs, prefer a fresh DB directory and record the exact commit, host, profile, command, and artifact path in the report.