bolt

NornicDB Bolt Protocol Server

Neo4j-compatible Bolt protocol server for NornicDB. Enables any Neo4j driver to connect to NornicDB without modifications.

✅ Implementation Status

Phase 1: Bolt Protocol Server - COMPLETE

• ✅ TCP Server & Protocol Handler
• ✅ PackStream Serialization (encoding & decoding)
• ✅ Message Handling (HELLO, RUN, PULL, DISCARD, BEGIN, COMMIT, ROLLBACK, RESET, GOODBYE)
• ✅ Authentication Handshake
• ✅ Session Management
• ✅ Result Streaming
• ✅ Comprehensive Unit Tests (2200+ lines)
• ✅ Integration Tests with Cypher Executor
• ✅ Stress Testing

Features

Protocol Support

• Bolt 4.x: Full support for Bolt 4.0, 4.1, 4.2, 4.3, 4.4
• PackStream: Complete binary serialization format
• Streaming: Efficient result streaming with PULL/DISCARD
• Transactions: BEGIN, COMMIT, ROLLBACK support
• Connection Pooling: Multiple concurrent connections

WebSocket transport

The Bolt port multiplexes four wire-level transports based on the first 5 bytes of every accepted connection — mirroring Neo4j's TransportSelectionHandler exactly:

How clients reach each branch:

• Official Neo4j drivers dial bolt:// / bolt+s:// (or the neo4j:// routing wrappers). The Node / JVM / Python builds produce raw TCP; the JS browser build produces a GET WebSocket upgrade from the same bolt:// URL.
• Third-party tools speaking raw WebSockets dial ws:// / wss:// directly and write Bolt frames into BinaryMessage payloads — same wire bytes the JS browser build produces internally.

A plain GET / (no Upgrade headers) returns a 200 OK discovery response — empty body when OAuth is not configured (Community parity), JSON describing the OAuth provider when it is.

WebSocket sessions speak the same Bolt wire format inside binary frames: same magic, same version negotiation, same chunked message framing.

TLS

Cert+key paths are read on every TLS handshake (the tls.Config.GetCertificate callback) and a 5-second background ticker re-loads them from disk. Operator update protocol: write to cert.pem.new then mv cert.pem.new cert.pem.

RequireTLS=true rejects every plaintext connection (raw or WS) with the canonical Neo4j error message. mTLS is opt-in via BoltTLSClientCAFile + a BoltTLSClientAuthMode enum (none / request / request_verify / require_verify).

Message Types

Response Messages

Usage

Starting the Server

Option 1: Command Line

# Build the server
cd cmd/nornicdb-bolt
go build

# Start with defaults (port 7687)
./nornicdb-bolt

# Start on custom port
./nornicdb-bolt -port 7688

# Custom data directory
./nornicdb-bolt -data ./mydata

Option 2: Programmatic

package main

import (
    "context"
    "github.com/orneryd/nornicdb/pkg/bolt"
    "github.com/orneryd/nornicdb/pkg/cypher"
    "github.com/orneryd/nornicdb/pkg/storage"
)

func main() {
    // Create storage
    store := storage.NewMemoryEngine()

    // Create Cypher executor
    cypherExec := cypher.NewStorageExecutor(store)

    // Wrap for Bolt
    executor := &MyBoltExecutor{cypher: cypherExec}

    // Configure server
    config := &bolt.Config{
        Port:            7687,
        MaxConnections:  100,
        ReadBufferSize:  8192,
        WriteBufferSize: 8192,
    }

    // Start server
    server := bolt.New(config, executor)
    if err := server.ListenAndServe(); err != nil {
        panic(err)
    }
}

// MyBoltExecutor implements bolt.QueryExecutor
type MyBoltExecutor struct {
    cypher *cypher.StorageExecutor
}

func (m *MyBoltExecutor) Execute(ctx context.Context, query string, params map[string]any) (*bolt.QueryResult, error) {
    result, err := m.cypher.Execute(ctx, query, params)
    if err != nil {
        return nil, err
    }
    return &bolt.QueryResult{
        Columns: result.Columns,
        Rows:    result.Rows,
    }, nil
}

Connecting with Neo4j Drivers

Python

from neo4j import GraphDatabase

# Connect to NornicDB
driver = GraphDatabase.driver("bolt://localhost:7687")

with driver.session() as session:
    # Create a node
    result = session.run(
        "CREATE (n:Person {name: $name, age: $age}) RETURN n",
        name="Alice",
        age=30
    )
    print(result.single()[0])

    # Query nodes
    result = session.run("MATCH (n:Person) RETURN n.name, n.age")
    for record in result:
        print(f"{record['n.name']}: {record['n.age']}")

driver.close()

JavaScript/TypeScript

const neo4j = require("neo4j-driver");

// Connect to NornicDB
const driver = neo4j.driver(
  "bolt://localhost:7687",
  neo4j.auth.basic("", ""), // Auth not required yet
);

const session = driver.session();

try {
  // Create a node
  const result = await session.run(
    "CREATE (n:Person {name: $name, age: $age}) RETURN n",
    { name: "Bob", age: 25 },
  );
  console.log(result.records[0].get("n"));

  // Query nodes
  const queryResult = await session.run("MATCH (n:Person) RETURN n");
  queryResult.records.forEach((record) => {
    console.log(record.get("n"));
  });
} finally {
  await session.close();
}

await driver.close();

Go

package main

import (
    "context"
    "fmt"
    "github.com/neo4j/neo4j-go-driver/v5/neo4j"
)

func main() {
    // Connect to NornicDB
    driver, err := neo4j.NewDriverWithContext(
        "bolt://localhost:7687",
        neo4j.NoAuth(),
    )
    if err != nil {
        panic(err)
    }
    defer driver.Close(context.Background())

    ctx := context.Background()
    session := driver.NewSession(ctx, neo4j.SessionConfig{})
    defer session.Close(ctx)

    // Create a node
    result, err := session.Run(ctx,
        "CREATE (n:Person {name: $name, age: $age}) RETURN n",
        map[string]any{"name": "Charlie", "age": 28},
    )
    if err != nil {
        panic(err)
    }

    if result.Next(ctx) {
        node := result.Record().Values[0]
        fmt.Printf("Created: %v\n", node)
    }

    // Query nodes
    result, _ = session.Run(ctx, "MATCH (n:Person) RETURN n", nil)
    for result.Next(ctx) {
        fmt.Println(result.Record().Values[0])
    }
}

Java

import org.neo4j.driver.*;

public class NornicDBExample {
    public static void main(String[] args) {
        // Connect to NornicDB
        Driver driver = GraphDatabase.driver(
            "bolt://localhost:7687",
            AuthTokens.none()
        );

        try (Session session = driver.session()) {
            // Create a node
            Result result = session.run(
                "CREATE (n:Person {name: $name, age: $age}) RETURN n",
                Values.parameters("name", "David", "age", 35)
            );
            System.out.println(result.single().get("n"));

            // Query nodes
            result = session.run("MATCH (n:Person) RETURN n");
            while (result.hasNext()) {
                System.out.println(result.next().get("n"));
            }
        }

        driver.close();
    }
}

Transaction Support

This example is atomic when the Bolt server is configured through NewWithDatabaseManager or a SessionExecutorFactory that returns a distinct TransactionalExecutor for each connection. A directly supplied TransactionalExecutor is supported only when MaxConnections is exactly one; cleanup failure or uncertain commit failure quarantines it against reuse. Multi-connection servers reject BEGIN for a shared raw executor. A plain QueryExecutor acknowledges transaction-control messages for protocol compatibility, but each RUN remains auto-committed and cannot be rolled back.

from neo4j import GraphDatabase

driver = GraphDatabase.driver("bolt://localhost:7687")

with driver.session() as session:
    # Explicit transaction
    tx = session.begin_transaction()

    try:
        tx.run("CREATE (n:Person {name: 'Eve'})")
        tx.run("CREATE (n:Person {name: 'Frank'})")
        tx.commit()
        print("Transaction committed")
    except Exception as e:
        tx.rollback()
        print(f"Transaction rolled back: {e}")

Architecture

┌─────────────────────────────────┐
│   Neo4j Driver (Any Language)   │
│  Python, JS, Go, Java, .NET...  │
└───────────────┬─────────────────┘
                │
                │ Bolt Protocol (TCP)
                │ PackStream Format
                │
┌───────────────▼─────────────────┐
│      Bolt Protocol Server        │
│  • Handshake & Authentication    │
│  • Session Management            │
│  • Message Routing               │
│  • Result Streaming              │
└───────────────┬─────────────────┘
                │
                │ QueryExecutor Interface
                │
┌───────────────▼─────────────────┐
│      Cypher Executor             │
│  • Query Parsing                 │
│  • Execution Planning            │
│  • Parameter Substitution        │
└───────────────┬─────────────────┘
                │
┌───────────────▼─────────────────┐
│      Storage Engine              │
│  • MemoryEngine (in-memory)      │
│  • BadgerEngine (persistent)     │
└─────────────────────────────────┘

Testing

Run Unit Tests

cd pkg/bolt
go test -v

Run Integration Tests

go test -v -run TestBoltCypherIntegration

Run Stress Tests

go test -v -run TestBoltServerStress

Test with Real Driver

# Terminal 1: Start server
cd cmd/nornicdb-bolt
go run main.go

# Terminal 2: Run Python test
pip install neo4j-driver
python3 << EOF
from neo4j import GraphDatabase
driver = GraphDatabase.driver("bolt://localhost:7687")
with driver.session() as session:
    result = session.run("CREATE (n:Test {id: 1}) RETURN n")
    print("Success:", result.single()[0])
driver.close()
EOF

Performance

Benchmarks

Why faster?

• In-memory storage (no disk I/O)
• Native Go implementation (no JVM overhead)
• Optimized PackStream encoding
• Efficient connection pooling

Scalability

• Concurrent connections: 100+ default, configurable up to 1000+
• Throughput: ~10K queries/sec on commodity hardware
• Memory: ~50MB base + ~1KB per connection
• Latency: P50: 0.5ms, P95: 2ms, P99: 5ms

Protocol Details

Handshake Flow

Client                              Server
  │                                   │
  ├─ Magic: 0x6060B017 ─────────────►│
  ├─ Versions: [4.4, 4.3, 4.2, 4.1] ─►│
  │                                   │
  │◄─────────── Selected: 4.4 ────────┤
  │                                   │
  ├─ HELLO {user_agent: ...} ────────►│
  │                                   │
  │◄─ SUCCESS {server: "NornicDB"} ───┤

Query Execution Flow

Client                              Server
  │                                   │
  ├─ RUN {query, params} ───────────►│
  │                                   │ Execute Query
  │◄─ SUCCESS {fields: [...]} ────────┤
  │                                   │
  ├─ PULL {n: 100} ─────────────────►│
  │                                   │ Stream Results
  │◄─ RECORD [row1] ──────────────────┤
  │◄─ RECORD [row2] ──────────────────┤
  │◄─ RECORD [row3] ──────────────────┤
  │◄─ SUCCESS {has_more: false} ──────┤

Transaction Flow

Client                              Server
  │                                   │
  ├─ BEGIN ─────────────────────────►│
  │◄─ SUCCESS ────────────────────────┤
  │                                   │
  ├─ RUN {query1} ──────────────────►│
  │◄─ SUCCESS ────────────────────────┤
  │                                   │
  ├─ RUN {query2} ──────────────────►│
  │◄─ SUCCESS ────────────────────────┤
  │                                   │
  ├─ COMMIT ─────────────────────────►│
  │◄─ SUCCESS ────────────────────────┤

BEGIN may include tx_timeout, a signed PackStream long measured in milliseconds, or null. Matching Neo4j 5.26, missing, null, zero, and negative values disable the client deadline; positive values beyond Go's duration range are accepted and saturated. The lifetime clock starts when a validated BEGIN is admitted to backend allocation, so allocation time, idle time, and active RUN work all count. If a backend accepts BEGIN after the deadline, BEGIN still succeeds, then NornicDB immediately owns rollback and the next transaction operation reports the timeout. Expiry cancels an active query, rolls the storage transaction back, and makes a later COMMIT fail with Neo4j's Neo.ClientError.Transaction.TransactionTimedOutClientConfiguration status. An absent, null, zero, or negative timeout does not schedule expiry.

Every explicit transaction has one terminal owner. COMMIT stops a pending deadline before entering storage; timeout, ROLLBACK, RESET, GOODBYE, and connection loss otherwise use the same exactly-once rollback path. Cleanup uses a five-second request context independent of connection cancellation. The session lifecycle owns operation admission for every supported per-session TransactionalExecutor. If a timeout cancels an active RUN or expires during a deferred explicit-transaction result flush, that operation performs the pending Badger/WAL rollback before the session responds or processes another message. RESET and connection teardown wait for that completion handoff. An admitted backend that does not honor the request context remains synchronously owned rather than being abandoned in a leaked cleanup goroutine, so the five seconds is cooperative rather than a hard wall-clock bound for such an implementation. If rollback errors or panics, or a commit returns an uncertain error, NornicDB does not claim that storage was released: it closes the connection, suppresses deferred flush, and quarantines a directly supplied single-connection executor. Timeout responses wait for owned cleanup; cleanup failure closes the connection instead of exposing a reusable timeout state. A deferred PULL/DISCARD flush error marks the explicit transaction failed until RESET rolls it back. Explicit terminal paths discard unconsumed result state, and CommitTransaction alone owns successful commit durability—rolled-back or committed writes are never sent through a later legacy flush. Idle expiry performs rollback before invoking its completion diagnostic; the separate cleanup-request diagnostic is emitted only for an active executor operation handoff, so an unresponsive logging callback cannot delay idle cleanup.

Compatibility

Supported Drivers

Known Limitations

1. No User Authentication: Currently accepts all connections (Phase 2)
2. No Cluster Routing: Single-node only (future enhancement)
3. No Streaming Large Results: Buffered in memory (optimization needed)

Roadmap

Completed ✅

• Bolt 4.x protocol implementation
• PackStream serialization
• Message handling (all types)
• Session management
• Result streaming
• Unit tests (2200+ lines)
• Integration tests
• Stress tests
• Command-line server
• Atomic explicit transactions with timeout and disconnect cleanup through the database-manager or per-connection TransactionalExecutor path

In Progress 🔄

• Schema management (constraints, indexes) - See Phase 2
• Built-in procedures (vector, fulltext, apoc) - See Phase 3

Planned 📋

• User authentication and RBAC
• TLS/SSL support
• Connection pooling optimizations
• Large result streaming (chunked)
• Query result caching
• Performance monitoring
• Cluster mode support

Troubleshooting

Connection Refused

# Check if server is running
lsof -i :7687

# Start server if not running
cd cmd/nornicdb-bolt
go run main.go

Driver Compatibility Issues

# Use latest driver version
pip install --upgrade neo4j-driver

# Verify connection
from neo4j import GraphDatabase
driver = GraphDatabase.driver("bolt://localhost:7687")
driver.verify_connectivity()

cypher-shell compatibility override

cypher-shell may reject a Bolt connection if the Bolt HELLO success metadata does not advertise a Neo4j server string. If that is the only blocker, opt into the announcement override:

export NORNICDB_BOLT_SERVER_ANNOUNCEMENT="Neo4j/5.26.0"
./nornicdb serve
cypher-shell -a bolt://localhost:7687 -u neo4j -p password

This changes only the announced Bolt server string. Leave it unset unless you need strict-client compatibility.

Performance Issues

// Increase connection pool size
config := &bolt.Config{
    Port:           7687,
    MaxConnections: 500,  // Increase from 100
}

Memory Issues

# Monitor memory usage
ps aux | grep nornicdb-bolt

# Reduce max connections if needed
./nornicdb-bolt -maxconn 50

Contributing

See IMPLEMENTATION_PLAN.md for the full development roadmap.

Running Tests

# All tests
go test ./pkg/bolt/...

# Verbose
go test -v ./pkg/bolt/...

# With coverage
go test -cover ./pkg/bolt/...

# Specific test
go test -run TestBoltCypherIntegration ./pkg/bolt/...

License

MIT License - See LICENSE for details.

Status: ✅ Phase 1 Complete - Ready for Phase 2 (Schema Management)
Last Updated: November 25, 2025
Version: 1.0.0