sop

SOP Data & Compute Platform

SOP is a Data & Compute Platform for high-performance applications and distributed storage workflows.

At its core, SOP combines an embedded storage engine, distributed coordination model, and AI/runtime tooling. It supports scaling from a single device to clustered deployments that share storage and coordination state.

Quickstart

The fastest way to see SOP work is the in-memory B-Tree. No servers, no config:

go run ./examples/quickstart

New to SOP? Read What is SOP, in plain words.

Every commit runs the delivery pipeline: build, tests, container packaging (GHCR), a staging smoke test, then a human-approved promotion to production.

The SOP Ecosystem

Packed inside the library is everything you need to build next-generation distributed systems:

• Swarm Computing Engine: A framework for distributed coordination, allowing applications to act as coherent parts of a greater whole.
• Polyglot Storage Engine: An ACID-compliant B-Tree storage system with Caching and Erasure Coding, optimized for performance. Read the Architecture Whitepaper.
• AI Scripting & Computing Engine: A versatile runtime that allows for creating intelligent, self-correcting workflows. See Platform Tools & Relational Intelligence.

Installation & Distribution

SOP is designed to be accessible regardless of your preferred technology stack. There are two primary ways to get the SOP Platform Suite, which includes both the SOP Code Library and the Platform Tools (Data Manager, Script VDE, AI Copilot).

Option 1: Language Packages

Best for developers who want to integrate SOP directly into their application code. Installing the package for your language automatically includes the Data Manager (sop-httpserver) and CLI tools.

Language	Installation	Description
Python	pip install sop4py	Full Python bindings with Data Manager & AI Scripts.
Go	go get github.com/sharedcode/sop	The core native library for maximum performance.
C#	dotnet add package Sop	Complete .NET Core integration.
Java	Maven/Gradle	(Coming Soon) Full JVM support.
Rust	cargo add sop	(Coming Soon) High-performance Rust bindings.

AI Copilot Configuration

The SOP Data Manager includes an embedded AI Copilot powered by LLMs (like Google Gemini). To use the Copilot features, you must configure your API key.

Setting the API Key

You can provide the API Key via the Setup Wizard (on first launch) or the configuration file. For a detailed guide on using the conversational interface, case sensitivity rules, and query examples, please refer to the AI Copilot User Guide.

Config File Add the generator configuration to your config.json file.

{
  "generator": {
    "provider": "gemini",
    "api_key": "your-api-key-here",
    "model": "gemini-2.0-flash-exp"
  },
  "port": 8080
}

Note: The Copilot requires an active internet connection to communicate with the LLM provider.

Option 2: Standalone Binary (GitHub Releases)

Best for infrastructure administrators, DevOps, or "Data-First" users who want to set up the management console immediately. You can download the sop-httpserver executable directly from our GitHub Releases Page.

• Universal Manager: The standalone binary acts as a central console to manage databases created by any language binding (Python, C#, Go, etc.).
• Zero-Dependency: No language runtimes (Python/DotNet) required to run the tool itself.
• Complete Bundle: The release archive includes not just the server binary, but also the native binaries for language bindings and essential documentation (READMEs), making it a self-contained "SOP Starter Kit".

---

Development Workflows: Code-First vs. Data-First

SOP supports both code-first and data-first workflows.

• Code-first: Define stores and logic in application code, then inspect and manage them with the Data Manager.
• Data-first: Create stores and seed data in the Data Manager, then open those stores from application code.
• Further Reading: See Workflows Guide for the detailed workflow patterns and deployment scenarios.

SOP Data Manager & AI Suite

SOP allows you to interact with your data using the SOP Data Manager—a web-based console that includes a SQL-like query engine and an AI Copilot.

Data Manager Capabilities

• Visual Management: Inspect B-Trees, manage Stores (Key-Value, Vector, Model), and explore the System DB.
• Recent Progress: The HTTP server now supports signed bearer access tokens for browser and remote API clients, plus store-management fixes that preserve runtime schema metadata and reduce false structural-change validation.
• Environment Manager: Switch between environments using portable JSON configuration files.
• Shared Intelligence: Manage permissions and connections to share databases across the network, allowing different teams to collaborate on the same "System Knowledge" base.

Designing for AI: Relations

SOP is not just a storage engine; it is designed to be the "Long Term Memory" for AI Agents. There are two ways to model relationships so that AI Agents can intuitively navigate your data:

1. Direct Relations (Metadata): Use this for standard One-to-Many relationships (e.g., Order.user_id -> User.id). By registering this relationship in the StoreOptions, the AI understands the detailed schema and can perform high-performance Joins automatically.
   • Pro Tip: This enables Bi-Directional Querying (Parent $\leftrightarrow$ Child) without the need for redundant "Link Stores" or double-indexing. Query "User's Orders" or "Order's User" with equal efficiency.
2. Link Store Pattern (Advanced): Use this for Many-to-Many relationships or complex graph traversals. Create a dedicated Link Store (e.g., User_Orders) to map IDs effectively without modifying the base tables.

This structure allows AI Agents to navigate data using simple "Chain of Thought" reasoning steps (e.g., "First find the User ID, then look up their Orders") rather than struggling to generate complex SQL Joins. The SOP Data Manager provides first-class support for visualizing and debugging these relationships.

• SQL Capabilities: Perform familiar SQL operations directly on your NoSQL B-Trees:
  • SELECT / SCAN: Filter data using rich criteria ($gt, $regex, $in).
  • JOIN: Perform high-performance connections between stores (e.g., Join 'Users' and 'Orders').
  • CRUD: Insert, Update, and Delete records via a query interface.

Storage Distribution & Redundancy

SOP separates storage responsibilities into registry/system data and user-data files.

• Registry and system data: Configured through StoresFolders with active/passive redundancy.
• User data files: Configured through ErasureConfigs for striping, parity, and store-level routing.
• Further Reading: See Configuration & Tuning Guide and Operations Guide for the full storage and failover configuration model.

The System Database (SystemDB)

All SOP environments come with a built-in SystemDB. It stores platform metadata and runtime artifacts such as:

• Scripts: Your automation workflows and compiled functions.
• LLM Knowledge: Domain knowledge and operational guidance used by the AI runtime.
• (Future) RBAC: Role-Based Access Control configurations for multi-user security.

Spaces & Active Memory

SOP uses Spaces as isolated semantic working sets rather than one shared monolithic vector space.

• Domain Scope: Spaces let the runtime keep memory, retrieval, and enrichment focused on a specific domain.
• Stateless Backend (BYOK): Credentials are supplied per request so the backend can remain stateless.
• Further Reading: See AI Copilot & Agent Architecture and SOP AI Kit for the detailed memory and Space model.

Partitioned Vector Search

SOP supports partitioned vector search using B-Tree-backed storage layouts that keep related vectors adjacent and allow metadata-aware scans.

For the detailed storage layout, optimization model, and vector-search tradeoffs, see SOP AI Kit, Vector Store Design, and Dynamic Vector Store Design.

Rich Key Structures (Metadata Carrier)

SOP supports complex B-Tree keys that carry metadata such as status, version, category, or routing fields.

This pattern is used across general-purpose stores and the AI/vector layers to support filtering and structural operations with minimal value reads. See Architecture Guide, SOP AI Kit, and Vector Store Design for the detailed key layouts.

AI Copilot & Scripts

The Data Manager includes an integrated AI Copilot that supports Natural Language Programming.

• Natural Language Queries: Ask for selections, joins, and CRUD operations in plain language.
• Script Drafting and Execution: Create, save, and run reusable workflows from the UI or command surface.
• Hybrid Execution: Deterministic tool steps and reasoning steps can be combined in the same workflow.
• Documentation: See AI Copilot & Agent Architecture, AI Script Architecture, and Store Orchestration Modes for runtime internals and orchestration details.

To launch the Data Manager:

# Data Manager is included in your language binding installation
sop-httpserver

Security & Enterprise Integration

SOP supports both local built-in auth and external identity-provider integration for enterprise deployments.

• Built-in auth: the embedded HTTP server supports signed bearer access tokens, short-lived refresh renewal, and a dedicated signing secret for the hot path.
• Enterprise directory support: the auth layer is designed to accept provider adapters for AD, Entra ID, LDAP, SAML, or OIDC without changing the core application logic.
• Secret handling: keep SOP_SESSION_SECRET (or session_secret) in a secret manager or environment variable; do not bake signing keys into source control.
• Remote clients: bearer tokens are suitable for CLI tools, scripts, and microservices that call the HTTP API directly over HTTPS.
• Deployment guidance: use HTTPS in front of the HTTP server, restrict token scopes/roles, and map external identities to internal roles before granting access.

This gives IT and security teams a clear path to adopt SOP in controlled environments while keeping the runtime flexible for future provider integrations.

Reference Guides

• AI Copilot & Agent Architecture - Canonical reference for the Copilot runtime, grounding model, and provider execution loops.
• Architecture Guide - Core engine, storage flow, replication, and backend design.
• Scalability & Limits - Capacity math and architectural scaling model.
• Operations Guide - Failover, recovery, and operational practices.
• Workflows Guide - Deployment and usage patterns from standalone to clustered environments.

🚀 Getting Started

Download & Installation Guide: The fastest way to get up and running with SOP.

⚡ Performance

SOP is designed for high throughput and low latency. Below are benchmark results running on a 2015 MacBook Pro (Dual-Core Intel Core i5, 8GB RAM) using the built-in benchmark tool (tools/benchmark).

Optimization Guide: Tuning SlotLength

The SlotLength parameter controls the number of items stored in each B-Tree node. Tuning this value can significantly impact performance depending on your dataset size and workload.

Configuration Used:

• IsValueDataInNodeSegment: true (Values stored directly in leaf nodes)
• CacheType: sop.InMemory (Persisted to disk with ACID support)

10,000 Items Benchmark

For smaller datasets, a SlotLength of 2,000 offers the best balance.

SlotLength	Insert (ops/sec)	Read (ops/sec)	Delete (ops/sec)
1,000	107,652	136,754	40,964
2,000	132,901	142,907	50,093
3,000	135,066	137,035	49,754
4,000	123,190	122,228	48,094
5,000	132,670	126,432	47,150

100,000 Items Benchmark

For larger datasets, increasing SlotLength to 4,000 yields higher throughput.

SlotLength	Insert (ops/sec)	Read (ops/sec)	Delete (ops/sec)
1,000	121,139	145,195	48,346
2,000	132,805	136,684	51,817
3,000	137,296	141,764	50,605
4,000	145,417	143,770	51,988
5,000	137,054	144,565	50,309

Recommendation: Start with a SlotLength of 2,000 for general use, and increase to 4,000+ for write-heavy workloads with large datasets.

Pro Tip for Massive Scale: For datasets reaching into the hundreds of billions or trillions of records, you can increase SlotLength up to 20,000. This maximizes node density, allowing a single B-Tree to manage massive amounts of data (calculated at 68% average load):

Hash Mod Value	Segment Size	Capacity (Key/Value Pairs)
750,000 (Max)	~3 GB	673,200,000,000 (673.2 Billion)

This enables managing petabytes of data with minimal metadata overhead. See Scalability & Limits for the full breakdown.

Why this matters

These benchmarks are running with Full ACID Transaction protection. Unlike simple Key-Value stores that optimize purely for random writes (often sacrificing order or safety), SOP provides a robust foundation for complex data access:

• Sorted Data: Native support for ORDER BY ASC/DESC.
• Fast Search: Efficient range scans and lookups.
• SQL-Ready: The ordered structure supports efficient Merge Joins and complex query patterns out of the box.
• Linear Scalability: SOP is built on SWARM (SOP exclusive) transaction technology, allowing it to scale linearly and horizontally across the network as nodes and hardware are added.

Competitive Landscape

How does SOP compare to other storage engines in the Go ecosystem?

Database	Type	Typical Batch Write (Ops/Sec)	Key Differences
SOP	B-Tree	~145,000	ACID, Ordered, SWARM Scalability
BadgerDB	LSM Tree	~150,000 - 300,000	Faster writes (LSM), but requires compaction and lacks native B-Tree ordering features.
SQLite	B-Tree (C)	~50,000 - 100,000	Slower due to SQL parsing overhead.
BoltDB	B-Tree	~10,000 - 50,000	Slower random writes due to copy-on-write page structure.
Go map	Hash Map	~10,000,000+	In-memory only. No persistence, no ACID, no ordering.

Note on SOP's Unique Value Proposition: While raw speed is comparable to top-tier engines, SOP distinguishes itself by combining features that usually don't exist together:

• Full ACID Transactions: Guarantees safety where others might trade it for speed.
• SWARM Technology: Unlike monolithic engines, SOP scales linearly across the network.
• SQL-Ready Structure: Data is stored ordered, enabling ORDER BY, range scans, and efficient joins without extra indexing overhead.

Interoperability & Data Management

SOP is designed as a Universal Data Platform. Whether you are writing in Go, Python, Java, C#, or Rust, your data should be accessible, manageable, and interoperable.

1. The Universal Approach (JSONDB)

This is the standard approach used by all language bindings (Python, Java, C#, Rust) and is also available to Go developers via the jsondb package.

• Mechanism: Data is serialized as JSON.
• Benefit: Native Data Manager Support. Since the format is standardized, the SOP Data Manager can automatically read, write, query, and visualize your B-Trees without any extra configuration.
• Interoperability: A B-Tree created in Python can be read by a Java app or managed by the Go-based Data Manager.

2. The Native Go Approach (Structs & Comparers)

Go developers often prefer storing native structs for maximum performance and type safety. SOP supports this fully but requires a bridge to be manageable by the generic Data Manager.

• SOP as a Library (Code First):

  • Usage: You define custom structs and a custom comparer in Go code.
  • Pros: Maximum flexibility and performance.
  • Cons: Hidden Logic. The Data Manager cannot inherently understand your compiled sorting logic.

• SOP as a Managed Platform (Data First):

  • Usage: You provide an IndexSpecification that describes your key fields and sorting order.
  • Pros: Full Data Manager Support. The IndexSpecification acts as the contract, allowing the UI and AI Agents to manage your data.
  • Workflow: You can use the Data Manager to create the B-Tree and define the IndexSpecification. Then, use the built-in Code Generator to generate the Go structs.

3. Spaces / Knowledge Bases (Knowledge Base Studio)

The same managed workflow extends to Spaces, which are SOP's Knowledge Base assets.

• Author in the UI: Use the SOP Data Manager (Knowledge Base Studio) to create, curate, and refine your Spaces / Knowledge Bases without writing application code.
• Consume in your app: Once authored, open the same Space from your Go application with the sop/ai package and use the rich KnowledgeBase API to manage, search, and digest those authored assets at runtime.
• Best practice: Treat the Data Manager as the authoring and governance layer, and the ai package as the runtime layer that retrieves, filters, and reasons over the curated Space content inside your product.

Typical flow:

1. Use the Data Manager to create or import a Space.
2. Curate categories, summaries, and items in the UI.
3. In your Go app, open the Space through ai/database / ai/memory, then use SearchKeywords or SearchSemantics to retrieve context for RAG, copilots, or domain tools.

This is how developers can keep the human-friendly authoring experience in SOP Data Manager while still embedding the resulting Knowledge Base directly into their application logic.

Bridging the Gap: From Code-First to Managed (Safe & Zero-Downtime)

SOP supports a hybrid workflow. You can start with a Code-First approach (using custom Go structs and comparers) and later make the store fully manageable by the Data Manager—without migration or downtime.

• The Feature: Use the Data Manager's Edit Store functionality to attach an IndexSpecification or CEL expression to an existing "Code-First" store.
• The Safety Mechanism: This operation is 100% safe but protected.
  1. Admin Unlock: For non-empty stores, these fields are locked by default. You must provide an Admin Token to unlock them, ensuring only authorized personnel can modify the schema of a live database.
  2. Metadata Only: The change only updates the Store Registry (metadata). It never touches the actual B-Tree data nodes or re-writes the data file.
  3. Non-Invasive: The underlying B-Tree structure remains identical. The Data Manager simply uses the new specification to interpret and sort the keys dynamically, matching the logic of your compiled code.
  4. Reversibility: Since the data is untouched, you can refine or remove the specification at any time.

This allows DBAs to take a "black box" store generated by application code and turn it into a transparent, queryable dataset for reporting, debugging, and AI analysis.

General Purpose & Strong Typing

SOP B-Trees are general-purpose storage engines. Similar to a Model Store, they can store "any" value type (e.g., interface{} in Go, Object in Java).

• Dynamic Typing: You can store mixed types in the same B-Tree if your application logic supports it.
• Strong Typing via "Seed": To enforce strong typing and enable rich Data Manager features, you should seed the B-Tree with an initial item.
  • Discovery: When you add a "seed" item (the first record), the Data Manager inspects it to discover the Key and Value types automatically.
  • Schema Enforcement: This effectively "locks in" the schema for the UI, allowing it to generate correct forms and validation rules.
  • Swarm Readiness: As noted below, this seed item also initializes the tree structure, enabling efficient "Swarm" transaction merging immediately.

See the API Cookbook for details.

SOP Data Manager

SOP includes a SOP Data Manager that provides full CRUD capabilities for B-Tree stores. It supports inspection, search, and day-to-day data management through a web UI.

• Web UI: A modern, responsive interface for browsing B-Trees, managing stores, and visualizing data.
• AI Copilot: Integrated directly into the UI, the AI Copilot can help you write queries (including SQL-like Joins), explain data structures, and even generate code snippets.
• Natural Language SQL: Perform complex Selects, Joins, and CRUD operations using plain English.
• Streaming Architecture: Results from Agents and Scripts are streamed in real-time, enabling Scripts as Views and efficient handling of large datasets with minimal memory footprint.
• SystemDB: View and manage internal system data, including registry information and transaction logs.
• Scripts: Record and replay complex data operations for testing or automation (now with Parameter support).

To launch the SOP Data Manager:

Development Mode (Offline / Dummy AI)

By default, the server runs in development mode. The AI Copilot and Space Management tools will use dummy/mock AI providers. This is perfect for local testing without incurring API costs.

# From the root of the repository
go run ./tools/httpserver

Production Mode (Real AI / BYOK)

To unlock the actual AI Copilot and manage Spaces (Memory Architecture), you need to start the server in Production Mode. Because SOP uses a BYOK (Bring Your Own Key) architecture, the backend remains stateless—your API keys are entered directly into the Web UI.

# From the root of the repository
go run ./tools/httpserver -production

Once running, you can enter your Embedder and LLM credentials during the Setup Wizard (Page 3), directly in the AI Chat window, or (in the future) via the Login screen.

Or use the pre-built binaries if available. See tools/httpserver/README.md for more details.

SOP AI Kit

The SOP AI Kit transforms SOP from a storage engine into a complete AI data platform.

• Vector Store: Native support for storing and searching high-dimensional vectors.
• RAG Agents: Build Retrieval-Augmented Generation applications with ease.
• Scripts: A functional AI runtime for drafting and executing complex workflows.

See ai/README.md for a deep dive into the AI capabilities.

Table of contents

• Introduction
• Polyglot Support
• Interoperability & Data Management
• Scalability & Limits
• Swarm Computing (Architecture)
• Workflows & Scenarios
• Engineering Culture & SDLC - New! How we use AI & Automation.
• High-level features and articles
• Architecture Guide
• Configuration & Tuning
• Operational Guide
• API Cookbook
• Examples (Go)
• Quick start
• Lifecycle: failures, failover, reinstate, EC auto-repair
• Prerequisites
• Running integration tests (Docker)
• Testing (unit, integration, stress)
• Usability
• SOP API discussions
• SOP for Python (sop4py)
• SOP for Java (sop4j)
• SOP for C# (sop4cs)
• SOP for Rust (sop4rs)
• SOP AI Kit - Includes Vector Store, RAG Agents, and Scripts (Record & Replay).
  • AI Script Architecture - Deep dive into our scalable, functional AI runtime.
  • Scripting Guide - Learn how to record and write scripts.
• SOP Data Manager - SOP HTTP Server & Web UI for local and clustered management workflows.
• Changelog
• Community & support
• Contributing & license
• Coordination model (OOA) and safety

Swarm Computing & Concurrent Transactions

SOP supports "Swarm Computing" where multiple distributed processes or threads can concurrently modify the same B-Tree without external locks. The library handles ACID transactions, conflict detection, and merging automatically.

Important Requirement for First Commit: To enable seamless concurrent merging on a newly created B-Tree, you must pre-seed the B-Tree with at least one item in a separate, initial transaction.

• Why? This establishes the root node and structure, preventing race conditions that can occur when multiple transactions attempt to initialize an empty tree simultaneously.
• Bonus: As mentioned above, this seed item also allows the Data Manager to auto-discover the Key/Value types, turning your general-purpose B-Tree into a strongly-typed, manageable store.
• Note: This requirement is simply to have at least one item in the tree. It can be a real application item or a dummy seed item.
• Safety: Your data remains ACID-compliant. This step simply ensures the "first commit" doesn't suffer from a "random drop" race condition where one transaction's initialization overwrites another's.
• After this single seed item is committed, the B-Tree is fully ready for high-concurrency "swarm" operations.

Cluster reboot procedure

When rebooting an entire cluster running applications that use SOP, follow this order to avoid stale locks and ensure clean recovery:

1. Gracefully stop all apps that use SOP across the cluster.
2. Stop the Redis service(s) used by these SOP apps.
3. Reboot hosts if needed (or proceed directly if not).
4. Start the Redis service(s) first and verify they are healthy.
5. Start the apps that use SOP.

Self-Healing & Reliability

SOP includes a robust background servicer that ensures database integrity even in the face of infrastructure failures like Redis restarts.

Redis Restart Detection & Lock Resurrection

In Clustered mode (using Redis), SOP employs a minimally intrusive "on Redis restart" detector. This mechanism:

• Detects Redis Restarts: Automatically identifies when the Redis cache has restarted or lost volatile data.
• Resurrects Locks: If a transaction was incomplete during a Redis failure, the system automatically "resurrects" the necessary locks for the transaction's priority logs.
• Prevents Corruption: This ensures that the registry sector does not become corrupted due to half-complete transactions.
• Self-Healing: The background servicer automatically handles this lifecycle maintenance, keeping the database "rock solid" without manual intervention.

Note: This feature is specific to Clustered mode. In Standalone mode, the application performs a similar cleanup sweep immediately upon startup.

Notes:

• SOP relies on Redis for coordination (locks, recovery bookkeeping). Bringing Redis up before SOP apps prevents unnecessary failovers or stale-lock handling during app startup.
• If any node was force-killed, SOP’s stale-lock and rollback paths will repair on next write; starting Redis first ensures that path has the needed state.

Introduction

What is SOP?

Scalable Objects Persistence (SOP) is a bare metal storage engine that bakes together a set of storage related features & algorithms in order to provide the most efficient & reliable (ACID attributes of transactions) technique (known) of storage management and rich search. It brings to the application the raw muscle of "raw storage" via direct I/O communications with disk drives, bypassing the overhead of intermediate database layers.

SOP V2 core is written in Go, but provides first-class bindings for Python, Java, C#, and Rust, making it a truly universal storage solution. It can be used for storage management by applications of many types across different hardware architectures & Operating Systems (OS).

Polyglot Support

SOP is designed as a "Write Once, Run Anywhere" architecture. The core engine is compiled into a shared library (libsop.so/.dylib/.dll) which is then consumed by language-specific bindings. This ensures that all languages benefit from the same high-performance, ACID-compliant core.

For a deep dive into our multi-language architecture, see Polyglot Support.

Supported Languages

• Go: The native core. Best for high-concurrency backend services.
• Python (sop4py): Ideal for AI/ML, RAG applications, and data science.
• Java (sop4j): Perfect for enterprise backends and legacy modernization.
• C# (sop4cs): Native integration for .NET Core and Windows environments.
• Rust (sop4rs): For systems programming and high-performance applications.

Scalability & Limits

SOP is architected to handle Petabyte-scale datasets and Trillions of objects.

• Capacity: Up to 495 Billion items per segment (with 1,000 segments = 495 Trillion items) per Btree.
• Throughput: Limited only by hardware (Redis Cluster + Storage I/O), not software.
• Design: Horizontal scaling via independent storage nodes and sharded registry.

See the full analysis in Scalability & Limits.

Key Use Cases

SOP is designed to be versatile, powering everything from small embedded tools to massive enterprise clusters.

For detailed architectural patterns, deployment lifecycles, and configuration examples, see Workflows & Scenarios.

1. Standalone App (Embedded DB)

• Scenario: Desktop apps, CLI tools, or single-node services needing rich indexing.
• Why SOP:
  • Bare Metal Performance: Direct B-Tree indexing on disk with minimal abstraction overhead.
  • Speed: "NoCheck" transaction mode. For build-once-read-many scenarios, skip conflict checks entirely for raw, unbridled read speed.
  • Simplicity: No external database dependencies (just a local file structure).

2. Enterprise Cluster App

• Scenario: Distributed systems requiring high availability and ACID guarantees.
• Why SOP:
  • ACID Transactions: Two-Phase Commit (2PC) across distributed nodes.
  • Multi-Tenancy: Native support for multi-tenancy (via Directories or Keyspaces) allows multiple tenants to share the same cluster while maintaining strict data isolation.
  • Scalability: Infinite metadata scaling via Sharded Registry (FileSystem) or Cassandra tables.
  • Resilience: Registry replication (Active/Passive or Quorum) and Erasure Coding for data blobs ensure zero data loss.
  • Operational Flexibility: Choose the backend that fits your ops stack:
    • FileSystem (infs): The most versatile option. Run on Local Disk for embedded/dev use, or mount a Network Drive (NAS/S3) for infinite cluster scalability. Requires only a shared mount and Redis. It also offers stronger all-or-nothing guarantees than incfs, thanks to the Registry-on-File-System design.
    • Cassandra (incfs): "Power up" your existing Cassandra cluster with SOP. Adds full ACID Transactions, B-Tree Indexing (ordered data, range queries), and efficient large item management to Cassandra's eventual consistency model. In practice, Cassandra can still exhibit rare dual-success races on batched commits, so the filesystem-backed Registry path is the safer choice when strict atomicity matters.

3. AI Vector Database

• Scenario: Storing and retrieving millions of vector embeddings for RAG (Retrieval-Augmented Generation) applications.
• Why SOP:
  • Transactional & ACID: Unlike eventual-consistency vector stores, SOP provides full ACID compliance for vector operations, ensuring no data loss or "ghost" vectors.
  • Novel Storage Schema: Uses a composite key strategy (CentroidID + DistanceToCentroid) to map high-dimensional vectors onto standard B-Trees, enabling efficient range scans and transactional integrity.
  • Ideal Random Sampling: Uses a novel "Lookup Tree" indexing algorithm to generate mathematically representative centroids, ensuring high-quality clustering even on sorted or skewed datasets.
  • Self-Healing Index: Automatically optimizes clusters and tracks distribution in real-time, maintaining optimal search speeds as data grows to terabytes.
  • Flexible Deployment: Run in Standalone mode (zero dependencies, in-memory cache) or Clustered mode (Redis-backed cache for distributed scale).

4. AI Agent with Local LLM (Ollama)

• Scenario: Privacy-focused or cost-sensitive AI agents that need to "understand" user input before searching.
• Why SOP:
  • Embedder Agent Pattern: SOP supports a dual-agent architecture where a specialized "Nurse" agent (powered by a local LLM like Llama 3 via Ollama) translates vague user queries (e.g., "my tummy hurts") into precise clinical terms (e.g., "abdominal pain") before the main "Doctor" agent searches the vector database.
  • Zero Cost: Run the entire stack (Vector DB + LLM + Application) on a single machine without any API fees.
  • Factory Reset Kit: SOP's "Fallback Config" pattern allows you to ship a self-contained "restore disk" (JSON with raw data) that automatically rebuilds the binary B-Tree database if it's ever corrupted or deleted.

5. Blob Store (Media & Large Files)

• Scenario: Storing and streaming massive files like 4K video, high-fidelity audio, or large datasets (1TB+).
• Why SOP:
  • Streaming Data Store: SOP's StreamingDataStore breaks large values into manageable chunks (e.g., 20MB) automatically.
  • Partial Updates: You can update specific chunks of a large file (e.g., editing a video segment) without rewriting the entire file.
  • ACID Transactions: Even for multi-gigabyte files, SOP guarantees transactional integrity. You can upload or update massive blobs in a transaction; if it fails, it rolls back cleanly.
  • Smart Resume: Built-in support for seeking to specific chunks allows for "resume download" or "seek to timestamp" functionality out of the box.

6. AI Model Registry

• Scenario: Managing versions of local AI models (weights, configurations) alongside the data they process.
• Why SOP:
  • Unified Storage: Store your training data (Vectors), metadata (Registry), and model artifacts (Blobs/JSON) in one ACID-compliant system.
  • Atomic Updates: Update your model weights and the vector index they correspond to in a single transaction, preventing version mismatch.
  • Versioning: Built-in support for versioning models (e.g., "v1.0", "v1.1") using composite keys.

7. Cassandra Power-Up (Layer 2 Database)

• Scenario: Enhancing existing Cassandra clusters with features it natively lacks.
• Why SOP:
  • Solves the "Blob Problem": Keeps Cassandra lean by storing only metadata (Registry) in tables, while offloading heavy data (B-Tree nodes, values) to the file system or object storage. This prevents compaction issues common with large blobs.
  • Rich Indexing: Adds full B-Tree capabilities (range queries, prefix search, ordering) which are natively missing or limited in Cassandra.
  • ACID Transactions: Provides strict ACID transactions (Two-Phase Commit) on top of Cassandra's eventually consistent architecture.
  • Multi-Tenancy: Native support for Keyspaces allows logical separation of data within the same cluster.

8. Embedded Search Engine

• Scenario: Adding "Search this wiki" or "Filter by text" features to an application without managing a separate Elasticsearch cluster.
• Why SOP:
  • Transactional Indexing: Index documents in the same transaction as you save them. No "eventual consistency" lag.
  • BM25 Scoring: Uses industry-standard ranking algorithms for relevance.
  • Zero Ops: It's just a library. No separate process to manage or monitor.

Core Innovations

SOP breaks the mold of traditional database architectures by combining the speed of embedded systems with the scalability of distributed clusters.

1. The Unified Native Core ("Write Once, Run Everywhere")

Unlike many databases that rely on slow TCP/IP protocols for local drivers, SOP runs inside your application process.

• Architecture: The core engine is written in Go and compiled into a high-performance Shared Library (.so / .dll).
• Language Agnostic: Whether you use Python, C#, or Java, your application loads this engine directly into memory using FFI.
• Zero Latency: Function calls (e.g., db.Get()) are direct memory operations, not network requests. This allows for millions of operations per second on a single node.

2. Swarm Computing Architecture

SOP is not just a storage engine; it is a coordination framework.

• Distributed State: Nodes in a cluster don't just store data; they share System Knowledge.
• Virtual Execution: The "OS" of the swarm allows you to run distributed workflows where logic moves to the data, rather than moving data to the logic.
• Linear Scalability: Add more nodes to increase both storage capacity (Storage Nodes) and processing power (Compute Nodes) linearly.

3. Rich Key Structures (Metadata-Carrying Keys)

Standard Key-Value stores treat keys as opaque strings. SOP treats them as First-Class Objects.

• Complex Keys: You can use complex structs (e.g., {Region, Dept, EmployeeID}) as keys.
• "Ride-on" Data: Critical metadata (like Version, Deleted flags, or CentroidID for vectors) is physically stored in the B-Tree Key node.
  • Performance: Operations like "List all non-deleted items" scan only the lightweight index, avoiding the expensive I/O of fetching the full data payload.
  • Consistency: This metadata serves as the source of truth for ACID transactions.

4. Hybrid AI Compute & Scripting (Self-Correcting)

SOP combines an AI-facing scripting engine with explicit execution controls.

• Deterministic Workflows: Multi-step reasoning can be captured as reusable scripts after validation.
• Memory and Repair: Runtime memory and repair patterns support iterative correction across asks.
• Operational Safety: Tool execution remains local and constrained by the application runtime.
• Further Reading: See AI Copilot & Agent Architecture and AI Script Architecture for the full runtime model.

5. Granular Durability & RAID

Moving beyond simple replication, SOP brings hardware-level reliability concepts into software.

• Erasure Coding: Split large objects (Blobs) across multiple physical drives or network locations with parity, ensuring data survival even if multiple drives fail.
• Store-Level RAID: You can configure redundancy policies per-store. Your "Logs" store can be ephemeral (fast, low safety), while your "Financials" store uses Reed-Solomon Erasure Coding (maximum safety) on the same cluster.

For a deeper dive into the system's design and package structure (including the Public vs. Internal split), please see the Architecture Guide.

For configuration options and performance tuning, see the Configuration Guide.

For operational best practices (failover, backups), see the Operational Guide.

For code examples, check out the API Cookbook.

See Summary for additional implementation notes and storage-engine details.

The Database Abstraction

SOP provides a high-level database package that simplifies configuration and management of your storage artifacts.

• Unified Entry Point: Manage B-Trees, Vector Stores, and Model Registries from a single Database instance.
• Deployment Modes:
  • Standalone: Uses in-memory caching and local storage. Ideal for single-node apps or development.
  • Clustered: Uses Redis for distributed caching and coordination. Ideal for production clusters.
• Simplified Transactions: db.BeginTransaction handles the complexity of configuring caching and replication for you.

Quick start

SOP is a NoSQL-like key/value storage engine with built-in indexing and transactions. You only need Go to start (Redis is optional for distributed setups).

1. Plan your environment

• Ensure sufficient disk capacity for your datasets. SOP stores on local filesystems and can replicate across drives.

1. Prerequisites

• Go 1.26.4 or later (module requires go 1.26.4)
• (Optional) Redis (recent version) - required only for distributed/cluster mode or if using Redis-backed caching. Note: Redis is NOT used for data storage, just for coordination & to offer built-in caching.

1. Install and run Redis (Optional)

• If using distributed features, install Redis locally or point to your cluster.

1. Add SOP to your Go app

• Import package:
  • github.com/sharedcode/sop/database (Recommended: Unified entry point for B-Trees, Vector Stores, and AI Models)
  • github.com/sharedcode/sop/ai (AI Toolkit: Vector Database, Agents, and RAG)
  • github.com/sharedcode/sop/infs (Low-level: Direct access to filesystem-backed B-Trees)
• Repo path: https://github.com/sharedcode/sop

1. Initialize and start coding

• Use the database package to initialize your environment.

  // Initialize (Standalone or Clustered)
  opts := sop.DatabaseOptions{
      Type:          sop.Standalone,
      StoresFolders: []string{"/var/lib/sop"},
  }
  
  // Start a Transaction
  tx, _ := database.BeginTransaction(ctx, opts, sop.ForWriting)
  
  // Open a Store (B-Tree, Vector, or Model)
  users, _ := database.NewBtree[string, string](ctx, opts, "users", tx, nil)
  
  // Perform Operations
  users.Add(ctx, "user1", "John Doe")
  
  // Commit
  tx.Commit(ctx)
  

1. Deploy

• Ship your app and SOP along your usual release flow (binary or container). If you expose SOP via a microservice, choose REST/gRPC as needed.

1. Permissions

• Ensure the process user has RW permissions on the target data directories/drives. SOP uses DirectIO and filesystem APIs with 4096-byte sector alignment.

Tip: Using Python? See “SOP for Python” below.

Lifecycle: failures, failover, reinstate, EC auto-repair

SOP is designed to keep your app online through common storage failures.

• Blob store with EC: B-tree nodes and large blobs are stored using Erasure Coding (EC). Up to the configured parity, reads/writes continue even when some drives are offline. When failures exceed parity, writes roll back (no failover is generated) and reads may require recovery.
• Registry and StoreRepository: These metadata files use Active/Passive replication. Only I/O errors on Registry/StoreRepository can generate a failover. On a failover, SOP flips to the passive path and continues. When you restore the failed drive, reinstate it as the passive side; SOP will fast‑forward any missing deltas and return it to rotation.
• Auto‑repair: With EC repair enabled, after replacing a failed blob drive, SOP reconstructs missing shards automatically and restores full redundancy in the background.

See the detailed lifecycle guide (failures, observability, reinstate/fast‑forward, and drive replacement) in GO_CORE_ENGINE.md: https://github.com/SharedCode/sop/blob/master/docs/GO_CORE_ENGINE.md#lifecycle-failures-failover-reinstate-and-ec-auto-repair

Also see Operational caveats: https://github.com/SharedCode/sop/blob/master/docs/GO_CORE_ENGINE.md#operational-caveats

For planned maintenance, see Cluster reboot procedure: Cluster reboot procedure.

Transaction idle maintenance (onIdle) & priority rollback sweeps

Each write or read transaction opportunistically invokes an internal onIdle() path at the start (Begin()). This lightweight pass performs two independent maintenance tasks:

1. Priority rollback sweeps (Lock Resurrection): Resurrects lost locks for interrupted higher-priority transactions by consulting per‑transaction priority log (.plg) files. This allows transactions to "self-heal" by enabling stale detection and rollback as necessary.

   • Cluster-wide coordination: This task is coordinated across the entire cluster (or all threads in standalone mode). Only one worker "wins" and performs the sweep at any given time, ensuring no redundant processing. This prevents unnecessary "swarm overload" on these onIdle services.
   • Restart fast path: On application start (in embedded mode), SOP triggers a one‑time sweep of all priority logs immediately, ignoring age. This accelerates recovery of any half‑committed writes that were waiting for the periodic window. In Clustered mode, a global coordinator ensures that only one process in the entire cluster pings the Redis 'notrestarted' flag and performs the actual lock resurrection service if needed.
   • Periodic path: Absent a restart, one worker periodically processes aged logs. Base interval is 5 minutes. If the previous sweep found work, a shorter 2 minute backoff is used to drain backlog faster. Intervals are governed by two atomically updated globals: lastPriorityOnIdleTime (Unix ms) and priorityLogFound (0/1 flag).
   • Rule: Priority logs older than 5 minutes are considered "abandoned" and are rolled back by this servicer.
   • Concurrency: A mutex plus atomic timestamp prevents overlapping sweeps; only one goroutine performs a rollback batch at a time even under high Begin() concurrency.
   • Rationale: Using onIdle piggybacks maintenance on natural transaction flow without a dedicated background goroutine, simplifying embedding into host applications that manage their own scheduling.

2. Expired transaction log cleanup: Removes obsolete commit/rollback artifacts (B-Tree node pages and data value pages).

   • Cluster-wide coordination: Like priority sweeps, this task is coordinated cluster-wide. Only one worker wins the right to perform the cleanup for a given interval (regular or accelerated).
   • Intervals:
     • 4 Hours (Default): B-Tree nodes and data pages modified in a transaction are temporary until the commit updates the Registry. Since the Registry is the source of truth for ACID transactions, cleaning up these temporary artifacts can be done at a "luxury of time" pace (4 hours) without affecting data integrity.
     • 5 Minutes (Accelerated): If recent activity suggests potential pending cleanup (e.g., known rollbacks), the interval accelerates to 5 minutes to reclaim space faster.
   • Timing uses an atomic lastOnIdleRunTime.

Thread safety: Earlier versions used unsynchronized globals; these now use atomic loads/stores (sync/atomic) to eliminate race detector warnings when tests force timer rewinds. Tests that manipulate timing (to speed up sweep scenarios) reset the atomic counters instead of writing plain globals.

Operational impact: You generally do not need to call anything explicitly—just ensure transactions continue to flow. If you embed SOP in a service that may become read‑only idle for long stretches but you still want prompt rollback of higher‑priority interruptions, periodically issue a lightweight read transaction to trigger onIdle.

Testing notes: Unit tests rewind lastPriorityOnIdleTime and priorityLogFound (atomically) to force immediate sweep execution; this pattern is acceptable only in test code. Production code should never reset these values manually.

Prerequisites

• Go 1.26.4+
• OS: macOS, Linux, or Windows.
  • Architectures: x64 (AMD64/Intel64) and ARM64 (Apple Silicon/Linux aarch64).
• (Optional) Redis server (local or cluster) - for distributed coordination
• Data directories on disks you intend SOP to use (4096-byte sector size recommended)

Running Integration Tests

You can run the SOP's integration tests from "infs" package using the following docker commands: NOTE: you need docker desktop running in your host machine for this to work. Go to the sop root folder, e.g. cd ~/sop, where sop is the folder where you cloned from github.

1. Build the docker image: docker build -t mydi .
2. Run the docker image in a container: docker run mydi

• Where "mydi" is the name of the docker image, you can use another name of your choice.

The docker image will be built with alpine (linux) and Redis server in it. Copy the SOP source codes to it. Setup target data folder and environment variable that tells the unit tests of the data folder path. On docker run, the shell script ensures that the Redis server is up & running then run the ("infs" package's integration) test files.

You can pattern how the test sets the (datapath) env't variable so you can run the same integration tests in your host machine, if needed, and yes, you need Redis running locally for this to work. See https://github.com/SharedCode/sop/blob/master/Dockerfile and https://github.com/SharedCode/sop/blob/master/docker-entrypoint.sh for more details.

If you’re using VS Code, there are ready-made tasks:

• Docker: Build and Test — builds image mydi
• Docker: Run Tests — runs tests in the container

Testing (unit, integration, stress)

Run tests locally without Docker using build tags:

• Unit tests (fast): go test ./...
• Integration tests (require Redis running on localhost and a writable data folder):
  • Set environment variable datapath to your data directory (defaults to a local path if unset).
  • Run: go test -tags=integration ./infs/integrationtests
• Stress tests (long-running): go test -timeout 2h -tags=stress ./infs/stresstests/...

VS Code tasks provided:

• Go: Test (Unit)
• Go: Test (Integration)
• Go: Test (Stress)
• Go: Test (Unit + Integration) runs both in sequence

CI note: GitHub Actions runs unit tests on pushes/PRs; a nightly/manual job runs the stress suite with -tags=stress.

Usability

See details here: https://github.com/sharedcode/sop/blob/master/docs/GO_CORE_ENGINE.md#usability

SOP API Discussions

See details here: https://github.com/sharedcode/sop/blob/master/docs/GO_CORE_ENGINE.md#simple-usage

SOP for Python (sop4py)

See details here: https://github.com/sharedcode/sop/tree/master/jsondb/python#readme Check out the [Python Cookbook](jsondb/python/COOKBOOK.md) for code recipes.

SOP for AI Kit

SOP includes a comprehensive AI toolkit for building local, privacy-first expert systems. - **AI Documentation**: [ai/README.md](ai/README.md) - Overview of the AI module, Vector Store, and Agent framework. - **AI Tutorial**: [ai/TUTORIAL.md](ai/TUTORIAL.md) - Step-by-step guide to building the "Doctor & Nurse" expert system.

Timeouts and deadlines

SOP commits are governed by two bounds:

• The caller context (deadline/cancellation)
• The transaction maxTime (commit max duration)

The commit ends when the earlier of these two is reached. Internal lock TTLs use maxTime to ensure locks are bounded even if the caller cancels early.

Recommendation: If you want replication/log cleanup to complete under the same budget, set your context deadline to at least maxTime plus a small grace period.

Reliability & Integrity

SOP implements a "Rock Solid" storage strategy ensuring data integrity and consistency across failures.

Checksums (CRC32)

Every data block written to disk is protected by a CRC32 checksum.

• Implementation: fs/marshaldata.go
• Mechanism: The marshalData function appends a crc32.ChecksumIEEE to every block. unmarshalData validates this checksum on read, returning an error if data corruption (bit rot) is detected.
• Zero-Copy Optimization: Sparse (all-zero) blocks are optimized to skip checksum calculation while maintaining validity.

Rollbacks (COW & Priority Logs)

SOP uses a robust rollback mechanism to recover from crashes or power failures during a transaction.

• Implementation: fs/hashmap.cow.go
• Copy-On-Write (COW): Before modifying a registry sector, SOP creates a .cow backup file (createCow). If a crash occurs, the next accessor detects the COW file, verifies its integrity (using the embedded CRC32), and restores the original state (restoreFromCow).
• Priority Logs: Transaction logs (.plg) track in-flight transactions. The onIdle maintenance process scans these logs to identify and roll back abandoned or expired transactions, ensuring the system returns to a consistent state.

Unified Locking (Cross-Platform)

SOP employs a "Redis-assisted, Storage-anchored" locking model that works consistently across operating systems (Linux, Windows, macOS).

• Storage Anchors: Exclusive access to storage sectors is enforced via claim markers on the disk itself, using standard filesystem APIs with 4096-byte sector alignment (DirectIO). This ensures that even if Redis (the coordination layer) is lost, the physical data remains protected by the filesystem's atomic guarantees.
• Redis Coordination: Redis is used for high-speed, ephemeral locking to reduce contention.
• Cross-Platform Consistency: By relying on standard file I/O and sector alignment rather than OS-specific locking primitives (like flock vs LockFile), SOP guarantees identical locking behavior on all supported platforms.

Coordination model (OOA) and safety

Coordination model: Redis-assisted, storage-anchored

SOP uses Redis for fast, ephemeral coordination and the filesystem for durable sector claims. Redis locks provide low-latency contention detection; per-sector claim markers on storage enforce exclusive access for CUD operations. This hybrid keeps coordination responsive without coupling correctness to Redis durability.

Why this is safe (despite Redis tail loss/failover)

• Locks are advisory; correctness is anchored in storage-sector claims and idempotent commit/rollback.
• On Redis restart, SOP detects it and performs cleanup sweeps (clearing stale sector claims) before resuming.
• Time-bounded lock TTLs, takeover checks, and rollback paths ensure progress without split-brain.
• Priority logs and deterministic rollback let workers resume or repair safely after interruptions.

Operational properties

• Decentralized: no leader or quorum; any node can coordinate on a sector independently.
• Horizontally scalable: sharded by registry sectors; no global hot spots.
• No single point of failure: loss of Redis state slows coordination briefly but doesn't corrupt data.
• Low latency: lock checks and claim writes are O(1) on hot path; no multi-round consensus.

When Redis is unavailable

• Writes that need exclusivity will wait/fail fast; storage remains consistent.
• On recovery, restart sweeps clear stale sector claims; workers resume.

Comparison to Paxos-style consensus

• SOP avoids global consensus, leader election, and replicated logs—lower coordination latency and cost.
• Better horizontal scaling for partitioned workloads (per-sector independence).
• No SPOF in the coordination layer; failover is trivial and stateless.
• If you need a globally ordered, cross-region commit log, consensus is still the right tool; SOP targets high-throughput, partition-aligned coordination. But then again, SOP is not a coordination engine, it is a storage engine. Its internal piece for coordination, e.g. - of handle (virtual ID) Registry, is what was described here.

TL;DR

SOP builds a fast, decentralized coordination layer using Redis only for ephemeral locks and relies on storage-anchored sector claims for correctness. It scales out naturally and avoids consensus overhead while remaining safe under failover.

Clustered Mode Compatibility

In Clustered Mode, SOP uses Redis to coordinate transactions across multiple nodes. This allows many machines to participate in data management for the same Database/B-Tree files on disk while maintaining ACID guarantees.

Note: The database files generated in Standalone and Clustered modes are fully compatible. You can switch between modes as needed but make sure if switching to Standalone mode, that there is only one process that writes to the database files.

Community & support

• Issues: https://github.com/SharedCode/sop/issues
• Discussions: https://github.com/SharedCode/sop/discussions (design/usage topics)

Contributing & license

• Contributing guide: see CONTRIBUTING.md
• Code of Conduct: see CODE_OF_CONDUCT.md
• License: MIT, see LICENSE