replication

package
v1.0.45 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 9, 2026 License: MIT Imports: 25 Imported by: 0

Documentation

Overview

Package replication provides distributed replication for NornicDB.

This package implements a Cassandra-style shared-nothing architecture with Raft consensus for strong consistency. It supports multiple deployment modes:

  • Standalone: Single node, no replication (default)
  • HAStandby: Hot standby with automatic failover (2 nodes)
  • Raft: Raft consensus cluster (3+ nodes)
  • MultiRegion: Raft clusters with cross-region async replication

The design follows these principles:

  • Shared-nothing: Each node owns its data, no shared storage
  • Tunable consistency: Choose between AP (fast) and CP (safe) per query
  • Linear scalability: Add nodes for increased capacity
  • Operational simplicity: Minimal configuration, sensible defaults

Environment Variables (NORNICDB_CLUSTER_*):

NORNICDB_CLUSTER_MODE=standalone|ha_standby|raft|multi_region
NORNICDB_CLUSTER_NODE_ID=node-1
NORNICDB_CLUSTER_BIND_ADDR=0.0.0.0:7000
NORNICDB_CLUSTER_ADVERTISE_ADDR=192.168.1.10:7000
NORNICDB_CLUSTER_REPLICATION_SECRET=your-shared-secret-min-32-chars

Hot Standby:

NORNICDB_CLUSTER_HA_ROLE=primary|standby
NORNICDB_CLUSTER_HA_PEER_ADDR=standby-host:7000
NORNICDB_CLUSTER_HA_SYNC_MODE=async|quorum
NORNICDB_CLUSTER_HA_HEARTBEAT_MS=1000
NORNICDB_CLUSTER_HA_FAILOVER_TIMEOUT=30s
NORNICDB_CLUSTER_HA_AUTO_FAILOVER=true

Raft Cluster:

NORNICDB_CLUSTER_RAFT_PEERS=node-2:7000,node-3:7000
NORNICDB_CLUSTER_RAFT_BOOTSTRAP=true
NORNICDB_CLUSTER_RAFT_ELECTION_TIMEOUT=1s
NORNICDB_CLUSTER_RAFT_HEARTBEAT_TIMEOUT=100ms
NORNICDB_CLUSTER_RAFT_SNAPSHOT_INTERVAL=300
NORNICDB_CLUSTER_RAFT_SNAPSHOT_THRESHOLD=10000

Multi-Region:

NORNICDB_CLUSTER_REGION_ID=us-east
NORNICDB_CLUSTER_REMOTE_REGIONS=eu-west:coord1:7000,ap-south:coord2:7000
NORNICDB_CLUSTER_CROSS_REGION_MODE=async|quorum

Example Usage:

// Load config from environment
config := replication.LoadFromEnv()

// Create replicator based on mode
replicator, err := replication.NewReplicator(config, storage)
if err != nil {
	log.Fatal(err)
}

// Start replication
if err := replicator.Start(ctx); err != nil {
	log.Fatal(err)
}

// Apply writes through replicator
if err := replicator.Apply(cmd); err != nil {
	if errors.Is(err, replication.ErrNotLeader) {
		// Forward to leader
		leaderAddr := replicator.LeaderAddr()
	}
}

ELI12 (Explain Like I'm 12):

Imagine you have a diary that you want to keep safe:

  1. **Standalone**: You have one diary. Simple, but if you lose it, everything's gone.

  2. **Hot Standby**: You write in your diary, and your friend copies everything you write into their diary. If you're sick, they can take over.

  3. **Raft Cluster**: You and 2 friends all have diaries. When you want to write something, everyone votes. If most agree, everyone writes the same thing. Even if one person is absent, the group still works.

  4. **Multi-Region**: Like Raft, but your friends are in different cities. Within each city, the Raft voting happens. Between cities, changes are copied but not voted on (it would be too slow).

The "shared-nothing" part means each person has their own complete diary - no one shares pages with anyone else. This makes it easy to add more friends without things getting complicated.

Package replication provides distributed replication for NornicDB.

Package replication provides cluster replication for NornicDB.

Transport Architecture:

NornicDB cluster communication uses a hybrid approach:

1. **Client Bolt Protocol (default port 7687)** - Used for:

  • Neo4j driver compatibility for client queries

  • Writes can be sent to any node: if a write hits a follower, the cluster transport automatically forwards it to the leader (ForwardApply) and returns the leader's response, so clients do not need to route writes to the leader.

2. **Cluster Protocol (default port 7000)** - Used for:

  • Raft consensus (RequestVote, AppendEntries)
  • Write forwarding (ForwardApply from follower to leader)
  • WAL streaming for HA standby
  • Heartbeats and health checks
  • Cluster coordination

This separation allows:

  • Client-facing Bolt remains pure Neo4j compatible
  • Cluster protocol is optimized for low-latency consensus
  • Existing Bolt infrastructure reused where appropriate

Example Configuration:

NORNICDB_CLUSTER_MODE=raft
NORNICDB_CLUSTER_BIND_ADDR=0.0.0.0:7000
NORNICDB_BOLT_PORT=7687

Index

Constants

This section is empty.

Variables

View Source
var (
	// ErrNotLeader is returned when a write is attempted on a non-leader node.
	ErrNotLeader = errors.New("not leader")

	// ErrNoLeader is returned when no leader is available in the cluster.
	ErrNoLeader = errors.New("no leader available")

	// ErrTimeout is returned when an operation times out.
	ErrTimeout = errors.New("operation timed out")

	// ErrClosed is returned when operating on a closed replicator.
	ErrClosed = errors.New("replicator is closed")

	// ErrStandbyMode is returned when writes are attempted on a standby node.
	ErrStandbyMode = errors.New("node is in standby mode")

	// ErrNotReady is returned when the replicator hasn't finished initialization.
	ErrNotReady = errors.New("replicator not ready")
)

Errors returned by replication operations.

Functions

func RegisterClusterHandlers

func RegisterClusterHandlers(t *ClusterTransport, r Replicator)

RegisterClusterHandlers wires a Replicator's handler methods into a ClusterTransport. This is intentionally transport-specific because other Transport implementations may not support message-type dispatch.

Types

type AppendEntriesRequest

type AppendEntriesRequest struct {
	Term         uint64          `json:"term"`
	LeaderID     string          `json:"leader_id"`
	LeaderAddr   string          `json:"leader_addr"`
	PrevLogIndex uint64          `json:"prev_log_index"`
	PrevLogTerm  uint64          `json:"prev_log_term"`
	Entries      []*RaftLogEntry `json:"entries"`
	LeaderCommit uint64          `json:"leader_commit"`
}

AppendEntriesRequest is sent by the leader to replicate log entries.

type AppendEntriesResponse

type AppendEntriesResponse struct {
	Term          uint64 `json:"term"`
	Success       bool   `json:"success"`
	MatchIndex    uint64 `json:"match_index"`
	ConflictIndex uint64 `json:"conflict_index"`
	ConflictTerm  uint64 `json:"conflict_term"`
	ResponderID   string `json:"responder_id"`
}

AppendEntriesResponse is the response to an append entries request.

type ClusterConnection

type ClusterConnection struct {
	// contains filtered or unexported fields
}

ClusterConnection implements PeerConnection for cluster communication.

func (*ClusterConnection) Close

func (c *ClusterConnection) Close() error

Close closes the connection.

func (*ClusterConnection) IsConnected

func (c *ClusterConnection) IsConnected() bool

IsConnected returns true if the connection is active.

func (*ClusterConnection) SendFence

func (c *ClusterConnection) SendFence(ctx context.Context, req *FenceRequest) (*FenceResponse, error)

SendFence sends a fence request to the peer.

func (*ClusterConnection) SendForwardApply

func (c *ClusterConnection) SendForwardApply(ctx context.Context, cmd *Command, timeout time.Duration) error

SendForwardApply sends a write command to the leader for application. Used by followers to forward writes to the leader automatically.

func (*ClusterConnection) SendHeartbeat

func (c *ClusterConnection) SendHeartbeat(ctx context.Context, req *HeartbeatRequest) (*HeartbeatResponse, error)

SendHeartbeat sends a heartbeat to the peer.

func (*ClusterConnection) SendPromote

func (c *ClusterConnection) SendPromote(ctx context.Context, req *PromoteRequest) (*PromoteResponse, error)

SendPromote sends a promote request to the peer.

func (*ClusterConnection) SendRaftAppendEntries

SendRaftAppendEntries sends Raft append entries to the peer.

func (*ClusterConnection) SendRaftVote

func (c *ClusterConnection) SendRaftVote(ctx context.Context, req *RaftVoteRequest) (*RaftVoteResponse, error)

SendRaftVote sends a Raft vote request to the peer.

func (*ClusterConnection) SendWALBatch

func (c *ClusterConnection) SendWALBatch(ctx context.Context, entries []*WALEntry) (*WALBatchResponse, error)

SendWALBatch sends WAL entries to the peer.

type ClusterMessage

type ClusterMessage struct {
	Type      ClusterMessageType
	NodeID    string
	Timestamp int64
	Signature string
	Payload   []byte
}

ClusterMessage is the on-wire format for cluster communication.

type ClusterMessageType

type ClusterMessageType uint8

ClusterMessageType identifies cluster protocol messages.

const (
	// Raft consensus messages
	ClusterMsgVoteRequest ClusterMessageType = iota + 1
	ClusterMsgVoteResponse
	ClusterMsgAppendEntries
	ClusterMsgAppendEntriesResponse

	// HA standby messages
	ClusterMsgWALBatch
	ClusterMsgWALBatchResponse
	ClusterMsgHeartbeat
	ClusterMsgHeartbeatResponse
	ClusterMsgFence
	ClusterMsgFenceResponse
	ClusterMsgPromote
	ClusterMsgPromoteResponse

	// Cluster management
	ClusterMsgJoin
	ClusterMsgJoinResponse
	ClusterMsgLeave
	ClusterMsgLeaveResponse
	ClusterMsgStatus
	ClusterMsgStatusResponse

	// Write forwarding: follower sends write to leader for application
	ClusterMsgForwardApply
	ClusterMsgForwardApplyResponse
)

type ClusterTransport

type ClusterTransport struct {
	// contains filtered or unexported fields
}

ClusterTransport handles cluster-to-cluster communication.

For client-facing queries, use the standard Bolt server (pkg/bolt). ClusterTransport is specifically for:

  • Raft consensus protocol
  • WAL streaming for HA
  • Cluster coordination

func NewClusterTransport

func NewClusterTransport(config *ClusterTransportConfig) *ClusterTransport

NewClusterTransport creates a cluster transport.

func (*ClusterTransport) Close

func (t *ClusterTransport) Close() error

Close shuts down the transport.

func (*ClusterTransport) Connect

func (t *ClusterTransport) Connect(ctx context.Context, addr string) (PeerConnection, error)

Connect establishes a connection to a peer node.

func (*ClusterTransport) Listen

func (t *ClusterTransport) Listen(ctx context.Context, addr string, handler ConnectionHandler) error

Listen starts accepting cluster connections.

func (*ClusterTransport) RegisterHandler

func (t *ClusterTransport) RegisterHandler(msgType ClusterMessageType, handler MessageHandler)

RegisterHandler registers a handler for a message type.

type ClusterTransportConfig

type ClusterTransportConfig struct {
	NodeID       string
	BindAddr     string
	DialTimeout  time.Duration
	ReadTimeout  time.Duration
	WriteTimeout time.Duration
	MaxMsgSize   int
	TLSServer    *tls.Config
	TLSClient    *tls.Config
	AuthSecret   []byte
	AuthMaxSkew  time.Duration
}

ClusterTransportConfig configures the cluster transport.

func DefaultClusterTransportConfig

func DefaultClusterTransportConfig() *ClusterTransportConfig

DefaultClusterTransportConfig returns production defaults.

type Command

type Command struct {
	// Type identifies the operation type.
	Type CommandType

	// Data is the serialized operation data.
	Data []byte

	// Timestamp when the command was created.
	Timestamp time.Time

	// RequestID for idempotency/deduplication.
	RequestID string
}

Command represents a write operation to be replicated.

type CommandType

type CommandType uint8

CommandType identifies the type of write operation.

const (
	// CmdUnknown is an unknown command type.
	CmdUnknown CommandType = iota

	// CmdCreateNode creates a new node.
	CmdCreateNode

	// CmdUpdateNode updates an existing node.
	CmdUpdateNode

	// CmdDeleteNode deletes a node.
	CmdDeleteNode

	// CmdCreateEdge creates a new edge.
	CmdCreateEdge

	// CmdDeleteEdge deletes an edge.
	CmdDeleteEdge

	// CmdUpdateEdge updates an existing edge.
	CmdUpdateEdge

	// CmdSetProperty sets a property on a node.
	CmdSetProperty

	// CmdBatchWrite is a batch of multiple writes.
	CmdBatchWrite

	// CmdCypher is a Cypher write query.
	CmdCypher

	// CmdVoteRequest is a Raft vote request.
	CmdVoteRequest

	// CmdVoteResponse is a Raft vote response.
	CmdVoteResponse

	// CmdAppendEntries is a Raft append entries request.
	CmdAppendEntries

	// CmdAppendEntriesResponse is a Raft append entries response.
	CmdAppendEntriesResponse

	// CmdDeleteByPrefix deletes all nodes/edges under an ID prefix (e.g. database drop).
	CmdDeleteByPrefix

	// CmdBulkCreateNodes creates multiple nodes.
	CmdBulkCreateNodes

	// CmdBulkCreateEdges creates multiple edges.
	CmdBulkCreateEdges

	// CmdBulkDeleteNodes deletes multiple nodes.
	CmdBulkDeleteNodes

	// CmdBulkDeleteEdges deletes multiple edges.
	CmdBulkDeleteEdges
)

type Config

type Config struct {
	// Mode selects the replication mode (default: standalone)
	// Environment: NORNICDB_CLUSTER_MODE
	Mode ReplicationMode

	// NodeID uniquely identifies this node in the cluster.
	// Auto-generated if not set.
	// Environment: NORNICDB_CLUSTER_NODE_ID
	NodeID string

	// BindAddr is the address to bind the replication server.
	// Environment: NORNICDB_CLUSTER_BIND_ADDR
	BindAddr string

	// AdvertiseAddr is the address advertised to other nodes.
	// Defaults to BindAddr if not set.
	// Environment: NORNICDB_CLUSTER_ADVERTISE_ADDR
	AdvertiseAddr string

	// ReplicationSecret is a shared secret used to authenticate cluster messages.
	// When set, nodes must present valid HMAC signatures for replication traffic.
	// Environment: NORNICDB_CLUSTER_REPLICATION_SECRET
	ReplicationSecret string

	// DataDir for replication state (Raft logs, snapshots).
	// Defaults to main database DataDir + "/replication"
	// Environment: NORNICDB_CLUSTER_DATA_DIR
	DataDir string

	// HAStandby holds hot standby configuration.
	HAStandby HAStandbyConfig

	// Raft holds Raft consensus configuration.
	Raft RaftConfig

	// MultiRegion holds multi-region configuration.
	MultiRegion MultiRegionConfig

	// Consistency holds default consistency levels.
	Consistency ConsistencyConfig

	// TLS holds security configuration for encrypted connections.
	TLS TLSConfig
}

Config holds all replication configuration. Designed to integrate with NornicDB's existing config patterns.

func DefaultConfig

func DefaultConfig() *Config

DefaultConfig returns a Config with sensible defaults for standalone mode. All values are tuned for single-node operation with zero overhead.

func LoadFromEnv

func LoadFromEnv() *Config

LoadFromEnv loads replication configuration from environment variables. Uses NORNICDB_CLUSTER_* prefix for all replication settings.

Example environment for HA Standby:

NORNICDB_CLUSTER_MODE=ha_standby
NORNICDB_CLUSTER_NODE_ID=primary-1
NORNICDB_CLUSTER_HA_ROLE=primary
NORNICDB_CLUSTER_HA_PEER_ADDR=standby-1:7000
NORNICDB_CLUSTER_HA_AUTO_FAILOVER=true

Example environment for Raft:

NORNICDB_CLUSTER_MODE=raft
NORNICDB_CLUSTER_NODE_ID=node-1
NORNICDB_CLUSTER_RAFT_BOOTSTRAP=true
NORNICDB_CLUSTER_RAFT_PEERS=node-2:node2:7000,node-3:node3:7000

func (*Config) IsStandalone

func (c *Config) IsStandalone() bool

IsStandalone returns true if running in standalone mode.

func (*Config) String

func (c *Config) String() string

String returns a safe string representation (no sensitive data).

func (*Config) Validate

func (c *Config) Validate() error

Validate checks the configuration for errors.

type ConnectionHandler

type ConnectionHandler func(conn PeerConnection)

ConnectionHandler handles incoming connections.

type ConsistencyConfig

type ConsistencyConfig struct {
	// DefaultWriteConsistency for write operations.
	// Environment: NORNICDB_CLUSTER_WRITE_CONSISTENCY
	DefaultWriteConsistency ConsistencyLevel

	// DefaultReadConsistency for read operations.
	// Environment: NORNICDB_CLUSTER_READ_CONSISTENCY
	DefaultReadConsistency ConsistencyLevel
}

ConsistencyConfig holds default consistency levels.

type ConsistencyLevel

type ConsistencyLevel string

ConsistencyLevel defines read/write consistency requirements. Follows Cassandra naming conventions.

const (
	// ConsistencyOne requires acknowledgment from one node.
	ConsistencyOne ConsistencyLevel = "ONE"

	// ConsistencyQuorum requires acknowledgment from majority (N/2+1).
	ConsistencyQuorum ConsistencyLevel = "QUORUM"

	// ConsistencyAll requires acknowledgment from all nodes.
	ConsistencyAll ConsistencyLevel = "ALL"

	// ConsistencyLocalOne requires acknowledgment from one node in local region.
	ConsistencyLocalOne ConsistencyLevel = "LOCAL_ONE"

	// ConsistencyLocalQuorum requires quorum from local region only.
	ConsistencyLocalQuorum ConsistencyLevel = "LOCAL_QUORUM"
)

type EdgePayload

type EdgePayload struct {
	ID            string
	StartNode     string
	EndNode       string
	Type          string
	Properties    map[string]any
	CreatedAt     int64
	Confidence    float64
	AutoGenerated bool
}

EdgePayload is the replication-safe representation of storage.Edge.

type FenceRequest

type FenceRequest struct {
	Reason    string
	RequestID string
}

FenceRequest requests the peer to stop accepting writes.

type FenceResponse

type FenceResponse struct {
	Fenced bool
}

FenceResponse is the response to a fence request.

type HAStandbyConfig

type HAStandbyConfig struct {
	// Role is "primary" or "standby".
	// Environment: NORNICDB_CLUSTER_HA_ROLE
	Role string

	// PeerAddr is the address of the other node.
	// Environment: NORNICDB_CLUSTER_HA_PEER_ADDR
	PeerAddr string

	// SyncMode controls replication synchronization.
	// Environment: NORNICDB_CLUSTER_HA_SYNC_MODE
	SyncMode SyncMode

	// HeartbeatInterval is how often to send heartbeats.
	// Environment: NORNICDB_CLUSTER_HA_HEARTBEAT_MS (in milliseconds)
	HeartbeatInterval time.Duration

	// FailoverTimeout is how long to wait before triggering failover.
	// Environment: NORNICDB_CLUSTER_HA_FAILOVER_TIMEOUT
	FailoverTimeout time.Duration

	// AutoFailover enables automatic failover on primary failure.
	// Environment: NORNICDB_CLUSTER_HA_AUTO_FAILOVER
	AutoFailover bool

	// WALBatchSize is max entries per WAL batch.
	// Environment: NORNICDB_CLUSTER_HA_WAL_BATCH_SIZE
	WALBatchSize int

	// WALBatchTimeout is max time to wait for batch fill.
	// Environment: NORNICDB_CLUSTER_HA_WAL_BATCH_TIMEOUT
	WALBatchTimeout time.Duration

	// ReconnectInterval is how often to retry connection to peer.
	// Environment: NORNICDB_CLUSTER_HA_RECONNECT_INTERVAL
	ReconnectInterval time.Duration

	// MaxReconnectBackoff is maximum backoff for reconnection.
	// Environment: NORNICDB_CLUSTER_HA_MAX_RECONNECT_BACKOFF
	MaxReconnectBackoff time.Duration
}

HAStandbyConfig configures 2-node hot standby replication.

type HAStandbyReplicator

type HAStandbyReplicator struct {
	// contains filtered or unexported fields
}

HAStandbyReplicator implements hot standby replication between 2 nodes. One node is primary (accepts writes), the other is standby (receives WAL). Supports automatic failover when primary fails.

func NewHAStandbyReplicator

func NewHAStandbyReplicator(config *Config, storage Storage) (*HAStandbyReplicator, error)

NewHAStandbyReplicator creates a new HA standby replicator.

func (*HAStandbyReplicator) Apply

func (r *HAStandbyReplicator) Apply(cmd *Command, timeout time.Duration) error

Apply applies a command to storage.

func (*HAStandbyReplicator) ApplyBatch

func (r *HAStandbyReplicator) ApplyBatch(cmds []*Command, timeout time.Duration) error

ApplyBatch applies multiple commands.

func (*HAStandbyReplicator) HandleFence

func (r *HAStandbyReplicator) HandleFence(req *FenceRequest) (*FenceResponse, error)

HandleFence handles incoming fence request (for primary).

func (*HAStandbyReplicator) HandleHeartbeat

func (r *HAStandbyReplicator) HandleHeartbeat(req *HeartbeatRequest) (*HeartbeatResponse, error)

HandleHeartbeat handles incoming heartbeat (for standby).

func (*HAStandbyReplicator) HandlePromote

func (r *HAStandbyReplicator) HandlePromote(req *PromoteRequest) (*PromoteResponse, error)

HandlePromote handles an incoming promote request from the peer. Today this is used as a lightweight coordination hook; the actual promotion decision remains local (based on health/auto-failover config).

func (*HAStandbyReplicator) HandleWALBatch

func (r *HAStandbyReplicator) HandleWALBatch(entries []*WALEntry) (*WALBatchResponse, error)

HandleWALBatch handles incoming WAL batch (for standby).

func (*HAStandbyReplicator) Health

func (r *HAStandbyReplicator) Health() *HealthStatus

Health returns health status.

func (*HAStandbyReplicator) IsLeader

func (r *HAStandbyReplicator) IsLeader() bool

IsLeader returns true if this is the primary.

func (*HAStandbyReplicator) LeaderAddr

func (r *HAStandbyReplicator) LeaderAddr() string

LeaderAddr returns the primary address.

func (*HAStandbyReplicator) LeaderID

func (r *HAStandbyReplicator) LeaderID() string

LeaderID returns the primary's node ID.

func (*HAStandbyReplicator) Mode

Mode returns the replication mode.

func (*HAStandbyReplicator) NodeID

func (r *HAStandbyReplicator) NodeID() string

NodeID returns this node's ID.

func (*HAStandbyReplicator) Promote

func (r *HAStandbyReplicator) Promote(ctx context.Context) error

Promote promotes this standby to primary.

func (*HAStandbyReplicator) SetTransport

func (r *HAStandbyReplicator) SetTransport(t Transport)

SetTransport sets the transport for peer communication. This must be called before Start() if not using the default transport.

func (*HAStandbyReplicator) Shutdown

func (r *HAStandbyReplicator) Shutdown() error

Shutdown stops the replicator.

func (*HAStandbyReplicator) Start

func (r *HAStandbyReplicator) Start(ctx context.Context) error

Start initializes and starts the HA standby replicator.

func (*HAStandbyReplicator) WaitForLeader

func (r *HAStandbyReplicator) WaitForLeader(ctx context.Context) error

WaitForLeader blocks until primary is available.

type HealthStatus

type HealthStatus struct {
	// Mode is the replication mode.
	Mode ReplicationMode `json:"mode"`

	// NodeID is this node's identifier.
	NodeID string `json:"node_id"`

	// Role is the current role (leader, follower, standby, etc.).
	Role string `json:"role"`

	// IsLeader indicates if this node accepts writes.
	IsLeader bool `json:"is_leader"`

	// LeaderID is the current leader's ID (if known).
	LeaderID string `json:"leader_id,omitempty"`

	// LeaderAddr is the current leader's address.
	LeaderAddr string `json:"leader_addr,omitempty"`

	// State is the current state (ready, initializing, etc.).
	State string `json:"state"`

	// Healthy indicates overall health.
	Healthy bool `json:"healthy"`

	// ReplicationLag is the lag behind leader (for followers).
	ReplicationLag time.Duration `json:"replication_lag,omitempty"`

	// LastContact is the last successful contact with leader/peer.
	LastContact time.Time `json:"last_contact,omitempty"`

	// Peers contains status of peer nodes.
	Peers []PeerStatus `json:"peers,omitempty"`

	// Region is the region ID (for multi-region).
	Region string `json:"region,omitempty"`

	// CommitIndex is the last committed log index (for Raft).
	CommitIndex uint64 `json:"commit_index,omitempty"`

	// AppliedIndex is the last applied log index.
	AppliedIndex uint64 `json:"applied_index,omitempty"`

	// Term is the current Raft term.
	Term uint64 `json:"term,omitempty"`
}

HealthStatus represents the health state of the replicator.

type HeartbeatRequest

type HeartbeatRequest struct {
	NodeID      string
	Role        string
	WALPosition uint64
	Timestamp   int64
}

HeartbeatRequest is a heartbeat message.

type HeartbeatResponse

type HeartbeatResponse struct {
	NodeID      string
	Role        string
	WALPosition uint64
	Lag         int64
}

HeartbeatResponse is the response to a heartbeat.

type MessageHandler

type MessageHandler func(ctx context.Context, nodeID string, msg *ClusterMessage) (*ClusterMessage, error)

MessageHandler processes incoming cluster messages.

type MultiRegionConfig

type MultiRegionConfig struct {
	// RegionID identifies this region.
	// Environment: NORNICDB_CLUSTER_REGION_ID
	RegionID string

	// LocalCluster is the local Raft cluster configuration.
	LocalCluster RaftConfig

	// RemoteRegions are other regions to replicate to.
	// Environment: NORNICDB_CLUSTER_REMOTE_REGIONS (format: "region:host:port,...")
	RemoteRegions []RemoteRegionConfig

	// CrossRegionSyncMode controls cross-region replication mode.
	// Environment: NORNICDB_CLUSTER_CROSS_REGION_MODE
	CrossRegionSyncMode SyncMode

	// CrossRegionBatchSize is entries per cross-region batch.
	// Environment: NORNICDB_CLUSTER_CROSS_REGION_BATCH_SIZE
	CrossRegionBatchSize int

	// CrossRegionBatchTimeout is max time to wait for batch fill.
	// Environment: NORNICDB_CLUSTER_CROSS_REGION_BATCH_TIMEOUT
	CrossRegionBatchTimeout time.Duration

	// ConflictStrategy handles write conflicts between regions.
	// Environment: NORNICDB_CLUSTER_CONFLICT_STRATEGY
	ConflictStrategy string // "last_write_wins", "manual"
}

MultiRegionConfig configures cross-region replication.

type MultiRegionReplicator

type MultiRegionReplicator struct {
	// contains filtered or unexported fields
}

MultiRegionReplicator implements multi-region replication. Each region runs a Raft cluster, with async replication between regions.

Architecture: - Each region has its own Raft cluster for strong local consistency - Cross-region replication is asynchronous via WAL streaming - One region is designated as "primary" for write coordination - Failover promotes a remote region to primary

func NewMultiRegionReplicator

func NewMultiRegionReplicator(config *Config, storage Storage) (*MultiRegionReplicator, error)

NewMultiRegionReplicator creates a new multi-region replicator.

func (*MultiRegionReplicator) Apply

func (r *MultiRegionReplicator) Apply(cmd *Command, timeout time.Duration) error

Apply applies a command through the local Raft cluster.

func (*MultiRegionReplicator) ApplyBatch

func (r *MultiRegionReplicator) ApplyBatch(cmds []*Command, timeout time.Duration) error

ApplyBatch applies multiple commands.

func (*MultiRegionReplicator) Health

func (r *MultiRegionReplicator) Health() *HealthStatus

Health returns health status.

func (*MultiRegionReplicator) IsLeader

func (r *MultiRegionReplicator) IsLeader() bool

IsLeader returns true if this node is the local Raft leader.

func (*MultiRegionReplicator) IsPrimaryRegion

func (r *MultiRegionReplicator) IsPrimaryRegion() bool

IsPrimaryRegion returns true if this is the primary region.

func (*MultiRegionReplicator) LeaderAddr

func (r *MultiRegionReplicator) LeaderAddr() string

LeaderAddr returns the address of the local leader.

func (*MultiRegionReplicator) LeaderID

func (r *MultiRegionReplicator) LeaderID() string

LeaderID returns the ID of the local leader.

func (*MultiRegionReplicator) Mode

Mode returns the replication mode.

func (*MultiRegionReplicator) NodeID

func (r *MultiRegionReplicator) NodeID() string

NodeID returns this node's ID.

func (*MultiRegionReplicator) RegionFailover

func (r *MultiRegionReplicator) RegionFailover(ctx context.Context) error

RegionFailover promotes this region to primary.

func (*MultiRegionReplicator) RegionID

func (r *MultiRegionReplicator) RegionID() string

RegionID returns this region's ID.

func (*MultiRegionReplicator) SetTransport

func (r *MultiRegionReplicator) SetTransport(transport Transport)

SetTransport sets the transport for cross-region communication.

func (*MultiRegionReplicator) Shutdown

func (r *MultiRegionReplicator) Shutdown() error

Shutdown stops the replicator.

func (*MultiRegionReplicator) Start

Start initializes and starts the multi-region replicator.

func (*MultiRegionReplicator) WaitForLeader

func (r *MultiRegionReplicator) WaitForLeader(ctx context.Context) error

WaitForLeader blocks until a leader is elected.

type NodePayload

type NodePayload struct {
	ID               string
	Labels           []string
	Properties       map[string]any
	NamedEmbeddings  map[string][]float32
	ChunkEmbeddings  [][]float32
	CreatedAtUnixNs  int64
	UpdatedAtUnixNs  int64
	DecayScore       float64
	LastAccessedUnix int64
	AccessCount      int64
}

NodePayload is the replication-safe representation of storage.Node, including embeddings.

type PeerConfig

type PeerConfig struct {
	// ID is the unique identifier for this peer.
	ID string

	// Addr is the network address (host:port).
	Addr string
}

PeerConfig holds configuration for a single peer node.

type PeerConnection

type PeerConnection interface {
	// SendWALBatch sends a batch of WAL entries to the peer.
	SendWALBatch(ctx context.Context, entries []*WALEntry) (*WALBatchResponse, error)

	// SendHeartbeat sends a heartbeat to the peer.
	SendHeartbeat(ctx context.Context, req *HeartbeatRequest) (*HeartbeatResponse, error)

	// SendFence sends a fence request to prevent split-brain.
	SendFence(ctx context.Context, req *FenceRequest) (*FenceResponse, error)

	// SendPromote notifies the peer to prepare for promotion.
	SendPromote(ctx context.Context, req *PromoteRequest) (*PromoteResponse, error)

	// SendRaftVote sends a Raft vote request and returns the response.
	SendRaftVote(ctx context.Context, req *RaftVoteRequest) (*RaftVoteResponse, error)

	// SendRaftAppendEntries sends Raft append entries and returns the response.
	SendRaftAppendEntries(ctx context.Context, req *RaftAppendEntriesRequest) (*RaftAppendEntriesResponse, error)

	// Close closes the connection.
	Close() error

	// IsConnected returns true if the connection is active.
	IsConnected() bool
}

PeerConnection represents a connection to a peer node.

type PeerStatus

type PeerStatus struct {
	// ID is the peer's identifier.
	ID string `json:"id"`

	// Address is the peer's network address.
	Address string `json:"address"`

	// Healthy indicates if the peer is reachable.
	Healthy bool `json:"healthy"`

	// Lag is the replication lag for this peer.
	Lag uint64 `json:"lag,omitempty"`

	// LastContact is the last successful contact with this peer.
	LastContact time.Time `json:"last_contact,omitempty"`

	// State is the peer's current state.
	State string `json:"state,omitempty"`
}

PeerStatus represents the status of a peer node.

type PromoteRequest

type PromoteRequest struct {
	Reason string
}

PromoteRequest notifies the peer to prepare for promotion.

type PromoteResponse

type PromoteResponse struct {
	Ready bool
}

PromoteResponse is the response to a promote request.

type RaftAppendEntriesRequest

type RaftAppendEntriesRequest struct {
	Term         uint64          `json:"term"`
	LeaderID     string          `json:"leader_id"`
	LeaderAddr   string          `json:"leader_addr"`
	PrevLogIndex uint64          `json:"prev_log_index"`
	PrevLogTerm  uint64          `json:"prev_log_term"`
	Entries      []*RaftLogEntry `json:"entries"`
	LeaderCommit uint64          `json:"leader_commit"`
}

RaftAppendEntriesRequest is a Raft AppendEntries RPC request.

type RaftAppendEntriesResponse

type RaftAppendEntriesResponse struct {
	Term          uint64 `json:"term"`
	Success       bool   `json:"success"`
	MatchIndex    uint64 `json:"match_index"`
	ConflictIndex uint64 `json:"conflict_index,omitempty"`
	ConflictTerm  uint64 `json:"conflict_term,omitempty"`
	ResponderID   string `json:"responder_id"`
}

RaftAppendEntriesResponse is a Raft AppendEntries RPC response.

type RaftConfig

type RaftConfig struct {
	// ClusterID identifies the Raft cluster.
	// Environment: NORNICDB_CLUSTER_RAFT_CLUSTER_ID
	ClusterID string

	// Bootstrap initializes a new cluster (only on first node).
	// Environment: NORNICDB_CLUSTER_RAFT_BOOTSTRAP
	Bootstrap bool

	// Peers is the initial set of peer addresses.
	// Format: "node-id:addr,node-id:addr"
	// Environment: NORNICDB_CLUSTER_RAFT_PEERS
	Peers []PeerConfig

	// ElectionTimeout is the time before starting an election.
	// Environment: NORNICDB_CLUSTER_RAFT_ELECTION_TIMEOUT
	ElectionTimeout time.Duration

	// HeartbeatTimeout is the interval for leader heartbeats.
	// Environment: NORNICDB_CLUSTER_RAFT_HEARTBEAT_TIMEOUT
	HeartbeatTimeout time.Duration

	// LeaderLeaseTimeout is how long a leader maintains its lease.
	// Environment: NORNICDB_CLUSTER_RAFT_LEADER_LEASE_TIMEOUT
	LeaderLeaseTimeout time.Duration

	// SnapshotInterval is seconds between automatic snapshots.
	// Environment: NORNICDB_CLUSTER_RAFT_SNAPSHOT_INTERVAL
	SnapshotInterval int

	// SnapshotThreshold is log entries before triggering snapshot.
	// Environment: NORNICDB_CLUSTER_RAFT_SNAPSHOT_THRESHOLD
	SnapshotThreshold uint64

	// TrailingLogs is entries to retain after snapshot.
	// Environment: NORNICDB_CLUSTER_RAFT_TRAILING_LOGS
	TrailingLogs uint64

	// MaxAppendEntries is max entries per AppendEntries RPC.
	// Environment: NORNICDB_CLUSTER_RAFT_MAX_APPEND_ENTRIES
	MaxAppendEntries uint64

	// CommitTimeout is max time to wait for commit.
	// Environment: NORNICDB_CLUSTER_RAFT_COMMIT_TIMEOUT
	CommitTimeout time.Duration

	// MaxInflightLogs is max in-flight log entries for replication.
	// Environment: NORNICDB_CLUSTER_RAFT_MAX_INFLIGHT_LOGS
	MaxInflightLogs int

	// SnapshotRetain is number of snapshots to retain.
	// Environment: NORNICDB_CLUSTER_RAFT_SNAPSHOT_RETAIN
	SnapshotRetain int
}

RaftConfig configures Raft consensus clustering.

type RaftLogEntry

type RaftLogEntry struct {
	Index   uint64   `json:"index"`
	Term    uint64   `json:"term"`
	Command *Command `json:"command"`
}

RaftLogEntry represents an entry in the Raft log.

type RaftRPCMessage

type RaftRPCMessage struct {
	Type    RaftRPCType `json:"type"`
	Payload []byte      `json:"payload"`
}

RaftRPCMessage wraps all Raft RPC messages for transport.

type RaftRPCType

type RaftRPCType uint8

RaftRPCType identifies the type of Raft RPC message.

const (
	RPCVoteRequest RaftRPCType = iota + 1
	RPCVoteResponse
	RPCAppendEntries
	RPCAppendEntriesResponse
)

type RaftReplicator

type RaftReplicator struct {
	// contains filtered or unexported fields
}

RaftReplicator implements full Raft consensus-based replication. This provides strong consistency with automatic leader election.

func NewRaftReplicator

func NewRaftReplicator(config *Config, storage Storage) (*RaftReplicator, error)

NewRaftReplicator creates a new Raft replicator.

func (*RaftReplicator) AddVoter

func (r *RaftReplicator) AddVoter(id, addr string) error

AddVoter adds a voting member to the cluster.

func (*RaftReplicator) Apply

func (r *RaftReplicator) Apply(cmd *Command, timeout time.Duration) error

Apply applies a command through Raft consensus. If this node is not the leader, it forwards the command to the leader automatically.

func (*RaftReplicator) ApplyBatch

func (r *RaftReplicator) ApplyBatch(cmds []*Command, timeout time.Duration) error

ApplyBatch applies multiple commands atomically.

func (*RaftReplicator) GetConfiguration

func (r *RaftReplicator) GetConfiguration() ([]PeerStatus, error)

GetConfiguration returns the current cluster configuration.

func (*RaftReplicator) HandleForwardApply

func (r *RaftReplicator) HandleForwardApply(cmd *Command, timeout time.Duration) error

HandleForwardApply applies a command received from a follower (write forwarding). Only the leader should receive these; it applies the command through Raft as usual.

func (*RaftReplicator) HandleRaftAppendEntries

func (r *RaftReplicator) HandleRaftAppendEntries(req *RaftAppendEntriesRequest) (*RaftAppendEntriesResponse, error)

HandleRaftAppendEntries handles an incoming AppendEntries RPC via the cluster transport.

func (*RaftReplicator) HandleRaftVote

func (r *RaftReplicator) HandleRaftVote(req *RaftVoteRequest) (*RaftVoteResponse, error)

HandleRaftVote handles an incoming RequestVote RPC via the cluster transport.

func (*RaftReplicator) Health

func (r *RaftReplicator) Health() *HealthStatus

Health returns health status.

func (*RaftReplicator) IsLeader

func (r *RaftReplicator) IsLeader() bool

IsLeader returns true if this node is the Raft leader.

func (*RaftReplicator) LeaderAddr

func (r *RaftReplicator) LeaderAddr() string

LeaderAddr returns the address of the current leader.

func (*RaftReplicator) LeaderID

func (r *RaftReplicator) LeaderID() string

LeaderID returns the ID of the current leader.

func (*RaftReplicator) Mode

func (r *RaftReplicator) Mode() ReplicationMode

Mode returns the replication mode.

func (*RaftReplicator) NodeID

func (r *RaftReplicator) NodeID() string

NodeID returns this node's ID.

func (*RaftReplicator) RemoveServer

func (r *RaftReplicator) RemoveServer(id string) error

RemoveServer removes a server from the cluster.

func (*RaftReplicator) SetTransport

func (r *RaftReplicator) SetTransport(t Transport)

SetTransport sets the transport for peer communication.

func (*RaftReplicator) Shutdown

func (r *RaftReplicator) Shutdown() error

Shutdown gracefully shuts down the replicator.

func (*RaftReplicator) Start

func (r *RaftReplicator) Start(ctx context.Context) error

Start initializes and starts the Raft replicator.

func (*RaftReplicator) WaitForLeader

func (r *RaftReplicator) WaitForLeader(ctx context.Context) error

WaitForLeader blocks until a leader is elected or context cancelled.

type RaftState

type RaftState int

RaftState represents the current state of a Raft node.

const (
	StateFollower RaftState = iota
	StateCandidate
	StateLeader
)

func (RaftState) String

func (s RaftState) String() string

type RaftVoteRequest

type RaftVoteRequest struct {
	Term         uint64 `json:"term"`
	CandidateID  string `json:"candidate_id"`
	LastLogIndex uint64 `json:"last_log_index"`
	LastLogTerm  uint64 `json:"last_log_term"`
}

RaftVoteRequest is a Raft RequestVote RPC request.

type RaftVoteResponse

type RaftVoteResponse struct {
	Term        uint64 `json:"term"`
	VoteGranted bool   `json:"vote_granted"`
	VoterID     string `json:"voter_id"`
}

RaftVoteResponse is a Raft RequestVote RPC response.

type RemoteRegionConfig

type RemoteRegionConfig struct {
	// RegionID identifies the remote region.
	RegionID string

	// Addrs are coordinator addresses for the remote region.
	Addrs []string

	// Priority for failover (lower = higher priority).
	Priority int
}

RemoteRegionConfig holds configuration for a remote region.

type ReplicatedEngine

type ReplicatedEngine struct {
	storage.Engine
	// contains filtered or unexported fields
}

ReplicatedEngine wraps a storage.Engine and routes write operations through a Replicator.

Design:

  • Reads are served locally from the embedded Engine.
  • Writes are turned into replication Commands and applied via the Replicator.
  • The embedded Engine is used only for reads; replicated writes are applied to the *inner* engine by the StorageAdapter on each node.

This wrapper intentionally operates on the *base* storage (the engine that stores fully-qualified IDs like "<db>:<id>") so multi-database isolation is preserved.

func NewReplicatedEngine

func NewReplicatedEngine(inner storage.Engine, replicator Replicator, timeout time.Duration) *ReplicatedEngine

func (*ReplicatedEngine) BulkCreateEdges

func (e *ReplicatedEngine) BulkCreateEdges(edges []*storage.Edge) error

func (*ReplicatedEngine) BulkCreateNodes

func (e *ReplicatedEngine) BulkCreateNodes(nodes []*storage.Node) error

func (*ReplicatedEngine) BulkDeleteEdges

func (e *ReplicatedEngine) BulkDeleteEdges(ids []storage.EdgeID) error

func (*ReplicatedEngine) BulkDeleteNodes

func (e *ReplicatedEngine) BulkDeleteNodes(ids []storage.NodeID) error

func (*ReplicatedEngine) CreateEdge

func (e *ReplicatedEngine) CreateEdge(edge *storage.Edge) error

func (*ReplicatedEngine) CreateNode

func (e *ReplicatedEngine) CreateNode(node *storage.Node) (storage.NodeID, error)

func (*ReplicatedEngine) DeleteByPrefix

func (e *ReplicatedEngine) DeleteByPrefix(prefix string) (nodesDeleted int64, edgesDeleted int64, err error)

func (*ReplicatedEngine) DeleteEdge

func (e *ReplicatedEngine) DeleteEdge(id storage.EdgeID) error

func (*ReplicatedEngine) DeleteNode

func (e *ReplicatedEngine) DeleteNode(id storage.NodeID) error

func (*ReplicatedEngine) IsLeader

func (e *ReplicatedEngine) IsLeader() bool

IsLeader reports whether this node can accept writes in the current replication mode. This is a convenience for higher-level components (e.g. multidb startup) that need to avoid performing metadata migrations on standby/followers.

func (*ReplicatedEngine) UpdateEdge

func (e *ReplicatedEngine) UpdateEdge(edge *storage.Edge) error

func (*ReplicatedEngine) UpdateNode

func (e *ReplicatedEngine) UpdateNode(node *storage.Node) error

type ReplicationMode

type ReplicationMode string

ReplicationMode defines how the node participates in replication.

const (
	// ModeStandalone is single-node operation with no replication (default).
	// This mode has zero overhead and is suitable for development, testing,
	// and small deployments where high availability is not required.
	ModeStandalone ReplicationMode = "standalone"

	// ModeHAStandby is hot standby replication between 2 nodes.
	// The primary accepts all writes and streams WAL to the standby.
	// Automatic failover promotes standby if primary fails.
	// Best for: Simple HA without the complexity of consensus.
	ModeHAStandby ReplicationMode = "ha_standby"

	// ModeRaft is Raft consensus with 3+ nodes.
	// Provides strong consistency with automatic leader election.
	// All writes go through Raft log for guaranteed ordering.
	// Best for: Production deployments requiring strong consistency.
	ModeRaft ReplicationMode = "raft"

	// ModeMultiRegion combines Raft clusters with cross-region replication.
	// Each region runs a Raft cluster for local strong consistency.
	// Cross-region replication is async for performance.
	// Best for: Global deployments with regional read replicas.
	ModeMultiRegion ReplicationMode = "multi_region"
)

type Replicator

type Replicator interface {
	// Start initializes and starts the replicator.
	// This should be called after the storage engine is ready.
	Start(ctx context.Context) error

	// Apply applies a write command to the cluster.
	// Returns ErrNotLeader if this node cannot accept writes.
	// The command is replicated according to the replication mode.
	Apply(cmd *Command, timeout time.Duration) error

	// ApplyBatch applies multiple commands atomically.
	ApplyBatch(cmds []*Command, timeout time.Duration) error

	// IsLeader returns true if this node can accept writes.
	// For standalone mode, this always returns true.
	// For HA standby, this returns true only on primary.
	// For Raft, this returns true only on the Raft leader.
	IsLeader() bool

	// LeaderAddr returns the address of the current leader.
	// Returns empty string if unknown or in standalone mode.
	LeaderAddr() string

	// LeaderID returns the ID of the current leader.
	LeaderID() string

	// Health returns the current health status of the replicator.
	Health() *HealthStatus

	// WaitForLeader blocks until a leader is elected or context is cancelled.
	WaitForLeader(ctx context.Context) error

	// Shutdown gracefully shuts down the replicator.
	// This should be called before closing the storage engine.
	Shutdown() error

	// Mode returns the current replication mode.
	Mode() ReplicationMode

	// NodeID returns this node's unique identifier.
	NodeID() string
}

Replicator is the unified interface for all replication modes. It abstracts the complexity of different replication strategies behind a simple interface that the rest of NornicDB uses.

The Replicator handles:

  • Write routing (to leader)
  • Read routing (based on consistency level)
  • Health monitoring
  • Failover orchestration

Example:

replicator, _ := replication.NewReplicator(config, storage)
replicator.Start(ctx)
defer replicator.Shutdown()

// All writes go through replicator
if err := replicator.Apply(cmd); err != nil {
    if errors.Is(err, replication.ErrNotLeader) {
        // Forward to leader
    }
}

func NewReplicator

func NewReplicator(config *Config, storage Storage) (Replicator, error)

NewReplicator creates the appropriate Replicator based on configuration. This is the main factory function for creating replicators.

For standalone mode, this returns a no-op replicator that has zero overhead. For other modes, it returns the appropriate distributed replicator.

Example:

config := replication.LoadFromEnv()
if err := config.Validate(); err != nil {
    log.Fatal(err)
}

replicator, err := replication.NewReplicator(config, storage)
if err != nil {
    log.Fatal(err)
}

type SnapshotReader

type SnapshotReader interface {
	Read(p []byte) (n int, err error)
}

SnapshotReader is used to read snapshot data.

type SnapshotWriter

type SnapshotWriter interface {
	Write(p []byte) (n int, err error)
}

SnapshotWriter is used to write snapshot data.

type StandaloneReplicator

type StandaloneReplicator struct {
	// contains filtered or unexported fields
}

StandaloneReplicator is a no-op replicator for single-node operation. It implements the Replicator interface with zero overhead. All writes are applied directly to storage without any replication.

func NewStandaloneReplicator

func NewStandaloneReplicator(config *Config, storage Storage) *StandaloneReplicator

NewStandaloneReplicator creates a new standalone (no-op) replicator.

func (*StandaloneReplicator) Apply

func (r *StandaloneReplicator) Apply(cmd *Command, timeout time.Duration) error

Apply applies a command directly to storage.

func (*StandaloneReplicator) ApplyBatch

func (r *StandaloneReplicator) ApplyBatch(cmds []*Command, timeout time.Duration) error

ApplyBatch applies multiple commands directly to storage.

func (*StandaloneReplicator) Health

func (r *StandaloneReplicator) Health() *HealthStatus

Health returns health status.

func (*StandaloneReplicator) IsLeader

func (r *StandaloneReplicator) IsLeader() bool

IsLeader always returns true for standalone mode.

func (*StandaloneReplicator) LeaderAddr

func (r *StandaloneReplicator) LeaderAddr() string

LeaderAddr returns empty string for standalone mode.

func (*StandaloneReplicator) LeaderID

func (r *StandaloneReplicator) LeaderID() string

LeaderID returns this node's ID for standalone mode.

func (*StandaloneReplicator) Mode

Mode returns the replication mode.

func (*StandaloneReplicator) NodeID

func (r *StandaloneReplicator) NodeID() string

NodeID returns this node's ID.

func (*StandaloneReplicator) Shutdown

func (r *StandaloneReplicator) Shutdown() error

Shutdown stops the standalone replicator.

func (*StandaloneReplicator) Start

func (r *StandaloneReplicator) Start(ctx context.Context) error

Start initializes the standalone replicator (no-op).

func (*StandaloneReplicator) WaitForLeader

func (r *StandaloneReplicator) WaitForLeader(ctx context.Context) error

WaitForLeader returns immediately for standalone mode.

type Storage

type Storage interface {
	// ApplyCommand applies a replicated command to storage.
	ApplyCommand(cmd *Command) error

	// GetWALPosition returns the current WAL position.
	GetWALPosition() (uint64, error)

	// GetWALEntries returns WAL entries starting from the given position.
	GetWALEntries(fromPosition uint64, maxEntries int) ([]*WALEntry, error)

	// WriteSnapshot writes a full snapshot to the given writer.
	WriteSnapshot(w SnapshotWriter) error

	// RestoreSnapshot restores state from a snapshot.
	RestoreSnapshot(r SnapshotReader) error
}

Storage is the interface that the storage engine must implement to work with replication. This is a subset of the full storage.Engine interface, containing only what replication needs.

type StorageAdapter

type StorageAdapter struct {
	// contains filtered or unexported fields
}

StorageAdapter bridges the replication.Storage interface to storage.Engine. It translates replication commands into storage operations and maintains WAL state.

func NewStorageAdapter

func NewStorageAdapter(engine storage.Engine) (*StorageAdapter, error)

NewStorageAdapter creates a new storage adapter wrapping the given engine. The WAL directory defaults to "data/replication/wal" if not specified.

func NewStorageAdapterWithWAL

func NewStorageAdapterWithWAL(engine storage.Engine, walDir string) (*StorageAdapter, error)

NewStorageAdapterWithWAL creates a new storage adapter with a custom WAL directory. If walDir is empty, defaults to "data/replication/wal".

func (*StorageAdapter) ApplyCommand

func (a *StorageAdapter) ApplyCommand(cmd *Command) error

ApplyCommand applies a replicated command to storage. WAL writes are now asynchronous for better performance.

func (*StorageAdapter) Close

func (a *StorageAdapter) Close() error

Close releases replication resources (WAL file handles/background goroutines).

func (*StorageAdapter) Engine

func (a *StorageAdapter) Engine() storage.Engine

Engine returns the underlying storage engine.

func (*StorageAdapter) FlushWAL

func (a *StorageAdapter) FlushWAL() error

FlushWAL waits for all pending WAL writes to complete. This is useful for tests or when you need to ensure durability before proceeding.

func (*StorageAdapter) GetWALEntries

func (a *StorageAdapter) GetWALEntries(fromPosition uint64, maxEntries int) ([]*WALEntry, error)

GetWALEntries returns WAL entries starting from the given position.

func (*StorageAdapter) GetWALPosition

func (a *StorageAdapter) GetWALPosition() (uint64, error)

GetWALPosition returns the current WAL position.

func (*StorageAdapter) PruneWALEntries

func (a *StorageAdapter) PruneWALEntries(uptoPosition uint64)

PruneWALEntries drops in-memory WAL entries up to (and including) uptoPosition. This keeps memory bounded while streaming and does not affect the persistent WAL.

func (*StorageAdapter) RestoreSnapshot

func (a *StorageAdapter) RestoreSnapshot(r SnapshotReader) error

RestoreSnapshot restores state from a snapshot.

func (*StorageAdapter) SetExecutor

func (a *StorageAdapter) SetExecutor(executor *cypher.StorageExecutor)

SetExecutor sets a custom Cypher executor for the adapter. This allows using an executor with additional configuration (e.g., database manager, embedder).

func (*StorageAdapter) WriteSnapshot

func (a *StorageAdapter) WriteSnapshot(w SnapshotWriter) error

WriteSnapshot writes a full snapshot to the given writer.

type SyncMode

type SyncMode string

SyncMode defines the synchronization strategy for replication.

const (
	// SyncAsync acknowledges writes immediately without waiting for replication.
	// Lowest latency, but potential data loss on failure.
	SyncAsync SyncMode = "async"

	// SyncQuorum waits until the replica acknowledges it has applied the write.
	// Highest durability, highest latency.
	SyncQuorum SyncMode = "quorum"
)

type TLSConfig

type TLSConfig struct {
	// Enabled enables TLS for all replication connections.
	// STRONGLY RECOMMENDED for production.
	// Environment: NORNICDB_CLUSTER_TLS_ENABLED
	Enabled bool

	// CertFile is path to the server certificate (PEM format).
	// Environment: NORNICDB_CLUSTER_TLS_CERT_FILE
	CertFile string

	// KeyFile is path to the server private key (PEM format).
	// Environment: NORNICDB_CLUSTER_TLS_KEY_FILE
	KeyFile string

	// CAFile is path to the CA certificate for client verification (mTLS).
	// If set, client certificates will be required and verified.
	// Environment: NORNICDB_CLUSTER_TLS_CA_FILE
	CAFile string

	// VerifyClient requires client certificate verification (mTLS).
	// Should be true in production for mutual authentication.
	// Environment: NORNICDB_CLUSTER_TLS_VERIFY_CLIENT
	VerifyClient bool

	// InsecureSkipVerify skips server certificate verification.
	// WARNING: Only use for testing. Never in production!
	// Environment: NORNICDB_CLUSTER_TLS_INSECURE_SKIP_VERIFY
	InsecureSkipVerify bool

	// ServerName is the expected server name for certificate verification.
	// Environment: NORNICDB_CLUSTER_TLS_SERVER_NAME
	ServerName string

	// MinVersion is the minimum TLS version (default: TLS 1.2).
	// Environment: NORNICDB_CLUSTER_TLS_MIN_VERSION
	MinVersion string // "1.2" or "1.3"

	// CipherSuites restricts allowed cipher suites (optional).
	// Leave empty to use secure defaults.
	// Environment: NORNICDB_CLUSTER_TLS_CIPHER_SUITES
	CipherSuites []string
}

TLSConfig configures TLS/mTLS for secure replication connections. ALL replication traffic should use TLS in production environments.

type Transport

type Transport interface {
	// Connect establishes a connection to a peer.
	Connect(ctx context.Context, addr string) (PeerConnection, error)

	// Listen starts accepting connections from peers.
	Listen(ctx context.Context, addr string, handler ConnectionHandler) error

	// Close shuts down the transport.
	Close() error
}

Transport is the interface for peer-to-peer communication. This allows mocking in tests.

func NewDefaultTransport

func NewDefaultTransport(config *ClusterTransportConfig) Transport

NewDefaultTransport creates a ClusterTransport for production use. See transport.go for the full implementation.

func NewDefaultTransportFromConfig

func NewDefaultTransportFromConfig(cfg *Config) (Transport, error)

NewDefaultTransportFromConfig creates a ClusterTransport using replication config.

type VoteRequest

type VoteRequest struct {
	Term         uint64 `json:"term"`
	CandidateID  string `json:"candidate_id"`
	LastLogIndex uint64 `json:"last_log_index"`
	LastLogTerm  uint64 `json:"last_log_term"`
}

VoteRequest is sent by candidates to request votes.

type VoteResponse

type VoteResponse struct {
	Term        uint64 `json:"term"`
	VoteGranted bool   `json:"vote_granted"`
	VoterID     string `json:"voter_id"`
}

VoteResponse is the response to a vote request.

type WALApplier

type WALApplier struct {
	// contains filtered or unexported fields
}

WALApplier applies WAL entries to storage.

func NewWALApplier

func NewWALApplier(storage Storage) *WALApplier

NewWALApplier creates a new WAL applier.

func (*WALApplier) ApplyBatch

func (a *WALApplier) ApplyBatch(entries []*WALEntry) (uint64, error)

ApplyBatch applies a batch of WAL entries.

func (*WALApplier) Flush

func (a *WALApplier) Flush() error

Flush ensures all pending entries are applied.

type WALBatchResponse

type WALBatchResponse struct {
	AckedPosition    uint64
	ReceivedPosition uint64
}

WALBatchResponse is the response to a WAL batch.

type WALEntry

type WALEntry struct {
	// Position is the unique, monotonically increasing position.
	Position uint64

	// Timestamp when the entry was created.
	Timestamp int64

	// Command is the replicated command.
	Command *Command
}

WALEntry represents a write-ahead log entry.

type WALStreamer

type WALStreamer struct {
	// contains filtered or unexported fields
}

WALStreamer manages WAL streaming from primary.

func NewWALStreamer

func NewWALStreamer(storage Storage, batchSize int) *WALStreamer

NewWALStreamer creates a new WAL streamer.

func (*WALStreamer) AcknowledgePosition

func (w *WALStreamer) AcknowledgePosition(pos uint64)

AcknowledgePosition marks entries up to this position as acknowledged.

func (*WALStreamer) GetPendingEntries

func (w *WALStreamer) GetPendingEntries(maxEntries int) ([]*WALEntry, error)

GetPendingEntries returns WAL entries that haven't been acknowledged.

func (*WALStreamer) LastAcked

func (w *WALStreamer) LastAcked() uint64

LastAcked returns the last acknowledged WAL position.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL