replication

package
v1.1.12 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 26, 2026 License: MIT Imports: 30 Imported by: 0

Documentation

Overview

Package replication provides distributed replication for NornicDB.

This package implements a Cassandra-style shared-nothing architecture with Raft consensus for strong consistency. It supports multiple deployment modes:

  • Standalone: Single node, no replication (default)
  • HAStandby: Hot standby with automatic failover (2 nodes)
  • Raft: Raft consensus cluster (3+ nodes)
  • MultiRegion: Raft clusters with cross-region async replication

The design follows these principles:

  • Shared-nothing: Each node owns its data, no shared storage
  • Tunable consistency: Choose between AP (fast) and CP (safe) per query
  • Linear scalability: Add nodes for increased capacity
  • Operational simplicity: Minimal configuration, sensible defaults

Environment Variables (NORNICDB_CLUSTER_*):

NORNICDB_CLUSTER_MODE=standalone|ha_standby|raft|multi_region
NORNICDB_CLUSTER_NODE_ID=node-1
NORNICDB_CLUSTER_BIND_ADDR=0.0.0.0:7000
NORNICDB_CLUSTER_ADVERTISE_ADDR=192.168.1.10:7000
NORNICDB_CLUSTER_REPLICATION_SECRET=your-shared-secret-min-32-chars

Hot Standby:

NORNICDB_CLUSTER_HA_ROLE=primary|standby
NORNICDB_CLUSTER_HA_PEER_ADDR=standby-host:7000
NORNICDB_CLUSTER_HA_SYNC_MODE=async|quorum
NORNICDB_CLUSTER_HA_HEARTBEAT_MS=1000
NORNICDB_CLUSTER_HA_FAILOVER_TIMEOUT=30s
NORNICDB_CLUSTER_HA_AUTO_FAILOVER=true

Raft Cluster:

NORNICDB_CLUSTER_RAFT_PEERS=node-2:7000,node-3:7000
NORNICDB_CLUSTER_RAFT_BOOTSTRAP=true
NORNICDB_CLUSTER_RAFT_ELECTION_TIMEOUT=1s
NORNICDB_CLUSTER_RAFT_HEARTBEAT_TIMEOUT=100ms
NORNICDB_CLUSTER_RAFT_SNAPSHOT_INTERVAL=300
NORNICDB_CLUSTER_RAFT_SNAPSHOT_THRESHOLD=10000

Multi-Region:

NORNICDB_CLUSTER_REGION_ID=us-east
NORNICDB_CLUSTER_REMOTE_REGIONS=eu-west:coord1:7000,ap-south:coord2:7000
NORNICDB_CLUSTER_CROSS_REGION_MODE=async|quorum

Example Usage:

// Load config from environment
config := replication.LoadFromEnv()

// Create replicator based on mode
replicator, err := replication.NewReplicator(config, storage)
if err != nil {
	log.Fatal(err)
}

// Start replication
if err := replicator.Start(ctx); err != nil {
	log.Fatal(err)
}

// Apply writes through replicator
if err := replicator.Apply(cmd); err != nil {
	if errors.Is(err, replication.ErrNotLeader) {
		// Forward to leader
		leaderAddr := replicator.LeaderAddr()
	}
}

ELI12 (Explain Like I'm 12):

Imagine you have a diary that you want to keep safe:

  1. **Standalone**: You have one diary. Simple, but if you lose it, everything's gone.

  2. **Hot Standby**: You write in your diary, and your friend copies everything you write into their diary. If you're sick, they can take over.

  3. **Raft Cluster**: You and 2 friends all have diaries. When you want to write something, everyone votes. If most agree, everyone writes the same thing. Even if one person is absent, the group still works.

  4. **Multi-Region**: Like Raft, but your friends are in different cities. Within each city, the Raft voting happens. Between cities, changes are copied but not voted on (it would be too slow).

The "shared-nothing" part means each person has their own complete diary - no one shares pages with anyone else. This makes it easy to add more friends without things getting complicated.

Plan 04-06-04: D-05b stale-peer GC lifecycle.Component populating no metric — instead, periodically calling DeleteLabelValues on the per-peer GaugeVecs in *observability.ReplicationMetrics to evict stale peers (peers we have not observed within the staleness threshold).

Why a GC component (RESEARCH §Q7 / Pitfall 3):

  • Per-peer label values accumulate on every reconnect — even when the peer label is stable per RISK-3, a misconfiguration that rotates PeerConfig.ID across restarts would explode cardinality without GC.
  • Without DeleteLabelValues, GaugeVec series persist forever in the registry — a multi-region churn scenario could push the cardinality past D-05a ceilings.
  • DeleteLabelValues races with concurrent observation per the prometheus client_golang documentation (the Bound observer cached by the replicator becomes stale after delete). The replicator's contract is to REBIND on reconnect — see replicator.go documentation.

Lifecycle semantics (mirrors pkg/storage.BytesMetricsSweeper — Plan 04-04 reference):

  • Start spawns a goroutine that ticks every interval (default 5min).
  • Each tick wraps `defer recover()` so any panic during DeleteLabelValues does not crash the supervisor errgroup (RISK-8 / T-04-08).
  • Shutdown closes the ticker and waits for the goroutine to exit within the supervisor's drain budget (5s typical).
  • Registered between pprof and workersC components per RESEARCH §Q4.

Plan 04-06-03: D-15a per-event observation chokepoint for the replicator.

This file holds the small "observability seam" that each Replicator implementation calls at the SAME sites that already emit role-transition log lines. No background polling — single-line additions at existing log sites per CONTEXT D-15a.

Why a thin observation helper instead of weaving metric calls through raft.go / ha_standby.go / multi_region.go directly:

  • **DRY**: the Role/Term/Counter triplet at every leader-boundary transition is identical across implementations. Centralizing in `observeRoleTransition` keeps the pkg/observability dependency surface narrow.
  • **Test seam**: TestRoleTransition_GaugeUpdate drives the helper directly without spinning up the full Raft state machine.
  • **Nil-safe**: production code injects metrics via SetReplicatorMetrics; test fixtures that build a Replicator without metrics get a clean no-op (existing replication tests do not need to know about metrics).

**Pitfall 3 mitigation (RESEARCH §Q7)**: this helper does NOT cache per-peer Bound observers. Per-peer observation goes through `observePeerLag` / `observePeerRTT` / `observeLastContact` which call WithLabelValues every time AND Mark the tracker. The peer_metrics_gc component evicts stale series via DeleteLabelValues; a held Bound observer would point to a detached series after eviction. Re-binding on every observation site is the simple, correct contract.

Package replication provides distributed replication for NornicDB.

Package replication provides cluster replication for NornicDB.

Transport Architecture:

NornicDB cluster communication uses a hybrid approach:

1. **Client Bolt Protocol (default port 7687)** - Used for:

  • Neo4j driver compatibility for client queries

  • Writes can be sent to any node: if a write hits a follower, the cluster transport automatically forwards it to the leader (ForwardApply) and returns the leader's response, so clients do not need to route writes to the leader.

2. **Cluster Protocol (default port 7000)** - Used for:

  • Raft consensus (RequestVote, AppendEntries)
  • Write forwarding (ForwardApply from follower to leader)
  • WAL streaming for HA standby
  • Heartbeats and health checks
  • Cluster coordination

This separation allows:

  • Client-facing Bolt remains pure Neo4j compatible
  • Cluster protocol is optimized for low-latency consensus
  • Existing Bolt infrastructure reused where appropriate

Example Configuration:

NORNICDB_CLUSTER_MODE=raft
NORNICDB_CLUSTER_BIND_ADDR=0.0.0.0:7000
NORNICDB_BOLT_PORT=7687

Index

Constants

View Source
const CurrentCodecVersion uint32 = 1

CurrentCodecVersion is the codec generation this binary speaks. Pre-version peers omit the field (wire value 0). Version 1 is the first versioned frame that supports the optional traceparent field added in Phase 8.

View Source
const DefaultPeerGCInterval = 5 * time.Minute

DefaultPeerGCInterval is the D-05b default sweep cadence.

View Source
const DefaultPeerGCStaleness = 24 * time.Hour

DefaultPeerGCStaleness is the D-05b default staleness threshold — peers not observed within this window are evicted from the registry.

Variables

View Source
var (
	// ErrNotLeader is returned when a write is attempted on a non-leader node.
	ErrNotLeader = errors.New("not leader")

	// ErrNoLeader is returned when no leader is available in the cluster.
	ErrNoLeader = errors.New("no leader available")

	// ErrTimeout is returned when an operation times out.
	ErrTimeout = errors.New("operation timed out")

	// ErrClosed is returned when operating on a closed replicator.
	ErrClosed = errors.New("replicator is closed")

	// ErrStandbyMode is returned when writes are attempted on a standby node.
	ErrStandbyMode = errors.New("node is in standby mode")

	// ErrNotReady is returned when the replicator hasn't finished initialization.
	ErrNotReady = errors.New("replicator not ready")
)

Errors returned by replication operations.

Functions

func RegisterClusterHandlers

func RegisterClusterHandlers(t *ClusterTransport, r Replicator)

RegisterClusterHandlers wires a Replicator's handler methods into a ClusterTransport. This is intentionally transport-specific because other Transport implementations may not support message-type dispatch.

Types

type AppendEntriesRequest

type AppendEntriesRequest struct {
	Term         uint64          `json:"term"`
	LeaderID     string          `json:"leader_id"`
	LeaderAddr   string          `json:"leader_addr"`
	PrevLogIndex uint64          `json:"prev_log_index"`
	PrevLogTerm  uint64          `json:"prev_log_term"`
	Entries      []*RaftLogEntry `json:"entries"`
	LeaderCommit uint64          `json:"leader_commit"`
	// CodecVersion identifies the wire format generation (TRC-21/TRC-22).
	// 0 (or absent via omitempty) = pre-version frame; 1 = first versioned frame.
	// Rolling-upgrade compat: receivers treat absent/0 as pre-version and accept
	// gracefully; leaders omit the field when any peer is at version 0.
	CodecVersion uint32 `json:"codec_version,omitempty"`
}

AppendEntriesRequest is sent by the leader to replicate log entries.

type AppendEntriesResponse

type AppendEntriesResponse struct {
	Term          uint64 `json:"term"`
	Success       bool   `json:"success"`
	MatchIndex    uint64 `json:"match_index"`
	ConflictIndex uint64 `json:"conflict_index"`
	ConflictTerm  uint64 `json:"conflict_term"`
	ResponderID   string `json:"responder_id"`
	// CodecVersion echoes the responder's supported codec version so the leader
	// can track peer capabilities for mixed-cluster compat (TRC-22).
	CodecVersion uint32 `json:"codec_version,omitempty"`
}

AppendEntriesResponse is the response to an append entries request.

type ClusterConnection

type ClusterConnection struct {
	// contains filtered or unexported fields
}

ClusterConnection implements PeerConnection for cluster communication.

func (*ClusterConnection) Close

func (c *ClusterConnection) Close() error

Close closes the connection.

func (*ClusterConnection) IsConnected

func (c *ClusterConnection) IsConnected() bool

IsConnected returns true if the connection is active.

func (*ClusterConnection) SendFence

func (c *ClusterConnection) SendFence(ctx context.Context, req *FenceRequest) (*FenceResponse, error)

SendFence sends a fence request to the peer.

func (*ClusterConnection) SendForwardApply

func (c *ClusterConnection) SendForwardApply(ctx context.Context, cmd *Command, timeout time.Duration) error

SendForwardApply sends a write command to the leader for application. Used by followers to forward writes to the leader automatically.

func (*ClusterConnection) SendHeartbeat

func (c *ClusterConnection) SendHeartbeat(ctx context.Context, req *HeartbeatRequest) (*HeartbeatResponse, error)

SendHeartbeat sends a heartbeat to the peer.

func (*ClusterConnection) SendPromote

func (c *ClusterConnection) SendPromote(ctx context.Context, req *PromoteRequest) (*PromoteResponse, error)

SendPromote sends a promote request to the peer.

func (*ClusterConnection) SendRaftAppendEntries

SendRaftAppendEntries sends Raft append entries to the peer.

func (*ClusterConnection) SendRaftVote

func (c *ClusterConnection) SendRaftVote(ctx context.Context, req *RaftVoteRequest) (*RaftVoteResponse, error)

SendRaftVote sends a Raft vote request to the peer.

func (*ClusterConnection) SendWALBatch

func (c *ClusterConnection) SendWALBatch(ctx context.Context, entries []*WALEntry) (*WALBatchResponse, error)

SendWALBatch sends WAL entries to the peer.

type ClusterMessage

type ClusterMessage struct {
	Type      ClusterMessageType
	NodeID    string
	Timestamp int64
	Signature string
	Payload   []byte
}

ClusterMessage is the on-wire format for cluster communication.

type ClusterMessageType

type ClusterMessageType uint8

ClusterMessageType identifies cluster protocol messages.

const (
	// Raft consensus messages
	ClusterMsgVoteRequest ClusterMessageType = iota + 1
	ClusterMsgVoteResponse
	ClusterMsgAppendEntries
	ClusterMsgAppendEntriesResponse

	// HA standby messages
	ClusterMsgWALBatch
	ClusterMsgWALBatchResponse
	ClusterMsgHeartbeat
	ClusterMsgHeartbeatResponse
	ClusterMsgFence
	ClusterMsgFenceResponse
	ClusterMsgPromote
	ClusterMsgPromoteResponse

	// Cluster management
	ClusterMsgJoin
	ClusterMsgJoinResponse
	ClusterMsgLeave
	ClusterMsgLeaveResponse
	ClusterMsgStatus
	ClusterMsgStatusResponse

	// Write forwarding: follower sends write to leader for application
	ClusterMsgForwardApply
	ClusterMsgForwardApplyResponse
)

type ClusterTransport

type ClusterTransport struct {
	// contains filtered or unexported fields
}

ClusterTransport handles cluster-to-cluster communication.

For client-facing queries, use the standard Bolt server (pkg/bolt). ClusterTransport is specifically for:

  • Raft consensus protocol
  • WAL streaming for HA
  • Cluster coordination

func NewClusterTransport

func NewClusterTransport(config *ClusterTransportConfig) *ClusterTransport

NewClusterTransport creates a cluster transport.

func (*ClusterTransport) Close

func (t *ClusterTransport) Close() error

Close shuts down the transport.

func (*ClusterTransport) Connect

func (t *ClusterTransport) Connect(ctx context.Context, addr string) (PeerConnection, error)

Connect establishes a connection to a peer node.

func (*ClusterTransport) Listen

func (t *ClusterTransport) Listen(ctx context.Context, addr string, handler ConnectionHandler) error

Listen starts accepting cluster connections.

func (*ClusterTransport) RegisterHandler

func (t *ClusterTransport) RegisterHandler(msgType ClusterMessageType, handler MessageHandler)

RegisterHandler registers a handler for a message type.

type ClusterTransportConfig

type ClusterTransportConfig struct {
	NodeID       string
	BindAddr     string
	DialTimeout  time.Duration
	ReadTimeout  time.Duration
	WriteTimeout time.Duration
	MaxMsgSize   int
	TLSServer    *tls.Config
	TLSClient    *tls.Config
	AuthSecret   []byte
	AuthMaxSkew  time.Duration
}

ClusterTransportConfig configures the cluster transport.

func DefaultClusterTransportConfig

func DefaultClusterTransportConfig() *ClusterTransportConfig

DefaultClusterTransportConfig returns production defaults.

type Command

type Command struct {
	// Type identifies the operation type.
	Type CommandType

	// Data is the serialized operation data.
	Data []byte

	// Timestamp when the command was created.
	Timestamp time.Time

	// RequestID for idempotency/deduplication.
	RequestID string
}

Command represents a write operation to be replicated.

type CommandType

type CommandType uint8

CommandType identifies the type of write operation.

const (
	// CmdUnknown is an unknown command type.
	CmdUnknown CommandType = iota

	// CmdCreateNode creates a new node.
	CmdCreateNode

	// CmdUpdateNode updates an existing node.
	CmdUpdateNode

	// CmdDeleteNode deletes a node.
	CmdDeleteNode

	// CmdCreateEdge creates a new edge.
	CmdCreateEdge

	// CmdDeleteEdge deletes an edge.
	CmdDeleteEdge

	// CmdUpdateEdge updates an existing edge.
	CmdUpdateEdge

	// CmdSetProperty sets a property on a node.
	CmdSetProperty

	// CmdBatchWrite is a batch of multiple writes.
	CmdBatchWrite

	// CmdCypher is a Cypher write query.
	CmdCypher

	// CmdVoteRequest is a Raft vote request.
	CmdVoteRequest

	// CmdVoteResponse is a Raft vote response.
	CmdVoteResponse

	// CmdAppendEntries is a Raft append entries request.
	CmdAppendEntries

	// CmdAppendEntriesResponse is a Raft append entries response.
	CmdAppendEntriesResponse

	// CmdDeleteByPrefix deletes all nodes/edges under an ID prefix (e.g. database drop).
	CmdDeleteByPrefix

	// CmdBulkCreateNodes creates multiple nodes.
	CmdBulkCreateNodes

	// CmdBulkCreateEdges creates multiple edges.
	CmdBulkCreateEdges

	// CmdBulkDeleteNodes deletes multiple nodes.
	CmdBulkDeleteNodes

	// CmdBulkDeleteEdges deletes multiple edges.
	CmdBulkDeleteEdges
)

type Config

type Config struct {
	// Mode selects the replication mode (default: standalone)
	// Environment: NORNICDB_CLUSTER_MODE
	Mode ReplicationMode

	// NodeID uniquely identifies this node in the cluster.
	// Auto-generated if not set.
	// Environment: NORNICDB_CLUSTER_NODE_ID
	NodeID string

	// BindAddr is the address to bind the replication server.
	// Environment: NORNICDB_CLUSTER_BIND_ADDR
	BindAddr string

	// AdvertiseAddr is the address advertised to other nodes.
	// Defaults to BindAddr if not set.
	// Environment: NORNICDB_CLUSTER_ADVERTISE_ADDR
	AdvertiseAddr string

	// ReplicationSecret is a shared secret used to authenticate cluster messages.
	// When set, nodes must present valid HMAC signatures for replication traffic.
	// Environment: NORNICDB_CLUSTER_REPLICATION_SECRET
	ReplicationSecret string

	// DataDir for replication state (Raft logs, snapshots).
	// Defaults to main database DataDir + "/replication"
	// Environment: NORNICDB_CLUSTER_DATA_DIR
	DataDir string

	// HAStandby holds hot standby configuration.
	HAStandby HAStandbyConfig

	// Raft holds Raft consensus configuration.
	Raft RaftConfig

	// MultiRegion holds multi-region configuration.
	MultiRegion MultiRegionConfig

	// Consistency holds default consistency levels.
	Consistency ConsistencyConfig

	// TLS holds security configuration for encrypted connections.
	TLS TLSConfig
}

Config holds all replication configuration. Designed to integrate with NornicDB's existing config patterns.

func DefaultConfig

func DefaultConfig() *Config

DefaultConfig returns a Config with sensible defaults for standalone mode. All values are tuned for single-node operation with zero overhead.

func LoadFromEnv

func LoadFromEnv() *Config

LoadFromEnv loads replication configuration from environment variables. Uses NORNICDB_CLUSTER_* prefix for all replication settings.

Example environment for HA Standby:

NORNICDB_CLUSTER_MODE=ha_standby
NORNICDB_CLUSTER_NODE_ID=primary-1
NORNICDB_CLUSTER_HA_ROLE=primary
NORNICDB_CLUSTER_HA_PEER_ADDR=standby-1:7000
NORNICDB_CLUSTER_HA_AUTO_FAILOVER=true

Example environment for Raft:

NORNICDB_CLUSTER_MODE=raft
NORNICDB_CLUSTER_NODE_ID=node-1
NORNICDB_CLUSTER_RAFT_BOOTSTRAP=true
NORNICDB_CLUSTER_RAFT_PEERS=node-2:node2:7000,node-3:node3:7000

func (*Config) IsStandalone

func (c *Config) IsStandalone() bool

IsStandalone returns true if running in standalone mode.

func (*Config) String

func (c *Config) String() string

String returns a safe string representation (no sensitive data).

func (*Config) Validate

func (c *Config) Validate() error

Validate checks the configuration for errors.

type ConnectionHandler

type ConnectionHandler func(conn PeerConnection)

ConnectionHandler handles incoming connections.

type ConsistencyConfig

type ConsistencyConfig struct {
	// DefaultWriteConsistency for write operations.
	// Environment: NORNICDB_CLUSTER_WRITE_CONSISTENCY
	DefaultWriteConsistency ConsistencyLevel

	// DefaultReadConsistency for read operations.
	// Environment: NORNICDB_CLUSTER_READ_CONSISTENCY
	DefaultReadConsistency ConsistencyLevel
}

ConsistencyConfig holds default consistency levels.

type ConsistencyLevel

type ConsistencyLevel string

ConsistencyLevel defines read/write consistency requirements. Follows Cassandra naming conventions.

const (
	// ConsistencyOne requires acknowledgment from one node.
	ConsistencyOne ConsistencyLevel = "ONE"

	// ConsistencyQuorum requires acknowledgment from majority (N/2+1).
	ConsistencyQuorum ConsistencyLevel = "QUORUM"

	// ConsistencyAll requires acknowledgment from all nodes.
	ConsistencyAll ConsistencyLevel = "ALL"

	// ConsistencyLocalOne requires acknowledgment from one node in local region.
	ConsistencyLocalOne ConsistencyLevel = "LOCAL_ONE"

	// ConsistencyLocalQuorum requires quorum from local region only.
	ConsistencyLocalQuorum ConsistencyLevel = "LOCAL_QUORUM"
)

type EdgePayload

type EdgePayload struct {
	ID            string
	StartNode     string
	EndNode       string
	Type          string
	Properties    map[string]any
	CreatedAt     int64
	Confidence    float64
	AutoGenerated bool
}

EdgePayload is the replication-safe representation of storage.Edge.

type FenceRequest

type FenceRequest struct {
	Reason    string
	RequestID string
}

FenceRequest requests the peer to stop accepting writes.

type FenceResponse

type FenceResponse struct {
	Fenced bool
}

FenceResponse is the response to a fence request.

type HAStandbyConfig

type HAStandbyConfig struct {
	// Role is "primary" or "standby".
	// Environment: NORNICDB_CLUSTER_HA_ROLE
	Role string

	// PeerAddr is the address of the other node.
	// Environment: NORNICDB_CLUSTER_HA_PEER_ADDR
	PeerAddr string

	// SyncMode controls replication synchronization.
	// Environment: NORNICDB_CLUSTER_HA_SYNC_MODE
	SyncMode SyncMode

	// HeartbeatInterval is how often to send heartbeats.
	// Environment: NORNICDB_CLUSTER_HA_HEARTBEAT_MS (in milliseconds)
	HeartbeatInterval time.Duration

	// FailoverTimeout is how long to wait before triggering failover.
	// Environment: NORNICDB_CLUSTER_HA_FAILOVER_TIMEOUT
	FailoverTimeout time.Duration

	// AutoFailover enables automatic failover on primary failure.
	// Environment: NORNICDB_CLUSTER_HA_AUTO_FAILOVER
	AutoFailover bool

	// WALBatchSize is max entries per WAL batch.
	// Environment: NORNICDB_CLUSTER_HA_WAL_BATCH_SIZE
	WALBatchSize int

	// WALBatchTimeout is max time to wait for batch fill.
	// Environment: NORNICDB_CLUSTER_HA_WAL_BATCH_TIMEOUT
	WALBatchTimeout time.Duration

	// ReconnectInterval is how often to retry connection to peer.
	// Environment: NORNICDB_CLUSTER_HA_RECONNECT_INTERVAL
	ReconnectInterval time.Duration

	// MaxReconnectBackoff is maximum backoff for reconnection.
	// Environment: NORNICDB_CLUSTER_HA_MAX_RECONNECT_BACKOFF
	MaxReconnectBackoff time.Duration
}

HAStandbyConfig configures 2-node hot standby replication.

type HAStandbyReplicator

type HAStandbyReplicator struct {
	// contains filtered or unexported fields
}

HAStandbyReplicator implements hot standby replication between 2 nodes. One node is primary (accepts writes), the other is standby (receives WAL). Supports automatic failover when primary fails.

func NewHAStandbyReplicator

func NewHAStandbyReplicator(config *Config, storage Storage) (*HAStandbyReplicator, error)

NewHAStandbyReplicator creates a new HA standby replicator.

func (*HAStandbyReplicator) Apply

func (r *HAStandbyReplicator) Apply(cmd *Command, timeout time.Duration) error

Apply applies a command to storage.

func (*HAStandbyReplicator) ApplyBatch

func (r *HAStandbyReplicator) ApplyBatch(cmds []*Command, timeout time.Duration) error

ApplyBatch applies multiple commands.

func (*HAStandbyReplicator) HandleFence

func (r *HAStandbyReplicator) HandleFence(req *FenceRequest) (*FenceResponse, error)

HandleFence handles incoming fence request (for primary).

func (*HAStandbyReplicator) HandleHeartbeat

func (r *HAStandbyReplicator) HandleHeartbeat(req *HeartbeatRequest) (*HeartbeatResponse, error)

HandleHeartbeat handles incoming heartbeat (for standby).

func (*HAStandbyReplicator) HandlePromote

func (r *HAStandbyReplicator) HandlePromote(req *PromoteRequest) (*PromoteResponse, error)

HandlePromote handles an incoming promote request from the peer. Today this is used as a lightweight coordination hook; the actual promotion decision remains local (based on health/auto-failover config).

func (*HAStandbyReplicator) HandleWALBatch

func (r *HAStandbyReplicator) HandleWALBatch(entries []*WALEntry) (*WALBatchResponse, error)

HandleWALBatch handles incoming WAL batch (for standby).

func (*HAStandbyReplicator) Health

func (r *HAStandbyReplicator) Health() *HealthStatus

Health returns health status.

func (*HAStandbyReplicator) IsLeader

func (r *HAStandbyReplicator) IsLeader() bool

IsLeader returns true if this is the primary.

func (*HAStandbyReplicator) LeaderAddr

func (r *HAStandbyReplicator) LeaderAddr() string

LeaderAddr returns the primary address.

func (*HAStandbyReplicator) LeaderID

func (r *HAStandbyReplicator) LeaderID() string

LeaderID returns the primary's node ID.

func (*HAStandbyReplicator) Mode

Mode returns the replication mode.

func (*HAStandbyReplicator) NodeID

func (r *HAStandbyReplicator) NodeID() string

NodeID returns this node's ID.

func (*HAStandbyReplicator) Promote

func (r *HAStandbyReplicator) Promote(ctx context.Context) error

Promote promotes this standby to primary.

func (*HAStandbyReplicator) SetReplicatorMetrics added in v1.1.0

func (r *HAStandbyReplicator) SetReplicatorMetrics(bag *observability.ReplicationMetrics, tracker *PeerTracker)

SetReplicatorMetrics implements MetricsAware — Plan-04-06 D-15a observation seam. Idempotent. Calling with nil disables observation.

func (*HAStandbyReplicator) SetTransport

func (r *HAStandbyReplicator) SetTransport(t Transport)

SetTransport sets the transport for peer communication.

Safe to call before or after Start(). The chaos / partition-recovery tests swap transports on a running replicator to simulate network failure and recovery; r.mu serializes those writes against the connectToStandbyLoop / Listen reads of r.transport.

func (*HAStandbyReplicator) Shutdown

func (r *HAStandbyReplicator) Shutdown() error

Shutdown stops the replicator.

func (*HAStandbyReplicator) Start

func (r *HAStandbyReplicator) Start(ctx context.Context) error

Start initializes and starts the HA standby replicator.

func (*HAStandbyReplicator) WaitForLeader

func (r *HAStandbyReplicator) WaitForLeader(ctx context.Context) error

WaitForLeader blocks until primary is available.

type HealthStatus

type HealthStatus struct {
	// Mode is the replication mode.
	Mode ReplicationMode `json:"mode"`

	// NodeID is this node's identifier.
	NodeID string `json:"node_id"`

	// Role is the current role (leader, follower, standby, etc.).
	Role string `json:"role"`

	// IsLeader indicates if this node accepts writes.
	IsLeader bool `json:"is_leader"`

	// LeaderID is the current leader's ID (if known).
	LeaderID string `json:"leader_id,omitempty"`

	// LeaderAddr is the current leader's address.
	LeaderAddr string `json:"leader_addr,omitempty"`

	// State is the current state (ready, initializing, etc.).
	State string `json:"state"`

	// Healthy indicates overall health.
	Healthy bool `json:"healthy"`

	// ReplicationLag is the lag behind leader (for followers).
	ReplicationLag time.Duration `json:"replication_lag,omitempty"`

	// LastContact is the last successful contact with leader/peer.
	LastContact time.Time `json:"last_contact,omitempty"`

	// Peers contains status of peer nodes.
	Peers []PeerStatus `json:"peers,omitempty"`

	// Region is the region ID (for multi-region).
	Region string `json:"region,omitempty"`

	// CommitIndex is the last committed log index (for Raft).
	CommitIndex uint64 `json:"commit_index,omitempty"`

	// AppliedIndex is the last applied log index.
	AppliedIndex uint64 `json:"applied_index,omitempty"`

	// Term is the current Raft term.
	Term uint64 `json:"term,omitempty"`
}

HealthStatus represents the health state of the replicator.

type HeartbeatRequest

type HeartbeatRequest struct {
	NodeID      string
	Role        string
	WALPosition uint64
	Timestamp   int64
}

HeartbeatRequest is a heartbeat message.

type HeartbeatResponse

type HeartbeatResponse struct {
	NodeID      string
	Role        string
	WALPosition uint64
	Lag         int64
}

HeartbeatResponse is the response to a heartbeat.

type MessageHandler

type MessageHandler func(ctx context.Context, nodeID string, msg *ClusterMessage) (*ClusterMessage, error)

MessageHandler processes incoming cluster messages.

type MetricsAware added in v1.1.0

type MetricsAware interface {
	// SetReplicatorMetrics injects the observability bag + peer tracker.
	// Idempotent. Calling with nil disables observation (defensive).
	SetReplicatorMetrics(bag *observability.ReplicationMetrics, tracker *PeerTracker)
}

MetricsAware is the optional interface a Replicator implementation implements to accept the observability bag. Implementations may embed *atomic.Pointer[replicatorMetrics] or guard with their own mutex; the interface itself is intentionally minimal.

type MultiRegionConfig

type MultiRegionConfig struct {
	// RegionID identifies this region.
	// Environment: NORNICDB_CLUSTER_REGION_ID
	RegionID string

	// LocalCluster is the local Raft cluster configuration.
	LocalCluster RaftConfig

	// RemoteRegions are other regions to replicate to.
	// Environment: NORNICDB_CLUSTER_REMOTE_REGIONS (format: "region:host:port,...")
	RemoteRegions []RemoteRegionConfig

	// CrossRegionSyncMode controls cross-region replication mode.
	// Environment: NORNICDB_CLUSTER_CROSS_REGION_MODE
	CrossRegionSyncMode SyncMode

	// CrossRegionBatchSize is entries per cross-region batch.
	// Environment: NORNICDB_CLUSTER_CROSS_REGION_BATCH_SIZE
	CrossRegionBatchSize int

	// CrossRegionBatchTimeout is max time to wait for batch fill.
	// Environment: NORNICDB_CLUSTER_CROSS_REGION_BATCH_TIMEOUT
	CrossRegionBatchTimeout time.Duration

	// ConflictStrategy handles write conflicts between regions.
	// Environment: NORNICDB_CLUSTER_CONFLICT_STRATEGY
	ConflictStrategy string // "last_write_wins", "manual"
}

MultiRegionConfig configures cross-region replication.

type MultiRegionReplicator

type MultiRegionReplicator struct {
	// contains filtered or unexported fields
}

MultiRegionReplicator implements multi-region replication. Each region runs a Raft cluster, with async replication between regions.

Architecture: - Each region has its own Raft cluster for strong local consistency - Cross-region replication is asynchronous via WAL streaming - One region is designated as "primary" for write coordination - Failover promotes a remote region to primary

func NewMultiRegionReplicator

func NewMultiRegionReplicator(config *Config, storage Storage) (*MultiRegionReplicator, error)

NewMultiRegionReplicator creates a new multi-region replicator.

func (*MultiRegionReplicator) Apply

func (r *MultiRegionReplicator) Apply(cmd *Command, timeout time.Duration) error

Apply applies a command through the local Raft cluster.

func (*MultiRegionReplicator) ApplyBatch

func (r *MultiRegionReplicator) ApplyBatch(cmds []*Command, timeout time.Duration) error

ApplyBatch applies multiple commands.

func (*MultiRegionReplicator) Health

func (r *MultiRegionReplicator) Health() *HealthStatus

Health returns health status.

func (*MultiRegionReplicator) IsLeader

func (r *MultiRegionReplicator) IsLeader() bool

IsLeader returns true if this node is the local Raft leader.

func (*MultiRegionReplicator) IsPrimaryRegion

func (r *MultiRegionReplicator) IsPrimaryRegion() bool

IsPrimaryRegion returns true if this is the primary region.

func (*MultiRegionReplicator) LeaderAddr

func (r *MultiRegionReplicator) LeaderAddr() string

LeaderAddr returns the address of the local leader.

func (*MultiRegionReplicator) LeaderID

func (r *MultiRegionReplicator) LeaderID() string

LeaderID returns the ID of the local leader.

func (*MultiRegionReplicator) Mode

Mode returns the replication mode.

func (*MultiRegionReplicator) NodeID

func (r *MultiRegionReplicator) NodeID() string

NodeID returns this node's ID.

func (*MultiRegionReplicator) RegionFailover

func (r *MultiRegionReplicator) RegionFailover(ctx context.Context) error

RegionFailover promotes this region to primary.

func (*MultiRegionReplicator) RegionID

func (r *MultiRegionReplicator) RegionID() string

RegionID returns this region's ID.

func (*MultiRegionReplicator) SetTransport

func (r *MultiRegionReplicator) SetTransport(transport Transport)

SetTransport sets the transport for cross-region communication.

func (*MultiRegionReplicator) Shutdown

func (r *MultiRegionReplicator) Shutdown() error

Shutdown stops the replicator.

func (*MultiRegionReplicator) Start

Start initializes and starts the multi-region replicator.

func (*MultiRegionReplicator) WaitForLeader

func (r *MultiRegionReplicator) WaitForLeader(ctx context.Context) error

WaitForLeader blocks until a leader is elected.

type NodePayload

type NodePayload struct {
	ID              string
	Labels          []string
	Properties      map[string]any
	NamedEmbeddings map[string][]float32
	ChunkEmbeddings [][]float32
	CreatedAtUnixNs int64
	UpdatedAtUnixNs int64
}

NodePayload is the replication-safe representation of storage.Node, including embeddings.

type PeerConfig

type PeerConfig struct {
	// ID is the unique identifier for this peer.
	ID string

	// Addr is the network address (host:port).
	Addr string
}

PeerConfig holds configuration for a single peer node.

type PeerConnection

type PeerConnection interface {
	// SendWALBatch sends a batch of WAL entries to the peer.
	SendWALBatch(ctx context.Context, entries []*WALEntry) (*WALBatchResponse, error)

	// SendHeartbeat sends a heartbeat to the peer.
	SendHeartbeat(ctx context.Context, req *HeartbeatRequest) (*HeartbeatResponse, error)

	// SendFence sends a fence request to prevent split-brain.
	SendFence(ctx context.Context, req *FenceRequest) (*FenceResponse, error)

	// SendPromote notifies the peer to prepare for promotion.
	SendPromote(ctx context.Context, req *PromoteRequest) (*PromoteResponse, error)

	// SendRaftVote sends a Raft vote request and returns the response.
	SendRaftVote(ctx context.Context, req *RaftVoteRequest) (*RaftVoteResponse, error)

	// SendRaftAppendEntries sends Raft append entries and returns the response.
	SendRaftAppendEntries(ctx context.Context, req *RaftAppendEntriesRequest) (*RaftAppendEntriesResponse, error)

	// Close closes the connection.
	Close() error

	// IsConnected returns true if the connection is active.
	IsConnected() bool
}

PeerConnection represents a connection to a peer node.

type PeerMetricsGC added in v1.1.0

type PeerMetricsGC struct {
	// contains filtered or unexported fields
}

PeerMetricsGC is the lifecycle.Component that GCs stale peer label values from the per-peer GaugeVecs in *observability.ReplicationMetrics.

func NewPeerMetricsGC added in v1.1.0

func NewPeerMetricsGC(metrics *observability.ReplicationMetrics, interval, staleness time.Duration) *PeerMetricsGC

NewPeerMetricsGC constructs the GC. interval ≤ 0 falls back to DefaultPeerGCInterval; staleness ≤ 0 falls back to DefaultPeerGCStaleness. metrics nil disables the sweep (Start returns nil immediately) — defensive for partial-init.

func (*PeerMetricsGC) Name added in v1.1.0

func (g *PeerMetricsGC) Name() string

Name implements lifecycle.Component.

func (*PeerMetricsGC) Shutdown added in v1.1.0

func (g *PeerMetricsGC) Shutdown(ctx context.Context) error

Shutdown implements lifecycle.Component. Idempotent; subsequent calls are no-ops.

func (*PeerMetricsGC) Start added in v1.1.0

func (g *PeerMetricsGC) Start(ctx context.Context) error

Start implements lifecycle.Component. Blocks until ctx is cancelled or Shutdown is called. Does NOT perform an initial sweep — peer state at startup is empty; sweeping immediately would be a no-op anyway.

func (*PeerMetricsGC) Tracker added in v1.1.0

func (g *PeerMetricsGC) Tracker() *PeerTracker

Tracker exposes the PeerTracker so the replicator can call Mark at observation sites. The GC owns the tracker so its sweep + Mark calls share the same mutex (no cross-component locking surprises).

type PeerStatus

type PeerStatus struct {
	// ID is the peer's identifier.
	ID string `json:"id"`

	// Address is the peer's network address.
	Address string `json:"address"`

	// Healthy indicates if the peer is reachable.
	Healthy bool `json:"healthy"`

	// Lag is the replication lag for this peer.
	Lag uint64 `json:"lag,omitempty"`

	// LastContact is the last successful contact with this peer.
	LastContact time.Time `json:"last_contact,omitempty"`

	// State is the peer's current state.
	State string `json:"state,omitempty"`
}

PeerStatus represents the status of a peer node.

type PeerTracker added in v1.1.0

type PeerTracker struct {
	// contains filtered or unexported fields
}

PeerTracker maintains a thread-safe map of peer-label → last-seen time. The replicator code calls Mark on every observation site (heartbeat, replicate, RTT measurement); the GC walks the map to find stale entries.

Rebind-on-reconnect contract (Pitfall 3 mitigation per RESEARCH §Q7): after PeerMetricsGC.sweep evicts a peer via DeleteLabelValues, any held Bound observer in the replicator goes stale — observations land on a detached series that will not be re-aggregated. The replicator MUST re-bind via WithLabelValues at the next observation site (or simply avoid caching Bound observers across reconnects).

func NewPeerTracker added in v1.1.0

func NewPeerTracker() *PeerTracker

NewPeerTracker constructs an empty tracker.

func (*PeerTracker) Forget added in v1.1.0

func (t *PeerTracker) Forget(peer string)

Forget removes a peer entry — called by the GC after eviction.

func (*PeerTracker) Len added in v1.1.0

func (t *PeerTracker) Len() int

Len reports the current number of tracked peers — diagnostic for tests.

func (*PeerTracker) Mark added in v1.1.0

func (t *PeerTracker) Mark(peer string)

Mark records that we just observed `peer`. Callers (replicator observation sites) invoke this every time they touch a per-peer metric.

func (*PeerTracker) StaleSince added in v1.1.0

func (t *PeerTracker) StaleSince(cutoff time.Time) []string

StaleSince returns peers not observed since `cutoff`. The result is a fresh slice owned by the caller — modifying it is safe.

type PromoteRequest

type PromoteRequest struct {
	Reason string
}

PromoteRequest notifies the peer to prepare for promotion.

type PromoteResponse

type PromoteResponse struct {
	Ready bool
}

PromoteResponse is the response to a promote request.

type RaftAppendEntriesRequest

type RaftAppendEntriesRequest struct {
	Term         uint64          `json:"term"`
	LeaderID     string          `json:"leader_id"`
	LeaderAddr   string          `json:"leader_addr"`
	PrevLogIndex uint64          `json:"prev_log_index"`
	PrevLogTerm  uint64          `json:"prev_log_term"`
	Entries      []*RaftLogEntry `json:"entries"`
	LeaderCommit uint64          `json:"leader_commit"`
	CodecVersion uint32          `json:"codec_version,omitempty"`
}

RaftAppendEntriesRequest is a Raft AppendEntries RPC request.

type RaftAppendEntriesResponse

type RaftAppendEntriesResponse struct {
	Term          uint64 `json:"term"`
	Success       bool   `json:"success"`
	MatchIndex    uint64 `json:"match_index"`
	ConflictIndex uint64 `json:"conflict_index,omitempty"`
	ConflictTerm  uint64 `json:"conflict_term,omitempty"`
	ResponderID   string `json:"responder_id"`
	CodecVersion  uint32 `json:"codec_version,omitempty"`
}

RaftAppendEntriesResponse is a Raft AppendEntries RPC response.

type RaftConfig

type RaftConfig struct {
	// ClusterID identifies the Raft cluster.
	// Environment: NORNICDB_CLUSTER_RAFT_CLUSTER_ID
	ClusterID string

	// Bootstrap initializes a new cluster (only on first node).
	// Environment: NORNICDB_CLUSTER_RAFT_BOOTSTRAP
	Bootstrap bool

	// Peers is the initial set of peer addresses.
	// Format: "node-id:addr,node-id:addr"
	// Environment: NORNICDB_CLUSTER_RAFT_PEERS
	Peers []PeerConfig

	// ElectionTimeout is the time before starting an election.
	// Environment: NORNICDB_CLUSTER_RAFT_ELECTION_TIMEOUT
	ElectionTimeout time.Duration

	// HeartbeatTimeout is the interval for leader heartbeats.
	// Environment: NORNICDB_CLUSTER_RAFT_HEARTBEAT_TIMEOUT
	HeartbeatTimeout time.Duration

	// LeaderLeaseTimeout is how long a leader maintains its lease.
	// Environment: NORNICDB_CLUSTER_RAFT_LEADER_LEASE_TIMEOUT
	LeaderLeaseTimeout time.Duration

	// SnapshotInterval is seconds between automatic snapshots.
	// Environment: NORNICDB_CLUSTER_RAFT_SNAPSHOT_INTERVAL
	SnapshotInterval int

	// SnapshotThreshold is log entries before triggering snapshot.
	// Environment: NORNICDB_CLUSTER_RAFT_SNAPSHOT_THRESHOLD
	SnapshotThreshold uint64

	// TrailingLogs is entries to retain after snapshot.
	// Environment: NORNICDB_CLUSTER_RAFT_TRAILING_LOGS
	TrailingLogs uint64

	// MaxAppendEntries is max entries per AppendEntries RPC.
	// Environment: NORNICDB_CLUSTER_RAFT_MAX_APPEND_ENTRIES
	MaxAppendEntries uint64

	// CommitTimeout is max time to wait for commit.
	// Environment: NORNICDB_CLUSTER_RAFT_COMMIT_TIMEOUT
	CommitTimeout time.Duration

	// MaxInflightLogs is max in-flight log entries for replication.
	// Environment: NORNICDB_CLUSTER_RAFT_MAX_INFLIGHT_LOGS
	MaxInflightLogs int

	// SnapshotRetain is number of snapshots to retain.
	// Environment: NORNICDB_CLUSTER_RAFT_SNAPSHOT_RETAIN
	SnapshotRetain int
}

RaftConfig configures Raft consensus clustering.

type RaftLogEntry

type RaftLogEntry struct {
	Index   uint64   `json:"index"`
	Term    uint64   `json:"term"`
	Command *Command `json:"command"`
}

RaftLogEntry represents an entry in the Raft log.

type RaftRPCMessage

type RaftRPCMessage struct {
	Type    RaftRPCType `json:"type"`
	Payload []byte      `json:"payload"`
}

RaftRPCMessage wraps all Raft RPC messages for transport.

type RaftRPCType

type RaftRPCType uint8

RaftRPCType identifies the type of Raft RPC message.

const (
	RPCVoteRequest RaftRPCType = iota + 1
	RPCVoteResponse
	RPCAppendEntries
	RPCAppendEntriesResponse
)

type RaftReplicator

type RaftReplicator struct {
	// contains filtered or unexported fields
}

RaftReplicator implements full Raft consensus-based replication. This provides strong consistency with automatic leader election.

func NewRaftReplicator

func NewRaftReplicator(config *Config, storage Storage) (*RaftReplicator, error)

NewRaftReplicator creates a new Raft replicator.

func (*RaftReplicator) AddVoter

func (r *RaftReplicator) AddVoter(id, addr string) error

AddVoter adds a voting member to the cluster.

func (*RaftReplicator) Apply

func (r *RaftReplicator) Apply(cmd *Command, timeout time.Duration) error

Apply applies a command through Raft consensus. If this node is not the leader, it forwards the command to the leader automatically.

func (*RaftReplicator) ApplyBatch

func (r *RaftReplicator) ApplyBatch(cmds []*Command, timeout time.Duration) error

ApplyBatch applies multiple commands atomically.

func (*RaftReplicator) GetConfiguration

func (r *RaftReplicator) GetConfiguration() ([]PeerStatus, error)

GetConfiguration returns the current cluster configuration.

func (*RaftReplicator) HandleForwardApply

func (r *RaftReplicator) HandleForwardApply(cmd *Command, timeout time.Duration) error

HandleForwardApply applies a command received from a follower (write forwarding). Only the leader should receive these; it applies the command through Raft as usual.

func (*RaftReplicator) HandleRaftAppendEntries

func (r *RaftReplicator) HandleRaftAppendEntries(req *RaftAppendEntriesRequest) (*RaftAppendEntriesResponse, error)

HandleRaftAppendEntries handles an incoming AppendEntries RPC via the cluster transport.

func (*RaftReplicator) HandleRaftVote

func (r *RaftReplicator) HandleRaftVote(req *RaftVoteRequest) (*RaftVoteResponse, error)

HandleRaftVote handles an incoming RequestVote RPC via the cluster transport.

func (*RaftReplicator) Health

func (r *RaftReplicator) Health() *HealthStatus

Health returns health status.

func (*RaftReplicator) IsLeader

func (r *RaftReplicator) IsLeader() bool

IsLeader returns true if this node is the Raft leader.

func (*RaftReplicator) LeaderAddr

func (r *RaftReplicator) LeaderAddr() string

LeaderAddr returns the address of the current leader.

func (*RaftReplicator) LeaderID

func (r *RaftReplicator) LeaderID() string

LeaderID returns the ID of the current leader.

func (*RaftReplicator) Mode

func (r *RaftReplicator) Mode() ReplicationMode

Mode returns the replication mode.

func (*RaftReplicator) NodeID

func (r *RaftReplicator) NodeID() string

NodeID returns this node's ID.

func (*RaftReplicator) RemoveServer

func (r *RaftReplicator) RemoveServer(id string) error

RemoveServer removes a server from the cluster.

func (*RaftReplicator) SetReplicatorMetrics added in v1.1.0

func (r *RaftReplicator) SetReplicatorMetrics(bag *observability.ReplicationMetrics, tracker *PeerTracker)

SetReplicatorMetrics implements MetricsAware — Plan-04-06 D-15a observation seam. Idempotent. Calling with nil disables observation.

func (*RaftReplicator) SetTransport

func (r *RaftReplicator) SetTransport(t Transport)

SetTransport sets the transport for peer communication.

func (*RaftReplicator) Shutdown

func (r *RaftReplicator) Shutdown() error

Shutdown gracefully shuts down the replicator.

func (*RaftReplicator) Start

func (r *RaftReplicator) Start(ctx context.Context) error

Start initializes and starts the Raft replicator.

func (*RaftReplicator) WaitForLeader

func (r *RaftReplicator) WaitForLeader(ctx context.Context) error

WaitForLeader blocks until a leader is elected or context cancelled.

type RaftState

type RaftState int

RaftState represents the current state of a Raft node.

const (
	StateFollower RaftState = iota
	StateCandidate
	StateLeader
)

func (RaftState) String

func (s RaftState) String() string

type RaftVoteRequest

type RaftVoteRequest struct {
	Term         uint64 `json:"term"`
	CandidateID  string `json:"candidate_id"`
	LastLogIndex uint64 `json:"last_log_index"`
	LastLogTerm  uint64 `json:"last_log_term"`
}

RaftVoteRequest is a Raft RequestVote RPC request.

type RaftVoteResponse

type RaftVoteResponse struct {
	Term        uint64 `json:"term"`
	VoteGranted bool   `json:"vote_granted"`
	VoterID     string `json:"voter_id"`
}

RaftVoteResponse is a Raft RequestVote RPC response.

type RemoteRegionConfig

type RemoteRegionConfig struct {
	// RegionID identifies the remote region.
	RegionID string

	// Addrs are coordinator addresses for the remote region.
	Addrs []string

	// Priority for failover (lower = higher priority).
	Priority int
}

RemoteRegionConfig holds configuration for a remote region.

type ReplicatedEngine

type ReplicatedEngine struct {
	storage.Engine
	// contains filtered or unexported fields
}

ReplicatedEngine wraps a storage.Engine and routes write operations through a Replicator.

Design:

  • Reads are served locally from the embedded Engine.
  • Writes are turned into replication Commands and applied via the Replicator.
  • The embedded Engine is used only for reads; replicated writes are applied to the *inner* engine by the StorageAdapter on each node.

This wrapper intentionally operates on the *base* storage (the engine that stores fully-qualified IDs like "<db>:<id>") so multi-database isolation is preserved.

func NewReplicatedEngine

func NewReplicatedEngine(inner storage.Engine, replicator Replicator, timeout time.Duration) *ReplicatedEngine

func (*ReplicatedEngine) BulkCreateEdges

func (e *ReplicatedEngine) BulkCreateEdges(edges []*storage.Edge) error

func (*ReplicatedEngine) BulkCreateNodes

func (e *ReplicatedEngine) BulkCreateNodes(nodes []*storage.Node) error

func (*ReplicatedEngine) BulkDeleteEdges

func (e *ReplicatedEngine) BulkDeleteEdges(ids []storage.EdgeID) error

func (*ReplicatedEngine) BulkDeleteNodes

func (e *ReplicatedEngine) BulkDeleteNodes(ids []storage.NodeID) error

func (*ReplicatedEngine) CreateEdge

func (e *ReplicatedEngine) CreateEdge(edge *storage.Edge) error

func (*ReplicatedEngine) CreateNode

func (e *ReplicatedEngine) CreateNode(node *storage.Node) (storage.NodeID, error)

func (*ReplicatedEngine) DeleteByPrefix

func (e *ReplicatedEngine) DeleteByPrefix(prefix string) (nodesDeleted int64, edgesDeleted int64, err error)

func (*ReplicatedEngine) DeleteEdge

func (e *ReplicatedEngine) DeleteEdge(id storage.EdgeID) error

func (*ReplicatedEngine) DeleteNode

func (e *ReplicatedEngine) DeleteNode(id storage.NodeID) error

func (*ReplicatedEngine) IsLeader

func (e *ReplicatedEngine) IsLeader() bool

IsLeader reports whether this node can accept writes in the current replication mode. This is a convenience for higher-level components (e.g. multidb startup) that need to avoid performing metadata migrations on standby/followers.

func (*ReplicatedEngine) UpdateEdge

func (e *ReplicatedEngine) UpdateEdge(edge *storage.Edge) error

func (*ReplicatedEngine) UpdateNode

func (e *ReplicatedEngine) UpdateNode(node *storage.Node) error

type ReplicationMode

type ReplicationMode string

ReplicationMode defines how the node participates in replication.

const (
	// ModeStandalone is single-node operation with no replication (default).
	// This mode has zero overhead and is suitable for development, testing,
	// and small deployments where high availability is not required.
	ModeStandalone ReplicationMode = "standalone"

	// ModeHAStandby is hot standby replication between 2 nodes.
	// The primary accepts all writes and streams WAL to the standby.
	// Automatic failover promotes standby if primary fails.
	// Best for: Simple HA without the complexity of consensus.
	ModeHAStandby ReplicationMode = "ha_standby"

	// ModeRaft is Raft consensus with 3+ nodes.
	// Provides strong consistency with automatic leader election.
	// All writes go through Raft log for guaranteed ordering.
	// Best for: Production deployments requiring strong consistency.
	ModeRaft ReplicationMode = "raft"

	// ModeMultiRegion combines Raft clusters with cross-region replication.
	// Each region runs a Raft cluster for local strong consistency.
	// Cross-region replication is async for performance.
	// Best for: Global deployments with regional read replicas.
	ModeMultiRegion ReplicationMode = "multi_region"
)

type Replicator

type Replicator interface {
	// Start initializes and starts the replicator.
	// This should be called after the storage engine is ready.
	Start(ctx context.Context) error

	// Apply applies a write command to the cluster.
	// Returns ErrNotLeader if this node cannot accept writes.
	// The command is replicated according to the replication mode.
	Apply(cmd *Command, timeout time.Duration) error

	// ApplyBatch applies multiple commands atomically.
	ApplyBatch(cmds []*Command, timeout time.Duration) error

	// IsLeader returns true if this node can accept writes.
	// For standalone mode, this always returns true.
	// For HA standby, this returns true only on primary.
	// For Raft, this returns true only on the Raft leader.
	IsLeader() bool

	// LeaderAddr returns the address of the current leader.
	// Returns empty string if unknown or in standalone mode.
	LeaderAddr() string

	// LeaderID returns the ID of the current leader.
	LeaderID() string

	// Health returns the current health status of the replicator.
	Health() *HealthStatus

	// WaitForLeader blocks until a leader is elected or context is cancelled.
	WaitForLeader(ctx context.Context) error

	// Shutdown gracefully shuts down the replicator.
	// This should be called before closing the storage engine.
	Shutdown() error

	// Mode returns the current replication mode.
	Mode() ReplicationMode

	// NodeID returns this node's unique identifier.
	NodeID() string
}

Replicator is the unified interface for all replication modes. It abstracts the complexity of different replication strategies behind a simple interface that the rest of NornicDB uses.

The Replicator handles:

  • Write routing (to leader)
  • Read routing (based on consistency level)
  • Health monitoring
  • Failover orchestration

Example:

replicator, _ := replication.NewReplicator(config, storage)
replicator.Start(ctx)
defer replicator.Shutdown()

// All writes go through replicator
if err := replicator.Apply(cmd); err != nil {
    if errors.Is(err, replication.ErrNotLeader) {
        // Forward to leader
    }
}

func NewReplicator

func NewReplicator(config *Config, storage Storage) (Replicator, error)

NewReplicator creates the appropriate Replicator based on configuration. This is the main factory function for creating replicators.

For standalone mode, this returns a no-op replicator that has zero overhead. For other modes, it returns the appropriate distributed replicator.

Example:

config := replication.LoadFromEnv()
if err := config.Validate(); err != nil {
    log.Fatal(err)
}

replicator, err := replication.NewReplicator(config, storage)
if err != nil {
    log.Fatal(err)
}

type SnapshotReader

type SnapshotReader interface {
	Read(p []byte) (n int, err error)
}

SnapshotReader is used to read snapshot data.

type SnapshotWriter

type SnapshotWriter interface {
	Write(p []byte) (n int, err error)
}

SnapshotWriter is used to write snapshot data.

type StandaloneReplicator

type StandaloneReplicator struct {
	// contains filtered or unexported fields
}

StandaloneReplicator is a no-op replicator for single-node operation. It implements the Replicator interface with zero overhead. All writes are applied directly to storage without any replication.

func NewStandaloneReplicator

func NewStandaloneReplicator(config *Config, storage Storage) *StandaloneReplicator

NewStandaloneReplicator creates a new standalone (no-op) replicator.

func (*StandaloneReplicator) Apply

func (r *StandaloneReplicator) Apply(cmd *Command, timeout time.Duration) error

Apply applies a command directly to storage.

func (*StandaloneReplicator) ApplyBatch

func (r *StandaloneReplicator) ApplyBatch(cmds []*Command, timeout time.Duration) error

ApplyBatch applies multiple commands directly to storage.

func (*StandaloneReplicator) Health

func (r *StandaloneReplicator) Health() *HealthStatus

Health returns health status.

func (*StandaloneReplicator) IsLeader

func (r *StandaloneReplicator) IsLeader() bool

IsLeader always returns true for standalone mode.

func (*StandaloneReplicator) LeaderAddr

func (r *StandaloneReplicator) LeaderAddr() string

LeaderAddr returns empty string for standalone mode.

func (*StandaloneReplicator) LeaderID

func (r *StandaloneReplicator) LeaderID() string

LeaderID returns this node's ID for standalone mode.

func (*StandaloneReplicator) Mode

Mode returns the replication mode.

func (*StandaloneReplicator) NodeID

func (r *StandaloneReplicator) NodeID() string

NodeID returns this node's ID.

func (*StandaloneReplicator) SetReplicatorMetrics added in v1.1.0

func (r *StandaloneReplicator) SetReplicatorMetrics(bag *observability.ReplicationMetrics, tracker *PeerTracker)

SetReplicatorMetrics implements MetricsAware. Standalone uses metrics only to emit an initial Role=standalone (well, follower with no peers) observation at Start; the bag is otherwise quiescent.

func (*StandaloneReplicator) Shutdown

func (r *StandaloneReplicator) Shutdown() error

Shutdown stops the standalone replicator.

func (*StandaloneReplicator) Start

func (r *StandaloneReplicator) Start(ctx context.Context) error

Start initializes the standalone replicator (no-op).

func (*StandaloneReplicator) WaitForLeader

func (r *StandaloneReplicator) WaitForLeader(ctx context.Context) error

WaitForLeader returns immediately for standalone mode.

type Storage

type Storage interface {
	// ApplyCommand applies a replicated command to storage.
	ApplyCommand(cmd *Command) error

	// GetWALPosition returns the current WAL position.
	GetWALPosition() (uint64, error)

	// GetWALEntries returns WAL entries starting from the given position.
	GetWALEntries(fromPosition uint64, maxEntries int) ([]*WALEntry, error)

	// WriteSnapshot writes a full snapshot to the given writer.
	WriteSnapshot(w SnapshotWriter) error

	// RestoreSnapshot restores state from a snapshot.
	RestoreSnapshot(r SnapshotReader) error
}

Storage is the interface that the storage engine must implement to work with replication. This is a subset of the full storage.Engine interface, containing only what replication needs.

type StorageAdapter

type StorageAdapter struct {
	// contains filtered or unexported fields
}

StorageAdapter bridges the replication.Storage interface to storage.Engine. It translates replication commands into storage operations and maintains WAL state.

func NewStorageAdapter

func NewStorageAdapter(engine storage.Engine) (*StorageAdapter, error)

NewStorageAdapter creates a new storage adapter wrapping the given engine. The WAL directory defaults to "data/replication/wal" if not specified.

func NewStorageAdapterWithWAL

func NewStorageAdapterWithWAL(engine storage.Engine, walDir string) (*StorageAdapter, error)

NewStorageAdapterWithWAL creates a new storage adapter with a custom WAL directory. If walDir is empty, defaults to "data/replication/wal".

func (*StorageAdapter) ApplyCommand

func (a *StorageAdapter) ApplyCommand(cmd *Command) error

ApplyCommand applies a replicated command to storage. WAL writes are now asynchronous for better performance.

func (*StorageAdapter) Close

func (a *StorageAdapter) Close() error

Close releases replication resources (WAL file handles/background goroutines).

func (*StorageAdapter) Engine

func (a *StorageAdapter) Engine() storage.Engine

Engine returns the underlying storage engine.

func (*StorageAdapter) FlushWAL

func (a *StorageAdapter) FlushWAL() error

FlushWAL waits for all pending WAL writes to complete. This is useful for tests or when you need to ensure durability before proceeding.

The previous version slept 20ms hoping walWriterLoop would drain the queue — on slow runners (CI) goroutine scheduling jitter could leave the tail request still in walQueue at sync time, so callers that immediately observed GetWALPosition saw a count one short of what they queued. The fix is to send a barrier request through the same channel and wait for its acknowledgement. Because the writer reads requests from walQueue in FIFO order, every request queued before the barrier is guaranteed to be flushed before the barrier itself completes.

func (*StorageAdapter) GetWALEntries

func (a *StorageAdapter) GetWALEntries(fromPosition uint64, maxEntries int) ([]*WALEntry, error)

GetWALEntries returns WAL entries starting from the given position.

func (*StorageAdapter) GetWALPosition

func (a *StorageAdapter) GetWALPosition() (uint64, error)

GetWALPosition returns the current WAL position.

func (*StorageAdapter) PruneWALEntries

func (a *StorageAdapter) PruneWALEntries(uptoPosition uint64)

PruneWALEntries drops in-memory WAL entries up to (and including) uptoPosition. This keeps memory bounded while streaming and does not affect the persistent WAL.

func (*StorageAdapter) RestoreSnapshot

func (a *StorageAdapter) RestoreSnapshot(r SnapshotReader) error

RestoreSnapshot restores state from a snapshot.

func (*StorageAdapter) SetExecutor

func (a *StorageAdapter) SetExecutor(executor *cypher.StorageExecutor)

SetExecutor sets a custom Cypher executor for the adapter. This allows using an executor with additional configuration (e.g., database manager, embedder).

func (*StorageAdapter) WriteSnapshot

func (a *StorageAdapter) WriteSnapshot(w SnapshotWriter) error

WriteSnapshot writes a full snapshot to the given writer.

type SyncMode

type SyncMode string

SyncMode defines the synchronization strategy for replication.

const (
	// SyncAsync acknowledges writes immediately without waiting for replication.
	// Lowest latency, but potential data loss on failure.
	SyncAsync SyncMode = "async"

	// SyncQuorum waits until the replica acknowledges it has applied the write.
	// Highest durability, highest latency.
	SyncQuorum SyncMode = "quorum"
)

type TLSConfig

type TLSConfig struct {
	// Enabled enables TLS for all replication connections.
	// STRONGLY RECOMMENDED for production.
	// Environment: NORNICDB_CLUSTER_TLS_ENABLED
	Enabled bool

	// CertFile is path to the server certificate (PEM format).
	// Environment: NORNICDB_CLUSTER_TLS_CERT_FILE
	CertFile string

	// KeyFile is path to the server private key (PEM format).
	// Environment: NORNICDB_CLUSTER_TLS_KEY_FILE
	KeyFile string

	// CAFile is path to the CA certificate for client verification (mTLS).
	// If set, client certificates will be required and verified.
	// Environment: NORNICDB_CLUSTER_TLS_CA_FILE
	CAFile string

	// VerifyClient requires client certificate verification (mTLS).
	// Should be true in production for mutual authentication.
	// Environment: NORNICDB_CLUSTER_TLS_VERIFY_CLIENT
	VerifyClient bool

	// InsecureSkipVerify skips server certificate verification.
	// WARNING: Only use for testing. Never in production!
	// Environment: NORNICDB_CLUSTER_TLS_INSECURE_SKIP_VERIFY
	InsecureSkipVerify bool

	// ServerName is the expected server name for certificate verification.
	// Environment: NORNICDB_CLUSTER_TLS_SERVER_NAME
	ServerName string

	// MinVersion is the minimum TLS version (default: TLS 1.2).
	// Environment: NORNICDB_CLUSTER_TLS_MIN_VERSION
	MinVersion string // "1.2" or "1.3"

	// CipherSuites restricts allowed cipher suites (optional).
	// Leave empty to use secure defaults.
	// Environment: NORNICDB_CLUSTER_TLS_CIPHER_SUITES
	CipherSuites []string
}

TLSConfig configures TLS/mTLS for secure replication connections. ALL replication traffic should use TLS in production environments.

type Transport

type Transport interface {
	// Connect establishes a connection to a peer.
	Connect(ctx context.Context, addr string) (PeerConnection, error)

	// Listen starts accepting connections from peers.
	Listen(ctx context.Context, addr string, handler ConnectionHandler) error

	// Close shuts down the transport.
	Close() error
}

Transport is the interface for peer-to-peer communication. This allows mocking in tests.

func NewDefaultTransport

func NewDefaultTransport(config *ClusterTransportConfig) Transport

NewDefaultTransport creates a ClusterTransport for production use. See transport.go for the full implementation.

func NewDefaultTransportFromConfig

func NewDefaultTransportFromConfig(cfg *Config) (Transport, error)

NewDefaultTransportFromConfig creates a ClusterTransport using replication config.

type VoteRequest

type VoteRequest struct {
	Term         uint64 `json:"term"`
	CandidateID  string `json:"candidate_id"`
	LastLogIndex uint64 `json:"last_log_index"`
	LastLogTerm  uint64 `json:"last_log_term"`
}

VoteRequest is sent by candidates to request votes.

type VoteResponse

type VoteResponse struct {
	Term        uint64 `json:"term"`
	VoteGranted bool   `json:"vote_granted"`
	VoterID     string `json:"voter_id"`
}

VoteResponse is the response to a vote request.

type WALApplier

type WALApplier struct {
	// contains filtered or unexported fields
}

WALApplier applies WAL entries to storage.

func NewWALApplier

func NewWALApplier(storage Storage) *WALApplier

NewWALApplier creates a new WAL applier.

func (*WALApplier) ApplyBatch

func (a *WALApplier) ApplyBatch(entries []*WALEntry) (uint64, error)

ApplyBatch applies a batch of WAL entries.

func (*WALApplier) Flush

func (a *WALApplier) Flush() error

Flush ensures all pending entries are applied.

type WALBatchResponse

type WALBatchResponse struct {
	AckedPosition    uint64
	ReceivedPosition uint64
}

WALBatchResponse is the response to a WAL batch.

type WALEntry

type WALEntry struct {
	// Position is the unique, monotonically increasing position.
	Position uint64

	// Timestamp when the entry was created.
	Timestamp int64

	// Command is the replicated command.
	Command *Command
}

WALEntry represents a write-ahead log entry.

type WALStreamer

type WALStreamer struct {
	// contains filtered or unexported fields
}

WALStreamer manages WAL streaming from primary.

func NewWALStreamer

func NewWALStreamer(storage Storage, batchSize int) *WALStreamer

NewWALStreamer creates a new WAL streamer.

func (*WALStreamer) AcknowledgePosition

func (w *WALStreamer) AcknowledgePosition(pos uint64)

AcknowledgePosition marks entries up to this position as acknowledged.

func (*WALStreamer) GetPendingEntries

func (w *WALStreamer) GetPendingEntries(maxEntries int) ([]*WALEntry, error)

GetPendingEntries returns WAL entries that haven't been acknowledged.

func (*WALStreamer) LastAcked

func (w *WALStreamer) LastAcked() uint64

LastAcked returns the last acknowledged WAL position.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL