ceph-volsync-plugin

ceph-volsync-plugin

A VolSync external mover plugin for Ceph storage, enabling efficient snapshot-based volume replication for CephFS and RBD volumes.

Overview

ceph-volsync-plugin extends VolSync with a Ceph-native data mover. Instead of copying entire volumes or relying on generic file-level rsync, it uses Ceph's snapshot diff APIs to identify and transfer only the changed data between replications.

It registers as an external mover with VolSync and handles ReplicationSource and ReplicationDestination custom resources for PVCs backed by supported Ceph CSI drivers.

Supported CSI providers:

Features

• Incremental sync — uses CephFS snapshot diff and RBD block diff APIs to transfer only changed data
• Concurrent pipeline — parallel gRPC data pipeline with configurable concurrency
• Direct TLS — data transfer uses ephemeral mTLS certificates exchanged at startup; stunnel handles only the initial handshake and rsync tunneling
• Zero-block optimization — zeroed and unchanged ranges are detected early and never transferred
• CephFS + RBD — CephFS routes non-empty regular files through the block pipeline and uses rsync for special files, zero-length files, and metadata; RBD operates directly on raw block devices
• OpenShift ready — automatically creates the required SecurityContextConstraints on OpenShift clusters

Architecture

  Source Cluster                         Destination Cluster
  ------------------------------         --------------------------------
  ReplicationSource CR                   ReplicationDestination CR
        |                                        |
  ReplicationSource                       ReplicationDestination
    Controller                               Controller
        |                                        |
  Ceph Mover Builder                      Ceph Mover Builder
        |                                        |
  K8s Job                                 K8s Job
  (Source Worker)                         (Destination Worker)
        |                                        |
        |<-- stunnel (handshake) + direct TLS -->|

Data flow:

1. Controllers watch ReplicationSource/ReplicationDestination CRs
2. The Ceph mover builder detects the CSI provider from spec.external.provider and creates a Mover
3. The mover orchestrates Kubernetes Jobs, PVCs, Services, and Secrets
4. Source worker reads snapshot diffs and streams changed blocks via gRPC
5. Destination worker receives and writes blocks, then commits

CephFS path: SnapshotDiffer categorises changes into non-empty regular files (block diff pipeline), special/zero-length files (rsync), deletions (gRPC Delete), and directory metadata (convergence rsync) -- all run concurrently.

RBD path: RBD diff iterator yields changed block ranges; these flow through a concurrent pipeline to the destination block device. Supports both full (no prior snapshot) and incremental (snapshot-to-snapshot) modes.

Prerequisites

• Kubernetes ≥ 1.28
• VolSync installed in the cluster
• Ceph cluster with ceph-csi drivers deployed
• VolumeSnapshot CRDs and external snapshot controller installed
• A StorageClass and VolumeSnapshotClass referencing a supported Ceph CSI driver

Quick Start

Install the operator

# Install CRDs (VolSync CRDs must already be present)
make install

# Deploy the operator using the published image
make deploy IMG=quay.io/ramendr/ceph-volsync-plugin-operator:latest \
  MOVER_IMG=quay.io/ramendr/ceph-volsync-plugin-mover:latest

Configure replication

Ceph credentials are automatically obtained from the CSI provisioner secret referenced in the ceph-csi-config ConfigMap — no manual secret creation required.

Create a ReplicationSource on the source cluster:

apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
  name: my-source
  namespace: my-namespace
spec:
  sourcePVC: my-pvc
  trigger:
    schedule: "*/30 * * * *"  # every 30 minutes
  external:
    provider: cephfs.csi.ceph.com
    parameters:
      storageClassName: cephfs-sc
      volumeSnapshotClassName: cephfs-snapclass
      copyMethod: Snapshot
      address: <copy from destination status.rsyncTLS.address>

Create a ReplicationDestination on the destination cluster:

apiVersion: volsync.backube/v1alpha1
kind: ReplicationDestination
metadata:
  name: my-dest
  namespace: my-namespace
spec:
  trigger:
    manual: first-sync
  external:
    provider: cephfs.csi.ceph.com
    parameters:
      storageClassName: cephfs-sc
      volumeSnapshotClassName: cephfs-snapclass
      copyMethod: Snapshot
      destinationPVC: my-dest-pvc

See docs/user-guide.md for a complete walkthrough including RBD volumes and incremental sync.

Configuration

CR external parameters

Set these in spec.external.parameters on both ReplicationSource and ReplicationDestination.

volumeName, baseSnapshotName, and targetSnapshotName are used only on the source with copyMethod: Direct. Snapshot mode tracks incremental state via snapshot status labels.

Operator environment variables

See docs/configuration.md for the full reference including mover environment variables and manager CLI flags.

Building

Protobuf generation is required before the first build:

make proto-generate        # generate gRPC/protobuf code (required first)
make generate              # generate deepcopy methods
make manifests             # generate CRDs and RBAC
make build                 # build manager binary

Container images:

make docker-build IMG=<registry>/ceph-volsync-plugin-operator:<tag>
make docker-build-mover MOVER_IMG=<registry>/ceph-volsync-plugin-mover:<tag>

# Multi-arch builds
make docker-buildx IMG=<registry>/ceph-volsync-plugin-operator:<tag>
make docker-buildx-mover MOVER_IMG=<registry>/ceph-volsync-plugin-mover:<tag>

Pinned versions: Go 1.25.7 · Ceph tentacle (quay.io/ceph/ceph:v20) · rsync 3.2.5 · stunnel 5.71

Testing

make test          # unit tests (uses envtest)
make test-e2e      # end-to-end tests (Kind cluster with Rook Ceph)
make lint          # golangci-lint
make lint-fix      # auto-fix linting issues

Local e2e tests use Kind (make test-e2e). CI uses Minikube. Both deploy Rook Ceph, install the snapshot controller, build and deploy the operator, then run the Ginkgo test suite. Requires Docker.

Project Structure

ceph-volsync-plugin/
├── cmd/
│   ├── manager/        # Kubernetes operator entry point
│   └── mover/          # Mover pod entry point (worker factory)
├── internal/
│   ├── ceph/           # go-ceph wrappers (CephFS diff, RBD diff, config)
│   ├── controller/     # ReplicationSource and ReplicationDestination controllers
│   ├── mover/ceph/     # Mover builder and K8s Job orchestration
│   ├── proto/          # gRPC/protobuf definitions (SyncService, VersionService)
│   └── worker/         # Data transfer workers
│       ├── cephfs/     # CephFS source and destination workers
│       ├── rbd/        # RBD source and destination workers
│       ├── pipeline/   # Concurrent Read->SendData pipeline
│       ├── tunnel/     # stunnel + rsync daemon management
│       ├── common/     # Shared worker interfaces and gRPC infrastructure
│       └── constant/   # Env var names, ports, paths
├── config/             # Kustomize: CRDs, RBAC, manager deployment
├── build/              # Containerfiles and build.env (pinned versions)
└── test/               # E2E tests and deployment scripts

Documentation

• Architecture — component diagram, data flow, pipeline internals
• User Guide — deployment, CephFS and RBD walkthroughs, troubleshooting
• Configuration Reference — all parameters, env vars, and CLI flags
• Contributing — dev setup, code generation workflow, PR process

License

Apache License 2.0 — see LICENSE.