rbd

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 1, 2026 License: MIT Imports: 19 Imported by: 0

README

RBD

Import github.com/JSYoo5B/ceph-testcontainers-go/rbd to give an application that uses RBD (librbd, go-ceph, the rbd CLI) a disposable Ceph cluster with the server side its integration test needs: initialized pools, namespaces, observation of image watchers and locks, backup transport between clusters and the immutable object cache. Image operations themselves (create, clone, snapshot, trash, migration, encryption, locks) belong to the client under test and are not wrapped. See API purpose and purpose tags.

go get github.com/JSYoo5B/ceph-testcontainers-go/rbd

Order of use

  1. Start a cluster with rbd.Run(ctx, ceph.DefaultImage, opts...). It is a ceph cluster (MON, MGR, two OSDs by default) with an initialized pool named rbd; rbd.WithPools selects other pools. On a cluster started by ceph.Run or another service package, prepare a pool with rbd.InitPool. Call WaitForClean before the first client I/O.
  2. Prepare the resources the scenario needs: rbd.CreateNamespace for tenant isolation, ceph.Container.CreateClient for least-privilege identities (profile rbd, profile rbd-read-only), rbd.StartImmutableObjectCache in a client container for parent caching.
  3. Hand the cluster to the SUT: run the application in a container attached with cluster.WithClient() or client.WithIdentity(), which provides ceph.conf and a keyring. Clients reach OSDs directly, so in the default bridge mode they must run in such a container.
  4. Observe and verify: rbd.ImageClients (watchers, lock owners), rbd.ListNamespaces, ImmutableObjectCacheLease.Status, the ceph observation APIs, or rbd.ExportBackup / rbd.RestoreBackup to move an image to another cluster and check it there.
  5. Clean up: terminate the cluster (testcontainers.CleanupContainer); namespaces and images go with it. Restore leases such as the immutable object cache before removing the caller-owned container they run in.
cluster, err := rbd.Run(ctx, ceph.DefaultImage)
testcontainers.CleanupContainer(t, cluster)
if err != nil {
	t.Fatal(err)
}
if err := cluster.WaitForClean(ctx); err != nil {
	t.Fatal(err)
}
app, err := testcontainers.Run(ctx, appImage, cluster.WithClient())
testcontainers.CleanupContainer(t, app)
if err != nil {
	t.Fatal(err)
}
// The application in app uses /etc/ceph/ceph.conf and the pool "rbd".

Concepts and limits

  • Everything in ceph applies to the returned cluster: clients and Cephx, pools, configuration, health, faults and topology changes. Mirroring between two clusters is in rbd/mirror.
  • Images: ceph.DefaultImage (Ceph Tentacle) supplies every role. With an explicit image set, essential supplies the daemons, client the ordinary rbd commands, and rbd-aux the immutable object cache and mirroring; see Releases and images.
  • No kernel RBD, rbd-nbd or /dev/mapper: containers run unprivileged with no host block devices. Native clients on the test host need host network mode on a Linux Docker host; see Host network.
  • Namespaces have no generation identity: do not delete and re-create a namespace that a descriptor still refers to. Incremental restore checks the baseline snapshot by name only.
  • Permissions: a read-only identity must open images read-only (a writable open registers a header watch, which needs write permission). A refused write through a read-only handle (EROFS) is a client-side check; to prove the server-side permission, test operations the server must deny (image creation, a direct RADOS write, another namespace: EPERM/EACCES). Cephx changes apply to new connections only.
  • Common pitfalls: one OSD with replica 1 passes WaitForClean but reports POOL_NO_REDUNDANCY; keep an override.AddBlocklist fence until the old lock owner has observed ESHUTDOWN.
Preparing for client features

The client feature under test decides what the fixture must provide:

Client feature Server-side preparation
Clone, flatten Format 2 images with layering; clone v1 needs a protected parent snapshot.
Trash move, restore, purge An initialized pool; a dedicated namespace so purges do not touch other tests' trash. The Python binding converts trash deferment times with local time: run it with TZ=UTC.
Live migration A replicated initialized pool, layering on source and target, source writers closed.
Group snapshots Format 2 images; flush and close writers first. It is a checkpoint of those images, not an application transaction.
Exclusive lock The exclusive-lock feature and two independent RADOS sessions; rbd.ImageClients shows the owner, fault.Pause and override.AddBlocklist freeze and fence it.
Encryption (LUKS1, LUKS2) librbd with libcryptsetup (client role and all images); no journaling on the image. librbd has no rekey API: run cryptsetup luksChangeKey on the exported header.
Erasure-coded data An EC pool with AllowOverwrites, passed as the image's data pool.
Backup to another cluster rbd.ExportBackup / ExportIncremental on one cluster, RestoreBackup / RestoreIncremental on the other.

Reference

Documentation

Overview

Package rbd prepares RBD server fixtures in disposable Ceph clusters: initialized image pools, namespaces and full or incremental backup transport. Mirroring between two clusters is in the rbd/mirror package.

Run starts a cluster with at least one initialized RBD pool. InitPool prepares a pool on a cluster created by ceph.Run or another service package's Run, and WithPools selects RBD pools for any of those Run functions.

Index

Examples

Constants

View Source
const (
	DefaultImmutableObjectCachePath    = "/var/cache/ceph/immutable-object-cache"
	DefaultImmutableObjectCacheSocket  = "/var/run/ceph/immutable-object-cache.sock"
	DefaultImmutableObjectCacheMaxSize = 1 << 30
)

Defaults of ImmutableObjectCache.

Variables

This section is empty.

Functions

func ExportBackup

func ExportBackup(ctx context.Context, client testcontainers.Container, image string, destination io.Writer) error

ExportBackup writes a CLI-generated format-2 archive to destination. The archive includes image metadata and snapshots. No mirror daemon is required. The caller owns the writer and decides where to retain the backup.

client is any container that has the rbd CLI and credentials for its own cluster, such as ceph.Container.ClientContainer or a container attached with WithClient, and image is an image spec such as volumes/disk or volumes/tenant-a/disk. The archive is written to a temporary file under /tmp in the client container and removed afterwards. The backup functions never close the caller's reader or writer, and they check cancellation between reads, so a reader or writer that can block indefinitely must support its own cancellation.

func ExportIncremental

func ExportIncremental(ctx context.Context, client testcontainers.Container, image, fromSnapshot string, destination io.Writer) error

ExportIncremental writes the CLI diff from fromSnapshot to image (which can include @snapshot). Restore requires the matching baseline snapshot. fromSnapshot is a bare snapshot name. With image@snapshot the diff ends at that snapshot, and restoring it creates that snapshot on the target. The client and stream rules are those of ExportBackup.

func InitPool

func InitPool(ctx context.Context, c *ceph.Container, pool string) error

InitPool initializes an existing replicated pool for RBD metadata. Pools registered to other applications are rejected; no force option is used. An existing RBD initialization is safe to repeat. EC pools may supply image data but cannot hold RBD metadata. Create the pool first, then wait for clean PGs before client I/O. Partial initialization never deletes pool data. Success is verified natively, by the pool's application tag and its rbd_trash object, not only by the CLI exit code. Pool names may contain letters, digits, _, . and -, must not start with . or -, and are at most 128 characters.

Example
package main

import (
	"context"
	"log"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/JSYoo5B/ceph-testcontainers-go/rbd"
	"github.com/testcontainers/testcontainers-go"
)

func main() {
	ctx := context.Background()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage,
		ceph.WithClusterConfig(ceph.ClusterConfig{}.WithPools(ceph.PoolConfig{Name: "images", Application: "rbd"})),
	)
	defer func() {
		if err := testcontainers.TerminateContainer(cluster); err != nil {
			log.Printf("failed to terminate container: %s", err)
		}
	}()
	if err != nil {
		panic(err)
	}
	if err := rbd.InitPool(ctx, cluster, "images"); err != nil {
		panic(err)
	}
	namespace, err := rbd.CreateNamespace(ctx, cluster, "images", "tenant-a")
	if err != nil {
		panic(err)
	}
	// Pass namespace.PoolName() and namespace.Name() to a native RBD client.
	// Images, snapshots and trash must be removed before deleting the namespace.
	if err := rbd.RemoveNamespace(ctx, cluster, namespace); err != nil {
		panic(err)
	}
}

func ListNamespaces

func ListNamespaces(ctx context.Context, c *ceph.Container, pool string) ([]string, error)

ListNamespaces returns the pool's native namespace names in sorted order, including names created outside this fixture. The unnamed default namespace is omitted by Ceph. Listing does not grant ownership for removal.

func RemoveNamespace

func RemoveNamespace(ctx context.Context, c *ceph.Container, ns *Namespace) error

RemoveNamespace removes a confirmed namespace owned by this cluster. Ceph refuses nonempty namespaces, including images in trash. No image purge or force option is used. Successful removal is idempotent; failed CLI removal can be retried with the same descriptor and a new context. Replacing its underlying pool is rejected. Do not delete and recreate the same named namespace through another client while its descriptor is in use: native RBD namespaces have no separate generation identity to distinguish replacements.

func RestoreBackup

func RestoreBackup(ctx context.Context, client testcontainers.Container, image string, archive io.Reader) error

RestoreBackup imports a format-2 archive into a new image on the client's own cluster. Pool creation and archive retention belong to the caller. It first spools the reader into a private temporary file on the test host, so a large archive is not held in memory, then copies it into the container; the client and stream rules are those of ExportBackup.

func RestoreIncremental

func RestoreIncremental(ctx context.Context, client testcontainers.Container, image string, archive io.Reader) error

RestoreIncremental applies a CLI diff to an existing image. Ceph checks that its baseline snapshot exists; this function does not invent a baseline. The check is by snapshot name only: nothing proves that the baseline has the same content.

func WithPools

func WithPools(configs ...ceph.PoolConfig) ceph.Option

WithPools creates replicated pools for RBD images during Run and initializes their RBD metadata. Application defaults to rbd and must not be another application; erasure-coded data pools are created with ceph.ClusterConfig.WithPools and referenced as an image's data pool. Repeated options replace the list, and passing the option replaces the default rbd pool of Run. The other ceph.PoolConfig fields keep their ceph meaning: a zero PGNum selects 8 PGs with the autoscaler off, zero Replicas and MinSize select the cluster defaults (2/1, or 1/1 with one OSD), and the placement fields are validated against the OSDs like those of any other pool.

Types

type Container

type Container struct {
	*ceph.Container
}

Container is a cluster started by Run. It embeds *ceph.Container, so every cluster method is available on it; pass its Container field to functions that take a *ceph.Container, such as those of this and the other service packages.

func Run

Run starts a disposable cluster for RBD clients. It accepts every ceph.Run option and the options of other service packages. When no WithPools option is given, it creates and initializes a replicated pool named rbd. Bootstrap options that leave the cluster without OSDs for the pool, such as ceph.ClusterConfig.WithNoInitialOSDs, are rejected; use ceph.Run for those stages. A non-nil Container returned with an error must still be terminated.

Run returns after the pools exist and their RBD metadata is initialized. PG peering may still be finishing, so call WaitForClean before the first client I/O.

Example
package main

import (
	"context"
	"log"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/JSYoo5B/ceph-testcontainers-go/rbd"
	"github.com/testcontainers/testcontainers-go"
)

func main() {
	ctx := context.Background()
	// Without WithPools, Run creates and initializes the replicated pool rbd.
	cluster, err := rbd.Run(ctx, ceph.DefaultImage,
		rbd.WithPools(ceph.PoolConfig{Name: "volumes"}),
	)
	defer func() {
		if err := testcontainers.TerminateContainer(cluster); err != nil {
			log.Printf("failed to terminate container: %s", err)
		}
	}()
	if err != nil {
		panic(err)
	}
	// Connect a native RBD client with cluster.ConnectionConfig().
}

type ImageClientStatus

type ImageClientStatus struct {
	Watchers []ImageWatcher
	Locks    []ImageLock
}

ImageClientStatus reports an image's watchers and locks. The two lists come from separate native reads, so a lock handoff between them can show a new owner without its watch or the reverse; poll for a stable answer.

func ImageClients

func ImageClients(ctx context.Context, c *ceph.Container, pool, namespace, image string) (*ImageClientStatus, error)

ImageClients reads which clients watch an image and which hold its locks, including clients outside this fixture. Namespace is empty for the default namespace. It reads native state only and grants no ownership.

func (*ImageClientStatus) ExclusiveOwner

func (status *ImageClientStatus) ExclusiveOwner() *ImageLock

ExclusiveOwner returns the managed exclusive lock, or nil if no client holds it.

type ImageLock

type ImageLock struct {
	// ID is the lock ID; for a managed lock it is "auto <watch cookie>".
	ID string
	// Locker is the owner's entity name, such as client.4235.
	Locker string
	// Address is the owner's IP:port/nonce, the form that
	// override.AddBlocklist takes.
	Address string
	// Managed marks the lock that librbd takes for exclusive-lock.
	Managed bool
}

ImageLock is one lock on an image header. Managed marks the lock librbd takes for the exclusive-lock feature; its ID carries the owner's watch cookie. Other locks are advisory locks added through rbd lock add.

type ImageWatcher

type ImageWatcher struct {
	Address  string
	ClientID uint64
	Cookie   uint64
}

ImageWatcher is one client watching an image header. Address is the client's IP:port/nonce, the same form BlocklistEntries reports. A client that stops answering keeps its watch until the OSD's osd_client_watch_timeout expires, 30 seconds by default.

type ImmutableObjectCache added in v0.4.0

type ImmutableObjectCache struct {
	// ID is the cephx ID, client.<ID>, that the daemon uses to read the
	// parent objects; empty means admin. The container's ceph.conf must name
	// its keyring, as ceph.ClientConfig.WithIdentity does.
	ID string
	// Path is the cache directory; empty means
	// DefaultImmutableObjectCachePath. It must be absent or empty.
	Path string
	// Socket is the path of the daemon's client socket; empty means
	// DefaultImmutableObjectCacheSocket. It must not exist, and it must not
	// be under Path.
	Socket string
	// MaxSize is immutable_object_cache_max_size in bytes; 0 means
	// DefaultImmutableObjectCacheMaxSize. The daemon treats it as a target,
	// not a hard limit.
	MaxSize int64
}

ImmutableObjectCache configures the RBD immutable object cache daemon, ceph-immutable-object-cache, that StartImmutableObjectCache starts in a client container. The daemon caches whole parent-image objects of clones for the librbd clients in that container that enable the parent cache.

type ImmutableObjectCacheLease added in v0.4.0

type ImmutableObjectCacheLease struct {
	// Config is the request with its defaults filled in.
	Config ImmutableObjectCache
	// contains filtered or unexported fields
}

ImmutableObjectCacheLease owns one daemon that StartImmutableObjectCache started, with its socket and cache directory. It signals the process only while its command line holds the lease's socket, so a reused PID is never stopped. To count cache use, read Status.RADOSReads: rados df read counters lag behind and do not show it.

func StartImmutableObjectCache added in v0.4.0

func StartImmutableObjectCache(ctx context.Context, client testcontainers.Container, config ImmutableObjectCache) (*ImmutableObjectCacheLease, error)

StartImmutableObjectCache starts the RBD immutable object cache daemon in client, a container that the caller owns, and returns its lease. A librbd client in the same container uses the cache when it sets the lease's ClientOptions; other clients read the parents from RADOS. The container must have ceph-immutable-object-cache on its PATH and the librbd parent_cache plugin in its plugin directory: the official Ceph image has both, and an image without them is refused with ceph.ErrImageCapability. An existing socket, a non-empty cache directory or an invalid path is refused with ceph.ErrInvalidRequest, and nothing is changed. A daemon that exits or never opens its socket fails the call with ceph.ErrNativeFailure, and what it left is removed.

func (*ImmutableObjectCacheLease) ClientOptions added in v0.4.0

func (l *ImmutableObjectCacheLease) ClientOptions() map[string]string

ClientOptions returns the Ceph options that make a librbd client in the lease's container use the cache, for a ceph.conf section, the rbd CLI (as --name=value) or rados.Rados(conf=...).

func (*ImmutableObjectCacheLease) Restore added in v0.4.0

Restore stops the daemon if it runs and removes its socket, admin socket, PID file and cached objects. It removes the cache directory only when the lease created it. A repeated call after success does nothing.

func (*ImmutableObjectCacheLease) Status added in v0.4.0

Status reports the daemon and the cache directory.

func (*ImmutableObjectCacheLease) Stop added in v0.4.0

Stop stops the daemon, with SIGTERM and then SIGKILL, and keeps the cache directory and socket file: librbd clients that open an image afterwards fail to register with the cache and read the parents from RADOS. A daemon that already exited needs no stop.

type ImmutableObjectCacheStatus added in v0.4.0

type ImmutableObjectCacheStatus struct {
	// Running means that the daemon the lease started still runs.
	Running bool
	// PID is the daemon's process ID in the client container.
	PID int
	// CachedObjects and CachedBytes count the files under the cache
	// directory: one per cached parent object.
	CachedObjects int
	CachedBytes   int64
	// RADOSReads is the daemon's objecter read count while Running: the
	// parent objects it read from RADOS to cache them. The daemon has no
	// cache hit counters of its own.
	RADOSReads uint64
}

ImmutableObjectCacheStatus is what the client container reports for the daemon and its cache directory.

type Namespace

type Namespace struct {
	// contains filtered or unexported fields
}

Namespace identifies an image namespace created by this cluster. Names and native pool identity are immutable. Namespace removal never deletes images, snapshots or trash; callers must remove those through their RBD client first. The cluster disposes of namespaces with all its data when it terminates.

func CreateNamespace

func CreateNamespace(ctx context.Context, c *ceph.Container, pool, name string) (*Namespace, error)

CreateNamespace creates a fresh image namespace in an initialized, replicated RBD pool. Existing names are rejected without changing their data. A non-nil descriptor returned with an error identifies an attempted creation, but cannot authorize removal unless creation was confirmed. Inspect native state or dispose of the cluster after an uncertain creation failure. Namespace names follow the rules of InitPool's pool names. Namespaces need not be removed for cleanup: terminating the cluster disposes of them with all their data.

func (*Namespace) Name

func (ns *Namespace) Name() string

Name returns the namespace name used by librbd and the RBD CLI.

func (*Namespace) PoolName

func (ns *Namespace) PoolName() string

PoolName returns the replicated metadata pool containing the namespace.

Directories

Path Synopsis
internal
resourcename
Package resourcename validates the pool, namespace and image names accepted by standalone RBD operations.
Package resourcename validates the pool, namespace and image names accepted by standalone RBD operations.
Package mirror connects two Ceph clusters with native RBD mirroring: setup CLI containers, one-way pool or image mirroring in snapshot or journal mode, any number of rbd-mirror receiver daemons, and typed readiness observation.
Package mirror connects two Ceph clusters with native RBD mirroring: setup CLI containers, one-way pool or image mirroring in snapshot or journal mode, any number of rbd-mirror receiver daemons, and typed readiness observation.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL