ceph

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: MIT Imports: 51 Imported by: 0

README

Ceph

The Testcontainers module for Ceph. It starts a small, disposable Ceph cluster for the integration tests of a service that uses Ceph.

What it is for

The module builds the Ceph side of an integration test, independently of the system under test and of the test code, and lets the test check afterwards that the cluster ended in the expected state. It reproduces as many operational conditions as it can, including some injected faults, but it is not a full Ceph deployment: large-scale experiments, performance measurement and tuning are out of scope, and it does not wrap object, image, file or S3 data APIs. Your service talks to the cluster with its own client (librados, go-ceph, librbd, libcephfs, the Ceph CLI or an S3 SDK). See API purpose and purpose tags.

Real ceph-mon, ceph-mgr and ceph-osd daemons run in separate containers on an isolated Docker network: one MON, one MGR and two OSDs by default, each OSD on a sparse BlueStore file inside its container. The daemons need no privileged mode, host disks, LVM, Docker socket or systemd; Testcontainers itself (and Ryuk) still needs the Docker engine. Dependencies, requirements, installation and the versioning policy are in the root README.

Usual order of use

  1. Start the cluster. ceph.Run(ctx, img, opts...) with ceph.DefaultImage (or role images through WithImageOptions) builds and waits for the whole topology. Settings are grouped values, each passed through one option: WithClusterConfig (MON/MGR/OSD counts, OSD layout, CRUSH root, pool defaults, initial pools), WithOSDStorage, WithMonitorStorage, WithNetworkConfig, WithImageOptions, WithZoneConfig (logical zones behind a routed boundary), plus WithAuth, WithConfigFile, WithHostNetwork, WithManagerPorts and WithStartupTimeout. Contradictory options fail before anything is created. For small functional OSDs, select OSDStorage{}.WithStartupBenchmark(OSDBenchmarkSkip) explicitly; OSD startup calibration explains native controls, configuration precedence and backing-storage costs. Register cleanup before checking the error: a partially started cluster is returned with the error and still owns containers.
  2. Prepare the state the test needs. WaitForClean before client I/O; CreatePool and the Pool handle (replication, quota, placement, PG count); CreateClient for a Cephx identity with exact caps; AddOSD, AddMonitor, AddManager and the daemon handles' Remove, Detach, Replace, SetIn for topology. The override package changes configuration with restoration handles, the fault package injects faults, and the rbd, cephfs and rgw packages add their services to the same Run.
  3. Hand the connection to the system under test. WithClient or ClientConfig.WithIdentity attach a client container to the cluster network with ceph.conf and a keyring; ConnectionConfig returns the same material for a process that connects itself (host mode, see Host networking). FactsOf reports the images, network and Ceph version.
  4. Observe and verify. Observe returns a resource-free Observer for readiness waits, tracked optional-service inventory, health, PG and pool reports, quorum, OSD states, MGR modules and services, watchers and blocklist entries. When no typed observation exists, Ceph runs the raw Ceph CLI in the control container, retrying only the connection to a MON (bounded by ctx and a minute), and RunNativeCommand runs a command under a supervisor that bounds its run time and output. The waits (WaitForClean, WaitForPGClean, WaitForQuorum, WaitMGRModuleReady) keep the cluster's startup timeout. The watches (WatchOSDs, Pool.WatchPGs, WatchHealth, WatchMonitors, WatchManagers) follow state while the SUT runs; their shared lifecycle and what each one fixes by its baseline are in watching state. Pool.OverridePGPrimary aims a fault at a known serving OSD (see fault injection). To check one captured observation against the condition the test states, use ceph/expect, for example expect.PGs.
  5. Finish. testcontainers.CleanupContainer runs Terminate, which removes every owned container, network and volume. The finalize package replaces it when the test needs post-test checks (with crash and cluster-log verification), a failure archive and caller cleanups in reverse order before Terminate; the diagnostics package collects the same reports standalone.
func TestRADOSRoundTrip(t *testing.T) {
	ctx := t.Context()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage,
		ceph.WithClusterConfig(ceph.ClusterConfig{}.WithPools(ceph.PoolConfig{Name: "app-data", Application: "rados"})),
	)
	testcontainers.CleanupContainer(t, cluster) // also owns a partial cluster
	if err != nil {
		t.Fatal(err)
	}
	observer, err := ceph.Observe(cluster)
	if err != nil {
		t.Fatal(err)
	}
	if err := observer.WaitForClean(ctx); err != nil { // Run does not wait for clean PGs
		t.Fatal(err)
	}
	// The client image must contain the tools the test uses. WithClient joins
	// the cluster network and copies ceph.conf and the admin keyring.
	client, err := testcontainers.Run(ctx, ceph.DefaultImage, cluster.WithClient(), ceph.WithIdleEntrypoint())
	testcontainers.CleanupContainer(t, client) // registered later, so removed first
	if err != nil {
		t.Fatal(err)
	}
	// ... run the system under test against the client, then check the result,
	// for example with ceph.Observe(cluster) or the client's own reads.
}

Starting a cluster takes tens of seconds and pulls a large image the first time; give go test a generous -timeout when several clusters run in one package.

Concepts and limits

  • One fixture per test. Call Run for each test and create the pools, namespaces and identities it needs. Stop clients the test attached before the cluster is terminated, and use a fresh bounded context in cleanups: t.Context() has ended by then. Keep non-nil handles returned with errors; an uncertain operation may still own a resource. See Contexts and cleanup.
  • Configuration composition. A grouped value records which fields were set; a later call in one value replaces an earlier one, and two carriers that set the same field are refused, even when they agree.
  • Images. One aggregate image supplies every role by default; an ImageSet selects essential, client, CephFS, RBD-aux and RGW images separately, and a missing optional role never falls back. Its MGR role is the one with a default: every manager runs the essential image unless ImageSet.MGR selects another, such as the official image or one with additional modules. See Releases and images.
  • Networks. Bridge mode by default: only MON a (and RGW gateways, and the MGRs with WithManagerPorts for a host-side client of a module listener such as the Dashboard) publishes ports, so native RADOS, RBD and CephFS clients run in a WithClient container. Host mode lets a process on a Linux Docker host connect directly. IPv6, dual-stack and a separate cluster network are bridge-mode options.
  • Placement is logical. All daemons share one Docker engine; CRUSH hosts, racks and zones test placement rules, not physical failure tolerance, and OSD files are sparse, so fullness is simulated with full ratios.
  • Regional zones. WithZoneConfig places one cluster's daemons and clients in logical availability zones with an owned routed boundary between them, which fault.ApplyZoneProfile delays or cuts and Container.Zones reports; Container.WithZone places a client or service daemon. See Regional zones.
  • Fixture defaults differ from a stock cluster (small pools, PG autoscaling off, replica-1 pools allowed, faster MGR stats); the configuration guide lists them.
  • Errors carry categories (ErrInvalidRequest, ErrUnsupported, ErrImageCapability, ErrEnvironmentUnavailable, ErrNativeFailure) and OperationError / NativeFailure details; see Errors.

Reference

Documentation

Overview

Package ceph provisions disposable Ceph clusters using testcontainers-go. It controls Ceph through container-local CLIs without linking host librados.

Run starts the MON, MGR and OSD topology with cluster-wide options: daemon counts, OSD storage and placement, networks, pools and Messenger mode. Pool policies, Cephx identities, configuration overrides, fault injection and health observations operate on the returned Container.

Client services live in their own packages, which add their options to the same Run and declare their own service types:

  • github.com/JSYoo5B/ceph-testcontainers-go/rbd: RBD pools, namespaces and backup.
  • github.com/JSYoo5B/ceph-testcontainers-go/cephfs: filesystems, MDS topology and subvolumes.
  • github.com/JSYoo5B/ceph-testcontainers-go/rgw: gateways, users, placement and storage classes.

Each of these service packages has its own Run that starts a cluster with that service's defaults. Options from several of them can be passed to one ceph.Run to start a cluster that serves them together.

Fixtures that span two existing clusters take those clusters in their own configuration instead; their options are not ceph.Run options:

  • github.com/JSYoo5B/ceph-testcontainers-go/rbd/mirror: rbd-mirror replication.
  • github.com/JSYoo5B/ceph-testcontainers-go/cephfs/mirror: CephFS snapshot mirroring.
  • github.com/JSYoo5B/ceph-testcontainers-go/rgw/multisite: RGW multisite realms, zones and sync policies.

Optional features that act on a running cluster have their own packages, so a cluster that uses none of them imports none:

  • github.com/JSYoo5B/ceph-testcontainers-go/override: reversible overrides of cluster policy.
  • github.com/JSYoo5B/ceph-testcontainers-go/fault: fault injection.
  • github.com/JSYoo5B/ceph-testcontainers-go/capture: packet capture.
  • github.com/JSYoo5B/ceph-testcontainers-go/ceph/diagnostics: read-only diagnostics reports, crash metadata and redaction.
  • github.com/JSYoo5B/ceph-testcontainers-go/ceph/finalize: the ordered end of a test's cluster with post-test checks, verification, a failure archive and termination.

These rules hold across the methods of a Container and its handles:

  • Methods are bounded by the caller's context and most also by WithStartupTimeout. Topology changes are serialized, and waiting for a concurrent one honours the context. A call canceled while waiting returns without native changes; one canceled after it sent a native command can leave that command applied.
  • An operation on one daemon, pool or Cephx identity is a method of its handle (OSDContainer, DetachedOSD, MonitorContainer, ManagerContainer, Pool, ClientConfig) and acts on the instance the handle names. The handle of a removed, detached, replaced or deleted resource, or of a terminated cluster, returns an error. Functions that create handles and cluster-wide operations stay on Container.
  • A method that creates something can return a non-nil result with an error. The result identifies what was attempted and stays owned by the cluster; inspect it, retry, or terminate the cluster. No method deletes data to roll back a partial failure.
  • Read methods return the native state at the moment of the call; several reads are not one atomic snapshot, and health, PG statistics and usage are reported asynchronously, so poll with a deadline. Several observations verify the cluster's original FSID and pool IDs.
  • Ceph has no compare-and-swap for configuration, flags, CRUSH or auth, so do not change the same state with raw commands while a helper applies or restores it.
  • Temporary changes (package override, package fault and ClientConfig.Revoke) return restoration handles, except the one-shot fault.InjectObjectDataError; keep a non-nil handle even with an error. Their shared contract is in docs/guides/contexts-and-cleanup.md.

This is an experimental module for application integration tests.

Index

Examples

Constants

View Source
const (
	MaxNativeCommandTimeout = time.Hour
	MaxNativeCommandOutput  = 64 << 20
	MaxNativeCommandInput   = 64 << 20
	// NativeCommandStopTimeout bounds one attempt to stop a command inside its
	// container: the canceler and, when it fails, one inspection of the
	// container. The attempt has this budget of its own, independent of the
	// caller's context, which has usually ended.
	NativeCommandStopTimeout = supervised.CancelTimeout + nativeInspectTimeout
)

Limits of a NativeCommand.

View Source
const CrashDir = evidence.CrashDir

CrashDir is the crash directory of every daemon of the cluster. The bootstrap ceph.conf sets it, and WithConfigFile refuses to change it.

View Source
const DefaultImage = "ghcr.io/jsyoo5b/ceph-testcontainers-images:official-20.2.4-all"

DefaultImage is the published official Tentacle aggregate, which supplies every dependency role on both native architectures. It is a release tag of ceph-testcontainers-images: a compatible rebuild replaces the image under the same tag, and the Docker engine's pull and cache policy decides when a host picks it up. Pass a digest reference or a local image ID to Run for an exact build. Explicit ImageSet selections avoid unused feature roles.

View Source
const MaxNativeFailureReason = 512

MaxNativeFailureReason bounds NativeFailure.Reason.

Variables

View Source
var (
	// ErrInvalidRequest identifies a classified argument or state refusal,
	// such as a terminated cluster or a container outside its ownership and
	// attachments. A refusal can follow native reads or partial setup. Consult
	// the operation's contract for preflight checks and partial handles.
	ErrInvalidRequest = errors.New("invalid request")
	// ErrUnsupported reports a native feature that the Ceph of the relevant
	// target lacks, established before any change from that target's own
	// capability, such as an option that its tool does not list, or from a
	// documented limit of the release it runs. AddressDualStack establishes
	// it after startup instead, from the addresses that the daemons
	// advertise. *OperationError names the feature, the target and its
	// version.
	ErrUnsupported = errors.New("unsupported by the target's Ceph")
	// ErrImageCapability reports an image that lacks what the operation
	// needs: an executable or runtime, such as the python3 of the
	// supervisor, or an optional extension that was not selected or does not
	// meet its contract.
	ErrImageCapability = errors.New("image capability unavailable")
	// ErrEnvironmentUnavailable reports that the Docker engine, the host, the
	// kernel or the permissions of a container cannot provide what the
	// operation needs, although the image may be valid. When it comes from a
	// call that had sent a native command, the command may have run.
	ErrEnvironmentUnavailable = errors.New("runtime environment unavailable")
	// ErrNativeFailure reports a native command that ran and failed: Ceph
	// refused the request, a resource was missing, permission was denied or
	// I/O failed. *NativeFailure carries the native status.
	ErrNativeFailure = errors.New("native operation failed")
)

Error categories. A classified error matches one of them with errors.Is, and errors.As with *OperationError or *NativeFailure returns its details. An error joined from several failures, such as a failed change and a failed restoration, can match several. Context errors, ErrNativeCommandIncomplete and *NativeCommandUnstoppedError keep their own contracts and belong to no category: a call that ctx ended is never classified by what it could not finish. An error that matches no category is a failed check of the fixture's own, such as output it could not parse, or a failure whose outcome is unknown, such as a container that disappeared after a native command was sent to it.

View Source
var ErrHostNetworkUnavailable error = &categoryError{text: "ceph host network is unavailable to the test process", kind: ErrEnvironmentUnavailable}

ErrHostNetworkUnavailable reports that host mode cannot serve native clients in the test process: the engine does not run Linux containers, the selected address is not assigned on the Docker host, or the first MON's advertised endpoint is not reachable from this process. Run returns it before creating MGR/OSD daemons; later MON/RGW port reservations may also return it. It belongs to ErrEnvironmentUnavailable.

View Source
var ErrNativeCommandIncomplete = errors.New("native command did not complete within its limits")

ErrNativeCommandIncomplete marks a NativeCommandResult that timed out or whose output was truncated: its bytes are not the command's complete output.

View Source
var ErrNativeInputClosed = errors.New("native command input is closed")

ErrNativeInputClosed reports a write to a NativeInput that was closed, or whose command no longer takes input.

View Source
var ErrWatchOverflow = watch.ErrOverflow

ErrWatchOverflow ends a watch whose consumer left the event buffer full for one polling interval while an event waited. The watch stops instead of replacing or dropping events; the error names the undelivered event.

Functions

func WithIdleEntrypoint

func WithIdleEntrypoint() testcontainers.CustomizeRequestOption

WithIdleEntrypoint keeps a client or helper container running with sleep infinity under Docker's init process, so commands can be executed in it later. As PID 1, sleep ignores SIGTERM and Docker waits for the full stop timeout (10 seconds by default) before killing it; init forwards the signal so termination finishes at once. Init also reaps background processes that exit inside the container, such as a test server started through Exec. The entrypoint and command remain sleep and infinity.

Types

type AddressFamily added in v0.2.0

type AddressFamily int

AddressFamily selects the addresses that the Messenger of every daemon binds and advertises.

const (
	// AddressIPv4 binds and advertises the containers' IPv4 addresses.
	AddressIPv4 AddressFamily = iota
	// AddressIPv6 runs the cluster on a dual-stack bridge and binds and
	// advertises only the containers' IPv6 addresses.
	AddressIPv6
	// AddressIPv4MappedIPv6 keeps an IPv4 bridge and binds and advertises the
	// containers' IPv4 addresses as IPv4-mapped IPv6 addresses
	// (::ffff:a.b.c.d) on IPv6 sockets.
	AddressIPv4MappedIPv6
	// AddressDualStack runs the cluster on a dual-stack bridge, and every
	// Messenger binds and advertises both the container's IPv4 and its IPv6
	// address, so that clients of either family can connect. MON address
	// vectors list the IPv6 address first; see NetworkConfig.WithAddressFamily.
	AddressDualStack
)

func (AddressFamily) String added in v0.2.0

func (family AddressFamily) String() string

type AuthConfig added in v0.2.0

type AuthConfig struct {
	// KeyType is the type of the keys the fixture generates itself: the
	// mon., client.admin and OSD keys and, unless ClientOptions.WithKeyType selects
	// another, CreateClient keys.
	KeyType CephxCipher
	// ServiceCipher encrypts service tickets. AllowedCiphers are the key
	// types that the MONs accept for authentication, and PreferredCipher is
	// the type the MONs use for keys they generate with auth get-or-create,
	// such as MGR, MDS and RGW keys. These MonMap settings need a KeyType and,
	// for anything but CipherAES, the image support that CipherAES256K
	// describes. ServiceCipher and PreferredCipher
	// default to KeyType, and AllowedCiphers to the distinct types among
	// KeyType, ServiceCipher and PreferredCipher. AllowedCiphers must contain
	// all three.
	ServiceCipher   CephxCipher
	AllowedCiphers  []CephxCipher
	PreferredCipher CephxCipher
	// MonTicketTTL and ServiceTicketTTL set auth_mon_ticket_ttl and
	// auth_service_ticket_ttl, in seconds with fractions kept, such as
	// 10500 * time.Millisecond. MONs rotate service secrets on their ticks, so
	// lifetimes of a few seconds also need a shorter mon_tick_interval, which
	// WithConfigFile can set. Keep MonTicketTTL at ten seconds or more: MON
	// clients renew their tickets only every few seconds, and a daemon whose
	// renewal comes too late is refused and does not authenticate again.
	MonTicketTTL     time.Duration
	ServiceTicketTTL time.Duration
}

AuthConfig selects the Cephx settings of a new cluster. Zero fields keep the defaults of the image's Ceph release.

type BlocklistEntry

type BlocklistEntry struct {
	Address string
	Range   bool
	Until   time.Time
}

BlocklistEntry reports native OSDMap fencing, including ranges created by external tools. Until is the MON's absolute expiration time. A listing grants no ownership to remove an existing entry.

type CephxCipher added in v0.2.0

type CephxCipher string

CephxCipher names a Cephx key type, which is also the cipher of the tickets made with such keys. A decoded key records its type in its first two bytes, little-endian: 1 for aes and 2 for aes256k.

const (
	// CipherAES is AES-128-CBC, the only key type of Squid and of Tentacle
	// before 20.2.4.
	CipherAES CephxCipher = "aes"
	// CipherAES256K is AES-256 with Kerberos-style key derivation
	// (AES256KRB5). It needs an image whose ceph-authtool has --key-type and
	// whose monmaptool has the cipher options, as in Ceph 20.2.4 and later,
	// where it is the default.
	CipherAES256K CephxCipher = "aes256k"
)

type ClientCaps

type ClientCaps struct {
	Mon string
	OSD string
	MGR string
	MDS string
}

ClientCaps contains Cephx capability expressions, passed as individual CLI arguments. For example, Mon: "allow r" and OSD: "allow rw pool=data namespace=tenant-a" restrict RADOS I/O to one pool and namespace. Empty fields grant no capabilities for that service. Capabilities are never broadened by this helper; Ceph validates their syntax.

type ClientConfig

type ClientConfig struct {
	// contains filtered or unexported fields
}

ClientConfig describes a client identity created by this cluster. Credentials remain private; ConnectionConfig returns copies for an external native client. The cluster owns the auth database, while callers own attached containers.

func (*ClientConfig) Capabilities added in v0.4.0

func (client *ClientConfig) Capabilities(ctx context.Context) (ClientCaps, error)

Capabilities returns the current caps of this confirmed, owned identity. It verifies the native key without returning auth database secrets.

func (*ClientConfig) ConnectionConfig

func (client *ClientConfig) ConnectionConfig() (ConnectionConfig, error)

ConnectionConfig returns independent ceph.conf and keyring copies. The config is a snapshot at creation; ClientConfig.WithIdentity uses the current MON addresses. Revocation does not erase already copied credentials or invalidate established sessions immediately. Test revoked credentials with a fresh connection. On error it returns the zero ConnectionConfig.

func (*ClientConfig) Delete added in v0.4.0

func (client *ClientConfig) Delete(ctx context.Context) error

Delete revokes this identity, which the cluster created, without deleting data. It checks for an externally replaced key before deleting; external auth edits must not race this operation. A repeat call after success is a no-op; existing sessions may retain issued tickets. The removal is fenced like the import of ReplaceKey: a failed call reports what the MONs hold once settled, and an uncertain one is resolved by the next operation on the identity.

func (ClientConfig) GoString

func (client ClientConfig) GoString() string

GoString excludes credentials from formatted Go representations as well.

func (*ClientConfig) KeyringPath

func (client *ClientConfig) KeyringPath() string

KeyringPath returns the per-identity path used in an attached client container.

func (*ClientConfig) Name

func (client *ClientConfig) Name() string

Name returns the fully qualified Cephx entity, for example client.reader.

func (*ClientConfig) ReplaceKey added in v0.4.0

func (client *ClientConfig) ReplaceKey(ctx context.Context, opts ...ClientOptions) (*ClientConfig, error)

ReplaceKey gives this client identity, which the cluster created, a fresh Cephx key and keeps its name and its exact capabilities, in one auth import, so that the MONs never lack the identity. The identity needs a capability, as auth import refuses an entry without any. The new key has the type of the old one unless ClientOptions.WithKeyType selects another. It returns a new descriptor for the new key, and the old descriptor goes stale: its ConnectionConfig still returns the old credentials, for tests that check their rejection. Repeating ReplaceKey on it returns the same successor while that successor is current. WithIdentity, Capabilities, UpdateCaps and Delete refuse the stale descriptor; use the successor. Containers that were attached with the old descriptor keep the old keyring.

A fresh connection with the old key fails at once. A client that authenticated before keeps its session and its tickets until it renews them within auth_mon_ticket_ttl (AuthConfig.MonTicketTTL), when the MONs reject the old key: Tentacle then drops the client's MON session, while Squid keeps an open one answering. The new key works for fresh MON and MGR clients at once.

The identity must still have the key of client; one changed outside the module is refused. External auth changes must not race this call. The import is fenced like the MON commands of RotateKey: when it fails, the call settles the MONs and reads whether it applied. When that stays uncertain, the call returns a descriptor of the attempted key that stays unusable, with the error, and a later call with client, or any other operation on the identity, settles first: ReplaceKey then returns a usable descriptor of that key once the import turns out to have applied, or replaces the key anew once it turns out not to have. A returned descriptor never changes afterwards.

func (*ClientConfig) Revoke added in v0.4.0

func (client *ClientConfig) Revoke(ctx context.Context) (*ClientRevocationHandle, error)

Revoke removes this client identity, which the cluster created, from the MONs until Restore puts back exactly its key and capabilities, so that a test can observe its rejection and then the recovery of the same credentials. The identity must still have the key of client; one changed outside the module is refused. While it is revoked, the descriptor works only for ConnectionConfig, whose credentials stay valid for after Restore; ClientConfig.WithIdentity, Capabilities, UpdateCaps and ReplaceKey fail, CreateClient refuses its name, and Delete makes the removal permanent.

A fresh connection with the client's key fails at once. A client that authenticated before keeps its sessions until it renews its tickets, within auth_mon_ticket_ttl (AuthConfig.MonTicketTTL), when the MONs reject the renewal; it keeps the MGR and OSD sessions it already holds. Tentacle also drops its MON session, and a librados client then does not authenticate again on its own, while Squid keeps an open MON session answering. A new connection with the same credentials works once Restore returns.

The removal and the restoration are fenced like the import of ReplaceKey. A failed call settles the MONs and reports what they hold. When that stays uncertain, Revoke returns the revocation with the error, and Restore settles first and then restores. The cluster needs no Restore before Terminate.

func (ClientConfig) String

func (client ClientConfig) String() string

String deliberately excludes credentials, including when a descriptor is logged.

func (*ClientConfig) UpdateCaps added in v0.4.0

func (client *ClientConfig) UpdateCaps(ctx context.Context, caps ClientCaps) error

UpdateCaps replaces all capabilities while retaining the identity/key. Omitted services lose their previous capabilities. External auth edits must not race this call. Already issued session tickets may retain previous caps; verify permission changes using a fresh client connection. The change is fenced like the import of ReplaceKey: a failed call reports what the MONs hold once settled, and an uncertain one is resolved by the next operation on the identity.

func (*ClientConfig) User

func (client *ClientConfig) User() string

User returns the ID without the client. prefix, for APIs such as librados rados_create and go-ceph rados.NewConnWithUser.

func (*ClientConfig) WithIdentity added in v0.4.0

WithIdentity attaches a container with only this client's keyring. It rejects a descriptor whose creation is incomplete, that is revoked or that ReplaceKey made stale. Caller owns the attached container and must terminate it before the cluster. The options customize the container's ceph.conf. WithZoneConfig requires Container.WithZone after this option. Omitting it fails enrollment after start with ErrInvalidRequest. Keep and terminate the live container returned alongside the error.

The container gets /etc/ceph/ceph.conf, with the current MON addresses and a [client.<id>] section that points to the keyring, and the keyring at KeyringPath. The client must select its identity, for example rados --id reader, CEPH_ARGS="--id reader", librados rados_create(&cluster, "reader") or go-ceph rados.NewConnWithUser("reader"). It also refuses an identity that Revoke removed until its Restore, or that has an unresolved change.

type ClientContainerOptions added in v0.4.0

type ClientContainerOptions struct {
	// contains filtered or unexported fields
}

ClientContainerOptions is the configuration that WithClient and ClientConfig.WithIdentity write into a client container. It is a plain value with chainable, value-returning methods; the zero value changes nothing. Within one value a later call of the same method replaces the earlier one. A field set by two values is refused with ErrInvalidRequest, even when they agree.

cluster.WithClient(ceph.ClientContainerOptions{}.WithMessengerPolicy(policy))

func (ClientContainerOptions) WithMessengerPolicy added in v0.4.0

WithMessengerPolicy writes the policy into the [client] section of the client container's ceph.conf. It applies only to the Ceph clients in that container, overrides the central configuration and MessengerV2Secure's global modes, and leaves the cluster's own control CLI alone. The value keeps its own copy of the lists; a nil list keeps Ceph's default and a non-nil empty list is refused.

type ClientMessengerPolicy added in v0.2.0

type ClientMessengerPolicy struct {
	// MonitorsAndManagers sets ms_mon_client_mode, which Ceph uses for
	// connections to MONs and MGRs.
	MonitorsAndManagers []MessengerConnectionMode
	// OtherDaemons sets ms_client_mode, which Ceph uses for connections to
	// the other daemons, such as OSDs.
	OtherDaemons []MessengerConnectionMode
}

ClientMessengerPolicy selects the Messenger v2 connection modes that a native client offers, in order of preference. A nil field keeps Ceph's default, which offers both modes.

type ClientOptions added in v0.4.0

type ClientOptions struct {
	// contains filtered or unexported fields
}

ClientOptions is the configuration of the key that CreateClient and ClientConfig.ReplaceKey generate. It is a plain value with chainable, value-returning methods; the zero value changes nothing. Within one value a later call of the same method replaces the earlier one. A field set by two values is refused with ErrInvalidRequest, even when they agree.

client, err := cluster.CreateClient(ctx, "app", caps, ceph.ClientOptions{}.WithKeyType(ceph.CipherAES256K))

func (ClientOptions) WithKeyType added in v0.4.0

func (o ClientOptions) WithKeyType(cipher CephxCipher) ClientOptions

WithKeyType selects the key type of the new key instead of the cluster's AuthConfig.KeyType or the release default for CreateClient, or the type of the current key for ReplaceKey. The MONs must allow the type for the client to authenticate. CipherAES256K needs an image with key type support (see CipherAES256K); without it the operation fails before any change. An unknown cipher, including the empty one, is refused.

type ClientRevocationHandle added in v0.4.0

type ClientRevocationHandle struct {
	// contains filtered or unexported fields
}

ClientRevocationHandle manages removal and restoration of a client identity that this cluster created. It retains ownership and possibly uncertain apply state; a non-nil handle does not guarantee a successful removal. Restore puts back the same key and capabilities. It holds the original credentials privately and never prints them. Copies share its state.

func (ClientRevocationHandle) GoString added in v0.4.0

func (r ClientRevocationHandle) GoString() string

GoString excludes credentials from formatted Go representations as well.

func (*ClientRevocationHandle) Name added in v0.4.0

func (r *ClientRevocationHandle) Name() string

Name returns the revoked entity, for example client.reader.

func (*ClientRevocationHandle) Restore added in v0.4.0

func (r *ClientRevocationHandle) Restore(ctx context.Context) error

Restore puts back the identity's original key and capabilities and makes its descriptor usable again. It succeeds again once restored, as long as the identity still holds them. It refuses an identity that was created again with another key or other capabilities, and one that ClientConfig.Delete removed for good. The restoration is fenced like the removal: when its outcome stays uncertain, the next Restore settles first.

func (ClientRevocationHandle) String added in v0.4.0

func (r ClientRevocationHandle) String() string

String deliberately excludes credentials.

type ClusterConfig added in v0.4.0

type ClusterConfig struct {
	// contains filtered or unexported fields
}

ClusterConfig is the topology and placement group of the Ceph configuration: how many MONs, MGRs and OSDs Run starts, where the initial OSDs sit in the CRUSH map, the default CRUSH root and pool size defaults, and the pools Run creates after the initial OSDs. It is a plain value with chainable, value-returning methods, not a customizer; pass it to Run through its one grouped option function, WithClusterConfig. The zero value changes nothing, and each field records whether it was set, so an explicit setting is different from an unspecified one.

cluster := ceph.ClusterConfig{}.WithMonitorCount(3).WithOSDCount(3).WithPoolDefaults(3, 2)
c, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithClusterConfig(cluster))

The managers (WithManagerCount, WithNoInitialManagers) are one field, and so are the initial OSDs (WithOSDCount, WithInitialOSDs, WithNoInitialOSDs). Within one value a later call of the same field replaces the earlier one. Run refuses a field that two WithClusterConfig options both set, even when they agree, so argument order never decides a conflict. Incompatible combinations (no initial OSDs with initial pools or services, no initial managers with initial CephFS or RGW services, a layout that cannot hold the pools) are refused before any resource is created. The composition rules are in docs/contributing/configuration-composition.md.

func (ClusterConfig) WithDefaultCRUSHRoot added in v0.4.0

func (c ClusterConfig) WithDefaultCRUSHRoot(root string) ClusterConfig

WithDefaultCRUSHRoot selects placement for pools that Ceph creates itself, including .mgr, and for omitted PoolConfig.CRUSHRoot/OSDConfig.Root fields. The initial layout must contain enough OSDs in this root for the default replica count. Without this setting the root is default.

func (ClusterConfig) WithInitialOSDs added in v0.4.0

func (c ClusterConfig) WithInitialOSDs(configs ...OSDConfig) ClusterConfig

WithInitialOSDs selects the count and logical CRUSH locations of the initial OSDs, one OSDConfig per OSD. An empty layout is refused, and so is a host that appears under different racks or roots. The value keeps its own copy of configs. Pools that need N distinct failure domains are validated against the layout before anything is created.

Run registers the initial OSDs one after another in layout order, so IDs, keys and CRUSH host buckets are allocated in that order on every run, and every host bucket exists before any OSD boots. It then starts up to four OSD containers at a time and waits until all are up and in. If an OSD fails to start, no further OSD is started and the error names it; every registered OSD and created container stays owned by the returned cluster. AddOSD adds one OSD at a time.

func (ClusterConfig) WithManagerCount added in v0.4.0

func (c ClusterConfig) WithManagerCount(count int) ClusterConfig

WithManagerCount starts one active manager and the remaining standbys (default: one manager). A count below one is refused; WithNoInitialManagers defers the first manager instead.

func (ClusterConfig) WithMonitorCount added in v0.4.0

func (c ClusterConfig) WithMonitorCount(count int) ClusterConfig

WithMonitorCount selects the initial monitor count (default: 1). Three enables quorum failover tests; even counts are allowed for deliberate quorum-loss scenarios. A count below one is refused.

func (ClusterConfig) WithNoInitialManagers added in v0.4.0

func (c ClusterConfig) WithNoInitialManagers() ClusterConfig

WithNoInitialManagers defers the first MGR until an explicit AddManager. Run waits for MON quorum and requested OSDs up/in, without implying PG clean or MGR module readiness. Initial CephFS/RGW services are refused before allocation; positive-OSD initial pools are allowed. The essential image must still contain the MGR and its required dependencies. WaitForClean and WaitForPGClean need an available manager, so they cannot succeed before one is added, and Ceph may report MGR_DOWN (after mon_mgr_mkfs_grace) or HEALTH_ERR while client data I/O still works. To simulate a manager outage in a configured cluster, stop the existing manager container instead.

func (ClusterConfig) WithNoInitialOSDs added in v0.4.0

func (c ClusterConfig) WithNoInitialOSDs() ClusterConfig

WithNoInitialOSDs starts only the selected MON/MGR topology. It keeps the ordinary prospective pool defaults (size 2, min_size 1) for later AddOSD calls; WithPoolDefaults may override them. Initial pools and CephFS/RGW services are refused before allocation. No storage readiness is implied: native internal pools may have pending PGs.

func (ClusterConfig) WithOSDCount added in v0.4.0

func (c ClusterConfig) WithOSDCount(count int) ClusterConfig

WithOSDCount sets the initial number of OSD containers at default locations (default: 2). A count below one is refused; WithNoInitialOSDs starts none. Each OSD gets a logical host osd-ID under the default root. Without WithPoolDefaults the cluster's default pool size is min(2, count) with min_size 1, so a one-OSD cluster gets replica-1 pools that can become clean.

func (ClusterConfig) WithPoolDefaults added in v0.4.0

func (c ClusterConfig) WithPoolDefaults(replicas, minSize int) ClusterConfig

WithPoolDefaults selects size/min_size for pools created by Ceph or callers. Without this setting the fast fixture uses size=min(2, OSD count), min_size=1 (2/1 with WithNoInitialOSDs). It requires 1 <= minSize <= replicas <= 10 and sets osd_pool_default_size and osd_pool_default_min_size, which WithConfigFile therefore rejects. CreatePool can override these defaults for each explicitly configured pool.

func (ClusterConfig) WithPools added in v0.4.0

func (c ClusterConfig) WithPools(configs ...PoolConfig) ClusterConfig

WithPools creates the selected pools after initial OSD startup. Configs and nested EC settings are copied; cluster defaults are resolved after all Run options have been applied.

type ClusterFacts added in v0.4.0

type ClusterFacts struct {
	AddressFamily             AddressFamily
	Images                    ImageSet
	MessengerMode             MessengerMode
	NetworkName               string
	UsesHostNetwork           bool
	PublicAddress             string
	ClusterNetworkName        string
	HasSeparateClusterNetwork bool
	CephVersion               string
}

ClusterFacts is an independent snapshot of the nine static cluster facts. It holds only values: no Container, Docker handle, context, lock, pointer, func or channel, so nothing in it can control the cluster or reactivate it, and it owns no resource or cleanup lifetime. Later changes to the cluster do not update an obtained snapshot, and changing a snapshot does not change the cluster or another snapshot.

The values describe the recorded configuration and identity of the cluster. They do not show that its resources still exist, are reachable or run, and they are not a native observation. Images and CephVersion describe the cluster's own settings and first MON; a daemon started with DaemonOptions.WithImage is not reflected (see the Image and Version methods of its handle).

func FactsOf added in v0.4.0

func FactsOf(c *Container) (ClusterFacts, error)

FactsOf returns an independent snapshot of the static facts of a non-nil cluster, including a terminated one. A nil cluster returns the zero ClusterFacts and an error that matches ErrInvalidRequest.

The facts are written by Run before it returns and are never cleared by Terminate, so FactsOf works on a terminated cluster or while Terminate runs, without a lock, command or Docker call. A fact that was never recorded stays empty (for example the CephVersion of a cluster that Run returned with an error); an empty value does not distinguish not recorded from recorded as empty, and no value reports whether the resources still exist.

type ConfigEntry

type ConfigEntry struct {
	Section            string `json:"section"`
	Mask               string `json:"mask"`
	Name               string `json:"name"`
	Value              string `json:"value"`
	Level              string `json:"level"`
	CanUpdateAtRuntime bool   `json:"can_update_at_runtime"`
}

ConfigEntry reports an exact stored entry rather than an inherited/default value. Configuration may contain sensitive values; log selected fields only.

type ConfigSetting

type ConfigSetting struct {
	Section, Mask, Name, Value string
}

ConfigSetting addresses one entry in the MON configuration database. Section is global, mon, mgr, osd, mds, client or a specific identity such as osd.0. Mask is optional: class:ssd, host:node-a or host:node-a/class:ssd. These stored settings have lower precedence than local files, argv and runtime overrides.

type ConnectionConfig added in v0.4.0

type ConnectionConfig struct {
	// Conf is the content of ceph.conf.
	Conf []byte
	// Keyring is the content of the keyring file.
	Keyring []byte
}

ConnectionConfig holds the files that an external native client needs to connect: the ceph.conf and the keyring. The fields are named so that the two byte slices cannot be swapped by a positional assignment. Each value returned by a ConnectionConfig method owns independent copies of the bytes. The keyring is a credential: it is not redacted, so do not log the value.

type Container

type Container struct {
	testcontainers.Container
	// contains filtered or unexported fields
}

Container embeds the MON/control container and owns the MGR, OSDs, optional RGW/MDS services and network. Use Terminate to clean up the entire cluster. Generic Run options customize the MON; module options configure the cluster. Use WithHostNetwork to change cluster networking, rather than changing only the MON's network mode.

func Run

Run creates configurable MON, MGR and OSD containers for a disposable cluster. img supplies all roles unless WithImageOptions selects explicit role images with ImageOptions.WithImageSet, in which case img must be empty or the set's essential image. Run recognises the Option values among opts (every Ceph setting, including each grouped configuration value carried by its option function), applies and validates them before it creates anything, and passes all of opts, in argument order, to the first MON's container request after the module's own request options and before the messenger environment, the WithConfigFile file and the ownership label. An Option has no effect on that request, so a setting passed as one is never lost and never reaches a container. Images contain runtime tools; this module supplies cluster setup.

Run resolves and validates the options, including local file reads, before checking ctx and starting the WithStartupTimeout timer. Bootstrap is bounded by both ctx and that timer. Run creates the owned bridge network (two with NetworkConfig.WithSeparateClusterNetwork) or checks the engine in host mode, starts MON a and records its configuration, admin keyring and Ceph version, adds the remaining MONs (and, with more than one, the separate CLI control container), starts the first MGR, registers and starts the OSDs and waits until they are up and in, then starts the remaining MGRs (the first stays active), waits for an available active MGR (or, with ClusterConfig.WithNoInitialManagers, for quorum and the OSDs), and finally creates the ClusterConfig.WithPools pools and the resources of the service options. Run does not wait for clean placement groups; call Observer.WaitForClean before client I/O.

Contradictory options fail before any Docker resource exists, regardless of their order: a host address without WithHostNetwork; host networking with a separate cluster network or with an address family other than AddressIPv4; a Messenger policy with MessengerV2Secure; AddressIPv4MappedIPv6 with a separate cluster network; one field set by two carriers of the same group; no initial OSDs with initial pools or services; no initial managers with initial filesystems or gateways; a default CRUSH root or an initial pool that the initial layout cannot satisfy; and duplicate pool, filesystem or gateway names, including the pools a filesystem creates. In host mode Run fails with ErrHostNetworkUnavailable when the engine does not run Linux containers, the selected address is not assigned on the Docker host, or the test process cannot reach the first MON.

The returned Container embeds the first MON's testcontainers.Container, so generic methods such as Exec, Logs, Host and MappedPort address MON a; Terminate is overridden and removes the whole cluster. A non-nil Container returned with an error owns every resource created so far and must still be terminated; register testcontainers.CleanupContainer before checking err.

Example
package main

import (
	"context"
	"log"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/testcontainers/testcontainers-go"
)

func main() {
	ctx := context.Background()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithClusterConfig(ceph.ClusterConfig{}.WithOSDCount(2)))
	defer func() {
		if err := testcontainers.TerminateContainer(cluster); err != nil {
			log.Printf("failed to terminate container: %s", err)
		}
	}()
	if err != nil {
		panic(err)
	}
	osd, err := cluster.AddOSD(ctx)
	if err != nil {
		panic(err)
	}
	if err := osd.Remove(ctx); err != nil {
		panic(err)
	}
}
Example (Services)

Options from several service packages start one cluster that serves RBD, CephFS and S3 clients together.

package main

import (
	"context"
	"log"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/JSYoo5B/ceph-testcontainers-go/cephfs"
	"github.com/JSYoo5B/ceph-testcontainers-go/rbd"
	"github.com/JSYoo5B/ceph-testcontainers-go/rgw"
	"github.com/testcontainers/testcontainers-go"
)

func main() {
	ctx := context.Background()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage,
		ceph.WithClusterConfig(ceph.ClusterConfig{}.WithOSDCount(3)),
		rbd.WithPools(ceph.PoolConfig{Name: "volumes"}),
		cephfs.WithFilesystems(cephfs.Config{Name: "shared"}),
		rgw.WithGateways(rgw.Config{Name: "s3"}),
	)
	defer func() {
		if err := testcontainers.TerminateContainer(cluster); err != nil {
			log.Printf("failed to terminate container: %s", err)
		}
	}()
	if err != nil {
		panic(err)
	}
	_ = cephfs.Filesystems(cluster)
	_, _ = rgw.Gateways(context.Background(), cluster)
}

func (*Container) AddManager

func (c *Container) AddManager(ctx context.Context, name string, opts ...DaemonOptions) (*ManagerContainer, error)

AddManager creates a named active/standby candidate and waits for its mgrmap registration. Existing daemon identities are rejected; it does not force active ownership away from an existing MGR. With AddressDualStack it then checks the active MGR's advertised addresses and fails with ErrUnsupported when one family has no concrete address. The MGR starts from the cluster's MGR image, ClusterFacts.Images.MGR (ImageSet.MGR, the essential image by default); DaemonOptions.WithImage starts this one MGR from another image instead. A MGR started from a cluster MGR image other than the essential image must report ClusterFacts.CephVersion from ceph-mgr --version in a short-lived probe container of that image; otherwise AddManager fails before any identity, key or MGR container is created and returns no handle. Options are validated before the topology lock is taken or any identity is created.

It never forces the active role away from a running manager, and success does not mean every MGR module is ready; use Observer.WaitMGRModuleReady or Observer.WaitForClean. On an AddressDualStack cluster it then checks the active MGR's advertised addresses.

func (*Container) AddMonitor

func (c *Container) AddMonitor(ctx context.Context, name string, opts ...DaemonOptions) (*MonitorContainer, error)

AddMonitor joins a new named MON from the live quorum's map and shared key. Existing daemon data is retained; a partial new member remains owned on error. DaemonOptions.WithImage starts this one MON from another image; the default is the essential image. Options are validated before the topology lock is taken or anything is created.

Names already in the monmap are refused, and a can never be reused. The first call also starts the independent control container. After joining, it copies the new quorum's addresses into the mon_host of every owned daemon's ceph.conf, including stopped daemons; if only that copy failed, retry RefreshMonitorConfig. Caller-owned client containers are not updated (see RefreshClientMonitorConfig). A partially started member stays owned on error.

func (*Container) AddOSD

func (c *Container) AddOSD(ctx context.Context, configs ...OSDOptions) (*OSDContainer, error)

AddOSD registers, formats and starts a new OSD, then waits for it to be up/in. With no argument the OSD takes the cluster defaults: host osd-ID under the default root and the essential image. OSDOptions values select an explicit logical CRUSH location (WithPlacement) or start this one OSD from another image (WithImage); several values compose as described at OSDOptions. Omitted host names resolve to osd-ID; logical hosts/racks are simulated placement domains on the same Docker engine, not physical failure domains.

The context is checked first, then the assembled configuration is validated once before the topology lock is taken, before any OSD is registered and before any container exists. Topology operations are serialized. An OSD returned with an error is tracked for cleanup and may be inspected; no data is silently purged on failure. The image override is kept in the OSD's recipe for DetachedOSD.Reopen.

func (*Container) Ceph

func (c *Container) Ceph(ctx context.Context, args ...string) ([]byte, error)

Ceph runs the Ceph CLI in the control container with --connect-timeout 5 and returns its standard output. Arguments are passed directly, without shell interpolation. Use --format json for machine-readable responses. A non-zero exit returns the standard output with a *NativeFailure (exit code, the errno Ceph reported and its reason without log lines, keys redacted); errors.Is(err, fs.ErrNotExist) holds for ENOENT. When no MON answers within 5 seconds the CLI exits before it sends the command, and Ceph connects again until a MON answers, ctx ends or a minute has passed, so a stalled or electing MON delays the command instead of failing it, and the command runs once.

func (*Container) ClientContainer added in v0.4.0

func (c *Container) ClientContainer(ctx context.Context) (testcontainers.Container, error)

ClientContainer returns the owned userspace toolkit executor. It starts lazily for a separate client image; aggregate images reuse the control-plane executor.

Run rados, rbd and the Python client bindings there. The first attachment is serialized with MON topology changes and honours cancellation, and the cluster owns the container's partial or successful teardown. RBD and CephFS operations, ObjectWatchers and PGInconsistencies use it, while daemon-local sockets and process observations stay on their daemon.

func (*Container) ConnectionConfig

func (c *Container) ConnectionConfig(ctx context.Context) (ConnectionConfig, error)

ConnectionConfig returns the independent bootstrap ceph.conf and admin keyring copies, waiting for topology and the configuration cache only while ctx remains active. On error it returns the zero ConnectionConfig.

It is for a native client the caller starts, such as a test process using go-ceph in host mode. The configuration contains the addresses the cluster advertises: container addresses in bridge mode, host addresses with the final MON ports in host mode. ctx bounds the wait for a concurrent topology change or configuration update. It fails after termination or before bootstrap completed and then returns the zero ConnectionConfig. For a restricted identity use ClientConfig.ConnectionConfig.

func (*Container) ControlContainer

func (c *Container) ControlContainer(ctx context.Context) (testcontainers.Container, error)

ControlContainer returns the stable CLI handle while bounding a concurrent control-plane creation or termination publication wait by ctx. The cluster owns the handle; this snapshot does not promise it remains live. In a single-MON cluster it is MON a; with more than one monitor it is an independent CLI container created by the first AddMonitor (also during Run), so commands keep working while MON a is stopped or removed. It runs the essential image, so it has the ceph CLI and bootstrap tools such as ceph-authtool. With split role images it has neither rados and rbd (use ClientContainer) nor radosgw-admin (use rgw.Gateway.Admin); an aggregate image may have them, but do not rely on that. Do not pause or stop it; fault.Pause refuses it.

func (*Container) CreateClient

func (c *Container) CreateClient(ctx context.Context, name string, caps ClientCaps, opts ...ClientOptions) (*ClientConfig, error)

CreateClient creates a fresh client with exactly the supplied capabilities. Existing identities are rejected before mutation; their keys and caps are never updated. A non-nil result with an error identifies an attempted or partial creation; inspect auth state or terminate the cluster rather than retrying with broader caps. ClientConfig.Delete can revoke a confirmed successful creation. The key has the cluster's AuthConfig.KeyType, the release default without one, or the type that ClientOptions.WithKeyType selects.

name is an ID or client.ID of letters, digits, dots, underscores or hyphens. The identity is created whether or not the MONs allow its key type; whether it can authenticate depends on the MonMap's auth_allowed_ciphers. CipherAES256K needs an essential image whose ceph-authtool has --key-type; without it the call fails before any change.

func (*Container) CreatePool

func (c *Container) CreatePool(ctx context.Context, config PoolConfig) (*Pool, error)

CreatePool creates a new pool with explicit placement and replication settings. It validates the entire config and requires enough eligible owned domains before sending any Ceph command. Existing pools, profiles and rules are rejected rather than changed. A non-nil Pool returned with an error identifies resources whose creation was attempted; inspect them or terminate the disposable cluster. That Pool may not know the native ID of its pool yet: it then binds to the pool of its name at its first successful operation, so use it only to clean up. A Pool returned without an error is always bound to its native pool. It never deletes data after a partial command failure. Use WaitForClean before application I/O. For RBD, create images in a replicated metadata pool and pass the EC pool as data_pool in the native client (or --data-pool to the RBD CLI).

The pool uses a dedicated CRUSH rule tc-<name>-replicated (or tc-<name>-ec with an EC profile of the same name) and is created with the autoscaler off. The returned Pool embeds the resolved PoolConfig and adds CRUSHRule and ErasureCodeProfile; it describes the requested settings at creation, while Pool.Status reports the current state. The handle identifies its native pool by name and native ID, and copies share that identity: after Pool.Remove, or when the name belongs to another pool, every method returns an error instead of acting on the pool that now has the name.

func (*Container) DeepScrubPG

func (c *Container) DeepScrubPG(ctx context.Context, pgid PGID) error

DeepScrubPG requests a native deep scrub of one PG and waits until the PG reports a deep-scrub stamp newer than before the request. Inconsistencies are reported, not repaired. The noscrub and nodeep-scrub flags or an unavailable primary can keep the wait from finishing before its deadline.

fault.InjectObjectDataError creates a damaged replica to observe with DeepScrubPG, Observer.PGInconsistencies and RepairPG; ceph osd map <pool> <object> --format json reports an object's PG and acting set.

func (*Container) Managers

func (c *Container) Managers(ctx context.Context) ([]*ManagerContainer, error)

Managers returns sorted owned candidate descriptors. The slice is independent; descriptors and their containers retain fixture ownership.

The returned slice is independent; descriptors and containers stay owned by the cluster. Stop the active manager's container (ManagerStatus names it) to make a standby take over, or the only manager to test behaviour without one.

func (*Container) Monitors

func (c *Container) Monitors(ctx context.Context) ([]*MonitorContainer, error)

Monitors returns the owned MON membership sorted by daemon name, waiting for the topology only while ctx remains active.

The returned slice is independent; descriptors and containers stay owned by the cluster. Stop and Start on a handle inject failures and keep the monitor's store; MonitorContainer.Remove changes membership. When a majority is stopped, quorum is lost and new clients cannot authenticate; after the monitors start again, existing and new sessions work. On a bridge network a MON returns at the address the monmap records for it whatever the order of Start calls, because the fixture requests its original address from Docker again; a MON whose endpoint fault.InterruptNetwork interrupted does not start until it is restored. MON a waits for a quorum when it starts, so after stopping all MONs start the others first.

func (*Container) OSDs

func (c *Container) OSDs(ctx context.Context) ([]*OSDContainer, error)

OSDs returns the owned OSD descriptors sorted by ID, waiting for the topology only while ctx remains active.

The returned slice is independent; descriptors and containers stay owned by the cluster. Stop and Start on a handle inject daemon failures and keep the OSD's identity and data, also with in-memory storage; Ceph marks a stopped OSD down after the heartbeat grace and out after mon_osd_down_out_interval unless noout is set. OSD bootstrap initializes only an empty directory: it refuses a non-empty partial store without a ready marker, and a ready store must match the OSD's ID, UUID and cluster FSID; failed initialization is not replayed (see docs/internals/osd-storage.md).

func (*Container) Pool added in v0.4.0

func (c *Container) Pool(ctx context.Context, name string) (*Pool, error)

Pool returns a handle for the existing pool with the exact native name. Use it for a pool that CreatePool did not create, such as an initial pool from WithPools or a default pool of a service. It fails when the pool does not exist. The handle identifies the pool by its native ID, so it does not act on a later pool that reuses the name.

The returned Pool describes what the cluster reports: Name, PGNum, Replicas (of a replicated pool), MinSize and ErasureCodeProfile. The other fields, CRUSHRule included, are zero; PoolState has the full native state.

func (*Container) RefreshClientMonitorConfig

func (c *Container) RefreshClientMonitorConfig(ctx context.Context, client testcontainers.Container) error

RefreshClientMonitorConfig updates only the explicit client's global mon_host entry in /etc/ceph/ceph.conf. The client's global FSID must match this fixture and its live quorum. Docker archive copies also work with stopped containers; this method never starts, stops, terminates or adopts the client's lifetime. Other settings and keyring files are retained. It does not update mirror peer attributes, network attachments, or private-section mon_host overrides.

Successful writes remain on partial failure and may be retried with a fresh context. External configuration writers must not race this operation: Docker has no file compare-and-swap. Callers own clients and their cleanup.

func (*Container) RefreshMonitorConfig

func (c *Container) RefreshMonitorConfig(ctx context.Context) error

RefreshMonitorConfig copies the live quorum's bootstrap addresses into the global mon_host entry of every owned daemon's ceph.conf, including stopped daemons, without starting or restarting them. Other entries and comments are retained. Caller-owned clients and multicluster daemon configs are not copied.

AddMonitor and MonitorContainer.Remove already refresh these files. Retry this method after an AddMonitor that joined native membership but could not copy every config; retry MonitorContainer.Remove itself after a partial removal. Successful copies and the future-client template are retained on partial failure. Repeating a refresh reads each current file and skips copies that are already up to date. External config writers must not race this operation: Docker has no file CAS.

func (*Container) RepairPG

func (c *Container) RepairPG(ctx context.Context, pgid PGID) error

RepairPG requests a native repair of one PG and waits until the PG reports a newer deep-scrub stamp and has finished repairing; it returns an error if the PG is still inconsistent then. Repair rewrites bad replicas from an authoritative copy; it cannot recover an object without one, and the wait then ends with the PG still inconsistent.

func (*Container) RunNativeCommand added in v0.2.0

func (c *Container) RunNativeCommand(ctx context.Context, target testcontainers.Container, command NativeCommand) (NativeCommandResult, error)

RunNativeCommand runs one native command in target under the supervisor that the module's own native queries use. target is the control container, a daemon or service container of this cluster, or a container of the caller attached to the cluster's network (the host network for a host-network cluster); the call never adopts, stops or removes the caller's container. target must be running, not paused, and have python3, which runs the supervisor. A malformed command or an ended ctx fails before anything runs.

The supervisor starts the command in its own process group, bounds each output stream inside the container and kills the group at the deadline, Timeout shortened to ctx's deadline. A command that times out or exceeds an output limit returns its result, marked by TimedOut or the Truncated flags, with an error that wraps ErrNativeCommandIncomplete. A descendant that starts its own session leaves the group and is not stopped.

When ctx ends first, the call stops the command inside the container before it returns, also when Docker had not started it yet, within its own NativeCommandStopTimeout. When that cannot be confirmed, for example because the container is paused or a signal killed the supervisor, the error is a *NativeCommandUnstoppedError, whose Stop retries. Terminate cancels running commands and stops those not confirmed. Stopping a command does not undo what it already did, such as a MON command it had sent, and nothing is retried. Errors name the command by Label; a failure of Docker or of target's implementation appears only by its kind, and errors.Unwrap returns the original error, whose text may quote anything. diagnostics.Redact redacts output that a test logs.

With Input, the supervisor runs in an exec of the Docker engine with its standard input attached, and passes what Input writes to the command; see NativeInput. The call stops taking writes when the command ends, and returns only after its input and output work ended. Its deadline and cancellation stop the command as without Input, and a write in progress then fails. A malformed limit or an Input that served another command fails before anything runs or any input is taken.

Failures before the command runs are classified: a malformed command, or a target that the cluster neither owns nor has attached, or that is stopped or paused, is ErrInvalidRequest; a target without python3, or a program that does not exist or is not executable, is ErrImageCapability; a program that could not start for want of a process or memory, or that a security policy refused, and other Docker failures are ErrEnvironmentUnavailable. A container that disappears or stops after the command was sent matches no category, since the command may have run, and neither do context errors, ErrNativeCommandIncomplete, ErrNativeInputClosed or *NativeCommandUnstoppedError. The essential and client roles and the RGW setup runtime provide python3; the cephfs role does not promise it.

func (*Container) Terminate

func (c *Container) Terminate(ctx context.Context, opts ...testcontainers.TerminateOption) error

Terminate removes all owned daemons before removing the isolated network. It also works on a partially initialized cluster returned by Run. The whole cluster is discarded, so each container gets 2 seconds instead of Docker's 10 seconds to stop before it is killed; a StopTimeout option passed by the caller still takes precedence. From its start, also when it then fails, Terminate refuses new native commands and cancels the running ones, including the command of an rgw.Gateway.Admin call.

It first restores network delays and interruptions and resumes paused containers, then releases host port reservations, stops services (RGW, MDS) before OSDs, removes every owned daemon and helper container, the OSD storage volumes and the shared admin-socket run volume, and finally the owned networks. Every container the cluster creates carries the org.testcontainers.ceph.cluster label with the cluster's ID, and Terminate also removes a labelled container it has no handle for, as Docker can create one when the reply to a create request was lost. A failed cleanup can be retried; an endpoint that cannot be restored keeps its network for the retry. Containers the caller attached with WithClient or ClientConfig.WithIdentity are the caller's: terminate them first, because Docker cannot remove a network that still has attached containers (with testcontainers.CleanupContainer, register the cluster first).

func (*Container) WithClient

WithClient connects another container to Ceph and copies its admin credentials. These privileged, ephemeral credentials are for trusted test containers only. Caller owns and terminates that container before terminating the cluster. The options customize the container's ceph.conf. WithZoneConfig requires Container.WithZone after this option. Omitting it fails enrollment after start with ErrInvalidRequest. Keep and terminate the live container returned alongside the error.

It copies the bootstrap /etc/ceph/ceph.conf and /etc/ceph/ceph.client.admin.keyring into the container and joins it to the cluster's public network (the host network in host mode). The container must already contain the Ceph clients it uses; WithClient installs nothing, and ceph.DefaultImage or ClusterFacts.Images.Client work for CLI and Python clients. Combine it with WithIdleEntrypoint for a container driven with Exec. Ceph clients connect directly to the OSD addresses the MONs advertise, which in bridge mode are container addresses, so a process on the Docker host cannot reach RADOS, RBD or CephFS through the MON's mapped port alone; use a WithClient container or WithHostNetwork on Linux.

func (*Container) WithZone added in v0.5.0

WithZone places a container in zone: Container.WithClient and ClientConfig.WithIdentity containers, whose post-start enrollment requires a zone, and service daemons such as an rgw.Start gateway, which otherwise join the quorum zone. Pass it after the attachment option. On every start the container is enrolled as a member: a route agent with NET_ADMIN joins its network namespace and the boundary installs its routes and its peers' routes to it; when it stops they are removed. A start fails when its enrollment does, also when a member already has a /32 route of its own to an address the boundary routes, which it does not replace. Such a container stays in ZoneLayout.Unzoned until a retried Start of it completes the enrollment. A cluster without zones refuses it, and so does an undeclared zone.

func (*Container) Zones added in v0.5.0

func (c *Container) Zones(ctx context.Context) (ZoneLayout, error)

Zones returns the zone topology of a cluster from WithZoneConfig. It reads Ceph's down-reporter settings with the control CLI and inspects the cluster's bridges for containers without a zone. Routes come from the boundary's cleanup records. To verify forwarding, inspect the member's kernel route or send traffic through it. A cluster without zones is refused.

type DaemonAddress added in v0.2.0

type DaemonAddress struct {
	Name  string
	Addrs []EntityAddress
}

DaemonAddress is the name of one MON or MGR and the public address vector that its cluster map advertises.

type DaemonAddresses added in v0.2.0

type DaemonAddresses struct {
	MonMapEpoch int
	// Monitors are the monmap members in rank order.
	Monitors    []DaemonAddress
	MgrMapEpoch int
	// ActiveManager is nil when no MGR is active. A MGR that has just become
	// active advertises its addresses only when it is available; until then
	// ManagerAvailable is false and its address vector can be empty. Standby
	// MGRs advertise no address in the MGRMap, so StandbyManagers lists only
	// their names.
	ActiveManager    *DaemonAddress
	ManagerAvailable bool
	StandbyManagers  []string
	OSDMapEpoch      int
	// OSDs are the OSDMap entries in ID order.
	OSDs []OSDAddress
}

DaemonAddresses lists the daemons of a cluster and the addresses that the MON, MGR and OSD maps advertise, with the epoch of each map. The three maps are read one after another, not as one snapshot.

type DaemonOptions added in v0.4.0

type DaemonOptions struct {
	// contains filtered or unexported fields
}

DaemonOptions is the configuration of one daemon that AddMonitor or AddManager creates, or that MonitorContainer.Replace starts again (AddOSD takes OSDOptions). It is a plain value with chainable, value-returning methods. It applies to that daemon only and never changes the cluster's settings or any other daemon. The zero value changes nothing.

mon, err := cluster.AddMonitor(ctx, "b", ceph.DaemonOptions{}.WithImage(image))

Within one value a later call of the same method replaces the earlier one. A field set by two values is refused with ErrInvalidRequest, even when they agree.

func (DaemonOptions) WithImage added in v0.4.0

func (o DaemonOptions) WithImage(image string) DaemonOptions

WithImage starts the one new daemon from image instead of the cluster's image for that daemon: the essential image for a MON, the MGR image of the cluster's ImageSet (Essential unless the set's MGR selects another) for a MGR. It takes precedence over that selection for this daemon only. The image must contain the daemon's binary and the tools the fixture's startup scripts use, as the essential image does; nothing here checks its release or compatibility with the running cluster.

The override belongs to that daemon: Stop and Start keep the container, and the daemon's handle reports it with Image. It does not change ClusterFacts, which describes the cluster's own settings and its first MON, and a CLI, client or helper container still uses the cluster's images.

The image reference must be non-empty and free of whitespace and control characters. A refusal matches ErrInvalidRequest and happens before the topology lock and before any daemon identity, key or container is created; nothing checks that the image exists. MGR containers are never recreated, so their image does not change for their lifetime; a MON keeps its image unless MonitorContainer.Replace starts it from another one. ClusterFacts.Images and ClusterFacts.CephVersion, and the images recorded in a finalization manifest or verification report, keep describing the cluster and its first MON; the diagnostics archive records each container's image from Docker. A client container, and a MGR started from the cluster's MGR image, are still refused when their release differs from ClusterFacts.CephVersion. This is a building block: no mixed-version or upgrade combination is claimed to work.

type DetachedOSD added in v0.4.0

type DetachedOSD struct {
	// contains filtered or unexported fields
}

DetachedOSD is the retained store and MON registration of an OSD whose container OSDContainer.Detach removed. DetachedOSD.Reopen of the same cluster starts a new container on it once. The cluster owns the store: Terminate discards it with the owned volume, and the MONs keep the registration down meanwhile.

func (*DetachedOSD) ClusterFSID added in v0.4.0

func (d *DetachedOSD) ClusterFSID() string

ClusterFSID returns the cluster FSID that the retained store holds.

func (*DetachedOSD) ID added in v0.4.0

func (d *DetachedOSD) ID() int

ID returns the retained numeric OSD ID.

func (*DetachedOSD) Image added in v0.4.0

func (d *DetachedOSD) Image() string

Image returns the image the detached OSD ran from, which DetachedOSD.Reopen uses again.

func (*DetachedOSD) Placement added in v0.4.0

func (d *DetachedOSD) Placement() OSDConfig

Placement returns the CRUSH location that DetachedOSD.Reopen starts the OSD in.

func (*DetachedOSD) Reopen added in v0.4.0

func (d *DetachedOSD) Reopen(ctx context.Context, configs ...OSDOptions) (*OSDContainer, error)

Reopen starts a new container on the store and registration that Detach retained, and waits for the OSD to be up/in. It returns a new *OSDContainer with the same ID, UUID and placement; the previous handle of the OSD keeps naming the removed container and is stale.

Before any write it requires the store's identity files, keyring and block inode and size to be unchanged since the detach, the MON registration to hold the same UUID and be down, and the MONs' key for the OSD to equal the stored one. A refusal there, a failed cleanup of the stale admin socket, or a canceled context before the container is requested leaves the OSD detached for another attempt; none of them dispatches a creation request. The cleanup runs under the supervisor. If its end stays unconfirmed, the detached OSD records it and every later Reopen first has to stop or confirm it, so that a late cleanup cannot remove the socket of the replacement; until then Reopen refuses without creating a container. Once the new container is requested the detached OSD is consumed: an error returns the new handle, which the cluster owns. If that handle has a container, detach it again, remove it or terminate the cluster. If the creation request returned no container, its reply may have been lost after the runtime accepted it, so it is never replayed and the handle has no container to detach: Detach and Reopen refuse it, Remove refuses the last OSD, and only Terminate recovers it. Reopen does not reconcile a container that such a lost reply may have created. The store is started through the ordinary bootstrap, which validates the stored identity again and never formats a store that has its ready marker.

With no option the new container runs the image the detached OSD ran. OSDOptions.WithImage starts it from another image instead, for example a newer release that opens the retained store; the store is not checked against that image's release, and nothing makes the change reversible. WithPlacement is refused: the registration keeps its placement.

func (*DetachedOSD) UUID added in v0.4.0

func (d *DetachedOSD) UUID() string

UUID returns the retained OSD UUID that the store and its MON registration hold.

type EntityAddress added in v0.2.0

type EntityAddress struct {
	// Type is the protocol: "v2" for msgr2, "v1" for the legacy protocol, or
	// "any".
	Type string
	// Addr is the IP address and port as Ceph prints it, such as
	// "172.20.0.3:6800" or "[fd00::3]:6800".
	Addr string
	// Nonce tells apart daemon instances that reuse an address. MONs use 0.
	Nonce uint32
}

EntityAddress is one entry of a Ceph address vector, as a cluster map advertises it.

type ErasureCodeConfig

type ErasureCodeConfig struct {
	K               int
	M               int
	AllowOverwrites bool
}

ErasureCodeConfig selects data (K) and parity (M) shards on separate failure domains using PoolConfig's placement fields. The fixture uses the portable Jerasure reed_sol_van plugin. Replicas must be zero; zero MinSize selects K+1. An explicit MinSize of K permits I/O without any spare shard during a failure. AllowOverwrites enables partial object writes and is required for RBD and CephFS data pools. Their metadata must remain in a replicated pool. K is at least 2, M at least 1 and K+M at most 128.

type FullRatioSnapshot

type FullRatioSnapshot struct {
	FSID   string
	Epoch  uint32
	Ratios FullRatios
}

FullRatioSnapshot is one native OSDMap observation. Epoch is not a generation for these thresholds: unrelated OSDMap changes can advance it too. The values describe map policy, not whether any OSD has reported the corresponding state.

type FullRatios

type FullRatios struct {
	NearFull, BackfillFull, Full float64
}

FullRatios are global OSDMap utilization thresholds, expressed as fractions of capacity. These are separate from central configuration and pool quotas. Native OSDMap storage uses float32; observations retain its normalized value.

type HealthCheck

type HealthCheck struct {
	Severity, Summary string
	Count             int64
	Details           []string
	Muted             bool
}

HealthCheck reports the native summary count and detail messages. Count is signed and need not equal the number of details, which Ceph may truncate.

type HealthMute

type HealthMute struct {
	Code, Summary, ExpiresAt string
	Count                    int64
	Sticky                   bool
}

HealthMute is a native stored mute, which can outlive its current check. ExpiresAt is empty for no TTL, otherwise the original Ceph timestamp: YYYY-MM-DDTHH:MM:SS.ffffff+HHMM (or -HHMM), or seconds.ffffff for an early epoch below 315360000 seconds. Expiry is not compared with the caller's clock; an expired or cleared nonsticky mute can remain visible before the MON tick.

type HealthSnapshot

type HealthSnapshot struct {
	FSID, Status string
	Checks       map[string]HealthCheck
	Mutes        []HealthMute
}

HealthSnapshot is one MON health observation. Status reflects the worst unmuted check; HEALTH_OK can therefore include muted warnings or errors. Checks retain native, case-sensitive codes, including unknown future codes.

type HealthWatch added in v0.5.0

type HealthWatch struct {
	// contains filtered or unexported fields
}

HealthWatch is a running watch of the cluster's health checks. One goroutine reads them and sends events; Events closes after that goroutine has stopped using the cluster. Stop ends the watch and waits for it.

func (*HealthWatch) Events added in v0.5.0

func (w *HealthWatch) Events() <-chan HealthWatchEvent

Events returns the receive-only event channel. The baseline is already in it when WatchHealth returns. The channel closes when the watch ends; Stop returns why.

func (*HealthWatch) Stop added in v0.5.0

func (w *HealthWatch) Stop() error

Stop cancels the watch if it is still running, waits until its goroutine has returned and Events is closed, and returns the cause that ended the watch: nil when Stop ended it, the context's error when the WatchHealth context ended, an ErrWatchOverflow error, or the read error that ended it (a terminated cluster or a foreign native FSID). Buffered events stay readable after Stop. Stop is safe to call more than once and from any goroutine; every call returns the same cause. Call it before the fixture's cleanup: cancelling the context alone does not wait.

type HealthWatchEvent added in v0.5.0

type HealthWatchEvent struct {
	Sequence          uint64
	Kind              WatchEventKind
	Started, Finished time.Time
	Health            HealthSnapshot
	Err               error
}

HealthWatchEvent is one event of a HealthWatch.

Sequence starts at 1 with the baseline and increases by one for each event, and no event is dropped, so consecutive numbers mean the consumer saw every event the watch produced. Started and Finished are the local clock around the read, not the time Ceph raised or cleared a check.

Health is the read as HealthDetails returns it, with Mutes sorted by code. Its FSID is the original cluster's, the same in every event. The watch compares Status, the set of check codes and, for each check, Severity, Count and Muted, and for each stored mute its Code, Count, Sticky and ExpiresAt. Summary and detail messages are not compared: a read that differs only in their text or order, such as a countdown or a reordered detail list, produces no event, so they are as fresh as the last event, not the last read. Each event has its own snapshot, which the consumer may keep or change. Messages can contain pool names, paths or other application data; do not print or publish events unfiltered. Two snapshots separated by a gap, or simply by the polling interval, do not establish the transitions in between: a check can appear and clear unseen. On a WatchGap, Health is zero and Err is the failure.

type ImageOptions added in v0.4.0

type ImageOptions struct {
	// contains filtered or unexported fields
}

ImageOptions is the image group of the Ceph configuration: a plain value with chainable, value-returning methods. It is not a customizer. Pass it to Run through its one grouped option function, WithImageOptions, which Run applies and validates like any Option before it creates a resource. The zero value changes nothing.

cluster, err := ceph.Run(ctx, "", ceph.WithImageOptions(ceph.ImageOptions{}.WithImageSet(images)))

Each method sets one field. Within one value a later call of the same method replaces the earlier one. Run refuses a field that two WithImageOptions options both set, even when they agree, so argument order never decides a conflict. The pattern for every configuration group, and the composition rules, are in docs/contributing/configuration-composition.md.

func (ImageOptions) WithCephFSImage added in v0.4.0

func (o ImageOptions) WithCephFSImage(image string) ImageOptions

WithCephFSImage selects the image of the CephFS role: every metadata server of the cluster, those started by Run with cephfs.WithFilesystems as well as those started later. Without it MDS daemons use the CephFS role of the cluster's images, which is the Run image unless WithImageSet selects another. The CephFS role in ClusterFacts.Images is the usual image argument of cephfs/mirror.Run. With WithImageSet it replaces the set's CephFS role. Use matching Ceph versions. A blank image is refused.

func (ImageOptions) WithImageSet added in v0.4.0

func (o ImageOptions) WithImageSet(images ImageSet) ImageOptions

WithImageSet selects explicit role images. The set is the one authoritative image configuration: Essential starts the MON and OSD daemons, and the MGR daemons unless the set's MGR selects another image, so the image argument of Run must be empty or equal to Essential and is not a second input. An All set supplies every role. A role that a requested feature needs and the set leaves empty is refused; there is no fallback to Essential, except the documented MGR default of ImageSet. An explicit empty set is refused, not read as unspecified. WithCephFSImage and WithRGWImage replace the matching role of the set. The ImageSet is copied.

func (ImageOptions) WithRGWImage added in v0.4.0

func (o ImageOptions) WithRGWImage(image string) ImageOptions

WithRGWImage selects the image of the RGW role: every gateway of the cluster, those started by Run with rgw.WithGateways as well as those started later with rgw.Start. Without it gateways use the RGW role of the cluster's images, which is the Run image unless WithImageSet selects another. Multisite setup clients use the RGW role recorded in ClusterFacts.Images. Multisite gateways use the image passed to multisite.Run. With WithImageSet it replaces the set's RGW role. Use matching Ceph versions. A blank image is refused.

type ImageSet added in v0.4.0

type ImageSet struct {
	Essential string
	Client    string
	CephFS    string
	RBDAux    string
	RGW       string
	MGR       string
	All       string
}

ImageSet selects the roles runtime contract. All explicitly supplies every role from one image. Otherwise Essential is required and optional roles are required when their feature is used; there is no fallback to Essential.

MGR is the one role with a default: the image of every manager the cluster starts, the initial active and standby candidates of Run as well as those that AddManager adds later. Left empty it is Essential (with All, the All image), so a set without MGR runs managers as before. All may be combined with MGR, which then replaces only the MGR role. The image is a whole MGR container image, not a sidecar, and must ship the ceph-mgr daemon and its modules of the cluster's Ceph release: when MGR is not Essential, each manager start first runs ceph-mgr --version in a short-lived probe container of the image and fails, before the manager's identity, key or container exists, when it reports another release than ClusterFacts.CephVersion. DaemonOptions.WithImage of one AddManager call takes precedence over MGR. After successful startup, FactsOf returns the resolved MGR image in the snapshot’s Images.MGR field.

type InconsistentObject

type InconsistentObject struct {
	Name, Namespace  string
	Errors           []string
	UnionShardErrors []string
	Shards           []InconsistentShard
}

InconsistentObject is one object reported by the last deep scrub of a PG. Errors are object-level native error names and UnionShardErrors combine the per-shard names, such as read_error or data_digest_mismatch.

type InconsistentShard

type InconsistentShard struct {
	OSD, Shard int
	Primary    bool
	Errors     []string
}

InconsistentShard retains one native shard report. Shard is -1 for replicated pools.

type MGRModuleState

type MGRModuleState struct {
	Name                             string
	Enabled, AlwaysOn, ForceDisabled bool
	Available, CanRun                bool
	ErrorString                      string
}

MGRModuleState combines configured module membership with the active MGR's advertised module metadata. Available means metadata is present, not that commands are ready. CanRun is meaningful only when Available is true. AlwaysOn modules can be force-disabled outside this fixture; Enabled reflects ForceDisabled too. ErrorString is the native dependency/load diagnostic.

type ManagerContainer

type ManagerContainer struct {
	testcontainers.Container
	DaemonName string
	// contains filtered or unexported fields
}

ManagerContainer is an owned active or standby manager.

func (*ManagerContainer) AdminSocket added in v0.4.0

func (m *ManagerContainer) AdminSocket(ctx context.Context, args ...string) ([]byte, error)

AdminSocket runs a command on the admin socket of this MGR, which its cluster must still own, and returns the command's output. args are the command words, such as "status", "version" or "config", "get", "osd_heartbeat_grace"; "help" lists the daemon's commands.

The command does not use the Messenger or Cephx: it reaches the daemon through its local socket, so it works while the daemon cannot reach a monitor and reports what the daemon itself has applied, for example the newest OSDMap epoch in an OSD's "status". It runs in the control-plane container, which shares the daemons' run directory, so it works whatever the daemon's own image contains. The daemon must be running; a stopped or removed daemon returns an error. A paused daemon accepts the connection but does not answer, so the call waits for ctx or the cluster's startup timeout, which also bounds every call. When the call ends early, AdminSocket stops its CLI in the control container before it returns, including one that Docker has not started yet. A nonzero exit returns the output with an error.

func (*ManagerContainer) Image added in v0.4.0

func (m *ManagerContainer) Image() string

Image returns the image this MGR container was started from: the DaemonOptions.WithImage override, or the cluster's MGR image, ClusterFacts.Images.MGR, as the reference it was requested by. A MGR of a cluster MGR image other than the essential image runs the image ID that its release check accepted, which the reference may no longer name.

func (*ManagerContainer) Remove added in v0.4.0

func (m *ManagerContainer) Remove(ctx context.Context) error

Remove deletes this owned manager after another running, registered candidate is available. Add a replacement before removing the last candidate. On failure the handle remains owned for retry or cluster cleanup. Do not edit these manager identities externally while a topology operation is in progress. A handle of a removed manager returns an error.

func (*ManagerContainer) RotateKey added in v0.4.0

func (m *ManagerContainer) RotateKey(ctx context.Context) error

RotateKey replaces the Cephx key of this running MGR without a restart, as cephadm's daemon key rotation does:

  1. auth get-or-create-pending creates the new key on the MONs, which then accept both keys.
  2. The daemon's keyring files in its container get the new key, so that a later restart authenticates with it.
  3. ceph tell <daemon> rotate-key hands the key to the running daemon.
  4. auth commit-pending makes it the entity's only key.

MONs share the mon. key, so *MonitorContainer has no RotateKey.

The MONs accept both keys until step 4, and they also commit the pending key on their own when the daemon authenticates with it, after a restart or a handoff. A failure before step 4 therefore keeps both keys valid, and the next call finishes the rotation with the same key; a failed step 4 is reported with the outcome that the MONs show once settled. A call returns only once none of its commands can still change the keys or the keyring files, or after recording such commands for the daemon: the next call stops them, waits for them and settles the MONs for them before it changes anything, and fails while it cannot. After a handoff that may still reach the daemon, a later call finishes the rotation with the same key and generates another key only once the daemon's container has restarted. A handle of a removed MGR returns an error.

func (*ManagerContainer) Version added in v0.4.0

func (m *ManagerContainer) Version(ctx context.Context) (string, error)

Version returns the release that this running MGR reports itself, such as "20.2.4". See OSDContainer.Version.

type ManagerMetrics added in v0.4.0

type ManagerMetrics struct {
	// URL is the scraped endpoint.
	URL string
	// Types maps a metric family to the type in its # TYPE line.
	Types map[string]string
	// Help maps a metric family to the text of its # HELP line.
	Help map[string]string
	// Samples are the samples in the order the module printed them.
	Samples []MetricSample
}

ManagerMetrics is one scrape of the active manager's prometheus module.

func (ManagerMetrics) Select added in v0.4.0

func (m ManagerMetrics) Select(name string, labels map[string]string) []MetricSample

Select returns the samples named name whose labels include every label in labels; nil labels select every sample of that name.

func (ManagerMetrics) Value added in v0.4.0

func (m ManagerMetrics) Value(name string, labels map[string]string) (float64, error)

Value returns the value of the one sample that Select finds; none, or more than one, is an error.

type ManagerStatus

type ManagerStatus struct {
	Available  bool   `json:"available"`
	ActiveName string `json:"active_name"`
	ActiveGID  uint64 `json:"active_gid"`
	Standbys   []struct {
		Name string `json:"name"`
		GID  uint64 `json:"gid"`
	} `json:"standbys"`
}

ManagerStatus reports active ownership and registered standbys.

type ManagerWatch added in v0.5.0

type ManagerWatch struct {
	// contains filtered or unexported fields
}

ManagerWatch is a running watch of the active MGR and its standbys. One goroutine reads the MGRMap and sends events; Events closes after that goroutine has stopped using the cluster. Stop ends the watch and waits for it.

func (*ManagerWatch) Events added in v0.5.0

func (w *ManagerWatch) Events() <-chan ManagerWatchEvent

Events returns the receive-only event channel. The baseline is already in it when WatchManagers returns. The channel closes when the watch ends; Stop returns why.

func (*ManagerWatch) Stop added in v0.5.0

func (w *ManagerWatch) Stop() error

Stop cancels the watch if it is still running, waits until its goroutine has returned and Events is closed, and returns the cause that ended the watch: nil when Stop ended it, the context's error when the WatchManagers context ended, an ErrWatchOverflow error, or the read error that ended it (a terminated cluster, or another cluster behind the control container). Buffered events stay readable after Stop. Stop is safe to call more than once and from any goroutine; every call returns the same cause. Call it before the fixture's cleanup: cancelling the context alone does not wait.

type ManagerWatchEvent added in v0.5.0

type ManagerWatchEvent struct {
	Sequence          uint64
	Kind              WatchEventKind
	Started, Finished time.Time
	Epoch             uint64
	Status            ManagerStatus
	Services          map[string]string
	Err               error
}

ManagerWatchEvent is one event of a ManagerWatch.

Sequence starts at 1 with the baseline and increases by one for each event, and no event is dropped, so consecutive numbers mean the consumer saw every event the watch produced. Started and Finished are the local clock around the read, not the time Ceph changed state. Epoch is the native MGRMap epoch of that read.

Status is the MGRMap's active manager, its availability and the registered standbys, as ManagerStatus reports them, with Standbys sorted by name and GID. An empty ActiveName with Available false is a successful observation of a map without an active manager, not a failed read. Services are the service URLs of the same MGRMap, keyed by module name: the URLs that this map's active manager reported, as ManagerServices returns them, not a separate query. An advertised URL does not prove that its listener accepts requests.

Status and Services are compared field by field; the same name with another GID is a change, and an epoch change that leaves them equal produces no event. Each event has its own slice and map, which the consumer may keep or change. Two events separated by a gap, or simply by the polling interval, do not establish the transitions in between.

type MessengerConnectionMode added in v0.2.0

type MessengerConnectionMode string

MessengerConnectionMode is one Messenger v2 connection mode.

const (
	// MessengerCRC checks frame integrity with CRC32C and does not encrypt.
	MessengerCRC MessengerConnectionMode = "crc"
	// MessengerSecure encrypts frames with AES-GCM.
	MessengerSecure MessengerConnectionMode = "secure"
)

type MessengerMode

type MessengerMode int

MessengerMode selects, at bootstrap, the Messenger protocols that every daemon listens on and advertises, and whether the connection modes stay configurable. Both are fixed for the life of the cluster.

const (
	// MessengerDefault retains Ceph's own Messenger defaults and v1/v2 MON
	// addresses. It does not imply that every connection uses encryption.
	MessengerDefault MessengerMode = iota
	// MessengerV2Secure allows only encrypted Messenger v2 connections, with
	// no CRC-mode negotiation or legacy Messenger v1 listener.
	MessengerV2Secure
	// MessengerV2Only listens on and advertises only Messenger v2, like
	// MessengerV2Secure, but keeps the connection modes configurable: Ceph's
	// defaults, NetworkConfig.WithMessengerPolicy, override.SetMessengerPolicy and
	// ClientContainerOptions.WithMessengerPolicy select them, as with MessengerDefault.
	MessengerV2Only
)

type MessengerPolicy added in v0.2.0

type MessengerPolicy struct {
	// Monitors sets ms_mon_service_mode for the MONs.
	Monitors []MessengerConnectionMode
	// Managers sets ms_mon_service_mode for the MGRs: Ceph applies the MON
	// option to connections from clients to MGRs.
	Managers []MessengerConnectionMode
	// OSDs sets ms_service_mode for the OSDs.
	OSDs []MessengerConnectionMode
}

MessengerPolicy selects, for each daemon type, the Messenger v2 connection modes that it accepts from clients. The order of a list does not matter: the connection uses the first mode in the client's order of preference that the daemon accepts. A nil field keeps Ceph's default, which accepts both modes. Connections between daemons keep their defaults, so every policy leaves the cluster itself working.

A client that offers no accepted mode fails at once against a MON (EACCES; the ceph CLI prints RADOS permission denied and exits with 13), but gets no error from a MGR or an OSD and keeps retrying until its own timeout, so bound such calls. The MON client's handle_auth_bad_method log line is not reliable in the ceph CLI's output, which exits without shutting librados down.

type MetricSample added in v0.4.0

type MetricSample struct {
	// Name is the sample's metric name, such as ceph_osd_op_w.
	Name string
	// Labels are the sample's labels, such as ceph_daemon="osd.0".
	Labels map[string]string
	// Value is the sample's value, which may be NaN or ±Inf.
	Value float64
	// Timestamp is the sample's timestamp in milliseconds, or 0 without one.
	Timestamp int64
}

MetricSample is one sample of the Prometheus text exposition format.

type MonitorContainer

type MonitorContainer struct {
	testcontainers.Container
	DaemonName string
	// contains filtered or unexported fields
}

MonitorContainer is an owned quorum member. Stop/Start preserves its store; MonitorContainer.Remove explicitly removes membership and deletes the container.

func (*MonitorContainer) AdminSocket added in v0.4.0

func (m *MonitorContainer) AdminSocket(ctx context.Context, args ...string) ([]byte, error)

AdminSocket runs a command on the admin socket of this MON, which its cluster must still own, and returns the command's output. args are the command words, such as "status", "version" or "config", "get", "osd_heartbeat_grace"; "help" lists the daemon's commands.

The command does not use the Messenger or Cephx: it reaches the daemon through its local socket, so it works while the daemon cannot reach a monitor and reports what the daemon itself has applied, for example the newest OSDMap epoch in an OSD's "status". It runs in the control-plane container, which shares the daemons' run directory, so it works whatever the daemon's own image contains. The daemon must be running; a stopped or removed daemon returns an error. A paused daemon accepts the connection but does not answer, so the call waits for ctx or the cluster's startup timeout, which also bounds every call. When the call ends early, AdminSocket stops its CLI in the control container before it returns, including one that Docker has not started yet. A nonzero exit returns the output with an error.

func (*MonitorContainer) Image added in v0.4.0

func (m *MonitorContainer) Image() string

Image returns the image this MON container was started from: the DaemonOptions.WithImage override, or the cluster's essential image.

func (*MonitorContainer) Remove added in v0.4.0

func (m *MonitorContainer) Remove(ctx context.Context) error

Remove deletes this owned monitor. It preserves a working quorum: add a replacement first when a stop would leave fewer than a majority of the current map running. A handle of a removed monitor returns an error.

It refuses a removal that would leave fewer than a majority of the current map running; add a replacement first. As in Ceph's manual procedure, the MON's process is killed before the mon remove, so that no warm daemon or client can take the leaving MON as its session while the survivors renumber their ranks and drop their sessions. MON a can be removed too; the embedded container of Container is then gone and fixture commands use the control container. If the mon remove fails or its reply is lost, the MON stays killed and the call returns an error without resending; calling Remove again with a fresh context reads the current membership, skips the completed native step and finishes the container and configuration cleanup. MON a is described by a fresh descriptor on every listing, so a handle is matched by name and container, not by pointer identity.

func (*MonitorContainer) Replace added in v0.4.0

func (m *MonitorContainer) Replace(ctx context.Context, opts ...DaemonOptions) (*MonitorContainer, error)

Replace starts this MON in a new container on the store and address it keeps, and waits for it to rejoin the quorum. The membership does not change: the MON keeps its name, rank and monmap entry. It needs MonitorStorage.WithVolume. DaemonOptions.WithImage starts the new container from another image, for example a newer release; with no option it uses the image this MON ran. The store is not checked against that image's release, and nothing makes the change reversible.

Before it touches the running MON it refuses a replacement that would leave fewer than a majority of the monmap in quorum, a store that is not attached to this handle's container, and a store with no recorded address. It then removes the old container, confirms that nothing mounts the volume, and creates the new one with the recorded IPv4 address, in a mode that refuses to format a missing store. It returns the new handle; this handle names the removed container and is stale, also for MON a, whose embedded container of Container is then gone and whose fixture commands use the control container.

If removal of the old container is unconfirmed, retry on this handle or terminate the cluster. When the old container is gone and the new one fails, the error comes with the new handle if a container exists: the store and its volume stay owned, nothing is formatted, and Replace on that handle or Terminate continues. A creation whose reply was lost is never replayed; only Terminate settles it.

func (*MonitorContainer) Version added in v0.4.0

func (m *MonitorContainer) Version(ctx context.Context) (string, error)

Version returns the release that this running MON reports itself, such as "20.2.4". See OSDContainer.Version.

type MonitorStorage added in v0.4.0

type MonitorStorage struct {
	// contains filtered or unexported fields
}

MonitorStorage is the MON storage group of the Ceph configuration: where every MON of the cluster keeps its store, including MONs added later. It is a plain value with chainable, value-returning methods, not a customizer; pass it to Run through its one grouped option function, WithMonitorStorage. The zero value changes nothing.

cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithMonitorStorage(ceph.MonitorStorage{}.WithVolume()))

Run refuses a second WithMonitorStorage that sets the volume again.

func (MonitorStorage) WithVolume added in v0.4.0

func (s MonitorStorage) WithVolume() MonitorStorage

WithVolume keeps each MON's store in a Docker volume of its own that the cluster owns, mounted at the MON data directory, instead of the container's writable layer. The volume outlives the MON's container, so MonitorContainer.Replace can start a new container, from another image if asked, on the same store and address. Terminate removes the volumes. It needs IPv4 addresses: the replacement binds the address the monmap pins. On a bridge that is the MON's IPv4 address on the public network; on the host network it is the public address and the MON's ports, which the replacement holds from the removal of the old container until the start of the new one.

type MonitorWatch added in v0.5.0

type MonitorWatch struct {
	// contains filtered or unexported fields
}

MonitorWatch is a running watch of the MON quorum and monmap membership. One goroutine reads the quorum status and sends events; Events closes after that goroutine has stopped using the cluster. Stop ends the watch and waits for it.

func (*MonitorWatch) Events added in v0.5.0

func (w *MonitorWatch) Events() <-chan MonitorWatchEvent

Events returns the receive-only event channel. The baseline is already in it when WatchMonitors returns. The channel closes when the watch ends; Stop returns why.

func (*MonitorWatch) Stop added in v0.5.0

func (w *MonitorWatch) Stop() error

Stop cancels the watch if it is still running, waits until its goroutine has returned and Events is closed, and returns the cause that ended the watch: nil when Stop ended it, the context's error when the WatchMonitors context ended, an ErrWatchOverflow error, or the read error that ended it (a terminated cluster, or a monmap of another cluster). Buffered events stay readable after Stop. Stop is safe to call more than once and from any goroutine; every call returns the same cause. Call it before the fixture's cleanup: cancelling the context alone does not wait.

type MonitorWatchEvent added in v0.5.0

type MonitorWatchEvent struct {
	Sequence          uint64
	Kind              WatchEventKind
	Started, Finished time.Time
	MonMapEpoch       uint64
	ElectionEpoch     uint64
	Leader            string
	Members           []DaemonAddress
	Quorum            []string
	Err               error
}

MonitorWatchEvent is one event of a MonitorWatch.

Sequence starts at 1 with the baseline and increases by one for each event, and no event is dropped, so consecutive numbers mean the consumer saw every event the watch produced. Started and Finished are the local clock around the read, not the time Ceph changed state.

MonMapEpoch and ElectionEpoch are the native monmap and election epochs of that read, and Leader is the quorum leader it names. Members are the monmap members, sorted by name, each with the public address vector the monmap advertises in native order; Quorum are the names of the members in quorum, sorted. Membership and quorum participation are separate facts: a stopped MON stays a member and leaves Quorum. Members, Quorum and Leader are compared; an epoch change that leaves them equal produces no event. Each event has its own slices, which the consumer may keep or change. Two events separated by a gap, or simply by the polling interval, do not establish the transitions in between.

type NativeCommand added in v0.2.0

type NativeCommand struct {
	// Argv is the program and its arguments. It runs without a shell.
	Argv []string
	// Timeout bounds the command's run time, at most
	// MaxNativeCommandTimeout. The caller's context can shorten it.
	Timeout time.Duration
	// MaxStdout and MaxStderr bound each output stream inside the container,
	// each at most MaxNativeCommandOutput bytes.
	MaxStdout, MaxStderr int
	// Label names the command in errors, which never contain its arguments
	// or output. It defaults to "native command".
	Label string
	// Input, when set, is the command's standard input, and MaxInput, at
	// most MaxNativeCommandInput, bounds the bytes written to it in all.
	// Without Input the command's standard input is empty, and MaxInput must
	// be 0. An Input serves one command.
	Input    *NativeInput
	MaxInput int
}

NativeCommand is one native command that RunNativeCommand runs under a supervisor inside a container.

type NativeCommandResult added in v0.2.0

type NativeCommandResult struct {
	Stdout, Stderr []byte
	// ExitCode is the command's exit code; a nonzero one is not an error.
	ExitCode int
	// TimedOut means that the supervisor killed the command's process group
	// at its deadline; the output is what it wrote until then.
	TimedOut bool
	// StdoutTruncated and StderrTruncated mean that a stream exceeded its
	// limit; the result holds its first bytes.
	StdoutTruncated, StderrTruncated bool
	// InputWritten counts the bytes of Input that the input stream to the
	// container took, and InputDelivered those that the supervisor wrote into
	// the command's standard input; the command may not have read all of
	// them. They differ when the command ended, or closed its standard input,
	// before it took everything. InputEOF means that the command's standard
	// input ended after all InputWritten bytes, because Input was closed.
	InputWritten, InputDelivered int64
	InputEOF                     bool
}

NativeCommandResult is the outcome of a supervised native command. Stdout and Stderr are separate, raw and binary-safe.

func (NativeCommandResult) Complete added in v0.2.0

func (r NativeCommandResult) Complete() bool

Complete reports whether the command finished within its time and output limits, so that its output is complete.

type NativeCommandUnstoppedError added in v0.2.0

type NativeCommandUnstoppedError struct {
	// contains filtered or unexported fields
}

NativeCommandUnstoppedError reports a native command that RunNativeCommand, or rgw.Gateway.Admin, could not confirm as stopped after its call ended early: it may still run in its container. Stop retries the confirmation; Terminate also tries it.

func (*NativeCommandUnstoppedError) Error added in v0.2.0

func (*NativeCommandUnstoppedError) Stop added in v0.2.0

Stop stops the command, or confirms that it ended with its container, and returns nil once the command can no longer run: after the container was removed, or once it runs again after a restart. A stopped container cannot confirm anything until then. Each attempt has its own NativeCommandStopTimeout, and waits for a concurrent attempt within ctx. Stop is idempotent and can be retried, also after Terminate.

func (*NativeCommandUnstoppedError) Unwrap added in v0.2.0

func (e *NativeCommandUnstoppedError) Unwrap() error

type NativeFailure added in v0.2.0

type NativeFailure struct {
	// Program is the command's program, such as "ceph" or "rbd"; the rest of
	// its arguments is never recorded.
	Program string
	// ExitCode is its exit code.
	ExitCode int
	// Errno is the error code that the tool reported, such as "ENOENT", or
	// "" when it reported none. It is read from the "Error ENOENT:" that
	// starts a line of Ceph and from the last "(2) No such file or directory"
	// of a line of the other tools, never guessed from the exit code.
	Errno string
	// Reason is the tool's own explanation: the lines of its error output
	// without Ceph's log lines, joined with "; ", at most
	// MaxNativeFailureReason bytes, with keys and the secrets that
	// diagnostics.Redact recognizes replaced. It can name resources of the
	// request, such as a pool.
	Reason string
}

NativeFailure is the status of a native command that ran and exited nonzero. errors.Is matches it with ErrNativeFailure, and with fs.ErrNotExist, fs.ErrPermission or fs.ErrExist when the tool reported ENOENT, EACCES or EPERM, or EEXIST. RunNativeCommand returns a nonzero exit as its result instead; the module's own commands and Ceph return this.

func (*NativeFailure) Error added in v0.2.0

func (f *NativeFailure) Error() string

func (*NativeFailure) Is added in v0.2.0

func (f *NativeFailure) Is(target error) bool

Is reports whether target is ErrNativeFailure, or the fs error that matches the reported errno.

type NativeInput added in v0.3.0

type NativeInput struct {
	// contains filtered or unexported fields
}

NativeInput is the standard input of one NativeCommand while it runs. Write sends data in order, and Close ends the input. The command gets exactly the bytes written, without a shell, an encoding or a terminal in between. Nothing that is written appears in arguments, the environment, files, errors, labels or diagnostics; the command's own output is returned as it wrote it, so a command that must not reveal its input must not echo it.

Inside the container the supervisor reads what arrives at once, up to MaxInput, and passes it on as the command reads, so a command that reads slowly or not at all does not hold up Write. Its methods are safe for concurrent use.

func NewNativeInput added in v0.3.0

func NewNativeInput() *NativeInput

NewNativeInput returns an open input for one NativeCommand.

func (*NativeInput) Close added in v0.3.0

func (in *NativeInput) Close() error

Close ends the input: the command gets the end of input after every byte written before. Close is idempotent and never fails; a command that ended before is not affected.

func (*NativeInput) Write added in v0.3.0

func (in *NativeInput) Write(ctx context.Context, data []byte) error

Write sends data to the command's standard input after the data of earlier writes, and returns once the input stream to the container took all of it. A write before RunNativeCommand starts the command waits for it. ctx bounds that wait; when ctx ends after the stream took the data, Write returns ctx's error, and the command may still get the data, as NativeCommandResult.InputWritten tells. Write copies data before it returns.

A write that would take the input past MaxInput is refused whole with an error that matches ErrInvalidRequest, and the input stays open. Once the input is closed, or the command ended or its input stream failed, Write returns an error that matches ErrNativeInputClosed. An empty write sends nothing.

type NetworkConfig added in v0.4.0

type NetworkConfig struct {
	// contains filtered or unexported fields
}

NetworkConfig is the network and Messenger group of the Ceph configuration: which addresses the daemons bind and advertise, whether OSD replication traffic uses its own bridge, the host address in host-network mode and the Messenger protocol and connection-mode policy. It is a plain value with chainable, value-returning methods, not a customizer; pass it to Run through its one grouped option function, WithNetworkConfig. The zero value changes nothing, and each field records whether it was set, so an explicit setting (including the default family AddressIPv4 or MessengerDefault) is different from an unspecified one.

network := ceph.NetworkConfig{}.WithAddressFamily(ceph.AddressDualStack).WithSeparateClusterNetwork()
cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithNetworkConfig(network))

Within one value a later call of the same method replaces the earlier one. Run refuses a field that two WithNetworkConfig options both set, even when they agree, so argument order never decides a conflict. Incompatible combinations (host network with a separate cluster network or a non-IPv4 family, a host address without WithHostNetwork, a Messenger policy with MessengerV2Secure) are refused before any resource is created. Docker execution policy such as WithHostNetwork stays a separate option function. The composition rules are in docs/contributing/configuration-composition.md.

func (NetworkConfig) WithAddressFamily added in v0.4.0

func (n NetworkConfig) WithAddressFamily(family AddressFamily) NetworkConfig

WithAddressFamily selects the addresses of the Messenger listeners of every MON, MGR, OSD and service daemon. Every family other than AddressIPv4 needs bridge mode, and AddressIPv6 and AddressDualStack a Docker Engine that can allocate IPv6 subnets. With WithSeparateClusterNetwork, AddressIPv6 makes both bridges dual-stack and their IPv6 subnets Ceph's public and cluster networks, and AddressDualStack makes both subnets of each bridge its network; AddressIPv4MappedIPv6 cannot be combined with it.

AddressDualStack sets ms_bind_ipv4, ms_bind_ipv6 and ms_bind_prefer_ipv4, and public_network to both subnets of the bridge. Every address vector has an entry of each family for each enabled Messenger protocol: v2, and v1 unless WithMessengerMode removes it. MON vectors list the IPv6 address first, which keeps a MGR from advertising the IPv6 wildcard address [::] that clients cannot reach; the other daemons list the IPv4 address first. Native clients connect to the first entry, so they reach MGRs, OSDs and MDSs over IPv4; ms_bind_prefer_ipv4 = false in a daemon's section of a WithConfigFile file makes that daemon list its IPv6 address first. Run, AddManager and ManagerContainer.Remove check that the MONs, the active MGR and the up OSDs advertise a concrete address of each family, and fail with ErrUnsupported otherwise. An unknown family is refused.

ClusterFacts.AddressFamily reports the selection. With AddressIPv6 and AddressIPv4MappedIPv6 the generated ceph.conf sets ms_bind_ipv4 = false and ms_bind_ipv6 = true, so MGRs, OSDs and MDSs bind [::] and advertise the address from which the MONs see them; no daemon has an IPv4 Messenger socket, and with AddressIPv4MappedIPv6 the IPv6 sockets accept the IPv4 connections the mapped addresses stand for. With AddressIPv6, Docker forwards an IPv4 connection to a published MON port to the container's IPv4 address, where no MON listens, so run clients in WithClient containers. Ceph prints an IPv4-mapped address in dotted form but parses only hexadecimal groups (::ffff:ac13:2); the fixture writes mon_host in the parseable form, so convert an address the same way before passing it to a Ceph tool. HTTP endpoints such as RGW's keep their addresses. Docker reuses the address of a removed or stopped container, so identify a daemon by its entry's nonce or its GID, not by address alone. See docs/guides/ceph-configuration.md for dual-stack address order and for cutting one family of a path.

func (NetworkConfig) WithHostAddress added in v0.4.0

func (n NetworkConfig) WithHostAddress(address string) NetworkConfig

WithHostAddress selects a local IPv4 address to bind and advertise in host mode, typically the Docker host's address when Docker is remote. The address must be assigned on the Docker daemon host and reachable from the test process; Run verifies both. Requires WithHostNetwork. An empty, unspecified, multicast, broadcast or non-IPv4 address is refused.

func (NetworkConfig) WithMessengerMode added in v0.4.0

func (n NetworkConfig) WithMessengerMode(mode MessengerMode) NetworkConfig

WithMessengerMode selects the Messenger policy at bootstrap. MessengerDefault retains Ceph's image defaults and v1/v2 addresses. MessengerV2Secure requires encrypted v2 connections without CRC-mode or v1 fallback, including later daemons and WithClient containers. MessengerV2Only removes the v1 listeners and addresses in the same way but leaves the connection modes to Ceph's defaults and the Messenger policy. This setting does not enable RGW HTTP TLS. An unknown mode is refused. Both settings are fixed for the cluster's lifetime.

In both v2 modes the monmap and mon_host contain only v2 addresses and no daemon listens on a v1 port: in bridge mode the MON publishes only 3300/tcp, so MappedPort(ctx, "6789/tcp") fails, and in host mode one MON port is reserved. Later daemons, WithClient and ClientConfig.WithIdentity containers and ConnectionConfig inherit the mode, and override.SetConfig refuses ms_bind_msgr1 and ms_bind_msgr2 (and, with MessengerV2Secure, the six ms_*_mode options) because the local file would override them. MessengerV2Only sets only ms_bind_msgr1 = false and ms_bind_msgr2 = true; MessengerV2Secure also sets ms_cluster_mode, ms_service_mode, ms_client_mode and their ms_mon_ forms to secure, so a CRC-only client is rejected with AUTH_BAD_METHOD rather than timing out. Cephx stays required in every mode. ClusterFacts.MessengerMode reports the selected mode, not live connections; on Tentacle a daemon's messenger dump admin command reports the mode of each connection.

func (NetworkConfig) WithMessengerPolicy added in v0.4.0

func (n NetworkConfig) WithMessengerPolicy(policy MessengerPolicy) NetworkConfig

WithMessengerPolicy selects the connection modes that MONs, MGRs and OSDs accept from clients. Run stores the policy in the central configuration right after the first MON starts, before any other daemon, so every daemon starts with it. override.SetMessengerPolicy changes it on a running cluster. It cannot be combined with MessengerV2Secure, whose local configuration fixes every mode. The value keeps its own copy of the lists, so changing the policy's slices afterwards does not change it; a nil list keeps Ceph's default and a non-nil empty list is refused.

func (NetworkConfig) WithSeparateClusterNetwork added in v0.4.0

func (n NetworkConfig) WithSeparateClusterNetwork() NetworkConfig

WithSeparateClusterNetwork creates a second owned bridge for OSD replication, recovery and heartbeat traffic. MON/MGR/MDS/RGW and WithClient use only the public bridge. Docker selects both subnets; with WithAddressFamily(AddressIPv6) both bridges are dual-stack and Ceph uses their IPv6 subnets. This setting requires bridge mode. Only OSDs join both bridges. ClusterFacts.NetworkName and ClusterFacts.ClusterNetworkName name them, and fault.InterruptNetwork with ClusterNetworkPlane disconnects one OSD's whole cluster endpoint (both families) and keeps its public one.

type NetworkPlane

type NetworkPlane string

NetworkPlane selects the client/public or OSD replication network.

const (
	// PublicNetworkPlane is the client/public bridge.
	PublicNetworkPlane NetworkPlane = "public"
	// ClusterNetworkPlane is the OSD replication and heartbeat bridge; with
	// a single bridge it selects the same endpoint as PublicNetworkPlane.
	ClusterNetworkPlane NetworkPlane = "cluster"
)

type OSDAddress added in v0.2.0

type OSDAddress struct {
	ID     int
	Up, In bool
	// Public serves clients; Cluster carries replication and recovery.
	Public, Cluster []EntityAddress
	// HeartbeatFront and HeartbeatBack carry OSD heartbeats on the public and
	// cluster sides.
	HeartbeatFront, HeartbeatBack []EntityAddress
}

OSDAddress is one OSD's entry in the OSDMap. The address vectors are those of the OSD's last boot; a down OSD keeps them.

type OSDBenchmarkPolicy added in v0.5.0

type OSDBenchmarkPolicy string

OSDBenchmarkPolicy selects the automatic mClock startup-calibration policy of the cluster's owned OSDs. It does not select a scheduler or supply measured IOPS capacity. MemStore is already excluded from automatic calibration by native Ceph; BlueStore on tmpfs is not MemStore.

const (
	// OSDBenchmarkNative permits Ceph's native calibration decision. It sets
	// osd_mclock_skip_benchmark=false, not force=true: a previously configured
	// non-default capacity can still avoid another calibration. MemStore and
	// other schedulers do not enter mClock's automatic benchmark path.
	OSDBenchmarkNative OSDBenchmarkPolicy = "native"
	// OSDBenchmarkSkip bypasses automatic mClock startup calibration while
	// retaining the scheduler and its configured capacity/bandwidth assumptions.
	// Those assumptions are not a measurement of the fixture's backing device.
	OSDBenchmarkSkip OSDBenchmarkPolicy = "skip"
)

type OSDConfig

type OSDConfig struct {
	Host        string
	Rack        string
	Zone        string
	Root        string
	DeviceClass string
}

OSDConfig places a container's OSD in the logical CRUSH hierarchy root > zone > rack > host. Root defaults to the cluster's selected default root, Host defaults to osd-ID, and Rack/DeviceClass are optional. Multiple OSDs may share a host, rack or zone for domain-failure tests.

Zone names a CRUSH bucket of type zone and, at the same time, the logical availability zone of WithZoneConfig whose network side the OSD's container joins. It is required in a cluster with zones and refused in one without.

type OSDContainer

type OSDContainer struct {
	testcontainers.Container
	ID int
	// contains filtered or unexported fields
}

OSDContainer is one storage daemon backed by a sparse BlueStore file by default, or by process memory with OSDStorage.WithObjectStore(OSDMemStore). The file is container-local unless OSDStorage.WithInMemory or OSDStorage.WithVolume is selected. Stop/Start can be used for failure injection; OSDContainer.Remove drains and purges it.

func (*OSDContainer) AdminSocket added in v0.4.0

func (o *OSDContainer) AdminSocket(ctx context.Context, args ...string) ([]byte, error)

AdminSocket runs a command on the admin socket of this OSD, which its cluster must still own, and returns the command's output. args are the command words, such as "status", "version" or "config", "get", "osd_heartbeat_grace"; "help" lists the daemon's commands.

The command does not use the Messenger or Cephx: it reaches the daemon through its local socket, so it works while the daemon cannot reach a monitor and reports what the daemon itself has applied, for example the newest OSDMap epoch in an OSD's "status". It runs in the control-plane container, which shares the daemons' run directory, so it works whatever the daemon's own image contains. The daemon must be running; a stopped or removed daemon, or one that Remove or Detach has begun to remove, returns an error. A paused daemon accepts the connection but does not answer, so the call waits for ctx or the cluster's startup timeout, which also bounds every call. When the call ends early, AdminSocket stops its CLI in the control container before it returns, including one that Docker has not started yet. A nonzero exit returns the output with an error.

func (*OSDContainer) Detach added in v0.4.0

func (o *OSDContainer) Detach(ctx context.Context) (*DetachedOSD, error)

Detach stops and removes this owned OSD's container but keeps its store in the owned volume and its MON registration, so that DetachedOSD.Reopen can start a new container on them. It needs OSDStorage.WithVolume. Detach marks the OSD neither out nor purged, but Ceph marks a down OSD out after mon_osd_down_out_interval; hold noout to keep it in.

Detach admits the native identity, refuses an OSD that a held network or pause fault targets, fences the instance against new native commands and drains its tracked ones, then stops it, waits for down and removes the container. Only after confirmed removal does it read the store's identity and publish the DetachedOSD; this *OSDContainer stays tied to the removed instance. On error the OSD stays owned with its fence: retry Detach with a fresh context or terminate the cluster. Remove refuses a detaching or detached OSD.

Detach only an OSD that is up: a daemon that is still booting can be killed at the end of the stop grace before it marks itself down, and in a one-OSD cluster no peer reports it, so Detach waits until its context ends. A detached OSD leaves Container.OSDs.

func (*OSDContainer) Image added in v0.4.0

func (o *OSDContainer) Image() string

Image returns the image this OSD container was started from: the OSDOptions.WithImage override, or the cluster's essential image. DetachedOSD.Reopen starts the OSD from the same image again.

func (*OSDContainer) Placement

func (o *OSDContainer) Placement() OSDConfig

Placement returns a copy of this OSD's resolved logical CRUSH location.

func (*OSDContainer) Remove added in v0.4.0

func (o *OSDContainer) Remove(ctx context.Context) error

Remove drains this owned OSD, waits for safe-to-destroy, stops it, waits for down, purges it from Ceph and removes its container. Another owned OSD must still have its container and native registration and be outside removal or detach. After identity admission, removal fences this Docker instance against new native commands and drains its tracked commands before native mutations. A failed drain retains ownership and its fence for retry; other targets remain admitted. This is permanent removal, not retained-store detach or reopen. Every removal phase verifies the captured registration UUID. External OSD replacement must not race removal: Ceph has no UUID compare-and-swap purge. An uncertain purge response remains tracked; a retry reconciles fresh native absence only after this handle issued purge. Completed purge retries only Docker cleanup. On error retry with a fresh context or terminate the cluster.

Remove acts on this handle's own instance. A handle of a removed or detached OSD, or one that Reopen replaced, returns an error and never acts on the container that owns the ID now. Remove refuses a detaching or detached OSD.

Once admitted, removal is permanent. SetIn and Detach on this handle then return ErrInvalidRequest, including after a failed drain. If a drain times out because the remaining OSDs lack space or placement domains, add the required capacity or adjust pool replication, then retry Remove with a fresh context. Terminate the cluster to discard the fixture instead.

func (*OSDContainer) RotateKey added in v0.4.0

func (o *OSDContainer) RotateKey(ctx context.Context) error

RotateKey replaces the Cephx key of this running OSD without a restart, as cephadm's daemon key rotation does. The rotation is described at ManagerContainer.RotateKey; the same steps and failure semantics apply. A handle of an OSD that Remove or Detach removed, or has begun to remove, returns an error.

func (*OSDContainer) SetIn added in v0.4.0

func (o *OSDContainer) SetIn(ctx context.Context, in bool) error

SetIn changes native OSDMap membership of this owned OSD. It does not stop, purge or change CRUSH capacity weights. Readback verifies the owned UUID and desired state, since native in/out can succeed for nonexistent IDs. WaitForPGClean can wait for recovery while an OSD remains out. Ceph preserves a nonzero map reweight across out/in. External replacement must not race this method; Ceph has no compare-and-swap operation for OSD membership. A handle of an OSD that Remove or Detach removed, or has begun to remove, returns an error.

func (*OSDContainer) Version added in v0.4.0

func (o *OSDContainer) Version(ctx context.Context) (string, error)

Version returns the release that this running OSD reports itself, such as "20.2.4", read from its own admin socket. It does not use CephVersion, which describes the first MON and the CLI, so it is the way to see the release of an OSD that OSDOptions.WithImage started from another image. It works under the same conditions as AdminSocket, and a stopped daemon, a stale handle or an unrecognized reply returns an error.

type OSDFlag added in v0.4.0

type OSDFlag string

OSDFlag is a global OSDMap flag token such as "noout". The constants name the flags that override.SetOSDFlag can change. OSDFlags returns the flags that the cluster reports, which can include other tokens, so the type stays open: a value outside the constants is representable and override.SetOSDFlag refuses it at run time.

const (
	OSDFlagNoUp        OSDFlag = "noup"
	OSDFlagNoDown      OSDFlag = "nodown"
	OSDFlagNoOut       OSDFlag = "noout"
	OSDFlagNoIn        OSDFlag = "noin"
	OSDFlagNoBackfill  OSDFlag = "nobackfill"
	OSDFlagNoRebalance OSDFlag = "norebalance"
	OSDFlagNoRecover   OSDFlag = "norecover"
	OSDFlagNoScrub     OSDFlag = "noscrub"
	OSDFlagNoDeepScrub OSDFlag = "nodeep-scrub"
	OSDFlagNoSnapTrim  OSDFlag = "nosnaptrim"
	OSDFlagNoAutoscale OSDFlag = "noautoscale"
)

The reversible global OSD flags that override.SetOSDFlag accepts. Composite pause and irreversible compatibility flags are not among them.

func (OSDFlag) String added in v0.4.0

func (f OSDFlag) String() string

String returns the native flag token.

type OSDObjectStore added in v0.2.0

type OSDObjectStore string

OSDObjectStore names the object store that OSDs keep their data in.

const (
	// OSDBlueStore keeps each OSD's data in a sparse BlueStore file. It is the
	// default.
	OSDBlueStore OSDObjectStore = "bluestore"
	// OSDMemStore keeps each OSD's data in the OSD process's memory, with
	// OSDStorage.WithBlockSize as its capacity. A clean stop saves the store to the
	// OSD's data directory and Start loads it again; a killed OSD loses it.
	// It is Ceph's test backend and needs no block file.
	OSDMemStore OSDObjectStore = "memstore"
)

type OSDOptions added in v0.4.0

type OSDOptions struct {
	// contains filtered or unexported fields
}

OSDOptions is the grouped configuration of AddOSD. The zero value selects every default: the placement that the cluster resolves for the new OSD and the cluster's essential image. Each With method returns a modified copy and leaves its receiver unchanged, so one base value can derive several independent configurations:

base := ceph.OSDOptions{}.WithPlacement(ceph.OSDConfig{Rack: "rack-a"})
first := base.WithImage(imageA)
second := base.WithImage(imageB)

AddOSD composes any number of values; the rules are in docs/contributing/configuration-composition.md. Every field tracks whether it was set, so an explicit empty setting differs from an unspecified one, and the value holds only strings, so a copy never shares mutable storage.

func (OSDOptions) WithImage added in v0.4.0

func (o OSDOptions) WithImage(image string) OSDOptions

WithImage starts the one new OSD from image instead of the cluster's essential image, with the same contract as DaemonOptions.WithImage: the image must contain the daemon's binary and the tools of the fixture's startup scripts, nothing here checks its release or compatibility with the running cluster, DetachedOSD.Reopen without WithImage starts the same OSD from the same image again (with WithImage, from that image), and the OSD's handle reports it with Image. The reference must be non-empty and free of whitespace and control characters; an empty image is an explicit invalid selection, not an unspecified one. A later call on the same value replaces the earlier image.

func (OSDOptions) WithPlacement added in v0.4.0

func (o OSDOptions) WithPlacement(config OSDConfig) OSDOptions

WithPlacement places the new OSD in the logical CRUSH location config describes (OSDConfig documents its defaults). All four fields are set together: an empty field is an explicit request for its default, and WithPlacement(OSDConfig{}) is an explicit default placement. A later call on the same value replaces the earlier placement entirely.

type OSDState

type OSDState struct {
	ID     int
	UUID   string
	Up, In bool
	Weight float64
}

OSDState reports authoritative OSDMap state for an owned OSD. Weight is the native map reweight in [0,1], not the CRUSH capacity weight. Out OSDs retain their containers, identities and data; absence or a replaced UUID is an error.

type OSDStorage added in v0.4.0

type OSDStorage struct {
	// contains filtered or unexported fields
}

OSDStorage is the OSD storage group of the Ceph configuration: where and how every OSD of the cluster keeps its data, including OSDs added later. It is a plain value with chainable, value-returning methods, not a customizer; pass it to Run through its one grouped option function, WithOSDStorage. The zero value changes nothing, and each field records whether it was set, so an explicit setting is different from an unspecified one.

storage := ceph.OSDStorage{}.WithBlockSize(256 << 20).WithObjectStore(ceph.OSDMemStore)
cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithOSDStorage(storage))

Within one value a later call of the same method replaces the earlier one. Run refuses a field that two WithOSDStorage options both set, even when they agree, so argument order never decides a conflict. Incompatible combinations (MemStore with in-memory or volume storage, in-memory with volume storage) are refused before any resource is created. The composition rules are in docs/contributing/configuration-composition.md.

func (OSDStorage) WithBlockSize added in v0.4.0

func (o OSDStorage) WithBlockSize(size int64) OSDStorage

WithBlockSize sets each sparse BlueStore file's logical size in bytes. The minimum is 64 MiB and the default remains 1 GiB. These are disposable files, not real disks. It is also the capacity of a MemStore OSD. Small functional fixtures can select WithStartupBenchmark(OSDBenchmarkSkip) before the initial OSDs or later AddOSD calls; native mClock calibration can exhaust a small file even without client data. A size below the minimum, including zero, is an explicit invalid setting, not an unspecified one.

func (OSDStorage) WithInMemory added in v0.4.0

func (o OSDStorage) WithInMemory(maxBytes int64) OSDStorage

WithInMemory keeps all OSD stores in one Docker-managed tmpfs volume. maxBytes is a positive cluster-wide filesystem allocation ceiling, independent of each sparse file's logical WithBlockSize. It does not reserve RAM or limit daemon RSS; tmpfs may swap, and a full volume fails I/O. An owned helper using the essential image preserves data across OSD Stop/Start. Cluster termination discards the volume. Helper loss or Docker host/VM restart can lose data. Requires a Linux Docker engine with local tmpfs volume support, including compatible Docker Desktop engines. No host mount, privileged container or additional image is required. The default remains container-local sparse files. ClusterConfig.WithNoInitialOSDs defers allocation until AddOSD. A non-positive maxBytes is refused. The volume shares the Docker daemon's memory (the Linux VM's on Docker Desktop, the remote host's on a remote engine) with every daemon; size the cap for replicas, metadata, recovery and mClock's startup benchmark, and note that deleting Ceph objects does not necessarily release tmpfs pages. The helper is an idle container without network, created before the first OSD; the cluster refuses to continue with a restarted helper. OSDContainer.Remove discards the purged OSD's directory. Rootless or managed engines may refuse tmpfs volumes.

func (OSDStorage) WithObjectStore added in v0.4.0

func (o OSDStorage) WithObjectStore(store OSDObjectStore) OSDStorage

WithObjectStore selects the object store of every OSD in the cluster, including OSDs added later. WithBlockSize sets the capacity of a MemStore OSD. MemStore cannot be combined with WithInMemory or WithVolume, which place BlueStore files in shared volumes. fault.InjectObjectDataError needs BlueStore. An unknown store, including the empty one, is refused. MemStore keeps data in the OSD process's memory and saves it to the data directory when the store is unmounted, so the fixture disables Ceph's fast OSD shutdown for MemStore OSDs: a stopped OSD keeps its data across Stop and Start, while a killed OSD or a removed container loses it. ceph osd metadata reports osd_objectstore as memstore.

func (OSDStorage) WithStartupBenchmark added in v0.5.0

func (o OSDStorage) WithStartupBenchmark(policy OSDBenchmarkPolicy) OSDStorage

WithStartupBenchmark chooses the startup benchmark policy of every owned OSD, including initial OSDs, AddOSD, same-container Stop/Start and a detached OSD's Reopen. Omitted, it preserves ordinary native configuration and adds no benchmark command-line setting. An explicit policy sets only osd_mclock_skip_benchmark on the daemon command line, before initialization; a local startup-input read checks that the binary accepts the option/value before store preparation. This is not a live calibration receipt. The flag takes precedence over local files and central configuration. Later override.SetConfig calls can change the stored entry while the command-line value stays effective. Other settings, including scheduler, force, IOPS capacity and sequential bandwidth, are unchanged. Native permits the native decision and does not force calibration.

Skip is fixture resource control for small functional BlueStore stores. The native benchmark can prefill 400 MiB of objects per OSD, additional to its timed writes and metadata. Sparse files may materialize physical allocation, and deleting benchmark objects need not release backing blocks or tmpfs pages. Use Native or omit the policy when native calibration or backing-device behavior is part of the test. Neither policy changes the automatic exclusion of MemStore or claims equivalent QoS behavior for every workload.

The value is copied with OSDStorage. Repeated calls on one value replace the earlier policy; two WithOSDStorage carriers that both set it are refused, even when they agree. Unknown policies, including an explicit empty value, are refused before allocating containers.

Example
package main

import (
	"context"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
)

func main() {
	ctx := context.Background()
	storage := ceph.OSDStorage{}.WithBlockSize(256 << 20).
		WithStartupBenchmark(ceph.OSDBenchmarkSkip)
	cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithOSDStorage(storage))
	if cluster != nil {
		defer func() { _ = cluster.Terminate(context.Background()) }()
	}
	if err != nil {
		panic(err)
	}
	// Initial and later OSDs retain mClock with their configured capacity
	// assumptions, without automatic startup calibration of the small store.
	_, err = cluster.AddOSD(ctx)
	if err != nil {
		panic(err)
	}
}

func (OSDStorage) WithVolume added in v0.4.0

func (o OSDStorage) WithVolume() OSDStorage

WithVolume keeps classic BlueStore OSD directories in one ordinary Docker-managed local named volume owned by the cluster. Allocation is lazy until the first OSD registration. Unlike tmpfs it needs no live keeper to preserve bytes; an isolated essential-image helper manages fresh/purged paths. Each OSD still gets its own sparse file with WithBlockSize as its logical capacity. The helper has no network and no cluster credentials; a stopped or restarted helper refuses further storage operations before a new OSD is registered and is not replaced automatically, while existing OSDs can still stop and start with their containers. OSDContainer.Remove drains and purges the native registration, removes the container and discards only its directory. Terminate removes every owned OSD and the helper, then deletes the volume without force; a busy or failed deletion keeps ownership for another Terminate. Changed volume metadata is refused before directory operations or deletion, so never modify or attach the volume externally. It cannot be combined with MemStore or WithInMemory. No caller-owned volumes, host mounts or devices are adopted. This is the only backend on which OSDContainer.Detach retains a store and DetachedOSD.Reopen restarts it in a new container; it gives no host-reboot durability, and Terminate also discards detached stores.

type OSDWatch added in v0.5.0

type OSDWatch struct {
	// contains filtered or unexported fields
}

OSDWatch is a running watch of owned OSD state. One goroutine reads the OSDMap and sends events; Events closes after that goroutine has stopped using the cluster. Stop ends the watch and waits for it.

func (*OSDWatch) Events added in v0.5.0

func (w *OSDWatch) Events() <-chan OSDWatchEvent

Events returns the receive-only event channel. The baseline is already in it when WatchOSDs returns. The channel closes when the watch ends; Stop returns why.

func (*OSDWatch) Stop added in v0.5.0

func (w *OSDWatch) Stop() error

Stop cancels the watch if it is still running, waits until its goroutine has returned and Events is closed, and returns the cause that ended the watch: nil when Stop ended it, the context's error when the WatchOSDs context ended, an ErrWatchOverflow error, or the read error that ended it (a terminated cluster, or an identity change of a watched OSD). Buffered events stay readable after Stop. Stop is safe to call more than once and from any goroutine; every call returns the same cause. Call it before the fixture's cleanup: cancelling the context alone does not wait.

type OSDWatchEvent added in v0.5.0

type OSDWatchEvent struct {
	Sequence          uint64
	Kind              WatchEventKind
	Started, Finished time.Time
	Epoch             uint64
	States            []OSDState
	Err               error
}

OSDWatchEvent is one event of an OSDWatch.

Sequence starts at 1 with the baseline and increases by one for each event, and no event is dropped, so consecutive numbers mean the consumer saw every event the watch produced. Started and Finished are the local clock around the read, not the time Ceph changed state. Epoch is the native OSDMap epoch of that read; an epoch change that leaves States equal produces no event.

States are the watched OSDs, sorted by ID, compared field by field (ID, UUID, Up, In, Weight). Each event has its own slice, which the consumer may keep or change. Two States separated by a gap, or simply by the polling interval, do not establish the transitions in between.

type ObjectWatcher

type ObjectWatcher struct {
	Address  string
	ClientID uint64
	Cookie   uint64
}

ObjectWatcher is one client watching a RADOS object. Address is the client's IP:port/nonce, the same form BlocklistEntries reports. A client that stops answering keeps its watch until the OSD's osd_client_watch_timeout expires, 30 seconds by default; until then a notify to the object waits for that client and times out.

type Observer added in v0.4.0

type Observer struct {
	// contains filtered or unexported fields
}

Observer groups read-only observations and readiness waits of one cluster. ServiceContainers reads tracked ownership handles, including stopped handles; it makes no liveness claim. Other queries read live native state. Each query reads current state rather than retaining a snapshot. It owns no resource and needs no cleanup. Queries keep the cluster's normal locking, liveness and command-outcome checks.

Configuration, BlocklistEntries, MGRModules, OSDFlags, OSDStates and ManagerServices take the topology lock and refuse a terminated cluster before running a command; MonitorBootstrapAddresses, HealthDetails and FullRatios do the same under ctx and keep their original-FSID checks; QuorumStatus takes no lock and fails on a terminated cluster with the error of the command path; PGInconsistencies and ObjectWatchers run rados in the client container. A refused or failed collection is an error, never an empty result.

Example
package main

import (
	"context"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/testcontainers/testcontainers-go"
)

func main() {
	ctx := context.Background()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage,
		ceph.WithClusterConfig(ceph.ClusterConfig{}.WithPools(ceph.PoolConfig{Name: "app"})),
	)
	defer func() { _ = testcontainers.TerminateContainer(cluster) }()
	if err != nil {
		panic(err)
	}
	observer, err := ceph.Observe(cluster)
	if err != nil {
		panic(err)
	}
	if err := observer.WaitForClean(ctx); err != nil {
		panic(err)
	}
	services, err := observer.ServiceContainers(ctx)
	if err != nil {
		panic(err)
	}
	_ = services // Tracked optional-service handles; no liveness claim.
}

func Observe added in v0.4.0

func Observe(cluster *Container) (*Observer, error)

Observe returns an Observer for cluster. A nil cluster returns an error and a nil Observer. Nothing is read from the cluster.

func (*Observer) BlocklistEntries added in v0.4.0

func (o *Observer) BlocklistEntries(ctx context.Context) ([]BlocklistEntry, error)

BlocklistEntries lists exact entries and CIDR ranges without mutating them.

It includes entries made by other tools or by Ceph itself, for example when CephFS or RBD evicts a client. Range is true for a CIDR range and Until is the MON's absolute expiration time. Listing grants no ownership to remove an entry; override.AddBlocklist adds one.

func (*Observer) Configuration added in v0.4.0

func (o *Observer) Configuration(ctx context.Context) ([]ConfigEntry, error)

Configuration lists exact central database entries, including masked entries. Use native config show/daemon config get to inspect a running daemon's effective value; this method does not claim a stored setting has overridden local config.

Entries can contain sensitive values.

func (*Observer) DaemonAddresses added in v0.4.0

func (o *Observer) DaemonAddresses(ctx context.Context) (DaemonAddresses, error)

DaemonAddresses reads the names and advertised addresses of the cluster's MONs, MGRs and OSDs from the native maps. They are the addresses a client learns from the cluster, which can differ from the addresses a container has: a host-network cluster advertises its host address, and an OSD that is down keeps its last addresses.

In bridge mode the addresses are the daemon containers' addresses on the fixture network; in host mode they are ClusterFacts.PublicAddress with the allocated ports. Standby MGRs advertise no address, and a MGR that has just become active advertises its vector only once it is available, so wait for ManagerAvailable after a failover. A down OSD keeps the addresses of its last boot; one that never booted has empty vectors. The three maps are read one after another, not as one snapshot; compare the epochs when the order matters. Each EntityAddress has Type (v2, v1 or any), Addr as Ceph prints it and Nonce, which MONs leave at 0.

func (*Observer) FullRatios added in v0.4.0

func (o *Observer) FullRatios(ctx context.Context) (FullRatioSnapshot, error)

FullRatios reads native thresholds through one captured control CLI handle, checking the original bootstrap FSID before and after the map query. Native zero, equal and misordered thresholds remain observable. It makes no writes, command retries or background workers (a MON connection that times out before a command is sent is reconnected, as in Container.Ceph) and returns a zero snapshot on any failure. These separate reads are not atomic; external configuration writers must not race the observation. Poll HealthDetails for eventual OSD fullness reports.

The ratios are Ceph's float32 values, so do not compare them with float64 literals for equality. They are OSDMap policy, separate from central configuration and pool quotas.

func (*Observer) HealthDetails added in v0.4.0

func (o *Observer) HealthDetails(ctx context.Context) (HealthSnapshot, error)

HealthDetails reads the original cluster's health through one captured control CLI handle, checking the bootstrap FSID before and after the query. It works before storage or MGR readiness, performs no writes and no command retries (a MON connection that times out before the command is sent is reconnected for up to a minute, as in Container.Ceph) and returns a zero snapshot on every failure, including post-query cancellation. Native command output is omitted from errors because it may contain secrets. External native/configuration writers must not race this observation: the three reads are not an atomic transaction. Poll with a caller deadline when asserting eventual health after changing a test topology.

Assert on a check's code and its Muted flag, since HEALTH_OK can include muted warnings; a sticky mute stays after its check clears. Ceph raises many checks only after a delay. Messages can contain pool names, paths or other application data.

func (*Observer) MGRModules added in v0.4.0

func (o *Observer) MGRModules(ctx context.Context) ([]MGRModuleState, error)

MGRModules lists configured and available modules, sorted by name. Enabled and CanRun do not prove command readiness; WaitMGRModuleReady probes selected built-in modules, and callers must probe their intended commands for others.

func (*Observer) ManagerMetrics added in v0.4.0

func (o *Observer) ManagerMetrics(ctx context.Context) (ManagerMetrics, error)

ManagerMetrics scrapes the active manager's prometheus module once and parses the Prometheus text exposition format. It never enables the module: enable it with override.SetMGRModule first, and wait until ManagerServices lists it. The module is not a baseline module of the essential role, so run the managers from an image that ships it in ImageSet.MGR: the mgr-modules role image of the release, which the official variant publishes, or the official image. A module that serves no endpoint is ErrInvalidRequest. The scrape runs in the control container, on the cluster's network, at the address that ManagerServices advertises; it is bounded to 30 seconds and 32 MiB, and a failed fetch or unparsable output is a *NativeFailure or an error.

A standby's empty page is an error. The module reports cluster-level metrics (pools such as ceph_pool_wr, OSD state, health, the MGRs); since Reef it leaves out daemon perf counters unless mgr/prometheus/exclude_perf_counters is false, and each daemon's own counters are in its AdminSocket perf dump. Pool counters come from the MGR's PG statistics, so poll until an expected value appears. In host mode the module first needs its own address and port per manager.

func (*Observer) ManagerServices added in v0.4.0

func (o *Observer) ManagerServices(ctx context.Context) (map[string]string, error)

ManagerServices returns the service URLs, keyed by module name such as prometheus or dashboard, that the active manager advertises in the MGR map. A module appears only after it has started its listener, and a failover replaces the URLs with the new active manager's. The URLs carry the address the module bound: a container address reachable from WithClient containers in bridge mode, or the configured address in host mode. They are not mapped to Docker host ports. An empty map means no module advertises a service.

func (*Observer) ManagerStatus added in v0.4.0

func (o *Observer) ManagerStatus(ctx context.Context) (ManagerStatus, error)

ManagerStatus reads active ownership and registered standbys from the native MGR map.

func (*Observer) MonitorBootstrapAddresses added in v0.4.0

func (o *Observer) MonitorBootstrapAddresses(ctx context.Context) (string, error)

MonitorBootstrapAddresses returns the current majority monmap's versioned address vectors, suitable for the Ceph mon_host setting. The live cluster FSID must match this fixture's original bootstrap configuration. This is a bounded native observation, not a network attachment or membership mutation.

func (*Observer) OSDFlags added in v0.4.0

func (o *Observer) OSDFlags(ctx context.Context) ([]OSDFlag, error)

OSDFlags lists exact global OSDMap flag tokens, including native compatibility flags that are not among the OSDFlag constants. Per-OSD, CRUSH-node or device-class flags are separate native state.

func (*Observer) OSDStates added in v0.4.0

func (o *Observer) OSDStates(ctx context.Context) ([]OSDState, error)

OSDStates returns owned OSD states sorted by native ID. It refuses an owned ID that is missing, purged or now belongs to a different native UUID.

func (*Observer) ObjectWatchers added in v0.4.0

func (o *Observer) ObjectWatchers(ctx context.Context, pool, namespace, object string) ([]ObjectWatcher, error)

ObjectWatchers lists the clients watching one object, including clients outside this fixture. Namespace is empty for the default namespace. It reads native state only and grants no ownership.

Each watcher's Address has the IP:port/nonce form of BlocklistEntries. A client that stops answering keeps its watch until the OSD's osd_client_watch_timeout expires (30 seconds by default); until then a notify on the object waits for it and times out.

func (*Observer) PGInconsistencies added in v0.4.0

func (o *Observer) PGInconsistencies(ctx context.Context, pgid PGID) ([]InconsistentObject, error)

PGInconsistencies lists the objects that the last deep scrub of a PG found inconsistent. An empty result means that scrub found none; it is not a new scrub. A PG whose results are no longer available returns an error.

func (*Observer) Pools added in v0.4.0

func (o *Observer) Pools(ctx context.Context) ([]*Pool, error)

Pools lists the pools that the cluster has, including pools that services or Ceph created, as one Pool for each. Every Pool identifies its native pool by name and native ID, as the Pool of Container.Pool does, and describes only what the listing reports (Name, PGNum, Replicas of a replicated pool, MinSize and ErasureCodeProfile); Pool.Status reads the full native state and Pool.Usage and Pool.PGs the usage and the PGs. The listing is one read of the native pool map, so the pools of one result are consistent with each other and later reads through them are separate observations.

func (*Observer) QuorumStatus added in v0.4.0

func (o *Observer) QuorumStatus(ctx context.Context) (QuorumStatus, error)

QuorumStatus reads the monitor quorum and monmap. It takes no topology lock and selects the control container for each read.

The query runs under a 5-second process limit inside the control container, so it returns an error rather than hanging while quorum is lost.

func (*Observer) ServiceContainers added in v0.5.0

func (o *Observer) ServiceContainers(ctx context.Context) ([]testcontainers.Container, error)

ServiceContainers returns a fresh slice of tracked optional-service handles (such as RGW and MDS), sorted by service name. It reads only the ownership registry: it does not query Docker or claim that a service is running.

The caller's context bounds acquisition of the topology lock. Partially started and terminated clusters remain inspectable; stopped handles and handles retained after failed cleanup are included. Successful service removal or termination can forget a handle. An empty inventory means no handles are tracked, not that Docker has no cluster-owned resources (a lost create reply can leave a resource without a handle). A missing tracked handle is an error, not an omitted entry. The cluster retains cleanup ownership.

func (*Observer) Status added in v0.4.0

func (o *Observer) Status(ctx context.Context) (Status, error)

Status queries the monitor's JSON API through the container-local CLI.

func (*Observer) WaitForClean added in v0.5.0

func (o *Observer) WaitForClean(ctx context.Context) error

WaitForClean waits until a manager is available, the cluster status counts as many OSDs in total, up and in as the fixture owns, at least one PG exists and every PG is exactly active+clean. HEALTH_WARN alone does not block it. PG states with extra flags, such as active+clean+scrubbing, do not count. It never succeeds while an owned OSD is intentionally stopped or out; use WaitForPGClean then. The cluster's startup timeout also bounds the wait.

The three OSD numbers are cluster-wide totals compared with the owned count. The wait reads no OSD ID or UUID, so it says how many OSDs are up and in, not which. To establish that one particular OSD has returned, read OSDStates and compare its entry, for example with the expect package.

Call it after creating pools and after topology changes, before client I/O. An OSD registered outside the fixture blocks the wait only while it makes one of the three numbers differ from the owned count; one that stands in for a missing owned registration does not. A pool whose replica count exceeds the OSD count never becomes clean, and after Pool.SetPGCount or Pool.SetPlacement it can pass before the new PG set is reported (see Pool.WaitForPGCount and Pool.PGs).

func (*Observer) WaitForPGClean added in v0.5.0

func (o *Observer) WaitForPGClean(ctx context.Context) error

WaitForPGClean waits for every existing PG to be exactly active+clean. Unlike WaitForClean it allows intentionally out/down OSDs, and does not assert any OSD topology. At least one PG and an available manager are required. This is a recovery barrier, not a promise of cluster health or daemon availability.

func (*Observer) WaitForQuorum added in v0.5.0

func (o *Observer) WaitForQuorum(ctx context.Context) error

WaitForQuorum waits for a majority of the current monmap, including fixtures with a stopped monitor. It does not require HEALTH_OK or every MON to run.

func (*Observer) WaitMGRModuleReady added in v0.5.0

func (o *Observer) WaitMGRModuleReady(ctx context.Context, name string) error

WaitMGRModuleReady waits for enabled/can-run metadata and successful native read-only commands. Supported modules are rbd_support (task and snapshot schedule lists) and volumes (volume list). Other modules need an application specific command probe. Each poll releases the cluster lock; cancellation and termination stop the wait. Success does not claim worker tasks have completed.

func (*Observer) WatchHealth added in v0.5.0

func (o *Observer) WatchHealth(ctx context.Context, cfg WatchConfig) (*HealthWatch, error)

WatchHealth polls the original cluster's health checks and reports their appearance, clearing, severity, count and mute changes. The default WatchConfig.Interval is 5 seconds.

Before it returns, WatchHealth makes one read under ctx and QueryTimeout. A failed first read returns its error and starts nothing. That read is the baseline: the first event, already buffered, which fixes the watched identity, the original cluster FSID. Checks are not pinned: any native code, unknown and future codes included, can appear or clear. Receive the baseline before triggering the change to observe.

Each later read is one HealthDetails observation, which takes the topology lock for its three CLI calls and releases it before the event is sent and during the wait for the next read, so the consumer may call any fixture operation while the watch runs. A successful read whose compared fields equal the previous successful read produces no event. A failed read, a malformed report included, produces a WatchGap and the watch continues; it is never a healthy or empty observation. The next successful read is WatchResumed. These end the watch instead: ctx ending, Stop, a terminated cluster, a native FSID of another cluster, and a full buffer (ErrWatchOverflow).

The watch only reads: it never mutes, clears or repairs a check. HEALTH_OK can include muted checks, and a sticky mute stays after its check clears; assert on a check's code and Muted flag. The three reads of one sample are not atomic. A check that appears or clears is what the monitor reported, not proof of the SUT's behavior, and sampling can miss a short warning, so the stream is no audit history and the absence of a check proves no absence of a fault.

Example

The baseline is buffered before WatchHealth returns, so the OSD stops only after the watch has a known starting state. The test then waits for the fault's own check code to appear and checks the SUT separately. Messages are not printed: they can contain application data. This example is compiled, not run.

package main

import (
	"context"
	"fmt"
	"time"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
)

func main() {
	ctx := context.Background()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithClusterConfig(ceph.ClusterConfig{}.WithOSDCount(3)))
	if cluster != nil {
		defer func() { _ = cluster.Terminate(context.Background()) }()
	}
	if err != nil {
		panic(err)
	}
	observer, err := ceph.Observe(cluster)
	if err != nil {
		panic(err)
	}
	osds, err := cluster.OSDs(ctx)
	if err != nil {
		panic(err)
	}
	watch, err := observer.WatchHealth(ctx, ceph.WatchConfig{Interval: 2 * time.Second})
	if err != nil {
		panic(err)
	}
	defer func() { _ = watch.Stop() }() // before the cluster's Terminate
	if baseline := <-watch.Events(); len(baseline.Health.Checks) != 0 {
		panic("the cluster is not healthy before the fault")
	}
	grace := 10 * time.Second
	if err := osds[0].Stop(ctx, &grace); err != nil {
		panic(err)
	}
	// ... the SUT runs ...
	for event := range watch.Events() {
		if event.Kind == ceph.WatchGap {
			continue
		}
		if check, ok := event.Health.Checks["OSD_DOWN"]; ok && !check.Muted {
			fmt.Printf("OSD_DOWN %s with count %d (event %d)\n", check.Severity, check.Count, event.Sequence)
			return
		}
	}
	panic(watch.Stop()) // the watch ended before OSD_DOWN was reported
}

func (*Observer) WatchManagers added in v0.5.0

func (o *Observer) WatchManagers(ctx context.Context, cfg WatchConfig) (*ManagerWatch, error)

WatchManagers polls the native MGRMap and reports changes of the active manager, its availability, the standbys and the advertised service URLs. The default WatchConfig.Interval is 3 seconds.

Before it returns, WatchManagers makes one read under ctx and QueryTimeout. A failed first read returns its error and starts nothing. That read is the baseline: the first event, already buffered. It pins the cluster, not a manager: each read runs ceph fsid, mgr dump and ceph fsid again through one control container, and both FSIDs must be the FSID of this fixture's original bootstrap configuration. A failover, a new GID, a standby that registers or leaves, including managers the fixture does not own, are changes the watch reports; a reported name does not make the fixture own that manager. Another FSID ends the watch. Receive the baseline before triggering the change to observe.

Each read takes no topology lock, as ManagerStatus: it waits for the control container's handle no longer than QueryTimeout allows, so the consumer may call any fixture operation while the watch runs. A successful read whose compared fields equal the previous successful read produces no event. A failed read produces a WatchGap and the watch continues; the next successful read is WatchResumed. A gap is not a map without managers. These end the watch instead: ctx ending, Stop, a terminated cluster, another cluster's FSID, and a full buffer (ErrWatchOverflow).

The watch is read-only: it never fails over a manager, enables a module or connects to a service. Available is the MGRMap's report, not proof that a module answers commands; use WaitMGRModuleReady or the module's own requests for that. The stream reports differences between samples, not a complete history of failovers.

Example

The baseline names the active manager before it is stopped. The event that names another available active GID advances the test, for example to a check that the SUT reconnects. This example is compiled, not run.

package main

import (
	"context"
	"fmt"
	"time"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
)

func main() {
	ctx := context.Background()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithClusterConfig(ceph.ClusterConfig{}.WithManagerCount(2)))
	if cluster != nil {
		defer func() { _ = cluster.Terminate(context.Background()) }()
	}
	if err != nil {
		panic(err)
	}
	observer, err := ceph.Observe(cluster)
	if err != nil {
		panic(err)
	}
	managers, err := cluster.Managers(ctx)
	if err != nil {
		panic(err)
	}
	watch, err := observer.WatchManagers(ctx, ceph.WatchConfig{})
	if err != nil {
		panic(err)
	}
	defer func() { _ = watch.Stop() }() // before the cluster's Terminate
	baseline := <-watch.Events()
	for _, manager := range managers {
		if manager.DaemonName == baseline.Status.ActiveName {
			grace := 10 * time.Second
			if err := manager.Stop(ctx, &grace); err != nil {
				panic(err)
			}
		}
	}
	for event := range watch.Events() {
		if event.Kind == ceph.WatchGap {
			continue
		}
		if event.Status.Available && event.Status.ActiveGID != baseline.Status.ActiveGID {
			fmt.Printf("mgr.%s (GID %d) took over from mgr.%s at epoch %d\n", event.Status.ActiveName, event.Status.ActiveGID, baseline.Status.ActiveName, event.Epoch)
			// ... check that the SUT reconnects ...
			return
		}
	}
	panic(watch.Stop()) // the watch ended before another manager took over
}

func (*Observer) WatchMonitors added in v0.5.0

func (o *Observer) WatchMonitors(ctx context.Context, cfg WatchConfig) (*MonitorWatch, error)

WatchMonitors polls the native quorum status and reports changes of the MON quorum and the monmap membership. The default WatchConfig.Interval is 1 second.

Before it returns, WatchMonitors makes one read under ctx and QueryTimeout. A failed first read returns its error and starts nothing. That read is the baseline: the first event, already buffered. It pins the cluster, not a member: its monmap must carry the FSID of this fixture's original bootstrap configuration, and every later read must too. MONs that are added, removed, replaced or moved to other addresses, including members the fixture does not own, are changes the watch reports; a monmap with another FSID ends it. Receive the baseline before triggering the change to observe.

Each read takes no topology lock, as QuorumStatus: it waits for the control container's handle no longer than QueryTimeout allows and runs quorum_status under the same 5-second process limit, so the consumer may call any fixture operation, including a long MON operation, while the watch runs. A successful read whose compared fields equal the previous successful read produces no event. A failed read produces a WatchGap and the watch continues; the next successful read is WatchResumed. Without a majority the query does not complete, so a gap, possibly a run of them, can be all a quorum loss shows: a gap is not an empty quorum or proof that a majority is lost. These end the watch instead: ctx ending, Stop, a terminated cluster, a monmap of another cluster, and a full buffer (ErrWatchOverflow).

The watch is read-only. Quorum is the MONs' own report, not proof that a daemon process runs, and an advertised address does not prove that clients can reach it. The stream reports differences between samples, not a complete history of elections or membership changes.

Example

The baseline is buffered before WatchMonitors returns, so the MON stops only after the watch has a known quorum. The event that shows the stopped MON out of the quorum but still in the monmap advances the test. This example is compiled, not run.

package main

import (
	"context"
	"fmt"
	"slices"
	"time"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
)

func main() {
	ctx := context.Background()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithClusterConfig(ceph.ClusterConfig{}.WithMonitorCount(3)))
	if cluster != nil {
		defer func() { _ = cluster.Terminate(context.Background()) }()
	}
	if err != nil {
		panic(err)
	}
	observer, err := ceph.Observe(cluster)
	if err != nil {
		panic(err)
	}
	monitors, err := cluster.Monitors(ctx)
	if err != nil {
		panic(err)
	}
	watch, err := observer.WatchMonitors(ctx, ceph.WatchConfig{})
	if err != nil {
		panic(err)
	}
	defer func() { _ = watch.Stop() }() // before the cluster's Terminate
	baseline := <-watch.Events()
	stopped := monitors[len(monitors)-1]
	grace := 10 * time.Second
	if err := stopped.Stop(ctx, &grace); err != nil {
		panic(err)
	}
	// ... the SUT runs with the remaining majority ...
	for event := range watch.Events() {
		if event.Kind == ceph.WatchGap {
			continue // an election in progress, not an empty quorum
		}
		member := slices.ContainsFunc(event.Members, func(m ceph.DaemonAddress) bool { return m.Name == stopped.DaemonName })
		if member && !slices.Contains(event.Quorum, stopped.DaemonName) {
			fmt.Printf("mon.%s left quorum %v at election epoch %d (baseline %d)\n", stopped.DaemonName, event.Quorum, event.ElectionEpoch, baseline.ElectionEpoch)
			return
		}
	}
	panic(watch.Stop()) // the watch ended before the MON left the quorum
}

func (*Observer) WatchOSDs added in v0.5.0

func (o *Observer) WatchOSDs(ctx context.Context, cfg WatchConfig) (*OSDWatch, error)

WatchOSDs polls the native OSDMap and reports changes of the owned OSDs. The default WatchConfig.Interval is 2 seconds.

Before it returns, WatchOSDs makes one read under ctx and QueryTimeout. A failed first read returns its error and starts nothing; a cluster that owns no OSD is ErrInvalidRequest. That read is the baseline: the first event, already buffered, and the fixed set of watched OSDs, pinned by ID and native UUID. OSDs added later are not watched. Receive the baseline before triggering the change to observe.

Each later read takes the topology lock for one osd dump, as OSDStates does, and releases it before the event is sent and during the wait for the next read, so the consumer may call any fixture operation while the watch runs. A successful read whose watched states equal the previous successful read produces no event. A failed read produces a WatchGap and the watch continues; the next successful read is WatchResumed. These end the watch instead: ctx ending, Stop, a terminated cluster, a watched OSD that is no longer owned (removed or detached) or whose native ID now has another UUID, and a full buffer (ErrWatchOverflow).

Up and down are the reported OSDMap state, not proof that the daemon process runs or that clients can reach it. The stream reports differences between samples, not a complete history of native transitions.

Example

The baseline is buffered before WatchOSDs returns, so the fault starts only after the watch has a known starting state. One event then advances the test while the SUT keeps running. This example is compiled, not run.

package main

import (
	"context"
	"fmt"
	"time"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
)

func main() {
	ctx := context.Background()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage)
	if cluster != nil {
		defer func() { _ = cluster.Terminate(context.Background()) }()
	}
	if err != nil {
		panic(err)
	}
	observer, err := ceph.Observe(cluster)
	if err != nil {
		panic(err)
	}
	osds, err := cluster.OSDs(ctx)
	if err != nil {
		panic(err)
	}
	watch, err := observer.WatchOSDs(ctx, ceph.WatchConfig{Interval: time.Second})
	if err != nil {
		panic(err)
	}
	defer func() { _ = watch.Stop() }() // before the cluster's Terminate
	baseline := <-watch.Events()
	grace := 10 * time.Second
	if err := osds[0].Stop(ctx, &grace); err != nil {
		panic(err)
	}
	// ... the SUT runs ...
	for event := range watch.Events() {
		if event.Kind == ceph.WatchGap {
			continue
		}
		for _, s := range event.States {
			if s.ID == osds[0].ID && !s.Up {
				fmt.Printf("osd.%d reported down at epoch %d (baseline %d)\n", s.ID, event.Epoch, baseline.Epoch)
				return
			}
		}
	}
	panic(watch.Stop()) // the watch ended before the OSD was reported down
}

type OperationError added in v0.2.0

type OperationError struct {
	// Kind is the category: ErrInvalidRequest, ErrUnsupported,
	// ErrImageCapability, ErrEnvironmentUnavailable, ErrNativeFailure.
	Kind error
	// Op names the operation, such as "create client", when the error's
	// text does not already start with it.
	Op string
	// Target names the container or cluster that the failure concerns, such
	// as "control", "osd.0", "destination cluster" or "netem helper".
	Target string
	// Version is the Ceph release observed on Target, when it matters.
	Version string
	// Image is the immutable ID of Target's image, when the fixture read it.
	Image string
	// Capability names the missing or unsupported capability, such as
	// "ceph-authtool --key-type" or "python3".
	Capability string
	// Err is the cause.
	Err error
}

OperationError is a classified failure with the details that identify it. Its text never includes command arguments, raw output or credentials.

func (*OperationError) Error added in v0.2.0

func (e *OperationError) Error() string

func (*OperationError) Is added in v0.2.0

func (e *OperationError) Is(target error) bool

Is reports whether target is the error's category.

func (*OperationError) Unwrap added in v0.2.0

func (e *OperationError) Unwrap() error

type Option

type Option func(*options) error

Option transfers cluster settings outside the monitor's container request.

func WithAuth added in v0.2.0

func WithAuth(config AuthConfig) Option

WithAuth selects the Cephx key type, the MonMap ciphers and the ticket lifetimes of a new cluster. The settings are part of the bootstrap: the key type and ciphers apply from the first MON, and the lifetimes are written to every daemon's and WithClient container's ceph.conf, so override.SetConfig cannot change them later. An image without key type support, such as Squid or Tentacle before 20.2.4, accepts a configuration in which every cipher is CipherAES, and Run fails before any daemon starts when one is not; Run checks the ceph-authtool --key-type and monmaptool cipher options themselves, not the release number. On Tentacle 20.2.4 and later every default key and cipher is aes256k. A PreferredCipher of aes also sets mon_auth_allow_insecure_key in [mon], which Tentacle requires before its MONs generate AES keys. WithConfigFile rejects auth_* keys, so set the lifetimes here. The MonMap ciphers can change on a running Tentacle cluster with ceph mon set (for example auth_allowed_ciphers), and ceph mon dump reports them.

func WithClusterConfig added in v0.4.0

func WithClusterConfig(cluster ClusterConfig) Option

WithClusterConfig is the grouped option function that carries a ClusterConfig to Run. Like every Option it changes the cluster settings only and never reaches a container request.

Example
package main

import (
	"context"
	"log"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/testcontainers/testcontainers-go"
)

func main() {
	ctx := context.Background()
	topology := ceph.ClusterConfig{}.WithMonitorCount(3).WithOSDCount(3).WithPoolDefaults(3, 2)
	cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithClusterConfig(topology))
	defer func() {
		if err := testcontainers.TerminateContainer(cluster); err != nil {
			log.Printf("failed to terminate container: %s", err)
		}
	}()
	if err != nil {
		panic(err)
	}
}

func WithConfigFile

func WithConfigFile(path string) Option

WithConfigFile reads a native ceph.conf on the test host during option preparation. The first MON uses the merged file when creating its store; each daemon and WithClient container receives it at startup. Entries replace the fixture's defaults for the same section and key, and later files replace earlier entries. The file is fixed at bootstrap and does not change a running cluster. A local value takes precedence over a stored MON database entry, even when override.SetConfig succeeds. Check a daemon's effective value with ceph config show.

Run rejects fixture-owned keys, including daemon command-line options. With AddressDualStack, ms_bind_prefer_ipv4 is allowed: false in a daemon's section makes it advertise IPv6 first. RGW uses [client] or [client.admin]. See docs/guides/ceph-configuration.md for the reserved keys and their replacements.

Spaces, hyphens and underscores are equivalent in key names. An unescaped # or ; starts a comment, even inside quotes, so a quoted value cut this way is rejected as ambiguous. Escape a literal marker as \# or \;. Validation rejects includes, continued lines, keys outside a section, duplicate keys in a section of one file, NUL bytes, invalid UTF-8 and individual files or merged results over 64 KiB.

func WithHostNetwork

func WithHostNetwork() Option

WithHostNetwork places every cluster daemon and WithClient container in the Docker daemon's host network. MON and RGW ports are selected automatically. The default advertised address is 127.0.0.1 for clients on that same host. Run fails with ErrHostNetworkUnavailable unless the engine runs Linux containers and the test process reaches the first MON's advertised endpoint. MON port reservation is retried up to five times when another process takes a port, and the reachability probe is retried for up to 5 seconds; later MON, MGR, OSD, MDS and RGW endpoints are not probed. MGR, OSD and MDS use Ceph's dynamic ports. Docker Desktop needs host networking enabled (4.34 or later), and a test process that itself runs in a bridge-network container cannot use host mode. It cannot be combined with NetworkConfig.WithSeparateClusterNetwork, and fault.InterruptNetwork and fault.ApplyNetworkProfile are unavailable in host mode.

func WithImageOptions added in v0.4.0

func WithImageOptions(config ImageOptions) Option

WithImageOptions is the grouped option function that carries an ImageOptions to Run. Like every Option it changes the cluster settings only and never reaches a container request.

Example

Role images are one ImageOptions value, passed to Run once through the grouped option function WithImageOptions. The image argument is empty because the set is the one image configuration. This example is compiled, so the snippet it shows cannot drift from the API; it is not run.

package main

import (
	"context"
	"log"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/testcontainers/testcontainers-go"
)

func main() {
	ctx := context.Background()
	images := ceph.ImageSet{Essential: "registry.example.com/ceph-core:20.2.4", Client: "registry.example.com/ceph-client:20.2.4"}
	cluster, err := ceph.Run(ctx, "", ceph.WithImageOptions(ceph.ImageOptions{}.WithImageSet(images)), ceph.WithClusterConfig(ceph.ClusterConfig{}.WithOSDCount(1)))
	defer func() {
		if err := testcontainers.TerminateContainer(cluster); err != nil {
			log.Printf("failed to terminate container: %s", err)
		}
	}()
	if err != nil {
		panic(err)
	}
}

func WithManagerPorts added in v0.5.0

func WithManagerPorts(ports ...int) Option

WithManagerPorts publishes the listed TCP ports of every MGR container, the initial ones and those of AddManager, on ports the Docker engine selects, so that a process on the Docker host reaches a MGR module listener such as the Dashboard's (dashboard.Dashboard.HostEndpoint). Without it a bridge-mode MGR publishes no port and only containers on the cluster network reach its listeners. Docker cannot publish a port of a created container, so the ports are fixed for the cluster's lifetime. Read a manager's mapped port with its handle's MappedPort; a stopped manager has none and a started one may get another. The ports must be the ones the module listens on in the container: WithManagerPorts does not configure a module.

A port outside 1 to 65535, a repeated port, an empty list, a second WithManagerPorts and the combination with WithHostNetwork, where the managers share the host's ports, are refused.

func WithMonitorStorage added in v0.4.0

func WithMonitorStorage(storage MonitorStorage) Option

WithMonitorStorage applies storage to every MON of the cluster.

func WithNetworkConfig added in v0.4.0

func WithNetworkConfig(network NetworkConfig) Option

WithNetworkConfig is the grouped option function that carries a NetworkConfig to Run. Like every Option it changes the cluster settings only and never reaches a container request.

Example

Network and Messenger settings are one NetworkConfig value, passed to Run once through the grouped option function WithNetworkConfig. Compiled, not run.

package main

import (
	"context"
	"log"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/testcontainers/testcontainers-go"
)

func main() {
	ctx := context.Background()
	network := ceph.NetworkConfig{}.WithAddressFamily(ceph.AddressDualStack).WithSeparateClusterNetwork()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithNetworkConfig(network), ceph.WithClusterConfig(ceph.ClusterConfig{}.WithOSDCount(1)))
	defer func() {
		if err := testcontainers.TerminateContainer(cluster); err != nil {
			log.Printf("failed to terminate container: %s", err)
		}
	}()
	if err != nil {
		panic(err)
	}
}

func WithOSDStorage added in v0.4.0

func WithOSDStorage(storage OSDStorage) Option

WithOSDStorage is the grouped option function that carries an OSDStorage to Run. Like every Option it changes the cluster settings only and never reaches a container request.

Example

OSD storage is one OSDStorage value, passed to Run once through the grouped option function WithOSDStorage. Compiled, not run.

package main

import (
	"context"
	"log"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/testcontainers/testcontainers-go"
)

func main() {
	ctx := context.Background()
	storage := ceph.OSDStorage{}.WithBlockSize(256 << 20).WithObjectStore(ceph.OSDMemStore)
	cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithOSDStorage(storage), ceph.WithClusterConfig(ceph.ClusterConfig{}.WithOSDCount(1)))
	defer func() {
		if err := testcontainers.TerminateContainer(cluster); err != nil {
			log.Printf("failed to terminate container: %s", err)
		}
	}()
	if err != nil {
		panic(err)
	}
}

func WithStartupTimeout

func WithStartupTimeout(timeout time.Duration) Option

WithStartupTimeout bounds bootstrap, readiness and topology operations. Run starts the timer after resolving and validating the options, including local file reads, so only the bootstrap that follows, up to the initial services, shares it. Each operation is also bounded by the caller's context. Default: 3 minutes.

It caps the context of Run's bootstrap and of each later topology, readiness and wait operation, such as AddOSD, WaitForClean, Pool.WaitForPGCount and DeepScrubPG. Waiting for the topology lock before the cap starts, and stopping an interrupted command, are outside it, so a call can return after the cap has passed. Increase it for slow engines, large pools or long recovery waits.

func WithZoneConfig added in v0.5.0

func WithZoneConfig(zones ZoneConfig) Option

WithZoneConfig is the grouped option function that carries a ZoneConfig to Run. Like every Option it changes the cluster settings only and never reaches a container request.

func (Option) Customize

Customize implements testcontainers.ContainerCustomizer.

type PGID added in v0.4.0

type PGID string

PGID identifies one placement group as "<pool>.<hex>": the decimal native pool ID, a dot and the lowercase hexadecimal PG number, for example "1.0" or "3.1f". Pool.PGs reports PGs with this type, so a reported PG passes straight to DeepScrubPG, RepairPG and PGInconsistencies. An untyped string constant converts to PGID without validation, so these methods check the grammar again and refuse an invalid value; ParsePGID checks it up front.

func ParsePGID added in v0.4.0

func ParsePGID(s string) (PGID, error)

ParsePGID returns s as a PGID when it matches the grammar "<pool>.<hex>", which is exactly what DeepScrubPG, RepairPG and PGInconsistencies accept. Otherwise it returns an error that matches ErrInvalidRequest and the zero PGID.

func (PGID) String added in v0.4.0

func (p PGID) String() string

String returns the native PG ID.

type PGPrimary added in v0.5.0

type PGPrimary struct {
	// PG is a PG of the pool, as Pool.PGs reports it.
	PG PGID
	// OSD is the owned OSD that is to serve the PG as its acting primary. It
	// must be up and a member of the PG's current acting set.
	OSD *OSDContainer
	// RaiseMinCompatClient lets the override raise the cluster's
	// require_min_compat_client to the release that the running MON names
	// as the minimum for pg-upmap-primary, when the cluster is below it.
	// Restore lowers it again once no override that relies on the raised
	// level remains. Without it, such a cluster is refused with
	// ErrInvalidRequest before any change.
	RaiseMinCompatClient bool
}

PGPrimary selects the acting primary that Pool.OverridePGPrimary gives one PG of the pool.

type PGPrimaryOverrideHandle added in v0.5.0

type PGPrimaryOverrideHandle struct {
	// contains filtered or unexported fields
}

PGPrimaryOverrideHandle owns one PG's pg-upmap-primary entry, and possibly a raised require_min_compat_client, for Pool.OverridePGPrimary. It retains the exact prior entry, including its absence, and possibly uncertain application: keep a non-nil handle returned with an error, because the change may have been applied. Copies share its state.

Restore puts back only what the handle introduced. It refuses an entry or a compatibility level that changed outside the override, rather than overwriting it. Ceph has no compare-and-swap for these values, so an outside write of the same value cannot be detected. Ceph itself cancels an entry whose OSD left the PG's raw placement or became its CRUSH primary; such a cancellation is an outside change when it differs from the prior entry. Terminate does not undo the override: it ends with the cluster.

func (*PGPrimaryOverrideHandle) Restore added in v0.5.0

func (h *PGPrimaryOverrideHandle) Restore(ctx context.Context) error

Restore puts back the PG's prior pg-upmap-primary entry: it removes an entry that did not exist, or sets the prior OSD again. When the override is the last holder of a raised require_min_compat_client, it then lowers the level to its prior release; another holder keeps it raised. It never runs rm-pg-upmap-primary-all and touches no other PG. It refuses an entry or a level that changed outside the override with ErrInvalidRequest and leaves it alone. A pool that was removed took its entries with it, which counts as restored.

Restore reads the stored OSDMap state back; it does not wait for the PG to be served by the prior primary, because a test may restore while that OSD is stopped. Use Observer.WaitForClean and Pool.PGs for that. It is idempotent across copies and reconciles a lost reply before any further change; retry with a fresh context. A terminated cluster refuses it with ErrInvalidRequest.

type PGState

type PGState struct {
	PGID                                        PGID
	State                                       string
	Up, Acting                                  []int
	UpPrimary, ActingPrimary                    int
	ReportedEpoch, MappingEpoch, LastEpochClean uint32
	ReportedSequence                            uint64
	StatsInvalid                                bool
	Stats                                       PGStats
}

PGState retains native reported placement and statistics. Up and Acting preserve order, empty vectors and repeated missing EC shard slots (CRUSH_ITEM_NONE, 2147483647). Primaries are explicit reported values, not inferred from the vectors; -1 means no primary. Epochs can be zero or older than the OSDMap brackets, and State retains unknown or future native states. A reported OSD may have been removed or be outside fixture ownership.

type PGStats

type PGStats struct {
	Objects, Bytes, ObjectCopies                      int64
	ObjectsDegraded, ObjectsMisplaced, ObjectsUnfound int64
}

PGStats contains signed native PG counters. Bytes counts logical payload, not physical disk allocation; ObjectCopies is separate from logical Objects. Reports arrive asynchronously, including when StatsInvalid is false, so these counters are not an I/O visibility or redistribution barrier.

type Pool

type Pool struct {
	PoolConfig
	CRUSHRule          string
	ErasureCodeProfile string
	// contains filtered or unexported fields
}

Pool identifies the pool, CRUSH rule and optional EC profile created for a fixture. PoolConfig contains the resolved defaults. Cluster termination disposes of its data together with the rest of the cluster.

The operations on a pool are methods of its handle. The handle identifies the native pool that it was created for or looked up as, by name and native pool ID, so a later pool that reuses the name is not reached through it. Copies of a Pool share that identity. A Pool that does not come from Container.CreatePool or Container.Pool has no cluster and its operations return an error.

func (*Pool) OverridePGPrimary added in v0.5.0

func (p *Pool) OverridePGPrimary(ctx context.Context, primary PGPrimary) (*PGPrimaryOverrideHandle, error)

OverridePGPrimary makes primary.OSD the acting primary of one PG of this replicated pool through the native pg-upmap-primary entry, so a test can aim a fault at, or route SUT I/O through, a known serving OSD. Replica placement is unchanged: only the order of the acting set changes.

The call validates the request and binds the original cluster FSID, the pool's native ID, the PG and the owned OSD's native UUID before any change. It refuses with ErrInvalidRequest: a nil or foreign OSD handle, an invalid or foreign PG, an erasure-coded pool, an OSD that is down or outside the PG's acting set, a PG with a primary_temp entry, a PG that another handle overrides, a terminated cluster, and a cluster whose require_min_compat_client is below what pg-upmap-primary needs unless primary.RaiseMinCompatClient is set. The needed release is the one that the running MON names in its refusal, not a release string of this module; the raise is refused natively, without --yes-i-really-mean-it, while connected clients look older. An OSD that already is the acting primary takes no native write: the handle then owns the unchanged entry.

It returns only when the condition is ready: the OSDMap holds the applied entry and maps primary.OSD first in the acting set, the serving PG answers pg query with primary.OSD as acting primary and active+clean, and the MGR PG report (Pool.PGs) agrees. That proves which OSD serves the PG at that moment. It does not prove that the SUT's client has received the new map (a client with an older map is redirected by the OSDs), that reads go to the primary only, or that the condition lasts: CRUSH, weight, primary affinity and balancer changes (in read or upmap-read mode the balancer writes these entries too) can move it again, and Ceph cancels the entry when its OSD leaves the PG's placement. It applies to replicated pools only and changes no balancer or other cluster policy.

A non-nil handle returned with an error owns a change that may have been applied; Restore it. Lock admission, every native command and the readiness wait are bounded by ctx and the cluster's startup timeout.

func (*Pool) PGs added in v0.4.0

func (p *Pool) PGs(ctx context.Context) (PoolPGSnapshot, error)

PGs observes the PGs that Ceph reports for this pool; PoolPGSnapshot describes the evidence. All failures, including post-query cancellation, return a zero snapshot and static errors that omit native output. There are no writes, command retries or workers (a MON connection that times out before a command is sent is reconnected, as in Container.Ceph). External native/configuration writers must not race these separate reads. Compare original fixture identities and data when asserting movement; these reported PGs are not current mappings or read-after-write evidence.

It works before any OSD exists.

func (*Pool) Remove added in v0.4.0

func (p *Pool) Remove(ctx context.Context) error

Remove deletes a pool created by CreatePool or WithPools, together with the CRUSH rule and erasure-code profile created for it, so a long-lived cluster can give each test its own pool without reaching PG limits. Pools that Ceph or a service created (.mgr, RGW zone pools, CephFS pools) and pools still used by a CephFS filesystem are refused. Every object in the pool is destroyed, including RBD images and namespaces; stop clients and mirroring that use it first. mon_allow_pool_delete is enabled in the central database only while the pool is removed and then restored; a local config file that sets it to false blocks removal. Ownership is recognized by the pool's tc-<name>-replicated or tc-<name>-ec CRUSH rule, so a pool that SetPlacement moved to another rule is refused until it moves back; placement rules are not removed. A retry after an error finishes removing whichever owned resources remain. External edits to the pool must not race this call.

After a complete removal the handle is stale and every operation through it returns an error, so a later pool that reuses the name is not affected.

func (*Pool) SetPGCount added in v0.4.0

func (p *Pool) SetPGCount(ctx context.Context, pgNum int) error

SetPGCount requests a new PG count for this pool with autoscaling off or warn. It records both native pg_num and pgp_num targets; the active manager then splits or merges PGs in steps, and merges wait for clean PGs. Success means the targets were accepted, not that PGs have changed. Use WaitForPGCount for the count and Observer.WaitForClean for the resulting movement. Without an active manager the targets remain pending. The native pool ID is checked before each mutation; on partial failure inspect Status. Native per-OSD PG limits can still reject the request.

func (*Pool) SetPlacement added in v0.4.0

func (p *Pool) SetPlacement(ctx context.Context, placement PoolPlacement) error

SetPlacement moves a replicated pool to a simple osd/host/rack/zone CRUSH rule. The size and min_size are unchanged and the new placement needs enough owned eligible domains in the native CRUSH map. It reuses the pool's CreatePool rule or an earlier rule from this method when its native steps match; otherwise it creates tc-<pool>-placement-<hash>. Rules are not deleted, so a pool can move back. Success means the pool references the rule; data moves asynchronously. Until PG reports reflect the new map, WaitForClean can pass on the previous placement, so poll PGs for the expected up/acting sets. Holding both nobackfill and norecover with override.SetOSDFlag keeps the move incomplete; nobackfill alone does not stop log-based recovery of small PGs. EC rules are bound to their profile and are refused. External pool/CRUSH edits must not race this call.

func (*Pool) SetQuota added in v0.4.0

func (p *Pool) SetQuota(ctx context.Context, quota PoolQuota) error

SetQuota replaces both limits, including zero values that remove a limit. Commands are sequential and may partially succeed; inspect Status after an error. Existing data is never removed. External pool edits must not race this operation; native pool IDs are checked before each mutation.

Quotas are enforced from asynchronously reported PG statistics, so writes can overshoot briefly; when a quota is reached Ceph marks the pool full and clients see writes blocked or EDQUOT, depending on the client.

func (*Pool) SetReplication added in v0.4.0

func (p *Pool) SetReplication(ctx context.Context, replicas, minSize int) error

SetReplication changes a replicated pool's size/min_size without changing data, PG count or placement. It requires enough owned placement domains in the native CRUSH rule. EC shard counts cannot be changed by this method. WaitForClean waits for resulting recovery. On partial failure, inspect Status; no rollback or data deletion is attempted. External pool/CRUSH edits must not race this call.

func (*Pool) Status added in v0.4.0

func (p *Pool) Status(ctx context.Context) (PoolState, error)

Status returns the native state of this pool. It fails when the pool does not exist or the name belongs to another pool now.

func (*Pool) Usage added in v0.4.0

func (p *Pool) Usage(ctx context.Context) (PoolUsageSnapshot, error)

Usage reports the pool's stored bytes and objects with the original bootstrap FSID and the exact native pool ID/name before and after the observation. A pool without a statistics row is unavailable, not empty. All failures, including cancellation after a query, return a zero snapshot. This method makes no writes, command retries or background workers (a MON connection that times out before a command is sent is reconnected, as in Container.Ceph). External pool or control configuration writers must not race it; the native reads are not an atomic transaction. Poll with a caller deadline for eventual assertions.

A pool without a statistics row is an error, not an empty result, and AllocatedBytes need not equal StoredBytes times the replica count.

func (*Pool) WaitForPGCount added in v0.4.0

func (p *Pool) WaitForPGCount(ctx context.Context, pgNum int) error

WaitForPGCount waits until the native pool reports pgNum as its current, pending and target PG and PGP counts and the manager reports exactly pgNum PGs for it. Until then Observer.WaitForClean can pass on the previous PG set. It fails if the name is bound to another pool than this handle identifies. The count does not imply clean PGs, completed backfill or data visibility; call WaitForClean afterwards or use PGs. Cluster startup timeout applies.

func (*Pool) WatchPGs added in v0.5.0

func (p *Pool) WatchPGs(ctx context.Context, cfg WatchConfig) (*PoolPGWatch, error)

WatchPGs polls the PGs that Ceph reports for this pool and reports their changes. The default WatchConfig.Interval is 5 seconds.

Before it returns, WatchPGs makes one read under ctx and QueryTimeout. A failed first read returns its error and starts nothing. That read is the baseline: the first event, already buffered, which fixes the watched identity, the original cluster FSID and the pool's native ID. PGs are not pinned: a split or merge of this pool changes the reported PG set, which is an event. Receive the baseline before triggering the change to observe.

Each later read is one Pool.PGs observation, which takes the topology lock for its five CLI calls and releases it before the event is sent and during the wait for the next read, so the consumer may call any fixture operation while the watch runs. A successful read whose compared fields equal the previous successful read produces no event. A failed read, a malformed report included, produces a WatchGap and the watch continues; the next successful read is WatchResumed. These end the watch instead: ctx ending, Stop, a terminated cluster, a native FSID of another cluster, a pool that this handle removed, that no longer has this name, or whose name now belongs to another native pool ID, and a full buffer (ErrWatchOverflow). The watch never follows a recreated pool by name and makes no change.

The reads of one sample are not atomic: the two OSDMap brackets can carry different epochs and policies, and the PG reports can lag either of them. A reported active+clean, up or acting state does not prove that a serving OSD applied a map, that a write is visible or that data is durable; check those with the SUT or the workload. The stream reports differences between samples, not a complete history of native transitions.

Example

The baseline is buffered before WatchPGs returns, so the OSD stops only after the watch has a known starting state. The first degraded report then advances the test while the SUT keeps writing; the SUT's own reads check the data. This example is compiled, not run.

package main

import (
	"context"
	"fmt"
	"strings"
	"time"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
)

func main() {
	ctx := context.Background()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithClusterConfig(ceph.ClusterConfig{}.WithOSDCount(3)))
	if cluster != nil {
		defer func() { _ = cluster.Terminate(context.Background()) }()
	}
	if err != nil {
		panic(err)
	}
	pool, err := cluster.CreatePool(ctx, ceph.PoolConfig{Name: "app", Application: "rados", Replicas: 2, MinSize: 1, PGNum: 8})
	if err != nil {
		panic(err)
	}
	osds, err := cluster.OSDs(ctx)
	if err != nil {
		panic(err)
	}
	watch, err := pool.WatchPGs(ctx, ceph.WatchConfig{Interval: 2 * time.Second})
	if err != nil {
		panic(err)
	}
	defer func() { _ = watch.Stop() }() // before the cluster's Terminate
	baseline := <-watch.Events()
	grace := 10 * time.Second
	if err := osds[0].Stop(ctx, &grace); err != nil {
		panic(err)
	}
	// ... the SUT writes and reads through the pool ...
	for event := range watch.Events() {
		if event.Kind == ceph.WatchGap {
			continue
		}
		for _, pg := range event.Snapshot.PGs {
			if strings.Contains(pg.State, "degraded") {
				fmt.Printf("pg %s reported %s at OSDMap epoch %d (baseline %d)\n",
					pg.PGID, pg.State, event.Snapshot.OSDMapEpochAfter, baseline.Snapshot.OSDMapEpochAfter)
				return
			}
		}
	}
	panic(watch.Stop()) // the watch ended before a degraded report
}

type PoolConfig

type PoolConfig struct {
	Name        string
	PGNum       int
	Replicas    int
	MinSize     int
	Application string
	// FailureDomain selects distinct osd, host, rack or zone buckets. Empty
	// uses osd.
	FailureDomain string
	// CRUSHRoot selects the hierarchy root; empty uses the cluster's default
	// root, which is default unless ClusterConfig.WithDefaultCRUSHRoot selects another.
	// DeviceClass optionally limits placement to OSDs explicitly configured
	// with that class.
	CRUSHRoot   string
	DeviceClass string
	ErasureCode *ErasureCodeConfig
	// NoHashPSPool clears the pool's hashpspool flag. Ceph then computes each
	// PG's placement seed with the legacy additive formula, the PG number plus
	// the pool ID, instead of hashing the PG number with the pool ID, as for
	// pools that predate the flag. It is set right after the pool is created,
	// before any data is written.
	NoHashPSPool bool
}

PoolConfig describes a pool in the disposable, file-backed cluster. Zero PGNum selects 8 PGs with autoscaling disabled. A nil ErasureCode creates a replicated pool, with the cluster's replica/min-size defaults when omitted (normally 2/1, or 1/1 for one initial OSD). Application can be rados, rbd, cephfs, rgw or a custom name; empty leaves it unassigned. Name uses letters, digits, '_', '.' and '-' and must not start with '.' or '-'. Replicas is 1 to 10 and must be zero for an erasure-coded pool. MinSize is 1 to Replicas for a replicated pool and K to K+M for an erasure-coded one. FailureDomain puts each replica or shard in a distinct osd, host, rack or zone bucket. For RBD pools that need rbd pool init, use rbd.WithPools or rbd.InitPool.

type PoolPGSnapshot

type PoolPGSnapshot struct {
	FSID                                string
	PoolBefore, PoolAfter               PoolState
	OSDMapEpochBefore, OSDMapEpochAfter uint32
	PGReady                             bool
	PGs                                 []PGState
}

PoolPGSnapshot contains one pool's reported PGs, bracketed by two native OSDMap observations. PoolBefore and PoolAfter reuse PoolState policy fields; their names and IDs agree, but policies and OSDMap epochs may differ. These reads are not an atomic placement snapshot. PGReady is the native MGR flag: false can accompany reports, and true can still accompany unknown PGs.

type PoolPGWatch added in v0.5.0

type PoolPGWatch struct {
	// contains filtered or unexported fields
}

PoolPGWatch is a running watch of one pool's reported PGs. One goroutine reads them and sends events; Events closes after that goroutine has stopped using the cluster. Stop ends the watch and waits for it.

func (*PoolPGWatch) Events added in v0.5.0

func (w *PoolPGWatch) Events() <-chan PoolPGWatchEvent

Events returns the receive-only event channel. The baseline is already in it when WatchPGs returns. The channel closes when the watch ends; Stop returns why.

func (*PoolPGWatch) Stop added in v0.5.0

func (w *PoolPGWatch) Stop() error

Stop cancels the watch if it is still running, waits until its goroutine has returned and Events is closed, and returns the cause that ended the watch: nil when Stop ended it, the context's error when the WatchPGs context ended, an ErrWatchOverflow error, or the read error that ended it (a terminated cluster, a foreign native FSID, or a pool that the handle no longer identifies). Buffered events stay readable after Stop. Stop is safe to call more than once and from any goroutine; every call returns the same cause. Call it before the fixture's cleanup: cancelling the context alone does not wait.

type PoolPGWatchEvent added in v0.5.0

type PoolPGWatchEvent struct {
	Sequence          uint64
	Kind              WatchEventKind
	Started, Finished time.Time
	Snapshot          PoolPGSnapshot
	Err               error
}

PoolPGWatchEvent is one event of a PoolPGWatch.

Sequence starts at 1 with the baseline and increases by one for each event, and no event is dropped, so consecutive numbers mean the consumer saw every event the watch produced. Started and Finished are the local clock around the read, not the time Ceph changed state.

Snapshot is the read as Pool.PGs returns it, with its PGs sorted by PGID (pool, then numeric seed) instead of in reported order. Its FSID and pool ID are those of the baseline, and its OSDMap epochs and per-PG reported epochs and sequences are the native values of that read. The watch compares only the pool policies of both brackets, PGReady, the set of PGIDs and, for each PG, State, Up, Acting (in order), UpPrimary and ActingPrimary. A read that differs only in epochs, reported sequences, StatsInvalid or Stats produces no event, so these fields are as fresh as the last event, not the last read. Each event has its own snapshot, which the consumer may keep or change. Two snapshots separated by a gap, or simply by the polling interval, do not establish the transitions in between. On a WatchGap, Snapshot is zero and Err is the failure.

type PoolPlacement

type PoolPlacement struct {
	FailureDomain string
	CRUSHRoot     string
	DeviceClass   string
}

PoolPlacement selects the CRUSH hierarchy of a replicated pool after creation. Its fields have PoolConfig's meaning and defaults: empty FailureDomain uses osd, empty CRUSHRoot uses the cluster's default root (default unless ClusterConfig.WithDefaultCRUSHRoot selects another) and empty DeviceClass leaves class unset.

type PoolQuota

type PoolQuota struct {
	MaxBytes   uint64
	MaxObjects uint64
}

PoolQuota limits logical bytes and object count. Zero removes that limit. Enforcement depends on asynchronously reported PG statistics, so the quota is not a strict per-write accounting boundary.

type PoolState

type PoolState struct {
	ID                   int64
	Name                 string
	Type                 string
	Size, MinSize, PGNum int
	PGNumTarget          int
	PGNumPending         int
	PGPlacementNum       int
	PGPlacementNumTarget int
	CRUSHRule            int
	AutoscaleMode        string
	ErasureCodeProfile   string
	Flags                string
	Quota                PoolQuota
}

PoolState is an authoritative snapshot of a native pool, including pools created by services or through Ceph. Size is replica count or EC shard count. It does not change when a policy is subsequently updated. PGNum is the current count; the target and pending fields show a split or merge still in progress and are zero when the native map omits them.

type PoolUsageSnapshot

type PoolUsageSnapshot struct {
	FSID, Name         string
	ID                 int64
	StoredBytes        uint64
	StoredDataBytes    uint64
	StoredOMAPBytes    uint64
	AllocatedBytes     uint64
	AllocatedDataBytes uint64
	AllocatedOMAPBytes uint64
	Objects            uint64
	MaxAvailableBytes  uint64
	UsedRatio          float64
}

PoolUsageSnapshot is one monitor-reported pool usage observation. StoredBytes estimates logical user data (DATA + OMAP); AllocatedBytes includes replication or EC and allocation overhead, but not the BlueStore database. All sizes are bytes, Objects is a logical count, and UsedRatio is a fraction in [0, 1]. MaxAvailableBytes estimates writable logical capacity. PG reports arrive asynchronously: this is neither a read-after-write barrier nor a quota check.

type QuorumStatus

type QuorumStatus struct {
	QuorumNames []string `json:"quorum_names"`
	MonMap      struct {
		FSID string `json:"fsid"`
		Mons []struct {
			Name        string `json:"name"`
			PublicAddrs struct {
				Addrvec []struct {
					Type string `json:"type"`
					Addr string `json:"addr"`
				} `json:"addrvec"`
			} `json:"public_addrs"`
		} `json:"mons"`
	} `json:"monmap"`
}

QuorumStatus contains the actual quorum and advertised monitor addresses.

type Status

type Status struct {
	FSID   string `json:"fsid"`
	Health struct {
		Status string `json:"status"`
	} `json:"health"`
	MgrMap struct {
		Available bool `json:"available"`
	} `json:"mgrmap"`
	OSDMap struct {
		NumOSDs   int `json:"num_osds"`
		NumUpOSDs int `json:"num_up_osds"`
		NumInOSDs int `json:"num_in_osds"`
	} `json:"osdmap"`
	PGMap struct {
		NumPGs     int `json:"num_pgs"`
		PGsByState []struct {
			StateName string `json:"state_name"`
			Count     int    `json:"count"`
		} `json:"pgs_by_state"`
	} `json:"pgmap"`
}

Status contains the subset of ceph status JSON used for readiness. Use Ceph for other commands or the full JSON response.

type WatchConfig added in v0.5.0

type WatchConfig struct {
	// Interval is the time between the starts of consecutive reads. Each
	// Watch documents its own default, chosen from the cost of its read.
	// Reads never overlap, and after a read the watch rests at least as long
	// as that read took, so it leaves the fixture's locks free at least half
	// the time: a read that takes longer than half the interval delays the
	// next one. A transition that is shorter than the interval can be missed.
	Interval time.Duration
	// QueryTimeout bounds one read, including its wait for fixture locks, 30
	// seconds by default. A read that exceeds it is a gap, not the end of the
	// watch.
	QueryTimeout time.Duration
	// Buffer is the capacity of the event channel, 16 by default.
	Buffer int
}

WatchConfig configures every Watch of this module: the Observer watches in this package and the watches of the service packages. A zero field takes its default; a negative field, an Interval below 100 milliseconds and a Buffer above 1024 are ErrInvalidRequest.

type WatchEventKind added in v0.5.0

type WatchEventKind string

WatchEventKind names what a watch event reports. Every Watch of this module uses the same kinds.

const (
	// WatchBaseline is the first event of every watch: the read that the
	// Watch call made before it returned, which fixes the watched identities.
	WatchBaseline WatchEventKind = WatchEventKind(watch.Baseline)
	// WatchChanged reports a successful read whose compared state differs
	// from the previous successful read.
	WatchChanged WatchEventKind = WatchEventKind(watch.Changed)
	// WatchGap reports one failed read. Err is the failure; the watch keeps
	// polling. A gap is never an empty or healthy observation.
	WatchGap WatchEventKind = WatchEventKind(watch.Gap)
	// WatchResumed reports the first successful read after one or more gaps,
	// whether or not the compared state changed across them.
	WatchResumed WatchEventKind = WatchEventKind(watch.Resumed)
)

The kinds of watch event.

type Zone added in v0.5.0

type Zone struct {
	Name string
	// Router forwards what leaves the zone.
	Router ZoneEndpoint
	// Members are sorted by name: mon.<name>, mgr.<name>, osd.<id>, control,
	// the service names of service daemons, and the container name of a
	// container placed with Container.WithZone.
	Members []ZoneEndpoint
	// ReporterSubtrees counts the distinct ReporterSubtree buckets (host by
	// default) of the zone's OSDs; zero when the level is not one of osd,
	// host, rack, zone or root.
	ReporterSubtrees int
}

Zone is one logical availability zone of a ZoneLayout.

type ZoneConfig added in v0.5.0

type ZoneConfig struct {
	// contains filtered or unexported fields
}

ZoneConfig is the logical availability zone group of the Ceph configuration: which zones one regional cluster spans, which zone holds the MON quorum, and the image of the boundary's routers and route agents. It is a plain value with chainable, value-returning methods, not a customizer; pass it to Run through its one grouped option function, WithZoneConfig. The zero value changes nothing, and each field records whether it was set.

zones := ceph.ZoneConfig{}.WithZones("a", "b").WithQuorumZone("a").WithHelperImage(fault.NetemHelperImage)
layout := ceph.ClusterConfig{}.WithInitialOSDs(ceph.OSDConfig{Zone: "a"}, ceph.OSDConfig{Zone: "b"})
cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithZoneConfig(zones), ceph.WithClusterConfig(layout))

A zone is at once a CRUSH bucket of type zone below the OSDs' root and a set of endpoints on the cluster's bridges. The cluster keeps its one public bridge (and its replication bridge with NetworkConfig.WithSeparateClusterNetwork), so every zone shares each plane's subnet and Ceph's public_network and cluster_network are unchanged. The boundary between the zones is owned by the cluster: one router per zone, which forwards the packets that leave its zone on every plane, and in the network namespace of every member one /32 route per address of every member of another zone, through the router of the member's own zone. Every cross-zone packet therefore crosses exactly one router, the one of the zone it leaves, and a packet within a zone goes directly. fault.ApplyZoneProfile shapes or cuts what one router forwards. See docs/guides/regional-zones.md for the membership rules, the route lifecycle and the limits of a single Docker engine.

All three settings are required together, and Run refuses zones with host networking, with an address family other than AddressIPv4, with OSDs that name no zone or an undeclared one, and with ClusterConfig.WithOSDCount (use WithInitialOSDs or WithNoInitialOSDs). Within one value a later call of the same method replaces the earlier one; Run refuses a field that two WithZoneConfig options both set.

func (ZoneConfig) WithHelperImage added in v0.5.0

func (z ZoneConfig) WithHelperImage(image string) ZoneConfig

WithHelperImage selects the image of the zone routers and of the route agents that join each member's network namespace with NET_ADMIN. It needs sleep, sh, cat, wc and iproute2's ip; fault.NetemHelperImage qualifies. The helper is never selected implicitly, and the daemon images stay unchanged.

func (ZoneConfig) WithQuorumZone added in v0.5.0

func (z ZoneConfig) WithQuorumZone(name string) ZoneConfig

WithQuorumZone selects the zone of every MON, and so of the quorum, of every MGR, of the control CLI and of the service daemons that are started without Container.WithZone. A partition then leaves the quorum on this side: the other zones' clients cannot authenticate, and their OSDs are marked down once enough of this zone's OSDs report them (see ZoneLayout.DownReporters).

func (ZoneConfig) WithZones added in v0.5.0

func (z ZoneConfig) WithZones(names ...string) ZoneConfig

WithZones declares the logical availability zones, at least two distinct CRUSH bucket names (letters, digits, underscores, dots or dashes). Each name becomes the CRUSH zone bucket of the OSDs that name it and the zone of the containers that Container.WithZone places there. The value keeps its own copy of names.

type ZoneEndpoint added in v0.5.0

type ZoneEndpoint struct {
	Name, ContainerID             string
	PublicAddress, ClusterAddress netip.Addr
}

ZoneEndpoint is a router or a member on the cluster's bridges. ClusterAddress is set only for an endpoint on a separate replication bridge: an OSD or a router.

type ZoneLayout added in v0.5.0

type ZoneLayout struct {
	// QuorumZone holds every MON, MGR and the control CLI.
	QuorumZone string
	// Zones are sorted by name.
	Zones []Zone
	// Routes lists the boundary's recorded /32 route intents, sorted by member
	// and destination. Records include installations with uncertain outcomes.
	// Inspect the member's route table or probe traffic for the effective path.
	Routes []ZoneRoute
	// DownReporters is mon_osd_min_down_reporters and ReporterSubtree is
	// mon_osd_reporter_subtree_level. Marking an OSD down through failure
	// reports requires DownReporters distinct ReporterSubtree buckets. A MON
	// can also mark an OSD down after mon_osd_report_timeout without a beacon.
	DownReporters   int
	ReporterSubtree string
	// Unzoned names the containers on the cluster's bridges that are neither
	// a router nor a member whose enrollment for its current start completed.
	// They reach every zone directly, or through the routes a partial
	// enrollment lacks; fault.ApplyZoneProfile refuses to shape while there
	// is one.
	Unzoned []string
}

ZoneLayout is the zone topology of a cluster from WithZoneConfig, as the fixture holds it: the zones with their routers and members, the routes the boundary recorded for cleanup, the MON quorum's zone, Ceph's down-reporter policy and the containers on the cluster's bridges that belong to no zone.

type ZoneRoute added in v0.5.0

type ZoneRoute struct {
	Member           string
	Plane            NetworkPlane
	Destination, Via netip.Addr
}

ZoneRoute records an intended route from Member to Destination on Plane through Via, the member's zone router. The boundary records the route before installation so it can clean up a partial enrollment. The effective route can differ after a failed installation or a change by the caller.

Source Files

Directories

Path Synopsis
Package diagnostics collects read-only evidence from a Ceph fixture for a test to save or report: a diagnostics report of the containers, networks, log tails and fixed Ceph queries of a cluster (Collect), the Ceph crash metadata in its daemon containers (CollectCrashMetadata), and the redaction of the secrets that the cluster knows (Redact).
Package diagnostics collects read-only evidence from a Ceph fixture for a test to save or report: a diagnostics report of the containers, networks, log tails and fixed Ceph queries of a cluster (Collect), the Ceph crash metadata in its daemon containers (CollectCrashMetadata), and the redaction of the secrets that the cluster knows (Redact).
Package expect evaluates captured observations of a ceph cluster against the conditions a test states, and explains each mismatch.
Package expect evaluates captured observations of a ceph cluster against the conditions a test states, and explains each mismatch.
Package finalize ends a test's dedicated Ceph cluster on one ordered cleanup path.
Package finalize ends a test's dedicated Ceph cluster on one ordered cleanup path.
internal
evidence
Package evidence holds what package ceph, the diagnostics collectors and the test finalization share about collecting evidence from a cluster: the inventory of the containers and secrets that a cluster tracks, the selection of the containers that one collection reads, and the redaction and bounding of collected text.
Package evidence holds what package ceph, the diagnostics collectors and the test finalization share about collecting evidence from a cluster: the inventory of the containers and secrets that a cluster tracks, the selection of the containers that one collection reads, and the redaction and bounding of collected text.
evidencehook
Package evidencehook gives the diagnostics and finalize packages the parts of a ceph.Container that are not public API: the inventory of the containers, networks and secrets that it tracks, the ID that labels its containers and the description of a failed native command.
Package evidencehook gives the diagnostics and finalize packages the parts of a ceph.Container that are not public API: the inventory of the containers, networks and secrets that it tracks, the ID that labels its containers and the description of a failed native command.
monstore
Package monstore keeps the cluster-owned MON data volumes of a Ceph fixture: one store record per MON (State and Volume), its state machine from creation to release, the generations that mount it (StartGeneration, StopGeneration, ReplaceGeneration) and the address it pins.
Package monstore keeps the cluster-owned MON data volumes of a Ceph fixture: one store record per MON (State and Volume), its state machine from creation to release, the generations that mount it (StartGeneration, StopGeneration, ReplaceGeneration) and the address it pins.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL