ceph

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 1, 2026 License: MIT Imports: 56 Imported by: 0

README

Ceph

The Testcontainers module for Ceph. It starts a small, disposable Ceph cluster for the integration tests of a service that uses Ceph.

What it is for

The module builds the Ceph side of an integration test, independently of the system under test and of the test code, and lets the test check afterwards that the cluster ended in the expected state. It reproduces as many operational conditions as it can, including some injected faults, but it is not a full Ceph deployment: large-scale experiments, performance measurement and tuning are out of scope, and it does not wrap object, image, file or S3 data APIs. Your service talks to the cluster with its own client (librados, go-ceph, librbd, libcephfs, the Ceph CLI or an S3 SDK). See API purpose and purpose tags.

Real ceph-mon, ceph-mgr and ceph-osd daemons run in separate containers on an isolated Docker network: one MON, one MGR and two OSDs by default, each OSD on a sparse BlueStore file inside its container. The daemons need no privileged mode, host disks, LVM, Docker socket or systemd; Testcontainers itself (and Ryuk) still needs the Docker engine. The Go module has no go-ceph, librados or cgo dependency and builds with CGO_ENABLED=0.

Requirements: Go 1.26 or later, a Docker engine that runs Linux containers, and access to pull the Ceph and Ryuk images. Releases follow semantic versioning; before v1.0.0 a minor release may change the API (see the published releases).

go get github.com/JSYoo5B/ceph-testcontainers-go/ceph

Usual order of use

  1. Start the cluster. ceph.Run(ctx, img, opts...) with ceph.DefaultImage (or role images through WithImageOptions) builds and waits for the whole topology. Settings are grouped values, each passed through one option: WithClusterConfig (MON/MGR/OSD counts, OSD layout, CRUSH root, pool defaults, initial pools), WithOSDStorage, WithMonitorStorage, WithNetworkConfig, WithImageOptions, plus WithAuth, WithConfigFile, WithHostNetwork and WithStartupTimeout. Contradictory options fail before anything is created. Register cleanup before checking the error: a partially started cluster is returned with the error and still owns containers.
  2. Prepare the state the test needs. WaitForClean before client I/O; CreatePool and the Pool handle (replication, quota, placement, PG count); CreateClient for a Cephx identity with exact caps; AddOSD, AddMonitor, AddManager and the daemon handles' Remove, Detach, Replace, SetIn for topology. The override package changes configuration with restoration handles, the fault package injects faults, and the rbd, cephfs and rgw packages add their services to the same Run.
  3. Hand the connection to the system under test. WithClient or ClientConfig.WithIdentity attach a client container to the cluster network with ceph.conf and a keyring; ConnectionConfig returns the same material for a process that connects itself (host mode, see Host networking). FactsOf reports the images, network and Ceph version.
  4. Observe and verify. Observe returns an Observer for health, PG and pool reports, quorum, OSD states, MGR modules and services, watchers and blocklist entries. Ceph and RunNativeCommand run bounded native commands when no typed observation exists.
  5. Finish. TerminateOnCleanup registers finalization: checks (with Finalization.Verify for crash and cluster-log expectations), an optional failure archive (CollectDiagnostics, CollectCrashMetadata), caller cleanups in reverse order, then Terminate, which removes every owned container, network and volume.
func TestRADOSRoundTrip(t *testing.T) {
	ctx := t.Context()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage,
		ceph.WithClusterConfig(ceph.ClusterConfig{}.WithPools(ceph.PoolConfig{Name: "app-data", Application: "rados"})),
	)
	testcontainers.CleanupContainer(t, cluster) // also owns a partial cluster
	if err != nil {
		t.Fatal(err)
	}
	if err := cluster.WaitForClean(ctx); err != nil { // Run does not wait for clean PGs
		t.Fatal(err)
	}
	// The client image must contain the tools the test uses. WithClient joins
	// the cluster network and copies ceph.conf and the admin keyring.
	client, err := testcontainers.Run(ctx, ceph.DefaultImage, cluster.WithClient(), ceph.WithIdleEntrypoint())
	testcontainers.CleanupContainer(t, client) // registered later, so removed first
	if err != nil {
		t.Fatal(err)
	}
	// ... run the system under test against the client, then check the result,
	// for example with ceph.Observe(cluster) or the client's own reads.
}

Starting a cluster takes tens of seconds and pulls a large image the first time; give go test a generous -timeout when several clusters run in one package.

Concepts and limits

  • One fixture per test. Call Run for each test and create the pools, namespaces and identities it needs. Stop clients the test attached before the cluster is terminated, and use a fresh bounded context in cleanups: t.Context() has ended by then. Keep non-nil handles returned with errors; an uncertain operation may still own a resource. See Contexts and cleanup.
  • Configuration composition. A grouped value records which fields were set; a later call in one value replaces an earlier one, and two carriers that set the same field are refused, even when they agree.
  • Images. One aggregate image supplies every role by default; an ImageSet selects essential, client, CephFS, RBD-aux and RGW images separately, and a missing optional role never falls back. See Releases and images.
  • Networks. Bridge mode by default: only MON a (and RGW gateways) publishes ports, so native RADOS, RBD and CephFS clients run in a WithClient container. Host mode lets a process on a Linux Docker host connect directly. IPv6, dual-stack and a separate cluster network are bridge-mode options.
  • Placement is logical. All daemons share one Docker engine; CRUSH hosts and racks test placement rules, not physical failure tolerance, and OSD files are sparse, so fullness is simulated with full ratios.
  • Fixture defaults differ from a stock cluster (small pools, PG autoscaling off, replica-1 pools allowed, faster MGR stats); the configuration guide lists them.
  • Errors carry categories (ErrInvalidRequest, ErrUnsupported, ErrImageCapability, ErrEnvironmentUnavailable, ErrNativeFailure) and OperationError / NativeFailure details; see Errors.

Reference

Documentation

Overview

Package ceph provisions disposable Ceph clusters using testcontainers-go. It controls Ceph through container-local CLIs without linking host librados.

Run starts the MON, MGR and OSD topology with cluster-wide options: daemon counts, OSD storage and placement, networks, pools and Messenger mode. Pool policies, Cephx identities, configuration overrides, fault injection and health observations operate on the returned Container.

Client services live in their own packages, which add their options to the same Run and declare their own service types:

  • github.com/JSYoo5B/ceph-testcontainers-go/rbd: RBD pools, namespaces and backup.
  • github.com/JSYoo5B/ceph-testcontainers-go/cephfs: filesystems, MDS topology and subvolumes.
  • github.com/JSYoo5B/ceph-testcontainers-go/rgw: gateways, users, placement and storage classes.

Each of these service packages has its own Run that starts a cluster with that service's defaults. Options from several of them can be passed to one ceph.Run to start a cluster that serves them together.

Fixtures that span two existing clusters take those clusters in their own configuration instead; their options are not ceph.Run options:

  • github.com/JSYoo5B/ceph-testcontainers-go/rbd/mirror: rbd-mirror replication.
  • github.com/JSYoo5B/ceph-testcontainers-go/cephfs/mirror: CephFS snapshot mirroring.
  • github.com/JSYoo5B/ceph-testcontainers-go/rgw/multisite: RGW multisite realms, zones and sync policies.

These rules hold across the methods of a Container and its handles:

  • Methods are bounded by the caller's context and most also by WithStartupTimeout. Topology changes are serialized, and waiting for a concurrent one honours the context. A call canceled while waiting returns without native changes; one canceled after it sent a native command can leave that command applied.
  • An operation on one daemon, pool or Cephx identity is a method of its handle (OSDContainer, DetachedOSD, MonitorContainer, ManagerContainer, Pool, ClientConfig) and acts on the instance the handle names. The handle of a removed, detached, replaced or deleted resource, or of a terminated cluster, returns an error. Functions that create handles and cluster-wide operations stay on Container.
  • A method that creates something can return a non-nil result with an error. The result identifies what was attempted and stays owned by the cluster; inspect it, retry, or terminate the cluster. No method deletes data to roll back a partial failure.
  • Read methods return the native state at the moment of the call; several reads are not one atomic snapshot, and health, PG statistics and usage are reported asynchronously, so poll with a deadline. Several observations verify the cluster's original FSID and pool IDs.
  • Ceph has no compare-and-swap for configuration, flags, CRUSH or auth, so do not change the same state with raw commands while a helper applies or restores it.
  • Temporary changes (package override, package fault and ClientConfig.Revoke) return restoration handles, except the one-shot fault.InjectObjectDataError; keep a non-nil handle even with an error. Their shared contract is in docs/guides/contexts-and-cleanup.md.

This is an experimental module for application integration tests.

Index

Examples

Constants

View Source 
const (
	MaxNativeCommandTimeout = time.Hour
	MaxNativeCommandOutput  = 64 << 20
	MaxNativeCommandInput   = 64 << 20
	// NativeCommandStopTimeout bounds one attempt to stop a command inside its
	// container: the canceler and, when it fails, one inspection of the
	// container. The attempt has this budget of its own, independent of the
	// caller's context, which has usually ended.
	NativeCommandStopTimeout = supervised.CancelTimeout + nativeInspectTimeout
)

Limits of a NativeCommand.

View Source 
const CrashDir = "/var/lib/ceph/crash"

CrashDir is the crash directory of every daemon of the cluster. The bootstrap ceph.conf sets it, and WithConfigFile refuses to change it.

View Source 
const DefaultImage = "ghcr.io/jsyoo5b/ceph-testcontainers-images@sha256:2e57651a5aa7a6791b2b6c67d63439a307e50a0f560f4c98323e2a17a7a9ab4b"

DefaultImage pins the published, pruned dependency-role aggregate on both native architectures. Explicit ImageSet selections avoid unused feature roles.

View Source 
const MaxNativeFailureReason = 512

MaxNativeFailureReason bounds NativeFailure.Reason.

Variables

View Source 
var (
	// ErrInvalidRequest reports a request that the fixture refused before
	// any change: malformed arguments, or an operation that the state of the
	// cluster or of the selected resource forbids, such as a terminated
	// cluster, a container that the cluster neither owns nor has attached,
	// or an override that is already held.
	ErrInvalidRequest = errors.New("invalid request")
	// ErrUnsupported reports a native feature that the Ceph of the relevant
	// target lacks, established before any change from that target's own
	// capability, such as an option that its tool does not list, or from a
	// documented limit of the release it runs. AddressDualStack establishes
	// it after startup instead, from the addresses that the daemons
	// advertise. *OperationError names the feature, the target and its
	// version.
	ErrUnsupported = errors.New("unsupported by the target's Ceph")
	// ErrImageCapability reports an image that lacks what the operation
	// needs: an executable or runtime, such as the python3 of the
	// supervisor, or an optional extension that was not selected or does not
	// meet its contract.
	ErrImageCapability = errors.New("image capability unavailable")
	// ErrEnvironmentUnavailable reports that the Docker engine, the host, the
	// kernel or the permissions of a container cannot provide what the
	// operation needs, although the image may be valid. When it comes from a
	// call that had sent a native command, the command may have run.
	ErrEnvironmentUnavailable = errors.New("runtime environment unavailable")
	// ErrNativeFailure reports a native command that ran and failed: Ceph
	// refused the request, a resource was missing, permission was denied or
	// I/O failed. *NativeFailure carries the native status.
	ErrNativeFailure = errors.New("native operation failed")
)

Error categories. A classified error matches one of them with errors.Is, and errors.As with *OperationError or *NativeFailure returns its details. An error joined from several failures, such as a failed change and a failed restoration, can match several. Context errors, ErrNativeCommandIncomplete and *NativeCommandUnstoppedError keep their own contracts and belong to no category: a call that ctx ended is never classified by what it could not finish. An error that matches no category is a failed check of the fixture's own, such as output it could not parse, or a failure whose outcome is unknown, such as a container that disappeared after a native command was sent to it. Only ErrInvalidRequest promises that nothing changed.

View Source 
var ErrHostNetworkUnavailable error = &categoryError{text: "ceph host network is unavailable to the test process", kind: ErrEnvironmentUnavailable}

ErrHostNetworkUnavailable reports that host mode cannot serve native clients in the test process: the engine does not run Linux containers, the selected address is not assigned on the Docker host, or the first MON's advertised endpoint is not reachable from this process. Run returns it before creating MGR/OSD daemons; later MON/RGW port reservations may also return it. It belongs to ErrEnvironmentUnavailable.

View Source 
var ErrNativeCommandIncomplete = errors.New("native command did not complete within its limits")

ErrNativeCommandIncomplete marks a NativeCommandResult that timed out or whose output was truncated: its bytes are not the command's complete output.

View Source 
var ErrNativeInputClosed = errors.New("native command input is closed")

ErrNativeInputClosed reports a write to a NativeInput that was closed, or whose command no longer takes input.

Functions

func WithIdleEntrypoint 

func WithIdleEntrypoint() testcontainers.CustomizeRequestOption

WithIdleEntrypoint keeps a client or helper container running with sleep infinity under Docker's init process, so commands can be executed in it later. As PID 1, sleep ignores SIGTERM and Docker waits for the full stop timeout (10 seconds by default) before killing it; init forwards the signal so termination finishes at once. Init also reaps background processes that exit inside the container, such as a test server started through Exec. The entrypoint and command remain sleep and infinity.

Types

type AddressFamily added in v0.2.0 

type AddressFamily int

AddressFamily selects the addresses that the Messenger of every daemon binds and advertises. 

const (
	// AddressIPv4 binds and advertises the containers' IPv4 addresses.
	AddressIPv4 AddressFamily = iota
	// AddressIPv6 runs the cluster on a dual-stack bridge and binds and
	// advertises only the containers' IPv6 addresses.
	AddressIPv6
	// AddressIPv4MappedIPv6 keeps an IPv4 bridge and binds and advertises the
	// containers' IPv4 addresses as IPv4-mapped IPv6 addresses
	// (::ffff:a.b.c.d) on IPv6 sockets.
	AddressIPv4MappedIPv6
	// AddressDualStack runs the cluster on a dual-stack bridge, and every
	// Messenger binds and advertises both the container's IPv4 and its IPv6
	// address, so that clients of either family can connect. MON address
	// vectors list the IPv6 address first; see NetworkConfig.WithAddressFamily.
	AddressDualStack
)

func (AddressFamily) String added in v0.2.0 

func (family AddressFamily) String() string

type AuthConfig added in v0.2.0 

type AuthConfig struct {
	// KeyType is the type of the keys the fixture generates itself: the
	// mon., client.admin and OSD keys and, unless ClientOptions.WithKeyType selects
	// another, CreateClient keys.
	KeyType CephxCipher
	// ServiceCipher encrypts service tickets. AllowedCiphers are the key
	// types that the MONs accept for authentication, and PreferredCipher is
	// the type the MONs use for keys they generate with auth get-or-create,
	// such as MGR, MDS and RGW keys. These MonMap settings need a KeyType and,
	// for anything but CipherAES, the image support that CipherAES256K
	// describes. ServiceCipher and PreferredCipher
	// default to KeyType, and AllowedCiphers to the distinct types among
	// KeyType, ServiceCipher and PreferredCipher. AllowedCiphers must contain
	// all three.
	ServiceCipher   CephxCipher
	AllowedCiphers  []CephxCipher
	PreferredCipher CephxCipher
	// MonTicketTTL and ServiceTicketTTL set auth_mon_ticket_ttl and
	// auth_service_ticket_ttl, in seconds with fractions kept, such as
	// 10500 * time.Millisecond. MONs rotate service secrets on their ticks, so
	// lifetimes of a few seconds also need a shorter mon_tick_interval, which
	// WithConfigFile can set. Keep MonTicketTTL at ten seconds or more: MON
	// clients renew their tickets only every few seconds, and a daemon whose
	// renewal comes too late is refused and does not authenticate again.
	MonTicketTTL     time.Duration
	ServiceTicketTTL time.Duration
}

AuthConfig selects the Cephx settings of a new cluster. Zero fields keep the defaults of the image's Ceph release.

type BlocklistEntry 

type BlocklistEntry struct {
	Address string
	Range   bool
	Until   time.Time
}

BlocklistEntry reports native OSDMap fencing, including ranges created by external tools. Until is the MON's absolute expiration time. A listing grants no ownership to remove an existing entry.

type CaptureFileSource added in v0.4.0 

type CaptureFileSource interface {
	CaptureSettler
	CaptureFile(ctx context.Context) (io.ReadCloser, int64, string, error)
}

CaptureFileSource is a settled capture with a read-only local file. CaptureFile returns a reader, the complete retrieved byte count and its SHA256. The descriptor must remain the settlement's frozen identity even if the file is later edited. It must not start or settle a capture, change or delete the caller's file, or expose a host path in errors. Opening and reading must honor ctx; Finalization waits for operations to return rather than abandoning them.

type CaptureSettler added in v0.4.0 

type CaptureSettler interface {
	// SettlementAllowance is the longest Settle runs, independent of the
	// context it is given. The finalization admits exactly this much for the
	// capture; it is not taken from the checks' budget.
	SettlementAllowance() time.Duration
	// Settle ends the capture, delivers its file and removes its helper. It
	// must not abandon work that is still running when its context ends, and
	// it returns the report, which must be JSON-marshalable and hold no host
	// path, also with an error. Settling twice returns the first report.
	Settle(ctx context.Context) (report any, err error)
}

CaptureSettler is a running packet capture that a Finalization settles. The capture package's Session implements it; the interface lives here because that package imports this one.

type CephxCipher added in v0.2.0 

type CephxCipher string

CephxCipher names a Cephx key type, which is also the cipher of the tickets made with such keys. A decoded key records its type in its first two bytes, little-endian: 1 for aes and 2 for aes256k. 

const (
	// CipherAES is AES-128-CBC, the only key type of Squid and of Tentacle
	// before 20.2.4.
	CipherAES CephxCipher = "aes"
	// CipherAES256K is AES-256 with Kerberos-style key derivation
	// (AES256KRB5). It needs an image whose ceph-authtool has --key-type and
	// whose monmaptool has the cipher options, as in Ceph 20.2.4 and later,
	// where it is the default.
	CipherAES256K CephxCipher = "aes256k"
)

type ClientCaps 

type ClientCaps struct {
	Mon string
	OSD string
	MGR string
	MDS string
}

ClientCaps contains Cephx capability expressions, passed as individual CLI arguments. For example, Mon: "allow r" and OSD: "allow rw pool=data namespace=tenant-a" restrict RADOS I/O to one pool and namespace. Empty fields grant no capabilities for that service. Capabilities are never broadened by this helper; Ceph validates their syntax.

type ClientConfig 

type ClientConfig struct {
	// contains filtered or unexported fields
}

ClientConfig describes a client identity created by this cluster. Credentials remain private; ConnectionConfig returns copies for an external native client. The cluster owns the auth database, while callers own attached containers.

func (*ClientConfig) Capabilities added in v0.4.0 

func (client *ClientConfig) Capabilities(ctx context.Context) (ClientCaps, error)

Capabilities returns the current caps of this confirmed, owned identity. It verifies the native key without returning auth database secrets.

func (*ClientConfig) ConnectionConfig 

func (client *ClientConfig) ConnectionConfig() (ConnectionConfig, error)

ConnectionConfig returns independent ceph.conf and keyring copies. The config is a snapshot at creation; ClientConfig.WithIdentity uses the current MON addresses. Revocation does not erase already copied credentials or invalidate established sessions immediately. Test revoked credentials with a fresh connection. On error it returns the zero ConnectionConfig.

func (*ClientConfig) Delete added in v0.4.0 

func (client *ClientConfig) Delete(ctx context.Context) error

Delete revokes this identity, which the cluster created, without deleting data. It checks for an externally replaced key before deleting; external auth edits must not race this operation. A repeat call after success is a no-op; existing sessions may retain issued tickets. The removal is fenced like the import of ReplaceKey: a failed call reports what the MONs hold once settled, and an uncertain one is resolved by the next operation on the identity.

func (ClientConfig) GoString 

func (client ClientConfig) GoString() string

GoString excludes credentials from formatted Go representations as well.

func (*ClientConfig) KeyringPath 

func (client *ClientConfig) KeyringPath() string

KeyringPath returns the per-identity path used in an attached client container.

func (*ClientConfig) Name 

func (client *ClientConfig) Name() string

Name returns the fully qualified Cephx entity, for example client.reader.

func (*ClientConfig) ReplaceKey added in v0.4.0 

func (client *ClientConfig) ReplaceKey(ctx context.Context, opts ...ClientOptions) (*ClientConfig, error)

ReplaceKey gives this client identity, which the cluster created, a fresh Cephx key and keeps its name and its exact capabilities, in one auth import, so that the MONs never lack the identity. The identity needs a capability, as auth import refuses an entry without any. The new key has the type of the old one unless ClientOptions.WithKeyType selects another. It returns a new descriptor for the new key, and the old descriptor goes stale: its ConnectionConfig still returns the old credentials, for tests that check their rejection, while every other operation on it fails. Containers that were attached with the old descriptor keep the old keyring.

A fresh connection with the old key fails at once. A client that authenticated before keeps its session and its tickets until it renews them within auth_mon_ticket_ttl (AuthConfig.MonTicketTTL), when the MONs reject the old key: Tentacle then drops the client's MON session, while Squid keeps an open one answering. The new key works for fresh MON and MGR clients at once.

The identity must still have the key of client; one changed outside the module is refused. External auth changes must not race this call. The import is fenced like the MON commands of RotateKey: when it fails, the call settles the MONs and reads whether it applied. When that stays uncertain, the call returns a descriptor of the attempted key that stays unusable, with the error, and a later call with client, or any other operation on the identity, settles first: ReplaceKey then returns a usable descriptor of that key once the import turns out to have applied, or replaces the key anew once it turns out not to have. A returned descriptor never changes afterwards.

func (*ClientConfig) Revoke added in v0.4.0 

func (client *ClientConfig) Revoke(ctx context.Context) (*ClientRevocationHandle, error)

Revoke removes this client identity, which the cluster created, from the MONs until Restore puts back exactly its key and capabilities, so that a test can observe its rejection and then the recovery of the same credentials. The identity must still have the key of client; one changed outside the module is refused. While it is revoked, the descriptor works only for ConnectionConfig, whose credentials stay valid for after Restore; ClientConfig.WithIdentity, Capabilities, UpdateCaps and ReplaceKey fail, CreateClient refuses its name, and Delete makes the removal permanent.

A fresh connection with the client's key fails at once. A client that authenticated before keeps its sessions until it renews its tickets, within auth_mon_ticket_ttl (AuthConfig.MonTicketTTL), when the MONs reject the renewal; it keeps the MGR and OSD sessions it already holds. Tentacle also drops its MON session, and a librados client then does not authenticate again on its own, while Squid keeps an open MON session answering. A new connection with the same credentials works once Restore returns.

The removal and the restoration are fenced like the import of ReplaceKey. A failed call settles the MONs and reports what they hold. When that stays uncertain, Revoke returns the revocation with the error, and Restore settles first and then restores. The cluster needs no Restore before Terminate.

func (ClientConfig) String 

func (client ClientConfig) String() string

String deliberately excludes credentials, including when a descriptor is logged.

func (*ClientConfig) UpdateCaps added in v0.4.0 

func (client *ClientConfig) UpdateCaps(ctx context.Context, caps ClientCaps) error

UpdateCaps replaces all capabilities while retaining the identity/key. Omitted services lose their previous capabilities. External auth edits must not race this call. Already issued session tickets may retain previous caps; verify permission changes using a fresh client connection. The change is fenced like the import of ReplaceKey: a failed call reports what the MONs hold once settled, and an uncertain one is resolved by the next operation on the identity.

func (*ClientConfig) User 

func (client *ClientConfig) User() string

User returns the ID without the client. prefix, for APIs such as librados rados_create and go-ceph rados.NewConnWithUser.

func (*ClientConfig) WithIdentity added in v0.4.0 

func (client *ClientConfig) WithIdentity(opts ...ClientContainerOptions) testcontainers.CustomizeRequestOption

WithIdentity attaches a container with only this client's keyring. It rejects a descriptor whose creation is incomplete, that is revoked or that ReplaceKey made stale. Caller owns the attached container and must terminate it before the cluster. The options customize the container's ceph.conf.

The container gets /etc/ceph/ceph.conf, with the current MON addresses and a [client.<id>] section that points to the keyring, and the keyring at KeyringPath. The client must select its identity, for example rados --id reader, CEPH_ARGS="--id reader", librados rados_create(&cluster, "reader") or go-ceph rados.NewConnWithUser("reader"). It also refuses an identity that Revoke removed until its Restore, or that has an unresolved change.

type ClientContainerOptions added in v0.4.0 

type ClientContainerOptions struct {
	// contains filtered or unexported fields
}

ClientContainerOptions is the configuration that WithClient and ClientConfig.WithIdentity write into a client container. It is a plain value with chainable, value-returning methods; the zero value changes nothing. Within one value a later call of the same method replaces the earlier one. A field set by two values is refused with ErrInvalidRequest, even when they agree. 

cluster.WithClient(ceph.ClientContainerOptions{}.WithMessengerPolicy(policy))

func (ClientContainerOptions) WithMessengerPolicy added in v0.4.0 

func (o ClientContainerOptions) WithMessengerPolicy(policy ClientMessengerPolicy) ClientContainerOptions

WithMessengerPolicy writes the policy into the [client] section of the client container's ceph.conf. It applies only to the Ceph clients in that container, overrides the central configuration and MessengerV2Secure's global modes, and leaves the cluster's own control CLI alone. The value keeps its own copy of the lists; a nil list keeps Ceph's default and a non-nil empty list is refused.

type ClientMessengerPolicy added in v0.2.0 

type ClientMessengerPolicy struct {
	// MonitorsAndManagers sets ms_mon_client_mode, which Ceph uses for
	// connections to MONs and MGRs.
	MonitorsAndManagers []MessengerConnectionMode
	// OtherDaemons sets ms_client_mode, which Ceph uses for connections to
	// the other daemons, such as OSDs.
	OtherDaemons []MessengerConnectionMode
}

ClientMessengerPolicy selects the Messenger v2 connection modes that a native client offers, in order of preference. A nil field keeps Ceph's default, which offers both modes.

type ClientOptions added in v0.4.0 

type ClientOptions struct {
	// contains filtered or unexported fields
}

ClientOptions is the configuration of the key that CreateClient and ClientConfig.ReplaceKey generate. It is a plain value with chainable, value-returning methods; the zero value changes nothing. Within one value a later call of the same method replaces the earlier one. A field set by two values is refused with ErrInvalidRequest, even when they agree. 

client, err := cluster.CreateClient(ctx, "app", caps, ceph.ClientOptions{}.WithKeyType(ceph.CipherAES256K))

func (ClientOptions) WithKeyType added in v0.4.0 

func (o ClientOptions) WithKeyType(cipher CephxCipher) ClientOptions

WithKeyType selects the key type of the new key instead of the cluster's AuthConfig.KeyType or the release default for CreateClient, or the type of the current key for ReplaceKey. The MONs must allow the type for the client to authenticate. CipherAES256K needs an image with key type support (see CipherAES256K); without it the operation fails before any change. An unknown cipher, including the empty one, is refused.

type ClientRevocationHandle added in v0.4.0 

type ClientRevocationHandle struct {
	// contains filtered or unexported fields
}

ClientRevocationHandle manages removal and restoration of a client identity that this cluster created. It retains ownership and possibly uncertain apply state; a non-nil handle does not guarantee a successful removal. Restore puts back the same key and capabilities. It holds the original credentials privately and never prints them. Copies share its state.

func (ClientRevocationHandle) GoString added in v0.4.0 

func (r ClientRevocationHandle) GoString() string

GoString excludes credentials from formatted Go representations as well.

func (*ClientRevocationHandle) Name added in v0.4.0 

func (r *ClientRevocationHandle) Name() string

Name returns the revoked entity, for example client.reader.

func (*ClientRevocationHandle) Restore added in v0.4.0 

func (r *ClientRevocationHandle) Restore(ctx context.Context) error

Restore puts back the identity's original key and capabilities and makes its descriptor usable again. It succeeds again once restored, as long as the identity still holds them. It refuses an identity that was created again with another key or other capabilities, and one that ClientConfig.Delete removed for good. The restoration is fenced like the removal: when its outcome stays uncertain, the next Restore settles first.

func (ClientRevocationHandle) String added in v0.4.0 

func (r ClientRevocationHandle) String() string

String deliberately excludes credentials.

type ClusterConfig added in v0.4.0 

type ClusterConfig struct {
	// contains filtered or unexported fields
}

ClusterConfig is the topology and placement group of the Ceph configuration: how many MONs, MGRs and OSDs Run starts, where the initial OSDs sit in the CRUSH map, the default CRUSH root and pool size defaults, and the pools Run creates after the initial OSDs. It is a plain value with chainable, value-returning methods, not a customizer; pass it to Run through its one grouped option function, WithClusterConfig. The zero value changes nothing, and each field records whether it was set, so an explicit setting is different from an unspecified one. 

cluster := ceph.ClusterConfig{}.WithMonitorCount(3).WithOSDCount(3).WithPoolDefaults(3, 2)
c, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithClusterConfig(cluster))

The managers (WithManagerCount, WithNoInitialManagers) are one field, and so are the initial OSDs (WithOSDCount, WithInitialOSDs, WithNoInitialOSDs). Within one value a later call of the same field replaces the earlier one. Run refuses a field that two WithClusterConfig options both set, even when they agree, so argument order never decides a conflict. Incompatible combinations (no initial OSDs with initial pools or services, no initial managers with initial CephFS or RGW services, a layout that cannot hold the pools) are refused before any resource is created. The composition rules are in docs/contributing/configuration-composition.md.

func (ClusterConfig) WithDefaultCRUSHRoot added in v0.4.0 

func (c ClusterConfig) WithDefaultCRUSHRoot(root string) ClusterConfig

WithDefaultCRUSHRoot selects placement for pools that Ceph creates itself, including .mgr, and for omitted PoolConfig.CRUSHRoot/OSDConfig.Root fields. The initial layout must contain enough OSDs in this root for the default replica count. Without this setting the root is default.

func (ClusterConfig) WithInitialOSDs added in v0.4.0 

func (c ClusterConfig) WithInitialOSDs(configs ...OSDConfig) ClusterConfig

WithInitialOSDs selects the count and logical CRUSH locations of the initial OSDs, one OSDConfig per OSD. An empty layout is refused, and so is a host that appears under different racks or roots. The value keeps its own copy of configs. Pools that need N distinct failure domains are validated against the layout before anything is created.

Run registers the initial OSDs one after another in layout order, so IDs, keys and CRUSH host buckets are allocated in that order on every run, and every host bucket exists before any OSD boots. It then starts up to four OSD containers at a time and waits until all are up and in. If an OSD fails to start, no further OSD is started and the error names it; every registered OSD and created container stays owned by the returned cluster. AddOSD adds one OSD at a time.

func (ClusterConfig) WithManagerCount added in v0.4.0 

func (c ClusterConfig) WithManagerCount(count int) ClusterConfig

WithManagerCount starts one active manager and the remaining standbys (default: one manager). A count below one is refused; WithNoInitialManagers defers the first manager instead.

func (ClusterConfig) WithMonitorCount added in v0.4.0 

func (c ClusterConfig) WithMonitorCount(count int) ClusterConfig

WithMonitorCount selects the initial monitor count (default: 1). Three enables quorum failover tests; even counts are allowed for deliberate quorum-loss scenarios. A count below one is refused.

func (ClusterConfig) WithNoInitialManagers added in v0.4.0 

func (c ClusterConfig) WithNoInitialManagers() ClusterConfig

WithNoInitialManagers defers the first MGR until an explicit AddManager. Run waits for MON quorum and requested OSDs up/in, without implying PG clean or MGR module readiness. Initial CephFS/RGW services are refused before allocation; positive-OSD initial pools are allowed. The essential image must still contain the MGR and its required dependencies. WaitForClean and WaitForPGClean need an available manager, so they cannot succeed before one is added, and Ceph may report MGR_DOWN (after mon_mgr_mkfs_grace) or HEALTH_ERR while client data I/O still works. To simulate a manager outage in a configured cluster, stop the existing manager container instead.

func (ClusterConfig) WithNoInitialOSDs added in v0.4.0 

func (c ClusterConfig) WithNoInitialOSDs() ClusterConfig

WithNoInitialOSDs starts only the selected MON/MGR topology. It keeps the ordinary prospective pool defaults (size 2, min_size 1) for later AddOSD calls; WithPoolDefaults may override them. Initial pools and CephFS/RGW services are refused before allocation. No storage readiness is implied: native internal pools may have pending PGs.

func (ClusterConfig) WithOSDCount added in v0.4.0 

func (c ClusterConfig) WithOSDCount(count int) ClusterConfig

WithOSDCount sets the initial number of OSD containers at default locations (default: 2). A count below one is refused; WithNoInitialOSDs starts none. Each OSD gets a logical host osd-ID under the default root. Without WithPoolDefaults the cluster's default pool size is min(2, count) with min_size 1, so a one-OSD cluster gets replica-1 pools that can become clean.

func (ClusterConfig) WithPoolDefaults added in v0.4.0 

func (c ClusterConfig) WithPoolDefaults(replicas, minSize int) ClusterConfig

WithPoolDefaults selects size/min_size for pools created by Ceph or callers. Without this setting the fast fixture uses size=min(2, OSD count), min_size=1 (2/1 with WithNoInitialOSDs). It requires 1 <= minSize <= replicas <= 10 and sets osd_pool_default_size and osd_pool_default_min_size, which WithConfigFile therefore rejects. CreatePool can override these defaults for each explicitly configured pool.

func (ClusterConfig) WithPools added in v0.4.0 

func (c ClusterConfig) WithPools(configs ...PoolConfig) ClusterConfig

WithPools creates the selected pools after initial OSD startup. Configs and nested EC settings are copied; cluster defaults are resolved after all Run options have been applied.

type ClusterFacts added in v0.4.0 

type ClusterFacts struct {
	AddressFamily             AddressFamily
	Images                    ImageSet
	MessengerMode             MessengerMode
	NetworkName               string
	UsesHostNetwork           bool
	PublicAddress             string
	ClusterNetworkName        string
	HasSeparateClusterNetwork bool
	CephVersion               string
}

ClusterFacts is an independent snapshot of the nine static cluster facts. It holds only values: no Container, Docker handle, context, lock, pointer, func or channel, so nothing in it can control the cluster or reactivate it, and it owns no resource or cleanup lifetime. Later changes to the cluster do not update an obtained snapshot, and changing a snapshot does not change the cluster or another snapshot.

The values describe the recorded configuration and identity of the cluster. They do not show that its resources still exist, are reachable or run, and they are not a native observation. Images and CephVersion describe the cluster's own settings and first MON; a daemon started with DaemonOptions.WithImage is not reflected (see the Image and Version methods of its handle).

func FactsOf added in v0.4.0 

func FactsOf(c *Container) (ClusterFacts, error)

FactsOf returns an independent snapshot of the static facts of a non-nil cluster, including a terminated one. A nil cluster returns the zero ClusterFacts and an error that matches ErrInvalidRequest.

It takes no lock, runs no command and makes no Docker call, and it never reads the closed flag: closing starts before teardown finishes, so the flag is no evidence about the state of the owned resources, and termination alone must not fail the capture. The recorded settings, version and network records are written before the cluster is returned to the caller and termination does not clear them, so reading them needs no synchronization with a concurrent Terminate. Retained and read are the run settings, the public and replication network records and the Ceph version recorded at startup. A fact never recorded, as in a cluster returned by Run with an error, stays empty and is not replaced by a default. TestClusterFactsRecordedSourcesAreNotRewrittenOutsideStartup guards that no later code path writes these sources.

The facts are written by Run before it returns and are never cleared by Terminate, so FactsOf works on a terminated cluster or while Terminate runs, without a lock, command or Docker call. A fact that was never recorded stays empty (for example the CephVersion of a cluster that Run returned with an error); an empty value does not distinguish not recorded from recorded as empty, and no value reports whether the resources still exist.

type ConfigEntry 

type ConfigEntry struct {
	Section            string `json:"section"`
	Mask               string `json:"mask"`
	Name               string `json:"name"`
	Value              string `json:"value"`
	Level              string `json:"level"`
	CanUpdateAtRuntime bool   `json:"can_update_at_runtime"`
}

ConfigEntry reports an exact stored entry rather than an inherited/default value. Configuration may contain sensitive values; log selected fields only.

type ConfigSetting 

type ConfigSetting struct {
	Section, Mask, Name, Value string
}

ConfigSetting addresses one entry in the MON configuration database. Section is global, mon, mgr, osd, mds, client or a specific identity such as osd.0. Mask is optional: class:ssd, host:node-a or host:node-a/class:ssd. These stored settings have lower precedence than local files, argv and runtime overrides.

type ConnectionConfig added in v0.4.0 

type ConnectionConfig struct {
	// Conf is the content of ceph.conf.
	Conf []byte
	// Keyring is the content of the keyring file.
	Keyring []byte
}

ConnectionConfig holds the files that an external native client needs to connect: the ceph.conf and the keyring. The fields are named so that the two byte slices cannot be swapped by a positional assignment. Each value returned by a ConnectionConfig method owns independent copies of the bytes. The keyring is a credential: it is not redacted, so do not log the value.

type Container 

type Container struct {
	testcontainers.Container
	// contains filtered or unexported fields
}

Container embeds the MON/control container and owns the MGR, OSDs, optional RGW/MDS services and network. Use Terminate to clean up the entire cluster. Generic Run options customize the MON; module options configure the cluster. Use WithHostNetwork to change cluster networking, rather than changing only the MON's network mode.

func Run 

func Run(ctx context.Context, img string, opts ...testcontainers.ContainerCustomizer) (*Container, error)

Run creates configurable MON, MGR and OSD containers for a disposable cluster. img supplies all roles unless WithImageOptions selects explicit role images with ImageOptions.WithImageSet, in which case img must be empty or the set's essential image. Run recognises the Option values among opts (every Ceph setting, including each grouped configuration value carried by its option function), applies and validates them before it creates anything, and passes all of opts, in argument order, to the first MON's container request after the module's own request options and before the messenger environment, the WithConfigFile file and the ownership label. An Option has no effect on that request, so a setting passed as one is never lost and never reaches a container. Images contain runtime tools; this module supplies cluster setup.

Run builds the whole cluster before it returns, bounded by ctx and by WithStartupTimeout: it validates every option together, creates the owned bridge network (two with NetworkConfig.WithSeparateClusterNetwork) or checks the engine in host mode, starts MON a and records its configuration, admin keyring and Ceph version, adds the remaining MONs (and, with more than one, the separate CLI control container), starts the first MGR, registers and starts the OSDs and waits until they are up and in, then starts the remaining MGRs (the first stays active), waits for an available active MGR (or, with ClusterConfig.WithNoInitialManagers, for quorum and the OSDs), and finally creates the ClusterConfig.WithPools pools and the resources of the service options. Run does not wait for clean placement groups; call WaitForClean before client I/O.

Contradictory options fail before any Docker resource exists, regardless of their order: a host address without WithHostNetwork; host networking with a separate cluster network or with an address family other than AddressIPv4; a Messenger policy with MessengerV2Secure; AddressIPv4MappedIPv6 with a separate cluster network; one field set by two carriers of the same group; no initial OSDs with initial pools or services; no initial managers with initial filesystems or gateways; a default CRUSH root or an initial pool that the initial layout cannot satisfy; and duplicate pool, filesystem or gateway names, including the pools a filesystem creates. In host mode Run fails with ErrHostNetworkUnavailable when the engine does not run Linux containers, the selected address is not assigned on the Docker host, or the test process cannot reach the first MON.

The returned Container embeds the first MON's testcontainers.Container, so generic methods such as Exec, Logs, Host and MappedPort address MON a; Terminate is overridden and removes the whole cluster. A non-nil Container returned with an error owns every resource created so far and must still be terminated; register testcontainers.CleanupContainer before checking err.

Example 
package main

import (
	"context"
	"log"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/testcontainers/testcontainers-go"
)

func main() {
	ctx := context.Background()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithClusterConfig(ceph.ClusterConfig{}.WithOSDCount(2)))
	defer func() {
		if err := testcontainers.TerminateContainer(cluster); err != nil {
			log.Printf("failed to terminate container: %s", err)
		}
	}()
	if err != nil {
		panic(err)
	}
	osd, err := cluster.AddOSD(ctx)
	if err != nil {
		panic(err)
	}
	if err := osd.Remove(ctx); err != nil {
		panic(err)
	}
}
Example (Services)

Options from several service packages start one cluster that serves RBD, CephFS and S3 clients together. 

package main

import (
	"context"
	"log"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/JSYoo5B/ceph-testcontainers-go/cephfs"
	"github.com/JSYoo5B/ceph-testcontainers-go/rbd"
	"github.com/JSYoo5B/ceph-testcontainers-go/rgw"
	"github.com/testcontainers/testcontainers-go"
)

func main() {
	ctx := context.Background()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage,
		ceph.WithClusterConfig(ceph.ClusterConfig{}.WithOSDCount(3)),
		rbd.WithPools(ceph.PoolConfig{Name: "volumes"}),
		cephfs.WithFilesystems(cephfs.Config{Name: "shared"}),
		rgw.WithGateways(rgw.Config{Name: "s3"}),
	)
	defer func() {
		if err := testcontainers.TerminateContainer(cluster); err != nil {
			log.Printf("failed to terminate container: %s", err)
		}
	}()
	if err != nil {
		panic(err)
	}
	_ = cephfs.Filesystems(cluster)
	_, _ = rgw.Gateways(context.Background(), cluster)
}

func (*Container) AddManager 

func (c *Container) AddManager(ctx context.Context, name string, opts ...DaemonOptions) (*ManagerContainer, error)

AddManager creates a named active/standby candidate and waits for its mgrmap registration. Existing daemon identities are rejected; it does not force active ownership away from an existing MGR. With AddressDualStack it then checks the active MGR's advertised addresses and fails with ErrUnsupported when one family has no concrete address. DaemonOptions.WithImage starts this one MGR from another image; the default is the essential image. Options are validated before the topology lock is taken or any identity is created.

It never forces the active role away from a running manager, and success does not mean every MGR module is ready; use WaitMGRModuleReady or WaitForClean. On an AddressDualStack cluster it then checks the active MGR's advertised addresses.

func (*Container) AddMonitor 

func (c *Container) AddMonitor(ctx context.Context, name string, opts ...DaemonOptions) (*MonitorContainer, error)

AddMonitor joins a new named MON from the live quorum's map and shared key. Existing daemon data is retained; a partial new member remains owned on error. DaemonOptions.WithImage starts this one MON from another image; the default is the essential image. Options are validated before the topology lock is taken or anything is created.

Names already in the monmap are refused, and a can never be reused. The first call also starts the independent control container. After joining, it copies the new quorum's addresses into the mon_host of every owned daemon's ceph.conf, including stopped daemons; if only that copy failed, retry RefreshMonitorConfig. Caller-owned client containers are not updated (see RefreshClientMonitorConfig). A partially started member stays owned on error.

func (*Container) AddOSD 

func (c *Container) AddOSD(ctx context.Context, configs ...OSDOptions) (*OSDContainer, error)

AddOSD registers, formats and starts a new OSD, then waits for it to be up/in. With no argument the OSD takes the cluster defaults: host osd-ID under the default root and the essential image. OSDOptions values select an explicit logical CRUSH location (WithPlacement) or start this one OSD from another image (WithImage); several values compose as described at OSDOptions. Omitted host names resolve to osd-ID; logical hosts/racks are simulated placement domains on the same Docker engine, not physical failure domains.

The context is checked first, then the assembled configuration is validated once before the topology lock is taken, before any OSD is registered and before any container exists. Topology operations are serialized. An OSD returned with an error is tracked for cleanup and may be inspected; no data is silently purged on failure. The image override is kept in the OSD's recipe for DetachedOSD.Reopen.

func (*Container) Ceph 

func (c *Container) Ceph(ctx context.Context, args ...string) ([]byte, error)

Ceph runs the Ceph CLI in the control container with --connect-timeout 5 and returns its standard output. Arguments are passed directly, without shell interpolation. Use --format json for machine-readable responses. A non-zero exit returns the standard output with a *NativeFailure (exit code, the errno Ceph reported and its reason without log lines, keys redacted); errors.Is(err, fs.ErrNotExist) holds for ENOENT. When no MON answers within 5 seconds the CLI exits before it sends the command, and Ceph connects again until a MON answers, ctx ends or a minute has passed, so a stalled or electing MON delays the command instead of failing it, and the command runs once.

func (*Container) ClientContainer added in v0.4.0 

func (c *Container) ClientContainer(ctx context.Context) (testcontainers.Container, error)

ClientContainer returns the owned userspace toolkit executor. It starts lazily for a separate client image; aggregate images reuse the control-plane executor.

Run rados, rbd and the Python client bindings there. The first attachment is serialized with MON topology changes and honours cancellation, and the cluster owns the container's partial or successful teardown. RBD and CephFS operations, ObjectWatchers and PGInconsistencies use it, while daemon-local sockets and process observations stay on their daemon.

func (*Container) CollectCrashMetadata added in v0.2.0 

func (c *Container) CollectCrashMetadata(ctx context.Context, config CrashMetadataConfig) (*CrashMetadataReport, error)

CollectCrashMetadata reads the Ceph crash metadata files in CrashDir of the cluster's MON, MGR, OSD, MDS and RGW containers and its client toolkit, and of config.AdditionalContainers, through Docker's archive API. It works on stopped containers and on a partially bootstrapped or terminated cluster, and starts, stops or reconfigures nothing. It reads only regular files named meta in a crash report directory, keeps their known fields and redacts them like CollectDiagnostics; it skips crash logs and every other file, and rejects links, unexpected paths and files that are not JSON objects. Each container's outcome, rejected files and errors stay in the report, which is returned also with an error for failed or incomplete reads. Crash metadata can still hold host names and paths; review a report before sharing it.

Ceph writes a report directory (a meta JSON file, a log file and a done marker) into CrashDir when a daemon dies on a fatal signal or a failed assertion; only meta files, also in posted/, are read, other entries are counted in IgnoredEntries, and rejected files are listed in Omitted with their reason. A record keeps the known crash fields (crash_id, timestamp, process_name, entity_name, ceph_version, utsname_*, os_*, backtrace, assert_*, io_error*, archived, stack_sig) and names other fields in OmittedFields without values. A failed or incomplete container makes the call return an error with the report and the other containers' records. A container's exit code alone does not tell a crash: a daemon that is PID 1, as here, exits 1 after writing its report. It is not called automatically outside TerminateOnCleanup's archive.

func (*Container) CollectDiagnostics 

func (c *Container) CollectDiagnostics(ctx context.Context, config DiagnosticsConfig) (*DiagnosticsReport, error)

CollectDiagnostics collects all tracked MON/MGR/OSD/MDS/RGW/control and client toolkit handles, container state, network addresses, bounded log tails and fixed Ceph queries. It supports partial bootstrap and stopped/removed containers. Individual failures return both a usable report and an aggregate error; truncation alone does not return an error. A canceled caller needs a fresh context to collect. No daemon is started, stopped or reconfigured, and no file, environment, keyring, auth database, mount source or arbitrary command is collected.

It collects the state and selected metadata of every tracked MON, MGR, OSD, MDS, RGW, control and client toolkit container, network metadata, log tails and fixed Ceph queries (version, status, health, pgs, quorum, monmap, mgrmap, osdmap, fsmap); it never collects files, environment variables, keyrings, the auth database, mount sources or arbitrary commands. TerminateOnCleanup with Archive calls it on failure; otherwise call it before cleanup with its own short context. When some queries fail, the report and an aggregate error are both returned, so save the report first; truncation alone is not an error. A ceph artifact holds the query's stdout followed by its stderr. When ctx ends, every running query is stopped in the control container before the call returns. Known admin and RGW secrets and every client key the cluster generated (also replaced or attempted ones) are masked with RedactValues and named secret patterns.

func (*Container) ConnectionConfig 

func (c *Container) ConnectionConfig(ctx context.Context) (ConnectionConfig, error)

ConnectionConfig returns the independent bootstrap ceph.conf and admin keyring copies, waiting for topology and the configuration cache only while ctx remains active. On error it returns the zero ConnectionConfig.

It is for a native client the caller starts, such as a test process using go-ceph in host mode. The configuration contains the addresses the cluster advertises: container addresses in bridge mode, host addresses with the final MON ports in host mode. ctx bounds the wait for a concurrent topology change or configuration update. It fails after termination or before bootstrap completed and then returns the zero ConnectionConfig. For a restricted identity use ClientConfig.ConnectionConfig.

func (*Container) ControlContainer 

func (c *Container) ControlContainer(ctx context.Context) (testcontainers.Container, error)

ControlContainer returns the stable CLI handle while bounding a concurrent control-plane creation or termination publication wait by ctx. The cluster owns the handle; this snapshot does not promise it remains live. In a single-MON cluster it is MON a; with more than one monitor it is an independent CLI container created by the first AddMonitor (also during Run), so commands keep working while MON a is stopped or removed. It runs the essential image, so it has the ceph CLI and bootstrap tools such as ceph-authtool. With split role images it has neither rados and rbd (use ClientContainer) nor radosgw-admin (use rgw.Gateway.Admin); an aggregate image may have them, but do not rely on that. Do not pause or stop it; fault.Pause refuses it.

func (*Container) CreateClient 

func (c *Container) CreateClient(ctx context.Context, name string, caps ClientCaps, opts ...ClientOptions) (*ClientConfig, error)

CreateClient creates a fresh client with exactly the supplied capabilities. Existing identities are rejected before mutation; their keys and caps are never updated. A non-nil result with an error identifies an attempted or partial creation; inspect auth state or terminate the cluster rather than retrying with broader caps. ClientConfig.Delete can revoke a confirmed successful creation. The key has the cluster's AuthConfig.KeyType, the release default without one, or the type that ClientOptions.WithKeyType selects.

name is an ID or client.ID of letters, digits, dots, underscores or hyphens. The identity is created whether or not the MONs allow its key type; whether it can authenticate depends on the MonMap's auth_allowed_ciphers. CipherAES256K needs an essential image whose ceph-authtool has --key-type; without it the call fails before any change.

func (*Container) CreatePool 

func (c *Container) CreatePool(ctx context.Context, config PoolConfig) (*Pool, error)

CreatePool creates a new pool with explicit placement and replication settings. It validates the entire config and requires enough eligible owned domains before sending any Ceph command. Existing pools, profiles and rules are rejected rather than changed. A non-nil Pool returned with an error identifies resources whose creation was attempted; inspect them or terminate the disposable cluster. That Pool may not know the native ID of its pool yet: it then binds to the pool of its name at its first successful operation, so use it only to clean up. A Pool returned without an error is always bound to its native pool. It never deletes data after a partial command failure. Use WaitForClean before application I/O. For RBD, create images in a replicated metadata pool and pass the EC pool as data_pool in the native client (or --data-pool to the RBD CLI).

The pool uses a dedicated CRUSH rule tc-<name>-replicated (or tc-<name>-ec with an EC profile of the same name) and is created with the autoscaler off. The returned Pool embeds the resolved PoolConfig and adds CRUSHRule and ErasureCodeProfile; it describes the requested settings at creation, while Pool.Status reports the current state. The handle identifies its native pool by name and native ID, and copies share that identity: after Pool.Remove, or when the name belongs to another pool, every method returns an error instead of acting on the pool that now has the name.

func (*Container) DeepScrubPG 

func (c *Container) DeepScrubPG(ctx context.Context, pgid PGID) error

DeepScrubPG requests a native deep scrub of one PG and waits until the PG reports a deep-scrub stamp newer than before the request. Inconsistencies are reported, not repaired. The noscrub and nodeep-scrub flags or an unavailable primary can keep the wait from finishing before its deadline.

fault.InjectObjectDataError creates a damaged replica to observe with DeepScrubPG, Observer.PGInconsistencies and RepairPG; ceph osd map <pool> <object> --format json reports an object's PG and acting set.

func (*Container) Managers 

func (c *Container) Managers(ctx context.Context) ([]*ManagerContainer, error)

Managers returns sorted owned candidate descriptors. The slice is independent; descriptors and their containers retain fixture ownership.

The returned slice is independent; descriptors and containers stay owned by the cluster. Stop the active manager's container (ManagerStatus names it) to make a standby take over, or the only manager to test behaviour without one.

func (*Container) Monitors 

func (c *Container) Monitors(ctx context.Context) ([]*MonitorContainer, error)

Monitors returns the owned MON membership sorted by daemon name, waiting for the topology only while ctx remains active.

The returned slice is independent; descriptors and containers stay owned by the cluster. Stop and Start on a handle inject failures and keep the monitor's store; MonitorContainer.Remove changes membership. When a majority is stopped, quorum is lost and new clients cannot authenticate; after the monitors start again, existing and new sessions work. On a bridge network a MON returns at the address the monmap records for it whatever the order of Start calls, because the fixture requests its original address from Docker again; a MON whose endpoint fault.InterruptNetwork interrupted does not start until it is restored. MON a waits for a quorum when it starts, so after stopping all MONs start the others first.

func (*Container) OSDs 

func (c *Container) OSDs(ctx context.Context) ([]*OSDContainer, error)

OSDs returns the owned OSD descriptors sorted by ID, waiting for the topology only while ctx remains active.

The returned slice is independent; descriptors and containers stay owned by the cluster. Stop and Start on a handle inject daemon failures and keep the OSD's identity and data, also with in-memory storage; Ceph marks a stopped OSD down after the heartbeat grace and out after mon_osd_down_out_interval unless noout is set. OSD bootstrap initializes only an empty directory: it refuses a non-empty partial store without a ready marker, and a ready store must match the OSD's ID, UUID and cluster FSID; failed initialization is not replayed (see docs/internals/osd-storage.md).

func (*Container) Pool added in v0.4.0 

func (c *Container) Pool(ctx context.Context, name string) (*Pool, error)

Pool returns a handle for the existing pool with the exact native name. Use it for a pool that CreatePool did not create, such as an initial pool from WithPools or a default pool of a service. It fails when the pool does not exist. The handle identifies the pool by its native ID, so it does not act on a later pool that reuses the name.

The returned Pool describes what the cluster reports: Name, PGNum, Replicas (of a replicated pool), MinSize and ErasureCodeProfile. The other fields, CRUSHRule included, are zero; PoolState has the full native state.

func (*Container) RedactSecrets added in v0.2.0 

func (c *Container) RedactSecrets(ctx context.Context, data []byte, values ...string) ([]byte, error)

RedactSecrets returns a copy of data in which the secrets that the cluster knows, and values, are replaced by [REDACTED], as in CollectDiagnostics: keyring keys, service credentials, and the private keys, secret assignments and bearer tokens that it recognizes. It is meant for output that a test logs or reports; an oracle compares the raw bytes. Redaction is best effort.

func (*Container) RefreshClientMonitorConfig 

func (c *Container) RefreshClientMonitorConfig(ctx context.Context, client testcontainers.Container) error

RefreshClientMonitorConfig updates only the explicit client's global mon_host entry in /etc/ceph/ceph.conf. The client's global FSID must match this fixture and its live quorum. Docker archive copies also work with stopped containers; this method never starts, stops, terminates or adopts the client's lifetime. Other settings and keyring files are retained. It does not update mirror peer attributes, network attachments, or private-section mon_host overrides.

Successful writes remain on partial failure and may be retried with a fresh context. External configuration writers must not race this operation: Docker has no file compare-and-swap. Callers own clients and their cleanup.

func (*Container) RefreshMonitorConfig 

func (c *Container) RefreshMonitorConfig(ctx context.Context) error

RefreshMonitorConfig copies the live quorum's bootstrap addresses into the global mon_host entry of every owned daemon's ceph.conf, including stopped daemons, without starting or restarting them. Other entries and comments are retained. Caller-owned clients and multicluster daemon configs are not copied.

AddMonitor and MonitorContainer.Remove already refresh these files. Retry this method after an AddMonitor that joined native membership but could not copy every config; retry MonitorContainer.Remove itself after a partial removal. Successful copies and the future-client template are retained on partial failure. Repeating a refresh reads each current file and skips copies that are already up to date. External config writers must not race this operation: Docker has no file CAS.

func (*Container) RepairPG 

func (c *Container) RepairPG(ctx context.Context, pgid PGID) error

RepairPG requests a native repair of one PG and waits until the PG reports a newer deep-scrub stamp and has finished repairing; it returns an error if the PG is still inconsistent then. Repair rewrites bad replicas from an authoritative copy; it cannot recover an object without one, and the wait then ends with the PG still inconsistent.

func (*Container) RunNativeCommand added in v0.2.0 

func (c *Container) RunNativeCommand(ctx context.Context, target testcontainers.Container, command NativeCommand) (NativeCommandResult, error)

RunNativeCommand runs one native command in target under the supervisor that the module's own native queries use. target is the control container, a daemon or service container of this cluster, or a container of the caller attached to the cluster's network (the host network for a host-network cluster); the call never adopts, stops or removes the caller's container. target must be running, not paused, and have python3, which runs the supervisor. A malformed command or an ended ctx fails before anything runs.

The supervisor starts the command in its own process group, bounds each output stream inside the container and kills the group at the deadline, Timeout shortened to ctx's deadline. A command that times out or exceeds an output limit returns its result, marked by TimedOut or the Truncated flags, with an error that wraps ErrNativeCommandIncomplete. A descendant that starts its own session leaves the group and is not stopped.

When ctx ends first, the call stops the command inside the container before it returns, also when Docker had not started it yet, within its own NativeCommandStopTimeout. When that cannot be confirmed, for example because the container is paused or a signal killed the supervisor, the error is a *NativeCommandUnstoppedError, whose Stop retries. Terminate cancels running commands and stops those not confirmed. Stopping a command does not undo what it already did, such as a MON command it had sent, and nothing is retried. Errors name the command by Label; a failure of Docker or of target's implementation appears only by its kind, and errors.Unwrap returns the original error, whose text may quote anything. RedactSecrets redacts output that a test logs.

With Input, the supervisor runs in an exec of the Docker engine with its standard input attached, and passes what Input writes to the command; see NativeInput. The call stops taking writes when the command ends, and returns only after its input and output work ended. Its deadline and cancellation stop the command as without Input, and a write in progress then fails. A malformed limit or an Input that served another command fails before anything runs or any input is taken.

Failures before the command runs are classified: a malformed command, or a target that the cluster neither owns nor has attached, or that is stopped or paused, is ErrInvalidRequest; a target without python3, or a program that does not exist or is not executable, is ErrImageCapability; a program that could not start for want of a process or memory, or that a security policy refused, and other Docker failures are ErrEnvironmentUnavailable. A container that disappears or stops after the command was sent matches no category, since the command may have run, and neither do context errors, ErrNativeCommandIncomplete, ErrNativeInputClosed or *NativeCommandUnstoppedError. The essential and client roles and the RGW setup runtime provide python3; the cephfs role does not promise it.

func (*Container) ServiceContainers 

func (c *Container) ServiceContainers() []testcontainers.Container

ServiceContainers returns owned optional services (such as RGW or MDS), sorted by service name. The cluster owns their cleanup.

func (*Container) Terminate 

func (c *Container) Terminate(ctx context.Context, opts ...testcontainers.TerminateOption) error

Terminate removes all owned daemons before removing the isolated network. It also works on a partially initialized cluster returned by Run. The whole cluster is discarded, so each container gets 2 seconds instead of Docker's 10 seconds to stop before it is killed; a StopTimeout option passed by the caller still takes precedence. From its start, also when it then fails, Terminate refuses new native commands and cancels the running ones, including the command of an rgw.Gateway.Admin call.

It first restores network delays and interruptions and resumes paused containers, then releases host port reservations, stops services (RGW, MDS) before OSDs, removes every owned daemon and helper container, the OSD storage volumes and the shared admin-socket run volume, and finally the owned networks. Every container the cluster creates carries the org.testcontainers.ceph.cluster label with the cluster's ID, and Terminate also removes a labelled container it has no handle for, as Docker can create one when the reply to a create request was lost. A failed cleanup can be retried; an endpoint that cannot be restored keeps its network for the retry. Containers the caller attached with WithClient or ClientConfig.WithIdentity are the caller's: terminate them first, because Docker cannot remove a network that still has attached containers (with testcontainers.CleanupContainer, register the cluster first).

func (*Container) TerminateOnCleanup added in v0.4.0 

func (c *Container) TerminateOnCleanup(t testing.TB, config FinalizationConfig) (*Finalization, error)

TerminateOnCleanup registers one t.Cleanup that finalizes and then terminates the cluster, and returns the Finalization to add checks and cleanups to. Register it right after Run instead of another cleanup that terminates the cluster, and route the removal of caller-owned clients that the archive should still see through Finalization.Cleanup: test cleanups registered later run before it.

Every phase uses a fresh context bounded by its own timeout, because the test's context has ended by then. Check, collection and storage errors never skip the cleanups or Terminate. A check's error and Terminate's error fail the test; collection and storage errors are logged and recorded in the manifest. Nothing runs after a forced process exit or a lost runner.

func (*Container) WaitForClean 

func (c *Container) WaitForClean(ctx context.Context) error

WaitForClean waits until a manager is available, the native OSD count equals the owned OSD count with every OSD up and in, at least one PG exists and every PG is exactly active+clean. HEALTH_WARN alone does not block it. PG states with extra flags, such as active+clean+scrubbing, do not count. It never succeeds while an owned OSD is intentionally stopped or out; use WaitForPGClean then. The cluster's startup timeout also bounds the wait.

Call it after creating pools and after topology changes, before client I/O. OSDs registered outside the fixture make the count differ, a pool whose replica count exceeds the OSD count never becomes clean, and after Pool.SetPGCount or Pool.SetPlacement it can pass before the new PG set is reported (see Pool.WaitForPGCount and Pool.PGs).

func (*Container) WaitForPGClean 

func (c *Container) WaitForPGClean(ctx context.Context) error

WaitForPGClean waits for every existing PG to be exactly active+clean. Unlike WaitForClean it allows intentionally out/down OSDs, and does not assert any OSD topology. At least one PG and an available manager are required. This is a recovery barrier, not a promise of cluster health or daemon availability.

func (*Container) WaitForQuorum 

func (c *Container) WaitForQuorum(ctx context.Context) error

WaitForQuorum waits for a majority of the current monmap, including fixtures with a stopped monitor. It does not require HEALTH_OK or every MON to run.

func (*Container) WaitMGRModuleReady 

func (c *Container) WaitMGRModuleReady(ctx context.Context, name string) error

WaitMGRModuleReady waits for enabled/can-run metadata and successful native read-only commands. Supported modules are rbd_support (task and snapshot schedule lists) and volumes (volume list). Other modules need an application specific command probe. Each poll releases the cluster lock; cancellation and termination stop the wait. Success does not claim worker tasks have completed.

func (*Container) WithClient 

func (c *Container) WithClient(opts ...ClientContainerOptions) testcontainers.CustomizeRequestOption

WithClient connects another container to Ceph and copies its admin credentials. These privileged, ephemeral credentials are for trusted test containers only. Caller owns and terminates that container before terminating the cluster. The options customize the container's ceph.conf.

It copies the bootstrap /etc/ceph/ceph.conf and /etc/ceph/ceph.client.admin.keyring into the container and joins it to the cluster's public network (the host network in host mode). The container must already contain the Ceph clients it uses; WithClient installs nothing, and ceph.DefaultImage or ClusterFacts.Images.Client work for CLI and Python clients. Combine it with WithIdleEntrypoint for a container driven with Exec. Ceph clients connect directly to the OSD addresses the MONs advertise, which in bridge mode are container addresses, so a process on the Docker host cannot reach RADOS, RBD or CephFS through the MON's mapped port alone; use a WithClient container or WithHostNetwork on Linux.

type CrashExpectation added in v0.4.0 

type CrashExpectation struct {
	Name     string
	Required bool
	MinCount int
	MaxCount int
	// Daemon is the source whose crash directory held the crash, such as
	// "osd.0".
	Daemon string
	// Process is the crashed process, such as "ceph-osd".
	Process string
	// Signature is a regular expression (RE2 syntax) that must match one
	// line of the crash's stack_sig, assert_condition, assert_func,
	// assert_msg or backtrace fields.
	Signature string
}

CrashExpectation permits, or with Required requires, new crashes that match all of its non-empty criteria; at least one of Daemon, Process and Signature is required.

type CrashMetadataConfig added in v0.2.0 

type CrashMetadataConfig struct {
	Timeout              time.Duration
	OperationTimeout     time.Duration
	MaxArchiveBytes      int64
	MaxEntries           int
	MaxRecordBytes       int
	MaxRecords           int
	AdditionalContainers []DiagnosticsContainer
	RedactValues         []string
}

CrashMetadataConfig bounds CollectCrashMetadata. Zero fields select a one minute total timeout, 20 seconds per container, 16 MiB of archive and 4096 archive entries per container, 64 KiB per metadata file and 64 records per container. AdditionalContainers are caller-owned containers, such as mirror daemons, whose CrashDir is read too; collection never adopts them. RedactValues removes additional opaque secrets from the collected fields.

The limits are 5 minutes total, 1 minute per container, 256 MiB of archive, 65536 entries, 1 MiB per metadata file, 1024 records per container (later ones are listed in Omitted), 64 additional containers and 128 redaction values of up to 4096 bytes. An additional container is read whatever its Role and Name; the control container, which runs no daemon, is read only when listed here.

type CrashMetadataContainer added in v0.2.0 

type CrashMetadataContainer struct {
	Role        string `json:"role"`
	Name        string `json:"name"`
	ContainerID string `json:"container_id,omitempty"`
	ImageID     string `json:"image_id,omitempty"`
	CrashDir    string `json:"crash_dir"`
	State       string `json:"state,omitempty"`
	ExitCode    int    `json:"exit_code"`
	OOMKilled   bool   `json:"oom_killed,omitempty"`
	StartedAt   string `json:"started_at,omitempty"`
	FinishedAt  string `json:"finished_at,omitempty"`

	Outcome CrashOutcome  `json:"outcome"`
	Error   string        `json:"error,omitempty"`
	Records []CrashRecord `json:"records"`
	// Omitted lists the metadata files that were rejected or not read.
	Omitted []CrashOmission `json:"omitted,omitempty"`
	// IgnoredEntries counts the other files, such as crash logs, whose
	// content was skipped.
	IgnoredEntries int `json:"ignored_entries"`
	// ArchiveBytes counts the archive bytes read.
	ArchiveBytes int64 `json:"archive_bytes"`
}

CrashMetadataContainer is what one container's crash directory held. The state fields are Docker's view at collection, for correlation with the records.

type CrashMetadataReport added in v0.2.0 

type CrashMetadataReport struct {
	SchemaVersion int                      `json:"schema_version"`
	StartedAt     time.Time                `json:"started_at"`
	FinishedAt    time.Time                `json:"finished_at"`
	Closed        bool                     `json:"closed"`
	Complete      bool                     `json:"complete"`
	Containers    []CrashMetadataContainer `json:"containers"`
}

CrashMetadataReport is an independent, JSON-serializable report of the Ceph crash metadata in the cluster's daemon containers. Complete means that every container's crash directory was read in full and every metadata file in it was accepted, or that the directory is absent. It does not mean that every crash in the cluster's history was captured: a removed container takes its records along.

type CrashOmission added in v0.2.0 

type CrashOmission struct {
	Path   string `json:"path"`
	Reason string `json:"reason"`
}

CrashOmission names a metadata file that was not accepted, and why.

type CrashOutcome added in v0.2.0 

type CrashOutcome string

CrashOutcome says what reading one container's crash directory found. 

const (
	// CrashRecordsFound means that the directory was read and held at least
	// one accepted record.
	CrashRecordsFound CrashOutcome = "records"
	// CrashEntriesRejected means that the directory was read, but every
	// metadata file in it was rejected; Omitted says why.
	CrashEntriesRejected CrashOutcome = "rejected"
	// CrashDirectoryEmpty means that the directory was read and held no
	// metadata file. It is not proof that the daemon never crashed.
	CrashDirectoryEmpty CrashOutcome = "empty"
	// CrashDirectoryAbsent means that the container has no crash directory.
	CrashDirectoryAbsent CrashOutcome = "absent"
	// CrashContainerUnavailable means that the container was never created
	// or was removed, with its crash directory.
	CrashContainerUnavailable CrashOutcome = "unavailable"
	// CrashReadFailed means that Docker or the container implementation
	// failed, for example without permission or after the context ended.
	CrashReadFailed CrashOutcome = "failed"
	// CrashArchiveIncomplete means that the archive was malformed, cut off or
	// over a limit; Records holds what was read before.
	CrashArchiveIncomplete CrashOutcome = "incomplete"
)

type CrashRecord added in v0.2.0 

type CrashRecord struct {
	CrashID       string         `json:"crash_id"`
	Path          string         `json:"path"`
	Entity        string         `json:"entity,omitempty"`
	Process       string         `json:"process,omitempty"`
	CephVersion   string         `json:"ceph_version,omitempty"`
	Timestamp     string         `json:"timestamp,omitempty"`
	Done          bool           `json:"done"`
	Posted        bool           `json:"posted,omitempty"`
	Fields        map[string]any `json:"fields"`
	OmittedFields []string       `json:"omitted_fields,omitempty"`
	// contains filtered or unexported fields
}

CrashRecord is one daemon-written crash metadata file. Fields holds the known Ceph crash fields, redacted; OmittedFields names the others, whose values are not kept. Done reports the done marker with which Ceph completes a crash report. Posted means that the report was in the posted directory, where ceph-crash moves the reports that it sent.

type DaemonAddress added in v0.2.0 

type DaemonAddress struct {
	Name  string
	Addrs []EntityAddress
}

DaemonAddress is the name of one MON or MGR and the public address vector that its cluster map advertises.

type DaemonAddresses added in v0.2.0 

type DaemonAddresses struct {
	MonMapEpoch int
	// Monitors are the monmap members in rank order.
	Monitors    []DaemonAddress
	MgrMapEpoch int
	// ActiveManager is nil when no MGR is active. A MGR that has just become
	// active advertises its addresses only when it is available; until then
	// ManagerAvailable is false and its address vector can be empty. Standby
	// MGRs advertise no address in the MGRMap, so StandbyManagers lists only
	// their names.
	ActiveManager    *DaemonAddress
	ManagerAvailable bool
	StandbyManagers  []string
	OSDMapEpoch      int
	// OSDs are the OSDMap entries in ID order.
	OSDs []OSDAddress
}

DaemonAddresses lists the daemons of a cluster and the addresses that the MON, MGR and OSD maps advertise, with the epoch of each map. The three maps are read one after another, not as one snapshot.

type DaemonOptions added in v0.4.0 

type DaemonOptions struct {
	// contains filtered or unexported fields
}

DaemonOptions is the configuration of one daemon that AddMonitor or AddManager creates, or that MonitorContainer.Replace starts again (AddOSD takes OSDOptions). It is a plain value with chainable, value-returning methods. It applies to that daemon only and never changes the cluster's settings or any other daemon. The zero value changes nothing. 

mon, err := cluster.AddMonitor(ctx, "b", ceph.DaemonOptions{}.WithImage(image))

Within one value a later call of the same method replaces the earlier one. A field set by two values is refused with ErrInvalidRequest, even when they agree.

func (DaemonOptions) WithImage added in v0.4.0 

func (o DaemonOptions) WithImage(image string) DaemonOptions

WithImage starts the one new daemon from image instead of the cluster's essential image. The image must contain the daemon's binary and the tools the fixture's startup scripts use, as the essential image does; nothing here checks its release or compatibility with the running cluster.

The override belongs to that daemon: Stop and Start keep the container, and the daemon's handle reports it with Image. It does not change ClusterFacts, which describes the cluster's own settings and its first MON, and a CLI, client or helper container still uses the cluster's images.

The image reference must be non-empty and free of whitespace and control characters. A refusal matches ErrInvalidRequest and happens before the topology lock and before any daemon identity, key or container is created; nothing checks that the image exists. MGR containers are never recreated, so their image does not change for their lifetime; a MON keeps its image unless MonitorContainer.Replace starts it from another one. ClusterFacts.Images and ClusterFacts.CephVersion, and the images recorded in a finalization manifest or verification report, keep describing the cluster and its first MON; the diagnostics archive records each container's image from Docker. A client container is still refused when its release differs from ClusterFacts.CephVersion. This is a building block: no mixed-version or upgrade combination is claimed to work.

type DebugLevelReceipt added in v0.4.0 

type DebugLevelReceipt struct {
	Daemon string `json:"daemon"`
	Name   string `json:"name"`
	// Generation is the daemon container's ID and start time that the
	// window was opened on.
	Generation string          `json:"generation"`
	Requested  string          `json:"requested"`
	Prior      string          `json:"prior"`
	Applied    string          `json:"applied,omitempty"`
	Restored   string          `json:"restored,omitempty"`
	State      DebugLevelState `json:"state"`
	// Reason is the refusal text of a DebugLevelNotApplied window and is
	// empty for every other state.
	Reason      string                 `json:"reason,omitempty"`
	Transitions []DebugLevelTransition `json:"transitions"`
}

DebugLevelReceipt is the JSON-serializable record of one window. Requested is the caller's text and Applied, Restored and each transition's Readback are the daemon's normalized effective values; an empty value was not read.

type DebugLevelReceipter added in v0.4.0 

type DebugLevelReceipter interface {
	Receipt() DebugLevelReceipt
}

DebugLevelReceipter is a source of one debug level window's receipt. A *override.DebugLevelHandle implements it. A registration is absent when it is a nil interface, when its dynamic value is a nil pointer, map, slice, func, chan or interface (Receipt is then never called, so a method that is not nil-safe, including a value-receiver method reached through a nil pointer, cannot panic), or when Receipt returns the zero receipt (State is empty). Absence is not decided by a nil comparison of the interface alone.

type DebugLevelSetting added in v0.4.0 

type DebugLevelSetting struct {
	Daemon, Name, Level string
}

DebugLevelSetting selects one runtime debug level of one running daemon. Daemon is an owned OSD such as "osd.0"; Name is "debug_ms" or "debug_osd"; Level is "N" or "N/M", each from 0 to 99, as Ceph's config set accepts it. Other roles and options are not supported yet.

type DebugLevelState added in v0.4.0 

type DebugLevelState string

DebugLevelState is the progress of an override.DebugLevelHandle. A window moves from DebugLevelPriorRead through DebugLevelApplyAttempted to DebugLevelApplied or DebugLevelApplyUncertain, and Restore moves it through DebugLevelRestoreAttempted to DebugLevelRestored, DebugLevelConflict, DebugLevelTargetChanged or DebugLevelUnconfirmed. 

const (
	// DebugLevelPriorRead: the daemon's effective prior level was read.
	DebugLevelPriorRead DebugLevelState = "prior-read"
	// DebugLevelApplyAttempted: the set was issued; its outcome is not
	// known yet.
	DebugLevelApplyAttempted DebugLevelState = "apply-attempted"
	// DebugLevelApplied: the effective readback equals the requested level.
	DebugLevelApplied DebugLevelState = "applied-verified"
	// DebugLevelApplyUncertain: the set or its readback failed or differs;
	// the level may have changed, and Restore reconciles it.
	DebugLevelApplyUncertain DebugLevelState = "apply-uncertain"
	// DebugLevelRestoreAttempted: Restore has started and not settled yet.
	DebugLevelRestoreAttempted DebugLevelState = "restore-attempted"
	// DebugLevelRestored: the effective readback equals the prior level.
	// Terminal.
	DebugLevelRestored DebugLevelState = "restored-verified"
	// DebugLevelConflict: the daemon showed a level that is neither the
	// prior nor the window's own, so it was left as it is. Terminal.
	DebugLevelConflict DebugLevelState = "conflict"
	// DebugLevelTargetChanged: the daemon's container generation was
	// stopped, restarted, replaced or removed, or the cluster terminated, so
	// the window's process is gone and nothing was restored. Terminal.
	DebugLevelTargetChanged DebugLevelState = "target-changed"
	// DebugLevelUnconfirmed: restoration could not be verified, for example
	// with a paused daemon or an unavailable socket; Restore may be retried.
	DebugLevelUnconfirmed DebugLevelState = "unconfirmed"
	// DebugLevelNotApplied: the request was refused before any native
	// step, so nothing was applied and there is nothing to restore. Reason
	// holds the refusal. A window that may have been applied never reports
	// this state.
	DebugLevelNotApplied DebugLevelState = "not-applied"
)

type DebugLevelTransition added in v0.4.0 

type DebugLevelTransition struct {
	State    DebugLevelState `json:"state"`
	At       time.Time       `json:"at"`
	Readback string          `json:"readback,omitempty"`
	Error    string          `json:"error,omitempty"`
}

DebugLevelTransition is one recorded state change of a window.

type DetachedOSD added in v0.4.0 

type DetachedOSD struct {
	// contains filtered or unexported fields
}

DetachedOSD is the retained store and MON registration of an OSD whose container OSDContainer.Detach removed. DetachedOSD.Reopen of the same cluster starts a new container on it once. The cluster owns the store: Terminate discards it with the owned volume, and the MONs keep the registration down meanwhile.

func (*DetachedOSD) ClusterFSID added in v0.4.0 

func (d *DetachedOSD) ClusterFSID() string

ClusterFSID returns the cluster FSID that the retained store holds.

func (*DetachedOSD) ID added in v0.4.0 

func (d *DetachedOSD) ID() int

ID returns the retained numeric OSD ID.

func (*DetachedOSD) Image added in v0.4.0 

func (d *DetachedOSD) Image() string

Image returns the image the detached OSD ran from, which DetachedOSD.Reopen uses again.

func (*DetachedOSD) Placement added in v0.4.0 

func (d *DetachedOSD) Placement() OSDConfig

Placement returns the CRUSH location that DetachedOSD.Reopen starts the OSD in.

func (*DetachedOSD) Reopen added in v0.4.0 

func (d *DetachedOSD) Reopen(ctx context.Context, configs ...OSDOptions) (*OSDContainer, error)

Reopen starts a new container on the store and registration that Detach retained, and waits for the OSD to be up/in. It returns a new *OSDContainer with the same ID, UUID and placement; the previous handle of the OSD keeps naming the removed container and is stale.

Before any write it requires the store's identity files, keyring and block inode and size to be unchanged since the detach, the MON registration to hold the same UUID and be down, and the MONs' key for the OSD to equal the stored one. A refusal there, a failed cleanup of the stale admin socket, or a canceled context before the container is requested leaves the OSD detached for another attempt; none of them dispatches a creation request. The cleanup runs under the supervisor. If its end stays unconfirmed, the detached OSD records it and every later Reopen first has to stop or confirm it, so that a late cleanup cannot remove the socket of the replacement; until then Reopen refuses without creating a container. Once the new container is requested the detached OSD is consumed: an error returns the new handle, which the cluster owns. If that handle has a container, detach it again, remove it or terminate the cluster. If the creation request returned no container, its reply may have been lost after the runtime accepted it, so it is never replayed and the handle has no container to detach: Detach and Reopen refuse it, Remove refuses the last OSD, and only Terminate recovers it. Reopen does not reconcile a container that such a lost reply may have created. The store is started through the ordinary bootstrap, which validates the stored identity again and never formats a store that has its ready marker.

With no option the new container runs the image the detached OSD ran. OSDOptions.WithImage starts it from another image instead, for example a newer release that opens the retained store; the store is not checked against that image's release, and nothing makes the change reversible. WithPlacement is refused: the registration keeps its placement.

func (*DetachedOSD) UUID added in v0.4.0 

func (d *DetachedOSD) UUID() string

UUID returns the retained OSD UUID that the store and its MON registration hold.

type DiagnosticArtifact 

type DiagnosticArtifact struct {
	Kind        string `json:"kind"`
	Name        string `json:"name"`
	Role        string `json:"role,omitempty"`
	ContainerID string `json:"container_id,omitempty"`
	Data        string `json:"data,omitempty"`
	Truncated   bool   `json:"truncated,omitempty"`
	Error       string `json:"error,omitempty"`
}

DiagnosticArtifact retains one observation, including incomplete output or an individual failure. Data is text; inspect observations are JSON when not truncated. Names and order are stable, but observations are not transactional.

type DiagnosticsConfig 

type DiagnosticsConfig struct {
	Timeout              time.Duration
	OperationTimeout     time.Duration
	MaxOutputBytes       int
	LogTail              int
	Concurrency          int
	MaxContainers        int
	AdditionalContainers []DiagnosticsContainer
	RedactValues         []string
}

DiagnosticsConfig bounds a read-only collection. Zero fields select a one minute total timeout, ten seconds per operation, 64 KiB per artifact, the last 1000 log lines, four workers and at most 128 containers. Additional containers are caller-owned clients or mirror daemons; collection never adopts them. RedactValues removes additional opaque application secrets. Redaction is best effort: arbitrary log contents still require review before sharing a report.

The limits are 5 minutes total, 1 minute per operation, 1 MiB per artifact, 10000 log lines, 8 workers, 128 containers, 64 additional containers and 128 redaction values of up to 4096 bytes; invalid limits are refused before collection, without a report. Artifact kinds are inventory, container- inspect, container-logs, network-inspect and ceph.

type DiagnosticsContainer 

type DiagnosticsContainer struct {
	Role      string
	Name      string
	Container testcontainers.Container
}

DiagnosticsContainer explicitly adds an external container to a report. Name and Role describe it; Container is only inspected and its logs read.

type DiagnosticsReport 

type DiagnosticsReport struct {
	SchemaVersion int                  `json:"schema_version"`
	StartedAt     time.Time            `json:"started_at"`
	FinishedAt    time.Time            `json:"finished_at"`
	HostNetwork   bool                 `json:"host_network"`
	Closed        bool                 `json:"closed"`
	Complete      bool                 `json:"complete"`
	Artifacts     []DiagnosticArtifact `json:"artifacts"`
}

DiagnosticsReport is an independent, JSON-serializable snapshot. Complete means every requested observation succeeded without byte truncation, not that Ceph is healthy or that replication has completed. Logs cover only LogTail.

type EntityAddress added in v0.2.0 

type EntityAddress struct {
	// Type is the protocol: "v2" for msgr2, "v1" for the legacy protocol, or
	// "any".
	Type string
	// Addr is the IP address and port as Ceph prints it, such as
	// "172.20.0.3:6800" or "[fd00::3]:6800".
	Addr string
	// Nonce tells apart daemon instances that reuse an address. MONs use 0.
	Nonce uint32
}

EntityAddress is one entry of a Ceph address vector, as a cluster map advertises it.

type ErasureCodeConfig 

type ErasureCodeConfig struct {
	K               int
	M               int
	AllowOverwrites bool
}

ErasureCodeConfig selects data (K) and parity (M) shards on separate failure domains using PoolConfig's placement fields. The fixture uses the portable Jerasure reed_sol_van plugin. Replicas must be zero; zero MinSize selects K+1. An explicit MinSize of K permits I/O without any spare shard during a failure. AllowOverwrites enables partial object writes and is required for RBD and CephFS data pools. Their metadata must remain in a replicated pool. K is at least 2, M at least 1 and K+M at most 128.

type ExpectationResult added in v0.4.0 

type ExpectationResult struct {
	Name      string `json:"name"`
	Required  bool   `json:"required"`
	MinCount  int    `json:"min_count"`
	MaxCount  int    `json:"max_count"`
	Matched   int    `json:"matched"`
	Satisfied bool   `json:"satisfied"`
}

ExpectationResult is how many events one expectation matched.

type Finalization added in v0.4.0 

type Finalization struct {
	// contains filtered or unexported fields
}

Finalization is the single cleanup path that TerminateOnCleanup registers. It runs, in this order: the checks in registration order, the failure archive, the cleanups in reverse registration order, then Terminate, so that a check registered after the test body still decides whether to archive and that a client removed by a cleanup is still there to inspect. Captures registered with Capture are settled after the checks and before the archive. The archive is written when the test has failed by then, so a failure reported by a check or a capture settlement triggers it too, and a cleanup that fails when nothing had failed before triggers an archive of what still runs, labelled after-cleanup. Test cleanups registered after TerminateOnCleanup run before all of this, as Go runs cleanups last-in first-out, so route the removal of clients the archive should see through Cleanup. A check fails the test by returning an error, by panicking or through t. Adding a step after the finalization has started fails the test.

func (*Finalization) Capture added in v0.4.0 

func (f *Finalization) Capture(name string, session CaptureSettler)

Capture registers a packet capture that the finalization settles after the checks and before the failure archive, so that a check or a failed test body that never called Stop still ends the capture before the archive and before any Cleanup removes a caller-owned target.

The settlement is a phase of its own. It does not use CheckTimeout or the expired test context: each capture gets a fresh context bounded by its SettlementAllowance, and the finalization waits for the settlement to return instead of abandoning it, because the capture's target must not be removed underneath a running settlement. An explicit Stop that already ran is not repeated; its report is the one kept. Settlement errors fail the test, and the report is kept for the archive even then. Nil sessions (including a typed nil pointer) and registration after finalization started fail the test instead.

With FinalizationConfig.Archive the archive keeps each settled report as capture-<n>-<name>.json, n being the registration order, and the manifest lists each settlement under captures with its error, failed_test, allowance and reported fields; a settlement failure marks the trigger capture_failed. The archived report names its output file by base name only, and its strings are masked like the manifest's errors (the configured RedactValues and the recognized secret patterns). Only metadata is archived: this never copies or retains the pcap file itself, which CaptureWithFile does on explicit consent.

func (*Finalization) CaptureWithFile added in v0.4.0 

func (f *Finalization) CaptureWithFile(name string, session CaptureFileSource, maxBytes int64)

CaptureWithFile registers a capture with explicit consent to copy its raw, unsanitized packet payloads and endpoint identifiers into the failure archive. Use Capture for metadata alone. maxBytes must be positive and at most 64 MiB; at most 16 files and 64 MiB together are admitted per finalization. There is no implicit truncation. Settlement, copying, archive retention and later upload are separate outcomes. A passing run settles without opening the source. Local filesystem calls may finish after a deadline (for example on a stalled filesystem); no background copying is abandoned underneath cleanup.

Each copy shares ArchiveTimeout, checks it between reads, verifies the complete retrieved size/SHA256 and publishes atomically. Invalid/partial pcap structure does not prevent preserving its exact retrieved bytes. The caller retains its original file and must keep it and its directory available until finalization returns. If cleanup produces the first failure, copying runs after cleanup and reports an unavailable source rather than recovering bytes that were removed.

func (*Finalization) Check added in v0.4.0 

func (f *Finalization) Check(name string, check func(context.Context) error)

Check adds a post-test check. Checks run before the archive decision, in registration order, while the cluster and every client still run. A check that returns an error, or panics, fails the test and triggers the archive; a check may also report through the test itself, including with t.Fatal, which ends only the check. Adding a check after finalization has started fails the test instead.

func (*Finalization) Cleanup added in v0.4.0 

func (f *Finalization) Cleanup(name string, cleanup func(context.Context) error)

Cleanup adds a cleanup that runs after the archive and before Terminate, in reverse registration order, such as the removal of a caller-owned client. Its error, or panic, fails the test; when nothing has failed before, it also triggers an archive of what still runs.

func (*Finalization) TrackDebugLevel added in v0.4.0 

func (f *Finalization) TrackDebugLevel(handle DebugLevelReceipter)

TrackDebugLevel registers a debug level window for the manifest. Every call is a separate window, in registration order, even for the same daemon, option and generation: windows are never merged or deduplicated. Register the handle right after override.SetDebugLevel returns, including the non-nil handle that comes with an apply error. A nil interface, a typed nil (a nil pointer, map, slice, func, chan or interface dynamic value), a zero-value handle, or any receipter whose receipt has no State is recorded as absent; Receipt is not called on a typed nil. Other panics in a foreign Receipt are not recovered, never as a restored window. A receipt in state DebugLevelNotApplied is recorded as present, with its Reason.

The manifest keeps a copy of the window's receipt when the failure archive is taken, labeled with the archive's phase, and a second copy after the cleanups, so that a restoration run by Finalization.Cleanup is a separate, later record. Taking a copy reads only the handle's local state; it never applies or restores a level and never queries the daemon. Passing runs archive nothing. Registering after finalization started fails the test.

func (*Finalization) Verify added in v0.4.0 

func (f *Finalization) Verify(ctx context.Context, config VerificationConfig) (*Verification, error)

Verify activates a post-test verification of the cluster and adds a check that evaluates it, so that a verification that is not clean fails the test and, with FinalizationConfig.Archive, triggers the failure archive before any cleanup. Activation records the baseline: the selected container generations (the cluster's daemon containers and config.AdditionalContainers), the crash IDs already in their crash directories, and the start of the window on the Docker engine's clock. It returns an error when the configuration is invalid, when the engine cannot be reached or after finalization has started.

The check fails the test both for an unexpected and for an incomplete verification. It never changes the cluster: it restarts, archives and removes nothing, and a failure never skips the cleanups or Terminate.

type FinalizationConfig added in v0.4.0 

type FinalizationConfig struct {
	// Archive enables the failure archive: when the test has failed, a
	// check failed or a finalization cleanup failed, the diagnostics, crash
	// metadata and a manifest are written before the cluster is terminated.
	// A passing run collects nothing.
	Archive bool
	// ArchiveDirectory is the root under which each archive gets its own
	// directory, named after the test, the cluster and a random execution
	// ID. Empty selects the test's ArtifactDir, which go test keeps only
	// with its -artifacts flag; otherwise it is removed after the test.
	ArchiveDirectory string
	// Scenario labels the manifest, for example with a batch or profile name.
	Scenario string
	// CheckTimeout bounds all checks together (default 2 minutes),
	// ArchiveTimeout the whole archive with both collectors and every write
	// (default 3 minutes), and CleanupTimeout the cleanups together and
	// Terminate separately (default 3 minutes). Each is at most 10 minutes;
	// invalid fields are refused before anything is registered.
	CheckTimeout   time.Duration
	ArchiveTimeout time.Duration
	CleanupTimeout time.Duration
	// Diagnostics and CrashMetadata configure the two collectors; select
	// caller-owned clients or mirror daemons with their AdditionalContainers.
	Diagnostics   DiagnosticsConfig
	CrashMetadata CrashMetadataConfig
}

FinalizationConfig selects what the cleanup that TerminateOnCleanup registers does before it terminates the cluster. Zero durations select two minutes for the checks together, three minutes for the failure archive, and three minutes for the cleanups together and again for Terminate; each is at most ten minutes.

type FullRatioSnapshot 

type FullRatioSnapshot struct {
	FSID   string
	Epoch  uint32
	Ratios FullRatios
}

FullRatioSnapshot is one native OSDMap observation. Epoch is not a generation for these thresholds: unrelated OSDMap changes can advance it too. The values describe map policy, not whether any OSD has reported the corresponding state.

type FullRatios 

type FullRatios struct {
	NearFull, BackfillFull, Full float64
}

FullRatios are global OSDMap utilization thresholds, expressed as fractions of capacity. These are separate from central configuration and pool quotas. Native OSDMap storage uses float32; observations retain its normalized value.

type HealthCheck 

type HealthCheck struct {
	Severity, Summary string
	Count             int64
	Details           []string
	Muted             bool
}

HealthCheck reports the native summary count and detail messages. Count is signed and need not equal the number of details, which Ceph may truncate.

type HealthMute 

type HealthMute struct {
	Code, Summary, ExpiresAt string
	Count                    int64
	Sticky                   bool
}

HealthMute is a native stored mute, which can outlive its current check. ExpiresAt is empty for no TTL, otherwise the original Ceph timestamp: YYYY-MM-DDTHH:MM:SS.ffffff+HHMM (or -HHMM), or seconds.ffffff for an early epoch below 315360000 seconds. Expiry is not compared with the caller's clock; an expired or cleared nonsticky mute can remain visible before the MON tick.

type HealthSnapshot 

type HealthSnapshot struct {
	FSID, Status string
	Checks       map[string]HealthCheck
	Mutes        []HealthMute
}

HealthSnapshot is one MON health observation. Status reflects the worst unmuted check; HEALTH_OK can therefore include muted warnings or errors. Checks retain native, case-sensitive codes, including unknown future codes.

type ImageOptions added in v0.4.0 

type ImageOptions struct {
	// contains filtered or unexported fields
}

ImageOptions is the image group of the Ceph configuration: a plain value with chainable, value-returning methods. It is not a customizer. Pass it to Run through its one grouped option function, WithImageOptions, which Run applies and validates like any Option before it creates a resource. The zero value changes nothing. 

cluster, err := ceph.Run(ctx, "", ceph.WithImageOptions(ceph.ImageOptions{}.WithImageSet(images)))

Each method sets one field. Within one value a later call of the same method replaces the earlier one. Run refuses a field that two WithImageOptions options both set, even when they agree, so argument order never decides a conflict. The pattern for every configuration group, and the composition rules, are in docs/contributing/configuration-composition.md.

func (ImageOptions) WithCephFSImage added in v0.4.0 

func (o ImageOptions) WithCephFSImage(image string) ImageOptions

WithCephFSImage selects the image of the CephFS role: every metadata server of the cluster, those started by Run with cephfs.WithFilesystems as well as those started later. Without it MDS daemons use the CephFS role of the cluster's images, which is the Run image unless WithImageSet selects another. FactsOf(c).Images.CephFS reports it, the usual image argument of cephfs/mirror.Run. With WithImageSet it replaces the set's CephFS role. Use matching Ceph versions. A blank image is refused.

func (ImageOptions) WithImageSet added in v0.4.0 

func (o ImageOptions) WithImageSet(images ImageSet) ImageOptions

WithImageSet selects explicit role images. The set is the one authoritative image configuration: Essential starts the MON, MGR and OSD daemons, so the image argument of Run must be empty or equal to Essential and is not a second input. An All set supplies every role. A role that a requested feature needs and the set leaves empty is refused; there is no fallback to Essential. An explicit empty set is refused, not read as unspecified. WithCephFSImage and WithRGWImage replace the matching role of the set. The ImageSet is copied.

func (ImageOptions) WithRGWImage added in v0.4.0 

func (o ImageOptions) WithRGWImage(image string) ImageOptions

WithRGWImage selects the image of the RGW role: every gateway of the cluster, those started by Run with rgw.WithGateways as well as those started later with rgw.Start. Without it gateways use the RGW role of the cluster's images, which is the Run image unless WithImageSet selects another. FactsOf(c).Images.RGW reports it, so the multisite setup clients use it too; multisite gateways take their image as an argument of multisite.Run instead. With WithImageSet it replaces the set's RGW role. Use matching Ceph versions. A blank image is refused.

type ImageSet added in v0.4.0 

type ImageSet struct {
	Essential string
	Client    string
	CephFS    string
	RBDAux    string
	RGW       string
	All       string
}

ImageSet selects the roles runtime contract. All explicitly supplies every role from one image. Otherwise Essential is required and optional roles are required when their feature is used; there is no fallback to Essential.

type InconsistentObject 

type InconsistentObject struct {
	Name, Namespace  string
	Errors           []string
	UnionShardErrors []string
	Shards           []InconsistentShard
}

InconsistentObject is one object reported by the last deep scrub of a PG. Errors are object-level native error names and UnionShardErrors combine the per-shard names, such as read_error or data_digest_mismatch.

type InconsistentShard 

type InconsistentShard struct {
	OSD, Shard int
	Primary    bool
	Errors     []string
}

InconsistentShard retains one native shard report. Shard is -1 for replicated pools.

type LogEvent added in v0.4.0 

type LogEvent string

LogEvent is the kind of a log line that verification evaluates. Lines of no kind are not events. 

const (
	// LogAssertion is a failed Ceph assertion or abort ("FAILED ceph_assert",
	// "ceph_abort").
	LogAssertion LogEvent = "assertion"
	// LogFatalSignal is a daemon's report of a fatal signal ("Caught
	// signal").
	LogFatalSignal LogEvent = "fatal-signal"
	// LogClusterError, LogClusterSecurity and LogClusterWarning are cluster
	// log entries of level ERR, SEC and WRN, which every daemon writes to
	// its output and the MON writes again as the cluster log. The level is
	// the one of the entry's own log envelope (see LogExpectation.Pattern);
	// a marker in the message of a lower level entry makes no event, and a
	// line with a marker but no recognized envelope is a gap, not an event.
	// The entry's channel (such as cluster or audit) does not select the
	// kind: a level ERR, SEC or WRN entry of any channel is an event.
	LogClusterError    LogEvent = "cluster-error"
	LogClusterSecurity LogEvent = "cluster-security"
	LogClusterWarning  LogEvent = "cluster-warning"
)

type LogExpectation added in v0.4.0 

type LogExpectation struct {
	// Name identifies the expectation in the report.
	Name string
	// Required makes the expectation fail the verification unless at least
	// MinCount events (at least one) match it. Without Required, MinCount
	// must be zero.
	Required bool
	MinCount int
	// MaxCount limits how many events it permits; zero is no limit. Further
	// events go on to the next expectations and are unexpected when none
	// permits them.
	MaxCount int
	// Daemon is the source as the report names it, such as "osd.0".
	Daemon string
	// Event is the kind of line.
	Event LogEvent
	// Pattern is a regular expression (RE2 syntax), searched unanchored in
	// the whole redacted line without its Docker timestamp, including the log
	// envelope; anchor it with ^ and $ to require a full match. Event, not Pattern, selects the structural kind.
	Pattern string
}

LogExpectation permits, or with Required requires, log events that match all of its non-empty criteria. Pattern is required.

type MGRModuleState 

type MGRModuleState struct {
	Name                             string
	Enabled, AlwaysOn, ForceDisabled bool
	Available, CanRun                bool
	ErrorString                      string
}

MGRModuleState combines configured module membership with the active MGR's advertised module metadata. Available means metadata is present, not that commands are ready. CanRun is meaningful only when Available is true. AlwaysOn modules can be force-disabled outside this fixture; Enabled reflects ForceDisabled too. ErrorString is the native dependency/load diagnostic.

type ManagerContainer 

type ManagerContainer struct {
	testcontainers.Container
	DaemonName string
	// contains filtered or unexported fields
}

ManagerContainer is an owned active or standby manager.

func (*ManagerContainer) AdminSocket added in v0.4.0 

func (m *ManagerContainer) AdminSocket(ctx context.Context, args ...string) ([]byte, error)

AdminSocket runs a command on the admin socket of this MGR, which its cluster must still own, and returns the command's output. args are the command words, such as "status", "version" or "config", "get", "osd_heartbeat_grace"; "help" lists the daemon's commands.

The command does not use the Messenger or Cephx: it reaches the daemon through its local socket, so it works while the daemon cannot reach a monitor and reports what the daemon itself has applied, for example the newest OSDMap epoch in an OSD's "status". It runs in the control-plane container, which shares the daemons' run directory, so it works whatever the daemon's own image contains. The daemon must be running; a stopped or removed daemon returns an error. A paused daemon accepts the connection but does not answer, so the call waits for ctx or the cluster's startup timeout, which also bounds every call. When the call ends early, AdminSocket stops its CLI in the control container before it returns, including one that Docker has not started yet. A nonzero exit returns the output with an error.

func (*ManagerContainer) Image added in v0.4.0 

func (m *ManagerContainer) Image() string

Image returns the image this MGR container was started from: the DaemonOptions.WithImage override, or the cluster's essential image.

func (*ManagerContainer) Remove added in v0.4.0 

func (m *ManagerContainer) Remove(ctx context.Context) error

Remove deletes this owned manager after another running, registered candidate is available. Add a replacement before removing the last candidate. On failure the handle remains owned for retry or cluster cleanup. Do not edit these manager identities externally while a topology operation is in progress. A handle of a removed manager returns an error.

func (*ManagerContainer) RotateKey added in v0.4.0 

func (m *ManagerContainer) RotateKey(ctx context.Context) error

RotateKey replaces the Cephx key of this running MGR without a restart, as cephadm's daemon key rotation does:

  1. auth get-or-create-pending creates the new key on the MONs, which then accept both keys.
  2. The daemon's keyring files in its container get the new key, so that a later restart authenticates with it.
  3. ceph tell <daemon> rotate-key hands the key to the running daemon.
  4. auth commit-pending makes it the entity's only key.

MONs share the mon. key, so *MonitorContainer has no RotateKey.

The MONs accept both keys until step 4, and they also commit the pending key on their own when the daemon authenticates with it, after a restart or a handoff. A failure before step 4 therefore keeps both keys valid, and the next call finishes the rotation with the same key; a failed step 4 is reported with the outcome that the MONs show once settled. A call returns only once none of its commands can still change the keys or the keyring files, or after recording such commands for the daemon: the next call stops them, waits for them and settles the MONs for them before it changes anything, and fails while it cannot. After a handoff that may still reach the daemon, a later call finishes the rotation with the same key and generates another key only once the daemon's container has restarted. A handle of a removed MGR returns an error.

func (*ManagerContainer) Version added in v0.4.0 

func (m *ManagerContainer) Version(ctx context.Context) (string, error)

Version returns the release that this running MGR reports itself, such as "20.2.4". See OSDContainer.Version.

type ManagerMetrics added in v0.4.0 

type ManagerMetrics struct {
	// URL is the scraped endpoint.
	URL string
	// Types maps a metric family to the type in its # TYPE line.
	Types map[string]string
	// Help maps a metric family to the text of its # HELP line.
	Help map[string]string
	// Samples are the samples in the order the module printed them.
	Samples []MetricSample
}

ManagerMetrics is one scrape of the active manager's prometheus module.

func (ManagerMetrics) Select added in v0.4.0 

func (m ManagerMetrics) Select(name string, labels map[string]string) []MetricSample

Select returns the samples named name whose labels include every label in labels; nil labels select every sample of that name.

func (ManagerMetrics) Value added in v0.4.0 

func (m ManagerMetrics) Value(name string, labels map[string]string) (float64, error)

Value returns the value of the one sample that Select finds; none, or more than one, is an error.

type ManagerStatus 

type ManagerStatus struct {
	Available  bool   `json:"available"`
	ActiveName string `json:"active_name"`
	ActiveGID  uint64 `json:"active_gid"`
	Standbys   []struct {
		Name string `json:"name"`
		GID  uint64 `json:"gid"`
	} `json:"standbys"`
}

ManagerStatus reports active ownership and registered standbys.

type MessengerConnectionMode added in v0.2.0 

type MessengerConnectionMode string

MessengerConnectionMode is one Messenger v2 connection mode. 

const (
	// MessengerCRC checks frame integrity with CRC32C and does not encrypt.
	MessengerCRC MessengerConnectionMode = "crc"
	// MessengerSecure encrypts frames with AES-GCM.
	MessengerSecure MessengerConnectionMode = "secure"
)

type MessengerMode 

type MessengerMode int

MessengerMode selects, at bootstrap, the Messenger protocols that every daemon listens on and advertises, and whether the connection modes stay configurable. Both are fixed for the life of the cluster. 

const (
	// MessengerDefault retains Ceph's own Messenger defaults and v1/v2 MON
	// addresses. It does not imply that every connection uses encryption.
	MessengerDefault MessengerMode = iota
	// MessengerV2Secure allows only encrypted Messenger v2 connections, with
	// no CRC-mode negotiation or legacy Messenger v1 listener.
	MessengerV2Secure
	// MessengerV2Only listens on and advertises only Messenger v2, like
	// MessengerV2Secure, but keeps the connection modes configurable: Ceph's
	// defaults, NetworkConfig.WithMessengerPolicy, override.SetMessengerPolicy and
	// ClientContainerOptions.WithMessengerPolicy select them, as with MessengerDefault.
	MessengerV2Only
)

type MessengerPolicy added in v0.2.0 

type MessengerPolicy struct {
	// Monitors sets ms_mon_service_mode for the MONs.
	Monitors []MessengerConnectionMode
	// Managers sets ms_mon_service_mode for the MGRs: Ceph applies the MON
	// option to connections from clients to MGRs.
	Managers []MessengerConnectionMode
	// OSDs sets ms_service_mode for the OSDs.
	OSDs []MessengerConnectionMode
}

MessengerPolicy selects, for each daemon type, the Messenger v2 connection modes that it accepts from clients. The order of a list does not matter: the connection uses the first mode in the client's order of preference that the daemon accepts. A nil field keeps Ceph's default, which accepts both modes. Connections between daemons keep their defaults, so every policy leaves the cluster itself working.

A client that offers no accepted mode fails at once against a MON (EACCES; the ceph CLI prints RADOS permission denied and exits with 13), but gets no error from a MGR or an OSD and keeps retrying until its own timeout, so bound such calls. The MON client's handle_auth_bad_method log line is not reliable in the ceph CLI's output, which exits without shutting librados down.

type MetricSample added in v0.4.0 

type MetricSample struct {
	// Name is the sample's metric name, such as ceph_osd_op_w.
	Name string
	// Labels are the sample's labels, such as ceph_daemon="osd.0".
	Labels map[string]string
	// Value is the sample's value, which may be NaN or ±Inf.
	Value float64
	// Timestamp is the sample's timestamp in milliseconds, or 0 without one.
	Timestamp int64
}

MetricSample is one sample of the Prometheus text exposition format.

type MonitorContainer 

type MonitorContainer struct {
	testcontainers.Container
	DaemonName string
	// contains filtered or unexported fields
}

MonitorContainer is an owned quorum member. Stop/Start preserves its store; MonitorContainer.Remove explicitly removes membership and deletes the container.

func (*MonitorContainer) AdminSocket added in v0.4.0 

func (m *MonitorContainer) AdminSocket(ctx context.Context, args ...string) ([]byte, error)

AdminSocket runs a command on the admin socket of this MON, which its cluster must still own, and returns the command's output. args are the command words, such as "status", "version" or "config", "get", "osd_heartbeat_grace"; "help" lists the daemon's commands.

The command does not use the Messenger or Cephx: it reaches the daemon through its local socket, so it works while the daemon cannot reach a monitor and reports what the daemon itself has applied, for example the newest OSDMap epoch in an OSD's "status". It runs in the control-plane container, which shares the daemons' run directory, so it works whatever the daemon's own image contains. The daemon must be running; a stopped or removed daemon returns an error. A paused daemon accepts the connection but does not answer, so the call waits for ctx or the cluster's startup timeout, which also bounds every call. When the call ends early, AdminSocket stops its CLI in the control container before it returns, including one that Docker has not started yet. A nonzero exit returns the output with an error.

func (*MonitorContainer) Image added in v0.4.0 

func (m *MonitorContainer) Image() string

Image returns the image this MON container was started from: the DaemonOptions.WithImage override, or the cluster's essential image.

func (*MonitorContainer) Remove added in v0.4.0 

func (m *MonitorContainer) Remove(ctx context.Context) error

Remove deletes this owned monitor. It preserves a working quorum: add a replacement first when a stop would leave fewer than a majority of the current map running. A handle of a removed monitor returns an error.

It refuses a removal that would leave fewer than a majority of the current map running; add a replacement first. MON a can be removed too; the embedded container of Container is then gone and fixture commands use the control container. If the mon remove reply is lost, the call returns an error without resending; calling Remove again with a fresh context reads the current membership, skips the completed native step and finishes the container and configuration cleanup. MON a is described by a fresh descriptor on every listing, so a handle is matched by name and container, not by pointer identity.

func (*MonitorContainer) Replace added in v0.4.0 

func (m *MonitorContainer) Replace(ctx context.Context, opts ...DaemonOptions) (*MonitorContainer, error)

Replace starts this MON in a new container on the store and address it keeps, and waits for it to rejoin the quorum. The membership does not change: the MON keeps its name, rank and monmap entry. It needs MonitorStorage.WithVolume. DaemonOptions.WithImage starts the new container from another image, for example a newer release; with no option it uses the image this MON ran. The store is not checked against that image's release, and nothing makes the change reversible.

Before it touches the running MON it refuses a replacement that would leave fewer than a majority of the monmap in quorum, a store that is not attached to this handle's container, and a store with no recorded address. It then removes the old container, confirms that nothing mounts the volume, and creates the new one with the recorded IPv4 address, in a mode that refuses to format a missing store. It returns the new handle; this handle names the removed container and is stale, also for MON a, whose embedded container of Container is then gone and whose fixture commands use the control container.

When the old container is gone and the new one fails, the error comes with the new handle if a container exists: the store and its volume stay owned, nothing is formatted, and Replace on that handle or Terminate continues. A creation whose reply was lost is never replayed; only Terminate settles it.

func (*MonitorContainer) Version added in v0.4.0 

func (m *MonitorContainer) Version(ctx context.Context) (string, error)

Version returns the release that this running MON reports itself, such as "20.2.4". See OSDContainer.Version.

type MonitorStorage added in v0.4.0 

type MonitorStorage struct {
	// contains filtered or unexported fields
}

MonitorStorage is the MON storage group of the Ceph configuration: where every MON of the cluster keeps its store, including MONs added later. It is a plain value with chainable, value-returning methods, not a customizer; pass it to Run through its one grouped option function, WithMonitorStorage. The zero value changes nothing. 

cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithMonitorStorage(ceph.MonitorStorage{}.WithVolume()))

Run refuses a second WithMonitorStorage that sets the volume again.

func (MonitorStorage) WithVolume added in v0.4.0 

func (s MonitorStorage) WithVolume() MonitorStorage

WithVolume keeps each MON's store in a Docker volume of its own that the cluster owns, mounted at the MON data directory, instead of the container's writable layer. The volume outlives the MON's container, so MonitorContainer.Replace can start a new container, from another image if asked, on the same store and address. Terminate removes the volumes. It needs IPv4 addresses: the replacement binds the address the monmap pins. On a bridge that is the MON's IPv4 address on the public network; on the host network it is the public address and the MON's ports, which the replacement holds from the removal of the old container until the start of the new one.

type NativeCommand added in v0.2.0 

type NativeCommand struct {
	// Argv is the program and its arguments. It runs without a shell.
	Argv []string
	// Timeout bounds the command's run time, at most
	// MaxNativeCommandTimeout. The caller's context can shorten it.
	Timeout time.Duration
	// MaxStdout and MaxStderr bound each output stream inside the container,
	// each at most MaxNativeCommandOutput bytes.
	MaxStdout, MaxStderr int
	// Label names the command in errors, which never contain its arguments
	// or output. It defaults to "native command".
	Label string
	// Input, when set, is the command's standard input, and MaxInput, at
	// most MaxNativeCommandInput, bounds the bytes written to it in all.
	// Without Input the command's standard input is empty, and MaxInput must
	// be 0. An Input serves one command.
	Input    *NativeInput
	MaxInput int
}

NativeCommand is one native command that RunNativeCommand runs under a supervisor inside a container.

type NativeCommandResult added in v0.2.0 

type NativeCommandResult struct {
	Stdout, Stderr []byte
	// ExitCode is the command's exit code; a nonzero one is not an error.
	ExitCode int
	// TimedOut means that the supervisor killed the command's process group
	// at its deadline; the output is what it wrote until then.
	TimedOut bool
	// StdoutTruncated and StderrTruncated mean that a stream exceeded its
	// limit; the result holds its first bytes.
	StdoutTruncated, StderrTruncated bool
	// InputWritten counts the bytes of Input that the input stream to the
	// container took, and InputDelivered those that the supervisor wrote into
	// the command's standard input; the command may not have read all of
	// them. They differ when the command ended, or closed its standard input,
	// before it took everything. InputEOF means that the command's standard
	// input ended after all InputWritten bytes, because Input was closed.
	InputWritten, InputDelivered int64
	InputEOF                     bool
}

NativeCommandResult is the outcome of a supervised native command. Stdout and Stderr are separate, raw and binary-safe.

func (NativeCommandResult) Complete added in v0.2.0 

func (r NativeCommandResult) Complete() bool

Complete reports whether the command finished within its time and output limits, so that its output is complete.

type NativeCommandUnstoppedError added in v0.2.0 

type NativeCommandUnstoppedError struct {
	// contains filtered or unexported fields
}

NativeCommandUnstoppedError reports a native command that RunNativeCommand, or rgw.Gateway.Admin, could not confirm as stopped after its call ended early: it may still run in its container. Stop retries the confirmation; Terminate also tries it.

func (*NativeCommandUnstoppedError) Error added in v0.2.0 

func (e *NativeCommandUnstoppedError) Error() string

func (*NativeCommandUnstoppedError) Stop added in v0.2.0 

func (e *NativeCommandUnstoppedError) Stop(ctx context.Context) error

Stop stops the command, or confirms that it ended with its container, and returns nil once the command can no longer run: after the container was removed, or once it runs again after a restart. A stopped container cannot confirm anything until then. Each attempt has its own NativeCommandStopTimeout, and waits for a concurrent attempt within ctx. Stop is idempotent and can be retried, also after Terminate.

func (*NativeCommandUnstoppedError) Unwrap added in v0.2.0 

func (e *NativeCommandUnstoppedError) Unwrap() error

type NativeFailure added in v0.2.0 

type NativeFailure struct {
	// Program is the command's program, such as "ceph" or "rbd"; the rest of
	// its arguments is never recorded.
	Program string
	// ExitCode is its exit code.
	ExitCode int
	// Errno is the error code that the tool reported, such as "ENOENT", or
	// "" when it reported none. It is read from the "Error ENOENT:" that
	// starts a line of Ceph and from the last "(2) No such file or directory"
	// of a line of the other tools, never guessed from the exit code.
	Errno string
	// Reason is the tool's own explanation: the lines of its error output
	// without Ceph's log lines, joined with "; ", at most
	// MaxNativeFailureReason bytes, with keys and the secrets that
	// RedactSecrets recognizes replaced. It can name resources of the
	// request, such as a pool.
	Reason string
}

NativeFailure is the status of a native command that ran and exited nonzero. errors.Is matches it with ErrNativeFailure, and with fs.ErrNotExist, fs.ErrPermission or fs.ErrExist when the tool reported ENOENT, EACCES or EPERM, or EEXIST. RunNativeCommand returns a nonzero exit as its result instead; the module's own commands and Ceph return this.

func (*NativeFailure) Error added in v0.2.0 

func (f *NativeFailure) Error() string

func (*NativeFailure) Is added in v0.2.0 

func (f *NativeFailure) Is(target error) bool

Is reports whether target is ErrNativeFailure, or the fs error that matches the reported errno.

type NativeInput added in v0.3.0 

type NativeInput struct {
	// contains filtered or unexported fields
}

NativeInput is the standard input of one NativeCommand while it runs. Write sends data in order, and Close ends the input. The command gets exactly the bytes written, without a shell, an encoding or a terminal in between. Nothing that is written appears in arguments, the environment, files, errors, labels or diagnostics; the command's own output is returned as it wrote it, so a command that must not reveal its input must not echo it.

Inside the container the supervisor reads what arrives at once, up to MaxInput, and passes it on as the command reads, so a command that reads slowly or not at all does not hold up Write. Its methods are safe for concurrent use.

func NewNativeInput added in v0.3.0 

func NewNativeInput() *NativeInput

NewNativeInput returns an open input for one NativeCommand.

func (*NativeInput) Close added in v0.3.0 

func (in *NativeInput) Close() error

Close ends the input: the command gets the end of input after every byte written before. Close is idempotent and never fails; a command that ended before is not affected.

func (*NativeInput) Write added in v0.3.0 

func (in *NativeInput) Write(ctx context.Context, data []byte) error

Write sends data to the command's standard input after the data of earlier writes, and returns once the input stream to the container took all of it. A write before RunNativeCommand starts the command waits for it. ctx bounds that wait; when ctx ends after the stream took the data, Write returns ctx's error, and the command may still get the data, as NativeCommandResult.InputWritten tells. Write copies data before it returns.

A write that would take the input past MaxInput is refused whole with an error that matches ErrInvalidRequest, and the input stays open. Once the input is closed, or the command ended or its input stream failed, Write returns an error that matches ErrNativeInputClosed. An empty write sends nothing.

type NetworkConfig added in v0.4.0 

type NetworkConfig struct {
	// contains filtered or unexported fields
}

NetworkConfig is the network and Messenger group of the Ceph configuration: which addresses the daemons bind and advertise, whether OSD replication traffic uses its own bridge, the host address in host-network mode and the Messenger protocol and connection-mode policy. It is a plain value with chainable, value-returning methods, not a customizer; pass it to Run through its one grouped option function, WithNetworkConfig. The zero value changes nothing, and each field records whether it was set, so an explicit setting (including the default family AddressIPv4 or MessengerDefault) is different from an unspecified one. 

network := ceph.NetworkConfig{}.WithAddressFamily(ceph.AddressDualStack).WithSeparateClusterNetwork()
cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithNetworkConfig(network))

Within one value a later call of the same method replaces the earlier one. Run refuses a field that two WithNetworkConfig options both set, even when they agree, so argument order never decides a conflict. Incompatible combinations (host network with a separate cluster network or a non-IPv4 family, a host address without WithHostNetwork, a Messenger policy with MessengerV2Secure) are refused before any resource is created. Docker execution policy such as WithHostNetwork stays a separate option function. The composition rules are in docs/contributing/configuration-composition.md.

func (NetworkConfig) WithAddressFamily added in v0.4.0 

func (n NetworkConfig) WithAddressFamily(family AddressFamily) NetworkConfig

WithAddressFamily selects the addresses of the Messenger listeners of every MON, MGR, OSD and service daemon. Every family other than AddressIPv4 needs bridge mode, and AddressIPv6 and AddressDualStack a Docker Engine that can allocate IPv6 subnets. With WithSeparateClusterNetwork, AddressIPv6 makes both bridges dual-stack and their IPv6 subnets Ceph's public and cluster networks, and AddressDualStack makes both subnets of each bridge its network; AddressIPv4MappedIPv6 cannot be combined with it.

AddressDualStack sets ms_bind_ipv4, ms_bind_ipv6 and ms_bind_prefer_ipv4, and public_network to both subnets of the bridge. Every address vector has an entry of each family for each enabled Messenger protocol: v2, and v1 unless WithMessengerMode removes it. MON vectors list the IPv6 address first, which keeps a MGR from advertising the IPv6 wildcard address [::] that clients cannot reach; the other daemons list the IPv4 address first. Native clients connect to the first entry, so they reach MGRs, OSDs and MDSs over IPv4; ms_bind_prefer_ipv4 = false in a daemon's section of a WithConfigFile file makes that daemon list its IPv6 address first. Run, AddManager and ManagerContainer.Remove check that the MONs, the active MGR and the up OSDs advertise a concrete address of each family, and fail with ErrUnsupported otherwise. An unknown family is refused.

ClusterFacts.AddressFamily reports the selection. With AddressIPv6 and AddressIPv4MappedIPv6 the generated ceph.conf sets ms_bind_ipv4 = false and ms_bind_ipv6 = true, so MGRs, OSDs and MDSs bind [::] and advertise the address from which the MONs see them; no daemon has an IPv4 Messenger socket, and with AddressIPv4MappedIPv6 the IPv6 sockets accept the IPv4 connections the mapped addresses stand for. With AddressIPv6, Docker forwards an IPv4 connection to a published MON port to the container's IPv4 address, where no MON listens, so run clients in WithClient containers. Ceph prints an IPv4-mapped address in dotted form but parses only hexadecimal groups (::ffff:ac13:2); the fixture writes mon_host in the parseable form, so convert an address the same way before passing it to a Ceph tool. HTTP endpoints such as RGW's keep their addresses. Docker reuses the address of a removed or stopped container, so identify a daemon by its entry's nonce or its GID, not by address alone. See docs/guides/ceph-configuration.md for dual-stack address order and for cutting one family of a path.

func (NetworkConfig) WithHostAddress added in v0.4.0 

func (n NetworkConfig) WithHostAddress(address string) NetworkConfig

WithHostAddress selects a local IPv4 address to bind and advertise in host mode, typically the Docker host's address when Docker is remote. The address must be assigned on the Docker daemon host and reachable from the test process; Run verifies both. Requires WithHostNetwork. An empty, unspecified, multicast, broadcast or non-IPv4 address is refused.

func (NetworkConfig) WithMessengerMode added in v0.4.0 

func (n NetworkConfig) WithMessengerMode(mode MessengerMode) NetworkConfig

WithMessengerMode selects the Messenger policy at bootstrap. MessengerDefault retains Ceph's image defaults and v1/v2 addresses. MessengerV2Secure requires encrypted v2 connections without CRC-mode or v1 fallback, including later daemons and WithClient containers. MessengerV2Only removes the v1 listeners and addresses in the same way but leaves the connection modes to Ceph's defaults and the Messenger policy. This setting does not enable RGW HTTP TLS. An unknown mode is refused. Both settings are fixed for the cluster's lifetime.

In both v2 modes the monmap and mon_host contain only v2 addresses and no daemon listens on a v1 port: in bridge mode the MON publishes only 3300/tcp, so MappedPort(ctx, "6789/tcp") fails, and in host mode one MON port is reserved. Later daemons, WithClient and ClientConfig.WithIdentity containers and ConnectionConfig inherit the mode, and override.SetConfig refuses ms_bind_msgr1 and ms_bind_msgr2 (and, with MessengerV2Secure, the six ms_*_mode options) because the local file would override them. MessengerV2Only sets only ms_bind_msgr1 = false and ms_bind_msgr2 = true; MessengerV2Secure also sets ms_cluster_mode, ms_service_mode, ms_client_mode and their ms_mon_ forms to secure, so a CRC-only client is rejected with AUTH_BAD_METHOD rather than timing out. Cephx stays required in every mode. ClusterFacts.MessengerMode reports the selected mode, not live connections; on Tentacle a daemon's messenger dump admin command reports the mode of each connection.

func (NetworkConfig) WithMessengerPolicy added in v0.4.0 

func (n NetworkConfig) WithMessengerPolicy(policy MessengerPolicy) NetworkConfig

WithMessengerPolicy selects the connection modes that MONs, MGRs and OSDs accept from clients. Run stores the policy in the central configuration right after the first MON starts, before any other daemon, so every daemon starts with it. override.SetMessengerPolicy changes it on a running cluster. It cannot be combined with MessengerV2Secure, whose local configuration fixes every mode. The value keeps its own copy of the lists, so changing the policy's slices afterwards does not change it; a nil list keeps Ceph's default and a non-nil empty list is refused.

func (NetworkConfig) WithSeparateClusterNetwork added in v0.4.0 

func (n NetworkConfig) WithSeparateClusterNetwork() NetworkConfig

WithSeparateClusterNetwork creates a second owned bridge for OSD replication, recovery and heartbeat traffic. MON/MGR/MDS/RGW and WithClient use only the public bridge. Docker selects both subnets; with WithAddressFamily(AddressIPv6) both bridges are dual-stack and Ceph uses their IPv6 subnets. This setting requires bridge mode. Only OSDs join both bridges. ClusterFacts.NetworkName and ClusterFacts.ClusterNetworkName name them, and fault.InterruptNetwork with ClusterNetworkPlane disconnects one OSD's whole cluster endpoint (both families) and keeps its public one.

type NetworkPlane 

type NetworkPlane string

NetworkPlane selects the client/public or OSD replication network. 

const (
	// PublicNetworkPlane is the client/public bridge.
	PublicNetworkPlane NetworkPlane = "public"
	// ClusterNetworkPlane is the OSD replication and heartbeat bridge; with
	// a single bridge it selects the same endpoint as PublicNetworkPlane.
	ClusterNetworkPlane NetworkPlane = "cluster"
)

type OSDAddress added in v0.2.0 

type OSDAddress struct {
	ID     int
	Up, In bool
	// Public serves clients; Cluster carries replication and recovery.
	Public, Cluster []EntityAddress
	// HeartbeatFront and HeartbeatBack carry OSD heartbeats on the public and
	// cluster sides.
	HeartbeatFront, HeartbeatBack []EntityAddress
}

OSDAddress is one OSD's entry in the OSDMap. The address vectors are those of the OSD's last boot; a down OSD keeps them.

type OSDConfig 

type OSDConfig struct {
	Host        string
	Rack        string
	Root        string
	DeviceClass string
}

OSDConfig places a container's OSD in the logical CRUSH hierarchy. Root defaults to the cluster's selected default root, Host defaults to osd-ID, and Rack/DeviceClass are optional. Multiple OSDs may share a host or rack for domain-failure tests.

type OSDContainer 

type OSDContainer struct {
	testcontainers.Container
	ID int
	// contains filtered or unexported fields
}

OSDContainer is one storage daemon backed by a sparse BlueStore file by default, or by process memory with OSDStorage.WithObjectStore(OSDMemStore). The file is container-local unless OSDStorage.WithInMemory or OSDStorage.WithVolume is selected. Stop/Start can be used for failure injection; OSDContainer.Remove drains and purges it.

func (*OSDContainer) AdminSocket added in v0.4.0 

func (o *OSDContainer) AdminSocket(ctx context.Context, args ...string) ([]byte, error)

AdminSocket runs a command on the admin socket of this OSD, which its cluster must still own, and returns the command's output. args are the command words, such as "status", "version" or "config", "get", "osd_heartbeat_grace"; "help" lists the daemon's commands.

The command does not use the Messenger or Cephx: it reaches the daemon through its local socket, so it works while the daemon cannot reach a monitor and reports what the daemon itself has applied, for example the newest OSDMap epoch in an OSD's "status". It runs in the control-plane container, which shares the daemons' run directory, so it works whatever the daemon's own image contains. The daemon must be running; a stopped or removed daemon returns an error. A paused daemon accepts the connection but does not answer, so the call waits for ctx or the cluster's startup timeout, which also bounds every call. When the call ends early, AdminSocket stops its CLI in the control container before it returns, including one that Docker has not started yet. A nonzero exit returns the output with an error.

func (*OSDContainer) Detach added in v0.4.0 

func (o *OSDContainer) Detach(ctx context.Context) (*DetachedOSD, error)

Detach stops and removes this owned OSD's container but keeps its store in the owned volume and its MON registration, so that DetachedOSD.Reopen can start a new container on them. It needs OSDStorage.WithVolume. Detach marks the OSD neither out nor purged, but Ceph marks a down OSD out after mon_osd_down_out_interval; hold noout to keep it in.

Detach admits the native identity, refuses an OSD that a held network or pause fault targets, fences the instance against new native commands and drains its tracked ones, then stops it, waits for down and removes the container. Only after confirmed removal does it read the store's identity and publish the DetachedOSD; this *OSDContainer stays tied to the removed instance. On error the OSD stays owned with its fence: retry Detach with a fresh context or terminate the cluster. Remove refuses a detaching or detached OSD.

Detach only an OSD that is up: a daemon that is still booting can be killed at the end of the stop grace before it marks itself down, and in a one-OSD cluster no peer reports it, so Detach waits until its context ends. A detached OSD leaves Container.OSDs.

func (*OSDContainer) Image added in v0.4.0 

func (o *OSDContainer) Image() string

Image returns the image this OSD container was started from: the OSDOptions.WithImage override, or the cluster's essential image. DetachedOSD.Reopen starts the OSD from the same image again.

func (*OSDContainer) Placement 

func (o *OSDContainer) Placement() OSDConfig

Placement returns a copy of this OSD's resolved logical CRUSH location.

func (*OSDContainer) Remove added in v0.4.0 

func (o *OSDContainer) Remove(ctx context.Context) error

Remove drains this owned OSD, waits for safe-to-destroy, stops it, waits for down, purges it from Ceph and removes its container. It refuses the last OSD. After identity admission, removal fences this Docker instance against new native commands and drains its tracked commands before native mutations. A failed drain retains ownership and its fence for retry; other targets remain admitted. This is permanent removal, not retained-store detach or reopen. Every removal phase verifies the captured registration UUID. External OSD replacement must not race removal: Ceph has no UUID compare-and-swap purge. An uncertain purge response remains tracked; a retry reconciles fresh native absence only after this handle issued purge. Completed purge retries only Docker cleanup. On error retry with a fresh context or terminate the cluster.

Remove acts on this handle's own instance. A handle of a removed or detached OSD, or one that Reopen replaced, returns an error and never acts on the container that owns the ID now. Remove refuses a detaching or detached OSD.

If the remaining OSDs cannot hold the data (replica count or free space), the drain does not finish and the call fails at the timeout; the OSD is left out and reweighted and is not marked in again automatically.

func (*OSDContainer) RotateKey added in v0.4.0 

func (o *OSDContainer) RotateKey(ctx context.Context) error

RotateKey replaces the Cephx key of this running OSD without a restart, as cephadm's daemon key rotation does. The rotation is described at ManagerContainer.RotateKey; the same steps and failure semantics apply. A handle of a removed or detached OSD returns an error.

func (*OSDContainer) SetIn added in v0.4.0 

func (o *OSDContainer) SetIn(ctx context.Context, in bool) error

SetIn changes native OSDMap membership of this owned OSD. It does not stop, purge or change CRUSH capacity weights. Readback verifies the owned UUID and desired state, since native in/out can succeed for nonexistent IDs. WaitForPGClean can wait for recovery while an OSD remains out. Ceph preserves a nonzero map reweight across out/in. External replacement must not race this method; Ceph has no compare-and-swap operation for OSD membership. A handle of a removed or detached OSD returns an error.

func (*OSDContainer) Version added in v0.4.0 

func (o *OSDContainer) Version(ctx context.Context) (string, error)

Version returns the release that this running OSD reports itself, such as "20.2.4", read from its own admin socket. It does not use CephVersion, which describes the first MON and the CLI, so it is the way to see the release of an OSD that OSDOptions.WithImage started from another image. It works under the same conditions as AdminSocket, and a stopped daemon, a stale handle or an unrecognized reply returns an error.

type OSDFlag added in v0.4.0 

type OSDFlag string

OSDFlag is a global OSDMap flag token such as "noout". The constants name the flags that override.SetOSDFlag can change. OSDFlags returns the flags that the cluster reports, which can include other tokens, so the type stays open: a value outside the constants is representable and override.SetOSDFlag refuses it at run time. 

const (
	OSDFlagNoUp        OSDFlag = "noup"
	OSDFlagNoDown      OSDFlag = "nodown"
	OSDFlagNoOut       OSDFlag = "noout"
	OSDFlagNoIn        OSDFlag = "noin"
	OSDFlagNoBackfill  OSDFlag = "nobackfill"
	OSDFlagNoRebalance OSDFlag = "norebalance"
	OSDFlagNoRecover   OSDFlag = "norecover"
	OSDFlagNoScrub     OSDFlag = "noscrub"
	OSDFlagNoDeepScrub OSDFlag = "nodeep-scrub"
	OSDFlagNoSnapTrim  OSDFlag = "nosnaptrim"
	OSDFlagNoAutoscale OSDFlag = "noautoscale"
)

The reversible global OSD flags that override.SetOSDFlag accepts. Composite pause and irreversible compatibility flags are not among them.

func (OSDFlag) String added in v0.4.0 

func (f OSDFlag) String() string

String returns the native flag token.

type OSDObjectStore added in v0.2.0 

type OSDObjectStore string

OSDObjectStore names the object store that OSDs keep their data in. 

const (
	// OSDBlueStore keeps each OSD's data in a sparse BlueStore file. It is the
	// default.
	OSDBlueStore OSDObjectStore = "bluestore"
	// OSDMemStore keeps each OSD's data in the OSD process's memory, with
	// OSDStorage.WithBlockSize as its capacity. A clean stop saves the store to the
	// OSD's data directory and Start loads it again; a killed OSD loses it.
	// It is Ceph's test backend and needs no block file.
	OSDMemStore OSDObjectStore = "memstore"
)

type OSDOptions added in v0.4.0 

type OSDOptions struct {
	// contains filtered or unexported fields
}

OSDOptions is the grouped configuration of AddOSD. The zero value selects every default: the placement that the cluster resolves for the new OSD and the cluster's essential image. Each With method returns a modified copy and leaves its receiver unchanged, so one base value can derive several independent configurations: 

base := ceph.OSDOptions{}.WithPlacement(ceph.OSDConfig{Rack: "rack-a"})
first := base.WithImage(imageA)
second := base.WithImage(imageB)

AddOSD composes any number of values; the rules are in docs/contributing/configuration-composition.md. Every field tracks whether it was set, so an explicit empty setting differs from an unspecified one, and the value holds only strings, so a copy never shares mutable storage.

func (OSDOptions) WithImage added in v0.4.0 

func (o OSDOptions) WithImage(image string) OSDOptions

WithImage starts the one new OSD from image instead of the cluster's essential image, with the same contract as DaemonOptions.WithImage: the image must contain the daemon's binary and the tools of the fixture's startup scripts, nothing here checks its release or compatibility with the running cluster, DetachedOSD.Reopen without WithImage starts the same OSD from the same image again (with WithImage, from that image), and the OSD's handle reports it with Image. The reference must be non-empty and free of whitespace and control characters; an empty image is an explicit invalid selection, not an unspecified one. A later call on the same value replaces the earlier image.

func (OSDOptions) WithPlacement added in v0.4.0 

func (o OSDOptions) WithPlacement(config OSDConfig) OSDOptions

WithPlacement places the new OSD in the logical CRUSH location config describes (OSDConfig documents its defaults). All four fields are set together: an empty field is an explicit request for its default, and WithPlacement(OSDConfig{}) is an explicit default placement. A later call on the same value replaces the earlier placement entirely.

type OSDState 

type OSDState struct {
	ID     int
	UUID   string
	Up, In bool
	Weight float64
}

OSDState reports authoritative OSDMap state for an owned OSD. Weight is the native map reweight in [0,1], not the CRUSH capacity weight. Out OSDs retain their containers, identities and data; absence or a replaced UUID is an error.

type OSDStorage added in v0.4.0 

type OSDStorage struct {
	// contains filtered or unexported fields
}

OSDStorage is the OSD storage group of the Ceph configuration: where and how every OSD of the cluster keeps its data, including OSDs added later. It is a plain value with chainable, value-returning methods, not a customizer; pass it to Run through its one grouped option function, WithOSDStorage. The zero value changes nothing, and each field records whether it was set, so an explicit setting is different from an unspecified one. 

storage := ceph.OSDStorage{}.WithBlockSize(256 << 20).WithObjectStore(ceph.OSDMemStore)
cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithOSDStorage(storage))

Within one value a later call of the same method replaces the earlier one. Run refuses a field that two WithOSDStorage options both set, even when they agree, so argument order never decides a conflict. Incompatible combinations (MemStore with in-memory or volume storage, in-memory with volume storage) are refused before any resource is created. The composition rules are in docs/contributing/configuration-composition.md.

func (OSDStorage) WithBlockSize added in v0.4.0 

func (o OSDStorage) WithBlockSize(size int64) OSDStorage

WithBlockSize sets each sparse BlueStore file's logical size in bytes. The minimum is 64 MiB and the default remains 1 GiB. These are disposable files, not real disks. It is also the capacity of a MemStore OSD. Small functional fixtures can use ClusterConfig.WithNoInitialOSDs and override.SetConfig to skip mClock's startup capacity benchmark before AddOSD; the benchmark can exhaust a small file even without client data. A size below the minimum, including zero, is an explicit invalid setting, not an unspecified one.

func (OSDStorage) WithInMemory added in v0.4.0 

func (o OSDStorage) WithInMemory(maxBytes int64) OSDStorage

WithInMemory keeps all OSD stores in one Docker-managed tmpfs volume. maxBytes is a positive cluster-wide filesystem allocation ceiling, independent of each sparse file's logical WithBlockSize. It does not reserve RAM or limit daemon RSS; tmpfs may swap, and a full volume fails I/O. An owned helper using the essential image preserves data across OSD Stop/Start. Cluster termination discards the volume. Helper loss or Docker host/VM restart can lose data. Requires a Linux Docker engine with local tmpfs volume support, including compatible Docker Desktop engines. No host mount, privileged container or additional image is required. The default remains container-local sparse files. ClusterConfig.WithNoInitialOSDs defers allocation until AddOSD. A non-positive maxBytes is refused. The volume shares the Docker daemon's memory (the Linux VM's on Docker Desktop, the remote host's on a remote engine) with every daemon; size the cap for replicas, metadata, recovery and mClock's startup benchmark, and note that deleting Ceph objects does not necessarily release tmpfs pages. The helper is an idle container without network, created before the first OSD; the cluster refuses to continue with a restarted helper. OSDContainer.Remove discards the purged OSD's directory. Rootless or managed engines may refuse tmpfs volumes.

func (OSDStorage) WithObjectStore added in v0.4.0 

func (o OSDStorage) WithObjectStore(store OSDObjectStore) OSDStorage

WithObjectStore selects the object store of every OSD in the cluster, including OSDs added later. WithBlockSize sets the capacity of a MemStore OSD. MemStore cannot be combined with WithInMemory or WithVolume, which place BlueStore files in shared volumes. fault.InjectObjectDataError needs BlueStore. An unknown store, including the empty one, is refused. MemStore keeps data in the OSD process's memory and saves it to the data directory when the store is unmounted, so the fixture disables Ceph's fast OSD shutdown for MemStore OSDs: a stopped OSD keeps its data across Stop and Start, while a killed OSD or a removed container loses it. ceph osd metadata reports osd_objectstore as memstore.

func (OSDStorage) WithVolume added in v0.4.0 

func (o OSDStorage) WithVolume() OSDStorage

WithVolume keeps classic BlueStore OSD directories in one ordinary Docker-managed local named volume owned by the cluster. Allocation is lazy until the first OSD registration. Unlike tmpfs it needs no live keeper to preserve bytes; an isolated essential-image helper manages fresh/purged paths. Each OSD still gets its own sparse file with WithBlockSize as its logical capacity. The helper has no network and no cluster credentials; a stopped or restarted helper refuses further storage operations before a new OSD is registered and is not replaced automatically, while existing OSDs can still stop and start with their containers. OSDContainer.Remove drains and purges the native registration, removes the container and discards only its directory. Terminate removes every owned OSD and the helper, then deletes the volume without force; a busy or failed deletion keeps ownership for another Terminate. Changed volume metadata is refused before directory operations or deletion, so never modify or attach the volume externally. It cannot be combined with MemStore or WithInMemory. No caller-owned volumes, host mounts or devices are adopted. This is the only backend on which OSDContainer.Detach retains a store and DetachedOSD.Reopen restarts it in a new container; it gives no host-reboot durability, and Terminate also discards detached stores.

type ObjectWatcher 

type ObjectWatcher struct {
	Address  string
	ClientID uint64
	Cookie   uint64
}

ObjectWatcher is one client watching a RADOS object. Address is the client's IP:port/nonce, the same form BlocklistEntries reports. A client that stops answering keeps its watch until the OSD's osd_client_watch_timeout expires, 30 seconds by default; until then a notify to the object waits for that client and times out.

type Observer added in v0.4.0 

type Observer struct {
	// contains filtered or unexported fields
}

Observer groups the read-only live observations of one cluster. Each query reads current state rather than retaining a snapshot. It owns no resource and needs no cleanup. Queries keep the cluster's normal locking, liveness and command-outcome checks.

Configuration, BlocklistEntries, MGRModules, OSDFlags, OSDStates and ManagerServices take the topology lock and refuse a terminated cluster before running a command; MonitorBootstrapAddresses, HealthDetails and FullRatios do the same under ctx and keep their original-FSID checks; QuorumStatus takes no lock and fails on a terminated cluster with the error of the command path; PGInconsistencies and ObjectWatchers run rados in the client container. A refused or failed collection is an error, never an empty result.

func Observe added in v0.4.0 

func Observe(cluster *Container) (*Observer, error)

Observe returns an Observer for cluster. A nil cluster returns an error and a nil Observer. Nothing is read from the cluster.

func (*Observer) BlocklistEntries added in v0.4.0 

func (o *Observer) BlocklistEntries(ctx context.Context) ([]BlocklistEntry, error)

BlocklistEntries lists exact entries and CIDR ranges without mutating them.

It includes entries made by other tools or by Ceph itself, for example when CephFS or RBD evicts a client. Range is true for a CIDR range and Until is the MON's absolute expiration time. Listing grants no ownership to remove an entry; override.AddBlocklist adds one.

func (*Observer) Configuration added in v0.4.0 

func (o *Observer) Configuration(ctx context.Context) ([]ConfigEntry, error)

Configuration lists exact central database entries, including masked entries. Use native config show/daemon config get to inspect a running daemon's effective value; this method does not claim a stored setting has overridden local config.

Entries can contain sensitive values.

func (*Observer) DaemonAddresses added in v0.4.0 

func (o *Observer) DaemonAddresses(ctx context.Context) (DaemonAddresses, error)

DaemonAddresses reads the names and advertised addresses of the cluster's MONs, MGRs and OSDs from the native maps. They are the addresses a client learns from the cluster, which can differ from the addresses a container has: a host-network cluster advertises its host address, and an OSD that is down keeps its last addresses.

In bridge mode the addresses are the daemon containers' addresses on the fixture network; in host mode they are ClusterFacts.PublicAddress with the allocated ports. Standby MGRs advertise no address, and a MGR that has just become active advertises its vector only once it is available, so wait for ManagerAvailable after a failover. A down OSD keeps the addresses of its last boot; one that never booted has empty vectors. The three maps are read one after another, not as one snapshot; compare the epochs when the order matters. Each EntityAddress has Type (v2, v1 or any), Addr as Ceph prints it and Nonce, which MONs leave at 0.

func (*Observer) FullRatios added in v0.4.0 

func (o *Observer) FullRatios(ctx context.Context) (FullRatioSnapshot, error)

FullRatios reads native thresholds through one captured control CLI handle, checking the original bootstrap FSID before and after the map query. Native zero, equal and misordered thresholds remain observable. It makes no writes, command retries or background workers (a MON connection that times out before a command is sent is reconnected, as in Container.Ceph) and returns a zero snapshot on any failure. These separate reads are not atomic; external configuration writers must not race the observation. Poll HealthDetails for eventual OSD fullness reports.

The ratios are Ceph's float32 values, so do not compare them with float64 literals for equality. They are OSDMap policy, separate from central configuration and pool quotas.

func (*Observer) HealthDetails added in v0.4.0 

func (o *Observer) HealthDetails(ctx context.Context) (HealthSnapshot, error)

HealthDetails reads the original cluster's health through one captured control CLI handle, checking the bootstrap FSID before and after the query. It works before storage or MGR readiness, performs no writes and no command retries (a MON connection that times out before the command is sent is reconnected for up to a minute, as in Container.Ceph) and returns a zero snapshot on every failure, including post-query cancellation. Native command output is omitted from errors because it may contain secrets. External native/configuration writers must not race this observation: the three reads are not an atomic transaction. Poll with a caller deadline when asserting eventual health after changing a test topology.

Assert on a check's code and its Muted flag, since HEALTH_OK can include muted warnings; a sticky mute stays after its check clears. Ceph raises many checks only after a delay. Messages can contain pool names, paths or other application data.

func (*Observer) MGRModules added in v0.4.0 

func (o *Observer) MGRModules(ctx context.Context) ([]MGRModuleState, error)

MGRModules lists configured and available modules, sorted by name. Enabled and CanRun do not prove command readiness; WaitMGRModuleReady probes selected built-in modules, and callers must probe their intended commands for others.

func (*Observer) ManagerMetrics added in v0.4.0 

func (o *Observer) ManagerMetrics(ctx context.Context) (ManagerMetrics, error)

ManagerMetrics scrapes the active manager's prometheus module once and parses the Prometheus text exposition format. It never enables the module: enable it with override.SetMGRModule first, and wait until ManagerServices lists it. A module that serves no endpoint is ErrInvalidRequest. The scrape runs in the control container, on the cluster's network, at the address that ManagerServices advertises; it is bounded to 30 seconds and 32 MiB, and a failed fetch or unparsable output is a *NativeFailure or an error.

A standby's empty page is an error. The module reports cluster-level metrics (pools such as ceph_pool_wr, OSD state, health, the MGRs); since Reef it leaves out daemon perf counters unless mgr/prometheus/exclude_perf_counters is false, and each daemon's own counters are in its AdminSocket perf dump. Pool counters come from the MGR's PG statistics, so poll until an expected value appears. In host mode the module first needs its own address and port per manager.

func (*Observer) ManagerServices added in v0.4.0 

func (o *Observer) ManagerServices(ctx context.Context) (map[string]string, error)

ManagerServices returns the service URLs, keyed by module name such as prometheus or dashboard, that the active manager advertises in the MGR map. A module appears only after it has started its listener, and a failover replaces the URLs with the new active manager's. The URLs carry the address the module bound: a container address reachable from WithClient containers in bridge mode, or the configured address in host mode. They are not mapped to Docker host ports. An empty map means no module advertises a service.

func (*Observer) ManagerStatus added in v0.4.0 

func (o *Observer) ManagerStatus(ctx context.Context) (ManagerStatus, error)

ManagerStatus reads active ownership and registered standbys from the native MGR map.

func (*Observer) MonitorBootstrapAddresses added in v0.4.0 

func (o *Observer) MonitorBootstrapAddresses(ctx context.Context) (string, error)

MonitorBootstrapAddresses returns the current majority monmap's versioned address vectors, suitable for the Ceph mon_host setting. The live cluster FSID must match this fixture's original bootstrap configuration. This is a bounded native observation, not a network attachment or membership mutation.

func (*Observer) OSDFlags added in v0.4.0 

func (o *Observer) OSDFlags(ctx context.Context) ([]OSDFlag, error)

OSDFlags lists exact global OSDMap flag tokens, including native compatibility flags that are not among the OSDFlag constants. Per-OSD, CRUSH-node or device-class flags are separate native state.

func (*Observer) OSDStates added in v0.4.0 

func (o *Observer) OSDStates(ctx context.Context) ([]OSDState, error)

OSDStates returns owned OSD states sorted by native ID. It refuses an owned ID that is missing, purged or now belongs to a different native UUID.

func (*Observer) ObjectWatchers added in v0.4.0 

func (o *Observer) ObjectWatchers(ctx context.Context, pool, namespace, object string) ([]ObjectWatcher, error)

ObjectWatchers lists the clients watching one object, including clients outside this fixture. Namespace is empty for the default namespace. It reads native state only and grants no ownership.

Each watcher's Address has the IP:port/nonce form of BlocklistEntries. A client that stops answering keeps its watch until the OSD's osd_client_watch_timeout expires (30 seconds by default); until then a notify on the object waits for it and times out.

func (*Observer) PGInconsistencies added in v0.4.0 

func (o *Observer) PGInconsistencies(ctx context.Context, pgid PGID) ([]InconsistentObject, error)

PGInconsistencies lists the objects that the last deep scrub of a PG found inconsistent. An empty result means that scrub found none; it is not a new scrub. A PG whose results are no longer available returns an error.

func (*Observer) Pools added in v0.4.0 

func (o *Observer) Pools(ctx context.Context) ([]*Pool, error)

Pools lists the pools that the cluster has, including pools that services or Ceph created, as one Pool for each. Every Pool identifies its native pool by name and native ID, as the Pool of Container.Pool does, and describes only what the listing reports (Name, PGNum, Replicas of a replicated pool, MinSize and ErasureCodeProfile); Pool.Status reads the full native state and Pool.Usage and Pool.PGs the usage and the PGs. The listing is one read of the native pool map, so the pools of one result are consistent with each other and later reads through them are separate observations.

func (*Observer) QuorumStatus added in v0.4.0 

func (o *Observer) QuorumStatus(ctx context.Context) (QuorumStatus, error)

QuorumStatus reads the monitor quorum and monmap. It takes no topology lock and selects the control container for each read.

The query runs under a 5-second process limit inside the control container, so it returns an error rather than hanging while quorum is lost.

func (*Observer) Status added in v0.4.0 

func (o *Observer) Status(ctx context.Context) (Status, error)

Status queries the monitor's JSON API through the container-local CLI.

type OperationError added in v0.2.0 

type OperationError struct {
	// Kind is the category: ErrInvalidRequest, ErrUnsupported,
	// ErrImageCapability, ErrEnvironmentUnavailable, ErrNativeFailure.
	Kind error
	// Op names the operation, such as "create client", when the error's
	// text does not already start with it.
	Op string
	// Target names the container or cluster that the failure concerns, such
	// as "control", "osd.0", "destination cluster" or "netem helper".
	Target string
	// Version is the Ceph release observed on Target, when it matters.
	Version string
	// Image is the immutable ID of Target's image, when the fixture read it.
	Image string
	// Capability names the missing or unsupported capability, such as
	// "ceph-authtool --key-type" or "python3".
	Capability string
	// Err is the cause.
	Err error
}

OperationError is a classified failure with the details that identify it. Its text never includes command arguments, raw output or credentials.

func (*OperationError) Error added in v0.2.0 

func (e *OperationError) Error() string

func (*OperationError) Is added in v0.2.0 

func (e *OperationError) Is(target error) bool

Is reports whether target is the error's category.

func (*OperationError) Unwrap added in v0.2.0 

func (e *OperationError) Unwrap() error

type Option 

type Option func(*options) error

Option transfers cluster settings outside the monitor's container request.

func WithAuth added in v0.2.0 

func WithAuth(config AuthConfig) Option

WithAuth selects the Cephx key type, the MonMap ciphers and the ticket lifetimes of a new cluster. The settings are part of the bootstrap: the key type and ciphers apply from the first MON, and the lifetimes are written to every daemon's and WithClient container's ceph.conf, so override.SetConfig cannot change them later. An image without key type support, such as Squid or Tentacle before 20.2.4, accepts a configuration in which every cipher is CipherAES, and Run fails before any daemon starts when one is not; Run checks the ceph-authtool --key-type and monmaptool cipher options themselves, not the release number. On Tentacle 20.2.4 and later every default key and cipher is aes256k. A PreferredCipher of aes also sets mon_auth_allow_insecure_key in [mon], which Tentacle requires before its MONs generate AES keys. WithConfigFile rejects auth_* keys, so set the lifetimes here. The MonMap ciphers can change on a running Tentacle cluster with ceph mon set (for example auth_allowed_ciphers), and ceph mon dump reports them.

func WithClusterConfig added in v0.4.0 

func WithClusterConfig(cluster ClusterConfig) Option

WithClusterConfig is the grouped option function that carries a ClusterConfig to Run. Like every Option it changes the cluster settings only and never reaches a container request.

Example 
package main

import (
	"context"
	"log"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/testcontainers/testcontainers-go"
)

func main() {
	ctx := context.Background()
	topology := ceph.ClusterConfig{}.WithMonitorCount(3).WithOSDCount(3).WithPoolDefaults(3, 2)
	cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithClusterConfig(topology))
	defer func() {
		if err := testcontainers.TerminateContainer(cluster); err != nil {
			log.Printf("failed to terminate container: %s", err)
		}
	}()
	if err != nil {
		panic(err)
	}
}

func WithConfigFile 

func WithConfigFile(path string) Option

WithConfigFile merges a native ceph.conf file from the test host into the bootstrap configuration of every daemon and WithClient container, like the WithConfigFile options of testcontainers-go modules. Entries replace the fixture's defaults for the same section and key, and later files override earlier files key by key. Ceph gives local files precedence over the MON configuration database, so override.SetConfig cannot change a key set here. Keys that the fixture owns or passes on a daemon command line are rejected: identity, addresses, networks, Cephx, Messenger modes, logging, data paths, pool size defaults and BlueStore file layout. Use the dedicated options for those. The exception is ms_bind_prefer_ipv4, which Run accepts with AddressDualStack only: false in a daemon's section, such as [osd.1] or [mgr.b], makes that daemon advertise its IPv6 address first, and native clients then connect to it over IPv6. RGW runs as client.admin, so RGW settings belong in [client] or [client.admin]. The merged file becomes /etc/ceph/ceph.conf before the first MON creates its store, so bootstrap-only settings work; it does not change a running cluster. Spaces, '-' and '_' in key names are equivalent, as in Ceph.

The file is validated before any container is created. Includes, lines continued with a trailing backslash, keys outside a section, duplicate keys within one section of one file, NUL bytes, invalid UTF-8 and files or merged results larger than 64 KiB are rejected. Ceph treats '#' and ';' as comment starts even inside quotes, so a value containing them is rejected unless they are escaped as \# and \;. Ceph silently ignores some values (for example an osd_memory_target below 896 MiB); check the effective value with ceph config show.

The rejected keys and their replacements: fsid, mon_host, mon_initial_members and mon_addr (generated; ClusterConfig.WithMonitorCount); public_network, cluster_network, public_addr, cluster_addr, public_bind_addr and cluster_bind_addr (WithHostNetwork, NetworkConfig.WithSeparateClusterNetwork, NetworkConfig.WithHostAddress); auth_*, keyring, key and keyfile (WithAuth); ms_*_mode, ms_mon_*_mode and ms_bind_* except ms_bind_prefer_ipv4 (NetworkConfig.WithMessengerMode, NetworkConfig.WithMessengerPolicy, ClientContainerOptions.WithMessengerPolicy, NetworkConfig.WithAddressFamily, WithHostNetwork); osd_pool_default_size and osd_pool_default_min_size (ClusterConfig.WithPoolDefaults); osd_pool_default_crush_rule (ClusterConfig.WithDefaultCRUSHRoot); bluestore_block_size (OSDStorage.WithBlockSize); osd_objectstore, bluestore_block_path and bluestore_block_create (fixed storage layout); log_file, log_to_file, log_to_stderr, err_to_stderr, mon_cluster_log_to_file and mon_cluster_log_to_stderr (logs always go to container output); admin_socket, run_dir, mon_data, osd_data, mgr_data and mds_data (fixed paths); crash_dir (fixed at CrashDir); mds_join_fs and mds_cache_memory_limit (MDS command line); rgw_frontends, rgw_thread_pool_size, rgw_exit_timeout_secs, rgw_realm, rgw_zonegroup, rgw_zone, rgw_sync_obj_etag_verify and rgw_sync_lease_period (RGW command line). ms_bind_prefer_ipv4 with any family but AddressDualStack is refused with ErrInvalidRequest.

func WithHostNetwork 

func WithHostNetwork() Option

WithHostNetwork places every cluster daemon and WithClient container in the Docker daemon's host network. MON and RGW ports are selected automatically. The default advertised address is 127.0.0.1 for clients on that same host. Run fails with ErrHostNetworkUnavailable unless the engine runs Linux containers and the test process reaches the first MON's advertised endpoint. MON port reservation is retried up to five times when another process takes a port, and the reachability probe is retried for up to 5 seconds; later MON, MGR, OSD, MDS and RGW endpoints are not probed. MGR, OSD and MDS use Ceph's dynamic ports. Docker Desktop needs host networking enabled (4.34 or later), and a test process that itself runs in a bridge-network container cannot use host mode. It cannot be combined with NetworkConfig.WithSeparateClusterNetwork, and fault.InterruptNetwork and fault.ApplyNetworkProfile are unavailable in host mode.

func WithImageOptions added in v0.4.0 

func WithImageOptions(config ImageOptions) Option

WithImageOptions is the grouped option function that carries an ImageOptions to Run. Like every Option it changes the cluster settings only and never reaches a container request.

Example

Role images are one ImageOptions value, passed to Run once through the grouped option function WithImageOptions. The image argument is empty because the set is the one image configuration. This example is compiled, so the snippet it shows cannot drift from the API; it is not run. 

package main

import (
	"context"
	"log"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/testcontainers/testcontainers-go"
)

func main() {
	ctx := context.Background()
	images := ceph.ImageSet{Essential: "registry.example.com/ceph-core:20.2.4", Client: "registry.example.com/ceph-client:20.2.4"}
	cluster, err := ceph.Run(ctx, "", ceph.WithImageOptions(ceph.ImageOptions{}.WithImageSet(images)), ceph.WithClusterConfig(ceph.ClusterConfig{}.WithOSDCount(1)))
	defer func() {
		if err := testcontainers.TerminateContainer(cluster); err != nil {
			log.Printf("failed to terminate container: %s", err)
		}
	}()
	if err != nil {
		panic(err)
	}
}

func WithMonitorStorage added in v0.4.0 

func WithMonitorStorage(storage MonitorStorage) Option

WithMonitorStorage applies storage to every MON of the cluster.

func WithNetworkConfig added in v0.4.0 

func WithNetworkConfig(network NetworkConfig) Option

WithNetworkConfig is the grouped option function that carries a NetworkConfig to Run. Like every Option it changes the cluster settings only and never reaches a container request.

Example

Network and Messenger settings are one NetworkConfig value, passed to Run once through the grouped option function WithNetworkConfig. Compiled, not run. 

package main

import (
	"context"
	"log"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/testcontainers/testcontainers-go"
)

func main() {
	ctx := context.Background()
	network := ceph.NetworkConfig{}.WithAddressFamily(ceph.AddressDualStack).WithSeparateClusterNetwork()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithNetworkConfig(network), ceph.WithClusterConfig(ceph.ClusterConfig{}.WithOSDCount(1)))
	defer func() {
		if err := testcontainers.TerminateContainer(cluster); err != nil {
			log.Printf("failed to terminate container: %s", err)
		}
	}()
	if err != nil {
		panic(err)
	}
}

func WithOSDStorage added in v0.4.0 

func WithOSDStorage(storage OSDStorage) Option

WithOSDStorage is the grouped option function that carries an OSDStorage to Run. Like every Option it changes the cluster settings only and never reaches a container request.

Example

OSD storage is one OSDStorage value, passed to Run once through the grouped option function WithOSDStorage. Compiled, not run. 

package main

import (
	"context"
	"log"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/testcontainers/testcontainers-go"
)

func main() {
	ctx := context.Background()
	storage := ceph.OSDStorage{}.WithBlockSize(256 << 20).WithObjectStore(ceph.OSDMemStore)
	cluster, err := ceph.Run(ctx, ceph.DefaultImage, ceph.WithOSDStorage(storage), ceph.WithClusterConfig(ceph.ClusterConfig{}.WithOSDCount(1)))
	defer func() {
		if err := testcontainers.TerminateContainer(cluster); err != nil {
			log.Printf("failed to terminate container: %s", err)
		}
	}()
	if err != nil {
		panic(err)
	}
}

func WithStartupTimeout 

func WithStartupTimeout(timeout time.Duration) Option

WithStartupTimeout bounds bootstrap, readiness and topology operations. Run's whole bootstrap, from validation to the initial services, shares it. Each operation is also bounded by the caller's context. Default: 3 minutes.

It bounds Run as a whole and each later topology, readiness and wait operation, such as AddOSD, WaitForClean, Pool.WaitForPGCount and DeepScrubPG, so a wait never outlives the shorter of the two limits. Increase it for slow engines, large pools or long recovery waits.

func (Option) Customize 

func (Option) Customize(*testcontainers.GenericContainerRequest) error

Customize implements testcontainers.ContainerCustomizer.

type PGID added in v0.4.0 

type PGID string

PGID identifies one placement group as "<pool>.<hex>": the decimal native pool ID, a dot and the lowercase hexadecimal PG number, for example "1.0" or "3.1f". Pool.PGs reports PGs with this type, so a reported PG passes straight to DeepScrubPG, RepairPG and PGInconsistencies. An untyped string constant converts to PGID without validation, so these methods check the grammar again and refuse an invalid value; ParsePGID checks it up front.

func ParsePGID added in v0.4.0 

func ParsePGID(s string) (PGID, error)

ParsePGID returns s as a PGID when it matches the grammar "<pool>.<hex>", which is exactly what DeepScrubPG, RepairPG and PGInconsistencies accept. Otherwise it returns an error that matches ErrInvalidRequest and the zero PGID.

func (PGID) String added in v0.4.0 

func (p PGID) String() string

String returns the native PG ID.

type PGState 

type PGState struct {
	PGID                                        PGID
	State                                       string
	Up, Acting                                  []int
	UpPrimary, ActingPrimary                    int
	ReportedEpoch, MappingEpoch, LastEpochClean uint32
	ReportedSequence                            uint64
	StatsInvalid                                bool
	Stats                                       PGStats
}

PGState retains native reported placement and statistics. Up and Acting preserve order, empty vectors and repeated missing EC shard slots (CRUSH_ITEM_NONE, 2147483647). Primaries are explicit reported values, not inferred from the vectors; -1 means no primary. Epochs can be zero or older than the OSDMap brackets, and State retains unknown or future native states. A reported OSD may have been removed or be outside fixture ownership.

type PGStats 

type PGStats struct {
	Objects, Bytes, ObjectCopies                      int64
	ObjectsDegraded, ObjectsMisplaced, ObjectsUnfound int64
}

PGStats contains signed native PG counters. Bytes counts logical payload, not physical disk allocation; ObjectCopies is separate from logical Objects. Reports arrive asynchronously, including when StatsInvalid is false, so these counters are not an I/O visibility or redistribution barrier.

type Pool 

type Pool struct {
	PoolConfig
	CRUSHRule          string
	ErasureCodeProfile string
	// contains filtered or unexported fields
}

Pool identifies the pool, CRUSH rule and optional EC profile created for a fixture. PoolConfig contains the resolved defaults. Cluster termination disposes of its data together with the rest of the cluster.

The operations on a pool are methods of its handle. The handle identifies the native pool that it was created for or looked up as, by name and native pool ID, so a later pool that reuses the name is not reached through it. Copies of a Pool share that identity. A Pool that does not come from Container.CreatePool or Container.Pool has no cluster and its operations return an error.

func (*Pool) PGs added in v0.4.0 

func (p *Pool) PGs(ctx context.Context) (PoolPGSnapshot, error)

PGs observes the PGs that Ceph reports for this pool; PoolPGSnapshot describes the evidence. All failures, including post-query cancellation, return a zero snapshot and static errors that omit native output. There are no writes, command retries or workers (a MON connection that times out before a command is sent is reconnected, as in Container.Ceph). External native/configuration writers must not race these separate reads. Compare original fixture identities and data when asserting movement; these reported PGs are not current mappings or read-after-write evidence.

It works before any OSD exists.

func (*Pool) Remove added in v0.4.0 

func (p *Pool) Remove(ctx context.Context) error

Remove deletes a pool created by CreatePool or WithPools, together with the CRUSH rule and erasure-code profile created for it, so a long-lived cluster can give each test its own pool without reaching PG limits. Pools that Ceph or a service created (.mgr, RGW zone pools, CephFS pools) and pools still used by a CephFS filesystem are refused. Every object in the pool is destroyed, including RBD images and namespaces; stop clients and mirroring that use it first. mon_allow_pool_delete is enabled in the central database only while the pool is removed and then restored; a local config file that sets it to false blocks removal. Ownership is recognized by the pool's tc-<name>-replicated or tc-<name>-ec CRUSH rule, so a pool that SetPlacement moved to another rule is refused until it moves back; placement rules are not removed. A retry after an error finishes removing whichever owned resources remain. External edits to the pool must not race this call.

After a complete removal the handle is stale and every operation through it returns an error, so a later pool that reuses the name is not affected.

func (*Pool) SetPGCount added in v0.4.0 

func (p *Pool) SetPGCount(ctx context.Context, pgNum int) error

SetPGCount requests a new PG count for this pool with autoscaling off or warn. It records both native pg_num and pgp_num targets; the active manager then splits or merges PGs in steps, and merges wait for clean PGs. Success means the targets were accepted, not that PGs have changed. Use WaitForPGCount for the count and Container.WaitForClean for the resulting movement. Without an active manager the targets remain pending. The native pool ID is checked before each mutation; on partial failure inspect Status. Native per-OSD PG limits can still reject the request.

func (*Pool) SetPlacement added in v0.4.0 

func (p *Pool) SetPlacement(ctx context.Context, placement PoolPlacement) error

SetPlacement moves a replicated pool to a simple osd/host/rack CRUSH rule. The size and min_size are unchanged and the new placement needs enough owned eligible domains in the native CRUSH map. It reuses the pool's CreatePool rule or an earlier rule from this method when its native steps match; otherwise it creates tc-<pool>-placement-<hash>. Rules are not deleted, so a pool can move back. Success means the pool references the rule; data moves asynchronously. Until PG reports reflect the new map, WaitForClean can pass on the previous placement, so poll PGs for the expected up/acting sets. Holding both nobackfill and norecover with override.SetOSDFlag keeps the move incomplete; nobackfill alone does not stop log-based recovery of small PGs. EC rules are bound to their profile and are refused. External pool/CRUSH edits must not race this call.

func (*Pool) SetQuota added in v0.4.0 

func (p *Pool) SetQuota(ctx context.Context, quota PoolQuota) error

SetQuota replaces both limits, including zero values that remove a limit. Commands are sequential and may partially succeed; inspect Status after an error. Existing data is never removed. External pool edits must not race this operation; native pool IDs are checked before each mutation.

Quotas are enforced from asynchronously reported PG statistics, so writes can overshoot briefly; when a quota is reached Ceph marks the pool full and clients see writes blocked or EDQUOT, depending on the client.

func (*Pool) SetReplication added in v0.4.0 

func (p *Pool) SetReplication(ctx context.Context, replicas, minSize int) error

SetReplication changes a replicated pool's size/min_size without changing data, PG count or placement. It requires enough owned placement domains in the native CRUSH rule. EC shard counts cannot be changed by this method. WaitForClean waits for resulting recovery. On partial failure, inspect Status; no rollback or data deletion is attempted. External pool/CRUSH edits must not race this call.

func (*Pool) Status added in v0.4.0 

func (p *Pool) Status(ctx context.Context) (PoolState, error)

Status returns the native state of this pool. It fails when the pool does not exist or the name belongs to another pool now.

func (*Pool) Usage added in v0.4.0 

func (p *Pool) Usage(ctx context.Context) (PoolUsageSnapshot, error)

Usage reports the pool's stored bytes and objects with the original bootstrap FSID and the exact native pool ID/name before and after the observation. A pool without a statistics row is unavailable, not empty. All failures, including cancellation after a query, return a zero snapshot. This method makes no writes, command retries or background workers (a MON connection that times out before a command is sent is reconnected, as in Container.Ceph). External pool or control configuration writers must not race it; the native reads are not an atomic transaction. Poll with a caller deadline for eventual assertions.

A pool without a statistics row is an error, not an empty result, and AllocatedBytes need not equal StoredBytes times the replica count.

func (*Pool) WaitForPGCount added in v0.4.0 

func (p *Pool) WaitForPGCount(ctx context.Context, pgNum int) error

WaitForPGCount waits until the native pool reports pgNum as its current, pending and target PG and PGP counts and the manager reports exactly pgNum PGs for it. Until then Container.WaitForClean can pass on the previous PG set. It fails if the name is bound to another pool than this handle identifies. The count does not imply clean PGs, completed backfill or data visibility; call WaitForClean afterwards or use PGs. Cluster startup timeout applies.

type PoolConfig 

type PoolConfig struct {
	Name        string
	PGNum       int
	Replicas    int
	MinSize     int
	Application string
	// FailureDomain selects distinct osd, host or rack buckets. Empty uses osd.
	FailureDomain string
	// CRUSHRoot selects the hierarchy root; empty uses the cluster's default
	// root, which is default unless ClusterConfig.WithDefaultCRUSHRoot selects another.
	// DeviceClass optionally limits placement to OSDs explicitly configured
	// with that class.
	CRUSHRoot   string
	DeviceClass string
	ErasureCode *ErasureCodeConfig
	// NoHashPSPool clears the pool's hashpspool flag. Ceph then computes each
	// PG's placement seed with the legacy additive formula, the PG number plus
	// the pool ID, instead of hashing the PG number with the pool ID, as for
	// pools that predate the flag. It is set right after the pool is created,
	// before any data is written.
	NoHashPSPool bool
}

PoolConfig describes a pool in the disposable, file-backed cluster. Zero PGNum selects 8 PGs with autoscaling disabled. A nil ErasureCode creates a replicated pool, with the cluster's replica/min-size defaults when omitted (normally 2/1, or 1/1 for one initial OSD). Application can be rados, rbd, cephfs, rgw or a custom name; empty leaves it unassigned. Name uses letters, digits, '_', '.' and '-' and must not start with '.' or '-'. Replicas is 1 to 10 and must be zero for an erasure-coded pool. MinSize is 1 to Replicas for a replicated pool and K to K+M for an erasure-coded one. FailureDomain puts each replica or shard in a distinct osd, host or rack bucket. For RBD pools that need rbd pool init, use rbd.WithPools or rbd.InitPool.

type PoolPGSnapshot 

type PoolPGSnapshot struct {
	FSID                                string
	PoolBefore, PoolAfter               PoolState
	OSDMapEpochBefore, OSDMapEpochAfter uint32
	PGReady                             bool
	PGs                                 []PGState
}

PoolPGSnapshot contains one pool's reported PGs, bracketed by two native OSDMap observations. PoolBefore and PoolAfter reuse PoolState policy fields; their names and IDs agree, but policies and OSDMap epochs may differ. These reads are not an atomic placement snapshot. PGReady is the native MGR flag: false can accompany reports, and true can still accompany unknown PGs.

type PoolPlacement 

type PoolPlacement struct {
	FailureDomain string
	CRUSHRoot     string
	DeviceClass   string
}

PoolPlacement selects the CRUSH hierarchy of a replicated pool after creation. Its fields have PoolConfig's meaning and defaults: empty FailureDomain uses osd, empty CRUSHRoot uses the cluster's default root (default unless ClusterConfig.WithDefaultCRUSHRoot selects another) and empty DeviceClass leaves class unset.

type PoolQuota 

type PoolQuota struct {
	MaxBytes   uint64
	MaxObjects uint64
}

PoolQuota limits logical bytes and object count. Zero removes that limit. Enforcement depends on asynchronously reported PG statistics, so the quota is not a strict per-write accounting boundary.

type PoolState 

type PoolState struct {
	ID                   int64
	Name                 string
	Type                 string
	Size, MinSize, PGNum int
	PGNumTarget          int
	PGNumPending         int
	PGPlacementNum       int
	PGPlacementNumTarget int
	CRUSHRule            int
	AutoscaleMode        string
	ErasureCodeProfile   string
	Flags                string
	Quota                PoolQuota
}

PoolState is an authoritative snapshot of a native pool, including pools created by services or through Ceph. Size is replica count or EC shard count. It does not change when a policy is subsequently updated. PGNum is the current count; the target and pending fields show a split or merge still in progress and are zero when the native map omits them.

type PoolUsageSnapshot 

type PoolUsageSnapshot struct {
	FSID, Name         string
	ID                 int64
	StoredBytes        uint64
	StoredDataBytes    uint64
	StoredOMAPBytes    uint64
	AllocatedBytes     uint64
	AllocatedDataBytes uint64
	AllocatedOMAPBytes uint64
	Objects            uint64
	MaxAvailableBytes  uint64
	UsedRatio          float64
}

PoolUsageSnapshot is one monitor-reported pool usage observation. StoredBytes estimates logical user data (DATA + OMAP); AllocatedBytes includes replication or EC and allocation overhead, but not the BlueStore database. All sizes are bytes, Objects is a logical count, and UsedRatio is a fraction in [0, 1]. MaxAvailableBytes estimates writable logical capacity. PG reports arrive asynchronously: this is neither a read-after-write barrier nor a quota check.

type QuorumStatus 

type QuorumStatus struct {
	QuorumNames []string `json:"quorum_names"`
	MonMap      struct {
		FSID string `json:"fsid"`
		Mons []struct {
			Name        string `json:"name"`
			PublicAddrs struct {
				Addrvec []struct {
					Type string `json:"type"`
					Addr string `json:"addr"`
				} `json:"addrvec"`
			} `json:"public_addrs"`
		} `json:"mons"`
	} `json:"monmap"`
}

QuorumStatus contains the actual quorum and advertised monitor addresses.

type Status 

type Status struct {
	FSID   string `json:"fsid"`
	Health struct {
		Status string `json:"status"`
	} `json:"health"`
	MgrMap struct {
		Available bool `json:"available"`
	} `json:"mgrmap"`
	OSDMap struct {
		NumOSDs   int `json:"num_osds"`
		NumUpOSDs int `json:"num_up_osds"`
		NumInOSDs int `json:"num_in_osds"`
	} `json:"osdmap"`
	PGMap struct {
		NumPGs     int `json:"num_pgs"`
		PGsByState []struct {
			StateName string `json:"state_name"`
			Count     int    `json:"count"`
		} `json:"pgs_by_state"`
	} `json:"pgmap"`
}

Status contains the subset of ceph status JSON used for readiness. Use Ceph for other commands or the full JSON response.

type Verification added in v0.4.0 

type Verification struct {
	// contains filtered or unexported fields
}

Verification is one activated post-test verification of a dedicated cluster. It keeps the baseline that its activation recorded.

func (*Verification) Evaluate added in v0.4.0 

func (v *Verification) Evaluate(ctx context.Context) (*VerificationReport, error)

Evaluate evaluates the window from the activation until now and returns its report. It changes nothing and can be repeated; each evaluation starts from the activation's baseline. Gaps make a report incomplete instead of returning an error: Evaluate returns an error only when it cannot evaluate at all.

type VerificationCheck added in v0.4.0 

type VerificationCheck struct {
	Outcome         VerificationOutcome   `json:"outcome"`
	Permitted       int                   `json:"permitted"`
	Unexpected      int                   `json:"unexpected"`
	Findings        []VerificationFinding `json:"findings"`
	OmittedFindings int                   `json:"omitted_findings"`
	Expectations    []ExpectationResult   `json:"expectations"`
	Gaps            []VerificationGap     `json:"gaps"`
}

VerificationCheck is the result of the log or the crash check. Findings keep the unexpected events first and then the permitted ones, up to MaxFindings; OmittedFindings counts the rest. Permitted and Unexpected count all of them.

type VerificationConfig added in v0.4.0 

type VerificationConfig struct {
	// Name labels the verification, its check and its report. Empty selects
	// "post-test". Names are unique within one Finalization.
	Name string
	// Logs and Crashes are the expected events. A log expectation never
	// matches a crash, nor a crash expectation a log line.
	Logs    []LogExpectation
	Crashes []CrashExpectation

	Timeout          time.Duration
	OperationTimeout time.Duration
	MaxLogBytes      int64
	MaxFindings      int
	// AdditionalContainers are caller-owned daemon containers, such as
	// mirror daemons, that are verified like the cluster's own. Verification
	// never adopts them.
	AdditionalContainers []DiagnosticsContainer
	// RedactValues removes additional opaque secrets from the report.
	RedactValues []string
}

VerificationConfig selects what Finalization.Verify checks after the test: the log lines and the new crashes of the cluster's daemons since activation, each against its own expectations. Zero fields select a two minute timeout for the activation and for each evaluation (at most five), 20 seconds per Docker operation (at most one minute), 32 MiB of log stream per container (at most 256 MiB) and 256 retained findings per check (at most 4096).

type VerificationFinding added in v0.4.0 

type VerificationFinding struct {
	Source      string `json:"source"`
	ContainerID string `json:"container_id,omitempty"`
	// Event is a LogEvent, or "crash".
	Event string `json:"event"`
	Time  string `json:"time,omitempty"`
	// Text is the redacted line, or a summary of the crash, at most 1 KiB.
	Text        string `json:"text"`
	Truncated   bool   `json:"truncated,omitempty"`
	CrashID     string `json:"crash_id,omitempty"`
	Expectation string `json:"expectation,omitempty"`
}

VerificationFinding is one evaluated event. Expectation names the expectation that permitted it; it is empty for an unexpected event.

type VerificationGap added in v0.4.0 

type VerificationGap struct {
	Source string `json:"source,omitempty"`
	Reason string `json:"reason"`
}

VerificationGap is an observation that could not be completed.

type VerificationOutcome added in v0.4.0 

type VerificationOutcome string

VerificationOutcome is the result of one check or of a whole verification. 

const (
	// VerificationClean means that no unexpected event was observed and
	// every required expectation was satisfied within the reported sources
	// and window, all of which were observed.
	VerificationClean VerificationOutcome = "clean"
	// VerificationUnexpected means that an observed event was not permitted
	// or that a required expectation was not satisfied.
	VerificationUnexpected VerificationOutcome = "unexpected"
	// VerificationIncomplete means that no unexpected event was found, but a
	// requested observation could not be completed; Gaps say which.
	VerificationIncomplete VerificationOutcome = "incomplete"
)

type VerificationReport added in v0.4.0 

type VerificationReport struct {
	SchemaVersion int                 `json:"schema_version"`
	Name          string              `json:"name"`
	Cluster       string              `json:"cluster"`
	Outcome       VerificationOutcome `json:"outcome"`
	Window        VerificationWindow  `json:"window"`
	// Images are the cluster's role images; a daemon started with DaemonOptions.WithImage
	// is not reflected.
	Images  ImageSet             `json:"images"`
	Sources []VerificationSource `json:"sources"`
	Logs    VerificationCheck    `json:"logs"`
	Crashes VerificationCheck    `json:"crashes"`
	// LogExpectations and CrashExpectations are the effective policy.
	LogExpectations   []LogExpectation   `json:"log_expectations"`
	CrashExpectations []CrashExpectation `json:"crash_expectations"`
	Notes             []string           `json:"notes"`
}

VerificationReport is the JSON-serializable result of one evaluation. Outcome is unexpected when a check is, else incomplete when a check is, else clean.

type VerificationSource added in v0.4.0 

type VerificationSource struct {
	Source       string `json:"source"`
	Role         string `json:"role"`
	Name         string `json:"name"`
	ContainerID  string `json:"container_id,omitempty"`
	ImageID      string `json:"image_id,omitempty"`
	StartedAt    string `json:"started_at,omitempty"`
	RestartCount int    `json:"restart_count"`
	State        string `json:"state,omitempty"`
	LogDriver    string `json:"log_driver,omitempty"`
	// LogBytes counts the log stream bytes read in the window.
	LogBytes int64 `json:"log_bytes"`
	// BaselineCrashes are the crash IDs that were present at activation.
	BaselineCrashes []string `json:"baseline_crashes"`
}

VerificationSource is one container generation that was selected at activation. StartedAt and RestartCount identify the generation.

type VerificationWindow added in v0.4.0 

type VerificationWindow struct {
	Phase     string    `json:"phase"`
	StartedAt time.Time `json:"started_at"`
	// EndedAt is zero when the engine's clock could not be read.
	EndedAt     time.Time `json:"ended_at"`
	EvaluatedAt time.Time `json:"evaluated_at"`
}

VerificationWindow is the observed interval, on the Docker engine's clock. Phase is "test": bootstrap before the activation and the cleanups and termination after the evaluation are outside it.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.