Documentation
¶
Overview ¶
Package cephfs prepares CephFS server fixtures in disposable Ceph clusters: filesystems, their MDS topology and data pools. Subvolume groups, subvolumes, snapshots, clones, subvolume principals and pins are in the cephfs/subvolume package, quiesce sets in cephfs/quiesce, client sessions and their limits in cephfs/session, snapshot schedules in cephfs/snapschedule, and snapshot mirroring between two clusters in cephfs/mirror.
Run starts a cluster with at least one filesystem. Start adds a filesystem to a cluster created by ceph.Run or another service package's Run, and WithFilesystems selects filesystems for any of those Run functions.
Index ¶
- Constants
- func WithFilesystems(configs ...Config) ceph.Option
- type Config
- type Container
- type DataPool
- type DataPoolState
- type Filesystem
- func (fs *Filesystem) AddDataPool(ctx context.Context, name string) (*DataPool, error)
- func (fs *Filesystem) DataPools(ctx context.Context) ([]DataPoolState, error)
- func (fs *Filesystem) MDSStatus(ctx context.Context) (*FilesystemStatus, error)
- func (fs *Filesystem) MDSs() []*MDS
- func (fs *Filesystem) OverrideMaxFileSize(ctx context.Context, bytes int64) (*MaxFileSizeOverrideHandle, error)
- func (fs *Filesystem) OverrideTiming(ctx context.Context, profile TimingProfile) ([]*override.ConfigHandle, error)
- func (fs *Filesystem) Remove(ctx context.Context) error
- func (fs *Filesystem) ScaleMDS(ctx context.Context, active, standby int) (returnErr error)
- func (fs *Filesystem) WaitReady(ctx context.Context) error
- func (fs *Filesystem) WatchMDSs(ctx context.Context, cfg ceph.WatchConfig) (*MDSWatch, error)
- type FilesystemStatus
- type MDS
- type MDSReadOnlyFault
- type MDSStatus
- type MDSWatch
- type MDSWatchEvent
- type MaxFileSizeOverrideHandle
- type TimingProfile
Examples ¶
Constants ¶
const MinMaxFileSize = 64 << 10
MinMaxFileSize is the MDSMonitor's lower bound for max_file_size, in bytes.
Variables ¶
This section is empty.
Functions ¶
func WithFilesystems ¶
WithFilesystems creates named filesystems and their MDS topologies during Run. No arguments selects the default filesystem. Repeated options replace the list; Start can add further filesystems after Run returns.
Types ¶
type Config ¶
type Config struct {
Name string
MetadataPool ceph.PoolConfig
DataPool ceph.PoolConfig
AdditionalDataPools []ceph.PoolConfig
ActiveMDS int
// NoInitialMDS creates the filesystem/pools without a metadata daemon.
// ActiveMDS and StandbyMDS must be omitted and StandbyReplay false.
// The first ScaleMDS must request 1 active / 0 standby; later scale is ordinary.
NoInitialMDS bool
StandbyMDS int
StandbyReplay bool
}
Config creates one named filesystem with dedicated pools and MDS daemons. Name defaults to tc-cephfs, ActiveMDS to one, and StandbyMDS to zero. Omitted pool names become <Name>-metadata and <Name>-data. Metadata and the default data pool must be replicated; additional pools may use erasure coding with AllowOverwrites. Each pool defaults to application=cephfs. Directory layouts select additional pools after filesystem creation. StandbyReplay assigns up to one journal follower per active rank; extra standbys stay idle.
Name is letters, digits, _, . or -, starting with a letter, digit or _. ActiveMDS (max_mds) is 1 to 256, and StandbyReplay needs StandbyMDS of at least one. Every pool gets application=cephfs and another application is refused; pools and OSD placements are validated before anything is created. The default data pool also stores inode backtraces and cannot be detached; additional pools are selected through subvolume, group or clone DataPool fields or directory layouts.
While a NoInitialMDS filesystem is cold its descriptor has a nil embedded container and an empty MDSs; do not call promoted container methods on it. WaitReady waits until its deadline, a fresh libcephfs mount fails after client_mount_timeout with errno 110 (ETIMEDOUT) and Ceph reports MDS_ALL_DOWN. ScaleMDS(ctx, 1, 0) starts the first MDS; after that the filesystem behaves like any other.
type Container ¶
Container is a cluster started by Run. It embeds *ceph.Container, so every cluster method is available on it; pass its Container field to functions that take a *ceph.Container, such as those of this and the other service packages.
func Run ¶
func Run(ctx context.Context, img string, opts ...testcontainers.ContainerCustomizer) (*Container, error)
Run starts a disposable cluster for CephFS clients. It accepts every ceph.Run option and the options of other service packages. When no WithFilesystems option is given, it creates the default filesystem tc-cephfs with one active MDS. Bootstrap options that leave the cluster without OSDs or managers for the filesystem, such as ceph.ClusterConfig.WithNoInitialOSDs, are rejected; use ceph.Run for those stages. A non-nil Container returned with an error must still be terminated.
type DataPool ¶
type DataPool struct {
Name, FilesystemName string
ID int64
// contains filtered or unexported fields
}
DataPool identifies an additional pool attached by AddDataPool. Its private identity and state are shared by copied handles. Exported fields are descriptive; editing them does not redirect removal to another pool.
func (*DataPool) RemoveUnused ¶ added in v0.4.0
RemoveUnused removes only the FSMap registration of a fresh helper attachment. A pool selected even once by a managed group/subvolume/clone is refused, including partial creations. Native groups/subvolumes must have no references and every RADOS namespace must be empty; unknown clone or retained snapshot information is conservatively refused. No pool, application tag or data is deleted. Native rm_data_pool cannot detect arbitrary POSIX layout references: the caller MUST stop external clients and reset all such layouts before removal, and prevent external edits from racing it. This method does not establish safety for a live workload. Default data pools are immutable. A copied handle shares removal state; retry reconciles a lost command reply.
type DataPoolState ¶
DataPoolState reports a currently registered native filesystem data pool. ID distinguishes a pool recreated under the same name. Default marks the immutable first data pool, which also stores inode backtraces.
type Filesystem ¶
type Filesystem struct {
testcontainers.Container
FilesystemName string
MetadataPool string
DataPool string
AdditionalDataPools []string
// contains filtered or unexported fields
}
Filesystem describes one filesystem and embeds its first MDS container for compatibility with Start. MDSs exposes every owned metadata daemon. The cluster owns their lifetime. Clients use libcephfs through WithClient without a kernel or FUSE mount on the host.
FilesystemName is what clients pass as libcephfs filesystem_name. The embedded container is nil while a NoInitialMDS filesystem is cold and moves to a surviving daemon when the original is retired; use MDSs for the topology.
Operations on the filesystem and its handles wait for each other within the caller's context; a call canceled while waiting returns without changing Ceph and can be retried. The Ceph CLI has no compare-and-swap for these resources, so external ceph fs edits must not run on the same resources while a fixture operation is in progress. Termination removes the owned resources; explicit cleanup is for tests that assert removal or inspect intermediate state.
func Filesystems ¶
func Filesystems(c *ceph.Container) []*Filesystem
Filesystems returns the cluster's owned filesystem descriptors sorted by filesystem name, including a descriptor whose setup partially failed. The cluster owns their MDS containers.
func Start ¶
func Start(ctx context.Context, c *ceph.Container, config Config, opts ...testcontainers.ContainerCustomizer) (*Filesystem, error)
Start creates one filesystem with dedicated pools and its ActiveMDS+StandbyMDS daemons on an existing cluster. A zero config selects the default filesystem tc-cephfs with one active MDS named a. Every pool and OSD placement is validated before mutations. Existing filesystems are rejected instead of modified. Creating a second filesystem enables Ceph's global enable_multiple flag; the existing filesystem and its configuration are preserved.
Customizers apply to each MDS and must preserve its identity, config, keyring, entrypoint and networking. A non-nil result with an error identifies attempted resources; the cluster owns cleanup, including partial daemon startup.
func (*Filesystem) AddDataPool ¶
AddDataPool attaches an existing pool with application=cephfs and no prior filesystem role. Replicated pools and EC pools with allow_ec_overwrites are accepted. Metadata pools, cross-filesystem reuse and repeated attachments are refused. It creates or deletes no pools. A handle returned with an error records an attempted attachment; retrying the same name reconciles an unknown successful command using the original native IDs. External edits must not race these operations. Ceph's retained application tag after detachment must be inspected and explicitly cleared through Ceph before a new attachment.
Clear the retained tag with ceph osd pool application rm <name> cephfs data before attaching a detached pool again.
func (*Filesystem) DataPools ¶
func (fs *Filesystem) DataPools(ctx context.Context) ([]DataPoolState, error)
DataPools reads the authoritative FSMap registration and current pool IDs. AdditionalDataPools in the creation descriptor describes initial settings; this method also reports subsequent native attachments and removals.
func (*Filesystem) MDSStatus ¶
func (fs *Filesystem) MDSStatus(ctx context.Context) (*FilesystemStatus, error)
MDSStatus reads one native FSMap, so rank ownership and standby state come from the same epoch. Active ranks include external daemons with Owned=false; standby lists contain only this fixture's daemons with matching affinity.
func (*Filesystem) MDSs ¶
func (fs *Filesystem) MDSs() []*MDS
MDSs returns a snapshot of this filesystem's owned daemons in creation order.
func (*Filesystem) OverrideMaxFileSize ¶ added in v0.5.0
func (fs *Filesystem) OverrideMaxFileSize(ctx context.Context, bytes int64) (*MaxFileSizeOverrideHandle, error)
OverrideMaxFileSize sets the MDSMap max_file_size of the filesystem to bytes, so that a client write or truncate that would make a file larger than bytes fails with EFBIG while smaller files stay usable. bytes must be at least MinMaxFileSize. The limit is filesystem-wide; it is not a quota, does not count used capacity and does not shrink existing files. Clients check the limit from the MDSMap they hold, so mount the client under test after the call returns. The previous value is captured, the applied one read back and the FSMap ID of the filesystem bound to the handle. Only one handle per filesystem may be active.
func (*Filesystem) OverrideTiming ¶ added in v0.5.0
func (fs *Filesystem) OverrideTiming(ctx context.Context, profile TimingProfile) ([]*override.ConfigHandle, error)
OverrideTiming applies profile to this filesystem's current owned active MDSs. DirectoryPins requires their effective mds_tick_interval to be five seconds and sets a separate central entry for each selected daemon to one. Every tracked MDS, including a standby, must retain confirmed ownership; an edited or partially started tracked daemon makes the profile unavailable. Every target's native version, ownership and effective value is checked before the first write; effective application is then awaited within the cluster's startup timeout. A local file or runtime override can prevent application. An already scheduled tick keeps its original deadline, so the first pin transition may still wait five seconds. Future scaled or replaced MDSs are outside the returned handles' scope.
The returned handles are the existing central-configuration overrides, not a new cleanup owner. Retain all of them even on error and Restore them in reverse order when the condition should end. An error can leave earlier or uncertain entries changed. Restoration restores database entries; allow the native daemons time to consume the restored values before timing-sensitive assertions. Do not concurrently edit these entries, restart daemons, change topology or remove the filesystem while applying the profile or restoring it. Cluster termination disposes of the configuration database.
Example ¶
package main
import (
"context"
"errors"
"time"
"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
"github.com/JSYoo5B/ceph-testcontainers-go/cephfs"
)
func main() {
ctx := context.Background()
cluster, err := cephfs.Run(ctx, ceph.DefaultImage, cephfs.WithFilesystems(cephfs.Config{Name: "pins", ActiveMDS: 2}))
if cluster != nil {
defer func() { _ = cluster.Terminate(context.Background()) }()
}
if err != nil {
panic(err)
}
fs := cephfs.Filesystems(cluster.Container)[0]
changes, err := fs.OverrideTiming(ctx, cephfs.TimingProfile{DirectoryPins: true})
// Keep every returned handle even when only part of the profile was applied.
defer func() {
restore, cancel := context.WithTimeout(context.Background(), time.Minute)
defer cancel()
var restoreErr error
for i := len(changes) - 1; i >= 0; i-- {
restoreErr = errors.Join(restoreErr, changes[i].Restore(restore))
}
if restoreErr != nil {
panic(restoreErr)
}
}()
if err != nil {
panic(err)
}
// Prepare several directory-pin transitions, then exercise the client SUT.
// This condition covers the current active MDSs; it is not a failover profile.
}
Output:
func (*Filesystem) Remove ¶ added in v0.4.0
func (fs *Filesystem) Remove(ctx context.Context) error
Remove deletes this owned filesystem: it gracefully stops its ranks and removes it in the FSMap, terminates its MDS daemons, deletes their Cephx keys and removes the metadata, default data and AdditionalDataPools that its configuration created, together with their CRUSH rules and EC profiles. Pools attached later with AddDataPool are detached but kept; remove them with ceph.Pool.Remove (see ceph.Container.Pool). File data in the removed pools is destroyed. Stop clients, CephFS mirrors and subvolume operations on the filesystem first; their handles fail afterwards. A retry after an error continues with whatever remains. If graceful shutdown cannot complete within the caller or startup timeout, removal returns an error without forcing the MDSs to fail. The filesystem may remain marked down and stays owned for a later retry.
Remove acts on this descriptor. One that Remove already removed, or that is not the one the cluster owns for its name, returns an error.
It requests fs set <name> down true and requires the filesystem's native up map to be empty before fs rm, so a rank that is still stopping or reconnecting blocks removal, and health warnings never authorize a forced fs fail. After fs rm it waits until every owned daemon of the other owned filesystems carries its own filesystem's ID as join_fscid in the FSMap again, since some monitors clear it on every remaining daemon until that daemon's next beacon. A retry checks this again even when the filesystem is already absent, so a removal that failed in this wait is never reported as done without it.
func (*Filesystem) ScaleMDS ¶
func (fs *Filesystem) ScaleMDS(ctx context.Context, active, standby int) (returnErr error)
ScaleMDS changes the active/standby daemon counts of this existing filesystem without recreating its pools or data. Active must be 1..256 and Standby must be non-negative. Additional MDSs preserve the initial customizers and affinity. Ceph hands retired ranks back to standbys before this method stops surplus owned daemons. Ordinary standbys are removed before replay followers.
The explicit topology request acknowledges Ceph's max_mds health-warning confirmation (--yes-i-really-mean-it) in this disposable fixture. Run scaling with healthy owned daemons and without concurrent external failure injection or FSMap edits. An error retains desired counts and every partial resource for cluster cleanup; it never deletes the filesystem, its pools or file data. Scaling to zero standby leaves the filesystem's replay preference unchanged.
func (*Filesystem) WaitReady ¶
func (fs *Filesystem) WaitReady(ctx context.Context) error
WaitReady waits for all configured ranks to be owned and active and all configured standbys to register. Replay mode also waits for journal followers. After a standby takes over a failed active daemon, use MDSStatus to check recovered ranks or restart a daemon before WaitReady; readiness includes the requested standby capacity.
Example (Select) ¶
package main
import (
"context"
"fmt"
"time"
"github.com/JSYoo5B/ceph-testcontainers-go/cephfs"
)
func main() {
// This unavailable descriptor verifies expected API-error delivery and the
// one-shot channel lifecycle without Docker. It is not a ready fixture or a
// replacement for Run/Start; real callers use their owned filesystem.
fs := &cephfs.Filesystem{}
waitCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
resultCh := make(chan error, 1)
var results <-chan error = resultCh // One result for one consumer, not a broadcast.
joined := make(chan struct{})
defer func() { cancel(); <-joined }() // Register before launching the worker.
go func() {
defer close(joined)
resultCh <- fs.WaitReady(waitCtx)
close(resultCh)
}()
var outcomeErr error
var open bool
select {
case outcomeErr, open = <-results:
case <-waitCtx.Done():
cancel()
<-joined
outcomeErr, open = <-results
}
cancel()
<-joined // Join before callbacks or fixture cleanup with a fresh context.
if !open {
panic("wait closed without a terminal result")
}
if _, open := <-results; open {
panic("wait sent more than one terminal result")
}
consume := func(err error) {
fmt.Println("consumer received error:", err != nil)
}
consume(outcomeErr) // Caller code after join, outside the wait's fixture gates.
fmt.Println("worker joined and channel closed")
}
Output: consumer received error: true worker joined and channel closed
func (*Filesystem) WatchMDSs ¶ added in v0.5.0
func (fs *Filesystem) WatchMDSs(ctx context.Context, cfg ceph.WatchConfig) (*MDSWatch, error)
WatchMDSs polls the native FSMap and reports changes of this filesystem's MDS ranks and standbys. The default WatchConfig.Interval is 2 seconds.
Before it returns, WatchMDSs makes one read under ctx and QueryTimeout. A failed first read returns its error and starts nothing; a filesystem that is not registered to an open cluster, or has no native identity, is ceph.ErrInvalidRequest. That read is the baseline: the first event, already buffered. The watched filesystem is fixed by its name and the native FSMap ID and pool IDs recorded when it was created, not by its daemons: a takeover, a restarted daemon with a new GID, a ScaleMDS or an external daemon taking a rank are events. Receive the baseline before triggering the change to observe.
Each later read takes the topology lock to snapshot the owned daemons, as MDSStatus does, releases it and then reads one fs dump; it takes no lock while the event is sent or during the wait for the next read, and never the setup gate, so the consumer may call any fixture operation while the watch runs, and reads continue while a ScaleMDS is in progress. A successful read whose Status equals the previous successful read produces no event. A failed read produces a WatchGap and the watch continues; the next successful read is WatchResumed. These end the watch instead: ctx ending, Stop, a terminated cluster, a filesystem that was removed or is absent from a decoded FSMap, one whose FSMap ID or original pools differ from those recorded at creation (recreated under the same name), and a full buffer (ceph.ErrWatchOverflow). Reaching the requested topology does not end it, so later degradation and recovery stay visible.
Ranks and states are what the MONs' FSMap reported, not proof that a daemon process runs, that its container is in a given Docker state or that clients can reach it; ownership is matched by daemon name against the roster snapshot taken just before the FSMap. The stream reports differences between samples, not a complete history of native transitions. It does not wait for readiness: use WaitReady for the requested standby capacity.
Example ¶
The baseline is buffered before WatchMDSs returns, so the fault starts only after the watch has a known starting topology. The takeover event then advances the test while the SUT keeps its mount. This example is compiled, not run.
package main
import (
"context"
"fmt"
"time"
"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
"github.com/JSYoo5B/ceph-testcontainers-go/cephfs"
)
func main() {
ctx := context.Background()
cluster, err := cephfs.Run(ctx, ceph.DefaultImage, cephfs.WithFilesystems(cephfs.Config{StandbyMDS: 1}))
if cluster != nil {
defer func() { _ = cluster.Terminate(context.Background()) }()
}
if err != nil {
panic(err)
}
fs := cephfs.Filesystems(cluster.Container)[0]
watch, err := fs.WatchMDSs(ctx, ceph.WatchConfig{Interval: time.Second})
if err != nil {
panic(err)
}
defer func() { _ = watch.Stop() }() // before the cluster's Terminate
baseline := <-watch.Events()
active := baseline.Status.Active[0]
for _, daemon := range fs.MDSs() {
if daemon.ID == active.Name {
kill := time.Duration(0)
if err := daemon.Stop(ctx, &kill); err != nil {
panic(err)
}
}
}
// ... the SUT runs ...
for event := range watch.Events() {
if event.Kind == ceph.WatchGap {
continue
}
for _, rank := range event.Status.Active {
if rank.Rank == 0 && rank.Owned && rank.Name != active.Name {
fmt.Printf("mds.%s took rank 0 from mds.%s at FSMap epoch %d (baseline %d)\n", rank.Name, active.Name, event.Epoch, baseline.Epoch)
return
}
}
}
panic(watch.Stop()) // the watch ended before a standby took the rank
}
Output:
type FilesystemStatus ¶
type FilesystemStatus struct {
FilesystemName string
FilesystemID int64
MaxMDS int
Active []MDSStatus
Standby []MDSStatus
StandbyReplay []MDSStatus
}
FilesystemStatus contains active ranks and this fixture's ordinary/replay standbys. Standbys belonging to another filesystem are never counted.
type MDS ¶
type MDS struct {
testcontainers.Container
ID string
FilesystemName string
// contains filtered or unexported fields
}
MDS is an owned metadata daemon. ID is its static Ceph daemon name; MDSStatus reports its current rank and GID. Stop/Start can inject failures. Cluster termination removes stopped and partially initialized daemons too.
On a ceph.AddressDualStack cluster every MDS advertises a concrete IPv4 and IPv6 address, IPv4 first; ms_bind_prefer_ipv4 = false in its [mds.<ID>] section of a ceph.WithConfigFile file makes clients use IPv6.
func (*MDS) AddReplacement ¶ added in v0.4.0
AddReplacement starts a new indexed MDS for the filesystem of this stopped MDS, its sole original member, preserving its pools, data and desired one-active/zero-standby policy. The old name must already be globally absent and rank zero failed; native failure is caller-controlled. No native fail, auth deletion or old CID removal is performed. Once the new rank is ready, MDS.RemoveStopped can retire the original old container using the new healthy survivor.
Any failure after the auth/start attempt leaves this request cleanup-only, including its use as typed scaling/retirement authority. A nonnil result remains owned even on error. Retrying the original or an exact copied handle never implicitly starts another daemon or completes a partial request. A completed retry reattests and returns only the exact new handle. Direct Start/Stop, native policy/failure edits and exported-field edits must not run concurrently. Native name/GID and original Docker CID are independent evidence; this method does not establish process-to-GID attribution.
Without a standby, Ceph can keep a dead daemon registered as laggy; fail its GID explicitly with ceph mds fail <gid> before calling it. Do not start, stop or fail daemons by hand while it runs.
func (*MDS) ForceReadOnly ¶ added in v0.5.0
func (daemon *MDS) ForceReadOnly(ctx context.Context) (*MDSReadOnlyFault, error)
ForceReadOnly makes the current process of this owned MDS read-only with the MDS command force_readonly, so that a client meets a metadata server that still answers but refuses metadata mutations with EROFS. The daemon must hold the only active rank of its filesystem (max_mds 1) as up:active. Its rank and GID are read from the FSMap and must be confirmed by the process that answers ceph tell mds.<gid> status. A standby, a daemon whose handle no longer matches what its filesystem created, a filesystem with more ranks, a terminated cluster and a removed filesystem are ceph.ErrInvalidRequest before any change, and so is a daemon whose health already reports MDS_READ_ONLY: that state is not this fault's. A second call for a process this fixture already faulted returns the existing fault with ceph.ErrInvalidRequest. A daemon whose own command list lacks force_readonly is ceph.ErrUnsupported, with the daemon's release in Version.
The command goes to the captured GID, never to the name, so a process that replaced the daemon in the meantime is not hit. After it, the call waits until the daemon reports MDS_READ_ONLY in its health, which arrives with its next beacon, and the same GID still holds the rank. An error after the command was sent returns the fault together with the error.
What the fault proves: the target process has set its in-memory read-only state, reports it to the MONs, and refuses new metadata mutations; a fresh client still reads existing data. What it does not prove: that the metadata pool failed (no OSD or pool is touched, so the cause a real read-only MDS has is absent), that a client which already held write capabilities or buffered data has lost them, how any single read or data write of a retained client ends, or anything about other ranks.
func (*MDS) RemoveStopped ¶ added in v0.4.0
RemoveStopped retires this original owned, stopped MDS container after its immutable daemon name has left every native filesystem and global standby list. Healthy original owned daemons must still serve every desired active rank. It neither fails a native MDS nor deletes its auth entity or changes desired capacity. Use Filesystem.ScaleMDS afterward to create missing replacement members.
Only successful original startups are eligible. A partial startup remains owned for cluster Terminate. An uncertain Docker removal retains the original handle for explicit retry; initial arbitrary absence is not success. Direct failure injection, Start/Stop and raw policy changes must not run concurrently.
type MDSReadOnlyFault ¶ added in v0.5.0
type MDSReadOnlyFault struct {
// contains filtered or unexported fields
}
MDSReadOnlyFault is the record of one MDS process that MDS.ForceReadOnly made read-only. It is a one-way fault, not an override: Ceph has no command that makes a read-only MDS writable again, so the handle has no Restore. The fault ends only when the process is replaced, by failing the daemon to a standby (MDS.Stop, then ceph mds fail <gid>) or by restarting it (MDS.Stop and MDS.Start), and WaitReplaced confirms that a new GID holds the rank. Copies share the record. Retain a non-nil result even on error: the command may have reached the daemon.
func (*MDSReadOnlyFault) String ¶ added in v0.5.0
func (fault *MDSReadOnlyFault) String() string
String reports the faulted process and how the fault ends.
func (*MDSReadOnlyFault) WaitReplaced ¶ added in v0.5.0
func (fault *MDSReadOnlyFault) WaitReplaced(ctx context.Context) (MDSStatus, error)
WaitReplaced waits until the process this fault made read-only no longer exists and another GID of an owned daemon holds its rank as up:active, and returns that daemon as MDSStatus reports it. It never replaces the process itself: stop the daemon and fail its GID so that a standby takes over, or restart it, before or while it waits. The wait ends with the context or the cluster's startup timeout, and its error then says that the faulted GID still holds the rank.
Replaced means, in one FSMap, that the faulted GID appears nowhere and the rank is held by another owned GID; that ceph tell mds.<gid> status of the new GID answers as that rank of this filesystem; and that the health read after it does not report MDS_READ_ONLY for the new holder. A new process starts without the read-only state, but if the cause of a real read-only MDS remained it would set it again: that case keeps the wait going. It does not prove that a client which held a session with the faulted process can write again; mount a new client to check. A rank taken by a daemon this fixture does not own, a filesystem that was removed or recreated under the same name and a terminated cluster are ceph.ErrInvalidRequest at once. WaitReplaced takes the topology lock only to snapshot the owned daemons, as MDSStatus does, so MDS.Stop, MDS.Start and other fixture calls may run while it waits.
type MDSStatus ¶
type MDSStatus struct {
Name string `json:"name"`
Rank int `json:"rank"`
GID uint64 `json:"gid"`
State string `json:"state"`
Owned bool `json:"owned"`
}
MDSStatus records a metadata daemon's current native FSMap identity. GID changes after process restart. Ordinary standbys have rank -1; replay standbys follow the rank they would replace. Owned identifies this fixture's daemons.
type MDSWatch ¶ added in v0.5.0
type MDSWatch struct {
// contains filtered or unexported fields
}
MDSWatch is a running watch of one filesystem's MDS ranks and standbys. One goroutine reads the FSMap and sends events; Events closes after that goroutine has stopped using the cluster. Stop ends the watch and waits for it.
func (*MDSWatch) Events ¶ added in v0.5.0
func (w *MDSWatch) Events() <-chan MDSWatchEvent
Events returns the receive-only event channel. The baseline is already in it when WatchMDSs returns. The channel closes when the watch ends; Stop returns why.
func (*MDSWatch) Stop ¶ added in v0.5.0
Stop cancels the watch if it is still running, waits until its goroutine has returned and Events is closed, and returns the cause that ended the watch: nil when Stop ended it, the context's error when the WatchMDSs context ended, a ceph.ErrWatchOverflow error, or the read error that ended it (a terminated cluster, a removed filesystem or one recreated under the same name). Buffered events stay readable after Stop. Stop is safe to call more than once and from any goroutine; every call returns the same cause. Call it before the fixture's cleanup: cancelling the context alone does not wait.
type MDSWatchEvent ¶ added in v0.5.0
type MDSWatchEvent struct {
Sequence uint64
Kind ceph.WatchEventKind
Started, Finished time.Time
Epoch, MDSMapEpoch uint64
Status FilesystemStatus
Err error
}
MDSWatchEvent is one event of an MDSWatch.
Sequence starts at 1 with the baseline and increases by one for each event, and no event is dropped, so consecutive numbers mean the consumer saw every event the watch produced. Started and Finished are the local clock around the read, not the time Ceph changed state. Epoch is the native FSMap epoch of that read and MDSMapEpoch the epoch of this filesystem's MDSMap in the same FSMap; an epoch change that leaves Status equal produces no event.
Status is what MDSStatus reports for that one FSMap. FilesystemName and FilesystemID are the watched filesystem and never change. MaxMDS and the Active, Standby and StandbyReplay lists are compared field by field (name, rank, GID, state, ownership), each list sorted by rank and then name, so a takeover, a new GID after a restart or a standby that registers or leaves is a change. As in MDSStatus, a rank whose daemon is not up:active (in replay, reconnect or rejoin during a takeover) is in no list, so a takeover shows as the rank leaving Active and coming back under the new name and GID. Each event has its own slices, which the consumer may keep or change. A WatchGap carries a zero Status and zero epochs. Two Statuses separated by a gap, or simply by the polling interval, do not establish the transitions in between.
type MaxFileSizeOverrideHandle ¶ added in v0.5.0
type MaxFileSizeOverrideHandle struct {
// contains filtered or unexported fields
}
MaxFileSizeOverrideHandle restores the max_file_size that OverrideMaxFileSize changed. Copies share restoration state. Retain a non-nil result even on error: a lost reply can leave the value changed.
func (*MaxFileSizeOverrideHandle) Restore ¶ added in v0.5.0
func (change *MaxFileSizeOverrideHandle) Restore(ctx context.Context) error
Restore returns max_file_size to the value captured before the change. It is idempotent across handle copies. A value that is neither the applied nor the captured one is an outside edit and is refused rather than overwritten; a filesystem recreated under the same name is refused too.
type TimingProfile ¶ added in v0.5.0
type TimingProfile struct {
// DirectoryPins reduces MDS upkeep ticks from five seconds to one for
// directory-pin setup on Ceph 19.2.5 and 20.2.4. At least two healthy,
// owned active ranks are required. Other MDS upkeep also runs more often;
// do not use it to qualify session expiry, reconnect or failover behavior.
DirectoryPins bool
}
TimingProfile selects qualified fixture conditions. Its zero value changes nothing. It does not speed up clocks, I/O, failure detection or lease expiry.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package expect evaluates captured observations of a CephFS filesystem against the conditions a test states, and explains each mismatch.
|
Package expect evaluates captured observations of a CephFS filesystem against the conditions a test states, and explains each mismatch. |
|
internal
|
|
|
fsbase
Package fsbase is the foundation that package cephfs and its feature packages share for one filesystem: its identity (the cluster, the authoritative name and the native pool identity captured at creation), the liveness check, the admission lifetime of an operation, the command transport of the MGR volumes module with the readers more than one feature needs, and the native pool reads with the subvolume data-pool checks.
|
Package fsbase is the foundation that package cephfs and its feature packages share for one filesystem: its identity (the cluster, the authoritative name and the native pool identity captured at creation), the liveness check, the admission lifetime of an operation, the command transport of the MGR volumes module with the readers more than one feature needs, and the native pool reads with the subvolume data-pool checks. |
|
fstest
Package fstest holds the unit-test fixtures that the tests of package cephfs and its feature packages share: the canned native output of one synthetic filesystem, a foundation core over it for the feature packages, which cannot construct a cephfs.Filesystem, and the assertions that an operation waits for a held admission gate without native effects.
|
Package fstest holds the unit-test fixtures that the tests of package cephfs and its feature packages share: the canned native output of one synthetic filesystem, a foundation core over it for the feature packages, which cannot construct a cephfs.Filesystem, and the assertions that an operation waits for a held admission gate without native effects. |
|
mdsmap
Package mdsmap decodes the native global MDS map (`ceph fs dump`) for the MDS lifecycle decisions of package cephfs: that a filesystem is still cold, that a stopped MDS is gone from it, and how far its replacement got.
|
Package mdsmap decodes the native global MDS map (`ceph fs dump`) for the MDS lifecycle decisions of package cephfs: that a filesystem is still cold, that a stopped MDS is gone from it, and how far its replacement got. |
|
volbase
Package volbase is the identity and the operation admission of the owned subvolumes and groups that package cephfs/subvolume creates, shared with the packages that act on subvolume handles, such as cephfs/quiesce.
|
Package volbase is the identity and the operation admission of the owned subvolumes and groups that package cephfs/subvolume creates, shared with the packages that act on subvolume handles, such as cephfs/quiesce. |
|
Package mirror mirrors CephFS snapshots of selected directories from a filesystem in one Ceph cluster to a filesystem in another, with native cephfs-mirror daemons: directory enrollment, peer management, MGR reassignment, process quiescence and typed readiness observation.
|
Package mirror mirrors CephFS snapshots of selected directories from a filesystem in one Ceph cluster to a filesystem in another, with native cephfs-mirror daemons: directory enrollment, peer management, MGR reassignment, process quiescence and typed readiness observation. |
|
Package quiesce pauses the I/O of owned CephFS subvolumes as one native quiesce set, so that a test can take a consistent checkpoint while the application under test keeps its clients connected.
|
Package quiesce pauses the I/O of owned CephFS subvolumes as one native quiesce set, so that a test can take a consistent checkpoint while the application under test keeps its clients connected. |
|
Package session observes the client sessions of an existing CephFS filesystem and overrides its MDSMap session policy, so that a test can see a silent or frozen client evicted within seconds, a new client refused while open sessions keep working, or only clients that lack a required client feature refused and evicted.
|
Package session observes the client sessions of an existing CephFS filesystem and overrides its MDSMap session policy, so that a test can see a silent or frozen client evicted within seconds, a new client refused while open sessions keep working, or only clients that lack a required client feature refused and evicted. |
|
Package snapschedule adds snapshot schedules, with their retention, to directories of an existing CephFS filesystem through the MGR snap_schedule module, so that a test can check how its application handles scheduled snapshots and their pruning.
|
Package snapschedule adds snapshot schedules, with their retention, to directories of an existing CephFS filesystem through the MGR snap_schedule module, so that a test can check how its application handles scheduled snapshots and their pruning. |
|
Package subvolume prepares the subvolume state of an existing CephFS filesystem: subvolume groups and subvolumes, their snapshots and clones, principals restricted to one subvolume, and directory pin policies of subvolumes and groups.
|
Package subvolume prepares the subvolume state of an existing CephFS filesystem: subvolume groups and subvolumes, their snapshots and clones, principals restricted to one subvolume, and directory pin policies of subvolumes and groups. |