Documentation
¶
Overview ¶
Package session observes the client sessions of an existing CephFS filesystem and overrides its MDSMap session policy, so that a test can see a silent or frozen client evicted within seconds, a new client refused while open sessions keep working, or only clients that lack a required client feature refused and evicted.
List, Timeouts and RequiredFeatures read native state; Watch polls the sessions and reports admission, state changes and removal from samples that two reads of the active-rank holders bracket. OverrideTimeouts changes both limits, OverrideRefusal makes the MDS reject new sessions and OverrideRequiredFeatures makes it reject and evict only clients that lack the named features; each returns a handle whose Restore puts back what it changed. These functions act on a filesystem that cephfs.Run, cephfs.Start or cephfs.WithFilesystems created; they start no container. Evicting the sessions of one subvolume principal is cephfs/subvolume.Authorization.EvictClients.
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Change ¶
Change is a session whose native state differs from the previous coherent sample: Session is the current row and PreviousState the state it had.
type Feature ¶
Feature is one CephFS client feature: its bit in the MDSMap's required_client_features and the name the cluster reports for it. A bit that the cluster's release does not name keeps the name the MDSMap prints for it.
func RequiredFeatures ¶
RequiredFeatures reads the client features that the MDSMap of fs requires now, sorted by bit. It includes bits that the fixture did not add and bits that the cluster's release does not name. It reads native state only.
type Limits ¶
Limits are the filesystem's MDSMap client session limits, session_timeout and session_autoclose. A silent client whose capabilities nobody wants stays open until Autoclose. Once another client asks for its capabilities, the MDS evicts it as soon as Timeout passes. With Ceph's default mds_session_blocklist_on_timeout the evicted address is blocklisted. These are filesystem settings, not central configuration, so override.SetConfig cannot change them. Native values are whole seconds.
type RankHolder ¶
RankHolder is the MDS daemon that held one active rank when a sample read the FSMap: the rank, the daemon's name and its GID. A daemon that restarts or takes over a rank has a new GID, so an unchanged GID means the same daemon instance kept the rank.
type RefusalOverrideHandle ¶
type RefusalOverrideHandle struct {
// contains filtered or unexported fields
}
RefusalOverrideHandle restores the refuse_client_session flag that OverrideRefusal set. Copies share restoration state. Retain a non-nil result even on error: a lost reply can leave the flag set.
func OverrideRefusal ¶
func OverrideRefusal(ctx context.Context, fs *cephfs.Filesystem) (*RefusalOverrideHandle, error)
OverrideRefusal sets the MDSMap flag refuse_client_session of fs, so that every MDS rank of the filesystem rejects a client that opens a new session: the client's mount fails with EPERM. Sessions that are already open keep working. The handle owns only the change from admitting to refusing and is bound to the filesystem's FSMap ID; a filesystem that already refuses new sessions is refused before any change. Only one handle per filesystem may be active.
While the flag is set, every new session is refused, including the fixture's own short-lived libcephfs probes and a session the MGR volumes module opens, so restore it before such fixture operations. An open session that must reconnect to a restarted or replaced MDS while the flag is set is refused like a new one: the MDS drops it and blocklists the client, whose I/O does not come back after Restore. A cluster whose MDSMap does not report the flag returns ceph.ErrUnsupported before any change.
func (*RefusalOverrideHandle) Restore ¶
func (change *RefusalOverrideHandle) Restore(ctx context.Context) error
Restore clears refuse_client_session again, so new sessions are admitted. It is idempotent across handle copies. A flag that is already clear is accepted without a command. A filesystem recreated under the same name is refused, and its flag is left alone.
type RequiredFeaturesOverrideHandle ¶
type RequiredFeaturesOverrideHandle struct {
// contains filtered or unexported fields
}
RequiredFeaturesOverrideHandle removes again the required client features that OverrideRequiredFeatures added. Copies share restoration state. Retain a non-nil result even on error: a lost reply or a failed later step can leave some features required.
func OverrideRequiredFeatures ¶
func OverrideRequiredFeatures(ctx context.Context, fs *cephfs.Filesystem, names ...string) (*RequiredFeaturesOverrideHandle, error)
OverrideRequiredFeatures adds the named client features to the MDSMap's required_client_features of fs, so that an MDS of the filesystem refuses a new session of a client that does not advertise one of them (its mount fails with EPERM) and evicts an open session that lacks one; with Ceph's default mds_session_blocklist_on_evict the evicted address is blocklisted. Clients that advertise every required feature are admitted as before.
The names must be ones that "fs feature ls" of the cluster's own image lists, other than its placeholder "reserved"; a name it does not list returns ceph.ErrUnsupported, and an empty, repeated or blank name ceph.ErrInvalidRequest, before any change. A feature that the filesystem already requires is recorded as AlreadyRequired: the handle owns nothing for it and its Restore leaves it required. Other bits, including ones the release does not name, are never changed. The handle is bound to the filesystem's FSMap ID. Only one handle per filesystem may be active, even one that owns nothing.
The native command adds one feature at a time, so the override is a sequence, not one atomic change: Steps reports each feature's outcome, and a failure stops the sequence and returns the handle with the error. The requirement is the filesystem's admission policy only. It proves that the MDSMap requires the features; whether a particular client advertises them is the client's own state, which the fixture neither reads nor changes, so observe the client's outcome separately.
func (*RequiredFeaturesOverrideHandle) Restore ¶
func (change *RequiredFeaturesOverrideHandle) Restore(ctx context.Context) error
Restore removes the features that this handle added, in reverse order, and leaves every other bit as it is, including features that were required before the override. It is idempotent across handle copies. A feature that the handle confirmed as added but that is no longer required is an outside edit and is refused, with ceph.ErrInvalidRequest, before any change; a filesystem recreated under the same name is refused too. A feature whose add was sent but never confirmed is removed when it is required and otherwise accepted as not applied.
Restore restores the policy only. A session that the requirement evicted stays closed and its blocklist entry stays; a client must mount again. The native commands address the filesystem by name and the feature by name without a precondition, so an outside remove and re-add of an owned feature between two reads cannot be told apart from no edit.
func (*RequiredFeaturesOverrideHandle) Steps ¶
func (change *RequiredFeaturesOverrideHandle) Steps() []RequirementStep
Steps reports, in request order, what the override found and did for each requested feature. It is a copy of the evidence of the apply; Restore does not change it.
type RequirementStep ¶
type RequirementStep struct {
Feature Feature
// AlreadyRequired reports that the filesystem required the feature
// before the override. The handle sent no command for it, owns nothing
// for it, and its Restore leaves it required.
AlreadyRequired bool
// Sent reports that the add command was attempted. The handle owns the
// bit from then on, even when the reply was lost or the command never ran.
Sent bool
// Confirmed reports that the readback after the command showed the
// feature required.
Confirmed bool
}
RequirementStep is the evidence of one feature that OverrideRequiredFeatures was asked to require, in request order.
type Session ¶
type Session struct {
Rank int
ID uint64
Address string
State string
EntityID string
Hostname string
Root string
PID string
Caps int
}
Session is one client session reported by an active MDS rank. A client with sessions on several ranks appears once per rank. Address is the session's IP:port/nonce, the same form BlocklistEntries reports, so an evicted session can be matched to its blocklist entry. EntityID, Hostname, Root and PID come from the metadata the client sent when it opened the session.
type TimeoutsOverrideHandle ¶
type TimeoutsOverrideHandle struct {
// contains filtered or unexported fields
}
TimeoutsOverrideHandle manages restoration of both session limits, retaining prior state and ownership even when application is uncertain. Copies share restoration state. Retain a non-nil result even on error: a lost reply can leave either value changed.
func OverrideTimeouts ¶
func OverrideTimeouts(ctx context.Context, fs *cephfs.Filesystem, requested Limits) (*TimeoutsOverrideHandle, error)
OverrideTimeouts sets both session limits of fs so a test can observe evicted clients in seconds instead of Ceph's default 60 and 300 seconds. Each value must be a whole number of seconds and at least 30 seconds, the native minimum. Only one handle per filesystem may be active. The native CLI changes one value per command, so apply and restore are not atomic.
func (*TimeoutsOverrideHandle) Restore ¶
func (change *TimeoutsOverrideHandle) Restore(ctx context.Context) error
Restore returns both limits to the values captured before the change. It is idempotent across handle copies. A value that is neither the applied nor the captured one is an outside edit and is refused rather than overwritten.
type WatchEvent ¶
type WatchEvent struct {
Sequence uint64
Kind ceph.WatchEventKind
Started, Finished time.Time
MaxMDS int
Before, After []RankHolder
Sessions []Session
Added, Removed []Session
Changed []Change
Rebaselined []int
Err error
}
WatchEvent is one event of a Watcher.
Sequence starts at 1 with the baseline and increases by one for each event, and no event is dropped, so consecutive numbers mean the consumer saw every event the watch produced. Started and Finished are the local clock around the read, not the time a session changed.
Before and After are the active ranks, sorted by rank, that the FSMap reads before and after the session listings reported; MaxMDS is the max_mds of both. A successful event comes from a coherent sample: Before and After are equal, every rank below MaxMDS is active and at least one rank is. On a WatchGap they show what was read, nil for a read that failed, and Err says why the sample was not used: a failed query, or a roster that changed or did not cover the ranks.
Sessions are the selected sessions of a coherent sample, sorted by rank, ID and address, one row per session per rank; empty but not nil when no selected session is listed, and nil on a gap. A session is identified by its rank, client ID and address (with its nonce): the same client ID on another rank, or with another address after a reconnect, is another session. Rank, ID, Address and State are compared; Caps and the client metadata travel with the event but cause none by themselves.
Added, Removed and Changed compare this sample with the previous coherent one, across any gaps in between, and only for a rank that the same daemon (GID) held in both. A session in Removed was listed by that daemon before and is not listed by it now: it closed, was evicted or killed, or reconnected under a new address (then it is also in Added). Removal does not establish why, nor blocklisting, flush completion or data survival; read those independently. Rebaselined lists, sorted, every rank whose holder differs between the two samples, including a rank that became or stopped being active: its sessions are in Sessions but in none of Added, Removed or Changed, because the sessions a new daemon lists, after a takeover for example, are its own reconnected view, not a continuation of the previous daemon's list. The baseline has only Sessions. Each event has its own slices, which the consumer may keep or change.
type Watcher ¶
type Watcher struct {
// contains filtered or unexported fields
}
Watcher is a running watch of the client sessions of one filesystem. One goroutine reads the sessions and sends events; Events closes after that goroutine has stopped using the filesystem. Stop ends the watch and waits for it.
func Watch ¶
func Watch(ctx context.Context, fs *cephfs.Filesystem, cfg ceph.WatchConfig, clients ...uint64) (*Watcher, error)
Watch polls the client sessions of fs, as List reads them, and reports sessions that appear, change native state (open, stale, killing, closing and the others the MDS reports) or disappear. clients, when given, selects the sessions whose Session.ID is one of them; without it every session is selected, including the fixture's own short-lived libcephfs probes. The default WatchConfig.Interval is 2 seconds.
Before it returns, Watch makes one read under ctx and QueryTimeout. A failed or incoherent first read returns its error and starts nothing; a filesystem that is not the cluster's active owned one, an invalid configuration and a zero client ID are ceph.ErrInvalidRequest. That read is the baseline: the first event, already buffered. Receive it before triggering the change to observe.
Each read takes the filesystem's admission, as List does, through its QueryTimeout context, reads the active ranks, lists the sessions of each active rank in turn and reads the active ranks again, and releases the admission before the event is sent and during the wait for the next read. The ranks are listed one after another, so a sample is not an atomic cross-rank snapshot: each rank's list is one native reply, taken at its own moment. The two rank reads bracket the listings: a sample whose roster of rank holders (rank and GID) or max_mds changed between them, in which a rank below max_mds is not active, or in which no rank is active, is a WatchGap, as is a failed query, and the watch continues; the next coherent read is WatchResumed. A gap never removes a session. An MDS failover therefore shows as gaps while the rank is not active and then as a resumed sample that rebaselines the rank. A coherent read whose selected sessions and roster equal the previous one produces no event.
These end the watch with ceph.ErrInvalidRequest instead: a terminated cluster, a filesystem that was removed or is no longer the registered one, and an FSMap that no longer lists the filesystem or lists it under another FSMap ID (recreated under the same name). ctx ending, Stop and a full buffer (ceph.ErrWatchOverflow) end it too.
The roster is read from the FSMap; the session listings are addressed by rank. Unchanged GIDs before and after mean that the daemon that answered each rank held it throughout, since a daemon instance never returns to a rank under the same GID, but the two reads are not one atomic snapshot. Polling misses a session that opens and closes between reads, and intermediate states that last shorter than the interval. Native tests cover one and two active ranks; more ranks follow the same rules but are exercised by unit tests only. The watch never evicts, blocklists or otherwise changes a session.
Example ¶
The watch selects one client by its native ID and reports its session until the MDS that held it no longer lists it. Gaps, such as samples taken while a standby takes over, never remove it. The blocklist is read on its own: the removal does not say why the session went away. This example is compiled, not run.
package main
import (
"context"
"fmt"
"slices"
"time"
"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
"github.com/JSYoo5B/ceph-testcontainers-go/cephfs"
"github.com/JSYoo5B/ceph-testcontainers-go/cephfs/session"
)
func main() {
ctx := context.Background()
cluster, err := cephfs.Run(ctx, ceph.DefaultImage)
if cluster != nil {
defer func() { _ = cluster.Terminate(context.Background()) }()
}
if err != nil {
panic(err)
}
fs := cephfs.Filesystems(cluster.Container)[0]
// The SUT mounts fs from a client with hostname "sut".
sessions, err := session.List(ctx, fs)
if err != nil {
panic(err)
}
i := slices.IndexFunc(sessions, func(s session.Session) bool { return s.Hostname == "sut" && s.ID != 0 })
if i < 0 {
panic("no mounted session for sut")
}
client := sessions[i]
watchCtx, cancel := context.WithTimeout(ctx, 5*time.Minute)
defer cancel()
w, err := session.Watch(watchCtx, fs, ceph.WatchConfig{}, client.ID)
if err != nil {
panic(err)
}
defer func() { _ = w.Stop() }() // before the cluster's Terminate
sameSession := func(s session.Session) bool {
return s.ID == client.ID && s.Rank == client.Rank && s.Address == client.Address
}
if !slices.ContainsFunc((<-w.Events()).Sessions, sameSession) {
panic("target session absent from baseline")
}
// ... freeze the SUT with fault.Pause ...
for event := range w.Events() {
if slices.Contains(event.Rebaselined, client.Rank) {
panic("target rank changed holder before removal")
}
if !slices.ContainsFunc(event.Removed, sameSession) {
continue
}
observer, err := ceph.Observe(cluster.Container)
if err != nil {
panic(err)
}
entries, err := observer.BlocklistEntries(ctx)
if err != nil {
panic(err)
}
blocklisted := slices.ContainsFunc(entries, func(e ceph.BlocklistEntry) bool { return e.Address == client.Address })
fmt.Println("session removed; address blocklisted:", blocklisted)
return
}
panic(fmt.Sprintf("watch ended before session removal: %v", w.Stop()))
}
Output:
func (*Watcher) Events ¶
func (w *Watcher) Events() <-chan WatchEvent
Events returns the receive-only event channel. The baseline is already in it when Watch returns. The channel closes when the watch ends; Stop returns why.
func (*Watcher) Stop ¶
Stop cancels the watch if it is still running, waits until its goroutine has returned and Events is closed, and returns the cause that ended the watch: nil when Stop ended it, the context's error when the Watch context ended, a ceph.ErrWatchOverflow error, or the ceph.ErrInvalidRequest read error that ended it (a terminated cluster, or a filesystem that was removed, is no longer the cluster's owned one or was recreated under the same name). Buffered events stay readable after Stop. Stop is safe to call more than once and from any goroutine; every call returns the same cause. Call it before the fixture's cleanup: cancelling the context alone does not wait.