expect

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: MIT Imports: 7 Imported by: 0

README

CephFS expect

Import github.com/JSYoo5B/ceph-testcontainers-go/cephfs/expect alongside cephfs when a test wants a precise verdict on an MDS observation it already captured, with an explanation of each mismatch. A filesystem needs nothing from this package to run, and a test that only constructs, queries or watches a filesystem can ignore it. See API purpose and purpose tags for where this fits, and ceph/expect for evaluations of cluster observations.

Every expectation has one Evaluate method that reads only its arguments: no query, wait, retry or testing.T. A mismatch is a result, listed in a deterministic order and summarized by String with a bounded number of entries. An error means there is no verdict: an invalid expectation is ceph.ErrInvalidRequest, and a malformed observation is an error of its own, never a mismatch. The result copies what it evaluated, so its diagnostics describe that observation even after the caller reuses the input. Polling, retries and assertion failures stay with the test.

MDS ranks and standbys

expect.MDSs states what one fs.MDSStatus read, or the Status of one WatchMDSs event, must report:

  • Ranks: ranks that must be up:active, each with an optional holder Name and GID and a NotGID that the holder must no longer have. A takeover is NotGID set to the killed daemon's GID; pinning the baseline GID makes a restarted or replaced daemon fail as gid.
  • Owned: every holder in Ranks is one of this fixture's daemons.
  • AllRanksActive: every rank below the reported max_mds is up:active.
  • Exact: no up:active rank outside Ranks.
  • FilesystemID and MaxMDS: the native filesystem ID, so a filesystem recreated under the same name fails, and max_mds.
  • Standby and StandbyReplay: the exact number of owned ordinary or replay standbys and the daemons, by name or GID, that must be among them. A nil list is not compared; &expect.MDSStandbys{} requires it empty.

MDSEvaluation lists each changed filesystem ID or max_mds, each inactive, renamed, replaced, unchanged, unowned or unselected rank, and each standby count or standby that differs.

baseline, err := fs.MDSStatus(ctx)
if err != nil {
	t.Fatal(err)
}
old := baseline.Active[0]
// ... SIGKILL the active MDS, fail it, and restart its container ...
want := expect.MDSs{
	FilesystemID: baseline.FilesystemID,
	Ranks:        []expect.MDSRank{{Rank: 0, NotGID: old.GID}},
	Owned:        true,
	Exact:        true,
	Standby:      &expect.MDSStandbys{Count: 1, Daemons: []expect.MDSDaemon{{Name: old.Name, NotGID: old.GID}}},
}
status, err := fs.MDSStatus(ctx)
if err != nil {
	t.Fatal(err) // a failed query is never a mismatch
}
result, err := want.Evaluate(*status)
if err != nil {
	t.Fatal(err)
}
if !result.Matched {
	t.Fatal(result) // ... standby count: want 1, observed 0; standby mds.tc-cephfs-0 absent
}

A match proves what that one FSMap reported. A rank still in replay, reconnect or rejoin is in no list and fails as inactive; up:active does not prove that clients reconnected. Unlike fs.WaitReady, which waits for the configured counts, an expectation checks the identities the test names.

Documentation

Overview

Package expect evaluates captured observations of a CephFS filesystem against the conditions a test states, and explains each mismatch.

It is optional: constructing, querying and watching a filesystem needs only package cephfs. A test that wants a precise verdict on one observation, such as "rank 0 is up:active on a daemon other than the one I killed and the restarted daemon is a standby again", states it as an expectation and evaluates the observation it already has:

status, err := fs.MDSStatus(ctx)
if err != nil {
	t.Fatal(err) // a failed query is never a mismatch
}
want := expect.MDSs{
	Ranks:   []expect.MDSRank{{Rank: 0, NotGID: old.GID}},
	Owned:   true,
	Exact:   true,
	Standby: &expect.MDSStandbys{Count: 1, Daemons: []expect.MDSDaemon{{Name: old.Name, NotGID: old.GID}}},
}
result, err := want.Evaluate(*status)
if err != nil {
	t.Fatal(err) // an invalid expectation or a malformed observation
}
if !result.Matched {
	t.Fatal(result) // names each rank or standby and the field that differs
}

Every Evaluate is pure: it reads only its arguments, performs no query, wait or retry, and returns a result that shares no slice with them. A mismatch is a result; an error means there is no verdict. Polling, retries and assertion failures stay with the test and its own tools. Evaluations of cluster observations are in package ceph/expect.

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type MDSDaemon

type MDSDaemon struct {
	Name   string
	GID    uint64
	NotGID uint64
}

MDSDaemon names one expected MDS daemon. Name, when set, selects the entry of that name; otherwise GID selects it, so one of them is required to select a standby. GID, when set, must equal the selected entry's GID and NotGID, when set, must differ from it. In an MDSMismatch it is the expectation of that rank or standby.

type MDSEvaluation

type MDSEvaluation struct {
	Expectation MDSs
	Observed    cephfs.FilesystemStatus
	Mismatches  []MDSMismatch
	Matched     bool
}

MDSEvaluation is the result of evaluating one observation. Expectation and Observed are copies of what was evaluated, so the diagnostics describe that observation even when the caller later reuses its slices. Mismatches are ordered filesystem ID, max_mds, the active ranks by rank, the standbys and then the replay standbys; within one rank in the order of the reason constants, and within one standby list the count before the expected daemons in their order. Matched means there are none.

func (MDSEvaluation) String

func (e MDSEvaluation) String() string

String summarizes the expectation, the observation and at most eight mismatches.

type MDSList

type MDSList string

MDSList names the list of a cephfs.FilesystemStatus a mismatch refers to.

const (
	// MDSListActive is FilesystemStatus.Active, the up:active ranks.
	MDSListActive MDSList = "active"
	// MDSListStandby is FilesystemStatus.Standby, the owned ordinary
	// standbys.
	MDSListStandby MDSList = "standby"
	// MDSListStandbyReplay is FilesystemStatus.StandbyReplay, the owned
	// journal followers.
	MDSListStandbyReplay MDSList = "standby-replay"
)

The lists of a cephfs.FilesystemStatus.

type MDSMismatch

type MDSMismatch struct {
	Reason                   MDSMismatchReason
	List                     MDSList
	Rank                     int
	Want                     MDSDaemon
	Observed                 cephfs.MDSStatus
	WantValue, ObservedValue int64
}

MDSMismatch is one reason an observation does not satisfy an MDSs expectation. List is the list it refers to, empty for MDSMismatchFilesystem and MDSMismatchMaxMDS. Rank is the active rank for MDSListActive and zero otherwise. Want is the expected daemon of that rank or standby, zero for MDSMismatchUnselected and for a rank expected only to be up:active. Observed is the evaluated rank holder or standby entry, zero when there is none. WantValue and ObservedValue are the compared numbers of MDSMismatchFilesystem, MDSMismatchMaxMDS and MDSMismatchCount, zero otherwise.

func (MDSMismatch) String

func (m MDSMismatch) String() string

String names the rank or standby, the reason and the evaluated values.

type MDSMismatchReason

type MDSMismatchReason string

MDSMismatchReason names why an observation does not satisfy an MDSs expectation.

const (
	// MDSMismatchFilesystem: the filesystem ID differs from the pinned one.
	MDSMismatchFilesystem MDSMismatchReason = "filesystem"
	// MDSMismatchMaxMDS: max_mds differs from the expectation.
	MDSMismatchMaxMDS MDSMismatchReason = "max_mds"
	// MDSMismatchInactive: an expected rank has no up:active holder.
	MDSMismatchInactive MDSMismatchReason = "inactive"
	// MDSMismatchName: the holder of a rank has another name.
	MDSMismatchName MDSMismatchReason = "name"
	// MDSMismatchGID: the rank holder or the standby has another GID than
	// the pinned one, so it is another daemon or the daemon restarted.
	MDSMismatchGID MDSMismatchReason = "gid"
	// MDSMismatchUnchanged: the rank holder or the standby still has the GID
	// it must no longer have.
	MDSMismatchUnchanged MDSMismatchReason = "unchanged"
	// MDSMismatchUnowned: Owned is set and the holder of a rank is not one of
	// this fixture's daemons.
	MDSMismatchUnowned MDSMismatchReason = "unowned"
	// MDSMismatchUnselected: an Exact expectation found an up:active rank
	// that Ranks does not name.
	MDSMismatchUnselected MDSMismatchReason = "unselected"
	// MDSMismatchCount: a standby list has another number of entries.
	MDSMismatchCount MDSMismatchReason = "count"
	// MDSMismatchAbsent: an expected standby is not in its list.
	MDSMismatchAbsent MDSMismatchReason = "absent"
)

The reasons an MDSEvaluation reports.

type MDSRank

type MDSRank struct {
	Rank   int
	Name   string
	GID    uint64
	NotGID uint64
}

MDSRank states one rank that must be up:active and, optionally, its holder. Name and GID, when set, must equal the holder's. NotGID, when set, must differ from the holder's GID: a baseline GID that a takeover or a restart replaces. A rank with none of them only has to be up:active.

type MDSStandbys

type MDSStandbys struct {
	Count   int
	Daemons []MDSDaemon
}

MDSStandbys states one standby list of an MDSs expectation. Count is the exact number of entries the list must hold, and Daemons are the daemons that must be among them, so Count may not be smaller than len(Daemons). A zero MDSStandbys requires an empty list.

type MDSs

type MDSs struct {
	FilesystemID   int64
	MaxMDS         int
	Ranks          []MDSRank
	Owned          bool
	AllRanksActive bool
	Exact          bool
	Standby        *MDSStandbys
	StandbyReplay  *MDSStandbys
}

MDSs states the MDS topology a test expects of one filesystem in one cephfs.Filesystem.MDSStatus observation, or the Status of one cephfs.MDSWatchEvent. Every field is optional, but an expectation must state at least one condition.

FilesystemID, when positive, pins the native filesystem ID (fscid), so a filesystem recreated under the same name does not match. MaxMDS, when positive, is the max_mds the MDSMap must report.

Ranks are the ranks that must be up:active, each with the holder it states: a Name or GID pins the daemon, and NotGID requires a holder other than that baseline GID, which is how a takeover or a restarted daemon shows. Owned additionally requires every holder in Ranks to be one of this fixture's daemons. AllRanksActive requires every rank below the observed MaxMDS to be up:active, and Exact requires no up:active rank outside Ranks.

Standby and StandbyReplay, when non-nil, state the fixture's ordinary and replay standbys of this filesystem: the exact count and daemons that must be among them.

A match proves only what that one FSMap reported. A rank in replay, reconnect or rejoin during a takeover is not up:active, so it does not match until the takeover completes; up:active is not proof that clients reconnected or that the daemon serves I/O. Standbys of other filesystems and external standbys are never in the observation. Unlike cephfs.Filesystem.WaitReady, which waits for the configured counts, an expectation checks the identities a test names in one captured observation and explains which rank or standby differs.

func (MDSs) Evaluate

func (e MDSs) Evaluate(observed cephfs.FilesystemStatus) (MDSEvaluation, error)

Evaluate checks one successful MDSStatus observation against the expectation. It reads only its arguments: no query, wait or retry, so a test composes it with its own polling or assertion helpers. A mismatch is a result; an error means no verdict.

An expectation that states nothing, has a negative FilesystemID, MaxMDS, rank or standby count, names a rank twice or at or above its own MaxMDS, sets Owned without Ranks, pins a GID equal to its NotGID, selects a standby by neither name nor GID, expects more standbys than its Count, or names one daemon or GID in two places is ceph.ErrInvalidRequest. An observation with a negative filesystem ID, a max_mds below 1 (such as the zero Status of a cephfs.MDSWatchEvent gap), an entry without a name or GID or in a state its list does not hold, an unowned standby, a repeated active rank, or a name or GID in two entries is a malformed observation, never a mismatch. The result shares no slice with its arguments.

Example

A takeover is rank 0 on a GID other than the killed one, and the restarted daemon is a standby again under a new GID. Evaluating the observation of a takeover that is still in progress names what is missing.

package main

import (
	"fmt"

	"github.com/JSYoo5B/ceph-testcontainers-go/cephfs"
	"github.com/JSYoo5B/ceph-testcontainers-go/cephfs/expect"
)

func main() {
	old := cephfs.MDSStatus{Name: "tc-cephfs-0", Rank: 0, GID: 4100, State: "up:active", Owned: true}
	want := expect.MDSs{
		FilesystemID: 1,
		Ranks:        []expect.MDSRank{{Rank: 0, NotGID: old.GID}},
		Owned:        true,
		Exact:        true,
		Standby:      &expect.MDSStandbys{Count: 1, Daemons: []expect.MDSDaemon{{Name: old.Name, NotGID: old.GID}}},
	}
	// The standby took rank 0; the killed daemon has not registered again.
	later := cephfs.FilesystemStatus{
		FilesystemName: "tc-cephfs", FilesystemID: 1, MaxMDS: 1,
		Active: []cephfs.MDSStatus{{Name: "tc-cephfs-1", Rank: 0, GID: 4200, State: "up:active", Owned: true}},
	}
	result, err := want.Evaluate(later)
	if err != nil {
		panic(err) // invalid expectation or malformed observation: no verdict
	}
	fmt.Println(result.Matched)
	fmt.Println(result)
}
Output:
false
mds want fs=1 ranks [0] owned exact standby=1 in "tc-cephfs" fs=1 max_mds=1 active=1 standby=0 standby-replay=0: standby count: want 1, observed 0; standby mds.tc-cephfs-0 absent
Example (Polling)

The test owns the polling: a failed query is kept apart from the last mismatch, so a timeout reports both instead of an empty "not ready". This example is compiled, not run.

package main

import (
	"context"
	"fmt"
	"time"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/JSYoo5B/ceph-testcontainers-go/cephfs"
	"github.com/JSYoo5B/ceph-testcontainers-go/cephfs/expect"
)

func main() {
	ctx := context.Background()
	cluster, err := cephfs.Run(ctx, ceph.DefaultImage, cephfs.WithFilesystems(cephfs.Config{StandbyMDS: 1}))
	if cluster != nil {
		defer func() { _ = cluster.Terminate(context.Background()) }()
	}
	if err != nil {
		panic(err)
	}
	fs := cephfs.Filesystems(cluster.Container)[0]
	baseline, err := fs.MDSStatus(ctx)
	if err != nil {
		panic(err)
	}
	old := baseline.Active[0]
	// ... stop the active MDS while the SUT runs ...
	want := expect.MDSs{FilesystemID: baseline.FilesystemID, Ranks: []expect.MDSRank{{Rank: 0, NotGID: old.GID}}, Owned: true, Exact: true}
	ctx, cancel := context.WithTimeout(ctx, 3*time.Minute)
	defer cancel()
	var last expect.MDSEvaluation
	var queryErr error
	for {
		status, err := fs.MDSStatus(ctx)
		if err == nil {
			last, err = want.Evaluate(*status)
		}
		if err == nil && last.Matched {
			return
		}
		if err != nil {
			queryErr = err
		}
		select {
		case <-ctx.Done():
			panic(fmt.Sprintf("rank 0 not taken over: last %v, last error %v", last, queryErr))
		case <-time.After(time.Second):
		}
	}
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL