expect

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: MIT Imports: 8 Imported by: 0

README

Expect

Import github.com/JSYoo5B/ceph-testcontainers-go/ceph/expect alongside ceph when a test wants a precise verdict on an observation it already captured, with an explanation of each mismatch. A cluster needs nothing from this package to run, and a test that only constructs, queries or watches a cluster can ignore it. See API purpose and purpose tags for where this fits.

Every expectation has one Evaluate method that reads only its arguments: no query, wait, retry or testing.T. A mismatch is a result, listed in a deterministic order and summarized by String with a bounded number of entries. An error means there is no verdict: an invalid expectation is ceph.ErrInvalidRequest, and a malformed observation is an error of its own, never a mismatch. The result copies what it evaluated, so its diagnostics describe that observation even after the caller reuses the input. Polling, retries and assertion failures stay with the test.

OSD membership

expect.OSDs states the up and in flags that selected owned OSDs must report in one ceph.Observer.OSDStates observation. Select the OSDs from an earlier OSDStates baseline to pin their native UUIDs, so an ID that a replacement re-registered does not pass, and set Exact when no other owned OSD may exist. OSDEvaluation lists each absent, replaced, down, out or unselected OSD.

baseline, err := observer.OSDStates(ctx)
if err != nil {
	t.Fatal(err)
}
// ... stop and restart an OSD while the SUT runs ...
want := expect.OSDs{Select: baseline, Up: true, In: true, Exact: true}
states, err := observer.OSDStates(ctx)
if err != nil {
	t.Fatal(err) // a failed query is never a mismatch
}
result, err := want.Evaluate(states)
if err != nil {
	t.Fatal(err)
}
if !result.Matched {
	t.Fatal(result) // osds [0 1] want up=true in=true exact in 2 observed: osd.1 up: want true, observed false
}

Unlike ceph.Observer.WaitForClean, which compares the cluster-wide total, up and in OSD counts of the cluster status with the number of owned OSDs, an expectation checks individual identities. When to use which is in What the waits prove about an OSD.

Health checks and mutes

expect.Health states the checks and stored mutes one ceph.Observer.HealthDetails observation must report. Require lists checks by native code, each with an optional severity, summary count and muted or unmuted state; Forbid lists codes that must be absent. Mutes and NoMutes list codes that must or must not have a stored mute, which a sticky mute can keep after its check clears. Status compares the overall status, which Ceph derives from the worst unmuted check, FSID pins the cluster, and Exact allows no other check or mute. Messages are never compared, since they carry countdowns and application data. HealthEvaluation lists each absent, forbidden or unexpected check and each severity, count, mute flag or stored mute that differs.

want := expect.Health{
	Require: []expect.HealthCheck{{Code: "OSD_DOWN", Severity: "HEALTH_WARN", Muted: expect.HealthUnmuted, Count: 1}},
	NoMutes: []string{"OSD_DOWN"},
}
health, err := observer.HealthDetails(ctx)
if err != nil {
	t.Fatal(err) // a failed query is never a mismatch
}
result, err := want.Evaluate(health)
if err != nil {
	t.Fatal(err)
}
if !result.Matched {
	t.Fatal(result) // health want require=[OSD_DOWN] nomutes=[OSD_DOWN] in HEALTH_OK with 0 checks, 0 mutes: OSD_DOWN absent
}

Ceph raises many checks only after a delay and clears them at a later monitor tick, so a test polls with its own deadline. A cleared check does not prove that the fault behind it is gone.

Pool placement groups

expect.PGs states the PGs one ceph.Pool.PGs observation must report. Count or Select names the PGs to check, so an empty report never matches by having nothing to check. Each checked PG must report the exact State (such as active+clean), carry the whole state tokens of Require and none of Forbid, report the exact ordered Up and Acting vectors or an ActingSize, contain the OSDs of InUp and InActing and none of NotInUp and NotInActing, and have one of the ActingPrimary values. Take Pool from an earlier observation, such as a baseline's PoolAfter, to pin the native pool ID, so a pool recreated under the same name does not pass. PGEvaluation lists each PG and condition that differs.

baseline, err := pool.PGs(ctx)
if err != nil {
	t.Fatal(err)
}
// ... stop and restart osd.2 while the SUT runs ...
want := expect.PGs{Pool: baseline.PoolAfter, Count: len(baseline.PGs), State: "active+clean", InActing: []int{2}}
snapshot, err := pool.PGs(ctx)
if err != nil {
	t.Fatal(err) // a failed query is never a mismatch
}
result, err := want.Evaluate(snapshot)
if err != nil {
	t.Fatal(err)
}
if !result.Matched {
	t.Fatal(result) // ... pg 3.1 state: want active+clean, observed active+undersized+degraded; pg 3.1 acting missing osd.2: observed [0 1]
}

Unlike ceph.Observer.WaitForClean, which waits for every PG of the cluster to report active+clean, an expectation checks one pool's PGs once. The PG reports arrive from the MGR asynchronously, so a match is no proof of write visibility or durability, and the native PGReady flag is not compared.

MON quorum

expect.Monitors states the monmap members and the quorum of one ceph.Observer.QuorumStatus observation. Members and Quorum are exact name sets; MemberCount and QuorumSize count them for a test whose MONs get new names, such as a rolling replacement. FSID pins the cluster, so a rebuilt cluster that reuses the MON names does not pass. Every set field is a condition and a zero field is not checked. MonitorEvaluation names each member that is absent, unexpected, out of the quorum or in it unexpectedly.

// ... stop mon.c while the SUT runs ...
want := expect.Monitors{Members: []string{"a", "b", "c"}, Quorum: []string{"a", "b"}}
status, err := observer.QuorumStatus(ctx)
if err != nil {
	t.Fatal(err) // a failed query is never a mismatch
}
result, err := want.Evaluate(status)
if err != nil {
	t.Fatal(err)
}
if !result.Matched {
	t.Fatal(result) // mons want members [a b c] quorum [a b] in monmap [a b c] quorum [a b c]: mon.c in quorum
}

A quorum that is not a majority of the expected members is an invalid request, since a QuorumStatus answer always comes from a majority. A match does not say which MON leads or that a member out of the quorum has stopped. Unlike ceph.Observer.WaitForQuorum, which accepts any majority, an expectation names the members the test requires.

MGR registrations

expect.Managers states the MGRMap of one ceph.Observer.ManagerStatus observation, or of the Status of a ceph.ManagerWatchEvent. Available is always compared. Active names the active MGR, ActiveGID pins its registration so a restart under the same name does not pass, and FormerGID requires an active MGR with any GID other than an earlier one, which states a failover. Standbys is the exact set of standby names, with an empty non-nil slice for none, and StandbyCount counts them. ManagerEvaluation names the field that differs and each absent or unexpected standby.

before, err := observer.ManagerStatus(ctx)
if err != nil {
	t.Fatal(err)
}
// ... fail the active MGR while the SUT runs ...
want := expect.Managers{Available: true, Active: before.Standbys[0].Name, FormerGID: before.ActiveGID}
status, err := observer.ManagerStatus(ctx)
if err != nil {
	t.Fatal(err)
}
result, err := want.Evaluate(status)
if err != nil {
	t.Fatal(err)
}
if !result.Matched {
	t.Fatal(result) // ...: active: want mgr.b, observed mgr.a; active gid: want other than 4100, observed 4100
}

A match does not prove that a module serves requests or that a stopped MGR has exited. Unlike ceph.Observer.WaitForClean, which requires only an available MGR, an expectation names the registrations the test requires.

Documentation

Overview

Package expect evaluates captured observations of a ceph cluster against the conditions a test states, and explains each mismatch.

It is optional: constructing, querying and watching a cluster needs only package ceph. A test that wants a precise verdict on one observation, such as "these pinned OSDs are up and in and no other owned OSD exists", states it as an expectation and evaluates the observation it already has:

states, err := observer.OSDStates(ctx)
if err != nil {
	t.Fatal(err) // a failed query is never a mismatch
}
result, err := expect.OSDs{Select: baseline, Up: true, In: true, Exact: true}.Evaluate(states)
if err != nil {
	t.Fatal(err) // an invalid expectation or a malformed observation
}
if !result.Matched {
	t.Fatal(result) // names each OSD and the field that differs
}

Every Evaluate is pure: it reads only its arguments, performs no query, wait or retry, and returns a result that shares no slice or map with them. A mismatch is a result; an error means there is no verdict. Polling, retries and assertion failures stay with the test and its own tools. Evaluations of service observations live in the service's own expect package, such as cephfs/expect, rbd/expect or rgw/expect.

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Health

type Health struct {
	FSID    string
	Status  string
	Require []HealthCheck
	Forbid  []string
	Mutes   []string
	NoMutes []string
	Exact   bool
}

Health states the health checks and stored mutes a test expects in one ceph.Observer.HealthDetails observation. FSID, when set, pins the cluster, so an observation of another cluster does not match. Status, when set, is the overall status the observation must report; Ceph derives it from the worst unmuted check, so HEALTH_OK can accompany muted warnings. Require lists checks that must be present and Forbid codes that must be absent. Mutes lists codes that must have a stored mute and NoMutes codes that must not; a stored sticky mute can outlive its check. Exact also requires no check outside Require and no stored mute outside Mutes.

A match proves only what this one observation reported. Ceph raises many checks after a delay and clears them at a later monitor tick, so a missing check is not proof that the condition behind it is gone, and a present check says nothing about the client data path. Unlike the readiness waits of package ceph, such as ceph.Observer.WaitForClean, an expectation never polls and names each check that differs.

func (Health) Evaluate

func (e Health) Evaluate(observed ceph.HealthSnapshot) (HealthEvaluation, error)

Evaluate checks one successful ceph.Observer.HealthDetails observation against the expectation. It reads only its arguments: no query, wait or retry, so a test composes it with its own polling or assertion helpers. A mismatch is a result; an error means no verdict. An expectation that states nothing, an invalid FSID, status, severity or mute state, an empty or repeated code, and a contradiction (a code both required and forbidden, both in Mutes and NoMutes, a muted required check without a stored mute or an unmuted one with a stored mute, or an unmuted required severity worse than Status) are ceph.ErrInvalidRequest. An observation with an invalid status or severity, an empty check code, an empty or repeated mute code, or an invalid FSID when the expectation pins one is a malformed observation, never a mismatch. The result shares no slice or map with its arguments.

Example

A health expectation compares check codes, severities, counts and mute state, never the messages, and names each check that differs.

package main

import (
	"fmt"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/JSYoo5B/ceph-testcontainers-go/ceph/expect"
)

func main() {
	observed := ceph.HealthSnapshot{
		FSID:   "44444444-4444-4444-8444-444444444444",
		Status: "HEALTH_WARN",
		Checks: map[string]ceph.HealthCheck{
			"OSD_DOWN": {Severity: "HEALTH_WARN", Summary: "1 osds down", Count: 1, Details: []string{"osd.1 is down"}},
			"SLOW_OPS": {Severity: "HEALTH_WARN", Summary: "2 slow ops", Count: 2},
		},
		Mutes: []ceph.HealthMute{},
	}
	want := expect.Health{
		Require: []expect.HealthCheck{{Code: "OSD_DOWN", Severity: "HEALTH_WARN", Muted: expect.HealthUnmuted, Count: 1}},
		Forbid:  []string{"SLOW_OPS"},
		NoMutes: []string{"OSD_DOWN"},
	}
	result, err := want.Evaluate(observed)
	if err != nil {
		panic(err) // invalid expectation or malformed observation: no verdict
	}
	fmt.Println(result.Matched)
	fmt.Println(result)
}
Output:
false
health want require=[OSD_DOWN] forbid=[SLOW_OPS] nomutes=[OSD_DOWN] in HEALTH_WARN with 2 checks, 0 mutes: SLOW_OPS present: HEALTH_WARN count=2 unmuted

type HealthCheck

type HealthCheck struct {
	Code     string
	Severity string
	Muted    HealthMuteState
	Count    int64
}

HealthCheck is one health check that a Health expectation requires. Code is the native, case-sensitive check code such as "OSD_DOWN". Severity, when set, is the native severity the check must report (HEALTH_OK, HEALTH_WARN or HEALTH_ERR). Muted states the reported mute flag. Count, when nonzero, is the native summary count the check must report; a count of zero cannot be required. The free-form summary and detail messages are never compared: they carry pool names, paths and other application data.

type HealthEvaluation

type HealthEvaluation struct {
	Expectation Health
	Observed    ceph.HealthSnapshot
	Mismatches  []HealthMismatch
	Matched     bool
}

HealthEvaluation is the result of evaluating one health observation. Expectation and Observed are deep copies of what was evaluated, so the diagnostics describe that observation even when the caller later reuses its slices and maps. Mismatches list HealthMismatchFSID and HealthMismatchStatus first, then the codes in ascending order, each code's reasons in the order of the reason constants; Matched means there are none.

func (HealthEvaluation) String

func (e HealthEvaluation) String() string

String summarizes the expectation and at most eight mismatches.

type HealthMismatch

type HealthMismatch struct {
	Reason         HealthMismatchReason
	Code           string
	Want, Observed string
}

HealthMismatch is one reason a health observation does not satisfy an expectation. Code is the check or mute code, empty for HealthMismatchFSID and HealthMismatchStatus. Want and Observed are the compared values as text, both empty where the reason alone says it: Want is empty for HealthMismatchPresent, HealthMismatchUnexpected, HealthMismatchMutePresent and HealthMismatchMuteUnexpected, whose Observed describes what was reported, and Observed is empty for HealthMismatchAbsent and HealthMismatchMuteAbsent.

func (HealthMismatch) String

func (m HealthMismatch) String() string

String names the code, the reason and the evaluated values.

type HealthMismatchReason

type HealthMismatchReason string

HealthMismatchReason names why a health observation does not satisfy a Health expectation.

const (
	// HealthMismatchFSID: the observation belongs to another cluster.
	HealthMismatchFSID HealthMismatchReason = "fsid"
	// HealthMismatchStatus: the overall status differs.
	HealthMismatchStatus HealthMismatchReason = "status"
	// HealthMismatchAbsent: a required check is not reported. Its severity,
	// mute flag and count are not compared.
	HealthMismatchAbsent HealthMismatchReason = "absent"
	// HealthMismatchSeverity: a required check reports another severity.
	HealthMismatchSeverity HealthMismatchReason = "severity"
	// HealthMismatchMuted: a required check reports another mute flag.
	HealthMismatchMuted HealthMismatchReason = "muted"
	// HealthMismatchCount: a required check reports another summary count.
	HealthMismatchCount HealthMismatchReason = "count"
	// HealthMismatchPresent: a forbidden check is reported.
	HealthMismatchPresent HealthMismatchReason = "present"
	// HealthMismatchUnexpected: an Exact expectation found a check that
	// neither Require nor Forbid names (a forbidden check is
	// HealthMismatchPresent).
	HealthMismatchUnexpected HealthMismatchReason = "unexpected"
	// HealthMismatchMuteAbsent: a code listed in Mutes has no stored mute.
	HealthMismatchMuteAbsent HealthMismatchReason = "mute absent"
	// HealthMismatchMutePresent: a code listed in NoMutes has a stored mute.
	HealthMismatchMutePresent HealthMismatchReason = "mute present"
	// HealthMismatchMuteUnexpected: an Exact expectation found a stored mute
	// that neither Mutes nor NoMutes names (a mute listed in NoMutes is
	// HealthMismatchMutePresent).
	HealthMismatchMuteUnexpected HealthMismatchReason = "mute unexpected"
)

The reasons a HealthEvaluation reports.

type HealthMuteState

type HealthMuteState string

HealthMuteState states whether a required health check must be muted.

const (
	// HealthMuteAny does not compare the check's Muted flag.
	HealthMuteAny HealthMuteState = ""
	// HealthMuted requires the check to be reported as muted.
	HealthMuted HealthMuteState = "muted"
	// HealthUnmuted requires the check to be reported as not muted, so it
	// counts towards the overall status.
	HealthUnmuted HealthMuteState = "unmuted"
)

The mute states a HealthCheck can require.

type ManagerEvaluation

type ManagerEvaluation struct {
	Expectation Managers
	Observed    ceph.ManagerStatus
	Mismatches  []ManagerMismatch
	Matched     bool
}

ManagerEvaluation is the result of evaluating one observation. Expectation and Observed are copies of what was evaluated, so the diagnostics describe that observation even when the caller later reuses its slices. Mismatches list the reasons about availability, the active MGR and the standby count first, in the order of the reason constants, then the per-standby reasons ordered by name and then in the order of the reason constants; Matched means there are none.

func (ManagerEvaluation) String

func (e ManagerEvaluation) String() string

String summarizes the expectation, the observed MGRMap and at most eight mismatches.

type ManagerMismatch

type ManagerMismatch struct {
	Reason            ManagerMismatchReason
	Name              string
	GID               uint64
	Count             int
	ObservedAvailable bool
	ObservedName      string
	ObservedGID       uint64
	ObservedCount     int
}

ManagerMismatch is one reason an observation does not satisfy a Managers expectation. The expected side is Name for ManagerMismatchActive and the standby reasons, GID for ManagerMismatchActiveGID and ManagerMismatchFormerGID, and Count for ManagerMismatchStandbyCount. The observed side is ObservedAvailable for ManagerMismatchAvailable, ObservedName for ManagerMismatchActive (empty without an active MGR), ObservedGID for the GID reasons (zero without an active MGR) and for ManagerMismatchStandbyUnexpected (that standby's GID), and ObservedCount for ManagerMismatchStandbyCount.

func (ManagerMismatch) String

func (m ManagerMismatch) String() string

String names the reason and the evaluated values.

type ManagerMismatchReason

type ManagerMismatchReason string

ManagerMismatchReason names why an observation does not satisfy a Managers expectation.

const (
	// ManagerMismatchAvailable: the reported availability differs from the
	// expectation.
	ManagerMismatchAvailable ManagerMismatchReason = "available"
	// ManagerMismatchActive: the active MGR has another name, or there is
	// none.
	ManagerMismatchActive ManagerMismatchReason = "active"
	// ManagerMismatchActiveGID: the active MGR carries another GID than the
	// pinned one, or there is none.
	ManagerMismatchActiveGID ManagerMismatchReason = "active-gid"
	// ManagerMismatchFormerGID: the active MGR still carries the former GID,
	// or there is none.
	ManagerMismatchFormerGID ManagerMismatchReason = "former-gid"
	// ManagerMismatchStandbyCount: the MGRMap lists another number of
	// standbys.
	ManagerMismatchStandbyCount ManagerMismatchReason = "standby-count"
	// ManagerMismatchStandbyAbsent: a name of Standbys is not a standby.
	ManagerMismatchStandbyAbsent ManagerMismatchReason = "standby-absent"
	// ManagerMismatchStandbyUnexpected: a standby that Standbys does not name.
	ManagerMismatchStandbyUnexpected ManagerMismatchReason = "standby-unexpected"
)

The reasons a ManagerEvaluation reports.

type Managers

type Managers struct {
	Available    bool
	Active       string
	ActiveGID    uint64
	FormerGID    uint64
	Standbys     []string
	StandbyCount int
}

Managers states the MGRMap ownership a test expects in one ceph.Observer.ManagerStatus observation, such as the status of a ceph.ManagerWatchEvent.

Available is always compared: the zero value expects an MGRMap with no available MGR. Active, when set, is the name the active MGR must have, and ActiveGID, when set, pins its registration: a GID from an earlier observation does not match an MGR that re-registered under the same name. FormerGID, when set, is the GID of an earlier active MGR that the current active one must not carry, which states a failover or a restart of the active MGR; it also requires an active MGR. Standbys, when not nil, is the exact set of standby names, so an empty non-nil slice requires that there is no standby. StandbyCount, when positive, is the number of standbys, for a test that does not know their names.

A match proves only what that one MGRMap reported: it does not prove that a module serves requests, that a standby process runs, or that a stopped MGR has exited. Unlike ceph.Observer.WaitForClean, which requires only an available MGR, an expectation names the MGR registrations the test requires.

func (Managers) Evaluate

func (e Managers) Evaluate(observed ceph.ManagerStatus) (ManagerEvaluation, error)

Evaluate checks one successful ceph.Observer.ManagerStatus observation against the expectation. It reads only its arguments: no query, wait or retry, so a test composes it with its own polling or assertion helpers. A mismatch is a result; an error means no verdict.

A negative StandbyCount, a StandbyCount that contradicts Standbys, an empty or repeated standby name, or a FormerGID equal to ActiveGID is ceph.ErrInvalidRequest. An observation whose active name and GID disagree about whether there is an active MGR, that is available without one, or that lists a standby with no name or GID or with a GID that another registration carries is a malformed observation, never a mismatch. The result shares no slice with its arguments.

Example

A failover expectation names the standby that must take over and the GID of the active MGR it must replace, taken from an earlier observation.

package main

import (
	"encoding/json"
	"fmt"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/JSYoo5B/ceph-testcontainers-go/ceph/expect"
)

func main() {
	var before, after ceph.ManagerStatus // as observer.ManagerStatus decodes them
	if err := json.Unmarshal([]byte(`{"available":true,"active_name":"a","active_gid":4100,
		"standbys":[{"name":"b","gid":4200}]}`), &before); err != nil {
		panic(err)
	}
	// ... fail the active MGR while the SUT runs ...
	if err := json.Unmarshal([]byte(`{"available":false,"active_name":"a","active_gid":4100,
		"standbys":[{"name":"b","gid":4200}]}`), &after); err != nil {
		panic(err)
	}
	want := expect.Managers{Available: true, Active: before.Standbys[0].Name, FormerGID: before.ActiveGID}
	result, err := want.Evaluate(after)
	if err != nil {
		panic(err) // invalid expectation or malformed observation: no verdict
	}
	fmt.Println(result.Matched)
	fmt.Println(result)
}
Output:
false
mgrs want available=true active mgr.b gid other than 4100 in available=false active mgr.a gid 4100 standbys [b]: available: want true, observed false; active: want mgr.b, observed mgr.a; active gid: want other than 4100, observed 4100

type MonitorEvaluation

type MonitorEvaluation struct {
	Expectation Monitors
	Observed    ceph.QuorumStatus
	Mismatches  []MonitorMismatch
	Matched     bool
}

MonitorEvaluation is the result of evaluating one observation. Expectation and Observed are copies of what was evaluated, so the diagnostics describe that observation even when the caller later reuses its slices. Mismatches list the reasons without a MON name first, in the order of the reason constants, then the per-member reasons ordered by name and then in the order of the reason constants; Matched means there are none.

func (MonitorEvaluation) String

func (e MonitorEvaluation) String() string

String summarizes the expectation, the observed membership and at most eight mismatches.

type MonitorMismatch

type MonitorMismatch struct {
	Reason               MonitorMismatchReason
	Name                 string
	FSID, ObservedFSID   string
	Count, ObservedCount int
}

MonitorMismatch is one reason an observation does not satisfy a Monitors expectation. Name is the MON for the per-member reasons and empty for the others. FSID and ObservedFSID are the pinned and the observed FSID for MonitorMismatchFSID. Count and ObservedCount are the expected and the observed number for MonitorMismatchMemberCount and MonitorMismatchQuorumSize.

func (MonitorMismatch) String

func (m MonitorMismatch) String() string

String names the reason and the evaluated values.

type MonitorMismatchReason

type MonitorMismatchReason string

MonitorMismatchReason names why an observation does not satisfy a Monitors expectation.

const (
	// MonitorMismatchFSID: the observation is of a cluster with another FSID.
	MonitorMismatchFSID MonitorMismatchReason = "fsid"
	// MonitorMismatchMemberCount: the monmap lists another number of members.
	MonitorMismatchMemberCount MonitorMismatchReason = "member-count"
	// MonitorMismatchQuorumSize: the quorum holds another number of members.
	MonitorMismatchQuorumSize MonitorMismatchReason = "quorum-size"
	// MonitorMismatchAbsent: a name of Members or Quorum is not in the monmap.
	// Its quorum membership is not reported separately.
	MonitorMismatchAbsent MonitorMismatchReason = "absent"
	// MonitorMismatchUnexpected: the monmap lists a name that Members does not.
	MonitorMismatchUnexpected MonitorMismatchReason = "unexpected"
	// MonitorMismatchOutOfQuorum: a name of Quorum is in the monmap but not in
	// the quorum.
	MonitorMismatchOutOfQuorum MonitorMismatchReason = "out-of-quorum"
	// MonitorMismatchInQuorum: the quorum holds a name that Quorum does not.
	MonitorMismatchInQuorum MonitorMismatchReason = "in-quorum"
)

The reasons a MonitorEvaluation reports.

type Monitors

type Monitors struct {
	FSID        string
	Members     []string
	MemberCount int
	Quorum      []string
	QuorumSize  int
}

Monitors states the monmap membership and quorum a test expects in one ceph.Observer.QuorumStatus observation. Every set field is a condition and a zero field is not checked; at least one must be set.

FSID pins the cluster: a QuorumStatus read from another cluster, such as a rebuilt one, does not match. Members is the exact set of MON names the monmap must list and MemberCount the number of members it must list, for a test whose members get new names. Quorum is the exact set of MON names the quorum must hold and QuorumSize the number it must hold. Names compare as sets, so their order is irrelevant.

A match proves only what that one quorum_status answer reported: it does not prove that a member outside the quorum is stopped, that one inside it accepts new sessions, or which MON leads. Unlike ceph.Observer.WaitForQuorum, which accepts any majority of the current monmap, an expectation names the members and the quorum that the test requires.

func (Monitors) Evaluate

func (e Monitors) Evaluate(observed ceph.QuorumStatus) (MonitorEvaluation, error)

Evaluate checks one successful ceph.Observer.QuorumStatus observation against the expectation. It reads only its arguments: no query, wait or retry, so a test composes it with its own polling or assertion helpers. A mismatch is a result; an error means no verdict.

An expectation that sets no field, a negative count, an empty or repeated name, an unparsable FSID, a Quorum name outside Members, a count that contradicts its name list, or a quorum that is not a majority of the expected members is ceph.ErrInvalidRequest: no observation could match it. An observation with no monmap member, an empty or repeated name, an empty quorum, a quorum name outside the monmap or an unparsable FSID is a malformed observation, never a mismatch. The result shares no slice with its arguments.

Example

A stopped MON leaves the quorum but stays in the monmap, and the evaluation names it.

package main

import (
	"encoding/json"
	"fmt"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/JSYoo5B/ceph-testcontainers-go/ceph/expect"
)

func main() {
	var observed ceph.QuorumStatus // as observer.QuorumStatus decodes it
	if err := json.Unmarshal([]byte(`{"quorum_names":["a","b"],"monmap":{"fsid":"44444444-4444-4444-8444-444444444444",
		"mons":[{"name":"a"},{"name":"b"},{"name":"c"}]}}`), &observed); err != nil {
		panic(err)
	}
	want := expect.Monitors{
		FSID:    "44444444-4444-4444-8444-444444444444",
		Members: []string{"a", "b", "c"},
		Quorum:  []string{"a", "b", "c"},
	}
	result, err := want.Evaluate(observed)
	if err != nil {
		panic(err) // invalid expectation or malformed observation: no verdict
	}
	fmt.Println(result.Matched)
	fmt.Println(result)
}
Output:
false
mons want fsid 44444444-4444-4444-8444-444444444444 members [a b c] quorum [a b c] in monmap [a b c] quorum [a b]: mon.c out of quorum

type OSDEvaluation

type OSDEvaluation struct {
	Expectation OSDs
	Observed    []ceph.OSDState
	Mismatches  []OSDMismatch
	Matched     bool
}

OSDEvaluation is the result of evaluating one observation. Expectation and Observed are copies of what was evaluated, so the diagnostics describe that observation even when the caller later reuses its slices. Mismatches are ordered by OSD ID, then in the order of the reason constants; Matched means there are none.

func (OSDEvaluation) String

func (e OSDEvaluation) String() string

String summarizes the expectation and at most eight mismatches.

type OSDMismatch

type OSDMismatch struct {
	Reason   OSDMismatchReason
	ID       int
	UUID     string
	Observed ceph.OSDState
}

OSDMismatch is one reason one OSD does not satisfy an expectation. UUID is the selected UUID, empty for a selection by ID alone and for OSDMismatchUnselected. Observed is the OSD's state in the evaluated observation and the zero value for OSDMismatchAbsent. For OSDMismatchUp and OSDMismatchIn the expected flag is the negation of the observed one.

func (OSDMismatch) String

func (m OSDMismatch) String() string

String names the OSD, the reason and the evaluated values.

type OSDMismatchReason

type OSDMismatchReason string

OSDMismatchReason names why one OSD does not satisfy an OSDs expectation.

const (
	// OSDMismatchAbsent: a selected ID is not in the observation.
	OSDMismatchAbsent OSDMismatchReason = "absent"
	// OSDMismatchReplaced: the ID now carries a different native UUID than the
	// selection pinned. Up and In of the replacement are not compared.
	OSDMismatchReplaced OSDMismatchReason = "replaced"
	// OSDMismatchUp: the reported up flag differs from the expectation.
	OSDMismatchUp OSDMismatchReason = "up"
	// OSDMismatchIn: the reported in flag differs from the expectation.
	OSDMismatchIn OSDMismatchReason = "in"
	// OSDMismatchUnselected: an Exact expectation found an owned OSD that the
	// selection does not name.
	OSDMismatchUnselected OSDMismatchReason = "unselected"
)

The reasons an OSDEvaluation reports.

type OSDs

type OSDs struct {
	Select []ceph.OSDState
	Up, In bool
	Exact  bool
}

OSDs states the OSDMap membership a test expects of selected owned OSDs. Select names the OSDs by ID and, when UUID is set, by native UUID: states from an earlier ceph.Observer.OSDStates baseline pin the exact registrations, so an OSD that a replacement re-registered under the same ID does not match. Only ID and UUID of a Select entry are read. Up and In are the reported OSDMap flags every selected OSD must have; they are not proof of process liveness or client reachability. Exact also requires the observation to contain no owned OSD outside the selection.

Unlike ceph.Observer.WaitForClean, which compares native OSD counts from the cluster status, an expectation checks individual identities, so it can tell which OSD is missing, replaced, down or out.

func (OSDs) Evaluate

func (e OSDs) Evaluate(observed []ceph.OSDState) (OSDEvaluation, error)

Evaluate checks one successful ceph.Observer.OSDStates observation against the expectation. It reads only its arguments: no query, wait or retry, so a test composes it with its own polling or assertion helpers. A mismatch is a result; an error means no verdict. An empty or duplicate selection or an unparsable selected UUID is ceph.ErrInvalidRequest, and an observation with a negative or repeated ID or an unparsable UUID is a malformed observation, never a mismatch. The result shares no slice with its arguments.

Example

An evaluation reads only the observation it is given, so its diagnostics name the field that failed in that observation.

package main

import (
	"fmt"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/JSYoo5B/ceph-testcontainers-go/ceph/expect"
)

func main() {
	baseline := []ceph.OSDState{
		{ID: 0, UUID: "11111111-1111-4111-8111-111111111111", Up: true, In: true, Weight: 1},
		{ID: 1, UUID: "22222222-2222-4222-8222-222222222222", Up: true, In: true, Weight: 1},
	}
	later := []ceph.OSDState{
		baseline[0],
		{ID: 1, UUID: "22222222-2222-4222-8222-222222222222", Up: false, In: true, Weight: 1},
	}
	result, err := expect.OSDs{Select: baseline, Up: true, In: true, Exact: true}.Evaluate(later)
	if err != nil {
		panic(err) // invalid expectation or malformed observation: no verdict
	}
	fmt.Println(result.Matched)
	fmt.Println(result)
}
Output:
false
osds [0 1] want up=true in=true exact in 2 observed: osd.1 up: want true, observed false
Example (Polling)

The test owns the polling: a failed query is kept apart from the last mismatch, so a timeout reports both instead of an empty "not ready". This example is compiled, not run.

package main

import (
	"context"
	"fmt"
	"time"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/JSYoo5B/ceph-testcontainers-go/ceph/expect"
)

func main() {
	ctx := context.Background()
	cluster, err := ceph.Run(ctx, ceph.DefaultImage)
	if cluster != nil {
		defer func() { _ = cluster.Terminate(context.Background()) }()
	}
	if err != nil {
		panic(err)
	}
	observer, err := ceph.Observe(cluster)
	if err != nil {
		panic(err)
	}
	baseline, err := observer.OSDStates(ctx)
	if err != nil {
		panic(err)
	}
	// ... stop and restart an OSD while the SUT runs ...
	want := expect.OSDs{Select: baseline, Up: true, In: true, Exact: true}
	ctx, cancel := context.WithTimeout(ctx, 2*time.Minute)
	defer cancel()
	var last expect.OSDEvaluation
	var queryErr error
	for {
		states, err := observer.OSDStates(ctx)
		if err == nil {
			last, err = want.Evaluate(states)
		}
		if err == nil && last.Matched {
			return
		}
		if err != nil {
			queryErr = err
		}
		select {
		case <-ctx.Done():
			panic(fmt.Sprintf("OSDs not restored: last %v, last error %v", last, queryErr))
		case <-time.After(time.Second):
		}
	}
}

type PGEvaluation

type PGEvaluation struct {
	Expectation PGs
	Observed    ceph.PoolPGSnapshot
	Mismatches  []PGMismatch
	Matched     bool
}

PGEvaluation is the result of evaluating one pool PG observation. Expectation and Observed are deep copies of what was evaluated, so the diagnostics describe that observation even when the caller later reuses its slices. Mismatches list PGMismatchFSID, PGMismatchPool and PGMismatchCount first, then the PGs by numeric PG number, each PG's reasons in the order of the reason constants and, within one reason, in the order of the expectation's list; Matched means there are none.

func (PGEvaluation) String

func (e PGEvaluation) String() string

String summarizes the expectation and at most eight mismatches.

type PGMismatch

type PGMismatch struct {
	Reason         PGMismatchReason
	PGID           ceph.PGID
	Want, Observed string
}

PGMismatch is one reason a pool PG observation does not satisfy an expectation. PGID names the PG, empty for PGMismatchFSID, PGMismatchPool and PGMismatchCount. Want and Observed are the compared values as text: the token for PGMismatchRequired and PGMismatchForbidden and the OSD ID for the missing and present reasons in Want, with the reported state or vector in Observed. Both are empty for PGMismatchAbsent.

func (PGMismatch) String

func (m PGMismatch) String() string

String names the PG, the reason and the evaluated values.

type PGMismatchReason

type PGMismatchReason string

PGMismatchReason names why a pool PG observation does not satisfy a PGs expectation.

const (
	// PGMismatchFSID: the observation belongs to another cluster.
	PGMismatchFSID PGMismatchReason = "fsid"
	// PGMismatchPool: the observation reports another pool name, or the
	// pinned name now carries another native pool ID.
	PGMismatchPool PGMismatchReason = "pool"
	// PGMismatchCount: the pool reports another number of PGs.
	PGMismatchCount PGMismatchReason = "count"
	// PGMismatchAbsent: a selected PG is not reported. Its conditions are not
	// compared.
	PGMismatchAbsent PGMismatchReason = "absent"
	// PGMismatchState: the PG reports another exact state.
	PGMismatchState PGMismatchReason = "state"
	// PGMismatchRequired: the PG state lacks a required token.
	PGMismatchRequired PGMismatchReason = "required"
	// PGMismatchForbidden: the PG state carries a forbidden token.
	PGMismatchForbidden PGMismatchReason = "forbidden"
	// PGMismatchUp: the PG reports another up vector.
	PGMismatchUp PGMismatchReason = "up"
	// PGMismatchActing: the PG reports another acting vector.
	PGMismatchActing PGMismatchReason = "acting"
	// PGMismatchActingSize: the acting vector has another length.
	PGMismatchActingSize PGMismatchReason = "acting size"
	// PGMismatchUpMissing: the up vector lacks an OSD of InUp.
	PGMismatchUpMissing PGMismatchReason = "up missing"
	// PGMismatchUpPresent: the up vector contains an OSD of NotInUp.
	PGMismatchUpPresent PGMismatchReason = "up present"
	// PGMismatchActingMissing: the acting vector lacks an OSD of InActing.
	PGMismatchActingMissing PGMismatchReason = "acting missing"
	// PGMismatchActingPresent: the acting vector contains an OSD of
	// NotInActing.
	PGMismatchActingPresent PGMismatchReason = "acting present"
	// PGMismatchPrimary: the acting primary is not one of ActingPrimary.
	PGMismatchPrimary PGMismatchReason = "primary"
)

The reasons a PGEvaluation reports.

type PGs

type PGs struct {
	FSID                                 string
	Pool                                 ceph.PoolState
	Select                               []ceph.PGID
	Count                                int
	State                                string
	Require, Forbid                      []string
	Up, Acting                           []int
	ActingSize                           int
	InUp, NotInUp, InActing, NotInActing []int
	ActingPrimary                        []int
}

PGs states the placement groups a test expects in one ceph.Pool.PGs observation of one pool.

FSID, when set, pins the cluster. Only Name and ID of Pool are read: Name, when set, must be the observed pool name, and ID, when nonzero, pins the native pool ID (Ceph numbers pools from 1), so a pool that was removed and recreated under the same name does not match. A PoolState from an earlier observation, such as the PoolAfter of a baseline, pins both.

Select names the PGs to check; when it is empty every reported PG is checked. Count, when nonzero, is the exact number of PGs the pool must report. An expectation selects no PG unless it sets Count or Select, so an empty report never matches by having nothing to check.

Each checked PG must satisfy every condition that is set:

  • State, when set, is the exact native state string, such as "active+clean"; "active+clean+scrubbing" does not match it.
  • Require and Forbid list native state tokens (the parts of the state between "+") the PG must and must not report, such as "active" or "degraded". A token matches only whole: "backfill" does not match "backfill_wait".
  • Up and Acting, when non-nil, are the exact ordered vectors the PG must report, including missing EC shard slots (2147483647); an empty non-nil vector requires an empty report.
  • ActingSize, when nonzero, is the length of the acting vector.
  • InUp and InActing list OSD IDs the vector must contain; NotInUp and NotInActing OSD IDs it must not contain. NotInActing can list 2147483647 to forbid a missing EC shard slot.
  • ActingPrimary, when non-empty, lists the reported acting primaries that are acceptable; -1 is the native value for no primary.

A match proves only what this one observation reported: the PG reports arrive from the MGR asynchronously, are not an atomic placement snapshot and are no read-after-write barrier, and the native PGReady flag is not compared. Unlike ceph.Observer.WaitForClean, which waits for the whole cluster to report clean, an expectation checks one pool's PGs once and names each PG and condition that differs.

func (PGs) Evaluate

func (e PGs) Evaluate(observed ceph.PoolPGSnapshot) (PGEvaluation, error)

Evaluate checks one successful ceph.Pool.PGs observation against the expectation. It reads only its arguments: no query, wait or retry, so a test composes it with its own polling or assertion helpers. A mismatch is a result; an error means no verdict. An expectation that selects no PG (neither Count nor Select), a negative count, size, pool ID or OSD ID, an invalid FSID, PGID or state token, a repeated entry, a selected PG of another pinned pool, and a contradiction (more selected PGs than Count, a token both required and forbidden or disagreeing with State, an OSD both wanted and forbidden or disagreeing with Up or Acting, an ActingSize other than the length of Acting) are ceph.ErrInvalidRequest. An observation whose two pool reads disagree on name or ID, a PG of another pool or with an invalid or repeated PGID, an empty state, a negative OSD in a vector, a primary below -1, or an invalid FSID when the expectation pins one is a malformed observation, never a mismatch. The result shares no slice with its arguments.

Example

A PG expectation pins the pool of a baseline, so a recreated pool of the same name does not pass, and names each PG and condition that differs.

package main

import (
	"fmt"

	"github.com/JSYoo5B/ceph-testcontainers-go/ceph"
	"github.com/JSYoo5B/ceph-testcontainers-go/ceph/expect"
)

func main() {
	pool := ceph.PoolState{ID: 3, Name: "orders", Type: "replicated", Size: 2, MinSize: 1, PGNum: 2}
	observed := ceph.PoolPGSnapshot{
		FSID:       "44444444-4444-4444-8444-444444444444",
		PoolBefore: pool, PoolAfter: pool,
		PGs: []ceph.PGState{
			{PGID: "3.0", State: "active+clean", Up: []int{0, 1}, Acting: []int{0, 1}, UpPrimary: 0, ActingPrimary: 0},
			{PGID: "3.1", State: "active+undersized+degraded", Up: []int{2}, Acting: []int{2}, UpPrimary: 2, ActingPrimary: 2},
		},
	}
	// Every PG of the baseline pool is clean on two OSDs, none on osd.2.
	want := expect.PGs{Pool: pool, Count: 2, State: "active+clean", ActingSize: 2, NotInActing: []int{2}}
	result, err := want.Evaluate(observed)
	if err != nil {
		panic(err) // invalid expectation or malformed observation: no verdict
	}
	fmt.Println(result.Matched)
	for _, m := range result.Mismatches {
		fmt.Println(m)
	}
}
Output:
false
pg 3.1 state: want active+clean, observed active+undersized+degraded
pg 3.1 acting size: want 2, observed [2]
pg 3.1 acting present osd.2: observed [2]

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL