pacemaker

package
v0.0.0-alpha.0....-54e8d0f Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 22, 2026 License: Apache-2.0 Imports: 29 Imported by: 0

Documentation

Overview

Package pacemaker provides health monitoring and status collection for Pacemaker-managed clusters in OpenShift's ExternalEtcd deployment topology.

Architecture

The package consists of two main components:

  1. Status Collector (statuscollector.go): Collects raw Pacemaker cluster status by executing "pcs status xml" and storing the results in a Pacemaker custom resource. The collector operates on a "collect everything" principle - it writes all data as-is without validation, allowing the health check to make informed decisions based on available data.

  2. Health Check Controller (healthcheck.go): Monitors the Pacemaker CR and evaluates cluster health. It is hardened to handle missing or incomplete data gracefully, treating absent information as "unknown" rather than "error". The health check updates operator conditions and records events based on cluster state.

Security Considerations

  • Command execution is hardcoded (no user input) and uses sudo -n (non-interactive)
  • XML parsing includes size limits (10MB) to prevent memory exhaustion
  • Command execution has timeouts to prevent hanging
  • Raw XML is stored in CRs but never logged to prevent information disclosure

Design Principles

  • Separation of concerns: Collection is separate from health evaluation
  • Missing data = unknown status, not error: Absence of information doesn't indicate failure
  • Type-safe enums: Use typed constants for all enumerated values
  • Defensive programming: All pointer dereferences are guarded with nil checks
  • Fail-safe: Transient collection failures don't immediately mark the cluster as degraded

Index

Constants

View Source
const (
	// Kubernetes API path for custom resources
	KubernetesAPIPath = "/apis"

	// PacemakerCluster resource name (plural) for API calls
	PacemakerResourceName = "pacemakerclusters"

	// PacemakerClusterResourceName is the name of the singleton PacemakerCluster CR
	PacemakerClusterResourceName = "cluster"
)

API constants - shared between healthcheck and statuscollector

View Source
const (
	ResourceAgentKubelet  = "systemd:kubelet"
	ResourceAgentEtcd     = "ocf:heartbeat:podman-etcd"
	ResourceAgentFencing  = "stonith:"
	ResourceAgentFenceAWS = "fence_aws"
)

Resource agent identifiers - used to identify pacemaker resources by type

View Source
const (
	// HealthCheckResyncInterval is how often the health check controller syncs.
	// Set lower than the CronJob schedule to detect CR updates promptly.
	HealthCheckResyncInterval = 30 * time.Second

	// StatusStalenessThreshold is how long before status is considered stale.
	// Should be > 2x the CronJob schedule to allow for transient collection failures.
	StatusStalenessThreshold = 5 * time.Minute

	// StatusUnknownDegradedThreshold is how long to wait in unknown status before marking degraded.
	// Matches StatusStalenessThreshold to provide consistent grace period behavior.
	StatusUnknownDegradedThreshold = 5 * time.Minute

	// EventDeduplicationWindowFencing is the window for fencing event deduplication
	EventDeduplicationWindowFencing = 24 * time.Hour

	// EventDeduplicationWindowDefault is the default event deduplication window
	EventDeduplicationWindowDefault = 5 * time.Minute

	// FailedActionTimeWindow is the window for detecting recent failed actions
	FailedActionTimeWindow = 5 * time.Minute

	// FencingEventTimeWindow is the window for detecting recent fencing events
	FencingEventTimeWindow = 24 * time.Hour
)

Time windows and thresholds

Timing Relationship:

  • The status collector CronJob runs every 2 minutes (configured in cronjob.yaml)
  • StatusStalenessThreshold (5 min) allows for ~2 missed collections before staleness
  • StatusUnknownDegradedThreshold (5 min) provides grace period for transient issues
  • HealthCheckResyncInterval (30s) ensures quick detection of CR updates
View Source
const (
	// ExpectedNodeCount is the expected number of nodes in a TNF cluster
	ExpectedNodeCount = 2

	// TargetNamespace is the namespace for events and resources
	TargetNamespace = "openshift-etcd"

	// StatusCollectorCronJobName is the name of the CronJob that runs the status collector
	StatusCollectorCronJobName = "pacemaker-status-collector"
)

Cluster configuration

View Source
const (
	// Event reasons for non-degrading conditions (informational/historical)
	EventReasonFailedAction    = "PacemakerFailedResourceAction"
	EventReasonFencingEvent    = "PacemakerFencingEvent"
	EventReasonWarning         = "PacemakerWarning"
	EventReasonHealthy         = "PacemakerHealthy"
	EventReasonWarningsCleared = "PacemakerWarningsCleared"

	// Event reasons for degrading conditions (current operational problems)
	EventReasonNodeOffline          = "PacemakerNodeOffline"
	EventReasonNodeUnhealthy        = "PacemakerNodeUnhealthy"
	EventReasonResourceUnhealthy    = "PacemakerResourceUnhealthy"
	EventReasonClusterUnhealthy     = "PacemakerClusterUnhealthy"
	EventReasonStatusStale          = "PacemakerStatusStale"
	EventReasonCRNotFound           = "PacemakerCRNotFound"
	EventReasonInsufficientNodes    = "PacemakerInsufficientNodes"
	EventReasonClusterInMaintenance = "PacemakerClusterInMaintenance"
	EventReasonGenericError         = "PacemakerError"

	// Event reasons for status collector
	EventReasonCollectionError = "PacemakerStatusCollectionError"
)

Event reasons - shared between healthcheck and statuscollector

Variables

This section is empty.

Functions

func FindCondition

func FindCondition(conditions []metav1.Condition, conditionType string) *metav1.Condition

FindCondition finds a condition by type from a list of conditions. Returns nil if the condition is not found.

func NewHealthCheck

func NewHealthCheck(
	operatorClient v1helpers.StaticPodOperatorClient,
	kubeClient kubernetes.Interface,
	eventRecorder events.Recorder,
	restConfig *rest.Config,
) (factory.Controller, cache.SharedIndexInformer, error)

NewHealthCheck creates a new HealthCheck for monitoring pacemaker status in clusters that use ExternalEtcd clusters. Returns the controller and the PacemakerCluster informer (which must be started separately).

func NewPacemakerStatusCollectorCommand

func NewPacemakerStatusCollectorCommand() *cobra.Command

Types

type Clone

type Clone struct {
	Resource []Resource `xml:"resource"`
}

type ClusterConfig

type ClusterConfig struct {
	ClusterName string              `json:"cluster_name"`
	ClusterUUID string              `json:"cluster_uuid"`
	Nodes       []ClusterConfigNode `json:"nodes"`
}

type ClusterConfigNode

type ClusterConfigNode struct {
	Name   string                  `json:"name"`
	NodeID string                  `json:"nodeid"`
	Addrs  []ClusterConfigNodeAddr `json:"addrs"`
}

type ClusterConfigNodeAddr

type ClusterConfigNodeAddr struct {
	Addr string `json:"addr"`
	Link string `json:"link"`
	Type string `json:"type"` // "IPv4", "IPv6"
}

type ClusterOptions

type ClusterOptions struct {
	MaintenanceMode string `xml:"maintenance-mode,attr"`
}

type ConditionSpec

type ConditionSpec struct {
	Type        string
	TrueReason  string
	FalseReason string
	TrueMsg     string
	FalseMsg    string
}

ConditionSpec defines the parameters for building a condition. Each spec maps a boolean state to appropriate reason/message pairs.

type CurrentDC

type CurrentDC struct {
	WithQuorum string `xml:"with_quorum,attr"`
}

type FenceEvent

type FenceEvent struct {
	Target     string `xml:"target,attr"`
	Action     string `xml:"action,attr"` // "reboot", "off", etc.
	Delegate   string `xml:"delegate,attr"`
	Client     string `xml:"client,attr"`
	Origin     string `xml:"origin,attr"`
	Status     string `xml:"status,attr"`      // "success", "failed", etc.
	ExitReason string `xml:"exit-reason,attr"` // Populated on failure
	Completed  string `xml:"completed,attr"`   // Timestamp: "2006-01-02 15:04:05.000000Z"
}

FenceEvent records a fencing operation (node isolation/reboot).

type FenceHistory

type FenceHistory struct {
	FenceEvent []FenceEvent `xml:"fence_event"`
}

type FencingAgentInfo

type FencingAgentInfo struct {
	Name       string
	TargetNode string
	Method     pacmkrv1.FencingMethod
	Resource   *Resource
	IsRunning  bool
}

FencingAgentInfo tracks a fencing agent parsed from pacemaker resources. Fencing agents are named like "master-0_redfish" where the prefix is the target node.

type HealthCheck

type HealthCheck struct {
	// contains filtered or unexported fields
}

HealthCheck monitors pacemaker status in ExternalEtcd topology clusters

type HealthStatus

type HealthStatus struct {
	OverallStatus HealthStatusValue
	Warnings      []string
	Errors        []string
	// CRLastUpdated is the timestamp from the PacemakerCluster CR's status.lastUpdated field.
	// Used for change detection (skip if unchanged) and grace period calculation (time since last valid).
	CRLastUpdated time.Time
}

HealthStatus represents the overall status of pacemaker in the ExternalEtcd cluster.

Warning vs Error Philosophy:

  • Errors: Conditions requiring immediate action or causing downstream degradation. Examples: node offline, fencing unavailable, etcd/kubelet unhealthy, maintenance mode.
  • Warnings: Degraded redundancy where the cluster remains functional but should be investigated. Examples: FencingHealthy=False with FencingAvailable=True (can still fence, but redundancy lost).

The operator degrades on Errors but not Warnings. Warnings generate events for observability.

type HealthStatusValue

type HealthStatusValue string

HealthStatusValue represents the overall health status of the pacemaker cluster.

type Node

type Node struct {
	Name             string `xml:"name,attr"`
	ID               string `xml:"id,attr"`
	Online           string `xml:"online,attr"`
	Standby          string `xml:"standby,attr"`
	StandbyOnFail    string `xml:"standby_onfail,attr"`
	Maintenance      string `xml:"maintenance,attr"`
	Pending          string `xml:"pending,attr"`
	Unclean          string `xml:"unclean,attr"`
	Shutdown         string `xml:"shutdown,attr"`
	ExpectedUp       string `xml:"expected_up,attr"`
	IsDC             string `xml:"is_dc,attr"`
	ResourcesRunning string `xml:"resources_running,attr"`
	Type             string `xml:"type,attr"` // "member" or "remote"
}

Node represents a pacemaker cluster node with its operational state. Boolean attributes are strings ("true"/"false") in pacemaker XML.

type NodeAttribute

type NodeAttribute struct {
	Name  string `xml:"name,attr"`
	Value string `xml:"value,attr"`
}

type NodeAttributeSet

type NodeAttributeSet struct {
	Name      string          `xml:"name,attr"`
	Attribute []NodeAttribute `xml:"attribute"`
}

type NodeAttributes

type NodeAttributes struct {
	Node []NodeAttributeSet `xml:"node"`
}

type NodeHistory

type NodeHistory struct {
	Node []NodeHistoryNode `xml:"node"`
}

type NodeHistoryNode

type NodeHistoryNode struct {
	Name            string            `xml:"name,attr"`
	ResourceHistory []ResourceHistory `xml:"resource_history"`
}

type NodeRef

type NodeRef struct {
	Name string `xml:"name,attr"`
}

type Nodes

type Nodes struct {
	Node []Node `xml:"node"`
}

type NodesConfigured

type NodesConfigured struct {
	Number string `xml:"number,attr"`
}

type OperationHistory

type OperationHistory struct {
	Call         string `xml:"call,attr"`
	Task         string `xml:"task,attr"` // "start", "stop", "monitor", etc.
	RC           string `xml:"rc,attr"`   // Return code: "0" = success
	RCText       string `xml:"rc_text,attr"`
	ExitReason   string `xml:"exit-reason,attr"`
	LastRCChange string `xml:"last-rc-change,attr"` // Timestamp: "Mon Jan 2 15:04:05 2006"
	ExecTime     string `xml:"exec-time,attr"`
	QueueTime    string `xml:"queue-time,attr"`
}

OperationHistory tracks resource operation results for failure detection.

type PacemakerResult

type PacemakerResult struct {
	XMLName        xml.Name       `xml:"pacemaker-result"`
	Summary        Summary        `xml:"summary"`
	Nodes          Nodes          `xml:"nodes"`
	Resources      Resources      `xml:"resources"`
	NodeAttributes NodeAttributes `xml:"node_attributes"`
	NodeHistory    NodeHistory    `xml:"node_history"`
	FenceHistory   FenceHistory   `xml:"fence_history"`
}

PacemakerResult is the root element of pcs status xml output.

type RecentFailedAction

type RecentFailedAction struct {
	ResourceID, Task, NodeName, RC, RCText, LastRCChange string
}

RecentFailedAction is a failed resource action that passed time-window filtering.

type RecentFencingEvent

type RecentFencingEvent struct {
	Action, Target, Status, ExitReason, Completed string
}

RecentFencingEvent is a fencing event that passed time-window filtering.

type Resource

type Resource struct {
	ID             string  `xml:"id,attr"`
	ResourceAgent  string  `xml:"resource_agent,attr"` // e.g., "systemd:kubelet", "stonith:fence_redfish"
	Role           string  `xml:"role,attr"`           // "Started", "Stopped", etc.
	TargetRole     string  `xml:"target_role,attr"`
	Active         string  `xml:"active,attr"`
	Orphaned       string  `xml:"orphaned,attr"`
	Blocked        string  `xml:"blocked,attr"`
	Maintenance    string  `xml:"maintenance,attr"` // "true" when resource is on a node in maintenance mode
	Managed        string  `xml:"managed,attr"`
	Failed         string  `xml:"failed,attr"`
	FailureIgnored string  `xml:"failure_ignored,attr"`
	NodesRunningOn string  `xml:"nodes_running_on,attr"`
	Node           NodeRef `xml:"node"`
}

Resource represents a pacemaker-managed resource (systemd service, container, etc).

type ResourceHistory

type ResourceHistory struct {
	ID               string             `xml:"id,attr"`
	OperationHistory []OperationHistory `xml:"operation_history"`
}

type ResourceStatePerNode

type ResourceStatePerNode struct {
	KubeletRunning  bool
	EtcdRunning     bool
	KubeletResource *Resource
	EtcdResource    *Resource
}

ResourceStatePerNode tracks resource state per node for condition building.

type Resources

type Resources struct {
	Clone    []Clone    `xml:"clone"`
	Resource []Resource `xml:"resource"`
}

type ResourcesConfigured

type ResourcesConfigured struct {
	Number string `xml:"number,attr"`
}

type Stack

type Stack struct {
	PacemakerdState string `xml:"pacemakerd-state,attr"`
}

type Summary

type Summary struct {
	Stack               Stack               `xml:"stack"`
	CurrentDC           CurrentDC           `xml:"current_dc"`
	NodesConfigured     NodesConfigured     `xml:"nodes_configured"`
	ResourcesConfigured ResourcesConfigured `xml:"resources_configured"`
	ClusterOptions      ClusterOptions      `xml:"cluster_options"`
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL