Documentation
¶
Overview ¶
Package pacemaker provides health monitoring and status collection for Pacemaker-managed clusters in OpenShift's ExternalEtcd deployment topology.
Architecture ¶
The package consists of two main components:
Status Collector (statuscollector.go): Collects raw Pacemaker cluster status by executing "pcs status xml" and storing the results in a Pacemaker custom resource. The collector operates on a "collect everything" principle - it writes all data as-is without validation, allowing the health check to make informed decisions based on available data.
Health Check Controller (healthcheck.go): Monitors the Pacemaker CR and evaluates cluster health. It is hardened to handle missing or incomplete data gracefully, treating absent information as "unknown" rather than "error". The health check updates operator conditions and records events based on cluster state.
Security Considerations ¶
- Command execution is hardcoded (no user input) and uses sudo -n (non-interactive)
- XML parsing includes size limits (10MB) to prevent memory exhaustion
- Command execution has timeouts to prevent hanging
- Raw XML is stored in CRs but never logged to prevent information disclosure
Design Principles ¶
- Separation of concerns: Collection is separate from health evaluation
- Missing data = unknown status, not error: Absence of information doesn't indicate failure
- Type-safe enums: Use typed constants for all enumerated values
- Defensive programming: All pointer dereferences are guarded with nil checks
- Fail-safe: Transient collection failures don't immediately mark the cluster as degraded
Index ¶
- Constants
- func FindCondition(conditions []metav1.Condition, conditionType string) *metav1.Condition
- func NewHealthCheck(operatorClient v1helpers.StaticPodOperatorClient, ...) (factory.Controller, cache.SharedIndexInformer, error)
- func NewPacemakerStatusCollectorCommand() *cobra.Command
- type Clone
- type ClusterConfig
- type ClusterConfigNode
- type ClusterConfigNodeAddr
- type ClusterOptions
- type ConditionSpec
- type CurrentDC
- type FenceEvent
- type FenceHistory
- type FencingAgentInfo
- type HealthCheck
- type HealthStatus
- type HealthStatusValue
- type Node
- type NodeAttribute
- type NodeAttributeSet
- type NodeAttributes
- type NodeHistory
- type NodeHistoryNode
- type NodeRef
- type Nodes
- type NodesConfigured
- type OperationHistory
- type PacemakerResult
- type RecentFailedAction
- type RecentFencingEvent
- type Resource
- type ResourceHistory
- type ResourceStatePerNode
- type Resources
- type ResourcesConfigured
- type Stack
- type Summary
Constants ¶
const ( // Kubernetes API path for custom resources KubernetesAPIPath = "/apis" // PacemakerCluster resource name (plural) for API calls PacemakerResourceName = "pacemakerclusters" // PacemakerClusterResourceName is the name of the singleton PacemakerCluster CR PacemakerClusterResourceName = "cluster" )
API constants - shared between healthcheck and statuscollector
const ( ResourceAgentKubelet = "systemd:kubelet" ResourceAgentEtcd = "ocf:heartbeat:podman-etcd" ResourceAgentFencing = "stonith:" ResourceAgentFenceAWS = "fence_aws" )
Resource agent identifiers - used to identify pacemaker resources by type
const ( // HealthCheckResyncInterval is how often the health check controller syncs. // Set lower than the CronJob schedule to detect CR updates promptly. HealthCheckResyncInterval = 30 * time.Second // StatusStalenessThreshold is how long before status is considered stale. // Should be > 2x the CronJob schedule to allow for transient collection failures. StatusStalenessThreshold = 5 * time.Minute // StatusUnknownDegradedThreshold is how long to wait in unknown status before marking degraded. // Matches StatusStalenessThreshold to provide consistent grace period behavior. StatusUnknownDegradedThreshold = 5 * time.Minute // EventDeduplicationWindowFencing is the window for fencing event deduplication EventDeduplicationWindowFencing = 24 * time.Hour // EventDeduplicationWindowDefault is the default event deduplication window EventDeduplicationWindowDefault = 5 * time.Minute // FailedActionTimeWindow is the window for detecting recent failed actions FailedActionTimeWindow = 5 * time.Minute // FencingEventTimeWindow is the window for detecting recent fencing events FencingEventTimeWindow = 24 * time.Hour )
Time windows and thresholds
Timing Relationship:
- The status collector CronJob runs every 2 minutes (configured in cronjob.yaml)
- StatusStalenessThreshold (5 min) allows for ~2 missed collections before staleness
- StatusUnknownDegradedThreshold (5 min) provides grace period for transient issues
- HealthCheckResyncInterval (30s) ensures quick detection of CR updates
const ( // ExpectedNodeCount is the expected number of nodes in a TNF cluster ExpectedNodeCount = 2 // TargetNamespace is the namespace for events and resources TargetNamespace = "openshift-etcd" // StatusCollectorCronJobName is the name of the CronJob that runs the status collector StatusCollectorCronJobName = "pacemaker-status-collector" )
Cluster configuration
const ( // Event reasons for non-degrading conditions (informational/historical) EventReasonFailedAction = "PacemakerFailedResourceAction" EventReasonFencingEvent = "PacemakerFencingEvent" EventReasonWarning = "PacemakerWarning" EventReasonHealthy = "PacemakerHealthy" EventReasonWarningsCleared = "PacemakerWarningsCleared" // Event reasons for degrading conditions (current operational problems) EventReasonNodeOffline = "PacemakerNodeOffline" EventReasonNodeUnhealthy = "PacemakerNodeUnhealthy" EventReasonResourceUnhealthy = "PacemakerResourceUnhealthy" EventReasonClusterUnhealthy = "PacemakerClusterUnhealthy" EventReasonStatusStale = "PacemakerStatusStale" EventReasonCRNotFound = "PacemakerCRNotFound" EventReasonInsufficientNodes = "PacemakerInsufficientNodes" EventReasonClusterInMaintenance = "PacemakerClusterInMaintenance" EventReasonGenericError = "PacemakerError" // Event reasons for status collector EventReasonCollectionError = "PacemakerStatusCollectionError" )
Event reasons - shared between healthcheck and statuscollector
Variables ¶
This section is empty.
Functions ¶
func FindCondition ¶
FindCondition finds a condition by type from a list of conditions. Returns nil if the condition is not found.
func NewHealthCheck ¶
func NewHealthCheck( operatorClient v1helpers.StaticPodOperatorClient, kubeClient kubernetes.Interface, eventRecorder events.Recorder, restConfig *rest.Config, ) (factory.Controller, cache.SharedIndexInformer, error)
NewHealthCheck creates a new HealthCheck for monitoring pacemaker status in clusters that use ExternalEtcd clusters. Returns the controller and the PacemakerCluster informer (which must be started separately).
Types ¶
type ClusterConfig ¶
type ClusterConfig struct {
ClusterName string `json:"cluster_name"`
ClusterUUID string `json:"cluster_uuid"`
Nodes []ClusterConfigNode `json:"nodes"`
}
type ClusterConfigNode ¶
type ClusterConfigNode struct {
Name string `json:"name"`
NodeID string `json:"nodeid"`
Addrs []ClusterConfigNodeAddr `json:"addrs"`
}
type ClusterConfigNodeAddr ¶
type ClusterOptions ¶
type ClusterOptions struct {
MaintenanceMode string `xml:"maintenance-mode,attr"`
}
type ConditionSpec ¶
type ConditionSpec struct {
Type string
TrueReason string
FalseReason string
TrueMsg string
FalseMsg string
}
ConditionSpec defines the parameters for building a condition. Each spec maps a boolean state to appropriate reason/message pairs.
type FenceEvent ¶
type FenceEvent struct {
Target string `xml:"target,attr"`
Action string `xml:"action,attr"` // "reboot", "off", etc.
Delegate string `xml:"delegate,attr"`
Client string `xml:"client,attr"`
Origin string `xml:"origin,attr"`
Status string `xml:"status,attr"` // "success", "failed", etc.
ExitReason string `xml:"exit-reason,attr"` // Populated on failure
Completed string `xml:"completed,attr"` // Timestamp: "2006-01-02 15:04:05.000000Z"
}
FenceEvent records a fencing operation (node isolation/reboot).
type FenceHistory ¶
type FenceHistory struct {
FenceEvent []FenceEvent `xml:"fence_event"`
}
type FencingAgentInfo ¶
type FencingAgentInfo struct {
Name string
TargetNode string
Method pacmkrv1.FencingMethod
Resource *Resource
IsRunning bool
}
FencingAgentInfo tracks a fencing agent parsed from pacemaker resources. Fencing agents are named like "master-0_redfish" where the prefix is the target node.
type HealthCheck ¶
type HealthCheck struct {
// contains filtered or unexported fields
}
HealthCheck monitors pacemaker status in ExternalEtcd topology clusters
type HealthStatus ¶
type HealthStatus struct {
OverallStatus HealthStatusValue
Warnings []string
Errors []string
// CRLastUpdated is the timestamp from the PacemakerCluster CR's status.lastUpdated field.
// Used for change detection (skip if unchanged) and grace period calculation (time since last valid).
CRLastUpdated time.Time
}
HealthStatus represents the overall status of pacemaker in the ExternalEtcd cluster.
Warning vs Error Philosophy:
- Errors: Conditions requiring immediate action or causing downstream degradation. Examples: node offline, fencing unavailable, etcd/kubelet unhealthy, maintenance mode.
- Warnings: Degraded redundancy where the cluster remains functional but should be investigated. Examples: FencingHealthy=False with FencingAvailable=True (can still fence, but redundancy lost).
The operator degrades on Errors but not Warnings. Warnings generate events for observability.
type HealthStatusValue ¶
type HealthStatusValue string
HealthStatusValue represents the overall health status of the pacemaker cluster.
type Node ¶
type Node struct {
Name string `xml:"name,attr"`
ID string `xml:"id,attr"`
Online string `xml:"online,attr"`
Standby string `xml:"standby,attr"`
StandbyOnFail string `xml:"standby_onfail,attr"`
Maintenance string `xml:"maintenance,attr"`
Pending string `xml:"pending,attr"`
Unclean string `xml:"unclean,attr"`
Shutdown string `xml:"shutdown,attr"`
ExpectedUp string `xml:"expected_up,attr"`
IsDC string `xml:"is_dc,attr"`
ResourcesRunning string `xml:"resources_running,attr"`
Type string `xml:"type,attr"` // "member" or "remote"
}
Node represents a pacemaker cluster node with its operational state. Boolean attributes are strings ("true"/"false") in pacemaker XML.
type NodeAttribute ¶
type NodeAttributeSet ¶
type NodeAttributeSet struct {
Name string `xml:"name,attr"`
Attribute []NodeAttribute `xml:"attribute"`
}
type NodeAttributes ¶
type NodeAttributes struct {
Node []NodeAttributeSet `xml:"node"`
}
type NodeHistory ¶
type NodeHistory struct {
Node []NodeHistoryNode `xml:"node"`
}
type NodeHistoryNode ¶
type NodeHistoryNode struct {
Name string `xml:"name,attr"`
ResourceHistory []ResourceHistory `xml:"resource_history"`
}
type NodesConfigured ¶
type NodesConfigured struct {
Number string `xml:"number,attr"`
}
type OperationHistory ¶
type OperationHistory struct {
Call string `xml:"call,attr"`
Task string `xml:"task,attr"` // "start", "stop", "monitor", etc.
RC string `xml:"rc,attr"` // Return code: "0" = success
RCText string `xml:"rc_text,attr"`
ExitReason string `xml:"exit-reason,attr"`
LastRCChange string `xml:"last-rc-change,attr"` // Timestamp: "Mon Jan 2 15:04:05 2006"
ExecTime string `xml:"exec-time,attr"`
QueueTime string `xml:"queue-time,attr"`
}
OperationHistory tracks resource operation results for failure detection.
type PacemakerResult ¶
type PacemakerResult struct {
XMLName xml.Name `xml:"pacemaker-result"`
Summary Summary `xml:"summary"`
Nodes Nodes `xml:"nodes"`
Resources Resources `xml:"resources"`
NodeAttributes NodeAttributes `xml:"node_attributes"`
NodeHistory NodeHistory `xml:"node_history"`
FenceHistory FenceHistory `xml:"fence_history"`
}
PacemakerResult is the root element of pcs status xml output.
type RecentFailedAction ¶
type RecentFailedAction struct {
ResourceID, Task, NodeName, RC, RCText, LastRCChange string
}
RecentFailedAction is a failed resource action that passed time-window filtering.
type RecentFencingEvent ¶
type RecentFencingEvent struct {
Action, Target, Status, ExitReason, Completed string
}
RecentFencingEvent is a fencing event that passed time-window filtering.
type Resource ¶
type Resource struct {
ID string `xml:"id,attr"`
ResourceAgent string `xml:"resource_agent,attr"` // e.g., "systemd:kubelet", "stonith:fence_redfish"
Role string `xml:"role,attr"` // "Started", "Stopped", etc.
TargetRole string `xml:"target_role,attr"`
Active string `xml:"active,attr"`
Orphaned string `xml:"orphaned,attr"`
Blocked string `xml:"blocked,attr"`
Maintenance string `xml:"maintenance,attr"` // "true" when resource is on a node in maintenance mode
Managed string `xml:"managed,attr"`
Failed string `xml:"failed,attr"`
FailureIgnored string `xml:"failure_ignored,attr"`
NodesRunningOn string `xml:"nodes_running_on,attr"`
Node NodeRef `xml:"node"`
}
Resource represents a pacemaker-managed resource (systemd service, container, etc).
type ResourceHistory ¶
type ResourceHistory struct {
ID string `xml:"id,attr"`
OperationHistory []OperationHistory `xml:"operation_history"`
}
type ResourceStatePerNode ¶
type ResourceStatePerNode struct {
KubeletRunning bool
EtcdRunning bool
KubeletResource *Resource
EtcdResource *Resource
}
ResourceStatePerNode tracks resource state per node for condition building.
type ResourcesConfigured ¶
type ResourcesConfigured struct {
Number string `xml:"number,attr"`
}
type Summary ¶
type Summary struct {
Stack Stack `xml:"stack"`
CurrentDC CurrentDC `xml:"current_dc"`
NodesConfigured NodesConfigured `xml:"nodes_configured"`
ResourcesConfigured ResourcesConfigured `xml:"resources_configured"`
ClusterOptions ClusterOptions `xml:"cluster_options"`
}