health

package
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 30, 2026 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	ComponentCollectorManager    = "collector_manager"
	ComponentBufferQueue         = "buffer_queue"
	ComponentDakrTransport       = "dakr_transport"
	ComponentMpaServer           = "mpa_server"
	ComponentPrometheus          = "prometheus"
	ComponentMonitor             = "monitor"
	ComponentEBPFTracer          = "ebpf_tracer"
	ComponentPodCache            = "pod_cache"
	ComponentKarpenterDeployment = "karpenter_deployment"
	ComponentGPURuntimeResolver  = "gpu_runtime_resolver"
	ComponentMemoryPressure      = "memory_pressure"
)

Component name constants used for HealthManager registration.

View Source 
const (
	// MemoryPressureCheckInterval matches NodeCollector's default resource-metrics
	// cadence (see NodeCollectorConfig.UpdateInterval) so cgroup polling runs at a
	// familiar frequency rather than an arbitrary one.
	MemoryPressureCheckInterval = 30 * time.Second

	// MemoryPressureThresholdPercent is the usage/limit ratio that triggers a
	// warning. Named so it's a one-line tune rather than a buried literal.
	MemoryPressureThresholdPercent = 85.0

	// MemoryPressureReaffirmInterval bounds how often a still-elevated reading is
	// re-reported, so sustained pressure doesn't spam Datadog on every tick.
	MemoryPressureReaffirmInterval = 10 * time.Minute
)

Variables

This section is empty.

Functions

func BuildHeartbeatRequest added in v0.0.64 

func BuildHeartbeatRequest(hm *HealthManager, clusterID string, operatorType gen.OperatorType, version, commit string, startTime time.Time) *gen.ReportHealthRequest

BuildHeartbeatRequest constructs a ReportHealthRequest from the current HealthManager state.

func BuildHeartbeatRequestFromReport added in v0.0.64 

func BuildHeartbeatRequestFromReport(report map[string]ComponentStatus, clusterID string, operatorType gen.OperatorType, version, commit string, startTime time.Time) *gen.ReportHealthRequest

BuildHeartbeatRequestFromReport constructs a ReportHealthRequest from an already-built report map. Use this when you need to log and send the same snapshot to avoid a double lock acquisition on HealthManager.

Types

type ComponentResponse added in v0.0.63 

type ComponentResponse struct {
	Status   string            `json:"status"`
	Message  string            `json:"message,omitempty"`
	Metadata map[string]string `json:"metadata,omitempty"`
}

ComponentResponse represents the JSON structure for /components/{component} responses

type ComponentStatus 

type ComponentStatus struct {
	Status   HealthStatus
	Message  string
	Metadata map[string]string
}

ComponentStatus holds the health status, message, and metadata for a component

type HealthManager 

type HealthManager struct {
	// contains filtered or unexported fields
}

func NewHealthManager 

func NewHealthManager() *HealthManager

NewHealthManager creates a new HealthManager

func (*HealthManager) BuildReport 

func (hm *HealthManager) BuildReport() map[string]ComponentStatus

BuildReport returns a snapshot of all component statuses

func (*HealthManager) CheckLiveness added in v0.0.63 

func (hm *HealthManager) CheckLiveness() (map[string]ComponentStatus, error)

CheckLiveness returns the report and liveness error atomically under a single lock acquisition, avoiding TOCTOU between BuildReport and LivenessCheck.

func (*HealthManager) CheckReadiness added in v0.0.63 

func (hm *HealthManager) CheckReadiness() (map[string]ComponentStatus, error)

CheckReadiness returns the report and readiness error atomically.

func (*HealthManager) ClearLivenessSuppression added in v0.0.63 

func (hm *HealthManager) ClearLivenessSuppression()

ClearLivenessSuppression removes any active grace period so LivenessCheck resumes normal evaluation. Call this after collectors are back up.

func (*HealthManager) ClearReadinessSuppression added in v0.0.65 

func (hm *HealthManager) ClearReadinessSuppression()

ClearReadinessSuppression removes any active readiness grace period so ReadinessCheck resumes normal evaluation.

func (*HealthManager) Deregister 

func (hm *HealthManager) Deregister(name string)

Deregister removes a component from the health registry

func (*HealthManager) GetStatus 

func (hm *HealthManager) GetStatus(name string) (ComponentStatus, bool)

GetStatus retrieves the current status for a component

func (*HealthManager) LivenessCheck added in v0.0.63 

func (hm *HealthManager) LivenessCheck() error

LivenessCheck checks if all components are at least Degraded (not Unhealthy). During an active grace period (set via SuppressLiveness) it always returns nil so that planned restarts do not trigger pod kills.

func (*HealthManager) ReadinessCheck added in v0.0.63 

func (hm *HealthManager) ReadinessCheck() error

ReadinessCheck checks if all required components are Healthy or Degraded.

func (*HealthManager) Register 

func (hm *HealthManager) Register(name string)

Register adds a component to the health registry

func (*HealthManager) SetStandby added in v0.0.74 

func (hm *HealthManager) SetStandby(standby bool)

SetStandby marks the pod as a standby (non-leader) replica. While in standby, ReadinessCheck passes unconditionally — the pod is healthy and ready to take over leadership, it just isn't running collectors yet. Call with false when leader election is won so normal readiness checks resume.

func (*HealthManager) SetTransitionObserver added in v0.0.77 

func (hm *HealthManager) SetTransitionObserver(obs TransitionObserver)

SetTransitionObserver registers (or clears, if nil) a callback invoked on every component status transition. Only one observer is held at a time. The observer runs outside the lock so it may safely re-enter the HealthManager.

func (*HealthManager) SuppressLiveness added in v0.0.63 

func (hm *HealthManager) SuppressLiveness(d time.Duration)

SuppressLiveness makes LivenessCheck pass unconditionally for the given duration. Use this before a planned collector restart so that the transient Unhealthy window does not trigger a pod kill. The grace period is cleared automatically when StartAll succeeds (via ClearLivenessSuppression) or when the deadline expires.

func (*HealthManager) SuppressReadiness added in v0.0.65 

func (hm *HealthManager) SuppressReadiness(d time.Duration)

SuppressReadiness makes ReadinessCheck pass unconditionally for the given duration. Use this at startup so the pod can become ready while waiting for leader election. The grace period is cleared automatically when collectors start (via ClearReadinessSuppression) or when the deadline expires.

func (*HealthManager) UpdateStatus 

func (hm *HealthManager) UpdateStatus(
	name string,
	status HealthStatus,
	message string,
	metadata map[string]string,
)

UpdateStatus updates the health status, message, and metadata for a component. If the new status differs from the previous one, the registered TransitionObserver (if any) is invoked outside the lock with old and new status.

type HealthResponse added in v0.0.63 

type HealthResponse struct {
	Status     string                       `json:"status"`
	Error      string                       `json:"error,omitempty"`
	Components map[string]ComponentResponse `json:"components,omitempty"`
}

HealthResponse represents the JSON structure for /healthz and /readyz responses

type HealthServer added in v0.0.63 

type HealthServer struct {
	// contains filtered or unexported fields
}

HealthServer serves health and readiness endpoints

func NewHealthServer added in v0.0.63 

func NewHealthServer(manager *HealthManager, addr string) *HealthServer

NewHealthServer creates a new HealthServer bound to the specified address

func (*HealthServer) Start added in v0.0.63 

func (s *HealthServer) Start() error

Start begins serving health endpoints

func (*HealthServer) Stop added in v0.0.63 

func (s *HealthServer) Stop(ctx context.Context) error

Stop gracefully shuts down the server

type HealthStatus 

type HealthStatus int

HealthStatus matches proto enum for easy mapping 

const (
	HealthStatusUnspecified HealthStatus = iota
	HealthStatusHealthy
	HealthStatusDegraded
	HealthStatusUnhealthy
)

HealthStatus values

func (HealthStatus) String added in v0.0.63 

func (s HealthStatus) String() string

String returns a human-readable representation of the HealthStatus

type MemoryPressureMonitor added in v0.1.1 

type MemoryPressureMonitor struct {
	// contains filtered or unexported fields
}

MemoryPressureMonitor periodically compares the process's cgroup memory usage against its configured limit and reports the crossing before the kubelet's OOM killer acts. automemlimit (imported for its init-time side effect in both zxporter entrypoints) only sets GOMEMLIMIT once at startup; this is the periodic re-check that was otherwise missing, giving an early warning ahead of the OOMKilled event instead of only learning about it in hindsight.

func NewMemoryPressureMonitor added in v0.1.1 

func NewMemoryPressureMonitor(
	logger logr.Logger,
	telemetryLogger telemetry_logger.Logger,
	healthManager *HealthManager,
) *MemoryPressureMonitor

NewMemoryPressureMonitor builds a monitor reading real cgroup files under /sys/fs/cgroup. telemetryLogger and healthManager may both be nil.

func (*MemoryPressureMonitor) Start added in v0.1.1 

func (m *MemoryPressureMonitor) Start(ctx context.Context) error

Start runs the check loop and blocks until ctx is cancelled, matching the controller-runtime Runnable shape used elsewhere for always-on, top-level processes (see GPURuntimeResolver.Start, EnvBasedController.Start) rather than returning immediately and requiring a separate Stop(). The controller-manager binary launches it as a goroutine from EnvBasedController.Start (internal/controller/custom.go), once the telemetry logger it needs has been initialized; zxporter-nodemon has no controller-runtime manager at all, so its entrypoint (cmd/zxporter-nodemon) launches it as a goroutine directly with a nil telemetry logger.

This intentionally does not use the collector-manager Start/Stop-with-ticker shape (NodeCollector, ContainerResourceCollector): that machinery exists to let CollectionPolicy dynamically add/replace/remove per-resource collectors, which doesn't apply here — this monitor is a single always-on process-level check wired once at startup in both binaries.

type NodeOperatorMonitor added in v0.0.68 

type NodeOperatorMonitor struct {
	// contains filtered or unexported fields
}

func NewNodeOperatorMonitor added in v0.0.68 

func NewNodeOperatorMonitor(logger logr.Logger, clientset kubernetes.Interface, httpClient *http.Client) *NodeOperatorMonitor

func (*NodeOperatorMonitor) BuildNodeOperatorReport added in v0.0.68 

func (m *NodeOperatorMonitor) BuildNodeOperatorReport(ctx context.Context) (map[string]ComponentStatus, string, string, time.Time)

type RestartOOMDetector added in v0.1.1 

type RestartOOMDetector struct {
	// contains filtered or unexported fields
}

RestartOOMDetector implements the retroactive half of OOM visibility: when zxporter itself gets OOM-killed, the process dies before it can report anything about its own death (MemoryPressureMonitor's proactive warning is necessarily best-effort — a sudden spike can still race past it). But Kubernetes independently persists pod.status.containerStatuses[].lastState.terminated on the Pod object regardless of whether the dying process got to do anything, so the *next* instance can read its own previous death from the API server on startup and report it. The reporting happens from the survivor, not the corpse.

func NewRestartOOMDetector added in v0.1.1 

func NewRestartOOMDetector(
	logger logr.Logger,
	telemetryLogger telemetry_logger.Logger,
	clientset kubernetes.Interface,
	namespace string,
	podName string,
	containerName string,
) *RestartOOMDetector

NewRestartOOMDetector builds a detector for the container named containerName inside the pod identified by namespace/podName. namespace and podName are typically sourced from the POD_NAMESPACE/POD_NAME downward-API env vars and may be empty (e.g. running outside a cluster); clientset and telemetryLogger may both be nil.

func (*RestartOOMDetector) Check added in v0.1.1 

func (d *RestartOOMDetector) Check(ctx context.Context)

Check runs the retroactive OOM check once. It is meant to be invoked a single time early in startup (see EnvBasedController.Start and cmd/zxporter-nodemon/main.go), not on a ticker like MemoryPressureMonitor — a previous termination only needs to be reported once, the first time the new instance notices it.

Every failure path here — missing env vars, no clientset, a failed API call, no previous termination, a termination that wasn't an OOM kill — is a silent, low-verbosity no-op. This check must never prevent or delay normal startup, so callers should invoke it from a goroutine rather than inline on the startup path.

type TransitionObserver added in v0.0.77 

type TransitionObserver func(component string, oldStatus, newStatus HealthStatus, message string, metadata map[string]string)

TransitionObserver is invoked whenever a component's status changes via UpdateStatus. It is called outside the HealthManager lock so observers may safely call back into the manager (e.g. to read other component statuses) without deadlocking. The observer is invoked synchronously, but the typical implementation should hand off to a queue so the UpdateStatus call site stays fast.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.