Documentation
¶
Overview ¶
Package nodefailure is the opt-in, deterministic policy for nodes that stay unreachable: it reschedules stateless apps, never moves anything holding node-local data, and fences stale copies when the node returns.
Index ¶
- Constants
- Variables
- func Classify(r Resource) (class, reason string)
- type Config
- type Controller
- func (c *Controller) MarkLost(ctx context.Context, nodeID string, dryRun bool) (Status, error)
- func (c *Controller) Resolve(ctx context.Context, nodeID, kind, name, action, target string) (Status, error)
- func (c *Controller) Run(ctx context.Context)
- func (c *Controller) SetPolicy(ctx context.Context, p store.NodeFailurePolicy) error
- func (c *Controller) States(ctx context.Context) (map[string]store.NodeFailureState, error)
- func (c *Controller) Status(ctx context.Context, nodeID string) (Status, error)
- func (c *Controller) Tick(ctx context.Context)
- type Executor
- type Option
- type Resource
- type Status
- type Store
Constants ¶
const ( ResolveWait = "wait" ResolveRestore = "restore" )
Resolution actions an operator can choose for a resource that needs a decision.
Variables ¶
var ( ErrNodeHealthy = errors.New("nodefailure: node is reachable") ErrNoFailure = errors.New("nodefailure: node has no failure in progress") ErrNoSuchAction = errors.New("nodefailure: no such resource on this node's failure list") )
Errors returned to operators.
Functions ¶
Types ¶
type Config ¶
type Config struct {
Mode string
Grace time.Duration
RecoveryHold time.Duration
Interval time.Duration
}
Config holds the control plane defaults a per-node policy overrides.
type Controller ¶
type Controller struct {
// contains filtered or unexported fields
}
Controller converges every node's failure state each tick; it is level-triggered and idempotent.
func (*Controller) MarkLost ¶
MarkLost declares an unreachable node lost: stateless apps move now and the rest wait for a per-resource decision. With dryRun it only returns the plan.
func (*Controller) Resolve ¶
func (c *Controller) Resolve(ctx context.Context, nodeID, kind, name, action, target string) (Status, error)
Resolve records an operator's decision for one resource of a failed node. ResolveRestore places it on target (or a picked node) so its data can be restored from backup there; the old copy is fenced when the node returns.
func (*Controller) SetPolicy ¶
func (c *Controller) SetPolicy(ctx context.Context, p store.NodeFailurePolicy) error
SetPolicy stores a node's policy override; an empty mode and zero grace restore the defaults.
func (*Controller) States ¶
func (c *Controller) States(ctx context.Context) (map[string]store.NodeFailureState, error)
States returns every live failure state keyed by node ID.
func (*Controller) Status ¶
Status reports a node's failure state and, while it is unreachable, the per-resource consequences (with would-be targets when nothing has run yet).
func (*Controller) Tick ¶
func (c *Controller) Tick(ctx context.Context)
Tick evaluates every node once.
type Executor ¶
type Executor interface {
IsLocal(nodeID string) bool
Resources(ctx context.Context, nodeID string) ([]Resource, error)
NodeOf(ctx context.Context, kind, name string) (string, error)
// PickTarget chooses a healthy node for res; commit false must not record anything.
PickTarget(ctx context.Context, res Resource, fromNodeID string, commit bool) (string, error)
Place(ctx context.Context, kind, name, nodeID string) error
// Fence stops the stale copy of a resource on nodeID.
Fence(ctx context.Context, kind, name, nodeID string) error
Audit(ctx context.Context, action, detail string)
}
Executor acts on the platform. Placement, teardown and audit stay with the caller.
type Status ¶
type Status struct {
NodeID string `json:"node_id"`
// Mode and GraceSeconds are the effective policy (override or default).
Mode string `json:"mode"`
GraceSeconds int `json:"grace_seconds"`
// Phase is "", "grace" (silent but inside the grace period), or a store.NodeFailurePhase*.
Phase string `json:"phase"`
// SilentSeconds and GraceRemainingSeconds drive the grace timer.
SilentSeconds int `json:"silent_seconds"`
GraceRemainingSeconds int `json:"grace_remaining_seconds"`
Reason string `json:"reason,omitempty"`
UnreachableAt *time.Time `json:"unreachable_at,omitempty"`
Consequences []store.NodeFailureAction `json:"consequences"`
Policy store.NodeFailurePolicy `json:"policy"`
State *store.NodeFailureState `json:"-"`
}
Status is what the dashboard and CLI show for one node.
type Store ¶
type Store interface {
ListNodes(ctx context.Context) ([]store.Node, error)
GetNode(ctx context.Context, id string) (*store.Node, error)
SetNodeSchedulable(ctx context.Context, id string, schedulable bool) error
GetNodeFailurePolicy(ctx context.Context, nodeID string) (store.NodeFailurePolicy, error)
SetNodeFailurePolicy(ctx context.Context, p store.NodeFailurePolicy) error
GetNodeFailureState(ctx context.Context, nodeID string) (*store.NodeFailureState, error)
ListNodeFailureStates(ctx context.Context) (map[string]store.NodeFailureState, error)
SaveNodeFailureState(ctx context.Context, s store.NodeFailureState) error
DeleteNodeFailureState(ctx context.Context, nodeID string) error
}
Store is the persistence surface the controller needs; *store.DB satisfies it.
Source Files