Documentation
¶
Overview ¶
Package reconciler contains the event-source foundation for a future self-healing reconciler (MINE-107). This file (events.go) implements a multi-node Docker Events API watcher: it connects to each node's Docker daemon, subscribes to container lifecycle events for containers managed by Carbon Panel, and translates them into a normalized ContainerEvent stream that a downstream consumer can act on.
This ticket (MINE-106) intentionally stops at the event source: no reconciliation logic, debouncing, or work queue is built here.
Package reconciler implements the level-triggered, self-healing reconciliation engine for Carbon Panel (MINE-107, Layer 2.2 of the Resilient Container Orchestrator epic, MINE-102).
The reconciler converges the desired state recorded in the database with the observed state of the Docker containers backing each server. It is level-triggered rather than edge-triggered: lifecycle events from the MINE-106 Watcher/Supervisor only schedule a per-server convergence pass (coalesced into a quiet window), while a periodic sweep guarantees that a dropped event can never leave a server permanently divergent.
Identity comes from MINE-103's deterministic ResolveContainer, so a container recreated by Watchtower, docker-compose, Portainer, or force-recreate is re-adopted rather than mistaken for a loss.
Index ¶
Constants ¶
const ( // DefaultDebounceWindow coalesces bursts of Docker events for the same // server into a single convergence pass. It is the dominant contributor to // the "external docker stop/kill is reflected within 500ms" budget. DefaultDebounceWindow = 300 * time.Millisecond // DefaultWorkers is the number of concurrent convergence goroutines. DefaultWorkers = 4 // DefaultSweepInterval is the safety-net period at which every server is // re-checked even in the absence of events. DefaultSweepInterval = 30 * time.Second )
Default tuning for the reconciler.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Client ¶
type Client interface {
ResolveContainer(ctx context.Context, serverID string) (*container.Summary, error)
ObserveContainer(ctx context.Context, containerID string) (*docker.ContainerState, error)
StartContainer(ctx context.Context, containerID string) error
RecreateContainer(ctx context.Context, oldContainerID string, server *db.Server, serverConfig *db.ServerConfig) (*docker.RecreateContainerResult, error)
StopContainerWithin(ctx context.Context, containerID string, window time.Duration) (bool, error)
}
Client is the slice of *docker.Client the Reconciler drives.
type ClientResolver ¶
ClientResolver resolves a node's Docker client without falling back to the local daemon (MINE-109's GetClientStrict). Returning an explicit error lets the reconciler distinguish "node unreachable" (leave the DB alone) from "container missing" (converge).
type Config ¶
type Config struct {
// DebounceWindow is the per-server quiet window that coalesces bursts of
// Docker events into a single convergence pass.
DebounceWindow time.Duration
// Workers is the number of concurrent convergence goroutines.
Workers int
// SweepInterval is the safety-net period at which every server is
// re-checked even without an event.
SweepInterval time.Duration
// DisableSelfHeal turns off automatic recreation of a missing container for
// a server the database still believes is running. Self-heal is ON by
// default; when disabled the divergence is only reported as an error.
DisableSelfHeal bool
}
Config tunes the reconciler. Zero-valued fields fall back to the defaults.
type ContainerEvent ¶
type ContainerEvent struct {
// NodeID identifies which node in the ClientPool this event came from.
NodeID string
// ServerID is populated from the carbon-panel.server.id label, if
// present. Empty when the event is for a module container.
ServerID string
// ModuleID is populated from the carbon-panel.module.id label, if
// present. Empty when the event is for a server container.
ModuleID string
// ContainerID is the Docker container ID (Actor.ID).
ContainerID string
// Action is the raw Docker event action string (e.g. "create",
// "start", "die", "destroy", "pause", "unpause", "rename",
// "health_status", "oom").
Action string
// ExitCode is parsed from Actor.Attributes["exitCode"] when Action is
// "die". Nil for all other actions, or if the attribute was missing
// or unparsable.
ExitCode *int
// Timestamp is when the event occurred, per the Docker daemon.
Timestamp time.Time
// Attributes carries the raw Actor.Attributes map for anything else a
// consumer might need (e.g. image name, container name).
Attributes map[string]string
}
ContainerEvent is the normalized representation of a Docker container lifecycle event, translated from the raw Docker Events API message. It is the contract consumed by the (future) reconciler.
type EventEmitter ¶
EventEmitter publishes lifecycle events on the central bus.
type LogMigrator ¶
type LogMigrator interface {
MigrateServerLogSubscriptions(serverID, oldContainerID, newContainerID string)
}
LogMigrator keeps a server's live console subscriptions attached to its container across a recreation (MINE-108): it moves the log streamer's subscriber registry and re-points the WebSocket hub's bookkeeping. It is optional — a nil LogMigrator simply disables the migration.
type PoolResolver ¶
type PoolResolver struct {
Pool *docker.ClientPool
}
PoolResolver adapts a *docker.ClientPool to the ClientResolver interface used by the Reconciler. Go method signatures are invariant, so the pool's concrete (*docker.Client, error) cannot satisfy (Client, error) on its own.
func (PoolResolver) GetClientStrict ¶
func (p PoolResolver) GetClientStrict(nodeID string) (Client, error)
GetClientStrict implements ClientResolver.
type ProxyRouter ¶
ProxyRouter re-points a server's proxy route at its current backend.
type Reconciler ¶
type Reconciler struct {
// contains filtered or unexported fields
}
Reconciler converges desired DB state with observed Docker state.
func New ¶
func New(store Store, clients ClientResolver, proxy ProxyRouter, bus EventEmitter, logMigrator LogMigrator, log *logger.Logger, cfg Config) *Reconciler
New constructs a Reconciler. It does not start any goroutines until Start is called. logMigrator may be nil, which disables console-subscription migration.
func (*Reconciler) ReconcileServer ¶
func (r *Reconciler) ReconcileServer(ctx context.Context, serverID string) error
ReconcileServer converges a single server's DB status with its container's observed state, adopting a drifted container ID, self-healing a missing container, and emitting lifecycle events on real transitions.
func (*Reconciler) Start ¶
func (r *Reconciler) Start(ctx context.Context, eventStream <-chan ContainerEvent)
Start launches the worker pool, consumes the supervisor's merged event stream (which may be nil), and begins the periodic sweep. It returns immediately; cancel ctx or call Stop to shut everything down.
func (*Reconciler) Stop ¶
func (r *Reconciler) Stop()
Stop halts the reconciler. It is safe to call more than once.
func (*Reconciler) Wait ¶
func (r *Reconciler) Wait()
Wait blocks until every reconciler goroutine has exited.
type Store ¶
type Store interface {
GetServer(ctx context.Context, id string) (*db.Server, error)
ListServers(ctx context.Context) ([]*db.Server, error)
GetServerConfig(ctx context.Context, serverID string) (*db.ServerConfig, error)
UpdateServerStatus(ctx context.Context, id string, status db.ServerStatus) error
UpdateServerContainerID(ctx context.Context, id string, containerID string) error
}
Store is the narrow database surface the Reconciler needs. *db.Store satisfies it.
Every write it uses is a targeted column update (MINE-110): the reconciler never calls the full-row UpdateServer, which would clobber a ContainerID written concurrently by another subsystem.
type Supervisor ¶
type Supervisor struct {
// contains filtered or unexported fields
}
Supervisor manages one Watcher per node and fans their output into a single merged event channel. Sync and WatchMembership add and remove nodes while the supervisor is running.
func NewSupervisor ¶
func NewSupervisor(nodeIDs []string, resolve func(nodeID string) (*docker.Client, error), log *logger.Logger) *Supervisor
NewSupervisor constructs a Supervisor for an explicit set of node IDs, using resolve to obtain a *docker.Client for each node ID. This is the most explicit constructor: callers who already know their node set (or who want to wire node discovery themselves, e.g. via a NodeStore's ListNodes) should use this directly.
func NewSupervisorFromNodeStore ¶
func NewSupervisorFromNodeStore(ctx context.Context, pool *docker.ClientPool, store docker.NodeStore, log *logger.Logger) (*Supervisor, error)
NewSupervisorFromNodeStore constructs a Supervisor that auto-discovers its node set from store.ListNodes (the same NodeStore backing pool; see internal/docker.NodeStore) and resolves each node's client via pool.GetClientStrict. Only enabled nodes are included. The initial set is read at call time; WatchMembership picks up nodes added or disabled later.
func (*Supervisor) Events ¶
func (s *Supervisor) Events() <-chan ContainerEvent
Events returns the merged, fanned-in channel of ContainerEvent values across all watched nodes. Valid after Start has been called.
func (*Supervisor) Start ¶
func (s *Supervisor) Start(ctx context.Context) error
Start resolves a client for each configured node, launches one Watcher per node, and begins fanning their output into Events(). It returns an error if any node's client could not be resolved; nodes that resolved successfully before the failure are still started (best-effort startup).
Start returns once all watchers are launched; it does not block for the lifetime of the supervisor. Cancel ctx to stop all watchers and close Events(). Sync may change membership until then.
func (*Supervisor) Sync ¶ added in v1.0.39
func (s *Supervisor) Sync(nodeIDs []string) error
Sync sets the watched node set. Before Start it only records the set. After Start it starts watchers for new nodes and stops watchers for nodes that disappeared. A node that fails to resolve is skipped and the first such error is returned; nodes that are already running stay up.
func (*Supervisor) WatchMembership ¶ added in v1.0.39
func (s *Supervisor) WatchMembership(ctx context.Context, every time.Duration, list func(context.Context) ([]string, error))
WatchMembership refreshes the watched set on every interval until ctx is cancelled. A failed listing keeps the previous watchers.
type Watcher ¶
type Watcher struct {
// contains filtered or unexported fields
}
Watcher streams container lifecycle events for a single node, translating them into ContainerEvent values sent on Out(). It reconnects with exponential backoff on stream errors/disconnects and respects context cancellation for clean shutdown.
func NewWatcher ¶
NewWatcher constructs a Watcher for a single node's Docker client. The returned Watcher does not start streaming until Run is called.
func (*Watcher) Out ¶
func (w *Watcher) Out() <-chan ContainerEvent
Out returns the channel Watcher sends translated events on. Callers should drain it; Run closes it when it returns.