reconciler

package
v1.0.39 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 28, 2026 License: MIT Imports: 16 Imported by: 0

Documentation

Overview

Package reconciler contains the event-source foundation for a future self-healing reconciler (MINE-107). This file (events.go) implements a multi-node Docker Events API watcher: it connects to each node's Docker daemon, subscribes to container lifecycle events for containers managed by Carbon Panel, and translates them into a normalized ContainerEvent stream that a downstream consumer can act on.

This ticket (MINE-106) intentionally stops at the event source: no reconciliation logic, debouncing, or work queue is built here.

Package reconciler implements the level-triggered, self-healing reconciliation engine for Carbon Panel (MINE-107, Layer 2.2 of the Resilient Container Orchestrator epic, MINE-102).

The reconciler converges the desired state recorded in the database with the observed state of the Docker containers backing each server. It is level-triggered rather than edge-triggered: lifecycle events from the MINE-106 Watcher/Supervisor only schedule a per-server convergence pass (coalesced into a quiet window), while a periodic sweep guarantees that a dropped event can never leave a server permanently divergent.

Identity comes from MINE-103's deterministic ResolveContainer, so a container recreated by Watchtower, docker-compose, Portainer, or force-recreate is re-adopted rather than mistaken for a loss.

Index

Constants

View Source
const (
	// DefaultDebounceWindow coalesces bursts of Docker events for the same
	// server into a single convergence pass. It is the dominant contributor to
	// the "external docker stop/kill is reflected within 500ms" budget.
	DefaultDebounceWindow = 300 * time.Millisecond
	// DefaultWorkers is the number of concurrent convergence goroutines.
	DefaultWorkers = 4
	// DefaultSweepInterval is the safety-net period at which every server is
	// re-checked even in the absence of events.
	DefaultSweepInterval = 30 * time.Second
)

Default tuning for the reconciler.

Variables

This section is empty.

Functions

This section is empty.

Types

type Client

type Client interface {
	ResolveContainer(ctx context.Context, serverID string) (*container.Summary, error)
	ObserveContainer(ctx context.Context, containerID string) (*docker.ContainerState, error)
	StartContainer(ctx context.Context, containerID string) error
	RecreateContainer(ctx context.Context, oldContainerID string, server *db.Server, serverConfig *db.ServerConfig) (*docker.RecreateContainerResult, error)
	StopContainerWithin(ctx context.Context, containerID string, window time.Duration) (bool, error)
}

Client is the slice of *docker.Client the Reconciler drives.

type ClientResolver

type ClientResolver interface {
	GetClientStrict(nodeID string) (Client, error)
}

ClientResolver resolves a node's Docker client without falling back to the local daemon (MINE-109's GetClientStrict). Returning an explicit error lets the reconciler distinguish "node unreachable" (leave the DB alone) from "container missing" (converge).

type Config

type Config struct {
	// DebounceWindow is the per-server quiet window that coalesces bursts of
	// Docker events into a single convergence pass.
	DebounceWindow time.Duration
	// Workers is the number of concurrent convergence goroutines.
	Workers int
	// SweepInterval is the safety-net period at which every server is
	// re-checked even without an event.
	SweepInterval time.Duration
	// DisableSelfHeal turns off automatic recreation of a missing container for
	// a server the database still believes is running. Self-heal is ON by
	// default; when disabled the divergence is only reported as an error.
	DisableSelfHeal bool
}

Config tunes the reconciler. Zero-valued fields fall back to the defaults.

type ContainerEvent

type ContainerEvent struct {
	// NodeID identifies which node in the ClientPool this event came from.
	NodeID string
	// ServerID is populated from the carbon-panel.server.id label, if
	// present. Empty when the event is for a module container.
	ServerID string
	// ModuleID is populated from the carbon-panel.module.id label, if
	// present. Empty when the event is for a server container.
	ModuleID string
	// ContainerID is the Docker container ID (Actor.ID).
	ContainerID string
	// Action is the raw Docker event action string (e.g. "create",
	// "start", "die", "destroy", "pause", "unpause", "rename",
	// "health_status", "oom").
	Action string
	// ExitCode is parsed from Actor.Attributes["exitCode"] when Action is
	// "die". Nil for all other actions, or if the attribute was missing
	// or unparsable.
	ExitCode *int
	// Timestamp is when the event occurred, per the Docker daemon.
	Timestamp time.Time
	// Attributes carries the raw Actor.Attributes map for anything else a
	// consumer might need (e.g. image name, container name).
	Attributes map[string]string
}

ContainerEvent is the normalized representation of a Docker container lifecycle event, translated from the raw Docker Events API message. It is the contract consumed by the (future) reconciler.

type EventEmitter

type EventEmitter interface {
	Emit(ctx context.Context, event events.Event)
}

EventEmitter publishes lifecycle events on the central bus.

type LogMigrator

type LogMigrator interface {
	MigrateServerLogSubscriptions(serverID, oldContainerID, newContainerID string)
}

LogMigrator keeps a server's live console subscriptions attached to its container across a recreation (MINE-108): it moves the log streamer's subscriber registry and re-points the WebSocket hub's bookkeeping. It is optional — a nil LogMigrator simply disables the migration.

type PoolResolver

type PoolResolver struct {
	Pool *docker.ClientPool
}

PoolResolver adapts a *docker.ClientPool to the ClientResolver interface used by the Reconciler. Go method signatures are invariant, so the pool's concrete (*docker.Client, error) cannot satisfy (Client, error) on its own.

func (PoolResolver) GetClientStrict

func (p PoolResolver) GetClientStrict(nodeID string) (Client, error)

GetClientStrict implements ClientResolver.

type ProxyRouter

type ProxyRouter interface {
	UpdateServerRoute(server *db.Server) error
}

ProxyRouter re-points a server's proxy route at its current backend.

type Reconciler

type Reconciler struct {
	// contains filtered or unexported fields
}

Reconciler converges desired DB state with observed Docker state.

func New

func New(store Store, clients ClientResolver, proxy ProxyRouter, bus EventEmitter, logMigrator LogMigrator, log *logger.Logger, cfg Config) *Reconciler

New constructs a Reconciler. It does not start any goroutines until Start is called. logMigrator may be nil, which disables console-subscription migration.

func (*Reconciler) ReconcileServer

func (r *Reconciler) ReconcileServer(ctx context.Context, serverID string) error

ReconcileServer converges a single server's DB status with its container's observed state, adopting a drifted container ID, self-healing a missing container, and emitting lifecycle events on real transitions.

func (*Reconciler) Start

func (r *Reconciler) Start(ctx context.Context, eventStream <-chan ContainerEvent)

Start launches the worker pool, consumes the supervisor's merged event stream (which may be nil), and begins the periodic sweep. It returns immediately; cancel ctx or call Stop to shut everything down.

func (*Reconciler) Stop

func (r *Reconciler) Stop()

Stop halts the reconciler. It is safe to call more than once.

func (*Reconciler) Wait

func (r *Reconciler) Wait()

Wait blocks until every reconciler goroutine has exited.

type Store

type Store interface {
	GetServer(ctx context.Context, id string) (*db.Server, error)
	ListServers(ctx context.Context) ([]*db.Server, error)
	GetServerConfig(ctx context.Context, serverID string) (*db.ServerConfig, error)
	UpdateServerStatus(ctx context.Context, id string, status db.ServerStatus) error
	UpdateServerContainerID(ctx context.Context, id string, containerID string) error
}

Store is the narrow database surface the Reconciler needs. *db.Store satisfies it.

Every write it uses is a targeted column update (MINE-110): the reconciler never calls the full-row UpdateServer, which would clobber a ContainerID written concurrently by another subsystem.

type Supervisor

type Supervisor struct {
	// contains filtered or unexported fields
}

Supervisor manages one Watcher per node and fans their output into a single merged event channel. Sync and WatchMembership add and remove nodes while the supervisor is running.

func NewSupervisor

func NewSupervisor(nodeIDs []string, resolve func(nodeID string) (*docker.Client, error), log *logger.Logger) *Supervisor

NewSupervisor constructs a Supervisor for an explicit set of node IDs, using resolve to obtain a *docker.Client for each node ID. This is the most explicit constructor: callers who already know their node set (or who want to wire node discovery themselves, e.g. via a NodeStore's ListNodes) should use this directly.

func NewSupervisorFromNodeStore

func NewSupervisorFromNodeStore(ctx context.Context, pool *docker.ClientPool, store docker.NodeStore, log *logger.Logger) (*Supervisor, error)

NewSupervisorFromNodeStore constructs a Supervisor that auto-discovers its node set from store.ListNodes (the same NodeStore backing pool; see internal/docker.NodeStore) and resolves each node's client via pool.GetClientStrict. Only enabled nodes are included. The initial set is read at call time; WatchMembership picks up nodes added or disabled later.

func (*Supervisor) Events

func (s *Supervisor) Events() <-chan ContainerEvent

Events returns the merged, fanned-in channel of ContainerEvent values across all watched nodes. Valid after Start has been called.

func (*Supervisor) Start

func (s *Supervisor) Start(ctx context.Context) error

Start resolves a client for each configured node, launches one Watcher per node, and begins fanning their output into Events(). It returns an error if any node's client could not be resolved; nodes that resolved successfully before the failure are still started (best-effort startup).

Start returns once all watchers are launched; it does not block for the lifetime of the supervisor. Cancel ctx to stop all watchers and close Events(). Sync may change membership until then.

func (*Supervisor) Sync added in v1.0.39

func (s *Supervisor) Sync(nodeIDs []string) error

Sync sets the watched node set. Before Start it only records the set. After Start it starts watchers for new nodes and stops watchers for nodes that disappeared. A node that fails to resolve is skipped and the first such error is returned; nodes that are already running stay up.

func (*Supervisor) WatchMembership added in v1.0.39

func (s *Supervisor) WatchMembership(ctx context.Context, every time.Duration, list func(context.Context) ([]string, error))

WatchMembership refreshes the watched set on every interval until ctx is cancelled. A failed listing keeps the previous watchers.

type Watcher

type Watcher struct {
	// contains filtered or unexported fields
}

Watcher streams container lifecycle events for a single node, translating them into ContainerEvent values sent on Out(). It reconnects with exponential backoff on stream errors/disconnects and respects context cancellation for clean shutdown.

func NewWatcher

func NewWatcher(nodeID string, cli *docker.Client, log *logger.Logger) *Watcher

NewWatcher constructs a Watcher for a single node's Docker client. The returned Watcher does not start streaming until Run is called.

func (*Watcher) Out

func (w *Watcher) Out() <-chan ContainerEvent

Out returns the channel Watcher sends translated events on. Callers should drain it; Run closes it when it returns.

func (*Watcher) Run

func (w *Watcher) Run(ctx context.Context)

Run streams events until ctx is cancelled, reconnecting on error with exponential backoff. It blocks until ctx is done, closing Out() before returning.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL