deployer

package
v0.8.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 28, 2026 License: Apache-2.0 Imports: 40 Imported by: 0

README

pkg/controller/deployer

NewDeployStack builds the three components that get validated configurations onto HAProxy pods:

  • DeploymentScheduler — decides when to deploy. Keeps the last validated config + last discovered endpoints and queues at most one pending deployment ("latest wins"). Also times out deployments that take longer than deploymentTimeout so a dropped DeploymentCompletedEvent can't wedge the pipeline forever.
  • Component (the deployer itself) — consumes DeploymentScheduledEvent and applies the render to every discovered pod through its HAPTIC agent (pkg/dataplane/agent/client).
  • DriftPreventionMonitor — fires a synthetic DriftPreventionTriggeredEvent every driftPreventionInterval, which re-renders and re-applies. That pass reads each pod's state with ?verify=1, so a file changed behind the controller's back is a digest difference the apply rewrites.

All three are leader-only — only the replica holding the Lease deploys, observers on other replicas stay idle.

How one pod is applied to

  1. GET /v1/state — the plan the pod applied, the plan its worker runs, the digest of every file it holds, its runtime inventory and its HAProxy version.
  2. The baseline is that applied plan, resolved through the deployer's plan cache; on a miss the opaque blob the pod stored is decoded, which is what makes a leader change cost no reload. A blob whose schema version or plan id does not check out is no baseline, and the pod gets the complete state plus a reload.
  3. deployplan.Diff answers runtime, file_only or reload plus the reasons. Pods reporting the same baseline and capabilities share one answer.
  4. POST /v1/apply carries the complete desired file set at digest granularity, a part only for a file the agent lacks (haproxy.cfg always travels whole), and all operation batches in one transaction. Agents without runtime_batches support receive a full reload for multi-batch changes. At most 16 pods concurrently, each bounded by syncTimeout.
  5. The ACK's applied and running plan ids, the mode and the reasons land in HAProxyCfg.status.deployedToPods[].

Every apply is fenced by a token — the leader epoch (a counter on the leader Lease, claimed per leadership term) and a per-term apply sequence — plus the baseline it was composed against. A 409 answers with the pod's actual baseline: prev_mismatch re-diffs once, unknown_baseline falls back to the complete state, and stale_epoch stands this controller down. A 409 naming missing file parts is answered by resending exactly those.

A refused apply (NACK) counts haptic_apply_rejected_total{pod}, carries HAProxy's own words into the pod's status, and drops that pod's baseline. An agent speaking a different API major gets the complete state and a reload; one missing a composed op kind reloads only for the plans that need it. Both count haptic_agent_version_skew_total and are never a refusal, which would fence the repair path.

The full contract, with the sequence diagram, is in docs/site/docs/development/design/deployment.md.

Construction

stack := deployer.NewDeployStack(bus, cfg, logger, domainMetrics, renderInputs, fence)

renderInputs is what the deploy side feeds back into the next render: the plan the fleet ACKed (server slots) and the capabilities the fleet's lowest HAProxy version supports. fence is the leadership term every apply is stamped with; nil means a single writer at epoch zero.

The controller registers the returned components with the leader lifecycle in pkg/controller/reconciliation.go.

Durations come from spec.dataplane and spec.controller on the CRD: deploymentTimeout, driftPreventionInterval and syncTimeout bound the controller; minDeploymentInterval and reloadVerificationTimeout configure the agent's reload pacing and reload deadline through the chart.

Event Flow

TemplateRenderedEvent ───────┐
RenderGateCompletedEvent ────┤
HAProxyPodsDiscoveredEvent ──┤
DriftPreventionTriggeredEvent┤
DeploymentCompletedEvent ────┤       (feedback edge)
                             ▼
                     DeploymentScheduler
                             │
                             ▼
                     DeploymentScheduledEvent
                             │
                             ▼
                         Component
                             │
             ▼
           DeploymentStartedEvent
           InstanceDeploymentFailedEvent (per pod)
           ConfigAppliedToPodEvent (per pod)
           DeploymentCompletedEvent
                             │
                             ▼
                   DriftPreventionMonitor
                             │
                             ▼
                   DriftPreventionTriggeredEvent (if idle for > interval)

Notable details:

  • The scheduler only deploys when it has all three inputs: a rendered config, a successful validation, and at least one discovered HAProxy endpoint. Partial state waits.
  • "Latest wins" is a single slot — concurrent changes don't queue up as a FIFO, they coalesce to the most recent one. One deployment is in flight at a time, which is the whole rate limit: reload pacing belongs to the agent, so an apply that needs no reload is not held back.
  • A gate verdict can become stale before delivery even when it was newest at the gate. The scheduler checks it against its held and dispatched render content before changing the latch, accepted render, or pending deployment. A stale verdict is debug-level tracing; an unauthenticated verdict remains an error.
  • DeploymentCompletedEvent.Succeeded counts the pods running the render, not the pods whose apply was accepted. A pod whose paced reload is still scheduled is neither converged nor failed.
  • The DeploymentCompletedEvent matching the active deployment ID closes the scheduler's in-progress flag. Every completion resets the drift monitor's idle timer, which is why the event is on the feedback edge in the diagram.
  • deploymentTimeout is a safety net, not an operational target — hitting it means a lost completion event or a stuck apply, both of which are bugs to investigate.
  • Cancellation uses a separate control subscription, so it reaches a blocked apply without waiting behind that call in the deployment mailbox. A timed-out deployment keeps the scheduler slot until its exact deployment ID reports termination.
  • An endpoint-authority change (a pod replaced under the same URL) retires the in-flight deployment: its pods are not the fleet any more.

Leadership Transitions

On LostLeadershipEvent the scheduler drops any pending deployment and clears its in-progress flag (otherwise a new leader would wait on a deployment the dead leader was handling); the drift monitor stops its timer. The deployer closes its pooled agent connections and restarts its apply sequence on the next term, under a new epoch.

On BecameLeaderEvent the scheduler is bootstrapped from two sides:

  • All-replica components that maintain state replay their last event so the new leader's scheduler doesn't have to wait. Currently that's HAProxyPodsDiscoveredEvent (from pkg/controller/discovery) and ConfigValidatedEvent (from pkg/controller/configchange). Grep for leadership.NewStateReplayer[ to see the canonical list.
  • Neither TemplateRenderedEvent nor RenderGateCompletedEvent is replayed — the render comes from the leader-only reconciler.Coordinator (ADR-0001) and the verdict from the leader-only rendergate, so both only exist on the leader to begin with. Instead, the reconciler triggers a fresh reconciliation on BecameLeaderEvent, which produces a fresh render (and a fresh verdict) rather than stale replays.

The new leader's first deployment reloads nothing: each pod reports the plan it applied, and the blob it stored is decodable by any leader.

See pkg/controller/LEADER_ONLY_COMPONENTS.md for the full replay/clear contract every leader-only component implements.

See Also

License

Apache-2.0 — see root LICENSE.

Documentation

Overview

Package deployer applies a validated render to the HAProxy fleet through the HAPTIC agent that owns each pod's file tree.

The Deployer is a stateless executor: it receives DeploymentScheduledEvent and, per pod, reads the agent's baseline, diffs the render against it (pkg/dataplane/deployplan) and sends the resulting fenced applies. All scheduling, rate limiting and queueing lives in the DeploymentScheduler.

Package deployer implements deployment scheduling and execution components.

Index

Constants

View Source 
const (
	// ComponentName is the unique identifier for this component.
	ComponentName = "deployer"

	// EventBufferSize is the size of the event subscription buffer.
	// Low-volume component (~1-2 deployment events per reconciliation cycle).
	EventBufferSize = busevents.StandardSubscriberBuffer
)
View Source 
const (
	// SchedulerComponentName is the unique identifier for the DeploymentScheduler component.
	SchedulerComponentName = "deployment-scheduler"

	// SchedulerEventBufferSize is the size of the event subscription buffer for the scheduler.
	// Moderate-volume component handling template, validation, and discovery events.
	// Named with "Scheduler" prefix to avoid conflict with EventBufferSize in this package.
	SchedulerEventBufferSize = busevents.StandardSubscriberBuffer
)
View Source 
const (
	// DriftMonitorComponentName is the unique identifier for the drift prevention monitor component.
	DriftMonitorComponentName = "drift-monitor"
)

Variables

This section is empty.

Functions

This section is empty.

Types

type AckedPlanSink 

type AckedPlanSink interface {
	SetAckedPlan(*renderplan.Plan)
}

AckedPlanSink receives the plan the fleet confirmed it is running. The renderer implements it; the deployer calls it after a deployment lands.

type Component 

type Component struct {
	*component.Base
	*component.ReadySignal
	// contains filtered or unexported fields
}

Component implements the deployer component.

Event subscriptions:

  • DeploymentScheduledEvent: apply the render to the scheduled endpoints
  • DeploymentCancelRequestEvent: cancel an in-progress deployment through a separate control loop that stays responsive while execution blocks

func New 

func New(eventBus *busevents.EventBus, logger *slog.Logger, syncTimeout time.Duration, domainMetrics *metrics.Metrics) *Component

New creates a new Deployer component.

syncTimeout bounds one pod's apply; the agent's own reload deadline is a chart-templated agent flag, not a client timeout. domainMetrics may be nil in tests; the per-pod counters are then not recorded.

func (*Component) CoalescesOn 

func (c *Component) CoalescesOn() []string

CoalescesOn implements component.CoalescingHandler: after each dispatch, the embedded component.Base drains the subscription channel and processes only the LATEST pending coalescible DeploymentScheduledEvent ("latest wins").

This coalescing is load-bearing: deployment is single-threaded, but the validator + scheduler upstream can fire many DeploymentScheduledEvents during a single deployment. Without it, the deployer would process every queued event in FIFO order — deploying the OLDEST pending config first and falling further and further behind under load.

func (*Component) Drain 

func (c *Component) Drain(ctx context.Context) error

Drain ends this term's deployments and returns once the one in flight has published its receipt, provided the term still holds the Lease.

func (*Component) HandleEvent 

func (c *Component) HandleEvent(event busevents.Event)

HandleEvent implements component.EventHandler: it routes events to the appropriate handler.

func (*Component) HealthCheck 

func (c *Component) HealthCheck() error

HealthCheck implements the lifecycle.HealthChecker interface. Returns an error if the component appears to be stalled (processing for > timeout). Returns nil when idle - idle is always healthy for event-driven components.

func (*Component) SetValidatedOccurrence 

func (c *Component) SetValidatedOccurrence(occurrence *rendercycle.Occurrence)

SetValidatedOccurrence records one exact render execution the gate passed.

func (*Component) Start 

func (c *Component) Start(ctx context.Context) error

Start begins the deployer's event loop and blocks until ctx is cancelled.

type DeployStack 

type DeployStack struct {
	Deployer     *Component
	Scheduler    *DeploymentScheduler
	DriftMonitor *DriftPreventionMonitor
}

DeployStack is the deploy-side component set, already wired together.

func NewDeployStack 

func NewDeployStack(
	eventBus *busevents.EventBus,
	cfg *coreconfig.Config,
	logger *slog.Logger,
	domainMetrics *metrics.Metrics,
	renderInputs RenderInputs,
	fence LeadershipFence,
	tls *transportsecurity.Source,
) *DeployStack

NewDeployStack builds the three deploy-side components and connects them to the metrics registry, the leadership fence and the render inputs.

The wiring lives here so a caller cannot forget one of the connections. domainMetrics is required. renderInputs and fence may be nil: the renderer then keeps rendering from its own plans and the local HAProxy probe, and applies are fenced at epoch zero (single writer, leader election disabled).

func (*DeployStack) EnableRetainedRecovery 

func (s *DeployStack) EnableRetainedRecovery(store RetainedStore, checker RetainedChecker, template *v1.HAProxyTemplateConfig, domainMetrics *metrics.Metrics, leaseName string)

EnableRetainedRecovery wires the leader's read, validation and bootstrap path.

type DeploymentScheduler 

type DeploymentScheduler struct {
	*component.ReadySignal
	// contains filtered or unexported fields
}

DeploymentScheduler decides which render is deployed next, and when.

It subscribes to events that trigger deployments, maintains the state of rendered and validated configurations, and holds a new render while the fleet's paced reloads are still pending. One deployment in flight is the only rate limit it applies; reload pacing lives in the agent.

Event subscriptions:

  • TemplateRenderedEvent: Cache the render and schedule deployment
  • RenderGateCompletedEvent: Move the gate's latch; a pass releases a held render
  • ValidationFailedEvent: Deploy cached config for drift prevention fallback
  • HAProxyPodsDiscoveredEvent: Update endpoints and schedule deployment

The component publishes DeploymentScheduledEvent when a deployment should execute.

func (*DeploymentScheduler) Drain 

func (s *DeploymentScheduler) Drain(ctx context.Context) error

Drain stops dispatching any plan other than the one last dispatched, which is the plan the deployment intent names, and returns once that deployment and its pending-reload follow-ups have completed: a paced reload confirms its worker receipt only on the follow-up.

func (*DeploymentScheduler) HealthCheck 

func (s *DeploymentScheduler) HealthCheck() error

HealthCheck implements the lifecycle.HealthChecker interface. Returns an error if the component appears to be stalled (processing for > timeout). Returns nil when idle (not processing) - idle is always healthy for event-driven components.

func (*DeploymentScheduler) Name 

func (s *DeploymentScheduler) Name() string

Name returns the unique identifier for this component. Implements the lifecycle.Component interface.

func (*DeploymentScheduler) Start 

func (s *DeploymentScheduler) Start(ctx context.Context) error

Start begins the deployment scheduler's event loop.

This method blocks until the context is cancelled or an error occurs. It subscribes to events when called (after leadership is acquired).

Parameters:

  • ctx: Context for cancellation and lifecycle management

Returns:

  • nil when context is cancelled (graceful shutdown)
  • Error only in exceptional circumstances

type DriftPreventionMonitor 

type DriftPreventionMonitor struct {
	*component.ReadySignal
	// contains filtered or unexported fields
}

DriftPreventionMonitor triggers periodic reconciliation to prevent configuration drift in HAProxy pods.

When no deployment has occurred within the configured interval, it publishes a DriftPreventionTriggeredEvent to trigger reconciliation. This helps detect and correct configuration drift caused by anything writing to the pod's file tree or manual changes.

This is a leader-only component that starts when leadership is acquired. Only the leader needs drift prevention since only the leader deploys. The Reconciler triggers fresh reconciliation on BecameLeaderEvent to provide current state.

Event subscriptions:

  • DeploymentCompletedEvent: Reset drift prevention timer
  • LostLeadershipEvent: Stop drift timer when losing leadership

The component publishes DriftPreventionTriggeredEvent when drift prevention is needed.

func NewDriftPreventionMonitor 

func NewDriftPreventionMonitor(eventBus *busevents.EventBus, logger *slog.Logger, driftPreventionInterval time.Duration) *DriftPreventionMonitor

NewDriftPreventionMonitor creates a new DriftPreventionMonitor component.

As a leader-only component, subscription happens in Start() after leadership is acquired, not during construction.

Parameters:

  • eventBus: The EventBus for subscribing to events and publishing triggers
  • logger: Structured logger for component logging
  • driftPreventionInterval: Interval after which to trigger drift prevention deployment

Returns:

  • A new DriftPreventionMonitor instance ready to be started

func (*DriftPreventionMonitor) HealthCheck 

func (m *DriftPreventionMonitor) HealthCheck() error

HealthCheck implements the lifecycle.HealthChecker interface. Returns an error if the component appears to be stalled (no timer tick for > stallTimeout). For a timer-based component like DriftPreventionMonitor, a healthy state means the timer is firing at the expected interval.

func (*DriftPreventionMonitor) Name 

func (m *DriftPreventionMonitor) Name() string

Name returns the unique identifier for this component. Implements the lifecycle.Component interface.

func (*DriftPreventionMonitor) Start 

func (m *DriftPreventionMonitor) Start(ctx context.Context) error

Start begins the drift prevention monitor's event loop.

This method blocks until the context is cancelled or an error occurs. As a leader-only component, it subscribes to events when started (after leadership is acquired).

Event handling:

  • DeploymentCompletedEvent: Resets the drift prevention timer
  • LostLeadershipEvent: Stops drift timer when losing leadership
  • Drift timer expiration: Publishes DriftPreventionTriggeredEvent

The component runs until the context is cancelled, at which point it performs cleanup and returns.

Parameters:

  • ctx: Context for cancellation and lifecycle management

Returns:

  • nil when context is cancelled (graceful shutdown)
  • Error only in exceptional circumstances

type FleetCapabilitiesSink 

type FleetCapabilitiesSink interface {
	SetCapabilities(dataplane.Capabilities)
}

FleetCapabilitiesSink receives what the fleet's HAProxy version supports. The renderer implements it; templates read the result as `capabilities`.

type LeadershipFence 

type LeadershipFence interface {
	Identity() string
	LeaderEpoch() uint64
	// Reclaim lifts the epoch past one a pod already accepted. It errors when
	// a newer leader — not a regressed counter — is behind the refusal, which
	// is the one case where the pod is right and this controller must stop.
	Reclaim(ctx context.Context, floor uint64) (uint64, error)
	// StandDown gives leadership up so a fresh term claims a fresh epoch.
	StandDown(reason string)
}

LeadershipFence is the current leadership term: who this controller is, the epoch every apply it sends is fenced by, and the two answers to a pod that refuses that epoch. A pod that has seen a higher epoch refuses this controller's writes.

type RenderInputs 

type RenderInputs interface {
	AckedPlanSink
	FleetCapabilitiesSink
}

RenderInputs is what the deploy side feeds back into the next render: the plan the fleet ACKed and the capabilities its HAProxy version supports. The renderer implements both.

type RetainedChecker 

type RetainedChecker interface {
	CheckOutput(context.Context, *renderoutput.Snapshot, string) error
}

RetainedChecker checks a complete snapshot with the current HAProxy binary.

type RetainedIntentStore 

type RetainedIntentStore interface {
	ClaimDeploymentAuthority(context.Context, *configpublisher.DeploymentAuthority) error
	RecordDeploymentIntent(context.Context, *configpublisher.DeploymentAuthority, string, string) error
	CheckDeploymentIntent(context.Context, *configpublisher.DeploymentAuthority, string, string) error
	CheckDeploymentAuthority(context.Context, *configpublisher.DeploymentAuthority) error
}

RetainedIntentStore fences writers and records output identity before apply.

type RetainedLoader 

type RetainedLoader interface {
	LoadRetained(context.Context, string, string, types.UID) ([]configpublisher.RetainedConfig, error)
}

RetainedLoader reads complete, acknowledged snapshots without publishing.

type RetainedStore 

type RetainedStore interface {
	RetainedLoader
	RetainedIntentStore
}

RetainedStore provides snapshots and the durable deployment barrier.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.