agentupgrade

package
v4.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 11, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package agentupgrade runs staged Agent upgrades (forward-sdk.md section 9, O4; owner decision H19): Control, never the node, decides when an Agent upgrades. A campaign pushes one signed Agent release to the enrolled nodes in batches (5% → 25% → 100% by default, at least 30 minutes each), canaries first in a deterministic order, as the agent.upgrade operation on the Agent Control stream of every Agent that negotiated upgrade.v1. A node succeeds when its Agent reconnects with the target version (and its configuration still applies); a batch in which more than 5% of the offered nodes fail — refused, failed to apply, or did not reconnect with the new version within 10 minutes — is rolled back: the campaign stops and the upgraded nodes of that batch are told to reinstate their previous release.

The campaign lives in two protected tables (v4_kernel_agent_upgrade_campaign, v4_kernel_agent_upgrade_node); the Worker drives it in the singleton-worker process, which holds the Agent streams.

Index

Constants

View Source
const (
	// FailureThresholdPercent: a batch rolls back when more than this share
	// of its offered nodes fail.
	FailureThresholdPercent = 5
	// ReconnectTimeout: an offered node must reconnect with the target
	// version within this time.
	ReconnectTimeout = 10 * time.Minute
	// MinBatchDuration is the shortest a batch may last.
	MinBatchDuration = 30 * time.Minute
	// FirstBatchMaxPercent caps the canary batch.
	FirstBatchMaxPercent = 5
	// MaxBatches bounds a campaign's batches.
	MaxBatches = 10
)

The H19 rules.

View Source
const (
	CodeBatchFailed       = "batch_failed"
	CodeAborted           = "aborted"
	CodeRejected          = "rejected"
	CodeApplyFailed       = "apply_failed"
	CodeReconnectTimeout  = "reconnect_timeout"
	CodeReverted          = "reverted"
	CodeConfigFailed      = "config_apply_failed"
	CodeUnsupported       = "upgrade_unsupported"
	CodeOffline           = "node_offline"
	CodeOfferFailed       = "offer_failed"
	CodeRollbackFailed    = "rollback_failed"
	CodeRollbackUnconfirm = "rollback_unconfirmed"
)

Error codes recorded on campaigns and nodes.

View Source
const (
	AuditActionStart    = "agent_upgrade_start"
	AuditActionPause    = "agent_upgrade_pause"
	AuditActionResume   = "agent_upgrade_resume"
	AuditActionAbort    = "agent_upgrade_abort"
	AuditActionRollback = "agent_upgrade_rollback"
	AuditActionFinish   = "agent_upgrade_finish"
)

Audit actions (v2_operation_log, module agent_upgrade).

View Source
const (
	// DefaultInterval is how often the worker evaluates the campaign.
	DefaultInterval = 5 * time.Second
	// ConfigGrace is how long a node that reconnected with the target
	// version and negotiated config.v1 has to report a failed
	// ConfigStatus before it counts as succeeded (it keeps being watched
	// for the rest of its batch).
	ConfigGrace = time.Minute
)

Worker timing.

Variables

View Source
var (
	ErrInvalid        = errors.New("invalid agent upgrade campaign")
	ErrActiveCampaign = errors.New("another agent upgrade campaign is active")
	ErrNoNodes        = errors.New("no node can be upgraded")
	ErrNotFound       = errors.New("agent upgrade campaign not found")
	ErrState          = errors.New("the campaign is not in a state that allows this")
)

Errors of the campaign API; handlers map them to answers.

Functions

func BatchBounds

func BatchBounds(total int, batches []Batch) []int

BatchBounds splits total nodes, ordered by OrderKey, into the batches: batch i takes the nodes from bounds[i] to bounds[i+1]. A cumulative share rounds up, so the canary batch has at least one node.

func Candidates

func Candidates(ctx context.Context, db *gorm.DB, exclude Exclude) ([]agentcontrol.AgentNode, error)

Candidates lists the nodes a campaign includes: enabled proxy and forward nodes whose Agent was seen on the Agent Control stream (v4_kernel_agent_transport, mtls-stream or apikey-stream), minus the excluded nodes and tags, in canary order.

func OrderKey

func OrderKey(node agentcontrol.AgentNode) string

OrderKey is a node's place in the canary order: the hex SHA-256 of its name, so the same nodes are the canaries of every campaign.

func ValidateBatches

func ValidateBatches(batches []Batch) error

ValidateBatches checks batches against H19: 1 to MaxBatches, cumulative percentages strictly increasing and ending at 100, the first at most FirstBatchMaxPercent, each at least MinBatchDuration.

Types

type Actor

type Actor struct {
	UserID uint
	Name   string
	IP     string
}

Actor is who changes a campaign: an administrator (UserID, Name, IP) or the command line (UserID 0).

type Batch

type Batch struct {
	Percent            int `json:"percent"`
	MinDurationSeconds int `json:"min_duration_seconds"`
}

Batch is one step of a campaign: the nodes up to Percent of the campaign's nodes (cumulative), which run for at least MinDurationSeconds before the next batch starts.

func DefaultBatches

func DefaultBatches() []Batch

DefaultBatches are H19's batches: 5%, 25% and 100% of the nodes (cumulative), 30 minutes each.

type BatchView

type BatchView struct {
	Index              int            `json:"index"`
	Percent            int            `json:"percent"`
	MinDurationSeconds int            `json:"min_duration_seconds"`
	Nodes              int            `json:"nodes"`
	States             map[string]int `json:"states"`
	// Offered and Failed are the batch's failure ratio's terms (H19):
	// skipped and pending nodes do not count.
	Offered int `json:"offered"`
	Failed  int `json:"failed"`
}

BatchView is one batch of a campaign with its nodes counted by state.

type CampaignView

type CampaignView struct {
	model.AgentUpgradeCampaign
	Artifacts []agentcontrol.UpgradeArtifact `json:"artifacts"`
	Batches   []BatchView                    `json:"batches"`
	Exclude   Exclude                        `json:"exclude"`
	Total     int                            `json:"total"`
	States    map[string]int                 `json:"states"`
	// BatchEndsAt is the earliest the current batch can pass.
	BatchEndsAt *time.Time `json:"batch_ends_at"`
	Nodes       []NodeView `json:"nodes,omitempty"`
}

CampaignView is a campaign as the API and the command line show it.

type Exclude

type Exclude struct {
	Nodes []string `json:"nodes"`
	Tags  []string `json:"tags"`
}

Exclude names nodes (proxy-<id>, forward-<id>) and node tags a campaign leaves out.

type NodeView

type NodeView struct {
	model.AgentUpgradeNode
	Node string `json:"node"`
}

NodeView is one node of a campaign.

type Service

type Service struct {
	DB *gorm.DB
	// Now is the clock; time.Now when nil.
	Now func() time.Time
}

Service is the campaign store and its rules.

func (*Service) Abort

func (s *Service) Abort(ctx context.Context, id string, actor Actor, rollback bool) (model.AgentUpgradeCampaign, error)

Abort stops a campaign. Without rollback it ends at once (aborted): nodes already offered finish their upgrade on their own. With rollback the current batch is rolled back first, as after a failure.

func (*Service) Active

func (s *Service) Active(ctx context.Context) (CampaignView, bool, error)

Active returns the active campaign, if any.

func (*Service) Get

func (s *Service) Get(ctx context.Context, id string) (CampaignView, error)

Get returns one campaign with its nodes in canary order.

func (*Service) List

func (s *Service) List(ctx context.Context, limit int) ([]CampaignView, error)

List returns the latest campaigns, newest first, without their nodes.

func (*Service) Pause

func (s *Service) Pause(ctx context.Context, id string, actor Actor) (model.AgentUpgradeCampaign, error)

Pause stops offering the upgrade and starting batches. Nodes already offered are still evaluated, and a failing batch still rolls back.

func (*Service) Resume

func (s *Service) Resume(ctx context.Context, id string, actor Actor) (model.AgentUpgradeCampaign, error)

Resume continues a paused campaign. The time the current batch was paused does not count towards its minimum duration.

func (*Service) Start

Start validates and records a campaign with its nodes and their batches. Only one campaign is active at a time.

type StartRequest

type StartRequest struct {
	TargetVersion string
	// ControlVersion is the running Control's release (H25: the Agent
	// release is Control's); a newer target is refused.
	ControlVersion string
	// Artifacts are the verified release assets
	// (agentinstall.UpgradeArtifacts).
	Artifacts []agentcontrol.UpgradeArtifact
	// Batches default to DefaultBatches.
	Batches []Batch
	Exclude Exclude
	Reason  string
	Actor   Actor
}

StartRequest starts a campaign.

type Worker

type Worker struct {
	Service *Service
	// Streams returns the Agent Control streams; a nil result skips the
	// evaluation.
	Streams func() agentstreams.Streams
	// Interval defaults to DefaultInterval.
	Interval time.Duration
}

Worker drives the active campaign. It runs in the singleton-worker process, which holds the Agent Control streams it dispatches on.

func (*Worker) HandleObserved

func (w *Worker) HandleObserved(node agentcontrol.AgentNode, observed *agentv1pb.ObservedState)

HandleObserved records the progress an Agent reports for an agent.upgrade operation (registered with the streams by Run).

func (*Worker) Run

func (w *Worker) Run(ctx context.Context)

Run evaluates the active campaign every Interval until ctx ends. It registers HandleObserved with the streams once per process.

func (*Worker) Tick

func (w *Worker) Tick(ctx context.Context) error

Tick evaluates the active campaign once.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL