Documentation
¶
Overview ¶
Package agentupgrade runs staged Agent upgrades (forward-sdk.md section 9, O4; owner decision H19): Control, never the node, decides when an Agent upgrades. A campaign pushes one signed Agent release to the enrolled nodes in batches (5% → 25% → 100% by default, at least 30 minutes each), canaries first in a deterministic order, as the agent.upgrade operation on the Agent Control stream of every Agent that negotiated upgrade.v1. A node succeeds when its Agent reconnects with the target version (and its configuration still applies); a batch in which more than 5% of the offered nodes fail — refused, failed to apply, or did not reconnect with the new version within 10 minutes — is rolled back: the campaign stops and the upgraded nodes of that batch are told to reinstate their previous release.
The campaign lives in two protected tables (v4_kernel_agent_upgrade_campaign, v4_kernel_agent_upgrade_node); the Worker drives it in the singleton-worker process, which holds the Agent streams.
Index ¶
- Constants
- Variables
- func BatchBounds(total int, batches []Batch) []int
- func Candidates(ctx context.Context, db *gorm.DB, exclude Exclude) ([]agentcontrol.AgentNode, error)
- func OrderKey(node agentcontrol.AgentNode) string
- func ValidateBatches(batches []Batch) error
- type Actor
- type Batch
- type BatchView
- type CampaignView
- type Exclude
- type NodeView
- type Service
- func (s *Service) Abort(ctx context.Context, id string, actor Actor, rollback bool) (model.AgentUpgradeCampaign, error)
- func (s *Service) Active(ctx context.Context) (CampaignView, bool, error)
- func (s *Service) Get(ctx context.Context, id string) (CampaignView, error)
- func (s *Service) List(ctx context.Context, limit int) ([]CampaignView, error)
- func (s *Service) Pause(ctx context.Context, id string, actor Actor) (model.AgentUpgradeCampaign, error)
- func (s *Service) Resume(ctx context.Context, id string, actor Actor) (model.AgentUpgradeCampaign, error)
- func (s *Service) Start(ctx context.Context, request StartRequest) (model.AgentUpgradeCampaign, error)
- type StartRequest
- type Worker
Constants ¶
const ( // FailureThresholdPercent: a batch rolls back when more than this share // of its offered nodes fail. FailureThresholdPercent = 5 // ReconnectTimeout: an offered node must reconnect with the target // version within this time. ReconnectTimeout = 10 * time.Minute // MinBatchDuration is the shortest a batch may last. MinBatchDuration = 30 * time.Minute // FirstBatchMaxPercent caps the canary batch. FirstBatchMaxPercent = 5 // MaxBatches bounds a campaign's batches. MaxBatches = 10 )
The H19 rules.
const ( CodeBatchFailed = "batch_failed" CodeAborted = "aborted" CodeRejected = "rejected" CodeApplyFailed = "apply_failed" CodeReconnectTimeout = "reconnect_timeout" CodeReverted = "reverted" CodeConfigFailed = "config_apply_failed" CodeUnsupported = "upgrade_unsupported" CodeOffline = "node_offline" CodeOfferFailed = "offer_failed" CodeRollbackFailed = "rollback_failed" CodeRollbackUnconfirm = "rollback_unconfirmed" )
Error codes recorded on campaigns and nodes.
const ( AuditActionStart = "agent_upgrade_start" AuditActionPause = "agent_upgrade_pause" AuditActionResume = "agent_upgrade_resume" AuditActionAbort = "agent_upgrade_abort" AuditActionRollback = "agent_upgrade_rollback" AuditActionFinish = "agent_upgrade_finish" )
Audit actions (v2_operation_log, module agent_upgrade).
const ( // DefaultInterval is how often the worker evaluates the campaign. DefaultInterval = 5 * time.Second // ConfigGrace is how long a node that reconnected with the target // version and negotiated config.v1 has to report a failed // ConfigStatus before it counts as succeeded (it keeps being watched // for the rest of its batch). ConfigGrace = time.Minute )
Worker timing.
Variables ¶
var ( ErrInvalid = errors.New("invalid agent upgrade campaign") ErrActiveCampaign = errors.New("another agent upgrade campaign is active") ErrNoNodes = errors.New("no node can be upgraded") ErrNotFound = errors.New("agent upgrade campaign not found") ErrState = errors.New("the campaign is not in a state that allows this") )
Errors of the campaign API; handlers map them to answers.
Functions ¶
func BatchBounds ¶
BatchBounds splits total nodes, ordered by OrderKey, into the batches: batch i takes the nodes from bounds[i] to bounds[i+1]. A cumulative share rounds up, so the canary batch has at least one node.
func Candidates ¶
func Candidates(ctx context.Context, db *gorm.DB, exclude Exclude) ([]agentcontrol.AgentNode, error)
Candidates lists the nodes a campaign includes: enabled proxy and forward nodes whose Agent was seen on the Agent Control stream (v4_kernel_agent_transport, mtls-stream or apikey-stream), minus the excluded nodes and tags, in canary order.
func OrderKey ¶
func OrderKey(node agentcontrol.AgentNode) string
OrderKey is a node's place in the canary order: the hex SHA-256 of its name, so the same nodes are the canaries of every campaign.
func ValidateBatches ¶
ValidateBatches checks batches against H19: 1 to MaxBatches, cumulative percentages strictly increasing and ending at 100, the first at most FirstBatchMaxPercent, each at least MinBatchDuration.
Types ¶
type Actor ¶
Actor is who changes a campaign: an administrator (UserID, Name, IP) or the command line (UserID 0).
type Batch ¶
type Batch struct {
Percent int `json:"percent"`
MinDurationSeconds int `json:"min_duration_seconds"`
}
Batch is one step of a campaign: the nodes up to Percent of the campaign's nodes (cumulative), which run for at least MinDurationSeconds before the next batch starts.
func DefaultBatches ¶
func DefaultBatches() []Batch
DefaultBatches are H19's batches: 5%, 25% and 100% of the nodes (cumulative), 30 minutes each.
type BatchView ¶
type BatchView struct {
Index int `json:"index"`
Percent int `json:"percent"`
MinDurationSeconds int `json:"min_duration_seconds"`
Nodes int `json:"nodes"`
States map[string]int `json:"states"`
// Offered and Failed are the batch's failure ratio's terms (H19):
// skipped and pending nodes do not count.
Offered int `json:"offered"`
Failed int `json:"failed"`
}
BatchView is one batch of a campaign with its nodes counted by state.
type CampaignView ¶
type CampaignView struct {
model.AgentUpgradeCampaign
Artifacts []agentcontrol.UpgradeArtifact `json:"artifacts"`
Batches []BatchView `json:"batches"`
Exclude Exclude `json:"exclude"`
Total int `json:"total"`
States map[string]int `json:"states"`
// BatchEndsAt is the earliest the current batch can pass.
BatchEndsAt *time.Time `json:"batch_ends_at"`
Nodes []NodeView `json:"nodes,omitempty"`
}
CampaignView is a campaign as the API and the command line show it.
type NodeView ¶
type NodeView struct {
model.AgentUpgradeNode
Node string `json:"node"`
}
NodeView is one node of a campaign.
type Service ¶
Service is the campaign store and its rules.
func (*Service) Abort ¶
func (s *Service) Abort(ctx context.Context, id string, actor Actor, rollback bool) (model.AgentUpgradeCampaign, error)
Abort stops a campaign. Without rollback it ends at once (aborted): nodes already offered finish their upgrade on their own. With rollback the current batch is rolled back first, as after a failure.
func (*Service) Pause ¶
func (s *Service) Pause(ctx context.Context, id string, actor Actor) (model.AgentUpgradeCampaign, error)
Pause stops offering the upgrade and starting batches. Nodes already offered are still evaluated, and a failing batch still rolls back.
func (*Service) Resume ¶
func (s *Service) Resume(ctx context.Context, id string, actor Actor) (model.AgentUpgradeCampaign, error)
Resume continues a paused campaign. The time the current batch was paused does not count towards its minimum duration.
func (*Service) Start ¶
func (s *Service) Start(ctx context.Context, request StartRequest) (model.AgentUpgradeCampaign, error)
Start validates and records a campaign with its nodes and their batches. Only one campaign is active at a time.
type StartRequest ¶
type StartRequest struct {
TargetVersion string
// ControlVersion is the running Control's release (H25: the Agent
// release is Control's); a newer target is refused.
ControlVersion string
// Artifacts are the verified release assets
// (agentinstall.UpgradeArtifacts).
Artifacts []agentcontrol.UpgradeArtifact
// Batches default to DefaultBatches.
Batches []Batch
Exclude Exclude
Reason string
Actor Actor
}
StartRequest starts a campaign.
type Worker ¶
type Worker struct {
Service *Service
// Streams returns the Agent Control streams; a nil result skips the
// evaluation.
Streams func() agentstreams.Streams
// Interval defaults to DefaultInterval.
Interval time.Duration
}
Worker drives the active campaign. It runs in the singleton-worker process, which holds the Agent Control streams it dispatches on.
func (*Worker) HandleObserved ¶
func (w *Worker) HandleObserved(node agentcontrol.AgentNode, observed *agentv1pb.ObservedState)
HandleObserved records the progress an Agent reports for an agent.upgrade operation (registered with the streams by Run).