deploysafety

package
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 2, 2026 License: Apache-2.0 Imports: 6 Imported by: 0

Documentation

Overview

Package deploysafety decides whether a new release may take traffic and whether a live one must be rolled back, and holds the shared reason strings.

Index

Constants

View Source
const (
	ExitPhaseReadiness = "readiness"
	ExitPhaseWindow    = "window"
)

Exit phases, the point in a release's life a container stopped.

View Source
const (
	PhaseReadiness = "readiness"
	PhaseStability = "stability"
	PhaseCutover   = "cutover"
	PhaseDrain     = "drain"
	PhaseWatch     = "watch"
	PhaseRollback  = "rollback"
	PhasePin       = "pin"
)

Phases of a deploy's gate timeline.

View Source
const (
	StatusStarted = "started"
	StatusPassed  = "passed"
	StatusFailed  = "failed"
	StatusInfo    = "info"
	// StatusStopped marks a watch an operator ended early.
	StatusStopped = "stopped"
)

Statuses a gate event can carry.

View Source
const (
	WatchModeOff    = "off"
	WatchModeNotify = "notify"
	WatchModeAuto   = "auto"
)

Watch modes: off skips the watch, notify alerts on a trip, auto also rolls back.

View Source
const (
	ReasonReadinessFailed          = "ReadinessFailed"
	ReasonOOMKilledDuringReadiness = "OOMKilledDuringReadiness"
	ReasonExitedDuringReadiness    = "ExitedDuringReadiness"
	ReasonOOMKilledDuringWindow    = "OOMKilledDuringStableWindow"
	ReasonExitedDuringWindow       = "ExitedDuringStableWindow"
	ReasonRestartedDuringWindow    = "RestartedDuringStableWindow"
	ReasonGateTimeout              = "StabilityGateTimeout"
	ReasonWatchCrashloop           = "PostDeployCrashloop"
	ReasonWatchOOMKilled           = "PostDeployOOMKilled"
	ReasonWatchExited              = "PostDeployExited"
	ReasonWatchErrorRate           = "PostDeployErrorRate"
	ReasonWatchLatency             = "PostDeployLatencyRegression"
	ReasonWatchStopped             = "WatchStopped"
	ReasonAutoRolledBack           = "AutoRolledBack"
	ReasonNoRollbackTarget         = "NoRollbackTarget"
	ReasonOperatorRollback         = "OperatorRollback"
)

Reasons a deploy fails its gate or a live release is rolled back. These land in deploy_attempts.reason, gate events and reconcile conditions.

View Source
const (
	EnvSuccessThreshold = "APP_DEPLOY_SUCCESS_THRESHOLD"
	EnvStableWindow     = "APP_DEPLOY_STABLE_WINDOW"
	EnvDrainTimeout     = "APP_DEPLOY_DRAIN_TIMEOUT"
	EnvWatchWindow      = "APP_DEPLOY_WATCH_WINDOW"
	EnvWatchMaxRestarts = "APP_DEPLOY_WATCH_MAX_RESTARTS"
	EnvWatchErrorRate   = "APP_DEPLOY_WATCH_ERROR_RATE"
	EnvWatchMinRequests = "APP_DEPLOY_WATCH_MIN_REQUESTS"
	EnvAutoRollback     = "APP_DEPLOY_AUTO_ROLLBACK"
	EnvWatchMode        = "APP_DEPLOY_WATCH_MODE"
	EnvWatchLatency     = "APP_DEPLOY_WATCH_LATENCY_RATIO"
	EnvWatchLatencyMin  = "APP_DEPLOY_WATCH_LATENCY_MIN_DELTA"
	EnvWatchSoftTicks   = "APP_DEPLOY_WATCH_SOFT_TICKS"
	EnvWatchSoftRoll    = "APP_DEPLOY_WATCH_SOFT_ROLLBACK"
	EnvRollbackPins     = "APP_DEPLOY_ROLLBACK_PINS"
	EnvWatchInterval    = "APP_DEPLOY_WATCH_INTERVAL"
)

Environment variables that set the platform-wide defaults.

View Source
const (
	StagePending      = "pending"
	StageGating       = "gating"
	StageGateFailed   = "gate_failed"
	StageWatching     = "watching"
	StageCleared      = "cleared"
	StageWatchFailed  = "watch_failed"
	StageRolledBack   = "rolled_back"
	StageWatchStopped = "watch_stopped"
)

Stages a release moves through, derived from its gate timeline.

View Source
const (
	WatchStateOff        = "off"
	WatchStateWaiting    = "waiting"
	WatchStateWatching   = "watching"
	WatchStateCleared    = "cleared"
	WatchStateTripped    = "tripped"
	WatchStateRolledBack = "rolled_back"
	WatchStateStopped    = "stopped"
)

Watch states a release's post-deploy watch can be in.

View Source
const MaxOverrideWindow = 30 * time.Minute

MaxOverrideWindow caps per-app durations so one app cannot stall the sequential reconcile loop or hold a watch open indefinitely.

View Source
const ReasonCrashLoopBackOff = "CrashLoopBackOff"

ReasonCrashLoopBackOff is the Ready condition reason while a container that keeps exiting waits out its restart delay.

Variables

GateFailureReasons is every reason that means a release never took traffic.

WatchTripReasons is every reason the post-deploy watch can trip on.

Functions

func CrashLoopMessage

func CrashLoopMessage(container string, restarts int, wait time.Duration) string

CrashLoopMessage is the condition message the controller writes; the CLI reads it back with ParseCrashLoop, so keep both in this file.

func LatencyRegressed

func LatencyRegressed(o WatchObservation, cfg Effective) (bool, string)

LatencyRegressed reports whether the new release's p95 is a real step up from the baseline: both sides need MinRequests of traffic, and the rise must clear both the ratio and the absolute floor.

func ParseCrashLoop

func ParseCrashLoop(msg string) (restarts int, wait time.Duration, ok bool)

ParseCrashLoop extracts the restart count and the delay before the next restart from a CrashLoopMessage.

func Stage

func Stage(events []store.DeployGateEvent) string

Stage folds a timeline (oldest first) into the stage the release is in now. A gate failure followed by a passing retry is gating again.

func StageFailed

func StageFailed(stage string) bool

StageFailed reports whether stage means the release did not hold up.

func StageTerminal

func StageTerminal(stage string) bool

StageTerminal reports whether nothing more will happen to a release in stage without an operator acting.

func ValidWatchMode

func ValidWatchMode(m string) bool

ValidWatchMode reports whether m is off, notify or auto.

func Validate

func Validate(s store.DeploySafetySettings) error

Validate rejects overrides outside the ranges the database enforces.

Types

type Effective

type Effective struct {
	// SuccessThreshold is how many consecutive readiness passes a new
	// container needs before it may take traffic.
	SuccessThreshold int `json:"success_threshold"`
	// StableWindow is how long a new container must stay running, without
	// a restart or OOM kill, before it may take traffic.
	StableWindow time.Duration `json:"stable_window"`
	// DrainTimeout is the minimum time the previous release keeps serving
	// after cutover, so in-flight requests finish before it stops.
	DrainTimeout     time.Duration `json:"drain_timeout"`
	WatchWindow      time.Duration `json:"watch_window"`
	WatchMaxRestarts int           `json:"watch_max_restarts"`
	WatchErrorRate   float64       `json:"watch_error_rate"`
	WatchMinRequests int           `json:"watch_min_requests"`
	// AutoRollback is derived from WatchMode (auto).
	AutoRollback bool `json:"auto_rollback"`
	// WatchMode is off, notify or auto.
	WatchMode string `json:"watch_mode"`
	// WatchLatencyRatio trips the soft latency signal when the new
	// release's p95 reaches this multiple of the previous release's; 0
	// turns the signal off.
	WatchLatencyRatio float64 `json:"watch_latency_ratio"`
	// WatchLatencyMinDelta is the absolute p95 increase also required, so
	// a 4ms to 9ms move is never a regression.
	WatchLatencyMinDelta time.Duration `json:"watch_latency_min_delta"`
	// WatchSoftTicks is how many consecutive evaluations a soft signal must
	// hold before it counts.
	WatchSoftTicks int `json:"watch_soft_ticks"`
	// WatchSoftRollback lets a soft signal roll back (mode auto only);
	// otherwise it only notifies.
	WatchSoftRollback bool `json:"watch_soft_rollback"`
	// RollbackPins is how many past releases per app stay pinned.
	RollbackPins  int           `json:"rollback_pins"`
	WatchInterval time.Duration `json:"watch_interval"`
}

Effective is a fully resolved set of gate and watch settings.

func BuiltinDefaults

func BuiltinDefaults() Effective

BuiltinDefaults are the defaults with no environment set.

func DefaultsFromEnv

func DefaultsFromEnv(lookup func(string) (string, bool)) (Effective, []error)

DefaultsFromEnv reads the APP_DEPLOY_* defaults over BuiltinDefaults. An invalid value keeps the builtin and is reported in warnings.

func Resolve

func Resolve(defaults Effective, s store.DeploySafetySettings) Effective

Resolve layers an app's overrides onto the platform defaults.

func (Effective) WithWindows

func (e Effective) WithWindows(w Windows) Effective

WithWindows returns e with the platform timings replaced, before any per-app override is resolved on top.

type ExitObservation

type ExitObservation struct {
	Running    bool
	OOMKilled  bool
	ExitCode   int
	StartedAt  time.Time
	FinishedAt time.Time
	// ObservedAt stands in for FinishedAt when the transport cannot report it.
	ObservedAt time.Time
}

ExitObservation is one container's observed run state.

type ExitVerdict

type ExitVerdict struct {
	// Exited is true once the container stopped running.
	Exited bool
	// Failed is true when the exit is a failure for this workload.
	Failed  bool
	Reason  string
	Message string
	// Ran is how long the container ran, zero when unknown.
	Ran time.Duration
}

ExitVerdict is ClassifyExit's result.

func ClassifyExit

func ClassifyExit(w Workload, phase string, obs ExitObservation) ExitVerdict

ClassifyExit decides whether obs is a failure for workload w. For a service it is any exit; phase picks the reason (readiness or window).

type GateObservation

type GateObservation struct {
	Running   bool
	OOMKilled bool
	ExitCode  int
	// RestartCount and StartedAt are compared against the values seen
	// when the gate began; a zero StartedAt means the runtime cannot tell.
	RestartCount         int
	BaselineRestartCount int
	StartedAt            time.Time
	BaselineStartedAt    time.Time
	ConsecutivePasses    int
	SinceStart           time.Duration
	BudgetExhausted      bool
}

GateObservation is what the reconciler saw of a new container on one tick of its stability gate.

type Verdict

type Verdict string

Verdict is the outcome of one evaluation of a gate or watch rule.

const (
	VerdictPending Verdict = "pending"
	VerdictPass    Verdict = "pass"
	VerdictFail    Verdict = "fail"
)

Verdicts.

func EvaluateGate

func EvaluateGate(o GateObservation, cfg Effective) (Verdict, string)

EvaluateGate decides whether a new container may take traffic yet.

func EvaluateWatch

func EvaluateWatch(o WatchObservation, cfg Effective) (Verdict, string, string)

EvaluateWatch decides whether a live release must be rolled back, has cleared its watch window, or is still being watched.

type WatchObservation

type WatchObservation struct {
	SinceCutover time.Duration
	Restarts     int
	Exited       bool
	OOMKilled    bool
	Requests     float64
	Errors5xx    float64
	// P95Ms is the new release's p95 since cutover; BaselineP95Ms and
	// BaselineRequests describe the release it replaced over the same span
	// before cutover. A zero baseline means there is nothing to compare to.
	P95Ms            float64
	BaselineP95Ms    float64
	BaselineRequests float64
}

WatchObservation is what the post-deploy watch saw of a live release since its cutover.

type WatchStatus

type WatchStatus struct {
	State         string
	Mode          string
	StartedAt     time.Time
	EndsAt        time.Time
	RemainingSecs int
	WindowSecs    int
	// Reason and Message describe the trip or the rollback; Notice carries a
	// soft signal that only alerted.
	Reason  string
	Message string
	Notice  string
}

WatchStatus is the operator-facing summary of one release's watch.

func WatchStatusOf

func WatchStatusOf(events []store.DeployGateEvent, cfg Effective, now time.Time) WatchStatus

WatchStatusOf folds a release's gate timeline (oldest first) into its watch status at now.

type Windows

type Windows struct {
	Drain, Stable, Watch, Hold time.Duration
}

Windows are the platform-wide deploy timings an operator can change at runtime; the hold lives beside them because it is read with the drain.

type Workload

type Workload string

Workload says what a stopped container was supposed to do.

const (
	// WorkloadService is a long-running app container: any exit, code 0
	// included, means it stopped serving.
	WorkloadService Workload = "service"
	// WorkloadJob is a one-shot container (a scheduled task or job run):
	// exit 0 is success.
	WorkloadJob Workload = "job"
)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL