Documentation
¶
Overview ¶
Package deploysafety decides whether a new release may take traffic and whether a live one must be rolled back, and holds the shared reason strings.
Index ¶
- Constants
- Variables
- func CrashLoopMessage(container string, restarts int, wait time.Duration) string
- func LatencyRegressed(o WatchObservation, cfg Effective) (bool, string)
- func ParseCrashLoop(msg string) (restarts int, wait time.Duration, ok bool)
- func Stage(events []store.DeployGateEvent) string
- func StageFailed(stage string) bool
- func StageTerminal(stage string) bool
- func ValidWatchMode(m string) bool
- func Validate(s store.DeploySafetySettings) error
- type Effective
- type ExitObservation
- type ExitVerdict
- type GateObservation
- type Verdict
- type WatchObservation
- type WatchStatus
- type Windows
- type Workload
Constants ¶
const ( ExitPhaseReadiness = "readiness" ExitPhaseWindow = "window" )
Exit phases, the point in a release's life a container stopped.
const ( PhaseReadiness = "readiness" PhaseStability = "stability" PhaseCutover = "cutover" PhaseDrain = "drain" PhaseWatch = "watch" PhaseRollback = "rollback" PhasePin = "pin" )
Phases of a deploy's gate timeline.
const ( StatusStarted = "started" StatusPassed = "passed" StatusFailed = "failed" StatusInfo = "info" // StatusStopped marks a watch an operator ended early. StatusStopped = "stopped" )
Statuses a gate event can carry.
const ( WatchModeOff = "off" WatchModeNotify = "notify" WatchModeAuto = "auto" )
Watch modes: off skips the watch, notify alerts on a trip, auto also rolls back.
const ( ReasonReadinessFailed = "ReadinessFailed" ReasonOOMKilledDuringReadiness = "OOMKilledDuringReadiness" ReasonExitedDuringReadiness = "ExitedDuringReadiness" ReasonOOMKilledDuringWindow = "OOMKilledDuringStableWindow" ReasonExitedDuringWindow = "ExitedDuringStableWindow" ReasonRestartedDuringWindow = "RestartedDuringStableWindow" ReasonGateTimeout = "StabilityGateTimeout" ReasonWatchCrashloop = "PostDeployCrashloop" ReasonWatchOOMKilled = "PostDeployOOMKilled" ReasonWatchExited = "PostDeployExited" ReasonWatchErrorRate = "PostDeployErrorRate" ReasonWatchLatency = "PostDeployLatencyRegression" ReasonWatchStopped = "WatchStopped" ReasonAutoRolledBack = "AutoRolledBack" ReasonNoRollbackTarget = "NoRollbackTarget" ReasonOperatorRollback = "OperatorRollback" )
Reasons a deploy fails its gate or a live release is rolled back. These land in deploy_attempts.reason, gate events and reconcile conditions.
const ( EnvSuccessThreshold = "APP_DEPLOY_SUCCESS_THRESHOLD" EnvStableWindow = "APP_DEPLOY_STABLE_WINDOW" EnvDrainTimeout = "APP_DEPLOY_DRAIN_TIMEOUT" EnvWatchWindow = "APP_DEPLOY_WATCH_WINDOW" EnvWatchMaxRestarts = "APP_DEPLOY_WATCH_MAX_RESTARTS" EnvWatchErrorRate = "APP_DEPLOY_WATCH_ERROR_RATE" EnvWatchMinRequests = "APP_DEPLOY_WATCH_MIN_REQUESTS" EnvAutoRollback = "APP_DEPLOY_AUTO_ROLLBACK" EnvWatchMode = "APP_DEPLOY_WATCH_MODE" EnvWatchLatency = "APP_DEPLOY_WATCH_LATENCY_RATIO" EnvWatchLatencyMin = "APP_DEPLOY_WATCH_LATENCY_MIN_DELTA" EnvWatchSoftTicks = "APP_DEPLOY_WATCH_SOFT_TICKS" EnvWatchSoftRoll = "APP_DEPLOY_WATCH_SOFT_ROLLBACK" EnvRollbackPins = "APP_DEPLOY_ROLLBACK_PINS" EnvWatchInterval = "APP_DEPLOY_WATCH_INTERVAL" )
Environment variables that set the platform-wide defaults.
const ( StagePending = "pending" StageGating = "gating" StageGateFailed = "gate_failed" StageWatching = "watching" StageCleared = "cleared" StageWatchFailed = "watch_failed" StageRolledBack = "rolled_back" StageWatchStopped = "watch_stopped" )
Stages a release moves through, derived from its gate timeline.
const ( WatchStateOff = "off" WatchStateWaiting = "waiting" WatchStateWatching = "watching" WatchStateCleared = "cleared" WatchStateTripped = "tripped" WatchStateRolledBack = "rolled_back" WatchStateStopped = "stopped" )
Watch states a release's post-deploy watch can be in.
const MaxOverrideWindow = 30 * time.Minute
MaxOverrideWindow caps per-app durations so one app cannot stall the sequential reconcile loop or hold a watch open indefinitely.
const ReasonCrashLoopBackOff = "CrashLoopBackOff"
ReasonCrashLoopBackOff is the Ready condition reason while a container that keeps exiting waits out its restart delay.
Variables ¶
var GateFailureReasons = []string{ ReasonReadinessFailed, ReasonOOMKilledDuringReadiness, ReasonExitedDuringReadiness, ReasonOOMKilledDuringWindow, ReasonExitedDuringWindow, ReasonRestartedDuringWindow, ReasonGateTimeout, }
GateFailureReasons is every reason that means a release never took traffic.
var WatchTripReasons = []string{ReasonWatchCrashloop, ReasonWatchOOMKilled, ReasonWatchExited, ReasonWatchErrorRate, ReasonWatchLatency}
WatchTripReasons is every reason the post-deploy watch can trip on.
Functions ¶
func CrashLoopMessage ¶
CrashLoopMessage is the condition message the controller writes; the CLI reads it back with ParseCrashLoop, so keep both in this file.
func LatencyRegressed ¶
func LatencyRegressed(o WatchObservation, cfg Effective) (bool, string)
LatencyRegressed reports whether the new release's p95 is a real step up from the baseline: both sides need MinRequests of traffic, and the rise must clear both the ratio and the absolute floor.
func ParseCrashLoop ¶
ParseCrashLoop extracts the restart count and the delay before the next restart from a CrashLoopMessage.
func Stage ¶
func Stage(events []store.DeployGateEvent) string
Stage folds a timeline (oldest first) into the stage the release is in now. A gate failure followed by a passing retry is gating again.
func StageFailed ¶
StageFailed reports whether stage means the release did not hold up.
func StageTerminal ¶
StageTerminal reports whether nothing more will happen to a release in stage without an operator acting.
func ValidWatchMode ¶
ValidWatchMode reports whether m is off, notify or auto.
func Validate ¶
func Validate(s store.DeploySafetySettings) error
Validate rejects overrides outside the ranges the database enforces.
Types ¶
type Effective ¶
type Effective struct {
// SuccessThreshold is how many consecutive readiness passes a new
// container needs before it may take traffic.
SuccessThreshold int `json:"success_threshold"`
// StableWindow is how long a new container must stay running, without
// a restart or OOM kill, before it may take traffic.
StableWindow time.Duration `json:"stable_window"`
// DrainTimeout is the minimum time the previous release keeps serving
// after cutover, so in-flight requests finish before it stops.
DrainTimeout time.Duration `json:"drain_timeout"`
WatchWindow time.Duration `json:"watch_window"`
WatchMaxRestarts int `json:"watch_max_restarts"`
WatchErrorRate float64 `json:"watch_error_rate"`
WatchMinRequests int `json:"watch_min_requests"`
// AutoRollback is derived from WatchMode (auto).
AutoRollback bool `json:"auto_rollback"`
// WatchMode is off, notify or auto.
WatchMode string `json:"watch_mode"`
// WatchLatencyRatio trips the soft latency signal when the new
// release's p95 reaches this multiple of the previous release's; 0
// turns the signal off.
WatchLatencyRatio float64 `json:"watch_latency_ratio"`
// WatchLatencyMinDelta is the absolute p95 increase also required, so
// a 4ms to 9ms move is never a regression.
WatchLatencyMinDelta time.Duration `json:"watch_latency_min_delta"`
// WatchSoftTicks is how many consecutive evaluations a soft signal must
// hold before it counts.
WatchSoftTicks int `json:"watch_soft_ticks"`
// WatchSoftRollback lets a soft signal roll back (mode auto only);
// otherwise it only notifies.
WatchSoftRollback bool `json:"watch_soft_rollback"`
// RollbackPins is how many past releases per app stay pinned.
RollbackPins int `json:"rollback_pins"`
WatchInterval time.Duration `json:"watch_interval"`
}
Effective is a fully resolved set of gate and watch settings.
func BuiltinDefaults ¶
func BuiltinDefaults() Effective
BuiltinDefaults are the defaults with no environment set.
func DefaultsFromEnv ¶
DefaultsFromEnv reads the APP_DEPLOY_* defaults over BuiltinDefaults. An invalid value keeps the builtin and is reported in warnings.
func Resolve ¶
func Resolve(defaults Effective, s store.DeploySafetySettings) Effective
Resolve layers an app's overrides onto the platform defaults.
func (Effective) WithWindows ¶
WithWindows returns e with the platform timings replaced, before any per-app override is resolved on top.
type ExitObservation ¶
type ExitObservation struct {
Running bool
OOMKilled bool
ExitCode int
StartedAt time.Time
FinishedAt time.Time
// ObservedAt stands in for FinishedAt when the transport cannot report it.
ObservedAt time.Time
}
ExitObservation is one container's observed run state.
type ExitVerdict ¶
type ExitVerdict struct {
// Exited is true once the container stopped running.
Exited bool
// Failed is true when the exit is a failure for this workload.
Failed bool
Reason string
Message string
// Ran is how long the container ran, zero when unknown.
Ran time.Duration
}
ExitVerdict is ClassifyExit's result.
func ClassifyExit ¶
func ClassifyExit(w Workload, phase string, obs ExitObservation) ExitVerdict
ClassifyExit decides whether obs is a failure for workload w. For a service it is any exit; phase picks the reason (readiness or window).
type GateObservation ¶
type GateObservation struct {
Running bool
OOMKilled bool
ExitCode int
// RestartCount and StartedAt are compared against the values seen
// when the gate began; a zero StartedAt means the runtime cannot tell.
RestartCount int
BaselineRestartCount int
StartedAt time.Time
BaselineStartedAt time.Time
ConsecutivePasses int
SinceStart time.Duration
BudgetExhausted bool
}
GateObservation is what the reconciler saw of a new container on one tick of its stability gate.
type Verdict ¶
type Verdict string
Verdict is the outcome of one evaluation of a gate or watch rule.
const ( VerdictPending Verdict = "pending" VerdictPass Verdict = "pass" VerdictFail Verdict = "fail" )
Verdicts.
func EvaluateGate ¶
func EvaluateGate(o GateObservation, cfg Effective) (Verdict, string)
EvaluateGate decides whether a new container may take traffic yet.
func EvaluateWatch ¶
func EvaluateWatch(o WatchObservation, cfg Effective) (Verdict, string, string)
EvaluateWatch decides whether a live release must be rolled back, has cleared its watch window, or is still being watched.
type WatchObservation ¶
type WatchObservation struct {
SinceCutover time.Duration
Restarts int
Exited bool
OOMKilled bool
Requests float64
Errors5xx float64
// P95Ms is the new release's p95 since cutover; BaselineP95Ms and
// BaselineRequests describe the release it replaced over the same span
// before cutover. A zero baseline means there is nothing to compare to.
P95Ms float64
BaselineP95Ms float64
BaselineRequests float64
}
WatchObservation is what the post-deploy watch saw of a live release since its cutover.
type WatchStatus ¶
type WatchStatus struct {
State string
Mode string
StartedAt time.Time
EndsAt time.Time
RemainingSecs int
WindowSecs int
// Reason and Message describe the trip or the rollback; Notice carries a
// soft signal that only alerted.
Reason string
Message string
Notice string
}
WatchStatus is the operator-facing summary of one release's watch.
func WatchStatusOf ¶
func WatchStatusOf(events []store.DeployGateEvent, cfg Effective, now time.Time) WatchStatus
WatchStatusOf folds a release's gate timeline (oldest first) into its watch status at now.
type Windows ¶
Windows are the platform-wide deploy timings an operator can change at runtime; the hold lives beside them because it is read with the drain.