Documentation
¶
Overview ¶
Package operation recovers server mutations that were interrupted by a panel crash or restart. A disconnected Docker request has an unknown outcome: the operation journal keeps the server locked until startup has observed the container, instead of allowing a blind retry.
Index ¶
- Constants
- Variables
- func ClientRequestID(op *db.ServerOperation) string
- func CountByOutcome(views []View) map[Outcome]int
- func Digest(kind string, server *db.Server) string
- func RedactError(text string) string
- func RetryReview(ctx context.Context, store ReviewStore, observer Observer, log *logger.Logger, ...)
- func RunJournaled(ctx context.Context, store Journal, log *logger.Logger, server *db.Server, ...) error
- func StorageKey(requestID, kind string) string
- func TrimRequestKey(key string) string
- func ValidOutcome(name string) bool
- func ValidateRequestID(id string) error
- type Journal
- type NextAction
- type Observed
- type Observer
- type Outcome
- type ReviewStore
- type SaveResumer
- type Store
- type Summary
- type View
Constants ¶
const ExecutionStaleAfter = 2 * time.Minute
ExecutionStaleAfter is how long an executing operation may stay unchanged before it stops being "in flight" and becomes an unknown outcome. A mutation that has not recorded a completion by then may still be running, but the panel can no longer tell, and an operator needs the escape hatch.
Variables ¶
var ErrAlreadyFailed = errors.New("matching lifecycle operation already failed")
ErrAlreadyFailed is returned when the same request id already failed. A new request id is required to try again.
var ErrInProgress = errors.New("matching lifecycle operation is already in progress")
ErrInProgress is returned when the same request id is already running.
Functions ¶
func ClientRequestID ¶ added in v1.0.40
func ClientRequestID(op *db.ServerOperation) string
ClientRequestID is the caller-supplied request id stored on a journal row. StorageKey concatenates that id with the kind; a generated key (empty request id) is returned as-is so an operator can still quote it.
func CountByOutcome ¶ added in v1.0.40
CountByOutcome counts the projected operations by outcome, which is what a bundle header and a health line want.
func RedactError ¶ added in v1.0.40
RedactError removes credentials from daemon and client error text before it is written to a diagnostic. Hosts, paths and messages are kept: they are the evidence an operator needs.
func RetryReview ¶
func RetryReview(ctx context.Context, store ReviewStore, observer Observer, log *logger.Logger, ids []string, every time.Duration)
RetryReview re-observes operations whose container state was unknown at startup. It stops when every id has been released or ctx is cancelled. Operations that are not in the list are left alone, so an in-flight mutation is never finalized by this loop.
func RunJournaled ¶
func RunJournaled(ctx context.Context, store Journal, log *logger.Logger, server *db.Server, kind, requestID string, mutate func() error) error
RunJournaled persists the intent of a server mutation before it is attempted, advances the journal through the attempt, and finalizes it. requestID is a client idempotency key: a repeat of a completed key does not run mutate again, and a repeat of a key that is still running returns ErrInProgress. An empty requestID is unique to this call.
If the process dies between the external effect and finalization, the entry stays active and startup recovery observes the container before releasing the per-server lock (see RecoverInterrupted). The active lock is the shared coordinator: no two lifecycle mutations of one world can interleave. A mutation error finalizes as failed so the user can retry without restarting the panel.
func StorageKey ¶
StorageKey is the journal key for a client request id and mutation kind. An empty request id is unique to this call, so it is not idempotent.
func TrimRequestKey ¶ added in v1.0.40
TrimRequestKey shortens a client request key for display without changing the value the journal matches on.
func ValidOutcome ¶ added in v1.0.40
ValidOutcome reports whether name is one this package produces. It exists so an RPC filter cannot silently match nothing because of a typo.
func ValidateRequestID ¶
ValidateRequestID rejects client request ids that do not fit the journal.
Types ¶
type Journal ¶
type Journal interface {
Store
BeginServerOperation(ctx context.Context, serverID, nodeID, containerID, key, kind, digest string) (*db.ServerOperation, bool, error)
}
Journal is the operation journal subset that RunJournaled needs. *db.Store satisfies it.
type NextAction ¶ added in v1.0.40
type NextAction string
NextAction is the safe next step for an operation in its current state. It is a stable code so a UI, a runbook or a support engineer can agree on the remedy without parsing an error string.
const ( // ActionNone means nothing is required. ActionNone NextAction = "none" // ActionWait means the operation is still in flight; wait for it. ActionWait NextAction = "wait" // ActionResolve means the process that owned the operation is gone. Startup // recovery observes the container before releasing the lock; an operator can // also resolve it explicitly. ActionResolve NextAction = "resolve_operation" // ActionRetryNewRequestID means the recorded attempt failed and the same // request id is refused, so a retry needs a new one. ActionRetryNewRequestID NextAction = "retry_with_new_request_id" // ActionContactOperator means the state is outside the automatic paths. ActionContactOperator NextAction = "contact_operator" )
type Observer ¶
type Observer interface {
Observe(ctx context.Context, serverID, containerID string) (Observed, error)
}
Observer resolves the current container state for a server. Implementations must return an error when the state cannot be established (for example the node is unreachable); recovery treats that as "leave the lock in place".
func NewDockerObserver ¶
NewDockerObserver adapts a Docker client to recovery's observation interface.
type Outcome ¶ added in v1.0.40
type Outcome string
Outcome is what an operator should understand an operation to have done. Unknown is a first-class answer: a mutation whose external effect was not recorded is neither a success nor a failure.
const ( // OutcomeSuccess means the mutation completed and was recorded. OutcomeSuccess Outcome = "success" // OutcomeFailed means the mutation returned an error and was recorded. OutcomeFailed Outcome = "failed" // OutcomeRunning means the mutation is still being attempted. It is the // only outcome that may still turn into a success or a failure on its own. OutcomeRunning Outcome = "running" // OutcomeUnknown means the intent was recorded but no completion or // failure was. The external effect may or may not have happened. OutcomeUnknown Outcome = "unknown" )
type ReviewStore ¶
type ReviewStore interface {
GetServerOperation(ctx context.Context, id string) (*db.ServerOperation, error)
AdvanceServerOperation(ctx context.Context, id, from, to, detail string) error
}
ReviewStore is the journal subset used to retry locks that could not be observed at startup. *db.Store satisfies it.
type SaveResumer ¶ added in v1.0.40
SaveResumer turns world saves back on after an interrupted backup. save-off does not survive a stopped JVM, so callers send it only while the container is still running.
type Store ¶
type Store interface {
ListActiveServerOperations(ctx context.Context) ([]db.ServerOperation, error)
AdvanceServerOperation(ctx context.Context, id, from, to, detail string) error
NoteActiveServerOperation(ctx context.Context, id, detail string) error
}
Store is the journal subset that recovery needs. *db.Store satisfies it.
type Summary ¶
type Summary struct {
Scanned int
Recovered int
NeedsReview int
// ReviewIDs are the active operations whose container state could not be
// established. A later pass may observe them once the daemon is reachable.
ReviewIDs []string
}
Summary reports what startup recovery did.
func RecoverInterrupted ¶
func RecoverInterrupted(ctx context.Context, store Store, observer Observer, log *logger.Logger) (Summary, error)
RecoverInterrupted observes every operation that still owns a server lock and finalizes it when the container state is known. Operations whose state cannot be observed keep their lock and are reported as requiring review; their server stays blocked until an operator or a later startup resolves them.
func ResumeInterruptedBackups ¶ added in v1.0.40
func ResumeInterruptedBackups(ctx context.Context, store Store, observer Observer, saves SaveResumer, log *logger.Logger) (Summary, error)
ResumeInterruptedBackups sends save-on for an interrupted backup whose container is still running, then releases that lock. A failed save-on leaves the lock in place. A stopped container is released without a command, because the next boot starts with saves enabled and this pass must not start the server.
type View ¶ added in v1.0.40
type View struct {
ID string `json:"id"`
ServerID string `json:"serverId"`
Kind string `json:"kind"`
Phase string `json:"phase"`
Outcome Outcome `json:"outcome"`
NextAction NextAction `json:"nextAction"`
Reason string `json:"reason"`
EndpointGeneration int64 `json:"endpointGeneration"`
HasContainer bool `json:"hasContainer"`
HoldsLock bool `json:"holdsLock"`
LastError string `json:"lastError,omitempty"`
StartedAt time.Time `json:"startedAt"`
UpdatedAt time.Time `json:"updatedAt"`
// RequestID is the client request id that keyed this row, when one was
// supplied. It is the join key with activity events.
RequestID string `json:"requestId,omitempty"`
}
View is the safe, redacted projection of one journal row. It carries what an operator needs to tell the outcomes apart and to pick a next step; it never carries credentials or a Docker host.
func Describe ¶ added in v1.0.40
func Describe(op *db.ServerOperation) View
Describe projects a journal row for diagnostics, judged against the wall clock. Prefer DescribeAt where the caller already has a time.
func DescribeAll ¶ added in v1.0.40
func DescribeAll(ops []db.ServerOperation) []View
DescribeAll projects every row, preserving order.
func DescribeAllAt ¶ added in v1.0.40
func DescribeAllAt(ops []db.ServerOperation, now time.Time) []View
DescribeAllAt projects every row at one instant, so a list never shows two rows judged by clocks that disagree.
func DescribeAt ¶ added in v1.0.40
func DescribeAt(op *db.ServerOperation, now time.Time) View
DescribeAt projects a journal row as it should be read at now. Whether an operation is still in flight depends on when it was last advanced, so the clock is a parameter instead of a hidden dependency.
func FilterOutcome ¶ added in v1.0.40
FilterOutcome returns the views with the given outcome.