operation

package
v1.0.40 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 17, 2026 License: MIT Imports: 12 Imported by: 0

Documentation

Overview

Package operation recovers server mutations that were interrupted by a panel crash or restart. A disconnected Docker request has an unknown outcome: the operation journal keeps the server locked until startup has observed the container, instead of allowing a blind retry.

Index

Constants

View Source
const ExecutionStaleAfter = 2 * time.Minute

ExecutionStaleAfter is how long an executing operation may stay unchanged before it stops being "in flight" and becomes an unknown outcome. A mutation that has not recorded a completion by then may still be running, but the panel can no longer tell, and an operator needs the escape hatch.

Variables

View Source
var ErrAlreadyFailed = errors.New("matching lifecycle operation already failed")

ErrAlreadyFailed is returned when the same request id already failed. A new request id is required to try again.

View Source
var ErrInProgress = errors.New("matching lifecycle operation is already in progress")

ErrInProgress is returned when the same request id is already running.

Functions

func ClientRequestID added in v1.0.40

func ClientRequestID(op *db.ServerOperation) string

ClientRequestID is the caller-supplied request id stored on a journal row. StorageKey concatenates that id with the kind; a generated key (empty request id) is returned as-is so an operator can still quote it.

func CountByOutcome added in v1.0.40

func CountByOutcome(views []View) map[Outcome]int

CountByOutcome counts the projected operations by outcome, which is what a bundle header and a health line want.

func Digest

func Digest(kind string, server *db.Server) string

Digest is the request payload fingerprint for a journaled mutation.

func RedactError added in v1.0.40

func RedactError(text string) string

RedactError removes credentials from daemon and client error text before it is written to a diagnostic. Hosts, paths and messages are kept: they are the evidence an operator needs.

func RetryReview

func RetryReview(ctx context.Context, store ReviewStore, observer Observer, log *logger.Logger, ids []string, every time.Duration)

RetryReview re-observes operations whose container state was unknown at startup. It stops when every id has been released or ctx is cancelled. Operations that are not in the list are left alone, so an in-flight mutation is never finalized by this loop.

func RunJournaled

func RunJournaled(ctx context.Context, store Journal, log *logger.Logger, server *db.Server, kind, requestID string, mutate func() error) error

RunJournaled persists the intent of a server mutation before it is attempted, advances the journal through the attempt, and finalizes it. requestID is a client idempotency key: a repeat of a completed key does not run mutate again, and a repeat of a key that is still running returns ErrInProgress. An empty requestID is unique to this call.

If the process dies between the external effect and finalization, the entry stays active and startup recovery observes the container before releasing the per-server lock (see RecoverInterrupted). The active lock is the shared coordinator: no two lifecycle mutations of one world can interleave. A mutation error finalizes as failed so the user can retry without restarting the panel.

func StorageKey

func StorageKey(requestID, kind string) string

StorageKey is the journal key for a client request id and mutation kind. An empty request id is unique to this call, so it is not idempotent.

func TrimRequestKey added in v1.0.40

func TrimRequestKey(key string) string

TrimRequestKey shortens a client request key for display without changing the value the journal matches on.

func ValidOutcome added in v1.0.40

func ValidOutcome(name string) bool

ValidOutcome reports whether name is one this package produces. It exists so an RPC filter cannot silently match nothing because of a typo.

func ValidateRequestID

func ValidateRequestID(id string) error

ValidateRequestID rejects client request ids that do not fit the journal.

Types

type Journal

type Journal interface {
	Store
	BeginServerOperation(ctx context.Context, serverID, nodeID, containerID, key, kind, digest string) (*db.ServerOperation, bool, error)
}

Journal is the operation journal subset that RunJournaled needs. *db.Store satisfies it.

type NextAction added in v1.0.40

type NextAction string

NextAction is the safe next step for an operation in its current state. It is a stable code so a UI, a runbook or a support engineer can agree on the remedy without parsing an error string.

const (
	// ActionNone means nothing is required.
	ActionNone NextAction = "none"
	// ActionWait means the operation is still in flight; wait for it.
	ActionWait NextAction = "wait"
	// ActionResolve means the process that owned the operation is gone. Startup
	// recovery observes the container before releasing the lock; an operator can
	// also resolve it explicitly.
	ActionResolve NextAction = "resolve_operation"
	// ActionRetryNewRequestID means the recorded attempt failed and the same
	// request id is refused, so a retry needs a new one.
	ActionRetryNewRequestID NextAction = "retry_with_new_request_id"
	// ActionContactOperator means the state is outside the automatic paths.
	ActionContactOperator NextAction = "contact_operator"
)

type Observed

type Observed struct {
	Exists      bool
	Running     bool
	ContainerID string
}

Observed is the container state recovery sees for an interrupted operation.

type Observer

type Observer interface {
	Observe(ctx context.Context, serverID, containerID string) (Observed, error)
}

Observer resolves the current container state for a server. Implementations must return an error when the state cannot be established (for example the node is unreachable); recovery treats that as "leave the lock in place".

func NewDockerObserver

func NewDockerObserver(client *docker.Client) Observer

NewDockerObserver adapts a Docker client to recovery's observation interface.

type Outcome added in v1.0.40

type Outcome string

Outcome is what an operator should understand an operation to have done. Unknown is a first-class answer: a mutation whose external effect was not recorded is neither a success nor a failure.

const (
	// OutcomeSuccess means the mutation completed and was recorded.
	OutcomeSuccess Outcome = "success"
	// OutcomeFailed means the mutation returned an error and was recorded.
	OutcomeFailed Outcome = "failed"
	// OutcomeRunning means the mutation is still being attempted. It is the
	// only outcome that may still turn into a success or a failure on its own.
	OutcomeRunning Outcome = "running"
	// OutcomeUnknown means the intent was recorded but no completion or
	// failure was. The external effect may or may not have happened.
	OutcomeUnknown Outcome = "unknown"
)

type ReviewStore

type ReviewStore interface {
	GetServerOperation(ctx context.Context, id string) (*db.ServerOperation, error)
	AdvanceServerOperation(ctx context.Context, id, from, to, detail string) error
}

ReviewStore is the journal subset used to retry locks that could not be observed at startup. *db.Store satisfies it.

type SaveResumer added in v1.0.40

type SaveResumer interface {
	ResumeWorldSaves(ctx context.Context, serverID string) error
}

SaveResumer turns world saves back on after an interrupted backup. save-off does not survive a stopped JVM, so callers send it only while the container is still running.

type Store

type Store interface {
	ListActiveServerOperations(ctx context.Context) ([]db.ServerOperation, error)
	AdvanceServerOperation(ctx context.Context, id, from, to, detail string) error
	NoteActiveServerOperation(ctx context.Context, id, detail string) error
}

Store is the journal subset that recovery needs. *db.Store satisfies it.

type Summary

type Summary struct {
	Scanned     int
	Recovered   int
	NeedsReview int
	// ReviewIDs are the active operations whose container state could not be
	// established. A later pass may observe them once the daemon is reachable.
	ReviewIDs []string
}

Summary reports what startup recovery did.

func RecoverInterrupted

func RecoverInterrupted(ctx context.Context, store Store, observer Observer, log *logger.Logger) (Summary, error)

RecoverInterrupted observes every operation that still owns a server lock and finalizes it when the container state is known. Operations whose state cannot be observed keep their lock and are reported as requiring review; their server stays blocked until an operator or a later startup resolves them.

func ResumeInterruptedBackups added in v1.0.40

func ResumeInterruptedBackups(ctx context.Context, store Store, observer Observer, saves SaveResumer, log *logger.Logger) (Summary, error)

ResumeInterruptedBackups sends save-on for an interrupted backup whose container is still running, then releases that lock. A failed save-on leaves the lock in place. A stopped container is released without a command, because the next boot starts with saves enabled and this pass must not start the server.

type View added in v1.0.40

type View struct {
	ID                 string     `json:"id"`
	ServerID           string     `json:"serverId"`
	Kind               string     `json:"kind"`
	Phase              string     `json:"phase"`
	Outcome            Outcome    `json:"outcome"`
	NextAction         NextAction `json:"nextAction"`
	Reason             string     `json:"reason"`
	EndpointGeneration int64      `json:"endpointGeneration"`
	HasContainer       bool       `json:"hasContainer"`
	HoldsLock          bool       `json:"holdsLock"`
	LastError          string     `json:"lastError,omitempty"`
	StartedAt          time.Time  `json:"startedAt"`
	UpdatedAt          time.Time  `json:"updatedAt"`
	// RequestID is the client request id that keyed this row, when one was
	// supplied. It is the join key with activity events.
	RequestID string `json:"requestId,omitempty"`
}

View is the safe, redacted projection of one journal row. It carries what an operator needs to tell the outcomes apart and to pick a next step; it never carries credentials or a Docker host.

func Describe added in v1.0.40

func Describe(op *db.ServerOperation) View

Describe projects a journal row for diagnostics, judged against the wall clock. Prefer DescribeAt where the caller already has a time.

func DescribeAll added in v1.0.40

func DescribeAll(ops []db.ServerOperation) []View

DescribeAll projects every row, preserving order.

func DescribeAllAt added in v1.0.40

func DescribeAllAt(ops []db.ServerOperation, now time.Time) []View

DescribeAllAt projects every row at one instant, so a list never shows two rows judged by clocks that disagree.

func DescribeAt added in v1.0.40

func DescribeAt(op *db.ServerOperation, now time.Time) View

DescribeAt projects a journal row as it should be read at now. Whether an operation is still in flight depends on when it was last advanced, so the clock is a parameter instead of a hidden dependency.

func FilterOutcome added in v1.0.40

func FilterOutcome(views []View, outcome Outcome) []View

FilterOutcome returns the views with the given outcome.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL