types

package
v0.74.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 19, 2026 License: MIT Imports: 3 Imported by: 0

Documentation

Index

Constants

View Source
const GaveUpProbeInterval = 30 * time.Minute

GaveUpProbeInterval is how often a given-up server is still probed by the periodic reconciliation. After MaxConnectionRetries consecutive failures the client stops its own retry ladder, but "never again" would leave an upstream silently dead after any outage longer than the ladder (laptop sleep, VPN drop, overnight maintenance) until a human notices and reconnects by hand — a failed upstream is invisible to the operator (#1013). One probe every 30 minutes keeps that self-healing at ~48 requests/day instead of ~2880.

The interval is deliberately FLAT. Parking is reserved for failures the classifier can PROVE deterministic (ConnectionInfo.Terminal, GH #1145); an escalating probe applied to the rest of the give-up tail would stretch unattended recovery to hours for exactly the failures that heal on their own — a stopped Docker daemon, a dropped VPN, a rate-limited upstream — and nothing in the tree shortens it again until a human looks.

View Source
const MaxConnectionRetries = 20

ConnectionInfo holds information about the current connection state MaxConnectionRetries is the maximum number of consecutive connection failures before giving up automatic reconnection. The server can still be reconnected manually or via reconnect-on-use.

View Source
const PermanentFailureAttempts = 2

PermanentFailureAttempts is how many classified-permanent connection failures are required before automatic reconnection stops.

Two, not one: the second attempt is a confirmation. A classifier false positive then costs one wasted retry instead of stranding a server that would have recovered — while still cutting the GH #1145 log from 55 attempts to 2.

Variables

This section is empty.

Functions

func OAuthRetryBackoffDuration added in v0.61.0

func OAuthRetryBackoffDuration(oauthRetryCount int) time.Duration

OAuthRetryBackoffDuration returns the (much longer) backoff to wait after the given number of consecutive OAuth failures: 5min, 15min, 1h, 4h, then 24h. An OAuth failure cannot be resolved by redialing quickly — it needs a token refresh or a human completing a login — so its ladder is deliberately coarse.

func RetryBackoffDuration added in v0.61.0

func RetryBackoffDuration(retryCount int) time.Duration

RetryBackoffDuration returns the exponential backoff to wait after the given number of consecutive connection failures: 1s, 2s, 4s, ... capped at 5 minutes.

Types

type ConnectionInfo

type ConnectionInfo struct {
	State            ConnectionState `json:"state"`
	LastError        error           `json:"last_error,omitempty"`
	RetryCount       int             `json:"retry_count"`
	LastRetryTime    time.Time       `json:"last_retry_time,omitempty"`
	ServerName       string          `json:"server_name,omitempty"`
	ServerVersion    string          `json:"server_version,omitempty"`
	LastOAuthAttempt time.Time       `json:"last_oauth_attempt,omitempty"`
	OAuthRetryCount  int             `json:"oauth_retry_count"`
	IsOAuthError     bool            `json:"is_oauth_error"`
	GaveUp           bool            `json:"gave_up"` // True when max retries exceeded
	// NetworkFailure reports that the current StateError was recorded by
	// SetNetworkError: the last connect attempt failed before any upstream
	// answered (DNS, dial, timeout). The connectivity watcher reconnects only
	// such servers when the network comes back (GH #1929). Reported by
	// GetConnectionInfo; state-change callback snapshots leave it false.
	NetworkFailure bool `json:"network_failure,omitempty"`
	// RetryAfter is the instant before which the upstream asked us not to come
	// back, captured from a `Retry-After` header on a rate-limited response
	// (#1040). Zero when the upstream gave no hint. It acts as a floor under the
	// retry ladder — the effective delay is max(backoff, Retry-After).
	RetryAfter time.Time `json:"retry_after,omitempty"`

	// Terminal marks a failure the diagnostics classifier proved deterministic:
	// the same attempt will fail identically until a human edits the config, the
	// image, or the machine. Automatic reconnection stops once the confirmation
	// budget (PermanentFailureAttempts) is spent — retrying a guaranteed failure
	// burns a container spawn, a package install and a log flood per attempt for
	// no chance of success (GH #1145: 55 identical attempts over 19 hours).
	//
	// It is CLEARED by every recovery path — a successful connection, an
	// explicit Reset (manual reconnect / disconnect), and client recreation on a
	// config change — so a user who fixes the problem always gets the server
	// back. See StateManager.SetTerminalError.
	Terminal bool `json:"terminal,omitempty"`
	// TerminalCode is the stable MCPX_* diagnostics code that proved the failure
	// permanent. Carried as a plain string so this package stays a leaf (no
	// import of internal/diagnostics). Empty unless Terminal is set.
	TerminalCode string `json:"terminal_code,omitempty"`

	// PendingAuthProbe marks a StatePendingAuth park whose OAuth requirement
	// was never confirmed by the upstream — no oauth block, no stored token,
	// and no HTTP 401 on the anonymous attempt (GH #1537). Such a park is
	// re-probed every GaveUpProbeInterval instead of waiting forever, so a
	// no-auth server misrouted into OAuth by a transient failure self-heals.
	// Meaningless outside StatePendingAuth.
	PendingAuthProbe bool `json:"pending_auth_probe,omitempty"`
}

func (*ConnectionInfo) ShouldAutoReconnect added in v0.61.0

func (ci *ConnectionInfo) ShouldAutoReconnect(now time.Time) bool

ShouldAutoReconnect reports whether an automatic (supervisor-driven) reconnect attempt is appropriate given the connection's failure history. It returns false while a backoff window from the last failure has not elapsed and for servers parked in PendingAuth — redialing cannot succeed until the user completes the OAuth login, and each attempt costs real requests against the upstream. Manual reconnects, login flows, and reconnect-on-use are not subject to this policy.

type ConnectionState

type ConnectionState int

ConnectionState represents the state of an upstream connection

const (
	// StateDisconnected indicates the upstream is not connected
	StateDisconnected ConnectionState = iota
	// StateConnecting indicates the upstream is attempting to connect
	StateConnecting
	// StatePendingAuth indicates the upstream requires OAuth authentication but is deferred (e.g., waiting for user action)
	StatePendingAuth
	// StateAuthenticating indicates the upstream is performing OAuth authentication
	StateAuthenticating
	// StateDiscovering indicates the upstream is discovering available tools
	StateDiscovering
	// StateReady indicates the upstream is connected and ready for requests
	StateReady
	// StateError indicates the upstream encountered an error
	StateError
)

func (ConnectionState) String

func (s ConnectionState) String() string

String returns the string representation of the connection state

type StateManager

type StateManager struct {
	// contains filtered or unexported fields
}

StateManager manages the state transitions for an upstream connection

func NewStateManager

func NewStateManager() *StateManager

NewStateManager creates a new state manager

func (*StateManager) ClearNetworkBackoff added in v0.74.0

func (sm *StateManager) ClearNetworkBackoff(now time.Time) bool

ClearNetworkBackoff restarts the retry ladder of a server whose current failure is still a network failure, and reports whether it did. The check and the clear happen under one lock, so a connect that started (Connecting), finished (Ready) or failed differently (OAuth, permanent, Retry-After) since the caller looked is never overwritten. The state stays StateError; the reconnect that follows moves it. Used by the connectivity watcher when the network demonstrably came back (GH #1929).

func (*StateManager) ClearTerminal added in v0.64.0

func (sm *StateManager) ClearTerminal()

ClearTerminal un-parks a terminally failed connection without otherwise touching the retry ladder. It exists for the "the user changed something" paths that keep the existing client instead of recreating it.

func (*StateManager) GetConnectionInfo

func (sm *StateManager) GetConnectionInfo() ConnectionInfo

GetConnectionInfo returns detailed connection information

func (*StateManager) GetState

func (sm *StateManager) GetState() ConnectionState

GetState returns the current connection state

func (*StateManager) GetStateChangeCallback

func (sm *StateManager) GetStateChangeCallback() func(oldState, newState ConnectionState, info *ConnectionInfo)

GetStateChangeCallback returns the current state change callback

func (*StateManager) IsConnecting

func (sm *StateManager) IsConnecting() bool

IsConnecting returns true if the connection is in progress

func (*StateManager) IsOAuthError

func (sm *StateManager) IsOAuthError() bool

IsOAuthError returns true if the last error was OAuth-related

func (*StateManager) IsReady

func (sm *StateManager) IsReady() bool

IsReady returns true if the connection is ready for requests

func (*StateManager) IsState

func (sm *StateManager) IsState(state ConnectionState) bool

IsState checks if the current state matches the given state

func (*StateManager) IsTerminal added in v0.64.0

func (sm *StateManager) IsTerminal() (bool, string)

IsTerminal reports whether the connection is parked on a classifier-proven permanent failure, and the code that proved it.

func (*StateManager) IsUserLoggedOut

func (sm *StateManager) IsUserLoggedOut() bool

IsUserLoggedOut returns true if the user has explicitly logged out

func (*StateManager) Reset

func (sm *StateManager) Reset()

Reset resets the state manager to disconnected state

func (*StateManager) ResetForReconnect added in v0.24.0

func (sm *StateManager) ResetForReconnect()

ResetForReconnect transitions to Disconnected state for a reconnection attempt while PRESERVING retryCount and lastRetryTime so exponential backoff is not defeated. Use this instead of Reset() when retrying a failed connection.

func (*StateManager) RetryAfter added in v0.61.0

func (sm *StateManager) RetryAfter() time.Time

RetryAfter returns the current park deadline, or the zero time when the upstream has given no rate-limit hint.

func (*StateManager) SetError

func (sm *StateManager) SetError(err error)

SetError sets an error and transitions to error state

func (*StateManager) SetNetworkError added in v0.74.0

func (sm *StateManager) SetNetworkError(err error)

SetNetworkError is SetError for a connect attempt that failed before any upstream answered — DNS, dial, connection refused, timeout (GH #1929). It advances the backoff exactly like SetError; it only marks the failure so the connectivity watcher may reconnect the server once the network is back.

func (*StateManager) SetOAuthError

func (sm *StateManager) SetOAuthError(err error)

SetOAuthError sets an OAuth-specific error with longer backoff periods

func (*StateManager) SetPendingAuth added in v0.61.0

func (sm *StateManager) SetPendingAuth(err error)

SetPendingAuth parks the connection in StatePendingAuth with the deferred-OAuth error attached. It exists because the obvious spelling — TransitionTo(StatePendingAuth) followed by SetError(err) — silently undoes itself: SetError forces StateError unconditionally, so the PendingAuth state survived microseconds and every consumer (health, notifications, the supervisor's reconnect gate) saw a plain error and kept redialing a server that cannot connect until a human logs in (#1013).

RetryCount is deliberately NOT bumped: a parked server is not retrying, so there is no ladder to advance and nothing to "give up" on. The wake paths are explicit user action (login, manual reconnect, reconnect-on-use) and the persisted-token scan in Manager.scanForNewTokens. A speculative park (SetPendingAuthProbe) is additionally re-probed by the reconnect loop every GaveUpProbeInterval.

func (*StateManager) SetPendingAuthProbe added in v0.71.0

func (sm *StateManager) SetPendingAuthProbe(err error, probe bool)

SetPendingAuthProbe is SetPendingAuth with the park's provenance: probe=true records that the OAuth requirement is unconfirmed (GH #1537), which lets ShouldAutoReconnect retry the server periodically.

func (*StateManager) SetRetryAfter added in v0.61.0

func (sm *StateManager) SetRetryAfter(deadline time.Time)

SetRetryAfter records an upstream-supplied park deadline captured from a `Retry-After` header on a rate-limited (429 / 503) response (#1040).

It only ever EXTENDS the window: a later, shorter hint must not shorten a wait the upstream already asked for, and an already-elapsed deadline is dropped rather than stored. Call it BEFORE SetError/SetOAuthError so the resulting ConnectionInfo the state-change callback publishes already carries the window.

func (*StateManager) SetServerInfo

func (sm *StateManager) SetServerInfo(name, version string)

SetServerInfo sets the server information

func (*StateManager) SetStateChangeCallback

func (sm *StateManager) SetStateChangeCallback(callback func(oldState, newState ConnectionState, info *ConnectionInfo))

SetStateChangeCallback sets a callback function that will be called on state changes

func (*StateManager) SetTerminalError added in v0.64.0

func (sm *StateManager) SetTerminalError(err error, code string)

SetTerminalError records a connection failure that the diagnostics classifier proved deterministic and unrecoverable — a missing binary, an image without the interpreter, an unparseable config. Retrying it cannot succeed until a human changes something, so after PermanentFailureAttempts confirmations both retry gates stop dialing and the server is surfaced as terminally failed.

It is otherwise exactly SetError: the state, error, counter and timestamp all advance identically, so every existing consumer keeps working. `code` is the stable MCPX_* string that justified the park, carried through to the API so the failure is visible with its reason rather than silently swallowed.

Call this ONLY from the connect path, and only for a code that diagnostics.ParkableCode accepts — permanence in the catalog is necessary but not sufficient, because the classifier's string fallbacks read the child process's own stderr. Plain SetError deliberately clears the park.

func (*StateManager) SetTransitionLoginClassifier added in v0.74.0

func (sm *StateManager) SetTransitionLoginClassifier(isLogin func(Transition) bool)

SetTransitionLoginClassifier installs the function that says whether a change into the error state is a login requirement. It is called under the state lock and must not block or call back into the state manager.

func (*StateManager) SetTransitionObserver added in v0.74.0

func (sm *StateManager) SetTransitionObserver(observer TransitionObserver)

SetTransitionObserver installs the observer. Transitions recorded before an observer is installed are not kept.

func (*StateManager) SetUserLoggedOut

func (sm *StateManager) SetUserLoggedOut(loggedOut bool)

SetUserLoggedOut marks that the user has explicitly logged out This prevents automatic reconnection until cleared (e.g., by explicit login)

func (*StateManager) ShouldRetry

func (sm *StateManager) ShouldRetry() bool

ShouldRetry returns true if the connection should be retried based on exponential backoff

func (*StateManager) ShouldRetryOAuth

func (sm *StateManager) ShouldRetryOAuth() bool

ShouldRetryOAuth returns true if OAuth should be retried with much longer backoff intervals

func (*StateManager) TransitionTo

func (sm *StateManager) TransitionTo(newState ConnectionState)

TransitionTo transitions to a new state

func (*StateManager) ValidateTransition

func (sm *StateManager) ValidateTransition(from, to ConnectionState) error

ValidateTransition validates if a state transition is allowed

type Transition added in v0.74.0

type Transition struct {
	From ConnectionState
	To   ConnectionState
	// At is the time of the change, taken under the state lock.
	At time.Time
	// Err is the connection's last error at that moment (nil on success).
	Err error
	// TerminalCode is the diagnostics code of a terminal failure, when set.
	TerminalCode string
	// Folded is the number of earlier transitions folded into this one
	// because the observer had stalled and the queue was full. A folded
	// record still starts where the first one started and ends where the
	// last one ended, so the observer sees every state the connection is
	// left in; what it loses is the intermediate states, and it is told how
	// many.
	Folded int
	// FoldedLost, FoldedRestored and FoldedLogin count the changes that folding
	// hid inside this record: a move out of Ready, a move into Ready and a
	// move into PendingAuth that From and To no longer show. Folding avoids
	// hiding one while any other pair can be folded, and counts it when it
	// cannot, so an outage is never erased by a stalled observer.
	FoldedLost     int
	FoldedRestored int
	FoldedLogin    int
	// FoldedLoginLast says that, of the hidden recoveries and login
	// requirements, a login requirement came last: no recovery followed it,
	// so it is still open after everything the record hid.
	FoldedLoginLast bool
	// Login marks a change into the error state whose error means "a login
	// is required" (set by the login classifier at capture). Such a change
	// is a login requirement exactly like a change into PendingAuth, and
	// folding must not hide it uncounted.
	Login bool
	// LoggedOut and OAuthError are the operator's logged-out flag and the
	// OAuth-error flag at that moment. A logout's disconnect is a login
	// requirement (release QA F1); an OAuth error flag alone is not, if the
	// error itself is a network failure.
	LoggedOut  bool
	OAuthError bool
}

Transition is one state change, captured when it happened.

type TransitionObserver added in v0.74.0

type TransitionObserver func(Transition)

TransitionObserver receives transitions in the order they happened, one at a time. It is called without the state lock and must not block for long: the next transition of the same connection waits behind it.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL