Documentation
¶
Overview ¶
Package hosterrors is the scan-wide dead-host circuit breaker.
A host:port that keeps failing at the transport level (refused, reset, DNS, i/o timeout) is quarantined so the scan stops spending requests and worker time on it. Unlike the cache it replaced, quarantine is not forever: after a cooldown one probe request is let through, and its outcome decides whether the host recovers or stays quarantined for a longer cooldown. A host whose probes keep failing is eventually declared dead for the rest of the scan.
Closed --(consecutive errors >= MaxHostError)--> Open(openedAt, cooldown) Open --(cooldown elapsed; first Admit wins)--> HalfOpen(probeAt) HalfOpen --(MarkSuccess)--> Closed (errors=0, cooldown=base) HalfOpen --(MarkFailed)--> Open (cooldown=min(2×cooldown, max)), trips++ HalfOpen --(probe unreported within ProbeTTL)--> another probe may be admitted HalfOpen --(MarkFailed, trips >= MaxTrips)--> Dead (sticky for the scan)
The key is host:port (see httpmsg.HostPortKey), with no method: a host that refuses connections refuses them for every method.
The API is split so that looking does not consume the probe:
- Quarantined is read-only. The executor uses it to drop work items for a quarantined host before they occupy a worker.
- Admit mutates. The requester calls it immediately before a send; it is the call that turns Open into HalfOpen, and it lets exactly one caller through as the probe.
Index ¶
- Constants
- Variables
- type Cache
- func (c *Cache) Admit(key string) bool
- func (c *Cache) AnyTripped() bool
- func (c *Cache) Close()
- func (c *Cache) LastError(key string) error
- func (c *Cache) MarkFailed(key string, err error, ignoreTimeout bool)
- func (c *Cache) MarkSuccess(key string)
- func (c *Cache) Quarantined(key string) bool
- func (c *Cache) QuarantinedCount() int
- func (c *Cache) SetObserver(fn func(Transition))
- func (c *Cache) SetVerbose(verbose bool)
- func (c *Cache) State(key string) State
- type Config
- type State
- type Transition
Constants ¶
const ( DefaultMaxHostsCount = 10000 DefaultCooldown = 30 * time.Second DefaultMaxCooldown = 5 * time.Minute DefaultMaxTrips = 5 DefaultProbeTTL = 30 * time.Second )
Defaults for the breaker timings. DefaultProbeTTL is 2 × the default request timeout (15s); callers that know the configured timeout pass 2 × that.
Variables ¶
var ErrUnresponsiveHost = errors.New("skipping as host is unresponsive")
ErrUnresponsiveHost is returned when a host is unresponsive
Functions ¶
This section is empty.
Types ¶
type Cache ¶
type Cache struct {
MaxHostError int
TrackError []string
// contains filtered or unexported fields
}
Cache is the host:port circuit breaker. All methods are safe for concurrent use; Quarantined, Admit and the healthy-host paths of MarkSuccess take no lock beyond a sync.Map load.
func NewWithConfig ¶ added in v0.5.3
NewWithConfig returns a breaker for cfg, filling zero fields with defaults.
func (*Cache) Admit ¶ added in v0.5.3
Admit reports whether a request to key may be sent now. Call it immediately before the send. For an Open host whose cooldown has elapsed it performs the Open→HalfOpen transition, and exactly one concurrent caller gets true — the probe. A half-open probe that has not reported within ProbeTTL is treated as lost, and one more caller is admitted in its place.
func (*Cache) AnyTripped ¶ added in v0.5.3
AnyTripped reports whether any host has ever opened. While it is false, Quarantined is false for every key, so a caller filtering many items can skip building their keys.
func (*Cache) LastError ¶ added in v0.5.3
LastError returns the most recent host error recorded for key, or nil.
func (*Cache) MarkFailed ¶
MarkFailed records a failed request to key. Errors that are not host errors (see checkError) are ignored, except that one arriving while a probe is in flight releases the probe slot: the probe's outcome was inconclusive, so the next caller may probe instead of waiting out ProbeTTL.
func (*Cache) MarkSuccess ¶
MarkSuccess records a successful request to key. A Closed host's error counter resets; a HalfOpen host recovers. A success while Open is a request that was in flight before the trip and is deliberately ignored: only the probe decides recovery.
func (*Cache) Quarantined ¶ added in v0.5.3
Quarantined reports whether work for key should be dropped now. Read-only: it never consumes the half-open probe. True while Open inside the cooldown, while HalfOpen with a probe in flight, and once Dead.
func (*Cache) QuarantinedCount ¶
QuarantinedCount returns how many hosts are currently not Closed (Open, HalfOpen or Dead). Informational, for logging cross-phase propagation.
func (*Cache) SetObserver ¶ added in v0.5.3
func (c *Cache) SetObserver(fn func(Transition))
SetObserver installs fn to receive every state transition. It is called outside the breaker's locks, on the goroutine that caused the change, so it must not block. Passing nil removes the observer.
func (*Cache) SetVerbose ¶
SetVerbose sets the cache to log at verbose level
type Config ¶ added in v0.5.3
type Config struct {
// MaxHostError is the consecutive tracked-error count that opens the
// breaker (--max-host-error). Must be > 0.
MaxHostError int
// MaxHostsCount bounds the ARC that holds error counters for healthy
// (Closed) hosts. Tripped hosts live outside it and are never evicted.
MaxHostsCount int
// TrackError lists extra error substrings that count as host errors.
TrackError []string
// Cooldown is the first quarantine period; MaxCooldown caps the doubling.
Cooldown time.Duration
MaxCooldown time.Duration
// MaxTrips is how many failed probes make a host dead. Negative disables
// the dead state (the host keeps being re-probed at MaxCooldown).
MaxTrips int
// ProbeTTL is how long a half-open probe may stay unreported before another
// probe is admitted (a probe whose caller vanished must not wedge the host).
ProbeTTL time.Duration
// Now is the clock; nil means time.Now. Tests inject a fake one.
Now func() time.Time
}
Config configures a Cache. Zero values take the defaults above.
type State ¶ added in v0.5.3
type State int32
State is a host's breaker state.
const ( // StateClosed — healthy; consecutive transport errors are being counted. StateClosed State = iota // StateOpen — quarantined until the cooldown elapses. StateOpen // StateHalfOpen — one probe request is in flight. StateHalfOpen // StateDead — every probe failed; quarantined for the rest of the scan. StateDead )
type Transition ¶ added in v0.5.3
type Transition struct {
Key string
From, To State
// Errors is the consecutive-error count at the time of the change.
Errors int32
// Trips is the number of failed probes so far.
Trips int32
// Cooldown is the quarantine period now in force (Open only).
Cooldown time.Duration
// Err is the error that caused the change, nil for a recovery.
Err error
}
Transition describes one breaker state change, delivered to the observer installed with SetObserver.
Source Files