Documentation
¶
Overview ¶
Package keypool implements per-key circuit-breaker state and configurable Pool selection over healthy keys. State is persisted in pkg/state under "secret_health:<keyHash>" (circuit records), "pool_rr:<poolName>" (round-robin counters), and "pool_lru:<poolName>:<keyHash>" (LRU timestamps).
Supported selection strategies (KeySelection):
- "prioritized" (default) — always pick the first healthy key in declaration order.
- "round-robin" — distribute traffic evenly using a counter.
- "least-recently-used" — pick the key with the oldest last-used timestamp.
The KeySelection enum lives in this package (not policy) so policy can import keypool without creating a cycle when its Service composes Selector + Limiter at runtime.
Index ¶
- Variables
- func ClearCircuit(ctx context.Context, store kv.Store, keyHash string) error
- func RegisterScripts(m *kv.Mem)
- type CircuitRecord
- type CircuitState
- type CooldownReason
- type FailureKind
- type KeySelection
- type Selector
- func (s *Selector) Pick(ctx context.Context, scope string, algo KeySelection, keys []*hostkey.HostKey, ...) (*hostkey.HostKey, error)
- func (s *Selector) PickWithExclude(ctx context.Context, scope string, algo KeySelection, keys []*hostkey.HostKey, ...) (*hostkey.HostKey, error)
- func (s *Selector) ReadCircuit(ctx context.Context, keyHash string) (CircuitRecord, bool)
- func (s *Selector) RecordFailure(ctx context.Context, keyHash string, kind FailureKind, ...)
- func (s *Selector) RecordLocalRateLimit(ctx context.Context, keyHash string, retryAfter time.Duration)
- func (s *Selector) RecordSuccess(ctx context.Context, keyHash string)
Constants ¶
This section is empty.
Variables ¶
var ErrNoHealthyKeys = errors.New("keypool: no healthy keys in pool")
Functions ¶
func ClearCircuit ¶
ClearCircuit deletes the circuit-breaker record for a key from the KV store. This is best-effort: if the DEL fails the error is returned to the caller, who should log a warning but not fail the outer operation.
Use this when a key is permanently deleted from the catalog so that orphaned secret_health:* keys do not accumulate in Redis indefinitely (R-8).
func RegisterScripts ¶ added in v0.9.8
RegisterScripts installs the Go emulators for keypool's Lua scripts on a MemStore, mirroring the Redis-side behaviour exactly. New registers them automatically when the store IS a *kv.Mem; test doubles that merely embed one must call this on the inner Mem themselves (same convention as pkgratelimit.RegisterScripts).
Types ¶
type CircuitRecord ¶
type CircuitRecord struct {
State CircuitState `json:"state"`
OpenUntil time.Time `json:"open_until,omitempty"`
BackoffStep int `json:"backoff_step"`
LastTransition time.Time `json:"last_transition"`
Indefinite bool `json:"indefinite,omitempty"`
Reason CooldownReason `json:"reason,omitempty"`
}
CircuitRecord is the per-key circuit-breaker state stored under "secret_health:<keyHash>" in pkg/state.
type CircuitState ¶
type CircuitState int
CircuitState describes the current health of a key.
const ( CircuitClosed CircuitState = iota // healthy, accepting traffic CircuitOpen // unhealthy, skip CircuitHalfOpen // single probe allowed )
func (CircuitState) String ¶
func (s CircuitState) String() string
String returns a stable lower-snake label for the circuit state, suitable for logs and API responses.
type CooldownReason ¶
type CooldownReason string
CooldownReason describes why a key was placed into cooldown (circuit open). The empty string means unknown / legacy record written before this field existed.
const ( ReasonUpstreamAuthFailed CooldownReason = "upstream_auth_failed" ReasonUpstreamRateLimited CooldownReason = "upstream_rate_limited" ReasonUpstreamServerError CooldownReason = "upstream_server_error" ReasonUpstreamNetworkError CooldownReason = "upstream_network_error" ReasonLocalRateLimited CooldownReason = "local_rl_exhausted" )
type FailureKind ¶
type FailureKind int
FailureKind classifies the upstream failure for circuit-breaker transitions.
const ( FailureAuth FailureKind = iota // 401/403 → open indefinitely FailureRateLimitShort // 429 with Retry-After ≤ 5s → stay closed FailureRateLimitLong // 429 with Retry-After > 5s → open for that duration FailureServerError // 5xx → exponential backoff FailureNetwork // net/timeout (post-connect) → treat as 5xx // FailureUpstreamUnreachable is a dial-phase failure (connection refused, // no route, DNS, TLS handshake): the connection was never established, so // it is a property of the host/baseURL, not the key. It NEVER trips a key // breaker — the pipeline retries the same host with backoff and reports an // unreachable status. Distinguishes a misconfigured baseURL from a bad key. FailureUpstreamUnreachable )
type KeySelection ¶
type KeySelection string
KeySelection is the algorithm a Selector uses to pick from healthy candidates. Persisted as a string in Policy.Spec.KeySelection.
const ( // KeySelectionPrioritized drains keys in declaration order — the // first healthy key in the candidate set wins. Default. KeySelectionPrioritized KeySelection = "prioritized" // KeySelectionRoundRobin rotates evenly across healthy keys via a // per-scope counter. KeySelectionRoundRobin KeySelection = "round-robin" // KeySelectionLeastRecentlyUsed prefers the key whose last successful // use was furthest in the past. KeySelectionLeastRecentlyUsed KeySelection = "least-recently-used" )
type Selector ¶
type Selector struct {
// contains filtered or unexported fields
}
Selector picks HostKeys from Pools and tracks per-key circuit-breaker state.
func New ¶
New constructs a Selector. clock and rng may be nil. When rng is nil, a new rand seeded from time.Now().UnixNano() is used.
func (*Selector) Pick ¶
func (s *Selector) Pick(ctx context.Context, scope string, algo KeySelection, keys []*hostkey.HostKey, exclude ...[]*hostkey.HostKey) (*hostkey.HostKey, error)
Pick returns a healthy HostKey from the candidate set. scope is the kv key tag (typically the owning Policy's Meta.Name) used for the round-robin counter and LRU timestamps. algo is the selection strategy; empty falls back to KeySelectionPrioritized.
exclude is an optional list of keys to skip even if healthy (e.g. for retry-with-exclusion in future callers). Pass nil to skip the check.
Open keys past their OpenUntil are auto-transitioned to HalfOpen and become eligible. Concurrent Picks may both pick the same half-open key; the caller's RecordSuccess/RecordFailure resolves the outcome (acceptable).
func (*Selector) PickWithExclude ¶
func (s *Selector) PickWithExclude(ctx context.Context, scope string, algo KeySelection, keys []*hostkey.HostKey, exclude []*hostkey.HostKey) (*hostkey.HostKey, error)
PickWithExclude is a convenience wrapper around Pick that accepts an explicit exclude list. Callers that always have an exclude slice can use this to avoid the variadic syntax.
func (*Selector) ReadCircuit ¶
ReadCircuit returns the stored circuit-breaker record for a key and whether a record actually exists in the state store. A missing or undecodable record yields a default-closed record with found=false — i.e. the key has never failed and is assumed healthy. This is a read-only accessor for the admin plane; it does not auto-transition expired-open records the way Pick does.
func (*Selector) RecordFailure ¶
func (s *Selector) RecordFailure(ctx context.Context, keyHash string, kind FailureKind, retryAfter time.Duration)
RecordFailure transitions according to kind. retryAfter is honoured only for RateLimit kinds.
func (*Selector) RecordLocalRateLimit ¶
func (s *Selector) RecordLocalRateLimit(ctx context.Context, keyHash string, retryAfter time.Duration)
RecordLocalRateLimit cools down a key because our own rate-limit rule fired (KeyQuotaExhausted from pkg/ratelimit). Distinct from upstream-driven cooldowns: no backoff escalation, no half-open probe — the duration is deterministic from Retry-After.
Used by the pipeline's post-Pick Reserve path (issue #89, future PR).
func (*Selector) RecordSuccess ¶
RecordSuccess transitions a key to CircuitClosed and resets backoff.
The read (prior state, for logging) and the write happen atomically in a single Lua round trip so a concurrent RecordFailure on another pod cannot be lost between our GET and SET. Falls back to the non-atomic Go-side GET+SET only when the store has no script runner (custom test doubles).
Source Files