Documentation
¶
Overview ¶
Package retry provides retry-queue and smart-retry types for the engine package. See ../REFACTOR_PLAN.md.
Note: hawk also has a top-level `github.com/GrayCodeAI/hawk/internal/resilience/retry` package for low-level HTTP/transport retry. This sub-package is specifically the engine's higher-level retry queue (work items deferred for later attempt).
Index ¶
- type FailureRecord
- type Item
- type Queue
- type RetryConfigFromProvider
- type RetryDecision
- type RetryItem
- type RetryQueue
- func (rq *RetryQueue) CalculateBackoff(attempts int) time.Duration
- func (rq *RetryQueue) Clear()
- func (rq *RetryQueue) Dequeue() *RetryItem
- func (rq *RetryQueue) Enqueue(operation string, args map[string]interface{}, err string, priority int) *RetryItem
- func (rq *RetryQueue) FormatQueue() string
- func (rq *RetryQueue) GetPending() []*RetryItem
- func (rq *RetryQueue) GetReady() []*RetryItem
- func (rq *RetryQueue) MarkFailed(id string, err string)
- func (rq *RetryQueue) MarkSuccess(id string)
- func (rq *RetryQueue) Prune()
- func (rq *RetryQueue) Size() int
- type RetryStrategy
- type SmartRetry
- func (sr *SmartRetry) AdaptStrategy(provider string)
- func (sr *SmartRetry) CalculateDelay(strategy *RetryStrategy, attempt int) time.Duration
- func (sr *SmartRetry) ClassifyError(err error) string
- func (sr *SmartRetry) ConfigureFromProvider(provider string, cfg RetryConfigFromProvider)
- func (sr *SmartRetry) Decide(provider, model string, err error, attempt int) *RetryDecision
- func (sr *SmartRetry) FormatStatus() string
- func (sr *SmartRetry) GetProviderHealth() map[string]string
- func (sr *SmartRetry) RecordRecovery(provider, model string, recoveryDelay time.Duration)
- func (sr *SmartRetry) ResetProvider(provider string)
- func (sr *SmartRetry) ShouldFallback(provider string) (bool, string)
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type FailureRecord ¶
type FailureRecord struct {
Provider string
Model string
ErrorType string // "rate_limit", "timeout", "server_error", "invalid_request", "auth"
ErrorMsg string
Timestamp time.Time
RetryCount int
Recovered bool
RecoveryDelay time.Duration
}
FailureRecord captures information about a failure for learning.
type Queue ¶
type Queue = RetryQueue
Queue is the FIFO of pending retry items with backoff and dedup.
type RetryConfigFromProvider ¶
type RetryConfigFromProvider interface {
BaseDelayMs() int
MaxDelayMs() int
MaxRetries() int
BackoffMultiplier() float64
JitterPct() int
}
RetryConfigFromProvider is the interface for provider retry configuration.
type RetryDecision ¶
type RetryDecision struct {
ShouldRetry bool
Delay time.Duration
Reason string
FallbackProvider string // switch provider if this one is failing
FallbackModel string
}
RetryDecision describes what the caller should do after a failure.
type RetryItem ¶
type RetryItem struct {
ID string
Operation string
Args map[string]interface{}
Error string
Attempts int
MaxAttempts int
NextRetry time.Time
Priority int
CreatedAt time.Time
Status string // "pending", "retrying", "succeeded", "failed_permanent"
}
RetryItem represents a single operation queued for retry.
type RetryQueue ¶
type RetryQueue struct {
Items []*RetryItem
MaxSize int
BackoffBase time.Duration
BackoffMax time.Duration
// contains filtered or unexported fields
}
RetryQueue manages failed operations with exponential backoff, priority ordering, and deduplication of identical operations.
func NewRetryQueue ¶
func NewRetryQueue() *RetryQueue
NewRetryQueue creates a RetryQueue with sensible defaults.
func (*RetryQueue) CalculateBackoff ¶
func (rq *RetryQueue) CalculateBackoff(attempts int) time.Duration
CalculateBackoff computes exponential backoff with jitter for the given attempt count. The result is capped at BackoffMax.
func (*RetryQueue) Dequeue ¶
func (rq *RetryQueue) Dequeue() *RetryItem
Dequeue returns the highest-priority item whose NextRetry time has passed. Returns nil if no items are ready.
func (*RetryQueue) Enqueue ¶
func (rq *RetryQueue) Enqueue(operation string, args map[string]interface{}, err string, priority int) *RetryItem
Enqueue adds an operation to the retry queue. If an identical operation+args combination is already queued, it increments the attempt count instead of creating a duplicate entry.
func (*RetryQueue) FormatQueue ¶
func (rq *RetryQueue) FormatQueue() string
FormatQueue returns a human-readable representation of the retry queue.
func (*RetryQueue) GetPending ¶
func (rq *RetryQueue) GetPending() []*RetryItem
GetPending returns all items that are still pending or retrying.
func (*RetryQueue) GetReady ¶
func (rq *RetryQueue) GetReady() []*RetryItem
GetReady returns all items that are ready to be retried now.
func (*RetryQueue) MarkFailed ¶
func (rq *RetryQueue) MarkFailed(id string, err string)
MarkFailed records a failure for an item. If the item has reached its max attempts, it is marked as permanently failed. Otherwise, the next retry time is recalculated with exponential backoff.
func (*RetryQueue) MarkSuccess ¶
func (rq *RetryQueue) MarkSuccess(id string)
MarkSuccess marks an item as successfully completed.
func (*RetryQueue) Prune ¶
func (rq *RetryQueue) Prune()
Prune removes succeeded and permanently failed items that are older than 1 hour.
func (*RetryQueue) Size ¶
func (rq *RetryQueue) Size() int
Size returns the total number of items in the queue.
type RetryStrategy ¶
type RetryStrategy struct {
Provider string
BaseDelay time.Duration
MaxDelay time.Duration
MaxRetries int
BackoffMultiplier float64
JitterPct float64 // 0-100, adds randomness
RetryOn []string // error patterns to retry on
AbortOn []string // error patterns to immediately fail on
}
RetryStrategy defines the retry behavior for a specific provider.
type SmartRetry ¶
type SmartRetry struct {
Strategies map[string]*RetryStrategy
FailureHistory []FailureRecord
MaxHistorySize int
// contains filtered or unexported fields
}
SmartRetry provides intelligent retry handling that learns from failure patterns and adapts retry strategies per-provider.
func NewSmartRetry ¶
func NewSmartRetry() *SmartRetry
NewSmartRetry creates a SmartRetry with default strategies per provider.
func (*SmartRetry) AdaptStrategy ¶
func (sr *SmartRetry) AdaptStrategy(provider string)
AdaptStrategy learns from failure history and adjusts the strategy for a provider.
func (*SmartRetry) CalculateDelay ¶
func (sr *SmartRetry) CalculateDelay(strategy *RetryStrategy, attempt int) time.Duration
CalculateDelay computes the delay for a given attempt using exponential backoff with jitter.
func (*SmartRetry) ClassifyError ¶
func (sr *SmartRetry) ClassifyError(err error) string
ClassifyError categorizes an error into a known error type.
func (*SmartRetry) ConfigureFromProvider ¶
func (sr *SmartRetry) ConfigureFromProvider(provider string, cfg RetryConfigFromProvider)
ConfigureFromProvider sets the retry strategy for a provider from a config source.
func (*SmartRetry) Decide ¶
func (sr *SmartRetry) Decide(provider, model string, err error, attempt int) *RetryDecision
Decide evaluates whether a request should be retried based on the provider, error, and attempt count. It classifies the error, checks strategy, calculates delay with jitter, and may suggest fallback providers.
func (*SmartRetry) FormatStatus ¶
func (sr *SmartRetry) FormatStatus() string
FormatStatus returns a human-readable summary of provider health.
func (*SmartRetry) GetProviderHealth ¶
func (sr *SmartRetry) GetProviderHealth() map[string]string
GetProviderHealth returns the health status of all known providers.
func (*SmartRetry) RecordRecovery ¶
func (sr *SmartRetry) RecordRecovery(provider, model string, recoveryDelay time.Duration)
RecordRecovery marks that a provider has recovered from failures.
func (*SmartRetry) ResetProvider ¶
func (sr *SmartRetry) ResetProvider(provider string)
ResetProvider clears failure history for a specific provider.
func (*SmartRetry) ShouldFallback ¶
func (sr *SmartRetry) ShouldFallback(provider string) (bool, string)
ShouldFallback checks if a provider has too many recent failures and suggests an alternative.