Documentation
¶
Overview ¶
Package retry provides retry-queue and smart-retry types for the engine package. See ../REFACTOR_PLAN.md.
Note: hawk also has a top-level `github.com/GrayCodeAI/hawk/internal/resilience/retry` package for low-level HTTP/transport retry. This sub-package is specifically the engine's higher-level retry queue (work items deferred for later attempt).
Index ¶
- func TempEscalationRetry(ctx context.Context, maxRetries int, ...) (string, error)
- type FailureRecord
- type Item
- type Queue
- type RetryConfigFromProvider
- type RetryDecision
- type RetryItem
- type RetryQueue
- func (rq *RetryQueue) CalculateBackoff(attempts int) time.Duration
- func (rq *RetryQueue) Clear()
- func (rq *RetryQueue) Dequeue() *RetryItem
- func (rq *RetryQueue) Enqueue(operation string, args map[string]interface{}, err string, priority int) *RetryItem
- func (rq *RetryQueue) FormatQueue() string
- func (rq *RetryQueue) GetPending() []*RetryItem
- func (rq *RetryQueue) GetReady() []*RetryItem
- func (rq *RetryQueue) MarkFailed(id string, err string)
- func (rq *RetryQueue) MarkSuccess(id string)
- func (rq *RetryQueue) Prune()
- func (rq *RetryQueue) Size() int
- type RetryStrategy
- type SmartRetry
- func (sr *SmartRetry) AdaptStrategy(provider string)
- func (sr *SmartRetry) CalculateDelay(strategy *RetryStrategy, attempt int) time.Duration
- func (sr *SmartRetry) ClassifyError(err error) string
- func (sr *SmartRetry) ConfigureFromProvider(provider string, cfg RetryConfigFromProvider)
- func (sr *SmartRetry) Decide(provider, model string, err error, attempt int) *RetryDecision
- func (sr *SmartRetry) FormatStatus() string
- func (sr *SmartRetry) GetProviderHealth() map[string]string
- func (sr *SmartRetry) RecordRecovery(provider, model string, recoveryDelay time.Duration)
- func (sr *SmartRetry) ResetProvider(provider string)
- func (sr *SmartRetry) ShouldFallback(provider string) (bool, string)
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func TempEscalationRetry ¶
func TempEscalationRetry( ctx context.Context, maxRetries int, attemptFn func(ctx context.Context, temp float64) (string, error), ) (string, error)
TempEscalationRetry calls attemptFn at temperature 0.0 first. If the result is empty or only whitespace, it retries up to maxRetries more times, linearly escalating the temperature toward 1.0 on each retry.
This is the Kimi-Dev pattern for stuck code-repair LLM calls: low temperature first to get deterministic output; escalate if the model stalls so it can explore less obvious repairs.
Temperature schedule (examples):
maxRetries=1 -> [0.0, 1.0] maxRetries=2 -> [0.0, 0.5, 1.0] maxRetries=3 -> [0.0, 0.33, 0.67, 1.0] maxRetries=4 -> [0.0, 0.25, 0.50, 0.75, 1.0]
If maxRetries <= 0 the function is called exactly once at temperature 0.0.
The function returns the first non-empty result. If every attempt returns an empty result, the last error (if any) is returned; otherwise a nil error is returned with the empty string (callers may treat empty as a signal to fall back to a different strategy).
Context cancellation is respected between attempts: if ctx is already done before an attempt, ctx.Err() is returned immediately.
Types ¶
type FailureRecord ¶
type FailureRecord struct {
Provider string
Model string
ErrorType string // "rate_limit", "timeout", "server_error", "invalid_request", "auth"
ErrorMsg string
Timestamp time.Time
RetryCount int
Recovered bool
RecoveryDelay time.Duration
}
FailureRecord captures information about a failure for learning.
type Queue ¶
type Queue = RetryQueue
Queue is the FIFO of pending retry items with backoff and dedup.
type RetryConfigFromProvider ¶
type RetryConfigFromProvider interface {
BaseDelayMs() int
MaxDelayMs() int
MaxRetries() int
BackoffMultiplier() float64
JitterPct() int
}
RetryConfigFromProvider is the interface for provider retry configuration.
type RetryDecision ¶
type RetryDecision struct {
ShouldRetry bool
Delay time.Duration
Reason string
FallbackProvider string // switch provider if this one is failing
FallbackModel string
}
RetryDecision describes what the caller should do after a failure.
type RetryItem ¶
type RetryItem struct {
ID string
Operation string
Args map[string]interface{}
Error string
Attempts int
MaxAttempts int
NextRetry time.Time
Priority int
CreatedAt time.Time
Status string // "pending", "retrying", "succeeded", "failed_permanent"
}
RetryItem represents a single operation queued for retry.
type RetryQueue ¶
type RetryQueue struct {
Items []*RetryItem
MaxSize int
BackoffBase time.Duration
BackoffMax time.Duration
// contains filtered or unexported fields
}
RetryQueue manages failed operations with exponential backoff, priority ordering, and deduplication of identical operations.
func NewRetryQueue ¶
func NewRetryQueue() *RetryQueue
NewRetryQueue creates a RetryQueue with sensible defaults.
func (*RetryQueue) CalculateBackoff ¶
func (rq *RetryQueue) CalculateBackoff(attempts int) time.Duration
CalculateBackoff computes exponential backoff with jitter for the given attempt count. The result is capped at BackoffMax.
func (*RetryQueue) Dequeue ¶
func (rq *RetryQueue) Dequeue() *RetryItem
Dequeue returns the highest-priority item whose NextRetry time has passed. Returns nil if no items are ready.
func (*RetryQueue) Enqueue ¶
func (rq *RetryQueue) Enqueue(operation string, args map[string]interface{}, err string, priority int) *RetryItem
Enqueue adds an operation to the retry queue. If an identical operation+args combination is already queued, it increments the attempt count instead of creating a duplicate entry.
func (*RetryQueue) FormatQueue ¶
func (rq *RetryQueue) FormatQueue() string
FormatQueue returns a human-readable representation of the retry queue.
func (*RetryQueue) GetPending ¶
func (rq *RetryQueue) GetPending() []*RetryItem
GetPending returns all items that are still pending or retrying.
func (*RetryQueue) GetReady ¶
func (rq *RetryQueue) GetReady() []*RetryItem
GetReady returns all items that are ready to be retried now.
func (*RetryQueue) MarkFailed ¶
func (rq *RetryQueue) MarkFailed(id string, err string)
MarkFailed records a failure for an item. If the item has reached its max attempts, it is marked as permanently failed. Otherwise, the next retry time is recalculated with exponential backoff.
func (*RetryQueue) MarkSuccess ¶
func (rq *RetryQueue) MarkSuccess(id string)
MarkSuccess marks an item as successfully completed.
func (*RetryQueue) Prune ¶
func (rq *RetryQueue) Prune()
Prune removes succeeded and permanently failed items that are older than 1 hour.
func (*RetryQueue) Size ¶
func (rq *RetryQueue) Size() int
Size returns the total number of items in the queue.
type RetryStrategy ¶
type RetryStrategy struct {
Provider string
BaseDelay time.Duration
MaxDelay time.Duration
MaxRetries int
BackoffMultiplier float64
JitterPct float64 // 0-100, adds randomness
RetryOn []string // error patterns to retry on
AbortOn []string // error patterns to immediately fail on
}
RetryStrategy defines the retry behavior for a specific provider.
type SmartRetry ¶
type SmartRetry struct {
Strategies map[string]*RetryStrategy
FailureHistory []FailureRecord
MaxHistorySize int
// contains filtered or unexported fields
}
SmartRetry provides intelligent retry handling that learns from failure patterns and adapts retry strategies per-provider.
func NewSmartRetry ¶
func NewSmartRetry() *SmartRetry
NewSmartRetry creates a SmartRetry with default strategies per provider.
func (*SmartRetry) AdaptStrategy ¶
func (sr *SmartRetry) AdaptStrategy(provider string)
AdaptStrategy learns from failure history and adjusts the strategy for a provider.
func (*SmartRetry) CalculateDelay ¶
func (sr *SmartRetry) CalculateDelay(strategy *RetryStrategy, attempt int) time.Duration
CalculateDelay computes the delay for a given attempt using exponential backoff with jitter.
func (*SmartRetry) ClassifyError ¶
func (sr *SmartRetry) ClassifyError(err error) string
ClassifyError categorizes an error into a known error type.
func (*SmartRetry) ConfigureFromProvider ¶
func (sr *SmartRetry) ConfigureFromProvider(provider string, cfg RetryConfigFromProvider)
ConfigureFromProvider sets the retry strategy for a provider from a config source.
func (*SmartRetry) Decide ¶
func (sr *SmartRetry) Decide(provider, model string, err error, attempt int) *RetryDecision
Decide evaluates whether a request should be retried based on the provider, error, and attempt count. It classifies the error, checks strategy, calculates delay with jitter, and may suggest fallback providers.
func (*SmartRetry) FormatStatus ¶
func (sr *SmartRetry) FormatStatus() string
FormatStatus returns a human-readable summary of provider health.
func (*SmartRetry) GetProviderHealth ¶
func (sr *SmartRetry) GetProviderHealth() map[string]string
GetProviderHealth returns the health status of all known providers.
func (*SmartRetry) RecordRecovery ¶
func (sr *SmartRetry) RecordRecovery(provider, model string, recoveryDelay time.Duration)
RecordRecovery marks that a provider has recovered from failures.
func (*SmartRetry) ResetProvider ¶
func (sr *SmartRetry) ResetProvider(provider string)
ResetProvider clears failure history for a specific provider.
func (*SmartRetry) ShouldFallback ¶
func (sr *SmartRetry) ShouldFallback(provider string) (bool, string)
ShouldFallback checks if a provider has too many recent failures and suggests an alternative.
Source Files