resile

package module
v1.0.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 6, 2026 License: MIT Imports: 11 Imported by: 0

README

Resile: Ergonomic Execution Resilience for Go

Go Reference License Build Status Codecov Go Report Card YouTube Dev.to

Resile is a resilience and retry library for Go, inspired by Python's stamina. It provides a type-safe, ergonomic, and highly observable way to handle transient failures in distributed systems.


See Resile in Action

Resile Demo Video

Subscribe to our YouTube channel for deep dives and tutorials.


The "Aha!" Snippet

Resile allows you to compose complex resilience strategies with a single, type-safe call. No interface{} casting, no reflection—just clean Go.

user, err := resile.Do(ctx, func(ctx context.Context) (*User, error) {
    return client.GetUser(ctx, id)
},
    resile.WithBulkhead(10),            // Max 10 concurrent requests
    resile.WithCircuitBreaker(cb),      // Stop if failure rate > 50%
    resile.WithRetry(3),                // 3 attempts with AWS Full Jitter
    resile.WithTimeout(1*time.Second),   // Each attempt max 1s
    resile.WithFallback(func(ctx context.Context, err error) (*User, error) {
        return cache.Get(id), nil       // Return stale data on failure
    }),
)

Table of Contents


Installation

go get github.com/cinar/resile

Requires Go 1.24+ (uses generics and errors.Join).


Why Resile?

In distributed systems, transient failures are a mathematical certainty. Resile simplifies the "Correct Way" to retry:

  • AWS Full Jitter: Uses the industry-standard algorithm to prevent "thundering herd" synchronization.
  • Adaptive Retries: Built-in token bucket rate limiting to prevent "retry storms" across a cluster.
  • Generic-First: No interface{} or reflection. Full compile-time type safety.
  • Context-Aware: Strictly respects context.Context cancellation and deadlines.
  • Zero-Dependency Core: The core resile, circuit, and chaos packages depend only on the Go standard library. Telemetry adapters (telemetry/) are opt-in.
  • Opinionated Defaults: Sensible defaults (5 attempts, exponential backoff).
  • Chaos-Ready: Built-in support for fault and latency injection to test your resilience policies.

Articles & Tutorials

Want to learn more about the philosophy behind Resile and advanced resilience patterns in Go? Check out these deep dives:

Also, check out our Dev.to space for more articles and discussions.


Examples

The examples/ directory contains standalone programs showing how to use Resile in various scenarios:


API Reference

Function Signature Description
Do[T] (ctx, action, ...Option) (T, error) Retry an action that returns a value.
DoErr (ctx, action, ...Option) error Retry an action that returns only an error.
DoState[T] (ctx, action, ...Option) (T, error) Like Do, with access to RetryState.
DoErrState (ctx, action, ...Option) error Like DoErr, with access to RetryState.
DoHedged[T] (ctx, action, ...Option) (T, error) Speculative retries for tail latency.
DoErrHedged (ctx, action, ...Option) error Error-only hedging variant.
DoStateHedged[T] (ctx, action, ...Option) (T, error) Stateful hedging (e.g., endpoint rotation).
DoErrStateHedged (ctx, action, ...Option) error Error-only stateful hedging.
New (...Option) Retryer Create a reusable Retryer instance.
NewPolicy (...Option) *Policy Create a composed resilience policy.
NewStateMachine[S,D,E] (S, D, TransitionFunc, ...Option) *StateMachine Create a resilient state machine.
FatalError (error) error Mark an error as non-retryable.
WithTestingBypass (context.Context) context.Context Skip all backoff delays in tests.
InjectDeadlineHeader (ctx, Header, string) Propagate deadline to outgoing requests.

Resilience Cookbook

1. Simple Retries

The Problem: A database connection or network request might fail intermittently due to transient blips.

The Recipe: Retry a simple operation that only returns an error. If all retries fail, Resile returns an aggregated error containing the failures from every attempt.

err := resile.DoErr(ctx, func(ctx context.Context) error {
    return db.PingContext(ctx)
})
2. Value-Yielding Retries (Generics)

The Problem: Fetching data from a microservice needs to be resilient and type-safe without boilerplate casting.

The Recipe: Fetch data with full type safety. The return type is inferred from your closure.

// val is automatically inferred as *User
user, err := resile.Do(ctx, func(ctx context.Context) (*User, error) {
    return apiClient.GetUser(ctx, userID)
}, resile.WithMaxAttempts(3))
3. Request Hedging (Speculative Retries)

The Problem: Long-tail latency (the 99th percentile) slows down your entire system even when most requests are fast.

The Recipe: Speculative retries reduce tail latency by starting a second request if the first one doesn't finish within a configured HedgingDelay. The first successful result is used, and the other is cancelled.

// For value-yielding operations
data, err := resile.DoHedged(ctx, action, 
    resile.WithMaxAttempts(3),
    resile.WithHedgingDelay(100 * time.Millisecond),
)

// For error-only operations
err := resile.DoErrHedged(ctx, action,
    resile.WithMaxAttempts(2),
    resile.WithHedgingDelay(50 * time.Millisecond),
)

Stateful hedging variants are also available for endpoint rotation with speculative retries:

data, err := resile.DoStateHedged(ctx, func(ctx context.Context, state resile.RetryState) (string, error) {
    url := endpoints[state.Attempt % uint(len(endpoints))]
    return client.Get(ctx, url)
}, resile.WithHedgingDelay(100 * time.Millisecond))

Read more: Beating Tail Latency: A Guide to Request Hedging in Go Microservices

4. Stateful Retries & Endpoint Rotation

The Problem: Retrying against the same failing endpoint is futile; you need to cycle through a list of healthy hosts.

The Recipe: Use DoState (or DoErrState) to access the RetryState, allowing you to rotate endpoints or fallback logic based on the failure history.

endpoints := []string{"api-v1.example.com", "api-v2.example.com"}

data, err := resile.DoState(ctx, func(ctx context.Context, state resile.RetryState) (string, error) {
    // Rotate endpoint based on attempt number
    url := endpoints[state.Attempt % uint(len(endpoints))]
    return client.Get(ctx, url)
})

Read more: Self-Healing State Machines: Resilient State Transitions in Go

5. Handling Rate Limits (Retry-After)

The Problem: Downstream services may return 429 Too Many Requests with a specific time you must wait.

The Recipe: Resile automatically detects if an error implements RetryAfterError. It can override the jittered backoff with a server-dictated duration.

type RateLimitError struct {
    WaitUntil time.Time
}

func (e *RateLimitError) Error() string { return "too many requests" }
func (e *RateLimitError) RetryAfter() time.Duration {
    return time.Until(e.WaitUntil)
}
func (e *RateLimitError) CancelAllRetries() bool {
    return false 
}

// Resile will sleep exactly until WaitUntil when this error is encountered.
6. Aborting Retries (Pushback Signal)

The Problem: Some errors are terminal (e.g., "Quota Exceeded") and retrying them just wastes resources and adds latency.

The Recipe: Implement CancelAllRetries() bool to abort the entire retry loop immediately.

type QuotaExceededError struct{}
func (e *QuotaExceededError) Error() string { return "quota exhausted" }
func (e *QuotaExceededError) CancelAllRetries() bool { return true }

// Resile will stop immediately if this error is encountered.
_, err := resile.Do(ctx, action, resile.WithMaxAttempts(10))
7. Fallback Strategies

The Problem: When a service is down, your application should degrade gracefully instead of failing completely.

The Recipe: Provide a fallback function to return stale data from a cache or a sensible default value when all attempts fail.

data, err := resile.Do(ctx, fetchData,
    resile.WithMaxAttempts(3),
    resile.WithFallback(func(ctx context.Context, err error) (string, error) {
        return cache.Get(ctx, key), nil 
    }),
)
8. Bulkhead Pattern

The Problem: One slow or failing resource (like a specific database) consumes all available goroutines, causing a "cascading failure" across the whole app.

The Recipe: Isolate failures by limiting the number of concurrent executions to a specific resource.

// Shared bulkhead with capacity of 10
bh := resile.NewBulkhead(10)

err := resile.DoErr(ctx, action, resile.WithBulkheadInstance(bh))

Read more: Stop the Domino Effect: Bulkhead Isolation in Go

9. Priority-Aware Bulkhead

The Problem: During high load, you want to ensure critical user requests succeed even if background tasks are dropped.

The Recipe: Implement load shedding based on traffic priority to protect critical paths during saturation.

thresholds := map[resile.Priority]float64{
    resile.PriorityLow:      0.5, // Shed when >50% full
    resile.PriorityStandard: 0.8, // Shed when >80% full
    resile.PriorityCritical: 1.0, // Allow until 100% full
}

err := resile.DoErr(ctx, action, 
    resile.WithPriorityBulkhead(20, thresholds),
)

Read more: Prioritize Your Traffic: Priority-Aware Bulkheads in Go

10. Rate Limiting Pattern

The Problem: Your application must strictly adhere to an external API's rate limits (e.g., 100 requests/sec).

The Recipe: Control the rate of executions using a time-based token bucket.

// Limit to 100 requests per second
rl := resile.NewRateLimiter(100, time.Second)

err := resile.DoErr(ctx, action, resile.WithRateLimiterInstance(rl))

Read more: Respecting Boundaries: Precise Rate Limiting in Go

11. Layered Defense with Circuit Breaker

The Problem: Retrying against a service experiencing a total outage just adds load and delays failure detection.

The Recipe: Combine retries with a mathematically rigorous sliding window circuit breaker. Resile supports both Count-based and Time-based windows.

import "github.com/cinar/resile/circuit"

// Trip if >50% of the last 100 calls fail.
cb := circuit.New(circuit.Config{
    WindowType:           circuit.WindowCountBased,
    WindowSize:           100,
    FailureRateThreshold: 50.0,
    MinimumCalls:         10,
    ResetTimeout:         30 * time.Second,
})

err := resile.DoErr(ctx, action, resile.WithCircuitBreaker(cb))

Read more: Resilience Beyond Counters: Sliding Window Circuit Breakers in Go

12. Macro-Level Protection (Adaptive Retries)

The Problem: A "retry storm" occurs when hundreds of clients all retry at once, preventing a struggling service from recovering.

The Recipe: Use a token bucket shared across your entire client. If the service degrades, the bucket depletes, causing clients to fail fast locally.

// Share this bucket across multiple executions
bucket := resile.DefaultAdaptiveBucket()

err := resile.DoErr(ctx, action, resile.WithAdaptiveBucket(bucket))

Read more: Preventing Microservice Meltdowns: Adaptive Retries and Circuit Breakers in Go

13. Adaptive Concurrency (TCP-Vegas)

The Problem: Setting static concurrency limits is hard; too high and you crash the server, too low and you waste throughput.

The Recipe: Automatically adjust concurrency limits based on Round-Trip Time (RTT). Resile increases concurrency when latency is stable and decreases it when queuing is detected.

// Shared limiter across multiple calls
al := resile.NewAdaptiveLimiter()

err := resile.DoErr(ctx, action, resile.WithAdaptiveLimiterInstance(al))

Read more: Beyond Static Limits: Adaptive Concurrency with TCP-Vegas in Go

14. Structured Logging & Telemetry

The Problem: You need to know when retries are happening and why, without cluttering your business logic.

The Recipe: Integrate with slog or OpenTelemetry seamlessly.

import "github.com/cinar/resile/telemetry/resileslog"

logger := slog.Default()
resile.Do(ctx, action, 
    resile.WithName("get-inventory"),
    resile.WithInstrumenter(resileslog.New(logger)),
)
import "github.com/cinar/resile/telemetry/resileotel"

instr, err := resileotel.New(tracerProvider, meterProvider)
if err != nil {
    log.Fatal(err)
}

resile.Do(ctx, action,
    resile.WithName("get-inventory"),
    resile.WithInstrumenter(instr),
)

Resile ships two built-in instrumenters: resileslog for structured logging and resileotel for OpenTelemetry traces and metrics.

15. Panic Recovery ("Let It Crash")

The Problem: A single unexpected panic in a request handler can take down an entire process.

The Recipe: Convert Go panics into retryable errors, allowing your application to reset to a known good state.

val, err := resile.Do(ctx, riskyAction, 
    resile.WithPanicRecovery(),
)
16. Fast Unit Testing

The Problem: Waiting for exponential backoff timers in CI/CD pipelines makes tests slow and flaky.

The Recipe: Use WithTestingBypass to make all retries execute instantly during tests.

func TestMyService(t *testing.T) {
    ctx := resile.WithTestingBypass(context.Background())
    err := service.Handle(ctx) // Retries 10 times instantly
}
17. Reusable Clients & Dependency Injection

The Problem: Passing individual retry parameters everywhere leads to inconsistent resilience policies.

The Recipe: Use resile.New() to create a Retryer interface for consistent, reusable strategies.

retryer := resile.New(
    resile.WithMaxAttempts(3),
    resile.WithBaseDelay(200 * time.Millisecond),
)

err := retryer.DoErr(ctx, service.Call)
18. Marking Errors as Fatal

The Problem: Some errors occur deep in your code that you know shouldn't be retried.

The Recipe: Use resile.FatalError() to abort the retry loop immediately from within your action.

err := resile.DoErr(ctx, func(ctx context.Context) error {
    if errors.Is(err, ErrAuthFailed) {
        return resile.FatalError(err)
    }
    return err
})
19. Custom Error Filtering

The Problem: You only want to retry on specific database errors (like ErrConnReset) but not on logic errors.

The Recipe: Control which errors trigger a retry using WithRetryIf or WithRetryIfFunc.

err := resile.DoErr(ctx, action,
    resile.WithRetryIf(ErrConnReset),
    // OR
    resile.WithRetryIfFunc(func(err error) bool {
        return isTransient(err)
    }),
)
20. Policy Composition & Chaining

The Problem: You need a specific execution order for your resilience layers (e.g., Timeout inside the Bulkhead).

The Recipe: Use the Policy API. The order of options in NewPolicy determines the execution hierarchy from outermost to innermost.

standardPolicy := resile.NewPolicy(
    resile.WithBulkhead(20),
    resile.WithCircuitBreaker(cb),
    resile.WithRetry(3),
    resile.WithTimeout(1*time.Second),
)

val, err := standardPolicy.Do(ctx, action)
21. Native Multi-Error Aggregation

The Problem: When a request fails after 5 retries, you usually only see the last error, losing the context of earlier failures.

The Recipe: Resile aggregates the complete timeline of failures using Go 1.20's errors.Join.

err := resile.DoErr(ctx, action, resile.WithMaxAttempts(3))

if multi, ok := err.(interface{ Unwrap() []error }); ok {
    for i, e := range multi.Unwrap() {
        fmt.Printf("Attempt %d failed: %v\n", i+1, e)
    }
}

Read more: Debugging the Timeline: Native Multi-Error Aggregation in Go

22. Native Chaos Engineering (Fault & Latency Injection)

The Problem: You don't know if your resilience policies actually work until a real production outage occurs.

The Recipe: Safely test your policies by synthetically inducing faults and latency into your application logic.

import "github.com/cinar/resile/chaos"

err := resile.DoErr(ctx, action, 
    resile.WithRetry(3),
    resile.WithChaos(chaos.Config{
        ErrorProbability: 0.1, 
        InjectedError:    errors.New("chaos!"),
    }),
)

Read more: Native Chaos Engineering: Testing Resilience with Fault & Latency Injection

23. Distributed Deadline Propagation

The Problem: A request times out at the gateway, but downstream microservices keep working on it, wasting CPU and memory.

The Recipe: Stop "zombie requests" by propagating remaining time budget across service boundaries.

// Early Abort if less than 10ms remains
_, err := resile.Do(ctx, action, 
    resile.WithMinDeadlineThreshold(10 * time.Millisecond),
)

// Inject into HTTP headers
resile.InjectDeadlineHeader(ctx, req.Header, "X-Request-Timeout")

Read more: Stopping the Zombie Requests: Distributed Deadline Propagation in Go

24. Reliable File Downloads (HTTP Resumption)

The Problem: Downloading a 1GB file fails at 900MB; starting over from zero is wasteful.

The Recipe: Combine DoErr with HTTP Range headers to resume downloads from the last successful byte.

var bytesReceived int64
err := resile.DoErr(ctx, func(ctx context.Context) error {
    if bytesReceived > 0 {
        req.Header.Set("Range", fmt.Sprintf("bytes=%d-", bytesReceived))
    }
    // ... do request and io.Copy ...
    bytesReceived += n
    return err
}, resile.WithMaxAttempts(10))

Read more: Reliable File Downloads with HTTP Range Resumption

25. SQL Resilience

The Problem: Databases are critical yet vulnerable to transient network errors and failovers.

The Recipe: Wrap standard database/sql calls with retries and a circuit breaker to protect against both blips and systemic outages.

_, err := resile.Do(ctx, func(ctx context.Context) (sql.Result, error) {
    return db.ExecContext(ctx, "UPDATE users SET active = ? WHERE id = ?", true, 42)
},
    resile.WithRetry(3),
    resile.WithCircuitBreaker(breaker),
)

Read more: Building Bulletproof Database Clients in Go: SQL Resilience with Resile

26. Redis Resilience

The Problem: Database connection pools (SQL or NoSQL like Redis) can be exhausted when the database slows down, leading to cascading failures.

The Recipe: Combine retries for transient blips with a shared bulkhead to strictly limit the number of concurrent operations hitting the connection pool.

// 1. Create a shared bulkhead matching your pool size
redisBulkhead := resile.NewBulkhead(20)

// 2. Wrap your Redis or SQL calls
val, err := resile.Do(ctx, func(ctx context.Context) (string, error) {
    return rdb.Get(ctx, "key").Result()
},
    resile.WithMaxAttempts(3),
    resile.WithBulkheadInstance(redisBulkhead),
)

Read more: Reliable Redis: Combining Retries and Bulkheads for Rock-Solid Caching


Built on Hyperscaler Research

Resile isn't just a collection of wrappers; it implements proven resilience algorithms used by the world's largest engineering organizations:

  • AWS Architecture: Exponential Backoff and Jitter (Full Jitter) to solve the thundering herd problem.
  • Google SRE: Adaptive Throttling logic to implement client-side rejection and protect degraded downstream services.
  • Uber Engineering: Adaptive Concurrency based on TCP-Vegas/Little's Law to dynamically adjust load without static limits.
  • Netflix: Bulkhead and Circuit Breaker patterns popularized by Hystrix, refined for modern Go concurrency.

Configuration Reference

Option Description Default
WithName(string) Identifies the operation in logs/metrics. ""
WithMaxAttempts(uint) Total number of attempts (initial + retries). 5
WithRetry(uint) Alias for WithMaxAttempts that adds a retry policy. -
WithBaseDelay(duration) Initial backoff duration. 100ms
WithMaxDelay(duration) Maximum possible backoff duration. 30s
WithBackoff(Backoff) Custom backoff algorithm (e.g. constant). Full Jitter
WithHedgingDelay(duration) Delay before speculative retries. 0
WithRetryIf(error) Only retry if errors.Is(err, target). All non-fatal
WithRetryIfFunc(func) Custom logic to decide if an error is retriable. nil
WithCircuitBreaker(cb) Attaches a circuit breaker state machine. nil
WithBulkhead(uint) Limits concurrent executions. nil
WithBulkheadInstance(b) Attaches a shared bulkhead instance. nil
WithPriorityBulkhead(uint, map) Limits concurrency based on priority. nil
WithPriorityBulkheadInstance(b) Attaches a shared priority bulkhead. nil
WithRateLimiter(limit, interval) Limits execution rate (token bucket). nil
WithRateLimiterInstance(rl) Attaches a shared rate limiter instance. nil
WithTimeout(duration) Sets an execution timeout for the operation. 0
WithAdaptiveBucket(b) Attaches a token bucket for adaptive retries. nil
WithAdaptiveLimiter() Creates an adaptive concurrency limiter (TCP-Vegas). nil
WithAdaptiveLimiterInstance(al) Attaches a shared adaptive concurrency limiter. nil
WithInstrumenter(inst) Attaches telemetry (slog/OTel/Prometheus). nil
WithFallback(f) Sets a generic fallback function. nil
WithFallbackErr(f) Sets a fallback function for error-only actions. nil
WithPanicRecovery() Enables "Let It Crash" panic handling. false
WithChaos(chaos.Config) Integrates a chaos injector for fault/latency injection. nil
WithMinDeadlineThreshold(d) Min remaining time required to start an attempt. 5ms

Sentinel Errors

Resile exports typed sentinel errors for programmatic error handling:

Error Returned When
ErrBulkheadFull Bulkhead capacity is reached and the request cannot be admitted.
ErrShedLoad Priority bulkhead sheds traffic below the priority threshold.
ErrRateLimitExceeded Rate limiter blocks execution due to quota exhaustion.

All sentinel errors support errors.Is() for matching.


Architecture & Design

Resile is built for high-performance, concurrent applications:

  • Memory Safety: Uses time.NewTimer with proper cleanup to prevent memory leaks in long-running loops.
  • Context Integrity: Every internal sleep is a select between the timer and ctx.Done().
  • Zero Allocations: Core execution loop is designed to be allocation-efficient.
  • Errors are Values: Leverage standard errors.Is and errors.As for all policy decisions.
  • Health Monitoring: Policy.Health() returns a channel of circuit.StateEvent for real-time circuit breaker state changes, enabling dashboard integration.
  • Benchmark-Proven: Policy execution runs in ~103 ns/op with 0 allocations; DoErr in ~561 ns/op (go test -bench . -benchmem).

Community


Acknowledgements

  • AWS Architecture Blog: For the definitive Exponential Backoff and Jitter algorithm (Full Jitter).
  • Stamina & Tenacity: For pioneering ergonomic retry APIs in the Python ecosystem that inspired the design of Resile.

License

Resile is released under the MIT License.

Documentation

Index

Constants

This section is empty.

Variables

View Source
var ErrBulkheadFull = errors.New("bulkhead capacity reached")

ErrBulkheadFull is returned when the bulkhead capacity is reached.

View Source
var ErrRateLimitExceeded = errors.New("rate limit exceeded")

ErrRateLimitExceeded is returned when the rate limit is exceeded.

View Source
var ErrShedLoad = errors.New("load shed: limit reached")

ErrShedLoad is returned when the adaptive concurrency or priority bulkhead limit is reached.

Functions

func Do

func Do[T any](ctx context.Context, action func(context.Context) (T, error), opts ...Option) (T, error)

Do executes an action with retry logic using the provided options. This generic function handles functions returning (T, error).

func DoErr

func DoErr(ctx context.Context, action func(context.Context) error, opts ...Option) error

DoErr executes an action with retry logic using the provided options. This function handles functions returning only error.

func DoErrHedged

func DoErrHedged(ctx context.Context, action func(context.Context) error, opts ...Option) error

DoErrHedged executes an action using speculative retries (hedging).

func DoErrState

func DoErrState(ctx context.Context, action func(context.Context, RetryState) error, opts ...Option) error

DoErrState executes a stateful action with retry logic using the provided options.

func DoErrStateHedged

func DoErrStateHedged(ctx context.Context, action func(context.Context, RetryState) error, opts ...Option) error

DoErrStateHedged executes a stateful action with speculative retries (hedging).

func DoHedged

func DoHedged[T any](ctx context.Context, action func(context.Context) (T, error), opts ...Option) (T, error)

DoHedged executes an action using speculative retries (hedging). It starts multiple attempts concurrently if previous ones take too long.

func DoState

func DoState[T any](ctx context.Context, action func(context.Context, RetryState) (T, error), opts ...Option) (T, error)

DoState executes a stateful action with retry logic using the provided options. The RetryState is passed to the closure, allowing it to adapt to failure history.

func DoStateHedged

func DoStateHedged[T any](ctx context.Context, action func(context.Context, RetryState) (T, error), opts ...Option) (T, error)

DoStateHedged executes a stateful action with speculative retries (hedging).

func FatalError

func FatalError(err error) error

FatalError wraps an error to indicate that the retry loop should terminate immediately.

func InjectDeadlineHeader added in v1.0.5

func InjectDeadlineHeader(ctx context.Context, header Header, key string)

InjectDeadlineHeader calculates the remaining time in the context and injects it into the provided header. This is useful for distributed deadline propagation. If the key is "Grpc-Timeout", it uses the gRPC-specific timeout format. Otherwise, it injects the remaining milliseconds as a string.

func WithPriority added in v1.0.6

func WithPriority(ctx context.Context, p Priority) context.Context

WithPriority attaches a Priority to the context.

func WithTestingBypass

func WithTestingBypass(ctx context.Context) context.Context

WithTestingBypass returns a new context that signals the retry loop to skip all sleep delays. This is intended for use in unit tests to prevent CI pipelines from being slowed down by backoff.

Types

type AdaptiveBucket

type AdaptiveBucket struct {
	// contains filtered or unexported fields
}

AdaptiveBucket implements a client-side token bucket rate limiter for retries. It protects downstream services from thundering herds by depleting tokens when retries occur and refilling them on successful responses. This allows a fleet of clients to quickly cut off traffic to a degraded system, providing macro-level protection.

func DefaultAdaptiveBucket

func DefaultAdaptiveBucket() *AdaptiveBucket

DefaultAdaptiveBucket creates a new AdaptiveBucket with standard defaults (max capacity: 500, retry cost: 5, success refill: 1).

func NewAdaptiveBucket

func NewAdaptiveBucket(maxCapacity, retryCost, successRefill float64) *AdaptiveBucket

NewAdaptiveBucket creates a new AdaptiveBucket with the specified configuration. A common AWS-style configuration is maxCapacity=500, retryCost=5, successRefill=1.

func (*AdaptiveBucket) AcquireRetryToken

func (b *AdaptiveBucket) AcquireRetryToken() bool

AcquireRetryToken attempts to consume a token for a retry. Returns true if a token was successfully consumed, false if the bucket is empty.

func (*AdaptiveBucket) AddSuccessToken

func (b *AdaptiveBucket) AddSuccessToken()

AddSuccessToken adds a fraction of a token back to the bucket upon a successful response.

type AdaptiveLimiter added in v1.0.4

type AdaptiveLimiter struct {
	// contains filtered or unexported fields
}

AdaptiveLimiter implements a dynamic concurrency limiter using Additive Increase, Multiplicative Decrease (AIMD) based on Round Trip Time (RTT), applying Little's Law.

func NewAdaptiveLimiter added in v1.0.4

func NewAdaptiveLimiter() *AdaptiveLimiter

NewAdaptiveLimiter creates a new AdaptiveLimiter with default configurations.

func (*AdaptiveLimiter) Execute added in v1.0.4

func (al *AdaptiveLimiter) Execute(ctx context.Context, action func() error) error

Execute wraps an action with adaptive concurrency control.

func (*AdaptiveLimiter) GetMaxConcurrency added in v1.0.4

func (al *AdaptiveLimiter) GetMaxConcurrency() int32

GetMaxConcurrency returns the current maximum concurrency limit.

func (*AdaptiveLimiter) Health added in v1.0.6

func (al *AdaptiveLimiter) Health() <-chan StateEvent

Health returns a channel that emits state change events. The channel is created lazily on first access. The caller should use a select with default to handle non-blocking receive.

type Backoff

type Backoff interface {
	// Next calculates the duration to wait before the specified attempt.
	// The attempt parameter is 0-indexed.
	Next(attempt uint) time.Duration
}

Backoff defines the interface for temporal distribution of retries.

func NewFullJitter

func NewFullJitter(base, cap time.Duration) Backoff

NewFullJitter returns a Backoff implementation using the AWS Full Jitter algorithm. The base duration dictates the initial delay, and the cap defines the absolute maximum.

type Bulkhead added in v1.0.3

type Bulkhead struct {
	// contains filtered or unexported fields
}

Bulkhead limits the number of concurrent executions to prevent overloading a resource.

func NewBulkhead added in v1.0.3

func NewBulkhead(capacity uint) *Bulkhead

NewBulkhead creates a new Bulkhead with the specified capacity.

func (*Bulkhead) Execute added in v1.0.3

func (b *Bulkhead) Execute(ctx context.Context, action func() error) error

Execute wraps an action with bulkhead concurrency control. It returns ErrBulkheadFull if the capacity is reached.

type Config

type Config struct {
	Name                 string
	MaxAttempts          uint
	BaseDelay            time.Duration
	MaxDelay             time.Duration
	HedgingDelay         time.Duration
	Backoff              Backoff
	Policy               *retryPolicy
	Instrumenter         Instrumenter
	CircuitBreaker       *circuit.Breaker
	Fallback             any
	AdaptiveBucket       *AdaptiveBucket
	RecoverPanics        bool
	Bulkhead             *Bulkhead
	PriorityBulkhead     *PriorityBulkhead
	Timeout              time.Duration
	RateLimiter          *RateLimiter
	AdaptiveLimiter      *AdaptiveLimiter
	Chaos                *chaos.Injector
	MinDeadlineThreshold time.Duration
	// contains filtered or unexported fields
}

Config represents the configuration for the retry execution.

func DefaultConfig

func DefaultConfig() *Config

DefaultConfig returns a reasonable production-grade configuration.

func (*Config) Do

func (c *Config) Do(ctx context.Context, action func(context.Context) (any, error)) (any, error)

Do satisfies the Retryer interface. Note: returns any for interface compliance.

func (*Config) DoErr

func (c *Config) DoErr(ctx context.Context, action func(context.Context) error) error

DoErr satisfies the Retryer interface.

func (*Config) DoErrHedged

func (c *Config) DoErrHedged(ctx context.Context, action func(context.Context) error) error

DoErrHedged satisfies the Retryer interface.

func (*Config) DoHedged

func (c *Config) DoHedged(ctx context.Context, action func(context.Context) (any, error)) (any, error)

DoHedged satisfies the Retryer interface.

type Header interface {
	Set(key, value string)
}

Header defines an interface for injecting metadata into outgoing requests. This allows Resile to remain agnostic of the underlying transport (HTTP, gRPC, etc.).

type HealthState added in v1.0.6

type HealthState int

HealthState represents the health state of a resilience component.

const (
	// HealthStateHealthy represents a healthy state where requests are allowed.
	HealthStateHealthy HealthState = iota
	// HealthStateDegraded represents a degraded but functional state.
	HealthStateDegraded
	// HealthStateUnhealthy represents an unhealthy state (e.g., circuit open).
	HealthStateUnhealthy
)

type Instrumenter

type Instrumenter interface {
	// BeforeAttempt is called before each execution attempt.
	// It can return a new context (e.g., to inject trace spans).
	BeforeAttempt(ctx context.Context, state RetryState) context.Context
	// AfterAttempt is called after each execution attempt.
	AfterAttempt(ctx context.Context, state RetryState)
	// OnBulkheadFull is called when the bulkhead capacity is reached.
	OnBulkheadFull(ctx context.Context, state RetryState)
	// OnRateLimitExceeded is called when the rate limit is exceeded.
	OnRateLimitExceeded(ctx context.Context, state RetryState)
}

Instrumenter defines the lifecycle hooks for monitoring retry executions. It is a zero-dependency interface to allow custom implementations for logging, metrics, and tracing.

type Option

type Option func(*Config)

Option defines a functional option for configuring a retry execution.

func WithAdaptiveBucket

func WithAdaptiveBucket(bucket *AdaptiveBucket) Option

WithAdaptiveBucket sets a token bucket for adaptive retries. The bucket should be shared across multiple executions to protect downstream services globally.

func WithAdaptiveLimiter added in v1.0.4

func WithAdaptiveLimiter() Option

WithAdaptiveLimiter integrates an adaptive concurrency limiter into the execution.

func WithAdaptiveLimiterInstance added in v1.0.4

func WithAdaptiveLimiterInstance(al *AdaptiveLimiter) Option

WithAdaptiveLimiterInstance integrates a shared adaptive concurrency limiter into the execution.

func WithBackoff

func WithBackoff(backoff Backoff) Option

WithBackoff sets a custom backoff algorithm.

func WithBaseDelay

func WithBaseDelay(delay time.Duration) Option

WithBaseDelay sets the initial delay for the backoff algorithm.

func WithBulkhead added in v1.0.3

func WithBulkhead(capacity uint) Option

WithBulkhead integrates a bulkhead into the execution.

func WithBulkheadInstance added in v1.0.3

func WithBulkheadInstance(bh *Bulkhead) Option

WithBulkheadInstance integrates a shared bulkhead into the execution.

func WithChaos added in v1.0.5

func WithChaos(cfg chaos.Config) Option

WithChaos integrates a chaos injector into the execution.

func WithCircuitBreaker

func WithCircuitBreaker(cb *circuit.Breaker) Option

WithCircuitBreaker integrates a circuit breaker into the retry execution.

func WithFallback

func WithFallback[T any](f func(context.Context, error) (T, error)) Option

WithFallback sets a function to be called if all retries are exhausted or if the circuit breaker is open. T must match the return type of the retry action.

func WithFallbackErr

func WithFallbackErr(f func(context.Context, error) error) Option

WithFallbackErr sets a fallback function for operations that only return an error.

func WithHedgingDelay

func WithHedgingDelay(delay time.Duration) Option

WithHedgingDelay sets the delay for speculative retries (hedging). If a response doesn't arrive within this delay, another attempt is started concurrently.

func WithInstrumenter

func WithInstrumenter(instr Instrumenter) Option

WithInstrumenter sets a telemetry instrumenter.

func WithMaxAttempts

func WithMaxAttempts(attempts uint) Option

WithMaxAttempts sets the maximum number of execution attempts.

func WithMaxDelay

func WithMaxDelay(delay time.Duration) Option

WithMaxDelay sets the maximum delay for the backoff algorithm.

func WithMinDeadlineThreshold added in v1.0.5

func WithMinDeadlineThreshold(threshold time.Duration) Option

WithMinDeadlineThreshold sets the minimum remaining time required to start a new attempt. If the context's deadline is sooner than this threshold, the execution is aborted early.

func WithName

func WithName(name string) Option

WithName sets the name for the operation. This is used in telemetry labels.

func WithPanicRecovery added in v1.0.1

func WithPanicRecovery() Option

WithPanicRecovery enables recovering from panics during execution. If a panic occurs, it is converted into a PanicError and treated as a retryable error.

func WithPriorityBulkhead added in v1.0.6

func WithPriorityBulkhead(capacity uint, thresholds map[Priority]float64) Option

WithPriorityBulkhead integrates a priority-aware bulkhead into the execution.

func WithPriorityBulkheadInstance added in v1.0.6

func WithPriorityBulkheadInstance(bh *PriorityBulkhead) Option

WithPriorityBulkheadInstance integrates a shared priority-aware bulkhead into the execution.

func WithRateLimiter added in v1.0.3

func WithRateLimiter(limit float64, interval time.Duration) Option

WithRateLimiter integrates a rate limiter into the execution.

func WithRateLimiterInstance added in v1.0.3

func WithRateLimiterInstance(rl *RateLimiter) Option

WithRateLimiterInstance integrates a shared rate limiter into the execution.

func WithRetry added in v1.0.3

func WithRetry(attempts uint) Option

WithRetry sets the maximum number of execution attempts and adds a retry policy to the pipeline.

func WithRetryIf

func WithRetryIf(target error) Option

WithRetryIf sets a specific error to trigger a retry.

func WithRetryIfFunc

func WithRetryIfFunc(f func(error) bool) Option

WithRetryIfFunc sets a custom function to determine if an error should be retried.

func WithTimeout added in v1.0.3

func WithTimeout(timeout time.Duration) Option

WithTimeout sets a timeout for the execution.

type PanicError added in v1.0.1

type PanicError struct {
	Value      any
	StackTrace string
}

PanicError represents a recovered panic during execution.

func (*PanicError) Error added in v1.0.1

func (p *PanicError) Error() string

Error implements the error interface.

type Policy added in v1.0.3

type Policy struct {
	// contains filtered or unexported fields
}

Policy represents a composed resilience strategy. It is thread-safe and reusable across multiple calls.

func NewPolicy added in v1.0.3

func NewPolicy(opts ...Option) *Policy

NewPolicy creates a new Policy with the given options. The order of options determines the execution order from outermost to innermost. Example: NewPolicy(WithFallback(f), WithRetry(3)) results in Fallback(Retry(Action)).

func (*Policy) Do added in v1.0.3

func (p *Policy) Do(ctx context.Context, action func(context.Context) (any, error)) (any, error)

Do executes an action within the resilience policy.

func (*Policy) DoErr added in v1.0.3

func (p *Policy) DoErr(ctx context.Context, action func(context.Context) error) error

DoErr executes an action within the resilience policy.

func (*Policy) Health added in v1.0.6

func (p *Policy) Health() <-chan circuit.StateEvent

Health returns a channel that emits state change events from the underlying resilience components. It returns nil if no health-reporting component is configured. The caller should use a select with default to handle non-blocking receive. Note: Only circuit breaker health events are exposed through Policy.Health(). For adaptive limiter events, access the limiter directly via AdaptiveLimiter.Health().

type Priority added in v1.0.6

type Priority int

Priority represents the importance of a request.

const (
	// PriorityLow is for non-critical, background, or asynchronous tasks.
	PriorityLow Priority = iota
	// PriorityStandard is the default priority for most requests.
	PriorityStandard
	// PriorityCritical is for high-priority, user-facing, or essential tasks.
	PriorityCritical
)

func GetPriority added in v1.0.6

func GetPriority(ctx context.Context) Priority

GetPriority retrieves the Priority from the context. Returns PriorityStandard if no priority is found.

type PriorityBulkhead added in v1.0.6

type PriorityBulkhead struct {
	// contains filtered or unexported fields
}

PriorityBulkhead limits concurrency based on priority levels. It preserves capacity for higher-priority traffic by shedding lower-priority traffic when the system approaches saturation.

func NewPriorityBulkhead added in v1.0.6

func NewPriorityBulkhead(capacity uint, thresholds map[Priority]float64) *PriorityBulkhead

NewPriorityBulkhead creates a new PriorityBulkhead with the specified capacity and thresholds. The thresholds map defines the maximum utilization (0.0 to 1.0) allowed for each priority. If a priority is not in the map, it defaults to 1.0 (no shedding until full capacity).

func (*PriorityBulkhead) Execute added in v1.0.6

func (b *PriorityBulkhead) Execute(ctx context.Context, action func() error) error

Execute wraps an action with priority-aware bulkhead concurrency control. It returns ErrShedLoad if the utilization threshold for the priority is exceeded. It returns ErrBulkheadFull if the absolute capacity is reached.

Performance Note: This implementation uses a "soft limit" design where the utilization check is decoupled from the actual semaphore acquisition. This provides high throughput and avoids global locks, which is ideal for load shedding. While this might allow slight over-utilization under extremely high contention, it maintains high performance for the majority of cases.

type RateLimiter added in v1.0.3

type RateLimiter struct {
	// contains filtered or unexported fields
}

RateLimiter implements a time-based token bucket rate limiter.

func NewRateLimiter added in v1.0.3

func NewRateLimiter(limit float64, interval time.Duration) *RateLimiter

NewRateLimiter creates a new RateLimiter with the specified limit and interval. Example: NewRateLimiter(100, time.Second) allows 100 requests per second.

func (*RateLimiter) Acquire added in v1.0.3

func (rl *RateLimiter) Acquire(ctx context.Context) bool

Acquire attempts to consume a token from the bucket. Returns true if a token was acquired, false otherwise.

type RetryAfterError

type RetryAfterError interface {
	error
	RetryAfter() time.Duration
	CancelAllRetries() bool
}

RetryAfterError is implemented by errors that specify how long to wait before retrying. This is commonly used with HTTP 429 (Too Many Requests) or 503 (Service Unavailable) to respect Retry-After headers. It also supports pushback signals to cancel all retries.

type RetryState

type RetryState struct {
	// Name is the optional identifier for the operation being retried.
	Name string
	// Attempt is the current 0-indexed retry iteration.
	Attempt uint
	// MaxAttempts is the maximum number of attempts allowed.
	MaxAttempts uint
	// LastError is the error encountered in the previous attempt.
	LastError error
	// TotalDuration is the cumulative time spent across all attempts and sleeps.
	TotalDuration time.Duration
	// NextDelay is the duration to be slept before the next attempt.
	NextDelay time.Duration
	// contains filtered or unexported fields
}

RetryState encapsulates the current state of a retry execution.

type Retryer

type Retryer interface {
	// Do executes a function that returns a value and an error.
	Do(ctx context.Context, action func(context.Context) (any, error)) (any, error)
	// DoHedged executes a function using speculative retries (hedging).
	DoHedged(ctx context.Context, action func(context.Context) (any, error)) (any, error)
	// DoErr executes a function that returns only an error.
	DoErr(ctx context.Context, action func(context.Context) error) error
	// DoErrHedged executes a function using speculative retries (hedging).
	DoErrHedged(ctx context.Context, action func(context.Context) error) error
}

Retryer defines the interface for executing actions with resilience.

func New

func New(opts ...Option) Retryer

New returns a new Retryer pre-configured with the provided options. This is useful for dependency injection and reusable resilience clients.

type StateEvent added in v1.0.6

type StateEvent struct {
	Component string
	State     HealthState
	Timestamp time.Time
	Message   string
}

StateEvent represents a state change event emitted by resilience components.

type StateMachine added in v1.0.2

type StateMachine[S any, D any, E any] struct {
	// contains filtered or unexported fields
}

StateMachine represents a resilient state machine inspired by Erlang's gen_statem. Every state transition is protected by the configured resilience policies.

func NewStateMachine added in v1.0.2

func NewStateMachine[S any, D any, E any](initialState S, initialData D, transition TransitionFunc[S, D, E], opts ...Option) *StateMachine[S, D, E]

NewStateMachine creates a new resilient state machine with the provided initial state, data, and transition function.

func (*StateMachine[S, D, E]) GetData added in v1.0.2

func (sm *StateMachine[S, D, E]) GetData() D

GetData returns the current data of the state machine.

func (*StateMachine[S, D, E]) GetState added in v1.0.2

func (sm *StateMachine[S, D, E]) GetState() S

GetState returns the current state of the state machine.

func (*StateMachine[S, D, E]) Handle added in v1.0.2

func (sm *StateMachine[S, D, E]) Handle(ctx context.Context, event E) error

Handle processes an event and performs a state transition. The transition is executed within a resilience envelope (Retries, Circuit Breakers, etc.). If the transition succeeds, the state machine updates its internal state and data.

type TransitionFunc added in v1.0.2

type TransitionFunc[S any, D any, E any] func(ctx context.Context, state S, data D, event E, rs RetryState) (S, D, error)

TransitionFunc defines the signature for a state transition function. It takes the current state, data, and event, and returns the next state and data. It also receives the current RetryState to allow transitions to adapt to failure history.

Directories

Path Synopsis
examples
adaptiveretry command
backpressure command
basic command
bulkhead command
chaos command
circuitbreaker command
deadline command
fallback command
hedging command
http command
panicrecovery command
pushback command
ratelimiter command
sql command
stateful command
statemachine command
telemetry

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL