httpx

package module
v5.0.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 1, 2026 License: Apache-2.0 Imports: 17 Imported by: 3

README

httpx

Go Reference Go version Test coverage Mutation OpenSSF Best Practices OpenSSF Scorecard

Resilient outbound-HTTP toolkit for Go: retry, backoff, transient-error classification, and more.

A resilient outbound-HTTP toolkit for Go providing jittered exponential backoff, transient-error classification, Retry-After parsing, HTTP status mapping, secret redaction, body draining, a transparent retrying http.RoundTripper with body replay, and a configurable redirect allowlist. Zero dependencies beyond the Go standard library and pgregory.net/rapid (test only).

The toolkit presents three retry doors sharing one option vocabulary:

  • Do[T] retries a typed operation you own (any closure returning (T, error)).
  • GetBytes retries an HTTP GET and returns bounded, redaction-safe body bytes.
  • NewRetryRoundTripper retries transparently inside a stdlib http.RoundTripper.

NewRetryClient assembles the retrying client (transport + an explicit, required redirect policy) in one call.

v4 tightens what counts as success: CheckHTTPStatus returns nil for 2xx only, so a 3xx surfaced by a non-following redirect policy is now an error (see Status checking and Migrating from v3). v5 hardens signatures with no behavior change: the redaction helpers take the secret as the named Secret type, and WithSameHost / WithPreserveMethod take a bool (see Migrating from v4).

Install

go get github.com/cplieger/httpx/v5@latest

Usage

// Bounded-bytes GET with retry
body, err := httpx.GetBytes(ctx, client, url,
    httpx.WithMaxAttempts(3),
    httpx.WithBaseDelay(time.Second),
)

// Generic typed retry
result, err := httpx.Do(ctx, func(ctx context.Context) (T, error) {
    return doWork(ctx)
}, httpx.WithMaxAttempts(3), httpx.WithLabel("fetch"))

// Retry rate limits too (opt-in): a *RateLimitError is retried after
// min(its Retry-After hint, maxWait)
result, err := httpx.Do(ctx, fn,
    httpx.WithRateLimitRetry(30*time.Second),
)

// Retry ONLY rate limits (transients fail fast; e.g. when transient retry is
// handled by an outer layer)
_, err := httpx.Do(ctx, func(ctx context.Context) (struct{}, error) {
    return struct{}{}, download(ctx)
}, httpx.WithRateLimitOnly(5*time.Minute), httpx.WithMaxAttempts(3))

// Retrying client: transport retry + an explicit redirect policy in one call.
// The policy parameter is required (there is no safe universal default; nil
// panics). No Client.Timeout is set; bound totals with a context deadline or
// TransportConfig.MaxElapsedTime, and single attempts on the base transport.
client := httpx.NewRetryClient(nil, httpx.DefaultRedirectPolicy, httpx.TransportConfig{
    MaxAttempts: 4,
    BaseDelay:   time.Second,
})

// Transparent retrying RoundTripper (inspired by hashicorp/go-retryablehttp)
rt := httpx.NewRetryRoundTripper(http.DefaultTransport, httpx.TransportConfig{
    MaxAttempts: 4,
    OnRetry: func(attempt int, req *http.Request, resp *http.Response, err error) {
        log.Printf("retry #%d for %s", attempt, req.URL)
    },
    PrepareRetry: func(req *http.Request) error {
        req.Header.Set("Authorization", "Bearer "+freshToken())
        return nil
    },
})
client := &http.Client{Transport: rt, CheckRedirect: httpx.DefaultRedirectPolicy}

// Retry POST/PUT with body replay (opt-in, requires GetBody)
client := httpx.NewRetryClient(nil, httpx.DefaultRedirectPolicy, httpx.TransportConfig{
    MaxAttempts:        4,
    RetryNonIdempotent: true,
})
payload := []byte(`{"key":"value"}`)
req, _ := http.NewRequest("POST", url, bytes.NewReader(payload))
req.GetBody = func() (io.ReadCloser, error) {
    return io.NopCloser(bytes.NewReader(payload)), nil
}
resp, err := client.Do(req)

// PermanentError: signal "do not retry" (mirrors cenkalti/backoff)
if configErr != nil {
    return httpx.Permanent(configErr) // will not be retried
}

// Custom redirect policy
policy := httpx.RedirectPolicyFunc(
    httpx.WithAllowedHosts("api.example.com"),
    httpx.WithAllowedSuffixes(".cdn.example.com"),
    httpx.WithMaxHops(3),
)

// Refuse a redirect that would change the method (POST -> GET across a
// 301/302/303) instead of silently re-issuing it as a GET. The refused hop
// surfaces the 3xx response itself, which CheckHTTPStatus reports as an error.
policy = httpx.RedirectPolicyFunc(httpx.WithSameHost(true), httpx.WithPreserveMethod(true))

// Status checking: nil for 2xx only. A 3xx (surfaced by a non-following
// policy) is an error.
if err := httpx.CheckHTTPStatus(resp); err != nil { /* typed: Auth/RateLimit/HTTPStatus */ }

// Pin a private / self-signed CA as the SOLE trust anchor (verification stays
// ON, TLS 1.2 minimum). The caller reads the PEM bytes (file, secret, env),
// keeping the helper I/O-free.
tr, err := httpx.CATransport(pemBytes)
client := &http.Client{Transport: tr, CheckRedirect: httpx.DefaultRedirectPolicy}
// ...or compose the pinned transport with retry:
client = httpx.NewRetryClient(tr, httpx.DefaultRedirectPolicy, httpx.TransportConfig{MaxAttempts: 3})

// Transient error classification
if httpx.IsTransient(err) { /* safe to retry */ }

// Limit response body size
rc := httpx.LimitedBody(resp, 1<<20) // 1 MB cap
defer rc.Close()

Migrating from v4

v5 is signature hardening only: no behavior changes. The redaction helpers take the secret as a named type, Secret, so the value-to-hide and the text-to-scan cannot be transposed at a call site (a reversed call turns the redactor into a leak); the two presence-style redirect options take the bool their neighbour WithAllowSchemeDowngrade always did; and the two capability interfaces embed error.

v4 v5
RedactSecretString(s, secret) (both plain string) secret is the named type Secret: RedactSecretString(s, httpx.Secret(token))
RedactTransportError(err, prefix, secret) (prefix and secret both plain string) RedactTransportError(err, prefix, httpx.Secret(token))
WithSameHost() WithSameHost(true); false is the no-option default (only allowlisted targets are followed)
WithPreserveMethod() WithPreserveMethod(true); false is the no-option default (a method-changing hop is followed as net/http rewrites it)
Transient = IsTransient() bool Transient = error + IsTransient() bool
RetryAfterHint = RetryAfterHint() time.Duration RetryAfterHint = error + RetryAfterHint() time.Duration
module path suffix /v4 github.com/cplieger/httpx/v5 in go.mod and every import
Go 1.26 toolchain (go 1.26.6) Go 1.27 toolchain (go 1.27.0); an older toolchain refuses to build the module

A call passing an untyped string constant (RedactSecretString(s, "token")) compiles unchanged; only a string variable in the secret position needs the httpx.Secret(...) conversion. A transposed call (the secret where the text belongs) is now a compile error, which is the point. RedactSecret(err, secret) is unchanged: its parameters already differ in type, so a swap never compiled.

The two interface changes need no work in a normal implementor: an implementor is an error already, or it could not appear in the chain errors.As walks. Only a type that declared IsTransient()/RetryAfterHint() without an Error() method stops satisfying; such a type was never reachable through either interface's only access path. What the change buys the caller is errors.AsType: errors.AsType[httpx.Transient](err) and errors.AsType[httpx.RetryAfterHint](err) now compile, where before errors.AsType's [E error] constraint refused both and go fix -errorsastype produced code that did not build.

Migrating from v3

Two changes, one of them breaking.

v3 v4
CheckHTTPStatus returns nil for 200-399 BREAKING: nil for 2xx only; every other status errors, a 3xx included (*HTTPStatusError)
a method-changing redirect hop is followed as a GET opt in to refusing it with RedirectPolicyFunc(..., WithPreserveMethod(true)) (additive, off by default)
github.com/cplieger/httpx/v3 github.com/cplieger/httpx/v5 in go.mod and every import; coming from v3 you cross the v5 signature changes in the same bump, so apply Migrating from v4 too

What changes: a 3xx that reaches your code now returns *HTTPStatusError{Code} instead of nil.

Who is affected: only callers that pair a non-following redirect policy (RefuseAllRedirects, or any CheckRedirect returning http.ErrUseLastResponse) with CheckHTTPStatus. A followed redirect never surfaces a 3xx, so a client on DefaultRedirectPolicy or an allowlist policy sees no difference. GetBytes and the RetryRoundTripper are untouched: GetBytes already returned a *StatusError for a 3xx (it now derives that from the classifier instead of restating the band), and the transport never classified statuses through it. DoConditional still errors on a 3xx, but the error is now *HTTPStatusError{Code} instead of the plain unexpected status %d fallback; a caller matching on that message must switch to errors.As.

What to do: delete the hand-rolled band check next to the call: a guard like if resp.StatusCode < 200 || resp.StatusCode >= 300 { ... } sitting after CheckHTTPStatus (written precisely because the classifier accepted 3xx) is now redundant. If a 3xx must stay non-fatal for one call site, check resp.StatusCode there instead of widening the classifier back.

API

Retry doors
  • Do[T]: generic retry with jittered exponential backoff. When a transient error implements RetryAfterHint, its pre-capped duration replaces the backoff for the next wait (the exponential base keeps advancing). Rate-limit handling is opt-in per call: WithRateLimitRetry(maxWait) adds *RateLimitError to the retryable set; WithRateLimitOnly(maxWait) retries nothing else. WithAttemptTimeout(d) adds a retryable per-attempt bound. Counts total attempts (a non-positive count clamps to 1).
  • GetBytes: HTTP GET with exponential backoff on 408/429/5xx and transient transport errors (timeouts, connection resets, DNS failures; see IsTransient); other 4xx and non-transient transport errors return immediately. Honors Retry-After (capped at RetryAfterCap). Counts total attempts.
  • NewRetryRoundTripper(base, TransportConfig{...}): create a retrying http.RoundTripper. TransportConfig{} is ready to use (3 attempts, 1s base delay, default policy); MaxAttempts: -1 means exactly one attempt.
Loop options

Shared by both loop doors (Option): WithMaxAttempts, WithBaseDelay, WithLogger, WithExhaustedLevel. Do-only (DoOption): WithLabel, WithAttemptTimeout, WithRateLimitRetry, WithRateLimitOnly. GetBytes-only (GetOption): WithHeaders, WithMaxBodyBytes. Passing an option to the wrong door is a compile error. A non-positive rate-limit maxWait falls back to RetryAfterCap (60s), so the inter-attempt wait is always positive (never a hot spin); supplying both rate-limit modes is a configuration error.

Clients
  • NewClient(timeout): simple client with the same-host DefaultRedirectPolicy preinstalled.
  • NewRetryClient(base, policy, cfg): retrying client; policy is required (nil panics; a nil CheckRedirect would silently mean net/http's follow-anywhere default). Sets no Client.Timeout (it would cap the whole retry sequence); see Timeouts and deadlines for bounding attempts and totals.
  • ContextWithDefaultTimeout(ctx, def): bound ctx by def only when the caller brought no deadline. A caller deadline is never undercut or extended; def <= 0 means no default; the returned cancel is always non-nil. The per-request timeout rule for an API client.
TLS transports
  • CATransport(pem): build an *http.Transport (cloned from http.DefaultTransport, so pooling, timeouts, and proxy settings are preserved) that pins the CA certificate(s) in pem as the sole trust anchors. Verification stays on (InsecureSkipVerify is never set) with a TLS 1.2 minimum. Returns the concrete, mutable transport so it composes with NewRetryRoundTripper. The caller reads the PEM bytes (file, secret, env), keeping the helper I/O-free.
  • ErrNoCertsInPEM: returned by CATransport when pem yields no certificates (a loud error instead of a silently-empty pool).
  • CloneDefaultTransport(): a private clone of http.DefaultTransport that is yours to mutate (a per-attempt ResponseHeaderTimeout, MaxIdleConnsPerHost, the base of NewRetryRoundTripper) without reconfiguring every other client in the process. Errors when http.DefaultTransport has been replaced by a non-*http.Transport.
Test helpers (certtest subpackage)

The github.com/cplieger/httpx/v5/certtest subpackage supplies throwaway self-signed CA material for tests, the companion to CATransport. Only _test.go files import it, so its certificate-generation code never links into a production binary.

  • certtest.SelfSignedCA(tb): a fresh self-signed CA certificate, PEM-encoded. Each call generates a new key, so two certs are mutually untrusted (handy for asserting a pin is enforced).
  • certtest.WriteSelfSignedCA(tb): the same certificate written to a ca.pem file under tb.TempDir(), returning the path.
Transport hooks & policies (TransportConfig fields)
  • CheckRetry: pluggable retry policy, func(ctx, resp, err) (bool, error). The default retries transient transport errors and 429/502/503/504 (deliberately narrower than GetBytes, which retries 408 and every 5xx).
  • OnRetry: per-attempt callback for observability/metrics (the transport's only seam; it logs nothing itself)
  • PrepareRetry: mutate the request before a retry (e.g., re-sign tokens)
  • MaxElapsedTime: hard total-time ceiling across retries, including honored Retry-After (checked between attempts)
  • RetryNonIdempotent: opt-in POST/PUT/PATCH/DELETE retry with GetBody replay
Classification & Parsing
  • IsTransient: classify errors as transient (retryable); respects PermanentError. A caller's expired or canceled context is terminal; an http.Client.Timeout or transport ResponseHeaderTimeout is a per-attempt bound and IS retried (see Timeouts and deadlines)
  • AttemptTimeout(err) / IsAttemptTimeout(err): mark a timeout as the expiry of a bound over ONE attempt, making it retryable, and test for that mark. The mirror of Permanent, and the only way an error that CARRIES context.DeadlineExceeded becomes retryable; the mark keeps the deadline visible to errors.Is(err, context.DeadlineExceeded) for the caller's own callers. WithAttemptTimeout applies it for you (see Timeouts and deadlines).
  • RetryAfterHint: an interface (error + RetryAfterHint() time.Duration) an error implements to supply the next retry wait. Do honors it when the error is transient and the duration is positive; the implementer must cap the value, since httpx applies no ceiling of its own here.
  • Transient: an interface (error + IsTransient() bool) an error implements to declare its own retryability, consulted by IsTransient. Both capability interfaces embed error, like net.Error, so a caller can write errors.AsType[httpx.Transient](err).
  • CheckHTTPStatus: map an HTTP status to a typed error; success is 2xx only (see Status checking)
  • IsRetryableStatus(code): does the retry loop treat this status as transient? True for 408, 429, and any 5xx. It is the same rule the built-in retry uses, not a copy of it: GetBytes's attempt function calls this function, so the two cannot drift. For the caller that runs GetBytes with WithMaxAttempts(1) inside its own retry budget and therefore classifies the returned *StatusError itself (see Nesting a door in your own retry loop). The RetryRoundTripper's default policy is narrower (429/502/503/504); pass this predicate through TransportConfig.CheckRetry to widen it.
  • ParseRetryAfter / ParseRetryAfterResponse: parse a Retry-After header (capped at RetryAfterCap / raw)
Status checking

CheckHTTPStatus(resp) returns nil for exactly 2xx (200-299) and an error for every other status:

Status Result
2xx nil
3xx *HTTPStatusError{Code} (not transient, not a client or server error)
401 / 403 *AuthError (the message carries (401) / (403))
429 *RateLimitError with the raw, uncapped Retry-After hint
other 4xx / 5xx *HTTPStatusError{Code} (transient for 502/503/504)
1xx *HTTPStatusError{Code} (not a completed response)

The 2xx-only window is a v4 breaking change (v3 returned nil for the whole 200-399 band). A 3xx only ever reaches a caller when the client is configured not to follow redirects: RefuseAllRedirects, or any CheckRedirect returning http.ErrUseLastResponse, which net/http hands back as the 3xx response itself with a nil error. Under the old window that redirect stub classified as success, so a token-bearing client that deliberately refuses the hop (exactly the client RefuseAllRedirects exists for) then treated the unfollowed redirect as a completed request. CheckHTTPStatus is the status handling that policy delegates to, and it now reports the 3xx as the failure it is.

A 3xx is deliberately an *HTTPStatusError rather than a new type, so it flows through the existing plumbing unchanged: IsTransient is false (only 502/503/504 are transient), IsServerError and IsClientError are both false, and LogSafeError and the redaction helpers pass it through (it embeds no URL). There is no second, stricter classifier; this is the only one. See Migrating from v3 for the migration.

Error Control
  • Permanent(err): wrap an error to signal "do not retry" (mirrors cenkalti/backoff)
  • IsPermanent(err): check whether an error is wrapped as permanent
  • PermanentError: the wrapper type (supports errors.Is/errors.As/Unwrap)
  • MarkTransient(err): wrap an error to signal "retry this". The mirror of Permanent, for a failure your operation knows is self-healing where the shared policy cannot know (a server-side fault delivered inside a 200 envelope, an upstream whose plain 500 always clears). It saves you declaring a one-method Transient wrapper, and cannot forget Unwrap the way a hand-rolled one does. The mark is the outermost verdict, so it overrides a non-transient verdict already on the error, but IsTransient's standing rejections still win: Permanent, an *AuthError, a *RateLimitError (retry a rate limit by naming a wait budget instead: WithRateLimitRetry), and a caller-context error stay terminal. For a per-attempt timeout carrying a deadline, use AttemptTimeout.
Nesting a door in your own retry loop

Running a door with WithMaxAttempts(1) inside your own retry loop is the sanctioned way to avoid multiplying the two attempt counts (a 3-attempt door inside a 3-attempt loop is 9 requests). The door then makes no retry decision, so you make it. The two exported predicates are the door's own rule, so your loop and the built-in one classify identically:

// One attempt per outer attempt; this loop owns the budget.
body, err := httpx.GetBytes(ctx, client, url, httpx.WithMaxAttempts(1))
if err != nil {
    // A self-healing status is worth another of MY attempts. GetBytes
    // deliberately does NOT mark its exhaustion error transient: after
    // WithMaxAttempts(1) that decision is the caller's policy.
    if se, ok := errors.AsType[*httpx.StatusError](err); ok && httpx.IsRetryableStatus(se.Code) {
        return httpx.MarkTransient(err)
    }
    return err // auth/config failure: terminal, fail on the first attempt
}

The exhaustion error still implements RetryAfterHint, so the upstream's already-capped Retry-After survives into the enclosing Do and is waited instead of the jittered backoff.

Backoff Primitives
  • JitteredBackoff: equal jitter, [backoff/2, backoff]
  • SafeDouble / SleepCtx: overflow-safe doubling, context-aware sleep
Body Helpers
  • Drain / DrainClose: drain a body for connection reuse (64 KB limit). A failed drain logs a bare Debug line and never the read error itself: that text is written by the far end (see URL redaction in logs and errors)
  • LimitedBody: wrap a response body with a size cap
  • ReadLimitedBody: read a body to a cap (closing it) with overflow detection, returning *ResponseTooLargeError instead of a silently truncated body
Conditional GET
  • Validators{ETag, LastModified}: the cache validators captured from a previous 200, replayed on the next request
  • ConditionalResult{Validators, Body, NotModified}: one conditional-request outcome
  • DoConditional(client, req, v, maxBodyBytes): one conditional attempt; v alone decides what is replayed (pre-existing conditional headers are cleared, empty fields unsent). A 304 returns NotModified with zero Validators (keep the ones you sent); a 200 returns the bounded body plus fresh validators; anything else is an error, with transport errors reduced via LogSafeError so no raw URL reaches caller error text. Single-shot by design: wrap it in Do for retry, rebuild the request per attempt, persist body and validators together, and send zero Validators when the cached body is unusable. Validators are checked in both directions (header field-value grammar, 1 KiB cap): an invalid upstream value is captured as empty and an invalid replayed field is unsent, so a poisoned validator degrades to an unconditional GET and self-heals on the next clean 200. Full semantics in the godoc.
Redirect Policies
  • DefaultRedirectPolicy: same-host-only (used by NewClient); refuses a same-host https->http downgrade, allows an http->https upgrade.
  • RefuseAllRedirects: follows no redirect; returns http.ErrUseLastResponse, so the client surfaces the 3xx response itself (nil error) and CheckHTTPStatus reports it as an error. The policy for a token-bearing client of an API that issues no redirects: Go forwards custom headers (X-Plex-Token, X-Api-Key) across redirects, so a hostile 302 would exfiltrate the credential.
  • DockerGitHubRedirectPolicy: example allowlist policy for docker.com/github.com.
  • RedirectPolicyFunc: build a custom redirect allowlist from functional options: WithAllowedHosts, WithAllowedSuffixes, WithSameHost(true) (also allow the original request's host; the same-origin building block; false is the no-option default), WithMaxHops, WithAllowSchemeDowngrade, and WithPreserveMethod. Every policy refuses an https->http downgrade by default, even to an allowlisted or same-host target, so an auth header is never forwarded onto a cleartext hop; an http->https upgrade is always allowed, and WithAllowSchemeDowngrade(true) opts out of the refusal.
  • WithPreserveMethod(true): refuses a hop that would change the request method instead of rewriting the method back (false is the no-option default: the hop is followed as net/http rewrites it). net/http downgrades a POST/PUT/PATCH/DELETE to a GET across a 301/302/303 and drops the body (RFC 9110 §15.4, Go issue 18570); only 307/308 carry the method forward. The refusal returns http.ErrUseLastResponse, so the 3xx surfaces to the caller (nil error) and errors under CheckHTTPStatus, the same pairing RefuseAllRedirects relies on. The comparison is against the original request, so a POST kept by a 307 and then downgraded by a 302 is refused at the second hop; an empty via chain fails closed. The hop cap, allowlist, and scheme-downgrade refusals (hard errors) keep precedence, and the option grants nothing on its own: with no allowlist and no WithSameHost(true) the policy still refuses everything.
  • CheckRedirect: the http.Client.CheckRedirect function shape as a type alias; every shipped policy is one.
Secret Redaction
  • RedactTransportError / RedactSecret / RedactSecretString: secret redaction (error- and string-level). RedactSecretString and RedactTransportError take the secret as the named Secret type, so the value-to-hide and the text-to-scan cannot be transposed. A reversed call would turn the redactor into a leak, and it no longer compiles.
  • LogSafeError: reduce a URL-embedding transport *url.Error to its underlying cause (everything else passes through, errors.Is/As preserved). The same reduction httpx applies to every transport error it logs; equivalent to RedactTransportError(err, "", "").
Error Types
  • AuthError / RateLimitError / HTTPStatusError / StatusError
  • ResponseTooLargeError: returned by GetBytes when the response exceeds WithMaxBodyBytes (carries Limit; no body is returned)
  • ErrRateLimited / ErrServerError: sentinel errors
  • PermanentError: do-not-retry sentinel wrapper

Logging

Do and GetBytes log via log/slog and accept WithLogger to override the default logger per call. Per-attempt "retrying" lines are logged at Debug; a retry that recovers is normal operation, not a degraded state. The terminal "retries exhausted" / "rate limit retries exhausted" lines are at Warn, except under WithMaxAttempts(1), where they drop to Debug: a one-attempt budget retried nothing, so the door is a single attempt inside the caller's own retry loop, and that loop owns both the retry policy and the warning. WithExhaustedLevel(level) overrides that line's level outright, for callers whose own failure log carries strictly more context than the library's can (the tracker, the item, an onset latch): demoting the library's copy keeps one report of one event without discarding the per-attempt Debug diagnostics a discard logger would also throw away. It applies to a multi-attempt budget too, and is the only way to raise the line above Warn. GetBytes also emits a Warn "slow upstream response" when a single attempt's response takes longer than 10s (timed per attempt, so backoff sleeps are not counted as upstream latency). The RetryRoundTripper logs nothing itself; observe its retries through the OnRetry hook, where redaction is the caller's responsibility. Drain/DrainClose emit one Debug line, failed to drain response body, carrying no attributes. A failed drain only forfeits connection reuse, and the read error that caused it is remote-authored text this package refuses to log (see below).

URL redaction in logs and errors

To avoid leaking credentials into logs (CWE-532, the class of go-retryablehttp CVE-2024-6104), GetBytes never logs or returns a raw request URL:

  • Every logged url attribute is redacted: the whole userinfo component is replaced with REDACTED (stronger than url.URL.Redacted, which masks only the password and would leave a username-only API token in the clear) and query values are replaced with REDACTED (query values commonly carry API keys and tokens). Query keys, scheme, host, and path are kept for debugging.
  • StatusError.Error() renders that same redacted URL, so the secret stays out of returned errors too; the raw StatusError.URL field remains available for programmatic use.
  • Transport errors (*url.Error, which embed the full URL) are reduced to their underlying cause before logging. The reduction is exported as LogSafeError so callers wrapping transport errors into their own messages can apply the same one.

That reduction is type-based (it strips an envelope this library itself added), so it cannot sanitize text the far end wrote. Drain/DrainClose therefore drop the body-read error instead of logging it: net/http renders a malformed chunked trailer as malformed MIME header: missing colon: "<remote bytes>", and for a URL that carries its credential in the path (a webhook token) an edge echoing the request URI puts that credential in those bytes. The drain site logs on the package-level default logger, which no option can reroute, so the value is dropped there rather than left for callers to handle. Nothing an operator can act on is lost, because a drain runs only where the body is already being discarded and its outcome is reported by the path that discarded it.

Retry exhaustion

GetBytes and the RetryRoundTripper report exhaustion differently; match your error handling to the one you use:

  • GetBytes returns nil body and a wrapped error: retries exhausted after <elapsed>: <lastErr> (unwrap with errors.Is/errors.As). A response that overflows WithMaxBodyBytes returns *ResponseTooLargeError (no body). When the last attempt carried a Retry-After, the exhaustion error also implements RetryAfterHint with that capped wait, so a caller running GetBytes with WithMaxAttempts(1) inside its own Do loop (the pattern that avoids multiplying the two attempt budgets) still honors the upstream-requested delay instead of falling back to jittered backoff. It deliberately does not implement Transient: whether an exhausted GET is worth another outer attempt stays the caller's policy.
  • RetryRoundTripper returns the last response with a nil error, even when that response is a retryable 5xx (e.g. a 503), mirroring how a non-retried request behaves. A caller that checks only err != nil will treat an exhausted 503 as success, so inspect resp.StatusCode and close the body. (A budget abort via MaxElapsedTime does return an error.)

Timeouts and deadlines

httpx retries transient failures, not budget expiry. IsTransient classifies a caller's expired or canceled context as non-transient, while a connection reset, a DNS error, a net.Error timeout, and a 429/5xx are transient. A caller's context deadline means "the budget is exhausted, stop"; the expiry of a bound over ONE attempt means "this attempt failed, try again".

The two are told apart by whether the context.DeadlineExceeded value is in the error's unwrap chain, not by errors.Is: net/http's own timeout error reports errors.Is(err, context.DeadlineExceeded) == true without ever carrying the sentinel, so errors.Is cannot distinguish the bounds net/http installs from the caller's own. Where the difference is genuinely unknowable (a bound the net package reports, since it maps every expired context onto one shared i/o timeout value), the verdict stays terminal, and AttemptTimeout(err) is how the code that installed the bound opts back in.

  • Total budget: a context deadline. Pass a context.WithTimeout (or a caller-supplied deadline) as the single authoritative bound. It spans every attempt and every backoff sleep (SleepCtx caps the backoff by it); on expiry the call ends, terminal, not retried.
  • Per-attempt bound under Do: WithAttemptTimeout(d). Each attempt runs under a context bounded by d, and that bound's expiry is retried, the gRPC per-try-timeout model. A caller deadline nearer than d still governs (context keeps the earlier deadline), so d caps one attempt and never extends the total. The expiry is marked as the attempt's only while the caller's own context is still live, so a caller out of budget stays terminal. The attempt context is canceled when fn returns, so read or drain the body inside fn.
  • Per-attempt bound on the client or transport: http.Client.Timeout and ResponseHeaderTimeout, retried. net/http reports both through its own timeout error, which claims to match context.DeadlineExceeded but never carries it, so both classify as per-attempt and are retried, including under the RetryRoundTripper's default policy. Use Permanent(err) in a TransportConfig.CheckRetry if you want one of them to stop the loop instead.
  • Per-attempt bound anywhere else: mark it with AttemptTimeout(err). A context.WithTimeout a caller derives itself inside fn carries the real sentinel, and a net-level bound (a net.Dialer Timeout) is indistinguishable from a caller deadline the net package mapped; both stay terminal until marked. WithAttemptTimeout is the first case done for you.
  • GetBytes takes no per-attempt option: the door owns its request and has no callback to bound. Bound it with the Client.Timeout of the client you pass, or run it as one attempt inside Do (WithMaxAttempts(1) on the GET, WithAttemptTimeout(d) on the Do), which is also the composition that avoids multiplying two attempt budgets.

Under NewRetryRoundTripper / NewRetryClient the retry loop runs inside client.Do, so http.Client.Timeout is not per-attempt at all: it caps the whole retry sequence and a slow attempt that trips it aborts the remaining retries, which is why NewRetryClient sets none. Put the per-attempt bound on the base transport (ResponseHeaderTimeout, retried) and the total on the caller's context or TransportConfig.MaxElapsedTime. Neither the between-attempt MaxElapsedTime check nor an expired context can interrupt an attempt already stalled inside the base transport; only a transport-level timeout can.

Unsupported by Design

The following features are intentionally not provided:

Feature Rationale
Circuit breaker Orthogonal pattern excluded by all comparables. Compose externally with sony/gobreaker.
Retry budget / token bucket None of the comparables implement it. Disproportionate complexity (~150 LOC + shared mutable state) for a focused library.
Multiple jitter strategies (full, decorrelated) Equal jitter is the recommended default per AWS Builders' Library. Full jitter risks near-zero delays.
ErrorHandler for exhaustion Current fmt.Errorf("retries exhausted: %w", lastErr) is sufficient. Callers unwrap.
Response body on error Adds API complexity (ownership of body close). Use Do[T] with custom logic.
Idempotency key injection Application-level concern, not a retry library's responsibility.
Configurable Retry-After cap A raisable cap would regress the fixed-60s DoS ceiling (ParseRetryAfter); rate-limit waits are capped by the caller-owned maxWait arguments.

Contributing

Issues and PRs are welcome. See CONTRIBUTING.md for the conventions and how to run the checks locally.

Disclaimer

This project is built with care and follows security best practices, but it is intended for personal / self-hosted use. No guarantees of fitness for production environments. Use at your own risk.

This project was built with AI-assisted tooling using Claude, GPT, and Kiro. The human maintainer defines architecture, supervises implementation, and makes all final decisions.

License

Apache-2.0. See LICENSE.

Documentation

Overview

Package httpx provides a resilient outbound-HTTP toolkit: transient-error classification, generic typed retry (Do) and a bounded-bytes GET (GetBytes) over one jittered-exponential-backoff loop, a transparent retrying RoundTripper, Retry-After parsing, HTTP status mapping, secret redaction, body draining, custom-CA TLS transports, and a configurable redirect allowlist.

The package deliberately keeps these concerns together: they compose into a single net/http.Client, whose configuration surface (Transport, CheckRedirect, per-request contexts) spans exactly this set. This overview maps the surface; the README carries usage examples, the v2 migration table, and the timeout model.

Retry doors

Three entry points with different ownership contracts (who builds the request, who owns the body, who sees the response). They share one option vocabulary and the same equal-jitter backoff progression; passing an option to the wrong door does not compile.

Shared loop options (Option): WithMaxAttempts, WithBaseDelay, WithLogger.

Clients

NewClient (timeout + same-host redirect policy preinstalled), NewRetryClient (retry transport + REQUIRED explicit redirect policy), ContextWithDefaultTimeout (request-deadline helper).

Classification and error control

IsTransient decides retryability; extend it for your own error types via the Transient and RetryAfterHint interfaces (both embed error, so errors.AsType reaches them), or mark one error at a time with MarkTransient. Permanent (and PermanentError, IsPermanent) marks an error non-retryable; AttemptTimeout (and IsAttemptTimeout) marks a timeout as the expiry of a bound over ONE attempt, which IS retryable — the only way an error CARRYING a context deadline becomes so, applied for you by WithAttemptTimeout. CheckHTTPStatus maps response codes to typed errors: AuthError, RateLimitError, StatusError, HTTPStatusError, ResponseTooLargeError, and the ErrRateLimited and ErrServerError sentinels. IsRetryableStatus answers the status-code half — the same rule the GetBytes loop applies — for a caller running one attempt inside its own retry budget.

Success is EXACTLY 2xx. CheckHTTPStatus returns nil only for [200, 300) and an error for every other status, a 3xx included — the v4 breaking change (v3 accepted the whole 200-399 band as success). A 3xx reaches a caller only under a non-following redirect policy (RefuseAllRedirects, or any CheckRedirect returning http.ErrUseLastResponse, which net/http surfaces as the 3xx response itself with a nil error), and the old window reported that redirect stub as success. It is now an *HTTPStatusError: still non-transient, still unchanged by the redaction helpers. Migrating from v3: a surfaced 3xx now errors, which affects only callers pairing a non-following policy with CheckHTTPStatus; a hand-rolled 2xx band check beside such a call is now redundant and can be deleted.

Retry-After

ParseRetryAfter and ParseRetryAfterResponse parse the header with a 60-second cap (RetryAfterCap). RateLimitError.RetryAfter carries the RAW uncapped hint; callers sleeping on it must bound it (the rate-limit retry modes do).

Redirect policies

A CheckRedirect policy is required knowledge the caller supplies: DefaultRedirectPolicy (same host), RefuseAllRedirects, DockerGitHubRedirectPolicy, or a custom allowlist built with RedirectPolicyFunc and RedirectOption (WithAllowedHosts, WithAllowedSuffixes, WithSameHost, WithMaxHops, WithAllowSchemeDowngrade, WithPreserveMethod).

TLS transports

CATransport pins PEM CA certificate(s) as the sole trust anchors on a cloned default transport (ErrNoCertsInPEM on empty input); CloneDefaultTransport yields a private mutable transport clone. The certtest subpackage generates throwaway CA material for tests.

Secret redaction

RedactSecret, RedactSecretString, RedactTransportError, and LogSafeError keep credentials out of logs and returned errors. GetBytes never logs or returns a raw URL. Secret types the value-to-hide apart from the text-to-scan, so a transposed call does not compile. The redactors match a literal value in the exact representation the text carries, so a needle held in one encoding never matches a haystack carrying another. A caller composing them with a normalizing transform or a byte cap follows the order documented on RedactSecretString: redact, normalize, redact, cap.

Body helpers

Drain, DrainClose, LimitedBody, ReadLimitedBody.

Conditional GET

DoConditional with Validators and ConditionalResult implements ETag/Last-Modified revalidation over one bounded read.

Backoff primitives

JitteredBackoff, SafeDouble, and SleepCtx are the exported building blocks the doors are made of, for callers composing their own loops.

Index

Examples

Constants

View Source
const (
	// DefaultBaseDelay is the production base for exponential-backoff retry.
	DefaultBaseDelay = time.Second
	// DefaultMaxAttempts caps the retry doors at three total attempts.
	DefaultMaxAttempts = 3
	// DefaultMaxBodyBytes caps response bodies at 10 MB.
	DefaultMaxBodyBytes int64 = 10 << 20
	// RetryAfterCap is the maximum Retry-After honor duration.
	RetryAfterCap = 60 * time.Second
)

Variables

View Source
var ErrNoCertsInPEM = errors.New("httpx: no PEM-encoded certificates found")

ErrNoCertsInPEM is returned by CATransport when the supplied PEM data contains no parseable certificates. A misconfigured or empty CA file therefore fails loudly rather than silently producing a transport that trusts nothing (which would reject every connection with an opaque error).

View Source
var ErrRateLimited = errors.New("rate limited")

ErrRateLimited is a sentinel callers use with errors.Is to detect 429 responses.

View Source
var ErrServerError = errors.New("server error")

ErrServerError is a sentinel for upstream 5xx responses.

Functions

func AttemptTimeout

func AttemptTimeout(err error) error

AttemptTimeout wraps err to declare that the timeout it reports bounded ONE ATTEMPT rather than the caller's total budget, making it retryable: IsTransient reports true for it (checked ahead of the context-error rejection), and the retry doors retry it. Returns nil for nil input. It is the mirror of Permanent, which forces the opposite verdict.

It exists because a context deadline alone cannot express the difference. A caller's expired deadline means "the budget is gone, stop"; the expiry of a per-attempt bound means "this attempt failed, try another" — opposite instructions from the same context.DeadlineExceeded. Only the code that installed the bound knows which one it is, so httpx cannot infer it and defaults to the safe reading (terminal). This mark is how the installer says otherwise, and the wrapper deliberately keeps the deadline visible to errors.Is instead of hiding it, which was the only way to opt in before.

Reach for it when the per-attempt bound is NOT the retry door's own and httpx cannot see whose bound it was:

  • a context.WithTimeout a caller derives itself inside a Do callback. Its expiry carries the deadline sentinel, indistinguishable from the caller's own. WithAttemptTimeout does this for you, mark included; hand-roll it only when the bound must cover something other than one whole attempt.
  • a net-level bound such as a net.Dialer Timeout. The net package maps every expired context onto one shared "i/o timeout" value, so a *net.OpError or *net.DNSError reporting a deadline cannot say whose it was and stays terminal (see IsTransient).

An http.Client.Timeout and a net/http.Transport ResponseHeaderTimeout do NOT need the mark: net/http reports those through its own timeout error, which never carries the sentinel, so IsTransient already reads them as per-attempt and retries them — including under the RetryRoundTripper's default policy. Use Permanent (or a TransportConfig.CheckRetry) if you want one of them to stop the loop instead.

The caller owns the judgment: mark ONLY a bound you installed over a single attempt. Marking a deadline that came from your own caller makes an exhausted budget look retryable, which is the failure mode the default classification prevents. WithAttemptTimeout avoids the judgment entirely by deciding from the two contexts it owns.

func CATransport

func CATransport(pem []byte) (*http.Transport, error)

CATransport builds an *http.Transport that trusts ONLY the CA certificate(s) in pem for TLS verification (pinning). It is cloned from http.DefaultTransport (via CloneDefaultTransport), so it keeps the standard connection pooling, dial/keepalive timeouts, HTTP/2 negotiation, and proxy support (ProxyFromEnvironment). A FRESH TLS config is then installed — RootCAs from pem, a TLS 1.2 minimum, and verification always ENABLED (InsecureSkipVerify is not set). Any TLS settings already on http.DefaultTransport are intentionally NOT carried over, so the returned transport's trust posture cannot be weakened by a program that globally mutated the default transport's *tls.Config (e.g. set InsecureSkipVerify or an accept-all VerifyPeerCertificate hook).

The supplied CA(s) are the SOLE trust anchors: the transport rejects any host not chaining to them, including public-CA hosts. This is the right setup for a known self-hosted endpoint presenting a private or self-signed certificate (a Plex server, an internal API).

The returned *http.Transport is concrete and mutable, so callers may tune fields such as MaxIdleConnsPerHost or pass it as the base RoundTripper of NewRetryRoundTripper:

tr, err := httpx.CATransport(pem)
client := httpx.NewRetryClient(tr, httpx.DefaultRedirectPolicy, httpx.TransportConfig{MaxAttempts: 3})

It returns ErrNoCertsInPEM when pem yields no certificates. The caller owns reading the PEM bytes (from a file, a secret, an env var), which keeps this function I/O-free and lets the caller bound the read as it sees fit.

CATransport requires http.DefaultTransport to be the standard library's *http.Transport (the default). If your program has replaced it with a wrapping RoundTripper (for example request instrumentation), CATransport returns an error.

func CheckHTTPStatus

func CheckHTTPStatus(resp *http.Response) error

CheckHTTPStatus classifies an HTTP response, mapping anything that is not a success to a typed error. Success is EXACTLY 2xx: a status in [200, 300) returns nil and EVERY other status returns an error — 401/403 → *AuthError, 429 → *RateLimitError, and everything else (a 3xx, any other 4xx, any 5xx, and an informational 1xx) → *HTTPStatusError carrying the code.

The 2xx-only window is a v4 change; v3 and earlier returned nil for the whole 200-399 band. A 3xx reaches a caller only when the client is configured NOT to follow redirects — RefuseAllRedirects, or any CheckRedirect returning http.ErrUseLastResponse, which net/http hands back as the 3xx response itself with a nil error. Under the old window that redirect stub classified as SUCCESS, so a token-bearing client that deliberately refuses the hop then treated the unfollowed redirect as a completed request; this is the "caller's own status handling" RefuseAllRedirects delegates to, and it now reports the 3xx as the failure it is. A caller that pairs a non-following policy with its own hand-rolled 2xx band check no longer needs it.

A 3xx is deliberately a *HTTPStatusError rather than a new error type or a second classifier, so it flows through the existing plumbing unchanged: IsTransient reports false (only 502/503/504 are transient), HTTPStatusError.IsServerError and HTTPStatusError.IsClientError both report false (a 3xx is neither ≥ 500 nor in [400, 500)), and LogSafeError and the redaction helpers pass it through untouched (it embeds no URL).

func CloneDefaultTransport

func CloneDefaultTransport() (*http.Transport, error)

CloneDefaultTransport returns a private clone of http.DefaultTransport, keeping the standard connection pooling, dial/keepalive timeouts, HTTP/2 negotiation, and proxy support (ProxyFromEnvironment). The clone is the caller's to mutate — set a per-attempt ResponseHeaderTimeout, tune MaxIdleConnsPerHost, or pass it as the base RoundTripper of NewRetryRoundTripper — without the global footgun of mutating http.DefaultTransport itself, which would silently reconfigure every other client in the process.

It returns an error when http.DefaultTransport has been replaced by a non-*http.Transport RoundTripper (request instrumentation, a test stub): a wrapping RoundTripper offers no concrete transport to clone, and failing loudly beats silently dropping the wrapper's behavior. It is the building block CATransport is assembled on.

func ContextWithDefaultTimeout

func ContextWithDefaultTimeout(ctx context.Context, def time.Duration) (context.Context, context.CancelFunc)

ContextWithDefaultTimeout bounds ctx by def ONLY when ctx carries no deadline; a deadline the caller already set is the authoritative budget and is never undercut (or extended). A def <= 0 means "no default" and leaves ctx unbounded. The returned cancel is always non-nil and must be called (a no-op on the passthrough paths), matching context.WithTimeout's contract.

It is the per-request timeout rule for an API client: per-attempt bounds live on the transport (ResponseHeaderTimeout), the total budget is the caller's ctx, and this default applies only when the caller brought no budget of its own. Extracted from the identical requestContext helpers in the plexapi and arrapi client libraries.

func DefaultRedirectPolicy

func DefaultRedirectPolicy(req *http.Request, via []*http.Request) error

DefaultRedirectPolicy is the default redirect policy: it allows a redirect only to the same host as the original request, and refuses a same-host https->http scheme downgrade (which would forward a custom auth header onto a cleartext hop). A cross-host redirect is refused (Go forwards a custom header across a redirect, so it would leak) and an http->https upgrade is allowed. It delegates to RedirectPolicyFunc(WithSameHost(true)), with one addition: a call with an empty via chain (which net/http never produces — via always carries at least the original request) is allowed rather than refused. Use RedirectPolicyFunc for a custom allowlist, a higher hop cap (WithMaxHops), or to permit downgrades (WithAllowSchemeDowngrade).

func Do

func Do[T any](ctx context.Context, fn func(ctx context.Context) (T, error), opts ...DoOption) (T, error)

Do calls fn up to WithMaxAttempts times (total, including the first call) with jittered exponential backoff, returning the first success. Non-retryable errors are returned immediately. By default the retryable set is IsTransient (a *RateLimitError is deliberately NOT transient; a generic operation must not blindly re-fire a rate-limited call) and a transient error carrying a positive RetryAfterHint waits that hint instead of the backoff, the exponential base still advancing. WithRateLimitRetry and WithRateLimitOnly opt into rate-limit retry per their docs.

Under the default and WithRateLimitRetry modes a context canceled after a failed attempt returns ctx.Err(); under WithRateLimitOnly the final attempt's error wins (the v2 RetryOnRateLimit contract). WithAttemptTimeout bounds each attempt and makes that bound's expiry retryable, the caller's own deadline staying terminal. Logging goes to WithLogger (default slog.Default()): per-attempt lines at Debug, the terminal exhaustion at Warn - or at Debug when the budget is a single attempt, since nothing was retried and the caller's own loop owns the warning (see exhaustedLevel).

Example
package main

import (
	"context"
	"fmt"
	"time"

	"github.com/cplieger/httpx/v5"
)

func main() {
	attempts := 0
	// context.Background() (not t.Context()): an Example has no *testing.T.
	result, err := httpx.Do(context.Background(), func(_ context.Context) (string, error) {
		attempts++
		if attempts < 2 {
			return "", &httpx.HTTPStatusError{Code: 503}
		}
		return "done", nil
	}, httpx.WithMaxAttempts(3), httpx.WithBaseDelay(time.Millisecond), httpx.WithLabel("example"))
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	fmt.Println(result)
}
Output:
done

func DockerGitHubRedirectPolicy

func DockerGitHubRedirectPolicy(req *http.Request, via []*http.Request) error

DockerGitHubRedirectPolicy is an OPTIONAL example redirect policy allowing docker.com and github.com hosts. Like every shipped policy it refuses an https->http scheme downgrade (judged against the original request's scheme, see WithAllowSchemeDowngrade), so a custom auth header never rides a cleartext hop even to an allowlisted host. Use it by assigning to Client.CheckRedirect or pass RedirectOption values to RedirectPolicyFunc for other allowlists.

func Drain

func Drain(body io.ReadCloser)

Drain reads and discards up to 64 KB of a response body to enable HTTP connection reuse. A failed drain forfeits only that reuse, and is reported as a bare Debug line on slog.Default(). The read error itself is deliberately NOT logged.

Go 1.27 made HTTP/1 Response.Body drain itself on Close, which raises the question of whether this still earns its place. It does, on two measured grounds. The stdlib drain is HTTP/1 only — net/http/internal/http2 has no equivalent — so on an HTTP/2 response, which every transport this package builds negotiates (CloneDefaultTransport keeps ForceAttemptHTTP2), this is the only drain there is. And Drain takes an io.ReadCloser, not an *http.Response, so it also covers a body the stdlib never wraps. Where both apply they compose rather than collide: this drain runs first and, for a body inside the 64 KB budget, reaches EOF, which short-circuits the stdlib's drain entirely.

The two limits differ deliberately and do not need reconciling. 64 KB is how much effort this function spends; net/http's own budget is 256 KiB with a 50 ms deadline, and it applies only when unread content remains AND resp.ContentLength <= 256 KiB. That gate is why an over-limit body refused by ReadLimitedBody reads nothing extra: a response declaring more than 256 KiB is excluded outright. Measured on go1.27.0 against go1.26.7, a declared 10 MB body read to a 1 MB cap pulls an identical 1,114,112 bytes on both. A CHUNKED over-limit body has ContentLength == -1, which passes the gate, so its worst case is the cap plus at most 256 KiB more. Either way the caller pays no latency for it: the drain runs on net/http's read-loop goroutine, and Close() measured 24-44 microseconds against a deliberately stalled chunked body that provably passes the gate.

That omission is a CWE-532 fix, not an oversight. A body-read error's text is written by the FAR END: net/http renders a malformed chunked trailer as

malformed MIME header: missing colon: "<remote bytes>"

and for a consumer whose request URL carries its credential in the PATH (a webhook token is the canonical shape) an edge that echoes the request URI puts that credential in those bytes. Logging it needs only the consumer to be running at Debug — the level an operator raises precisely while diagnosing failing deliveries. Three properties make the site uncloseable from anywhere else, which is why the value is dropped HERE rather than by the caller:

  • Nobody can route it. Drain takes no logger (it is called from defer at consumer call sites and from four sites inside this package, one of them a RoundTripper that has no logger at all), and the retry doors' WithLogger governs their own loop lines, not slog.Default().
  • LogSafeError does not reduce it. That boundary is TYPE-based: it strips the *url.Error envelope this package's own machinery adds. A body-read error is not a *url.Error — a malformed trailer surfaces as textproto.ProtocolError, whose entire value IS the message — so the reduction returns it byte-identically.
  • Redaction needs a secret to redact, and Drain is handed none.

No diagnosis a caller can act on is lost. Drain runs only where the body is being thrown away, so the response's own outcome is already reported by the path that discarded it (CheckHTTPStatus's typed error, the retry lines, ReadLimitedBody's error); a drain error is never the only signal of a failure, and no caller can repair a drain. This is the same call the package makes in urlErrorCause, which substitutes a contentless stand-in rather than log the one field it holds.

func DrainClose

func DrainClose(rc io.ReadCloser)

DrainClose reads remaining bytes (up to drainLimit) from rc before closing it. It inherits Drain's logging contract: a failed drain logs a bare Debug line carrying no remote-authored text.

func GetBytes

func GetBytes(ctx context.Context, client *http.Client, reqURL string, opts ...GetOption) ([]byte, error)

GetBytes performs an HTTP GET with bounded exponential-backoff retry on 429 and 5xx responses and on transient transport errors (timeouts, connection resets, DNS failures - see IsTransient). 4xx (non-429) and non-transient transport errors are returned immediately. Honors Retry-After (capped at RetryAfterCap). The response body is read to WithMaxBodyBytes and returned; an over-limit body fails loud with *ResponseTooLargeError (no body). Every logged url attribute and every returned error is redacted (see the package's URL redaction docs).

GetBytes deliberately keeps its own retry loop rather than delegating to RetryRoundTripper.RoundTrip. It is a decorator over the same shared primitives (resolveWait, JitteredBackoff, SafeDouble, SleepCtx, ParseRetryAfter, IsTransient, Drain), not a thin wrapper over the RoundTripper cycle, because GetBytes carries behavior the transparent RoundTripper has no equivalent for and which must stay byte-for-byte stable for existing consumers:

  • []byte return with the body capped at WithMaxBodyBytes (the RoundTripper hands back an *http.Response and never reads the body);
  • URL/secret redaction on every log "url" attr (redactURL) and every returned/wrapped error (LogSafeError, StatusError.Error()), the CWE-532 hardening the RoundTripper path does not perform;
  • rich per-attempt slog logging plus the "retries exhausted after %s: %w" wrapper, which the RoundTripper exposes only as an OnRetry hook;
  • classification of every 5xx (not just 502/503/504) as retryable, and of every non-2xx as a permanent *StatusError carrying the request URL (the 2xx success band is CheckHTTPStatus's, delegated to since v4; only the error value differs, because this door's errors render a redacted URL). A 3xx reaches here only when the client refuses redirects, and is an error: the redirect stub is not the requested resource and GetBytes cannot surface Location.

Routing GetBytes through RoundTrip would silently change one or more of these, so the loop is intentionally not merged.

Example
package main

import (
	"context"
	"fmt"
	"net/http"
	"net/http/httptest"
	"time"

	"github.com/cplieger/httpx/v5"
)

func main() {
	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
		fmt.Fprint(w, "hello")
	}))
	defer srv.Close()

	// context.Background() (not t.Context()): an Example has no *testing.T.
	body, err := httpx.GetBytes(context.Background(), srv.Client(), srv.URL,
		httpx.WithMaxAttempts(3),
		httpx.WithBaseDelay(100*time.Millisecond),
	)
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	fmt.Println(string(body))
}
Output:
hello

func IsAttemptTimeout

func IsAttemptTimeout(err error) bool

IsAttemptTimeout reports whether err (or any wrapped error) was marked by AttemptTimeout as a per-attempt timeout.

func IsPermanent

func IsPermanent(err error) bool

IsPermanent reports whether err (or any wrapped error) is a *PermanentError.

func IsRetryableStatus

func IsRetryableStatus(code int) bool

IsRetryableStatus reports whether a response status is one the retry loop treats as a transient failure worth another attempt: 408 Request Timeout, 429 Too Many Requests, and any 5xx (a code >= 500). Everything else — every 2xx, every 3xx, an informational 1xx, and every other 4xx — is a settled answer this package does not repeat.

It is the SAME rule the built-in retry uses, not a restatement of it: GetBytes's own attempt function calls this function to classify a response, so a caller's verdict and the door's verdict cannot disagree.

It exists for the caller that runs GetBytes under WithMaxAttempts(1) inside its own outer retry budget (the sanctioned way to avoid multiplying the two attempt counts). That caller receives the door's *StatusError and owns the retry decision — GetBytes deliberately does not mark its exhaustion error Transient, because whether an exhausted GET is worth another outer attempt is the caller's policy — so it needs to ask the question the door would have asked:

if se, ok := errors.AsType[*httpx.StatusError](err); ok && httpx.IsRetryableStatus(se.Code) {
	return httpx.MarkTransient(err) // spend another of MY attempts
}

408 is included because it is the server reporting that IT gave up waiting, which self-heals and is safe to repeat on an idempotent GET. 429 is here because this door retries a rate limit by default; Do does not (its *RateLimitError is retryable only under WithRateLimitRetry or WithRateLimitOnly), so pair this predicate with the door whose policy it describes.

The RetryRoundTripper is the one deliberate divergence: its default policy retries 429/502/503/504 only, a narrower set than this, so a plain 500 that this predicate calls retryable is not retried by the transport. Widen the transport with TransportConfig.CheckRetry (this predicate is a ready-made one) rather than assuming the two agree.

func IsTransient

func IsTransient(err error) bool

IsTransient returns true for errors likely caused by temporary server or network issues worth retrying. Auth, rate-limit, permanent, and caller-context errors are never transient.

"Caller-context error" is decided by the sentinel actually being in the error's unwrap chain, not by errors.Is — net/http's timeout errors match context.DeadlineExceeded without carrying it, so an http.Client.Timeout and a transport ResponseHeaderTimeout are per-ATTEMPT bounds and ARE transient (see isCallerContextError). A caller's own expired deadline stays terminal; AttemptTimeout marks a bound whose expiry this package cannot otherwise tell apart from one.

func JitteredBackoff

func JitteredBackoff(backoff time.Duration) time.Duration

JitteredBackoff returns a duration in [backoff/2, backoff] using the "equal jitter" strategy (per AWS Builders' Library). Full jitter and decorrelated jitter are intentionally not provided — equal jitter is the recommended default for HTTP retry as it avoids thundering herd while maintaining a minimum backoff floor.

func LimitedBody

func LimitedBody(resp *http.Response, limit int64) io.ReadCloser

LimitedBody wraps resp.Body with an io.LimitReader capped at limit bytes, preserving the original Close method.

func LogSafeError

func LogSafeError(err error) error

LogSafeError returns an error whose message is safe to log. A transport *url.Error embeds the full request URL (with any userinfo/query secrets), so it is reduced to its underlying cause. Nil returns nil, and a non-nil error NEVER reduces to nil (see urlErrorCause); *StatusError already renders a redacted URL via Error(), so it (and everything else) passes through unchanged — preserving errors.Is/As chains for callers.

httpx applies this reduction to every transport error it logs or wraps; it is exported so a caller wrapping transport errors into its own messages can apply the same one (equivalent to RedactTransportError(err, "", "") — reach for that variant when a known secret must also be scrubbed from the text).

func MarkTransient

func MarkTransient(err error) error

MarkTransient wraps err in a value satisfying the Transient interface with a true verdict, so the retry doors treat it as a transient failure. Returns nil for nil input. It is the mirror of Permanent, which forces the opposite verdict; the name is not the bare adjective only because Transient is the interface it satisfies.

It exists so a caller widening the retryable set for its own operation does not have to declare a one-method wrapper type to do it. That wrapper is pure boilerplate — an Error, an Unwrap and an IsTransient returning true — and hand-rolling it invites the two mistakes this function cannot make: forgetting Unwrap (which hides the cause from every errors.Is and errors.As the caller's own callers run) and marking a nil error.

The mark is the OUTERMOST verdict, so it overrides a non-transient verdict already on the error: MarkTransient(&HTTPStatusError{Code: 500}) is retryable even though a plain 500 is not. Use it for a failure your operation knows is self-healing where the shared policy cannot know — a server-side fault delivered inside a 200 envelope, for example, which no status-based classification can see:

if env := serverFault(body); env != nil {
	return httpx.MarkTransient(env) // this clears; spend another attempt
}

IsTransient's standing rejections still outrank the mark, because they answer questions the caller is not the authority on:

  • Permanent wins. Permanent(MarkTransient(err)) is not retried.
  • An *AuthError and a *RateLimitError stay terminal. Credentials do not fix themselves, and a rate limit is retried by naming a wait budget (WithRateLimitRetry, WithRateLimitOnly), not by claiming transience.
  • A caller-context error stays terminal: the budget that authorized the work is gone, so another attempt has nothing to spend. For a timeout that bounded ONE attempt and therefore carries a deadline that is not the caller's, use AttemptTimeout — IsTransient consults that mark ahead of the context rejection, which is the whole reason it is a separate mark.

func NewClient

func NewClient(timeout time.Duration) *http.Client

NewClient returns an *http.Client with the given timeout and the DefaultRedirectPolicy (same-host only). For custom redirect allowlists, configure CheckRedirect with RedirectPolicyFunc or assign DockerGitHubRedirectPolicy.

func NewRetryClient

func NewRetryClient(base http.RoundTripper, policy CheckRedirect, cfg TransportConfig) *http.Client

NewRetryClient returns an *http.Client whose Transport is a RetryRoundTripper over base (nil base means http.DefaultTransport) and whose CheckRedirect is policy. It is the one-call form of pairing NewRetryRoundTripper with an explicit redirect policy.

policy is REQUIRED and must be non-nil: NewRetryClient panics on a nil policy, because a nil CheckRedirect silently means net/http's default follow-anywhere behavior (up to 10 hops to any host, custom auth headers forwarded) — exactly the unsafe omission this constructor exists to prevent. Pass DefaultRedirectPolicy (same-host), RefuseAllRedirects, or a RedirectPolicyFunc allowlist.

The returned client sets no Client.Timeout: a Client.Timeout above a retrying transport caps the WHOLE retry sequence and defeats the retries beneath it. Note that neither MaxElapsedTime nor the request context can interrupt a stalled in-flight attempt from between attempts: bound single attempts on the base transport (e.g. ResponseHeaderTimeout on a CloneDefaultTransport()) and bound the total with a context deadline (http.NewRequestWithContext) or TransportConfig.MaxElapsedTime.

func ParseRetryAfter

func ParseRetryAfter(h string) time.Duration

ParseRetryAfter parses a Retry-After header value (delta-seconds or HTTP-date). Returns zero for missing/malformed values. Caps at RetryAfterCap for safety (prevents unbounded waits in retry loops). For raw uncapped values, use ParseRetryAfterResponse.

func ParseRetryAfterResponse

func ParseRetryAfterResponse(resp *http.Response) time.Duration

ParseRetryAfterResponse parses the Retry-After header from an *http.Response. Returns zero if absent or unparseable. Does NOT cap — preserves the raw duration so callers (e.g., CheckHTTPStatus) can make their own decisions. For capped values suitable for retry loops, use ParseRetryAfter.

func Permanent

func Permanent(err error) error

Permanent wraps err to indicate it should never be retried. Mirrors cenkalti/backoff.Permanent().

func ReadLimitedBody

func ReadLimitedBody(body io.ReadCloser, limit int64) ([]byte, error)

ReadLimitedBody reads body up to limit bytes, always closes body, and returns the bytes read. It reads one byte past limit to detect an over-limit body and returns *ResponseTooLargeError (with nil bytes) rather than a silently truncated payload — a truncated body indistinguishable from a complete one is a corruption hazard. A limit of math.MaxInt64 means "effectively unlimited" and is guarded against probe-size overflow.

It is the read-all-with-overflow-detection companion to LimitedBody (which only caps the stream and leaves reading and overflow handling to the caller), and is the same cap+1 read GetBytes applies internally — exposed for callers that issue their own request and decode outside GetBytes but still want the fail-loud size bound. On any error the body is already closed.

func RedactSecret

func RedactSecret(err error, secret string) error

RedactSecret replaces occurrences of secret in err's message with "REDACTED". It funnels into RedactSecretString, whose doc comment carries the ordering and representation rules the caller owns.

func RedactSecretString

func RedactSecretString(s string, secret Secret) string

RedactSecretString replaces every occurrence of secret in s with "REDACTED" and returns the result. It is the string-level building block behind RedactSecret and RedactTransportError, exposed for callers that must redact a secret from a plain string — a captured HTTP response body destined for an error field or a log line — rather than from an error value. An empty secret is a no-op (s is returned unchanged), matching the error-shaped variants.

The replacement is literal and byte-exact, which puts three obligations on the caller:

  1. Hold the needle in the SAME representation the haystack carries. A decoded token does not match a JSON-escaped or percent-encoded haystack, and the mismatch is silent: nothing is replaced and no error is reported.
  2. Call this on BOTH sides of any normalizing transform (runesafe Sanitize/SanitizeSingleLine, unicode/norm NFC/NFKC, strconv.Unquote, JSON/URL/HTML unescape, case folding, whitespace collapse), passing the needle in the representation each side carries. A call placed only before the transform misses a secret the transform CONSTRUCTS out of text the needle never matched; a call placed only after it misses a secret the transform REWRITES, leaving a near-complete fragment of the value.
  3. Put any byte cap (ReadLimitedBody, a bounded sanitizer preset, a plain truncation) LAST. A cut that splits the value leaves a surviving prefix this function can no longer match.

Composed, that order is: redact, normalize, redact, cap.

This package performs one of those transforms

Every URL httpx renders — *StatusError.Error(), and every "url" attribute in a GetBytes log line — goes through a parse-and-re-serialize step that blanks query values and drops userinfo. That step is a URL-encoding transform in rule 2's sense, so redacting a known secret out of one of those strings is the "only after the transform" mistake. Measured on go1.27.0 (identical on go1.26.7): for a secret in the request PATH, which is the one credential-bearing position the rendering keeps verbatim, a secret containing a space renders as tok%20en, a non-ASCII byte as tok%C3%A9n, and a '#' truncates the value at the fragment boundary — so a byte-exact needle held in the caller's own representation matches none of them and the value reaches the log percent-encoded. A stray '%' is the safe case: the URL fails to parse and the whole rendering collapses to a fixed placeholder.

So a caller holding a credential that rides in a URL PATH must redact it before the URL reaches this package (or pass the needle in its percent-encoded form as well). A credential in the query string or the userinfo needs nothing: the rendering removes those wholesale, without matching anything.

func RedactTransportError

func RedactTransportError(err error, prefix string, secret Secret) error

RedactTransportError unwraps *url.Error and redacts the secret from the error message. Returns nil for nil input, and never nil for a non-nil one (see urlErrorCause for the *url.Error that carries no cause of its own). The replacement runs through RedactSecretString, whose doc comment carries the ordering and representation rules for composing redaction with a normalizing transform or a byte cap. An empty secret disables redaction: the (possibly prefixed) error is returned with its message untouched, since there is nothing to scan for.

func RedirectPolicyFunc

func RedirectPolicyFunc(opts ...RedirectOption) func(*http.Request, []*http.Request) error

RedirectPolicyFunc returns a CheckRedirect function configured with the given options. A redirect is followed only when its target host is allowed — an exact WithAllowedHosts entry, a WithAllowedSuffixes match, or (with WithSameHost(true)) the original request's own host — and, unless WithAllowSchemeDowngrade is set, the redirect does not downgrade https->http. With no allowlist and no WithSameHost(true), all redirects are refused. The hop cap is WithMaxHops (default 5). WithPreserveMethod(true) additionally refuses (via http.ErrUseLastResponse, so the 3xx surfaces to the caller) a hop that would change the request method.

func RefuseAllRedirects

func RefuseAllRedirects(*http.Request, []*http.Request) error

RefuseAllRedirects is a CheckRedirect policy that follows NO redirect: it returns http.ErrUseLastResponse, so the client hands the caller the redirect response itself (status 3xx, body open, nil error) instead of the followed hop. It is the policy for a token-bearing client of an API that issues no redirects: Go's client forwards custom request headers (an X-Plex-Token, an X-Api-Key) across redirects — only Authorization, Cookie, and WWW-Authenticate are stripped, and only on a cross-domain hop — so a hostile 302 (MITM, DNS poisoning) would exfiltrate the credential to an attacker-chosen origin. With the hop refused, the credential never leaves the configured host and the unexpected 3xx surfaces to the caller's own status handling — which is CheckHTTPStatus: since v4 it classifies a 3xx as an error (*HTTPStatusError), so a surfaced redirect stub is reported as the failure it is instead of passing as success.

func SafeDouble

func SafeDouble(d time.Duration) time.Duration

SafeDouble doubles a duration, guarding against int64 overflow.

func SleepCtx

func SleepCtx(ctx context.Context, d time.Duration) error

SleepCtx sleeps for d or returns early on context cancellation.

Types

type AuthError

type AuthError struct{ Msg string }

AuthError indicates invalid or expired credentials.

func (*AuthError) Error

func (e *AuthError) Error() string

type CheckRedirect

type CheckRedirect = func(req *http.Request, via []*http.Request) error

CheckRedirect is the http.Client.CheckRedirect function shape. It is a type alias, so values are assignable in both directions; every shipped policy (DefaultRedirectPolicy, RefuseAllRedirects, DockerGitHubRedirectPolicy, and anything built with RedirectPolicyFunc) is a CheckRedirect.

type CheckRetry

type CheckRetry func(ctx context.Context, resp *http.Response, err error) (bool, error)

CheckRetry is the signature for a retry policy function. It receives the context, the response (may be nil on transport error), and the error (nil on successful response). It returns whether to retry and an optional error to short-circuit with. Mirrors hashicorp/go-retryablehttp CheckRetry.

type ConditionalResult

type ConditionalResult struct {
	// Validators are the fresh validators captured from a 200 response's ETag /
	// Last-Modified headers (either may be empty when the server sent none).
	// Zero on a 304: the caller keeps the validators it already holds.
	Validators Validators
	// Body is the full response body of a 200, bounded by the maxBodyBytes
	// given to DoConditional. Nil on a 304.
	Body []byte
	// NotModified reports a 304: the cached representation is still current.
	NotModified bool
}

ConditionalResult is one conditional-request outcome. Fields are ordered largest-alignment-first for govet fieldalignment.

func DoConditional

func DoConditional(client *http.Client, req *http.Request, v Validators, maxBodyBytes int64) (ConditionalResult, error)

DoConditional executes req as a single conditional request: it owns both conditional headers — any pre-existing If-None-Match / If-Modified-Since on req is removed, then each is set from v (an empty field is not sent), so v alone decides what is replayed — performs the request on client, and classifies the response.

  • 304 -> NotModified=true (body drained and closed; Validators zero — keep the ones you sent).
  • 200 -> the bounded body plus the response's fresh validators. A body over maxBodyBytes fails loud with *ResponseTooLargeError rather than being silently truncated; maxBodyBytes <= 0 means DefaultMaxBodyBytes.
  • Anything else -> an error: the CheckHTTPStatus mapping for every non-2xx status (*AuthError for 401/403, *RateLimitError for 429, *HTTPStatusError otherwise — transient only for 502/503/504, and since v4 that mapping covers a 3xx from a redirect-refusing client too), or a plain non-transient error for a 2xx that is neither usable content nor a revalidation (a 204, a 206). The body is always closed.

A transport error from the request itself is reduced via LogSafeError before it is returned: a *url.Error embeds the full request URL, so the reduction keeps query-string secrets out of caller error text (the same contract GetBytes applies to every error it returns), while preserving the cause for transient classification when composed with Do.

It is deliberately a SINGLE attempt so the caller owns the retry and cache policy: wrap it in Do (transient classification composes through the returned errors), rebuild req per attempt, and decide app-side when a cached copy may be reused on failure (stale-on-error) and whether validators may be sent at all (send the zero Validators when the cached body is unusable, so an empty cache can never be "revalidated" into a 304 with nothing to reuse). Intended for GET (or HEAD, where Body stays empty).

Validator hygiene: validators are validated in BOTH directions against the header field-value grammar (RFC 9110: no control bytes other than HTAB, no DEL) plus a 1 KiB per-value cap. A 200's ETag / Last-Modified that fails the check is captured as empty, so a hostile or corrupt upstream value can never enter the caller's persisted cache state; a replayed v field that fails it is not sent, so a validator poisoned in a store outside this package degrades to an unconditional GET and self-heals through the next 200's clean capture instead of failing at net/http's request-write validation on every subsequent attempt. Both drops are silent - the cost is one full re-download, never a correctness fault.

type DoOption

type DoOption interface {
	// contains filtered or unexported methods
}

DoOption configures Do. Options constructed as Option values apply to both loop doors; Do-only options (WithLabel, WithRateLimitRetry, WithRateLimitOnly) implement only this interface, so passing one to GetBytes is a compile error.

func WithAttemptTimeout

func WithAttemptTimeout(d time.Duration) DoOption

WithAttemptTimeout bounds EACH attempt by d and makes that bound's expiry RETRYABLE: the per-attempt (per-try) timeout, as distinct from the caller's total budget. It is the piece that joins a per-attempt deadline to the retry loop; without it a deadline inside fn is classified terminal (see AttemptTimeout for why an error alone cannot say otherwise). A non-positive d means no per-attempt bound, which is the option's absence.

fn receives a context derived from the caller's with d applied. Context keeps the EARLIER deadline, so a caller deadline nearer than d still governs and the total budget is never extended — d caps one attempt, never the call.

The expiry is marked as the attempt's ONLY when the caller's own context is still live, so retrying a caller that is out of budget stays impossible: an expired or canceled caller context is never marked, Do returns ctx.Err() before it classifies anything, and SleepCtx refuses to wait on a dead context. The decision is made from the two contexts rather than from the error, because a context.DeadlineExceeded value is identical either way.

The attempt context is canceled as soon as fn returns, so fn must not return a value that still depends on it — read, decode, or drain a response body inside fn (the same rule a caller-derived per-attempt context already imposes).

It is a DoOption: GetBytes owns its own request and has no callback to bound. Bound that door by running it as one attempt inside Do — GetBytes(ctx, ...) with WithMaxAttempts(1) inside a Do callback carrying this option — the same composition that avoids multiplying two attempt budgets.

func WithLabel

func WithLabel(s string) DoOption

WithLabel sets the operation label used in Do's log lines ("<label> failed, retrying", "<label> retries exhausted"). Default: "operation".

func WithRateLimitOnly

func WithRateLimitOnly(maxWait time.Duration) DoOption

WithRateLimitOnly makes Do retry ONLY *RateLimitError (matched through wrapped errors); every other error, including transient transport errors, is returned immediately. It absorbs v2's RetryOnRateLimit: the wait semantics match WithRateLimitRetry, the terminal Warn is "rate limit retries exhausted", and, matching the v2 contract, the FINAL attempt's error wins even under an already-canceled context (cancellation is observed in the always-positive inter-attempt sleep instead). Mutually exclusive with WithRateLimitRetry.

func WithRateLimitRetry

func WithRateLimitRetry(maxWait time.Duration) DoOption

WithRateLimitRetry makes Do additionally treat *RateLimitError as retryable, alongside the transient set. The wait before a rate-limit retry is min(err.RetryAfter, maxWait) when the error carries a positive hint, else maxWait; a non-positive maxWait falls back to RetryAfterCap, so the wait is always positive and a canceled context is observed before every retry. Transient errors keep their jittered-backoff waits (with the RetryAfterHint override), and the exponential base advances on every retry. Mutually exclusive with WithRateLimitOnly (Do returns a configuration error when both are supplied).

type GetOption

type GetOption interface {
	// contains filtered or unexported methods
}

GetOption configures GetBytes. GetBytes-only options (WithHeaders, WithMaxBodyBytes) implement only this interface, so passing one to Do is a compile error.

func WithHeaders

func WithHeaders(fn func(*http.Request)) GetOption

WithHeaders sets a function that is called to set headers on each request.

func WithMaxBodyBytes

func WithMaxBodyBytes(n int64) GetOption

WithMaxBodyBytes sets the maximum response body size to read. Default: DefaultMaxBodyBytes (10 MB). A non-positive value falls back to the default.

type HTTPStatusError

type HTTPStatusError struct {
	Code int
}

HTTPStatusError represents a non-2xx HTTP response not covered by AuthError or RateLimitError. Implements the Transient interface for 502/503/504.

func (*HTTPStatusError) Error

func (e *HTTPStatusError) Error() string

func (*HTTPStatusError) IsClientError

func (e *HTTPStatusError) IsClientError() bool

IsClientError reports whether the status code is 4xx.

func (*HTTPStatusError) IsServerError

func (e *HTTPStatusError) IsServerError() bool

IsServerError reports whether the status code is 5xx.

func (*HTTPStatusError) IsTransient

func (e *HTTPStatusError) IsTransient() bool

IsTransient reports whether the status code is a retryable server failure (502/503/504).

type OnRetry

type OnRetry func(attempt int, req *http.Request, resp *http.Response, err error)

OnRetry is called before each retry attempt (not the initial request). attempt is 1-indexed (first retry = 1). resp may be nil on transport errors.

type Option

type Option interface {
	DoOption
	GetOption
}

Option is a retry option accepted by BOTH loop doors, Do and GetBytes: WithMaxAttempts, WithBaseDelay, and WithLogger construct Option values.

func WithBaseDelay

func WithBaseDelay(d time.Duration) Option

WithBaseDelay sets the initial backoff delay. Default: DefaultBaseDelay (1s). A non-positive value falls back to the default.

func WithExhaustedLevel

func WithExhaustedLevel(l slog.Level) Option

WithExhaustedLevel overrides the level of the terminal "retries exhausted" line (see exhaustedLevel for the default rule).

It exists for the caller that reports the same terminal failure itself, with more context than this door can have. Such a caller wants the per-attempt retry diagnostics and NOT the verdict: leaving both produces two log lines for one failure, and silencing the whole logger to stop the second throws the diagnostics away with it. Pass slog.LevelDebug to keep the line for diagnosis while the caller's own line carries the alarm.

It does not change what is returned, and it does not suppress the line: a level below the logger's threshold simply is not emitted, which is the caller's decision to make.

func WithLogger

func WithLogger(l *slog.Logger) Option

WithLogger sets the logger for retry diagnostics. Default: slog.Default(). A nil logger falls back to the default.

func WithMaxAttempts

func WithMaxAttempts(n int) Option

WithMaxAttempts sets the maximum number of attempts (TOTAL, including the first call). Default: DefaultMaxAttempts (3). A value below 1 is treated as 1, so the operation always runs at least once (never a silent no-op).

type PermanentError

type PermanentError struct {
	Err error
}

PermanentError wraps an error to signal that it should NOT be retried, regardless of other retry policies. Mirrors cenkalti/backoff.PermanentError. Use Permanent(err) to wrap.

func (*PermanentError) Error

func (e *PermanentError) Error() string

func (*PermanentError) Is

func (e *PermanentError) Is(target error) bool

Is allows errors.Is matching against other PermanentErrors.

func (*PermanentError) Unwrap

func (e *PermanentError) Unwrap() error

type PrepareRetry

type PrepareRetry func(req *http.Request) error

PrepareRetry is called before each retry to allow request mutation (e.g., re-signing with a fresh token). Mirrors go-retryablehttp PrepareRetry.

type RateLimitError

type RateLimitError struct {
	Msg        string
	RetryAfter time.Duration
}

RateLimitError indicates a rate limit was exceeded. RetryAfter, when non-zero, is the RAW, UNCAPPED hint from the upstream's Retry-After header (populated via ParseRetryAfterResponse). The upstream controls this value; a hostile or misconfigured server can supply a very large duration (CWE-400 uncontrolled resource consumption). Callers that sleep on it directly MUST bound it first, e.g. min(err.RetryAfter, cap). Do's rate-limit modes (WithRateLimitRetry, WithRateLimitOnly) already do this (they cap at their maxWait argument). For a pre-capped value use ParseRetryAfter (bounded at RetryAfterCap = 60s).

func (*RateLimitError) Error

func (e *RateLimitError) Error() string

type RedirectOption

type RedirectOption func(*redirectCfg)

RedirectOption configures a redirect policy created by RedirectPolicyFunc.

func WithAllowSchemeDowngrade

func WithAllowSchemeDowngrade(allow bool) RedirectOption

WithAllowSchemeDowngrade permits a redirect that downgrades the scheme (https on the original request -> http on the target). The default (false) refuses such a downgrade so a credential carried in a custom request header (which Go forwards across a redirect, stripping only Authorization/Cookie) is never sent over a cleartext hop. A scheme upgrade (http->https) is always allowed regardless of this setting. The downgrade is judged against the ORIGINAL request's scheme (via[0]).

func WithAllowedHosts

func WithAllowedHosts(hosts ...string) RedirectOption

WithAllowedHosts sets the exact hostnames allowed as redirect targets.

func WithAllowedSuffixes

func WithAllowedSuffixes(suffixes ...string) RedirectOption

WithAllowedSuffixes sets the domain suffixes allowed (e.g. ".docker.com").

func WithMaxHops

func WithMaxHops(n int) RedirectOption

WithMaxHops sets the maximum number of redirect hops. Default: 5.

func WithPreserveMethod

func WithPreserveMethod(preserve bool) RedirectOption

WithPreserveMethod REFUSES a redirect hop that would change the request method, rather than rewriting the method back. The default (false) matches leaving the option out: a method-changing hop is followed as net/http rewrites it. net/http downgrades a POST (or PUT/PATCH/DELETE) to a GET across a 301, 302, or 303 and drops the body, per RFC 9110 §15.4 and Go's issue 18570 compatibility rule; only a 307 or 308 carries the method and body forward. For an API call whose meaning IS its method, that silent downgrade turns a write into a read against a URL the caller never named, so WithPreserveMethod(true) makes the client stop instead.

A refused hop returns http.ErrUseLastResponse, not an error: net/http then hands the caller the 3xx response itself (status, Location header, body open, nil error), exactly as RefuseAllRedirects does, and the surfaced 3xx is an error under CheckHTTPStatus. The method is never rewritten back to the original — the request is simply not re-sent.

The comparison is against the ORIGINAL request (via[0]), so a chain that preserves the method once and changes it later is still refused: a POST through a 307 (method kept) followed by a 302 (method downgraded to GET) is refused at the second hop. A same-method hop (a GET chain, or a POST through a 307/308) is unaffected and still subject to the hop cap, the target allowlist, and the scheme-downgrade rule, which all take precedence — a cross-host or downgrading hop is refused as a hard error even when the method also changes. When via is empty (net/http never does this; a hand-built chain can) the original method is unknowable, so the hop is refused too: the option fails closed.

It only ever narrows what a policy follows, so it grants nothing on its own: RedirectPolicyFunc with WithPreserveMethod(true) and no allowlist and no WithSameHost entry still refuses every redirect.

func WithSameHost

func WithSameHost(same bool) RedirectOption

WithSameHost permits a redirect whose target host equals the original request's host (ASCII case-insensitive, RFC 3986 §6.2.2.1), in addition to any WithAllowedHosts / WithAllowedSuffixes entries. The default (false) matches leaving the option out: only allowlisted targets are followed. WithSameHost(true) is the building block for a same-origin policy: combined with the default scheme-downgrade refusal (see WithAllowSchemeDowngrade), it follows a service's own same-host redirects (including an http->https upgrade) while refusing a cross-host hop that would forward a custom auth header to another origin. A policy built with only WithSameHost(true) (no allowlisted hosts) permits exactly the same-host set.

Like every RedirectOption, later values overwrite earlier ones: appending WithSameHost(false) to an option slice that already carries WithSameHost(true) turns the permission back off.

type ResponseTooLargeError

type ResponseTooLargeError struct {
	Limit int64
}

ResponseTooLargeError is returned by GetBytes when the response body exceeds the configured maximum (WithMaxBodyBytes, default DefaultMaxBodyBytes). The body is not returned: a truncated payload indistinguishable from a complete one is a silent-corruption hazard, so GetBytes fails loud instead. Limit is the cap that was exceeded, mirroring the stdlib *http.MaxBytesError shape.

func (*ResponseTooLargeError) Error

func (e *ResponseTooLargeError) Error() string

type RetryAfterHint

type RetryAfterHint interface {
	error
	RetryAfterHint() time.Duration
}

RetryAfterHint is implemented by errors that carry an explicit wait duration for the next retry, typically a parsed and capped Retry-After. When fn's returned error is transient AND implements this interface with a positive duration, Do waits that duration before the next attempt instead of its jittered exponential backoff. The exponential base still advances, so a later transient error without a hint resumes the normal progression. The hint MUST already be capped by the implementer (e.g. via ParseRetryAfter); Do sleeps on it verbatim and applies no ceiling of its own, so an uncapped value is an unbounded-wait hazard.

It embeds error for the same reason Transient does: the interface is read through errors.As/errors.AsType over an error tree, so it can only ever name an error.

type RetryRoundTripper

type RetryRoundTripper struct {
	// contains filtered or unexported fields
}

RetryRoundTripper implements http.RoundTripper with automatic retry.

By default, only idempotent methods (GET, HEAD, OPTIONS, TRACE) are retried. Use TransportConfig.RetryNonIdempotent to also retry POST/PUT/PATCH/DELETE when the request has a GetBody function for body replay.

Inspired by hashicorp/go-retryablehttp, but operates directly on stdlib *http.Request without a custom request type, and counts TOTAL attempts (TransportConfig.MaxAttempts) rather than go-retryablehttp's retries-beyond-first.

func NewRetryRoundTripper

func NewRetryRoundTripper(next http.RoundTripper, cfg TransportConfig) *RetryRoundTripper

NewRetryRoundTripper creates a RetryRoundTripper wrapping next with the given configuration. If next is nil, http.DefaultTransport is used. TransportConfig{} gives the defaults (see its field docs).

func (*RetryRoundTripper) RoundTrip

func (rt *RetryRoundTripper) RoundTrip(req *http.Request) (*http.Response, error)

RoundTrip implements http.RoundTripper. It retries eligible requests on transient failures with jittered exponential backoff, honoring Retry-After. Per the http.RoundTripper contract, the caller's request is never mutated.

When retries are exhausted the final response is returned, not an error: if the last attempt produced a retryable response (e.g. a 503), RoundTrip returns that *http.Response with a nil error, exactly as a non-retried request would. The caller owns the returned body (must Close it) and MUST inspect resp.StatusCode - a nil error does not imply a 2xx response.

type Secret

type Secret string

Secret is a credential value being redacted: the needle the redaction helpers scan FOR, never the text they scan. It is a distinct type so the value-to-hide and the text-to-scan cannot be transposed at a call site — a reversed call turns the redactor into a leak, scanning the credential for occurrences of the log text and returning the credential untouched. With the positions typed apart, a plain string variable in the secret position is a compile error; the caller converts exactly the value that is the credential (httpx.Secret(token)) and nothing else fits there.

Convert at the call boundary, never store one: Secret carries no logging protection of its own — formatting a Secret with %v/%s or logging it via slog prints the credential verbatim. The zero value ("") disables redaction: every helper returns its input unredacted when the secret is empty, because there is nothing to scan for. Callers redacting a possibly-absent credential must treat empty as "no redaction happened", not as "redaction succeeded".

type StatusError

type StatusError struct {
	URL  string
	Code int
}

StatusError represents a non-2xx response with URL context. Used by GetBytes. Supports errors.Is matching against ErrRateLimited and ErrServerError.

func (*StatusError) Error

func (e *StatusError) Error() string

func (*StatusError) Is

func (e *StatusError) Is(target error) bool

Is reports whether this StatusError matches ErrRateLimited or ErrServerError.

type Transient

type Transient interface {
	error
	IsTransient() bool
}

Transient is the interface an error implements to report whether it represents a transient (retryable) failure. It embeds error because every value this interface can ever name IS one: it is read through errors.As/errors.AsType over an error tree, whose nodes are errors by construction, so a non-error implementation is unreachable. Declaring the error half makes the type say that, keeps Error() available on a recovered value for diagnostics, and is what lets a caller write errors.AsType[Transient](err) — errors.AsType constrains its type parameter to error, where the older errors.As accepted any target. Same shape as net.Error.

type TransportConfig

type TransportConfig struct {
	// CheckRetry overrides the retry policy. nil means the default policy:
	// transient transport errors plus 429/502/503/504 responses (deliberately
	// narrower than GetBytes, which retries every 5xx; supply a custom policy
	// to broaden the set when an upstream returns transient 500s).
	CheckRetry CheckRetry
	// OnRetry is a hook called before each retry attempt, the transport's only
	// observability seam (it logs nothing itself).
	OnRetry OnRetry
	// PrepareRetry is called before each retry to mutate the cloned request
	// (e.g. refresh an auth token).
	PrepareRetry PrepareRetry
	// MaxAttempts is the TOTAL attempt count including the initial request.
	// Zero means unset and takes DefaultMaxAttempts (3); a NEGATIVE value
	// means exactly one attempt (the "try once" configuration — v2 expressed
	// it as WithRTMaxAttempts(0), but a zero struct field cannot distinguish
	// absent from zero, so v3 moves try-once to negatives).
	MaxAttempts int
	// BaseDelay is the initial backoff delay; non-positive means
	// DefaultBaseDelay (1s). Waits are equal-jitter with overflow-safe
	// doubling, and a retryable response's Retry-After header (capped at
	// RetryAfterCap) overrides the computed wait.
	BaseDelay time.Duration
	// MaxElapsedTime is a hard ceiling on total time across retries,
	// including any honored Retry-After: when now+wait would meet or pass it,
	// the round-tripper aborts with an error instead of sleeping. It is
	// checked BETWEEN attempts and cannot interrupt a stalled in-flight
	// attempt; bound single attempts on the base transport (e.g.
	// ResponseHeaderTimeout) and the whole call with the request context.
	// Zero means no ceiling.
	MaxElapsedTime time.Duration
	// RetryNonIdempotent enables retry of non-idempotent methods (POST, PUT,
	// PATCH, DELETE) when the request has a GetBody function for body replay.
	RetryNonIdempotent bool
}

TransportConfig configures a RetryRoundTripper (and, through NewRetryClient, the retrying client). The zero value is ready to use and behaves exactly like an unconfigured v2 round-tripper: three total attempts, one-second base delay, the default retry policy, no hooks, no elapsed ceiling, no non-idempotent replay.

type Validators

type Validators struct {
	// ETag is replayed as If-None-Match.
	ETag string
	// LastModified is replayed as If-Modified-Since.
	LastModified string
}

Validators carries the cache validators captured from a previous 200 response, replayed on the next conditional request so an unchanged resource is a cheap 304 instead of a re-download. Persist them alongside the cached body; the zero value sends no conditional headers (forcing a full 200). DoConditional validates both fields in both directions (see its validator hygiene contract), so values it captured are always safe to persist and replay verbatim.

Directories

Path Synopsis
Package certtest provides throwaway self-signed certificate material for tests.
Package certtest provides throwaway self-signed certificate material for tests.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL