Documentation
¶
Overview ¶
Package httpx is a resilient outbound-HTTP toolkit: transient-error classification, retries over one jittered exponential backoff loop, Retry-After parsing, status mapping, secret redaction, body helpers, custom-CA TLS transports and redirect allowlists. They live together because they compose into one net/http.Client.
Retry doors ¶
Three entry points share one option vocabulary and one backoff progression, and passing an option to the wrong door does not compile:
- Do retries a typed operation. You build the requests and keep the typed result.
- GetBytes is a bounded-bytes GET with redacted diagnostics. It owns the request, the body cap and the close.
- NewRetryRoundTripper retries transparently beneath any client, under a TransportConfig.
NewClient and NewRetryClient build clients with a redirect policy in place.
Errors ¶
IsTransient decides retryability, extended through the Transient and RetryAfterHint interfaces or MarkTransient. Permanent marks an error non-retryable. A context deadline is the total budget and is never retried, while AttemptTimeout marks the end of one attempt, which is. CheckHTTPStatus returns nil only for a 2xx status and a typed error for every other one, matched with errors.As or errors.Is.
Secrets ¶
RedactSecret, RedactSecretString, RedactTransportError and LogSafeError keep credentials out of logs and errors, and GetBytes never logs or returns a raw URL. A caller adding a normalizing transform or a byte cap follows the order on RedactSecretString: redact, normalize, redact, cap.
The README lists the whole API. docs/retries.md, docs/timeouts.md, docs/responses.md and docs/security-model.md hold the full contracts.
Index ¶
- Constants
- Variables
- func AttemptTimeout(err error) error
- func CATransport(pem []byte) (*http.Transport, error)
- func CheckHTTPStatus(resp *http.Response) error
- func CloneDefaultTransport() (*http.Transport, error)
- func ContextWithDefaultTimeout(ctx context.Context, def time.Duration) (context.Context, context.CancelFunc)
- func DefaultRedirectPolicy(req *http.Request, via []*http.Request) error
- func Do[T any](ctx context.Context, fn func(ctx context.Context) (T, error), opts ...DoOption) (T, error)
- func DockerGitHubRedirectPolicy(req *http.Request, via []*http.Request) error
- func Drain(body io.ReadCloser)
- func DrainClose(rc io.ReadCloser)
- func GetBytes(ctx context.Context, client *http.Client, reqURL string, opts ...GetOption) ([]byte, error)
- func IsAttemptTimeout(err error) bool
- func IsPermanent(err error) bool
- func IsRetryableStatus(code int) bool
- func IsTransient(err error) bool
- func JitteredBackoff(backoff time.Duration) time.Duration
- func LimitedBody(resp *http.Response, limit int64) io.ReadCloser
- func LogSafeError(err error) error
- func MarkTransient(err error) error
- func NewClient(timeout time.Duration) *http.Client
- func NewRetryClient(base http.RoundTripper, policy CheckRedirect, cfg TransportConfig) *http.Client
- func ParseRetryAfter(h string) time.Duration
- func ParseRetryAfterResponse(resp *http.Response) time.Duration
- func Permanent(err error) error
- func ReadLimitedBody(body io.ReadCloser, limit int64) ([]byte, error)
- func RedactSecret(err error, secret string) error
- func RedactSecretString(s string, secret Secret) string
- func RedactTransportError(err error, prefix string, secret Secret) error
- func RedirectPolicyFunc(opts ...RedirectOption) func(*http.Request, []*http.Request) error
- func RefuseAllRedirects(*http.Request, []*http.Request) error
- func SafeDouble(d time.Duration) time.Duration
- func SleepCtx(ctx context.Context, d time.Duration) error
- type AuthError
- type CheckRedirect
- type CheckRetry
- type ConditionalResult
- type DoOption
- type GetOption
- type HTTPStatusError
- type OnRetry
- type Option
- type PermanentError
- type PrepareRetry
- type RateLimitError
- type RedirectOption
- func WithAllowSchemeDowngrade(allow bool) RedirectOption
- func WithAllowedHosts(hosts ...string) RedirectOption
- func WithAllowedSuffixes(suffixes ...string) RedirectOption
- func WithMaxHops(n int) RedirectOption
- func WithPreserveMethod(preserve bool) RedirectOption
- func WithSameHost(same bool) RedirectOption
- func WithSameOrigin(same bool) RedirectOption
- type RedirectReason
- type RedirectRefusedError
- type ResponseTooLargeError
- type RetryAfterHint
- type RetryRoundTripper
- type Secret
- type StatusError
- type Transient
- type TransportConfig
- type Validators
Examples ¶
Constants ¶
const ( // DefaultBaseDelay is the production base for exponential-backoff retry. DefaultBaseDelay = time.Second // DefaultMaxAttempts caps the retry doors at three total attempts. DefaultMaxAttempts = 3 // DefaultMaxBodyBytes caps response bodies at 10 MB. DefaultMaxBodyBytes int64 = 10 << 20 // RetryAfterCap is the maximum Retry-After honor duration. RetryAfterCap = 60 * time.Second )
Variables ¶
var ErrNoCertsInPEM = errors.New("httpx: no PEM-encoded certificates found")
ErrNoCertsInPEM is returned by CATransport when the supplied PEM data contains no parseable certificates. A misconfigured or empty CA file therefore fails loudly rather than silently producing a transport that trusts nothing (which would reject every connection with an opaque error).
var ErrRateLimited = errors.New("rate limited")
ErrRateLimited is a sentinel callers use with errors.Is to detect 429 responses.
var ErrServerError = errors.New("server error")
ErrServerError is a sentinel for upstream 5xx responses.
Functions ¶
func AttemptTimeout ¶
AttemptTimeout wraps err to declare that the timeout it reports bounded ONE ATTEMPT rather than the caller's total budget, making it retryable: IsTransient reports true for it (checked ahead of the context-error rejection), and the retry doors retry it. Returns nil for nil input. It is the mirror of Permanent, which forces the opposite verdict.
It exists because a context deadline alone cannot express the difference. A caller's expired deadline means "the budget is gone, stop"; the expiry of a per-attempt bound means "this attempt failed, try another" — opposite instructions from the same context.DeadlineExceeded. Only the code that installed the bound knows which one it is, so httpx cannot infer it and defaults to the safe reading (terminal). This mark is how the installer says otherwise, and the wrapper deliberately keeps the deadline visible to errors.Is instead of hiding it, which was the only way to opt in before.
Reach for it when the per-attempt bound is NOT the retry door's own and httpx cannot see whose bound it was:
- a context.WithTimeout a caller derives itself inside a Do callback. Its expiry carries the deadline sentinel, indistinguishable from the caller's own. WithAttemptTimeout does this for you, mark included; hand-roll it only when the bound must cover something other than one whole attempt.
- a net-level bound such as a net.Dialer Timeout. The net package maps every expired context onto one shared "i/o timeout" value, so a *net.OpError or *net.DNSError reporting a deadline cannot say whose it was and stays terminal (see IsTransient).
An http.Client.Timeout and a net/http.Transport ResponseHeaderTimeout do NOT need the mark: net/http reports those through its own timeout error, which never carries the sentinel, so IsTransient already reads them as per-attempt and retries them — including under the RetryRoundTripper's default policy. Use Permanent (or a TransportConfig.CheckRetry) if you want one of them to stop the loop instead.
The caller owns the judgment: mark ONLY a bound you installed over a single attempt. Marking a deadline that came from your own caller makes an exhausted budget look retryable, which is the failure mode the default classification prevents. WithAttemptTimeout avoids the judgment entirely by deciding from the two contexts it owns.
func CATransport ¶
CATransport builds an *http.Transport that trusts ONLY the CA certificate(s) in pem for TLS verification (pinning). It is cloned from http.DefaultTransport (via CloneDefaultTransport), so it keeps the standard connection pooling, dial/keepalive timeouts, HTTP/2 negotiation, and proxy support (ProxyFromEnvironment). A FRESH TLS config is then installed — RootCAs from pem, a TLS 1.2 minimum, and verification always ENABLED (InsecureSkipVerify is not set). Any TLS settings already on http.DefaultTransport are intentionally NOT carried over, so the returned transport's trust posture cannot be weakened by a program that globally mutated the default transport's *tls.Config (e.g. set InsecureSkipVerify or an accept-all VerifyPeerCertificate hook).
The supplied CA(s) are the SOLE trust anchors: the transport rejects any host not chaining to them, including public-CA hosts. This is the right setup for a known self-hosted endpoint presenting a private or self-signed certificate (a Plex server, an internal API).
The returned *http.Transport is concrete and mutable, so callers may tune fields such as MaxIdleConnsPerHost or pass it as the base RoundTripper of NewRetryRoundTripper:
tr, err := httpx.CATransport(pem)
client := httpx.NewRetryClient(tr, httpx.DefaultRedirectPolicy, httpx.TransportConfig{MaxAttempts: 3})
It returns ErrNoCertsInPEM when pem yields no certificates. The caller owns reading the PEM bytes (from a file, a secret, an env var), which keeps this function I/O-free and lets the caller bound the read as it sees fit.
CATransport requires http.DefaultTransport to be the standard library's *http.Transport (the default). If your program has replaced it with a wrapping RoundTripper (for example request instrumentation), CATransport returns an error.
func CheckHTTPStatus ¶
CheckHTTPStatus returns nil for a 2xx and otherwise a typed error: 401/403 → *AuthError, 429 → *RateLimitError, anything else (1xx, 3xx, other 4xx, 5xx) → *HTTPStatusError carrying the code, transient only for 502/503/504. A 3xx reaches it only from a client that does not follow redirects (see RefuseAllRedirects) and is reported as a failure, not a completed request.
func CloneDefaultTransport ¶
CloneDefaultTransport returns a private clone of http.DefaultTransport, keeping the standard connection pooling, dial/keepalive timeouts, HTTP/2 negotiation, and proxy support (ProxyFromEnvironment). The clone is the caller's to mutate — set a per-attempt ResponseHeaderTimeout, tune MaxIdleConnsPerHost, or pass it as the base RoundTripper of NewRetryRoundTripper — without the global footgun of mutating http.DefaultTransport itself, which would silently reconfigure every other client in the process.
It returns an error when http.DefaultTransport has been replaced by a non-*http.Transport RoundTripper (request instrumentation, a test stub): a wrapping RoundTripper offers no concrete transport to clone, and failing loudly beats silently dropping the wrapper's behavior. It is the building block CATransport is assembled on.
func ContextWithDefaultTimeout ¶
func ContextWithDefaultTimeout(ctx context.Context, def time.Duration) (context.Context, context.CancelFunc)
ContextWithDefaultTimeout bounds ctx by def ONLY when ctx carries no deadline; a deadline the caller already set is the authoritative budget and is never undercut (or extended). A def <= 0 means "no default" and leaves ctx unbounded. The returned cancel is always non-nil and must be called (a no-op on the passthrough paths), matching context.WithTimeout's contract.
It is the per-request timeout rule for an API client: per-attempt bounds live on the transport (ResponseHeaderTimeout), the total budget is the caller's ctx, and this default applies only when the caller brought no budget of its own. Extracted from the identical requestContext helpers in the plexapi and arrapi client libraries.
func DefaultRedirectPolicy ¶
DefaultRedirectPolicy is the default redirect policy: it allows a redirect only to the same host as the original request, and refuses a same-host https->http scheme downgrade (which would forward a custom auth header onto a cleartext hop). A cross-host redirect is refused (Go forwards a custom header across a redirect, so it would leak) and an http->https upgrade is allowed. It delegates to RedirectPolicyFunc(WithSameHost(true)), with one addition: a call with an empty via chain (which net/http never produces — via always carries at least the original request) is allowed rather than refused. Use RedirectPolicyFunc for a custom allowlist, a higher hop cap (WithMaxHops), or to permit downgrades (WithAllowSchemeDowngrade).
func Do ¶
func Do[T any](ctx context.Context, fn func(ctx context.Context) (T, error), opts ...DoOption) (T, error)
Do calls fn up to WithMaxAttempts times (total) with jittered exponential backoff and returns the first success; a non-retryable error returns at once. By default only IsTransient errors retry (a *RateLimitError is not transient) and a positive RetryAfterHint replaces the backoff wait. After a failed attempt a canceled context returns ctx.Err(), except under WithRateLimitOnly, where the final attempt's error wins. Retries log at Debug to WithLogger; exhaustion logs at Warn, or Debug for a single-attempt budget (see exhaustedLevel).
Example ¶
package main
import (
"context"
"fmt"
"time"
"github.com/cplieger/httpx/v5"
)
func main() {
attempts := 0
// context.Background() (not t.Context()): an Example has no *testing.T.
result, err := httpx.Do(context.Background(), func(_ context.Context) (string, error) {
attempts++
if attempts < 2 {
return "", &httpx.HTTPStatusError{Code: 503}
}
return "done", nil
}, httpx.WithMaxAttempts(3), httpx.WithBaseDelay(time.Millisecond), httpx.WithLabel("example"))
if err != nil {
fmt.Println("error:", err)
return
}
fmt.Println(result)
}
Output: done
func DockerGitHubRedirectPolicy ¶
DockerGitHubRedirectPolicy is an OPTIONAL example redirect policy allowing docker.com and github.com hosts. Like every shipped policy it refuses an https->http scheme downgrade (judged against the original request's scheme, see WithAllowSchemeDowngrade), so a custom auth header never rides a cleartext hop even to an allowlisted host. Use it by assigning to Client.CheckRedirect or pass RedirectOption values to RedirectPolicyFunc for other allowlists.
func Drain ¶
func Drain(body io.ReadCloser)
Drain reads and discards up to 64 KB of a response body to enable HTTP connection reuse. A failed drain forfeits only that reuse, and is reported as a bare Debug line on slog.Default(). The read error itself is deliberately NOT logged.
Go 1.27 made HTTP/1 Response.Body drain itself on Close, which raises the question of whether this still earns its place. It does, on two measured grounds. The stdlib drain is HTTP/1 only — net/http/internal/http2 has no equivalent — so on an HTTP/2 response, which every transport this package builds negotiates (CloneDefaultTransport keeps ForceAttemptHTTP2), this is the only drain there is. And Drain takes an io.ReadCloser, not an *http.Response, so it also covers a body the stdlib never wraps. Where both apply they compose rather than collide: this drain runs first and, for a body inside the 64 KB budget, reaches EOF, which short-circuits the stdlib's drain entirely.
The two limits differ deliberately and do not need reconciling. 64 KB is how much effort this function spends; net/http's own budget is 256 KiB with a 50 ms deadline, and it applies only when unread content remains AND resp.ContentLength <= 256 KiB. That gate is why an over-limit body refused by ReadLimitedBody reads nothing extra: a response declaring more than 256 KiB is excluded outright. Measured on go1.27.0 against go1.26.7, a declared 10 MB body read to a 1 MB cap pulls an identical 1,114,112 bytes on both. A CHUNKED over-limit body has ContentLength == -1, which passes the gate, so its worst case is the cap plus at most 256 KiB more. Either way the caller pays no latency for it: the drain runs on net/http's read-loop goroutine, and Close() measured 24-44 microseconds against a deliberately stalled chunked body that provably passes the gate.
That omission is a CWE-532 fix, not an oversight. A body-read error's text is written by the FAR END: net/http renders a malformed chunked trailer as
malformed MIME header: missing colon: "<remote bytes>"
and for a consumer whose request URL carries its credential in the PATH (a webhook token is the canonical shape) an edge that echoes the request URI puts that credential in those bytes. Logging it needs only the consumer to be running at Debug — the level an operator raises precisely while diagnosing failing deliveries. Three properties make the site uncloseable from anywhere else, which is why the value is dropped HERE rather than by the caller:
- Nobody can route it. Drain takes no logger (it is called from defer at consumer call sites and from four sites inside this package, one of them a RoundTripper that has no logger at all), and the retry doors' WithLogger governs their own loop lines, not slog.Default().
- LogSafeError does not reduce it. That boundary is TYPE-based: it strips the *url.Error envelope this package's own machinery adds. A body-read error is not a *url.Error — a malformed trailer surfaces as textproto.ProtocolError, whose entire value IS the message — so the reduction returns it byte-identically.
- Redaction needs a secret to redact, and Drain is handed none.
No diagnosis a caller can act on is lost. Drain runs only where the body is being thrown away, so the response's own outcome is already reported by the path that discarded it (CheckHTTPStatus's typed error, the retry lines, ReadLimitedBody's error); a drain error is never the only signal of a failure, and no caller can repair a drain. This is the same call the package makes in urlErrorCause, which substitutes a contentless stand-in rather than log the one field it holds.
func DrainClose ¶
func DrainClose(rc io.ReadCloser)
DrainClose reads remaining bytes (up to drainLimit) from rc before closing it. It inherits Drain's logging contract: a failed drain logs a bare Debug line carrying no remote-authored text.
func GetBytes ¶
func GetBytes(ctx context.Context, client *http.Client, reqURL string, opts ...GetOption) ([]byte, error)
GetBytes performs a GET, retrying 429, 5xx and transient transport errors with bounded backoff (honoring Retry-After up to RetryAfterCap), and returns the body read to WithMaxBodyBytes (*ResponseTooLargeError past it). Any other non-2xx is a permanent *StatusError. Every logged url and returned error is redacted. It keeps its own loop rather than wrapping RetryRoundTripper, which has no body cap, redaction or 5xx-wide retry, so merging the two would change this door's behavior.
Example ¶
package main
import (
"context"
"fmt"
"net/http"
"net/http/httptest"
"time"
"github.com/cplieger/httpx/v5"
)
func main() {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
fmt.Fprint(w, "hello")
}))
defer srv.Close()
// context.Background() (not t.Context()): an Example has no *testing.T.
body, err := httpx.GetBytes(context.Background(), srv.Client(), srv.URL,
httpx.WithMaxAttempts(3),
httpx.WithBaseDelay(100*time.Millisecond),
)
if err != nil {
fmt.Println("error:", err)
return
}
fmt.Println(string(body))
}
Output: hello
func IsAttemptTimeout ¶
IsAttemptTimeout reports whether err (or any wrapped error) was marked by AttemptTimeout as a per-attempt timeout.
func IsPermanent ¶
IsPermanent reports whether err (or any wrapped error) is a *PermanentError.
func IsRetryableStatus ¶
IsRetryableStatus reports whether a response status is one the retry loop treats as a transient failure worth another attempt: 408 Request Timeout, 429 Too Many Requests, and any 5xx (a code >= 500). Everything else — every 2xx, every 3xx, an informational 1xx, and every other 4xx — is a settled answer this package does not repeat.
It is the SAME rule the built-in retry uses, not a restatement of it: GetBytes's own attempt function calls this function to classify a response, so a caller's verdict and the door's verdict cannot disagree.
It exists for the caller that runs GetBytes under WithMaxAttempts(1) inside its own outer retry budget (the sanctioned way to avoid multiplying the two attempt counts). That caller receives the door's *StatusError and owns the retry decision — GetBytes deliberately does not mark its exhaustion error Transient, because whether an exhausted GET is worth another outer attempt is the caller's policy — so it needs to ask the question the door would have asked:
if se, ok := errors.AsType[*httpx.StatusError](err); ok && httpx.IsRetryableStatus(se.Code) {
return httpx.MarkTransient(err) // spend another of MY attempts
}
408 is included because it is the server reporting that IT gave up waiting, which self-heals and is safe to repeat on an idempotent GET. 429 is here because this door retries a rate limit by default; Do does not (its *RateLimitError is retryable only under WithRateLimitRetry or WithRateLimitOnly), so pair this predicate with the door whose policy it describes.
The RetryRoundTripper is the one deliberate divergence: its default policy retries 429/502/503/504 only, a narrower set than this, so a plain 500 that this predicate calls retryable is not retried by the transport. Widen the transport with TransportConfig.CheckRetry (this predicate is a ready-made one) rather than assuming the two agree.
func IsTransient ¶
IsTransient returns true for errors likely caused by temporary server or network issues worth retrying. Auth, rate-limit, permanent, and caller-context errors are never transient.
"Caller-context error" is decided by the sentinel actually being in the error's unwrap chain, not by errors.Is — net/http's timeout errors match context.DeadlineExceeded without carrying it, so an http.Client.Timeout and a transport ResponseHeaderTimeout are per-ATTEMPT bounds and ARE transient (see isCallerContextError). A caller's own expired deadline stays terminal; AttemptTimeout marks a bound whose expiry this package cannot otherwise tell apart from one.
func JitteredBackoff ¶
JitteredBackoff returns a duration in [backoff/2, backoff] using the "equal jitter" strategy (per AWS Builders' Library). Full jitter and decorrelated jitter are intentionally not provided — equal jitter is the recommended default for HTTP retry as it avoids thundering herd while maintaining a minimum backoff floor.
func LimitedBody ¶
func LimitedBody(resp *http.Response, limit int64) io.ReadCloser
LimitedBody wraps resp.Body with an io.LimitReader capped at limit bytes, preserving the original Close method.
func LogSafeError ¶
LogSafeError returns an error whose message is safe to log. A transport *url.Error embeds the full request URL (with any userinfo/query secrets), so it is reduced to its underlying cause. Nil returns nil, and a non-nil error NEVER reduces to nil (see urlErrorCause); *StatusError already renders a redacted URL via Error(), so it (and everything else) passes through unchanged — preserving errors.Is/As chains for callers.
httpx applies this reduction to every transport error it logs or wraps; it is exported so a caller wrapping transport errors into its own messages can apply the same one (equivalent to RedactTransportError(err, "", "") — reach for that variant when a known secret must also be scrubbed from the text).
func MarkTransient ¶
MarkTransient wraps err in a value satisfying the Transient interface with a true verdict, so the retry doors treat it as a transient failure. Returns nil for nil input. It is the mirror of Permanent, which forces the opposite verdict; the name is not the bare adjective only because Transient is the interface it satisfies.
It exists so a caller widening the retryable set for its own operation does not have to declare a one-method wrapper type to do it. That wrapper is pure boilerplate — an Error, an Unwrap and an IsTransient returning true — and hand-rolling it invites the two mistakes this function cannot make: forgetting Unwrap (which hides the cause from every errors.Is and errors.As the caller's own callers run) and marking a nil error.
The mark is the OUTERMOST verdict, so it overrides a non-transient verdict already on the error: MarkTransient(&HTTPStatusError{Code: 500}) is retryable even though a plain 500 is not. Use it for a failure your operation knows is self-healing where the shared policy cannot know — a server-side fault delivered inside a 200 envelope, for example, which no status-based classification can see:
if env := serverFault(body); env != nil {
return httpx.MarkTransient(env) // this clears; spend another attempt
}
IsTransient's standing rejections still outrank the mark, because they answer questions the caller is not the authority on:
- Permanent wins. Permanent(MarkTransient(err)) is not retried.
- An *AuthError and a *RateLimitError stay terminal. Credentials do not fix themselves, and a rate limit is retried by naming a wait budget (WithRateLimitRetry, WithRateLimitOnly), not by claiming transience.
- A caller-context error stays terminal: the budget that authorized the work is gone, so another attempt has nothing to spend. For a timeout that bounded ONE attempt and therefore carries a deadline that is not the caller's, use AttemptTimeout — IsTransient consults that mark ahead of the context rejection, which is the whole reason it is a separate mark.
func NewClient ¶
NewClient returns an *http.Client with the given timeout and the DefaultRedirectPolicy (same-host only). For custom redirect allowlists, configure CheckRedirect with RedirectPolicyFunc or assign DockerGitHubRedirectPolicy.
func NewRetryClient ¶
func NewRetryClient(base http.RoundTripper, policy CheckRedirect, cfg TransportConfig) *http.Client
NewRetryClient returns an *http.Client whose Transport is a RetryRoundTripper over base (nil means http.DefaultTransport) and whose CheckRedirect is policy. It panics on a nil policy, because nil means net/http's follow-anywhere default; pass DefaultRedirectPolicy, RefuseAllRedirects or a RedirectPolicyFunc. It sets no Client.Timeout, which would cap the whole retry sequence: bound one attempt on the base transport and the total with a context deadline or TransportConfig.MaxElapsedTime.
func ParseRetryAfter ¶
ParseRetryAfter parses a Retry-After header value (delta-seconds or HTTP-date). Returns zero for missing/malformed values. Caps at RetryAfterCap for safety (prevents unbounded waits in retry loops). For raw uncapped values, use ParseRetryAfterResponse.
func ParseRetryAfterResponse ¶
ParseRetryAfterResponse parses the Retry-After header from an *http.Response. Returns zero if absent or unparseable. Does NOT cap — preserves the raw duration so callers (e.g., CheckHTTPStatus) can make their own decisions. For capped values suitable for retry loops, use ParseRetryAfter.
func Permanent ¶
Permanent wraps err to indicate it should never be retried. Mirrors cenkalti/backoff.Permanent().
func ReadLimitedBody ¶
func ReadLimitedBody(body io.ReadCloser, limit int64) ([]byte, error)
ReadLimitedBody reads body up to limit bytes, always closes body, and returns the bytes read. It reads one byte past limit to detect an over-limit body and returns *ResponseTooLargeError (with nil bytes) rather than a silently truncated payload — a truncated body indistinguishable from a complete one is a corruption hazard. A limit of math.MaxInt64 means "effectively unlimited" and is guarded against probe-size overflow.
It is the read-all-with-overflow-detection companion to LimitedBody (which only caps the stream and leaves reading and overflow handling to the caller), and is the same cap+1 read GetBytes applies internally — exposed for callers that issue their own request and decode outside GetBytes but still want the fail-loud size bound. On any error the body is already closed.
func RedactSecret ¶
RedactSecret replaces occurrences of secret in err's message with "REDACTED". It funnels into RedactSecretString, whose doc comment carries the ordering and representation rules the caller owns.
func RedactSecretString ¶
RedactSecretString replaces every occurrence of secret in s with "REDACTED" and returns the result. It is the string-level building block behind RedactSecret and RedactTransportError, exposed for callers that must redact a secret from a plain string — a captured HTTP response body destined for an error field or a log line — rather than from an error value. An empty secret is a no-op (s is returned unchanged), matching the error-shaped variants.
The replacement is literal and byte-exact, which puts three obligations on the caller:
- Hold the needle in the SAME representation the haystack carries. A decoded token does not match a JSON-escaped or percent-encoded haystack, and the mismatch is silent: nothing is replaced and no error is reported.
- Call this on BOTH sides of any normalizing transform (runesafe Sanitize/SanitizeSingleLine, unicode/norm NFC/NFKC, strconv.Unquote, JSON/URL/HTML unescape, case folding, whitespace collapse), passing the needle in the representation each side carries. A call placed only before the transform misses a secret the transform CONSTRUCTS out of text the needle never matched; a call placed only after it misses a secret the transform REWRITES, leaving a near-complete fragment of the value.
- Put any byte cap (ReadLimitedBody, a bounded sanitizer preset, a plain truncation) LAST. A cut that splits the value leaves a surviving prefix this function can no longer match.
Composed, that order is: redact, normalize, redact, cap.
This package performs one of those transforms ¶
Every URL httpx renders — *StatusError.Error(), and every "url" attribute in a GetBytes log line — goes through a parse-and-re-serialize step that blanks query values and drops userinfo. That step is a URL-encoding transform in rule 2's sense, so redacting a known secret out of one of those strings is the "only after the transform" mistake. Measured on go1.27.0 (identical on go1.26.7): for a secret in the request PATH, which is the one credential-bearing position the rendering keeps verbatim, a secret containing a space renders as tok%20en, a non-ASCII byte as tok%C3%A9n, and a '#' truncates the value at the fragment boundary — so a byte-exact needle held in the caller's own representation matches none of them and the value reaches the log percent-encoded. A stray '%' is the safe case: the URL fails to parse and the whole rendering collapses to a fixed placeholder.
So a caller holding a credential that rides in a URL PATH must redact it before the URL reaches this package (or pass the needle in its percent-encoded form as well). A credential in the query string or the userinfo needs nothing: the rendering removes those wholesale, without matching anything.
func RedactTransportError ¶
RedactTransportError unwraps *url.Error and redacts the secret from the error message. Returns nil for nil input, and never nil for a non-nil one (see urlErrorCause for the *url.Error that carries no cause of its own). The replacement runs through RedactSecretString, whose doc comment carries the ordering and representation rules for composing redaction with a normalizing transform or a byte cap. An empty secret disables redaction: the (possibly prefixed) error is returned with its message untouched, since there is nothing to scan for.
func RedirectPolicyFunc ¶
RedirectPolicyFunc returns a CheckRedirect function configured with the given options. A redirect is followed only to an allowed target (a WithAllowedHosts entry, a WithAllowedSuffixes match, the original host under WithSameHost, the original origin under WithSameOrigin) and, unless WithAllowSchemeDowngrade is set, never https->http. With none of those permissions every redirect is refused. The hop cap is WithMaxHops. Every hard refusal is a *RedirectRefusedError; WithPreserveMethod(true) instead answers http.ErrUseLastResponse for a hop that would change the method.
func RefuseAllRedirects ¶
RefuseAllRedirects is a CheckRedirect policy that follows no redirect: the client returns the 3xx response itself with a nil error. Use it for a token-bearing client of an API that never redirects, because net/http forwards custom headers (X-Plex-Token, X-Api-Key) across a redirect and a hostile 302 would leak them. CheckHTTPStatus reports the surfaced 3xx as an error.
func SafeDouble ¶
SafeDouble doubles a duration, guarding against int64 overflow.
Types ¶
type AuthError ¶
type AuthError struct{ Msg string }
AuthError indicates invalid or expired credentials.
type CheckRedirect ¶
CheckRedirect is the http.Client.CheckRedirect function shape. It is a type alias, so values are assignable in both directions; every shipped policy (DefaultRedirectPolicy, RefuseAllRedirects, DockerGitHubRedirectPolicy, and anything built with RedirectPolicyFunc) is a CheckRedirect.
type CheckRetry ¶
CheckRetry is the signature for a retry policy function. It receives the context, the response (may be nil on transport error), and the error (nil on successful response). It returns whether to retry and an optional error to short-circuit with. Mirrors hashicorp/go-retryablehttp CheckRetry.
type ConditionalResult ¶
type ConditionalResult struct {
// Validators are the fresh validators captured from a 200 response's ETag /
// Last-Modified headers (either may be empty when the server sent none).
// Zero on a 304: the caller keeps the validators it already holds.
Validators Validators
// Body is the full response body of a 200, bounded by the maxBodyBytes
// given to DoConditional. Nil on a 304.
Body []byte
// NotModified reports a 304: the cached representation is still current.
NotModified bool
}
ConditionalResult is one conditional-request outcome. Fields are ordered largest-alignment-first for govet fieldalignment.
func DoConditional ¶
func DoConditional(client *http.Client, req *http.Request, v Validators, maxBodyBytes int64) (ConditionalResult, error)
DoConditional sends req once with If-None-Match/If-Modified-Since set only from v (replacing any on req) and classifies the response: 304 is NotModified; 200 returns the body (*ResponseTooLargeError over maxBodyBytes, where <= 0 means DefaultMaxBodyBytes) and its validators; a non-2xx is CheckHTTPStatus's error and any other 2xx a plain error. Transport errors pass through LogSafeError. The caller owns retry (wrap in Do, rebuild req per attempt) and must send zero Validators when its cached body is unusable. A validator failing RFC 9110 field-value grammar or 1 KiB is silently dropped.
type DoOption ¶
type DoOption interface {
// contains filtered or unexported methods
}
DoOption configures Do. Options constructed as Option values apply to both loop doors; Do-only options (WithLabel, WithRateLimitRetry, WithRateLimitOnly) implement only this interface, so passing one to GetBytes is a compile error.
func WithAttemptTimeout ¶
WithAttemptTimeout bounds EACH attempt by d and makes that bound's expiry RETRYABLE: the per-attempt (per-try) timeout, as distinct from the caller's total budget. It is the piece that joins a per-attempt deadline to the retry loop; without it a deadline inside fn is classified terminal (see AttemptTimeout for why an error alone cannot say otherwise). A non-positive d means no per-attempt bound, which is the option's absence.
fn receives a context derived from the caller's with d applied. Context keeps the EARLIER deadline, so a caller deadline nearer than d still governs and the total budget is never extended — d caps one attempt, never the call.
The expiry is marked as the attempt's ONLY when the caller's own context is still live, so retrying a caller that is out of budget stays impossible: an expired or canceled caller context is never marked, Do returns ctx.Err() before it classifies anything, and SleepCtx refuses to wait on a dead context. The decision is made from the two contexts rather than from the error, because a context.DeadlineExceeded value is identical either way.
The attempt context is canceled as soon as fn returns, so fn must not return a value that still depends on it — read, decode, or drain a response body inside fn (the same rule a caller-derived per-attempt context already imposes).
It is a DoOption: GetBytes owns its own request and has no callback to bound. Bound that door by running it as one attempt inside Do — GetBytes(ctx, ...) with WithMaxAttempts(1) inside a Do callback carrying this option — the same composition that avoids multiplying two attempt budgets.
func WithLabel ¶
WithLabel sets the operation label used in Do's log lines ("<label> failed, retrying", "<label> retries exhausted"). Default: "operation".
func WithRateLimitOnly ¶
WithRateLimitOnly makes Do retry ONLY *RateLimitError (matched through wrapped errors); every other error, including transient transport errors, is returned immediately. The wait semantics match WithRateLimitRetry, the terminal Warn is "rate limit retries exhausted", and the FINAL attempt's error wins even under an already-canceled context (cancellation is observed in the always-positive inter-attempt sleep instead). Mutually exclusive with WithRateLimitRetry.
func WithRateLimitRetry ¶
WithRateLimitRetry makes Do additionally treat *RateLimitError as retryable, alongside the transient set. The wait before a rate-limit retry is min(err.RetryAfter, maxWait) when the error carries a positive hint, else maxWait; a non-positive maxWait falls back to RetryAfterCap, so the wait is always positive and a canceled context is observed before every retry. Transient errors keep their jittered-backoff waits (with the RetryAfterHint override), and the exponential base advances on every retry. Mutually exclusive with WithRateLimitOnly (Do returns a configuration error when both are supplied).
type GetOption ¶
type GetOption interface {
// contains filtered or unexported methods
}
GetOption configures GetBytes. GetBytes-only options (WithHeaders, WithMaxBodyBytes) implement only this interface, so passing one to Do is a compile error.
func WithHeaders ¶
WithHeaders sets a function that is called to set headers on each request.
func WithMaxBodyBytes ¶
WithMaxBodyBytes sets the maximum response body size to read. Default: DefaultMaxBodyBytes (10 MB). A non-positive value falls back to the default.
type HTTPStatusError ¶
type HTTPStatusError struct {
Code int
}
HTTPStatusError represents a non-2xx HTTP response not covered by AuthError or RateLimitError. Implements the Transient interface for 502/503/504.
func (*HTTPStatusError) Error ¶
func (e *HTTPStatusError) Error() string
func (*HTTPStatusError) IsClientError ¶
func (e *HTTPStatusError) IsClientError() bool
IsClientError reports whether the status code is 4xx.
func (*HTTPStatusError) IsServerError ¶
func (e *HTTPStatusError) IsServerError() bool
IsServerError reports whether the status code is 5xx.
func (*HTTPStatusError) IsTransient ¶
func (e *HTTPStatusError) IsTransient() bool
IsTransient reports whether the status code is a retryable server failure (502/503/504).
type OnRetry ¶
OnRetry is called before each retry attempt (not the initial request). attempt is 1-indexed (first retry = 1). resp may be nil on transport errors.
type Option ¶
Option is a retry option accepted by BOTH loop doors, Do and GetBytes: WithMaxAttempts, WithBaseDelay, and WithLogger construct Option values.
func WithBaseDelay ¶
WithBaseDelay sets the initial backoff delay. Default: DefaultBaseDelay (1s). A non-positive value falls back to the default.
func WithExhaustedLevel ¶
WithExhaustedLevel overrides the level of the terminal "retries exhausted" line (see exhaustedLevel for the default rule).
It exists for the caller that reports the same terminal failure itself, with more context than this door can have. Such a caller wants the per-attempt retry diagnostics and NOT the verdict: leaving both produces two log lines for one failure, and silencing the whole logger to stop the second throws the diagnostics away with it. Pass slog.LevelDebug to keep the line for diagnosis while the caller's own line carries the alarm.
It does not change what is returned, and it does not suppress the line: a level below the logger's threshold simply is not emitted, which is the caller's decision to make.
func WithLogger ¶
WithLogger sets the logger for retry diagnostics. Default: slog.Default(). A nil logger falls back to the default.
func WithMaxAttempts ¶
WithMaxAttempts sets the maximum number of attempts (TOTAL, including the first call). Default: DefaultMaxAttempts (3). A value below 1 is treated as 1, so the operation always runs at least once (never a silent no-op).
type PermanentError ¶
type PermanentError struct {
Err error
}
PermanentError wraps an error to signal that it should NOT be retried, regardless of other retry policies. Mirrors cenkalti/backoff.PermanentError. Use Permanent(err) to wrap.
func (*PermanentError) Error ¶
func (e *PermanentError) Error() string
func (*PermanentError) Is ¶
func (e *PermanentError) Is(target error) bool
Is allows errors.Is matching against other PermanentErrors.
func (*PermanentError) Unwrap ¶
func (e *PermanentError) Unwrap() error
type PrepareRetry ¶
PrepareRetry is called before each retry to allow request mutation (e.g., re-signing with a fresh token). Mirrors go-retryablehttp PrepareRetry.
type RateLimitError ¶
RateLimitError indicates a rate limit was exceeded. RetryAfter, when non-zero, is the RAW, UNCAPPED hint from the upstream's Retry-After header (populated via ParseRetryAfterResponse). The upstream controls this value; a hostile or misconfigured server can supply a very large duration (CWE-400 uncontrolled resource consumption). Callers that sleep on it directly MUST bound it first, e.g. min(err.RetryAfter, cap). Do's rate-limit modes (WithRateLimitRetry, WithRateLimitOnly) already do this (they cap at their maxWait argument). For a pre-capped value use ParseRetryAfter (bounded at RetryAfterCap = 60s).
func (*RateLimitError) Error ¶
func (e *RateLimitError) Error() string
type RedirectOption ¶
type RedirectOption func(*redirectCfg)
RedirectOption configures a redirect policy created by RedirectPolicyFunc.
func WithAllowSchemeDowngrade ¶
func WithAllowSchemeDowngrade(allow bool) RedirectOption
WithAllowSchemeDowngrade permits a redirect that downgrades the scheme (https on the original request -> http on the target). The default (false) refuses such a downgrade so a credential carried in a custom request header (which Go forwards across a redirect, stripping only Authorization/Cookie) is never sent over a cleartext hop. A scheme upgrade (http->https) is always allowed regardless of this setting. The downgrade is judged against the ORIGINAL request's scheme (via[0]).
func WithAllowedHosts ¶
func WithAllowedHosts(hosts ...string) RedirectOption
WithAllowedHosts sets the exact hostnames allowed as redirect targets.
func WithAllowedSuffixes ¶
func WithAllowedSuffixes(suffixes ...string) RedirectOption
WithAllowedSuffixes sets the domain suffixes allowed (e.g. ".docker.com").
func WithMaxHops ¶
func WithMaxHops(n int) RedirectOption
WithMaxHops makes the policy follow at most n redirects in one request and refuse the next with RedirectTooMany. n <= 0 means the default, 5.
func WithPreserveMethod ¶
func WithPreserveMethod(preserve bool) RedirectOption
WithPreserveMethod refuses, when true, a hop that would change the request method, as net/http does to a POST, PUT, PATCH or DELETE across a 301, 302 or 303 (RFC 9110 §15.4, https://go.dev/issue/18570). It returns http.ErrUseLastResponse, so the caller gets the 3xx with a nil error, as with RefuseAllRedirects. The method is compared with the original request (via[0]) and an empty via fails closed. The hop cap, target and downgrade refusals take precedence, and it grants nothing: with no allowlist, WithSameHost or WithSameOrigin every redirect is still refused.
func WithSameHost ¶
func WithSameHost(same bool) RedirectOption
WithSameHost permits a redirect whose target host equals the original request's host (ASCII case-insensitive, RFC 3986 §6.2.2.1), beside any allowlist entries. It follows that host on any port and an http->https upgrade there, so a custom auth header can reach another service on the host; WithSameOrigin keeps it on one origin. An https->http hop stays under WithAllowSchemeDowngrade. Later options win, so a later WithSameHost(false) turns the permission back off.
func WithSameOrigin ¶ added in v5.1.0
func WithSameOrigin(same bool) RedirectOption
WithSameOrigin permits a redirect to the original request's origin (RFC 6454): the same scheme, host (ASCII case-insensitive) and effective port, an absent port reading as 80 for http and 443 for https. Unlike WithSameHost it refuses another port on that host and an http->https upgrade. Under WithSameOrigin(true) the origin alone decides a hop to the original host, whatever WithSameHost or WithAllowedHosts say; the allowlist and the downgrade rule judge other hosts. A refusal is RedirectOtherOrigin.
type RedirectReason ¶ added in v5.1.0
type RedirectReason int
RedirectReason names the rule of a redirect policy that refused a hop.
const ( // RedirectNotAllowed means the policy follows no redirect at all (no // allowlist, no WithSameHost(true), no WithSameOrigin(true)). RedirectNotAllowed RedirectReason = iota + 1 // RedirectOtherOrigin means the target is neither an allowed host nor, under // WithSameHost or WithSameOrigin, the original request's host or origin. RedirectOtherOrigin // RedirectDowngrade means an https->http hop to an otherwise allowed target. RedirectDowngrade // RedirectTooMany means the hop would exceed the policy's hop cap. RedirectTooMany )
type RedirectRefusedError ¶ added in v5.1.0
type RedirectRefusedError struct {
// Origin is the refused target's origin, "scheme://host" plus ":port"
// when the port is not the scheme's default, escaped so [url.Parse]
// reads it back. It never holds the target's userinfo, path or query, so
// Error is safe to log for a target that carried a credential there.
Origin string
// Reason is the rule that refused the hop.
Reason RedirectReason
}
RedirectRefusedError is a shipped redirect policy's hard refusal of one hop: the hop cap, target permission and downgrade checks of a RedirectPolicyFunc policy, DefaultRedirectPolicy and DockerGitHubRedirectPolicy answer one, so a caller tells them apart with errors.As and Reason alone. RefuseAllRedirects and a WithPreserveMethod refusal answer http.ErrUseLastResponse instead. An http.Client returns it inside a *url.Error, and it is never transient.
func (*RedirectRefusedError) Error ¶ added in v5.1.0
func (e *RedirectRefusedError) Error() string
Error renders the refusal from Reason and Origin only.
type ResponseTooLargeError ¶
type ResponseTooLargeError struct {
Limit int64
}
ResponseTooLargeError is returned by GetBytes when the response body exceeds the configured maximum (WithMaxBodyBytes, default DefaultMaxBodyBytes). The body is not returned: a truncated payload indistinguishable from a complete one is a silent-corruption hazard, so GetBytes fails loud instead. Limit is the cap that was exceeded, mirroring the stdlib *http.MaxBytesError shape.
func (*ResponseTooLargeError) Error ¶
func (e *ResponseTooLargeError) Error() string
type RetryAfterHint ¶
RetryAfterHint is implemented by errors that carry an explicit wait duration for the next retry, typically a parsed and capped Retry-After. When fn's returned error is transient AND implements this interface with a positive duration, Do waits that duration before the next attempt instead of its jittered exponential backoff. The exponential base still advances, so a later transient error without a hint resumes the normal progression. The hint MUST already be capped by the implementer (e.g. via ParseRetryAfter); Do sleeps on it verbatim and applies no ceiling of its own, so an uncapped value is an unbounded-wait hazard.
It embeds error for the same reason Transient does: the interface is read through errors.As/errors.AsType over an error tree, so it can only ever name an error.
type RetryRoundTripper ¶
type RetryRoundTripper struct {
// contains filtered or unexported fields
}
RetryRoundTripper implements http.RoundTripper with automatic retry.
By default, only idempotent methods (GET, HEAD, OPTIONS, TRACE) are retried. Use TransportConfig.RetryNonIdempotent to also retry POST/PUT/PATCH/DELETE when the request has a GetBody function for body replay.
Inspired by hashicorp/go-retryablehttp, but operates directly on stdlib *http.Request without a custom request type, and counts TOTAL attempts (TransportConfig.MaxAttempts) rather than go-retryablehttp's retries-beyond-first.
func NewRetryRoundTripper ¶
func NewRetryRoundTripper(next http.RoundTripper, cfg TransportConfig) *RetryRoundTripper
NewRetryRoundTripper creates a RetryRoundTripper wrapping next with the given configuration. If next is nil, http.DefaultTransport is used. TransportConfig{} gives the defaults (see its field docs).
func (*RetryRoundTripper) RoundTrip ¶
RoundTrip implements http.RoundTripper. It retries eligible requests on transient failures with jittered exponential backoff, honoring Retry-After. Per the http.RoundTripper contract, the caller's request is never mutated.
When retries are exhausted the final response is returned, not an error: if the last attempt produced a retryable response (e.g. a 503), RoundTrip returns that *http.Response with a nil error, exactly as a non-retried request would. The caller owns the returned body (must Close it) and MUST inspect resp.StatusCode - a nil error does not imply a 2xx response.
type Secret ¶
type Secret string
Secret is a credential value being redacted: the needle the redaction helpers scan FOR, never the text they scan. It is a distinct type so the value-to-hide and the text-to-scan cannot be transposed at a call site — a reversed call turns the redactor into a leak, scanning the credential for occurrences of the log text and returning the credential untouched. With the positions typed apart, a plain string variable in the secret position is a compile error; the caller converts exactly the value that is the credential (httpx.Secret(token)) and nothing else fits there.
Convert at the call boundary, never store one: Secret carries no logging protection of its own — formatting a Secret with %v/%s or logging it via slog prints the credential verbatim. The zero value ("") disables redaction: every helper returns its input unredacted when the secret is empty, because there is nothing to scan for. Callers redacting a possibly-absent credential must treat empty as "no redaction happened", not as "redaction succeeded".
type StatusError ¶
StatusError represents a non-2xx response with URL context. Used by GetBytes. Supports errors.Is matching against ErrRateLimited and ErrServerError.
func (*StatusError) Error ¶
func (e *StatusError) Error() string
func (*StatusError) Is ¶
func (e *StatusError) Is(target error) bool
Is reports whether this StatusError matches ErrRateLimited or ErrServerError.
type Transient ¶
Transient is the interface an error implements to report whether it represents a transient (retryable) failure. It embeds error because every value this interface can ever name IS one: it is read through errors.As/errors.AsType over an error tree, whose nodes are errors by construction, so a non-error implementation is unreachable. Declaring the error half makes the type say that, keeps Error() available on a recovered value for diagnostics, and is what lets a caller write errors.AsType[Transient](err) — errors.AsType constrains its type parameter to error, where the older errors.As accepted any target. Same shape as net.Error.
type TransportConfig ¶
type TransportConfig struct {
// CheckRetry overrides the retry policy. nil means the default policy:
// transient transport errors plus 429/502/503/504 responses (deliberately
// narrower than GetBytes, which retries every 5xx; supply a custom policy
// to broaden the set when an upstream returns transient 500s).
CheckRetry CheckRetry
// OnRetry is a hook called before each retry attempt, the transport's only
// observability seam (it logs nothing itself).
OnRetry OnRetry
// PrepareRetry is called before each retry to mutate the cloned request
// (e.g. refresh an auth token).
PrepareRetry PrepareRetry
// MaxAttempts is the TOTAL attempt count including the initial request.
// Zero means unset and takes DefaultMaxAttempts (3); a NEGATIVE value
// means exactly one attempt (a zero struct field cannot distinguish absent
// from zero, so try-once is spelled as a negative).
MaxAttempts int
// BaseDelay is the initial backoff delay; non-positive means
// DefaultBaseDelay (1s). Waits are equal-jitter with overflow-safe
// doubling, and a retryable response's Retry-After header (capped at
// RetryAfterCap) overrides the computed wait.
BaseDelay time.Duration
// MaxElapsedTime is a hard ceiling on total time across retries,
// including any honored Retry-After: when now+wait would meet or pass it,
// the round-tripper aborts with an error instead of sleeping. It is
// checked BETWEEN attempts and cannot interrupt a stalled in-flight
// attempt; bound single attempts on the base transport (e.g.
// ResponseHeaderTimeout) and the whole call with the request context.
// Zero means no ceiling.
MaxElapsedTime time.Duration
// RetryNonIdempotent enables retry of non-idempotent methods (POST, PUT,
// PATCH, DELETE) when the request has a GetBody function for body replay.
RetryNonIdempotent bool
}
TransportConfig configures a RetryRoundTripper (and, through NewRetryClient, the retrying client). The zero value is ready to use: three total attempts, one-second base delay, the default retry policy, no hooks, no elapsed ceiling, no non-idempotent replay.
type Validators ¶
type Validators struct {
// ETag is replayed as If-None-Match.
ETag string
// LastModified is replayed as If-Modified-Since.
LastModified string
}
Validators carries the cache validators captured from a previous 200 response, replayed on the next conditional request so an unchanged resource is a cheap 304 instead of a re-download. Persist them alongside the cached body; the zero value sends no conditional headers (forcing a full 200). DoConditional validates both fields in both directions (see its validator hygiene contract), so values it captured are always safe to persist and replay verbatim.