Documentation
¶
Overview ¶
h1client.go
Index ¶
- Constants
- Variables
- func SerializeRequest(req *http.Request) ([]byte, error)
- type ClientConfig
- type ConnectionPool
- type CredentialsProvider
- type CustomClient
- type H1Client
- func (c *H1Client) Close() error
- func (c *H1Client) Connect(ctx context.Context) error
- func (c *H1Client) Do(ctx context.Context, req *http.Request) (*http.Response, error)
- func (c *H1Client) IsIdle(timeout time.Duration) bool
- func (c *H1Client) ReadPipelinedResponses(ctx context.Context, expectedCount int) ([]*http.Response, error)
- func (c *H1Client) SendRaw(ctx context.Context, data []byte) error
- type H2Client
- type H2Settings
- type H2StreamResetError
- type RetryPolicy
Constants ¶
const ( // MaxResponseBodyBytes defines a reasonable limit for response bodies. MaxResponseBodyBytes = 32 * 1024 * 1024 // 32 MB // H2 specific constants (RFC 7540) DefaultH2InitialWindowSize = 65535 DefaultH2MaxFrameSize = 16384 // Define target receive window sizes for better throughput. // These values are advertised to the server during connection setup. TargetH2ConnWindowSize = 8 * 1024 * 1024 // 8 MB TargetH2StreamWindowSize = 4 * 1024 * 1024 // 4 MB )
const MaxReplayableBodyBytes = 2 * 1024 * 1024 // 2 MB
Define a reasonable limit for buffering request bodies in memory for replayability. This prevents excessive memory usage when handling large uploads that might need retries/redirects.
Variables ¶
var ErrCredentialsNotFound = errors.New("credentials not found")
ErrCredentialsNotFound is a sentinel error returned by a CredentialsProvider when it does not have credentials for a given host and realm. This signals to the client that authentication cannot be handled and the original 401/407 response should be returned.
Functions ¶
func SerializeRequest ¶
SerializeRequest converts an `http.Request` object into its raw HTTP/1.1 wire format as a byte slice. It correctly handles the request line, headers, and body, ensuring the `Content-Length` is set and the `Connection: keep-alive` header is present for persistent connections.
This utility is essential for manual request sending, such as in HTTP pipelining.
Types ¶
type ClientConfig ¶
type ClientConfig struct {
// DialerConfig holds the low-level configuration for establishing TCP and
// TLS connections, including proxy settings.
DialerConfig *network.DialerConfig
// CookieJar specifies the cookie jar for the client. If nil, cookies are not managed.
CookieJar http.CookieJar
// RequestTimeout sets the timeout for a single HTTP request attempt.
RequestTimeout time.Duration
// IdleConnTimeout is the maximum duration a connection can remain idle in the
// pool before it is closed and evicted.
IdleConnTimeout time.Duration
// InsecureSkipVerify controls whether to skip TLS certificate verification.
InsecureSkipVerify bool
// CheckRedirect provides a function to define a custom redirect policy. If nil,
// the client's default policy is used.
CheckRedirect func(req *http.Request, via []*http.Request) error
// RetryPolicy defines the rules for retrying failed requests.
RetryPolicy *RetryPolicy
// CredentialsProvider is an interface for dynamically supplying credentials for
// HTTP authentication.
CredentialsProvider CredentialsProvider
// H2Config holds settings specific to HTTP/2 connections.
H2Config H2Settings
}
ClientConfig is the primary configuration struct for a CustomClient. It aggregates all configurable aspects of the client, including dialing, cookies, timeouts, redirection, retries, authentication, and HTTP/2 settings.
func NewBrowserClientConfig ¶
func NewBrowserClientConfig() *ClientConfig
NewBrowserClientConfig creates a new ClientConfig with defaults that are optimized to emulate the behavior of a modern web browser. This includes a pre-configured cookie jar, sensible timeouts, a default retry policy, and standard HTTP/2 settings.
func (*ClientConfig) SetProxy ¶
func (c *ClientConfig) SetProxy(proxyURL *url.URL)
SetProxy is a convenience method to configure an HTTP/HTTPS proxy for the client. It sets the `ProxyURL` field in the underlying `DialerConfig`.
type ConnectionPool ¶
type ConnectionPool interface {
// Close immediately closes all connections in the pool.
Close() error
// IsIdle returns true if the pool has been idle for at least the specified duration.
IsIdle(timeout time.Duration) bool
}
ConnectionPool defines a minimal interface for a connection pool, used by the CustomClient's connection evictor to manage and close idle connections.
type CredentialsProvider ¶
type CredentialsProvider interface {
// GetCredentials is called when the client receives a 401 (Unauthorized) or
// 407 (Proxy Authentication Required) response.
//
// Parameters:
// - host: The host (e.g., "example.com:443") that issued the challenge.
// - realm: The authentication realm specified in the WWW-Authenticate header.
//
// It should return the username, password, and nil on success. If credentials
// are not available, it must return ErrCredentialsNotFound. Other errors
// will halt the request.
GetCredentials(host string, realm string) (username string, password string, err error)
}
CredentialsProvider defines an interface for dynamically supplying credentials in response to an HTTP authentication challenge (e.g., Basic Auth). This allows the client to support authentication without hardcoding usernames and passwords.
type CustomClient ¶
type CustomClient struct {
Config *ClientConfig
Logger *zap.Logger
// MaxRedirects specifies the maximum number of redirects to follow for a single request.
MaxRedirects int
// contains filtered or unexported fields
}
CustomClient is a sophisticated, low level HTTP client designed for fine grained control over HTTP/1.1 and HTTP/2 connections. It is the core of the browser's networking stack, managing persistent connections on a per-host basis and handling the full request lifecycle, including cookies, redirects, retries, and authentication.
It maintains separate pools of `H1Client` and `H2Client` instances, allowing it to transparently handle different protocol versions. A background goroutine periodically evicts idle connections to conserve resources.
func NewCustomClient ¶
func NewCustomClient(config *ClientConfig, logger *zap.Logger) *CustomClient
NewCustomClient creates and initializes a new CustomClient with the given configuration. It also starts a background goroutine to periodically close idle connections from its pools.
func (*CustomClient) CloseAll ¶
func (c *CustomClient) CloseAll()
CloseAll shuts down the client, closing all active and idle connections and stopping the background connection evictor goroutine. It waits for the evictor to terminate cleanly before returning.
func (*CustomClient) ConnectionCount ¶
func (c *CustomClient) ConnectionCount() int
ConnectionCount returns the total number of active connections (H1 + H2). This method is thread-safe and primarily intended for testing purposes.
func (*CustomClient) Do ¶
Do executes a single HTTP request and returns the response. This is the main entry point for the client. It orchestrates the entire request lifecycle, including:
1. Making the request body replayable for retries and redirects. 2. Adding cookies from the client's cookie jar. 3. Executing the request with a configurable retry policy for transient errors. 4. Storing cookies from the response. 5. Handling authentication challenges (e.g., HTTP Basic Auth). 6. Following HTTP redirects up to a configured limit.
Parameters:
- ctx: The context for the entire request operation, including all retries and redirects.
- req: The HTTP request to send.
Returns the final HTTP response or an error if the request could not be completed.
type H1Client ¶
type H1Client struct {
Conn net.Conn
Config *ClientConfig
TargetURL *url.URL
Address string
Logger *zap.Logger
// contains filtered or unexported fields
}
H1Client manages a single, persistent HTTP/1.1 connection to a specific host. It is designed to emulate the behavior of a browser's connection handling for HTTP/1.1, including support for Keep-Alive (reusing a single TCP/TLS connection for multiple sequential requests) and offering primitives for manual HTTP pipelining.
The client establishes its connection lazily (on the first request) and is thread-safe, ensuring that requests over the single connection are properly serialized.
func NewH1Client ¶
NewH1Client creates a new, un-connected H1Client for a given target URL and configuration. The actual network connection is established lazily when the first request is made.
func (*H1Client) Close ¶
Close terminates the client's network connection. This method is thread-safe.
func (*H1Client) Connect ¶
Connect establishes the underlying TCP and TLS connection to the target host if it is not already connected. It is called automatically by `Do` and other methods and is idempotent.
For HTTPS connections, it uses ALPN to explicitly negotiate HTTP/1.1, ensuring that servers that support both H2 and H1.1 will select the correct protocol for this client.
func (*H1Client) Do ¶
Do sends a single HTTP request over the persistent connection and waits for the response. It serializes access to the underlying connection, ensuring that multiple goroutines calling `Do` will have their requests sent sequentially, respecting HTTP/1.1 Keep-Alive semantics.
This method handles the entire request-response cycle, including serializing the request, writing it to the wire, parsing the response, and handling context cancellation during I/O operations.
func (*H1Client) IsIdle ¶
IsIdle determines if the connection has been idle for a duration longer than the specified timeout. This method is used by the `CustomClient`'s connection evictor to manage the connection pool and implements the `ConnectionPool` interface.
func (*H1Client) ReadPipelinedResponses ¶
func (c *H1Client) ReadPipelinedResponses(ctx context.Context, expectedCount int) ([]*http.Response, error)
ReadPipelinedResponses reads a specified number of HTTP responses from the connection's buffered reader. This method is the counterpart to `SendRaw` and is used to retrieve the results of a pipelined request sequence.
Parameters:
- ctx: The context for the read operation.
- expectedCount: The number of responses to parse from the stream.
Returns a slice of the parsed `http.Response` objects or an error if parsing fails or the connection is closed prematurely.
func (*H1Client) SendRaw ¶
SendRaw provides a low level mechanism to send arbitrary raw bytes over the connection. This is primarily intended for implementing manual HTTP/1.1 pipelining, where multiple serialized requests are sent without waiting for individual responses.
The caller is responsible for ensuring the data is a valid sequence of HTTP requests. Access to the connection is serialized. NOTE: SendRaw/ReadPipelinedResponses rely on the deadlines set based on the context/config and do not implement the active context cancellation monitoring used in Do().
type H2Client ¶
type H2Client struct {
Conn net.Conn
Config *ClientConfig
TargetURL *url.URL
Address string
Logger *zap.Logger
Framer *http2.Framer
HPEncoder *hpack.Encoder
HPDecoder *hpack.Decoder
// contains filtered or unexported fields
}
H2Client manages a single, persistent HTTP/2 connection to a specific host. It is a low level client that provides full control over the HTTP/2 framing layer, including concurrent request multiplexing, stream management, and manual flow control.
A background goroutine reads and processes incoming frames, dispatching them to the appropriate streams. Another optional goroutine handles keep-alive PINGs to maintain connection liveness. The client is thread-safe and designed for high concurrency scenarios.
func NewH2Client ¶
NewH2Client creates a new, un-connected H2Client for a given target URL and configuration. The actual network connection and H2 preface exchange happen lazily on the first request.
func (*H2Client) Close ¶
Close gracefully shuts down the H2 connection. It sends a GOAWAY frame with no error code, signals the background loops to terminate, closes the underlying network connection, and waits for the loops to exit. It implements the `ConnectionPool` interface.
func (*H2Client) Connect ¶
Connect establishes the full H2 connection if it is not already active. This is the primary setup method and is called lazily by `Do`. It is idempotent.
The connection process involves:
- Establishing a TCP connection and performing a TLS handshake with ALPN to negotiate "h2".
- Sending the client preface string.
- Sending initial SETTINGS and WINDOW_UPDATE frames to configure the connection.
- Starting background goroutines to read incoming frames (`readLoop`) and handle keep-alive PINGs (`pingLoop`).
func (*H2Client) Do ¶
Do executes a single HTTP request over the multiplexed H2 connection. It is the primary method for interacting with the client and is safe for concurrent use.
The process involves:
- Lazily establishing the H2 connection if needed.
- Initializing a new stream for the request.
- Starting a new goroutine to serialize and send the request headers and body, respecting H2 flow control.
- Waiting for the response to be received by the background `readLoop`, or for the request context to be cancelled.
This concurrent design allows multiple `Do` calls to be in flight simultaneously over the single TCP connection.
func (*H2Client) IsIdle ¶
IsIdle determines if the connection has been idle for a duration longer than the specified timeout. A connection is considered idle only if it is active and has no streams in progress. This method is used by the `CustomClient`'s connection evictor and implements the `ConnectionPool` interface.
type H2Settings ¶
type H2Settings struct {
// PingInterval specifies the time between sending HTTP/2 PING frames to the
// server to check for connection liveness.
PingInterval time.Duration
// PingTimeout is the maximum time to wait for a PING acknowledgment before
// considering the connection dead and closing it.
PingTimeout time.Duration
}
H2Settings defines configuration parameters specific to HTTP/2 connections.
func DefaultH2Settings ¶
func DefaultH2Settings() H2Settings
DefaultH2Settings returns a default H2Settings configuration with a 30-second ping interval and a 5-second ping timeout.
type H2StreamResetError ¶
H2StreamResetError is returned when an H2 stream is reset by the server. This allows higher-level clients (like CustomClient) to inspect the error code and determine if the request is retryable (e.g., REFUSED_STREAM).
func (H2StreamResetError) Error ¶
func (e H2StreamResetError) Error() string
type RetryPolicy ¶
type RetryPolicy struct {
// MaxRetries is the maximum number of retry attempts after the initial request fails.
MaxRetries int
// InitialBackoff is the base duration to wait before the first retry.
InitialBackoff time.Duration
// MaxBackoff is the upper limit for the backoff duration, preventing excessively long waits.
MaxBackoff time.Duration
// BackoffFactor is the multiplier for the exponential backoff calculation (e.g., 2.0).
BackoffFactor float64
// Jitter, if true, adds a random factor to the backoff duration to prevent
// multiple clients from retrying in synchronized waves (thundering herd problem).
Jitter bool
// RetryableStatusCodes is a set of HTTP status codes that should trigger a
// retry for idempotent requests (e.g., GET, PUT, DELETE).
RetryableStatusCodes map[int]bool
}
RetryPolicy encapsulates the rules for retrying failed or transient HTTP requests. It supports exponential backoff with jitter to prevent overwhelming a server that is temporarily unavailable.
func NewDefaultRetryPolicy ¶
func NewDefaultRetryPolicy() *RetryPolicy
NewDefaultRetryPolicy creates and returns a RetryPolicy with sensible defaults, such as 3 max retries, exponential backoff starting at 500ms, and retries for common transient server errors like 502, 503, and 504.
Source Files