Documentation
¶
Overview ¶
browser/network/compression.go
browser/network/dialer.go
internal/network/http_parser.go
browser/network/httpclient.go
browser/network/proxy.go
Index ¶
- Constants
- func DecompressResponse(resp *http.Response) error
- func DialContext(ctx context.Context, network, address string, config *DialerConfig) (net.Conn, error)
- func DialTCPContext(ctx context.Context, network, address string, config *DialerConfig) (net.Conn, error)
- func NewClient(config *ClientConfig) *http.Client
- func NewHTTPTransport(config *ClientConfig) *http.Transport
- type ClientConfig
- type CompressionMiddleware
- type DialerConfig
- type HTTPParser
- type InterceptionProxy
- type Logger
- type NopLogger
- type ProxyTransportConfig
- type RequestHandler
- type ResponseHandler
Constants ¶
const ( DefaultDialTimeout = 15 * time.Second DefaultKeepAliveInterval = 30 * time.Second DefaultTLSHandshakeTimeout = 10 * time.Second DefaultResponseHeaderTimeout = 30 * time.Second DefaultRequestTimeout = 120 * time.Second // Overall timeout for resource loading // Connection Pool Configuration for a browser. DefaultMaxIdleConns = 200 // Total connections across all hosts DefaultMaxIdleConnsPerHost = 10 // Common browser limit DefaultMaxConnsPerHost = 15 DefaultIdleConnTimeout = 90 * time.Second )
Constants optimized for browser behavior.
const SecureMinTLSVersion = tls.VersionTLS12
SecureMinTLSVersion defines the lowest TLS version considered secure by default.
Variables ¶
This section is empty.
Functions ¶
func DecompressResponse ¶
DecompressResponse inspects the `Content-Encoding` header of an http.Response and wraps its Body with the appropriate decompression reader(s). It is the core decompression logic used by the CompressionMiddleware.
This function handles multiple, layered encodings (e.g., gzip applied over deflate) by applying decoders in the reverse order. It supports gzip, brotli, and both zlib-wrapped and raw deflate streams. For performance, it uses pooled readers for gzip and brotli.
After successfully wrapping the body, it removes the `Content-Encoding` and `Content-Length` headers from the response and sets `resp.Uncompressed` to true.
NOTE: If this function returns an error (e.g., due to an invalid header or unsupported encoding), the `resp.Body` may have been partially read and should be considered corrupted. The caller is responsible for closing the body and discarding the response in such cases.
func DialContext ¶
func DialContext(ctx context.Context, network, address string, config *DialerConfig) (net.Conn, error)
DialContext provides a complete, high-level dialing function that establishes a connection and performs a TLS handshake if a TLSConfig is provided. It handles both direct and proxied connections transparently.
This function orchestrates the entire connection process:
- Establishes a raw TCP connection (or a proxy tunnel) via DialTCPContext.
- If `config.TLSConfig` is not nil, it performs a TLS handshake over the established connection to upgrade it to a secure `tls.Conn`.
It is suitable for creating connections for protocols that handle their own application layer logic, such as WebSockets.
Parameters:
- ctx: The context to control the dialing and TLS handshake process.
- network: The network type, typically "tcp".
- address: The target address in "host:port" format.
- config: The dialer configuration.
Returns the final, possibly TLS-wrapped, net.Conn or an error.
func DialTCPContext ¶
func DialTCPContext(ctx context.Context, network, address string, config *DialerConfig) (net.Conn, error)
DialTCPContext establishes a raw TCP connection to a given address, correctly handling proxy configuration. It is designed to be used as the `DialContext` function in an `http.Transport`.
If a proxy is specified in the DialerConfig, this function will first connect to the proxy and establish a TCP tunnel using the HTTP CONNECT method. Otherwise, it will establish a direct TCP connection to the target address.
Parameters:
- ctx: The context to control the dialing process.
- network: The network type, typically "tcp".
- address: The target address in "host:port" format.
- config: The dialer configuration.
Returns the established net.Conn or an error if the connection fails.
func NewClient ¶
func NewClient(config *ClientConfig) *http.Client
NewClient creates a new `http.Client` fully configured according to the provided ClientConfig. It builds a layered transport stack, starting with the base `http.Transport` from `NewHTTPTransport`, and wraps it with the `CompressionMiddleware` to provide transparent decompression.
The returned client is configured to not follow redirects automatically, allowing the caller to inspect and handle redirect responses manually, which is a common requirement in browser automation and security scanning.
func NewHTTPTransport ¶
func NewHTTPTransport(config *ClientConfig) *http.Transport
NewHTTPTransport creates a new `http.Transport` based on the provided ClientConfig. It is the foundational layer of the HTTP client, responsible for connection pooling, dialing, TLS handshakes, and proxying.
This function configures the transport with the custom dialer (`DialTCPContext`), a secure TLS configuration, and robust connection pool settings. It also explicitly disables the transport's built-in compression handling, as that functionality is managed by the `CompressionMiddleware` which wraps this transport.
Types ¶
type ClientConfig ¶
type ClientConfig struct {
// InsecureSkipVerify controls whether the client will skip TLS certificate
// verification. Setting this to true is insecure and should only be used in
// controlled testing environments.
InsecureSkipVerify bool
// TLSConfig provides a custom TLS configuration for the client. If nil, a
// secure default configuration will be generated.
TLSConfig *tls.Config
// RequestTimeout specifies the total time limit for a single HTTP request,
// including connection time, redirects, and reading the response body.
RequestTimeout time.Duration
// DialerConfig provides the low-level configuration for establishing TCP
// connections.
DialerConfig *DialerConfig
// MaxIdleConns is the maximum number of idle (keep-alive) connections across all hosts.
MaxIdleConns int
// MaxIdleConnsPerHost is the maximum number of idle connections to a single host.
MaxIdleConnsPerHost int
// MaxConnsPerHost is the maximum number of connections (idle + active) to a single host.
MaxConnsPerHost int
// IdleConnTimeout is the maximum amount of time an idle connection will remain
// in the pool before being closed.
IdleConnTimeout time.Duration
// DisableKeepAlives, if true, prevents the transport from reusing TCP
// connections after a request has completed.
DisableKeepAlives bool
// ProxyURL specifies the proxy server for the client to use.
ProxyURL *url.URL
// CookieJar is the cookie jar used to store and send cookies for HTTP requests.
// If nil, cookies will not be handled automatically.
CookieJar http.CookieJar
// Logger is the logger instance for the client to use.
Logger Logger
}
ClientConfig holds the high-level configuration for creating a customized HTTP client. It consolidates settings for security, timeouts, connection pooling, proxying, and state management (cookies).
func NewBrowserClientConfig ¶
func NewBrowserClientConfig() *ClientConfig
NewBrowserClientConfig creates a new ClientConfig with settings specifically optimized for emulating a modern web browser. This includes a large connection pool, aggressive keep-alives, a default in-memory cookie jar, and a secure low-level dialer configuration.
type CompressionMiddleware ¶
type CompressionMiddleware struct {
// Transport is the underlying http.RoundTripper to which requests will be
// sent after the Accept-Encoding header is added. If nil, http.DefaultTransport
// is used.
Transport http.RoundTripper
}
CompressionMiddleware is an http.RoundTripper that transparently handles HTTP response decompression. It automatically adds an `Accept-Encoding` header to outgoing requests to negotiate compression with the server and then decompresses the response body based on the `Content-Encoding` header received.
This middleware supports gzip, deflate (both zlib and raw), and brotli encoding, utilizing sync.Pool for readers to optimize performance and reduce garbage collection.
func NewCompressionMiddleware ¶
func NewCompressionMiddleware(transport http.RoundTripper) *CompressionMiddleware
NewCompressionMiddleware creates a new CompressionMiddleware that wraps the provided http.RoundTripper. If the provided transport is nil, it defaults to http.DefaultTransport.
type DialerConfig ¶
type DialerConfig struct {
// Timeout is the maximum amount of time a dial will wait for a connect to complete.
Timeout time.Duration
// KeepAlive specifies the interval between keep-alive probes for active connections.
KeepAlive time.Duration
// TLSConfig specifies the TLS configuration to use for secure connections.
// If nil, TLS will not be enabled for the connection.
TLSConfig *tls.Config
// NoDelay controls whether the TCP_NODELAY option is set on the connection,
// which is crucial for low-latency browser interactions.
NoDelay bool
// Resolver allows specifying a custom DNS resolver. If nil, the system's
// default resolver is used.
Resolver *net.Resolver
// ProxyURL specifies the URL of an HTTP/HTTPS proxy server. If set, connections
// will be established through the proxy using the CONNECT method.
ProxyURL *url.URL
}
DialerConfig holds the complete configuration for establishing network connections, including timeouts, keep-alive settings, TLS parameters, TCP options, custom DNS resolvers, and proxy settings.
func NewDialerConfig ¶
func NewDialerConfig() *DialerConfig
NewDialerConfig creates a new DialerConfig with secure and performant defaults suitable for a modern web browser. This includes a robust TLS 1.2+ configuration with a preference for modern cipher suites, a reasonable connection timeout, TCP KeepAlive enabled, and TCP_NODELAY enabled for responsiveness.
func (*DialerConfig) Clone ¶
func (c *DialerConfig) Clone() *DialerConfig
Clone creates a deep copy of the DialerConfig, ensuring that mutable fields like TLSConfig and ProxyURL are duplicated, preventing side effects when configurations are modified.
type HTTPParser ¶
type HTTPParser struct {
// contains filtered or unexported fields
}
HTTPParser provides functionality for parsing raw HTTP messages from a stream, with a specific focus on handling sequences of responses from HTTP pipelining.
func NewHTTPParser ¶
func NewHTTPParser(logger *zap.Logger) *HTTPParser
NewHTTPParser creates and returns a new HTTPParser.
func (*HTTPParser) ParsePipelinedResponses ¶
func (p *HTTPParser) ParsePipelinedResponses(conn io.Reader, expectedTotal int) ([]*http.Response, error)
ParsePipelinedResponses reads a sequence of HTTP responses from a single `io.Reader` (e.g., a TCP connection) and parses them. This is essential for handling HTTP/1.1 pipelining, where multiple requests are sent without waiting for each response.
The function iteratively reads and parses each response from the buffered reader. For each response, it:
- Initializes the appropriate decompression stream based on `Content-Encoding`.
- Reads the entire response body to advance the reader to the start of the next response.
- Replaces the consumed body with a new, readable `io.NopCloser` containing the fully read and decompressed body content.
This ensures that the caller receives a slice of complete, decompressed `http.Response` objects, ready for immediate use.
Parameters:
- conn: The `io.Reader` from which to read the response stream.
- expectedTotal: The number of responses to attempt to parse.
Returns a slice of parsed `http.Response` objects or an error if a non-EOF parsing or decompression error occurs.
type InterceptionProxy ¶
type InterceptionProxy struct {
// contains filtered or unexported fields
}
InterceptionProxy implements a full Man-in-the-Middle (MITM) proxy server that can intercept, inspect, and modify both HTTP and HTTPS traffic. It uses a dynamically-pluggable hook system to allow for custom processing of requests and responses.
When provided with a CA certificate and key, it can perform TLS interception for HTTPS traffic. If not, it will operate as a simple tunneling proxy for HTTPS requests.
func NewInterceptionProxy ¶
func NewInterceptionProxy(caCert, caKey []byte, transportConfig *ProxyTransportConfig, logger *zap.Logger) (*InterceptionProxy, error)
NewInterceptionProxy creates, configures, and returns a new InterceptionProxy. It initializes the underlying `goproxy` server, sets up the transport for upstream connections (including support for proxy chaining), and configures the MITM capabilities if a CA certificate and key are provided.
Parameters:
- caCert: The PEM-encoded CA certificate for signing intercepted TLS connections.
- caKey: The PEM-encoded private key for the CA certificate.
- transportConfig: Configuration for the proxy's upstream connections.
- logger: The logger for the proxy to use.
Returns the configured InterceptionProxy or an error if configuration fails.
func (*InterceptionProxy) AddRequestHook ¶
func (ip *InterceptionProxy) AddRequestHook(handler RequestHandler)
AddRequestHook registers a new RequestHandler to be executed on incoming requests. Handlers are executed in the order they are added. This method is thread-safe.
func (*InterceptionProxy) AddResponseHook ¶
func (ip *InterceptionProxy) AddResponseHook(handler ResponseHandler)
AddResponseHook registers a new ResponseHandler to be executed on incoming responses. Handlers are executed in the order they are added. This method is thread-safe.
type Logger ¶
type Logger interface {
// Warn logs a warning message.
Warn(msg string, args ...interface{})
// Info logs an informational message.
Info(msg string, args ...interface{})
// Debug logs a debug message.
Debug(msg string, args ...interface{})
// Error logs an error message.
Error(msg string, args ...interface{})
}
Logger defines a minimal logging interface that this package uses to report warnings, errors, and informational messages. This allows users to integrate the client with their own logging framework (e.g., zap, logrus).
type NopLogger ¶
type NopLogger struct{}
NopLogger is a no-op implementation of the Logger interface that discards all log messages. It is used as the default logger to prevent nil panics if no logger is provided.
type ProxyTransportConfig ¶
type ProxyTransportConfig struct {
// DialerConfig specifies the low-level dialing configuration for the proxy's
// outgoing connections.
DialerConfig *DialerConfig
}
ProxyTransportConfig defines the configuration for the proxy's own upstream connections. This allows for scenarios like proxy chaining, where the interception proxy forwards its traffic through another downstream proxy.
type RequestHandler ¶
RequestHandler is the function signature for a hook that can inspect and/or modify an HTTP request as it passes through the proxy. It can either return a modified request to be forwarded, or it can generate a response directly, effectively blocking the request from reaching its destination.
Source Files