README
¶
httputil
The package contains various helpers related to http protocol
Documentation
¶
Index ¶
- Constants
- Variables
- func AllHTTPMethods() []string
- func ChangePoolSize(x int64) error
- func DrainResponseBody(resp *http.Response)
- func DumpRequest(req *http.Request) (string, error)
- func DumpResponseHeadersAndRaw(resp *http.Response) (headers, fullresp []byte, err error)
- func DumpResponseIntoBuffer(resp *http.Response, body bool, buff *bytes.Buffer) (err error)
- func GetPoolSize() int64
- func SetBufferSize(size int64)
- func SetChunkSize(size int)
- func SetMaxLargeBuffers(max int)
- type ChainItem
- type CookieJar
- type Option
- type ResponseChain
- func (r *ResponseChain) Body() *bytes.Buffer
- func (r *ResponseChain) BodyBytes() []byte
- func (r *ResponseChain) BodyString() string
- func (r *ResponseChain) Close()
- func (r *ResponseChain) Fill() error
- func (r *ResponseChain) FullResponse() *bytes.Buffer
- func (r *ResponseChain) FullResponseBytes() []byte
- func (r *ResponseChain) FullResponseString() string
- func (r *ResponseChain) Has() bool
- func (r *ResponseChain) Headers() *bytes.Buffer
- func (r *ResponseChain) HeadersBytes() []byte
- func (r *ResponseChain) HeadersString() string
- func (r *ResponseChain) Previous() bool
- func (r *ResponseChain) Request() *http.Request
- func (r *ResponseChain) Response() *http.Response
Constants ¶
const ( // DefaultBufferSize is the default size of bytes buffer used for response // body storage. // // Use [SetBufferSize] to adjust the size. DefaultBufferSize = int64(10000) // DefaultMaxBodySize is the default maximum size of HTTP response body to // read. // // Responses larger than this will be truncated. DefaultMaxBodySize = 8 * 1024 * 1024 // 8 MB // DefaultMaxLargeBuffers is the maximum number of buffers at [maxBodyRead] // size that will be kept in the pool. // // This prevents pool pollution from accumulating many large buffers while // still allowing buffer reuse during burst workloads (e.g., nuclei scans // with compression bombs). Excess large buffers are discarded and handled // by GC. // // Default of 20 balances memory usage (~160MB max for large buffers) with // performance for typical concurrent workloads. // // Tuning: // - Increase for higher concurrency workloads (e.g., 50+ concurrent reqs) // - Decrease for memory-constrained environments (min. 10 recommended) // // Use [SetMaxLargeBuffers] to adjust the size. DefaultMaxLargeBuffers = 20 )
const ( // DefaultChunkSize defines the default chunk size for reading response // bodies. // // Use [SetChunkSize] to adjust the size. DefaultChunkSize = 32 * 1024 // 32KB )
Variables ¶
var ( // DefaultBytesBufferAlloc is the default size of bytes buffer used for // response body storage. // // Deprecated: Use [DefaultBufferSize] instead. DefaultBytesBufferAlloc = int64(10000) )
var ( // MaxBodyRead is the maximum size of HTTP response body to read. // // Responses larger than this will be truncated. // // Deprecated: Use [DefaultMaxBodySize] instead. MaxBodyRead = 4 * 1024 * 1024 // 4 MB )
Functions ¶
func AllHTTPMethods ¶
func AllHTTPMethods() []string
AllHTTPMethods contains all available HTTP methods
func ChangePoolSize ¶ added in v0.0.84
func DrainResponseBody ¶
DrainResponseBody drains the response body and closes it.
This reads and discards up to MaxBodyRead bytes to check for any remaining data, then closes the connection. This prevents connection reuse for responses that exceed the expected size (potential DoS).
func DumpResponseHeadersAndRaw ¶
DumpResponseHeadersAndRaw returns http headers and response as strings
func DumpResponseIntoBuffer ¶ added in v0.0.80
DumpResponseIntoBuffer dumps a http response without allocating a new buffer for the response body.
func GetPoolSize ¶ added in v0.0.84
func GetPoolSize() int64
func SetBufferSize ¶ added in v0.7.0
func SetBufferSize(size int64)
SetBufferSize sets the size of bytes buffer used for response body storage.
Changing the size will reset the buffer pool.
If size is less than 1000, it will be set to 1000.
func SetChunkSize ¶ added in v0.7.2
func SetChunkSize(size int)
SetChunkSize sets the chunk size for reading response bodies.
If size is less than or equal to zero, it resets to the default chunk size.
func SetMaxLargeBuffers ¶ added in v0.7.0
func SetMaxLargeBuffers(max int)
SetMaxLargeBuffers adjusts the maximum number of large buffers that can be pooled.
This should be called before making HTTP requests. Changing the limit will drain existing pooled buffers to ensure clean state.
If max is less than DefaultMaxLargeBuffers, it will be set to DefaultMaxLargeBuffers.
Types ¶
type ChainItem ¶
type ChainItem struct {
Request []byte
Response []byte
StatusCode int
Location string
RequestURL string
}
ChainItem request=>response Deprecated: use ResponseChain instead which is more efficient and lazy
type CookieJar ¶ added in v0.6.0
type CookieJar struct {
// contains filtered or unexported fields
}
CookieJar is a thread-safe wrapper around http.CookieJar
func NewCookieJar ¶ added in v0.6.0
New creates a new thread-safe cookie jar with the given options If no jar is provided, creates a simple in-memory cookie jar
type Option ¶ added in v0.6.0
type Option func(*CookieJar)
Option represents a configuration option for the cookie jar
func WithCookieJar ¶ added in v0.6.0
WithCookieJar sets an existing cookie jar to wrap
type ResponseChain ¶ added in v0.0.80
type ResponseChain struct {
// contains filtered or unexported fields
}
ResponseChain is a response chain for a http request on every call to previous it returns the previous response if it was redirected.
func NewResponseChain ¶ added in v0.0.80
func NewResponseChain(resp *http.Response, maxBody int64) *ResponseChain
NewResponseChain creates a new response chain for a http request with a maximum body size.
If maxBody is less than or equal to zero, it defaults to DefaultMaxBodySize.
func (*ResponseChain) Body ¶ added in v0.0.80
func (r *ResponseChain) Body() *bytes.Buffer
Body returns the current response body buffer in the chain.
Warning: The returned buffer is pooled and must not be modified or retained. Prefer BodyBytes() or BodyString() for safe read-only access.
func (*ResponseChain) BodyBytes ¶ added in v0.7.0
func (r *ResponseChain) BodyBytes() []byte
BodyBytes returns the current response body as byte slice in the chain.
The returned slice is valid only until Close() is called.
func (*ResponseChain) BodyString ¶ added in v0.7.0
func (r *ResponseChain) BodyString() string
BodyString returns the current response body as string in the chain.
The returned string is valid only until Close() is called. This is a zero-copy operation for performance.
func (*ResponseChain) Close ¶ added in v0.0.80
func (r *ResponseChain) Close()
Close the response chain and releases the buffers.
func (*ResponseChain) FullResponse ¶ added in v0.0.80
func (r *ResponseChain) FullResponse() *bytes.Buffer
FullResponse returns a new buffer containing headers+body.
Warning: The caller is responsible for managing the returned buffer's lifecycle. The buffer should be returned to the pool using putBuffer() or allowed to be garbage collected. Prefer FullResponseBytes() or FullResponseString() for safe read-only access.
func (*ResponseChain) FullResponseBytes ¶ added in v0.7.0
func (r *ResponseChain) FullResponseBytes() []byte
FullResponseBytes returns the current response (headers+body) as byte slice.
The returned slice is a copy and remains valid even after Close() is called.
func (*ResponseChain) FullResponseString ¶ added in v0.7.0
func (r *ResponseChain) FullResponseString() string
FullResponseString returns the current response as string in the chain.
The returned string is a copy and remains valid even after Close() is called. This is a zero-copy operation from the byte slice.
func (*ResponseChain) Has ¶ added in v0.0.80
func (r *ResponseChain) Has() bool
Has returns true if the response chain has a response
func (*ResponseChain) Headers ¶ added in v0.0.80
func (r *ResponseChain) Headers() *bytes.Buffer
Headers returns the current response headers buffer in the chain.
Warning: The returned buffer is pooled and must not be modified or retained. Prefer HeadersBytes() or HeadersString() for safe read-only access.
func (*ResponseChain) HeadersBytes ¶ added in v0.7.0
func (r *ResponseChain) HeadersBytes() []byte
HeadersBytes returns the current response headers as byte slice in the chain.
The returned slice is valid only until Close() is called.
func (*ResponseChain) HeadersString ¶ added in v0.7.0
func (r *ResponseChain) HeadersString() string
HeadersString returns the current response headers as string in the chain.
The returned string is valid only until Close() is called. This is a zero-copy operation for performance.
func (*ResponseChain) Previous ¶ added in v0.0.80
func (r *ResponseChain) Previous() bool
previous updates response pointer to previous response if it was redirected and returns true else false
func (*ResponseChain) Request ¶ added in v0.0.80
func (r *ResponseChain) Request() *http.Request
Request is request of current response
func (*ResponseChain) Response ¶ added in v0.0.80
func (r *ResponseChain) Response() *http.Response
Response is response of current response
Source Files