Documentation
¶
Overview ¶
Package http provides the Requester — Vigolium's HTTP client used by scanner modules to send probes. It layers a middleware pipeline, per-host rate limiting and error tracking, retries, redirect control, and a response-chain abstraction that owns response-body lifecycle over the standard library transport. Requests can be bound to a context (WithContext / ExecuteContext) so in-flight calls abort on scan shutdown or phase deadline.
Index ¶
- Constants
- Variables
- func ClustererSizeForConcurrency(concurrency int) int
- func FinalRequestOf(chain *httpUtils.ResponseChain, resp *httpmsg.HttpResponse) *httpmsg.HttpRequestResponse
- func IsBlockedResponse(chain *httpUtils.ResponseChain) bool
- func RedirectChainHops(chain *httpUtils.ResponseChain) []*httpmsg.HttpRequestResponse
- func ResolveRedirectMode(options *types.Options) string
- func ValidRedirectMode(s string) bool
- func WasRedirected(chain *httpUtils.ResponseChain) bool
- type BlockNotice
- type CachedResponse
- type ClustererStats
- type ObservedResponse
- type Options
- type PoolStatsSnapshot
- type RequestClusterer
- type Requester
- func (r *Requester) CloneForScan() (*Requester, error)
- func (r *Requester) CloneWithoutCredentials() (*Requester, error)
- func (r *Requester) CloseIdleConnections()
- func (r *Requester) Clusterer() *RequestClusterer
- func (r *Requester) Execute(input *httpmsg.HttpRequestResponse, opts Options) (*httpUtils.ResponseChain, time.Duration, error)
- func (r *Requester) ExecuteContext(ctx context.Context, input *httpmsg.HttpRequestResponse, opts Options) (*httpUtils.ResponseChain, time.Duration, error)
- func (r *Requester) LogPoolStats()
- func (r *Requester) PoolStats() PoolStatsSnapshot
- func (r *Requester) RequestsSent() int64
- func (r *Requester) SetBlockNotifier(sink func(BlockNotice))
- func (r *Requester) SetCarriedSessions(sessions map[string]httpmsg.CarriedSession)
- func (r *Requester) SetResponseObserver(sink func(ObservedResponse))
- func (r *Requester) WithAbandonFlag(flag *atomic.Bool) *Requester
- func (r *Requester) WithContext(ctx context.Context) *Requester
Constants ¶
const ( MaxBodyRead = int64(30 * 1024 * 1024) // 30MB // TransportProfileSweep is the many-hosts-one-request-each connection // profile. See types.Options.TransportProfile for why the two profiles // cannot be collapsed into one. TransportProfileSweep = "sweep" )
const ( // RedirectModeOff records the 3xx itself and never follows it. RedirectModeOff = "off" // RedirectModeSameHost follows only when scheme-host-port is unchanged. RedirectModeSameHost = "same-host" // RedirectModeSameApex follows while the registrable domain (eTLD+1) is // unchanged. Cross-scheme and cross-port hops within that domain follow. RedirectModeSameApex = "same-apex" // RedirectModeAny follows every redirect up to the hop cap. This is the // historical default and stays the default, so no existing scan changes // behaviour by upgrading. RedirectModeAny = "any" )
Redirect modes accepted by Options.RedirectMode / --redirect-mode.
These replace the FollowRedirects/FollowHostRedirects bool pair, which had no spelling for the case a host sweep needs: follow inside the registrable domain. `www.example.com -> example.com` is the same application answering under its canonical name, while `example.com -> tracker.example.net` is a different party — and exact-host matching cannot tell those apart, so the only options were "follow everything" or "follow nothing".
Variables ¶
var ErrRequestAbandoned = errors.New("request abandoned: module call already timed out")
ErrRequestAbandoned is returned by Execute when the calling module has already been abandoned by the executor (its per-module timeout fired, or the phase was cancelled), so any result it produces will be discarded.
var RedirectModes = []string{ RedirectModeOff, RedirectModeSameHost, RedirectModeSameApex, RedirectModeAny, }
RedirectModes lists every accepted spelling, in the order help text should present them (most restrictive first). Error messages derive from this rather than hand-listing, so a mode added here reaches the CLI for free.
Functions ¶
func ClustererSizeForConcurrency ¶
ClustererSizeForConcurrency returns the LRU entry cap for a clusterer given the scan's concurrency. The set of TTL-fresh entries at any instant grows with in-flight request throughput, so the cache must be at least a few multiples of concurrency to avoid evicting fresh entries under wide fan-out — bounded by clusterCacheSizeMax to keep retained body memory in check.
func FinalRequestOf ¶
func FinalRequestOf(chain *httpUtils.ResponseChain, resp *httpmsg.HttpResponse) *httpmsg.HttpRequestResponse
FinalRequestOf returns the request that produced the chain's CURRENT response, paired with resp, or nil when the chain carries no request.
Redirect chains are why this exists. The executor pairs the ORIGINAL request with the FINAL response, so a followed chain produces a record whose URL says where the scan started and whose body came from wherever it ended up — the two halves of one row describing different servers. When hop recording is on, the caller re-pairs the final response with its own request through this, and the intermediate URLs survive as their own rows instead of being lost.
Must be called BEFORE RedirectChainHops, which rewinds the chain.
func IsBlockedResponse ¶
func IsBlockedResponse(chain *httpUtils.ResponseChain) bool
IsBlockedResponse reports whether a response chain was classified as a WAF/CDN block (captcha, bot-detection, or challenge page).
Exported for the one consumer that dispatches its own scan traffic and therefore cannot rely on the requester's internal per-response classification firing for it: known-issue-scan probes each target host through this requester before and during its nuclei run, and needs the verdict back rather than only its side effects. It runs the same classifier the hot path does, so a probe and a scan request can never disagree about what a block looks like.
func RedirectChainHops ¶
func RedirectChainHops(chain *httpUtils.ResponseChain) []*httpmsg.HttpRequestResponse
RedirectChainHops materializes every INTERMEDIATE hop of a followed redirect chain as its own request/response pair, oldest first. The final response is NOT included — its caller already owns that one.
Returns nil when the request was not redirected.
This consumes the chain ¶
ResponseChain.Previous() rewinds the chain IN PLACE (resp = resp.Request.Response) and Fill() resets the shared header/body buffers to re-load the hop it landed on. So by the time this returns, the chain no longer refers to the final response and its buffers hold the OLDEST hop. Callers must have copied everything they need out of the chain first, and must treat the chain as write-off-and-Close afterwards. Getting that order wrong silently substitutes a 301's headers for the 200's.
Intermediate hops carry no body ¶
Go's client has already drained the bodies of the hops it followed, and respChain.Fill skips the body read entirely once `reloaded` is set. Each hop therefore yields its status line and headers — crucially including Location — and an empty body. That is the whole content of a redirect response in practice (RFC 9110 does not require one and servers rarely send one), but it does mean a hop record has content_length 0 and no body-derived signals.
Hops carry no duration. The requester reports ONE elapsed time for the whole logical operation, which the caller attributes to the final response; there is no per-hop timing to recover, and dividing the total among hops would be inventing data. Zero here means unknown, per HttpResponse.Duration.
func ResolveRedirectMode ¶
ResolveRedirectMode collapses the redirect options into one mode.
An explicit RedirectMode always wins. Otherwise the legacy bools are mapped so nothing that set them changes behaviour: DisableRedirects means off, FollowHostRedirects means same-host, and anything else means any — which is what the old makeRedirectFunc(false, n) already did, since it only ever refused on the hop cap.
func ValidRedirectMode ¶
ValidRedirectMode reports whether s names a redirect mode.
func WasRedirected ¶
func WasRedirected(chain *httpUtils.ResponseChain) bool
WasRedirected reports whether the chain's current response was reached by following at least one redirect. Cheap: it inspects the linked request/ response pointers without touching buffers or rewinding anything.
Types ¶
type BlockNotice ¶
type BlockNotice struct {
Host string // host the block was observed on
WAFType string // detected WAF/CDN vendor (e.g. "cloudflare", "akamai", "generic")
Status int // HTTP status code of the blocking response
}
BlockNotice describes a WAF/CDN block observed on scan traffic. It is passed to the sink registered via SetBlockNotifier, once per host.
type CachedResponse ¶
type CachedResponse struct {
StatusCode int
Proto string
Header http.Header
Request *http.Request
Duration time.Duration
CachedAt time.Time
// contains filtered or unexported fields
}
CachedResponse holds a snapshot of response data that can be used to reconstruct independent ResponseChain instances. The body is a plain shared slice — its lifetime is managed by the LRU + GC (an entry's body is freed once the entry is evicted and every reconstructed chain referencing it is dropped).
func (*CachedResponse) Body ¶
func (c *CachedResponse) Body() []byte
Body returns the cached response body bytes (shared, do not modify).
func (*CachedResponse) ToResponseChain ¶
func (c *CachedResponse) ToResponseChain() *httpUtils.ResponseChain
ToResponseChain reconstructs an independent ResponseChain from cached data. The caller must call Close() on the returned chain when done. Uses reference-counted shared buffers to avoid copying body/header bytes.
type ClustererStats ¶
type ClustererStats struct {
Total int64
Clustered int64
CacheHits int64
RetainedBytes int64 // approximate response-body bytes currently cached
ByteEvictions int64 // entries evicted to stay under the byte budget
SizeRejects int64 // responses too large to post-cache
}
ClustererStats holds clusterer metrics.
type ObservedResponse ¶
type ObservedResponse struct {
Host string
URL string
ContentType string
Status int
RequestRaw []byte
Body []byte
}
ObservedResponse is the payload handed to a response observer for a server-error response. RequestRaw and Body are the observer's to read synchronously; a sink that retains them must copy (the underlying buffers are reused after Execute).
type Options ¶
type Options struct {
NoRedirects bool
RawRequest bool
IgnoreTimeoutTracking bool
NoClustering bool
DisableCompression bool // skip Accept-Encoding header so Go auto-decompresses
// RawRequestTarget, when non-empty, is written verbatim as the HTTP
// request-line target (request-URI) while the TCP/TLS connection still goes
// to the request's real host. It enables routing-based SSRF / "Cracking the
// lens" request-line attacks — e.g. connecting to a victim proxy but writing
// an absolute-form target "http://127.0.0.1:8080/", a userinfo trick
// "@collab.net/", or a protocol-relative "//collab.net/". Requires
// RawRequest=true (it is routed through the rawhttp client); ignored otherwise.
RawRequestTarget string
// contains filtered or unexported fields
}
Options per-request
type PoolStatsSnapshot ¶
type PoolStatsSnapshot struct {
Requests int64
ConnNew int64
ConnReused int64
ConnDialed int64
TLSHandshakes int64
}
PoolStatsSnapshot is a point-in-time copy of the connection-pool counters.
func (PoolStatsSnapshot) ReuseRatio ¶
func (s PoolStatsSnapshot) ReuseRatio() float64
ReuseRatio is the share of round trips served on an already-open connection: (roundTrips - dialed) / roundTrips. A low value signals connection churn — the class of regression that made anonymous-clone transport fragmentation invisible before this telemetry existed.
Derived from ConnDialed (the transport's real dials) rather than the httptrace new/reused split, which systematically misattributes new connections as reuses.
func (PoolStatsSnapshot) RoundTrips ¶
func (s PoolStatsSnapshot) RoundTrips() int64
RoundTrips is the number of HTTP transactions that obtained a connection.
This is the correct denominator for the churn metrics below, and it is NOT Requests: `requests` ticks once per logical doRequest call, while the client follows redirects and retries *inside* that call, each of which is its own round trip against its own connection. GotConn fires once per round trip, so while the new/reused split is misattributed (see poolStats.connDialed), their sum is an accurate transaction count.
func (PoolStatsSnapshot) RoundTripsPerConnection ¶
func (s PoolStatsSnapshot) RoundTripsPerConnection() float64
RoundTripsPerConnection is how many HTTP transactions each established connection carried. 1.0 means every transaction paid a fresh TCP (+TLS) setup; higher is better. Returns 0 when nothing was dialed through the traced path.
type RequestClusterer ¶
type RequestClusterer struct {
// contains filtered or unexported fields
}
RequestClusterer deduplicates concurrent identical HTTP requests using singleflight for in-flight dedup and an LRU cache with TTL for near-concurrent dedup.
func NewRequestClusterer ¶
func NewRequestClusterer() *RequestClusterer
NewRequestClusterer creates a new RequestClusterer with the default LRU size.
func NewRequestClustererWithSize ¶
func NewRequestClustererWithSize(size int) *RequestClusterer
NewRequestClustererWithSize creates a RequestClusterer with an explicit LRU entry cap. Sizes below the floor (clusterCacheSize) are raised to it. Callers typically derive size from scan concurrency via ClustererSizeForConcurrency.
func (*RequestClusterer) Execute ¶
func (rc *RequestClusterer) Execute( input *httpmsg.HttpRequestResponse, opts Options, doExecute func(*httpmsg.HttpRequestResponse, Options) (*httpUtils.ResponseChain, time.Duration, error), ) (*httpUtils.ResponseChain, time.Duration, error)
Execute checks the cache and singleflight group before delegating to the actual HTTP executor. Returns (ResponseChain, duration, error). Cache hits receive duration=0.
func (*RequestClusterer) LogStats ¶
func (rc *RequestClusterer) LogStats()
LogStats logs clusterer statistics at info level.
func (*RequestClusterer) Stats ¶
func (rc *RequestClusterer) Stats() ClustererStats
Stats returns current clusterer metrics.
type Requester ¶
type Requester struct {
// contains filtered or unexported fields
}
Requester executes HTTP requests with rate limiting and host error tracking
func NewRequester ¶
NewRequester creates a new Requester with all HTTP clients initialized
func (*Requester) CloneForScan ¶
CloneForScan returns a per-scan requester that SHARES the expensive transport (connection pool), dialer, and host rate limiter with r, but gives the scan its OWN behavioral state that concurrent scans would otherwise corrupt on a single shared requester: a fresh cookie jar, a fresh response observer (the 5xx corroboration sink — installed per-Execute, so on a shared requester the last writer wins and the first to finish clears it for the rest), and a fresh request clusterer (keyed on the request hash — sharing it would serve one scan's response to another scan's byte-identical request).
The server holds one shared requester and runs multiple lightweight scans (scan-url / scan-request) concurrently; this isolates their evidence and cookies. The edge-pacer and block-notifier stay shared: they dedup per host across the whole process, and giving each scan its own edge-pacer would register a new (never-removed) evict subscription on the shared limiter and leak subscriptions over the server's lifetime.
func (*Requester) CloneWithoutCredentials ¶
CloneWithoutCredentials returns a lightweight anonymous VIEW over the same scan-scoped requester: it shares the transport (connection pool), rate limiter, response observer, block notifier, edge-pacing dedup, and request clusterer, and isolates only the credential surface — a fresh cookie jar and credential-stripped headers. Authorization-differential modules must not reuse the primary requester's cookie jar/custom auth headers: deleting Authorization from one raw request is otherwise undone by doRequest, which reapplies r.customHeaders immediately before sending.
Sharing rather than rebuilding (the previous NewRequester clone) matters on a large corpus: each old clone minted its own http.Transport + four HTTP clients + jar with no idle-connection reclamation, so probe traffic fragmented the pool (extra TCP/TLS handshakes) and — because it also got a fresh response observer / block state — was invisible to the executor's scan-wide 5xx corroboration and edge pacing. The view fixes both: probes reuse the warm pool and stay observed. The clusterer is shared, but NOT because the raw-request hash is sufficient on its own — it is not; see Options.clusterScope. Isolation comes from the scope stamped below, so the shared clusterer only ever coalesces requests that really are equivalent.
func (*Requester) CloseIdleConnections ¶
func (r *Requester) CloseIdleConnections()
CloseIdleConnections returns this requester's idle keep-alive connections to the OS. Safe to call more than once and on a nil receiver.
Call it from the component that OWNS the requester, once its work has drained — never from a WithContext/credential view, which shares the same transport and would pull the sockets out from under the owner. It does not touch the process-global fastdialer (that is reference-counted by network.Init/Close) and does not close the raw clients, whose Close would close a fastdialer they may have only borrowed.
Without this, a finished scan left up to MaxIdleConns sockets pinned for IdleConnTimeout (90s). One scan recovers on its own; a server process running scans back to back, or a host sweep that touched thousands of origins, accumulates them.
func (*Requester) Clusterer ¶
func (r *Requester) Clusterer() *RequestClusterer
Clusterer returns the request clusterer (nil if clustering is disabled).
func (*Requester) Execute ¶
func (r *Requester) Execute(input *httpmsg.HttpRequestResponse, opts Options) (*httpUtils.ResponseChain, time.Duration, error)
Execute sends HTTP request with rate limiting, host error tracking, and optional request clustering to deduplicate concurrent identical requests. It uses the context bound via WithContext (if any) so callers that never touch ExecuteContext still honour scan/module cancellation; otherwise it is equivalent to the non-cancellable legacy behaviour.
func (*Requester) ExecuteContext ¶
func (r *Requester) ExecuteContext(ctx context.Context, input *httpmsg.HttpRequestResponse, opts Options) (*httpUtils.ResponseChain, time.Duration, error)
ExecuteContext is the cancellable variant of Execute: ctx is attached to the outgoing HTTP request, so cancelling it (scan shutdown or a per-module/active timeout) aborts the in-flight request and its retry loop instead of leaving the goroutine to drain on its own. A context.Background() ctx is equivalent to the legacy non-cancellable Execute.
func (*Requester) LogPoolStats ¶
func (r *Requester) LogPoolStats()
LogPoolStats logs connection-pool telemetry at info level, a companion to the request clusterer's LogStats at end of scan. A no-op when nothing was traced.
func (*Requester) PoolStats ¶
func (r *Requester) PoolStats() PoolStatsSnapshot
PoolStats returns a snapshot of the connection-pool counters, or a zero value when telemetry is unavailable.
func (*Requester) RequestsSent ¶
RequestsSent returns how many requests this requester (and every clone sharing its counter) has put on the wire. Safe on a nil Requester and from any goroutine; the counter is monotonic for the life of the scan.
func (*Requester) SetBlockNotifier ¶
func (r *Requester) SetBlockNotifier(sink func(BlockNotice))
SetBlockNotifier installs a sink invoked once per host the first time a response on that host is classified as a WAF/CDN block (captcha, bot-detection, or challenge page). It lets the scan warn the operator that traffic is being filtered and results against that host may be incomplete. The sink runs on the request goroutine, so it must be cheap and non-blocking. Call once during scan setup, before scanning concurrency starts; a nil sink is a no-op. The installed sink is shared by every WithContext clone of this requester.
func (*Requester) SetCarriedSessions ¶
func (r *Requester) SetCarriedSessions(sessions map[string]httpmsg.CarriedSession)
SetCarriedSessions installs browser-harvested per-host sessions on the requester. Keys are normalized to bare lowercase hostnames; a request is matched against them by host so a session only ever reaches the host it was harvested from. Cookies are merged into (never over) a request's existing Cookie header, and a non-empty UserAgent is pinned — both applied before the operator's -H custom headers so an explicit -H Cookie/User-Agent still wins. Safe to call once during scan setup; a nil/empty map is a no-op.
func (*Requester) SetResponseObserver ¶
func (r *Requester) SetResponseObserver(sink func(ObservedResponse))
SetResponseObserver installs a sink invoked for every server-error (5xx) response on this requester's traffic, so the executor can corroborate a leaked database error from ANY module's probe rather than only the module that sent it. The sink runs synchronously on the request goroutine — it must be cheap and non-blocking, and must copy any request/response bytes it retains. Call during scan setup; a nil sink clears the observer. The installed sink is shared by every WithContext clone of this requester.
func (*Requester) WithAbandonFlag ¶
WithAbandonFlag returns a shallow clone whose Execute refuses to START new requests once flag is set. Requests already in flight are untouched.
This exists because a per-module timeout does not stop the module. The executor's watchdog returns (nil, false) and the caller discards the result, but the module's goroutine keeps running — and kept issuing fresh requests against the phase-bound requester. On a real scan that was 725+ timed-out modules still hammering the target, competing for the same per-host concurrency as live work, to produce findings that are thrown away by definition.
The signal is separate from WithContext, and only gates request INITIATION, for one specific reason: the request clusterer dedups concurrent identical requests through a singleflight group, so one in-flight request is shared by every module that asked for it. Attaching a per-module deadline to the outgoing request would let the first caller's timeout cancel a request other modules are legitimately waiting on. Gating initiation instead stops the abandoned module's future load without severing anyone's shared socket.
A flag rather than a context: this is only ever read as a boolean — never selected on, never attached to a request — and it is set once per module dispatch, the hottest fan-out loop in the scan. A context.WithCancel child there would allocate a context plus a CancelFunc and take the shared phase context's children-map lock twice per call, serializing hundreds of concurrent dispatches on one mutex to carry a single bit.
func (*Requester) WithContext ¶
WithContext returns a shallow copy of the Requester whose context-less Execute uses ctx for cancellation. The copy shares the underlying HTTP clients, rate limiter, clusterer, and headers — only the default context differs — so it is cheap to create per scan task. The executor hands each active-module task a context-bound requester so a per-module timeout or scan shutdown aborts the module's in-flight requests even when the module calls Execute directly. A nil ctx returns the receiver unchanged.
Source Files