http

package
v0.5.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 24, 2026 License: MIT Imports: 34 Imported by: 0

Documentation

Overview

Package http provides the Requester — Vigolium's HTTP client used by scanner modules to send probes. It layers a middleware pipeline, per-host rate limiting and error tracking, retries, redirect control, and a response-chain abstraction that owns response-body lifecycle over the standard library transport. Requests can be bound to a context (WithContext / ExecuteContext) so in-flight calls abort on scan shutdown or phase deadline.

Index

Constants

View Source
const (
	MaxBodyRead = int64(30 * 1024 * 1024) // 30MB

	// TransportProfileSweep is the many-hosts-one-request-each connection
	// profile. See types.Options.TransportProfile for why the two profiles
	// cannot be collapsed into one.
	TransportProfileSweep = "sweep"
)
View Source
const (
	// RedirectModeOff records the 3xx itself and never follows it.
	RedirectModeOff = "off"
	// RedirectModeSameHost follows only when scheme-host-port is unchanged.
	RedirectModeSameHost = "same-host"
	// RedirectModeSameApex follows while the registrable domain (eTLD+1) is
	// unchanged. Cross-scheme and cross-port hops within that domain follow.
	RedirectModeSameApex = "same-apex"
	// RedirectModeAny follows every redirect up to the hop cap. This is the
	// historical default and stays the default, so no existing scan changes
	// behaviour by upgrading.
	RedirectModeAny = "any"
)

Redirect modes accepted by Options.RedirectMode / --redirect-mode.

These replace the FollowRedirects/FollowHostRedirects bool pair, which had no spelling for the case a host sweep needs: follow inside the registrable domain. `www.example.com -> example.com` is the same application answering under its canonical name, while `example.com -> tracker.example.net` is a different party — and exact-host matching cannot tell those apart, so the only options were "follow everything" or "follow nothing".

Variables

View Source
var ErrRequestAbandoned = errors.New("request abandoned: module call already timed out")

ErrRequestAbandoned is returned by Execute when the calling module has already been abandoned by the executor (its per-module timeout fired, or the phase was cancelled), so any result it produces will be discarded.

RedirectModes lists every accepted spelling, in the order help text should present them (most restrictive first). Error messages derive from this rather than hand-listing, so a mode added here reaches the CLI for free.

Functions

func ChainRelocated added in v0.4.9

func ChainRelocated(chain *httpUtils.ResponseChain, requested *url.URL) (relocated, linked bool)

ChainRelocated reports whether the exchange ended somewhere other than requested, and whether the chain still carries its redirect linkage.

Two answers because the callers need different ones. Re-pairing the response with the request that produced it only needs "did we end up elsewhere"; materializing the intermediate hops needs the linked responses themselves.

The two can disagree, which is the whole reason this exists. The request clusterer caches a flattened snapshot and reconstructs a chain with no resp.Request.Response — deliberately, since retaining that pointer would pin every prior response and body for the cache entry's lifetime. The snapshot does keep the FINAL hop's request, so the destination URL survives even though the linkage does not.

Compares URL fields rather than strings: this runs on every baseline fetch, and url.URL.String() allocates on both sides for an answer that is almost always "no".

func ClustererSizeForConcurrency

func ClustererSizeForConcurrency(concurrency int) int

ClustererSizeForConcurrency returns the LRU entry cap for a clusterer given the scan's concurrency. The set of TTL-fresh entries at any instant grows with in-flight request throughput, so the cache must be at least a few multiples of concurrency to avoid evicting fresh entries under wide fan-out — bounded by clusterCacheSizeMax to keep retained body memory in check.

func FinalRequestOf

FinalRequestOf returns the request that produced the chain's CURRENT response, paired with resp, or nil when the chain carries no request.

Redirect chains are why this exists. The executor pairs the ORIGINAL request with the FINAL response, so a followed chain produces a record whose URL says where the scan started and whose body came from wherever it ended up — the two halves of one row describing different servers. When hop recording is on, the caller re-pairs the final response with its own request through this, and the intermediate URLs survive as their own rows instead of being lost.

Must be called BEFORE RedirectChainHops, which rewinds the chain.

func IsBlockedResponse

func IsBlockedResponse(chain *httpUtils.ResponseChain) bool

IsBlockedResponse reports whether a response chain was classified as a WAF/CDN block (captcha, bot-detection, or challenge page).

Exported for the one consumer that dispatches its own scan traffic and therefore cannot rely on the requester's internal per-response classification firing for it: known-issue-scan probes each target host through this requester before and during its nuclei run, and needs the verdict back rather than only its side effects. It runs the same classifier the hot path does, so a probe and a scan request can never disagree about what a block looks like.

func RedirectChainHops

func RedirectChainHops(chain *httpUtils.ResponseChain) []*httpmsg.HttpRequestResponse

RedirectChainHops materializes every INTERMEDIATE hop of a followed redirect chain as its own request/response pair, oldest first. The final response is NOT included — its caller already owns that one.

Returns nil when the request was not redirected.

This consumes the chain

ResponseChain.Previous() rewinds the chain IN PLACE (resp = resp.Request.Response) and Fill() resets the shared header/body buffers to re-load the hop it landed on. So by the time this returns, the chain no longer refers to the final response and its buffers hold the OLDEST hop. Callers must have copied everything they need out of the chain first, and must treat the chain as write-off-and-Close afterwards. Getting that order wrong silently substitutes a 301's headers for the 200's.

Intermediate hops carry no body

Go's client has already drained the bodies of the hops it followed, and respChain.Fill skips the body read entirely once `reloaded` is set. Each hop therefore yields its status line and headers — crucially including Location — and an empty body. That is the whole content of a redirect response in practice (RFC 9110 does not require one and servers rarely send one), but it does mean a hop record has content_length 0 and no body-derived signals.

Hops carry no duration. The requester reports ONE elapsed time for the whole logical operation, which the caller attributes to the final response; there is no per-hop timing to recover, and dividing the total among hops would be inventing data. Zero here means unknown, per HttpResponse.Duration.

func ResolveRedirectMode

func ResolveRedirectMode(options *types.Options) string

ResolveRedirectMode collapses the redirect options into one mode.

An explicit RedirectMode always wins. Otherwise the legacy bools are mapped so nothing that set them changes behaviour: DisableRedirects means off, FollowHostRedirects means same-host, and anything else means any — which is what the old makeRedirectFunc(false, n) already did, since it only ever refused on the hop cap.

func StoppedAtAuthWall added in v0.4.9

func StoppedAtAuthWall(requestURL string, statusCode int, location string) (string, bool)

StoppedAtAuthWall reports whether resp is a redirect the policy refused to follow because its destination is a login / SSO wall, and returns that destination.

Callers use this to attribute the stop rather than to make it: the decision already happened inside CheckRedirect, which can only signal through ErrUseLastResponse and therefore cannot report a reason.

Re-deriving beats recording it inside the policy, which would be the obvious alternative, because the request clusterer replays a cached response WITHOUT running CheckRedirect at all — anything the policy recorded would be silently absent on every cache hit. Reading it back off the stored 3xx is uniform across live and replayed responses, and costs one URL parse on a path already gated to redirects.

Sharing stopsAtAuthWall with the policy is what keeps the two from drifting into disagreeing about what a wall is.

func ValidRedirectMode

func ValidRedirectMode(s string) bool

ValidRedirectMode reports whether s names a redirect mode.

func WasRedirected

func WasRedirected(chain *httpUtils.ResponseChain) bool

WasRedirected reports whether the chain's current response was reached by following at least one redirect. Cheap: it inspects the linked request/ response pointers without touching buffers or rewinding anything.

Types

type BlockNotice

type BlockNotice struct {
	Host    string // host the block was observed on
	WAFType string // detected WAF/CDN vendor (e.g. "cloudflare", "akamai", "generic")
	Status  int    // HTTP status code of the blocking response
}

BlockNotice describes a WAF/CDN block observed on scan traffic. It is passed to the sink registered via SetBlockNotifier, once per host.

type CachedResponse

type CachedResponse struct {
	StatusCode int
	Proto      string
	Header     http.Header
	// Close mirrors http.Response.Close: the server signalled it is closing the
	// connection rather than keeping it alive. Go strips the hop-by-hop
	// Connection header from Header before a module ever sees it, so this flag is
	// the only surviving record of keep-alive state - without it every cache hit
	// reports a reusable connection, which silently defeats any module gating on
	// connection reuse (request smuggling, RQP cache poisoning).
	Close bool

	Request  *http.Request
	Duration time.Duration
	CachedAt time.Time
	// contains filtered or unexported fields
}

CachedResponse holds a snapshot of response data that can be used to reconstruct independent ResponseChain instances. The body is a plain shared slice — its lifetime is managed by the LRU + GC (an entry's body is freed once the entry is evicted and every reconstructed chain referencing it is dropped).

func (*CachedResponse) Body

func (c *CachedResponse) Body() []byte

Body returns the cached response body bytes (shared, do not modify).

func (*CachedResponse) ToResponseChain

func (c *CachedResponse) ToResponseChain() *httpUtils.ResponseChain

ToResponseChain reconstructs an independent ResponseChain from cached data. The caller must call Close() on the returned chain when done. Uses reference-counted shared buffers to avoid copying body/header bytes.

type ChainRow added in v0.4.9

type ChainRow struct {
	RR *httpmsg.HttpRequestResponse
	// Truncated marks a row whose destination is NOT the row after it: either
	// hops between them were dropped, or the chain stopped here. It is the one
	// bit that distinguishes "this 3xx pointed somewhere that never answered"
	// from "we stopped walking", which otherwise produce an identical row.
	Truncated bool
}

ChainRow is one redirect hop selected for storage.

func ShapeRedirectChain added in v0.4.9

func ShapeRedirectChain(hops []*httpmsg.HttpRequestResponse, finalTarget string) []ChainRow

ShapeRedirectChain selects which of a chain's intermediate hops to store.

Two reductions, in this order, and the order matters: collapsing first means the cap spends its budget on hops that say something.

Canonical hops are dropped

A hop whose destination is the same resource under its canonical spelling — a scheme upgrade, a trailing slash, www. gained or dropped — carries no information the destination row does not already carry. `curl -L ctdtoolkit.netflix.net` is the everyday shape: one 301 whose only content is "say it with https". Storing it produced a contentless row with no title, no technology and surface_score 0, sitting in the output next to rows that mean something, and it is why the row for a submitted target so often looked dead.

Including the first hop. Keeping it was the obvious hedge — it is the only row carrying the submitted target's own status — but it is the wrong trade now that every row names its Target: the destination row already says which line produced it, so the 301 adds a contentless row and nothing else. A target whose only redirect is canonical therefore yields exactly one record, at the canonical URL, which is what an operator running `curl -L` sees.

Then the cap keeps the ends

If more rows remain than the budget allows, the FIRST and the LAST survive and the middle goes: the first is the fact the caller asked for, the last is the fact they wanted. A cap that kept the first four and discarded the terminal response would be strictly worse than no cap at all.

finalTarget is the URL of the response the caller pairs with these hops; it is what the last intermediate is classified against.

type ClustererStats

type ClustererStats struct {
	Total         int64
	Clustered     int64
	CacheHits     int64
	RetainedBytes int64 // approximate response-body bytes currently cached
	ByteEvictions int64 // entries evicted to stay under the byte budget
	SizeRejects   int64 // responses too large to post-cache
}

ClustererStats holds clusterer metrics.

type ObservedResponse

type ObservedResponse struct {
	Host        string
	URL         string
	ContentType string
	Status      int
	RequestRaw  []byte
	Body        []byte
}

ObservedResponse is the payload handed to a response observer for a server-error response. RequestRaw and Body are the observer's to read synchronously; a sink that retains them must copy (the underlying buffers are reused after Execute).

type Options

type Options struct {
	NoRedirects           bool
	RawRequest            bool
	IgnoreTimeoutTracking bool
	NoClustering          bool
	DisableCompression    bool // skip Accept-Encoding header so Go auto-decompresses
	// RawRequestTarget, when non-empty, is written verbatim as the HTTP
	// request-line target (request-URI) while the TCP/TLS connection still goes
	// to the request's real host. It enables routing-based SSRF / "Cracking the
	// lens" request-line attacks — e.g. connecting to a victim proxy but writing
	// an absolute-form target "http://127.0.0.1:8080/", a userinfo trick
	// "@collab.net/", or a protocol-relative "//collab.net/". Implies
	// RawRequest: it is routed through the rawhttp client. RawBytes, which
	// carries its own request line, takes precedence if both are set.
	RawRequestTarget string
	// RawBytes, when non-empty, is written to the connection verbatim as the
	// ENTIRE request — request line, headers and body — with no framing
	// normalization whatsoever. It exists for attacks whose payload IS the
	// framing, where any rewrite destroys the test: request smuggling (CL.TE /
	// TE.CL / TE.TE) needs Content-Length and Transfer-Encoding to DISAGREE, and
	// needs duplicate or oddly-cased Transfer-Encoding headers to reach the wire.
	//
	// Neither of those can travel any other path. net/http computes framing from
	// req.ContentLength and req.TransferEncoding and ignores the corresponding
	// headers: a manually set "Transfer-Encoding: chunked" is dropped outright and
	// "Content-Length: 4" is overwritten with the real body length. Even rawhttp's
	// normal path re-derives Content-Length (AutomaticContentLength) and carries
	// headers in an http.Header map, which canonicalizes key case and so cannot
	// express "Transfer-Encoding" and "Transfer-encoding" as two distinct headers.
	// Only these verbatim bytes preserve all of it.
	//
	// Implies RawRequest (rawhttp is the only client that can send them) and
	// disables clustering: the cache keys on the parsed request, which by
	// construction no longer describes what goes out.
	//
	// The caller owns correctness — nothing validates or repairs these bytes.
	// They must be a complete HTTP/1.1 message with CRLF line endings.
	RawBytes []byte
	// contains filtered or unexported fields
}

Options per-request

type PoolStatsSnapshot

type PoolStatsSnapshot struct {
	Requests      int64
	ConnNew       int64
	ConnReused    int64
	ConnDialed    int64
	TLSHandshakes int64
}

PoolStatsSnapshot is a point-in-time copy of the connection-pool counters.

func (PoolStatsSnapshot) ReuseRatio

func (s PoolStatsSnapshot) ReuseRatio() float64

ReuseRatio is the share of round trips served on an already-open connection: (roundTrips - dialed) / roundTrips. A low value signals connection churn — the class of regression that made anonymous-clone transport fragmentation invisible before this telemetry existed.

Derived from ConnDialed (the transport's real dials) rather than the httptrace new/reused split, which systematically misattributes new connections as reuses.

func (PoolStatsSnapshot) RoundTrips

func (s PoolStatsSnapshot) RoundTrips() int64

RoundTrips is the number of HTTP transactions that obtained a connection.

This is the correct denominator for the churn metrics below, and it is NOT Requests: `requests` ticks once per logical doRequest call, while the client follows redirects and retries *inside* that call, each of which is its own round trip against its own connection. GotConn fires once per round trip, so while the new/reused split is misattributed (see poolStats.connDialed), their sum is an accurate transaction count.

func (PoolStatsSnapshot) RoundTripsPerConnection

func (s PoolStatsSnapshot) RoundTripsPerConnection() float64

RoundTripsPerConnection is how many HTTP transactions each established connection carried. 1.0 means every transaction paid a fresh TCP (+TLS) setup; higher is better. Returns 0 when nothing was dialed through the traced path.

type RequestClusterer

type RequestClusterer struct {
	// contains filtered or unexported fields
}

RequestClusterer deduplicates concurrent identical HTTP requests using singleflight for in-flight dedup and an LRU cache with TTL for near-concurrent dedup.

func NewRequestClusterer

func NewRequestClusterer() *RequestClusterer

NewRequestClusterer creates a new RequestClusterer with the default LRU size.

func NewRequestClustererWithSize

func NewRequestClustererWithSize(size int) *RequestClusterer

NewRequestClustererWithSize creates a RequestClusterer with an explicit LRU entry cap. Sizes below the floor (clusterCacheSize) are raised to it. Callers typically derive size from scan concurrency via ClustererSizeForConcurrency.

func (*RequestClusterer) Execute

Execute checks the cache and singleflight group before delegating to the actual HTTP executor. Returns (ResponseChain, duration, error). Cache hits receive duration=0.

func (*RequestClusterer) LogStats

func (rc *RequestClusterer) LogStats()

LogStats logs clusterer statistics at info level.

func (*RequestClusterer) Stats

func (rc *RequestClusterer) Stats() ClustererStats

Stats returns current clusterer metrics.

type Requester

type Requester struct {
	// contains filtered or unexported fields
}

Requester executes HTTP requests with rate limiting and host error tracking

func NewRequester

func NewRequester(options *types.Options, services *services.Services) (*Requester, error)

NewRequester creates a new Requester with all HTTP clients initialized

func (*Requester) CloneForScan

func (r *Requester) CloneForScan() (*Requester, error)

CloneForScan returns a per-scan requester that SHARES the expensive transport (connection pool), dialer, and host rate limiter with r, but gives the scan its OWN behavioral state that concurrent scans would otherwise corrupt on a single shared requester: a fresh cookie jar, a fresh response observer (the 5xx corroboration sink — installed per-Execute, so on a shared requester the last writer wins and the first to finish clears it for the rest), and a fresh request clusterer (keyed on the request hash — sharing it would serve one scan's response to another scan's byte-identical request).

The server holds one shared requester and runs multiple lightweight scans (scan-url / scan-request) concurrently; this isolates their evidence and cookies. The edge-pacer and block-notifier stay shared: they dedup per host across the whole process, and giving each scan its own edge-pacer would register a new (never-removed) evict subscription on the shared limiter and leak subscriptions over the server's lifetime.

func (*Requester) CloneWithoutCredentials

func (r *Requester) CloneWithoutCredentials() (*Requester, error)

CloneWithoutCredentials returns a lightweight anonymous VIEW over the same scan-scoped requester: it shares the transport (connection pool), rate limiter, response observer, block notifier, edge-pacing dedup, and request clusterer, and isolates only the credential surface — a fresh cookie jar and credential-stripped headers. Authorization-differential modules must not reuse the primary requester's cookie jar/custom auth headers: deleting Authorization from one raw request is otherwise undone by doRequest, which reapplies r.customHeaders immediately before sending.

Sharing rather than rebuilding (the previous NewRequester clone) matters on a large corpus: each old clone minted its own http.Transport + four HTTP clients + jar with no idle-connection reclamation, so probe traffic fragmented the pool (extra TCP/TLS handshakes) and — because it also got a fresh response observer / block state — was invisible to the executor's scan-wide 5xx corroboration and edge pacing. The view fixes both: probes reuse the warm pool and stay observed. The clusterer is shared, but NOT because the raw-request hash is sufficient on its own — it is not; see Options.clusterScope. Isolation comes from the scope stamped below, so the shared clusterer only ever coalesces requests that really are equivalent.

func (*Requester) CloseIdleConnections

func (r *Requester) CloseIdleConnections()

CloseIdleConnections returns this requester's idle keep-alive connections to the OS. Safe to call more than once and on a nil receiver.

Call it from the component that OWNS the requester, once its work has drained — never from a WithContext/credential view, which shares the same transport and would pull the sockets out from under the owner. It does not touch the process-global fastdialer (that is reference-counted by network.Init/Close) and does not close the raw clients, whose Close would close a fastdialer they may have only borrowed.

Without this, a finished scan left up to MaxIdleConns sockets pinned for IdleConnTimeout (90s). One scan recovers on its own; a server process running scans back to back, or a host sweep that touched thousands of origins, accumulates them.

func (*Requester) Clusterer

func (r *Requester) Clusterer() *RequestClusterer

Clusterer returns the request clusterer (nil if clustering is disabled).

func (*Requester) Execute

Execute sends HTTP request with rate limiting, host error tracking, and optional request clustering to deduplicate concurrent identical requests. It uses the context bound via WithContext (if any) so callers that never touch ExecuteContext still honour scan/module cancellation; otherwise it is equivalent to the non-cancellable legacy behaviour.

func (*Requester) ExecuteContext

ExecuteContext is the cancellable variant of Execute: ctx is attached to the outgoing HTTP request, so cancelling it (scan shutdown or a per-module/active timeout) aborts the in-flight request and its retry loop instead of leaving the goroutine to drain on its own. A context.Background() ctx is equivalent to the legacy non-cancellable Execute.

func (*Requester) LogPoolStats

func (r *Requester) LogPoolStats()

LogPoolStats logs connection-pool telemetry at info level, a companion to the request clusterer's LogStats at end of scan. A no-op when nothing was traced.

func (*Requester) PoolStats

func (r *Requester) PoolStats() PoolStatsSnapshot

PoolStats returns a snapshot of the connection-pool counters, or a zero value when telemetry is unavailable.

func (*Requester) RequestsSent

func (r *Requester) RequestsSent() int64

RequestsSent returns how many requests this requester (and every clone sharing its counter) has put on the wire. Safe on a nil Requester and from any goroutine; the counter is monotonic for the life of the scan.

func (*Requester) SetBlockNotifier

func (r *Requester) SetBlockNotifier(sink func(BlockNotice))

SetBlockNotifier installs a sink invoked once per host the first time a response on that host is classified as a WAF/CDN block (captcha, bot-detection, or challenge page). It lets the scan warn the operator that traffic is being filtered and results against that host may be incomplete. The sink runs on the request goroutine, so it must be cheap and non-blocking. Call once during scan setup, before scanning concurrency starts; a nil sink is a no-op. The installed sink is shared by every WithContext clone of this requester.

func (*Requester) SetCarriedSessions

func (r *Requester) SetCarriedSessions(sessions map[string]httpmsg.CarriedSession)

SetCarriedSessions installs browser-harvested per-host sessions on the requester. Keys are normalized to bare lowercase hostnames; a request is matched against them by host so a session only ever reaches the host it was harvested from. Cookies are merged into (never over) a request's existing Cookie header, and a non-empty UserAgent is pinned — both applied before the operator's -H custom headers so an explicit -H Cookie/User-Agent still wins. Safe to call once during scan setup; a nil/empty map is a no-op.

func (*Requester) SetResponseObserver

func (r *Requester) SetResponseObserver(sink func(ObservedResponse))

SetResponseObserver installs a sink invoked for every server-error (5xx) response on this requester's traffic, so the executor can corroborate a leaked database error from ANY module's probe rather than only the module that sent it. The sink runs synchronously on the request goroutine — it must be cheap and non-blocking, and must copy any request/response bytes it retains. Call during scan setup; a nil sink clears the observer. The installed sink is shared by every WithContext clone of this requester.

func (*Requester) WithAbandonFlag

func (r *Requester) WithAbandonFlag(flag *atomic.Bool) *Requester

WithAbandonFlag returns a shallow clone whose Execute refuses to START new requests once flag is set. Requests already in flight are untouched.

This exists because a per-module timeout does not stop the module. The executor's watchdog returns (nil, false) and the caller discards the result, but the module's goroutine keeps running — and kept issuing fresh requests against the phase-bound requester. On a real scan that was 725+ timed-out modules still hammering the target, competing for the same per-host concurrency as live work, to produce findings that are thrown away by definition.

The signal is separate from WithContext, and only gates request INITIATION, for one specific reason: the request clusterer dedups concurrent identical requests through a singleflight group, so one in-flight request is shared by every module that asked for it. Attaching a per-module deadline to the outgoing request would let the first caller's timeout cancel a request other modules are legitimately waiting on. Gating initiation instead stops the abandoned module's future load without severing anyone's shared socket.

A flag rather than a context: this is only ever read as a boolean — never selected on, never attached to a request — and it is set once per module dispatch, the hottest fan-out loop in the scan. A context.WithCancel child there would allocate a context plus a CancelFunc and take the shared phase context's children-map lock twice per call, serializing hundreds of concurrent dispatches on one mutex to carry a single bit.

func (*Requester) WithContext

func (r *Requester) WithContext(ctx context.Context) *Requester

WithContext returns a shallow copy of the Requester whose context-less Execute uses ctx for cancellation. The copy shares the underlying HTTP clients, rate limiter, clusterer, and headers — only the default context differs — so it is cheap to create per scan task. The executor hands each active-module task a context-bound requester so a per-module timeout or scan shutdown aborts the module's in-flight requests even when the module calls Execute directly. A nil ctx returns the receiver unchanged.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL