Documentation
¶
Overview ¶
Package urlcand provides shared URL candidate extraction for reputation-feed checkers (urlhaus, threatfox). A single Extract call replaces the per-checker redundant regex walk + defang copy that the old code performed on every buffer.
The extraction logic is identical to what the old per-checker Check methods did inline: FindAll on the raw buffer (raw candidates), then — only when the cheap byte-gate fires — FindAll on the defanged copy (deobfuscated candidates). All raw candidates come first; deobfuscated ones follow. A shared budget caps the total across both passes.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NormalizeHTTPURL ¶ added in v1.1.0
NormalizeHTTPURL returns a canonical http(s) URL for feed set comparison, the bare lowercase hostname, and the raw IPv4 hostname when present. It lowercases scheme/host, strips default ports and fragments, removes a bare trailing "/", and trims common trailing punctuation from regex captures.
Types ¶
type Candidate ¶
type Candidate struct {
Raw string // the raw URL string as found in the buffer
Deobf bool // true when found only in the defanged copy
Norm string // canonical http(s) URL form for feed lookup, "" when invalid
Host string // lowercase hostname from Norm, "" when invalid
IP string // Host when it is a dotted-decimal IPv4 address, else ""
// contains filtered or unexported fields
}
Candidate is one URL string extracted from a buffer.
func Extract ¶
Extract extracts URL candidates from data. If maxURLs <= 0 it defaults to 64. Raw candidates (Deobf=false) come first; defanged candidates (Deobf=true) follow using the remaining budget. The total number of candidates never exceeds maxURLs.
The extraction mirrors the semantics of the old per-checker inline loop: budget is decremented once per regex match (not per normalized/valid URL), so the same first-N matches are produced regardless of which checker subsequently processes them.
func NewCandidate ¶ added in v1.1.0
NewCandidate builds a candidate. Normalized lookup fields are filled lazily on first feed use so clean extraction avoids URL parsing until it is needed.
Source Files