Documentation
¶
Overview ¶
Package core contains the scan Executor — the central orchestrator of a native scan. The Executor receives HttpRequestResponse work items, dispatches them to registered active and passive modules through a bounded worker pool, and collects ResultEvent findings. It owns scope filtering, per-host rate limiting, pre/post hooks, per-module timeouts, context-based cancellation, and the post-EOF feedback drain that lets passive modules re-inject discovered requests. Subpackages provide the rate limiter, network dialer, service container, and scan statistics.
Index ¶
- func SuggestWorkerCount(moduleCount, maxWorkers int) int
- type Executor
- func (e *Executor) ConsideredModuleCount() int64
- func (e *Executor) Execute(ctx context.Context) (bool, error)
- func (e *Executor) FeedbackDropped() int64
- func (e *Executor) InFlight() int64
- func (e *Executor) ModuleMetrics() map[string]stats.ModuleStatsSnapshot
- func (e *Executor) Processed() int64
- func (e *Executor) ResolveRequestUUID(requestHash string) string
- func (e *Executor) Responded() int64
- func (e *Executor) SuppressedFindings() int64
- type ExecutorConfig
- type HookRunner
- type InsertionPointCache
- func (c *InsertionPointCache) Add(key string, points []httpmsg.InsertionPoint)
- func (c *InsertionPointCache) Get(key string) ([]httpmsg.InsertionPoint, bool)
- func (c *InsertionPointCache) GetOrCompute(raw []byte, requestID string, includeNested bool) ([]httpmsg.InsertionPoint, error)
- func (c *InsertionPointCache) Stats() InsertionPointCacheStats
- type InsertionPointCacheStats
- type OASTFlusher
- type PauseController
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func SuggestWorkerCount ¶
SuggestWorkerCount returns a heuristic worker count for rescans. It scales with the number of active modules but caps at maxWorkers. The floor is 2 to ensure at least minimal parallelism.
Types ¶
type Executor ¶
type Executor struct {
// contains filtered or unexported fields
}
Executor orchestrates scanning with worker pool.
func NewExecutor ¶
func NewExecutor( cfg ExecutorConfig, src source.InputSource, activeModules []modules.ActiveModule, passiveModules []modules.PassiveModule, ) *Executor
NewExecutor creates a new Executor with the given configuration.
func (*Executor) ConsideredModuleCount ¶
ConsideredModuleCount returns the number of distinct modules whose CanProcess has been evaluated at least once during this scan. Reaches parity with the total enabled-module count once every module has been seen — including those whose CanProcess always rejects the input shape (e.g., POST-only modules in a GET-only scan). Use this for "modules scanned X/Y" status displays.
func (*Executor) Execute ¶
Execute runs the scan. Blocks until all inputs are processed or context is cancelled.
func (*Executor) FeedbackDropped ¶
FeedbackDropped returns the number of feedback items dropped due to channel capacity.
func (*Executor) InFlight ¶
InFlight returns the number of worker goroutines currently processing items. Zero means the executor is quiescent (all workers idle, no items in flight). Exposed for status reporting — callers typically combine this with source-level idle signals to distinguish "still doing work" from "waiting for input."
func (*Executor) ModuleMetrics ¶
func (e *Executor) ModuleMetrics() map[string]stats.ModuleStatsSnapshot
ModuleMetrics returns a point-in-time snapshot of per-module performance metrics.
func (*Executor) ResolveRequestUUID ¶
ResolveRequestUUID resolves a request hash to its database record UUID. Implements modkit.RequestUUIDResolver.
func (*Executor) Responded ¶
Responded returns how many input items produced an HTTP response, which is always <= Processed(). Use this, not Processed(), to report how many targets answered: Processed() counts attempts, including the ones that never reached a server. See the responded field for why the two differ.
func (*Executor) SuppressedFindings ¶
SuppressedFindings returns the number of candidate findings dropped by the body-differential safety net because a real payload-vs-baseline difference could not be re-confirmed.
type ExecutorConfig ¶
type ExecutorConfig struct {
Workers int
OnResult func(*output.ResultEvent)
OnCandidate func(*output.ResultEvent)
OnObservation func(*output.ResultEvent)
OnTraffic func(method, url string, statusCode int, contentType string) // Optional: called for each processed item
// OnAuthWall is called once per item whose redirect chain the transport
// refused to follow into a login / SSO wall (http.StoppedAtAuthWall). It
// carries the target that bounced and the wall it bounced to.
//
// The executor only reports; the phase decides what to do. That split is
// deliberate: the wall host is a scan-wide fact (it feeds the fuzz-target
// exclusion the spidering phase already maintains) and the executor has no
// business owning scan-wide state, while re-deriving it later from stored
// records would mean re-parsing every 3xx in the corpus.
OnAuthWall func(target, wall string)
Services *services.Services
HTTPRequester *http.Requester
Repository *database.Repository // Optional: database storage
RecordWriter *database.RecordWriter // Optional: batched record writer (preferred over Repository.SaveRecord)
FindingWriter *database.FindingWriter // Optional: batched finding writer (preferred over Repository.SaveFinding)
ScanUUID string
// ProjectUUID owns every record and finding this executor writes. An empty
// value is silently coerced to the DEFAULT project by the repository (see
// database.defaultProjectUUID), so a config carrying a Repository/
// RecordWriter/FindingWriter MUST set it or its output lands in a project
// nobody is reading. Enforced by TestExecutorConfigAlwaysSetsProjectUUID.
ProjectUUID string
Hooks HookRunner // Optional: pre/post hooks
ScopeMatcher *config.ScopeMatcher // Optional: scope filtering
ScopeOnIngest bool // When true, skip both save and scan for out-of-scope items
StaticFileMatcher *config.ScopeMatcher // Optional: always-on static file filtering (independent of ScopeMatcher)
FollowSubdomains bool // When true, subdomain_harvest adds discovered subdomains to scope and feeds them (needs ScopeMatcher + feedback)
SkipBaseline bool // When true, skip HTTP fetch if response already attached (Phase 3 DB source)
// RecordRedirectChain persists each followed redirect hop as its own record,
// chained through parent_uuid, and re-pairs the final response with the
// request that actually produced it. Off by default: it multiplies row
// counts on every redirecting URL, and only a host-sweep style phase wants
// the intermediate hops. See saveRedirectHops.
RecordRedirectChain bool
// RecordSource is the http_records.source label this executor stamps. Empty
// means "scanner", which is what every phase used before the label became
// configurable — so a phase that says nothing is unchanged.
RecordSource string
OASTProvider modkit.OASTProvider // Optional: OAST callback URL generator for blind vuln detection
OASTService OASTFlusher // Optional: OAST service to flush after scanning
PauseCtrl *PauseController // Optional: cooperative pause/resume controller
MaxFindingsPerModule int // When > 0, suppress findings after this many per module
MaxDuration time.Duration // When > 0, cancel execution after this duration
FeedbackDrainTimeout time.Duration // Idle timeout for draining feedback after source EOF (default: 100ms)
FeedbackDrainMaxStall time.Duration // Hard cap on draining with workers in-flight but making no progress (0 = 2x active module timeout). Guards against a module that ignores cancellation.
WorkerExitGrace time.Duration // Grace for worker goroutines to exit after the item channel is closed (0 = default 60s, capped at FeedbackDrainMaxStall). Kept short and separate from the drain-stall cap so shutdown stays responsive.
IPCacheSize int // LRU cache size for parsed insertion points (default: 4096)
IPCache *InsertionPointCache // Optional: shared IP cache (if nil, a new one is created)
ParallelPassive bool // When true, run passive per-request modules concurrently
PassiveModuleTimeout time.Duration // Timeout per passive module call (default: 5s). 0 uses default.
ActiveModuleTimeout time.Duration // Timeout per active module call (default: 300s). 0 uses default. Modules may raise via TimeoutHinter.
AdaptiveWorkers bool // When true, dynamically scale worker count based on queue depth
MinWorkers int // Floor for adaptive scaling (default: 2)
MaxWorkers int // Ceiling for adaptive scaling (default: Workers*4)
ActiveTaskLimit int // Max concurrent active module tasks across host/request/IP scopes
OnStatus func(processed, total, findings, distinctModules, activeCount, passiveCount, timedOut int64, elapsed time.Duration) // Optional: periodic status callback
StatusInterval time.Duration // Interval for OnStatus callback (default: 30s)
// ModuleTimeouts, when non-nil, is an externally-owned counter the executor
// increments on each active-module timeout (instead of its own private one).
// A multi-round phase passes one shared counter so the timed-out total in the
// status line accumulates across the per-round executors it creates.
ModuleTimeouts *atomic.Int64
// FirstStatusInterval, when > 0 and shorter than StatusInterval, fires the
// very first status tick after this duration instead of waiting a full
// StatusInterval. Useful for long-cadence status (e.g., 2m) where users
// shouldn't have to stare at silence before the first line appears.
FirstStatusInterval time.Duration
// DisableFeedback drops every URL discovered by passive modules instead of
// re-injecting it into the scan. With this flag the executor scans exactly
// the items the input source delivers and nothing more — matching the
// shallow per-request semantics of `vigolium scan-request`. Used by
// scan-on-receive's default mode to avoid the cascade of new HTTP records
// that link extractors and redirect followers would otherwise generate.
DisableFeedback bool
// TechFilterDisabled bypasses the tech-stack allowlist gate (see modules.TechAware).
// Set by --no-tech-filter and automatically by --intensity=deep so deep scans
// run every module regardless of detected stack.
TechFilterDisabled bool
// OnTechDetected fires once per (host, tag) when a passive fingerprint
// module first publishes a stack detection.
OnTechDetected func(host, tag string)
// DeepScan mirrors --intensity=deep into ScanContext so modules can unlock
// heavier probing (e.g. dashboard_exposure's full mount-path sweep).
DeepScan bool
// ContentClassByHost seeds the per-host content-class registry from the
// heuristics root probe (host → modkit.ContentClass string). It is the
// fallback consulted by the content-class gate when a record's own
// Content-Type is indeterminate. Honors the same TechFilterDisabled switch.
ContentClassByHost map[string]string
}
ExecutorConfig configures the Executor behavior.
func DefaultExecutorConfig ¶
func DefaultExecutorConfig() ExecutorConfig
DefaultExecutorConfig returns sensible defaults.
type HookRunner ¶
type HookRunner interface {
RunPreHooks(req *httpmsg.HttpRequestResponse) (*httpmsg.HttpRequestResponse, error)
RunPostHooks(result *output.ResultEvent) (*output.ResultEvent, error)
}
HookRunner transforms requests before scanning and filters results after scanning.
type InsertionPointCache ¶
type InsertionPointCache struct {
// contains filtered or unexported fields
}
InsertionPointCache is a byte-aware LRU of insertion-point sets, keyed by request hash. It bounds both entry count (via the underlying LRU) and total retained bytes, and rejects individual entries above a per-entry ceiling.
A nil *InsertionPointCache is usable: every method degrades to a miss, so a caller with no cache configured needs no branch of its own.
All access is serialized on one mutex. That is deliberate: the byte counter and the LRU must move together or the budget drifts, and the eviction callback updates the counter while the LRU's own lock is held — so it must not attempt to re-acquire this mutex. Because every caller path holds the mutex around the LRU call, the callback runs with exclusive access already and updates the counter directly. Contention is not a concern: lookups are per-request, not per-payload.
func NewInsertionPointCache ¶
func NewInsertionPointCache(entries int) *InsertionPointCache
NewInsertionPointCache creates a byte-aware insertion-point cache holding at most entries items. A non-positive entries value falls back to a safe default.
func (*InsertionPointCache) Add ¶
func (c *InsertionPointCache) Add(key string, points []httpmsg.InsertionPoint)
Add caches points under key unless the set is too large to be worth retaining. Oversized sets are counted as rejections and left uncached — the caller's copy is unaffected either way, so a rejection costs only a later re-derivation.
func (*InsertionPointCache) Get ¶
func (c *InsertionPointCache) Get(key string) ([]httpmsg.InsertionPoint, bool)
Get returns the cached insertion points for key.
func (*InsertionPointCache) GetOrCompute ¶
func (c *InsertionPointCache) GetOrCompute(raw []byte, requestID string, includeNested bool) ([]httpmsg.InsertionPoint, error)
GetOrCompute returns the insertion points for requestID, deriving and caching them on a miss.
This owns the cache-key convention (the includeNested variants are separate entries) so no caller has to reproduce it. Both the per-insertion-point scan stage and the module-facing provider go through here; deriving a key inline instead is how the two ends up disagreeing about which variant they stored.
func (*InsertionPointCache) Stats ¶
func (c *InsertionPointCache) Stats() InsertionPointCacheStats
Stats returns a snapshot of the cache's current size and cumulative counters.
type InsertionPointCacheStats ¶
type InsertionPointCacheStats struct {
Entries int
RetainedBytes int64
Hits int64
Misses int64
// Evictions counts only byte-budget evictions. Ordinary count-bound LRU
// evictions are not counted: they are the cache working as configured, while a
// byte eviction means the budget — not the entry cap — is the binding limit.
Evictions int64
// Rejected counts sets never cached because they exceeded the per-entry ceiling.
Rejected int64
}
InsertionPointCacheStats is a point-in-time snapshot of cache behaviour, for logging and metrics. Counters are cumulative for the cache's lifetime.
type OASTFlusher ¶
type OASTFlusher interface {
Flush()
Close()
}
OASTFlusher is implemented by the OAST service to allow the executor to flush pending interactions after scanning completes.
type PauseController ¶
type PauseController struct {
// contains filtered or unexported fields
}
PauseController provides cooperative pause/resume for worker goroutines. Workers call WaitIfPaused between items; Pause blocks until all active items finish, then holds future workers until Resume is called.
stateMu serializes the Pause/Resume transitions in full (including the mu Lock/Unlock that spans a pause), so concurrent Pause+Resume — reachable, e.g. a stop-triggered Runner.Close resuming while an API pause request is in flight — can never close a nil channel, unlock an unlocked mutex, or leave pausedCh stale relative to the paused flag.
func NewPauseController ¶
func NewPauseController() *PauseController
NewPauseController creates a new PauseController in the unpaused state.
func (*PauseController) AcquireWorker ¶
func (p *PauseController) AcquireWorker()
AcquireWorker acquires a read lock for the duration of item processing. Call ReleaseWorker when done. Returns false if paused (caller should call WaitIfPaused first).
func (*PauseController) IsPaused ¶
func (p *PauseController) IsPaused() bool
IsPaused returns the current pause state.
func (*PauseController) Pause ¶
func (p *PauseController) Pause()
Pause blocks new workers and waits for in-flight items to finish. Safe to call multiple times (idempotent) and concurrently with Resume.
func (*PauseController) ReleaseWorker ¶
func (p *PauseController) ReleaseWorker()
ReleaseWorker releases the read lock after item processing.
func (*PauseController) Resume ¶
func (p *PauseController) Resume()
Resume unblocks paused workers. Safe to call when not paused and concurrently with Pause.
func (*PauseController) WaitIfPaused ¶
func (p *PauseController) WaitIfPaused(ctx context.Context) bool
WaitIfPaused blocks the caller if the controller is paused. Returns false if the context is cancelled while waiting. Workers should call this between processing items.