Documentation
¶
Overview ¶
Package core contains the scan Executor — the central orchestrator of a native scan. The Executor receives HttpRequestResponse work items, dispatches them to registered active and passive modules through a bounded worker pool, and collects ResultEvent findings. It owns scope filtering, per-host rate limiting, pre/post hooks, per-module timeouts, context-based cancellation, and the post-EOF feedback drain that lets passive modules re-inject discovered requests. Subpackages provide the rate limiter, network dialer, service container, and scan statistics.
Index ¶
- func SuggestWorkerCount(moduleCount, maxWorkers int) int
- type ExecutionReport
- type Executor
- func (e *Executor) ConsideredModuleCount() int64
- func (e *Executor) Execute(ctx context.Context) (bool, error)
- func (e *Executor) FeedbackDropped() int64
- func (e *Executor) InFlight() int64
- func (e *Executor) ModuleMetrics() map[string]stats.ModuleStatsSnapshot
- func (e *Executor) Processed() int64
- func (e *Executor) Report() ExecutionReport
- func (e *Executor) ResolveRequestUUID(requestHash string) string
- func (e *Executor) Responded() int64
- func (e *Executor) StoreFailed() int64
- func (e *Executor) Stored() int64
- func (e *Executor) SuppressedFindings() int64
- type ExecutorConfig
- type FindingAdmission
- type HookRunner
- type InsertionPointCache
- func (c *InsertionPointCache) Add(key string, points []httpmsg.InsertionPoint)
- func (c *InsertionPointCache) Get(key string) ([]httpmsg.InsertionPoint, bool)
- func (c *InsertionPointCache) GetOrCompute(raw []byte, requestID string, includeNested bool) ([]httpmsg.InsertionPoint, error)
- func (c *InsertionPointCache) Stats() InsertionPointCacheStats
- type InsertionPointCacheStats
- type OASTFlusher
- type PauseController
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func SuggestWorkerCount ¶
SuggestWorkerCount returns a heuristic worker count for rescans. It scales with the number of active modules but caps at maxWorkers. The floor is 2 to ensure at least minimal parallelism.
Types ¶
type ExecutionReport ¶ added in v0.5.2
type ExecutionReport struct {
// StoppedEarly — the input source was not read to EOF (cancelled, deadline,
// or a paused scan that was stopped while waiting).
StoppedEarly bool
// DrainStalled — the feedback drain was abandoned with workers still in
// flight and making no progress.
DrainStalled bool
// WorkersAbandoned — one or more workers did not exit within the shutdown
// grace and were leaked.
WorkersAbandoned bool
// DeferredFlushSkipped — the end-of-run passive flush was skipped because a
// worker was abandoned, so deferred findings (secret detection, anomaly
// ranking) may be incomplete.
DeferredFlushSkipped bool
// Unacked counts work items a worker dequeued but deliberately did NOT
// acknowledge, because the context was already cancelled when it got them.
// Not acknowledging is what keeps the durable cursor behind them so a later
// run re-serves them; this number is how many that was.
Unacked int64
}
ExecutionReport says whether one Execute call processed its whole input, and what it gave up on if not.
Execute returns (bool, error) where the bool is "produced findings" and the error is nil on every one of these paths: a cancelled feed, an abandoned drain, abandoned workers, a skipped deferred flush. Each of those silently reduces coverage, and the caller had no way to learn about any of them — a scan whose workers were abandoned mid-flight reported exactly what a clean one did. This is that missing channel.
func (ExecutionReport) Clean ¶ added in v0.5.2
func (r ExecutionReport) Clean() bool
Clean reports whether the run covered its whole input with nothing abandoned.
A convenience for callers and tests that want the one-line question. The runner's own consumer (phaseTracker.noteExecution) deliberately does not use it: each flag maps to a distinct outcome reason code, so it has to read them individually.
type Executor ¶
type Executor struct {
// contains filtered or unexported fields
}
Executor orchestrates scanning with worker pool.
func NewExecutor ¶
func NewExecutor( cfg ExecutorConfig, src source.InputSource, activeModules []modules.ActiveModule, passiveModules []modules.PassiveModule, ) *Executor
NewExecutor creates a new Executor with the given configuration.
func (*Executor) ConsideredModuleCount ¶
ConsideredModuleCount returns the number of distinct modules whose CanProcess has been evaluated at least once during this scan. Reaches parity with the total enabled-module count once every module has been seen — including those whose CanProcess always rejects the input shape (e.g., POST-only modules in a GET-only scan). Use this for "modules scanned X/Y" status displays.
func (*Executor) Execute ¶
Execute runs the scan. Blocks until all inputs are processed or context is cancelled.
func (*Executor) FeedbackDropped ¶
FeedbackDropped returns the number of feedback items dropped due to channel capacity.
func (*Executor) InFlight ¶
InFlight returns the number of worker goroutines currently processing items. Zero means the executor is quiescent (all workers idle, no items in flight). Exposed for status reporting — callers typically combine this with source-level idle signals to distinguish "still doing work" from "waiting for input."
func (*Executor) ModuleMetrics ¶
func (e *Executor) ModuleMetrics() map[string]stats.ModuleStatsSnapshot
ModuleMetrics returns a point-in-time snapshot of per-module performance metrics.
func (*Executor) Report ¶ added in v0.5.2
func (e *Executor) Report() ExecutionReport
Report returns this executor's coverage self-report. Call it after Execute has returned; before that the fields are still being written.
func (*Executor) ResolveRequestUUID ¶
ResolveRequestUUID resolves a request hash to its database record UUID. Implements modkit.RequestUUIDResolver.
func (*Executor) StoreFailed ¶ added in v0.5.2
StoreFailed returns the number of input items whose primary record could not be written. Distinct from an item that was never stored because it got no response (Processed() minus Responded()) or because a filter dropped it: this one reached the database and the database refused it.
func (*Executor) Stored ¶ added in v0.5.2
Responded returns how many input items produced an HTTP response, which is always <= Processed(). Use this, not Processed(), to report how many targets answered: Processed() counts attempts, including the ones that never reached a server. See the responded field for why the two differ. Stored returns the number of input items whose primary record is in the database when this executor finished with them. Use this, not Processed(), to report how many records a run PERSISTED: Processed() counts attempts, and an attempt that never reached a host, or whose write failed, stores nothing.
func (*Executor) SuppressedFindings ¶
SuppressedFindings returns the number of candidate findings dropped by the body-differential safety net because a real payload-vs-baseline difference could not be re-confirmed.
type ExecutorConfig ¶
type ExecutorConfig struct {
Workers int
OnResult func(*output.ResultEvent)
OnCandidate func(*output.ResultEvent)
OnObservation func(*output.ResultEvent)
OnTraffic func(method, url string, statusCode int, contentType string) // Optional: called for each processed item
// OnAuthWall is called once per item whose redirect chain the transport
// refused to follow into a login / SSO wall (http.StoppedAtAuthWall). It
// carries the target that bounced and the wall it bounced to.
//
// The executor only reports; the phase decides what to do. That split is
// deliberate: the wall host is a scan-wide fact (it feeds the fuzz-target
// exclusion the spidering phase already maintains) and the executor has no
// business owning scan-wide state, while re-deriving it later from stored
// records would mean re-parsing every 3xx in the corpus.
OnAuthWall func(target, wall string)
Services *services.Services
HTTPRequester *http.Requester
Repository *database.Repository // Optional: database storage
RecordWriter *database.RecordWriter // Optional: batched record writer (preferred over Repository.SaveRecord)
FindingWriter *database.FindingWriter // Optional: batched finding writer (preferred over Repository.SaveFinding)
ScanUUID string
// ProjectUUID owns every record and finding this executor writes. An empty
// value is silently coerced to the DEFAULT project by the repository (see
// database.defaultProjectUUID), so a config carrying a Repository/
// RecordWriter/FindingWriter MUST set it or its output lands in a project
// nobody is reading. Enforced by TestExecutorConfigAlwaysSetsProjectUUID.
ProjectUUID string
Hooks HookRunner // Optional: pre/post hooks
ScopeMatcher *config.ScopeMatcher // Optional: scope filtering
ScopeOnIngest bool // When true, skip both save and scan for out-of-scope items
StaticFileMatcher *config.ScopeMatcher // Optional: always-on static file filtering (independent of ScopeMatcher)
FollowSubdomains bool // When true, subdomain_harvest adds discovered subdomains to scope and feeds them (needs ScopeMatcher + feedback)
SkipBaseline bool // When true, skip HTTP fetch if response already attached (Phase 3 DB source)
// RecordRedirectChain persists each followed redirect hop as its own record,
// chained through parent_uuid, and re-pairs the final response with the
// request that actually produced it. Off by default: it multiplies row
// counts on every redirecting URL, and only a host-sweep style phase wants
// the intermediate hops. See saveRedirectHops.
RecordRedirectChain bool
// RecordSource is the http_records.source label this executor stamps. Empty
// means "scanner", which is what every phase used before the label became
// configurable — so a phase that says nothing is unchanged.
RecordSource string
OASTProvider modkit.OASTProvider // Optional: OAST callback URL generator for blind vuln detection
OASTService OASTFlusher // Optional: OAST service to flush after scanning
PauseCtrl *PauseController // Optional: cooperative pause/resume controller
MaxFindingsPerModule int // When > 0, suppress findings after this many per module
// FindingAdmission optionally shares the per-module cap and the
// already-admitted identity set with other executors. nil = executor-local,
// which is right for a phase that runs exactly one Execute. The
// dynamic-assessment phase passes ONE admission to every feedback round, so
// the cap spans the phase and a root cause re-found in a later round does not
// re-fire its callbacks. See FindingAdmission.
FindingAdmission *FindingAdmission
MaxDuration time.Duration // When > 0, cancel execution after this duration
FeedbackDrainTimeout time.Duration // Idle timeout for draining feedback after source EOF (default: 100ms)
FeedbackDrainMaxStall time.Duration // Hard cap on draining with workers in-flight but making no progress (0 = 2x active module timeout). Guards against a module that ignores cancellation.
WorkerExitGrace time.Duration // Grace for worker goroutines to exit after the item channel is closed (0 = default 60s, capped at FeedbackDrainMaxStall). Kept short and separate from the drain-stall cap so shutdown stays responsive.
IPCacheSize int // LRU cache size for parsed insertion points (default: 4096)
IPCache *InsertionPointCache // Optional: shared IP cache (if nil, a new one is created)
ParallelPassive bool // When true, run passive per-request modules concurrently
PassiveModuleTimeout time.Duration // Timeout per passive module call (default: 5s). 0 uses default.
ActiveModuleTimeout time.Duration // Timeout per active module call (default: 300s). 0 uses default. Modules may raise via TimeoutHinter.
AdaptiveWorkers bool // When true, dynamically scale worker count based on queue depth
MinWorkers int // Floor for adaptive scaling (default: 2)
MaxWorkers int // Ceiling for adaptive scaling (default: Workers*4)
ActiveTaskLimit int // Max concurrent active module tasks across host/request/IP scopes
OnStatus func(processed, total, findings, distinctModules, activeCount, passiveCount, timedOut int64, elapsed time.Duration) // Optional: periodic status callback
StatusInterval time.Duration // Interval for OnStatus callback (default: 30s)
// ModuleTimeouts, when non-nil, is an externally-owned counter the executor
// increments on each active-module timeout (instead of its own private one).
// A multi-round phase passes one shared counter so the timed-out total in the
// status line accumulates across the per-round executors it creates.
ModuleTimeouts *atomic.Int64
// FirstStatusInterval, when > 0 and shorter than StatusInterval, fires the
// very first status tick after this duration instead of waiting a full
// StatusInterval. Useful for long-cadence status (e.g., 2m) where users
// shouldn't have to stare at silence before the first line appears.
FirstStatusInterval time.Duration
// DisableFeedback drops every URL discovered by passive modules instead of
// re-injecting it into the scan. With this flag the executor scans exactly
// the items the input source delivers and nothing more — matching the
// shallow per-request semantics of `vigolium scan-request`. Used by
// scan-on-receive's default mode to avoid the cascade of new HTTP records
// that link extractors and redirect followers would otherwise generate.
DisableFeedback bool
// TechFilterDisabled bypasses the tech-stack allowlist gate (see modules.TechAware).
// Set by --no-tech-filter and automatically by --intensity=deep so deep scans
// run every module regardless of detected stack.
TechFilterDisabled bool
// OnTechDetected fires once per (host, tag) when a passive fingerprint
// module first publishes a stack detection.
OnTechDetected func(host, tag string)
// DeepScan mirrors --intensity=deep into ScanContext so modules can unlock
// heavier probing (e.g. dashboard_exposure's full mount-path sweep).
DeepScan bool
// ContentClassByHost seeds the per-host content-class registry from the
// heuristics root probe (host → modkit.ContentClass string). It is the
// fallback consulted by the content-class gate when a record's own
// Content-Type is indeterminate. Honors the same TechFilterDisabled switch.
ContentClassByHost map[string]string
}
ExecutorConfig configures the Executor behavior.
func DefaultExecutorConfig ¶
func DefaultExecutorConfig() ExecutorConfig
DefaultExecutorConfig returns sensible defaults.
type FindingAdmission ¶ added in v0.5.2
type FindingAdmission struct {
// contains filtered or unexported fields
}
FindingAdmission is the state behind the per-module finding cap: how many findings each module has had admitted, and which root-cause identities have already been admitted.
It is a separate type so it can OUTLIVE one Executor. The dynamic-assessment phase builds a fresh executor per feedback round, so this state used to reset between rounds: a cap documented as holding "for the remainder of the scan" actually restarted every round, and a root cause re-found in round 2 re-fired every callback and notification for a finding already reported in round 1. Sharing one admission across the phase's rounds is what makes the documented contract true (triage C7: per dynamic-assessment phase, shared across rounds; other phases are uncapped).
The zero value is ready to use, so an Executor built without a shared one gets its own by construction and a literal &Executor{cfg: …} in a test still works. Safe for concurrent use: both fields are sync.Maps.
func NewFindingAdmission ¶ added in v0.5.2
func NewFindingAdmission() *FindingAdmission
NewFindingAdmission returns admission state a caller can share across several executors — one per dynamic-assessment phase, covering all of its rounds.
type HookRunner ¶
type HookRunner interface {
RunPreHooks(req *httpmsg.HttpRequestResponse) (*httpmsg.HttpRequestResponse, error)
RunPostHooks(result *output.ResultEvent) (*output.ResultEvent, error)
}
HookRunner transforms requests before scanning and filters results after scanning.
type InsertionPointCache ¶
type InsertionPointCache struct {
// contains filtered or unexported fields
}
InsertionPointCache is a byte-aware LRU of insertion-point sets, keyed by request hash. It bounds both entry count (via the underlying LRU) and total retained bytes, and rejects individual entries above a per-entry ceiling.
A nil *InsertionPointCache is usable: every method degrades to a miss, so a caller with no cache configured needs no branch of its own.
All access is serialized on one mutex. That is deliberate: the byte counter and the LRU must move together or the budget drifts, and the eviction callback updates the counter while the LRU's own lock is held — so it must not attempt to re-acquire this mutex. Because every caller path holds the mutex around the LRU call, the callback runs with exclusive access already and updates the counter directly. Contention is not a concern: lookups are per-request, not per-payload.
func NewInsertionPointCache ¶
func NewInsertionPointCache(entries int) *InsertionPointCache
NewInsertionPointCache creates a byte-aware insertion-point cache holding at most entries items. A non-positive entries value falls back to a safe default.
func (*InsertionPointCache) Add ¶
func (c *InsertionPointCache) Add(key string, points []httpmsg.InsertionPoint)
Add caches points under key unless the set is too large to be worth retaining. Oversized sets are counted as rejections and left uncached — the caller's copy is unaffected either way, so a rejection costs only a later re-derivation.
func (*InsertionPointCache) Get ¶
func (c *InsertionPointCache) Get(key string) ([]httpmsg.InsertionPoint, bool)
Get returns the cached insertion points for key.
func (*InsertionPointCache) GetOrCompute ¶
func (c *InsertionPointCache) GetOrCompute(raw []byte, requestID string, includeNested bool) ([]httpmsg.InsertionPoint, error)
GetOrCompute returns the insertion points for requestID, deriving and caching them on a miss.
This owns the cache-key convention (the includeNested variants are separate entries) so no caller has to reproduce it. Both the per-insertion-point scan stage and the module-facing provider go through here; deriving a key inline instead is how the two ends up disagreeing about which variant they stored.
func (*InsertionPointCache) Stats ¶
func (c *InsertionPointCache) Stats() InsertionPointCacheStats
Stats returns a snapshot of the cache's current size and cumulative counters.
type InsertionPointCacheStats ¶
type InsertionPointCacheStats struct {
Entries int
RetainedBytes int64
Hits int64
Misses int64
// Evictions counts only byte-budget evictions. Ordinary count-bound LRU
// evictions are not counted: they are the cache working as configured, while a
// byte eviction means the budget — not the entry cap — is the binding limit.
Evictions int64
// Rejected counts sets never cached because they exceeded the per-entry ceiling.
Rejected int64
}
InsertionPointCacheStats is a point-in-time snapshot of cache behaviour, for logging and metrics. Counters are cumulative for the cache's lifetime.
type OASTFlusher ¶
type OASTFlusher interface {
Flush()
Close()
}
OASTFlusher is implemented by the OAST service to allow the executor to flush pending interactions after scanning completes.
type PauseController ¶
type PauseController struct {
// contains filtered or unexported fields
}
PauseController provides cooperative pause/resume for worker goroutines. Workers call WaitIfPaused between items; Pause blocks until all active items finish, then holds future workers until Resume is called.
stateMu serializes the Pause/Resume transitions in full (including the mu Lock/Unlock that spans a pause), so concurrent Pause+Resume — reachable, e.g. a stop-triggered Runner.Close resuming while an API pause request is in flight — can never close a nil channel, unlock an unlocked mutex, or leave pausedCh stale relative to the paused flag.
func NewPauseController ¶
func NewPauseController() *PauseController
NewPauseController creates a new PauseController in the unpaused state.
func (*PauseController) AcquireWorker ¶
func (p *PauseController) AcquireWorker()
AcquireWorker acquires a read lock for the duration of item processing. Call ReleaseWorker when done. Returns false if paused (caller should call WaitIfPaused first).
func (*PauseController) IsPaused ¶
func (p *PauseController) IsPaused() bool
IsPaused returns the current pause state.
func (*PauseController) Pause ¶
func (p *PauseController) Pause()
Pause blocks new workers and waits for in-flight items to finish. Safe to call multiple times (idempotent) and concurrently with Resume.
func (*PauseController) ReleaseWorker ¶
func (p *PauseController) ReleaseWorker()
ReleaseWorker releases the read lock after item processing.
func (*PauseController) Resume ¶
func (p *PauseController) Resume()
Resume unblocks paused workers. Safe to call when not paused and concurrently with Pause.
func (*PauseController) WaitIfPaused ¶
func (p *PauseController) WaitIfPaused(ctx context.Context) bool
WaitIfPaused blocks the caller if the controller is paused. Returns false if the context is cancelled while waiting. Workers should call this between processing items.