core

package
v0.4.9 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 20, 2026 License: MIT Imports: 34 Imported by: 0

Documentation

Overview

Package core contains the scan Executor — the central orchestrator of a native scan. The Executor receives HttpRequestResponse work items, dispatches them to registered active and passive modules through a bounded worker pool, and collects ResultEvent findings. It owns scope filtering, per-host rate limiting, pre/post hooks, per-module timeouts, context-based cancellation, and the post-EOF feedback drain that lets passive modules re-inject discovered requests. Subpackages provide the rate limiter, network dialer, service container, and scan statistics.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func SuggestWorkerCount

func SuggestWorkerCount(moduleCount, maxWorkers int) int

SuggestWorkerCount returns a heuristic worker count for rescans. It scales with the number of active modules but caps at maxWorkers. The floor is 2 to ensure at least minimal parallelism.

Types

type Executor

type Executor struct {
	// contains filtered or unexported fields
}

Executor orchestrates scanning with worker pool.

func NewExecutor

func NewExecutor(
	cfg ExecutorConfig,
	src source.InputSource,
	activeModules []modules.ActiveModule,
	passiveModules []modules.PassiveModule,
) *Executor

NewExecutor creates a new Executor with the given configuration.

func (*Executor) ConsideredModuleCount

func (e *Executor) ConsideredModuleCount() int64

ConsideredModuleCount returns the number of distinct modules whose CanProcess has been evaluated at least once during this scan. Reaches parity with the total enabled-module count once every module has been seen — including those whose CanProcess always rejects the input shape (e.g., POST-only modules in a GET-only scan). Use this for "modules scanned X/Y" status displays.

func (*Executor) Execute

func (e *Executor) Execute(ctx context.Context) (bool, error)

Execute runs the scan. Blocks until all inputs are processed or context is cancelled.

func (*Executor) FeedbackDropped

func (e *Executor) FeedbackDropped() int64

FeedbackDropped returns the number of feedback items dropped due to channel capacity.

func (*Executor) InFlight

func (e *Executor) InFlight() int64

InFlight returns the number of worker goroutines currently processing items. Zero means the executor is quiescent (all workers idle, no items in flight). Exposed for status reporting — callers typically combine this with source-level idle signals to distinguish "still doing work" from "waiting for input."

func (*Executor) ModuleMetrics

func (e *Executor) ModuleMetrics() map[string]stats.ModuleStatsSnapshot

ModuleMetrics returns a point-in-time snapshot of per-module performance metrics.

func (*Executor) Processed

func (e *Executor) Processed() int64

Processed returns the number of items processed by the executor.

func (*Executor) ResolveRequestUUID

func (e *Executor) ResolveRequestUUID(requestHash string) string

ResolveRequestUUID resolves a request hash to its database record UUID. Implements modkit.RequestUUIDResolver.

func (*Executor) Responded

func (e *Executor) Responded() int64

Responded returns how many input items produced an HTTP response, which is always <= Processed(). Use this, not Processed(), to report how many targets answered: Processed() counts attempts, including the ones that never reached a server. See the responded field for why the two differ.

func (*Executor) SuppressedFindings

func (e *Executor) SuppressedFindings() int64

SuppressedFindings returns the number of candidate findings dropped by the body-differential safety net because a real payload-vs-baseline difference could not be re-confirmed.

type ExecutorConfig

type ExecutorConfig struct {
	Workers       int
	OnResult      func(*output.ResultEvent)
	OnCandidate   func(*output.ResultEvent)
	OnObservation func(*output.ResultEvent)
	OnTraffic     func(method, url string, statusCode int, contentType string) // Optional: called for each processed item
	// OnAuthWall is called once per item whose redirect chain the transport
	// refused to follow into a login / SSO wall (http.StoppedAtAuthWall). It
	// carries the target that bounced and the wall it bounced to.
	//
	// The executor only reports; the phase decides what to do. That split is
	// deliberate: the wall host is a scan-wide fact (it feeds the fuzz-target
	// exclusion the spidering phase already maintains) and the executor has no
	// business owning scan-wide state, while re-deriving it later from stored
	// records would mean re-parsing every 3xx in the corpus.
	OnAuthWall    func(target, wall string)
	Services      *services.Services
	HTTPRequester *http.Requester
	Repository    *database.Repository    // Optional: database storage
	RecordWriter  *database.RecordWriter  // Optional: batched record writer (preferred over Repository.SaveRecord)
	FindingWriter *database.FindingWriter // Optional: batched finding writer (preferred over Repository.SaveFinding)
	ScanUUID      string
	// ProjectUUID owns every record and finding this executor writes. An empty
	// value is silently coerced to the DEFAULT project by the repository (see
	// database.defaultProjectUUID), so a config carrying a Repository/
	// RecordWriter/FindingWriter MUST set it or its output lands in a project
	// nobody is reading. Enforced by TestExecutorConfigAlwaysSetsProjectUUID.
	ProjectUUID       string
	Hooks             HookRunner           // Optional: pre/post hooks
	ScopeMatcher      *config.ScopeMatcher // Optional: scope filtering
	ScopeOnIngest     bool                 // When true, skip both save and scan for out-of-scope items
	StaticFileMatcher *config.ScopeMatcher // Optional: always-on static file filtering (independent of ScopeMatcher)
	FollowSubdomains  bool                 // When true, subdomain_harvest adds discovered subdomains to scope and feeds them (needs ScopeMatcher + feedback)
	SkipBaseline      bool                 // When true, skip HTTP fetch if response already attached (Phase 3 DB source)
	// RecordRedirectChain persists each followed redirect hop as its own record,
	// chained through parent_uuid, and re-pairs the final response with the
	// request that actually produced it. Off by default: it multiplies row
	// counts on every redirecting URL, and only a host-sweep style phase wants
	// the intermediate hops. See saveRedirectHops.
	RecordRedirectChain bool
	// RecordSource is the http_records.source label this executor stamps. Empty
	// means "scanner", which is what every phase used before the label became
	// configurable — so a phase that says nothing is unchanged.
	RecordSource          string
	OASTProvider          modkit.OASTProvider                                                                                                 // Optional: OAST callback URL generator for blind vuln detection
	OASTService           OASTFlusher                                                                                                         // Optional: OAST service to flush after scanning
	PauseCtrl             *PauseController                                                                                                    // Optional: cooperative pause/resume controller
	MaxFindingsPerModule  int                                                                                                                 // When > 0, suppress findings after this many per module
	MaxDuration           time.Duration                                                                                                       // When > 0, cancel execution after this duration
	FeedbackDrainTimeout  time.Duration                                                                                                       // Idle timeout for draining feedback after source EOF (default: 100ms)
	FeedbackDrainMaxStall time.Duration                                                                                                       // Hard cap on draining with workers in-flight but making no progress (0 = 2x active module timeout). Guards against a module that ignores cancellation.
	WorkerExitGrace       time.Duration                                                                                                       // Grace for worker goroutines to exit after the item channel is closed (0 = default 60s, capped at FeedbackDrainMaxStall). Kept short and separate from the drain-stall cap so shutdown stays responsive.
	IPCacheSize           int                                                                                                                 // LRU cache size for parsed insertion points (default: 4096)
	IPCache               *InsertionPointCache                                                                                                // Optional: shared IP cache (if nil, a new one is created)
	ParallelPassive       bool                                                                                                                // When true, run passive per-request modules concurrently
	PassiveModuleTimeout  time.Duration                                                                                                       // Timeout per passive module call (default: 5s). 0 uses default.
	ActiveModuleTimeout   time.Duration                                                                                                       // Timeout per active module call (default: 300s). 0 uses default. Modules may raise via TimeoutHinter.
	AdaptiveWorkers       bool                                                                                                                // When true, dynamically scale worker count based on queue depth
	MinWorkers            int                                                                                                                 // Floor for adaptive scaling (default: 2)
	MaxWorkers            int                                                                                                                 // Ceiling for adaptive scaling (default: Workers*4)
	ActiveTaskLimit       int                                                                                                                 // Max concurrent active module tasks across host/request/IP scopes
	OnStatus              func(processed, total, findings, distinctModules, activeCount, passiveCount, timedOut int64, elapsed time.Duration) // Optional: periodic status callback
	StatusInterval        time.Duration                                                                                                       // Interval for OnStatus callback (default: 30s)
	// ModuleTimeouts, when non-nil, is an externally-owned counter the executor
	// increments on each active-module timeout (instead of its own private one).
	// A multi-round phase passes one shared counter so the timed-out total in the
	// status line accumulates across the per-round executors it creates.
	ModuleTimeouts *atomic.Int64
	// FirstStatusInterval, when > 0 and shorter than StatusInterval, fires the
	// very first status tick after this duration instead of waiting a full
	// StatusInterval. Useful for long-cadence status (e.g., 2m) where users
	// shouldn't have to stare at silence before the first line appears.
	FirstStatusInterval time.Duration
	// DisableFeedback drops every URL discovered by passive modules instead of
	// re-injecting it into the scan. With this flag the executor scans exactly
	// the items the input source delivers and nothing more — matching the
	// shallow per-request semantics of `vigolium scan-request`. Used by
	// scan-on-receive's default mode to avoid the cascade of new HTTP records
	// that link extractors and redirect followers would otherwise generate.
	DisableFeedback bool
	// TechFilterDisabled bypasses the tech-stack allowlist gate (see modules.TechAware).
	// Set by --no-tech-filter and automatically by --intensity=deep so deep scans
	// run every module regardless of detected stack.
	TechFilterDisabled bool
	// OnTechDetected fires once per (host, tag) when a passive fingerprint
	// module first publishes a stack detection.
	OnTechDetected func(host, tag string)
	// DeepScan mirrors --intensity=deep into ScanContext so modules can unlock
	// heavier probing (e.g. dashboard_exposure's full mount-path sweep).
	DeepScan bool
	// ContentClassByHost seeds the per-host content-class registry from the
	// heuristics root probe (host → modkit.ContentClass string). It is the
	// fallback consulted by the content-class gate when a record's own
	// Content-Type is indeterminate. Honors the same TechFilterDisabled switch.
	ContentClassByHost map[string]string
}

ExecutorConfig configures the Executor behavior.

func DefaultExecutorConfig

func DefaultExecutorConfig() ExecutorConfig

DefaultExecutorConfig returns sensible defaults.

type HookRunner

type HookRunner interface {
	RunPreHooks(req *httpmsg.HttpRequestResponse) (*httpmsg.HttpRequestResponse, error)
	RunPostHooks(result *output.ResultEvent) (*output.ResultEvent, error)
}

HookRunner transforms requests before scanning and filters results after scanning.

type InsertionPointCache

type InsertionPointCache struct {
	// contains filtered or unexported fields
}

InsertionPointCache is a byte-aware LRU of insertion-point sets, keyed by request hash. It bounds both entry count (via the underlying LRU) and total retained bytes, and rejects individual entries above a per-entry ceiling.

A nil *InsertionPointCache is usable: every method degrades to a miss, so a caller with no cache configured needs no branch of its own.

All access is serialized on one mutex. That is deliberate: the byte counter and the LRU must move together or the budget drifts, and the eviction callback updates the counter while the LRU's own lock is held — so it must not attempt to re-acquire this mutex. Because every caller path holds the mutex around the LRU call, the callback runs with exclusive access already and updates the counter directly. Contention is not a concern: lookups are per-request, not per-payload.

func NewInsertionPointCache

func NewInsertionPointCache(entries int) *InsertionPointCache

NewInsertionPointCache creates a byte-aware insertion-point cache holding at most entries items. A non-positive entries value falls back to a safe default.

func (*InsertionPointCache) Add

func (c *InsertionPointCache) Add(key string, points []httpmsg.InsertionPoint)

Add caches points under key unless the set is too large to be worth retaining. Oversized sets are counted as rejections and left uncached — the caller's copy is unaffected either way, so a rejection costs only a later re-derivation.

func (*InsertionPointCache) Get

Get returns the cached insertion points for key.

func (*InsertionPointCache) GetOrCompute

func (c *InsertionPointCache) GetOrCompute(raw []byte, requestID string, includeNested bool) ([]httpmsg.InsertionPoint, error)

GetOrCompute returns the insertion points for requestID, deriving and caching them on a miss.

This owns the cache-key convention (the includeNested variants are separate entries) so no caller has to reproduce it. Both the per-insertion-point scan stage and the module-facing provider go through here; deriving a key inline instead is how the two ends up disagreeing about which variant they stored.

func (*InsertionPointCache) Stats

Stats returns a snapshot of the cache's current size and cumulative counters.

type InsertionPointCacheStats

type InsertionPointCacheStats struct {
	Entries       int
	RetainedBytes int64
	Hits          int64
	Misses        int64
	// Evictions counts only byte-budget evictions. Ordinary count-bound LRU
	// evictions are not counted: they are the cache working as configured, while a
	// byte eviction means the budget — not the entry cap — is the binding limit.
	Evictions int64
	// Rejected counts sets never cached because they exceeded the per-entry ceiling.
	Rejected int64
}

InsertionPointCacheStats is a point-in-time snapshot of cache behaviour, for logging and metrics. Counters are cumulative for the cache's lifetime.

type OASTFlusher

type OASTFlusher interface {
	Flush()
	Close()
}

OASTFlusher is implemented by the OAST service to allow the executor to flush pending interactions after scanning completes.

type PauseController

type PauseController struct {
	// contains filtered or unexported fields
}

PauseController provides cooperative pause/resume for worker goroutines. Workers call WaitIfPaused between items; Pause blocks until all active items finish, then holds future workers until Resume is called.

stateMu serializes the Pause/Resume transitions in full (including the mu Lock/Unlock that spans a pause), so concurrent Pause+Resume — reachable, e.g. a stop-triggered Runner.Close resuming while an API pause request is in flight — can never close a nil channel, unlock an unlocked mutex, or leave pausedCh stale relative to the paused flag.

func NewPauseController

func NewPauseController() *PauseController

NewPauseController creates a new PauseController in the unpaused state.

func (*PauseController) AcquireWorker

func (p *PauseController) AcquireWorker()

AcquireWorker acquires a read lock for the duration of item processing. Call ReleaseWorker when done. Returns false if paused (caller should call WaitIfPaused first).

func (*PauseController) IsPaused

func (p *PauseController) IsPaused() bool

IsPaused returns the current pause state.

func (*PauseController) Pause

func (p *PauseController) Pause()

Pause blocks new workers and waits for in-flight items to finish. Safe to call multiple times (idempotent) and concurrently with Resume.

func (*PauseController) ReleaseWorker

func (p *PauseController) ReleaseWorker()

ReleaseWorker releases the read lock after item processing.

func (*PauseController) Resume

func (p *PauseController) Resume()

Resume unblocks paused workers. Safe to call when not paused and concurrently with Pause.

func (*PauseController) WaitIfPaused

func (p *PauseController) WaitIfPaused(ctx context.Context) bool

WaitIfPaused blocks the caller if the controller is paused. Returns false if the context is cancelled while waiting. Workers should call this between processing items.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL