core

package
v0.5.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 3, 2026 License: MIT Imports: 34 Imported by: 0

Documentation

Overview

Package core contains the scan Executor — the central orchestrator of a native scan. The Executor receives HttpRequestResponse work items, dispatches them to registered active and passive modules through a bounded worker pool, and collects ResultEvent findings. It owns scope filtering, per-host rate limiting, pre/post hooks, per-module timeouts, context-based cancellation, and the post-EOF feedback drain that lets passive modules re-inject discovered requests. Subpackages provide the rate limiter, network dialer, service container, and scan statistics.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func SuggestWorkerCount

func SuggestWorkerCount(moduleCount, maxWorkers int) int

SuggestWorkerCount returns a heuristic worker count for rescans. It scales with the number of active modules but caps at maxWorkers. The floor is 2 to ensure at least minimal parallelism.

Types

type ExecutionReport added in v0.5.2

type ExecutionReport struct {
	// StoppedEarly — the input source was not read to EOF (cancelled, deadline,
	// or a paused scan that was stopped while waiting).
	StoppedEarly bool
	// DrainStalled — the feedback drain was abandoned with workers still in
	// flight and making no progress.
	DrainStalled bool
	// WorkersAbandoned — one or more workers did not exit within the shutdown
	// grace and were leaked.
	WorkersAbandoned bool
	// DeferredFlushSkipped — the end-of-run passive flush was skipped because a
	// worker was abandoned, so deferred findings (secret detection, anomaly
	// ranking) may be incomplete.
	DeferredFlushSkipped bool
	// Unacked counts work items a worker dequeued but deliberately did NOT
	// acknowledge, because the context was already cancelled when it got them.
	// Not acknowledging is what keeps the durable cursor behind them so a later
	// run re-serves them; this number is how many that was.
	Unacked int64
}

ExecutionReport says whether one Execute call processed its whole input, and what it gave up on if not.

Execute returns (bool, error) where the bool is "produced findings" and the error is nil on every one of these paths: a cancelled feed, an abandoned drain, abandoned workers, a skipped deferred flush. Each of those silently reduces coverage, and the caller had no way to learn about any of them — a scan whose workers were abandoned mid-flight reported exactly what a clean one did. This is that missing channel.

func (ExecutionReport) Clean added in v0.5.2

func (r ExecutionReport) Clean() bool

Clean reports whether the run covered its whole input with nothing abandoned.

A convenience for callers and tests that want the one-line question. The runner's own consumer (phaseTracker.noteExecution) deliberately does not use it: each flag maps to a distinct outcome reason code, so it has to read them individually.

type Executor

type Executor struct {
	// contains filtered or unexported fields
}

Executor orchestrates scanning with worker pool.

func NewExecutor

func NewExecutor(
	cfg ExecutorConfig,
	src source.InputSource,
	activeModules []modules.ActiveModule,
	passiveModules []modules.PassiveModule,
) *Executor

NewExecutor creates a new Executor with the given configuration.

func (*Executor) ConsideredModuleCount

func (e *Executor) ConsideredModuleCount() int64

ConsideredModuleCount returns the number of distinct modules whose CanProcess has been evaluated at least once during this scan. Reaches parity with the total enabled-module count once every module has been seen — including those whose CanProcess always rejects the input shape (e.g., POST-only modules in a GET-only scan). Use this for "modules scanned X/Y" status displays.

func (*Executor) Execute

func (e *Executor) Execute(ctx context.Context) (bool, error)

Execute runs the scan. Blocks until all inputs are processed or context is cancelled.

func (*Executor) FeedbackDropped

func (e *Executor) FeedbackDropped() int64

FeedbackDropped returns the number of feedback items dropped due to channel capacity.

func (*Executor) InFlight

func (e *Executor) InFlight() int64

InFlight returns the number of worker goroutines currently processing items. Zero means the executor is quiescent (all workers idle, no items in flight). Exposed for status reporting — callers typically combine this with source-level idle signals to distinguish "still doing work" from "waiting for input."

func (*Executor) ModuleMetrics

func (e *Executor) ModuleMetrics() map[string]stats.ModuleStatsSnapshot

ModuleMetrics returns a point-in-time snapshot of per-module performance metrics.

func (*Executor) Processed

func (e *Executor) Processed() int64

Processed returns the number of items processed by the executor.

func (*Executor) Report added in v0.5.2

func (e *Executor) Report() ExecutionReport

Report returns this executor's coverage self-report. Call it after Execute has returned; before that the fields are still being written.

func (*Executor) ResolveRequestUUID

func (e *Executor) ResolveRequestUUID(requestHash string) string

ResolveRequestUUID resolves a request hash to its database record UUID. Implements modkit.RequestUUIDResolver.

func (*Executor) Responded

func (e *Executor) Responded() int64

func (*Executor) StoreFailed added in v0.5.2

func (e *Executor) StoreFailed() int64

StoreFailed returns the number of input items whose primary record could not be written. Distinct from an item that was never stored because it got no response (Processed() minus Responded()) or because a filter dropped it: this one reached the database and the database refused it.

func (*Executor) Stored added in v0.5.2

func (e *Executor) Stored() int64

Responded returns how many input items produced an HTTP response, which is always <= Processed(). Use this, not Processed(), to report how many targets answered: Processed() counts attempts, including the ones that never reached a server. See the responded field for why the two differ. Stored returns the number of input items whose primary record is in the database when this executor finished with them. Use this, not Processed(), to report how many records a run PERSISTED: Processed() counts attempts, and an attempt that never reached a host, or whose write failed, stores nothing.

func (*Executor) SuppressedFindings

func (e *Executor) SuppressedFindings() int64

SuppressedFindings returns the number of candidate findings dropped by the body-differential safety net because a real payload-vs-baseline difference could not be re-confirmed.

type ExecutorConfig

type ExecutorConfig struct {
	Workers       int
	OnResult      func(*output.ResultEvent)
	OnCandidate   func(*output.ResultEvent)
	OnObservation func(*output.ResultEvent)
	OnTraffic     func(method, url string, statusCode int, contentType string) // Optional: called for each processed item
	// OnAuthWall is called once per item whose redirect chain the transport
	// refused to follow into a login / SSO wall (http.StoppedAtAuthWall). It
	// carries the target that bounced and the wall it bounced to.
	//
	// The executor only reports; the phase decides what to do. That split is
	// deliberate: the wall host is a scan-wide fact (it feeds the fuzz-target
	// exclusion the spidering phase already maintains) and the executor has no
	// business owning scan-wide state, while re-deriving it later from stored
	// records would mean re-parsing every 3xx in the corpus.
	OnAuthWall    func(target, wall string)
	Services      *services.Services
	HTTPRequester *http.Requester
	Repository    *database.Repository    // Optional: database storage
	RecordWriter  *database.RecordWriter  // Optional: batched record writer (preferred over Repository.SaveRecord)
	FindingWriter *database.FindingWriter // Optional: batched finding writer (preferred over Repository.SaveFinding)
	ScanUUID      string
	// ProjectUUID owns every record and finding this executor writes. An empty
	// value is silently coerced to the DEFAULT project by the repository (see
	// database.defaultProjectUUID), so a config carrying a Repository/
	// RecordWriter/FindingWriter MUST set it or its output lands in a project
	// nobody is reading. Enforced by TestExecutorConfigAlwaysSetsProjectUUID.
	ProjectUUID       string
	Hooks             HookRunner           // Optional: pre/post hooks
	ScopeMatcher      *config.ScopeMatcher // Optional: scope filtering
	ScopeOnIngest     bool                 // When true, skip both save and scan for out-of-scope items
	StaticFileMatcher *config.ScopeMatcher // Optional: always-on static file filtering (independent of ScopeMatcher)
	FollowSubdomains  bool                 // When true, subdomain_harvest adds discovered subdomains to scope and feeds them (needs ScopeMatcher + feedback)
	SkipBaseline      bool                 // When true, skip HTTP fetch if response already attached (Phase 3 DB source)
	// RecordRedirectChain persists each followed redirect hop as its own record,
	// chained through parent_uuid, and re-pairs the final response with the
	// request that actually produced it. Off by default: it multiplies row
	// counts on every redirecting URL, and only a host-sweep style phase wants
	// the intermediate hops. See saveRedirectHops.
	RecordRedirectChain bool
	// RecordSource is the http_records.source label this executor stamps. Empty
	// means "scanner", which is what every phase used before the label became
	// configurable — so a phase that says nothing is unchanged.
	RecordSource         string
	OASTProvider         modkit.OASTProvider // Optional: OAST callback URL generator for blind vuln detection
	OASTService          OASTFlusher         // Optional: OAST service to flush after scanning
	PauseCtrl            *PauseController    // Optional: cooperative pause/resume controller
	MaxFindingsPerModule int                 // When > 0, suppress findings after this many per module
	// FindingAdmission optionally shares the per-module cap and the
	// already-admitted identity set with other executors. nil = executor-local,
	// which is right for a phase that runs exactly one Execute. The
	// dynamic-assessment phase passes ONE admission to every feedback round, so
	// the cap spans the phase and a root cause re-found in a later round does not
	// re-fire its callbacks. See FindingAdmission.
	FindingAdmission      *FindingAdmission
	MaxDuration           time.Duration                                                                                                       // When > 0, cancel execution after this duration
	FeedbackDrainTimeout  time.Duration                                                                                                       // Idle timeout for draining feedback after source EOF (default: 100ms)
	FeedbackDrainMaxStall time.Duration                                                                                                       // Hard cap on draining with workers in-flight but making no progress (0 = 2x active module timeout). Guards against a module that ignores cancellation.
	WorkerExitGrace       time.Duration                                                                                                       // Grace for worker goroutines to exit after the item channel is closed (0 = default 60s, capped at FeedbackDrainMaxStall). Kept short and separate from the drain-stall cap so shutdown stays responsive.
	IPCacheSize           int                                                                                                                 // LRU cache size for parsed insertion points (default: 4096)
	IPCache               *InsertionPointCache                                                                                                // Optional: shared IP cache (if nil, a new one is created)
	ParallelPassive       bool                                                                                                                // When true, run passive per-request modules concurrently
	PassiveModuleTimeout  time.Duration                                                                                                       // Timeout per passive module call (default: 5s). 0 uses default.
	ActiveModuleTimeout   time.Duration                                                                                                       // Timeout per active module call (default: 300s). 0 uses default. Modules may raise via TimeoutHinter.
	AdaptiveWorkers       bool                                                                                                                // When true, dynamically scale worker count based on queue depth
	MinWorkers            int                                                                                                                 // Floor for adaptive scaling (default: 2)
	MaxWorkers            int                                                                                                                 // Ceiling for adaptive scaling (default: Workers*4)
	ActiveTaskLimit       int                                                                                                                 // Max concurrent active module tasks across host/request/IP scopes
	OnStatus              func(processed, total, findings, distinctModules, activeCount, passiveCount, timedOut int64, elapsed time.Duration) // Optional: periodic status callback
	StatusInterval        time.Duration                                                                                                       // Interval for OnStatus callback (default: 30s)
	// ModuleTimeouts, when non-nil, is an externally-owned counter the executor
	// increments on each active-module timeout (instead of its own private one).
	// A multi-round phase passes one shared counter so the timed-out total in the
	// status line accumulates across the per-round executors it creates.
	ModuleTimeouts *atomic.Int64
	// FirstStatusInterval, when > 0 and shorter than StatusInterval, fires the
	// very first status tick after this duration instead of waiting a full
	// StatusInterval. Useful for long-cadence status (e.g., 2m) where users
	// shouldn't have to stare at silence before the first line appears.
	FirstStatusInterval time.Duration
	// DisableFeedback drops every URL discovered by passive modules instead of
	// re-injecting it into the scan. With this flag the executor scans exactly
	// the items the input source delivers and nothing more — matching the
	// shallow per-request semantics of `vigolium scan-request`. Used by
	// scan-on-receive's default mode to avoid the cascade of new HTTP records
	// that link extractors and redirect followers would otherwise generate.
	DisableFeedback bool
	// TechFilterDisabled bypasses the tech-stack allowlist gate (see modules.TechAware).
	// Set by --no-tech-filter and automatically by --intensity=deep so deep scans
	// run every module regardless of detected stack.
	TechFilterDisabled bool
	// OnTechDetected fires once per (host, tag) when a passive fingerprint
	// module first publishes a stack detection.
	OnTechDetected func(host, tag string)
	// DeepScan mirrors --intensity=deep into ScanContext so modules can unlock
	// heavier probing (e.g. dashboard_exposure's full mount-path sweep).
	DeepScan bool
	// ContentClassByHost seeds the per-host content-class registry from the
	// heuristics root probe (host → modkit.ContentClass string). It is the
	// fallback consulted by the content-class gate when a record's own
	// Content-Type is indeterminate. Honors the same TechFilterDisabled switch.
	ContentClassByHost map[string]string
}

ExecutorConfig configures the Executor behavior.

func DefaultExecutorConfig

func DefaultExecutorConfig() ExecutorConfig

DefaultExecutorConfig returns sensible defaults.

type FindingAdmission added in v0.5.2

type FindingAdmission struct {
	// contains filtered or unexported fields
}

FindingAdmission is the state behind the per-module finding cap: how many findings each module has had admitted, and which root-cause identities have already been admitted.

It is a separate type so it can OUTLIVE one Executor. The dynamic-assessment phase builds a fresh executor per feedback round, so this state used to reset between rounds: a cap documented as holding "for the remainder of the scan" actually restarted every round, and a root cause re-found in round 2 re-fired every callback and notification for a finding already reported in round 1. Sharing one admission across the phase's rounds is what makes the documented contract true (triage C7: per dynamic-assessment phase, shared across rounds; other phases are uncapped).

The zero value is ready to use, so an Executor built without a shared one gets its own by construction and a literal &Executor{cfg: …} in a test still works. Safe for concurrent use: both fields are sync.Maps.

func NewFindingAdmission added in v0.5.2

func NewFindingAdmission() *FindingAdmission

NewFindingAdmission returns admission state a caller can share across several executors — one per dynamic-assessment phase, covering all of its rounds.

type HookRunner

type HookRunner interface {
	RunPreHooks(req *httpmsg.HttpRequestResponse) (*httpmsg.HttpRequestResponse, error)
	RunPostHooks(result *output.ResultEvent) (*output.ResultEvent, error)
}

HookRunner transforms requests before scanning and filters results after scanning.

type InsertionPointCache

type InsertionPointCache struct {
	// contains filtered or unexported fields
}

InsertionPointCache is a byte-aware LRU of insertion-point sets, keyed by request hash. It bounds both entry count (via the underlying LRU) and total retained bytes, and rejects individual entries above a per-entry ceiling.

A nil *InsertionPointCache is usable: every method degrades to a miss, so a caller with no cache configured needs no branch of its own.

All access is serialized on one mutex. That is deliberate: the byte counter and the LRU must move together or the budget drifts, and the eviction callback updates the counter while the LRU's own lock is held — so it must not attempt to re-acquire this mutex. Because every caller path holds the mutex around the LRU call, the callback runs with exclusive access already and updates the counter directly. Contention is not a concern: lookups are per-request, not per-payload.

func NewInsertionPointCache

func NewInsertionPointCache(entries int) *InsertionPointCache

NewInsertionPointCache creates a byte-aware insertion-point cache holding at most entries items. A non-positive entries value falls back to a safe default.

func (*InsertionPointCache) Add

func (c *InsertionPointCache) Add(key string, points []httpmsg.InsertionPoint)

Add caches points under key unless the set is too large to be worth retaining. Oversized sets are counted as rejections and left uncached — the caller's copy is unaffected either way, so a rejection costs only a later re-derivation.

func (*InsertionPointCache) Get

Get returns the cached insertion points for key.

func (*InsertionPointCache) GetOrCompute

func (c *InsertionPointCache) GetOrCompute(raw []byte, requestID string, includeNested bool) ([]httpmsg.InsertionPoint, error)

GetOrCompute returns the insertion points for requestID, deriving and caching them on a miss.

This owns the cache-key convention (the includeNested variants are separate entries) so no caller has to reproduce it. Both the per-insertion-point scan stage and the module-facing provider go through here; deriving a key inline instead is how the two ends up disagreeing about which variant they stored.

func (*InsertionPointCache) Stats

Stats returns a snapshot of the cache's current size and cumulative counters.

type InsertionPointCacheStats

type InsertionPointCacheStats struct {
	Entries       int
	RetainedBytes int64
	Hits          int64
	Misses        int64
	// Evictions counts only byte-budget evictions. Ordinary count-bound LRU
	// evictions are not counted: they are the cache working as configured, while a
	// byte eviction means the budget — not the entry cap — is the binding limit.
	Evictions int64
	// Rejected counts sets never cached because they exceeded the per-entry ceiling.
	Rejected int64
}

InsertionPointCacheStats is a point-in-time snapshot of cache behaviour, for logging and metrics. Counters are cumulative for the cache's lifetime.

type OASTFlusher

type OASTFlusher interface {
	Flush()
	Close()
}

OASTFlusher is implemented by the OAST service to allow the executor to flush pending interactions after scanning completes.

type PauseController

type PauseController struct {
	// contains filtered or unexported fields
}

PauseController provides cooperative pause/resume for worker goroutines. Workers call WaitIfPaused between items; Pause blocks until all active items finish, then holds future workers until Resume is called.

stateMu serializes the Pause/Resume transitions in full (including the mu Lock/Unlock that spans a pause), so concurrent Pause+Resume — reachable, e.g. a stop-triggered Runner.Close resuming while an API pause request is in flight — can never close a nil channel, unlock an unlocked mutex, or leave pausedCh stale relative to the paused flag.

func NewPauseController

func NewPauseController() *PauseController

NewPauseController creates a new PauseController in the unpaused state.

func (*PauseController) AcquireWorker

func (p *PauseController) AcquireWorker()

AcquireWorker acquires a read lock for the duration of item processing. Call ReleaseWorker when done. Returns false if paused (caller should call WaitIfPaused first).

func (*PauseController) IsPaused

func (p *PauseController) IsPaused() bool

IsPaused returns the current pause state.

func (*PauseController) Pause

func (p *PauseController) Pause()

Pause blocks new workers and waits for in-flight items to finish. Safe to call multiple times (idempotent) and concurrently with Resume.

func (*PauseController) ReleaseWorker

func (p *PauseController) ReleaseWorker()

ReleaseWorker releases the read lock after item processing.

func (*PauseController) Resume

func (p *PauseController) Resume()

Resume unblocks paused workers. Safe to call when not paused and concurrently with Pause.

func (*PauseController) WaitIfPaused

func (p *PauseController) WaitIfPaused(ctx context.Context) bool

WaitIfPaused blocks the caller if the controller is paused. Returns false if the context is cancelled while waiting. Workers should call this between processing items.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL