database

package
v0.5.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 10, 2026 License: MIT Imports: 50 Imported by: 0

Documentation

Overview

Package database provides the repository layer over SQLite (default) or PostgreSQL via the Bun ORM. It persists scans, HTTP records, findings, scopes, and OAST interactions — all scoped to a project_uuid for multi-tenancy. It supports batched HTTP-record ingestion (SaveRecordBatch, RecordWriter) and per-source deduplication (DeduplicateRecordsBySource).

Index

Constants

View Source 
const (
	RecordSourceIngestServer = "ingest-server" // POST /api/ingest-http
	RecordSourceIngestProxy  = "ingest-proxy"  // transparent ingest proxy capture
	RecordSourceIngestCLI    = "ingest-cli"    // `vigolium ingest ...` command
	RecordSourceScanner      = "scanner"       // executor feedback re-injection
	RecordSourceProbe        = "probe"         // `vigolium run probe` host sweep (one request per target, no fuzzing)
	RecordSourceHarvest      = "harvest"       // external-harvest archive URL; request-only until harvest-probe completes it
	RecordSourceFinding      = "finding"       // request/response attached to a finding
	RecordSourceBurp         = "burp"          // Burp Suite, via the loopback bridge or a push to /api/ingest-http
	RecordSourceCaido        = "caido"         // Caido, via the loopback bridge or a push to /api/ingest-http
	RecordSourceParamMining  = "param-mining"  // request carrying a hidden parameter the param-mining phase confirmed
)

Canonical values for http_records.source. Kept here (rather than inline string literals at call sites) so the scan-on-receive filter and the ingest writers agree on a single set of labels.

View Source 
const (
	ModuleTypeActive         = "active"
	ModuleTypePassive        = "passive"
	ModuleTypeNuclei         = "nuclei"
	ModuleTypeAgent          = "agent"
	ModuleTypeOAST           = "oast"
	ModuleTypeExtension      = "extension"
	ModuleTypeKnownIssueScan = "known-issue-scan"
	ModuleTypeWhitebox       = "whitebox"
)

ModuleType constants classify the kind of module that created a finding.

View Source 
const (
	FindingSourceDynamicAssessment = "dynamic-assessment"
	FindingSourceKnownIssueScan    = "known-issue-scan"
	FindingSourceAgent             = "agent"
	FindingSourceOAST              = "oast"
	FindingSourceExtension         = "extension"
	FindingSourceAudit             = "audit"
	FindingSourceImport            = "import"
	FindingSourceSpidering         = "spidering"
)

FindingSource constants identify which phase/component produced a finding.

View Source 
const (
	RecordKindFinding     = "finding"
	RecordKindCandidate   = "candidate"
	RecordKindObservation = "observation"
)

Record-kind constants distinguish reportable vulnerabilities from retained reconnaissance and unconfirmed hypotheses stored in the same table.

View Source 
const (
	StatusDraft         = "draft"
	StatusTriaged       = "triaged"
	StatusConfirmed     = "confirmed"
	StatusFalsePositive = "false_positive"
	StatusAcceptedRisk  = "accepted_risk"
	StatusFixed         = "fixed"
)

Status constants represent the lifecycle state of a Finding.

Lifecycle: 

draft ──triage──▶ triaged ──verify──▶ confirmed ──fix──▶ fixed
             ╲──▶ false_positive
             ╲──▶ accepted_risk

confirmed means a human or a re-verification reproduced the finding; it is a status, not a verification result (that is Finding.Verification, kept apart so re-running `finding verify` never silently rewrites a triage verdict).

Trust model: deterministic engines (native scan, SAST tools) produce findings as `triaged`. LLM/agent-driven and externally imported findings land as `draft` and require triage to be promoted.

View Source 
const (
	SeverityCritical = "critical"
	SeverityHigh     = "high"
	SeverityMedium   = "medium"
	SeverityLow      = "low"
	SeverityInfo     = "info"
	SeveritySuspect  = "suspect"
)

Severity constants represent the canonical severity strings stored on a Finding. They match the lowercase keys used by `vigolium finding --severity` and the severity counters on Scan.

View Source 
const (
	SourceTypeLocal  = "local"
	SourceTypeGitURL = "git-url"
	SourceTypeGCS    = "gcs"
)

SourceType constants classify how source code was provided.

View Source 
const (
	VerdictConfirmed     = "confirmed"
	VerdictNotReproduced = "not_reproduced"
	VerdictInconclusive  = "inconclusive"
	VerdictUnsupported   = "unsupported"
)

Verification verdicts recorded in FindingVerification.Verdict.

View Source 
const (
	PersistenceWriterRecords  = "records"
	PersistenceWriterFindings = "findings"
	PersistenceWriterMixed    = "mixed"
)

Writer names used by the two writers and by a merged outcome.

View Source 
const (
	ReasonPhaseDeadline        = "phase_deadline"         // the phase's own max_duration / ceiling fired
	ReasonScanBudget           = "scan_budget"            // --scanning-max-duration fired
	ReasonCancelled            = "cancelled"              // Ctrl-C / caller cancellation
	ReasonError                = "error"                  // the phase returned an error
	ReasonNoModules            = "no_modules"             // nothing to execute
	ReasonDrainStalled         = "drain_stalled"          // feedback drain abandoned with workers in flight
	ReasonWorkersAbandoned     = "workers_abandoned"      // workers did not exit within the shutdown grace
	ReasonDeferredFlushSkipped = "deferred_flush_skipped" // passive flush skipped after abandoning workers
	ReasonProducerAbandoned    = "producer_abandoned"     // an input producer did not exit in time
	ReasonTargetsFailed        = "targets_failed"         // one or more targets errored
	ReasonTargetsSkipped       = "targets_skipped"        // one or more targets were never attempted
	ReasonPersistence          = "persistence_incomplete" // a writer could not account for every write
	ReasonRoundError           = "round_error"            // a dynamic-assessment feedback round failed
	ReasonCheckpointFailed     = "checkpoint_failed"      // the durable scan cursor could not be advanced
	ReasonInputIncomplete      = "input_incomplete"       // the executor stopped before its source was exhausted
	ReasonAuthUnavailable      = "auth_unavailable"       // configured authentication could not be applied
	ReasonLoginBudget          = "login_budget"           // the total login budget elapsed before every session hydrated
	ReasonTargetsOutOfScope    = "targets_out_of_scope"   // one or more CLI targets were rejected by scope
	ReasonTargetsUnreachable   = "targets_unreachable"    // one or more admitted targets got no HTTP answer
	ReasonOASTUnavailable      = "oast_unavailable"       // OAST registration failed; out-of-band checks did not run (non-stopping)
	ReasonRequestBudget        = "request_budget"         // the scan-wide request budget (--max-requests) ran out
	ReasonAuthLost             = "auth_lost"              // the session expired mid-scan and re-login did not restore it
	ReasonPhaseError           = "phase_error"            // a phase failed and the scan carried on without it
	ReasonExtensionsAllFailed  = "extensions_all_failed"  // every requested extension failed to load or run
	ReasonAbandoned            = "abandoned"              // the scan's process died without finalizing; set by `log ls --reap`

	// Stop reasons: the scan did nothing useful and ends with exit 1.
	ReasonNoTargetsInScope   = "no_targets_in_scope"  // scope admitted zero targets
	ReasonNoTargetsReachable = "no_targets_reachable" // zero admitted targets answered

	LimitTargetBudget            = "target_budget"              // informational: a target used its whole budget
	LimitHostRequestBudget       = "host_request_budget"        // a host used its whole request budget
	LimitJSRequestCap            = "js_request_cap"             // JS analysis hit its per-target request cap
	LimitJSInputOverHardLimit    = "js_input_over_hard_limit"   // a JS file exceeded the hard size limit and was not parsed
	LimitJSTemplatePersistCap    = "js_template_persist_cap"    // JS-derived request templates hit the persist cap
	LimitCDXMaxRows              = "cdx_max_rows"               // an archive (CDX) query hit its row cap
	LimitSitemapCap              = "sitemap_cap"                // sitemap expansion hit its URL cap
	LimitResponsesWhileLoggedOut = "responses_while_logged_out" // responses observed while the session was logged out
)

Reason and limit codes. These are PERSISTED on the scan row and EMITTED on the event stream, so they are a compatibility surface: add codes, never rename or repurpose one. A reason says why a phase did not cover everything; a limit says a configured bound was reached, which is designed behaviour rather than a defect, and is reported separately so a consumer can tell them apart.

View Source 
const (
	CompletenessComplete = "complete"
	CompletenessPartial  = "partial"
)

Completeness values. Empty means unknown and must never be read as complete.

View Source 
const (
	CandidateStatusProposed      = "proposed"
	CandidateStatusVerifying     = "verifying"
	CandidateStatusConfirmed     = "confirmed"
	CandidateStatusRejected      = "rejected"
	CandidateStatusNeedsEvidence = "needs_evidence"
)

Candidate lifecycle status values.

View Source 
const (
	SectionStatusRunning     = "running"
	SectionStatusCompleted   = "completed"
	SectionStatusInterrupted = "interrupted"
)

Section lifecycle status values.

View Source 
const (
	AgenticScanStatusTimeout         = "timeout"
	AgenticScanStatusBudgetExhausted = "budget_exhausted"
	AgenticScanStatusCancelled       = "cancelled"
	AgenticScanStatusFailed          = "failed"
)

Agentic-run statuses beyond the completed set. "timeout" and "budget_exhausted" are runs that ran but stopped at a limit (the CLI exits 5 for them); "cancelled" is an interrupt; "failed" an error.

View Source 
const (
	// DiffNew: observed by the head scan, not by the base scan.
	DiffNew = "new"
	// DiffUnchanged: observed by both.
	DiffUnchanged = "unchanged"
	// DiffFixed: observed by the base scan and not by a head scan that covered
	// at least what the base did, to completion (Comparable).
	DiffFixed = "fixed"
	// DiffNotReobserved: observed by the base scan and not by a head scan that
	// cannot vouch for its absence (not Comparable). Not evidence of a fix.
	DiffNotReobserved = "not_reobserved"
)

Diff classes, one per finding in a scan diff.

View Source 
const (
	ComparabilityHeadNotFinished       = "head_not_finished"         // head is still running, paused, failed or cancelled
	ComparabilityHeadPartial           = "head_partial"              // head finished with completeness partial
	ComparabilityHeadCompletenessUnset = "head_completeness_unknown" // head row carries no completeness verdict
	ComparabilityTargetsUnknown        = "targets_unknown"           // a scan row has no target list to compare
	ComparabilityTargetsNotCovered     = "head_targets_not_superset" // head did not cover every base target
	ComparabilityPhasesUnknown         = "phases_unknown"            // a scan row has no phase outcomes to compare
	ComparabilityPhasesNotCovered      = "head_phases_not_superset"  // a phase that ran in base did not complete in head
)

Reasons a head scan is not comparable to its base, so absence from it is reported as not_reobserved rather than fixed.

View Source 
const (
	DiffHistoryComplete = "complete"
	// DiffHistoryPartial: a scan in the diff started before the store recorded
	// every observation (schema_meta.observation_history_since), so a finding
	// that scan re-observed may be missing from its set.
	DiffHistoryPartial = "partial"
)

Diff history labels.

View Source 
const (
	ScanRefLatest   = "latest"
	ScanRefPrevious = "previous"
)

Scan reference keywords accepted by ResolveScanRef.

View Source 
const (
	ScanNameScanURL     = "scan-url"
	ScanNameScanRequest = "scan-request"
	ScanModeSingle      = "single"
)

Scan-row names and modes for single-request scans: the scan-url / scan-request direct path in the CLI and the REST /api/scan-url and /api/scan-request handlers. One spelling each, so `db ls --table scans` groups them the same way whichever surface started the scan.

View Source 
const AnalysisArtifactKindSourceMapOriginal = sourcemap.ArtifactKindOriginal

AnalysisArtifactKindSourceMapOriginal labels an original source file recovered from a source map's sourcesContent. Aliased from the sourcemap package, which owns the name for the writers on the other side of this package boundary.

View Source 
const (

	// DefaultMaxEndpoints bounds the distinct endpoints one aggregate holds.
	// Records that would open a new endpoint past it are counted as unbucketed.
	DefaultMaxEndpoints = 100000
)
View Source 
const DefaultMaxParamShapeSamples = 3

DefaultMaxParamShapeSamples is the per-shape representative cap used when param-shape coalescing is enabled without an explicit override.

View Source 
const (

	// DefaultMaxParams bounds the distinct parameters one aggregate holds.
	DefaultMaxParams = 200000
)
View Source 
const DefaultProjectUUID = "00000000-0000-0000-defa-c01001000001"

DefaultProjectUUID is the UUID for the default project created during init. Lowercased for consistency with uuid.New().String() output and byte-wise SQL comparisons.

View Source 
const DefaultUserUUID = "00000000-0000-0000-0000-000000000001"

DefaultUserUUID is the well-known UUID for the default local user created during init.

View Source 
const EvidenceSeparator = output.EvidenceSeparator

EvidenceSeparator is the delimiter between request and response inside an AdditionalEvidence entry. Re-exported from pkg/output (the canonical owner) so the storage-layer dedup/merge path splits evidence the same way modules emit it.

View Source 
const JSFactsArtifactKind = "jstangle-facts"

JSFactsArtifactKind is the analysis artifact discovery stores per JS asset with the asset's non-request jstangle facts (routes, GraphQL operations, WebSocket/SSE endpoints, asset references, path literals, request hints).

View Source 
const MinFindingHashPrefix = 12

MinFindingHashPrefix is the shortest finding_hash prefix a status write accepts. finding_hash is a 40-char hex digest and is stable across databases (unlike the autoincrement id), so it is the identity an agent should carry; twelve characters keep a typo from matching by accident while staying short enough to copy reliably.

View Source 
const OutcomeMessageMax = 512

OutcomeMessageMax bounds the stored first-error text. A phase error can carry a wrapped chain with a full response body in it, and the scan row is read by every listing.

View Source 
const ParamMiningRemarkPrefix = "param-mining:"

ParamMiningRemarkPrefix starts the remark a param-mining record carries per hidden parameter: "param-mining:name=<n>;signal=<s>".

View Source 
const (

	// SQLChunkSize is how many ids or uuids to put in one IN() list. It keeps the
	// generated SQL to a few hundred KB while staying under maxSQLParams with
	// room for the handful of other values a query carries (project scoping, a
	// date range, a LIMIT) — so one constant is correct on both kinds of path.
	SQLChunkSize = 8000
)

Chunking policy for the id/uuid IN() lists this package builds.

The reason is NOT the bound-parameter ceiling, and assuming it was is how the wrong chunk size got picked before. Two facts, both measured against the dependencies in go.mod rather than inferred:

  • Through database/sql and the modernc driver directly, a statement binds at most 32766 parameters; 32767 fails with "too many SQL variables". That is the ceiling maxSQLParams records, and it applies to the raw-SQL paths (merge's pinned connection, the scratch-database probes).
  • Through bun — which is every query builder call in this package — `?` and bun.List do not produce bound parameters at all. The formatter interpolates the values as SQL literals, so a 100k-element IN() list executes happily.

So an unchunked list does not fail: it silently becomes a multi-megabyte SQL string, built and escaped element by element in Go and then parsed by SQLite, for a statement whose result is the same either way. Chunking bounds that string. It also bounds the raw-SQL paths below the hard ceiling, so a caller does not have to know which kind of path it is on.

If a future bun upgrade switches to real parameter binding, the ceiling starts applying to these calls too — TestBunInterpolatesRatherThanBinds is the test that notices.

View Source 
const TargetIssueCap = 20

TargetIssueCap bounds each TargetTally issue list. The tally is stored on the scan row and read by every listing; the counts stay exact past the cap, only the examples are clipped.

Variables

View Source 
var (
	// ErrTableNotFound indicates the requested table is unknown to the generic API.
	ErrTableNotFound = errors.New("table not found")
	// ErrInvalidColumn indicates a supplied column does not exist on the target table.
	ErrInvalidColumn = errors.New("invalid column")
	// ErrNoValidFields indicates no usable fields were supplied for an insert/update.
	ErrNoValidFields = errors.New("no valid fields provided")
	// ErrCompositePKUnsupported indicates a single-value PK operation was attempted
	// on a table whose primary key spans multiple columns.
	ErrCompositePKUnsupported = errors.New("single-value operation not supported for composite primary key table")
	// ErrImmutablePrimaryKey indicates an attempt to update a primary-key column.
	ErrImmutablePrimaryKey = errors.New("cannot update primary key column")
)

Sentinel errors returned by the generic table API. Callers (notably the REST DB handlers) discriminate on these with errors.Is to map failures to HTTP status codes, rather than matching on error-message substrings. They classify client-fixable input problems (unknown table/column, empty field set, single-value ops on composite-PK tables) distinctly from internal failures.

View Source 
var AllowedCleanTables = map[string]cleanableTable{
	"http_records":             {SQLName: "http_records", CascadeFirst: []string{"finding_records"}},
	"findings":                 {SQLName: "findings", CascadeFirst: findingChildTables},
	"finding_records":          {SQLName: "finding_records"},
	"finding_observations":     {SQLName: "finding_observations"},
	"finding_status_events":    {SQLName: "finding_status_events"},
	"oast_payloads":            {SQLName: "oast_payloads"},
	"scans":                    {SQLName: "scans"},
	"agentic_scans":            {SQLName: "agentic_scans", CascadeFirst: []string{"agent_sections", "agent_finding_candidates"}},
	"agent_sections":           {SQLName: "agent_sections"},
	"agent_finding_candidates": {SQLName: "agent_finding_candidates"},
	"oast_interactions":        {SQLName: "oast_interactions"},
	"scan_logs":                {SQLName: "scan_logs"},
	"authentication_hostnames": {SQLName: "authentication_hostnames"},
	"scopes":                   {SQLName: "scopes"},
}

AllowedCleanTables maps user-facing table names to their SQL table name and any dependent tables that must be cleaned first.

View Source 
var BridgeRecordSources = []string{
	RecordSourceBurp,
	RecordSourceCaido,
}

BridgeRecordSources lists the http_records.source values a loopback bridge listener can produce. Which one a given record gets is decided by the implementation identity the listener reports (see pkg/burpbridge), not by the flag spelling the user typed — --burp-bridge-url and --caido-bridge-url are one flag pointing at one protocol.

View Source 
var CompletedAgenticScanStatuses = []string{
	"completed",
	"completed_with_warnings",
	"completed_with_errors",
}

CompletedAgenticScanStatuses are the terminal statuses of a run that ran to the end: cleanly, degraded, or with some drivers failed. Code that means "finished successfully" should match this set, not the literal "completed", or degraded runs silently drop out.

View Source 
var DefaultRetryPolicy = RetryPolicy{
	MaxAttempts: 12,
	BaseDelay:   50 * time.Millisecond,
	MaxDelay:    2 * time.Second,
}

DefaultRetryPolicy is the budget the merge path has always used: 12 attempts, 50ms doubling to 2s, about 14s in total. Long enough to ride out another process's write transaction, short enough that a wedged store surfaces.

View Source 
var DiffChanges = []string{DiffNew, DiffFixed, DiffNotReobserved, DiffUnchanged}

DiffChanges lists the classes in report order.

View Source 
var DismissedFindingStatuses = []string{StatusFalsePositive, StatusAcceptedRisk}

DismissedFindingStatuses are the verdicts that take a finding out of the answer to "what is still wrong": an operator or agent has ruled it not a bug (false_positive) or decided to live with it (accepted_risk). The --fail-on gate, `scan diff` and `finding --new-since` leave these out by default; reports keep them, labelled.

View Source 
var EndpointSorts = []string{"records", "findings", "template", "last_seen"}

EndpointSorts are the accepted EndpointOptions.Sort values.

View Source 
var ErrCompactUnsupported = errors.New("db compact is only supported for SQLite")

ErrCompactUnsupported reports a database driver compaction does not handle.

View Source 
var ErrDatabaseLocked = errors.New("database is locked by another writer")

ErrDatabaseLocked is the sentinel every exhausted busy retry unwraps to.

View Source 
var ErrRecordWriterClosed = errors.New("record writer is closed")

ErrRecordWriterClosed is returned when writes are attempted after shutdown starts.

View Source 
var ErrScanNotFound = errors.New("scan not found")

ErrScanNotFound is returned when a scan reference matches no scan in scope.

View Source 
var ErrScanProjectMismatch = errors.New("scan UUID exists under a different project")

ErrScanProjectMismatch is returned by CreateScan / CreateAgenticScan when the caller pins a UUID that already exists under a different project. This guards against cross-project record corruption when remote nodes sync via --scan-uuid.

View Source 
var ErrSchemaNotInitialized = errSchemaNotInitialized

ErrSchemaNotInitialized marks a store with no vigolium tables at all: a fresh file rather than a stale one. CheckSchema returns it so the opener can decide between initializing (its own default store) and refusing (a pinned path).

View Source 
var ErrSchemaOutdated = errors.New("database schema is older than this vigolium")

ErrSchemaOutdated marks a database written by an older vigolium that this handle cannot migrate because it was opened read-only. It is a distinct error because it is FATAL to the command — every query that follows would fail on a missing column — whereas a migration failure on a writable handle is best-effort and worth only a warning.

A read-intent open reports it as *SchemaOutdatedError (remedy: `vigolium db migrate`); a --read-only evidence open wraps it with that mode's remedy.

View Source 
var ErrSchemaTooNew = errors.New("database schema is newer than this vigolium")

ErrSchemaTooNew is the sentinel every *SchemaTooNewError unwraps to.

View Source 
var EvidenceRecordSources = []string{RecordKindFinding, RecordKindCandidate, RecordKindObservation}

EvidenceRecordSources are the http_records.source values the executor stamps on a finding's own evidence row (the record kind of the result that saved it), as opposed to a record that was scanned. `finding verify` uses the difference to tell a finding's baseline from its payload request on rows written before findings.origin_record_uuid existed.

View Source 
var FindingStatuses = []string{
	StatusDraft, StatusTriaged, StatusConfirmed, StatusFalsePositive, StatusAcceptedRisk, StatusFixed,
}

FindingStatuses is the closed finding-status vocabulary, in lifecycle order. IsValidFindingStatus, the REST validation message and any closed-enum consumer read this list rather than keeping their own.

View Source 
var HostFactsFieldNames = []string{"a", "aaaa", "cname", "tls"}

HostFactsFieldNames are the JSON keys AsMap can produce, for the --fields allowlist. Derived from the struct so a new fact reaches the allowlist with its emitter.

View Source 
var IngestRecordSources = []string{
	RecordSourceIngestServer,
	RecordSourceIngestProxy,
	RecordSourceIngestCLI,
	RecordSourceBurp,
	RecordSourceCaido,
}

IngestRecordSources lists the http_records.source values that represent user-ingested traffic. Scan-on-receive filters the DBInputSource by this list so it only processes traffic the user actually ingested — excluding scanner-produced artefacts that would otherwise cause fan-out.

The bridge vendors belong here for the same reason ingest-server does: a record that arrived from Burp or Caido is traffic the user deliberately handed over, whether it was pulled in by `traffic --save-to-vigolium-db` / `import -B` or pushed to /api/ingest-http by the plugin. Omitting them is silent — the rows land in the database and scan-on-receive simply never polls them, with no error to explain why nothing was scanned.

View Source 
var OutcomeCodes = []string{
	ReasonPhaseDeadline,
	ReasonScanBudget,
	ReasonCancelled,
	ReasonError,
	ReasonNoModules,
	ReasonDrainStalled,
	ReasonWorkersAbandoned,
	ReasonDeferredFlushSkipped,
	ReasonProducerAbandoned,
	ReasonTargetsFailed,
	ReasonTargetsSkipped,
	ReasonPersistence,
	ReasonRoundError,
	ReasonCheckpointFailed,
	ReasonInputIncomplete,
	ReasonAuthUnavailable,
	ReasonLoginBudget,
	ReasonTargetsOutOfScope,
	ReasonTargetsUnreachable,
	ReasonOASTUnavailable,
	ReasonRequestBudget,
	ReasonAuthLost,
	ReasonPhaseError,
	ReasonExtensionsAllFailed,
	ReasonAbandoned,
	ReasonNoTargetsInScope,
	ReasonNoTargetsReachable,
	LimitTargetBudget,
	LimitHostRequestBudget,
	LimitJSRequestCap,
	LimitJSInputOverHardLimit,
	LimitJSTemplatePersistCap,
	LimitCDXMaxRows,
	LimitSitemapCap,
	LimitResponsesWhileLoggedOut,
}

OutcomeCodes lists every reason and limit code exactly once. It exists so a consumer can enumerate the vocabulary, and so a drift guard in the tests can assert that a newly added constant was not forgotten here.

View Source 
var TerminalAgenticScanStatuses = []string{
	"completed",
	"completed_with_errors",
	"completed_with_warnings",
	"failed",
	"cancelled",
	"timeout",
	"budget_exhausted",
	"error",
}

TerminalAgenticScanStatuses is the authoritative set of agentic_scans.status values meaning the run has finished. Two subsystems must agree on it — the SSE log follower, which tails runtime.log until the run is terminal, and the retention sweep below, which only deletes finished runs — and a status present in one and absent from the other is either a stream that never ends or a row that is never cleaned. Both were true of "completed_with_errors", what a multi-driver audit writes when one leg fails: following its log spun until the two-hour cap and its row outlived every retention window. Add new terminal statuses here, not at either call site.

View Source 
var VerificationVerdicts = []string{VerdictConfirmed, VerdictNotReproduced, VerdictInconclusive, VerdictUnsupported}

VerificationVerdicts is the closed `finding verify` verdict vocabulary.

Functions

func AgenticRunStatus added in v0.5.3 

func AgenticRunStatus(e AgenticRunEnd) string

AgenticRunStatus is the agentic_scans.status for a finished run. It is the one place that decides it, for the CLI finalizers and the swarm runner alike, so a timed-out run is "timeout" whichever path wrote it (it used to be "failed" in the database while the JSON said "timed_out").

func AllTablesDeleteOrder 

func AllTablesDeleteOrder() []string

AllTablesDeleteOrder returns the ordered list of cleanable table names.

func AuthenticationHostnameToSession 

func AuthenticationHostnameToSession(sh *AuthenticationHostname) *authentication.Session

AuthenticationHostnameToSession converts a DB AuthenticationHostname row to a native authentication.Session.

A row commonly carries BOTH a login flow and the headers that flow produced last time (SessionToAuthenticationHostname writes both, and the stored headers are what `vigolium session` shows the operator). A Session may name only ONE credential source, though — Session.Validate rejects more, and Session.IsHydrated treats stored headers as "already logged in" — so exactly one is restored, in this order: 

login flow (login_url) → raw login request → static headers

A row with a login flow therefore restores with NIL headers, which is what makes the login RE-RUN: the stored token is from an earlier scan and is very likely expired, and the scan that reused it spent its whole run getting 401s. Carrying both instead made NewManager reject the session outright and failed session initialization for the entire scan, so a DB-sourced login session never worked at all.

Note: the DB schema does not carry the type/token_path shorthand fields — only the expanded ExtractRules. SessionToAuthenticationHostname normalizes shorthand into explicit rules on write so they round-trip correctly.

func AuthenticationHostnamesToSessionConfig 

func AuthenticationHostnamesToSessionConfig(rows []*AuthenticationHostname) *authentication.SessionConfig

AuthenticationHostnamesToSessionConfig converts a slice of DB rows (typically for one hostname) into a authentication.SessionConfig with ordered sessions.

func CachedDNS 

func CachedDNS(hostname string) (a, aaaa, cname []string)

CachedDNS returns the full DNS answer this PROCESS resolved for hostname, or nothing when it never looked it up (or the entry has aged out).

Deliberately not persisted. `ip` — one address per record — is the only DNS the schema stores. A host sweep writes many records per host (a redirect chain alone is three), and a corpus of thousands of records across a few hundred hosts would otherwise store the same A/AAAA/CNAME arrays thousands of times, inflating the database for data that is identical on every row and stale the moment the TTL expires. Resolving is cheap and repeatable; storing it is not.

So this is a read-time attachment: the run that did the resolving reports the full answer inline in its output, and reading the same database back later in a fresh process correctly shows only `ip`. Expiry is deliberately NOT checked — this reports what was resolved for the records being emitted, and a sweep long enough to cross the TTL should still describe the rows it wrote.

func CompletedAgenticScanStatus added in v0.5.1 

func CompletedAgenticScanStatus(degraded bool) string

CompletedAgenticScanStatus is the status of a run that finished without an error. A degraded run (audit failed to start, discovery failed, the model never called a tool...) is "completed_with_warnings", so it can be told apart from a clean pass. Every writer - server handlers, the CLI finalizers, the swarm runner - goes through here so one outcome gets one status.

func CountFindingsByAgenticScan 

func CountFindingsByAgenticScan(ctx context.Context, db *DB, agenticScanUUID string) (map[string]int64, error)

CountFindingsByAgenticScan returns finding counts grouped by severity for one agentic-scan run. Severity strings are lowercased/trimmed so callers get a canonical key set regardless of how the finding was inserted. Empty agenticScanUUID returns an empty map without querying.

func CountFindingsByAgenticScans 

func CountFindingsByAgenticScans(ctx context.Context, db *DB, agenticScanUUIDs []string) (map[string]int64, error)

CountFindingsByAgenticScans is like CountFindingsByAgenticScan but counts across several agentic-scan UUIDs at once (a parent run plus its driver / sub-run children). Returns severity→count with lowercase, trimmed keys.

func CountFindingsByConfidence 

func CountFindingsByConfidence(ctx context.Context, db *DB, projectUUID string) (map[string]int64, error)

CountFindingsByConfidence returns finding counts grouped by confidence.

func CountFindingsByModule 

func CountFindingsByModule(ctx context.Context, db *DB, agenticScanUUID string) (map[string]int64, error)

CountFindingsByModule returns finding counts grouped by module_id, scoped to one agentic-scan run when agenticScanUUID is non-empty (empty = all rows).

func CountFindingsBySeverity 

func CountFindingsBySeverity(ctx context.Context, db *DB, projectUUID string, hostnames ...string) (map[string]int64, error)

CountFindingsBySeverity returns finding counts grouped by severity, filtered by project and, when hostnames is non-empty, restricted to findings on those in-scope hosts. Empty hostnames means no host filter. Findings are scoped by hostname only (the findings table carries no scheme/port column), unlike the origin-precise record scoping; this lets the scan-completion summary count only findings on the hosts actually scanned, not leftovers from prior scans in the same project.

func CountFindingsByURL 

func CountFindingsByURL(ctx context.Context, db *DB, agenticScanUUID string) (map[string]int64, error)

CountFindingsByURL returns finding counts grouped by URL, scoped to one agentic-scan run when agenticScanUUID is non-empty (empty = all rows). Findings with an empty URL are skipped — they aren't endpoint-attributable.

func CountRecordsByColumn 

func CountRecordsByColumn(ctx context.Context, db *DB, projectUUID, column string) (map[string]int64, error)

CountRecordsByColumn returns http_record counts grouped by a single column (one of method, status_code, response_content_type), filtered by project (empty = every row, e.g. a --glob-db merge). Keys are the column values as text. Powers the traffic listing's status/method/content-type summary.

func CountSaved 

func CountSaved(uuids []string) int

CountSaved reports how many records in a SaveRecordBatch result were actually persisted. The result is index-aligned with the input and carries "" for a record that was skipped, so len() over-reports — this is the only correct way to turn that slice into a count.

func CurrentSchemaVersion 

func CurrentSchemaVersion() int

CurrentSchemaVersion exposes the schema revision this binary expects, so a consumer can fail loudly on drift at startup (`vigolium version --json`) rather than discovering it as a query error mid-run.

func DeleteGenericRecord 

func DeleteGenericRecord(ctx context.Context, db *DB, tableName, pkValue string) error

DeleteGenericRecord deletes a record by primary key.

func EndpointID added in v0.5.3 

func EndpointID(origin, template string) string

EndpointID is the stable id of an endpoint: the first 12 hex digits of sha1(origin + " " + template). It depends on nothing but the two strings, so the same endpoint has the same id in every database.

func EndpointOrigin added in v0.5.3 

func EndpointOrigin(scheme, host string, port int) string

EndpointOrigin renders scheme://host[:port], omitting the scheme's default port, with the host lower-cased.

func EndpointTemplate added in v0.5.3 

func EndpointTemplate(path string) (template string, segments []string, kinds []pathshape.SegKind)

EndpointTemplate is the path's structural template (query stripped).

func EscapeLikeLiteral 

func EscapeLikeLiteral(s string) string

EscapeLikeLiteral neutralizes the LIKE metacharacters in s so the string matches itself. The escape character is escaped first; doing it last would double the backslashes this function just introduced.

func ExpandPath 

func ExpandPath(path string) string

ExpandPath resolves ~ and environment variables in a filesystem path using the same rules as the SQLite DSN builder. Exported so callers (e.g. the merge lock) can derive sidecar paths next to the real database file.

func ExtractPrimaryToken 

func ExtractPrimaryToken(headers map[string]string) string

extractPrimaryToken extracts the primary session token value from a headers map. It checks Authorization (Bearer/token) first, then Cookie, then falls back to the first header value. Returns empty string if no headers.

func FormatStats 

func FormatStats(stats *DatabaseStats) string

FormatStats formats statistics as a human-readable string

func GetGenericRecord 

func GetGenericRecord(ctx context.Context, db *DB, tableName, pkValue string) (map[string]interface{}, error)

GetGenericRecord fetches a single record by primary key.

func GroupRecordsByField 

func GroupRecordsByField(field string) (string, error)

GroupRecordsByField resolves a public field name to its physical column, or explains what the accepted names are.

Exported so the CLI can reject a bad --group-by before it opens a database: a name typo is knowable from the flags alone, and finding out after the open is how a read command comes to touch a file it had no business touching.

func GroupableFields 

func GroupableFields() []string

GroupableFields lists the accepted --group-by names, sorted, for help text and for the error a rejected name produces.

func HasDynamicSegment 

func HasDynamicSegment(path string) bool

HasDynamicSegment returns true if the path contains at least one dynamic segment — one PathToTemplate would replace with "*". Version-like segments (v1, v2, v10) are not considered dynamic.

func HasSQLiteHeader 

func HasSQLiteHeader(b []byte) bool

HasSQLiteHeader reports whether b begins with the SQLite 3 file magic. Callers that already hold the leading bytes of a file use this to avoid a second read.

func HostnamesOf 

func HostnamesOf(hosts []HostTarget) []string

HostnamesOf returns the distinct hostnames of the given origins, order-preserving. Used to derive a hostname-only filter (e.g. for findings, which carry no port column) from a set of in-scope origins.

func InferSourceType 

func InferSourceType(sourcePath string) string

InferSourceType returns the source type for a given path or URL. Both "gs://" (canonical) and "gcs://" (alias) are recognized as cloud storage.

func InsertGenericRecord 

func InsertGenericRecord(ctx context.Context, db *DB, tableName string, fields map[string]interface{}) error

InsertGenericRecord inserts a record into the given table. fields is a map of column_name -> value. Column names are validated.

func IsBridgeRecordSource 

func IsBridgeRecordSource(source string) bool

IsBridgeRecordSource reports whether a source label names a bridge vendor. Case-insensitive, since it answers both a user-typed --source and a header value written by a plugin.

This is the single answer to "is this label a bridge vendor" — the traffic and replay record selection ask it to decide whether a --source filter can be satisfied by live bridge traffic at all, and the ingest endpoint asks it to decide whether a client may claim the label. A second copy of the list is how a third vendor ends up working on one path and not the other.

func IsBusy added in v0.5.3 

func IsBusy(err error) bool

IsBusy reports whether err is a transient SQLite busy/locked error, the kind another connection's write transaction produces once busy_timeout runs out. Matched on text because the driver wraps them as plain errors; SQLite's own messages and result codes are fixed in its source, not localized.

func IsEvidenceRecordSource added in v0.5.3 

func IsEvidenceRecordSource(source string) bool

IsEvidenceRecordSource reports whether source marks a finding-evidence record.

func IsRedirectStatus 

func IsRedirectStatus(code int) bool

IsRedirectStatus reports whether a status code is one that carries a Location.

304 is excluded: it is a cache validator, not a redirect, and has no destination to show. Shared so the write path (which decides what to store in response_location) and the read path (which decides what to render) can never disagree about what counts as a redirect — a disagreement whose symptom is a blank destination rather than an error.

func IsSQLiteFile 

func IsSQLiteFile(path string) (bool, error)

IsSQLiteFile reports whether the file at path is a SQLite 3 database, detected by its magic header rather than its extension — so a .sqlite/.sqlite3/.db or a bare filename all resolve correctly. Files shorter than the header, or with any other leading bytes (e.g. a JSONL export), read as not-SQLite; a genuine open/read error is surfaced.

func IsValidFindingSeverity 

func IsValidFindingSeverity(s string) bool

IsValidFindingSeverity reports whether s is a recognised Finding severity level.

func IsValidFindingStatus 

func IsValidFindingStatus(s string) bool

IsValidFindingStatus reports whether s is a recognised Finding lifecycle status.

func LikeContains 

func LikeContains(term string) string

LikeContains renders term as a pattern matching any row that CONTAINS it as a literal substring. This is the shape almost every search filter wants.

func LikeGlob 

func LikeGlob(pattern string) string

LikeGlob renders a user-facing glob as a LIKE pattern, where `*` is the only wildcard. It backs the filters documented as accepting wildcards (--host, --path): `*` becomes `%`, and every other metacharacter — including a `%` or `_` the user typed — stays literal.

The caller decides whether a pattern without a `*` means "contains" or "equals"; this function only translates.

func ListTables 

func ListTables(ctx context.Context, db *DB) ([]string, error)

ListTables returns the names of all user tables in the database.

func MinReaderVersion added in v0.5.3 

func MinReaderVersion() int

MinReaderVersion exposes the reader floor this binary writes, for `vigolium version --json`.

func MultiDriverAgenticScanStatus added in v0.5.1 

func MultiDriverAgenticScanStatus(failed, total int) string

MultiDriverAgenticScanStatus is the parent status of a run made of several drivers: "failed" when every driver failed (nothing ran, so "completed_with_errors" overstated it), "completed_with_errors" when some did, "completed" otherwise.

func ParamMiningRemark added in v0.5.3 

func ParamMiningRemark(name, signal string) string

ParamMiningRemark renders the remark for one confirmed hidden parameter.

func ParamValueTypes added in v0.5.3 

func ParamValueTypes() []string

ParamValueTypes lists the dominant_type names --type accepts.

func ParameterTypeFromParamType 

func ParameterTypeFromParamType(ptype httpmsg.ParamType) string

ParameterTypeFromParamType converts ParamType to database parameter type string

func PathToTemplate 

func PathToTemplate(path string) string

PathToTemplate normalizes a URL path for related-record lookup by replacing dynamic path segments (IDs, UUIDs, tokens) with a wildcard "*". Static segments and version segments (v1, v2, v10) are preserved. The resulting template uses "*" which callers can replace with "%" for SQL LIKE.

It is a thin wrapper over pkg/httpmsg/pathshape (whose "{kind}" placeholders all become "*") plus the legacy long-token rule:

  • pathshape kinds: integers, UUIDs, all-hex ≥ 8 chars, ISO dates and timestamps, emails, JWTs, base64 ≥ 16 chars
  • Long alphanumeric ≥ 20 chars (token-like): `^[A-Za-z0-9_-]{20,}$`

Version-like segments (v1, v2, v10) are always kept as-is.

func QueryGenericTable 

func QueryGenericTable(ctx context.Context, db *DB, tableName string, limit, offset int) ([]map[string]interface{}, []string, int64, error)

QueryGenericTable runs a paginated SELECT * on the given table. Returns rows as ordered maps, column names, and total row count. The tableName must be validated against ListTables before calling.

func QueryGenericTableFiltered 

func QueryGenericTableFiltered(ctx context.Context, db *DB, tableName string, opts GenericQueryOptions) ([]map[string]interface{}, []string, int64, error)

QueryGenericTableFiltered runs a filtered, sorted, paginated SELECT on the given table. The tableName and all column references are validated before use.

func RecordToHttpRequestResponse 

func RecordToHttpRequestResponse(record *HTTPRecord) (*httpmsg.HttpRequestResponse, error)

RecordToHttpRequestResponse converts an HTTPRecord back to HttpRequestResponse. Exported for use by the agent input normalizer and other packages.

func ResolveHostnameNow 

func ResolveHostnameNow(ctx context.Context, hostname string) (ipv4, ipv6, cname []string)

ResolveHostnameNow resolves hostname synchronously and caches the answer, returning the full answer. Returns the cached value when one is fresh.

This is the DNS-prefetch entry point for a host sweep, where DNS is part of the ANSWER rather than incidental metadata: the probe phase resolves its whole target list through here (bounded, before any HTTP) so every record it writes carries a complete A/AAAA/CNAME set. Every other path keeps using the background best-effort resolver, because on a normal scan's write path a blocking lookup would stall the record writer for a full DNS timeout per dead host.

ctx cancels the WAIT, not the lookup: a resolution in flight is shared with every other caller asking the same question, so one caller walking away must not take the answer with it. The lookup itself is bounded by dnsLookupTimeout.

func RetryBusy added in v0.5.3 

func RetryBusy(ctx context.Context, fn func() error, policy RetryPolicy) error

RetryBusy runs fn, retrying with exponential backoff while policy.Retryable accepts the error. It returns nil on the first success and the error itself on the first non-retryable failure. When the budget runs out on a busy error the result is a *LockedError (code database_locked); when it runs out on any other retryable error, that error is returned unchanged. A cancelled ctx stops the wait and returns ctx.Err().

func ScanComparability added in v0.5.3 

func ScanComparability(base, head *Scan) (bool, []string)

ScanComparability reports whether head can vouch for the absence of base's findings: it finished, to completion, over at least base's targets and every phase that ran in base. Unknown is never comparable — "fixed" is a claim, and a missing verdict cannot support it.

func SetBinaryVersion added in v0.5.3 

func SetBinaryVersion(v string)

SetBinaryVersion records the running vigolium version for schema_meta.written_by.

func SortedCodes added in v0.5.2 

func SortedCodes(set map[string]struct{}) []string

SortedCodes returns the keys of a code set, sorted and deduplicated, or nil for an empty set — so a PhaseOutcome's Reasons/Limits are stable across runs and comparable between two scans of the same target.

func TruncateOutcomeMessage added in v0.5.2 

func TruncateOutcomeMessage(msg string) string

TruncateOutcomeMessage clips a message to OutcomeMessageMax bytes on a rune boundary, so a persisted outcome can never carry an unbounded error string or an invalid UTF-8 tail.

func UpdateGenericRecord 

func UpdateGenericRecord(ctx context.Context, db *DB, tableName, pkValue string, fields map[string]interface{}) error

UpdateGenericRecord updates a record by primary key. fields is a map of column_name -> new_value. PK columns cannot be updated.

func ValidateEndpointSort added in v0.5.3 

func ValidateEndpointSort(s string) error

ValidateEndpointSort reports whether s is an accepted sort ("" is).

func ValidateFindingHashPrefix added in v0.5.3 

func ValidateFindingHashPrefix(h string) error

ValidateFindingHashPrefix rejects a hash selector that is too short or not hex. Exported so the CLI can reject it as a usage error before opening the store.

func ValidateTableName 

func ValidateTableName(ctx context.Context, db *DB, tableName string) error

ValidateTableName checks that the given name is in the list of real tables.

func WhereScanRecords added in v0.5.3 

func WhereScanRecords(q *bun.SelectQuery, scanUUID string) *bun.SelectQuery

WhereScanRecords narrows an http_records query (alias "r") to the exchanges one scan accounts for: the rows it stored first (http_records.scan_uuid) plus any row one of its findings cites as evidence. The second half matters because records are deduplicated across scans: a rescan's finding can point at an exchange an earlier scan stored, and a scan-scoped export that dropped it would ship the finding without its proof.

CountRecordsForScan deliberately counts only the first half — it answers "how many records did this scan write".

func WithFacts 

func WithFacts(src *StoredHostFacts, r *HTTPRecord) any

WithFacts pairs a record with what src observed about its host. Returns the bare record when nothing was observed, so a process that probed nothing emits exactly the bytes it always did. A nil src answers from the process caches.

func WithLikeEscape 

func WithLikeEscape(pred string) string

WithLikeEscape appends the ESCAPE clause to every `LIKE ?` placeholder in a predicate.

It is a rewrite rather than 30 hand-edited SQL strings on purpose. The predicates in this package span multi-line raw strings with up to twelve placeholders each, and an ESCAPE clause missed on one branch is a filter that silently keeps the old over-matching behavior — the exact failure mode this whole file exists to remove. Rewriting makes the guarantee positional: if the text says LIKE ?, it carries an ESCAPE.

Call it once per predicate at package level where possible; the per-query cost is a single pass over a short constant string.

func WithMergeLock 

func WithMergeLock(destPath string, timeout time.Duration, fn func() error) error

WithMergeLock runs fn while holding a best-effort cross-process advisory lock for destPath, serializing concurrent merges into the same destination database. The lock is an O_EXCL sentinel file (<destPath>.merge-lock), which is portable across platforms and needs no syscall-level file locking.

It is intentionally best-effort: if the lock cannot be acquired within timeout, fn runs anyway and relies on SQLite's own busy_timeout + the merge's retry-on-busy loop for correctness. The lock is an optimization that keeps staggered finishers from contending, not a correctness requirement, so a stale or contended lock degrades to DB-level serialization rather than failing the merge.

Types

type AgentFindingCandidate 

type AgentFindingCandidate struct {
	bun.BaseModel `bun:"table:agent_finding_candidates,alias:acand" json:"-"`

	ID              int64  `bun:"id,pk,autoincrement" json:"id"`
	UUID            string `bun:"uuid,notnull,unique" json:"uuid"`
	AgenticScanUUID string `bun:"agentic_scan_uuid,nullzero" json:"agentic_scan_uuid,omitempty"`
	ProjectUUID     string `bun:"project_uuid,nullzero" json:"project_uuid,omitempty"`
	SectionUUID     string `bun:"section_uuid,nullzero" json:"section_uuid,omitempty"`

	Title       string `bun:"title,nullzero" json:"title,omitempty"`
	Severity    string `bun:"severity,nullzero" json:"severity,omitempty"`
	Description string `bun:"description,nullzero" json:"description,omitempty"`
	Remediation string `bun:"remediation,nullzero" json:"remediation,omitempty"`
	CWEID       string `bun:"cwe_id,nullzero" json:"cwe_id,omitempty"`
	SourceFile  string `bun:"source_file,nullzero" json:"source_file,omitempty"`
	URL         string `bun:"url,nullzero" json:"url,omitempty"`
	Hostname    string `bun:"hostname,nullzero" json:"hostname,omitempty"`
	Confidence  string `bun:"confidence,nullzero" json:"confidence,omitempty"`
	Class       string `bun:"class,nullzero" json:"class,omitempty"` // idor | xss | sqli | ssrf | rce | auth | other

	Status        string `bun:"status,notnull,default:'proposed'" json:"status"` // proposed | verifying | confirmed | rejected | needs_evidence
	VerdictReason string `bun:"verdict_reason,nullzero" json:"verdict_reason,omitempty"`
	EvidenceGrade string `bun:"evidence_grade,nullzero" json:"evidence_grade,omitempty"`

	RecordUUIDs []string `bun:"record_uuids,type:jsonb,nullzero" json:"record_uuids,omitempty"`
	OASTIDs     []string `bun:"oast_ids,type:jsonb,nullzero" json:"oast_ids,omitempty"`
	Request     string   `bun:"request,nullzero" json:"request,omitempty"`
	Response    string   `bun:"response,nullzero" json:"response,omitempty"`

	DedupHash         string   `bun:"dedup_hash,notnull,default:''" json:"dedup_hash"`
	PromotedFindingID int64    `bun:"promoted_finding_id,notnull,default:0" json:"promoted_finding_id"`
	Tags              []string `bun:"tags,type:jsonb,nullzero" json:"tags,omitempty"`

	CreatedAt  time.Time `bun:"created_at,notnull,default:current_timestamp" json:"created_at"`
	VerifiedAt time.Time `bun:"verified_at,nullzero" json:"verified_at,omitempty"`
}

AgentFindingCandidate is a proposed finding awaiting fresh-context verification in the durable-autopilot verify-before-promote pipeline. The operator's propose_candidate tool writes rows here (status=proposed); a skeptic verifier grades them, and confirmed ones are promoted into a real Finding (promoted_finding_id set, status=confirmed). Rows are only written when agent.olium.autopilot_mode != legacy. Cascade-deleted with the parent agentic scan.

type AgentSection 

type AgentSection struct {
	bun.BaseModel `bun:"table:agent_sections,alias:asec" json:"-"`

	ID              int64  `bun:"id,pk,autoincrement" json:"id"`
	UUID            string `bun:"uuid,notnull,unique" json:"uuid"`
	AgenticScanUUID string `bun:"agentic_scan_uuid,nullzero" json:"agentic_scan_uuid,omitempty"`
	ProjectUUID     string `bun:"project_uuid,nullzero" json:"project_uuid,omitempty"`

	Seq            int    `bun:"seq,notnull,default:0" json:"seq"`
	Kind           string `bun:"kind,nullzero" json:"kind,omitempty"`
	Status         string `bun:"status,notnull,default:'running'" json:"status"` // running | completed | interrupted
	Task           string `bun:"task,nullzero" json:"task,omitempty"`
	ClosingSummary string `bun:"closing_summary,nullzero" json:"closing_summary,omitempty"`
	RotationReason string `bun:"rotation_reason,nullzero" json:"rotation_reason,omitempty"`
	TurnCount      int    `bun:"turn_count,notnull,default:0" json:"turn_count"`
	InputTokens    int64  `bun:"input_tokens,notnull,default:0" json:"input_tokens"`
	OutputTokens   int64  `bun:"output_tokens,notnull,default:0" json:"output_tokens"`
	ErrorMessage   string `bun:"error_message,nullzero" json:"error_message,omitempty"`

	StartedAt time.Time `bun:"started_at,nullzero" json:"started_at,omitempty"`
	EndedAt   time.Time `bun:"ended_at,nullzero" json:"ended_at,omitempty"`
	CreatedAt time.Time `bun:"created_at,notnull,default:current_timestamp" json:"created_at"`
}

AgentSection represents one bounded operator section within a durable autopilot run (an engine Reset() + reconstructed-brief cycle). Rows are only written when agent.olium.autopilot_mode != legacy; legacy runs never touch this table. Cascade-deleted with the parent agentic scan.

type AgenticRunEnd added in v0.5.3 

type AgenticRunEnd struct {
	// Err is the run's error, nil on a clean finish. A time or budget stop
	// still counts as that stop when Err is set: the loop often reports the
	// cancellation it caused as an error.
	Err error
	// TimedOut: the wall-clock budget (--max-duration) fired.
	TimedOut bool
	// BudgetExhausted: a command, token or cost budget fired.
	BudgetExhausted bool
	// Cancelled: an interrupt (Ctrl-C, SIGTERM) stopped the run.
	Cancelled bool
	// Degraded: the run finished but with warnings (see CompletedAgenticScanStatus).
	Degraded bool
	// FailedDrivers/TotalDrivers: a multi-driver run (audit), see
	// MultiDriverAgenticScanStatus. Zero TotalDrivers means single-driver.
	FailedDrivers, TotalDrivers int
}

AgenticRunEnd describes how an agentic run ended, for AgenticRunStatus.

type AgenticScan 

type AgenticScan struct {
	bun.BaseModel `bun:"table:agentic_scans,alias:ar" json:"-"`

	ID          int64  `bun:"id,pk,autoincrement" json:"id"`
	UUID        string `bun:"uuid,notnull,unique" json:"uuid"`
	ProjectUUID string `bun:"project_uuid,notnull" json:"project_uuid"`
	ScanUUID    string `bun:"scan_uuid,nullzero" json:"scan_uuid,omitempty"`

	// Config
	Mode        string   `bun:"mode,notnull" json:"mode"` // query, autopilot, pipeline, scan
	AgentName   string   `bun:"agent_name,notnull" json:"agent_name"`
	Protocol    string   `bun:"protocol,nullzero" json:"protocol,omitempty"` // engine identifier — currently always "olium-engine" (subprocess SDK backends removed)
	Model       string   `bun:"model,nullzero" json:"model,omitempty"`       // model name used for this run (for cost audit)
	InputRaw    string   `bun:"input_raw,nullzero" json:"input_raw,omitempty"`
	InputType   string   `bun:"input_type,nullzero" json:"input_type,omitempty"` // url, curl, burp, raw, record_uuid
	TargetURL   string   `bun:"target_url,nullzero" json:"target_url,omitempty"`
	VulnType    string   `bun:"vuln_type,nullzero" json:"vuln_type,omitempty"`
	ModuleNames []string `bun:"module_names,type:jsonb,nullzero" json:"module_names,omitempty"`
	TemplateID  string   `bun:"template_id,nullzero" json:"template_id,omitempty"`

	// Execution
	Status       string   `bun:"status,notnull,default:'pending'" json:"status"` // pending, running, completed, failed, cancelled
	CurrentPhase string   `bun:"current_phase,nullzero" json:"current_phase,omitempty"`
	PhasesRun    []string `bun:"phases_run,type:jsonb,nullzero" json:"phases_run,omitempty"`

	// Results
	FindingCount int `bun:"finding_count,default:0" json:"finding_count"`
	RecordCount  int `bun:"record_count,default:0" json:"record_count"`
	SavedCount   int `bun:"saved_count,default:0" json:"saved_count"`

	// Agent context
	SourcePath            string                 `bun:"source_path,nullzero" json:"source_path,omitempty"`            // source code path used
	SourceType            string                 `bun:"source_type,nullzero" json:"source_type,omitempty"`            // local, git-url, gcs
	TokenUsage            map[string]interface{} `bun:"token_usage,type:jsonb,nullzero" json:"token_usage,omitempty"` // input/output token counts per phase
	TotalInputTokens      int64                  `bun:"total_input_tokens,default:0" json:"total_input_tokens"`
	TotalOutputTokens     int64                  `bun:"total_output_tokens,default:0" json:"total_output_tokens"`
	EstimatedCostUSD      float64                `bun:"estimated_cost_usd,default:0" json:"estimated_cost_usd"`
	RetryCount            int                    `bun:"retry_count,default:0" json:"retry_count"`
	ParentAgenticScanUUID string                 `bun:"parent_run_uuid,nullzero" json:"parent_agentic_scan_uuid,omitempty"` // for swarm sub-runs
	InputRecordCount      int                    `bun:"input_record_count,default:0" json:"input_record_count"`

	// Agent session ID (for resume)
	SessionID string `bun:"session_id,nullzero" json:"session_id,omitempty"`

	// Filesystem path to the session directory holding agent artifacts
	// (output.md, audit-stream.jsonl, extensions/, etc.). Populated by the
	// server handlers so API clients can locate run outputs.
	SessionDir string `bun:"session_dir,nullzero" json:"session_dir,omitempty"`

	// Debug (stored as JSON text blobs)
	AttackPlan     string `bun:"attack_plan,nullzero" json:"attack_plan,omitempty"`
	TriageResult   string `bun:"triage_result,nullzero" json:"triage_result,omitempty"`
	PromptSent     string `bun:"prompt_sent,nullzero" json:"prompt_sent,omitempty"`
	AgentRawOutput string `bun:"agent_raw_output,nullzero" json:"agent_raw_output,omitempty"`
	ErrorMessage   string `bun:"error_message,nullzero" json:"error_message,omitempty"`

	// Pipeline/scan result (JSON blob for full result objects)
	ResultJSON string `bun:"result_json,nullzero" json:"result_json,omitempty"`

	StorageURL string `bun:"storage_url,nullzero" json:"storage_url,omitempty"`

	// Timing
	StartedAt   time.Time `bun:"started_at,nullzero" json:"started_at,omitempty"`
	CompletedAt time.Time `bun:"completed_at,nullzero" json:"completed_at,omitempty"`
	DurationMs  int64     `bun:"duration_ms,default:0" json:"duration_ms"`
	CreatedAt   time.Time `bun:"created_at,notnull,default:current_timestamp" json:"created_at"`
}

AgenticScan represents a single agent execution for debugging and status tracking. It replaces the in-memory agent run status map with persistent DB storage.

type AmbiguousFindingHashError added in v0.5.3 

type AmbiguousFindingHashError struct {
	Prefix     string
	Candidates []string // finding_hash values, at most a handful
	More       bool     // more candidates exist than are listed
}

AmbiguousFindingHashError is returned when a hash prefix matches more than one finding. Nothing is written: a verdict meant for one finding must never land on several.

func (*AmbiguousFindingHashError) Error added in v0.5.3 

func (e *AmbiguousFindingHashError) Error() string

type AnalysisArtifact 

type AnalysisArtifact struct {
	bun.BaseModel `bun:"table:analysis_artifacts,alias:aa" json:"-"`

	ID             int64     `bun:"id,pk,autoincrement" json:"id"`
	ProjectUUID    string    `bun:"project_uuid,notnull" json:"project_uuid"`
	ScanUUID       string    `bun:"scan_uuid,nullzero" json:"scan_uuid,omitempty"`
	HTTPRecordUUID string    `bun:"http_record_uuid,notnull" json:"http_record_uuid"`
	Kind           string    `bun:"kind,notnull" json:"kind"`
	Filename       string    `bun:"filename,nullzero" json:"filename,omitempty"`
	MediaType      string    `bun:"media_type,nullzero" json:"media_type,omitempty"`
	SHA256         string    `bun:"sha256,notnull" json:"sha256"`
	ByteLength     int64     `bun:"byte_length,notnull" json:"byte_length"`
	Content        []byte    `bun:"content,type:bytea,notnull" json:"content,omitempty"`
	Metadata       string    `bun:"metadata,type:jsonb,nullzero" json:"metadata,omitempty"`
	CreatedAt      time.Time `bun:"created_at,notnull,default:current_timestamp" json:"created_at"`
}

AnalysisArtifact stores immutable derived content linked to an HTTP record. It deliberately lives beside, rather than inside, HTTPRecord so analysis can never replace the raw evidence captured from the target.

type AuthenticationHostname 

type AuthenticationHostname struct {
	bun.BaseModel `bun:"table:authentication_hostnames,alias:sh" json:"-"`

	ID          int64  `bun:"id,pk,autoincrement" json:"id"`
	ProjectUUID string `bun:"project_uuid,notnull" json:"project_uuid"`
	ScanUUID    string `bun:"scan_uuid,nullzero" json:"scan_uuid,omitempty"`

	Hostname string `bun:"hostname,notnull" json:"hostname"`

	SessionName string `bun:"session_name,notnull" json:"session_name"`
	SessionRole string `bun:"session_role,nullzero,default:''" json:"session_role,omitempty"`
	Position    int    `bun:"position,default:0" json:"position"`

	// Primary session token (JWT, cookie value, API key) for quick access.
	SessionToken string `bun:"session_token,nullzero" json:"session_token,omitempty"`

	// Static auth headers (JSON map)
	Headers map[string]string `bun:"headers,type:jsonb,nullzero" json:"headers,omitempty"`

	// Login flow fields (flat for queryability)
	LoginURL         string `bun:"login_url,nullzero" json:"login_url,omitempty"`
	LoginMethod      string `bun:"login_method,nullzero" json:"login_method,omitempty"`
	LoginContentType string `bun:"login_content_type,nullzero" json:"login_content_type,omitempty"`
	LoginBody        string `bun:"login_body,nullzero" json:"login_body,omitempty"`
	LoginRequest     string `bun:"login_request,nullzero" json:"login_request,omitempty"`
	LoginResponse    string `bun:"login_response,nullzero" json:"login_response,omitempty"`

	// Extract rules (JSON array)
	ExtractRules string `bun:"extract_rules,type:jsonb,nullzero" json:"extract_rules,omitempty"`

	Source     string     `bun:"source,nullzero,default:''" json:"source,omitempty"`
	HydratedAt *time.Time `bun:"hydrated_at,nullzero" json:"hydrated_at,omitempty"`
	CreatedAt  time.Time  `bun:"created_at,notnull,default:current_timestamp" json:"created_at"`
	UpdatedAt  time.Time  `bun:"updated_at,notnull,default:current_timestamp" json:"updated_at"`
}

AuthenticationHostname persists per-hostname session auth configs in the DB.

func SessionToAuthenticationHostname 

func SessionToAuthenticationHostname(s *authentication.Session, position int) *AuthenticationHostname

SessionToAuthenticationHostname converts a native authentication.Session to a DB AuthenticationHostname row. The caller must set ProjectUUID, Hostname, and optionally ScanUUID on the returned row.

func SessionsToAuthenticationHostnames 

func SessionsToAuthenticationHostnames(sessions []*authentication.Session, projectUUID, hostname string) []*AuthenticationHostname

SessionsToAuthenticationHostnames converts a slice of authentication.Session objects to DB rows for a given hostname. Sets ProjectUUID and Hostname on each row.

type BodyCompaction added in v0.5.3 

type BodyCompaction struct {
	Version int `json:"version"`
	// CompactedAt is when the most recent part was compacted.
	CompactedAt time.Time `json:"compacted_at"`
	// KeepHead is the head size of the most recent pass that compacted a part.
	KeepHead int64 `json:"keep_head"`
	// Response and Request are present only for the parts that were compacted.
	Response *CompactedBody `json:"response,omitempty"`
	Request  *CompactedBody `json:"request,omitempty"`
}

BodyCompaction is the marker stored in http_records.body_compaction.

type ClientRoute added in v0.5.3 

type ClientRoute struct {
	Scheme   string
	Hostname string
	Port     int

	Path           string
	FullPath       string
	FullPathStatic bool
	RouteType      string
	Extractor      string
	Confidence     string
	SourceURL      string
	ScanUUID       string
}

ClientRoute is one client-side route a JS asset declared, read back from a jstangle-facts artifact, together with the origin of the record the artifact is anchored to (the asset's own record or, for an unrecorded asset, a record of its origin). Path is the route as the router declares it; FullPath is the helper's normalized absolute route (parents joined) when it reported one.

type ColumnInfo 

type ColumnInfo struct {
	Name     string `json:"name"`
	Type     string `json:"type"`
	Nullable string `json:"nullable"`
	Default  string `json:"default,omitempty"`
}

ColumnInfo describes a single column in a database table.

func ListColumns 

func ListColumns(ctx context.Context, db *DB, tableName string) ([]ColumnInfo, error)

ListColumns returns column metadata for the given table.

type CompactOptions added in v0.5.3 

type CompactOptions struct {
	// ProjectUUID confines the pass to one project; empty means every project
	// in the file.
	ProjectUUID string
	// BodyOver is the body size, in bytes, above which a body is compacted.
	BodyOver int64
	// KeepHead is how many body bytes a compacted body keeps. Must be less than
	// BodyOver.
	KeepHead int64
	// SkipContentTypes / OnlyContentTypes are globs over the response media
	// type; SkipPaths / OnlyPaths are globs over the URL path. See
	// compactFilter for how they combine.
	SkipContentTypes []string
	OnlyContentTypes []string
	SkipPaths        []string
	OnlyPaths        []string
	// OlderThan, when set, restricts the pass to records sent strictly before it.
	OlderThan *time.Time
	// MaxDBSize, when positive, turns the pass into a budget: records are
	// compacted oldest first until the estimated size after VACUUM is at most
	// this many bytes, and the pass stops there.
	MaxDBSize int64
	// DryRun counts what the pass would do and changes nothing.
	DryRun bool
	// Vacuum runs VACUUM (and a WAL checkpoint) after a pass that changed rows,
	// so the freed pages are returned to the filesystem.
	Vacuum bool
	// Now stamps compacted_at; nil means time.Now.
	Now func() time.Time
}

CompactOptions selects which bodies a compaction pass replaces.

func (*CompactOptions) Validate added in v0.5.3 

func (o *CompactOptions) Validate() error

Validate checks the option combination.

type CompactSummary added in v0.5.3 

type CompactSummary struct {
	DryRun bool `json:"dry_run"`
	// RecordsCompacted counts records with at least one body replaced (or, on a
	// dry run, that would be).
	RecordsCompacted        int64 `json:"records_compacted"`
	ResponseBodiesCompacted int64 `json:"response_bodies_compacted"`
	RequestBodiesCompacted  int64 `json:"request_bodies_compacted"`
	// BodyBytesBefore / BodyBytesAfter sum the compacted bodies' lengths before
	// and after.
	BodyBytesBefore int64 `json:"body_bytes_before"`
	BodyBytesAfter  int64 `json:"body_bytes_after"`
	// DBBytesBefore / DBBytesAfter are the database file plus its WAL. On a dry
	// run DBBytesAfter is the estimated size after compaction and VACUUM.
	DBBytesBefore int64 `json:"db_bytes_before"`
	DBBytesAfter  int64 `json:"db_bytes_after"`
	// MaxDBSize echoes the budget; 0 means none was set.
	MaxDBSize int64 `json:"max_db_size"`
	// BudgetMet is true when no budget was set, or when the database ends at or
	// under it. False means every selected body was compacted and the file is
	// still over budget.
	BudgetMet bool `json:"budget_met"`
	// Remaining counts selected records that still hold a body over BodyOver
	// after this pass — non-zero when a budget stopped the pass early.
	Remaining int64 `json:"remaining"`
	Vacuumed  bool  `json:"vacuumed"`
}

CompactSummary reports what a pass did. It is the `items` payload of `vigolium db compact -j`.

type CompactedBody added in v0.5.3 

type CompactedBody struct {
	// OriginalLength is the body's length in bytes before compaction, headers
	// excluded.
	OriginalLength int64 `json:"original_length"`
	// SHA256 is the lowercase hex SHA-256 of the original body bytes.
	SHA256 string `json:"sha256"`
	// KeptLength is how many body bytes the stored message still holds.
	KeptLength int64 `json:"kept_length"`
}

CompactedBody describes one replaced body.

type DB 

type DB struct {
	*bun.DB
	// contains filtered or unexported fields
}

DB wraps bun.DB with additional metadata

func NewDB 

func NewDB(cfg *config.DatabaseConfig) (*DB, error)

NewDB creates database connection based on config

func NewDBFromBun 

func NewDBFromBun(bunDB *bun.DB, driver string) *DB

NewDBFromBun wraps an existing bun.DB for use in tests or external tooling.

func (*DB) CheckSchema added in v0.5.3 

func (db *DB) CheckSchema(ctx context.Context) error

CheckSchema answers, with catalog reads only, whether this store can be read as-is by this binary. It never takes a write lock and never migrates.

  • nil: current (or newer but readable — additive migrations only add);
  • *SchemaTooNewError: min_reader_version is above this binary;
  • ErrSchemaNotInitialized: no vigolium tables at all;
  • *SchemaOutdatedError: tables, indexes, columns or the version are behind.

The read commands' opener decides what to do with the last two: migrate through a separate writer (database.auto_migrate_on_read) or refuse with the `vigolium db migrate` remedy.

func (*DB) Close 

func (db *DB) Close() error

Close closes database connection

func (*DB) CompactBodies added in v0.5.3 

func (db *DB) CompactBodies(ctx context.Context, opts CompactOptions) (*CompactSummary, error)

CompactBodies runs one compaction pass. See the package comment above.

func (*DB) CreateRecordIndexes 

func (db *DB) CreateRecordIndexes(ctx context.Context) error

CreateRecordIndexes builds the http_records read indexes. Idempotent (every statement is IF NOT EXISTS), so calling it on a store that never deferred is harmless.

func (*DB) CreateSchema 

func (db *DB) CreateSchema(ctx context.Context) error

CreateSchema creates all database tables and indexes if they don't exist.

It is a no-op on a read-only handle. The read commands call it on open to self-heal a fresh project database; against a source opened --read-only that would be a schema write to someone else's evidence, and a source genuinely missing the tables surfaces as a query error rather than being silently migrated into the current shape.

It refuses (SchemaTooNewError) a store a newer binary has stamped: DDL and backfills written for an older schema are not safe to run against one this binary does not know.

func (*DB) DeferRecordIndexes 

func (db *DB) DeferRecordIndexes()

DeferRecordIndexes tells CreateSchema to leave the http_records read indexes uncreated, so a bulk load runs against the primary key alone. The caller MUST call CreateRecordIndexes once loading is done — a store left without them is schema-complete but plans every read as a full scan.

Only for disposable stores that are filled once and then read (the --glob-db scratch merge). A persistent store must not defer: it is written to continuously, so there is no "after the load" to build them in.

func (*DB) Driver 

func (db *DB) Driver() string

Driver returns the database driver name

func (*DB) EnsureSchemaCurrent 

func (db *DB) EnsureSchemaCurrent(ctx context.Context) error

EnsureSchemaCurrent brings an existing database up to date, doing NOTHING when it already is.

The "do nothing" path is the whole point. CreateSchema issues DDL, and DDL takes SQLite's WRITE lock — so calling it unconditionally on every open makes every read command queue behind any concurrent writer for up to busy_timeout (60s by default). Detecting staleness first is a read-only question, so the common case takes no lock at all.

A read-only handle cannot be migrated (mode=ro refuses DDL); CreateSchema's own read-only branch turns that into ErrSchemaOutdated naming the missing columns, so it is reached through the same call rather than special-cased here.

Indexes and tables are deliberately not part of the check: only a missing COLUMN makes the read commands this serves fail outright, and a read command meeting an old file wants the cheapest question that unblocks it.

That makes this the WEAKER of the package's two readiness checks, and it is not the one to reach for when the answer gates a write workload — a missing table or a missing UNIQUE index passes it. See EnsureSchemaReady, which checks every object CreateSchema creates and is what the scan entry points use now that they no longer run CreateSchema unconditionally.

func (*DB) EnsureSchemaReady added in v0.5.0 

func (db *DB) EnsureSchemaReady(ctx context.Context) error

EnsureSchemaReady makes this database usable, running schema DDL only when a read-only check shows something is actually missing.

Use it wherever a process opens a store it expects to already be initialized, so starting a scan alongside another writer does not queue on the write lock just to re-assert a schema that is already correct. A fresh file, a missing table, a missing index, a missing migrated column or an older migration version all fall through to the full CreateSchema, so this never accepts an incomplete schema.

ctx bounds the read-only readiness question only. The repair deliberately runs without that deadline: CreateSchema's gated branches are O(rows) — whole-table backfills and index builds — so a large database that genuinely owes migrations needs however long they take, and cutting it off at a deadline derived from a lock-wait budget would turn a slow success into a failure.

func (*DB) GetStats 

func (db *DB) GetStats(ctx context.Context, filters QueryFilters) (*DatabaseStats, error)

GetStats retrieves database statistics

func (*DB) GetTopHosts 

func (db *DB) GetTopHosts(ctx context.Context, filters QueryFilters, limit int) ([]HostStats, error)

GetTopHosts retrieves top hosts by request count with per-host finding counts. It uses a constant TWO queries — one for the top hosts, one grouped finding-count query — instead of the previous 1 + N (a per-host count query), so its cost no longer grows with the requested host limit.

func (*DB) HasFTS 

func (db *DB) HasFTS() bool

HasFTS reports whether full-text search is available on this handle (FTS5 for SQLite, tsvector for PostgreSQL).

It DISCOVERS the answer rather than remembering whether this process created the index. It used to be a plain field assigned only by SeedDefaults, which made the answer depend on how the handle was opened rather than on what the database contains: `vigolium server` seeds and reported true, every read command (traffic, finding, db ls) does not seed and reported false on the very same file. Two commands then took different query paths over one database.

Discovery is lazy and cached: a handle that never searches pays nothing, and one that does pays a single sqlite_master lookup.

func (*DB) HasVigoliumSchema 

func (db *DB) HasVigoliumSchema(ctx context.Context) bool

HasVigoliumSchema reports whether this store has been initialized as a vigolium database, by the presence of the core table every read touches.

It is the question "is this the right file?", which is NOT the same as "does it have rows": a foreign SQLite file and a scanned target both open cleanly, and only this distinguishes them before vigolium's tables get created inside the former.

func (*DB) IsReader added in v0.5.3 

func (db *DB) IsReader() bool

IsReader reports whether this handle is a read-intent pool.

func (*DB) Migrate added in v0.5.3 

func (db *DB) Migrate(ctx context.Context) (*MigrateResult, error)

Migrate brings this store up to the binary's schema and reports what it changed. It is the explicit form of the migration every writer runs on open, and the remedy SchemaOutdatedError names.

The migration itself runs without ctx's deadline, for the reason EnsureSchemaReady gives: its gated work is O(rows), and a large store that owes it needs however long it takes.

func (*DB) OpenReader added in v0.5.3 

func (db *DB) OpenReader() (*DB, bool, error)

OpenReader opens a second, read-intent pool on the store this handle opened, for a long-running process that wants its list/get traffic off the writer's pool. The caller owns closing it.

It returns (db, false) — the handle itself, nothing new to close — where a separate pool would add nothing: in-memory SQLite (each connection is its own database), a handle that is already read-only or a reader, or a handle that was not opened by NewDB.

func (*DB) Optimize 

func (db *DB) Optimize(ctx context.Context)

Optimize refreshes the query-planner statistics so subsequent dedup and finding queries plan against real row counts. Call it after a bulk ingest phase (e.g. discovery) and before the heavy dedup passes; it also runs on Close and on the WAL checkpointer cadence. Best-effort — a failure degrades plans, never correctness.

SQLite uses `PRAGMA optimize` (incremental: only re-analyzes tables that changed enough to matter). PostgreSQL analyzes just the hot ingest tables rather than the whole database, so a short-lived CLI run against a shared server doesn't trigger a blocking full-DB ANALYZE on every close.

func (*DB) Path 

func (db *DB) Path() string

Path returns the resolved SQLite file backing this handle, or "" for a driver that is not a single file (Postgres).

It exists because DatabaseStats.Database.Path and .Size were declared and never filled: `db stats -j` reported `"path": "", "size": 0` for a store it had open and had just counted 60 records in. A consumer reading that to confirm WHICH database it was looking at — the whole reason the -j envelope carries db_path — was told nothing, in a field that looked like an answer.

func (*DB) SQLDB 

func (db *DB) SQLDB() *sql.DB

SQLDB returns the underlying *sql.DB. Used by callers that need a pinned *sql.Conn for connection-scoped operations (e.g. ATTACH DATABASE during a SQLite-to-SQLite merge) that must run on a single physical connection.

func (*DB) SchemaInfo added in v0.5.3 

func (db *DB) SchemaInfo(ctx context.Context) (SchemaInfo, error)

SchemaInfo reads schema_meta. A store with no schema_meta row (fresh, or created before versioning) reports the zero value and no error; a store from before the v2 columns reports its version with min_reader_version 1, which is what every such store honestly requires.

func (*DB) SeedDefaults 

func (db *DB) SeedDefaults(ctx context.Context) error

SeedDefaults creates the default user and project if they don't exist. This is called during initialization to ensure CLI has a working project context.

func (*DB) ServerVersion 

func (db *DB) ServerVersion(ctx context.Context) (string, error)

ServerVersion returns the database server version string (e.g. the Postgres "PostgreSQL 16.1 ..." banner or the SQLite library version).

type DBInputSource 

type DBInputSource struct {
	// contains filtered or unexported fields
}

DBInputSource polls the database for HTTP records after the scan cursor and provides them as WorkItems. It implements source.InputSource.

func NewDBInputSource 

func NewDBInputSource(db *DB, repo *Repository, scanUUID string, pollInterval time.Duration) *DBInputSource

NewDBInputSource creates a new DBInputSource that polls for records after the scan cursor at the given interval.

func NewOneShotDBInputSource 

func NewOneShotDBInputSource(db *DB, repo *Repository, scanUUID string) *DBInputSource

NewOneShotDBInputSource creates a DBInputSource that returns io.EOF when no records remain after the cursor, instead of polling indefinitely.

func (*DBInputSource) Close 

func (s *DBInputSource) Close() error

Close stops the source. After Close, Next will return io.EOF.

func (*DBInputSource) Count added in v0.5.3 

func (s *DBInputSource) Count() int64

Count implements source.Countable: the number of records that were eligible when the source first read from the database, or 0 before its first Next.

An upper bound at the moment it is taken, not a promise: a polling source keeps serving records that arrive later, so Done can pass it, and a consumer reporting progress must clamp (the phase tracker does). It is deliberately NOT computed when the executor is built — callers size caches off a known total at construction, and a database round-trip there would change that behaviour for every DB-fed scan.

func (*DBInputSource) IdleFor 

func (s *DBInputSource) IdleFor() time.Duration

IdleFor returns how long it has been since the source last observed a new row (or since creation, if none have arrived yet). Useful for status reporting to tell the user "scan is idle — server is still listening for more records."

func (*DBInputSource) Next 

func (s *DBInputSource) Next(ctx context.Context) (*work.WorkItem, error)

Next returns the next record after the scan cursor as a WorkItem. It blocks (polling) until a record is available, the context is cancelled, or the source is closed.

func (*DBInputSource) WithHostScopes 

func (s *DBInputSource) WithHostScopes(hosts []HostTarget) *DBInputSource

WithHostScopes sets an in-scope origin filter so only records matching these (scheme, hostname, port) origins are returned. This keeps HTTP records from unrelated origins — including the same host on a different port left over from a previous scan (e.g. localhost:8080 when targeting localhost:3000) — out of the scan.

func (*DBInputSource) WithIdleTimeout 

func (s *DBInputSource) WithIdleTimeout(timeout time.Duration) *DBInputSource

WithIdleTimeout configures the source to return io.EOF after this long without any new rows arriving from the database. Only honored in polling mode (not oneShot). Zero (the default) keeps the original behavior — poll forever. Typical use: one-shot scan-on-receive where the caller wants the scan to terminate once the ingestion pipeline has settled.

func (*DBInputSource) WithIncludeSources 

func (s *DBInputSource) WithIncludeSources(sources []string) *DBInputSource

WithIncludeSources restricts the source to records whose http_records.source value is in the given list. Used by scan-on-receive shallow mode to exclude scanner-produced artefacts (source="finding", source="scanner", etc.) so the scan only processes user-ingested traffic and doesn't fan out across records that were created as byproducts of scanning itself. An empty slice disables the filter (default).

func (*DBInputSource) WithOnActivity 

func (s *DBInputSource) WithOnActivity(fn func(recordCount int, idleFor time.Duration, firstBatch bool)) *DBInputSource

WithOnActivity registers a callback invoked once per non-empty fetch batch. firstBatch is true for the very first batch ever returned by this source. idleFor is the duration the source had been quiet *before* this batch landed. Callers typically use this to print a "scan started / resumed" line so the user gets immediate feedback rather than waiting for the next status tick.

func (*DBInputSource) WithPageSize 

func (s *DBInputSource) WithPageSize(pageSize int) *DBInputSource

WithPageSize sets the number of records fetched from the database per batch.

type DatabaseInfo 

type DatabaseInfo struct {
	Path    string `json:"path"`
	Driver  string `json:"driver"`
	Version string `json:"version,omitempty"`
	Size    int64  `json:"size"`
}

DatabaseInfo holds database metadata

type DatabaseStats 

type DatabaseStats struct {
	Database     DatabaseInfo        `json:"database"`
	Records      RecordCounts        `json:"records"`
	Findings     FindingsStats       `json:"findings"`
	HTTPMethods  map[string]int64    `json:"http_methods"`
	StatusCodes  StatusCodeBreakdown `json:"status_codes"`
	Performance  PerformanceMetrics  `json:"performance"`
	DateRange    DateRangeStats      `json:"date_range"`
	ScanSessions []ScanSessionStats  `json:"scan_sessions,omitempty"`
	TopHosts     []HostStats         `json:"top_hosts,omitempty"`
}

DatabaseStats holds database statistics

type DateRangeStats 

type DateRangeStats struct {
	FirstRequest time.Time     `json:"first_request"`
	LastRequest  time.Time     `json:"last_request"`
	Duration     time.Duration `json:"duration"`
}

DateRangeStats holds date range information

type DeleteBuilder 

type DeleteBuilder struct {
	// contains filtered or unexported fields
}

DeleteBuilder builds delete queries with filters

func NewDeleteBuilder 

func NewDeleteBuilder(db *DB, filters QueryFilters) *DeleteBuilder

NewDeleteBuilder creates a new delete builder

func (*DeleteBuilder) DeleteAllTables 

func (db *DeleteBuilder) DeleteAllTables(ctx context.Context, dryRun bool) (map[string]int64, error)

DeleteAllTables deletes all data from every data table.

func (*DeleteBuilder) DeleteFindings 

func (db *DeleteBuilder) DeleteFindings(ctx context.Context, dryRun bool) (int64, error)

DeleteFindings deletes findings matching the configured filters. If dryRun is true, returns the count without deleting.

func (*DeleteBuilder) DeleteOrphans 

func (db *DeleteBuilder) DeleteOrphans(ctx context.Context, dryRun bool) (int64, error)

DeleteOrphans deletes orphaned findings (findings where none of the http_record_uuids exist in http_records).

It honors filters.ProjectUUID. Orphan cleanup reads as store maintenance, but the rows it removes belong to projects: an unscoped sweep run from one engagement deleted another engagement's findings, which is the same cross-project leak the scoped delete paths exist to prevent. With no project filter set the behavior is unchanged and the whole store is swept.

func (*DeleteBuilder) DeleteRecords 

func (db *DeleteBuilder) DeleteRecords(ctx context.Context, dryRun bool) (int64, error)

DeleteRecords deletes the HTTP records matching filters, along with the findings whose only evidence they were.

The order below is the policy, and it is not the obvious one. Deleting every finding that merely *touches* a deleted record destroys findings that also cite records the caller is keeping — one module reporting the same issue across five endpoints would lose the whole finding because one endpoint was pruned. So the junction is narrowed first, and a finding is removed only once nothing it links to survives:

  1. collect the findings linked to the doomed records (bounded by findings touched, not by records deleted);
  2. drop those records' junction rows, so the junction describes surviving evidence only;
  3. delete the records;
  4. delete the collected findings that now link to nothing.

Step 4 is scoped to the collected candidates rather than sweeping every evidence-less finding, because a finding with no HTTP record is normal — audit and source-scan findings never have one, and this is not the command that removes them (see DeleteOrphans).

Every IN() list is chunked (see SQLChunkSize). The filters can select an unbounded number of records, and bun renders a list as SQL literals rather than bound parameters — so an unchunked delete builds one statement holding every uuid as text, tens of megabytes of it on a large corpus.

func (*DeleteBuilder) DeleteTable 

func (db *DeleteBuilder) DeleteTable(ctx context.Context, tableName string, dryRun bool) (int64, error)

DeleteTable deletes all rows from a specific table.

type DeparosStatusPolicy 

type DeparosStatusPolicy struct {
	// KeepOnePerHost statuses collapse to a single representative per
	// (hostname, status_code) — the shortest path. Use where "this host has one"
	// is enough signal: authentication/authorization walls (401/403) and pacing
	// (429). Route identity across paths is intentionally NOT preserved here — an
	// authz wall is a host-level fact, and keeping every 403 would reintroduce
	// catch-all floods.
	KeepOnePerHost []int
	// KeepPerPath statuses preserve every DISTINCT path (only exact-duplicate
	// paths collapse), capped at PerPathCap records per (hostname, status_code) to
	// bound pathological floods. Use for "endpoint exists" evidence where the path
	// itself is the attack surface: 405/415/422 mean the route is real but the
	// method/payload was wrong, so dynamic assessment should retry it with the
	// right method. Routers only emit these for matched routes, so fuzz-flood risk
	// is low — the cap is a backstop, not the primary control.
	KeepPerPath []int
	// PerPathCap bounds the KeepPerPath tier per (hostname, status_code). <=0
	// disables the cap (keep all distinct paths).
	PerPathCap int
}

DeparosStatusPolicy controls how discovery client-error (4xx) records are retained. It replaces a blanket "drop all 4xx" with priority tiers so that distinct route identities carrying real attack-surface signal survive, while fuzz noise (nonexistent paths → 404/400) is still pruned. Statuses in neither tier are dropped entirely.

type EmbeddedParam 

type EmbeddedParam struct {
	Name       string `json:"name"`
	Value      string `json:"value,omitempty"`
	Type       string `json:"type"`
	NameStart  int    `json:"name_start,omitempty"`
	NameEnd    int    `json:"name_end,omitempty"`
	ValueStart int    `json:"value_start,omitempty"`
	ValueEnd   int    `json:"value_end,omitempty"`
	Metadata   string `json:"metadata,omitempty"`
	// Hidden marks a parameter no observed request carried: the param-mining
	// phase found the endpoint acting on it (see pkg/parammine).
	Hidden bool `json:"hidden,omitempty"`
}

EmbeddedParam represents a parameter stored as JSON within HTTPRecord

type EndpointAuth added in v0.5.3 

type EndpointAuth struct {
	Authenticated int64 `json:"authenticated"`
	Anonymous     int64 `json:"anonymous"`
}

EndpointAuth counts a endpoint's records with and without credentials.

type EndpointOptions added in v0.5.3 

type EndpointOptions struct {
	// MinRecords drops endpoints seen fewer times.
	MinRecords int64
	// Sort is one of EndpointSorts; "" means records.
	Sort string
	// Collapse, when >= 2, merges sibling literal segments that share an
	// identical set of child templates once at least Collapse of them do
	// (/blog/a, /blog/b, … → /blog/{var}). 0 disables it: it can merge real
	// routes, so it is opt-in.
	Collapse int
	// Limit/Offset page the endpoint list; Limit 0 returns every endpoint.
	Limit, Offset int
	// SkipFindings leaves the finding counts out (no findings pass).
	SkipFindings bool
	// MaxEndpoints overrides DefaultMaxEndpoints (0 = default).
	MaxEndpoints int
}

EndpointOptions shapes an Endpoints call.

type EndpointResult added in v0.5.3 

type EndpointResult struct {
	// Endpoints is the requested page.
	Endpoints []EndpointView
	// Total counts every endpoint that passed MinRecords, whatever the page.
	Total int64
	// RecordsScanned is how many records the filters matched.
	RecordsScanned int64
	// Capped reports the distinct-endpoint cap was reached; UnbucketedRecords
	// were not attributed to any endpoint.
	Capped            bool
	UnbucketedRecords int64
}

EndpointResult is a bounded endpoint inventory.

type EndpointView added in v0.5.3 

type EndpointView struct {
	ID       string   `json:"id"`
	Origin   string   `json:"origin"`
	Host     string   `json:"host"`
	Template string   `json:"template"`
	Methods  []string `json:"methods"`

	Records       int64    `json:"records"`
	DistinctPaths int64    `json:"distinct_paths"`
	PathsCapped   bool     `json:"distinct_paths_capped,omitempty"`
	SamplePaths   []string `json:"sample_paths"`

	// Params lists parameter names by location (query, body, json, cookie,
	// xml, multipart, path), each capped at 50.
	Params     map[string][]string `json:"params"`
	ParamCount int                 `json:"param_count"`

	RequestContentTypes  []string `json:"request_content_types"`
	ResponseContentTypes []string `json:"response_content_types"`

	Status    map[string]int64 `json:"status"`
	TopStatus int              `json:"top_status"`
	Auth      EndpointAuth     `json:"auth"`

	// Sources are the record sources (scanner, deparos, jstangle, ingest-*…);
	// JSOutcomes are the jstangle:* outcome remarks (found, not_found,
	// withheld, …) of JS-derived records, so a route known only from code is
	// visible as such.
	Sources    []string `json:"sources"`
	JSOutcomes []string `json:"js_outcomes,omitempty"`

	// Findings counts live findings (not false_positive / accepted_risk) whose
	// URL maps onto this endpoint.
	Findings           int64            `json:"findings"`
	FindingsBySeverity map[string]int64 `json:"findings_by_severity,omitempty"`
	MaxSeverity        string           `json:"max_severity,omitempty"`

	FirstSeen string `json:"first_seen"`
	LastSeen  string `json:"last_seen"`
}

EndpointView is one endpoint: an origin plus a path template, with every method seen on it.

type Finding 

type Finding struct {
	bun.BaseModel `bun:"table:findings,alias:f" json:"-"`

	ID          int64  `bun:"id,pk,autoincrement" json:"id"`
	ProjectUUID string `bun:"project_uuid,notnull" json:"project_uuid"`

	// UUID references to HTTPRecords (not FK, allows flexibility)
	HTTPRecordUUIDs []string `bun:"http_record_uuids,type:jsonb,notnull" json:"http_record_uuids"`

	// Scan reference (soft, no FK)
	ScanUUID        string `bun:"scan_uuid,nullzero" json:"scan_uuid,omitempty"`
	AgenticScanUUID string `bun:"agentic_scan_uuid,nullzero" json:"agentic_scan_uuid,omitempty"` // link to agentic_scan that produced this finding

	// Denormalized target info (for fast display/filtering without joining)
	URL      string `bun:"url,nullzero" json:"url,omitempty"`
	Hostname string `bun:"hostname,nullzero" json:"hostname,omitempty"`

	// Module info
	ModuleID      string   `bun:"module_id,notnull" json:"module_id"`
	ModuleName    string   `bun:"module_name,notnull" json:"module_name"`
	ModuleType    string   `bun:"module_type,nullzero" json:"module_type,omitempty"`
	FindingSource string   `bun:"finding_source,nullzero" json:"finding_source,omitempty"`
	RecordKind    string   `bun:"record_kind,notnull,default:'finding'" json:"record_kind,omitempty"`
	EvidenceGrade string   `bun:"evidence_grade,nullzero" json:"evidence_grade,omitempty"`
	ModuleShort   string   `bun:"module_short,nullzero" json:"module_short,omitempty"`
	Description   string   `bun:"description,nullzero" json:"description,omitempty"`
	Severity      string   `bun:"severity,notnull" json:"severity"`
	Confidence    string   `bun:"confidence,notnull,default:'firm'" json:"confidence"`
	Tags          []string `bun:"tags,type:jsonb,nullzero" json:"tags,omitempty"`

	// Finding lifecycle
	Status      string `bun:"status,nullzero,default:'triaged'" json:"status,omitempty"` // one of FindingStatuses
	Remediation string `bun:"remediation,nullzero" json:"remediation,omitempty"`
	// Provenance of the current status: why, by whom, when. The full history is
	// finding_status_events.
	StatusNote      string     `bun:"status_note,nullzero" json:"status_note,omitempty"`
	StatusActor     string     `bun:"status_actor,nullzero" json:"status_actor,omitempty"`
	StatusUpdatedAt *time.Time `bun:"status_updated_at,nullzero" json:"status_updated_at,omitempty"`

	// Classification
	CWEID     string  `bun:"cwe_id,nullzero" json:"cwe_id,omitempty"`
	CVSSScore float64 `bun:"cvss_score,default:0" json:"cvss_score"`

	// Source info (for audit findings)
	SourceFile string `bun:"source_file,nullzero" json:"source_file,omitempty"`
	RepoName   string `bun:"repo_name,nullzero" json:"repo_name,omitempty"` // repository name or URL for audit findings

	MatchedAt          []string `bun:"matched_at,type:jsonb,nullzero" json:"matched_at,omitempty"`
	ExtractedResults   []string `bun:"extracted_results,type:jsonb,nullzero" json:"extracted_results,omitempty"`
	AdditionalEvidence []string `bun:"additional_evidence,type:jsonb,nullzero" json:"additional_evidence,omitempty"`

	Request  string `bun:"request,nullzero" json:"request,omitempty"`
	Response string `bun:"response,nullzero" json:"response,omitempty"`

	// Replay context, for re-verifying the finding: the baseline record the
	// module scanned (as opposed to the evidence records in HTTPRecordUUIDs),
	// the insertion point it mutated and the payload it sent. Empty on findings
	// written before schema v3 or by producers that do not know them.
	OriginRecordUUID string                 `bun:"origin_record_uuid,nullzero" json:"origin_record_uuid,omitempty"`
	InsertionPoint   *FindingInsertionPoint `bun:"insertion_point,type:jsonb,nullzero" json:"insertion_point,omitempty"`
	Payload          string                 `bun:"payload,nullzero" json:"payload,omitempty"`

	// Verification is the latest re-verification result, separate from Status.
	// VerifiedAt duplicates its timestamp as a column for sorting and filtering.
	Verification *FindingVerification `bun:"verification,type:jsonb,nullzero" json:"verification,omitempty"`
	VerifiedAt   *time.Time           `bun:"verified_at,nullzero" json:"verified_at,omitempty"`

	FindingHash string `bun:"finding_hash,notnull" json:"finding_hash"`

	FoundAt   time.Time `bun:"found_at,notnull,default:current_timestamp" json:"found_at"`
	CreatedAt time.Time `bun:"created_at,notnull,default:current_timestamp" json:"created_at"`
}

Finding represents a vulnerability finding (no FK, soft UUID reference)

func (*Finding) FromResultEvent 

func (f *Finding) FromResultEvent(event *output.ResultEvent) error

FromResultEvent converts output.ResultEvent to Finding

type FindingInsertionPoint added in v0.5.3 

type FindingInsertionPoint struct {
	Type string `json:"type"`
	Name string `json:"name,omitempty"`
}

FindingInsertionPoint names the insertion point a module mutated, e.g. {type: "URL_PARAM", name: "id"}. Type uses the httpmsg insertion-point type names.

type FindingObservation added in v0.5.3 

type FindingObservation struct {
	bun.BaseModel `bun:"table:finding_observations,alias:fo" json:"-"`

	ProjectUUID string    `bun:"project_uuid,notnull" json:"project_uuid"`
	FindingID   int64     `bun:"finding_id,pk" json:"finding_id"`
	FindingHash string    `bun:"finding_hash,notnull" json:"finding_hash"`
	ScanUUID    string    `bun:"scan_uuid,pk" json:"scan_uuid"`
	ObservedAt  time.Time `bun:"observed_at,notnull,default:current_timestamp" json:"observed_at"`
}

FindingObservation records that a scan observed a finding. One row per (finding, scan); see the finding_observations DDL for why findings.scan_uuid alone cannot answer it.

type FindingSaveResult 

type FindingSaveResult struct {
	Inserted bool  // a new finding row was written (false on a dedup-append to an existing finding)
	Err      error // non-nil if this finding failed to save
}

FindingSaveResult reports the outcome of persisting one finding in a batched direct save, aligned by index with the input slice passed to SaveFindingsDirectBatch.

type FindingStatusChange added in v0.5.3 

type FindingStatusChange struct {
	ID          int64  `json:"id"`
	FindingHash string `json:"finding_hash"`
	From        string `json:"from"`
	To          string `json:"to"`
}

FindingStatusChange is one finding a status write touched (or would have).

type FindingStatusEvent added in v0.5.3 

type FindingStatusEvent struct {
	bun.BaseModel `bun:"table:finding_status_events,alias:fse" json:"-"`

	ID          int64     `bun:"id,pk,autoincrement" json:"id"`
	FindingID   int64     `bun:"finding_id,notnull" json:"finding_id"`
	ProjectUUID string    `bun:"project_uuid,notnull" json:"project_uuid"`
	FromStatus  string    `bun:"from_status,nullzero" json:"from_status,omitempty"`
	ToStatus    string    `bun:"to_status,notnull" json:"to_status"`
	Note        string    `bun:"note,nullzero" json:"note,omitempty"`
	Actor       string    `bun:"actor,nullzero" json:"actor,omitempty"`
	CreatedAt   time.Time `bun:"created_at,notnull,default:current_timestamp" json:"created_at"`
}

FindingStatusEvent is one entry of a finding's append-only status history.

type FindingStatusResult added in v0.5.3 

type FindingStatusResult struct {
	Changed   []FindingStatusChange
	Unchanged []FindingStatusChange
	NotFound  []string
}

FindingStatusResult reports a status write. Changed findings moved to the new status (and got an event row); Unchanged already had it and were left alone, note included; NotFound lists the selectors (an id as its decimal string, a hash as given) that matched nothing in scope.

type FindingStatusUpdate added in v0.5.3 

type FindingStatusUpdate struct {
	IDs    []int64
	Hashes []string
	Status string
	// Note says why (stored as findings.status_note and on the event row). A
	// new verdict replaces the previous note, so an empty Note clears it.
	Note string
	// Actor names who decided: a person, "agent-triage", "api", ….
	Actor string
}

FindingStatusUpdate names the findings to move and the verdict to give them. IDs and Hashes may both be set; a finding named twice is updated once. Hashes are full finding_hash values or prefixes of at least MinFindingHashPrefix hex characters.

type FindingVerification added in v0.5.3 

type FindingVerification struct {
	Verdict       string    `json:"verdict"`
	At            time.Time `json:"at"`
	ModuleVersion string    `json:"module_version,omitempty"`
	ScanUUID      string    `json:"scan_uuid,omitempty"`
	Reason        string    `json:"reason,omitempty"`
}

FindingVerification is the result of re-running a finding's module against its target (`finding verify`).

type FindingWrite 

type FindingWrite struct {
	Event           *output.ResultEvent
	HTTPRecordUUIDs []string
	ScanUUID        string
	ProjectUUID     string
}

FindingWrite bundles the arguments needed to persist a single finding, allowing a batch of findings to be coalesced into one transaction. See Repository.SaveFindingsBatch.

type FindingWriter 

type FindingWriter struct {
	// contains filtered or unexported fields
}

FindingWriter decouples finding persistence from scan workers. SaveFinding is a multi-statement, dedup-aware operation; calling it synchronously on every worker blocks the worker on a database round-trip. FindingWriter funnels findings through a single background goroutine that coalesces them into batch transactions (one fsync for many findings) via Repository.SaveFindingsBatch.

Findings are low-volume relative to HTTP records, so a single writer goroutine keeps up while also eliminating write contention between concurrent finding transactions. This mirrors RecordWriter's lifecycle (buffered channel, background flush, bounded shutdown drain) but is fire-and-forget: the result UUID is never needed by the caller, so Save does not block on persistence.

func NewFindingWriter 

func NewFindingWriter(repo *Repository, cfg FindingWriterConfig) *FindingWriter

NewFindingWriter creates and starts a FindingWriter. Call Close() to flush remaining findings and stop the background goroutine.

func (*FindingWriter) Close 

func (w *FindingWriter) Close()

Close stops accepting new findings, flushes the buffer, and returns. After Close, Save persists synchronously. Prefer Shutdown when the caller can report what happened; Close is the fire-and-forget form kept for existing call sites.

func (*FindingWriter) Metrics 

func (w *FindingWriter) Metrics() FindingWriterMetrics

Metrics returns a snapshot of the writer's counters.

func (*FindingWriter) Save 

func (w *FindingWriter) Save(ctx context.Context, event *output.ResultEvent, recordUUIDs []string, scanUUID, projectUUID string) error

Save enqueues a finding for batched persistence. It does not block on the database: if the buffer is full or the writer is shutting down, the finding is persisted synchronously so nothing is dropped. Safe for concurrent callers.

func (*FindingWriter) Shutdown added in v0.5.2 

func (w *FindingWriter) Shutdown() PersistenceOutcome

Shutdown stops accepting new findings, drains the buffer, and reports what became of everything it accepted.

One absolute deadline (now+FlushTimeout) governs the whole shutdown and is published before the drain is woken, so the drain's flushes and the wait for the goroutine to exit share a single budget instead of each starting its own.

Idempotent: the outcome is computed once and every later or concurrent caller gets that same value, so two phases closing the same writer cannot report the same findings twice. A nil writer is a no-op reporting a zero (clean) outcome — see RecordWriter.Shutdown for why that lives here rather than at every call site.

type FindingWriterConfig 

type FindingWriterConfig struct {
	// BufferSize is the channel capacity. When full, Save persists inline rather
	// than blocking the scan worker. Default: 1024.
	BufferSize int

	// BatchSize is the maximum number of findings coalesced into one transaction.
	// Default: 64.
	BatchSize int

	// FlushInterval is the maximum time a finding waits in the buffer before
	// being flushed, even if the batch isn't full. Default: 50ms.
	FlushInterval time.Duration

	// FlushTimeout bounds the shutdown drain so a wedged database can't block
	// Close() forever. Steady-state flushes are bounded by OperationTimeout
	// instead, and remain immune to scan cancellation. Default: 2m.
	FlushTimeout time.Duration

	// OperationTimeout bounds ONE steady-state flush. Steady flushes deliberately
	// do not inherit the scan's context — cancelling a scan mid-transaction would
	// abort findings already pulled off the channel — but "not cancellable" was
	// implemented as context.Background(), i.e. no bound at all: a wedged database
	// parked the writer goroutine for the rest of the process. This is the bound,
	// set far above SQLite's 60s busy_timeout so a merely busy database still
	// succeeds. Default: 2m.
	OperationTimeout time.Duration
	// contains filtered or unexported fields
}

FindingWriterConfig configures the batching behavior of FindingWriter.

type FindingWriterMetrics 

type FindingWriterMetrics struct {
	Enqueued     int64 // findings handed to the background writer
	Written      int64 // findings the background writer persisted (or deduped onto an existing row)
	Inline       int64 // findings persisted synchronously (buffer full / shutting down)
	InlineFailed int64 // inline saves that returned an error
	Errors       int64 // findings that individually failed to persist in a flush
}

FindingWriterMetrics exposes counters for monitoring.

Written and Errors are exact per finding, not per batch: a flush asks for the per-finding outcomes and counts each one. They used to be charged by batch size on any batch error, so one unconvertible finding reported the whole batch as dropped and Written+Errors could exceed Enqueued.

type FindingsQueryBuilder 

type FindingsQueryBuilder struct {
	// contains filtered or unexported fields
}

FindingsQueryBuilder builds filtered queries for the findings table

func NewFindingsQueryBuilder 

func NewFindingsQueryBuilder(db *DB, filters QueryFilters) *FindingsQueryBuilder

NewFindingsQueryBuilder creates a new findings query builder

func (*FindingsQueryBuilder) Count 

func (fqb *FindingsQueryBuilder) Count(ctx context.Context) (int64, error)

Count returns total number of matching findings

func (*FindingsQueryBuilder) Execute 

func (fqb *FindingsQueryBuilder) Execute(ctx context.Context) ([]*Finding, error)

Execute executes the query and returns findings

func (*FindingsQueryBuilder) ExecuteWithCount 

func (fqb *FindingsQueryBuilder) ExecuteWithCount(ctx context.Context) ([]*Finding, int64, error)

ExecuteWithCount runs the filtered query and returns matching findings alongside the total unfiltered count, in a single round-trip via Bun's ScanAndCount. Use this when callers need both the page and the total (paginated views) — saves one DB roundtrip vs. Execute + Count.

type FindingsStats 

type FindingsStats struct {
	BySeverity map[string]int64 `json:"by_severity"`
	Total      int64            `json:"total"`
}

FindingsStats holds finding statistics by severity

type GenericFilter 

type GenericFilter struct {
	Column   string
	Operator string // "eq", "like", "gt", "lt", "gte", "lte", "in", "neq"
	Value    string
}

GenericFilter represents a single column filter.

type GenericQueryOptions 

type GenericQueryOptions struct {
	Limit      int
	Offset     int
	SortBy     string // Column to sort by
	SortAsc    bool
	Columns    []string // Column whitelist (empty = all)
	SearchTerm string   // Fuzzy search across text columns
	Truncate   int      // Truncate []byte fields to this length (0 = no truncation)
	Filters    []GenericFilter
}

GenericQueryOptions holds options for filtered, sorted, paginated generic queries.

type GroupFindingOptions 

type GroupFindingOptions struct {
	// PerHost groups within each hostname — a value seen on two hosts stays two
	// findings. When false, grouping is project-wide regardless of host.
	PerHost bool
	// Tags, when non-empty, restricts grouping to findings carrying at least one
	// matching tag (case-insensitive).
	Tags []string
	// ByModule lists module IDs whose findings collapse to a single finding per
	// (module, severity[, host]) regardless of the per-URL extracted value — for
	// modules that fire once per asset (e.g. sourcemap-detect, one .map filename
	// per JS bundle). These bypass the value-identity requirement and the Tags
	// gate; module + severity is the guardrail.
	ByModule []string
	// ByRule lists module IDs whose findings collapse per (module, rule_name,
	// severity[, host]) regardless of value — like ByModule, but the rule
	// (module_name) stays part of the key, so DISTINCT rules remain distinct
	// findings while repeats of the SAME rule fold together. This is for
	// secret-detect, whose one module_id carries many rule_names (secret-scan
	// rules): a "Looker Client ID" rule matching every chunk hash in a minified
	// bundle's content-hash map yields dozens of identical findings differing only
	// by value, which collapse to one finding (all values unioned on), while a
	// genuinely different secret — an AWS key, a Slack token — keeps its own row.
	ByRule []string
	// BundleSuspect lists module IDs whose GENERIC Suspect-severity findings
	// collapse by (module, severity[, host]) — the rule dropped from the key — so
	// the family-less low-signal noise on a host folds into ONE bundle. Only
	// findings carrying output.SuspectBundleTag qualify (secret-detect sets it on
	// its generic-namespace rules); a NAMED provider family at Suspect severity is
	// left untagged and stays per-rule via ByRule, so distinct families are never
	// merged into one rollup. The module's higher-severity findings also stay
	// per-rule. The bundle survivor's module_name is relabeled to
	// suspectBundleFindingName and its description to suspectBundleDescription so it
	// isn't mislabeled with just the first rule's name/text. Default: secret-detect.
	BundleSuspect []string
	// MaxURLs caps the merged matched-URL list on the survivor (0 = unlimited).
	MaxURLs int
	// Hostnames, when non-empty, scopes the pass to findings on those hosts. The
	// dynamic-assessment feedback loop sets it to the hosts being scanned so a
	// growing findings table isn't loaded in full each round. Empty = project-wide.
	Hostnames []string
}

GroupFindingOptions configures value-based finding grouping (see GroupFindingsByValue).

type HTTPRecord 

type HTTPRecord struct {
	bun.BaseModel `bun:"table:http_records,alias:r" json:"-"`

	UUID        string `bun:"uuid,pk,notnull" json:"uuid"`
	ProjectUUID string `bun:"project_uuid,notnull" json:"project_uuid"`
	ScanUUID    string `bun:"scan_uuid,nullzero" json:"scan_uuid,omitempty"` // link to scan that produced this record

	// Host info (embedded, replaces hosts table)
	Scheme   string `bun:"scheme,notnull" json:"scheme"`
	Hostname string `bun:"hostname,notnull" json:"hostname"`
	Port     int    `bun:"port,notnull" json:"port"`
	IP       string `bun:"ip,nullzero" json:"ip,omitempty"` // resolved IP address the request went to (httpx: host_ip)

	// Request fields
	Method               string `bun:"method,notnull" json:"method"`
	Path                 string `bun:"path,notnull" json:"path"`
	URL                  string `bun:"url,notnull" json:"url"`
	HTTPVersion          string `bun:"http_version,notnull" json:"http_version"`
	RequestContentType   string `bun:"request_content_type,nullzero" json:"request_content_type,omitempty"`
	RequestContentLength int64  `bun:"request_content_length,default:0" json:"request_content_length"`
	RawRequest           []byte `bun:"raw_request,type:bytea,nullzero" json:"raw_request,omitempty"`
	RequestHash          string `bun:"request_hash,notnull" json:"request_hash"`
	RequestAuthorization string `bun:"request_authorization,nullzero" json:"request_authorization,omitempty"`

	// Response fields
	StatusCode            int    `bun:"status_code,default:0" json:"status_code"`
	StatusPhrase          string `bun:"status_phrase,nullzero" json:"status_phrase,omitempty"`
	ResponseHTTPVersion   string `bun:"response_http_version,nullzero" json:"response_http_version,omitempty"`
	ResponseContentType   string `bun:"response_content_type,nullzero" json:"response_content_type,omitempty"`
	ResponseContentLength int64  `bun:"response_content_length,default:0" json:"response_content_length"`
	RawResponse           []byte `bun:"raw_response,type:bytea,nullzero" json:"raw_response,omitempty"`
	ResponseHash          string `bun:"response_hash,nullzero" json:"response_hash,omitempty"`
	ResponseNormHash      string `bun:"response_norm_hash,nullzero" json:"response_norm_hash,omitempty"` // hash of the body with reflected URL/path + dynamic runs stripped, for reflected-URL-robust dedup
	// ResponseTimeMs is 0 when the exchange was never timed, which is not the
	// same as "it took no time". Only the requester measures, so every record
	// rebuilt from stored bytes — and every intermediate redirect hop, which
	// the transport followed inside one measurement — has nothing to report.
	//
	// omitempty so an unmeasured row omits the field entirely. A silent zero is
	// worse than an absent one: a consumer averaging over a chain gets a number
	// that is wrong rather than one it knows it cannot compute.
	ResponseTimeMs int64  `bun:"response_time_ms,default:0" json:"response_time_ms,omitempty"`
	ResponseWords  int64  `bun:"response_words,default:0" json:"response_words"`
	HasResponse    bool   `bun:"has_response,notnull,default:false" json:"has_response"`
	ResponseTitle  string `bun:"response_title,nullzero" json:"response_title,omitempty"`

	// ResponseLocation is the Location header of a 3xx response, stored verbatim
	// — relative values included, because rewriting them to absolute would be
	// this column asserting something the server did not say.
	//
	// It exists because the redirect tree needs exactly one header out of a
	// response whose body it has no use for. Without a column the destination is
	// only recoverable by reading raw_response back, which on a `--glob-db` read
	// means reopening every source file the selected redirects came from and
	// holding an 8 KiB prefix per row until rendering finishes. Parsing it once at
	// ingestion and storing ~30 bytes replaces all of that.
	//
	// Empty means "no Location", which for a record written by this version is
	// also the answer for a non-redirect. It does NOT distinguish "not backfilled"
	// on a record written before this column existed — see
	// redirectLocationsAreBackfilled for how the read path handles that.
	ResponseLocation string `bun:"response_location,nullzero" json:"response_location,omitempty"`

	// Parameters (JSON array, replaces http_parameters table)
	Parameters []EmbeddedParam `bun:"parameters,type:jsonb,nullzero" json:"parameters,omitempty"`

	// Timestamps
	SentAt     time.Time `bun:"sent_at,notnull,default:current_timestamp" json:"sent_at"`
	ReceivedAt time.Time `bun:"received_at,nullzero" json:"received_at,omitempty"`
	CreatedAt  time.Time `bun:"created_at,notnull,default:current_timestamp" json:"created_at"`

	Source string `bun:"source,nullzero,default:''" json:"source,omitempty"`

	// Analysis & enrichment
	Technology      []string `bun:"technology,type:jsonb,nullzero" json:"technology,omitempty"`     // detected technologies/frameworks
	ContentHash     string   `bun:"content_hash,nullzero" json:"content_hash,omitempty"`            // hash of meaningful response content for change detection
	IsAuthenticated bool     `bun:"is_authenticated,notnull,default:false" json:"is_authenticated"` // whether request was sent with valid auth
	ParentUUID      string   `bun:"parent_uuid,nullzero" json:"parent_uuid,omitempty"`              // parent record UUID (crawl/spider parent)

	// Target is the input line this record's chain started from, as it appears
	// in the run's target list (normalized to an absolute URL). Empty for
	// records that did not come from a submitted target — crawl discoveries,
	// ingest.
	//
	// It exists because hostname is THIS HOP's hostname, not the one that was
	// asked about. A followed redirect that leaves the host produces records
	// under names nobody submitted, and a consumer keying on hostname then
	// reports a host it invented. Recovering the input otherwise means walking
	// parent_uuid to the root, which no single record can answer and which a
	// truncated chain can break outright.
	Target string `bun:"target,nullzero" json:"target,omitempty"`
	// RootUUID is the first record of this record's redirect chain — itself for
	// a root. Redundant with walking ParentUUID, and deliberately so: the walk
	// is O(chain) and needs the whole stream, this is O(1) on one record and
	// survives the middle of a chain being dropped by the storage cap.
	RootUUID string `bun:"root_uuid,nullzero" json:"root_uuid,omitempty"`
	// ChainTruncated marks the last stored row of a redirect chain that did not
	// reach its destination — the follow cap was hit, or the storage cap
	// dropped hops after this one.
	//
	// Without it a 3xx at the end of a chain is ambiguous in a way that reads
	// as an answer: "this redirect pointed somewhere that never replied" and
	// "we stopped walking" produce an identical row, and the obvious consumer
	// reading ("the terminal hop is the row nothing claims as a parent") takes
	// the 3xx as final in both cases.
	ChainTruncated bool `bun:"chain_truncated,notnull,default:false" json:"chain_truncated,omitempty"`

	// Risk labeling (populated by background analysis)
	Remarks   []string `bun:"remarks,type:jsonb,nullzero" json:"remarks,omitempty"`
	RiskScore int      `bun:"risk_score,default:0" json:"risk_score"`

	// SurfaceScore is the deterministic attack-surface score (0-100) written by
	// the surface_scoring passive module, which owns the signal set and the
	// scale — see pkg/modules/passive/surface_scoring. It
	// answers "is this record worth attacking" from properties of the record
	// itself, and is reproducible across scans.
	//
	// Deliberately NOT merged into RiskScore: that column holds anomaly_ranking's
	// rank-within-batch percentile, which measures statistical rarity against
	// whichever records happened to be flushed alongside it. The two are different
	// scales from different inputs; sharing one column would mean whichever module
	// flushed last silently overwrote the other.
	SurfaceScore int `bun:"surface_score,default:0" json:"surface_score"`

	// BodyCompaction is set when `vigolium db compact` replaced a stored body
	// with its head, and records what the original was. Nil means RawRequest
	// and RawResponse hold the bodies exactly as captured. See BodyCompaction.
	BodyCompaction *BodyCompaction `bun:"body_compaction,type:jsonb,nullzero" json:"body_compaction,omitempty"`
}

HTTPRecord represents a single HTTP request/response record (denormalized)

func (*HTTPRecord) EnrichFromHttpRequestResponse 

func (r *HTTPRecord) EnrichFromHttpRequestResponse(ctx *httpmsg.HttpRequestResponse) error

EnrichFromHttpRequestResponse fills everything PrepareIdentity left out: the owned raw bytes, all response-derived columns, request authorization, parameters, resolved IP, and timestamps. Run it only for a record that is actually going to be stored. Safe to call on a record whose identity half has already run; it does not recompute it.

func (*HTTPRecord) FromHttpRequestResponse 

func (r *HTTPRecord) FromHttpRequestResponse(ctx *httpmsg.HttpRequestResponse) error

FromHttpRequestResponse populates an HTTPRecord from httpmsg.HttpRequestResponse.

The record OWNS its RawRequest/RawResponse: both are copied out of the source message rather than aliased. This is a lifetime requirement, not a style choice. A converted record routinely outlives the call that produced it — RecordWriter.admit hands it to a shard channel and returns, the flush goroutine inserts it later, and Repository.emitRecordSaved passes it to the filesystem mirror, which writes it from yet another goroutine. Meanwhile the executor builds its responses over a POOLED buffer (see pkg/core, getResponseBuffer) that processItem returns to the pool as soon as it is done with the item. Aliasing therefore let a record be persisted or mirrored with another response's bytes whenever the write outlived the caller — on scan cancellation the caller stops waiting while the record is still queued, which is exactly when this happened.

The copy (bytes.Clone below) costs one memcpy per converted record, which is small next to the two SHA-256 passes below, and is only paid for records that get enriched at all.

Conversion is split in two halves, and a caller that deduplicates should use them separately rather than calling this: PrepareIdentity fills only the columns that decide WHICH row this is, and EnrichFromHttpRequestResponse does the expensive response-derived work. See PrepareIdentity.

func (*HTTPRecord) MarshalJSON 

func (r *HTTPRecord) MarshalJSON() ([]byte, error)

MarshalJSON preserves the external JSON contract by injecting four derived fields (request_headers, request_body, response_headers, response_body) alongside the stored struct fields.

func (*HTTPRecord) ParsedRequest 

func (r *HTTPRecord) ParsedRequest() *httpmsg.HttpRequest

ParsedRequest parses RawRequest into an httpmsg.HttpRequest. Returns nil when there is no raw request. Each call constructs a new parser; callers that need multiple fields should hoist the result into a local, or use parsedView for the standard four-field expansion.

func (*HTTPRecord) ParsedResponse 

func (r *HTTPRecord) ParsedResponse() *httpmsg.HttpResponse

ParsedResponse parses RawResponse into an httpmsg.HttpResponse, or nil when no raw response is present.

func (*HTTPRecord) ParsedView 

func (r *HTTPRecord) ParsedView() (reqHdrs, respHdrs map[string][]string, reqBody, respBody []byte)

ParsedView decodes RawRequest and RawResponse exactly once each and returns the four derived fields. Use this whenever a caller needs more than one derived field — calling individual accessors for each field re-parses.

When raw bytes are absent (e.g., list endpoints exclude raw_* columns), returned values are nil — that is what makes the JSON projection in HandleListRecords drop the four derived fields.

func (*HTTPRecord) PrepareIdentity 

func (r *HTTPRecord) PrepareIdentity(ctx *httpmsg.HttpRequestResponse) error

PrepareIdentity fills the columns that establish a record's identity — the ones recordDedupKey builds its key from and findDuplicateRecordUUIDs matches on — and nothing else. Cheap: a URL parse, a few header reads, and one SHA-256 over the raw request.

It exists so a duplicate can be recognized WITHOUT paying for enrichment. EnrichFromHttpRequestResponse runs two more passes over the response body (a normalized-body-hash regex, a word count), tokenizes HTML to pull out a title, and copies the raw bytes. Profiling a 64 KiB HTML response put ~57% of conversion CPU in the normalizing regex alone, and the whole conversion at ~2.6ms and ~786 KiB allocated. On duplicate-heavy ingestion — discovery and spidering re-encountering the same URLs, which is the case the dedup cache was added for — all of that was spent on records thrown away microseconds later.

Deliberately NOT here: hostname resolution. The IP is stored metadata, not identity, and scheduling it from this half would spawn a background DNS goroutine for every duplicate too.

Callers must set Source (and ProjectUUID) before building a dedup key from the result: recordDedupKey consults both.

func (*HTTPRecord) RedirectLocation 

func (r *HTTPRecord) RedirectLocation() string

RedirectLocation returns where this record's response points, or "" when it is not a redirect and when the destination is not recoverable.

The stored column answers first. Parsing raw_response is the fallback for records written before response_location existed, whose column is empty whether or not the response carried a Location — the two are indistinguishable, so a legacy redirect with no Location costs one parse of bytes already in hand.

It lives here, beside the write path that fills the column, so the question "where does this redirect go" has ONE answer. The read path used to carry its own copy of the parse, which is exactly the drift this file exists to prevent.

func (*HTTPRecord) RequestBodyBytes 

func (r *HTTPRecord) RequestBodyBytes() []byte

RequestBodyBytes returns the request body bytes parsed from RawRequest.

func (*HTTPRecord) RequestHeadersMap 

func (r *HTTPRecord) RequestHeadersMap() map[string][]string

RequestHeadersMap returns request headers as a map[name][]values.

func (*HTTPRecord) ResponseBodyBytes 

func (r *HTTPRecord) ResponseBodyBytes() []byte

ResponseBodyBytes returns the response body bytes parsed from RawResponse.

func (*HTTPRecord) ResponseHeadersMap 

func (r *HTTPRecord) ResponseHeadersMap() map[string][]string

ResponseHeadersMap returns response headers as a map[name][]values.

type HostFacts 

type HostFacts struct {
	A     []string       `json:"a,omitempty"`
	AAAA  []string       `json:"aaaa,omitempty"`
	CNAME []string       `json:"cname,omitempty"`
	TLS   *tlsprobe.Info `json:"tls,omitempty"`
}

HostFacts are the per-host observations a run collects alongside its records: the full DNS answer and, under --tls-probe, the TLS handshake and leaf certificate.

They are NOT columns, and not attached by HTTPRecord.MarshalJSON. Both decisions follow from the same fact — these describe the HOST, not the record:

  • Storing them per row would put many identical copies of one answer in the database (a sweep writes several records per host; a redirect chain alone is three), and they are stale the moment their TTL passes. Only the single `ip` column is persisted.
  • Attaching them inside MarshalJSON would decorate every record that every caller in the process serializes from a global cache. The ingest server marshals `[]*HTTPRecord` straight to the HTTP response, so one project's rows would carry a name resolved at an unknown time while ingesting another's.

So the emit sites that KNOW this process did the probing opt in, and everyone else keeps serializing a record as itself.

func LookupHostFacts 

func LookupHostFacts(hostname string, port int) HostFacts

LookupHostFacts returns what this PROCESS observed for a record's host, or zero facts when it observed nothing (a run without --tls-probe, a plaintext endpoint, or any process that did no probing — an `export` of an old database, for instance).

func (HostFacts) AsMap 

func (h HostFacts) AsMap() map[string]any

AsMap renders the facts as JSON keys, for the map-shaped agent views. Keys match httpx's names, and match the struct tags above — one definition of the key set, so the two output shapes cannot drift.

func (HostFacts) Empty 

func (h HostFacts) Empty() bool

Empty reports whether this process observed nothing for the host. Empty facts are omitted entirely rather than emitted as empty arrays: "not probed" and "no records" are different answers.

type HostObservation 

type HostObservation struct {
	bun.BaseModel `bun:"table:host_observations,alias:ho" json:"-"`

	ID          int64     `bun:"id,pk,autoincrement" json:"id"`
	ProjectUUID string    `bun:"project_uuid,notnull" json:"project_uuid"`
	ScanUUID    string    `bun:"scan_uuid,nullzero" json:"scan_uuid,omitempty"`
	Hostname    string    `bun:"hostname,notnull" json:"hostname"`
	Port        int       `bun:"port,notnull" json:"port"`
	ObservedAt  time.Time `bun:"observed_at,notnull,default:current_timestamp" json:"observed_at"`

	// DNS answers as jsonb columns, the same way Finding.Tags and
	// HTTPRecord.Technology store their slices. Stored whole rather than as three
	// child tables: they are read as a whole or not at all, and never queried by
	// element.
	DNSA     []string `bun:"dns_a,type:jsonb,nullzero" json:"dns_a,omitempty"`
	DNSAAAA  []string `bun:"dns_aaaa,type:jsonb,nullzero" json:"dns_aaaa,omitempty"`
	DNSCNAME []string `bun:"dns_cname,type:jsonb,nullzero" json:"dns_cname,omitempty"`

	// TLS is the handshake and leaf certificate, absent when the endpoint was not
	// handshaked (no --tls-probe, or a plaintext endpoint).
	TLS *tlsprobe.Info `bun:"tls,type:jsonb,nullzero" json:"tls,omitempty"`

	// Complete reports that the lookup actually ran to an answer. False marks an
	// observation the run never finished - a cancelled or budget-limited
	// prefetch - which is a different fact from a host that genuinely resolves to
	// nothing, and the two are indistinguishable without it.
	Complete  bool      `bun:"complete,notnull" json:"complete"`
	CreatedAt time.Time `bun:"created_at,notnull,default:current_timestamp" json:"created_at"`
}

HostObservation is what one scan observed about one endpoint: the full DNS answer and, under --tls-probe, the handshake and leaf certificate.

It is the durable counterpart to the process-local caches HostFacts reads. Those caches are bounded (8192 entries each) and evict in insertion order, so on a sweep larger than that the hosts resolved FIRST had their answers dropped before the export that wanted them - the work was done, the answer was known, and the output silently omitted it. A row here survives eviction, the process, and the run.

One row per (scan, hostname, port). Re-probing a host adds an observation, it does not replace one: the addresses a host resolved to during an earlier scan are that scan's evidence, and an upsert keyed on the host alone would erase it.

func (*HostObservation) Facts 

func (o *HostObservation) Facts() HostFacts

Facts renders a stored observation back into the output shape. The columns are bun jsonb fields, so this is a field copy rather than a decode.

type HostObservationInput 

type HostObservationInput struct {
	Hostname string
	Port     int
	HostFacts

	// Complete reports that the lookup ran to an answer. A cut-short prefetch
	// records incomplete observations rather than none, so a later reader can
	// tell "no AAAA record" from "we never got there".
	Complete bool
}

HostObservationInput is one endpoint's observation, as the probe collects it, before it becomes a row.

HostFacts is embedded rather than restated: the answer being stored IS the answer that will later be emitted, and copying its fields here meant two places to edit for every new fact and two Empty() methods that could silently disagree. What this type adds is the two things HostFacts cannot carry - which endpoint the answer describes, and whether the lookup finished.

func (HostObservationInput) Empty 

func (o HostObservationInput) Empty() bool

Empty reports whether the observation carries nothing worth storing. A row with no answer AND no completeness claim states nothing that the absence of a row does not already state.

type HostParamNames added in v0.5.3 

type HostParamNames struct {
	// JS holds names from the target's JavaScript: jstangle-derived records
	// and the request hints in jstangle-facts artifacts.
	JS []string
	// Observed holds names from every other record of the host (crawl,
	// discovery, ingested traffic).
	Observed []string
}

HostParamNames are the parameter names one host's records already use, split by where they came from and each ordered by how often it appeared.

type HostStats 

type HostStats struct {
	Scheme       string `json:"scheme"`
	Hostname     string `json:"hostname"`
	Port         int    `json:"port"`
	RequestCount int64  `json:"request_count"`
	FindingCount int64  `json:"finding_count"`
}

HostStats holds statistics for a host

type HostTarget 

type HostTarget struct {
	Scheme   string `bun:"scheme"`
	Hostname string `bun:"hostname"`
	Port     int    `bun:"port"`
}

HostTarget represents a distinct scheme+hostname+port combination from HTTP records.

type InsertionPointSample added in v0.5.3 

type InsertionPointSample struct {
	Records         int
	AvgParams       float64
	AvgPathSegments float64
}

InsertionPointSample summarizes a sample of stored records for the scan plan's request estimate: how many records were read, and how many parameters and path segments an average one carries (each is an insertion point an active module may test).

type LockedError added in v0.5.3 

type LockedError struct {
	Op       string
	Attempts int
	Err      error
}

LockedError reports a busy retry budget that ran out. It carries the stable code database_locked, so the CLI's error document and a REST body name the condition the same way, and it unwraps to both ErrDatabaseLocked and the last driver error.

func (*LockedError) Code added in v0.5.3 

func (e *LockedError) Code() apicontract.ErrorCode

Code is the stable error code (database_locked).

func (*LockedError) Error added in v0.5.3 

func (e *LockedError) Error() string

func (*LockedError) Unwrap added in v0.5.3 

func (e *LockedError) Unwrap() []error

Unwrap exposes both the sentinel and the driver error.

type MergeOptions 

type MergeOptions struct {
	// SkipHTTPRecords omits the http_records table from the copy. It is safe for
	// readers that never resolve a finding's linked records: finding_records
	// carries no foreign key to http_records (it is a bare
	// PRIMARY KEY (finding_id, record_uuid) table), and findings embed their own
	// request/response inline, so the junction and every finding still merge
	// intact — a later record lookup simply resolves to nothing.
	//
	// This exists because http_records holds the raw request/response blobs and
	// is ~98% of the bytes in a typical result database. A read that renders
	// purely from findings (the finding tree/table/JSON views) would otherwise
	// pay to copy every blob into the destination only to discard it.
	SkipHTTPRecords bool

	// SkipRecordBodies copies http_records without its raw_request/raw_response
	// columns, which are nullable and hold ~96% of the table's bytes. Every row
	// still lands, with its metadata intact, so row counts and any predicate over
	// metadata (status/method/host/content-type) are unaffected — unlike dropping
	// rows, this changes nothing observable for such a reader.
	//
	// It is only safe when nothing downstream reads the raw corpus: a filter that
	// LIKEs over it (see QueryFilters.UsesRawCorpus) silently matches nothing, and
	// a renderer that prints it silently prints empty. Ignored when
	// SkipHTTPRecords is set, which already omits the table.
	SkipRecordBodies bool

	// SkipFindings omits findings, the finding_records junction, the finding
	// history tables (finding_observations, finding_status_events) and
	// oast_interactions from the copy. Safe only for a reader that shows no
	// finding and filters no record by one.
	//
	// It is the only option here that bounds the MERGING process's memory rather
	// than the destination's size. The three tables it drops are the ones that
	// cannot be copied set-based — findings.id is AUTOINCREMENT and is referenced
	// by both the junction and the OAST rows, so the id remapping needs every
	// source row in Go before the write transaction opens. Each finding row
	// carries its inline request, response, and additional_evidence, so a corpus
	// with many findings is buffered in full, per source, whether or not anything
	// will read one.
	SkipFindings bool
}

MergeOptions tunes what a SQLite-to-SQLite merge copies.

type MergeStats 

type MergeStats struct {
	ProjectsMerged       int
	ScansMerged          int
	AgenticScansMerged   int
	RecordsMerged        int
	FindingsMerged       int
	FindingsDeduped      int
	FindingRecordsMerged int
	OASTMerged           int
	ObservationsMerged   int
	StatusEventsMerged   int
}

MergeStats reports what a SQLite-to-SQLite merge copied into the destination.

func MergeSQLiteFile 

func MergeSQLiteFile(ctx context.Context, dest *DB, srcPath string) (*MergeStats, error)

MergeSQLiteFile merges every native-scan result row from the SQLite database at srcPath into dest (which must be an already-open, schema-current SQLite database). It is safe to call repeatedly with the same source: rows dedup on their natural keys (uuid for records/scans, (project_uuid, finding_hash) for findings), so a re-merge is idempotent. project_uuid values are copied verbatim, so the merged rows stay scoped to whatever project the scan ran under.

The whole operation runs on a single pinned connection inside one BEGIN IMMEDIATE transaction, and retries on SQLITE_BUSY/locked so it tolerates other processes briefly holding the destination's write lock.

func MergeSQLiteFileWithOptions 

func MergeSQLiteFileWithOptions(ctx context.Context, dest *DB, srcPath string, opts MergeOptions) (*MergeStats, error)

MergeSQLiteFileWithOptions is MergeSQLiteFile with control over what is copied; see MergeOptions.

func (*MergeStats) Add 

func (s *MergeStats) Add(o *MergeStats)

Add accumulates o's counters into s, for aggregating the stats of several merges (e.g. a multi-source import) into a single total.

type MigrateResult added in v0.5.3 

type MigrateResult struct {
	FromVersion      int      `json:"from_version"`
	ToVersion        int      `json:"to_version"`
	MinReaderVersion int      `json:"min_reader_version"`
	WrittenBy        string   `json:"written_by,omitempty"`
	Applied          []string `json:"applied"`
	AlreadyCurrent   bool     `json:"already_current"`
}

MigrateResult describes one explicit migration (`vigolium db migrate`).

type NewSinceBaseline added in v0.5.3 

type NewSinceBaseline struct {
	ScanUUID       string
	PartialHistory bool
}

NewSinceBaseline is the `finding --new-since <scan>` predicate: keep findings the baseline scan did not observe and whose earliest observation belongs to a scan that started after the baseline did. A finding with no observations at all (an import, an out-of-band agent finding) falls back to "stored after the baseline finished".

PartialHistory is set when the baseline predates the store's observation history; the predicate then also requires the finding to have been stored after the baseline started, since a backfilled finding's only observation is its LATEST scan and would otherwise read as new.

type OASTInteraction 

type OASTInteraction struct {
	bun.BaseModel `bun:"table:oast_interactions,alias:oi" json:"-"`

	ID            int64     `bun:"id,pk,autoincrement" json:"id"`
	ProjectUUID   string    `bun:"project_uuid,notnull" json:"project_uuid"`
	ScanUUID      string    `bun:"scan_uuid,nullzero" json:"scan_uuid,omitempty"`
	UniqueID      string    `bun:"unique_id,notnull" json:"unique_id"`
	FullID        string    `bun:"full_id,notnull" json:"full_id"`
	Protocol      string    `bun:"protocol,notnull" json:"protocol"`
	QType         string    `bun:"q_type,nullzero" json:"q_type,omitempty"`
	RawRequest    string    `bun:"raw_request,nullzero" json:"raw_request,omitempty"`
	RawResponse   string    `bun:"raw_response,nullzero" json:"raw_response,omitempty"`
	RemoteAddress string    `bun:"remote_address,nullzero" json:"remote_address,omitempty"`
	InteractedAt  time.Time `bun:"interacted_at,notnull" json:"interacted_at"`

	// Correlated context from payload tracker
	TargetURL     string `bun:"target_url,nullzero" json:"target_url,omitempty"`
	ParameterName string `bun:"parameter_name,nullzero" json:"parameter_name,omitempty"`
	InjectionType string `bun:"injection_type,nullzero" json:"injection_type,omitempty"`
	ModuleID      string `bun:"module_id,nullzero" json:"module_id,omitempty"`
	FindingID     int64  `bun:"finding_id,nullzero" json:"finding_id,omitempty"` // link to finding this interaction proves
	Payload       string `bun:"payload,nullzero" json:"payload,omitempty"`       // exact payload that triggered the interaction

	CreatedAt time.Time `bun:"created_at,notnull,default:current_timestamp" json:"created_at"`
}

OASTInteraction records an out-of-band interaction received from an interactsh server.

type OASTPayload added in v0.5.3 

type OASTPayload struct {
	bun.BaseModel `bun:"table:oast_payloads,alias:op" json:"-"`

	Nonce       string    `bun:"nonce,pk" json:"nonce"`
	ProjectUUID string    `bun:"project_uuid,notnull" json:"project_uuid"`
	ScanUUID    string    `bun:"scan_uuid,nullzero" json:"scan_uuid,omitempty"`
	Class       string    `bun:"class,nullzero" json:"class,omitempty"`
	ModuleID    string    `bun:"module_id,nullzero" json:"module_id,omitempty"`
	TargetURL   string    `bun:"target_url,nullzero" json:"target_url,omitempty"`
	Param       string    `bun:"param,nullzero" json:"param,omitempty"`
	RequestHash string    `bun:"request_hash,nullzero" json:"request_hash,omitempty"`
	Payload     string    `bun:"payload,nullzero" json:"payload,omitempty"`
	CreatedAt   time.Time `bun:"created_at,notnull,default:current_timestamp" json:"created_at"`
}

OASTPayload is the context of one planted OAST payload, keyed by the nonce embedded in its callback host or path, so an interaction that arrives out of band can be attributed after the planting process is gone.

type ParamOptions added in v0.5.3 

type ParamOptions struct {
	// Endpoint keeps one endpoint, by EndpointView id or by template.
	Endpoint string
	// Locations keeps these locations (query, body, json, cookie, xml,
	// multipart, path, header, other).
	Locations []string
	// Types keeps parameters whose dominant type is listed (ValueType names:
	// integer, uuid, sequential_id, email, jwt, …).
	Types []string
	// MinOccurrences drops parameters seen fewer times.
	MinOccurrences int64
	// ShowValues disables sample redaction.
	ShowValues bool
	// Limit/Offset page the list; Limit 0 returns every parameter.
	Limit, Offset int
	// MaxParams overrides DefaultMaxParams (0 = default).
	MaxParams int
}

ParamOptions shapes a Params call.

type ParamResult added in v0.5.3 

type ParamResult struct {
	Params            []ParamView
	Total             int64
	RecordsScanned    int64
	Capped            bool
	UnbucketedRecords int64
}

ParamResult is a bounded parameter inventory.

type ParamView added in v0.5.3 

type ParamView struct {
	EndpointID string   `json:"endpoint_id"`
	Origin     string   `json:"origin"`
	Template   string   `json:"template"`
	Methods    []string `json:"methods"`
	Location   string   `json:"location"`
	Name       string   `json:"name"`
	// After is the literal segment before a path parameter ("users" for
	// {int}@1 in /users/{int}).
	After string `json:"after,omitempty"`

	Occurrences    int64 `json:"occurrences"`
	DistinctValues int64 `json:"distinct_values"`
	DistinctCapped bool  `json:"distinct_capped"`

	// Types is a histogram of mutation.Classify over at most 200 values;
	// Classified says how many were classified.
	Types        map[string]int64 `json:"types"`
	Classified   int64            `json:"classified"`
	DominantType string           `json:"dominant_type"`

	// Samples are up to three values, at most 64 characters each. Cookie and
	// header values and secret-looking names are redacted unless ShowValues.
	Samples  []string `json:"samples"`
	Redacted bool     `json:"redacted,omitempty"`
}

ParamView is one parameter of one endpoint.

type PathTarget 

type PathTarget struct {
	Scheme   string `bun:"scheme"`
	Hostname string `bun:"hostname"`
	Port     int    `bun:"port"`
	Path     string `bun:"path"`
}

PathTarget represents a distinct scheme+hostname+port+path combination from HTTP records.

type PerformanceMetrics 

type PerformanceMetrics struct {
	AvgResponseTime int64 `json:"avg_response_time"`
	MinResponseTime int64 `json:"min_response_time"`
	MaxResponseTime int64 `json:"max_response_time"`
	P50ResponseTime int64 `json:"p50_response_time"`
	P95ResponseTime int64 `json:"p95_response_time"`
	P99ResponseTime int64 `json:"p99_response_time"`
}

PerformanceMetrics holds response time statistics

type PersistenceOutcome added in v0.5.2 

type PersistenceOutcome struct {
	Writer    string `json:"writer"` // "records" | "findings" | "mixed"
	Accepted  int64  `json:"accepted"`
	Committed int64  `json:"committed"` // persisted, or deduplicated onto an existing row
	Failed    int64  `json:"failed,omitempty"`
	Unknown   int64  `json:"unknown,omitempty"`
	TimedOut  bool   `json:"timed_out,omitempty"`
}

PersistenceOutcome is what a writer can say, after shutdown, about writes it accepted. Committed+Failed+Unknown == Accepted. Unknown = still queued or in flight when the close deadline passed: it may or may not have landed.

The distinction between Failed and Unknown is the point. Before this existed, Close() returned nothing: a writer that drained cleanly, a writer that lost a batch to a failed transaction, and a writer whose deadline expired with records still queued were indistinguishable to the caller, so a scan reported success either way. Unknown is deliberately not folded into Failed — a timed-out drain may well have committed everything a moment later, and reporting those as lost would be as wrong as reporting them as stored.

func (PersistenceOutcome) Clean added in v0.5.2 

func (o PersistenceOutcome) Clean() bool

Clean reports whether everything the writer accepted reached a known-good end. A clean outcome is the only one a caller may report as an unqualified success.

func (*PersistenceOutcome) Merge added in v0.5.2 

func (o *PersistenceOutcome) Merge(p PersistenceOutcome)

Merge folds p into o, so a phase that shuts down several writers can report one number. The writer label collapses to "mixed" once two different writers are combined; merging into a zero outcome adopts p's label.

type PersistenceShutdowner added in v0.5.2 

type PersistenceShutdowner interface {
	Shutdown() PersistenceOutcome
}

PersistenceShutdowner is a writer that can be drained and asked what happened. Both RecordWriter and FindingWriter satisfy it, which is what lets a phase shut down whatever writers it owns through one call.

type PhaseOutcome added in v0.5.2 

type PhaseOutcome struct {
	Phase   string     `json:"phase"`
	State   PhaseState `json:"state"`
	Reasons []string   `json:"reasons,omitempty"` // sorted, unique
	Limits  []string   `json:"limits,omitempty"`  // sorted, unique
	Errors  int        `json:"errors,omitempty"`
	Message string     `json:"message,omitempty"` // first error, truncated
	// Unprocessed counts work items the executor dequeued but did not process —
	// deliberately NOT acknowledged, so a later run re-serves them.
	Unprocessed int64               `json:"unprocessed,omitempty"`
	DurationMS  int64               `json:"duration_ms"`
	Persistence *PersistenceOutcome `json:"persistence,omitempty"`
	// Targets is the CLI-target tally (scope admission, reachability); set only
	// on the "preflight" pseudo-phase. See TargetTally.
	Targets *TargetTally `json:"targets,omitempty"`
	// Counts are phase-specific tallies keyed by a stable snake_case name (the
	// harvest-probe step reports hosts_dead, kept, dropped_not_found, …). Only
	// phases whose result is a set of numbers rather than records set it.
	Counts map[string]int64 `json:"counts,omitempty"`
	// BudgetExhaustedHosts are the hosts whose --max-requests-per-host ran out
	// during the phase, sorted; set alongside the host_request_budget limit.
	BudgetExhaustedHosts []string `json:"budget_exhausted_hosts,omitempty"`
}

PhaseOutcome is one phase's result, in the terms a consumer can act on. It is persisted on the scan row (scans.phase_outcomes) and summarised on the event stream, so every field is additive-only.

func (PhaseOutcome) BenignSkip added in v0.5.2 

func (o PhaseOutcome) BenignSkip() bool

BenignSkip reports whether this phase was skipped for having nothing to do, rather than because the scan ran out of time, was cancelled, or failed.

It is the one rule behind "does this phase count against the scan's coverage", and it lives here because three places need the same answer: the completeness verdict, the partial banner, and the union of reasons on the terminal event. The third did not have it, so an already-partial scan's scan.finished could carry a `no_modules` that the banner and the phase line both — correctly — left out: one rule, three answers.

The state check is part of the rule. A phase that FAILED without recording a reason is not a benign skip; it is the most serious thing that can happen to a phase, reported with no explanation.

func (*PhaseOutcome) Merge added in v0.5.2 

func (o *PhaseOutcome) Merge(b PhaseOutcome)

Merge folds another outcome for the SAME phase into o.

It exists for full-native-scan-on-receive, which runs the whole plan once per batch of arriving records: without coalescing, the scan row would accumulate one outcome per phase per iteration and grow without bound for the life of the run. Coalescing also gives the more useful answer — "what happened to the discovery phase in this scan" rather than a thousand separate verdicts — and for an ordinary scan, where each phase appears exactly once, it never fires.

Reasons and limits union, counts sum, and the state is the worse of the two. The first message is kept: it is the first failure, which is normally the cause rather than a consequence of it.

type PhaseState added in v0.5.2 

type PhaseState string

PhaseState is what became of one scan phase. It is deliberately NOT the scan status vocabulary ("running"/"completed"/"failed"/"cancelled"): a phase can end having done most of its work and none of those four words says so, which is the whole reason a time-boxed phase used to report a clean completion. 

const (
	// PhaseCompleted — the phase consumed its whole input and nothing was lost.
	PhaseCompleted PhaseState = "completed"
	// PhasePartial — the phase ran but did not cover everything it was given.
	PhasePartial PhaseState = "partial"
	// PhaseFailed — the phase reported an error. It may still have produced work.
	PhaseFailed PhaseState = "failed"
	// PhaseSkipped — the phase never ran (nothing to do, or the budget was gone).
	PhaseSkipped PhaseState = "skipped"
)

func ResolvePhaseState added in v0.5.2 

func ResolvePhaseState(errs int, reasons []string, skipped bool) PhaseState

ResolvePhaseState turns the three things a phase tracker observes into a state.

Precedence is errors → skipped → reasons → completed. An error outranks everything because a phase that failed is not a phase that merely came up short; a skipped phase never ran, so its reasons describe why it was skipped rather than what it missed; a reason without an error is the partial case this vocabulary exists for. LIMITS are deliberately not an input: reaching a configured bound is designed behaviour and must not downgrade a phase.

type PrimaryKeyInfo 

type PrimaryKeyInfo struct {
	Columns []string `json:"columns"`
}

PrimaryKeyInfo describes the primary key column(s) of a table.

func DetectPrimaryKey 

func DetectPrimaryKey(ctx context.Context, db *DB, tableName string) (PrimaryKeyInfo, error)

DetectPrimaryKey returns the primary key column(s) for the given table.

type Project 

type Project struct {
	bun.BaseModel `bun:"table:projects,alias:p" json:"-"`

	UUID        string `bun:"uuid,pk,notnull" json:"uuid"`
	Name        string `bun:"name,notnull" json:"name"`
	Description string `bun:"description,nullzero" json:"description,omitempty"`
	OwnerUUID   string `bun:"owner_uuid,nullzero" json:"owner_uuid,omitempty"` // soft FK → users
	ConfigPath  string `bun:"config_path,nullzero" json:"config_path,omitempty"`

	Tags          []string  `bun:"tags,type:jsonb,nullzero" json:"tags,omitempty"`
	DefaultTarget string    `bun:"default_target,nullzero" json:"default_target,omitempty"`
	LastScanAt    time.Time `bun:"last_scan_at,nullzero" json:"last_scan_at,omitempty"`

	CreatedAt time.Time `bun:"created_at,notnull,default:current_timestamp" json:"created_at"`
	UpdatedAt time.Time `bun:"updated_at,notnull,default:current_timestamp" json:"updated_at"`
}

Project represents a logical grouping for all scan data.

type ProjectStatsRow 

type ProjectStatsRow struct {
	ProjectUUID      string `bun:"project_uuid"`
	HTTPRecords      int64  `bun:"http_records"`
	HTTP2xx          int64  `bun:"http_2xx"`
	HTTP3xx          int64  `bun:"http_3xx"`
	HTTP4xx          int64  `bun:"http_4xx"`
	HTTP5xx          int64  `bun:"http_5xx"`
	Findings         int64  `bun:"findings"`
	Critical         int64  `bun:"critical"`
	High             int64  `bun:"high"`
	Medium           int64  `bun:"medium"`
	Low              int64  `bun:"low"`
	Info             int64  `bun:"info"`
	Scans            int64  `bun:"scans"`
	AgenticScans     int64  `bun:"agentic_scans"`
	OASTInteractions int64  `bun:"oast_interactions"`
}

ProjectStatsRow holds per-project aggregated counts used by GetAllProjectsStats.

type QueryBuilder 

type QueryBuilder struct {
	// contains filtered or unexported fields
}

QueryBuilder builds filtered database queries

func NewQueryBuilder 

func NewQueryBuilder(db *DB, filters QueryFilters) *QueryBuilder

NewQueryBuilder creates a new query builder

func (*QueryBuilder) BuildRecordsQuery 

func (qb *QueryBuilder) BuildRecordsQuery() *bun.SelectQuery

BuildRecordsQuery builds a query for http_records table

func (*QueryBuilder) Count 

func (qb *QueryBuilder) Count(ctx context.Context) (int64, error)

Count returns total number of matching records

func (*QueryBuilder) Endpoints added in v0.5.3 

func (qb *QueryBuilder) Endpoints(ctx context.Context, opts EndpointOptions) (EndpointResult, error)

Endpoints aggregates the filtered records into an endpoint inventory.

func (*QueryBuilder) Execute 

func (qb *QueryBuilder) Execute(ctx context.Context) ([]*HTTPRecord, error)

Execute executes the query and returns results

func (*QueryBuilder) ExecuteWithCount 

func (qb *QueryBuilder) ExecuteWithCount(ctx context.Context) ([]*HTTPRecord, int64, error)

ExecuteWithCount runs the filtered query and returns the matching page of records alongside the total count for the same filters.

It is NOT one round trip, despite what this comment used to claim. Bun's ScanAndCount collapses to a single statement only when there is no limit and no offset; a paginated call — the only reason to use this method — takes the two-query path, and with no explicit conn it runs the scan and the count CONCURRENTLY, so it occupies two pool connections for its duration. On SQLite that is worth remembering when this runs alongside a busy RecordWriter.

Still preferable to Execute + Count, which serializes the same two queries. Callers that do not need an exact total on every refresh should use Execute alone rather than paying the count.

func (*QueryBuilder) GroupRecordsBy 

func (qb *QueryBuilder) GroupRecordsBy(ctx context.Context, field string, limit int) (RecordGrouping, error)

GroupRecordsBy counts the filtered records by one column, largest bucket first, capped at limit (<= 0 means no cap).

The totals come from a second query ONLY when the cap actually bit. Summing the returned buckets describes the whole result exactly when every bucket came back — each filtered row lands in exactly one bucket, since COALESCE removes the NULL/empty split — and describes only the visible part otherwise, which is how a bounded view comes to report a complete-looking number that is wrong. So the expensive case is paid for and the common one is not: the filters here are the listing's filters, which include `LIKE '%…%'` over the raw request/response corpus, and running those twice reads every matched record's blobs twice to produce a dozen integers.

func (*QueryBuilder) OmitBodies 

func (qb *QueryBuilder) OmitBodies() *QueryBuilder

OmitBodies drops raw_request/raw_response from the SELECT list, for callers that read only metadata off the returned records. Those two columns hold the whole request/response corpus and dwarf every other column, so hydrating them for a list view costs memory proportional to the traffic scanned rather than to the page size.

Filters are unaffected: they run in SQL against the stored columns, so this is safe even with --search/--header/--body active (contrast MergeOptions. SkipRecordBodies, which removes the columns from the destination and so cannot be used with those filters). The only requirement is that the caller not read HTTPRecord.RawRequest/RawResponse — they come back empty.

This mirrors FindingsQueryBuilder.ExecuteWithCount, which always excludes its equivalent blob columns; records can't default to that because --raw/--burp and the record views legitimately need them.

func (*QueryBuilder) Params added in v0.5.3 

func (qb *QueryBuilder) Params(ctx context.Context, opts ParamOptions) (ParamResult, error)

Params aggregates the filtered records into a parameter inventory.

type QueryFilters 

type QueryFilters struct {
	// Project scoping
	ProjectUUID string // Required: filter all queries to this project

	// Identity filtering. RecordUUIDs selects exact stored records by UUID and is
	// applied as a SQL predicate, so it narrows BEFORE pagination — a record is
	// never reported absent merely because it fell outside the current page.
	// Without it the read surface had no identity selector at all: the only flags
	// that took a record UUID (`replay -u`, `fuzz -u`) both send traffic, and a
	// UUID passed as the fuzzy search term matched zero rows because that term
	// searches URL/path/body, not identity.
	RecordUUIDs []string

	// URLsExact selects records whose stored url matches EXACTLY — an equality
	// IN (), never a LIKE.
	//
	// It exists because `--url` was the name callers reached for and the one name
	// the read surface did not have: --host takes a hostname, --path a fuzzy
	// pattern, and a URL passed as the positional term is a substring search that
	// also matches every URL containing it. So "show me this one endpoint" had no
	// spelling, and the nearest miss was answered by the flag suggester with
	// "did you mean --all?", which lifts the result cap.
	//
	// Repeated values are OR-ed within the field and AND-ed against every other
	// filter, matching how --uuid and --method already behave. No normalization,
	// no case folding, no trailing-slash equivalence: equality is against the
	// bytes that were stored, because a comparison that quietly canonicalizes is
	// one the caller cannot predict from the value they passed.
	URLsExact []string

	// Host filtering
	HostPattern string // Hostname pattern (supports wildcards)

	// Request filtering
	Methods          []string // HTTP methods (GET, POST, etc.)
	PathPattern      string   // Path pattern (supports wildcards)
	StatusCodes      []int    // Status codes (200, 404, etc.)
	ScanUUID         string   // Scan UUID: findings.scan_uuid for findings, http_records.scan_uuid (first-stored-by) for records
	AgenticScanUUIDs []string // Agentic-scan UUID(s) (links agent-produced findings; pass the scan tree for nested audit/swarm runs)
	Source           string   // Filter by record source (e.g. scanner, ingest-cli, ingest-server)

	// Response filtering
	ContentType string // Filter by response content type

	// Risk filtering
	MinRiskScore    int      // Minimum risk score filter (anomaly_ranking's batch percentile)
	MinSurfaceScore int      // Minimum attack-surface score filter (surface_scoring's 0-100 signal score)
	Remark          string   // Filter by remark substring (single)
	Remarks         []string // Filter by multiple remarks (AND: record must have all)

	// Finding filtering
	FindingID      int      // Filter by finding ID
	FindingIDAfter int64    // Filter findings with ID greater than this value
	Severity       []string // Finding severity (critical, high, medium, low, info)
	Confidence     []string // Finding confidence (certain, firm, tentative)
	ModuleName     string   // Filter findings by module name
	ModuleType     string   // Filter findings by module type (active, passive, nuclei, etc.)
	FindingSource  string   // Filter findings by source (audit, spa, agent, etc.)
	RecordKinds    []string // finding (default), candidate, observation
	RepoName       string   // Filter findings by repo name
	Status         []string // Filter findings by lifecycle status (one of FindingStatuses)
	// ExcludeStatuses drops findings whose lifecycle status is listed (a NULL
	// status reads as triaged, the column default). The --fail-on gate, scan
	// diff and --new-since pass DismissedFindingStatuses here.
	ExcludeStatuses []string
	// ObservedInScans keeps findings that any of these scans observed
	// (finding_observations), not merely the scan a finding was last
	// attributed to (findings.scan_uuid).
	ObservedInScans []string
	// NewSince keeps findings first observed after a baseline scan; see
	// NewSinceBaseline. Built by Repository.ResolveNewSince.
	NewSince *NewSinceBaseline
	// CreatedAfter keeps findings first stored after this instant
	// (`finding --new-since <duration>`).
	CreatedAfter *time.Time

	// Date range filtering
	DateFrom *time.Time
	DateTo   *time.Time

	// Full-text search
	SearchTerm   string   // Search across URLs, paths
	SearchTerms  []string // Findings: repeatable search terms, AND-combined (each further narrows the match)
	FuzzyTerm    string   // Broad fuzzy search across multiple fields
	HeaderSearch string   // Search in headers
	BodySearch   string   // Search in request/response body

	// Negative / exclusion search — drop rows where the term appears in the same
	// corpus the positive counterpart scans.
	ExcludeTerms        []string // Repeatable, AND-combined: a row is dropped if ANY term matches (inverse of SearchTerms)
	ExcludeHeaderSearch string   // Drop rows whose header/raw corpus contains the term (inverse of HeaderSearch)
	ExcludeBodySearch   string   // Drop rows whose body/raw corpus contains the term (inverse of BodySearch)

	// Pagination
	Limit  int
	Offset int

	// Sorting
	SortBy  string // Field to sort by
	SortAsc bool   // Sort ascending (default: descending)
}

QueryFilters holds filter criteria for database queries

func (QueryFilters) EffectiveExcludeTerms 

func (f QueryFilters) EffectiveExcludeTerms() []string

EffectiveExcludeTerms returns the exclusion search terms with blanks dropped, so callers can loop the result directly. Each term becomes an independent NOT (...) conjunct — a row is dropped if ANY term matches.

func (QueryFilters) EffectiveSearchTerms 

func (f QueryFilters) EffectiveSearchTerms() []string

EffectiveSearchTerms returns the search terms to AND-combine: the repeatable SearchTerms when set, otherwise the single SearchTerm. Blank terms are dropped, so callers can loop the result directly.

func (QueryFilters) UsesLinkedFindings 

func (f QueryFilters) UsesLinkedFindings() bool

UsesLinkedFindings reports whether any active filter reaches into findings when the query is over HTTP RECORDS. It is the mirror of UsesLinkedRecords: records have no finding columns of their own, so --severity resolves through a join over the finding_records junction (see applyFilters).

A caller that would otherwise omit the finding tables entirely (globDBSkipSet.Findings, MergeOptions.SkipFindings) MUST check this first. The failure is silent: with no findings to join, a --severity record listing returns nothing rather than erroring.

Keep in sync with applyFilters.

func (QueryFilters) UsesLinkedRecords 

func (f QueryFilters) UsesLinkedRecords() bool

UsesLinkedRecords reports whether any active filter reaches into http_records when the query is over FINDINGS. It is the findings-side counterpart to UsesRawCorpus and a strictly wider question: a findings query has no record columns of its own, so host/path/method/status/source all resolve through an EXISTS over the finding_records junction (see applyFindingFilters), and every raw-corpus filter does too.

The question does not arise for traffic, whose query is over http_records directly — there the rows are always present by construction.

A caller that would otherwise omit the table entirely (globDBSkipSet.Records, MergeOptions.SkipHTTPRecords) MUST check this first. The failure is silent: with no rows to satisfy them, every EXISTS predicate is false, so --host returns nothing and the NOT EXISTS exclude forms return everything.

Keep in sync with applyFindingFilters.

func (QueryFilters) UsesRawCorpus 

func (f QueryFilters) UsesRawCorpus() bool

UsesRawCorpus reports whether any active filter matches against the raw_request/raw_response blob columns — the FuzzyTerm branches, recordSearchPredicate (--search) and the region predicates behind --header/--body (see message_regions.go), plus all three exclude forms.

A caller that would otherwise avoid materializing those blobs (a projected SELECT, or a merge that omits the columns) MUST check this first and keep them when it returns true. The failure is silent, not an error: with the blobs absent every LIKE matches nothing, so --search returns an empty result and the --exclude-* forms — which are negated — return everything.

Keep in sync with applyFilters; this is the reason to add a field here rather than open-code the flag checks at each call site.

type ReSpiderCandidate 

type ReSpiderCandidate struct {
	UUID                string `bun:"uuid"`
	URL                 string `bun:"url"`
	Scheme              string `bun:"scheme"`
	Hostname            string `bun:"hostname"`
	Port                int    `bun:"port"`
	Source              string `bun:"source"`
	ResponseContentType string `bun:"response_content_type"`
	RawResponse         []byte `bun:"raw_response"`
}

ReSpiderCandidate is a lightweight projection of an HTTP record used by the targeted re-spider phase. It carries just enough to evaluate rich/SPA content and dedup app shells (source + raw_response) without materializing every column.

type RecordCleanupResult added in v0.5.2 

type RecordCleanupResult struct {
	// Deleted is the number of http_records rows actually removed.
	Deleted int64
	// KeptReferenced counts records the pass selected as redundant but left in
	// place because a finding or an analysis artifact still references them.
	KeptReferenced int64
	// ByStatus breaks Deleted down by HTTP status code, for operator feedback.
	ByStatus map[int]int64
}

RecordCleanupResult reports what one discovery record-cleanup pass removed and what it deliberately spared.

type RecordCounts 

type RecordCounts struct {
	HTTPRecords     int64   `json:"http_records"`
	WithResponse    int64   `json:"with_response"`
	ResponsePercent float64 `json:"response_percent"`
	Findings        int64   `json:"findings"`
}

RecordCounts holds counts of each record type

type RecordGroup 

type RecordGroup struct {
	Value string `json:"value"`
	Count int64  `json:"count"`
}

RecordGroup is one bucket: a column value and how many records carry it.

type RecordGrouping 

type RecordGrouping struct {
	// Field is the public name that was grouped on, echoed back.
	Field string `json:"field"`
	// Groups are the largest buckets first, at most the requested limit.
	Groups []RecordGroup `json:"groups"`
	// DistinctGroups is how many buckets exist in total, whatever Groups holds.
	DistinctGroups int64 `json:"distinct_groups"`
	// TotalRecords is how many records the filters matched, whatever was bucketed.
	TotalRecords int64 `json:"total_records"`
	// OtherGroups / OtherRecords describe the tail beyond the limit. Both zero
	// means the grouping is complete.
	OtherGroups  int64 `json:"other_groups"`
	OtherRecords int64 `json:"other_records"`
}

RecordGrouping is a bounded answer, and says so.

A grouping is not automatically small — one recorded case grouped 56 rows into 54 distinct buckets, which is a listing wearing a summary's name. Reporting only the top N without saying what was left out turns that into a wrong answer, so the tail is counted rather than dropped.

type RecordLineage added in v0.4.9 

type RecordLineage struct {
	// ParentUUID is the record this one redirected from. Empty for a root.
	ParentUUID string
	// RootUUID is the first record of the chain. Empty means "this row is the
	// root", which applyTo resolves to the row's own UUID.
	RootUUID string
	// Target is the submitted input line the chain started from, verbatim.
	Target string
	// ChainTruncated marks a row whose destination is not the row after it.
	ChainTruncated bool
	// ScanUUID is the scan that fetched this exchange. Stamped on a NEW row
	// only, never on a dedup hit (AdoptRecord ignores it): records are shared
	// across scans, so http_records.scan_uuid means "the first scan to store this
	// exchange". That is what lets a scan-scoped export and records_written count
	// what a run produced instead of the whole project.
	ScanUUID string
}

RecordLineage is the chain metadata one record carries: where it sits in a redirect chain, and which submitted target the chain started from.

Grouped into a struct rather than added to the parameter lists because the four fields are one fact about one row and are always decided together — a hop's parent, root and target all come from the same walk, and splitting them across positional arguments is how a caller ends up passing a root from one chain with a parent from another.

type RecordResponseUpdate 

type RecordResponseUpdate struct {
	StatusCode            int
	StatusPhrase          string
	ResponseHTTPVersion   string
	ResponseContentType   string
	ResponseContentLength int64
	RawResponse           []byte
	ResponseHash          string
	ResponseTimeMs        int64
}

RecordResponseUpdate contains response fields to update on an existing HTTP record. Used by the traffic --replay --in-replace feature.

type RecordWithHostFacts 

type RecordWithHostFacts struct {
	Record *HTTPRecord
	Facts  HostFacts
}

RecordWithHostFacts is an HTTPRecord serialized together with its host facts. Marshals as the record's own object with the fact keys merged in at the top level, which is where httpx puts them and therefore where a consumer written against that output looks.

func (RecordWithHostFacts) MarshalJSON 

func (rw RecordWithHostFacts) MarshalJSON() ([]byte, error)

MarshalJSON merges the fact keys into the record's own object rather than nesting them under a wrapper, so the emitted shape is the record a consumer already parses plus the extra keys.

type RecordWriter 

type RecordWriter struct {
	// contains filtered or unexported fields
}

RecordWriter serializes database writes through sharded goroutines that coalesce individual SaveRecord calls into batch transactions. Records are routed to shards by hashing the host name, so writes for the same host are serialized within a shard. With Shards=1 (default), behavior is identical to a single-goroutine writer. This eliminates SQLite SQLITE_BUSY errors under concurrent ingestion.

func NewRecordWriter 

func NewRecordWriter(repo *Repository, cfg RecordWriterConfig) *RecordWriter

NewRecordWriter creates and starts a RecordWriter. Call Close() to flush remaining records and stop the background goroutines.

func (*RecordWriter) Close 

func (w *RecordWriter) Close()

Close stops accepting new writes, flushes remaining records, and returns.

Ordering matters: it marks the writer closed under the admission write lock (so no new write can admit past this point), waits for every already-admitted enqueue to finish landing in a shard channel, and only THEN cancels the flush context. That guarantees the shutdown drain observes every admitted record — none is orphaned in a channel after the flush loop exits, and every admitted write receives its real result instead of a spurious "closed" error. Prefer Shutdown when the caller can report what happened; Close is the fire-and-forget form kept for existing call sites.

func (*RecordWriter) Metrics 

func (w *RecordWriter) Metrics() RecordWriterMetrics

Metrics returns a snapshot of the writer's counters.

func (*RecordWriter) SaveRecord 

func (w *RecordWriter) SaveRecord(ctx context.Context, rr *httpmsg.HttpRequestResponse, source string, projectUUID string) (string, error)

SaveRecord implements the network.RecordSaver interface by delegating to Write.

func (*RecordWriter) SaveRecordBatch 

func (w *RecordWriter) SaveRecordBatch(ctx context.Context, records []*httpmsg.HttpRequestResponse, source string, projectUUID string) ([]string, error)

SaveRecordBatch enqueues records to their shards BEFORE awaiting any result, so a single bulk producer's records land in the same writer batch instead of paying one flush interval per record (the trap of a Write-in-a-loop: Write blocks until its record is flushed, so a lone producer never fills a batch). Results are returned positionally aligned with the input — uuids[i] corresponds to records[i] — including deduplicated, invalid, and failed entries, so callers can't misassociate a UUID. The first error encountered is returned alongside the (still complete) uuid slice.

Submissions are admitted in BOUNDED slices rather than all at once. Callers hand this whole collections: discovery submits an entire provenance group in one call, which on a large corpus is tens of thousands of records. Admitting them together made every one of their converted HTTPRecords — each owning a full raw request and response — plus a result channel each live simultaneously, so peak memory scaled with the corpus rather than with anything configured. Slices stay large enough to fill many writer batches, so the coalescing this method exists for is unaffected.

func (*RecordWriter) Shutdown added in v0.5.2 

func (w *RecordWriter) Shutdown() PersistenceOutcome

Shutdown is Close that reports what became of every record it accepted.

One absolute deadline (now+FlushTimeout) governs the entire shutdown: the wait for in-flight enqueues to settle, every flush the drain performs, and the wait for the flush loops to exit. Each of those used to start its own FlushTimeout, so a wedged database could hold Close for three times the configured budget — and the caller still learned nothing about what was lost.

Idempotent: the outcome is computed once and every later or concurrent caller gets the same value. A nil writer is a no-op reporting a zero (clean) outcome: a phase that had no repository created no writer, and it accepted nothing, so there is nothing to report. Keeping that here rather than at every call site is what lets a caller pass the writers it owns without first proving each one exists — and a typed-nil pointer inside a PersistenceShutdowner interface is not caught by a nil check there anyway.

func (*RecordWriter) Write 

func (w *RecordWriter) Write(ctx context.Context, rr *httpmsg.HttpRequestResponse, source string, projectUUID string) (string, error)

Write enqueues a record for batched insertion. It blocks until the record is persisted (or the context is cancelled). This is safe to call from multiple goroutines concurrently.

func (*RecordWriter) WriteWithLineage added in v0.4.9 

func (w *RecordWriter) WriteWithLineage(ctx context.Context, rr *httpmsg.HttpRequestResponse, source string, projectUUID string, lineage RecordLineage) (string, error)

WriteWithLineage is Write carrying the record's full chain metadata. See RecordLineage, and admit for why it is applied to new rows only.

type RecordWriterConfig 

type RecordWriterConfig struct {
	// BufferSize is the channel capacity. Senders block when the buffer is full,
	// providing natural backpressure. Default: 4096.
	BufferSize int

	// BatchSize is the maximum number of records flushed in a single transaction.
	// Default: 128.
	BatchSize int

	// Shards is the number of independent flush goroutines. Each shard has its
	// own channel and flushLoop. Records are routed to shards by hashing the
	// host name, so writes for the same host are serialized within a shard.
	// Default (when <= 0): 1 for SQLite, 4 for PostgreSQL. SQLite is a
	// single-writer database, so multiple flush goroutines would only contend
	// on BEGIN IMMEDIATE / busy_timeout without gaining write parallelism;
	// PostgreSQL supports genuinely concurrent writers, so it fans out.
	Shards int

	// FlushTimeout bounds the shutdown drain, not steady-state flushes. Normal
	// flushes run on an uncancellable context.Background() so a slow insert never
	// drops records; only the drain triggered by Close() is capped by this single
	// budget, so a wedged database can't block Close() forever while a healthy one
	// still drains in full. Default: 2m (far longer than any healthy drain).
	FlushTimeout time.Duration

	// OperationTimeout bounds ONE steady-state flush. Steady flushes deliberately
	// do not inherit the scan's context — cancelling a scan mid-transaction would
	// abort records already pulled off the channel and fail writers waiting on
	// their results — but "uncancellable" was implemented as context.Background(),
	// i.e. no bound at all: one wedged transaction parked a shard's flush loop for
	// the rest of the process, its channel filled, and every Write routed to that
	// host blocked behind it. This is the bound, set far above SQLite's 60s
	// busy_timeout so a merely busy database still succeeds. Default: 2m.
	OperationTimeout time.Duration

	// DedupCacheSize bounds the in-memory dedup cache (LRU) that maps a record's
	// dedup key to its UUID. A cache hit lets Write skip the per-record SELECT and
	// the redundant insert for a key already seen by this writer (common in
	// discovery/spidering, which re-encounter the same URLs). Zero means "unset"
	// and selects the default of 50000; pass a NEGATIVE size to disable the cache.
	DedupCacheSize int

	// MaxSubmitRecords and MaxSubmitBytes bound ONE SaveRecordBatch submission:
	// how many records (and how many raw request+response bytes) it admits before
	// waiting for that slice to resolve. They cap the peak memory of a bulk caller
	// that hands over a whole collection at once — without them, peak retention is
	// set by the caller's corpus size and nothing else.
	//
	// Both defaults sit well above BatchSize, so a submission still fills many
	// writer batches and the coalescing SaveRecordBatch exists for is preserved.
	// Defaults: 1024 records, 128 MiB.
	MaxSubmitRecords int
	MaxSubmitBytes   int64

	// ScanUUID, when set, is stamped on every row this writer INSERTS, unless
	// the write's own lineage names a scan. A dedup hit (cache or batched lookup)
	// keeps the existing row's scan_uuid: the column records the first scan to
	// store the exchange. One writer belongs to one scan phase, so the runner
	// sets it once rather than threading it through every caller (the browser
	// crawler writes through this writer without knowing about scans).
	ScanUUID string
	// contains filtered or unexported fields
}

RecordWriterConfig configures the batching behavior of RecordWriter.

type RecordWriterMetrics 

type RecordWriterMetrics struct {
	Enqueued    int64
	Flushed     int64
	FlushFailed int64
	FlushErrors int64
	BatchCount  int64
	BufferDepth int64
}

RecordWriterMetrics exposes counters for monitoring.

Flushed and FlushFailed partition the batch entries that reached a terminal outcome: Flushed counts entries that ended up persisted or deduplicated onto an existing row, FlushFailed counts entries whose insert transaction failed. FlushErrors counts failing BATCHES, so it is not comparable with either — one failed batch fails every entry in it.

type Repository 

type Repository struct {

	// OnRecordSaved, when non-nil, is invoked with each newly inserted HTTP
	// record (deduplicated saves and dedup hits do NOT fire). OnFindingSaved is
	// the equivalent for newly inserted findings (a dedup-append to an existing
	// finding does not fire). Both run synchronously on the save goroutine, so
	// implementations must be fast and non-blocking (e.g. enqueue and return).
	// Used by the server's --mirror-fs filesystem mirror; left nil elsewhere, so
	// CLI scans and other repo users are unaffected.
	OnRecordSaved  func(*HTTPRecord)
	OnFindingSaved func(*Finding)
	// contains filtered or unexported fields
}

Repository handles HTTP record and finding storage

func NewRepository 

func NewRepository(db *DB) *Repository

NewRepository creates a new repository instance

func (*Repository) AdoptRecord added in v0.4.9 

func (r *Repository) AdoptRecord(ctx context.Context, uuid string, lineage RecordLineage, source string)

AdoptRecord repairs a row that lost a dedup race, in ONE statement: it backfills the chain metadata the winning row never had, and promotes a finding-evidence source to the crawl source of the phase that just fetched it.

Why any of this is needed. A passive module persists the response it reports on BEFORE the item's own save runs (processItem runs the passive stage first), so on a passive-heavy phase the evidence row routinely wins the dedup and the phase's own row — its lineage and its source label — is discarded. The visible symptom was that `vigolium traffic --source probe` returned a sweep's redirect hops and hid every page it actually fetched.

The deeper fix is to save before the passive stage rather than repair afterwards; that is a behavioural change to processItem and belongs in its own commit. Until then this is the repair, and it must stay cheap: it runs once per deduped record, which on a sweep is most of them.

One statement, not five. Every column here is conditional in SQL rather than in Go, so a no-op costs nothing beyond the row lock and the caller pays one round trip instead of one per column. Each condition encodes the same promote-only contract:

  • parent_uuid, root_uuid, target: backfilled when empty, NEVER overwritten. A shared redirect destination already belongs to the chain it was first recorded in; re-rooting it to the most recent traversal would rewrite history for every earlier chain through the same URL. A root_uuid equal to the row's own uuid counts as empty — that is the "no chain" default.
  • chain_truncated: only ever raised. A row once known to be cut short does not become whole because a later, shorter walk reached it.
  • source: a finding-evidence source is displaced because it describes HOW the row first reached the database, not what the row IS. A crawl source is never displaced by another crawl source, or "which phase found this" would degrade into "which phase saw it last".

Best-effort: a failure leaves the row readable under its old lineage, which reads as less information rather than wrong information, and must not fail the save that prompted it.

func (*Repository) AdoptRecordParent 

func (r *Repository) AdoptRecordParent(ctx context.Context, uuid, parentUUID string)

AdoptRecordParent links an existing record to a parent, but ONLY when it has none. A no-op for an empty uuid or parent.

Backfill-never-overwrite is the whole contract. Filling an empty parent adds information and cannot contradict anything. Overwriting a set one would let the most recent scan to traverse a shared redirect target silently re-parent it out of the chain it was first recorded in, rewriting history for every earlier chain that ran through the same URL.

Best-effort: a failure to link leaves a shorter chain, which reads correctly as less lineage rather than as wrong lineage, and must not fail the save that prompted it.

A thin wrapper on AdoptRecord, which applies every column in one statement; kept because "link this orphan to its parent" is a question callers ask on its own.

func (*Repository) AdvanceScanCursor 

func (r *Repository) AdvanceScanCursor(ctx context.Context, scanUUID string, recordCreatedAt time.Time, recordUUID string) error

AdvanceScanCursor updates the cursor position and increments ProcessedCount.

func (*Repository) AdvanceScanCursorBy 

func (r *Repository) AdvanceScanCursorBy(ctx context.Context, scanUUID string, recordCreatedAt time.Time, recordUUID string, delta int64) error

AdvanceScanCursorBy updates the cursor position and increments ProcessedCount by delta.

func (*Repository) AgenticScanStatusByUUID 

func (r *Repository) AgenticScanStatusByUUID(ctx context.Context, uuid string) (string, error)

AgenticScanStatusByUUID returns just the status column for a run.

It exists for the SSE log follower, which asks "is this run finished yet?" twice a second for as long as a client watches: doing that through GetAgenticScan reads the run's full row — blobs included — so N followers on a finished-but-still-watched autopilot run pull 2N megabyte-scale reads per second out of the database to learn one short string.

func (*Repository) AppendRemarks 

func (r *Repository) AppendRemarks(ctx context.Context, annotations map[string][]string) error

AppendRemarks batch-appends remarks to HTTPRecords identified by UUID. Existing remarks are preserved and duplicates within each record are deduplicated.

The read and the merged write happen in ONE transaction. Remarks are a growing set written by several analysis passes against the same records, and reading the current value outside the transaction turns every append into a read- modify-write race: two passes both read ["a"], both write their own merge, and whichever commits second silently drops the other's remark. The symptom is a missing annotation, which looks like the analysis never fired.

SQLite's write lock serializes the transactions outright. On PostgreSQL the rows are locked FOR UPDATE, so a concurrent appender blocks on the read rather than proceeding from a stale copy.

func (*Repository) ApplyDeparosStatusPolicy 

func (r *Repository) ApplyDeparosStatusPolicy(ctx context.Context, projectUUID string, policy DeparosStatusPolicy) (RecordCleanupResult, error)

ApplyDeparosStatusPolicy enforces the discovery status-retention policy on stored deparos records. A fuzzed path the server answers with a 4xx is usually not a discovered resource — it's a rejection (malformed/ambiguous path → 400, not-found → 404). But some 4xx statuses carry real attack surface: 401/403 prove an authz boundary exists, 405/415/422 prove a route exists but was hit with the wrong method/payload, and 429 is a pacing signal. The policy retains those (see DeparosStatusPolicy) and drops the rest, rather than deleting every 4xx and losing the signal.

Returns the number of deleted records and a status-code breakdown of them.

func (*Repository) AttributeScanToAgenticScan 

func (r *Repository) AttributeScanToAgenticScan(ctx context.Context, scanUUID, agenticScanUUID string) error

AttributeScanToAgenticScan links a native scan (and every finding it produced) to the agentic scan that launched it. The run_native_scan tool calls this after a blocking LaunchScan returns, so the child scan's findings show up under `finding --agentic-scan <parent>` — which filters on finding.agentic_scan_uuid directly rather than joining through scans. Both the scan row and its findings are stamped; a no-op when either UUID is empty.

func (*Repository) BackfillRecordResponse 

func (r *Repository) BackfillRecordResponse(ctx context.Context, uuid string, rr *httpmsg.HttpRequestResponse) error

BackfillRecordResponse populates the response columns of an existing request-only http_record stub from a freshly fetched request/response, computing every response-derived field (content-type/length, hashes, normalized hash, word count, title) via the shared FromHttpRequestResponse converter so the backfilled row matches a normally-saved record.

Discovery stores endpoints synthesized from a parsed OpenAPI/Swagger/Postman spec as request-only stubs (status 0, empty body); the executor then fetches each baseline to scan the endpoint, but without this the stub stays at status 0 with an empty response in the traffic view. The update is guarded on has_response = false so it only ever fills a genuine stub — it never clobbers a record that already captured a response, making it idempotent across the discovery and dynamic-assessment phases. An rr with no response is a no-op.

func (*Repository) CompleteScan 

func (r *Repository) CompleteScan(ctx context.Context, scanUUID string, errMsg string) error

CompleteScan marks a scan as completed (or failed if errMsg is non-empty) and populates severity counts from the findings table.

Leaves the completeness columns untouched, so a caller with no phase-outcome data (the server's own finalization paths, an agent rescan) records "unknown" rather than asserting a coverage verdict it cannot know. The runner, which does see every phase, calls CompleteScanWithOutcome instead.

func (*Repository) CompleteScanWithOutcome added in v0.5.2 

func (r *Repository) CompleteScanWithOutcome(ctx context.Context, scanUUID string, errMsg string, c ScanCompletion) error

CompleteScanWithOutcome is CompleteScan plus the scan's coverage verdict: how complete the run was, the single reason it stopped short, and the per-phase outcomes behind that verdict.

The status vocabulary is deliberately unchanged — "completed" still means the scan reached its end, and existing consumers keep working. Completeness is a SECOND axis alongside it, because the two are independent facts: a run can reach its end having covered half its input, and conflating that into a new status value would break every caller that switches on the four it knows.

A zero ScanCompletion writes nothing to the three columns, which is how a caller says "I don't know" without overwriting a verdict with a guess.

func (*Repository) CountRecordsAfterCursor 

func (r *Repository) CountRecordsAfterCursor(ctx context.Context, projectUUID string, cursorAt time.Time, cursorUUID string, hosts ...HostTarget) (int64, error)

CountRecordsAfterCursor counts a project's records after the given cursor position. A zero cursorAt means count all of them. When hosts is non-empty, only records matching those in-scope origins (scheme+hostname+port) are counted.

projectUUID is required rather than optional so that spanning projects has to be asked for: passing "" still counts every project, for a merge or an export that deliberately does. (This query had no project predicate at all, and returned the whole table on any multi-project store — see TestCountRecords_IsProjectScoped for what that broke.)

func (*Repository) CountRecordsAfterCursorBySource 

func (r *Repository) CountRecordsAfterCursorBySource(ctx context.Context, projectUUID string, cursorAt time.Time, cursorUUID string, sources []string, hosts []HostTarget) (int64, error)

CountRecordsAfterCursorBySource is like CountRecordsAfterCursor but also filters on http_records.source. Used by scan-on-receive shallow mode to report only user-ingested traffic in the "new ingested records" status, excluding finding/scanner artefacts produced by the scan itself.

func (*Repository) CountRecordsByStatusCode 

func (r *Repository) CountRecordsByStatusCode(ctx context.Context, projectUUID string, hosts ...HostTarget) (map[int]int64, error)

CountRecordsByStatusCode returns http_record counts grouped by HTTP status code, restricted to the same project and in-scope origins as CountRecordsAfterCursor (empty hosts = every origin, empty projectUUID = every project). Keys are the raw numeric status codes (0 for a missing/unset status); callers bucket them into 2xx/3xx/… classes.

It powers the scan-completion summary's status-class line, printed as a breakdown OF the record count beside it, so its project and host scope has to match that count's exactly or the classes do not sum to the total.

func (*Repository) CountRecordsForScan 

func (r *Repository) CountRecordsForScan(ctx context.Context, projectUUID, scanUUID string) (int64, error)

CountRecordsForScan returns how many HTTP records this scan persisted.

func (*Repository) CreateAgenticScan 

func (r *Repository) CreateAgenticScan(ctx context.Context, run *AgenticScan) error

CreateAgenticScan stores a new agent run record. When a row with run.UUID already exists, the call is a no-op as long as the existing project_uuid matches — this is the get-or-create path used for cross-node sync via --scan-uuid. Returns ErrScanProjectMismatch when the existing row belongs to a different project.

func (*Repository) CreateProject 

func (r *Repository) CreateProject(ctx context.Context, project *Project) error

CreateProject inserts a new project.

func (*Repository) CreateScan 

func (r *Repository) CreateScan(ctx context.Context, scan *Scan) error

CreateScan inserts a new scan record. When a row with scan.UUID already exists, the call is a no-op as long as the existing project_uuid matches — this is the get-or-create path used for cross-node sync via --scan-uuid. Returns ErrScanProjectMismatch when the existing row belongs to a different project.

func (*Repository) CreateScanLog 

func (r *Repository) CreateScanLog(ctx context.Context, log *ScanLog) error

CreateScanLog inserts a scan log entry.

func (*Repository) CreateScanLogBatch 

func (r *Repository) CreateScanLogBatch(ctx context.Context, logs []*ScanLog) error

CreateScanLogBatch inserts multiple scan log entries in a single bulk insert.

func (*Repository) CreateScanWithCursor 

func (r *Repository) CreateScanWithCursor(ctx context.Context, scan *Scan) error

CreateScanWithCursor creates a Scan record. If mode is "incremental", copies cursor from the last completed scan with matching Modules. Otherwise starts at zero.

func (*Repository) CreateUser 

func (r *Repository) CreateUser(ctx context.Context, user *User) error

CreateUser inserts a new user.

func (*Repository) DB 

func (r *Repository) DB() *DB

DB returns the underlying database handle.

func (*Repository) DeduplicateDeparosByNormHash 

func (r *Repository) DeduplicateDeparosByNormHash(ctx context.Context, projectUUID string) (RecordCleanupResult, error)

DeduplicateDeparosByNormHash collapses deparos records whose NORMALIZED response body is identical — records that differ only by a reflected request URL/path or per-request dynamic tokens (timestamps, ids). This is the case the exact (hostname, status, content_length, response_hash) dedup misses: an error or echo page that mirrors the requested URI gives every probed path a distinct body, length, and hash, so they all survive as separate records even though they are the same page. response_norm_hash (computed at storage time with the reflected URL/path stripped and dynamic runs collapsed) makes them identical.

Grouping is on (hostname, method, status_code, response_content_type, response_norm_hash); the shortest path per group survives. Records with no normalized hash (empty bodies) are left to exact-hash dedup. Returns the number of deleted records and a status-code breakdown.

func (*Repository) DeduplicateDeparosRecords 

func (r *Repository) DeduplicateDeparosRecords(ctx context.Context, projectUUID string) (int64, error)

DeduplicateDeparosRecords removes duplicate deparos HTTP records. Delegates to DeduplicateRecordsBySource with source "deparos".

func (*Repository) DeduplicateFindings 

func (r *Repository) DeduplicateFindings(ctx context.Context, projectUUID string, hostnames ...string) (deleted int64, grouped int64, err error)

DeduplicateFindings merges duplicate findings that share the same (module_id, severity, matched_at URL) within a project. This collapses findings where the same module fires many times on the same URL with different payloads (e.g., input-behavior-probe producing dozens of results per endpoint). Within each group, the earliest finding is kept and the request/response pairs from duplicates are collected into its AdditionalEvidence field.

When one or more hostnames are passed, the pass is scoped to findings on those hosts (an empty/omitted list dedupes the whole project). The dynamic-assessment feedback loop scopes each round to the hosts being scanned — coverage-equivalent there, since every finding produced during DA is on an in-scope host — so a growing findings table isn't full-scanned once per round. Returns the count of deleted findings and the number of groups that were merged.

func (*Repository) DeduplicateRecordsBySource 

func (r *Repository) DeduplicateRecordsBySource(ctx context.Context, projectUUID, source string) (int64, error)

DeduplicateRecordsBySource removes duplicate HTTP records for a given source that share identical (hostname, method, status_code, response_content_length, response_hash). Within each group, the record with the shortest path is kept. Returns the number of deleted records.

func (*Repository) DeduplicateSoftDeparosRecords 

func (r *Repository) DeduplicateSoftDeparosRecords(ctx context.Context, projectUUID string) (RecordCleanupResult, error)

DeduplicateSoftDeparosRecords removes deparos HTTP records that are "soft duplicates": same response characteristics (status, word count, content type) under the same 2-segment path prefix. This catches cases where the server echoes part of the URL in the response body, causing different response_hash values for functionally identical pages. Exact response_content_length is deliberately NOT part of the key: a reflected URL/path shifts the body length by a handful of bytes per probe, which would otherwise split an otherwise-identical family into singletons (the very case this pass exists to collapse) — word count + content type + status + path prefix is the stable shape signal. Only groups with 3+ members are collapsed. The shortest path per group is kept.

func (*Repository) DeleteAuthenticationHostname 

func (r *Repository) DeleteAuthenticationHostname(ctx context.Context, id int64) error

DeleteAuthenticationHostname deletes a single session hostname by ID.

func (*Repository) DeleteAuthenticationHostnamesByHostname 

func (r *Repository) DeleteAuthenticationHostnamesByHostname(ctx context.Context, projectUUID, hostname string) error

DeleteAuthenticationHostnamesByHostname deletes all session hostnames for a project+hostname.

func (*Repository) DeleteFinding 

func (r *Repository) DeleteFinding(ctx context.Context, id int64) error

DeleteFinding deletes a finding by its numeric ID, including its findingChildTables rows (evidence junction, observations, status history).

func (*Repository) DeleteFindingByHash 

func (r *Repository) DeleteFindingByHash(ctx context.Context, projectUUID, findingHash string) error

DeleteFindingByHash deletes the finding(s) with the given content hash within a project, including their findingChildTables rows. Used by the OAST collector to replace a weaker finding (e.g. a DNS-resolution lead) with a stronger one (the HTTP-fetch confirmation) for the same callback payload, so each payload yields exactly one finding reflecting the strongest evidence. No-op when findingHash is empty or no such finding exists.

func (*Repository) DeleteOASTInteraction 

func (r *Repository) DeleteOASTInteraction(ctx context.Context, id int64) error

DeleteOASTInteraction deletes an OAST interaction by its numeric ID.

func (*Repository) DeleteOldAgenticScans 

func (r *Repository) DeleteOldAgenticScans(ctx context.Context, olderThan time.Duration) (int, error)

DeleteOldAgenticScans removes finished agent runs older than the given duration. "Finished" is TerminalAgenticScanStatuses, not a literal list — see that variable.

func (*Repository) DeleteProject 

func (r *Repository) DeleteProject(ctx context.Context, uuid string) error

DeleteProject deletes a project by UUID.

func (*Repository) DeleteRecord 

func (r *Repository) DeleteRecord(ctx context.Context, uuid string) error

DeleteRecord deletes an HTTP record by UUID, including any finding_records junction rows.

func (*Repository) DeleteRequestOnlyStubs added in v0.5.3 

func (r *Repository) DeleteRequestOnlyStubs(ctx context.Context, projectUUID, source string, uuids []string) (int64, error)

DeleteRequestOnlyStubs removes request-only records that a check against the live target found not worth scanning: a 404/410, a soft-404, or a host that did not answer at all. Returns how many rows were deleted.

Every row is guarded on project, source AND has_response = false, so the delete can only ever take a stub the caller's own phase wrote. A harvested URL that deduped onto an existing record adopts that row's UUID — which may be a fully fetched record from another phase or an earlier scan — and the guard is what keeps a dead archive URL from deleting real evidence that happens to share its identity. finding_records junction rows go in the same transaction, mirroring DeleteRecord.

func (*Repository) DeleteScan 

func (r *Repository) DeleteScan(ctx context.Context, uuid string) error

DeleteScan deletes a scan record by UUID.

func (*Repository) DiffScans added in v0.5.3 

func (r *Repository) DiffScans(ctx context.Context, opts ScanDiffOptions) (*ScanDiff, error)

DiffScans classifies every finding observed by either scan: new = head \ base, unchanged = base ∩ head, and for base \ head fixed when the head is comparable (ScanComparability), not_reobserved otherwise. Dismissed findings are left out unless IncludeDismissed. Entries are ordered by class (DiffChanges), then severity, then id.

func (*Repository) ExistingRecordUUIDs 

func (r *Repository) ExistingRecordUUIDs(ctx context.Context, projectUUID string, uuids []string) ([]string, error)

ExistingRecordUUIDs returns the subset of uuids that exist. It selects the uuid column straight into a []string, so validating a caller-supplied list costs neither the raw_request/raw_response blobs nor an HTTPRecord struct per row.

The answer is always scoped to a project, blank meaning the default one as everywhere else in this package. The scoping is not optional because enforcing ownership is this method's whole purpose: record UUIDs are globally unique, so an unscoped check answers "does this row exist anywhere" when the question is "does the caller own it", and a caller who learns one UUID from another engagement could pull its traffic into their own scan. An opt-out parameter would make the safe and unsafe calls look identical.

func (*Repository) FindingProjectUUID 

func (r *Repository) FindingProjectUUID(ctx context.Context, id int64) (string, error)

FindingProjectUUID returns just the project a finding belongs to, for the existence-plus-scope check the mutating handlers perform before acting on it. Returns sql.ErrNoRows when the finding does not exist, like GetFindingByID.

Projected on purpose: the callers read only this one field, and a finding carries its own inline request/response blobs.

func (*Repository) FindingsInScan added in v0.5.3 

func (r *Repository) FindingsInScan(ctx context.Context, projectUUID, scanUUID string) ([]*Finding, error)

FindingsInScan returns the reportable findings (record_kind finding) a scan produced inside one project, ordered by id. projectUUID == "" drops the project predicate (a standalone `-S --db` store).

func (*Repository) GetAgenticScan 

func (r *Repository) GetAgenticScan(ctx context.Context, uuid string) (*AgenticScan, error)

GetAgenticScan retrieves an agent run by UUID.

func (*Repository) GetAllProjectsStats 

func (r *Repository) GetAllProjectsStats(ctx context.Context) (map[string]*ProjectStatsRow, error)

GetAllProjectsStats returns aggregated stats for all projects in bulk. Uses GROUP BY to avoid N+1 queries when listing projects.

func (*Repository) GetAuthenticationHostnamesByHostname 

func (r *Repository) GetAuthenticationHostnamesByHostname(ctx context.Context, projectUUID, hostname string) ([]*AuthenticationHostname, error)

GetAuthenticationHostnamesByHostname returns session hostnames for a project+hostname, ordered by position.

func (*Repository) GetAuthenticationHostnamesByProject 

func (r *Repository) GetAuthenticationHostnamesByProject(ctx context.Context, projectUUID string) ([]*AuthenticationHostname, error)

GetAuthenticationHostnamesByProject returns all session hostnames for a project, ordered by hostname then position.

func (*Repository) GetAuthenticationHostnamesByScan 

func (r *Repository) GetAuthenticationHostnamesByScan(ctx context.Context, projectUUID, scanUUID string) ([]*AuthenticationHostname, error)

GetAuthenticationHostnamesByScan returns session hostnames for a project+scan, ordered by hostname then position.

func (*Repository) GetChildAgenticScans 

func (r *Repository) GetChildAgenticScans(ctx context.Context, parentUUID string) ([]*AgenticScan, error)

GetChildAgenticScans returns agent runs whose ParentAgenticScanUUID matches the given UUID.

func (*Repository) GetDistinctHosts 

func (r *Repository) GetDistinctHosts(ctx context.Context, projectUUID string, since ...time.Time) ([]HostTarget, error)

GetDistinctHosts returns distinct scheme+hostname+port combinations from HTTP records, filtered by project. When a non-zero `since` is supplied, only records created at/after that time are considered (used to find origins discovered during a specific scan).

func (*Repository) GetDistinctPaths 

func (r *Repository) GetDistinctPaths(ctx context.Context, projectUUID string, hosts ...HostTarget) ([]PathTarget, error)

GetDistinctPaths returns distinct scheme+hostname+port+path combinations from HTTP records, filtered by project and, when hosts is non-empty, restricted to those in-scope origins (matching the executor's WithHostScopes convention). Empty hosts means no origin filter.

func (*Repository) GetFindingByID 

func (r *Repository) GetFindingByID(ctx context.Context, id int64) (*Finding, error)

GetFindingByID retrieves a single finding by numeric ID.

func (*Repository) GetFindingsByRecordUUID 

func (r *Repository) GetFindingsByRecordUUID(ctx context.Context, uuid string) ([]*Finding, error)

GetFindingsByRecordUUID retrieves findings that reference a specific HTTP record UUID. Since http_record_uuids is a JSONB array, we use json_each to search inside it.

func (*Repository) GetFindingsBySeverity 

func (r *Repository) GetFindingsBySeverity(ctx context.Context, projectUUID, sev string, limit int) ([]*Finding, error)

GetFindingsBySeverity retrieves findings filtered by severity within a project.

func (*Repository) GetOASTInteractionByID 

func (r *Repository) GetOASTInteractionByID(ctx context.Context, id int64) (*OASTInteraction, error)

GetOASTInteractionByID retrieves a single OAST interaction by its numeric ID.

func (*Repository) GetOASTInteractionsByScan 

func (r *Repository) GetOASTInteractionsByScan(ctx context.Context, scanUUID string) ([]*OASTInteraction, error)

GetOASTInteractionsByScan retrieves OAST interactions for a specific scan.

func (*Repository) GetOASTPayloadsByNonce added in v0.5.3 

func (r *Repository) GetOASTPayloadsByNonce(ctx context.Context, projectUUID string, nonces []string) (map[string]*OASTPayload, error)

GetOASTPayloadsByNonce returns the stored payload contexts for the given nonces, keyed by nonce. Nonces with no stored row are absent from the map. A non-empty projectUUID restricts the lookup to that project; an empty one (a stateless database) searches every row.

func (*Repository) GetProjectByName 

func (r *Repository) GetProjectByName(ctx context.Context, name string) (*Project, error)

GetProjectByName retrieves a project by name. Returns an error if zero or multiple projects match (names are not guaranteed to be unique).

func (*Repository) GetProjectByUUID 

func (r *Repository) GetProjectByUUID(ctx context.Context, uuid string) (*Project, error)

GetProjectByUUID retrieves a project by UUID.

func (*Repository) GetProjectStats 

func (r *Repository) GetProjectStats(ctx context.Context, projectUUID string) (*ProjectStatsRow, error)

GetProjectStats returns aggregated stats for a single project.

func (*Repository) GetReSpiderCandidates 

func (r *Repository) GetReSpiderCandidates(ctx context.Context, projectUUID, afterUUID string, limit int, hosts ...HostTarget) ([]ReSpiderCandidate, error)

GetReSpiderCandidates returns records that carry a response body, for the targeted re-spider phase, using UUID-based cursor pagination (afterUUID="" for the first page). It deliberately includes spidering-sourced records too, so the caller can pre-seed its shell-dedup set with pages the browser already crawled. Ordered by uuid so paging is stable and per-host capping deterministic. Each returned row carries a full raw_response body, so the caller pages with a modest limit and reduces each page before reading the next rather than materializing every body at once.

func (*Repository) GetRecordByRequestHash 

func (r *Repository) GetRecordByRequestHash(ctx context.Context, projectUUID, requestHash string) (*HTTPRecord, error)

GetRecordByRequestHash retrieves the most recent HTTP record whose raw request hashes to requestHash (the SHA-256 over the raw request bytes, matching httpmsg.HttpRequest.ID). It recovers the originating request for an out-of-band (OAST) finding when the in-memory hash→UUID resolver is no longer available — e.g. a callback that lands after the executor that planted the payload has been torn down. projectUUID may be empty to search across projects.

func (*Repository) GetRecordByUUID 

func (r *Repository) GetRecordByUUID(ctx context.Context, uuid string) (*HTTPRecord, error)

GetRecordByUUID retrieves a single HTTP record by UUID

func (*Repository) GetRecordByUUIDOrPrefix added in v0.5.1 

func (r *Repository) GetRecordByUUIDOrPrefix(ctx context.Context, idOrPrefix string) (*HTTPRecord, error)

GetRecordByUUIDOrPrefix retrieves a record by full UUID, falling back to expanding an unambiguous prefix. Returns sql.ErrNoRows when nothing matches, and an error naming the candidates when a prefix is ambiguous.

Agents read record UUIDs out of a JSON tool result and re-type them into the next call; transcribing 36 hex characters is something models get wrong regularly (an observed run produced one UUID with an invented head and another that spliced two records together). A prefix is short enough to copy reliably.

The full-UUID path costs exactly one primary-key lookup, same as before - the prefix scan only runs once that has missed.

func (*Repository) GetRecordMetadataByHostname 

func (r *Repository) GetRecordMetadataByHostname(ctx context.Context, projectUUID, hostname string, limit int) ([]*HTTPRecord, error)

GetRecordMetadataByHostname is GetRecordsByHostname without the raw_request / raw_response BLOB columns, for callers that only need request-line metadata (method, URL, path, status). It avoids pulling multi-KB bodies for every row when they're immediately discarded — notably the autopilot coverage probe, which scans up to 5000 rows just to build (method, URL) signatures. Use the full GetRecordsByHostname when the bodies are needed.

func (*Repository) GetRecordsByHostname 

func (r *Repository) GetRecordsByHostname(ctx context.Context, projectUUID, hostname string, limit int) ([]*HTTPRecord, error)

GetRecordsByHostname retrieves HTTP records for a hostname within a project.

func (*Repository) GetRecordsByUUIDs 

func (r *Repository) GetRecordsByUUIDs(ctx context.Context, uuids []string) ([]*HTTPRecord, error)

GetRecordsByUUIDs retrieves HTTP records matching the given UUIDs.

func (*Repository) GetRecordsWithResponseBody 

func (r *Repository) GetRecordsWithResponseBody(ctx context.Context, projectUUID, afterUUID string, limit int, hosts ...HostTarget) ([]*HTTPRecord, error)

GetRecordsWithResponseBody returns HTTP records that have a non-empty response body, using UUID-based cursor pagination. Only columns needed for batch secret scanning are selected.

func (*Repository) GetRelatedRecords 

func (r *Repository) GetRelatedRecords(ctx context.Context, uuid string, limit int) ([]*HTTPRecord, error)

GetRelatedRecords finds HTTP records with the same hostname and a path matching the path-template of the given UUID's record. Default limit 10; excludes the source record itself. Results are filtered to the same path depth as the source record.

func (*Repository) GetScanByUUID 

func (r *Repository) GetScanByUUID(ctx context.Context, uuid string) (*Scan, error)

GetScanByUUID retrieves a scan by its UUID

func (*Repository) GetScanRecordsByUUIDs 

func (r *Repository) GetScanRecordsByUUIDs(ctx context.Context, uuids []string) ([]*HTTPRecord, error)

GetScanRecordsByUUIDs retrieves HTTP records matching the given UUIDs, projecting only scanRecordColumns — what the dynamic-assessment scan feed consumes. This avoids loading (and JSON-deserializing) the parameters/technology/remarks jsonb columns plus ~25 unused scalar columns per record. Modules receive the reconstructed HttpRequestResponse (via recordToHttpRequestResponse), never the HTTPRecord, so no downstream consumer reads the dropped fields.

func (*Repository) GetUnprobedRecordsBySource 

func (r *Repository) GetUnprobedRecordsBySource(ctx context.Context, projectUUID, source, hostname string, limit int) ([]*HTTPRecord, error)

GetUnprobedRecordsBySource returns records with has_response=false for the given source and hostname.

func (*Repository) GetUserByUUID 

func (r *Repository) GetUserByUUID(ctx context.Context, uuid string) (*User, error)

GetUserByUUID retrieves a user by UUID.

func (*Repository) GroupFindingsByValue 

func (r *Repository) GroupFindingsByValue(ctx context.Context, projectUUID string, opts GroupFindingOptions) (deleted int64, grouped int64, err error)

GroupFindingsByValue collapses findings that share an identical extracted value (e.g. the same leaked secret reported once per URL) into a single finding. The survivor (earliest by created_at) absorbs every duplicate's matched URLs into MatchedAt and a sample of their request/response pairs into AdditionalEvidence; the duplicates are then deleted. Grouping is keyed on (module_id, severity[, hostname], normalized extracted_results) — the value must match byte-for-byte, so findings with distinct or empty extracted values are never merged.

Modules listed in opts.ByModule are an exception: their findings collapse on (module_id, severity[, hostname]) alone, regardless of (or absent) an extracted value, and skip the Tags gate. This is for modules that fire once per asset where the per-URL value is noise (e.g. sourcemap-detect, a distinct .map filename per JS bundle).

Modules listed in opts.ByRule collapse on (module_id, rule_name, severity[, hostname]) — the same as ByModule but with the rule (module_name) kept in the key, so a module whose single id fronts many rules (secret-detect) folds repeats of one rule while keeping different rules apart. Their distinct values are unioned onto the survivor just like ByModule.

Returns the count of deleted findings and the number of groups that were collapsed.

func (*Repository) HasHostObservations 

func (r *Repository) HasHostObservations(ctx context.Context) bool

HasHostObservations reports whether this database can answer endpoint lookups at all.

A database written before this table existed has nothing stored, which is a fact about it and not a failure. Asked once up front so a read of an older file degrades to the cache-only behaviour silently, instead of failing a query per endpoint about a table it was never going to have. A read-only handle cannot be migrated into having one either, and missingColumns only tracks ALTER-added columns, never whole tables.

func (*Repository) HydrateFindingRequests 

func (r *Repository) HydrateFindingRequests(ctx context.Context, findings []*Finding)

HydrateFindingRequests populates the Request field on findings that were loaded without it — the list query (FindingsQueryBuilder) omits the request/response blobs for speed. It fetches id→request for the given findings in as few round-trips as one statement's worth of ids allows; findings that already carry a Request, or that have no id, are skipped. Used by display paths that need a finding's own PoC request (e.g. the tree's attack-URL rendering) without pulling the full raw view.

func (*Repository) InScopeHosts 

func (r *Repository) InScopeHosts(ctx context.Context, scopeCfg config.ScopeConfig, targets []string, projectUUID, scanUUID string) []HostTarget

InScopeHosts returns the distinct (scheme, hostname, port) origins in the project that are in scope for the supplied CLI targets. An origin is in scope when its hostname passes the scope matcher AND EITHER its (scheme, port) matches one of the targets' origins (default ports normalized like stored records: https→443, http→80) OR it was discovered during the current scan (records created at/after the scan's start). Returns nil when there are no targets (no filter — a project-wide pass). This is the single source of truth for the origin scoping the executor applies via WithHostScopes, so DynamicAssessment, KnownIssueScan, spidering/discovery seeds, and the scan-completion record count all derive the same set.

The full-origin match keeps e.g. localhost:3000 separate from localhost:8080 left in the project by a prior scan, while the current-scan provenance carve-out still lets THIS scan's own discoveries on a different port (e.g. the --intensity deep port sweep, or a followed cross-port link) be scanned. Records carry no scan_uuid, so provenance is keyed on the scan's start time. If no target yields a determinable scheme/port (e.g. all bare hosts), the scheme/port constraint is dropped — a safe fallback that never over-excludes.

func (*Repository) IncrementProcessedCount 

func (r *Repository) IncrementProcessedCount(ctx context.Context, scanUUID string, delta int64) error

IncrementProcessedCount adds delta to the scan's processed_count. Use this for phases that don't advance the cursor (discovery, spidering, etc.).

func (*Repository) LastScanLogTimes added in v0.5.3 

func (r *Repository) LastScanLogTimes(ctx context.Context, projectUUIDs, scanUUIDs []string) (map[string]time.Time, error)

LastScanLogTimes returns, per scan, the newest scan_logs.created_at — one GROUP BY over every listed scan, for liveness readers that list many scans at once. Scans with no log lines are absent from the map.

projectUUIDs narrows the read to the projects the scans belong to, which is what lets the (project_uuid, scan_uuid) index serve it; pass none to read across projects (a stateless read of a standalone file).

func (*Repository) ListAgentSections 

func (r *Repository) ListAgentSections(ctx context.Context, agenticScanUUID string) ([]*AgentSection, error)

ListAgentSections returns all sections for an agentic scan ordered by seq (execution order). Empty agenticScanUUID returns an empty slice.

func (*Repository) ListAgenticScanSummaries 

func (r *Repository) ListAgenticScanSummaries(ctx context.Context, projectUUID string, mode string, limit, offset int) ([]*AgenticScan, int64, error)

ListAgenticScanSummaries is ListAgenticScans with agenticScanBlobColumns dropped — same rows, same order, same total. Use it for any view that renders a run rather than replaying it; the omitted fields come back empty, so a caller that serializes the *AgenticScan verbatim (the `--json` CLI paths) must keep using ListAgenticScans.

func (*Repository) ListAgenticScans 

func (r *Repository) ListAgenticScans(ctx context.Context, projectUUID string, mode string, limit, offset int) ([]*AgenticScan, int64, error)

ListAgenticScans returns paginated agent runs for a project, ordered by created_at DESC, with every column populated. Callers that only render the run should prefer ListAgenticScanSummaries.

func (*Repository) ListCandidates 

func (r *Repository) ListCandidates(ctx context.Context, agenticScanUUID string, statuses ...string) ([]*AgentFindingCandidate, error)

ListCandidates returns candidates for an agentic scan, optionally filtered to the given statuses (empty statuses = all). Ordered by id so the verifier processes them in proposal order.

func (*Repository) ListClientRoutes added in v0.5.3 

func (r *Repository) ListClientRoutes(ctx context.Context, scanUUID, projectUUID string, hosts []HostTarget, limit int) ([]ClientRoute, error)

ListClientRoutes returns the client routes projectUUID's jstangle-facts artifacts declare, oldest artifact first, deduplicated per origin and route. scanUUID, when set, keeps only artifacts stamped with that scan (their anchor record's scan); hosts, when set, keeps only routes anchored on those origins. limit caps the routes returned (0 = no cap). Routes are returned as declared: filtering redirects and low-confidence guesses and making parameters concrete is the caller's job (see clientroute).

func (*Repository) ListFindingObservations added in v0.5.3 

func (r *Repository) ListFindingObservations(ctx context.Context, projectUUID string, findingID int64) ([]FindingObservation, error)

ListFindingObservations returns the observations of one finding, oldest first. projectUUID == "" drops the project predicate (scoping off).

func (*Repository) ListFindingStatusEvents added in v0.5.3 

func (r *Repository) ListFindingStatusEvents(ctx context.Context, projectUUID string, findingID int64) ([]FindingStatusEvent, error)

ListFindingStatusEvents returns a finding's status history, oldest first. projectUUID == "" drops the project predicate (scoping off).

func (*Repository) ListFindings 

func (r *Repository) ListFindings(ctx context.Context, filters QueryFilters) ([]*Finding, int64, error)

ListFindings runs a filtered findings query and returns the matching page plus the total unfiltered count, in a single round-trip. The canonical entry point for callers that want paginated results — keeps FindingsQueryBuilder behind the repository boundary so they don't need to reach for Repository.DB().

func (*Repository) ListOASTInteractions 

func (r *Repository) ListOASTInteractions(ctx context.Context, projectUUID, scanUUID, protocol, moduleID, search string, limit, offset int) ([]*OASTInteraction, int64, error)

ListOASTInteractions returns a paginated, filtered list of OAST interactions. Heavy columns (raw_request, raw_response) are excluded for list performance.

func (*Repository) ListOASTInteractionsByUniqueIDs added in v0.5.3 

func (r *Repository) ListOASTInteractionsByUniqueIDs(ctx context.Context, projectUUID string, ids []string) ([]*OASTInteraction, error)

ListOASTInteractionsByUniqueIDs returns the stored callbacks whose unique_id is one of ids (raw_response is not loaded). A non-empty projectUUID restricts the lookup to that project. `oast ingest` uses it to skip callbacks it already stored and to know the strongest protocol each nonce already saw.

func (*Repository) ListProjects 

func (r *Repository) ListProjects(ctx context.Context, ownerUUID string) ([]*Project, error)

ListProjects returns projects, optionally filtered by owner.

func (*Repository) ListRunningScans added in v0.5.3 

func (r *Repository) ListRunningScans(ctx context.Context, projectUUID string) ([]*Scan, error)

ListRunningScans returns every scan row whose stored status is still "running", newest first — the candidates a reaper checks for liveness. An empty projectUUID reads across projects (a stateless read of a standalone file), like ListScans.

func (*Repository) ListScanLogs 

func (r *Repository) ListScanLogs(ctx context.Context, scanUUID string, level, phase string, limit, offset int) ([]*ScanLog, int64, error)

ListScanLogs returns log entries for a scan, ordered by created_at ascending. Both level and phase are optional filters; pass "" to skip.

func (*Repository) ListScans 

func (r *Repository) ListScans(ctx context.Context, projectUUID string, limit, offset int) ([]*Scan, int64, error)

ListScans returns scans ordered by created_at descending with limit/offset, filtered by project.

func (*Repository) ListUsers 

func (r *Repository) ListUsers(ctx context.Context) ([]*User, error)

ListUsers returns all users.

func (*Repository) LoadEnabledScopes 

func (r *Repository) LoadEnabledScopes(ctx context.Context, projectUUID string) ([]*Scope, error)

LoadEnabledScopes loads enabled scope rules for a project, ordered by priority. An empty projectUUID means the default project — never "all projects". Falls back to the default project's scopes when a non-default project has none, but if the default project itself has no enabled scopes it returns empty rather than an unscoped query that would leak every project's rules across engagements.

func (*Repository) LookupHostObservation 

func (r *Repository) LookupHostObservation(ctx context.Context, projectUUID, scanUUID, hostname string, port int) (HostFacts, error)

LookupHostObservation returns the observation for one endpoint, or zero facts when none is stored.

One endpoint per call, rather than loading a run's observations up front. Preloading looked right for a streaming emit path - fetch once, then answer every record from memory - and was wrong in both directions. This table accumulates a row per endpoint per RUN, so the project-wide preload returned endpoints × runs rows to build a map of endpoints: on 20k endpoints across 25 runs, exporting a two-record database peaked at 842 MB against a 120 MB baseline, with export memory growing in scan history rather than in the corpus being exported. Resolving "latest per endpoint" in SQL instead still cost a full scan and a temp b-tree. And a paginated reader (`traffic -j`, 100 rows) paid all of it to ask about a handful of hosts.

A point lookup is index-only against idx_host_obs_project_host_id, and the caller memoizes, so the cost is one query per DISTINCT endpoint actually emitted and the memory is one entry per the same.

scanUUID empty widens the read to the whole project and takes the most recently RECORDED observation - the right answer for `export` of a database whose runs this process did not perform.

func (*Repository) ObservationHistorySince added in v0.5.3 

func (r *Repository) ObservationHistorySince(ctx context.Context) (time.Time, error)

ObservationHistorySince reports when this store started recording every finding observation (the schema v3 migration). Observations before it are the backfill's single row per finding, so a comparison involving a scan that started earlier is partial. The zero time means the marker is unset (a store not yet migrated).

func (*Repository) OverwriteRecordResponseBody 

func (r *Repository) OverwriteRecordResponseBody(ctx context.Context, uuid string, rawResponse []byte) error

OverwriteRecordResponseBody replaces the stored raw response of the record with the given UUID and recomputes its derived fields (response_hash, response_norm_hash, response_words, response_content_length), keeping them consistent with dedup/resume/fingerprinting that key off those columns.

rawResponse must be a complete HTTP response (status line + headers + body). Used by the passive js-beautify module to overwrite a minified JS body with its beautified form in place. The derivation mirrors Record.FromRequestResponse so a rewritten record hashes identically to one ingested with the new body. (Distinct from UpdateRecordResponse, which is the replay feature's full response-field swap.)

func (*Repository) ParamMiningCandidates added in v0.5.3 

func (r *Repository) ParamMiningCandidates(ctx context.Context, scanUUID string, hosts []HostTarget, limit int) ([]HTTPRecord, error)

ParamMiningCandidates returns the records the param-mining phase may mine: the ones the dynamic-assessment phase of scanUUID will see (after the scan cursor, in the project, within hosts), answered 2xx, GET or POST, and not param-mining output themselves. Ordered by risk_score, then surface_score, then age; at most limit rows. Content-type and per-endpoint/per-host selection is left to the caller.

func (*Repository) ParamNamesForHost added in v0.5.3 

func (r *Repository) ParamNamesForHost(ctx context.Context, projectUUID, hostname string) (HostParamNames, error)

ParamNamesForHost collects the parameter names projectUUID's records use on hostname, the first tier of the param-mining wordlist.

func (*Repository) PauseScan 

func (r *Repository) PauseScan(ctx context.Context, scanUUID string) error

PauseScan sets a scan's status to "paused".

func (*Repository) PurgeProjectData 

func (r *Repository) PurgeProjectData(ctx context.Context, projectUUID string) error

PurgeProjectData deletes every row tied to projectUUID across all per-project tables. finding_records has no project_uuid, so it's pruned via a subquery on findings.id before the findings rows themselves are removed. Runs in a single transaction so a partial failure leaves the project intact.

func (*Repository) ReapAbandonedScan added in v0.5.3 

func (r *Repository) ReapAbandonedScan(ctx context.Context, scan *Scan, lastWriteAt time.Time) (reaped bool, err error)

ReapAbandonedScan finalizes a scan whose row still says "running" but whose process is gone (the caller's liveness verdict): status failed, completeness partial, stop_reason abandoned, and an error_message naming the last write. finished_at is that last write when known (when the scan actually stopped), else now; phase_outcomes and the severity counts are left as they are.

Only a row still "running" is touched, so a scan that finalized between the caller's read and this write keeps its own verdict. reaped reports whether this call wrote the row.

func (*Repository) ReassignProjectData 

func (r *Repository) ReassignProjectData(ctx context.Context, sourceUUID, targetUUID string) error

ReassignProjectData moves all data owned by sourceUUID to targetUUID. This should be called before deleting a project so its records are not orphaned.

func (*Repository) RecordProjectUUID 

func (r *Repository) RecordProjectUUID(ctx context.Context, uuid string) (string, error)

RecordProjectUUID returns just the project a record belongs to, for the existence-plus-scope check the mutating handlers perform before acting on it. Returns sql.ErrNoRows when the record does not exist, like GetRecordByUUID.

Projected on purpose: the callers read only this one field, and a record's raw_response alone can be megabytes.

func (*Repository) RefreshScanStats 

func (r *Repository) RefreshScanStats(ctx context.Context, scanUUID string) error

RefreshScanStats updates running scan stats during long-running scans where CompleteScan hasn't been called yet.

func (*Repository) ReplayBaselineCandidates added in v0.5.3 

func (r *Repository) ReplayBaselineCandidates(ctx context.Context, projectUUID, scanUUID, hostname, method string, limit int) ([]*HTTPRecord, error)

ReplayBaselineCandidates returns recent scanned (non-evidence) records for one host and method inside a project, optionally narrowed to a scan, newest first: the pool `finding verify` searches for the baseline of a legacy finding whose only linked record is its own payload request.

func (*Repository) ResetScanCursor 

func (r *Repository) ResetScanCursor(ctx context.Context, scanUUID string) error

ResetScanCursor resets the scan cursor to the beginning so all records are re-read on the next iteration (e.g., between seed and audit phases).

func (*Repository) ResolveFindings added in v0.5.3 

func (r *Repository) ResolveFindings(ctx context.Context, projectUUID string, ids []int64, hashes []string) ([]*Finding, []string, error)

ResolveFindings loads the findings named by id and/or finding_hash (full or ≥MinFindingHashPrefix prefix) inside one project, ordered by id, with the same matching and ambiguity rules as UpdateFindingStatusScoped. Selectors that match nothing in scope come back in notFound.

func (*Repository) ResolveNewSince added in v0.5.3 

func (r *Repository) ResolveNewSince(ctx context.Context, baseline *Scan) (*NewSinceBaseline, error)

ResolveNewSince builds the --new-since predicate for a baseline scan already resolved in scope.

func (*Repository) ResolveRecordUUID added in v0.5.1 

func (r *Repository) ResolveRecordUUID(ctx context.Context, prefix string) (string, error)

ResolveRecordUUID expands a UUID prefix to the full UUID of the one record it matches. Returns sql.ErrNoRows when nothing matches, and an error naming the candidates when the prefix is ambiguous.

Implemented as a range scan over the primary key rather than LIKE 'prefix%': the btree is ordered by uuid, so a range is index-seekable under any collation, while LIKE falls back to a full table scan on both SQLite (BINARY collation, no case_sensitive_like pragma) and Postgres (no text_pattern_ops index). http_records is the largest table in the schema and its rows carry request/response bodies, so that scan is not cheap. A range also sidesteps LIKE metacharacters entirely - a prefix containing % or _ is just a prefix here, which matters when the input is a string a model may have mangled.

func (*Repository) ResolveScanRef added in v0.5.3 

func (r *Repository) ResolveScanRef(ctx context.Context, projectUUID, ref string) (*Scan, error)

ResolveScanRef resolves a scan reference within the project: a full UUID, a UUID prefix of at least 8 characters, or the keyword "latest" (the most recently started scan that is not still running) / "previous" (the one before it). A miss — including a scan in another project — is ErrScanNotFound; an ambiguous prefix names its candidates.

func (*Repository) ResumeScan 

func (r *Repository) ResumeScan(ctx context.Context, scanUUID string) error

ResumeScan sets a scan's status back to "running".

func (*Repository) SampleInsertionPoints added in v0.5.3 

func (r *Repository) SampleInsertionPoints(ctx context.Context, projectUUID string, hosts []HostTarget, limit int) (InsertionPointSample, error)

SampleInsertionPoints reads at most limit records of the project on the given in-scope origins (empty = every origin) — the parameters and path columns only, never a body — and averages their insertion points. Read-only.

func (*Repository) SaveAgentSection 

func (r *Repository) SaveAgentSection(ctx context.Context, section *AgentSection) error

SaveAgentSection inserts a new bounded-section row. The uuid carries the section identity; a duplicate uuid is a no-op (idempotent on resume).

func (*Repository) SaveAnalysisArtifact 

func (r *Repository) SaveAnalysisArtifact(
	ctx context.Context,
	httpRecordUUID, kind, filename, mediaType, sha256 string,
	content []byte,
	metadata string,
) error

SaveAnalysisArtifact is the primitive-argument adapter used by input sources that deliberately avoid importing the database package in their interfaces.

func (*Repository) SaveAnalysisArtifactForRecord 

func (r *Repository) SaveAnalysisArtifactForRecord(ctx context.Context, artifact *AnalysisArtifact) error

SaveAnalysisArtifactForRecord stores a content-addressed immutable artifact beside an existing HTTP record. Duplicate writes of the same record/kind/hash are idempotent.

func (*Repository) SaveAuthenticationHostname 

func (r *Repository) SaveAuthenticationHostname(ctx context.Context, sh *AuthenticationHostname) error

SaveAuthenticationHostname upserts a single session hostname record. Conflict key: (project_uuid, hostname, session_name).

func (*Repository) SaveAuthenticationHostnames 

func (r *Repository) SaveAuthenticationHostnames(ctx context.Context, rows []*AuthenticationHostname) error

SaveAuthenticationHostnames batch-upserts session hostname records in a transaction.

func (*Repository) SaveCandidate 

func (r *Repository) SaveCandidate(ctx context.Context, cand *AgentFindingCandidate) (bool, error)

SaveCandidate inserts a proposed finding candidate, deduping on (agentic_scan_uuid, dedup_hash) via ON CONFLICT DO NOTHING so a looping model that re-proposes the same bug produces exactly one row. Returns inserted=true only when a new row was written (false on a dedup hit), so callers can keep an accurate proposed-count.

func (*Repository) SaveFinding 

func (r *Repository) SaveFinding(ctx context.Context, event *output.ResultEvent, httpRecordUUIDs []string, scanUUID string, projectUUID string) error

SaveFinding stores a vulnerability finding linked to HTTP records by UUIDs. Uses INSERT ON CONFLICT for atomic dedup when finding_hash is non-empty.

func (*Repository) SaveFindingDirect 

func (r *Repository) SaveFindingDirect(ctx context.Context, finding *Finding) error

SaveFindingDirect inserts a pre-built Finding directly (without ResultEvent conversion). Uses INSERT ON CONFLICT for atomic dedup when finding_hash is non-empty.

func (*Repository) SaveFindingReported 

func (r *Repository) SaveFindingReported(ctx context.Context, finding *Finding) (bool, error)

SaveFindingReported is SaveFindingDirect that additionally reports whether a new finding row was inserted (true) vs collapsed into an existing finding by dedup (false). The autopilot report_finding tool uses it so its run counter only tallies distinct findings.

func (*Repository) SaveFindingsBatch 

func (r *Repository) SaveFindingsBatch(ctx context.Context, writes []FindingWrite) error

SaveFindingsBatch persists a batch of findings in a single transaction, coalescing what would otherwise be one transaction (and fsync) per finding. Findings are converted up front so a conversion error skips just that finding rather than aborting the batch. If the transaction itself fails (e.g. a database-level error), every finding is retried individually so one bad finding can't drop the rest — preserving the error isolation of per-finding SaveFinding while keeping the fast path a single transaction.

func (*Repository) SaveFindingsBatchResults added in v0.5.2 

func (r *Repository) SaveFindingsBatchResults(ctx context.Context, writes []FindingWrite) []FindingSaveResult

SaveFindingsBatchResults is SaveFindingsBatch with the per-finding outcomes, aligned 1:1 with writes — results[i] describes writes[i], including the entries this method itself rejects (a nil Event or one that will not convert, each carrying its own Err at its own index).

The alignment is what makes a partial batch countable. SaveFindingsBatch collapses to a single error, so FindingWriter could only ever attribute a failure to the whole batch: one bad finding in a batch of sixty-four was reported as sixty-four dropped findings, and a transaction that failed and then succeeded on retry for all but one was reported the same way. Operators read those counters to decide whether a scan's results are complete.

func (*Repository) SaveFindingsDirectBatch 

func (r *Repository) SaveFindingsDirectBatch(ctx context.Context, findings []*Finding) ([]FindingSaveResult, error)

SaveFindingsDirectBatch persists a batch of pre-built findings in a single transaction — the batch analogue of SaveFindingDirect, used by bulk importers to coalesce what would otherwise be one transaction (and fsync) per finding. The returned slice is aligned with the input; the top-level error is the first per-finding error, if any.

func (*Repository) SaveHostObservations 

func (r *Repository) SaveHostObservations(ctx context.Context, projectUUID, scanUUID string, obs []HostObservationInput) error

SaveHostObservations persists a run's host observations in bounded batches.

ON CONFLICT (scan_uuid, hostname, port) updates rather than inserting: within ONE run an endpoint has one observation, so a second write for it is a refinement (the TLS handshake finishing after the DNS answer, say) and not a second event. ACROSS runs the conflict cannot fire, which is the point - re-probing a host next month adds that month's observation and leaves this one intact.

func (*Repository) SaveOASTInteraction 

func (r *Repository) SaveOASTInteraction(ctx context.Context, interaction *OASTInteraction) error

SaveOASTInteraction stores an OAST interaction record.

func (*Repository) SaveOASTPayloads added in v0.5.3 

func (r *Repository) SaveOASTPayloads(ctx context.Context, payloads []*OASTPayload) error

SaveOASTPayloads stores the payload contexts of planted OAST payloads in one insert. A nonce that is already stored is left as it is: nonces are minted unique, so a conflict can only be the same payload written twice.

func (*Repository) SaveParamMiningRecord added in v0.5.3 

func (r *Repository) SaveParamMiningRecord(ctx context.Context, rr *httpmsg.HttpRequestResponse, projectUUID, scanUUID string, hidden []string, remarks []string) (uuid string, created bool, err error)

SaveParamMiningRecord stores the exchange that confirmed hidden parameters as a record with source param-mining: its EmbeddedParam entries for hidden are marked Hidden, and remarks are attached. scanUUID is stamped on a new row so the scan's own exports count it. created is false when an identical exchange was already stored (its UUID is returned, remarks appended).

func (*Repository) SaveRecord 

func (r *Repository) SaveRecord(ctx context.Context, httpRR *httpmsg.HttpRequestResponse, source string, projectUUID string) (string, error)

SaveRecord stores a denormalized HTTP record (request + response + host + parameters). The source identifies the origin of the record (e.g. "scanner", "ingest-cli", "ingest-server", "ingest-proxy"). Returns the UUID of the saved record. If a matching record already exists (same method, hostname, path, URL, and request body), the existing UUID is returned without inserting.

func (*Repository) SaveRecordBatch 

func (r *Repository) SaveRecordBatch(ctx context.Context, records []*httpmsg.HttpRequestResponse, source string, projectUUID string) ([]string, error)

SaveRecordBatch converts httpmsg.HttpRequestResponse objects to HTTPRecord models and batch-inserts them. This is the high-level batch equivalent of SaveRecord.

func (*Repository) SaveRecordWithLineage added in v0.4.9 

func (r *Repository) SaveRecordWithLineage(ctx context.Context, httpRR *httpmsg.HttpRequestResponse, source string, projectUUID string, lineage RecordLineage) (string, error)

SaveRecordWithLineage is SaveRecord carrying the record's full chain metadata.

On a duplicate hit the existing row is ADOPTED rather than replaced: see AdoptRecord for which columns that repairs and why the row it repairs is so often the one that matters.

func (*Repository) SaveRecordsBatch 

func (r *Repository) SaveRecordsBatch(ctx context.Context, records []*HTTPRecord) ([]string, error)

SaveRecordsBatch inserts multiple HTTP records in a single transaction. Returns the UUIDs of all successfully inserted records.

func (*Repository) SaveRecordsBatchSkipExisting added in v0.5.2 

func (r *Repository) SaveRecordsBatchSkipExisting(ctx context.Context, records []*HTTPRecord) ([]string, error)

SaveRecordsBatchSkipExisting inserts multiple HTTP records in one transaction, skipping any whose uuid is already stored, and returns the uuids it actually inserted. len(records)-len(inserted) is the number skipped as duplicates.

SaveRecordsBatch aborts the whole batch on a duplicate uuid, which is right for a scanner writing records it has just generated and wrong for an import: a JSONL export re-imported, or two overlapping exports imported together, share record uuids by design. That used to fail the run with a UNIQUE constraint error after an arbitrary number of earlier batches had already committed — neither idempotent nor atomic. Here the overlap is simply not inserted.

Only the inserted uuids fire emitRecordSaved: a row that was already there is not a new record, and a mirror that saw it would duplicate on every re-import.

func (*Repository) ScanFindingTotal added in v0.5.3 

func (r *Repository) ScanFindingTotal(ctx context.Context, scanUUID string) (int, map[string]int, error)

ScanFindingTotal returns how many findings a scan persisted, with the same per-severity breakdown ScanSeverityTally gives, read from the findings table rather than the scan row.

It is the ONE count every end-of-scan summary prints. The runner's "Scan finished … findings: N" banner and the CLI's "N issues found" line used to read two different things — the scan row's total_findings (stale until the row is finalized) and a project-wide count over the in-scope hosts — and disagreed on the same run. Both now call this, so they cannot.

Reading the table rather than the row also means a scan whose terminal CompleteScanWithOutcome write failed still reports its true count. The error is the query's: a caller should print no count rather than a wrong zero.

func (*Repository) ScanSeverityTally 

func (r *Repository) ScanSeverityTally(ctx context.Context, scanUUID string) map[string]int

ScanSeverityTally returns a scan's persisted finding counts keyed by severity name, omitting severities with no findings. It reads the same rows the scan row's own counters are computed from, so a consumer reading the machine event stream's scan.finished and a consumer querying the database afterwards agree.

func (*Repository) ScansForTarget added in v0.5.3 

func (r *Repository) ScansForTarget(ctx context.Context, projectUUID, target string, n int) ([]*Scan, error)

ScansForTarget returns the last n finished scans (newest first) whose target list includes target. Matching is on the normalized target (case, trailing slash); a target given without a scheme matches either scheme.

func (*Repository) SetFindingStatus added in v0.5.3 

func (r *Repository) SetFindingStatus(ctx context.Context, projectUUID string, id int64, status, note, actor string) error

SetFindingStatus is UpdateFindingStatusScoped for one finding by id: it returns sql.ErrNoRows when the id matches nothing in scope, and nil when the finding was updated or already had the status.

func (*Repository) SetFindingVerification added in v0.5.3 

func (r *Repository) SetFindingVerification(ctx context.Context, projectUUID string, id int64, v *FindingVerification) error

SetFindingVerification stores a `finding verify` result on one finding: the verification JSON and verified_at (its timestamp, for sorting). Status is not touched; a verdict reaches status only through the scoped status writer (`finding verify --mark`). A finding outside the project is left alone; callers resolve findings in scope first.

func (*Repository) SetOASTPayloadValue added in v0.5.3 

func (r *Repository) SetOASTPayloadValue(ctx context.Context, nonce, payload string) error

SetOASTPayloadValue records the literal value planted for nonce (see oast.Service.RecordPayload) on an already stored payload row.

func (*Repository) SetRecordTechnology 

func (r *Repository) SetRecordTechnology(ctx context.Context, technology map[string][]string) error

SetRecordTechnology batch-writes the detected technology stack onto HTTPRecords identified by UUID, replacing whatever the column held.

Replace rather than merge (the shape AppendRemarks uses) because technology is not accumulated analyst annotation: it is the fingerprint modules' verdict about the host, recomputed from scratch on every scan, and a merge would make a record carry a stack the target stopped running two scans ago.

Records are grouped by their technology list before writing, because the source is host-scoped: every record on one host shares one list, so a corpus of N records over H hosts costs H UPDATEs rather than N. The per-value UUID set is still chunked so a single busy host cannot build an IN() clause past the driver's parameter limit.

func (*Repository) StreamAnalysisArtifactsByKind added in v0.4.9 

func (r *Repository) StreamAnalysisArtifactsByKind(
	ctx context.Context,
	projectUUID, kind string,
	batchSize int,
	fn func(artifact *AnalysisArtifact) error,
) error

StreamAnalysisArtifactsByKind walks stored artifacts of one kind in id order, handing each to fn. It streams rather than returning a slice because recovered source content is unbounded in aggregate — one SPA source map can carry hundreds of files — and the callers scan each body once and discard it.

A zero-length page ends the walk; fn returning an error aborts it.

func (*Repository) UpdateAgentSection 

func (r *Repository) UpdateAgentSection(ctx context.Context, section *AgentSection) error

UpdateAgentSection updates a section row identified by uuid. OmitZero lets the caller pass a partial struct (e.g. only status + closing_summary + ended_at) without clobbering columns set at BeginSection time.

func (*Repository) UpdateAgenticScan 

func (r *Repository) UpdateAgenticScan(ctx context.Context, run *AgenticScan) error

UpdateAgenticScan updates an agent run record by UUID. Uses OmitZero so callers (SwarmRunner.Run and the API enrichAgenticScanRecord helpers) can pass partial structs without zeroing fields they don't intend to touch. To explicitly clear a field, use a column-list update via NewUpdate().

func (*Repository) UpdateAgenticScanStorageURL 

func (r *Repository) UpdateAgenticScanStorageURL(ctx context.Context, agenticScanUUID, storageURL string) error

UpdateAgenticScanStorageURL sets the storage_url field on an agentic scan record.

func (*Repository) UpdateCandidateStatus 

func (r *Repository) UpdateCandidateStatus(ctx context.Context, uuid, status, verdictReason string, promotedFindingID int64) error

UpdateCandidateStatus records a verifier verdict on a candidate by uuid, setting status, verdict_reason, promoted_finding_id (when > 0), and stamping verified_at. Used by the fresh-context verifier to mark confirmed / rejected / needs_evidence.

func (*Repository) UpdateFindingSeverity 

func (r *Repository) UpdateFindingSeverity(ctx context.Context, id int64, severity string) error

UpdateFindingSeverity sets the severity of a single finding by ID. Returns sql.ErrNoRows if no finding matches.

func (*Repository) UpdateFindingStatusByHash 

func (r *Repository) UpdateFindingStatusByHash(ctx context.Context, projectUUID, agenticScanUUID, findingHash, status string) (int64, error)

UpdateFindingStatusByHash sets the lifecycle status of findings matching a finding_hash within a project and an agentic_scan_uuid scope. Used by the swarm triage writeback to promote draft findings to triaged / false_positive based on agent verdicts. Either scope may be empty (dropping that predicate); the swarm path passes both so a hash shared across projects only ever touches the run's own project. Returns the number of rows updated.

func (*Repository) UpdateFindingStatusScoped added in v0.5.3 

func (r *Repository) UpdateFindingStatusScoped(ctx context.Context, projectUUID string, req FindingStatusUpdate) (*FindingStatusResult, error)

UpdateFindingStatusScoped sets the lifecycle status of the findings named by id and/or finding_hash (full or ≥MinFindingHashPrefix prefix), inside one project, in one transaction. Each finding whose status actually changes gets status_note/status_actor/status_updated_at set and one finding_status_events row; a finding already at the status is reported Unchanged and not touched.

The project predicate is part of every lookup, so an id or hash from another project matches nothing and is reported in NotFound — indistinguishable from a missing finding, revealing nothing about other projects. projectUUID == "" drops the predicate; only callers with scoping deliberately off (a standalone `-S --db file` store) pass it.

An ambiguous hash prefix fails the whole call with *AmbiguousFindingHashError before anything is written.

This is the single status writer for the CLI (`finding mark`, `agent triage`), REST PATCH and the olium update_finding tool. The swarm triage writeback, which scopes by agentic scan rather than by finding, keeps UpdateFindingStatusByHash.

func (*Repository) UpdateFindingTriage 

func (r *Repository) UpdateFindingTriage(ctx context.Context, projectUUID string, id int64, severity, description string) error

UpdateFindingTriage sets severity and description on a single finding in one statement. Used by the agent triage flow so a false-positive verdict can't land a half-updated row (severity downgraded but reasoning lost) if the process is killed between two separate UPDATEs. Project-scoped like UpdateFindingStatusScoped: projectUUID == "" drops the predicate, and a finding outside the project returns sql.ErrNoRows.

func (*Repository) UpdateProject 

func (r *Repository) UpdateProject(ctx context.Context, project *Project) error

UpdateProject updates an existing project.

func (*Repository) UpdateRecordAnnotations 

func (r *Repository) UpdateRecordAnnotations(ctx context.Context, uuid string, riskScore *int, remarks []string) error

UpdateRecordAnnotations updates the risk_score and/or remarks of an HTTP record. Only non-nil fields are updated. Returns an error if no record matches the UUID.

func (*Repository) UpdateRecordResponse 

func (r *Repository) UpdateRecordResponse(ctx context.Context, uuid string, update *RecordResponseUpdate) error

UpdateRecordResponse replaces the response fields of an existing HTTP record.

func (*Repository) UpdateRiskScores 

func (r *Repository) UpdateRiskScores(ctx context.Context, scores map[string]int) error

UpdateRiskScores batch-updates risk_score on HTTPRecords identified by UUID. Uses CASE/WHEN SQL to update up to 500 records per statement, minimizing roundtrips.

func (*Repository) UpdateScan 

func (r *Repository) UpdateScan(ctx context.Context, scan *Scan) error

UpdateScan updates an existing scan record

func (*Repository) UpdateScanHeartbeat added in v0.5.3 

func (r *Repository) UpdateScanHeartbeat(ctx context.Context, scanUUID string, hb ScanHeartbeat) (updated bool, err error)

UpdateScanHeartbeat stamps a running scan's row with its latest beat: one small UPDATE, by uuid, touching only the liveness columns (and total_requests). updated_at is deliberately left alone — it is the row's edit time, and a heartbeat every few seconds would make it meaningless as one; heartbeat_at is the liveness column.

Only a row still "running" is touched: a late beat must never re-stamp a finalized row. updated reports whether a row was written, which is how a heartbeat learns its scan was finalized by someone else.

func (*Repository) UpdateScanPartial 

func (r *Repository) UpdateScanPartial(ctx context.Context, scan *Scan) error

UpdateScanPartial updates a scan row by UUID, skipping fields whose Go value is the zero value. Use this when the caller only wants to touch a subset of fields (e.g. an API PATCH that should leave omitted fields unchanged). To explicitly clear a field, use a column-level Set via NewUpdate().

func (*Repository) UpdateScanStorageURL 

func (r *Repository) UpdateScanStorageURL(ctx context.Context, scanUUID, storageURL string) error

UpdateScanStorageURL sets the storage_url field on a scan record.

func (*Repository) UpdateSurfaceScores 

func (r *Repository) UpdateSurfaceScores(ctx context.Context, scores map[string]int) error

UpdateSurfaceScores batch-updates surface_score on HTTPRecords identified by UUID. Written by the surface_scoring passive module; see HTTPRecord.SurfaceScore for why this is a separate column from risk_score rather than a second writer of the same one.

func (*Repository) UpsertSnapshotRecord 

func (r *Repository) UpsertSnapshotRecord(ctx context.Context, httpRR *httpmsg.HttpRequestResponse, source string, projectUUID string) (uuid, outcome string, err error)

UpsertSnapshotRecord stores a Burp snapshot record idempotently. When an existing request later gains a response, or its response changes, the response-derived columns are refreshed in place while the stable UUID and finding links are preserved. outcome is inserted, updated, or unchanged.

func (*Repository) UpsertUser 

func (r *Repository) UpsertUser(ctx context.Context, user *User) error

UpsertUser inserts a new user or updates name/email if the UUID already exists. Returns the user's UUID.

func (*Repository) WalkJavaScriptRecords 

func (r *Repository) WalkJavaScriptRecords(ctx context.Context, projectUUID, hostname string, limit int, fn func(recordURL, contentType string, body []byte) error) error

WalkJavaScriptRecords streams JavaScript responses already stored for a host by earlier phases (spidering, proxy/Burp ingestion) so the discovery engine can feed browser-collected bundles through JSTangle even though they live in the main DB rather than the ephemeral discovery sitemap. Each callback receives the record URL, its response content-type, and the decoded (gzip-inflated) response body. JavaScript is matched by content-type or a .js/.mjs URL suffix; the walk is bounded by limit and ordered by uuid for determinism. A callback error stops the walk and is returned. This satisfies the source.spideredJSProvider optional interface structurally (primitive-typed) so neither package imports the other.

type RetryPolicy added in v0.5.3 

type RetryPolicy struct {
	// Op names the operation in debug logs and in the final error.
	Op string
	// MaxAttempts is the total number of calls to fn, including the first.
	MaxAttempts int
	// BaseDelay is the wait after the first failure; it doubles per attempt up
	// to MaxDelay.
	BaseDelay time.Duration
	MaxDelay  time.Duration
	// Retryable decides whether an error is worth another attempt. Nil means
	// IsBusy: only SQLite busy/locked errors are retried, anything else returns
	// at once.
	Retryable func(error) bool
	// Wait sleeps between attempts. Nil means a context-aware timer. A caller
	// that holds a mutex across fn supplies one that releases it while waiting.
	Wait func(ctx context.Context, d time.Duration) error
}

RetryPolicy bounds RetryBusy. The zero value is usable: every unset field falls back to DefaultRetryPolicy's.

type RiskPrioritizedDBInputSource 

type RiskPrioritizedDBInputSource struct {
	// contains filtered or unexported fields
}

RiskPrioritizedDBInputSource processes high-risk records first, then falls back to normal cursor-based order. It implements source.InputSource.

Delivery is AT-LEAST-ONCE. The durable cursor advances only once every item in the round has been acknowledged or written off, so a round cut short re-serves its tail on the next pass rather than skipping it — duplicate work being far cheaper than a silent coverage hole. Two consequences follow from that choice and are accepted rather than fixed here:

  • The checkpoint is `processed_count = processed_count + ?`, which is not idempotent. A write that commits but reports an error (a connection lost after the server applied it) is retried by FlushCursor and can double-count processed_count. The cursor position itself is an absolute (created_at, uuid) assignment and stays correct; only the counter can overshoot, and a wrong progress number is a far smaller problem than a cursor past unprocessed records.
  • A record served twice is scanned twice. Findings dedup downstream, so the visible cost is time rather than duplicate results.

func NewRiskPrioritizedDBInputSource 

func NewRiskPrioritizedDBInputSource(db *DB, repo *Repository, scanUUID string) *RiskPrioritizedDBInputSource

NewRiskPrioritizedDBInputSource creates a DBInputSource that processes records with risk_score > 0 first (highest risk first), then continues with the normal cursor-based order for remaining records.

func (*RiskPrioritizedDBInputSource) Close 

func (s *RiskPrioritizedDBInputSource) Close() error

Close stops the source.

func (*RiskPrioritizedDBInputSource) CoalescedDropped 

func (s *RiskPrioritizedDBInputSource) CoalescedDropped() int

CoalescedDropped returns how many records the param-shape coalescing pass removed from this scan's iteration list. Valid after the snapshot loads (i.e. after the first Next / after Execute). Zero when coalescing is disabled.

func (*RiskPrioritizedDBInputSource) Count added in v0.5.3 

func (s *RiskPrioritizedDBInputSource) Count() int64

Count implements source.Countable: the number of records in this round's snapshot window, or 0 before the first Next (and for an empty snapshot). An upper bound on what the round serves, since param-shape coalescing removes records from it as pages are read.

func (*RiskPrioritizedDBInputSource) FlushCursor added in v0.5.2 

func (s *RiskPrioritizedDBInputSource) FlushCursor(ctx context.Context) error

FlushCursor makes one last, reported attempt to checkpoint the round's cursor.

The steady path commits from the ack that resolves the final item, on a background context, and logs a failure at warn level. That is the right place for it — the cursor must advance whether or not anyone is watching — but it leaves a phase with no way to learn that the checkpoint never landed. This is that call: the phase runs it after its executor returns and can turn a failure into a reported partial run instead of a clean one whose cursor lies.

A no-op once committed, and a no-op while items are still unresolved (the commit has not become due yet, so there is nothing to flush). Returns the last error when the retries are exhausted.

func (*RiskPrioritizedDBInputSource) Next 

func (s *RiskPrioritizedDBInputSource) Next(ctx context.Context) (*work.WorkItem, error)

Next streams records high-risk-first then chronologically, via bounded keyset pages, so memory scales with the page size (not the whole eligible table).

func (*RiskPrioritizedDBInputSource) Unresolved added in v0.5.2 

func (s *RiskPrioritizedDBInputSource) Unresolved() int

Unresolved reports how many of this round's items ended neither acknowledged nor skipped — items a worker took and never finished, because the phase was cancelled or ran out of budget. Valid after the round's executor has returned.

func (*RiskPrioritizedDBInputSource) WithHostScopes 

func (s *RiskPrioritizedDBInputSource) WithHostScopes(hosts []HostTarget) *RiskPrioritizedDBInputSource

WithHostScopes sets an in-scope origin filter so only records matching these (scheme, hostname, port) origins are returned.

func (*RiskPrioritizedDBInputSource) WithParamShapeCoalescing 

func (s *RiskPrioritizedDBInputSource) WithParamShapeCoalescing(maxSamples int) *RiskPrioritizedDBInputSource

WithParamShapeCoalescing enables param-shape coalescing of the snapshot: records that share a param shape (host + path shape + method + query/form/JSON param-name-set; see paramShapeRepresentative) are reduced to at most maxSamples value-distinct representatives, cutting redundant dynamic-assessment fan-out over value-only-different requests (e.g. /search?q=1..N, /users/1..N). Records stay in the database; only this scan's iteration list is pruned. The scan cursor still advances past every record, so coalesced-away records are not re-scanned in a later feedback round. maxSamples <= 0 (the default) disables it.

type Scan 

type Scan struct {
	bun.BaseModel `bun:"table:scans,alias:sc" json:"-"`

	UUID        string `bun:"uuid,pk,notnull" json:"uuid"`
	ProjectUUID string `bun:"project_uuid,notnull" json:"project_uuid"`
	Name        string `bun:"name,nullzero" json:"name"`
	Description string `bun:"description,nullzero" json:"description"`
	Status      string `bun:"status,notnull,default:'running'" json:"status"` // running, paused, completed, failed, cancelled
	// Target can be a URL, domain, IP, CIDR, or file path (for imported requests)
	Target  string `bun:"target,nullzero" json:"target"`
	Modules string `bun:"modules,nullzero" json:"modules"`
	Threads int    `bun:"threads,default:0" json:"threads"`

	// Scan context
	// ScopeOriginMode is the RESOLVED origin-scope mode the scan ran under (all,
	// relaxed, balanced, strict) — never the raw --scope-origin value, which is
	// empty on the common run that takes the default. Persisted because it is the
	// setting that decides which HOSTS the scan was allowed to touch: on a
	// database carrying traffic from earlier scans, "balanced" admits every
	// sibling subdomain of the target's eTLD+1, so a result set cannot be read
	// correctly without it. Written by the runner (the single owner of the
	// fallback chain) for CLI, server and scan-on-receive runs alike, and carried
	// out through the `scan` envelope of --format jsonl and the REST scan object.
	ScopeOriginMode string   `bun:"scope_origin_mode,nullzero" json:"scope_origin_mode"`
	Profile         string   `bun:"profile,nullzero" json:"profile"`                     // scanning profile used (light, full, api, etc.)
	SourcePath      string   `bun:"source_path,nullzero" json:"source_path"`             // source code path for audit/agent scans
	SourceType      string   `bun:"source_type,nullzero" json:"source_type"`             // local, git-url, gcs
	Tags            []string `bun:"tags,type:jsonb,nullzero" json:"tags"`                // arbitrary tags for filtering/grouping
	TriggeredBy     string   `bun:"triggered_by,nullzero" json:"triggered_by"`           // user, schedule, webhook, agent
	AgenticScanUUID string   `bun:"agentic_scan_uuid,nullzero" json:"agentic_scan_uuid"` // link to agentic_scan that spawned this scan

	// Soft reference to the HTTP record that triggered this scan (scan-on-receive)
	HTTPRecordUUID string `bun:"http_record_uuid,nullzero" json:"http_record_uuid"`

	// Cursor-based scan tracking
	ScanSource      string    `bun:"scan_source,nullzero" json:"scan_source"`         // "cli", "api", "scan-on-receive"
	ScanMode        string    `bun:"scan_mode,nullzero" json:"scan_mode"`             // "incremental" or "full"
	StartCursorAt   time.Time `bun:"start_cursor_at,nullzero" json:"start_cursor_at"` // cursor position at scan start
	StartCursorUUID string    `bun:"start_cursor_uuid,nullzero" json:"start_cursor_uuid"`
	CursorAt        time.Time `bun:"cursor_at,nullzero" json:"cursor_at"` // current cursor position
	CursorUUID      string    `bun:"cursor_uuid,nullzero" json:"cursor_uuid"`
	ProcessedCount  int64     `bun:"processed_count,default:0" json:"processed_count"`

	StartedAt  time.Time `bun:"started_at,notnull,default:current_timestamp" json:"started_at"`
	FinishedAt time.Time `bun:"finished_at,nullzero" json:"finished_at"`
	DurationMs int64     `bun:"duration_ms,default:0" json:"duration_ms"`
	// Summary fields (populated at end of scan)
	TotalRequests int64 `bun:"total_requests,default:0" json:"total_requests"`
	TotalFindings int64 `bun:"total_findings,default:0" json:"total_findings"`

	// Risk level counts (populated at end of scan)
	CriticalCount int64 `bun:"critical_count,default:0" json:"critical_count"`
	HighCount     int64 `bun:"high_count,default:0" json:"high_count"`
	MediumCount   int64 `bun:"medium_count,default:0" json:"medium_count"`
	LowCount      int64 `bun:"low_count,default:0" json:"low_count"`
	InfoCount     int64 `bun:"info_count,default:0" json:"info_count"`
	SuspectCount  int64 `bun:"suspect_count,default:0" json:"suspect_count"`
	// Error count (populated at end of scan)
	ErrorMessage string `bun:"error_message,nullzero" json:"error_message"`

	StorageURL string `bun:"storage_url,nullzero" json:"storage_url"`

	// Scan completeness — whether the scan covered everything it was given, and
	// if not, why and where. Written by the runner's finalizer, which is the only
	// component that sees every phase's outcome.
	//
	// Status alone could never answer this: "completed" meant "the process
	// reached the end", so a scan whose total budget fired after two of seven
	// phases, or whose dynamic assessment was cut off at its own deadline, was
	// indistinguishable from one that finished the whole plan. That mattered
	// beyond reporting — scan-on-receive treats a completed scan as a valid
	// predecessor and inherits its cursor, so a curtailed run that looked clean
	// made the NEXT run skip the records the first one never processed.
	//
	// An EMPTY Completeness means unknown (an older binary, or an agent/server
	// caller with no outcome data) and must never be read as "complete".
	Completeness  string         `bun:"completeness,nullzero" json:"completeness,omitempty"`
	StopReason    string         `bun:"stop_reason,nullzero" json:"stop_reason,omitempty"`
	PhaseOutcomes []PhaseOutcome `bun:"phase_outcomes,type:jsonb,nullzero" json:"phase_outcomes,omitempty"`

	// Plan is the resolved scan plan (phases, budgets, pacing) as JSON, written
	// once when the run starts. Opaque here: the plan resolver owns its shape.
	Plan json.RawMessage `bun:"plan,type:jsonb,nullzero" json:"plan,omitempty"`

	// Run liveness, refreshed by the runner's heartbeat while the scan runs.
	// A row whose heartbeat stopped advancing while status is still "running"
	// belongs to a process that died; PID and RunnerHost let a reader on the
	// same host confirm that without waiting out the staleness window.
	HeartbeatAt  *time.Time `bun:"heartbeat_at,nullzero" json:"heartbeat_at,omitempty"`
	CurrentPhase string     `bun:"current_phase,nullzero" json:"current_phase,omitempty"`
	PID          int        `bun:"pid,nullzero" json:"pid,omitempty"`
	RunnerHost   string     `bun:"runner_host,nullzero" json:"runner_host,omitempty"`

	// ModuleErrors counts errors per module ID over the run; ExtensionsFailed
	// lists extensions that failed to load. scan.finished reads both back from
	// the row.
	ModuleErrors     map[string]int64 `bun:"module_errors,type:jsonb,nullzero" json:"module_errors,omitempty"`
	ExtensionsFailed []string         `bun:"extensions_failed,type:jsonb,nullzero" json:"extensions_failed,omitempty"`

	CreatedAt time.Time `bun:"created_at,notnull,default:current_timestamp" json:"created_at"`
	UpdatedAt time.Time `bun:"updated_at,notnull,default:current_timestamp" json:"updated_at"`
}

Scan represents a scan session

func NewSingleRequestScan added in v0.5.3 

func NewSingleRequestScan(uuid, projectUUID, name, target, modules, source string) *Scan

NewSingleRequestScan builds the scan row for a single-request scan.

It is the one constructor for that row. The REST handlers used to build it inline, twice, and the CLI direct path created no row at all, so a finding carried a scan_uuid nothing could look up. name is ScanNameScanURL or ScanNameScanRequest; source is the ScanSource label ("cli" or "api"). The row starts "running" with StartedAt now; CompleteScanWithOutcome closes it.

type ScanCompletion added in v0.5.2 

type ScanCompletion struct {
	// Completeness is "complete", "partial", or "" (unknown — written by a caller
	// that has no outcome data, e.g. an older binary or an agent-driven scan).
	Completeness string         `json:"completeness,omitempty"`
	StopReason   string         `json:"stop_reason,omitempty"`
	Phases       []PhaseOutcome `json:"phases,omitempty"`
}

ScanCompletion is the scan-level roll-up of its phases: whether everything was covered, the single most useful explanation when it was not, and the per-phase detail behind that verdict.

type ScanDiff added in v0.5.3 

type ScanDiff struct {
	Base, Head           *Scan
	Comparable           bool
	ComparabilityReasons []string
	History              string
	Entries              []ScanDiffEntry
	// Counts has one key per DiffChanges class, zero included.
	Counts map[string]int
}

ScanDiff is the result of DiffScans.

type ScanDiffEntry added in v0.5.3 

type ScanDiffEntry struct {
	Finding *Finding
	Change  string
	// FirstSeenScan is the scan of the finding's earliest observation.
	FirstSeenScan string
}

ScanDiffEntry is one finding in a diff, as it stands now.

type ScanDiffOptions added in v0.5.3 

type ScanDiffOptions struct {
	// ProjectUUID scopes both scans and every finding; "" = scoping off.
	ProjectUUID string
	// Base is scan a (the earlier run), Head is scan b (the later run).
	Base, Head *Scan
	// IncludeDismissed keeps false_positive/accepted_risk findings.
	IncludeDismissed bool
}

ScanDiffOptions selects the two scans and what to include.

type ScanHeartbeat added in v0.5.3 

type ScanHeartbeat struct {
	At time.Time
	// Phase is the phase running now; "" between phases (and on the final beat)
	// clears current_phase.
	Phase string
	// RequestsSent is the scan's requests so far, written to total_requests.
	// Nil leaves total_requests untouched, for a caller that does not count.
	RequestsSent *int64
	PID          int
	Host         string
}

ScanHeartbeat is one liveness beat of a running scan, written onto its row.

type ScanLog 

type ScanLog struct {
	bun.BaseModel `bun:"table:scan_logs,alias:sl" json:"-"`

	ID          int64     `bun:"id,pk,autoincrement" json:"id"`
	ProjectUUID string    `bun:"project_uuid,notnull" json:"project_uuid"`
	ScanUUID    string    `bun:"scan_uuid,notnull" json:"scan_uuid"`
	Level       string    `bun:"level,notnull" json:"level"`            // trace, info, warn, error
	Phase       string    `bun:"phase,nullzero" json:"phase,omitempty"` // discovery, spidering, dynamic-assessment, etc.
	Message     string    `bun:"message,notnull" json:"message"`
	Metadata    string    `bun:"metadata,nullzero" json:"metadata,omitempty"` // JSON blob for extra context
	CreatedAt   time.Time `bun:"created_at,notnull,default:current_timestamp" json:"created_at"`
}

ScanLog represents a log entry for a scan session.

type ScanLogger 

type ScanLogger struct {
	// contains filtered or unexported fields
}

ScanLogger provides a convenient interface for writing scan log entries. It is safe to use with a nil Repository (all methods become no-ops).

func NewScanLogger 

func NewScanLogger(repo *Repository, scanUUID string) *ScanLogger

NewScanLogger creates a ScanLogger for the given scan. If repo is nil, all logging methods are no-ops so callers don't need nil checks.

func (*ScanLogger) Close 

func (l *ScanLogger) Close()

Close stops the batcher goroutine and flushes any remaining entries.

func (*ScanLogger) Error 

func (l *ScanLogger) Error(phase, message string)

Error logs an error message.

func (*ScanLogger) Flush 

func (l *ScanLogger) Flush()

Flush writes all buffered trace entries to the database.

func (*ScanLogger) Info 

func (l *ScanLogger) Info(phase, message string)

Info logs an informational message.

func (*ScanLogger) InfoWithMeta 

func (l *ScanLogger) InfoWithMeta(phase, message string, meta map[string]interface{})

InfoWithMeta logs an informational message with structured metadata.

func (*ScanLogger) StartBatcher 

func (l *ScanLogger) StartBatcher()

StartBatcher launches a background goroutine that periodically flushes buffered trace entries to the database. Call Close() to stop and flush.

func (*ScanLogger) Trace 

func (l *ScanLogger) Trace(phase, message string)

Trace logs a trace-level message (raw console output). These entries are buffered and flushed in batches to avoid per-line DB inserts.

func (*ScanLogger) Warn 

func (l *ScanLogger) Warn(phase, message string)

Warn logs a warning message.

type ScanSessionStats 

type ScanSessionStats struct {
	ScanUUID       string `json:"scan_uuid"`
	Name           string `json:"name,omitempty"`
	Status         string `json:"status"`
	ProcessedCount int64  `json:"processed_count"`
	ScanMode       string `json:"scan_mode,omitempty"`
}

ScanSessionStats holds statistics for a scan session

type ScanView 

type ScanView struct {
	*Scan
	Target              string `json:"target"`
	Modules             string `json:"modules"`
	TotalActiveModules  int    `json:"total_active_modules"`
	TotalPassiveModules int    `json:"total_passive_modules"`
}

ScanView is a display-oriented wrapper around Scan used by listing endpoints (CLI `db ls --table scans --json` and REST `GET /api/scans`). It overrides the `modules` JSON field so a fully-populated module list renders as "all" instead of a multi-kilobyte CSV, and surfaces active/passive module counts for quick tracking without forcing callers to split the CSV themselves.

type SchemaInfo added in v0.5.3 

type SchemaInfo struct {
	Version          int    `json:"version"`
	MinReaderVersion int    `json:"min_reader_version"`
	WrittenBy        string `json:"written_by,omitempty"`
	UpdatedAt        string `json:"updated_at,omitempty"`
}

SchemaInfo is what schema_meta says about a store.

type SchemaOutdatedError added in v0.5.3 

type SchemaOutdatedError struct {
	DBVersion int
	Missing   []string
}

SchemaOutdatedError reports a store written by an older vigolium that a read-intent open will not migrate. It unwraps to ErrSchemaOutdated.

func (*SchemaOutdatedError) Code added in v0.5.3 

func (e *SchemaOutdatedError) Code() apicontract.ErrorCode

func (*SchemaOutdatedError) Error added in v0.5.3 

func (e *SchemaOutdatedError) Error() string

func (*SchemaOutdatedError) Is added in v0.5.3 

func (e *SchemaOutdatedError) Is(target error) bool

type SchemaTooNewError added in v0.5.3 

type SchemaTooNewError struct {
	DBVersion        int
	MinReaderVersion int
	Binary           int
	WrittenBy        string
	Write            bool
}

SchemaTooNewError reports a store this binary must not touch: either it cannot read it at all (min_reader_version above this binary), or it can read but must not write a schema it does not know (Write).

func (*SchemaTooNewError) Code added in v0.5.3 

func (e *SchemaTooNewError) Code() apicontract.ErrorCode

func (*SchemaTooNewError) Error added in v0.5.3 

func (e *SchemaTooNewError) Error() string

func (*SchemaTooNewError) Is added in v0.5.3 

func (e *SchemaTooNewError) Is(target error) bool

type Scope 

type Scope struct {
	bun.BaseModel `bun:"table:scopes,alias:s" json:"-"`

	ID          int64  `bun:"id,pk,autoincrement" json:"id"`
	ProjectUUID string `bun:"project_uuid,notnull" json:"project_uuid"`
	Name        string `bun:"name,notnull" json:"name"`
	Description string `bun:"description,nullzero" json:"description,omitempty"`

	RuleType string `bun:"rule_type,notnull" json:"rule_type"` // "include" or "exclude"

	HostPattern        string   `bun:"host_pattern,nullzero" json:"host_pattern,omitempty"`
	PathPattern        string   `bun:"path_pattern,nullzero" json:"path_pattern,omitempty"`
	ContentTypePattern string   `bun:"content_type_pattern,nullzero" json:"content_type_pattern,omitempty"` // e.g., "image/*" to exclude
	Methods            []string `bun:"methods,type:jsonb,nullzero" json:"methods,omitempty"`
	Ports              []int    `bun:"ports,type:jsonb,nullzero" json:"ports,omitempty"`
	Schemes            []string `bun:"schemes,type:jsonb,nullzero" json:"schemes,omitempty"`

	Priority      int       `bun:"priority,notnull,default:100" json:"priority"`
	Enabled       bool      `bun:"enabled,notnull,default:true" json:"enabled"`
	HitCount      int64     `bun:"hit_count,default:0" json:"hit_count"`
	LastMatchedAt time.Time `bun:"last_matched_at,nullzero" json:"last_matched_at,omitempty"`

	CreatedAt time.Time `bun:"created_at,notnull,default:current_timestamp" json:"created_at"`
	UpdatedAt time.Time `bun:"updated_at,notnull,default:current_timestamp" json:"updated_at"`
}

Scope defines URL/request scope rules (firewall-style: first match wins)

type ScopeEvaluator 

type ScopeEvaluator struct {
	// contains filtered or unexported fields
}

ScopeEvaluator evaluates URLs against scope rules (firewall-style: first match wins, default=exclude)

func NewScopeEvaluator 

func NewScopeEvaluator(rules []*Scope) *ScopeEvaluator

NewScopeEvaluator creates a new evaluator from pre-loaded scope rules

func (*ScopeEvaluator) IsInScope 

func (e *ScopeEvaluator) IsInScope(scheme, hostname string, port int, method, path string) bool

IsInScope checks if a URL matches the scope rules. Iterates rules by priority. First matching rule wins. If no rule matches, returns false (default=exclude).

type SeverityCount 

type SeverityCount struct {
	Severity string `bun:"severity" json:"severity"`
	Count    int64  `bun:"count" json:"count"`
}

SeverityCount holds a severity label and its count.

type StatusCodeBreakdown 

type StatusCodeBreakdown struct {
	Success   int64         `json:"success"`    // 2xx
	Redirect  int64         `json:"redirect"`   // 3xx
	ClientErr int64         `json:"client_err"` // 4xx
	ServerErr int64         `json:"server_err"` // 5xx
	ByCode    map[int]int64 `json:"by_code"`
}

StatusCodeBreakdown holds status code statistics

type StoredHostFacts 

type StoredHostFacts struct {
	// contains filtered or unexported fields
}

StoredHostFacts serves a run's persisted observations, falling back to the process caches.

The fallback order is deliberate. The stored row is the durable answer and survives both cache eviction and the process; the caches can still be ahead of it for an endpoint observed after the observations were written. Preferring the stored row means a large sweep emits the same facts for its first host as for its last, which the caches alone could not do: they hold 8192 entries and evict in insertion order, so on a bigger sweep the hosts resolved EARLIEST had their answers dropped before the export that wanted them.

Resolution is LAZY and memoized: one query per distinct endpoint the caller actually asks about, not a preload of the run's observations. See Repository.LookupHostObservation for why preloading was the wrong shape.

Every method is nil-receiver safe and a nil *StoredHostFacts behaves as "process caches only", so callers never branch on it.

func NewStoredHostFacts 

func NewStoredHostFacts(ctx context.Context, repo *Repository, projectUUID, scanUUID string) *StoredHostFacts

NewStoredHostFacts prepares an endpoint-fact reader for an emit pass. A nil repo, or a database predating the observations table, yields a reader that answers from the process caches alone.

func (*StoredHostFacts) Facts 

func (s *StoredHostFacts) Facts(hostname string, port int) HostFacts

Facts returns what was observed about one endpoint.

func (*StoredHostFacts) FactsMap 

func (s *StoredHostFacts) FactsMap(hostname string, port int) map[string]any

FactsMap is Facts rendered as JSON keys for the map-shaped agent views, memoized alongside the facts so the render cost is per endpoint, not per record.

func (*StoredHostFacts) MissingCount 

func (s *StoredHostFacts) MissingCount() int

MissingCount is how many distinct endpoints this pass could describe from neither the stored observations nor the process caches.

func (*StoredHostFacts) Stored 

func (s *StoredHostFacts) Stored() int

Stored reports how many distinct endpoints this pass could describe.

type TableInfo 

type TableInfo struct {
	Name     string `json:"name"`
	RowCount int64  `json:"row_count"`
}

TableInfo describes a table with its row count.

func ListTablesWithCounts 

func ListTablesWithCounts(ctx context.Context, db *DB) ([]TableInfo, error)

ListTablesWithCounts returns all user tables with their row counts.

type TargetIssue added in v0.5.3 

type TargetIssue struct {
	Target string `json:"target"`
	// Code is the scope rule for a rejected target, or the transport error
	// class for an unreachable one.
	Code   string `json:"code"`
	Detail string `json:"detail,omitempty"`
}

TargetIssue is one target that was refused or did not answer.

type TargetTally added in v0.5.3 

type TargetTally struct {
	Requested int `json:"requested"`
	Admitted  int `json:"admitted"`
	Reached   int `json:"reached"`
	// ReachabilityChecked says every admitted target was asked whether it
	// answers (the runner's preflight GET, or the direct path's executor), so
	// Admitted-Reached is the exact unreachable count. False when the check was
	// skipped (--skip preflight, a probe-only or target-silent plan, an
	// interrupted preflight): Reached is then not a verdict.
	ReachabilityChecked bool `json:"reachability_checked,omitempty"`
	// Rejected are targets scope refused; Code is the scope rule
	// (e.g. "scope.host.include"). Capped at TargetIssueCap.
	Rejected []TargetIssue `json:"rejected,omitempty"`
	// Unreachable are admitted targets that got no HTTP answer; Code is the
	// transport error class (dns, refused, timeout, tls, reset, other). Capped
	// at TargetIssueCap.
	Unreachable []TargetIssue `json:"unreachable,omitempty"`
}

TargetTally is what happened to a scan's CLI targets before the phases ran: how many were asked for, how many scope admitted, how many answered, and the ones that did not make it. It rides on the "preflight" pseudo-phase's PhaseOutcome (scans.phase_outcomes, additive JSON, no migration).

func (*TargetTally) AddRejected added in v0.5.3 

func (t *TargetTally) AddRejected(issue TargetIssue)

AddRejected records a scope-refused target, keeping at most TargetIssueCap.

func (*TargetTally) AddUnreachable added in v0.5.3 

func (t *TargetTally) AddUnreachable(issue TargetIssue)

AddUnreachable records a target that did not answer, keeping at most TargetIssueCap.

func (*TargetTally) Merge added in v0.5.3 

func (t *TargetTally) Merge(b TargetTally)

Merge folds another tally for the same phase into t: counts sum, issue lists concatenate up to the cap. See PhaseOutcome.Merge.

func (TargetTally) UnreachableCount added in v0.5.3 

func (t TargetTally) UnreachableCount() (int, bool)

UnreachableCount is the number of admitted targets that did not answer, and whether that number is known (see ReachabilityChecked). Exact past the Unreachable list's cap.

type UUIDListDBInputSource 

type UUIDListDBInputSource struct {
	// contains filtered or unexported fields
}

UUIDListDBInputSource iterates over a pre-defined list of HTTP record UUIDs, fetching each from the database and converting to WorkItems. It implements source.InputSource.

func NewUUIDListDBInputSource 

func NewUUIDListDBInputSource(repo *Repository, uuids []string) *UUIDListDBInputSource

NewUUIDListDBInputSource creates a new UUIDListDBInputSource for the given UUIDs.

func (*UUIDListDBInputSource) Close 

func (s *UUIDListDBInputSource) Close() error

Close stops the source. After Close, Next will return io.EOF.

func (*UUIDListDBInputSource) Next 

func (s *UUIDListDBInputSource) Next(ctx context.Context) (*work.WorkItem, error)

Next returns the next record from the UUID list as a WorkItem. Skips invalid or missing UUIDs. Returns io.EOF when all UUIDs have been processed.

type User 

type User struct {
	bun.BaseModel `bun:"table:users,alias:u" json:"-"`

	UUID      string    `bun:"uuid,pk,notnull" json:"uuid"`
	Email     string    `bun:"email,nullzero" json:"email,omitempty"`
	Name      string    `bun:"name,nullzero" json:"name,omitempty"`
	CreatedAt time.Time `bun:"created_at,notnull,default:current_timestamp" json:"created_at"`
	UpdatedAt time.Time `bun:"updated_at,notnull,default:current_timestamp" json:"updated_at"`
}

User represents a system user.

type WriteResult 

type WriteResult struct {
	UUID string
	Err  error
}

WriteResult is the outcome of a single record write.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.