database

func (*DB) Close ¶

func (db *DB) Close() error

Close closes database connection

func (*DB) CreateRecordIndexes ¶

func (db *DB) CreateRecordIndexes(ctx context.Context) error

CreateRecordIndexes builds the http_records read indexes. Idempotent (every statement is IF NOT EXISTS), so calling it on a store that never deferred is harmless.

func (*DB) CreateSchema ¶

func (db *DB) CreateSchema(ctx context.Context) error

CreateSchema creates all database tables and indexes if they don't exist.

It is a no-op on a read-only handle. The read commands call it on open to self-heal a fresh project database; against a source opened --read-only that would be a schema write to someone else's evidence, and a source genuinely missing the tables surfaces as a query error rather than being silently migrated into the current shape.

func (*DB) DeferRecordIndexes ¶

func (db *DB) DeferRecordIndexes()

DeferRecordIndexes tells CreateSchema to leave the http_records read indexes uncreated, so a bulk load runs against the primary key alone. The caller MUST call CreateRecordIndexes once loading is done — a store left without them is schema-complete but plans every read as a full scan.

Only for disposable stores that are filled once and then read (the --glob-db scratch merge). A persistent store must not defer: it is written to continuously, so there is no "after the load" to build them in.

func (*DB) Driver ¶

func (db *DB) Driver() string

Driver returns the database driver name

func (*DB) EnsureSchemaCurrent ¶

func (db *DB) EnsureSchemaCurrent(ctx context.Context) error

EnsureSchemaCurrent brings an existing database up to date, doing NOTHING when it already is.

The "do nothing" path is the whole point. CreateSchema issues DDL, and DDL takes SQLite's WRITE lock — so calling it unconditionally on every open makes every read command queue behind any concurrent writer for up to busy_timeout (60s by default). Detecting staleness first is a read-only question, so the common case takes no lock at all.

A read-only handle cannot be migrated (mode=ro refuses DDL); CreateSchema's own read-only branch turns that into ErrSchemaOutdated naming the missing columns, so it is reached through the same call rather than special-cased here.

Indexes are deliberately not part of the check: they are a performance concern, not a correctness one, and the write commands still call CreateSchema directly. Only a missing TABLE or COLUMN can make a query fail outright.

func (*DB) GetStats ¶

func (db *DB) GetStats(ctx context.Context, filters QueryFilters) (*DatabaseStats, error)

GetStats retrieves database statistics

func (*DB) GetTopHosts ¶

func (db *DB) GetTopHosts(ctx context.Context, filters QueryFilters, limit int) ([]HostStats, error)

GetTopHosts retrieves top hosts by request count with per-host finding counts. It uses a constant TWO queries — one for the top hosts, one grouped finding-count query — instead of the previous 1 + N (a per-host count query), so its cost no longer grows with the requested host limit.

func (*DB) HasFTS ¶

func (db *DB) HasFTS() bool

HasFTS reports whether full-text search is available on this handle (FTS5 for SQLite, tsvector for PostgreSQL).

It DISCOVERS the answer rather than remembering whether this process created the index. It used to be a plain field assigned only by SeedDefaults, which made the answer depend on how the handle was opened rather than on what the database contains: `vigolium server` seeds and reported true, every read command (traffic, finding, db ls) does not seed and reported false on the very same file. Two commands then took different query paths over one database.

Discovery is lazy and cached: a handle that never searches pays nothing, and one that does pays a single sqlite_master lookup.

func (*DB) HasVigoliumSchema ¶

func (db *DB) HasVigoliumSchema(ctx context.Context) bool

HasVigoliumSchema reports whether this store has been initialized as a vigolium database, by the presence of the core table every read touches.

It is the question "is this the right file?", which is NOT the same as "does it have rows": a foreign SQLite file and a scanned target both open cleanly, and only this distinguishes them before vigolium's tables get created inside the former.

func (*DB) Optimize ¶

func (db *DB) Optimize(ctx context.Context)

Optimize refreshes the query-planner statistics so subsequent dedup and finding queries plan against real row counts. Call it after a bulk ingest phase (e.g. discovery) and before the heavy dedup passes; it also runs on Close and on the WAL checkpointer cadence. Best-effort — a failure degrades plans, never correctness.

SQLite uses `PRAGMA optimize` (incremental: only re-analyzes tables that changed enough to matter). PostgreSQL analyzes just the hot ingest tables rather than the whole database, so a short-lived CLI run against a shared server doesn't trigger a blocking full-DB ANALYZE on every close.

func (*DB) Path ¶

func (db *DB) Path() string

Path returns the resolved SQLite file backing this handle, or "" for a driver that is not a single file (Postgres).

It exists because DatabaseStats.Database.Path and .Size were declared and never filled: `db stats -j` reported `"path": "", "size": 0` for a store it had open and had just counted 60 records in. A consumer reading that to confirm WHICH database it was looking at — the whole reason the -j envelope carries db_path — was told nothing, in a field that looked like an answer.

func (*DB) SQLDB ¶

func (db *DB) SQLDB() *sql.DB

SQLDB returns the underlying *sql.DB. Used by callers that need a pinned *sql.Conn for connection-scoped operations (e.g. ATTACH DATABASE during a SQLite-to-SQLite merge) that must run on a single physical connection.

func (*DB) SeedDefaults ¶

func (db *DB) SeedDefaults(ctx context.Context) error

SeedDefaults creates the default user and project if they don't exist. This is called during initialization to ensure CLI has a working project context.

func (*DB) ServerVersion ¶

func (db *DB) ServerVersion(ctx context.Context) (string, error)

ServerVersion returns the database server version string (e.g. the Postgres "PostgreSQL 16.1 ..." banner or the SQLite library version).

func (*DBInputSource) Close ¶

func (s *DBInputSource) Close() error

Close stops the source. After Close, Next will return io.EOF.

func (*DBInputSource) IdleFor ¶

func (s *DBInputSource) IdleFor() time.Duration

IdleFor returns how long it has been since the source last observed a new row (or since creation, if none have arrived yet). Useful for status reporting to tell the user "scan is idle — server is still listening for more records."

func (*DBInputSource) Next ¶

func (s *DBInputSource) Next(ctx context.Context) (*work.WorkItem, error)

Next returns the next record after the scan cursor as a WorkItem. It blocks (polling) until a record is available, the context is cancelled, or the source is closed.

func (*DBInputSource) WithHostScopes ¶

func (s *DBInputSource) WithHostScopes(hosts []HostTarget) *DBInputSource

WithHostScopes sets an in-scope origin filter so only records matching these (scheme, hostname, port) origins are returned. This keeps HTTP records from unrelated origins — including the same host on a different port left over from a previous scan (e.g. localhost:8080 when targeting localhost:3000) — out of the scan.

func (*DBInputSource) WithIdleTimeout ¶

func (s *DBInputSource) WithIdleTimeout(timeout time.Duration) *DBInputSource

WithIdleTimeout configures the source to return io.EOF after this long without any new rows arriving from the database. Only honored in polling mode (not oneShot). Zero (the default) keeps the original behavior — poll forever. Typical use: one-shot scan-on-receive where the caller wants the scan to terminate once the ingestion pipeline has settled.

func (*DBInputSource) WithIncludeSources ¶

func (s *DBInputSource) WithIncludeSources(sources []string) *DBInputSource

WithIncludeSources restricts the source to records whose http_records.source value is in the given list. Used by scan-on-receive shallow mode to exclude scanner-produced artefacts (source="finding", source="scanner", etc.) so the scan only processes user-ingested traffic and doesn't fan out across records that were created as byproducts of scanning itself. An empty slice disables the filter (default).

func (*DBInputSource) WithOnActivity ¶

func (s *DBInputSource) WithOnActivity(fn func(recordCount int, idleFor time.Duration, firstBatch bool)) *DBInputSource

WithOnActivity registers a callback invoked once per non-empty fetch batch. firstBatch is true for the very first batch ever returned by this source. idleFor is the duration the source had been quiet *before* this batch landed. Callers typically use this to print a "scan started / resumed" line so the user gets immediate feedback rather than waiting for the next status tick.

func (*DBInputSource) WithPageSize ¶

func (s *DBInputSource) WithPageSize(pageSize int) *DBInputSource

WithPageSize sets the number of records fetched from the database per batch.

func (*DeleteBuilder) DeleteAllTables ¶

func (db *DeleteBuilder) DeleteAllTables(ctx context.Context, dryRun bool) (map[string]int64, error)

DeleteAllTables deletes all data from every data table.

func (*DeleteBuilder) DeleteFindings ¶

func (db *DeleteBuilder) DeleteFindings(ctx context.Context, dryRun bool) (int64, error)

DeleteFindings deletes findings matching the configured filters. If dryRun is true, returns the count without deleting.

func (*DeleteBuilder) DeleteOrphans ¶

func (db *DeleteBuilder) DeleteOrphans(ctx context.Context, dryRun bool) (int64, error)

DeleteOrphans deletes orphaned findings (findings where none of the http_record_uuids exist in http_records).

It honors filters.ProjectUUID. Orphan cleanup reads as store maintenance, but the rows it removes belong to projects: an unscoped sweep run from one engagement deleted another engagement's findings, which is the same cross-project leak the scoped delete paths exist to prevent. With no project filter set the behavior is unchanged and the whole store is swept.

func (*DeleteBuilder) DeleteRecords ¶

func (db *DeleteBuilder) DeleteRecords(ctx context.Context, dryRun bool) (int64, error)

DeleteRecords deletes the HTTP records matching filters, along with the findings whose only evidence they were.

The order below is the policy, and it is not the obvious one. Deleting every finding that merely *touches* a deleted record destroys findings that also cite records the caller is keeping — one module reporting the same issue across five endpoints would lose the whole finding because one endpoint was pruned. So the junction is narrowed first, and a finding is removed only once nothing it links to survives:

1. collect the findings linked to the doomed records (bounded by findings touched, not by records deleted);
2. drop those records' junction rows, so the junction describes surviving evidence only;
3. delete the records;
4. delete the collected findings that now link to nothing.

Step 4 is scoped to the collected candidates rather than sweeping every evidence-less finding, because a finding with no HTTP record is normal — audit and source-scan findings never have one, and this is not the command that removes them (see DeleteOrphans).

Every IN() list is chunked (see SQLChunkSize). The filters can select an unbounded number of records, and bun renders a list as SQL literals rather than bound parameters — so an unchunked delete builds one statement holding every uuid as text, tens of megabytes of it on a large corpus.

func (*DeleteBuilder) DeleteTable ¶

func (db *DeleteBuilder) DeleteTable(ctx context.Context, tableName string, dryRun bool) (int64, error)

DeleteTable deletes all rows from a specific table.

func (*Finding) FromResultEvent ¶

func (f *Finding) FromResultEvent(event *output.ResultEvent) error

FromResultEvent converts output.ResultEvent to Finding

func (*FindingWriter) Close ¶

func (w *FindingWriter) Close()

Close stops accepting new findings, flushes the buffer, and returns. After Close, Save persists synchronously. The drain is bounded by FlushTimeout so a wedged database can't hang Close forever.

func (*FindingWriter) Metrics ¶

func (w *FindingWriter) Metrics() FindingWriterMetrics

Metrics returns a snapshot of the writer's counters.

func (*FindingWriter) Save ¶

func (w *FindingWriter) Save(ctx context.Context, event *output.ResultEvent, recordUUIDs []string, scanUUID, projectUUID string) error

Save enqueues a finding for batched persistence. It does not block on the database: if the buffer is full or the writer is shutting down, the finding is persisted synchronously so nothing is dropped. Safe for concurrent callers.

func (*FindingsQueryBuilder) Count ¶

func (fqb *FindingsQueryBuilder) Count(ctx context.Context) (int64, error)

Count returns total number of matching findings

func (*FindingsQueryBuilder) Execute ¶

func (fqb *FindingsQueryBuilder) Execute(ctx context.Context) ([]*Finding, error)

Execute executes the query and returns findings

func (*FindingsQueryBuilder) ExecuteWithCount ¶

func (fqb *FindingsQueryBuilder) ExecuteWithCount(ctx context.Context) ([]*Finding, int64, error)

ExecuteWithCount runs the filtered query and returns matching findings alongside the total unfiltered count, in a single round-trip via Bun's ScanAndCount. Use this when callers need both the page and the total (paginated views) — saves one DB roundtrip vs. Execute + Count.

func (*HTTPRecord) EnrichFromHttpRequestResponse ¶

func (r *HTTPRecord) EnrichFromHttpRequestResponse(ctx *httpmsg.HttpRequestResponse) error

EnrichFromHttpRequestResponse fills everything PrepareIdentity left out: the owned raw bytes, all response-derived columns, request authorization, parameters, resolved IP, and timestamps. Run it only for a record that is actually going to be stored. Safe to call on a record whose identity half has already run; it does not recompute it.

func (*HTTPRecord) FromHttpRequestResponse ¶

func (r *HTTPRecord) FromHttpRequestResponse(ctx *httpmsg.HttpRequestResponse) error

FromHttpRequestResponse populates an HTTPRecord from httpmsg.HttpRequestResponse.

The record OWNS its RawRequest/RawResponse: both are copied out of the source message rather than aliased. This is a lifetime requirement, not a style choice. A converted record routinely outlives the call that produced it — RecordWriter.admit hands it to a shard channel and returns, the flush goroutine inserts it later, and Repository.emitRecordSaved passes it to the filesystem mirror, which writes it from yet another goroutine. Meanwhile the executor builds its responses over a POOLED buffer (see pkg/core, getResponseBuffer) that processItem returns to the pool as soon as it is done with the item. Aliasing therefore let a record be persisted or mirrored with another response's bytes whenever the write outlived the caller — on scan cancellation the caller stops waiting while the record is still queued, which is exactly when this happened.

The copy (bytes.Clone below) costs one memcpy per converted record, which is small next to the two SHA-256 passes below, and is only paid for records that get enriched at all.

Conversion is split in two halves, and a caller that deduplicates should use them separately rather than calling this: PrepareIdentity fills only the columns that decide WHICH row this is, and EnrichFromHttpRequestResponse does the expensive response-derived work. See PrepareIdentity.

func (*HTTPRecord) MarshalJSON ¶

func (r *HTTPRecord) MarshalJSON() ([]byte, error)

MarshalJSON preserves the external JSON contract by injecting four derived fields (request_headers, request_body, response_headers, response_body) alongside the stored struct fields.

func (*HTTPRecord) ParsedRequest ¶

func (r *HTTPRecord) ParsedRequest() *httpmsg.HttpRequest

ParsedRequest parses RawRequest into an httpmsg.HttpRequest. Returns nil when there is no raw request. Each call constructs a new parser; callers that need multiple fields should hoist the result into a local, or use parsedView for the standard four-field expansion.

func (*HTTPRecord) ParsedResponse ¶

func (r *HTTPRecord) ParsedResponse() *httpmsg.HttpResponse

ParsedResponse parses RawResponse into an httpmsg.HttpResponse, or nil when no raw response is present.

func (*HTTPRecord) ParsedView ¶

func (r *HTTPRecord) ParsedView() (reqHdrs, respHdrs map[string][]string, reqBody, respBody []byte)

ParsedView decodes RawRequest and RawResponse exactly once each and returns the four derived fields. Use this whenever a caller needs more than one derived field — calling individual accessors for each field re-parses.

When raw bytes are absent (e.g., list endpoints exclude raw_* columns), returned values are nil — that is what makes the JSON projection in HandleListRecords drop the four derived fields.

func (*HTTPRecord) PrepareIdentity ¶

func (r *HTTPRecord) PrepareIdentity(ctx *httpmsg.HttpRequestResponse) error

PrepareIdentity fills the columns that establish a record's identity — the ones recordDedupKey builds its key from and findDuplicateRecordUUIDs matches on — and nothing else. Cheap: a URL parse, a few header reads, and one SHA-256 over the raw request.

It exists so a duplicate can be recognized WITHOUT paying for enrichment. EnrichFromHttpRequestResponse runs two more passes over the response body (a normalized-body-hash regex, a word count), tokenizes HTML to pull out a title, and copies the raw bytes. Profiling a 64 KiB HTML response put ~57% of conversion CPU in the normalizing regex alone, and the whole conversion at ~2.6ms and ~786 KiB allocated. On duplicate-heavy ingestion — discovery and spidering re-encountering the same URLs, which is the case the dedup cache was added for — all of that was spent on records thrown away microseconds later.

Deliberately NOT here: hostname resolution. The IP is stored metadata, not identity, and scheduling it from this half would spawn a background DNS goroutine for every duplicate too.

Callers must set Source (and ProjectUUID) before building a dedup key from the result: recordDedupKey consults both.

func (*HTTPRecord) RedirectLocation ¶

func (r *HTTPRecord) RedirectLocation() string

RedirectLocation returns where this record's response points, or "" when it is not a redirect and when the destination is not recoverable.

The stored column answers first. Parsing raw_response is the fallback for records written before response_location existed, whose column is empty whether or not the response carried a Location — the two are indistinguishable, so a legacy redirect with no Location costs one parse of bytes already in hand.

It lives here, beside the write path that fills the column, so the question "where does this redirect go" has ONE answer. The read path used to carry its own copy of the parse, which is exactly the drift this file exists to prevent.

func (*HTTPRecord) RequestBodyBytes ¶

func (r *HTTPRecord) RequestBodyBytes() []byte

RequestBodyBytes returns the request body bytes parsed from RawRequest.

func (*HTTPRecord) RequestHeadersMap ¶

func (r *HTTPRecord) RequestHeadersMap() map[string][]string

RequestHeadersMap returns request headers as a map[name][]values.

func (*HTTPRecord) ResponseBodyBytes ¶

func (r *HTTPRecord) ResponseBodyBytes() []byte

ResponseBodyBytes returns the response body bytes parsed from RawResponse.

func (*HTTPRecord) ResponseHeadersMap ¶

func (r *HTTPRecord) ResponseHeadersMap() map[string][]string

ResponseHeadersMap returns response headers as a map[name][]values.

func (HostFacts) AsMap ¶

func (h HostFacts) AsMap() map[string]any

AsMap renders the facts as JSON keys, for the map-shaped agent views. Keys match httpx's names, and match the struct tags above — one definition of the key set, so the two output shapes cannot drift.

func (HostFacts) Empty ¶

func (h HostFacts) Empty() bool

Empty reports whether this process observed nothing for the host. Empty facts are omitted entirely rather than emitted as empty arrays: "not probed" and "no records" are different answers.

func (*HostObservation) Facts ¶

func (o *HostObservation) Facts() HostFacts

Facts renders a stored observation back into the output shape. The columns are bun jsonb fields, so this is a field copy rather than a decode.

func (HostObservationInput) Empty ¶

func (o HostObservationInput) Empty() bool

Empty reports whether the observation carries nothing worth storing. A row with no answer AND no completeness claim states nothing that the absence of a row does not already state.

func (*MergeStats) Add ¶

func (s *MergeStats) Add(o *MergeStats)

Add accumulates o's counters into s, for aggregating the stats of several merges (e.g. a multi-source import) into a single total.

func (*QueryBuilder) BuildRecordsQuery ¶

func (qb *QueryBuilder) BuildRecordsQuery() *bun.SelectQuery

BuildRecordsQuery builds a query for http_records table

func (*QueryBuilder) Count ¶

func (qb *QueryBuilder) Count(ctx context.Context) (int64, error)

Count returns total number of matching records

func (*QueryBuilder) Execute ¶

func (qb *QueryBuilder) Execute(ctx context.Context) ([]*HTTPRecord, error)

Execute executes the query and returns results

func (*QueryBuilder) ExecuteWithCount ¶

func (qb *QueryBuilder) ExecuteWithCount(ctx context.Context) ([]*HTTPRecord, int64, error)

ExecuteWithCount runs the filtered query and returns the matching page of records alongside the total count for the same filters.

It is NOT one round trip, despite what this comment used to claim. Bun's ScanAndCount collapses to a single statement only when there is no limit and no offset; a paginated call — the only reason to use this method — takes the two-query path, and with no explicit conn it runs the scan and the count CONCURRENTLY, so it occupies two pool connections for its duration. On SQLite that is worth remembering when this runs alongside a busy RecordWriter.

Still preferable to Execute + Count, which serializes the same two queries. Callers that do not need an exact total on every refresh should use Execute alone rather than paying the count.

func (*QueryBuilder) GroupRecordsBy ¶

func (qb *QueryBuilder) GroupRecordsBy(ctx context.Context, field string, limit int) (RecordGrouping, error)

GroupRecordsBy counts the filtered records by one column, largest bucket first, capped at limit (<= 0 means no cap).

The totals come from a second query ONLY when the cap actually bit. Summing the returned buckets describes the whole result exactly when every bucket came back — each filtered row lands in exactly one bucket, since COALESCE removes the NULL/empty split — and describes only the visible part otherwise, which is how a bounded view comes to report a complete-looking number that is wrong. So the expensive case is paid for and the common one is not: the filters here are the listing's filters, which include `LIKE '%…%'` over the raw request/response corpus, and running those twice reads every matched record's blobs twice to produce a dozen integers.

func (*QueryBuilder) OmitBodies ¶

func (qb *QueryBuilder) OmitBodies() *QueryBuilder

OmitBodies drops raw_request/raw_response from the SELECT list, for callers that read only metadata off the returned records. Those two columns hold the whole request/response corpus and dwarf every other column, so hydrating them for a list view costs memory proportional to the traffic scanned rather than to the page size.

Filters are unaffected: they run in SQL against the stored columns, so this is safe even with --search/--header/--body active (contrast MergeOptions. SkipRecordBodies, which removes the columns from the destination and so cannot be used with those filters). The only requirement is that the caller not read HTTPRecord.RawRequest/RawResponse — they come back empty.

This mirrors FindingsQueryBuilder.ExecuteWithCount, which always excludes its equivalent blob columns; records can't default to that because --raw/--burp and the record views legitimately need them.

func (QueryFilters) EffectiveExcludeTerms ¶

func (f QueryFilters) EffectiveExcludeTerms() []string

EffectiveExcludeTerms returns the exclusion search terms with blanks dropped, so callers can loop the result directly. Each term becomes an independent NOT (...) conjunct — a row is dropped if ANY term matches.

func (QueryFilters) EffectiveSearchTerms ¶

func (f QueryFilters) EffectiveSearchTerms() []string

EffectiveSearchTerms returns the search terms to AND-combine: the repeatable SearchTerms when set, otherwise the single SearchTerm. Blank terms are dropped, so callers can loop the result directly.

func (QueryFilters) UsesLinkedFindings ¶

func (f QueryFilters) UsesLinkedFindings() bool

UsesLinkedFindings reports whether any active filter reaches into findings when the query is over HTTP RECORDS. It is the mirror of UsesLinkedRecords: records have no finding columns of their own, so --severity resolves through a join over the finding_records junction (see applyFilters).

A caller that would otherwise omit the finding tables entirely (globDBSkipSet.Findings, MergeOptions.SkipFindings) MUST check this first. The failure is silent: with no findings to join, a --severity record listing returns nothing rather than erroring.

Keep in sync with applyFilters.

func (QueryFilters) UsesLinkedRecords ¶

func (f QueryFilters) UsesLinkedRecords() bool

UsesLinkedRecords reports whether any active filter reaches into http_records when the query is over FINDINGS. It is the findings-side counterpart to UsesRawCorpus and a strictly wider question: a findings query has no record columns of its own, so host/path/method/status/source all resolve through an EXISTS over the finding_records junction (see applyFindingFilters), and every raw-corpus filter does too.

The question does not arise for traffic, whose query is over http_records directly — there the rows are always present by construction.

A caller that would otherwise omit the table entirely (globDBSkipSet.Records, MergeOptions.SkipHTTPRecords) MUST check this first. The failure is silent: with no rows to satisfy them, every EXISTS predicate is false, so --host returns nothing and the NOT EXISTS exclude forms return everything.

Keep in sync with applyFindingFilters.

func (QueryFilters) UsesRawCorpus ¶

func (f QueryFilters) UsesRawCorpus() bool

UsesRawCorpus reports whether any active filter matches against the raw_request/raw_response blob columns — the FuzzyTerm branches, recordSearchPredicate (--search) and the region predicates behind --header/--body (see message_regions.go), plus all three exclude forms.

A caller that would otherwise avoid materializing those blobs (a projected SELECT, or a merge that omits the columns) MUST check this first and keep them when it returns true. The failure is silent, not an error: with the blobs absent every LIKE matches nothing, so --search returns an empty result and the --exclude-* forms — which are negated — return everything.

Keep in sync with applyFilters; this is the reason to add a field here rather than open-code the flag checks at each call site.

func (RecordWithHostFacts) MarshalJSON ¶

func (rw RecordWithHostFacts) MarshalJSON() ([]byte, error)

MarshalJSON merges the fact keys into the record's own object rather than nesting them under a wrapper, so the emitted shape is the record a consumer already parses plus the extra keys.

func (*RecordWriter) Close ¶

func (w *RecordWriter) Close()

Close stops accepting new writes, flushes remaining records, and returns.

Ordering matters: it marks the writer closed under the admission write lock (so no new write can admit past this point), waits for every already-admitted enqueue to finish landing in a shard channel, and only THEN cancels the flush context. That guarantees the shutdown drain observes every admitted record — none is orphaned in a channel after the flush loop exits, and every admitted write receives its real result instead of a spurious "closed" error.

func (*RecordWriter) Metrics ¶

func (w *RecordWriter) Metrics() RecordWriterMetrics

Metrics returns a snapshot of the writer's counters.

func (*RecordWriter) SaveRecord ¶

func (w *RecordWriter) SaveRecord(ctx context.Context, rr *httpmsg.HttpRequestResponse, source string, projectUUID string) (string, error)

SaveRecord implements the network.RecordSaver interface by delegating to Write.

func (*RecordWriter) SaveRecordBatch ¶

func (w *RecordWriter) SaveRecordBatch(ctx context.Context, records []*httpmsg.HttpRequestResponse, source string, projectUUID string) ([]string, error)

SaveRecordBatch enqueues records to their shards BEFORE awaiting any result, so a single bulk producer's records land in the same writer batch instead of paying one flush interval per record (the trap of a Write-in-a-loop: Write blocks until its record is flushed, so a lone producer never fills a batch). Results are returned positionally aligned with the input — uuids[i] corresponds to records[i] — including deduplicated, invalid, and failed entries, so callers can't misassociate a UUID. The first error encountered is returned alongside the (still complete) uuid slice.

Submissions are admitted in BOUNDED slices rather than all at once. Callers hand this whole collections: discovery submits an entire provenance group in one call, which on a large corpus is tens of thousands of records. Admitting them together made every one of their converted HTTPRecords — each owning a full raw request and response — plus a result channel each live simultaneously, so peak memory scaled with the corpus rather than with anything configured. Slices stay large enough to fill many writer batches, so the coalescing this method exists for is unaffected.

func (*RecordWriter) Write ¶

func (w *RecordWriter) Write(ctx context.Context, rr *httpmsg.HttpRequestResponse, source string, projectUUID string) (string, error)

Write enqueues a record for batched insertion. It blocks until the record is persisted (or the context is cancelled). This is safe to call from multiple goroutines concurrently.

func (*RecordWriter) WriteWithParent ¶

func (w *RecordWriter) WriteWithParent(ctx context.Context, rr *httpmsg.HttpRequestResponse, source string, projectUUID string, parentUUID string) (string, error)

WriteWithParent is Write with an explicit parent_uuid link, used to chain the hops of a followed redirect into one walkable sequence. See admit for why the link is applied to new rows only.

func (*Repository) AdoptRecordParent ¶

func (r *Repository) AdoptRecordParent(ctx context.Context, uuid, parentUUID string)

AdoptRecordParent links an existing record to a parent, but ONLY when it has none. A no-op for an empty uuid or parent.

Backfill-never-overwrite is the whole contract. Filling an empty parent adds information and cannot contradict anything. Overwriting a set one would let the most recent scan to traverse a shared redirect target silently re-parent it out of the chain it was first recorded in, rewriting history for every earlier chain that ran through the same URL.

Best-effort: a failure to link leaves a shorter chain, which reads correctly as less lineage rather than as wrong lineage, and must not fail the save that prompted it.

func (*Repository) AdvanceScanCursor ¶

func (r *Repository) AdvanceScanCursor(ctx context.Context, scanUUID string, recordCreatedAt time.Time, recordUUID string) error

AdvanceScanCursor updates the cursor position and increments ProcessedCount.

func (*Repository) AdvanceScanCursorBy ¶

func (r *Repository) AdvanceScanCursorBy(ctx context.Context, scanUUID string, recordCreatedAt time.Time, recordUUID string, delta int64) error

AdvanceScanCursorBy updates the cursor position and increments ProcessedCount by delta.

func (*Repository) AgenticScanStatusByUUID ¶

func (r *Repository) AgenticScanStatusByUUID(ctx context.Context, uuid string) (string, error)

AgenticScanStatusByUUID returns just the status column for a run.

It exists for the SSE log follower, which asks "is this run finished yet?" twice a second for as long as a client watches: doing that through GetAgenticScan reads the run's full row — blobs included — so N followers on a finished-but-still-watched autopilot run pull 2N megabyte-scale reads per second out of the database to learn one short string.

func (*Repository) AppendRemarks ¶

func (r *Repository) AppendRemarks(ctx context.Context, annotations map[string][]string) error

AppendRemarks batch-appends remarks to HTTPRecords identified by UUID. Existing remarks are preserved and duplicates within each record are deduplicated.

The read and the merged write happen in ONE transaction. Remarks are a growing set written by several analysis passes against the same records, and reading the current value outside the transaction turns every append into a read- modify-write race: two passes both read ["a"], both write their own merge, and whichever commits second silently drops the other's remark. The symptom is a missing annotation, which looks like the analysis never fired.

SQLite's write lock serializes the transactions outright. On PostgreSQL the rows are locked FOR UPDATE, so a concurrent appender blocks on the read rather than proceeding from a stale copy.

func (*Repository) ApplyDeparosStatusPolicy ¶

func (r *Repository) ApplyDeparosStatusPolicy(ctx context.Context, projectUUID string, policy DeparosStatusPolicy) (int64, map[int]int64, error)

ApplyDeparosStatusPolicy enforces the discovery status-retention policy on stored deparos records. A fuzzed path the server answers with a 4xx is usually not a discovered resource — it's a rejection (malformed/ambiguous path → 400, not-found → 404). But some 4xx statuses carry real attack surface: 401/403 prove an authz boundary exists, 405/415/422 prove a route exists but was hit with the wrong method/payload, and 429 is a pacing signal. The policy retains those (see DeparosStatusPolicy) and drops the rest, rather than deleting every 4xx and losing the signal.

Returns the number of deleted records and a status-code breakdown of them.

func (*Repository) AttributeScanToAgenticScan ¶

func (r *Repository) AttributeScanToAgenticScan(ctx context.Context, scanUUID, agenticScanUUID string) error

AttributeScanToAgenticScan links a native scan (and every finding it produced) to the agentic scan that launched it. The run_scan tool calls this after a blocking LaunchScan returns, so the child scan's findings show up under `finding --agentic-scan <parent>` — which filters on finding.agentic_scan_uuid directly rather than joining through scans. Both the scan row and its findings are stamped; a no-op when either UUID is empty.

func (*Repository) BackfillRecordResponse ¶

func (r *Repository) BackfillRecordResponse(ctx context.Context, uuid string, rr *httpmsg.HttpRequestResponse) error

BackfillRecordResponse populates the response columns of an existing request-only http_record stub from a freshly fetched request/response, computing every response-derived field (content-type/length, hashes, normalized hash, word count, title) via the shared FromHttpRequestResponse converter so the backfilled row matches a normally-saved record.

Discovery stores endpoints synthesized from a parsed OpenAPI/Swagger/Postman spec as request-only stubs (status 0, empty body); the executor then fetches each baseline to scan the endpoint, but without this the stub stays at status 0 with an empty response in the traffic view. The update is guarded on has_response = false so it only ever fills a genuine stub — it never clobbers a record that already captured a response, making it idempotent across the discovery and dynamic-assessment phases. An rr with no response is a no-op.

func (*Repository) CompleteScan ¶

func (r *Repository) CompleteScan(ctx context.Context, scanUUID string, errMsg string) error

CompleteScan marks a scan as completed (or failed if errMsg is non-empty) and populates severity counts from the findings table.

func (*Repository) CountRecordsAfterCursor ¶

func (r *Repository) CountRecordsAfterCursor(ctx context.Context, cursorAt time.Time, cursorUUID string, hosts ...HostTarget) (int64, error)

CountRecordsAfterCursor counts records after the given cursor position. A zero cursorAt means count all records. When hosts is non-empty, only records matching those in-scope origins (scheme+hostname+port) are counted.

func (*Repository) CountRecordsAfterCursorBySource ¶

func (r *Repository) CountRecordsAfterCursorBySource(ctx context.Context, cursorAt time.Time, cursorUUID string, sources []string, hosts []HostTarget) (int64, error)

CountRecordsAfterCursorBySource is like CountRecordsAfterCursor but also filters on http_records.source. Used by scan-on-receive shallow mode to report only user-ingested traffic in the "new ingested records" status, excluding finding/scanner artefacts produced by the scan itself.

func (*Repository) CountRecordsByStatusCode ¶

func (r *Repository) CountRecordsByStatusCode(ctx context.Context, hosts ...HostTarget) (map[int]int64, error)

CountRecordsByStatusCode returns http_record counts grouped by HTTP status code, restricted to the same in-scope origins as CountRecordsAfterCursor (empty hosts = every record). Keys are the raw numeric status codes (0 for a missing/unset status); callers bucket them into 2xx/3xx/… classes. Powers the scan-completion summary's status-class line, so its host scope matches the record count printed alongside it.

func (*Repository) CountRecordsForScan ¶

func (r *Repository) CountRecordsForScan(ctx context.Context, projectUUID, scanUUID string) (int64, error)

CountRecordsForScan returns how many HTTP records this scan persisted.

func (*Repository) CreateAgenticScan ¶

func (r *Repository) CreateAgenticScan(ctx context.Context, run *AgenticScan) error

CreateAgenticScan stores a new agent run record. When a row with run.UUID already exists, the call is a no-op as long as the existing project_uuid matches — this is the get-or-create path used for cross-node sync via --scan-uuid. Returns ErrScanProjectMismatch when the existing row belongs to a different project.

func (*Repository) CreateProject ¶

func (r *Repository) CreateProject(ctx context.Context, project *Project) error

CreateProject inserts a new project.

func (*Repository) CreateScan ¶

func (r *Repository) CreateScan(ctx context.Context, scan *Scan) error

CreateScan inserts a new scan record. When a row with scan.UUID already exists, the call is a no-op as long as the existing project_uuid matches — this is the get-or-create path used for cross-node sync via --scan-uuid. Returns ErrScanProjectMismatch when the existing row belongs to a different project.

func (*Repository) CreateScanLog ¶

func (r *Repository) CreateScanLog(ctx context.Context, log *ScanLog) error

CreateScanLog inserts a scan log entry.

func (*Repository) CreateScanLogBatch ¶

func (r *Repository) CreateScanLogBatch(ctx context.Context, logs []*ScanLog) error

CreateScanLogBatch inserts multiple scan log entries in a single bulk insert.

func (*Repository) CreateScanWithCursor ¶

func (r *Repository) CreateScanWithCursor(ctx context.Context, scan *Scan) error

CreateScanWithCursor creates a Scan record. If mode is "incremental", copies cursor from the last completed scan with matching Modules. Otherwise starts at zero.

func (*Repository) CreateUser ¶

func (r *Repository) CreateUser(ctx context.Context, user *User) error

CreateUser inserts a new user.

func (*Repository) DB ¶

func (r *Repository) DB() *DB

DB returns the underlying database handle.

func (*Repository) DeduplicateDeparosByNormHash ¶

func (r *Repository) DeduplicateDeparosByNormHash(ctx context.Context, projectUUID string) (int64, map[int]int64, error)

DeduplicateDeparosByNormHash collapses deparos records whose NORMALIZED response body is identical — records that differ only by a reflected request URL/path or per-request dynamic tokens (timestamps, ids). This is the case the exact (hostname, status, content_length, response_hash) dedup misses: an error or echo page that mirrors the requested URI gives every probed path a distinct body, length, and hash, so they all survive as separate records even though they are the same page. response_norm_hash (computed at storage time with the reflected URL/path stripped and dynamic runs collapsed) makes them identical.

Grouping is on (hostname, method, status_code, response_content_type, response_norm_hash); the shortest path per group survives. Records with no normalized hash (empty bodies) are left to exact-hash dedup. Returns the number of deleted records and a status-code breakdown.

func (*Repository) DeduplicateDeparosRecords ¶

func (r *Repository) DeduplicateDeparosRecords(ctx context.Context, projectUUID string) (int64, error)

DeduplicateDeparosRecords removes duplicate deparos HTTP records. Delegates to DeduplicateRecordsBySource with source "deparos".

func (*Repository) DeduplicateFindings ¶

func (r *Repository) DeduplicateFindings(ctx context.Context, projectUUID string, hostnames ...string) (deleted int64, grouped int64, err error)

DeduplicateFindings merges duplicate findings that share the same (module_id, severity, matched_at URL) within a project. This collapses findings where the same module fires many times on the same URL with different payloads (e.g., input-behavior-probe producing dozens of results per endpoint). Within each group, the earliest finding is kept and the request/response pairs from duplicates are collected into its AdditionalEvidence field.

When one or more hostnames are passed, the pass is scoped to findings on those hosts (an empty/omitted list dedupes the whole project). The dynamic-assessment feedback loop scopes each round to the hosts being scanned — coverage-equivalent there, since every finding produced during DA is on an in-scope host — so a growing findings table isn't full-scanned once per round. Returns the count of deleted findings and the number of groups that were merged.

func (*Repository) DeduplicateRecordsBySource ¶

func (r *Repository) DeduplicateRecordsBySource(ctx context.Context, projectUUID, source string) (int64, error)

DeduplicateRecordsBySource removes duplicate HTTP records for a given source that share identical (hostname, method, status_code, response_content_length, response_hash). Within each group, the record with the shortest path is kept. Returns the number of deleted records.

func (*Repository) DeduplicateSoftDeparosRecords ¶

func (r *Repository) DeduplicateSoftDeparosRecords(ctx context.Context, projectUUID string) (int64, map[int]int64, error)

DeduplicateSoftDeparosRecords removes deparos HTTP records that are "soft duplicates": same response characteristics (status, word count, content type) under the same 2-segment path prefix. This catches cases where the server echoes part of the URL in the response body, causing different response_hash values for functionally identical pages. Exact response_content_length is deliberately NOT part of the key: a reflected URL/path shifts the body length by a handful of bytes per probe, which would otherwise split an otherwise-identical family into singletons (the very case this pass exists to collapse) — word count + content type + status + path prefix is the stable shape signal. Only groups with 3+ members are collapsed. The shortest path per group is kept.

func (*Repository) DeleteAuthenticationHostname ¶

func (r *Repository) DeleteAuthenticationHostname(ctx context.Context, id int64) error

DeleteAuthenticationHostname deletes a single session hostname by ID.

func (*Repository) DeleteAuthenticationHostnamesByHostname ¶

func (r *Repository) DeleteAuthenticationHostnamesByHostname(ctx context.Context, projectUUID, hostname string) error

DeleteAuthenticationHostnamesByHostname deletes all session hostnames for a project+hostname.

func (*Repository) DeleteFinding ¶

func (r *Repository) DeleteFinding(ctx context.Context, id int64) error

DeleteFinding deletes a finding by its numeric ID, including any finding_records junction rows.

func (*Repository) DeleteFindingByHash ¶

func (r *Repository) DeleteFindingByHash(ctx context.Context, projectUUID, findingHash string) error

DeleteFindingByHash deletes the finding(s) with the given content hash within a project, including their finding_records junction rows. Used by the OAST collector to replace a weaker finding (e.g. a DNS-resolution lead) with a stronger one (the HTTP-fetch confirmation) for the same callback payload, so each payload yields exactly one finding reflecting the strongest evidence. No-op when findingHash is empty or no such finding exists.

func (*Repository) DeleteOASTInteraction ¶

func (r *Repository) DeleteOASTInteraction(ctx context.Context, id int64) error

DeleteOASTInteraction deletes an OAST interaction by its numeric ID.

func (*Repository) DeleteOldAgenticScans ¶

func (r *Repository) DeleteOldAgenticScans(ctx context.Context, olderThan time.Duration) (int, error)

DeleteOldAgenticScans removes finished agent runs older than the given duration. "Finished" is TerminalAgenticScanStatuses, not a literal list — see that variable.

func (*Repository) DeleteProject ¶

func (r *Repository) DeleteProject(ctx context.Context, uuid string) error

DeleteProject deletes a project by UUID.

func (*Repository) DeleteRecord ¶

func (r *Repository) DeleteRecord(ctx context.Context, uuid string) error

DeleteRecord deletes an HTTP record by UUID, including any finding_records junction rows.

func (*Repository) DeleteScan ¶

func (r *Repository) DeleteScan(ctx context.Context, uuid string) error

DeleteScan deletes a scan record by UUID.

func (*Repository) ExistingRecordUUIDs ¶

func (r *Repository) ExistingRecordUUIDs(ctx context.Context, projectUUID string, uuids []string) ([]string, error)

ExistingRecordUUIDs returns the subset of uuids that exist. It selects the uuid column straight into a []string, so validating a caller-supplied list costs neither the raw_request/raw_response blobs nor an HTTPRecord struct per row.

The answer is always scoped to a project, blank meaning the default one as everywhere else in this package. The scoping is not optional because enforcing ownership is this method's whole purpose: record UUIDs are globally unique, so an unscoped check answers "does this row exist anywhere" when the question is "does the caller own it", and a caller who learns one UUID from another engagement could pull its traffic into their own scan. An opt-out parameter would make the safe and unsafe calls look identical.

func (*Repository) FindingProjectUUID ¶

func (r *Repository) FindingProjectUUID(ctx context.Context, id int64) (string, error)

FindingProjectUUID returns just the project a finding belongs to, for the existence-plus-scope check the mutating handlers perform before acting on it. Returns sql.ErrNoRows when the finding does not exist, like GetFindingByID.

Projected on purpose: the callers read only this one field, and a finding carries its own inline request/response blobs.

func (*Repository) GetAgenticScan ¶

func (r *Repository) GetAgenticScan(ctx context.Context, uuid string) (*AgenticScan, error)

GetAgenticScan retrieves an agent run by UUID.

func (*Repository) GetAllProjectsStats ¶

func (r *Repository) GetAllProjectsStats(ctx context.Context) (map[string]*ProjectStatsRow, error)

GetAllProjectsStats returns aggregated stats for all projects in bulk. Uses GROUP BY to avoid N+1 queries when listing projects.

func (*Repository) GetAuthenticationHostnamesByHostname ¶

func (r *Repository) GetAuthenticationHostnamesByHostname(ctx context.Context, projectUUID, hostname string) ([]*AuthenticationHostname, error)

GetAuthenticationHostnamesByHostname returns session hostnames for a project+hostname, ordered by position.

func (*Repository) GetAuthenticationHostnamesByProject ¶

func (r *Repository) GetAuthenticationHostnamesByProject(ctx context.Context, projectUUID string) ([]*AuthenticationHostname, error)

GetAuthenticationHostnamesByProject returns all session hostnames for a project, ordered by hostname then position.

func (*Repository) GetAuthenticationHostnamesByScan ¶

func (r *Repository) GetAuthenticationHostnamesByScan(ctx context.Context, projectUUID, scanUUID string) ([]*AuthenticationHostname, error)

GetAuthenticationHostnamesByScan returns session hostnames for a project+scan, ordered by hostname then position.

func (*Repository) GetChildAgenticScans ¶

func (r *Repository) GetChildAgenticScans(ctx context.Context, parentUUID string) ([]*AgenticScan, error)

GetChildAgenticScans returns agent runs whose ParentAgenticScanUUID matches the given UUID.

func (*Repository) GetDistinctHosts ¶

func (r *Repository) GetDistinctHosts(ctx context.Context, projectUUID string, since ...time.Time) ([]HostTarget, error)

GetDistinctHosts returns distinct scheme+hostname+port combinations from HTTP records, filtered by project. When a non-zero `since` is supplied, only records created at/after that time are considered (used to find origins discovered during a specific scan).

func (*Repository) GetDistinctPaths ¶

func (r *Repository) GetDistinctPaths(ctx context.Context, projectUUID string, hosts ...HostTarget) ([]PathTarget, error)

GetDistinctPaths returns distinct scheme+hostname+port+path combinations from HTTP records, filtered by project and, when hosts is non-empty, restricted to those in-scope origins (matching the executor's WithHostScopes convention). Empty hosts means no origin filter.

func (*Repository) GetFindingByID ¶

func (r *Repository) GetFindingByID(ctx context.Context, id int64) (*Finding, error)

GetFindingByID retrieves a single finding by numeric ID.

func (*Repository) GetFindingsByRecordUUID ¶

func (r *Repository) GetFindingsByRecordUUID(ctx context.Context, uuid string) ([]*Finding, error)

GetFindingsByRecordUUID retrieves findings that reference a specific HTTP record UUID. Since http_record_uuids is a JSONB array, we use json_each to search inside it.

func (*Repository) GetFindingsBySeverity ¶

func (r *Repository) GetFindingsBySeverity(ctx context.Context, projectUUID, sev string, limit int) ([]*Finding, error)

GetFindingsBySeverity retrieves findings filtered by severity within a project.

func (*Repository) GetOASTInteractionByID ¶

func (r *Repository) GetOASTInteractionByID(ctx context.Context, id int64) (*OASTInteraction, error)

GetOASTInteractionByID retrieves a single OAST interaction by its numeric ID.

func (*Repository) GetOASTInteractionsByScan ¶

func (r *Repository) GetOASTInteractionsByScan(ctx context.Context, scanUUID string) ([]*OASTInteraction, error)

GetOASTInteractionsByScan retrieves OAST interactions for a specific scan.

func (*Repository) GetProjectByName ¶

func (r *Repository) GetProjectByName(ctx context.Context, name string) (*Project, error)

GetProjectByName retrieves a project by name. Returns an error if zero or multiple projects match (names are not guaranteed to be unique).

func (*Repository) GetProjectByUUID ¶

func (r *Repository) GetProjectByUUID(ctx context.Context, uuid string) (*Project, error)

GetProjectByUUID retrieves a project by UUID.

func (*Repository) GetProjectStats ¶

func (r *Repository) GetProjectStats(ctx context.Context, projectUUID string) (*ProjectStatsRow, error)

GetProjectStats returns aggregated stats for a single project.

func (*Repository) GetReSpiderCandidates ¶

func (r *Repository) GetReSpiderCandidates(ctx context.Context, projectUUID, afterUUID string, limit int, hosts ...HostTarget) ([]ReSpiderCandidate, error)

GetReSpiderCandidates returns records that carry a response body, for the targeted re-spider phase, using UUID-based cursor pagination (afterUUID="" for the first page). It deliberately includes spidering-sourced records too, so the caller can pre-seed its shell-dedup set with pages the browser already crawled. Ordered by uuid so paging is stable and per-host capping deterministic. Each returned row carries a full raw_response body, so the caller pages with a modest limit and reduces each page before reading the next rather than materializing every body at once.

func (*Repository) GetRecordByRequestHash ¶

func (r *Repository) GetRecordByRequestHash(ctx context.Context, projectUUID, requestHash string) (*HTTPRecord, error)

GetRecordByRequestHash retrieves the most recent HTTP record whose raw request hashes to requestHash (the SHA-256 over the raw request bytes, matching httpmsg.HttpRequest.ID). It recovers the originating request for an out-of-band (OAST) finding when the in-memory hash→UUID resolver is no longer available — e.g. a callback that lands after the executor that planted the payload has been torn down. projectUUID may be empty to search across projects.

func (*Repository) GetRecordByUUID ¶

func (r *Repository) GetRecordByUUID(ctx context.Context, uuid string) (*HTTPRecord, error)

GetRecordByUUID retrieves a single HTTP record by UUID

func (*Repository) GetRecordMetadataByHostname ¶

func (r *Repository) GetRecordMetadataByHostname(ctx context.Context, projectUUID, hostname string, limit int) ([]*HTTPRecord, error)

GetRecordMetadataByHostname is GetRecordsByHostname without the raw_request / raw_response BLOB columns, for callers that only need request-line metadata (method, URL, path, status). It avoids pulling multi-KB bodies for every row when they're immediately discarded — notably the autopilot coverage probe, which scans up to 5000 rows just to build (method, URL) signatures. Use the full GetRecordsByHostname when the bodies are needed.

func (*Repository) GetRecordsByHostname ¶

func (r *Repository) GetRecordsByHostname(ctx context.Context, projectUUID, hostname string, limit int) ([]*HTTPRecord, error)

GetRecordsByHostname retrieves HTTP records for a hostname within a project.

func (*Repository) GetRecordsByUUIDs ¶

func (r *Repository) GetRecordsByUUIDs(ctx context.Context, uuids []string) ([]*HTTPRecord, error)

GetRecordsByUUIDs retrieves HTTP records matching the given UUIDs.

func (*Repository) GetRecordsWithResponseBody ¶

func (r *Repository) GetRecordsWithResponseBody(ctx context.Context, projectUUID, afterUUID string, limit int, hosts ...HostTarget) ([]*HTTPRecord, error)

GetRecordsWithResponseBody returns HTTP records that have a non-empty response body, using UUID-based cursor pagination. Only columns needed for batch secret scanning are selected.

func (*Repository) GetRelatedRecords ¶

func (r *Repository) GetRelatedRecords(ctx context.Context, uuid string, limit int) ([]*HTTPRecord, error)

GetRelatedRecords finds HTTP records with the same hostname and a path matching the path-template of the given UUID's record. Default limit 10; excludes the source record itself. Results are filtered to the same path depth as the source record.

func (*Repository) GetScanByUUID ¶

func (r *Repository) GetScanByUUID(ctx context.Context, uuid string) (*Scan, error)

GetScanByUUID retrieves a scan by its UUID

func (*Repository) GetScanRecordsByUUIDs ¶

func (r *Repository) GetScanRecordsByUUIDs(ctx context.Context, uuids []string) ([]*HTTPRecord, error)

GetScanRecordsByUUIDs retrieves HTTP records matching the given UUIDs, projecting only scanRecordColumns — what the dynamic-assessment scan feed consumes. This avoids loading (and JSON-deserializing) the parameters/technology/remarks jsonb columns plus ~25 unused scalar columns per record. Modules receive the reconstructed HttpRequestResponse (via recordToHttpRequestResponse), never the HTTPRecord, so no downstream consumer reads the dropped fields.

func (*Repository) GetUnprobedRecordsBySource ¶

func (r *Repository) GetUnprobedRecordsBySource(ctx context.Context, projectUUID, source, hostname string, limit int) ([]*HTTPRecord, error)

GetUnprobedRecordsBySource returns records with has_response=false for the given source and hostname.

func (*Repository) GetUserByUUID ¶

func (r *Repository) GetUserByUUID(ctx context.Context, uuid string) (*User, error)

GetUserByUUID retrieves a user by UUID.

func (*Repository) GroupFindingsByValue ¶

func (r *Repository) GroupFindingsByValue(ctx context.Context, projectUUID string, opts GroupFindingOptions) (deleted int64, grouped int64, err error)

GroupFindingsByValue collapses findings that share an identical extracted value (e.g. the same leaked secret reported once per URL) into a single finding. The survivor (earliest by created_at) absorbs every duplicate's matched URLs into MatchedAt and a sample of their request/response pairs into AdditionalEvidence; the duplicates are then deleted. Grouping is keyed on (module_id, severity[, hostname], normalized extracted_results) — the value must match byte-for-byte, so findings with distinct or empty extracted values are never merged.

Modules listed in opts.ByModule are an exception: their findings collapse on (module_id, severity[, hostname]) alone, regardless of (or absent) an extracted value, and skip the Tags gate. This is for modules that fire once per asset where the per-URL value is noise (e.g. sourcemap-detect, a distinct .map filename per JS bundle).

Modules listed in opts.ByRule collapse on (module_id, rule_name, severity[, hostname]) — the same as ByModule but with the rule (module_name) kept in the key, so a module whose single id fronts many rules (secret-detect) folds repeats of one rule while keeping different rules apart. Their distinct values are unioned onto the survivor just like ByModule.

Returns the count of deleted findings and the number of groups that were collapsed.

func (*Repository) HasHostObservations ¶

func (r *Repository) HasHostObservations(ctx context.Context) bool

HasHostObservations reports whether this database can answer endpoint lookups at all.

A database written before this table existed has nothing stored, which is a fact about it and not a failure. Asked once up front so a read of an older file degrades to the cache-only behaviour silently, instead of failing a query per endpoint about a table it was never going to have. A read-only handle cannot be migrated into having one either, and missingColumns only tracks ALTER-added columns, never whole tables.

func (*Repository) HydrateFindingRequests ¶

func (r *Repository) HydrateFindingRequests(ctx context.Context, findings []*Finding)

HydrateFindingRequests populates the Request field on findings that were loaded without it — the list query (FindingsQueryBuilder) omits the request/response blobs for speed. It fetches id→request for the given findings in as few round-trips as one statement's worth of ids allows; findings that already carry a Request, or that have no id, are skipped. Used by display paths that need a finding's own PoC request (e.g. the tree's attack-URL rendering) without pulling the full raw view.

func (*Repository) InScopeHosts ¶

func (r *Repository) InScopeHosts(ctx context.Context, scopeCfg config.ScopeConfig, targets []string, projectUUID, scanUUID string) []HostTarget

InScopeHosts returns the distinct (scheme, hostname, port) origins in the project that are in scope for the supplied CLI targets. An origin is in scope when its hostname passes the scope matcher AND EITHER its (scheme, port) matches one of the targets' origins (default ports normalized like stored records: https→443, http→80) OR it was discovered during the current scan (records created at/after the scan's start). Returns nil when there are no targets (no filter — a project-wide pass). This is the single source of truth for the origin scoping the executor applies via WithHostScopes, so DynamicAssessment, KnownIssueScan, spidering/discovery seeds, and the scan-completion record count all derive the same set.

The full-origin match keeps e.g. localhost:3000 separate from localhost:8080 left in the project by a prior scan, while the current-scan provenance carve-out still lets THIS scan's own discoveries on a different port (e.g. the --intensity deep port sweep, or a followed cross-port link) be scanned. Records carry no scan_uuid, so provenance is keyed on the scan's start time. If no target yields a determinable scheme/port (e.g. all bare hosts), the scheme/port constraint is dropped — a safe fallback that never over-excludes.

func (*Repository) IncrementProcessedCount ¶

func (r *Repository) IncrementProcessedCount(ctx context.Context, scanUUID string, delta int64) error

IncrementProcessedCount adds delta to the scan's processed_count. Use this for phases that don't advance the cursor (discovery, spidering, etc.).

func (*Repository) ListAgentSections ¶

func (r *Repository) ListAgentSections(ctx context.Context, agenticScanUUID string) ([]*AgentSection, error)

ListAgentSections returns all sections for an agentic scan ordered by seq (execution order). Empty agenticScanUUID returns an empty slice.

func (*Repository) ListAgenticScanSummaries ¶

func (r *Repository) ListAgenticScanSummaries(ctx context.Context, projectUUID string, mode string, limit, offset int) ([]*AgenticScan, int64, error)

ListAgenticScanSummaries is ListAgenticScans with agenticScanBlobColumns dropped — same rows, same order, same total. Use it for any view that renders a run rather than replaying it; the omitted fields come back empty, so a caller that serializes the *AgenticScan verbatim (the `--json` CLI paths) must keep using ListAgenticScans.

func (*Repository) ListAgenticScans ¶

func (r *Repository) ListAgenticScans(ctx context.Context, projectUUID string, mode string, limit, offset int) ([]*AgenticScan, int64, error)

ListAgenticScans returns paginated agent runs for a project, ordered by created_at DESC, with every column populated. Callers that only render the run should prefer ListAgenticScanSummaries.

func (*Repository) ListCandidates ¶

func (r *Repository) ListCandidates(ctx context.Context, agenticScanUUID string, statuses ...string) ([]*AgentFindingCandidate, error)

ListCandidates returns candidates for an agentic scan, optionally filtered to the given statuses (empty statuses = all). Ordered by id so the verifier processes them in proposal order.

func (*Repository) ListFindings ¶

func (r *Repository) ListFindings(ctx context.Context, filters QueryFilters) ([]*Finding, int64, error)

ListFindings runs a filtered findings query and returns the matching page plus the total unfiltered count, in a single round-trip. The canonical entry point for callers that want paginated results — keeps FindingsQueryBuilder behind the repository boundary so they don't need to reach for Repository.DB().

func (*Repository) ListOASTInteractions ¶

func (r *Repository) ListOASTInteractions(ctx context.Context, projectUUID, scanUUID, protocol, moduleID, search string, limit, offset int) ([]*OASTInteraction, int64, error)

ListOASTInteractions returns a paginated, filtered list of OAST interactions. Heavy columns (raw_request, raw_response) are excluded for list performance.

func (*Repository) ListProjects ¶

func (r *Repository) ListProjects(ctx context.Context, ownerUUID string) ([]*Project, error)

ListProjects returns projects, optionally filtered by owner.

func (*Repository) ListScanLogs ¶

func (r *Repository) ListScanLogs(ctx context.Context, scanUUID string, level, phase string, limit, offset int) ([]*ScanLog, int64, error)

ListScanLogs returns log entries for a scan, ordered by created_at ascending. Both level and phase are optional filters; pass "" to skip.

func (*Repository) ListScans ¶

func (r *Repository) ListScans(ctx context.Context, projectUUID string, limit, offset int) ([]*Scan, int64, error)

ListScans returns scans ordered by created_at descending with limit/offset, filtered by project.

func (*Repository) ListUsers ¶

func (r *Repository) ListUsers(ctx context.Context) ([]*User, error)

ListUsers returns all users.

func (*Repository) LoadEnabledScopes ¶

func (r *Repository) LoadEnabledScopes(ctx context.Context, projectUUID string) ([]*Scope, error)

LoadEnabledScopes loads enabled scope rules for a project, ordered by priority. An empty projectUUID means the default project — never "all projects". Falls back to the default project's scopes when a non-default project has none, but if the default project itself has no enabled scopes it returns empty rather than an unscoped query that would leak every project's rules across engagements.

func (*Repository) LookupHostObservation ¶

func (r *Repository) LookupHostObservation(ctx context.Context, projectUUID, scanUUID, hostname string, port int) (HostFacts, error)

LookupHostObservation returns the observation for one endpoint, or zero facts when none is stored.

One endpoint per call, rather than loading a run's observations up front. Preloading looked right for a streaming emit path - fetch once, then answer every record from memory - and was wrong in both directions. This table accumulates a row per endpoint per RUN, so the project-wide preload returned endpoints × runs rows to build a map of endpoints: on 20k endpoints across 25 runs, exporting a two-record database peaked at 842 MB against a 120 MB baseline, with export memory growing in scan history rather than in the corpus being exported. Resolving "latest per endpoint" in SQL instead still cost a full scan and a temp b-tree. And a paginated reader (`traffic -j`, 100 rows) paid all of it to ask about a handful of hosts.

A point lookup is index-only against idx_host_obs_project_host_id, and the caller memoizes, so the cost is one query per DISTINCT endpoint actually emitted and the memory is one entry per the same.

scanUUID empty widens the read to the whole project and takes the most recently RECORDED observation - the right answer for `export` of a database whose runs this process did not perform.

func (*Repository) OverwriteRecordResponseBody ¶

func (r *Repository) OverwriteRecordResponseBody(ctx context.Context, uuid string, rawResponse []byte) error

OverwriteRecordResponseBody replaces the stored raw response of the record with the given UUID and recomputes its derived fields (response_hash, response_norm_hash, response_words, response_content_length), keeping them consistent with dedup/resume/fingerprinting that key off those columns.

rawResponse must be a complete HTTP response (status line + headers + body). Used by the passive js-beautify module to overwrite a minified JS body with its beautified form in place. The derivation mirrors Record.FromRequestResponse so a rewritten record hashes identically to one ingested with the new body. (Distinct from UpdateRecordResponse, which is the replay feature's full response-field swap.)

func (*Repository) PauseScan ¶

func (r *Repository) PauseScan(ctx context.Context, scanUUID string) error

PauseScan sets a scan's status to "paused".

func (*Repository) PurgeProjectData ¶

func (r *Repository) PurgeProjectData(ctx context.Context, projectUUID string) error

PurgeProjectData deletes every row tied to projectUUID across all per-project tables. finding_records has no project_uuid, so it's pruned via a subquery on findings.id before the findings rows themselves are removed. Runs in a single transaction so a partial failure leaves the project intact.

func (*Repository) ReassignProjectData ¶

func (r *Repository) ReassignProjectData(ctx context.Context, sourceUUID, targetUUID string) error

ReassignProjectData moves all data owned by sourceUUID to targetUUID. This should be called before deleting a project so its records are not orphaned.

func (*Repository) RecordProjectUUID ¶

func (r *Repository) RecordProjectUUID(ctx context.Context, uuid string) (string, error)

RecordProjectUUID returns just the project a record belongs to, for the existence-plus-scope check the mutating handlers perform before acting on it. Returns sql.ErrNoRows when the record does not exist, like GetRecordByUUID.

Projected on purpose: the callers read only this one field, and a record's raw_response alone can be megabytes.

func (*Repository) RefreshScanStats ¶

func (r *Repository) RefreshScanStats(ctx context.Context, scanUUID string) error

RefreshScanStats updates running scan stats during long-running scans where CompleteScan hasn't been called yet.

func (*Repository) ResetScanCursor ¶

func (r *Repository) ResetScanCursor(ctx context.Context, scanUUID string) error

ResetScanCursor resets the scan cursor to the beginning so all records are re-read on the next iteration (e.g., between seed and audit phases).

func (*Repository) ResumeScan ¶

func (r *Repository) ResumeScan(ctx context.Context, scanUUID string) error

ResumeScan sets a scan's status back to "running".

func (*Repository) SaveAgentSection ¶

func (r *Repository) SaveAgentSection(ctx context.Context, section *AgentSection) error

SaveAgentSection inserts a new bounded-section row. The uuid carries the section identity; a duplicate uuid is a no-op (idempotent on resume).

func (*Repository) SaveAnalysisArtifact ¶

func (r *Repository) SaveAnalysisArtifact(
	ctx context.Context,
	httpRecordUUID, kind, filename, mediaType, sha256 string,
	content []byte,
	metadata string,
) error

SaveAnalysisArtifact is the primitive-argument adapter used by input sources that deliberately avoid importing the database package in their interfaces.

func (*Repository) SaveAnalysisArtifactForRecord ¶

func (r *Repository) SaveAnalysisArtifactForRecord(ctx context.Context, artifact *AnalysisArtifact) error

SaveAnalysisArtifactForRecord stores a content-addressed immutable artifact beside an existing HTTP record. Duplicate writes of the same record/kind/hash are idempotent.

func (*Repository) SaveAuthenticationHostname ¶

func (r *Repository) SaveAuthenticationHostname(ctx context.Context, sh *AuthenticationHostname) error

SaveAuthenticationHostname upserts a single session hostname record. Conflict key: (project_uuid, hostname, session_name).

func (*Repository) SaveAuthenticationHostnames ¶

func (r *Repository) SaveAuthenticationHostnames(ctx context.Context, rows []*AuthenticationHostname) error

SaveAuthenticationHostnames batch-upserts session hostname records in a transaction.

func (*Repository) SaveCandidate ¶

func (r *Repository) SaveCandidate(ctx context.Context, cand *AgentFindingCandidate) (bool, error)

SaveCandidate inserts a proposed finding candidate, deduping on (agentic_scan_uuid, dedup_hash) via ON CONFLICT DO NOTHING so a looping model that re-proposes the same bug produces exactly one row. Returns inserted=true only when a new row was written (false on a dedup hit), so callers can keep an accurate proposed-count.

func (*Repository) SaveFinding ¶

func (r *Repository) SaveFinding(ctx context.Context, event *output.ResultEvent, httpRecordUUIDs []string, scanUUID string, projectUUID string) error

SaveFinding stores a vulnerability finding linked to HTTP records by UUIDs. Uses INSERT ON CONFLICT for atomic dedup when finding_hash is non-empty.

func (*Repository) SaveFindingDirect ¶

func (r *Repository) SaveFindingDirect(ctx context.Context, finding *Finding) error

SaveFindingDirect inserts a pre-built Finding directly (without ResultEvent conversion). Uses INSERT ON CONFLICT for atomic dedup when finding_hash is non-empty.

func (*Repository) SaveFindingReported ¶

func (r *Repository) SaveFindingReported(ctx context.Context, finding *Finding) (bool, error)

SaveFindingReported is SaveFindingDirect that additionally reports whether a new finding row was inserted (true) vs collapsed into an existing finding by dedup (false). The autopilot report_finding tool uses it so its run counter only tallies distinct findings.

func (*Repository) SaveFindingsBatch ¶

func (r *Repository) SaveFindingsBatch(ctx context.Context, writes []FindingWrite) error

SaveFindingsBatch persists a batch of findings in a single transaction, coalescing what would otherwise be one transaction (and fsync) per finding. Findings are converted up front so a conversion error skips just that finding rather than aborting the batch. If the transaction itself fails (e.g. a database-level error), every finding is retried individually so one bad finding can't drop the rest — preserving the error isolation of per-finding SaveFinding while keeping the fast path a single transaction.

func (*Repository) SaveFindingsDirectBatch ¶

func (r *Repository) SaveFindingsDirectBatch(ctx context.Context, findings []*Finding) ([]FindingSaveResult, error)

SaveFindingsDirectBatch persists a batch of pre-built findings in a single transaction — the batch analogue of SaveFindingDirect, used by bulk importers to coalesce what would otherwise be one transaction (and fsync) per finding. The returned slice is aligned with the input; the top-level error is the first per-finding error, if any.

func (*Repository) SaveHostObservations ¶

func (r *Repository) SaveHostObservations(ctx context.Context, projectUUID, scanUUID string, obs []HostObservationInput) error

SaveHostObservations persists a run's host observations in bounded batches.

ON CONFLICT (scan_uuid, hostname, port) updates rather than inserting: within ONE run an endpoint has one observation, so a second write for it is a refinement (the TLS handshake finishing after the DNS answer, say) and not a second event. ACROSS runs the conflict cannot fire, which is the point - re-probing a host next month adds that month's observation and leaves this one intact.

func (*Repository) SaveOASTInteraction ¶

func (r *Repository) SaveOASTInteraction(ctx context.Context, interaction *OASTInteraction) error

SaveOASTInteraction stores an OAST interaction record.

func (*Repository) SaveRecord ¶

func (r *Repository) SaveRecord(ctx context.Context, httpRR *httpmsg.HttpRequestResponse, source string, projectUUID string) (string, error)

SaveRecord stores a denormalized HTTP record (request + response + host + parameters). The source identifies the origin of the record (e.g. "scanner", "ingest-cli", "ingest-server", "ingest-proxy"). Returns the UUID of the saved record. If a matching record already exists (same method, hostname, path, URL, and request body), the existing UUID is returned without inserting.

func (*Repository) SaveRecordBatch ¶

func (r *Repository) SaveRecordBatch(ctx context.Context, records []*httpmsg.HttpRequestResponse, source string, projectUUID string) ([]string, error)

SaveRecordBatch converts httpmsg.HttpRequestResponse objects to HTTPRecord models and batch-inserts them. This is the high-level batch equivalent of SaveRecord.

func (*Repository) SaveRecordWithParent ¶

func (r *Repository) SaveRecordWithParent(ctx context.Context, httpRR *httpmsg.HttpRequestResponse, source string, projectUUID string, parentUUID string) (string, error)

SaveRecordWithParent is SaveRecord with an explicit parent_uuid link, used to chain the hops of a followed redirect into one walkable sequence.

On a duplicate hit the existing row's parent is BACKFILLED but never overwritten (see AdoptRecordParent). A row can lose the race to its own chain: a passive module reporting on the final response persists it as finding evidence before the item's own save runs, and that earlier row then wins the dedup — leaving the last link of the chain dangling unless the parent is applied after the fact.

func (*Repository) SaveRecordsBatch ¶

func (r *Repository) SaveRecordsBatch(ctx context.Context, records []*HTTPRecord) ([]string, error)

SaveRecordsBatch inserts multiple HTTP records in a single transaction. Returns the UUIDs of all successfully inserted records.

func (*Repository) ScanSeverityTally ¶

func (r *Repository) ScanSeverityTally(ctx context.Context, scanUUID string) map[string]int

ScanSeverityTally returns a scan's persisted finding counts keyed by severity name, omitting severities with no findings. It reads the same rows the scan row's own counters are computed from, so a consumer reading the machine event stream's scan.finished and a consumer querying the database afterwards agree.

func (*Repository) SetRecordTechnology ¶

func (r *Repository) SetRecordTechnology(ctx context.Context, technology map[string][]string) error

SetRecordTechnology batch-writes the detected technology stack onto HTTPRecords identified by UUID, replacing whatever the column held.

Replace rather than merge (the shape AppendRemarks uses) because technology is not accumulated analyst annotation: it is the fingerprint modules' verdict about the host, recomputed from scratch on every scan, and a merge would make a record carry a stack the target stopped running two scans ago.

Records are grouped by their technology list before writing, because the source is host-scoped: every record on one host shares one list, so a corpus of N records over H hosts costs H UPDATEs rather than N. The per-value UUID set is still chunked so a single busy host cannot build an IN() clause past the driver's parameter limit.

func (*Repository) UpdateAgentSection ¶

func (r *Repository) UpdateAgentSection(ctx context.Context, section *AgentSection) error

UpdateAgentSection updates a section row identified by uuid. OmitZero lets the caller pass a partial struct (e.g. only status + closing_summary + ended_at) without clobbering columns set at BeginSection time.

func (*Repository) UpdateAgenticScan ¶

func (r *Repository) UpdateAgenticScan(ctx context.Context, run *AgenticScan) error

UpdateAgenticScan updates an agent run record by UUID. Uses OmitZero so callers (SwarmRunner.Run and the API enrichAgenticScanRecord helpers) can pass partial structs without zeroing fields they don't intend to touch. To explicitly clear a field, use a column-list update via NewUpdate().

func (*Repository) UpdateAgenticScanStorageURL ¶

func (r *Repository) UpdateAgenticScanStorageURL(ctx context.Context, agenticScanUUID, storageURL string) error

UpdateAgenticScanStorageURL sets the storage_url field on an agentic scan record.

func (*Repository) UpdateCandidateStatus ¶

func (r *Repository) UpdateCandidateStatus(ctx context.Context, uuid, status, verdictReason string, promotedFindingID int64) error

UpdateCandidateStatus records a verifier verdict on a candidate by uuid, setting status, verdict_reason, promoted_finding_id (when > 0), and stamping verified_at. Used by the fresh-context verifier to mark confirmed / rejected / needs_evidence.

func (*Repository) UpdateFindingSeverity ¶

func (r *Repository) UpdateFindingSeverity(ctx context.Context, id int64, severity string) error

UpdateFindingSeverity sets the severity of a single finding by ID. Returns sql.ErrNoRows if no finding matches.

func (*Repository) UpdateFindingStatus ¶

func (r *Repository) UpdateFindingStatus(ctx context.Context, id int64, status string) error

UpdateFindingStatus sets the lifecycle status of a single finding by ID. Returns sql.ErrNoRows if no finding matches.

func (*Repository) UpdateFindingStatusByHash ¶

func (r *Repository) UpdateFindingStatusByHash(ctx context.Context, agenticScanUUID, findingHash, status string) (int64, error)

UpdateFindingStatusByHash sets the lifecycle status of findings matching a finding_hash within an agentic_scan_uuid scope. Used by the swarm triage writeback to promote draft findings to triaged / false_positive based on agent verdicts. Returns the number of rows updated.

func (*Repository) UpdateFindingTriage ¶

func (r *Repository) UpdateFindingTriage(ctx context.Context, id int64, severity, description string) error

UpdateFindingTriage sets severity and description on a single finding in one statement. Used by the agent triage flow so a false-positive verdict can't land a half-updated row (severity downgraded but reasoning lost) if the process is killed between two separate UPDATEs.

func (*Repository) UpdateProject ¶

func (r *Repository) UpdateProject(ctx context.Context, project *Project) error

UpdateProject updates an existing project.

func (*Repository) UpdateRecordAnnotations ¶

func (r *Repository) UpdateRecordAnnotations(ctx context.Context, uuid string, riskScore *int, remarks []string) error

UpdateRecordAnnotations updates the risk_score and/or remarks of an HTTP record. Only non-nil fields are updated. Returns an error if no record matches the UUID.

func (*Repository) UpdateRecordResponse ¶

func (r *Repository) UpdateRecordResponse(ctx context.Context, uuid string, update *RecordResponseUpdate) error

UpdateRecordResponse replaces the response fields of an existing HTTP record.

func (*Repository) UpdateRiskScores ¶

func (r *Repository) UpdateRiskScores(ctx context.Context, scores map[string]int) error

UpdateRiskScores batch-updates risk_score on HTTPRecords identified by UUID. Uses CASE/WHEN SQL to update up to 500 records per statement, minimizing roundtrips.

func (*Repository) UpdateScan ¶

func (r *Repository) UpdateScan(ctx context.Context, scan *Scan) error

UpdateScan updates an existing scan record

func (*Repository) UpdateScanPartial ¶

func (r *Repository) UpdateScanPartial(ctx context.Context, scan *Scan) error

UpdateScanPartial updates a scan row by UUID, skipping fields whose Go value is the zero value. Use this when the caller only wants to touch a subset of fields (e.g. an API PATCH that should leave omitted fields unchanged). To explicitly clear a field, use a column-level Set via NewUpdate().

func (*Repository) UpdateScanStorageURL ¶

func (r *Repository) UpdateScanStorageURL(ctx context.Context, scanUUID, storageURL string) error

UpdateScanStorageURL sets the storage_url field on a scan record.

func (*Repository) UpdateSurfaceScores ¶

func (r *Repository) UpdateSurfaceScores(ctx context.Context, scores map[string]int) error

UpdateSurfaceScores batch-updates surface_score on HTTPRecords identified by UUID. Written by the surface_scoring passive module; see HTTPRecord.SurfaceScore for why this is a separate column from risk_score rather than a second writer of the same one.

func (*Repository) UpsertSnapshotRecord ¶

func (r *Repository) UpsertSnapshotRecord(ctx context.Context, httpRR *httpmsg.HttpRequestResponse, source string, projectUUID string) (uuid, outcome string, err error)

UpsertSnapshotRecord stores a Burp snapshot record idempotently. When an existing request later gains a response, or its response changes, the response-derived columns are refreshed in place while the stable UUID and finding links are preserved. outcome is inserted, updated, or unchanged.

func (*Repository) UpsertUser ¶

func (r *Repository) UpsertUser(ctx context.Context, user *User) error

UpsertUser inserts a new user or updates name/email if the UUID already exists. Returns the user's UUID.

func (*Repository) WalkJavaScriptRecords ¶

func (r *Repository) WalkJavaScriptRecords(ctx context.Context, projectUUID, hostname string, limit int, fn func(recordURL, contentType string, body []byte) error) error

WalkJavaScriptRecords streams JavaScript responses already stored for a host by earlier phases (spidering, proxy/Burp ingestion) so the discovery engine can feed browser-collected bundles through JSTangle even though they live in the main DB rather than the ephemeral discovery sitemap. Each callback receives the record URL, its response content-type, and the decoded (gzip-inflated) response body. JavaScript is matched by content-type or a .js/.mjs URL suffix; the walk is bounded by limit and ordered by uuid for determinism. A callback error stops the walk and is returned. This satisfies the source.spideredJSProvider optional interface structurally (primitive-typed) so neither package imports the other.

func (*RiskPrioritizedDBInputSource) Close ¶

func (s *RiskPrioritizedDBInputSource) Close() error

Close stops the source.

func (*RiskPrioritizedDBInputSource) CoalescedDropped ¶

func (s *RiskPrioritizedDBInputSource) CoalescedDropped() int

CoalescedDropped returns how many records the param-shape coalescing pass removed from this scan's iteration list. Valid after the snapshot loads (i.e. after the first Next / after Execute). Zero when coalescing is disabled.

func (*RiskPrioritizedDBInputSource) Next ¶

func (s *RiskPrioritizedDBInputSource) Next(ctx context.Context) (*work.WorkItem, error)

Next streams records high-risk-first then chronologically, via bounded keyset pages, so memory scales with the page size (not the whole eligible table).

func (*RiskPrioritizedDBInputSource) WithHostScopes ¶

func (s *RiskPrioritizedDBInputSource) WithHostScopes(hosts []HostTarget) *RiskPrioritizedDBInputSource

WithHostScopes sets an in-scope origin filter so only records matching these (scheme, hostname, port) origins are returned.

func (*RiskPrioritizedDBInputSource) WithParamShapeCoalescing ¶

func (s *RiskPrioritizedDBInputSource) WithParamShapeCoalescing(maxSamples int) *RiskPrioritizedDBInputSource

WithParamShapeCoalescing enables param-shape coalescing of the snapshot: GET records that share a (host, path, query-param-name-set) are reduced to at most maxSamples value-distinct representatives, cutting redundant dynamic-assessment fan-out over value-only-different URLs (e.g. /search?q=1..N). Records stay in the database; only this scan's iteration list is pruned. The scan cursor still advances past every record, so coalesced-away records are not re-scanned in a later feedback round. maxSamples <= 0 (the default) disables it.

func (*ScanLogger) Close ¶

func (l *ScanLogger) Close()

Close stops the batcher goroutine and flushes any remaining entries.

func (*ScanLogger) Error ¶

func (l *ScanLogger) Error(phase, message string)

Error logs an error message.

func (*ScanLogger) Flush ¶

func (l *ScanLogger) Flush()

Flush writes all buffered trace entries to the database.

func (*ScanLogger) Info ¶

func (l *ScanLogger) Info(phase, message string)

Info logs an informational message.

func (*ScanLogger) InfoWithMeta ¶

func (l *ScanLogger) InfoWithMeta(phase, message string, meta map[string]interface{})

InfoWithMeta logs an informational message with structured metadata.

func (*ScanLogger) StartBatcher ¶

func (l *ScanLogger) StartBatcher()

StartBatcher launches a background goroutine that periodically flushes buffered trace entries to the database. Call Close() to stop and flush.

func (*ScanLogger) Trace ¶

func (l *ScanLogger) Trace(phase, message string)

Trace logs a trace-level message (raw console output). These entries are buffered and flushed in batches to avoid per-line DB inserts.

func (*ScanLogger) Warn ¶

func (l *ScanLogger) Warn(phase, message string)

Warn logs a warning message.

func (*ScopeEvaluator) IsInScope ¶

func (e *ScopeEvaluator) IsInScope(scheme, hostname string, port int, method, path string) bool

IsInScope checks if a URL matches the scope rules. Iterates rules by priority. First matching rule wins. If no rule matches, returns false (default=exclude).

func (*StoredHostFacts) Facts ¶

func (s *StoredHostFacts) Facts(hostname string, port int) HostFacts

Facts returns what was observed about one endpoint.

func (*StoredHostFacts) FactsMap ¶

func (s *StoredHostFacts) FactsMap(hostname string, port int) map[string]any

FactsMap is Facts rendered as JSON keys for the map-shaped agent views, memoized alongside the facts so the render cost is per endpoint, not per record.

func (*StoredHostFacts) MissingCount ¶

func (s *StoredHostFacts) MissingCount() int

MissingCount is how many distinct endpoints this pass could describe from neither the stored observations nor the process caches.

func (*StoredHostFacts) Stored ¶

func (s *StoredHostFacts) Stored() int

Stored reports how many distinct endpoints this pass could describe.

func (*UUIDListDBInputSource) Close ¶

func (s *UUIDListDBInputSource) Close() error

Close stops the source. After Close, Next will return io.EOF.

func (*UUIDListDBInputSource) Next ¶

func (s *UUIDListDBInputSource) Next(ctx context.Context) (*work.WorkItem, error)

Next returns the next record from the UUID list as a WorkItem. Skips invalid or missing UUIDs. Returns io.EOF when all UUIDs have been processed.