Documentation
¶
Overview ¶
Package dbimport provides reusable importers for audit folders, archive bundles (.tar.gz, .tgz, .zip), and JSONL exports. Both the CLI (`vigolium import`) and the REST API (`POST /api/import`) call into this package so the parsing, validation, and DB-write logic stays in one place.
Index ¶
- Constants
- func ArchiveExt(path string) string
- func CopyDirContents(src, dst string) error
- func ExtractArchiveToDir(archivePath string) (dir string, cleanup func(), err error)
- type Options
- type Result
- func ImportArchive(ctx context.Context, repo *database.Repository, ...) (*Result, error)
- func ImportAudit(ctx context.Context, repo *database.Repository, folderPath, projectUUID string, ...) (*Result, error)
- func ImportJSONL(ctx context.Context, repo *database.Repository, r io.Reader, ...) (*Result, error)
- func ImportPath(ctx context.Context, repo *database.Repository, path, projectUUID string, ...) (*Result, error)
- func ImportSQLite(ctx context.Context, repo *database.Repository, srcPath, projectUUID string, ...) (*Result, error)
- func MergeResults(results []*Result) *Result
Constants ¶
const DefaultMaxLineBytes = 256 << 20
DefaultMaxLineBytes is the ceiling ImportJSONL applies to one line when Options.MaxLineBytes is unset. 256 MiB is far above any real exported record (the largest observed response body is ~2 orders of magnitude smaller) and far below the point where the allocation itself is the failure.
Variables ¶
This section is empty.
Functions ¶
func ArchiveExt ¶
ArchiveExt inspects a path and returns one of: ".tar.gz", ".tgz", ".zip", ".jsonl", ".ndjson", ".json", or "" (unknown). Lowercased, includes leading dot.
func CopyDirContents ¶
CopyDirContents recursively copies the contents of src into dst, creating dst if needed. Symlinks are skipped; only regular files and dirs are copied. File modes are preserved best-effort.
func ExtractArchiveToDir ¶
ExtractArchiveToDir extracts a .tar.gz / .tgz / .zip into a fresh temp dir and returns its path. The caller must defer cleanup().
Types ¶
type Options ¶
type Options struct {
// AgenticScanUUID, if non-empty, attaches imported findings (and HTTP
// records for JSONL) to an existing agentic_scan row instead of creating a
// new one. For audit imports the existing row's metadata is preserved and
// only finding counts/storage_url are touched.
AgenticScanUUID string
// OriginalSource carries the user-supplied input string (e.g. gs:// URL or
// archive path). Recorded as StorageURL on audit scan rows when it has a
// gs:// prefix.
OriginalSource string
// SessionDirArchiver, if non-nil, is invoked after the agentic scan UUID is
// determined. It receives that UUID and the on-disk audit source dir,
// and returns the absolute session directory where the source was copied.
// Used by audit imports only.
SessionDirArchiver func(scanUUID, srcDir string) (sessionDir string, err error)
// Source identifies the audit harness flavor (audit vs piolium). When
// zero-valued, audit.DefaultSource() is used.
Source *audit.FindingSource
// SkipHTTPRecords omits the http_records table from a SQLite→SQLite merge;
// see database.MergeOptions. Ignored by the JSONL/audit/archive importers,
// which don't carry a separate record table to skip.
SkipHTTPRecords bool
// SkipRecordBodies omits the raw_request/raw_response columns from a
// SQLite→SQLite merge of http_records; see database.MergeOptions. Ignored by
// the JSONL/audit/archive importers.
SkipRecordBodies bool
// SkipFindings omits findings, the finding_records junction, and
// oast_interactions from a SQLite→SQLite merge; see database.MergeOptions.
// Ignored by the JSONL/audit/archive importers, whose whole payload is
// findings.
SkipFindings bool
// PreserveProjectUUID keeps each JSONL row's own project_uuid instead of
// re-homing it onto the projectUUID argument. A row with no project_uuid of
// its own still gets the argument.
//
// It exists for the read-only scratch databases: a `-S --db export.jsonl`
// load and a --glob-db merge exist to show a file's contents as they are, and
// stamping the default project over every row made `project_uuid` in the
// output a statement about the reader rather than about the data — which also
// made an explicit project filter over such a source meaningless.
//
// Persistent `vigolium import` deliberately does NOT set it: importing into a
// project means the rows join that project.
PreserveProjectUUID bool
// MaxLineBytes caps a single JSONL line. 0 means DefaultMaxLineBytes.
//
// The reader grows its buffer to whatever a line needs, which is required —
// an exported http_record carries its whole response body on one line. With
// no ceiling at all, though, a truncated or non-JSONL file (a tarball, a core
// dump, a file whose newlines were stripped) is read into memory in full
// before the first parse error can be reported, so the symptom of pointing
// `import` at the wrong file is the OOM killer rather than a message.
MaxLineBytes int
}
Options carries optional knobs that apply to both audit and JSONL imports.
type Result ¶
type Result struct {
AgenticScan *database.AgenticScan
CreatedNew bool
RecordsImported int
FindingsTotal int
FindingsSaved int
FindingsSkipped int
ParseErrors int
// RecordsSkippedDuplicate counts HTTP records whose uuid was already in the
// destination, so they were not inserted. Reported separately from
// RecordsImported because "0 imported" out of a 10,000-line export means
// something very different from "0 imported, 10,000 already present".
RecordsSkippedDuplicate int
// FindingsFailed counts findings the destination refused to store. Distinct
// from FindingsSkipped, which is the benign dedup-append case: a failed
// finding is data that was in the source and is NOT in the destination, and
// folding the two together reported a lossy import as a clean one.
FindingsFailed int
SeverityCounts map[string]int
SkippedTypes map[string]int
SessionDir string
StorageURL string
// MergeStats is populated only for SQLite-database imports (ImportSQLite):
// a lossless SQLite→SQLite merge of another vigolium result database into
// the destination. Nil for audit and JSONL imports.
MergeStats *database.MergeStats
}
Result captures what was imported during a single ImportXxx call.
AgenticScan is populated for audit imports (whether a new scan was created or an existing one was attached to); for JSONL imports it is populated only when Options.AgenticScanUUID was supplied so the caller can correlate imported findings with an existing scan row. CreatedNew distinguishes a freshly-created agentic scan from an attach.
func ImportArchive ¶
func ImportArchive(ctx context.Context, repo *database.Repository, archivePath, projectUUID string, opts Options) (*Result, error)
ImportArchive extracts a .tar.gz / .tgz / .zip and dispatches its contents to the audit or JSONL importers. Results across nested imports are merged.
func ImportAudit ¶
func ImportAudit(ctx context.Context, repo *database.Repository, folderPath, projectUUID string, opts Options) (*Result, error)
ImportAudit imports an audit output folder. When opts.AgenticScanUUID is set, the existing scan row is loaded (and project-validated by the caller) and findings are attached to it instead of a new row being created.
func ImportJSONL ¶
func ImportJSONL(ctx context.Context, repo *database.Repository, r io.Reader, projectUUID string, opts Options) (*Result, error)
ImportJSONL imports HTTP records and findings from a JSONL stream of envelopes ({"type": "...", "data": {...}}). When opts.AgenticScanUUID is supplied, findings are tagged with that UUID; HTTP records are not tagged (the schema does not carry an agentic_scan_uuid on records).
func ImportPath ¶
func ImportPath(ctx context.Context, repo *database.Repository, path, projectUUID string, opts Options) (*Result, error)
ImportPath dispatches based on filesystem inspection of path: directory → audit folder, .tar.gz/.tgz/.zip → archive, a vigolium SQLite result database → SQLite merge, anything else → JSONL.
func ImportSQLite ¶
func ImportSQLite(ctx context.Context, repo *database.Repository, srcPath, projectUUID string, opts Options) (*Result, error)
ImportSQLite merges another vigolium SQLite result database at srcPath into the database behind repo. It is a lossless, idempotent SQLite→SQLite merge of the scan-result tables (http_records, findings, finding_records, scans, agentic_scans, oast_interactions, projects), deduping rows on their natural keys — so importing the same database twice adds nothing the second time.
Each row keeps its original project_uuid, so imported data stays scoped to whatever project it was scanned under; the projectUUID argument (the caller's active project) is intentionally not applied here. opts is likewise not applied: AgenticScanUUID (attach) and OriginalSource (storage-URL recording) are single-scan/audit concepts that don't map onto a bulk merge of a whole database of scans — findings keep their own agentic_scan_uuid from the source. Requires a SQLite destination — a Postgres destination returns a clear error from database.MergeSQLiteFile. The destination schema must already exist (callers run CreateSchema before ImportPath).
func MergeResults ¶
MergeResults folds several import Results into one aggregate: counts, the severity/skipped maps and MergeStats are summed, while the last non-nil AgenticScan and session/storage references win (see mergeResult). nil entries are ignored. Used to summarize a multi-source import as a single Result.
func (*Result) AgenticScanUUID ¶
AgenticScanUUID returns the UUID of the result's agentic scan, or "" if no scan was created or attached (the JSONL-without-attach case).