dbimport

package
v0.5.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 3, 2026 License: MIT Imports: 18 Imported by: 0

Documentation

Overview

Package dbimport provides reusable importers for audit folders, archive bundles (.tar.gz, .tgz, .zip), and JSONL exports. Both the CLI (`vigolium import`) and the REST API (`POST /api/import`) call into this package so the parsing, validation, and DB-write logic stays in one place.

Index

Constants

View Source
const DefaultMaxLineBytes = 256 << 20

DefaultMaxLineBytes is the ceiling ImportJSONL applies to one line when Options.MaxLineBytes is unset. 256 MiB is far above any real exported record (the largest observed response body is ~2 orders of magnitude smaller) and far below the point where the allocation itself is the failure.

Variables

This section is empty.

Functions

func ArchiveExt

func ArchiveExt(path string) string

ArchiveExt inspects a path and returns one of: ".tar.gz", ".tgz", ".zip", ".jsonl", ".ndjson", ".json", or "" (unknown). Lowercased, includes leading dot.

func CopyDirContents

func CopyDirContents(src, dst string) error

CopyDirContents recursively copies the contents of src into dst, creating dst if needed. Symlinks are skipped; only regular files and dirs are copied. File modes are preserved best-effort.

func ExtractArchiveToDir

func ExtractArchiveToDir(archivePath string) (dir string, cleanup func(), err error)

ExtractArchiveToDir extracts a .tar.gz / .tgz / .zip into a fresh temp dir and returns its path. The caller must defer cleanup().

Types

type Options

type Options struct {
	// AgenticScanUUID, if non-empty, attaches imported findings (and HTTP
	// records for JSONL) to an existing agentic_scan row instead of creating a
	// new one. For audit imports the existing row's metadata is preserved and
	// only finding counts/storage_url are touched.
	AgenticScanUUID string

	// OriginalSource carries the user-supplied input string (e.g. gs:// URL or
	// archive path). Recorded as StorageURL on audit scan rows when it has a
	// gs:// prefix.
	OriginalSource string

	// SessionDirArchiver, if non-nil, is invoked after the agentic scan UUID is
	// determined. It receives that UUID and the on-disk audit source dir,
	// and returns the absolute session directory where the source was copied.
	// Used by audit imports only.
	SessionDirArchiver func(scanUUID, srcDir string) (sessionDir string, err error)

	// Source identifies the audit harness flavor (audit vs piolium). When
	// zero-valued, audit.DefaultSource() is used.
	Source *audit.FindingSource

	// SkipHTTPRecords omits the http_records table from a SQLite→SQLite merge;
	// see database.MergeOptions. Ignored by the JSONL/audit/archive importers,
	// which don't carry a separate record table to skip.
	SkipHTTPRecords bool

	// SkipRecordBodies omits the raw_request/raw_response columns from a
	// SQLite→SQLite merge of http_records; see database.MergeOptions. Ignored by
	// the JSONL/audit/archive importers.
	SkipRecordBodies bool

	// SkipFindings omits findings, the finding_records junction, and
	// oast_interactions from a SQLite→SQLite merge; see database.MergeOptions.
	// Ignored by the JSONL/audit/archive importers, whose whole payload is
	// findings.
	SkipFindings bool

	// PreserveProjectUUID keeps each JSONL row's own project_uuid instead of
	// re-homing it onto the projectUUID argument. A row with no project_uuid of
	// its own still gets the argument.
	//
	// It exists for the read-only scratch databases: a `-S --db export.jsonl`
	// load and a --glob-db merge exist to show a file's contents as they are, and
	// stamping the default project over every row made `project_uuid` in the
	// output a statement about the reader rather than about the data — which also
	// made an explicit project filter over such a source meaningless.
	//
	// Persistent `vigolium import` deliberately does NOT set it: importing into a
	// project means the rows join that project.
	PreserveProjectUUID bool

	// MaxLineBytes caps a single JSONL line. 0 means DefaultMaxLineBytes.
	//
	// The reader grows its buffer to whatever a line needs, which is required —
	// an exported http_record carries its whole response body on one line. With
	// no ceiling at all, though, a truncated or non-JSONL file (a tarball, a core
	// dump, a file whose newlines were stripped) is read into memory in full
	// before the first parse error can be reported, so the symptom of pointing
	// `import` at the wrong file is the OOM killer rather than a message.
	MaxLineBytes int
}

Options carries optional knobs that apply to both audit and JSONL imports.

type Result

type Result struct {
	AgenticScan *database.AgenticScan
	CreatedNew  bool

	RecordsImported int
	FindingsTotal   int
	FindingsSaved   int
	FindingsSkipped int
	ParseErrors     int

	// RecordsSkippedDuplicate counts HTTP records whose uuid was already in the
	// destination, so they were not inserted. Reported separately from
	// RecordsImported because "0 imported" out of a 10,000-line export means
	// something very different from "0 imported, 10,000 already present".
	RecordsSkippedDuplicate int

	// FindingsFailed counts findings the destination refused to store. Distinct
	// from FindingsSkipped, which is the benign dedup-append case: a failed
	// finding is data that was in the source and is NOT in the destination, and
	// folding the two together reported a lossy import as a clean one.
	FindingsFailed int

	SeverityCounts map[string]int
	SkippedTypes   map[string]int

	SessionDir string
	StorageURL string

	// MergeStats is populated only for SQLite-database imports (ImportSQLite):
	// a lossless SQLite→SQLite merge of another vigolium result database into
	// the destination. Nil for audit and JSONL imports.
	MergeStats *database.MergeStats
}

Result captures what was imported during a single ImportXxx call.

AgenticScan is populated for audit imports (whether a new scan was created or an existing one was attached to); for JSONL imports it is populated only when Options.AgenticScanUUID was supplied so the caller can correlate imported findings with an existing scan row. CreatedNew distinguishes a freshly-created agentic scan from an attach.

func ImportArchive

func ImportArchive(ctx context.Context, repo *database.Repository, archivePath, projectUUID string, opts Options) (*Result, error)

ImportArchive extracts a .tar.gz / .tgz / .zip and dispatches its contents to the audit or JSONL importers. Results across nested imports are merged.

func ImportAudit

func ImportAudit(ctx context.Context, repo *database.Repository, folderPath, projectUUID string, opts Options) (*Result, error)

ImportAudit imports an audit output folder. When opts.AgenticScanUUID is set, the existing scan row is loaded (and project-validated by the caller) and findings are attached to it instead of a new row being created.

func ImportJSONL

func ImportJSONL(ctx context.Context, repo *database.Repository, r io.Reader, projectUUID string, opts Options) (*Result, error)

ImportJSONL imports HTTP records and findings from a JSONL stream of envelopes ({"type": "...", "data": {...}}). When opts.AgenticScanUUID is supplied, findings are tagged with that UUID; HTTP records are not tagged (the schema does not carry an agentic_scan_uuid on records).

func ImportPath

func ImportPath(ctx context.Context, repo *database.Repository, path, projectUUID string, opts Options) (*Result, error)

ImportPath dispatches based on filesystem inspection of path: directory → audit folder, .tar.gz/.tgz/.zip → archive, a vigolium SQLite result database → SQLite merge, anything else → JSONL.

func ImportSQLite

func ImportSQLite(ctx context.Context, repo *database.Repository, srcPath, projectUUID string, opts Options) (*Result, error)

ImportSQLite merges another vigolium SQLite result database at srcPath into the database behind repo. It is a lossless, idempotent SQLite→SQLite merge of the scan-result tables (http_records, findings, finding_records, scans, agentic_scans, oast_interactions, projects), deduping rows on their natural keys — so importing the same database twice adds nothing the second time.

Each row keeps its original project_uuid, so imported data stays scoped to whatever project it was scanned under; the projectUUID argument (the caller's active project) is intentionally not applied here. opts is likewise not applied: AgenticScanUUID (attach) and OriginalSource (storage-URL recording) are single-scan/audit concepts that don't map onto a bulk merge of a whole database of scans — findings keep their own agentic_scan_uuid from the source. Requires a SQLite destination — a Postgres destination returns a clear error from database.MergeSQLiteFile. The destination schema must already exist (callers run CreateSchema before ImportPath).

func MergeResults

func MergeResults(results []*Result) *Result

MergeResults folds several import Results into one aggregate: counts, the severity/skipped maps and MergeStats are summed, while the last non-nil AgenticScan and session/storage references win (see mergeResult). nil entries are ignored. Used to summarize a multi-source import as a single Result.

func (*Result) AgenticScanUUID

func (r *Result) AgenticScanUUID() string

AgenticScanUUID returns the UUID of the result's agentic scan, or "" if no scan was created or attached (the JSONL-without-attach case).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL