importer

package
v1.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 12, 2026 License: Apache-2.0 Imports: 22 Imported by: 6

Documentation

Overview

Package importer converts the files other security tools export into CTIS 1.4 reports: Nessus v2 XML, Qualys host detection XML (joined with the Qualys KnowledgeBase), CycloneDX and SPDX SBOMs, OSV records, CSAF 2.0 and OpenVEX documents, DefectDojo Generic Findings JSON, SARIF 2.1.0, and the native JSON of trivy, nuclei, semgrep, betterleaks and vuls. It is the one conversion entry point: a receiver's upload and CI endpoints and a sensor's parser tool all call Parse.

A code report (SARIF, semgrep, betterleaks, a trivy file-system scan) is filed on Options.Repository when it is set; a receiver that knows the repository from a verified identity sets it, so a file can never move its findings onto another asset.

Detect names the format of a file from its first bytes; Parse converts it. A Result carries the CTIS report, the VEX statements of a VEX document (which describe products, not findings, and so are not findings of the report), counts, problems with their line numbers, and the source fields the file held that the format's mapping spec does not know.

Mapping discipline

Every format has a mapping spec (Specs): one row per source field, saying which CTIS member holds it, or why it is deliberately not kept. The package tests walk every fixture and fail when a fixture holds a field the spec does not list, so an importer can never drop a field silently. The spec tables are rendered to docs/importers/, with the coverage of each format, and a test keeps the documents in step with the code.

The native value always travels next to the normalized one: the source's severity, status, ids and scores go to finding.native and finding.scores, and fields without a CTIS member go to finding.source_extra (bounded).

Hostile input

Every file is treated as hostile. The limits (Limits) bound the input size, nesting depth, element and attribute counts, text sizes and the number of findings, assets and statements. XML is read with document type declarations limited to an external identifier (an internal subset, and so any entity declaration, is refused, and no external resource is ever read), only the predefined entities, and only UTF-8, US-ASCII or ISO-8859-1 input. JSON must be valid UTF-8 and is depth-checked before it is decoded. Archives (OpenZip) refuse path traversal, absolute paths, links, nested archives and decompression bombs.

Credentials and scan operator details are never copied into a report: the Nessus and Qualys mapping specs list the policy, preference and user fields as ignored, and the parsers skip them.

Index

Constants

View Source
const Container = "(container)"

Container is the Target of a field that only groups other fields.

View Source
const SniffLen = 64 << 10

SniffLen is how many leading bytes Detect needs at most.

Variables

View Source
var (
	// ErrUnknownFormat: Detect does not recognize the input, or the format
	// named is not one this package reads.
	ErrUnknownFormat = errors.New("unknown import format")

	// ErrTooLarge: the input or one of its parts is over a limit.
	ErrTooLarge = errors.New("import input over a limit")

	// ErrUnsafe: the input uses a construct this package never reads (an
	// XML internal subset or entity declaration, an unsupported charset,
	// a link or traversal in an archive).
	ErrUnsafe = errors.New("import input uses a refused construct")

	// ErrMalformed: the input is not well-formed for its format.
	ErrMalformed = errors.New("malformed import input")
)

Errors a caller can test with errors.Is.

Functions

func IsZip

func IsZip(head []byte) bool

IsZip reports whether head starts like a ZIP archive.

Types

type ArchiveFile

type ArchiveFile struct {
	// The file's path inside the archive, cleaned. It is only a label: it is
	// never used to create a file.
	Name string
	// Uncompressed size as the archive declares it (the reader enforces the
	// limits on the real size).
	DeclaredSize int64
	// contains filtered or unexported fields
}

ArchiveFile is one file of an archive.

func OpenZip

func OpenZip(r io.ReaderAt, size int64, lim ArchiveLimits) ([]ArchiveFile, error)

OpenZip lists the regular files of a ZIP archive. It refuses the whole archive when an entry has an absolute path, a ".." element, a backslash, a drive letter or a control character in its name, is a link or another non-regular file, is encrypted, or is itself an archive, and when there are more than MaxEntries files. Nothing is ever written to disk.

func (ArchiveFile) Open

func (a ArchiveFile) Open() (io.ReadCloser, error)

Open returns the file's content. The reader fails with ErrTooLarge as soon as the file passes MaxEntryBytes, the archive passes MaxTotalBytes, or the file decompresses at more than MaxRatio.

type ArchiveLimits

type ArchiveLimits struct {
	// Most files in the archive (directories are not counted).
	MaxEntries int
	// Largest uncompressed file.
	MaxEntryBytes int64
	// Largest sum of the uncompressed files.
	MaxTotalBytes int64
	// Largest ratio of uncompressed to compressed size of one file, checked
	// as the file is read, once it is past 1 MiB.
	MaxRatio int64
}

ArchiveLimits bound what OpenZip accepts. Zero fields take the defaults of DefaultArchiveLimits.

func DefaultArchiveLimits

func DefaultArchiveLimits() ArchiveLimits

DefaultArchiveLimits returns the limits a zero ArchiveLimits stands for.

type Coverage

type Coverage struct {
	Mapped  int
	Ignored int
}

Coverage is the share of a spec's fields that are kept.

func (Coverage) Percent

func (c Coverage) Percent() float64

Percent is Mapped as a share of Mapped plus Ignored.

type Field

type Field struct {
	Path    string
	Target  string
	Ignored string
}

Field is one row of a mapping spec.

Path is the field's path in the source: XML as "/Root/Child/@attr" (elements keyed by an attribute as "tag[name]"), JSON as "/member/array[]/ member". A "*" matches any run of characters inside one path segment; a "**" segment matches zero or more segments (a trailing "/**" matches everything below).

Exactly one of Target and Ignored is set: Target names the CTIS member or members the value goes to; Ignored says why the value is deliberately not kept. A container element whose children are listed has Target "(container)".

func (Field) Mapped

func (f Field) Mapped() bool

Mapped reports whether the field's value is kept.

type Format

type Format string

Format names a file format this package reads.

const (
	// Nessus v2 XML (.nessus), written by Nessus and Tenable.sc.
	FormatNessus Format = "nessus"
	// Qualys VM host list detection XML (HOST_LIST_VM_DETECTION_OUTPUT).
	FormatQualys Format = "qualys"
	// Qualys KnowledgeBase XML (KNOWLEDGE_BASE_VULN_LIST_OUTPUT). Read only
	// as the companion of a detection file (Options.QualysKnowledgeBase).
	FormatQualysKB Format = "qualys_kb"
	// CycloneDX 1.4 to 1.6 JSON: an SBOM, a VDR or a VEX.
	FormatCycloneDX Format = "cyclonedx"
	// SPDX 2.2 / 2.3 JSON.
	FormatSPDX Format = "spdx"
	// osv-scanner JSON results (packages with their OSV records).
	FormatOSV Format = "osv"
	// CSAF 2.0 JSON (the VEX profile and the security advisory profile).
	FormatCSAF Format = "csaf"
	// OpenVEX JSON.
	FormatOpenVEX Format = "openvex"
	// DefectDojo Generic Findings Import JSON.
	FormatDefectDojo Format = "defectdojo"
	// SARIF 2.1.0 (any static analysis tool that writes SARIF).
	FormatSARIF Format = "sarif"
	// trivy JSON (image, file-system, repository and config scans).
	FormatTrivy Format = "trivy"
	// nuclei JSON lines (-jsonl) or a JSON array (-json-export).
	FormatNuclei Format = "nuclei"
	// semgrep JSON (--json).
	FormatSemgrep Format = "semgrep"
	// betterleaks JSON report (the gitleaks-compatible array).
	FormatBetterleaks Format = "betterleaks"
	// vuls JSON scan result (one server).
	FormatVuls Format = "vuls"
	// gitleaks JSON report (an array of leaks). Every gitleaks-compatible
	// scanner writes the same shape; Options.ToolName names the tool.
	FormatGitleaks Format = "gitleaks"
	// grype JSON output.
	FormatGrype Format = "grype"
	// ZAP traditional JSON or XML report.
	FormatZAP Format = "zap"
)

func AllFormats

func AllFormats() []Format

AllFormats returns every format Parse reads, sorted.

func Detect

func Detect(head []byte) (Format, bool)

Detect names the format of an input from its leading bytes (at most SniffLen are read). It recognizes the formats of AllFormats and FormatQualysKB. It never errors on hostile input; it returns false.

func (Format) IsValid

func (f Format) IsValid() bool

IsValid reports whether f is a format Parse reads.

type Issue

type Issue struct {
	// 1-based line and column of the place in the input, when known.
	Line   int `json:"line,omitempty"`
	Column int `json:"column,omitempty"`

	// The record the problem is in: an XML element path or a JSON pointer.
	Path string `json:"path,omitempty"`

	Message string `json:"message"`
}

Issue is a problem at a place in the input.

func (Issue) String

func (i Issue) String() string

type Limits

type Limits struct {
	// Largest input, in bytes.
	MaxInputBytes int64

	// Deepest nesting of XML elements or JSON objects and arrays.
	MaxDepth int

	// Most XML elements in one document.
	MaxElements int

	// Most attributes on one XML element.
	MaxAttributes int

	// Largest text of one XML element or one JSON string, in bytes. Longer
	// values are refused, not truncated, so a file that needs them is seen
	// as what it is.
	MaxTextBytes int

	// Most findings, assets, dependencies and VEX statements in one result.
	MaxFindings   int
	MaxAssets     int
	MaxComponents int
	MaxStatements int

	// Most distinct source field paths recorded for the coverage check.
	MaxObservedPaths int

	// Most problems reported in Result.Issues.
	MaxIssues int
}

Limits bound what a parser accepts. A zero field takes its default (DefaultLimits). The defaults are sized for a real export of a large estate; a receiver that accepts uploads from people should pass smaller values.

func DefaultLimits

func DefaultLimits() Limits

DefaultLimits returns the limits a zero Limits stands for.

type Options

type Options struct {
	// Format of the input. Empty: Detect decides from the first bytes.
	Format Format

	// Limits on the input; zero fields take DefaultLimits.
	Limits Limits

	// Report timestamp. Zero: the time of the parse (UTC).
	Now time.Time

	// metadata.id of the report (an import or scan session id).
	ReportID string

	// metadata.source_type of the report. Empty: "manual" (a file someone
	// imports).
	SourceType string

	// tool.name of the report. Empty: the format's tool (nessus, qualys,
	// cyclonedx, ...).
	ToolName string

	// Findings below this severity are counted in Stats.Skipped and left
	// out. Empty keeps every finding.
	MinSeverity ctis.Severity

	// The asset of records that name none (a DefectDojo finding without an
	// endpoint). Nil: an unclassified asset named after the tool.
	DefaultAsset *ctis.Asset

	// The repository a code report (SARIF, semgrep, betterleaks, a trivy
	// file-system scan) is filed on, with its branch and commit. It wins
	// over what the file names (a SARIF versionControlProvenance, a trivy
	// artifact). A receiver that knows the repository from a verified
	// identity (a CI workload token) sets it from there and never from the
	// file; the importer only records the file's claim.
	Repository string
	Branch     string
	CommitSHA  string

	// SARIF only. ToolType is the kind of tool that wrote the log: "sast",
	// "sca", "secret", "iac" or "web3". It decides the type of every finding
	// and the tool's capabilities instead of the rule tags and the tool name
	// (a scanner runtime that knows what it ran sets it). Other values are
	// ignored. DefaultConfidence (1 to 100) is the confidence of a result
	// whose rule states no precision; zero or out of range: 90.
	ToolType          string
	DefaultConfidence int

	// The Qualys KnowledgeBase XML for a Qualys detection file. Optional:
	// without it, detections keep their QID and results but have no title
	// beyond the QID, CVEs or CVSS.
	QualysKnowledgeBase io.Reader
}

Options configure Parse. The zero value detects the format and uses the default limits.

type ParseError

type ParseError struct {
	Format Format
	Line   int
	Column int
	Msg    string
	Err    error
}

ParseError is the error of an input that could not be read. Line and Column point into the input when known.

func (*ParseError) Error

func (e *ParseError) Error() string

func (*ParseError) Unwrap

func (e *ParseError) Unwrap() error

type Product

type Product struct {
	// Package URL.
	PURL string `json:"purl,omitempty"`

	// CPE 2.3 name.
	CPE string `json:"cpe,omitempty"`

	// Name and version as the document states them, when it gives no
	// identifier.
	Name    string `json:"name,omitempty"`
	Version string `json:"version,omitempty"`

	// When set, the statement is about these components inside the
	// product only (an OpenVEX subcomponent, a CSAF relationship's
	// component), not about the product as a whole or about the components
	// anywhere else. A receiver matches a finding on a subcomponent's PURL
	// or CPE only when the finding's asset is the product.
	Subcomponents []Product `json:"subcomponents,omitempty"`
}

Product is a product of a VEX statement.

type Result

type Result struct {
	// Format of the file.
	Format Format `json:"format"`

	// The CTIS report: assets, findings and dependencies. Never nil; a VEX
	// document gives a report without findings.
	Report *ctis.Report `json:"report"`

	// The statements of a VEX document (CSAF VEX, OpenVEX, the analysis
	// members of a CycloneDX VEX). A receiver applies them to its own
	// findings that match a statement's vulnerability and product.
	VEX []VEXStatement `json:"vex,omitempty"`

	// Counts of what the file held and what was kept.
	Stats Stats `json:"stats"`

	// Problems that did not stop the import (an item without a required
	// field, a value out of range), at most Limits.MaxIssues.
	Issues []Issue `json:"issues,omitempty"`

	// Source field paths the file held that the format's mapping spec does
	// not list, sorted. Their values are not in the report; a receiver can
	// show them so the spec is extended.
	Unmapped []string `json:"unmapped,omitempty"`
}

Result is what Parse returns for one file.

func Parse

func Parse(ctx context.Context, r io.Reader, opts Options) (*Result, error)

Parse reads one exported file and converts it to CTIS.

It returns a *ParseError (wrapping ErrUnknownFormat, ErrTooLarge, ErrUnsafe or ErrMalformed) when the input cannot be read at all. Problems with single records do not fail the parse; they are in Result.Issues and the record is skipped.

type Spec

type Spec struct {
	Format Format
	// Human name of the source format and the versions the spec was written
	// against.
	Title         string
	SourceVersion string
	// Notes on normalization rules that are not per field (severity tables,
	// status tables, asset identity, deduplication).
	Rules []string
	// The fields, in document order.
	Fields []Field
}

Spec is the mapping spec of one format: every source field the importer knows, and what becomes of it.

func SpecFor

func SpecFor(f Format) (Spec, bool)

SpecFor returns the mapping spec of a format.

func Specs

func Specs() []Spec

Specs returns the mapping spec of every format Parse reads (plus the Qualys KnowledgeBase), sorted by format.

func (Spec) Coverage

func (s Spec) Coverage() Coverage

Coverage counts the spec's mapped and ignored fields (containers are not counted).

func (Spec) Markdown

func (s Spec) Markdown() string

Markdown renders the spec as a document.

func (Spec) Match

func (s Spec) Match(p string) (Field, bool)

Match returns the first field whose path matches p.

func (Spec) Unmapped

func (s Spec) Unmapped(observed []string) []string

Unmapped returns the paths of observed that no field of s matches.

type Stats

type Stats struct {
	// Records of the source: Nessus ReportItems, Qualys detections, CycloneDX
	// vulnerabilities, OSV affected packages, DefectDojo findings.
	Records int `json:"records"`

	Assets     int `json:"assets"`
	Findings   int `json:"findings"`
	Components int `json:"components"`
	Statements int `json:"statements"`

	// Records skipped (below the minimum severity, or unusable; the reason
	// of an unusable record is in Issues).
	Skipped int `json:"skipped"`

	// Findings per CTIS severity.
	BySeverity map[ctis.Severity]int `json:"by_severity,omitempty"`
}

Stats counts what an import read and produced.

type VEXStatement

type VEXStatement struct {
	// The vulnerability, under every id the document gives it.
	VulnerabilityIDs []ctis.VulnerabilityID `json:"vulnerability_ids"`

	// The products the statement is about. A receiver matches a finding
	// on a shared PURL (version-less when the statement's PURL has no
	// version) or CPE.
	Products []Product `json:"products"`

	// The statement.
	VEX ctis.VEX `json:"vex"`
}

VEXStatement is one exploitability statement of a VEX document: the status of the products below for the vulnerability below.

Directories

Path Synopsis
Package mapping turns the JSON or JSON Lines output of any command-line tool into a CTIS report through a declarative mapping file, so a tool that writes JSON needs no parser code.
Package mapping turns the JSON or JSON Lines output of any command-line tool into a CTIS report through a declarative mapping file, so a tool that writes JSON needs no parser code.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL