Documentation
¶
Overview ¶
Package importer converts the files other security tools export into CTIS 1.4 reports: Nessus v2 XML, Qualys host detection XML (joined with the Qualys KnowledgeBase), CycloneDX and SPDX SBOMs, OSV records, CSAF 2.0 and OpenVEX documents, DefectDojo Generic Findings JSON, SARIF 2.1.0, and the native JSON of trivy, nuclei, semgrep, betterleaks and vuls. It is the one conversion entry point: a receiver's upload and CI endpoints and a sensor's parser tool all call Parse.
A code report (SARIF, semgrep, betterleaks, a trivy file-system scan) is filed on Options.Repository when it is set; a receiver that knows the repository from a verified identity sets it, so a file can never move its findings onto another asset.
Detect names the format of a file from its first bytes; Parse converts it. A Result carries the CTIS report, the VEX statements of a VEX document (which describe products, not findings, and so are not findings of the report), counts, problems with their line numbers, and the source fields the file held that the format's mapping spec does not know.
Mapping discipline ¶
Every format has a mapping spec (Specs): one row per source field, saying which CTIS member holds it, or why it is deliberately not kept. The package tests walk every fixture and fail when a fixture holds a field the spec does not list, so an importer can never drop a field silently. The spec tables are rendered to docs/importers/, with the coverage of each format, and a test keeps the documents in step with the code.
The native value always travels next to the normalized one: the source's severity, status, ids and scores go to finding.native and finding.scores, and fields without a CTIS member go to finding.source_extra (bounded).
Hostile input ¶
Every file is treated as hostile. The limits (Limits) bound the input size, nesting depth, element and attribute counts, text sizes and the number of findings, assets and statements. XML is read with document type declarations limited to an external identifier (an internal subset, and so any entity declaration, is refused, and no external resource is ever read), only the predefined entities, and only UTF-8, US-ASCII or ISO-8859-1 input. JSON must be valid UTF-8 and is depth-checked before it is decoded. Archives (OpenZip) refuse path traversal, absolute paths, links, nested archives and decompression bombs.
Credentials and scan operator details are never copied into a report: the Nessus and Qualys mapping specs list the policy, preference and user fields as ignored, and the parsers skip them.
Index ¶
Constants ¶
const Container = "(container)"
Container is the Target of a field that only groups other fields.
const SniffLen = 64 << 10
SniffLen is how many leading bytes Detect needs at most.
Variables ¶
var ( // ErrUnknownFormat: Detect does not recognize the input, or the format // named is not one this package reads. ErrUnknownFormat = errors.New("unknown import format") // ErrTooLarge: the input or one of its parts is over a limit. ErrTooLarge = errors.New("import input over a limit") // ErrUnsafe: the input uses a construct this package never reads (an // XML internal subset or entity declaration, an unsupported charset, // a link or traversal in an archive). ErrUnsafe = errors.New("import input uses a refused construct") // ErrMalformed: the input is not well-formed for its format. ErrMalformed = errors.New("malformed import input") )
Errors a caller can test with errors.Is.
Functions ¶
Types ¶
type ArchiveFile ¶
type ArchiveFile struct {
// The file's path inside the archive, cleaned. It is only a label: it is
// never used to create a file.
Name string
// Uncompressed size as the archive declares it (the reader enforces the
// limits on the real size).
DeclaredSize int64
// contains filtered or unexported fields
}
ArchiveFile is one file of an archive.
func OpenZip ¶
func OpenZip(r io.ReaderAt, size int64, lim ArchiveLimits) ([]ArchiveFile, error)
OpenZip lists the regular files of a ZIP archive. It refuses the whole archive when an entry has an absolute path, a ".." element, a backslash, a drive letter or a control character in its name, is a link or another non-regular file, is encrypted, or is itself an archive, and when there are more than MaxEntries files. Nothing is ever written to disk.
func (ArchiveFile) Open ¶
func (a ArchiveFile) Open() (io.ReadCloser, error)
Open returns the file's content. The reader fails with ErrTooLarge as soon as the file passes MaxEntryBytes, the archive passes MaxTotalBytes, or the file decompresses at more than MaxRatio.
type ArchiveLimits ¶
type ArchiveLimits struct {
// Most files in the archive (directories are not counted).
MaxEntries int
// Largest uncompressed file.
MaxEntryBytes int64
// Largest sum of the uncompressed files.
MaxTotalBytes int64
// Largest ratio of uncompressed to compressed size of one file, checked
// as the file is read, once it is past 1 MiB.
MaxRatio int64
}
ArchiveLimits bound what OpenZip accepts. Zero fields take the defaults of DefaultArchiveLimits.
func DefaultArchiveLimits ¶
func DefaultArchiveLimits() ArchiveLimits
DefaultArchiveLimits returns the limits a zero ArchiveLimits stands for.
type Field ¶
Field is one row of a mapping spec.
Path is the field's path in the source: XML as "/Root/Child/@attr" (elements keyed by an attribute as "tag[name]"), JSON as "/member/array[]/ member". A "*" matches any run of characters inside one path segment; a "**" segment matches zero or more segments (a trailing "/**" matches everything below).
Exactly one of Target and Ignored is set: Target names the CTIS member or members the value goes to; Ignored says why the value is deliberately not kept. A container element whose children are listed has Target "(container)".
type Format ¶
type Format string
Format names a file format this package reads.
const ( // Nessus v2 XML (.nessus), written by Nessus and Tenable.sc. FormatNessus Format = "nessus" // Qualys VM host list detection XML (HOST_LIST_VM_DETECTION_OUTPUT). FormatQualys Format = "qualys" // Qualys KnowledgeBase XML (KNOWLEDGE_BASE_VULN_LIST_OUTPUT). Read only // as the companion of a detection file (Options.QualysKnowledgeBase). FormatQualysKB Format = "qualys_kb" // CycloneDX 1.4 to 1.6 JSON: an SBOM, a VDR or a VEX. FormatCycloneDX Format = "cyclonedx" // SPDX 2.2 / 2.3 JSON. FormatSPDX Format = "spdx" // osv-scanner JSON results (packages with their OSV records). FormatOSV Format = "osv" // CSAF 2.0 JSON (the VEX profile and the security advisory profile). FormatCSAF Format = "csaf" // OpenVEX JSON. FormatOpenVEX Format = "openvex" // DefectDojo Generic Findings Import JSON. FormatDefectDojo Format = "defectdojo" // SARIF 2.1.0 (any static analysis tool that writes SARIF). FormatSARIF Format = "sarif" // trivy JSON (image, file-system, repository and config scans). FormatTrivy Format = "trivy" // nuclei JSON lines (-jsonl) or a JSON array (-json-export). FormatNuclei Format = "nuclei" // semgrep JSON (--json). FormatSemgrep Format = "semgrep" // betterleaks JSON report (the gitleaks-compatible array). FormatBetterleaks Format = "betterleaks" // vuls JSON scan result (one server). FormatVuls Format = "vuls" // gitleaks JSON report (an array of leaks). Every gitleaks-compatible // scanner writes the same shape; Options.ToolName names the tool. FormatGitleaks Format = "gitleaks" // grype JSON output. FormatGrype Format = "grype" // ZAP traditional JSON or XML report. FormatZAP Format = "zap" )
type Issue ¶
type Issue struct {
// 1-based line and column of the place in the input, when known.
Line int `json:"line,omitempty"`
Column int `json:"column,omitempty"`
// The record the problem is in: an XML element path or a JSON pointer.
Path string `json:"path,omitempty"`
Message string `json:"message"`
}
Issue is a problem at a place in the input.
type Limits ¶
type Limits struct {
// Largest input, in bytes.
MaxInputBytes int64
// Deepest nesting of XML elements or JSON objects and arrays.
MaxDepth int
// Most XML elements in one document.
MaxElements int
// Most attributes on one XML element.
MaxAttributes int
// Largest text of one XML element or one JSON string, in bytes. Longer
// values are refused, not truncated, so a file that needs them is seen
// as what it is.
MaxTextBytes int
// Most findings, assets, dependencies and VEX statements in one result.
MaxFindings int
MaxAssets int
MaxComponents int
MaxStatements int
// Most distinct source field paths recorded for the coverage check.
MaxObservedPaths int
// Most problems reported in Result.Issues.
MaxIssues int
}
Limits bound what a parser accepts. A zero field takes its default (DefaultLimits). The defaults are sized for a real export of a large estate; a receiver that accepts uploads from people should pass smaller values.
func DefaultLimits ¶
func DefaultLimits() Limits
DefaultLimits returns the limits a zero Limits stands for.
type Options ¶
type Options struct {
// Format of the input. Empty: Detect decides from the first bytes.
Format Format
// Limits on the input; zero fields take DefaultLimits.
Limits Limits
// Report timestamp. Zero: the time of the parse (UTC).
Now time.Time
// metadata.id of the report (an import or scan session id).
ReportID string
// metadata.source_type of the report. Empty: "manual" (a file someone
// imports).
SourceType string
// tool.name of the report. Empty: the format's tool (nessus, qualys,
// cyclonedx, ...).
ToolName string
// Findings below this severity are counted in Stats.Skipped and left
// out. Empty keeps every finding.
MinSeverity ctis.Severity
// The asset of records that name none (a DefectDojo finding without an
// endpoint). Nil: an unclassified asset named after the tool.
DefaultAsset *ctis.Asset
// The repository a code report (SARIF, semgrep, betterleaks, a trivy
// file-system scan) is filed on, with its branch and commit. It wins
// over what the file names (a SARIF versionControlProvenance, a trivy
// artifact). A receiver that knows the repository from a verified
// identity (a CI workload token) sets it from there and never from the
// file; the importer only records the file's claim.
Repository string
Branch string
CommitSHA string
// SARIF only. ToolType is the kind of tool that wrote the log: "sast",
// "sca", "secret", "iac" or "web3". It decides the type of every finding
// and the tool's capabilities instead of the rule tags and the tool name
// (a scanner runtime that knows what it ran sets it). Other values are
// ignored. DefaultConfidence (1 to 100) is the confidence of a result
// whose rule states no precision; zero or out of range: 90.
ToolType string
DefaultConfidence int
// The Qualys KnowledgeBase XML for a Qualys detection file. Optional:
// without it, detections keep their QID and results but have no title
// beyond the QID, CVEs or CVSS.
QualysKnowledgeBase io.Reader
}
Options configure Parse. The zero value detects the format and uses the default limits.
type ParseError ¶
ParseError is the error of an input that could not be read. Line and Column point into the input when known.
func (*ParseError) Error ¶
func (e *ParseError) Error() string
func (*ParseError) Unwrap ¶
func (e *ParseError) Unwrap() error
type Product ¶
type Product struct {
// Package URL.
PURL string `json:"purl,omitempty"`
// CPE 2.3 name.
CPE string `json:"cpe,omitempty"`
// Name and version as the document states them, when it gives no
// identifier.
Name string `json:"name,omitempty"`
Version string `json:"version,omitempty"`
// When set, the statement is about these components inside the
// product only (an OpenVEX subcomponent, a CSAF relationship's
// component), not about the product as a whole or about the components
// anywhere else. A receiver matches a finding on a subcomponent's PURL
// or CPE only when the finding's asset is the product.
Subcomponents []Product `json:"subcomponents,omitempty"`
}
Product is a product of a VEX statement.
type Result ¶
type Result struct {
// Format of the file.
Format Format `json:"format"`
// The CTIS report: assets, findings and dependencies. Never nil; a VEX
// document gives a report without findings.
Report *ctis.Report `json:"report"`
// The statements of a VEX document (CSAF VEX, OpenVEX, the analysis
// members of a CycloneDX VEX). A receiver applies them to its own
// findings that match a statement's vulnerability and product.
VEX []VEXStatement `json:"vex,omitempty"`
// Counts of what the file held and what was kept.
Stats Stats `json:"stats"`
// Problems that did not stop the import (an item without a required
// field, a value out of range), at most Limits.MaxIssues.
Issues []Issue `json:"issues,omitempty"`
// Source field paths the file held that the format's mapping spec does
// not list, sorted. Their values are not in the report; a receiver can
// show them so the spec is extended.
Unmapped []string `json:"unmapped,omitempty"`
}
Result is what Parse returns for one file.
func Parse ¶
Parse reads one exported file and converts it to CTIS.
It returns a *ParseError (wrapping ErrUnknownFormat, ErrTooLarge, ErrUnsafe or ErrMalformed) when the input cannot be read at all. Problems with single records do not fail the parse; they are in Result.Issues and the record is skipped.
type Spec ¶
type Spec struct {
Format Format
// Human name of the source format and the versions the spec was written
// against.
Title string
SourceVersion string
// Notes on normalization rules that are not per field (severity tables,
// status tables, asset identity, deduplication).
Rules []string
// The fields, in document order.
Fields []Field
}
Spec is the mapping spec of one format: every source field the importer knows, and what becomes of it.
func Specs ¶
func Specs() []Spec
Specs returns the mapping spec of every format Parse reads (plus the Qualys KnowledgeBase), sorted by format.
func (Spec) Coverage ¶
Coverage counts the spec's mapped and ignored fields (containers are not counted).
type Stats ¶
type Stats struct {
// Records of the source: Nessus ReportItems, Qualys detections, CycloneDX
// vulnerabilities, OSV affected packages, DefectDojo findings.
Records int `json:"records"`
Assets int `json:"assets"`
Findings int `json:"findings"`
Components int `json:"components"`
Statements int `json:"statements"`
// Records skipped (below the minimum severity, or unusable; the reason
// of an unusable record is in Issues).
Skipped int `json:"skipped"`
// Findings per CTIS severity.
BySeverity map[ctis.Severity]int `json:"by_severity,omitempty"`
}
Stats counts what an import read and produced.
type VEXStatement ¶
type VEXStatement struct {
// The vulnerability, under every id the document gives it.
VulnerabilityIDs []ctis.VulnerabilityID `json:"vulnerability_ids"`
// The products the statement is about. A receiver matches a finding
// on a shared PURL (version-less when the statement's PURL has no
// version) or CPE.
Products []Product `json:"products"`
// The statement.
VEX ctis.VEX `json:"vex"`
}
VEXStatement is one exploitability statement of a VEX document: the status of the products below for the vulnerability below.
Source Files
¶
- archive.go
- betterleaks.go
- codeasset.go
- csaf.go
- cyclonedx.go
- defectdojo.go
- doc.go
- format.go
- gitleaks.go
- grype.go
- guard.go
- jsonsafe.go
- leaks.go
- limits.go
- nessus.go
- nuclei.go
- openvex.go
- osv.go
- parse.go
- qualys.go
- result.go
- sarif.go
- sbom_shared.go
- semgrep.go
- shared.go
- spdx.go
- spec.go
- spec_betterleaks.go
- spec_csaf.go
- spec_cyclonedx.go
- spec_defectdojo.go
- spec_gitleaks.go
- spec_grype.go
- spec_nessus.go
- spec_nuclei.go
- spec_openvex.go
- spec_osv.go
- spec_qualys.go
- spec_qualys_kb.go
- spec_sarif.go
- spec_semgrep.go
- spec_spdx.go
- spec_trivy.go
- spec_vuls.go
- spec_zap.go
- trivy.go
- vuls.go
- xmlsafe.go
- zap.go
Directories
¶
| Path | Synopsis |
|---|---|
|
Package mapping turns the JSON or JSON Lines output of any command-line tool into a CTIS report through a declarative mapping file, so a tool that writes JSON needs no parser code.
|
Package mapping turns the JSON or JSON Lines output of any command-line tool into a CTIS report through a declarative mapping file, so a tool that writes JSON needs no parser code. |