Documentation
¶
Overview ¶
Package sealed builds what the sealed coverage tier runs against: advisory databases generated from one file of our own text, a container with no network, the expectations a scenario states, and the comparison between those and what a scan produced.
The tier exists to assert exact results. A test against the live databases can only assert structure, because an advisory published tomorrow changes today's answer. Against a database generated from test/integration/testdata/ecosystems/advisories.yaml, a scan either produces the findings the scenario names or it has a bug, and running it with no network makes any fetch the design did not account for fail rather than drift.
Index ¶
- Constants
- func AWSAccessKeyID() (string, error)
- func Check(exp Expected, anns []Annotation, got []Finding) ([]string, error)
- func CheckErrors(exp []ErrorExpectation, report []byte) ([]string, error)
- func CheckInit(exp, got InitExpectation) []string
- func Diff(want, got string) string
- func RunReplacements(work string, start time.Time) map[string]string
- func ValidateSARIF(raw []byte) error
- func WriteGoVulnDB(home string, advs Advisories, fetchedAt time.Time) error
- func WriteRetireRepo(home string, advs Advisories, fetchedAt time.Time) error
- func WriteTrivyDB(cacheDir string, advs Advisories) error
- type Advisories
- type Advisory
- type Annotation
- type ComponentExpectation
- type Container
- type ErrorExpectation
- type Expected
- type Finding
- type FindingExpectation
- type InitExpectation
- type Normalizer
- type RunOptions
- type Scenario
- type SecretExpectation
Constants ¶
const Cleared = "<cleared>"
Cleared is what a normalized field is replaced with.
const FailingToolMessage = "sealed fixture: this program failed on purpose"
FailingToolMessage is what a program replaced by Fail writes before exiting 2.
const FixtureSuffix = ".fixture"
FixtureSuffix marks a dependency manifest stored under another name. A requirements.txt, go.mod or package-lock.json anywhere in this repository is read by the forge's dependency graph and reported as a dependency of Draugr, vulnerabilities included; stored with the suffix it is a file nothing parses, and Prepare restores its name in the copy a scan reads.
const Image = "ubuntu:24.04@sha256:008173c23f95b170204355c12626cb5a965d779a7e1283b09e9cffbb1bf33ca3"
Image is the container the sealed tier runs in, pinned by digest. It supplies only the root of a filesystem: /usr, /etc and /opt are the host's, mounted read-only, so the tools, interpreters and libraries inside are the ones installed on the machine running the test, and the only thing the container takes away is the network.
const TrivySchemaVersion = 2
TrivySchemaVersion is the database schema the pinned Trivy reads. A Trivy that moves to another fails to open this database rather than reading it wrongly; bump it with the tool and regenerate.
Variables ¶
This section is empty.
Functions ¶
func AWSAccessKeyID ¶
AWSAccessKeyID returns a random string in the shape of an AWS access key id, which no AWS account issued. Regenerated until its entropy clears the threshold secret scanners apply, so a low-entropy draw cannot make a scenario fail by chance.
func Check ¶
func Check(exp Expected, anns []Annotation, got []Finding) ([]string, error)
Check compares what a scan reported with what the scenario expects, and returns one line per difference: an expected result that is missing, a result nothing expected, and a result on a line an annotation marks as one its rule must not report.
func CheckErrors ¶
func CheckErrors(exp []ErrorExpectation, report []byte) ([]string, error)
CheckErrors compares the controls a report says could not run with the ones the scenario expects to fail, one line per difference. A control that errors unexpectedly is a difference, and so is an expected failure that ran, or failed for another reason.
func CheckInit ¶
func CheckInit(exp, got InitExpectation) []string
CheckInit compares what `draugr init` wrote with what the scenario expects, one line per difference.
func Diff ¶
Diff renders the lines that differ between two documents, as removals and additions around the longest run of lines they share, capped at a screenful.
func RunReplacements ¶
RunReplacements are the strings particular to one sealed run: its directory and the dates the run could have stamped.
func ValidateSARIF ¶
ValidateSARIF checks a document against the SARIF 2.1.0 schema.
The published schema declares JSON Schema draft-04, which the validator does not read. The two differences that matter are its "$schema" and its top-level "id", renamed "$id" in later drafts; the schema uses no other draft-04 construct, so both are rewritten on load and nothing else is.
func WriteGoVulnDB ¶
func WriteGoVulnDB(home string, advs Advisories, fetchedAt time.Time) error
WriteGoVulnDB writes the Go advisories in advs as a Go vulnerability database in the offline layout govulncheck's -db reads (index/db.json, index/modules.json, index/vulns.json, ID/<id>.json) into home/.draugr/feeds/govulndb, and records it in the feed manifest as fetched at fetchedAt, which is where a scan looks for it.
func WriteRetireRepo ¶
func WriteRetireRepo(home string, advs Advisories, fetchedAt time.Time) error
WriteRetireRepo writes the js advisories in advs as a retire.js repository at home/.draugr/data/retirejs/jsrepository.json, with the cache index naming it, which is the copy Draugr hands retire.js with --jsrepo when offline. Each library is identified by its file name and by a banner comment.
func WriteTrivyDB ¶
func WriteTrivyDB(cacheDir string, advs Advisories) error
WriteTrivyDB writes a Trivy vulnerability database holding advs into cacheDir/db, the layout `trivy --cache-dir <cacheDir> --skip-db-update` reads. Advisories in an ecosystem Trivy does not cover (js) are left out.
Types ¶
type Advisories ¶
type Advisories struct {
Advisories []Advisory `yaml:"advisories"`
}
Advisories is the parsed advisories file.
func LoadAdvisories ¶
func LoadAdvisories(path string) (Advisories, error)
LoadAdvisories reads and checks an advisories file.
func (Advisories) For ¶
func (a Advisories) For(ecosystem string) []Advisory
For returns the advisories in one ecosystem, in file order.
type Advisory ¶
type Advisory struct {
// ID is the identifier a scanner reports: a CVE where there is one.
ID string `yaml:"id"`
// GoID is the Go vulnerability database's own identifier, for an advisory govulncheck reads.
GoID string `yaml:"goID,omitempty"`
// Ecosystem is the package ecosystem: pip, npm, go, rubygems, cargo, composer, nuget, maven,
// or js for a library retire.js identifies by its file.
Ecosystem string `yaml:"ecosystem"`
// Package is the name the ecosystem gives the package.
Package string `yaml:"package"`
// Fixed is the first version without the vulnerability; every earlier version is affected.
Fixed string `yaml:"fixed"`
// Severity is CRITICAL, HIGH, MEDIUM or LOW.
Severity string `yaml:"severity"`
// Title is a one-line summary, written for the fixture.
Title string `yaml:"title"`
// Symbols maps a Go package path to the functions the vulnerability is in, which is what
// govulncheck follows to decide whether it is reachable.
Symbols map[string][]string `yaml:"symbols,omitempty"`
// CWE is the weakness, which retire.js requires on every entry.
CWE string `yaml:"cwe,omitempty"`
}
Advisory is one vulnerability the sealed databases carry, in the terms every generator needs.
type Annotation ¶
type Annotation struct {
// File is relative to the repository, with any fixture suffix removed.
File string
// Line is the line the annotation is about: the first line after it that is neither blank nor
// another annotation.
Line int
Rule string
// Want is true for `ruleid:` and false for `ok:`.
Want bool
}
Annotation is an expectation written in a fixture's own code, the way Semgrep tests its rules: a comment naming a rule that must (`ruleid:`) or must not (`ok:`) report the next line of code.
func ReadAnnotations ¶
func ReadAnnotations(dir string) ([]Annotation, error)
ReadAnnotations collects the annotations in every file under dir, skipping vendor/ and node_modules/, which hold other people's code.
type ComponentExpectation ¶
type ComponentExpectation struct {
Name string `yaml:"name"`
// Repositories are the repository urls, relative to the descriptor.
Repositories []string `yaml:"repositories"`
}
ComponentExpectation is one component `draugr init` declares.
type Container ¶
type Container struct {
// Work is the one writable directory, mounted at the same path inside. The run's HOME is
// Work/home, so every cache and dataset Draugr reads is the one the test put there.
Work string
// HostHome is the invoking user's home directory, mounted read-only for the tools installed
// under it.
HostHome string
// Path is the PATH inside the container.
Path string
// Extra are further read-only mounts, such as a GOROOT outside the directories already mounted.
Extra []string
// Env is added to the environment inside the container.
Env map[string]string
// UID and GID are who the container runs as, so what it writes belongs to the invoking user.
UID, GID int
}
Container describes one sealed run.
func HostContainer ¶
HostContainer describes a sealed run over work using the invoking user's tools: PATH as it is, behind the tools Draugr installed and the directory of the semgrep on PATH.
Semgrep's CLI runs a second program, pysemgrep, by looking it up on PATH. Beside the semgrep binary is the one from the same installation; a stale one elsewhere on PATH belongs to another interpreter, which the host may find through its own home directory and the container cannot.
func (*Container) Fail ¶
Fail replaces tool inside the container with a program that writes FailingToolMessage to stderr and exits 2.
type ErrorExpectation ¶
type ErrorExpectation struct {
Control string `yaml:"control"`
// Contains is text the control's error must include.
Contains string `yaml:"contains"`
}
ErrorExpectation is one control that must fail to run.
type Expected ¶
type Expected struct {
// Init is what `draugr init` writes for the repository.
Init InitExpectation `yaml:"init"`
// Secrets are credentials the harness writes into the repository before it is committed.
// Each must be reported by the secrets control.
Secrets []SecretExpectation `yaml:"secrets"`
// Findings are every other result the scan must report, and together with the secrets and the
// fixture's inline sast annotations, every result it may report.
Findings []FindingExpectation `yaml:"findings"`
// Sealed changes what the container provides, for a scenario about a failure.
Sealed RunOptions `yaml:"sealed"`
// Errors are the controls that must fail to run, each with text its error must contain. Every
// other control must run.
Errors []ErrorExpectation `yaml:"errors"`
}
Expected is a scenario's expected.yaml: what `draugr init` proposes for the repository and what a sealed scan of it reports.
type Finding ¶
type Finding struct {
Control string
Tool string
Rule string
File string
Line int
Package string
Reachability string
}
Finding is one SARIF result, reduced to what an expectation can name.
type FindingExpectation ¶
type FindingExpectation struct {
Control string `yaml:"control"`
// Tool is the scanner that reported it, as the SARIF names it.
Tool string `yaml:"tool"`
Rule string `yaml:"rule"`
// Location is file:line relative to the repository, or the file alone for a result about a
// whole file.
Location string `yaml:"location"`
// Package is "ecosystem name version", for a finding about a dependency.
Package string `yaml:"package,omitempty"`
// Reachability is the verdict reachability analysis reached, where it ran.
Reachability string `yaml:"reachability,omitempty"`
}
FindingExpectation is one result a scan must report.
type InitExpectation ¶
type InitExpectation struct {
// Controls are the controls it enables, in any order.
Controls []string `yaml:"controls"`
// Components are the components it declares.
Components []ComponentExpectation `yaml:"components"`
}
InitExpectation is the descriptor `draugr init` writes.
func ObserveInit ¶
func ObserveInit(path string) (InitExpectation, error)
ObserveInit reads the descriptor `draugr init` wrote, through the loader a scan uses, as the controls it enables and the components it declares.
type Normalizer ¶
type Normalizer struct {
// Clear are paths to fields replaced with Cleared. "*" matches every element of an array or
// every value of an object. Each path must match at least one field.
Clear [][]string
// Sort are paths to arrays sorted by their elements' JSON.
Sort [][]string
// Replace maps a string to what it becomes, in every string value.
Replace map[string]string
// AllowMissing accepts a Clear path that matches nothing. For a run where a control failed to
// start, which leaves no scanner version and no finding for that control to clear.
AllowMissing bool
}
Normalizer makes a report comparable across runs: it clears the fields that vary by machine or by moment, replaces run-specific strings wherever they appear, and sorts the arrays whose order depends on which job finished first.
A path that matches nothing is an error rather than a no-op. A field that moved would otherwise stop being cleared without anybody noticing, and the golden would start pinning a value that changes on the next run, or would stop pinning anything at the old path.
func ReportNormalizer ¶
func ReportNormalizer(replace map[string]string) Normalizer
ReportNormalizer is the normalizer for Draugr's report.json.
func SARIFNormalizer ¶
func SARIFNormalizer(replace map[string]string) Normalizer
SARIFNormalizer is the normalizer for Draugr's results.sarif.
type RunOptions ¶
type RunOptions struct {
// WithoutTool is a program removed from PATH inside the container.
WithoutTool string `yaml:"withoutTool"`
// FailingTool is a program replaced by one that writes FailingToolMessage and exits 2.
FailingTool string `yaml:"failingTool"`
// WithoutTrivyDB leaves Trivy's vulnerability database unwritten.
WithoutTrivyDB bool `yaml:"withoutTrivyDB"`
// GoVulnDBAge is how long before the run the local Go vulnerability database is recorded as
// fetched, as a Go duration. Empty means at the start of the run.
GoVulnDBAge string `yaml:"goVulnDBAge"`
}
RunOptions take something away from a sealed run, so a scenario can assert that its absence is reported rather than passed.
type Scenario ¶
type Scenario struct {
// Name is the directory name, which is also the component and the repository's directory.
Name string
// Dir is the scenario directory.
Dir string
// Expected is the parsed expected.yaml.
Expected Expected
}
Scenario is one directory under ecosystems/: a repository to scan, the descriptor to scan it with, and what the scan must produce.
func LoadScenario ¶
LoadScenario reads the scenario in dir.
func (Scenario) Prepare ¶
Prepare copies the scenario's repository to work/<name>, restores the names of its manifests, writes the secrets the scenario asks for, and commits the result. It returns the repository directory.
The secrets are generated here and never committed to this repository: a credential-shaped string in a public tree is refused by push protection and reported by every scanner that reads it, including the one being tested.
type SecretExpectation ¶
type SecretExpectation struct {
// File is where the harness writes it, relative to the repository. It is written on line 1.
File string `yaml:"file"`
// Rules are the rules that must report it.
Rules []string `yaml:"rules"`
}
SecretExpectation is one generated credential.