Documentation
¶
Overview ¶
Package accuracyeval loads an embedded golden corpus of labeled detection cases and reduces the owned engine's produced-vs-expected results to detection-accuracy metrics (precision, recall, false-discovery / false-negative rates), overall and per ecosystem group. The engine itself is injected as a CaseScanner so this stays in the usecase layer (domain + benchmark only, no infrastructure). The SCA golden gate and the nightly worker job both consume this loader, so the measured and gated corpora can never drift apart.
Index ¶
- Constants
- func CaseAdvisories(c Case) []advisory.Advisory
- func CaseSBOM(c Case) *sbom.SBOM
- func Evaluate(ctx context.Context, scanner CaseScanner) (benchmark.AccuracyReport, error)
- func Observe(ctx context.Context, scanner CaseScanner, c Case) (benchmark.AccuracyObservation, error)
- func ToRun(id string, ranAt time.Time, rep benchmark.AccuracyReport) (accuracy.Run, error)
- type Advisory
- type Case
- type CaseScanner
- type Component
- type Expected
- type Range
Constants ¶
const CorpusVersion = "detection-golden-v1"
CorpusVersion identifies the embedded golden corpus. Bump it when the corpus set changes so a persisted accuracy run records which corpus produced it.
Variables ¶
This section is empty.
Functions ¶
func CaseAdvisories ¶
CaseAdvisories converts a case's pinned corpus advisories into domain advisories, the input a CaseScanner runs the owned engine against.
func Evaluate ¶
func Evaluate(ctx context.Context, scanner CaseScanner) (benchmark.AccuracyReport, error)
Evaluate runs the engine over the whole embedded corpus and reduces it to an accuracy report (overall + per-group precision/recall/F1/false-discovery/false-negative). Pure and offline given an offline scanner.
func Observe ¶
func Observe(ctx context.Context, scanner CaseScanner, c Case) (benchmark.AccuracyObservation, error)
Observe runs the engine over one case and keys both the ground truth and the produced findings by "component@version|advisory-id". A produced finding is keyed STRICTLY by its own primary id (ownadvisory.Source reports the CVE when the advisory carries one), with no alias rescue, so a wrong advisory cannot be laundered into a true positive through a shared alias.
Types ¶
type Advisory ¶
type Advisory struct {
ID string `json:"id"`
Aliases []string `json:"aliases,omitempty"`
Summary string `json:"summary,omitempty"`
CVSSScore float64 `json:"cvss_score,omitempty"`
CVSSVector string `json:"cvss_vector,omitempty"`
Ecosystem string `json:"ecosystem"` // OSV ecosystem / distro key (e.g. "npm", "PyPI", "Debian:11")
Package string `json:"package"`
Ranges []Range `json:"ranges,omitempty"`
Versions []string `json:"versions,omitempty"`
Fixed string `json:"fixed_version,omitempty"`
Withdrawn bool `json:"withdrawn,omitempty"`
}
Advisory is a compact, authorable advisory record pinned into a case.
type Case ¶
type Case struct {
Name string `json:"name"`
Group string `json:"group"` // ecosystem, for the per-group ratchet
Notes string `json:"notes,omitempty"`
Components []Component `json:"components"`
Advisories []Advisory `json:"advisories"`
Expected []Expected `json:"expected"`
}
Case is one labeled evaluation unit.
type CaseScanner ¶
type CaseScanner interface {
ScanCase(ctx context.Context, advisories []advisory.Advisory, doc *sbom.SBOM) ([]vulnerability.RawFinding, error)
}
CaseScanner runs the owned detection engine over one case's SBOM against its pinned advisories and returns the raw findings. It is injected (the concrete implementation wraps ownadvisory.Source in the infrastructure layer) so this package stays free of any infrastructure import.
type Component ¶
type Component struct {
Name string `json:"name"`
Version string `json:"version"`
PURL string `json:"purl"`
}
Component is one SBOM component in a case.
type Expected ¶
type Expected struct {
Component string `json:"component"`
Version string `json:"version"`
CVE string `json:"cve"`
}
Expected is one ground-truth detection that MUST be produced, keyed to a specific component VERSION so a false positive on a patched version of the same package is not masked by a true positive on the vulnerable version. CVE is the advisory id the source is expected to REPORT.
Source Files