README
¶
Acceptance Testing
This package provides a testing framework for validating vulnerability analyzers.
See the full documentation for details on:
- Creating fixtures
- Running tests
- Implementing an auditor
Quick Start
go test -tags=integration -v ./test/acceptance/...
Documentation
¶
Overview ¶
Package acceptance provides a testing framework for vulnerability auditors. It loads test fixtures from OCI registries using the Referrers API, runs auditors against them, and compares results with the expected fixture
Example ¶
package main
import (
"context"
"io"
"iter"
"testing"
"github.com/quay/claircore/test/acceptance"
)
// ExampleAuditor demonstrates how to implement a custom Auditor.
// This file exists to ensure the example in the documentation compiles.
type ExampleAuditor struct{}
func (a *ExampleAuditor) Audit(ctx context.Context, t testing.TB, ref string, csafDocs iter.Seq[io.Reader]) ([]acceptance.Result, error) {
for r := range csafDocs {
_ = r
}
return nil, nil
}
func main() {
ctx := context.Background()
t := &testing.T{}
auditor := &ExampleAuditor{}
acceptance.Run(ctx, t, auditor, []string{
"quay.io/projectquay/clair-fixtures:python-test",
})
}
Output:
Index ¶
- Variables
- func FetchVEXDocs(ctx context.Context, client *http.Client, urls []string) ([][]byte, error)
- func Run(ctx context.Context, t *testing.T, a Auditor, refs []string, ...)
- type Auditor
- type ClaircoreAuditor
- type ClaircoreConfig
- type Comparison
- type Fixture
- type LoaderOption
- type Match
- type Mismatch
- type Result
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ErrNotAFixture = errors.New("image is not a test fixture")
ErrNotAFixture is returned when an image lacks the required referrer artifacts (vulnerability manifest and/or CSAF documents) to be treated as a test fixture. This is acceptable as existing repos can contain non-fixture images.
Functions ¶
func FetchVEXDocs ¶
FetchVEXDocs fetches CSAF/VEX documents from the given URLs using client. The returned slice can be assigned directly to Fixture.VEXDocuments.
Types ¶
type Auditor ¶
type Auditor interface {
// Audit analyses the image at ref using the provided CSAF documents
// and returns vulnerability findings.
Audit(ctx context.Context, t testing.TB, ref string, csafDocs iter.Seq[io.Reader]) ([]Result, error)
}
Auditor is the interface that vulnerability auditors must implement.
type ClaircoreAuditor ¶
type ClaircoreAuditor struct {
// contains filtered or unexported fields
}
ClaircoreAuditor implements Auditor using claircore's libraries directly.
func NewClaircoreAuditor ¶
func NewClaircoreAuditor(ctx context.Context, t testing.TB, cfg *ClaircoreConfig, client *http.Client) (*ClaircoreAuditor, error)
NewClaircoreAuditor creates a new auditor backed by claircore libraries. The testing.TB is used for layer caching - layers are cached in the global test cache directory and persist across test runs.
type ClaircoreConfig ¶
type ClaircoreConfig struct {
IndexerDSN string
MatcherDSN string
Platform string // Default: "linux/amd64"
// IndexerPool and MatcherPool allow passing pre-existing pools (e.g., from test helpers).
// If set, the corresponding DSN field is ignored.
IndexerPool *pgxpool.Pool
MatcherPool *pgxpool.Pool
}
ClaircoreConfig holds configuration for creating a ClaircoreAuditor.
type Comparison ¶
type Comparison struct {
// Fixture is the reference that was tested.
Fixture string
// Matches are results that matched expectations.
Matches []Match
// Mismatches are results with the wrong status.
Mismatches []Mismatch
// Misses are expected results that the scanner did not report.
Misses []fixtures.ManifestRecord
// Extras are results reported by the scanner but not in the fixture.
// These do not cause test failure (scanner may find more than fixture defines).
// TODO (crozzy): Maybe a strict mode that fails the test if there are any extras?
Extras []Result
}
Comparison is the result of comparing expected vs actual scanner results.
func Compare ¶
func Compare(expected []fixtures.ManifestRecord, actual []Result) *Comparison
Compare checks auditor results against expected results from a fixture.
The comparison logic:
- StatusAffected: MUST appear in results as affected. Missing or wrong status is failure.
- StatusNotAffected: MUST appear in results as not-affected. Missing or wrong status is failure.
- StatusAbsent: MUST NOT appear in results. If present, it's a failure.
- Results not in expected are Extras (reported but not a failure).
func (*Comparison) Passed ¶
func (c *Comparison) Passed() bool
Passed returns true if there were no mismatches or missing results.
type Fixture ¶
type Fixture struct {
// Reference is the OCI reference (e.g., "registry.io/repo:tag").
Reference string
// Manifest is the digest of the OCI manifest.
Manifest string
// VEXDocuments contains the CSAF/VEX documents for vulnerability matching.
VEXDocuments [][]byte
// Expected is the parsed vulnerability manifest (expected output).
Expected []fixtures.ManifestRecord
}
Fixture is a loaded test fixture from an OCI registry.
func LoadFixture ¶
LoadFixture fetches a fixture from an OCI registry using the Referrers API.
The reference can be:
- A registry reference: "registry.io/repo:tag" or "registry.io/repo@sha256:..."
- An OCI Layout directory: "ocidir:///path/to/layout"
If WithFixture is provided, that fixture is returned directly without fetching.
type LoaderOption ¶
type LoaderOption func(*loaderConfig)
LoaderOption configures the fixture loader.
func WithDockerCerts ¶
func WithDockerCerts() LoaderOption
WithDockerCerts uses TLS certificates from local cert directories.
func WithDockerCreds ¶
func WithDockerCreds() LoaderOption
WithDockerCreds uses credentials from local machine.
func WithFixture ¶
func WithFixture(f *Fixture) LoaderOption
WithFixture provides a pre-loaded fixture, bypassing registry fetching. This is primarily useful for testing.
Note: When using WithFixture with Run, only a single reference should be provided. The same fixture is returned for all references, ignoring which reference was requested. For testing multiple fixtures, use real registry fixtures with OCI referrers or call Run multiple times.
To build a fixture from live VEX URLs rather than OCI referrers, use FetchVEXDocs to populate Fixture.VEXDocuments and set Fixture.Expected inline, then pass the result to Run.
Example ¶
ExampleWithFixture demonstrates building a fixture from live VEX URLs using acceptance.FetchVEXDocs and acceptance.WithFixture, without storing VEX documents as OCI referrers.
package main
import (
"context"
"io"
"iter"
"net/http"
"testing"
"github.com/quay/claircore/test/acceptance"
"github.com/quay/claircore/toolkit/fixtures"
)
// ExampleAuditor demonstrates how to implement a custom Auditor.
// This file exists to ensure the example in the documentation compiles.
type ExampleAuditor struct{}
func (a *ExampleAuditor) Audit(ctx context.Context, t testing.TB, ref string, csafDocs iter.Seq[io.Reader]) ([]acceptance.Result, error) {
for r := range csafDocs {
_ = r
}
return nil, nil
}
func main() {
ctx := context.Background()
t := &testing.T{}
auditor := &ExampleAuditor{}
docs, err := acceptance.FetchVEXDocs(ctx, http.DefaultClient, []string{
"https://security.example.com/advisories/CVE-2024-1234.json",
})
if err != nil {
t.Fatal(err)
}
fix := &acceptance.Fixture{
Reference: "registry.example.com/namespace/image@sha256:abc123...",
VEXDocuments: docs,
Expected: []fixtures.ManifestRecord{
{ID: "CVE-2024-1234", Product: "mypackage-1.2.3", Status: fixtures.StatusAffected},
},
}
acceptance.Run(ctx, t, auditor, nil, acceptance.WithFixture(fix))
}
Output:
type Match ¶
type Match struct {
TrackingID string
ProductID string
Status fixtures.VulnerabilityStatus
}
Match is a result that matched expectations.
type Mismatch ¶
type Mismatch struct {
TrackingID string
ProductID string
Expected fixtures.VulnerabilityStatus
Actual fixtures.VulnerabilityStatus
}
Mismatch is a result with a status different from expected.
type Result ¶
type Result struct {
// TrackingID is the CSAF trackingID.
TrackingID string
// ProductID is the CSAF productID from the product tree.
ProductID string
// Status is the scanner's determination.
Status fixtures.VulnerabilityStatus
// Package is the matched package name@version for debugging.
Package string
}
Result is a single vulnerability finding from a scanner.
Source Files
Directories