Documentation
¶
Overview ¶
Package assurance implements Bomly's release assurance framework: the shared check-result contract every quality check emits, the declarative catalog that declares which checks must exist, and the report the public assurance page renders for each release.
Index ¶
- Constants
- func ArchiveName(binary, version, goos, goarch string) string
- func ExpectedAssets(version string) []string
- func MatchJobURL(payload []byte, runnerName string) (string, error)
- func RenderMarkdown(report Report, opts MarkdownOptions) string
- func RenderResultMarkdown(result CheckResult, reproduce [][]string) string
- func StatusIcon(status Status) string
- func UntrustedNativeArchives(outcome ChecksumOutcome, version string) []string
- type Area
- type AreaReport
- type Artifact
- type AssetPresence
- type BinaryProbe
- type BuildOptions
- type Catalog
- type Check
- type CheckResult
- type ChecksumEntries
- type ChecksumOutcome
- type Coverage
- type Detail
- type EcosystemCoverage
- type Environment
- type Evidence
- type EvidenceArtifact
- type EvidenceLevel
- type ExpectedInstance
- type GoTestSummary
- type Index
- type IndexEntry
- type Input
- type InstanceReport
- type Level
- type Link
- type MarkdownOptions
- type MetricTrend
- type Release
- type Report
- type ReportCheck
- type ReportEvidence
- type Runner
- type Source
- type Stage
- type StageReport
- type Status
- type StatusChange
- type Trends
- type UnknownResult
- type Verdict
Constants ¶
const BenchmarkRunSchema = "bomly.benchmark-run/v1"
BenchmarkRunSchema is the schema of the performance sample manifest produced by internal/assurance/perfrun.
const CatalogSchema = "bomly.assurance-catalog/v1"
CatalogSchema is the schema identifier of the assurance catalog.
const CheckSchema = "bomly.assurance-check/v1"
CheckSchema is the schema identifier every check result carries.
const DefaultCatalogPath = "docs/assurance/catalog.json"
DefaultCatalogPath is the repository-relative catalog location.
const IndexSchema = "bomly.assurance-index/v1"
IndexSchema is the schema identifier of the release index.
const MaxCatalogBytes = 4 << 20
MaxCatalogBytes bounds the catalog document.
const MaxGoTestDetails = 200
MaxGoTestDetails caps how many individual test outcomes a check result records, so one slice with thousands of subtests cannot bloat the report.
const MaxReportBytes = 16 << 20
MaxReportBytes bounds a report document.
const MaxResultBytes = 8 << 20
MaxResultBytes bounds a single check-result document.
const ReportSchema = "bomly.assurance-report/v1"
ReportSchema is the schema identifier of a per-release assurance report.
const SBOMAssuranceSchema = "bomly.sbom-assurance-run/v1"
SBOMAssuranceSchema is the schema of the SBOM interoperability run manifest produced by internal/assurance/sbominterop.
Variables ¶
This section is empty.
Functions ¶
func ArchiveName ¶
ArchiveName returns the release archive file name for one binary, version, and platform. version is the tag without its leading "v".
func ExpectedAssets ¶
ExpectedAssets lists every file a published release must carry, sorted. version is the release tag without its leading "v".
func MatchJobURL ¶
MatchJobURL picks the URL of the job a check is running inside, so a reported count links to the exact log that produced it rather than to the whole run.
A runner executes one job at a time, so the running job on this runner is the caller's own job — that holds for matrix legs and for jobs contributed by a called workflow, which is why the runner name is matched rather than the job name (display names differ between direct and nested invocations). Returns an empty string when there is no confident match; callers fall back to the run.
func RenderMarkdown ¶
func RenderMarkdown(report Report, opts MarkdownOptions) string
RenderMarkdown renders a report as the markdown used for job summaries and the per-release tracking issue.
func RenderResultMarkdown ¶
func RenderResultMarkdown(result CheckResult, reproduce [][]string) string
RenderResultMarkdown renders one check result for a workflow step summary. reproduce is the catalog's local reproduction command for the check, shown so a reader can run the same thing without hunting for it.
func StatusIcon ¶
StatusIcon returns the emoji shown next to a status in markdown summaries.
func UntrustedNativeArchives ¶
func UntrustedNativeArchives(outcome ChecksumOutcome, version string) []string
UntrustedNativeArchives returns the archives for the running platform that this release's checksum verification did not vouch for — mismatched, or absent from SHA256SUMS entirely. Their contents must not be executed.
Types ¶
type Area ¶
type Area struct {
ID string `json:"id"`
Title string `json:"title"`
Description string `json:"description"`
}
Area groups related checks and evidence for presentation.
type AreaReport ¶
type AreaReport struct {
ID string `json:"id"`
Title string `json:"title"`
Description string `json:"description"`
Status Status `json:"status"`
Verdict Verdict `json:"verdict"`
CheckIDs []string `json:"check_ids,omitempty"`
EvidenceIDs []string `json:"evidence_ids,omitempty"`
}
AreaReport summarises one subject area: the checks that cover it and the evidence claims made about it. Areas are what the published report is organised by, because "what does this tell me about Bomly" is a more useful question for a reader than "when in the release did this run".
type Artifact ¶
type Artifact struct {
Name string `json:"name"`
SHA256 string `json:"sha256,omitempty"`
Bytes int64 `json:"bytes,omitempty"`
}
Artifact records a file a check produced or verified.
type AssetPresence ¶
AssetPresence reports which expected assets a directory holds.
func InspectAssets ¶
func InspectAssets(dir, version string) (AssetPresence, error)
InspectAssets compares the files in dir against the expected asset list.
type BinaryProbe ¶
BinaryProbe is the outcome of extracting one release archive and asking the binary inside it for its version.
func ProbeNativeBinaries ¶
func ProbeNativeBinaries(ctx context.Context, dir, version, workDir string) ([]BinaryProbe, error)
ProbeNativeBinaries extracts the archives built for the host platform and checks that each binary reports the released version.
type BuildOptions ¶
type BuildOptions struct {
// Release identifies the release being reported on.
Release Release
// Stages limits the report to these stages; empty means every stage.
Stages []Stage
// StageRuns maps a stage to the workflow run that produced its results.
StageRuns map[Stage]string
// Previous is the previous release's report, used for trends.
Previous *Report
// IncludeEvidence attaches the public evidence claims to the report.
IncludeEvidence bool
// GeneratedBy names the tool version that produced the report.
GeneratedBy string
// Now is the report timestamp; the zero value means time.Now().
Now time.Time
}
BuildOptions configures how a report is assembled from check results.
type Catalog ¶
type Catalog struct {
SchemaVersion string `json:"schema_version"`
Areas []Area `json:"areas"`
Checks []Check `json:"checks"`
Evidence []Evidence `json:"evidence"`
}
Catalog declares every check the framework expects and every public evidence claim those checks back. It is the single source the report is built against.
func LoadCatalog ¶
LoadCatalog reads and validates the catalog at path.
func ParseCatalog ¶
ParseCatalog decodes and structurally validates a catalog document. It does not touch the filesystem; use VerifyFiles for the repository-side checks.
func (Catalog) ChecksForStage ¶
ChecksForStage returns the catalog checks that belong to one stage.
func (Catalog) VerifyFiles ¶
VerifyFiles confirms every file the catalog names -- each claim's artifacts and fixture inputs -- exists inside root as a regular file, and returns how many it checked. A claim that points at a file that was renamed or deleted would otherwise keep reading as proven.
The count is returned so a caller can refuse a run that reached nothing (ADR-0044): a catalog with no files to check and a catalog whose files are all present look the same from a nil error.
type Check ¶
type Check struct {
ID string `json:"id"`
Title string `json:"title"`
Area string `json:"area"`
Stage Stage `json:"stage"`
Level Level `json:"level"`
Description string `json:"description"`
Source Source `json:"source"`
ExpectedInstances []ExpectedInstance `json:"expected_instances,omitempty"`
Reproduce [][]string `json:"reproduce,omitempty"`
Proves []string `json:"proves"`
Limitations []string `json:"limitations"`
}
Check declares one quality check: which stage runs it, whether it gates the release, which instances must report, and what it does and does not prove.
type CheckResult ¶
type CheckResult struct {
SchemaVersion string `json:"schema_version"`
ID string `json:"id"`
Instance string `json:"instance,omitempty"`
Stage Stage `json:"stage"`
Level Level `json:"level,omitempty"`
Status Status `json:"status"`
StartedAt string `json:"started_at,omitempty"`
FinishedAt string `json:"finished_at,omitempty"`
DurationMS float64 `json:"duration_ms,omitempty"`
Ref string `json:"ref,omitempty"`
Tag string `json:"tag,omitempty"`
Commit string `json:"commit,omitempty"`
Version string `json:"version,omitempty"`
RunURL string `json:"run_url,omitempty"`
Job string `json:"job,omitempty"`
Runner Runner `json:"runner,omitempty"`
Summary string `json:"summary"`
Metrics map[string]float64 `json:"metrics,omitempty"`
Details []Detail `json:"details,omitempty"`
Artifacts []Artifact `json:"artifacts,omitempty"`
Links []Link `json:"links,omitempty"`
}
CheckResult is what a single check instance writes when it finishes. One file per instance; instances sharing an ID merge into one reported check.
func ConvertBenchmarkRun ¶
func ConvertBenchmarkRun(data []byte, base CheckResult) (CheckResult, error)
ConvertBenchmarkRun turns a performance sample manifest into a check result.
func ConvertSBOMAssurance ¶
func ConvertSBOMAssurance(data []byte, base CheckResult) (CheckResult, error)
ConvertSBOMAssurance turns an SBOM interoperability run manifest into a check result.
func LoadResults ¶
func LoadResults(dir string) ([]CheckResult, error)
LoadResults reads every check-result document under dir, recursively, so a directory of merged CI artifacts can be consumed directly. Files that are not check results are ignored; malformed check results are an error.
func ParseCheckResult ¶
func ParseCheckResult(data []byte) (CheckResult, error)
ParseCheckResult decodes and validates one check-result document.
func (CheckResult) Encode ¶
func (r CheckResult) Encode() ([]byte, error)
Encode renders the check result as indented JSON with a trailing newline.
func (CheckResult) FileName ¶
func (r CheckResult) FileName() string
FileName is the on-disk name a check result is written under.
func (CheckResult) Key ¶
func (r CheckResult) Key() string
Key identifies a check instance uniquely within one run.
func (CheckResult) Validate ¶
func (r CheckResult) Validate() error
Validate reports whether the check result satisfies the contract.
type ChecksumEntries ¶
ChecksumEntries maps asset file names to their recorded SHA-256 hashes.
func ParseSHA256SUMS ¶
func ParseSHA256SUMS(data []byte) (ChecksumEntries, error)
ParseSHA256SUMS reads a GoReleaser SHA256SUMS document.
type ChecksumOutcome ¶
type ChecksumOutcome struct {
Verified []string
Mismatched []string
Unlisted []string
// NotDownloaded names files SHA256SUMS lists that are absent locally, which
// is expected when only the platform-native assets were fetched.
NotDownloaded []string
}
ChecksumOutcome is the result of hashing the assets present in a directory.
func VerifyChecksums ¶
func VerifyChecksums(dir string, entries ChecksumEntries) (ChecksumOutcome, error)
VerifyChecksums hashes every file in dir and compares it to SHA256SUMS.
type Coverage ¶
type Coverage struct {
Ecosystems []EcosystemCoverage `json:"ecosystems"`
}
Coverage lists every ecosystem any check exercised, in name order. Each carries the worst status among the checks that covered it.
type Detail ¶
type Detail struct {
Name string `json:"name"`
Status Status `json:"status"`
Note string `json:"note,omitempty"`
DurationMS float64 `json:"duration_ms,omitempty"`
}
Detail is one named sub-result inside a check instance, such as a single smoke test, fuzz target, or cross-build target.
type EcosystemCoverage ¶
EcosystemCoverage is whether one language or package format was exercised for this release. Which check did the exercising is deliberately not part of it: the reader's question is "was my ecosystem covered", and answering it with a grid of checks invites the false conclusion that a blank cell is a gap when the ecosystem was covered by another check.
type Environment ¶
type Environment struct {
Runners []Runner `json:"runners,omitempty"`
GeneratedBy string `json:"generated_by,omitempty"`
}
Environment records where the checks ran.
type Evidence ¶
type Evidence struct {
ID string `json:"id"`
Title string `json:"title"`
Area string `json:"area"`
Description string `json:"description"`
EvidenceLevel EvidenceLevel `json:"evidence_level"`
CheckID string `json:"check_id"`
Instance string `json:"instance,omitempty"`
Inputs []Input `json:"inputs"`
RequiredTools []string `json:"required_tools,omitempty"`
Reproduce [][]string `json:"reproduce"`
Artifacts []EvidenceArtifact `json:"artifacts"`
Proves []string `json:"proves"`
Limitations []string `json:"limitations"`
}
Evidence is one public claim about Bomly's behavior, proven by a pinned input and a committed result file, and backed by a check whose per-release status shows whether the claim still holds. A claim that would only restate what its check already reports does not belong here — the check card says it once.
type EvidenceArtifact ¶
type EvidenceArtifact struct {
Path string `json:"path"`
}
EvidenceArtifact is a repository file that records an evidence claim's result.
It is named by path and nothing else. A file in this repository is already content-addressed by git: a release tag and a path identify exact bytes, and the report records the commit. A checksum stored beside the path would be a second, hand-kept copy of what git knows, and the only thing it could ever do is fall behind the file -- which it did, on every golden update, until it was removed. What cannot be addressed by this repository's history is still pinned where it is named: a git input by revision, a container by digest.
type EvidenceLevel ¶
type EvidenceLevel string
EvidenceLevel describes how strong a public evidence claim is.
const ( // EvidenceDeterministic is reproducible in process with no external input. EvidenceDeterministic EvidenceLevel = "deterministic" // EvidencePinnedInput depends on a pinned repository revision or fixture. EvidencePinnedInput EvidenceLevel = "pinned-input" // EvidenceSnapshot depends on an upstream tag that can move. EvidenceSnapshot EvidenceLevel = "snapshot" // EvidenceLiveService depends on data from a live service. EvidenceLiveService EvidenceLevel = "live-service" )
Evidence strength levels, from strongest to weakest guarantee.
func (EvidenceLevel) Valid ¶
func (e EvidenceLevel) Valid() bool
Valid reports whether the evidence level is known.
type ExpectedInstance ¶
type ExpectedInstance struct {
Name string `json:"name"`
Ecosystems []string `json:"ecosystems,omitempty"`
Platform string `json:"platform,omitempty"`
}
ExpectedInstance declares one matrix leg a check must report.
type GoTestSummary ¶
type GoTestSummary struct {
Packages int
Total int
Passed int
Failed int
Skipped int
ElapsedSec float64
Details []Detail
Truncated int
// FailedTests names the tests that failed, in report order.
FailedTests []string
// Anomalies records non-JSON lines, which usually mean a build failure.
Anomalies []string
}
GoTestSummary is the outcome of one `go test -json` stream.
func ParseGoTestEvents ¶
ParseGoTestEvents reads a `go test -json` stream and summarises it. Lines that are not JSON events (build errors, toolchain notices) are recorded as anomalies instead of failing the parse. When echo is non-nil, test output is replayed to it so CI logs stay readable.
func (GoTestSummary) ToCheckResult ¶
func (s GoTestSummary) ToCheckResult(base CheckResult, exitCode int) CheckResult
ToCheckResult turns a `go test -json` summary into a check result. exitCode is the test command's exit status, which catches build failures that produce no test events at all.
type Index ¶
type Index struct {
SchemaVersion string `json:"schema_version"`
Latest string `json:"latest"`
GeneratedAt string `json:"generated_at"`
Releases []IndexEntry `json:"releases"`
}
Index lists every release that has a published assurance report, newest first.
func ParseIndex ¶
ParseIndex decodes and validates an index document.
type IndexEntry ¶
type IndexEntry struct {
Tag string `json:"tag"`
Version string `json:"version"`
PublishedAt string `json:"published_at,omitempty"`
GeneratedAt string `json:"generated_at"`
Verdict Status `json:"verdict"`
Gates int `json:"gates_failed"`
Path string `json:"path"`
}
IndexEntry is one release listed in the assurance index.
type Input ¶
type Input struct {
Kind string `json:"kind"`
Location string `json:"location"`
Ref string `json:"ref,omitempty"`
Revision string `json:"revision,omitempty"`
}
Input names a pinned repository, fixture, container image, or service an evidence claim depends on.
type InstanceReport ¶
type InstanceReport struct {
Name string `json:"name"`
Status Status `json:"status"`
Summary string `json:"summary,omitempty"`
DurationMS float64 `json:"duration_ms,omitempty"`
RunURL string `json:"run_url,omitempty"`
Runner Runner `json:"runner,omitempty"`
Metrics map[string]float64 `json:"metrics,omitempty"`
Details []Detail `json:"details,omitempty"`
Artifacts []Artifact `json:"artifacts,omitempty"`
Links []Link `json:"links,omitempty"`
}
InstanceReport is one reported leg of a check.
type Level ¶
type Level string
Level distinguishes checks that block a release from advisory observations.
type MarkdownOptions ¶
type MarkdownOptions struct {
// Heading is the top-level title; a default is used when empty.
Heading string
// IncludeChecks lists every check in a per-stage table.
IncludeChecks bool
// IncludeTrends adds the comparison against the previous release.
IncludeTrends bool
}
MarkdownOptions controls how a report renders as markdown.
type MetricTrend ¶
type MetricTrend struct {
CheckID string `json:"check_id"`
Metric string `json:"metric"`
Previous float64 `json:"previous"`
Current float64 `json:"current"`
Delta float64 `json:"delta"`
DeltaPct float64 `json:"delta_pct,omitempty"`
Better string `json:"better"`
}
MetricTrend compares one metric against the previous release's report.
type Release ¶
type Release struct {
Tag string `json:"tag"`
Version string `json:"version"`
Commit string `json:"commit,omitempty"`
URL string `json:"url,omitempty"`
PublishedAt string `json:"published_at,omitempty"`
}
Release identifies the release a report describes.
type Report ¶
type Report struct {
SchemaVersion string `json:"schema_version"`
GeneratedAt string `json:"generated_at"`
Release Release `json:"release"`
Verdict Verdict `json:"verdict"`
Stages []StageReport `json:"stages"`
Areas []AreaReport `json:"areas"`
Checks []ReportCheck `json:"checks"`
Evidence []ReportEvidence `json:"evidence"`
Coverage Coverage `json:"coverage"`
Trends *Trends `json:"trends,omitempty"`
Unknown []UnknownResult `json:"unknown_results,omitempty"`
Environment Environment `json:"environment"`
}
Report is the per-release document the public assurance page renders.
func BuildReport ¶
func BuildReport(catalog Catalog, results []CheckResult, opts BuildOptions) Report
BuildReport merges check results into the catalog and produces the report that the public assurance page renders. Declared checks without a result are reported as missing; reported checks the catalog does not declare are listed separately so nothing is silently dropped.
func LoadReport ¶
LoadReport reads and validates the report at path.
func ParseReport ¶
ParseReport decodes and structurally validates a report document.
type ReportCheck ¶
type ReportCheck struct {
ID string `json:"id"`
Title string `json:"title"`
Area string `json:"area"`
Stage Stage `json:"stage"`
Level Level `json:"level"`
Description string `json:"description"`
Status Status `json:"status"`
Summary string `json:"summary,omitempty"`
DurationMS float64 `json:"duration_ms,omitempty"`
Source Source `json:"source"`
Instances []InstanceReport `json:"instances,omitempty"`
MissingInstances []string `json:"missing_instances,omitempty"`
Metrics map[string]float64 `json:"metrics,omitempty"`
Reproduce [][]string `json:"reproduce,omitempty"`
Proves []string `json:"proves"`
Limitations []string `json:"limitations"`
}
ReportCheck is one catalog check merged with the results it received.
type ReportEvidence ¶
type ReportEvidence struct {
ID string `json:"id"`
Title string `json:"title"`
Area string `json:"area"`
Description string `json:"description"`
EvidenceLevel EvidenceLevel `json:"evidence_level"`
CheckID string `json:"check_id"`
Instance string `json:"instance,omitempty"`
Status Status `json:"status"`
Inputs []Input `json:"inputs"`
RequiredTools []string `json:"required_tools,omitempty"`
Reproduce [][]string `json:"reproduce"`
Artifacts []EvidenceArtifact `json:"artifacts"`
Proves []string `json:"proves"`
Limitations []string `json:"limitations"`
}
ReportEvidence is one public claim with the status of the check backing it.
type Runner ¶
type Runner struct {
OS string `json:"os,omitempty"`
Arch string `json:"arch,omitempty"`
GoVersion string `json:"go_version,omitempty"`
}
Runner records the machine a check instance executed on.
type Stage ¶
type Stage string
Stage names the release phase a check belongs to.
const ( // StagePrerequisites runs on the source tree before a release tag exists. StagePrerequisites Stage = "prerequisites" // StagePreRelease runs inside the release pipeline against the draft release. StagePreRelease Stage = "pre-release" // StagePostRelease runs after publication against the shipped artifacts. StagePostRelease Stage = "post-release" )
The three release assurance stages.
type StageReport ¶
type StageReport struct {
ID Stage `json:"id"`
Title string `json:"title"`
Status Status `json:"status"`
Verdict Verdict `json:"verdict"`
RunURL string `json:"run_url,omitempty"`
CheckIDs []string `json:"check_ids"`
}
StageReport is one release stage and the checks it contributed.
type Status ¶
type Status string
Status is the outcome of a check or check instance.
const ( // StatusPass means the check completed and every expectation held. StatusPass Status = "pass" // StatusSkip means the check did not run for a declared reason. StatusSkip Status = "skip" // StatusMissing means no result was reported for a declared check. StatusMissing Status = "missing" // StatusDegraded means the check completed with reduced confidence. StatusDegraded Status = "degraded" // StatusFail means the check did not hold. StatusFail Status = "fail" )
Check outcomes, ordered from best to worst by Severity.
type StatusChange ¶
type StatusChange struct {
CheckID string `json:"check_id"`
Previous Status `json:"previous"`
Current Status `json:"current"`
}
StatusChange records a check whose status moved between releases.
type Trends ¶
type Trends struct {
PreviousTag string `json:"previous_tag"`
Metrics []MetricTrend `json:"metrics,omitempty"`
Changed []StatusChange `json:"changed,omitempty"`
}
Trends compares this report against the previous release's report.
type UnknownResult ¶
type UnknownResult struct {
ID string `json:"id"`
Instance string `json:"instance,omitempty"`
Stage Stage `json:"stage"`
Status Status `json:"status"`
}
UnknownResult records a reported check that the catalog does not declare.
type Verdict ¶
type Verdict struct {
Overall Status `json:"overall"`
Checks int `json:"checks"`
Passed int `json:"passed"`
Failed int `json:"failed"`
Degraded int `json:"degraded"`
Skipped int `json:"skipped"`
Missing int `json:"missing"`
GatesFailed []string `json:"gates_failed,omitempty"`
AdvisoriesFailed []string `json:"advisories_failed,omitempty"`
MissingChecks []string `json:"missing_checks,omitempty"`
}
Verdict counts outcomes and names the checks that need attention.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Command assurance drives Bomly's release assurance framework.
|
Command assurance drives Bomly's release assurance framework. |
|
Command perfrun repeatedly executes one deterministic assurance case and records cold/warm samples in a versioned machine-readable manifest.
|
Command perfrun repeatedly executes one deterministic assurance case and records cold/warm samples in a versioned machine-readable manifest. |
|
Command sbominterop generates canonical SBOMs and validates them with checksum-pinned upstream tools.
|
Command sbominterop generates canonical SBOMs and validates them with checksum-pinned upstream tools. |