Documentation
¶
Overview ¶
Package ctrf provides Go types and utilities for the Common Test Report Format (CTRF). See https://ctrf.io/ and https://github.com/ctrf-io/ctrf/blob/main/schema/ctrf.schema.json
Index ¶
- Constants
- func ConfigMapName(runID, phase string) string
- func DeleteCTRFConfigMap(ctx context.Context, clientset kubernetes.Interface, ...) error
- func ExitCodeToCTRFStatus(code int32) string
- func IsFailingStatus(status string) bool
- func WriteCTRFConfigMap(ctx context.Context, clientset kubernetes.Interface, ...) error
- type Builder
- type Environment
- type Report
- type Results
- type RuntimeProvenance
- type Summary
- type TestResult
- type Tool
- type ValidatorResult
Constants ¶
const ( // ExtraKeySkipReason carries the closed-set code of a skipped result. ExtraKeySkipReason = "skipReason" // ExtraKeyObservationOnly marks a passed result as an observation, not a // demonstration; the value is ExtraValueTrue. ExtraKeyObservationOnly = "observationOnly" // ExtraKeyResultSchema is the result-contract version the validator image // spoke, stamped on every result before the check runs; the value is // ResultSchemaVersion. A reader that knows a check to be observation-only // reads its unmarked pass as an observation whether or not this key is // present; the key only tells the reader whether the image predates // ExtraKeyObservationOnly or dropped the marker. ExtraKeyResultSchema = "resultSchema" // ExtraValueTrue is the published form of a true boolean extra. ExtraValueTrue = "true" // ResultSchemaVersion is the current value of ExtraKeyResultSchema. Bump // it when the meaning of a marker changes or a reader needs to tell one // image generation from another: // - 1: every result carries this key; an observation-only pass carries // ExtraKeyObservationOnly. ResultSchemaVersion = "1" )
TestResult.Extra keys that both ends of the result contract read: the validator images publish them (validators.EmitExtra) and the evidence readers consume them (pkg/evidence/cncf, pkg/evidence/redact). They live here, the one package every side already imports, so the publisher and the readers cannot drift apart; validators re-exports them as ExtraKey*.
const ( // ReportFormatCTRF is the document format identifier. ReportFormatCTRF = "CTRF" // SpecVersion is the CTRF specification version implemented by this package. SpecVersion = "0.0.1" // StatusPassed indicates the test passed. StatusPassed = "passed" // StatusFailed indicates the test failed. StatusFailed = "failed" // StatusSkipped indicates the test was skipped. StatusSkipped = "skipped" // StatusPending indicates the test is pending execution. StatusPending = "pending" // StatusOther indicates an unexpected outcome (crash, OOM, timeout, etc.). StatusOther = "other" // ExtraLinePrefix marks a stdout line carrying one JSON object for // TestResult.Extra. pkg/validator/job.ExtractResult merges every such line, // a later value winning per key, and strips them from Stdout. // // The prefix lives here (not in validators/ or pkg/validator/job) because it // is the shared contract between the in-pod producer and the orchestrator // consumer, and both packages import ctrf. ExtraLinePrefix = "##AICR-EXTRA## " // ProvenanceLinePrefix marks a stdout line carrying one JSON // RuntimeProvenance record. Every such line is stripped from Stdout, and // the last valid one wins. ProvenanceLinePrefix = "##AICR-PROVENANCE## " )
const (
// ConfigMapKeyReport is the key used to store the CTRF JSON in the ConfigMap.
ConfigMapKeyReport = "report.json"
)
Variables ¶
This section is empty.
Functions ¶
func ConfigMapName ¶
ConfigMapName returns the canonical ConfigMap name for a CTRF report.
func DeleteCTRFConfigMap ¶
func DeleteCTRFConfigMap( ctx context.Context, clientset kubernetes.Interface, namespace, runID, phase string, ) error
DeleteCTRFConfigMap removes a CTRF ConfigMap. Ignores NotFound errors.
func ExitCodeToCTRFStatus ¶
ExitCodeToCTRFStatus maps a container exit code to a CTRF status string.
func IsFailingStatus ¶
IsFailingStatus reports whether a check or phase status must block progress — i.e., fail a readiness gate or trip cross-phase fail-fast.
Both StatusFailed and StatusOther are blocking. StatusOther means the check could not be executed to a verdict (crash, OOM, timeout, or a Job that reached a terminal Failed state with no inspectable pod). For a gate, an inconclusive outcome must fail closed: treating it as non-blocking lets a not-yet-ready cluster masquerade as passing. StatusSkipped stays non-blocking (a legitimately inapplicable check, e.g. no GPU nodes present).
func WriteCTRFConfigMap ¶
func WriteCTRFConfigMap( ctx context.Context, clientset kubernetes.Interface, namespace, runID, phase string, report *Report, ) error
WriteCTRFConfigMap serializes a Report to JSON and writes it to a ConfigMap. Uses create-or-update semantics: creates the ConfigMap if it does not exist, updates it if it already exists from a previous run.
Types ¶
type Builder ¶
type Builder struct {
// contains filtered or unexported fields
}
Builder accumulates test results and produces a CTRF Report.
func NewBuilder ¶
NewBuilder creates a builder for a single phase's CTRF report.
func (*Builder) AddResult ¶
func (b *Builder) AddResult(r *ValidatorResult)
AddResult converts a ValidatorResult to a CTRF TestResult and appends it.
func (*Builder) AddSkipped ¶
AddSkipped appends a skipped entry for a validator that was not executed.
func (*Builder) AddSkippedWithExtra ¶
AddSkippedWithExtra is AddSkipped with a structured Extra map attached.
The human reason goes in Message, which the default (minimal) redaction policy blanks unconditionally, so a reason that must reach a signed evidence bundle also has to ride Extra, the only per-test channel redaction preserves, and only for keys and values on pkg/evidence/redact's fail-closed allowlist. Pass a low-cardinality enum code or count, never free-form text.
func (*Builder) SetEnvironment ¶
func (b *Builder) SetEnvironment(env *Environment)
SetEnvironment sets the optional environment metadata for the report.
type Environment ¶
type Environment struct {
// AppName is the name of the application under test.
AppName string `json:"appName,omitempty"`
// AppVersion is the version of the application under test.
AppVersion string `json:"appVersion,omitempty"`
// TestEnvironment is the logical test environment (e.g., "eks-h100-training").
TestEnvironment string `json:"testEnvironment,omitempty"`
}
Environment describes the execution environment and build context.
type Report ¶
type Report struct {
// ReportFormat is the document format identifier. Always "CTRF".
ReportFormat string `json:"reportFormat"`
// SpecVersion is the CTRF specification version in SemVer format.
SpecVersion string `json:"specVersion"`
// Timestamp is the report generation time in RFC 3339 format.
Timestamp string `json:"timestamp"`
// GeneratedBy is the tool or system that produced this document.
GeneratedBy string `json:"generatedBy"`
// Results contains the test execution results.
Results Results `json:"results"`
}
Report is the top-level CTRF document.
func MergeReports ¶
MergeReports combines multiple CTRF reports into a single report. It aggregates test results, summaries, and picks the earliest timestamp.
func ReadCTRFConfigMap ¶
func ReadCTRFConfigMap( ctx context.Context, clientset kubernetes.Interface, namespace, runID, phase string, ) (*Report, error)
ReadCTRFConfigMap reads a CTRF Report from a ConfigMap.
type Results ¶
type Results struct {
// Tool identifies the testing tool or framework.
Tool Tool `json:"tool"`
// Summary provides aggregated statistics for the test run.
Summary Summary `json:"summary"`
// Tests is the list of individual test case results.
Tests []TestResult `json:"tests"`
// Environment describes the execution environment.
Environment *Environment `json:"environment,omitempty"`
}
Results holds the tool metadata, summary, test entries, and environment.
type RuntimeProvenance ¶
type RuntimeProvenance struct {
// ShippedDigest is sha256 (lowercase hex) of the normalized worker template
// read from the deployed artifact.
ShippedDigest string `json:"shippedDigest"`
// DerivedDigest is sha256 (lowercase hex) of the normalized worker template
// as applied for the measurement — scheduling stamped.
DerivedDigest string `json:"derivedDigest"`
// OverriddenPaths are the paths at which the applied template differs from
// the shipped one (benchmark-owned overrides plus stamped scheduling).
OverriddenPaths []string `json:"overriddenPaths,omitempty"`
// InheritedPaths are the shipped template's leaf paths carried unchanged.
InheritedPaths []string `json:"inheritedPaths,omitempty"`
}
RuntimeProvenance is the bounded, digest-bound evidence carrier for a benchmark runtime DERIVED from an artifact the recipe ships (#2297). It binds the result to the exact templates compared without shipping them: content identities of the normalized shipped and applied worker PodTemplateSpecs, the paths at which they differ, and the inventory of shipped paths carried into the applied runtime unchanged — an inherited field is equal on both sides and therefore invisible to a diff, so the inventory is what proves the wiring was measured rather than re-injected.
Paths are template KEYS (dotted, named-list elements addressed as [name]), never values: no network name, node name, or env value appears. Names the operator authors — list element names, and keys of user-keyed maps such as labels, annotations, nodeSelector and resource limits — are collapsed by pkg/evidence/redact in minimal evidence unless they are on its exact allowlists (the shipped runtime's fabric env names and vendor keys). This is an AICR extension alongside the CTRF `extra` object; it is absent for every result that did not derive a runtime.
type Summary ¶
type Summary struct {
// Tests is the total number of tests executed.
Tests int `json:"tests"`
// Passed is the count of tests with status "passed".
Passed int `json:"passed"`
// Failed is the count of tests with status "failed".
Failed int `json:"failed"`
// Skipped is the count of tests with status "skipped".
Skipped int `json:"skipped"`
// Pending is the count of tests with status "pending".
Pending int `json:"pending"`
// Other is the count of tests with status "other".
Other int `json:"other"`
// Start is the run start time in milliseconds since Unix epoch.
Start int64 `json:"start"`
// Stop is the run end time in milliseconds since Unix epoch.
Stop int64 `json:"stop"`
}
Summary provides aggregated statistics and timing for a test run.
type TestResult ¶
type TestResult struct {
// Name is the name or title of the test case.
Name string `json:"name"`
// Status is the final outcome of the test case.
// One of: "passed", "failed", "skipped", "pending", "other".
Status string `json:"status"`
// Duration is the test execution time in milliseconds.
Duration int `json:"duration"`
// Suite is the suite hierarchy from top-level to immediate parent.
Suite []string `json:"suite,omitempty"`
// Message is the error or failure message (e.g., from termination log).
Message string `json:"message,omitempty"`
// Stdout contains standard output lines from test execution.
Stdout []string `json:"stdout,omitempty"`
// Extra holds low-cardinality outcome data, mirroring the CTRF spec's
// extra object. Each value must be one of:
// - a count ("2")
// - a number rounded to one decimal place ("305.4", "+1.8")
// - "true" or "false"
// - a "v"-prefixed numeric version ("v25.3.0", "v25.3.0-rc.1")
// - a closed-set code ("no-gpu-nodes")
//
// Never node names, IPs, hostnames, digests of cluster-specific content,
// or other operator-identifying text. Keys are the validators.ExtraKey*
// constants.
Extra map[string]string `json:"extra,omitempty"`
// RuntimeProvenance binds a derived-runtime measurement to the templates it
// compared (see RuntimeProvenance). It survives minimal redaction after the
// shape and key-collapsing rules in pkg/evidence/redact; nil otherwise.
RuntimeProvenance *RuntimeProvenance `json:"runtimeProvenance,omitempty"`
}
TestResult represents an individual test case result. Message and Stdout are free-form text that minimal evidence redaction strips, while Extra is the low-cardinality outcome data it keeps.
type Tool ¶
type Tool struct {
// Name is the name of the testing tool.
Name string `json:"name"`
// Version is the version of the testing tool.
Version string `json:"version,omitempty"`
}
Tool identifies the testing tool that produced the results.
type ValidatorResult ¶
type ValidatorResult struct {
// Name is the validator name from the catalog entry.
Name string
// Phase is the validation phase from the catalog entry.
Phase string
// ExitCode is the container exit code. -1 indicates the container never ran.
ExitCode int32
// TerminationMsg is the content of /dev/termination-log.
TerminationMsg string
// Stdout contains the standard output lines from the container.
Stdout []string
// Extra is the merged payload of the container's ExtraLinePrefix lines,
// copied into TestResult.Extra.
Extra map[string]string
// RuntimeProvenance is the derived-runtime evidence carrier parsed from the
// container's ProvenanceLinePrefix sentinel line; nil when none was emitted.
RuntimeProvenance *RuntimeProvenance
// Duration is the wall-clock execution time.
Duration time.Duration
// StartTime is when the container started.
StartTime time.Time
// CompletionTime is when the container finished.
CompletionTime time.Time
}
ValidatorResult is the outcome of running a single validator container. Populated by the job package after extracting exit code, termination message, and stdout from the completed pod.
func (*ValidatorResult) CTRFStatus ¶
func (r *ValidatorResult) CTRFStatus() string
CTRFStatus maps the exit code to a CTRF status string.