ctrf

package
v1.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 16, 2026 License: Apache-2.0 Imports: 13 Imported by: 0

Documentation

Overview

Package ctrf provides Go types and utilities for the Common Test Report Format (CTRF). See https://ctrf.io/ and https://github.com/ctrf-io/ctrf/blob/main/schema/ctrf.schema.json

Index

Constants

View Source
const (
	// ExtraKeySkipReason carries the closed-set code of a skipped result.
	ExtraKeySkipReason = "skipReason"

	// ExtraKeyObservationOnly marks a passed result as an observation, not a
	// demonstration; the value is ExtraValueTrue.
	ExtraKeyObservationOnly = "observationOnly"

	// ExtraKeyResultSchema is the result-contract version the validator image
	// spoke, stamped on every result before the check runs; the value is
	// ResultSchemaVersion. A reader that knows a check to be observation-only
	// reads its unmarked pass as an observation whether or not this key is
	// present; the key only tells the reader whether the image predates
	// ExtraKeyObservationOnly or dropped the marker.
	ExtraKeyResultSchema = "resultSchema"

	// ExtraValueTrue is the published form of a true boolean extra.
	ExtraValueTrue = "true"

	// ResultSchemaVersion is the current value of ExtraKeyResultSchema. Bump
	// it when the meaning of a marker changes or a reader needs to tell one
	// image generation from another:
	//   - 1: every result carries this key; an observation-only pass carries
	//     ExtraKeyObservationOnly.
	ResultSchemaVersion = "1"
)

TestResult.Extra keys that both ends of the result contract read: the validator images publish them (validators.EmitExtra) and the evidence readers consume them (pkg/evidence/cncf, pkg/evidence/redact). They live here, the one package every side already imports, so the publisher and the readers cannot drift apart; validators re-exports them as ExtraKey*.

View Source
const (
	// ReportFormatCTRF is the document format identifier.
	ReportFormatCTRF = "CTRF"

	// SpecVersion is the CTRF specification version implemented by this package.
	SpecVersion = "0.0.1"

	// StatusPassed indicates the test passed.
	StatusPassed = "passed"

	// StatusFailed indicates the test failed.
	StatusFailed = "failed"

	// StatusSkipped indicates the test was skipped.
	StatusSkipped = "skipped"

	// StatusPending indicates the test is pending execution.
	StatusPending = "pending"

	// StatusOther indicates an unexpected outcome (crash, OOM, timeout, etc.).
	StatusOther = "other"

	// ExtraLinePrefix marks a stdout line carrying one JSON object for
	// TestResult.Extra. pkg/validator/job.ExtractResult merges every such line,
	// a later value winning per key, and strips them from Stdout.
	//
	// The prefix lives here (not in validators/ or pkg/validator/job) because it
	// is the shared contract between the in-pod producer and the orchestrator
	// consumer, and both packages import ctrf.
	ExtraLinePrefix = "##AICR-EXTRA## "

	// ProvenanceLinePrefix marks a stdout line carrying one JSON
	// RuntimeProvenance record. Every such line is stripped from Stdout, and
	// the last valid one wins.
	ProvenanceLinePrefix = "##AICR-PROVENANCE## "
)
View Source
const (
	// ConfigMapKeyReport is the key used to store the CTRF JSON in the ConfigMap.
	ConfigMapKeyReport = "report.json"
)

Variables

This section is empty.

Functions

func ConfigMapName

func ConfigMapName(runID, phase string) string

ConfigMapName returns the canonical ConfigMap name for a CTRF report.

func DeleteCTRFConfigMap

func DeleteCTRFConfigMap(
	ctx context.Context,
	clientset kubernetes.Interface,
	namespace, runID, phase string,
) error

DeleteCTRFConfigMap removes a CTRF ConfigMap. Ignores NotFound errors.

func ExitCodeToCTRFStatus

func ExitCodeToCTRFStatus(code int32) string

ExitCodeToCTRFStatus maps a container exit code to a CTRF status string.

func IsFailingStatus

func IsFailingStatus(status string) bool

IsFailingStatus reports whether a check or phase status must block progress — i.e., fail a readiness gate or trip cross-phase fail-fast.

Both StatusFailed and StatusOther are blocking. StatusOther means the check could not be executed to a verdict (crash, OOM, timeout, or a Job that reached a terminal Failed state with no inspectable pod). For a gate, an inconclusive outcome must fail closed: treating it as non-blocking lets a not-yet-ready cluster masquerade as passing. StatusSkipped stays non-blocking (a legitimately inapplicable check, e.g. no GPU nodes present).

func WriteCTRFConfigMap

func WriteCTRFConfigMap(
	ctx context.Context,
	clientset kubernetes.Interface,
	namespace, runID, phase string,
	report *Report,
) error

WriteCTRFConfigMap serializes a Report to JSON and writes it to a ConfigMap. Uses create-or-update semantics: creates the ConfigMap if it does not exist, updates it if it already exists from a previous run.

Types

type Builder

type Builder struct {
	// contains filtered or unexported fields
}

Builder accumulates test results and produces a CTRF Report.

func NewBuilder

func NewBuilder(toolName, toolVersion, phase string) *Builder

NewBuilder creates a builder for a single phase's CTRF report.

func (*Builder) AddResult

func (b *Builder) AddResult(r *ValidatorResult)

AddResult converts a ValidatorResult to a CTRF TestResult and appends it.

func (*Builder) AddSkipped

func (b *Builder) AddSkipped(name, phase, reason string)

AddSkipped appends a skipped entry for a validator that was not executed.

func (*Builder) AddSkippedWithExtra

func (b *Builder) AddSkippedWithExtra(name, phase, reason string, extra map[string]string)

AddSkippedWithExtra is AddSkipped with a structured Extra map attached.

The human reason goes in Message, which the default (minimal) redaction policy blanks unconditionally, so a reason that must reach a signed evidence bundle also has to ride Extra, the only per-test channel redaction preserves, and only for keys and values on pkg/evidence/redact's fail-closed allowlist. Pass a low-cardinality enum code or count, never free-form text.

func (*Builder) Build

func (b *Builder) Build() *Report

Build produces the final CTRF Report with computed summary.

func (*Builder) SetEnvironment

func (b *Builder) SetEnvironment(env *Environment)

SetEnvironment sets the optional environment metadata for the report.

type Environment

type Environment struct {
	// AppName is the name of the application under test.
	AppName string `json:"appName,omitempty"`

	// AppVersion is the version of the application under test.
	AppVersion string `json:"appVersion,omitempty"`

	// TestEnvironment is the logical test environment (e.g., "eks-h100-training").
	TestEnvironment string `json:"testEnvironment,omitempty"`
}

Environment describes the execution environment and build context.

type Report

type Report struct {
	// ReportFormat is the document format identifier. Always "CTRF".
	ReportFormat string `json:"reportFormat"`

	// SpecVersion is the CTRF specification version in SemVer format.
	SpecVersion string `json:"specVersion"`

	// Timestamp is the report generation time in RFC 3339 format.
	Timestamp string `json:"timestamp"`

	// GeneratedBy is the tool or system that produced this document.
	GeneratedBy string `json:"generatedBy"`

	// Results contains the test execution results.
	Results Results `json:"results"`
}

Report is the top-level CTRF document.

func MergeReports

func MergeReports(toolName, toolVersion string, reports []*Report) *Report

MergeReports combines multiple CTRF reports into a single report. It aggregates test results, summaries, and picks the earliest timestamp.

func ReadCTRFConfigMap

func ReadCTRFConfigMap(
	ctx context.Context,
	clientset kubernetes.Interface,
	namespace, runID, phase string,
) (*Report, error)

ReadCTRFConfigMap reads a CTRF Report from a ConfigMap.

type Results

type Results struct {
	// Tool identifies the testing tool or framework.
	Tool Tool `json:"tool"`

	// Summary provides aggregated statistics for the test run.
	Summary Summary `json:"summary"`

	// Tests is the list of individual test case results.
	Tests []TestResult `json:"tests"`

	// Environment describes the execution environment.
	Environment *Environment `json:"environment,omitempty"`
}

Results holds the tool metadata, summary, test entries, and environment.

type RuntimeProvenance

type RuntimeProvenance struct {
	// ShippedDigest is sha256 (lowercase hex) of the normalized worker template
	// read from the deployed artifact.
	ShippedDigest string `json:"shippedDigest"`
	// DerivedDigest is sha256 (lowercase hex) of the normalized worker template
	// as applied for the measurement — scheduling stamped.
	DerivedDigest string `json:"derivedDigest"`
	// OverriddenPaths are the paths at which the applied template differs from
	// the shipped one (benchmark-owned overrides plus stamped scheduling).
	OverriddenPaths []string `json:"overriddenPaths,omitempty"`
	// InheritedPaths are the shipped template's leaf paths carried unchanged.
	InheritedPaths []string `json:"inheritedPaths,omitempty"`
}

RuntimeProvenance is the bounded, digest-bound evidence carrier for a benchmark runtime DERIVED from an artifact the recipe ships (#2297). It binds the result to the exact templates compared without shipping them: content identities of the normalized shipped and applied worker PodTemplateSpecs, the paths at which they differ, and the inventory of shipped paths carried into the applied runtime unchanged — an inherited field is equal on both sides and therefore invisible to a diff, so the inventory is what proves the wiring was measured rather than re-injected.

Paths are template KEYS (dotted, named-list elements addressed as [name]), never values: no network name, node name, or env value appears. Names the operator authors — list element names, and keys of user-keyed maps such as labels, annotations, nodeSelector and resource limits — are collapsed by pkg/evidence/redact in minimal evidence unless they are on its exact allowlists (the shipped runtime's fabric env names and vendor keys). This is an AICR extension alongside the CTRF `extra` object; it is absent for every result that did not derive a runtime.

type Summary

type Summary struct {
	// Tests is the total number of tests executed.
	Tests int `json:"tests"`

	// Passed is the count of tests with status "passed".
	Passed int `json:"passed"`

	// Failed is the count of tests with status "failed".
	Failed int `json:"failed"`

	// Skipped is the count of tests with status "skipped".
	Skipped int `json:"skipped"`

	// Pending is the count of tests with status "pending".
	Pending int `json:"pending"`

	// Other is the count of tests with status "other".
	Other int `json:"other"`

	// Start is the run start time in milliseconds since Unix epoch.
	Start int64 `json:"start"`

	// Stop is the run end time in milliseconds since Unix epoch.
	Stop int64 `json:"stop"`
}

Summary provides aggregated statistics and timing for a test run.

type TestResult

type TestResult struct {
	// Name is the name or title of the test case.
	Name string `json:"name"`

	// Status is the final outcome of the test case.
	// One of: "passed", "failed", "skipped", "pending", "other".
	Status string `json:"status"`

	// Duration is the test execution time in milliseconds.
	Duration int `json:"duration"`

	// Suite is the suite hierarchy from top-level to immediate parent.
	Suite []string `json:"suite,omitempty"`

	// Message is the error or failure message (e.g., from termination log).
	Message string `json:"message,omitempty"`

	// Stdout contains standard output lines from test execution.
	Stdout []string `json:"stdout,omitempty"`

	// Extra holds low-cardinality outcome data, mirroring the CTRF spec's
	// extra object. Each value must be one of:
	//   - a count ("2")
	//   - a number rounded to one decimal place ("305.4", "+1.8")
	//   - "true" or "false"
	//   - a "v"-prefixed numeric version ("v25.3.0", "v25.3.0-rc.1")
	//   - a closed-set code ("no-gpu-nodes")
	//
	// Never node names, IPs, hostnames, digests of cluster-specific content,
	// or other operator-identifying text. Keys are the validators.ExtraKey*
	// constants.
	Extra map[string]string `json:"extra,omitempty"`

	// RuntimeProvenance binds a derived-runtime measurement to the templates it
	// compared (see RuntimeProvenance). It survives minimal redaction after the
	// shape and key-collapsing rules in pkg/evidence/redact; nil otherwise.
	RuntimeProvenance *RuntimeProvenance `json:"runtimeProvenance,omitempty"`
}

TestResult represents an individual test case result. Message and Stdout are free-form text that minimal evidence redaction strips, while Extra is the low-cardinality outcome data it keeps.

type Tool

type Tool struct {
	// Name is the name of the testing tool.
	Name string `json:"name"`

	// Version is the version of the testing tool.
	Version string `json:"version,omitempty"`
}

Tool identifies the testing tool that produced the results.

type ValidatorResult

type ValidatorResult struct {
	// Name is the validator name from the catalog entry.
	Name string

	// Phase is the validation phase from the catalog entry.
	Phase string

	// ExitCode is the container exit code. -1 indicates the container never ran.
	ExitCode int32

	// TerminationMsg is the content of /dev/termination-log.
	TerminationMsg string

	// Stdout contains the standard output lines from the container.
	Stdout []string

	// Extra is the merged payload of the container's ExtraLinePrefix lines,
	// copied into TestResult.Extra.
	Extra map[string]string

	// RuntimeProvenance is the derived-runtime evidence carrier parsed from the
	// container's ProvenanceLinePrefix sentinel line; nil when none was emitted.
	RuntimeProvenance *RuntimeProvenance

	// Duration is the wall-clock execution time.
	Duration time.Duration

	// StartTime is when the container started.
	StartTime time.Time

	// CompletionTime is when the container finished.
	CompletionTime time.Time
}

ValidatorResult is the outcome of running a single validator container. Populated by the job package after extracting exit code, termination message, and stdout from the completed pod.

func (*ValidatorResult) CTRFStatus

func (r *ValidatorResult) CTRFStatus() string

CTRFStatus maps the exit code to a CTRF status string.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL