registry-conformance

registry-conformance

Status: Phase 1 — the test catalog still carries not started and optional rows.

A Go CLI and library that validates an artifact registry implementation against the Maven, npm, and OCI distribution protocols. It runs a catalog of conformance tests against a registry under test and reports which behaviors pass, fail, or are skipped, so registry operators can check their implementation against the suite's test catalog instead of guessing.

Supported formats:

• Maven
• npm
• OCI distribution

The test catalog is the single source of truth for the suite's Phase 1 inventory; see the catalog documentation for its schema and lifecycle rules.

Install

From source

go build -o registry-conformance ./cmd/conformance

With Go install

go install gitlab.com/gitlab-org/ops/registry-conformance/cmd/conformance@latest

This installs the binary as conformance (named after the command package). To get the registry-conformance name, use go build above or download a release artifact.

Each release publishes binaries, checksums, SBOMs, and cosign signatures to the project registry.

Prerequisites

Some tests drive the real client instead of raw HTTP. Put the matching binary on PATH before running that format's suite:

When a driver is missing, its tests report SKIP and the run still exits 0. Check the Skipped: count in the summary block before treating a green run as full coverage.

Quick start

List the tests for a format

registry-conformance list --format maven

list prints one test name per line from the format's test catalog — including the negative-auth test (<format>.negative.unauthenticated) that run also executes — and never contacts a registry. list and run therefore agree on the test inventory for a format, so an allow-list or coverage count built from list output reconciles with a run's counts.

Run the conformance suite

registry-conformance run \
  --format npm \
  --registry-url https://registry.example.com \
  --credential bearer:env:NPM_TOKEN

registry-conformance prints each test result to stdout as PASS, FAIL, or SKIP lines followed by a summary block. In a terminal without NO_COLOR set, it colors the status prefixes. stderr carries structured JSON logs and, on a tool error, a single-line message — redirect the two streams separately in CI.

The suite is state-changing: it publishes and deletes real artifacts in the target registry under a per-run conformance-<run-id>-* namespace and does not remove them afterwards. Point the quick start at a test registry rather than production.

To fail CI-style consumers, use the exit code: 0 when all tests passed or were skipped, 1 when at least one test failed, and 2 for a tool error (bad configuration, no matching tests, the run exceeding --timeout, or a failure writing the report). A registry that cannot be reached fails its tests rather than stopping the run, so it exits 1, not 2.

For a machine-readable result, write JUnit XML:

registry-conformance run \
  --format oci \
  --registry-url https://registry.example.com \
  --report junit.xml

A timed-out or otherwise interrupted run still writes the partial report, marked with a synthetic <testcase> named <format>.run.interrupted carrying an <error> element with the cancellation reason — rendered as an errored test by GitLab CI's JUnit parser — plus errors="1" on the count attributes. The stdout summary block names the same reason as its last line (Interrupted: <reason>). A JUnit file with tests="0" and errors="0", by contrast, is a completed run that matched no tests.

Usage

Flags

Every flag also reads its REGISTRY_CONFORMANCE_<UPPER_SNAKE> environment variable (for example REGISTRY_CONFORMANCE_FORMAT); --run-id additionally falls back to CI_JOB_ID. Run registry-conformance run --help for the full flag reference — the generated help, not this table, is the authority when they disagree.

Testing a remote repository

A remote repository serves no content of its own: it proxies one upstream, caches what it serves, and refuses every write. Because every read the suite makes has to read something it put there, a remote run needs a second, writable repository to seed into:

registry-conformance run \
  --format=maven \
  --repository-kind=remote \
  --registry-url=https://registry.example.com/remote-repo \
  --upstream-url=https://registry.example.com/fixtures-repo \
  --credential basic:env:CONFORMANCE_USER+env:CONFORMANCE_TOKEN

Point --registry-url at the remote repository and --upstream-url at the repository it proxies. Both must already be linked; the suite does not configure them. If you have no writable upstream, pass --upstream-free-only instead: the run then exercises only the absence and write-refusal rows and cannot tell you whether proxying works. Its exit 0 is not proxying evidence; a CI gate that needs a proxying claim must gate on a run with an upstream, not on this one. A --filter or --priority that leaves only those same rows is the same weak green for the same reason, even with an upstream configured: nothing was relayed, so nothing about proxying was established.

One host serving both repositories, on OCI. The example above gives each repository its own base URL, which is what Maven and npm always need and why the example uses Maven. OCI does not work that way: /v2/ has to be the first path segment, so an OCI --registry-url can only be the host and the repository lives inside the image name. A registry that serves both repositories on one host therefore needs identical --registry-url and --upstream-url, and tells the two apart by name:

registry-conformance run \
  --format=oci \
  --repository-kind=remote \
  --registry-url=https://registry.example.com \
  --upstream-url=https://registry.example.com \
  --oci-name-prefix=my-org/container/remote-repo/my-org/container/fixtures-repo \
  --oci-upstream-name-prefix=my-org/container/fixtures-repo \
  --credential bearer:env:CONFORMANCE_TOKEN

The under-test prefix ends with the upstream's, and that is not a typo. A pull-through cache strips its own routing prefix off the incoming name and forwards what is left to the upstream it was configured with, so what is left has to already be the upstream's own path. Give the two repositories sibling prefixes instead and, once an OCI ## Remote row runs, the preflight reads 404 MANIFEST_UNKNOWN against an upstream you seeded correctly, with nothing in the message pointing at the name. S08 §Addressing two repositories on one host property 5 has the rule, and why the suite will not compute the nesting for you.

Identical base URLs are otherwise rejected, and they stay rejected here unless the two prefixes differ. That is not pedantry about the flags: one prefix on both sides means the suite seeds its fixture into the repository it is testing, so every relay row passes without the proxy ever contacting the upstream. The run reports green and establishes nothing. If you see upstream-url: must differ from registry-url, or --oci-upstream-name-prefix must be set and differ from --oci-name-prefix, that is this check, and setting the second prefix is the fix.

This invocation is currently accepted, not exercised: no OCI ## Remote row has shipped yet, so the run executes the hosted catalog against --oci-name-prefix and never reads --upstream-url or --oci-upstream-name-prefix. Its exit 0 is not proxying evidence, for the same reason an --upstream-free-only run's is not. A CI gate that needs a proxying claim must wait for the remote rows.

Reusing a run ID (--run-id, REGISTRY_CONFORMANCE_RUN_ID, or Config.RunID) against the same pair of URLs silently degrades the run. Coordinates embed the run ID, and a remote cache holds what it has fetched, so a second run with an already-used ID reads a warm cache. What that does depends on the upstream's duplicate policy: under GitLab's defaults OCI's seeding push is idempotent and Maven accepts the duplicate publish, so the relay rows pass without the proxy ever contacting the upstream (a silent green), while npm answers 403 to the duplicate publish and the run exits 2 on the first seeding row with a credential-shaped reason. Against an upstream that refuses every duplicate publish row-attributably instead, which is npmjs.org's 409 or a GitLab Maven upstream running maven_duplicates_allowed=false, every seeding row skips and the run exits 2 reporting that it established nothing. That third outcome does not apply to --format=oci under GitLab's defaults, whose seeding push is idempotent there; an upstream that answers 409 on the blob upload path (S07 reserves it for BLOB_UPLOAD_INVALID) reaches it like the others. Let the suite generate the ID, or vary it per invocation.

The run credential reaches more than one host. Use a minimally-scoped credential: read on the repository under test, write only on the upstream project. Nothing needs more than that, and three things reach further than the single --registry-url you might expect:

• --upstream-url receives the same credential, on every seeding write. Do not point it at a registry that should not have it.
• The repository under test receives it on every request. It is the party being tested, so it is the least-trusted one; a target that logs or echoes Authorization would obtain a credential that can write to your upstream.
• For OCI, a 401 from either base URL is answered by a token request to the realm host that response advertises, carrying the credential. That is the standard Docker token-auth flow and it is how docker pull behaves, but note that the realm host need not be either of the two you passed, and --allow-redirect-host does not gate it. Only https realms are used. Choosing a base URL is choosing the realm it advertises, which matters most when the upstream is a shared repository someone else configured.

Credential spec

--credential takes a <kind>:<spec> value. bearer (token) takes exactly one source; basic (username/password) takes exactly two joined by +, username first. none, or omitting the flag, runs unauthenticated. Sources are:

Examples:

--credential basic:env:MAVEN_USER+env:MAVEN_PASSWORD
--credential bearer:file:/run/secrets/npm-token
--credential bearer:stdin

Library usage

pkg/conformance exposes the suite as a Go API. Pick a per-format module and call RunModule with a Config:

import (
    "gitlab.com/gitlab-org/ops/registry-conformance/pkg/conformance"
    "gitlab.com/gitlab-org/ops/registry-conformance/pkg/conformance/oci"
)

report, err := conformance.RunModule(ctx, oci.NewModule(), conformance.Config{
    Format:      "oci",
    RegistryURL: "https://registry.example.com",
})

RunModule returns a *Report with per-test results; it writes no report output — no stdout, no files. Rendering (human-readable stdout, JUnit XML) lives in the CLI, which also maps the (Report, error) pair to an exit code. A SettleTimeout of 0 (the zero value, as above) disables polling; the CLI's --settle-timeout defaults to 30s, so a library consumer against an eventually-consistent registry must set it explicitly. docs/specs/S04-contracts.md pins the CLI/library contract.

Documentation

Contributing

See CONTRIBUTING.md. For setup, follow the developer setup guide.

License

MIT © GitLab, Inc.