conformance/

directory
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 13, 2026 License: MIT

README

Manual conformance

These manifests authorize nothing by themselves. Agents must execute a named real-provider or GitHub scenario only after the user explicitly requests it. Outside the tagged release workflow, the executable refuses CI and requires --allow-live plus fresh named-scenario authorization. --release-ci permits only the expected repository's release.yml, stable version-tag push, clean matching harness build and explicitly named nonpublication scenario. PR/main CI and the deterministic standalone pre-release workflow have no live credentials. Publication additionally requires --allow-publication, an exact existing --disposable-repository owner/repo, and matching configured policy. It never creates repositories, refreshes login, merges, or deletes remote data.

Build separately with go build -o bin/workflow-conformance ./cmd/workflow-conformance. Run from an intentionally private evidence location outside this repository:

workflow-conformance --scenario /engine/conformance/scenarios/native-claude-smoke.json \
  --config /private/providers.yml --role claude --allow-live \
  --engine-commit ENGINE_COMMIT --evidence /private/conformance-evidence

Use the packaged matching --engine binary for nested/CLI/MCP scenarios. CLI smoke and session scenarios call their selected provider adapters directly. Managed scenarios use a conformance controller with registered Claude actions and the shared production controller/worker/broker wiring at capacity one. Native agent/run remains restricted to Codex definitions. Mixed workflows verify both Claude and Codex physical invocation identities and versions. Workflow scenarios clone an explicitly supplied local --repository-path independently and require --workflow relative to that clone. The mixed-provider template is workflows/mixed-development.yml in this directory; copy it into the disposable fixture repository and add native builder.toml and reviewer.toml Codex definitions; the registered Claude action selects the admitted claude SDK role. Publication uses the existing development workflow and matching allowlisted policy in an explicitly supplied disposable repository. Container owner-loss requires a container role with a long-running approved shell tool; only the newly launched owned controller process is killed, after observing provider usage. The release orchestrator supplies --runtime-control for its own disposable daemon/service. It verifies outage uncertainty, restart, cleanup recovery and rejected stale ownership, and emits both runtime-recovery records. It never stops the runner's default Docker daemon or uses shared Podman storage.

Session tests verify stable identity/context. Interrupted tests retain partial usage. Discovery tests require managed policy and use native hook/MCP canaries. Nested tests verify a child/grandchild tree at capacity one. The external MCP uncertain-response test hosts a local effect fixture and drops its response; passing requires one effect and needs_attention. Container tests require an already available digest-pinned compatible image; none is pulled implicitly. Unsupported upstream policy remains a failed gate, never a backend substitution.

Claude SDK roles can use authentication.mode: native with the operator's own CLAUDE_CODE_OAUTH_TOKEN from claude setup-token; credential-env can select a different token variable. An API endpoint is required only for mode: gateway. SDK native authentication keeps managed settings/discovery isolation. Reports record the selected mode; the historical claude-gateway-structured scenario ID also exercises structured output through native authentication.

Evidence records build/commit, OS/architecture, provider versions, runtime/image, configuration fingerprint, checked assertions, usage completeness/reported cost, and private artifact references. SQLite state and raw provider transcripts stay private. Commit only reviewed sanitized evidence summaries; no credentials or raw private transcripts belong in this repository. Failed/unverified assertions stay false. Protocol observations useful as regressions should be redacted into deterministic fixtures before committing them.

Shared network runtime

The Linux/rootful Podman and normal-Chrome HTTP/2 check is a separate explicit opt-in: node conformance/network-runtime/acceptance.mjs --run-privileged. It uses uniquely owned fixtures, no models and no GitHub publication. See network runtime verification.

The separate production Codex plugin experiment uses installed Codex, production assets and the real Go helper with disposable configuration/trust, local fake Responses/API services and no external route. Its root-authenticated control/status fixture is fake; the Podman harness above separately verifies the production helper against an actual prepared runtime. Hook skip/failure/bypass remains a routing limitation of the plugin. The independent BPF LSM socket boundary prevents host Unix API fallback, and admission rejects saved non-Unix container API routes. Deterministic policy fixtures do not certify LSM attachment or syscall interception; the kernel test and Podman/Chrome harness remain separate opt-ins.

Tagged release automation

release_e2e.py installs no login flows and never writes GitHub secrets. The workflow installs the locked Claude CLI, Codex CLI and packaged SDK dependencies, extracts the actual release binaries, then runs seven smoke/session records before the nine remaining provider/runtime records. Each scenario has one attempt and a five-minute deadline; the job has a 120-minute limit. Runtime fixtures load the scanned distribution archive by its exact image ID. Only the sanitized aggregate is uploaded. Credentials, workspaces, journals, transcripts and runtime storage live under an exclusively created runner-temporary root and are removed on completion. See the release runbook for the separate human signature/publication-evidence boundary.

Directories

Path Synopsis
network-runtime
fixture command
Standalone, model-free fixture for the explicit privileged network acceptance.
Standalone, model-free fixture for the explicit privileged network acceptance.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL