e2e/

E2E Tests

End-to-end tests for the entire CLI against real agents (Claude Code, Antigravity, OpenCode, Codex, Cursor, Factory AI Droid, Copilot CLI).

Commands

mise run test:e2e [filter]                          # run filtered (or omit filter for all agents)
mise run test:e2e --agent claude-code [filter]       # Claude Code only
mise run test:e2e --agent antigravity [filter]       # Antigravity only
mise run test:e2e --agent opencode [filter]          # OpenCode only
mise run test:e2e --agent codex [filter]             # Codex only
mise run test:e2e --agent cursor [filter]            # Cursor only
mise run test:e2e --agent factoryai-droid [filter]   # Factory AI Droid only
mise run test:e2e --agent copilot-cli [filter]       # Copilot CLI only
mise run test:e2e:controlplane [filter]              # control-plane commands against production (no agent)
go build ./...                                      # compile check (no agent CLI needed)

Do NOT run E2E tests proactively. They make real API calls that consume tokens and cost money. Only run when explicitly asked.

Structure

e2e/
├── agents/       # Agent abstraction (Agent interface, tmux sessions, concurrency gates)
├── bootstrap/    # CI pre-test setup (auth config, warmup)
├── controlplane/ # Control-plane tests (login, org/project/repo) with their own TestMain
├── entire/       # `entire` CLI wrapper (enable, explain, etc.)
├── exploratory/  # Experimental tests, not run by CI
├── tests/        # Blessed test files (run by CI)
└── testutil/     # Repo setup, assertions, artifact capture

Key Patterns

• Every test uses testutil.ForEachAgent which runs it per registered agent with repo setup, concurrency gating, and timeout scaling.
• All operations go through RepoState (s.RunPrompt, s.Git) so they're logged to console.log.
• Use the entire package for CLI interactions, not raw exec.Command.
• Skip tests pending CLI fixes with t.Skip("ENT-XXX: reason").

Agents see only real agent signals

Whether a git hook may prompt is decided from the agent's process alone, as on a user's machine, so the harness hands agents no shortcut:

• Agent processes and tmux sessions start without ENTIRE_TEST_TTY, CI, GITHUB_ACTIONS or GIT_TERMINAL_PROMPT (agents.harnessOnlyEnv). Build a runner's env with agentEnv, never from bare os.Environ(); NewTmuxSession unsets them itself.
• Headless agents start detached from the runner's terminal (setupProcessGroup), as an IDE or desktop app starts them: on Unix in a new session with no controlling terminal, on Windows in a console with no window (CREATE_NO_WINDOW), which the agent's git and hooks inherit.
• Test repos keep the default commit_linking ("prompt"). Every E2E commit comes from an agent or the harness, so a test fails if any hook logged strategy.CommitLinkingPromptLogMsg (failIfCommitLinkingPrompted). The prompt gives up after a timeout, so without this a misjudged agent shell only shows as a slower commit.
• Vogon behaves like a real agent: in interactive mode it turns off canonical input on its terminal as a TUI does, and it sets no "don't prompt" variable for its git commands.

These rules exist because every one of those shortcuts hid a hang users hit: Windows git commit from Claude Code and Codex desktop blocked on an invisible prompt (#2725, #2726) while E2E stayed green.

OpenCode dependency cache

OpenCode setup shares a version-keyed plugin dependency tree under the system temporary directory. A reusable tree must contain a node_modules directory and regular package.json and package-lock.json files. Incomplete trees are removed and rebuilt in staging; setup validates the staged tree before publishing it and validates any competing process's tree before reusing it. An interprocess lock beside the cache serializes validation, cleanup, installation, and publication; its lock file is retained so waiters keep using the same lock. Cache installs explicitly enable npm lockfile generation even when local npm settings disable it.

Control-Plane Tests

controlplane/ runs the entire binary against the production control plane with no coding agent involved. Its TestMain logs in once per run with entire login --device, completing GitHub sign-in and the device approval in headless Chromium (playwright-go) as the GitHub test user named by E2E_GH_USERNAME / E2E_GH_PASSWORD / E2E_GH_TOTP_SECRET; every test then starts from that session. The account has authenticator-app 2FA enabled on purpose: GitHub skips its emailed new-device verification for 2FA accounts, and the test computes the one-time code from the secret. The CLI's config and token store live in a temp dir outside e2e/artifacts/, which CI uploads.

Tests create real resources named e2e-cp-<timestamp> and delete them in reverse order (repo, project, org) through t.Cleanup. The native-mirror lifecycle test also places a real mirror and removes it; seeding one took 9 seconds when measured against production, so it needs no extra time budget. The test account may own at most three orgs, so a run that is killed before cleanup (package timeout, cancelled job, lost runner) would block later ones; each run therefore starts by sweeping e2e-cp-* orgs older than 30 minutes, and everything under them.

Run it with mise run test:e2e:controlplane [filter]; the task installs the Playwright driver and Chromium on first use.

Adding a New Agent

1. Create agents/<name>.go implementing the Agent interface.
2. Register it in init() with Register(&YourAgent{}).
3. Add a Bootstrap() method for any CI-specific setup (auth config, warmup).
4. Add a RegisterGate("<name>", N) call if concurrency needs limiting.
5. Ensure the agent name is accepted by mise run test:e2e --agent <name>.
6. Add the agent to .github/workflows/e2e.yml matrix and dispatch options.

Environment Variables

Antigravity credentials

Antigravity E2E resolves its auth mode from the environment, in this order:

1. ADC — GOOGLE_APPLICATION_CREDENTIALS set: agy runs against its default (cloudcode-pa) backend with USE_ADC=1, an isolated per-repo HOME, and --project from E2E_ANTIGRAVITY_PROJECT, GOOGLE_CLOUD_PROJECT, or the ADC JSON project_id. This path needs a Gemini Code Assist entitlement on the project (see caveat below). In GitHub Actions it is enabled by the optional ANTIGRAVITY_GOOGLE_APPLICATION_CREDENTIALS_JSON secret.
2. Gemini API key — GEMINI_API_KEY set: agy ≥ 1.1.13 talks to the Gemini API directly with no account session. The harness isolates HOME per repo, writes {"modelProvider":"gemini"} into that home's ~/.gemini/antigravity-cli/settings.json, passes GEMINI_API_KEY (and GOOGLE_GEMINI_BASE_URL if set) through, and scrubs GOOGLE_API_KEY — agy prefers it over GEMINI_API_KEY when both are set. Needs agy ≥ 1.1.25: up to 1.1.24 agy loaded .agents/hooks.json on this route but never executed the hooks (google-antigravity/antigravity-cli#893, still open upstream; the fix shipped silently in 1.1.25 — verified by bisecting the release binaries with a probe hook), so Entire recorded nothing. CI installs the latest release, so this is the default CI mode. The hook probe (E2E_ANTIGRAVITY_HOOK_PROBE=1, log at .entire/logs/agy-hook-probe.log in artifacts) is how to tell "hooks not executed" from "hooks not loaded".

Both isolated modes also pre-trust the test repo (trustedWorkspaces in that settings.json, agy's equivalent of GEMINI_CLI_TRUST_WORKSPACE=true): agy only loads a workspace's .agents/hooks.json for a trusted workspace, so in a fresh HOME a headless run would otherwise do the work with no Entire hooks firing. They also seed cache/onboarding.json as already complete: interactive agy in a never-run HOME shows a color-scheme chooser and a Terms of Service consent before the prompt (headless -p skips both), and the tmux-driven TestInteractive* tests otherwise time out with agy sat on the Terms screen. The runner can also click through both screens if the seed stops being honoured. 3. OAuth — neither set: the developer's real HOME (existing agy login) is used for local runs.

The harness fails fast (no scenario restart) on the walls that don't clear on retry: Individual quota reached / SERVICE_DISABLED / AUTH_PERMISSION_DENIED (ADC/OAuth entitlement), GEMINI_API_KEY environment variable is not set (API-key misconfiguration) and API_KEY_INVALID.

ADC entitlement caveat: agy's default backend (cloudcode-pa.googleapis.com) is a gated private API that cannot be enabled with gcloud services enable (fails with AUTH_PERMISSION_DENIED, subject 110002) — project Owner is not sufficient. ADC only works once a Gemini Code Assist Standard/Enterprise subscription is provisioned on the project; Code Assist licenses attach to user identities, so plain service accounts are not entitled. Prefer the API-key mode unless you specifically need the Code Assist backend.

For local runs, either authenticate agy with OAuth, or:

export GEMINI_API_KEY=...            # API-key mode (agy >= 1.1.13)
mise run test:e2e --agent antigravity TestSingleSessionManualCommit

Debugging Failures

Artifacts are captured to e2e/artifacts/ on every run (git-log, git-tree, console.log, checkpoint metadata, entire logs). Set E2E_KEEP_REPOS=1 to preserve the temp repo — a symlink appears in the artifact dir pointing to it.

Use the debug-e2e skill (.claude/skills/debug-e2e/) for a structured workflow when investigating failures.

Reading artifacts

• console.log — full operation transcript including agent stdout/stderr
• git-log.txt — commit history at time of failure
• git-tree.txt — working tree state
• entire-logs/ — internal CLI logs

Fixing flaky tests

When a test passes on retry but failed once, the problem is usually agent non-determinism, not a CLI bug. Common patterns:

• Agent asked for confirmation instead of acting: The model output contains "Does this look right?" or "Should I proceed?". Fix: append "Do not ask for confirmation, just make the change." to the prompt.
• Agent wrote to wrong path or created extra files: Fix: be more explicit about exact file paths and what not to do.
• Agent committed when it shouldn't have: Fix: add "Do not commit" to the prompt.
• Checkpoint wait timeout: WaitForCheckpoint or WaitForCheckpointAdvanceFrom exceeded deadline. Fix: increase the timeout argument.

To diagnose: read console.log in the failing test's artifact directory. Compare what the agent actually did vs what the test expected.

CI Workflows

• .github/workflows/e2e.yml runs the standard agent suite on pushes to main. Antigravity runs in the standard suite in agy's Gemini API-key mode (see Antigravity credentials).
• For debugging a single test, dispatch .github/workflows/e2e.yml with an agent and the optional test regex. An empty regex keeps the normal suite. The filter also reaches Windows when running Claude; selecting another agent skips the Windows Claude job.
• .github/workflows/e2e-controlplane.yml runs the control-plane tests on pushes to main and on manual dispatch. Runs are serialized and never cancelled mid-flight, because the shared test account's resources are cleaned up by the test itself.

Unix workflows install Codex and Copilot using their official shell installers; Codex prompts are disabled with CODEX_NON_INTERACTIVE=1. Installer downloads use --compressed, and E2E installs retain retries with pipefail. Nightly OpenCode installs deliberately keep npm to avoid the shell installer's unauthenticated GitHub API rate limits. Windows also keeps npm for Codex, Copilot, and OpenCode rather than using Unix shell installers.

The E2E workflow bootstraps agents before testing. ci.yml covers both checkpoint backends with the free canary on Linux, and runs the Vogon canary on Windows (test-canary-windows; roger-roger stays Linux-only); nightly-e2e.yml checks the published nightly installation.