e2e/

End-to-End Tests

E2E tests validate the full MCP server against a real GitLab instance by driving the real cmd/server binary. Build tag: e2e.

There are five modules, answering different questions:

Each tag has to be listed in GO_ANALYSIS_TAGS in the Makefile and in e2eTags in cmd/gen_testing_docs, or the module is invisible to go vet, to golangci-lint and to the generated test metrics. A file behind a tag nothing names is analysed by nothing.

Those five tags are the whole list. Every file under test/e2e/gitlab and test/e2e/internal carries e2e alone, with two exceptions: the package doc.go files carry no constraint, and the harness's race seam pair (server_race.go and server_norace.go) carries e2e && race and e2e && !race, of which a run sees the half its race setting selects. So one compile and one analysis run see all of them, and the runtime a test needs is decided by the package it is in rather than by a tag. The suite this replaced used to be two halves behind a second tag that excluded each other, which meant a single run could only see one half and the Enterprise one needed a compile step and an analysis pass of its own; that half was ported to test/e2e/gitlab/ee and deleted, and both passes went with it.

One build, four packages

The four packages that drive the real binary — test/e2e/gitlab through the harness, and test/e2e/http, test/e2e/stdio and test/e2e/collector through one of their own — each build cmd/server once per process, or drive the build E2E_SERVER_BINARY names. Every make target that runs one of them stages that build first through e2e-server-binary and hands the path over: make test-e2e-ce and its siblings for the rebuilt suite, and make test-e2e-stdio, make test-e2e-http and make test-e2e-collector for the three transport modules, which ignored the variable until they were taught to read it and then went on paying for a compile each. Running a module as go test -tags … stages nothing, which is what CI does for the two push-triggered ones, so there the compile is still per module. A path that names nothing, a directory, or a file that cannot be executed is refused rather than answered with a build of their own, because a typo would otherwise leave the run testing a binary nobody staged; the refusal names the variable, since the path alone does not say where it came from.

What the variable means is the same in all four packages. How it is resolved is not: the harness reads it through the run's settings, which overlay .env, test/e2e/.env.docker and E2E_ENV_FILE on the process environment, while the three transport modules read the process environment alone. A value written into .env therefore reaches the rebuilt suite and not them.

A go test -race run refuses the variable outright. The detector instruments the test binary and nothing else, so each of these packages passes -race on to the build it does itself through a build-tag seam; a staged binary was almost certainly compiled plainly, and driving it would leave the run watching nothing and reporting no race for that reason. .github/workflows/race.yml sets it nowhere, and the modules no longer depend on that staying true.

HTTP transport module

make test-e2e-http

No GitLab and no credentials: the module builds the binary, starts it with the flags each case needs, and stands up a fake instance for the few behaviours that only appear once GitLab answers — the failure budget is charged when a credential is rejected, and deliberately not when the instance is merely unreachable.

It exists because the in-memory suite cannot reach any of this. The handler chain is assembled in package main, so a unit test could not import it and a test that reassembled it would be testing its own copy. Most cases in the module correspond to something that shipped broken: a preflight answered 401, which made --trusted-origins useless in a browser; a throttled GitLab reported as an invalid token; an invalid token relayed upstream on every retry; and an x-mcp-header annotation carrying a prefix the transport also adds.

A second kind of case pins what the go-sdk this module is built against does with this server's answers today, where that behaviour is a defect recorded in docs/development/upstream-bugs.md (session_not_found_test.go for row 68). These are meant to fail on a dependency bump, not on a change of ours: the bump that changes the SDK's behaviour fails in its own pull request, with a message naming the entry and every page that has to change with it. A new pin of that kind belongs here or in the stdio module rather than in a unit test, because what it holds is the SDK meeting the real binary. A defect the server works around is the exception, since the binary no longer shows it: result_type_test.go holds the answer the binary gives for row 66 with its workaround in place, the resultType of a rate-limited tools/call following the revision as a served call's does, and the pin of the SDK's own behaviour is a unit test on a bare SDK server, TestSDK_MiddlewareToolResult_GoesOutUnlabeled in internal/toolutil.

The TestProxy_* cases run a real nginx in Docker in front of the server, and skip when Docker is unavailable rather than modelling one. That layer is not optional detail: a proxy answering OPTIONS itself hides a server that cannot, and a proxy adding its own CORS headers collides with the server's to produce a response curl reports as 200 and a browser refuses outright.

stdio transport module

make test-e2e-stdio

Also no GitLab and no credentials. It builds the binary, starts it with the environment each case needs, and speaks JSON-RPC to it over real pipes.

stdio is this project's primary transport and nothing anywhere drove it until this module existed. The in-memory suite runs in one process, so it has no pipes, no separation of stdout from stderr, no process to exit, and no environment-variable configuration, since HTTP mode takes flags instead. Apart from the go-sdk pins described under the HTTP module (listen_end_test.go holds row 10 here, and elicitation_declared_version_test.go row 11), every case corresponds to something that shipped broken and was found by hand against a binary: a nil dereference that killed the process on an ordinary eliciting tool call; a keepalive ping that closed the session of any client speaking 2026-07-28 after 45 idle seconds, held in place by a unit test asserting the ping ought to be there; an exit status of 1 on every normal shutdown; and a log stream whose steady state was entirely the SDK's own session chatter.

Collector module

make test-e2e-collector

No GitLab and no credentials either, but Docker is required: it starts a pinned otel/opentelemetry-collector-contrib with an OTLP receiver and file exporters, points the real binary at it, drives traffic, and asserts on the OTLP JSON the collector wrote after parsing what it was sent. Without Docker it skips, with the reason, rather than modeling a receiver.

It exists because every other telemetry test in this repository is graded by code we wrote. The in-process receiver in test/e2e/http answers 200 to whatever it is handed and stores the bytes, which is right for the two questions it asks (was the collector credential sent, did anything private leak) and is no evidence that the export is well formed. A malformed protobuf, a resource missing an attribute a backend requires, or a metric whose unit contradicts its name all pass a stub and all ship telemetry an operator cannot use. So this module asserts acceptance and shape, and deliberately repeats none of the credential or privacy assertions.

Unlike the two transport modules it is not in the push-triggered CI jobs, because it pulls a container image. It belongs with the Docker-mode targets.

Rebuilt suite (test/e2e/gitlab)

make test-e2e-ce        # ephemeral GitLab CE: the common and ce packages
make test-e2e-ee        # ephemeral GitLab EE with the cached license: the common and ee packages
make test-e2e-gitlab    # a self-hosted instance from .env; a package the instance cannot serve skips
make e2e-clean-orphans  # delete what runs started two hours ago or more left behind

The suite. Every test drives the real cmd/server binary over stdio through test/e2e/internal/harness, names its actions by canonical catalog ID, and runs each scenario on the dynamic, meta and individual surfaces as subtests. The runtime a test needs is decided by its package: common runs on every runtime (Free actions are verified on the CE catalog and on the licensed EE one, where schema pruning differs), ce holds the few facts that only hold without a license, and ee needs a Premium or Ultimate one. A package pointed at the wrong runtime refuses before it writes anything, naming what it found and the target to run instead; E2E_RUNTIME_MISMATCH=skip turns that into skips.

Assert a refusal's wording through assertMentions, mentionsAny, containsAny or the contains of harness.ExpectToolError whenever the substring names a tool, an alias or an action ID: make check-action-ids (cmd/audit_action_ids) reads the substrings those helpers are handed and no other call, and holds each one to the catalog the way it holds the hint it quotes. A tool name, a registered alias or a dotted ID nothing resolves fails the push rather than the next licensed run, which is how issue 901 was found a month late; the excuses are the hint rule's own declaration tables, consulted without keeping any entry alive (a gitlab_-shaped token that is no tool, hintToolExemptions; a dotted token that is no ID, proseExemptions, and the params. class, proseNonDomains), plus one more: an alias a Usage line names by design, declared in declaredAliasMentions, which a test quoting that line quotes faithfully. A bare strings.Contains on served text is outside the gate: the suite uses it for needles that name nothing the catalog holds, a status code (403) or one of the server's fixed phrases (unknown action, confirm=true), and a quotation of a tool or an action written that way would go unread. Quote the canonical action ID the server writes, never a tool name, and assert each fact on its own rather than as one member of an either-or check whose other member always matches. A new assertion helper is read only once it is declared in servedTextAssertions, and a renamed one fails the gate until its entry follows it.

The read and preview sweeps, the resource and subscription sweeps and the prompt sweep bind what they call from one shared, read-only World (fixture.SharedWorld), built once per package and checked by a digest at the end of the run. Its core (a group, a project, a branch with a commit, a merge request, an issue, a label, a milestone) is what every read test stands on, so a core object the instance will not make fails the run. Beside it the World makes its extras best effort, each under its own bounded context: a tag and its release, an environment and a deployment into it, a feature flag, a deploy key, a project board, a project and a personal snippet, a wiki page, a group label and milestone, and a canceled pipeline with its one job, from a configuration whose workflow rule admits only API-created pipelines and whose job asks for a runner tag no runner carries, so an instance with a runner and one without end with the same pipeline. An extra the World cannot make (a pipeline that never reached a terminal state, a disabled wiki's 403) is left unbound with its reason. A name that means one object across the catalog (pipeline_id, job_id, tag_name, environment_id, deployment_id, deploy_key_id) is a plain binding, and the read, preview, resource and subscription sweeps log an unmade extra's reason beside every resource template naming it and every action requiring it. One that does not (board_id, a feature flag's name, slug, snippet_id, and the group templates' label_id and milestone_iid) is bound per resource template only, so the reason is logged beside that template alone: the disabled wiki's 403 appears beside the wiki template, and an action requiring a wiki's slug, a board_id, a feature flag's name or a snippet_id is not bound from the World at all and is logged as naming a parameter the World has no binding for. Outside the group templates, label_id and milestone_iid bind the project's label and milestone for every action, group actions included. Two more names are bound per template, beside the plain bindings: the release template's tag_name stands on the release rather than the tag, and the file template's path is the README the project was created with, which is always bound and so never logs a reason. The release domain's actions that address an existing release bind tag_name to the release as well, so an instance with the tag and no release logs the release's reason beside the release template and beside release.get, release.link_list, release.update and release.delete rather than reading or previewing them against a tag with no release. release.link_get and the release link mutations address the release the same way and also need a link id, the links, or a name and a URL, which the World never binds, so they are logged as naming a parameter the World has no binding for whatever became of the release. release.create keeps the tag, since its tag_name is the tag a new release will stand on, and so do the tag actions. One template is gated on an extra none of its variables names: the latest pipeline template reads whatever pipeline the project holds, so it is left unbound with the pipeline's reason only when GitLab created none, and stays bound when the World's pipeline was created and never settled. A subscription counts as accepted only once the server acknowledges it, and the subscription sweep subscribes and closes one template at a time on a session of its own, since the server holds at most ten watchers per credential. It fails on a decline its knownDeclines table does not name, since one past those is the watcher cap or a first read the World should have satisfied, and on a declared template the server acknowledges or no longer advertises, so an entry goes with the defect it names; a declared template the World could not bind on a run is logged as not judged there.

Coverage is what the server dispatched, in a test that passed, on a named runtime, surface and mode: every child runs with telemetry on and the harness reads its spans, and the calls are recorded under dist/e2e-calls/<target> for cmd/audit_e2e_coverage. A test's flush waits for the server's own span of every traced call the test made, whatever its method, for at most ten seconds and in practice about one export interval: a resource read, a prompt or a completion is spanned like a tool call and counts as arrived the moment its span lands, though only a span that names the tool or the action it ran is written as a dispatch line. A call made on a context that had already ended carries no trace, since neither transport sends it and no span of it can come. Each session line says whether the server's span of any of its calls arrived (dispatch_observed) and whether it made a traced call at all (idle when it made none; a subscribe on protocol 2026-07-28 carries no trace); the coverage command names an idle session apart instead of letting it hold its surface and mode unobserved, though a row whose sessions were all idle reads dispatch_observed false, with idle_sessions equal to sessions, since nothing about its telemetry was seen. Neither flag decides any credit, since a tool call is credited on the action its own span named. The shard lines carry schema version 2, which is when dispatch_observed came to count a span of any method and idle was added; a shard of version 1 is refused rather than folded, and a run recorded before it is re-recorded. The coverage command's -baseline alone still reads one, dropping its session lines, since it compares only what the calls credited and the old suite's baseline can never be recorded again. GitLab state is built through client-go by test/e2e/internal/fixture, never through the server under test, so fixture traffic is never coverage. Every file under test/e2e/gitlab and test/e2e/internal carries exactly //go:build e2e (doc.go carries none, and the harness's race seam pair carries e2e && race and e2e && !race), so one compile and one analysis run see the whole suite; the harness's own tests hold that, and hold the runtime packages to never assembling a server of their own and never taking an Env from harness.NewDetached, which skips the bootstrap and exists so the fixture library's tests can drive a whole builder against a stub GitLab. The Docker lifecycle both targets share lives in test/e2e/scripts/run-docker-e2e.sh, and every run carries -p 1 -count=1: capability locks are process-local, and a cached PASS records no calls. The licensed run stays local; CI runs common and ce in the e2e-gitlab job of e2e.yml, which blocks and which release.yml reaches by calling that workflow.

A complete run

make test-e2e-ce and make test-e2e-ee are the complete runs: the ones whose record says what the suite covers, and so the ones nothing may quietly leave out. The two runs of de1ab3b49 ended with eleven skipped tests in seven test functions on CE and ten in five on EE, and nothing failed or even warned about any of them: the CE run passed, and the EE run failed for reasons that had nothing to do with its skips (issue 1014 listed five and three of those functions; the gate below found the rest). Five things make a run complete now, and a sixth stops any run, complete or not, against a GitLab the fine-grained action table has fallen behind.

• The images are the ones their tags name today. docker compose up pulls only a missing image, so a host kept testing the release it first pulled, five weeks old by then, and every scenario gated on a newer GitLab skipped. Before compose starts anything, run-docker-e2e.sh asks the registry which image the GitLab and runner tags name, without pulling (docker buildx imagetools inspect, which answers Docker Hub anonymously), and compares that digest with the local image's RepoDigests. It pulls only when they differ, and then removes the image the pull replaced, by its ID, unless a tag still names it or a container was created from it; it says which. A missing image is left to compose, a reference pinned by digest asks nothing, and a registry that cannot be reached keeps the local image with a warning rather than failing the run; the question is bounded by timeout(1) at sixty seconds where the host has it, so a registry on a path that drops packets is a warning too rather than a hung run. The pull itself has no deadline, since the registry has just answered and a GitLab image is gigabytes a slow link can legitimately take a long time over. GITLAB_IMAGE still pins a release (a pinned tag is checked the same way) and GITLAB_RUNNER_IMAGE does the same for the runner. The run prints the release it tested, read from /api/v4/version rather than off the tag, before the tests and into the saved output. The decision lives in test/e2e/scripts/docker-images.sh, which scripts/e2e_docker_images_test.py drives with a stand-in docker answering what the daemon and Docker Hub answered for the images the issue found stale. The other images the stack starts (the LDAP and Bitbucket fixtures, the nginx fixture service) are fixtures whose version decides nothing a scenario asserts, and are left to compose.
• The scenarios that call a public URL run. The GitHub, Gists and Bitbucket Cloud importers need E2E_EXTERNAL_NETWORK=true, which the harness keeps opt-in because such a test fails for reasons that have nothing to do with this server. A complete run turns it on unless the caller set it, since the host running one has Internet, and looks for the caller's value where the harness would have: the environment and the make command line, then the file E2E_ENV_FILE names, then the repository .env. A value written in .env is therefore honoured, which a default put into the process environment would have outranked. The credentials those scenarios also need are read from the repository .env as before.
• Every fixture starts on both runtimes. The Bitbucket Data Center fixture used to start under CE only, to keep a second JVM of about a gigabyte off the licensed run, and the Bitbucket Server import skipped on EE for it. The import is a Free action the common package drives on either runtime, so both targets start the fixture now.
• A run that cannot be complete is refused before it starts. With E2E_GATE_SKIPS=true, run-docker-e2e.sh refuses to start when the caller turned E2E_EXTERNAL_NETWORK off or left the Bitbucket fixture out with E2E_BITBUCKET=false, naming which: either would skip scenarios nothing declares, and the gate would fail the run an hour later for a reason known before it began. An offline host runs make test-e2e-ce E2E_GATE_SKIPS=false, which is not a complete run: its skips are held to nothing and its record says nothing about what the suite covers.
• A skip nobody declared fails the run. After the tests, run-docker-e2e.sh hands the go test -json stream it just wrote to cmd/audit_e2e_coverage -check-skips, which fails on every skipped test that no entry of cmd/audit_e2e_coverage/skip_declarations.go covers for the run's runtime, and on every entry for that runtime no skip matched. An entry names the test, or a test whose every subtest it covers, a fragment its printed reason must contain, a category and a reason, so it excuses one skip for the reason it gives and for no other. The gate runs whatever the tests did and decides the status only of a run whose tests passed, so a failing run still prints its undeclared skips and is never reported as a skip finding. The targets turn it on with E2E_GATE_SKIPS=true unless the caller set it; the model evaluation targets share the script and do not. The table is empty. It held two entries, the saved view lifecycle on each runtime, whose create GitLab answered with a 500 on 19.3.0 and 19.3.1-ee, until the teardown's copy of GitLab's exceptions log (e2e-<runtime>-gitlab-exceptions.json, taken before the volume holding it goes) named the exception on 19.4.1: the step that subscribes a user to a view locks the caller's user row while sign-in tracking has left unsaved changes on it, which every create and subscribe made with a token meets, and the create has already saved the view by then (upstream bugs, entry 72). Nothing a client sends avoids it, so the scenario asserts what it leaves: the view in the listing after a failed create, and nothing recorded after a failed subscribe.
• A GitLab newer than the fine-grained table stops the run before the tests (issue 1155). The table the server judges a fine-grained grant by (internal/tools/actiongrants/table_gen.go) records one release (its Version), and the grant is judged only on that release's major.minor and on the prerelease of the release right after it; anywhere else every fine-grained session runs in phase A, and the scenarios that hold what a grant lists and may call would pass without checking either. So once the instance answers its version, run-docker-e2e.sh asks cmd/audit_e2e_coverage -check-release (through test/e2e/scripts/release-gate.sh) before go test starts, on every run and not only a complete one, and a release newer than both stops the run with exit 1 after the boot, the stack torn down as usual. The message names the release the instance runs and the one the table records, and the way back: make gen-api-live (it boots gitlab/gitlab-ee:latest; go run ./cmd/gen_api_live/ -image <image> boots another release) rewrites docs/development/gitlab-api-live.json, make gen-action-grants rewrites the table, docs/development/action-requests.json and the fine-grained permissions reference (site/src/content/docs/reference/fine-grained-permissions.mdx and its Spanish twin) and writes nothing while the derivation has a finding, which is where a release that changed an authorization shows up; commit both and start the run again. Meanwhile GITLAB_IMAGE pinned to the release the table records runs, and the message spells the two images (gitlab/gitlab-ce:<release>-ce.0, gitlab/gitlab-ee:<release>-ee.0). A release older than the table is that kind of pin rather than a stale table, so it runs, in phase A, and the gate and the harness each say so in one line; a version the instance did not answer, or answered in a shape that does not start with a major.minor as GitLab writes one (two numbers, neither with a leading zero), stops the run with exit 2 rather than passing it. The harness asks the same rule (internal/testutil/e2erelease) in its environment setup, so make test-e2e against a self-hosted instance stops with the same words before anything is prepared on it, as a fatal refusal that E2E_RUNTIME_MISMATCH=skip does not turn into skips: the harness bootstraps on the first test that asks it for the instance, that test fails with the refusal, every other one skips with a pointer to it, and the package exits 2. The release workflow's E2E gate runs e2e.yml on gitlab/gitlab-ce:latest, so no release can be cut while the table is behind the GitLab a user installs; that is intended, and the remedy is the re-record above rather than pinning the gate's image. The model evaluation targets share the script and the harness and stop the same way.

A scenario that cannot run is therefore a decision someone wrote down, not a silence. Two that used to skip on CE assert instead: TestTierPin_DetectionAgreesWithTheRuntimeProbe holds the server to Free on a Community Edition image, which can hold no license, and TestWorkItems_Lifecycle_IssueTypeCreateListGetUpdateDelete pins the refusal client-go's work item documents cause there (entry 45 of docs/development/upstream-bugs.md), so the day a client-go release with the fix is taken here, it fails and says to run the whole lifecycle on CE again.

The CI job, e2e-gitlab in e2e.yml, is not a complete run and sets neither switch: it leaves the Bitbucket fixture out and holds none of the importers' credentials by design, and it keeps the -check floors. It pulls every image fresh, a hosted runner having none cached.

Go statement coverage of the binary

make test-e2e-ce COVER=1     # or test-e2e-ee, or test-e2e-gitlab
make test-e2e-stdio COVER=1  # or test-e2e-http, or test-e2e-collector
make e2e-go-coverage         # merge, then the per-package table and the total

A second and different coverage question. cmd/audit_e2e_coverage says which catalog actions dispatched; this says which statements of the program ran: the startup path, the catalog build, the middleware chain, the error branches no live-GitLab scenario reaches. Nothing else in the tree asks it.

COVER=1 builds the shared binary with -cover and gives every server child a GOCOVERDIR under dist/e2e-cover/<target>, through E2E_COVER_DIR. Every run target reads it: the three that drive the rebuilt suite against a GitLab, and test-e2e-stdio, test-e2e-http and test-e2e-collector, which are the only thing in the tree that exercises the transport and so the only run that can measure it. Two names, the way the calls directory has two: E2E_COVER_ROOT is the make variable naming the parent make e2e-go-coverage searches, and E2E_COVER_DIR is the leaf one run is handed, exported to the harness, to the three transport modules and to the Docker script. The exported one must be absolute, like the calls directory and for the same reason (a child runs in a working directory of its own), and each target clears its own leaf first, the Docker script doing it for the two Docker targets since CI invokes that script directly.

What the variable means is the same in all four packages and how it is resolved is not, exactly as with E2E_SERVER_BINARY: the harness reads it through the run's settings and the three transport modules read the process environment alone. Each of them keeps its own reader (test/e2e/{stdio,http,collector}/coverage_test.go) because test/e2e/internal/harness is behind //go:build e2e and nothing compiled under another tag can import it, and a package all four could import would have to carry no build constraint at all, which is the one shape the gate over test/e2e/internal refuses. A copy that drifts is not silent: its target's leaf stays empty, and make e2e-go-coverage refuses an empty read rather than reporting zero percent.

A binary built with -cover writes one meta-data file when it starts and one counter file when it exits normally, which is why every harness here asks its children to stop and waits before it cancels their context: the cancel is a kill, a killed process runs no exit hook, and everything it executed is missing from the profile. The stdio module closes the client's end of stdin, the HTTP one sends the termination signal it already uses for its shutdown cases, and the collector one sends os.Interrupt, which also lets the server flush its last telemetry batch. The rebuilt suite's run ends with the count of counter files against the count of children started, which is where a child that died anyway is visible.

CI runs the gate instrumented, with no switch, which is the one place -cover is not opt-in. The blocking e2e-gitlab job is the only run of this suite that happens on every push, so a figure conditioned on anything would be a figure nobody ever sees; what it costs is a gate whose binary is not byte-identical to the one the release ships, and the instrumentation does nothing of its own, only counting statements and writing the tally at exit. A developer reproducing a CI failure with make test-e2e-ce COVER=1 therefore drives the same binary CI did. The transport modules are the other way round: CI runs them as go test -tags …, which stages no binary and so builds no instrumented one, and their share of the figure comes from a local make test-e2e-http COVER=1 and its siblings.

make e2e-go-coverage merges every directory holding coverage data, so a CE run and a licensed one fold into one figure, which is the honest one: a statement only the licensed catalog reaches was still reached. It writes dist/e2e-reports/e2e-go-coverage.out, prints the per-package table and the total, and fails rather than reporting zero when there is nothing to read, which is the shape of an uninstrumented build. Nothing is committed, no floor is applied, and the profile is never merged into coverage.out, which Sonar publishes as the unit suite's figure.

Quick Start

Self-Hosted Mode

Requires a running GitLab instance with a Personal Access Token that has create/delete project permissions.

# Create .env in project root
cat > .env <<EOF
GITLAB_URL=https://gitlab.example.com
GITLAB_TOKEN=glpat-...
# Required when running webhook/custom-emoji tests outside Docker mode.
# Must be reachable from the GitLab instance, not just from the test process.
E2E_FIXTURE_URL=https://fixture.example.com
# Optional when GitLab must reach itself through a different URL for push mirror tests.
E2E_GITLAB_INTERNAL_URL=https://gitlab.example.com
EOF

# Run
make test-e2e-gitlab

Docker Mode

Uses an ephemeral GitLab CE container plus a Bitbucket Data Center import fixture. Requires Docker and the memory for them: the GitLab container runs four Puma workers (puma['worker_processes']), peaks at about 7.5 GiB in a complete CE run and 10 GiB in an EE one, and is capped at 12 GiB with no swap beyond it (mem_limit and memswap_limit), and the runner and the fixture services, Bitbucket among them, need memory beside it. Measured on 2026-10-04 against the GitLab 19.4.1 images, CE and EE.

All Docker infrastructure is version-controlled in this directory:

• docker-compose.yml — GitLab CE + Runner + Bitbucket + fixture service definition
• scripts/setup-gitlab.sh — Creates test user, PAT, generates test/e2e/.env.docker
• scripts/setup-bitbucket.sh — Provisions the Bitbucket import fixture, appends BITBUCKET_SERVER_*
• scripts/register-runner.sh — Registers CI runner
• scripts/wait-for-gitlab.sh — Polls readiness endpoint
• scripts/docker-images.sh: holds the GitLab and runner images to the images their tags name today, pulling only a newer one, and reads the release the run tested
• scripts/skip-gate.sh: hands a complete run's go test -json stream to the skip gate, and settles what a complete run is started with or refused for (see A complete run)
• scripts/gitlab-exceptions.sh: copies GitLab's exceptions log into the reports before the teardown deletes it

The manual steps below start whatever images the daemon holds, like a bare docker compose up; run-docker-e2e.sh, which every target runs, checks them against the registry first.

All commands run from the project root. The Bitbucket fixture needs an ephemeral admin password shared between compose and the provisioning script (make test-e2e-docker generates one per run automatically):

export E2E_BITBUCKET_ADMIN_PASSWORD=$(openssl rand -hex 16)
docker compose -f test/e2e/docker-compose.yml --profile bitbucket up -d
./test/e2e/scripts/wait-for-gitlab.sh
./test/e2e/scripts/setup-gitlab.sh
./test/e2e/scripts/register-runner.sh
./test/e2e/scripts/setup-bitbucket.sh

set -a && source test/e2e/.env.docker && set +a
go test -v -tags e2e -p 1 -timeout 2700s ./test/e2e/gitlab/common/ ./test/e2e/gitlab/ce/

# Cleanup
docker compose -f test/e2e/docker-compose.yml --profile bitbucket down -v

Or use the Makefile target:

make test-e2e-docker

Running the fixture on another host

The containers need about 5 GB and a few cores for ten minutes of boot, which is a poor fit for a busy workstation. Docker follows DOCKER_HOST (or the active context), so the whole fixture can run elsewhere while the suite runs here; the Makefile only has to know where GitLab is reachable from this machine:

DOCKER_HOST=ssh://truenas \
E2E_DOCKER_GITLAB_URL=http://192.168.0.40:8929 \
make test-e2e-docker          # or test-e2e-ce, test-e2e-ee

E2E_DOCKER_GITLAB_URL is also handed to the container as its external_url, so the web_url fields GitLab answers with name the address the tests reach. The registry and Bitbucket follow the same host: test/e2e/scripts/run-docker-e2e.sh reads the host out of that URL whatever port or path it carries, gives the registry http:// that host on port 5050 as its external URL, and looks for Bitbucket at http:// that host on port 7990, printing the address it derived. It publishes the Bitbucket port on the loopback the Bitbucket URL (the derived one, or E2E_DOCKER_BITBUCKET_URL when set) names: 127.0.0.1 for localhost, whatever its case, and the address itself for a 127.x.y.z address and for [::1], since a port published on one loopback address refuses a connection to another. A host name is resolved on this machine, and one that resolves to loopback addresses alone (a Debian hostname mapped to 127.0.1.1, an /etc/hosts alias of 127.0.0.1) is published on that loopback, its first IPv4 one or [::1]. Any other URL is published on 0.0.0.0, since the setup script on this machine has to reach a remote container's port: a name that also resolves to another address, one that resolves to nothing here, and a name that only begins with 127., which is never read as the address it is spelled like. A URL naming this machine by its LAN address is published on 0.0.0.0 too, and overriding E2E_DOCKER_BITBUCKET_URL moves the bind with it unless E2E_BITBUCKET_BIND is set as well. The import test itself never dials Bitbucket: GitLab does, over the compose network, at the address the setup script recorded. E2E_REGISTRY_EXTERNAL_URL, E2E_DOCKER_BITBUCKET_URL and E2E_BITBUCKET_BIND override each. Before the derivation a remote run had to set both by hand, and one that did not still passed: the import test skipped and the coverage record came out one action short. The published ports are open on the remote host's network: use it on a LAN you trust.

Docker Enterprise Mode

Enterprise mode uses the same Docker topology with the EE image and a local Ultimate subscription. Store a 24-character activation code in .env as ENTERPRISE_LICENSE or GITLAB_ACTIVATION_CODE, or export it in the shell; the lifecycle script passes it to the GitLab EE container during startup. make test-e2e-ee is the licensed run, and make test-e2e-docker-enterprise is the same target under its older name. It runs the common and ee packages of the rebuilt suite against the real binary: there is no Enterprise build tag, and the package decides the runtime, so ee refuses an unlicensed instance before it writes anything while common runs its Free actions on the licensed catalog too. make test-e2e-docker is the CE run under its older name. After a successful activation-code run, the setup script exports the generated license key to test/e2e/.enterprise-license with owner-only permissions. Future runs prefer that ignored local cache and install it through the License API, so they do not need to spend the activation code again. Delete the cache file to force a fresh activation-code flow. Legacy .gitlab-license keys can still be stored in ENTERPRISE_LICENSE; the setup script installs those through the License API without writing the secret into test/e2e/.env.docker.

make test-e2e-ee

Equivalent manual setup, which is what test/e2e/scripts/run-docker-e2e.sh ee does; in Docker mode the harness reads test/e2e/.env.docker itself, and builds cmd/server once per process unless E2E_SERVER_BINARY names a build:

GITLAB_ACTIVATION_CODE="$ENTERPRISE_LICENSE" env GITLAB_IMAGE=gitlab/gitlab-ee:latest docker compose -f test/e2e/docker-compose.yml up -d
./test/e2e/scripts/wait-for-gitlab.sh
GITLAB_ENTERPRISE=true ./test/e2e/scripts/setup-gitlab.sh
./test/e2e/scripts/register-runner.sh

E2E_MODE=docker go test -v -tags e2e -p 1 -count=1 -timeout 3600s ./test/e2e/gitlab/common/ ./test/e2e/gitlab/ee/

env GITLAB_IMAGE=gitlab/gitlab-ee:latest docker compose -f test/e2e/docker-compose.yml down -v

Docker mode enables pipeline and job tests that require a CI runner, and starts an internal fixture service used by webhook and custom emoji tests. The setup script also writes E2E_FIXTURE_URL and E2E_GITLAB_INTERNAL_URL into .env.docker so CI runs all non-EE tests without public Internet dependencies.

What setup-gitlab.sh provisions

Beyond the test user and PAT, the setup script prepares the instance so coverage that used to skip can run:

• Instance settings: enables the github, bitbucket, and bitbucket_server importer sources (GitLab ships with them disabled) and sets default_branch_protection=0 so unprotect-then-push flows cannot race GitLab's asynchronous default-branch protection job.
• Container registry seed: creates the e2e-registry-seed project and pushes busybox:stable with two tags (seed-a, seed-b) through the plain-HTTP registry at localhost:5050, recording E2E_REGISTRY_PROJECT in .env.docker. The push writes its auth into a scratch DOCKER_CONFIG instead of calling docker login, which needs an unlocked keychain on macOS. Without a docker CLI the registry tag tests skip.
• GitLab Pages is enabled in the omnibus config; the Pages write test publishes a real deployment through a pages CI job before asserting.
• Pending schema migration seed: deletes the newest schema_migrations row via gitlab-psql and records its version as E2E_DB_MIGRATION_VERSION, so the mutating db_migration_mark path runs for real — the test's mark call re-inserts the row, leaving the instance as it started. The newest version is used because GitLab squashes old migrations and only recent versions still have a resolvable migration file.

What setup-bitbucket.sh provisions

Both Docker targets also start a real Bitbucket Data Center container (e2e-bitbucket, compose profile bitbucket) so import_bitbucket_server runs instead of skipping, on CE and on EE alike:

• Unattended setup is driven by a bitbucket.properties file written before the stock entrypoint runs, using Atlassian's public 3-hour timebomb Data Center test license — long enough for an ephemeral run, torn down with the stack. Search is disabled and the embedded eval database is used, keeping the container at one ~1 GB JVM.
• Fixture content: project E2E, repository fixture with an initial commit seeded through the browse edit REST API, and an admin HTTP access token for the importer.
• The BITBUCKET_SERVER_* variables are appended to .env.docker with the compose-network URL (http://e2e-bitbucket:7990) because GitLab — not the test process — dereferences it. Provisioning is best-effort: on any failure it warns and the suite falls back to the documented skip.

Optional operator credentials

Tests that talk to external services read the repository root .env (loaded as a fallback in Docker mode; .env.docker values keep precedence) and skip when unset:

Imported projects are registered for permanent deletion in the per-test resource ledger.

When to run a Docker run

Run one when a change touches tool registration, catalog projection or the schemas a surface serves, or the pipeline, job, webhook, custom emoji or mirror behaviour that only a runner and the fixture service exercise. A change to a documentation page, or to a unit test against mocked GitLab responses, does not need one. It is slower than the unit suite, and it is the check to reach for in an architecture migration, because it drives the real GitLab API, the MCP transport and every tool surface together.

Reports, keeping the stack, and what to check

run-docker-e2e.sh writes three files per run under dist/e2e-reports/ (E2E_REPORT_DIR), named after the runtime unless E2E_REPORT_NAME says otherwise: e2e-<runtime>-junit.xml, the go test -json stream e2e-<runtime>-log.json that the skip gate and cmd/audit_e2e_coverage read, and the console output e2e-<runtime>-output.txt, beside the copy of GitLab's exceptions log. The stack is torn down with its volumes whatever the run ended with; E2E_KEEP_STACK=true leaves it up, so GitLab's data and the runner's logs can be read after a failure. Remove it afterwards with docker compose -f test/e2e/docker-compose.yml --profile bitbucket down -v.

Architecture

Layout

A scenario names each action by its typed harness.ActionID and runs once per surface: harness.SurfacesWith builds the fixture once on the parent test and runs the body again on the dynamic, meta and individual surfaces as subtests, so the three are held to the same assertions.

Sessions

A test asks its Env for a session by harness.ServerConfig: a tool surface, a protective mode (default, read-only or safe), a capability surface (full or minimal), a meta parameter-schema mode, and when it needs one a token, an exclusion list, an elicitation policy, a tier pin or the HTTP transport. Every field the server sees is a variable or a flag the released binary reads, and there is no way to hand the server a catalog of the test's own. Three fields configure the harness rather than the server: the elicitation policy, which is what the harness client advertises and does with a request, its Responder, which answers a scripted one, and Private, which is the session's lifetime. Sessions of one configuration are shared by the tests of a package, since nothing one test calls changes what the next is served; Private: true gives a test a server nobody else touches, stopped when the test ends.

The four guided flows (issue, merge request, release, project) are answered by a scripted client on every surface, in a private session per test and surface that ends with the test, with the auto-accept and no-elicitation policies beside them.

A session can run on a fine-grained personal access token (issue 952). fixture.NewFineGrainedToken mints one for a fixture user with exactly the scopes a test writes, through the route GitLab offers a token's own user, since the administrator route takes no grant; fixture.StartupScopes are the four permissions the server needs to learn what the token is (User, Namespace and Personal Access Token: Read for the user, Metadata: Read for the instance), and a scenario adds what it is about to them. The harness reads the token's grant and the instance's version the way the binary does, holds the session's listing to the decision the binary's middleware takes for it, and gives the session its reading (Session.Authority), so a scenario asserts the phase with Session.FineGrainedPhase, which answers what the grant decides (harness.FineGrainedJudgement): the listing and the calls (phase B) where the instance runs the release the action table was recorded from, the listing alone on the prerelease of the release right after it (what a nightly image and GitLab.com report), where every call phase A allows is handed to GitLab, and neither (phase A, naming the recorded release) on a release older than the table, which is how one scenario holds every state and skips none. Which release is which is internal/testutil/e2erelease, written apart from the server's own reading so the two are held to each other, and a release newer than both judged states never reaches a scenario: the run stops before any scenario reaches the instance until the table is re-recorded (see A complete run). A scenario that sends a fine-grained token's request GitLab is to refuse for a project the grant does not cover first waits, with fixture.AwaitProjectAccess, until the user sees that project: until the background job that refreshes the user's authorizations has run, GitLab answers 404 for it rather than the 403 naming the permission. A refusal of a fine-grained session is classified under the server's own reason, fine_grained, and harness.Withheld accepts it on every surface. The session's lines record credential: fine-grained, which cmd/audit_e2e_coverage counts apart from every classic figure. fixture.ProbeREST and fixture.ProbeGraphQL ask GitLab directly what it answers a token, bypassing the server: the direct probes in common/fine_grained_tokens_test.go and ee/fine_grained_graphql_test.go hold each cause the table records to GitLab's own answer.

Resource subscriptions are driven through the real binary too. TestSubscriptions_Sweep subscribes to every subscribable template the World binds, one at a time on a session of its own, and counts one accepted only once the server acknowledges it; TestSubscriptionDelivery_ChangedIssue_NotifiesTheSubscriber changes an issue it owns and waits for the notification.

Safety guardrails

• Names: Env.Name names every object a test creates after the run, the package and the test, so two runs and three packages can share an instance without one deleting another's fixtures.
• Ledger: every fixture builder registers the undo for what it created on the test's Env, and the ledger runs it when the test ends, under a bounded cleanup budget. Projects and groups are permanently removed.
• Orphans: a run sweeps what carries its own run ID at its exit, after every test's cleanup: every project and group the token owns, every personal snippet of the token's user (which no deletion of a project or a group takes along) and, with an administrator's token, every user. A run that was killed never reaches its exit, so what it left is swept by hand with make e2e-clean-orphans, against the instance .env names. That sweep knows no run ID: it reads the one each name carries and the second that run started, and deletes the same four kinds wherever the run started more than E2E_SWEEP_MIN_AGE ago, two hours by default. Each package stamps its own start, and its fixtures are in use no longer than its timeout plus the exit hooks' budgets: the timeout's alarm is a fatal panic, so the hooks run only after tests that ended in time, and then on the exit sweep's 5m and the World teardown's 3m. go test's own backstop only shortens that, with a SIGQUIT a tenth of the timeout past it (never less than a minute, 6m at 3600s) and a kill up to a tenth later. So a run started with the Makefile's default timeouts, 3600s for every package, is never touched while it goes; after a run started with a longer timeout, set E2E_SWEEP_MIN_AGE above that timeout plus 8m (on the make command line, in the environment or in .env, read in that order before the 2h default), and while a run started with -timeout 0 is going, do not run the sweep at all. A self-hosted run going at the same time is not failed by it either: such a run fails at its end when a group or project it saw at its start is gone or renamed, and it leaves out of that check every object whose name carries a run identifier the harness minted (not one an E2E_RUN_ID override named), so an older run's leftover the sweep deletes does not count against it. The listings it cannot search for a shape are read whole, every user on the instance included with an administrator's token. A run started with E2E_RUN_ID carries no date in its names and is never reached that way: the harness sets a letter before any stamp such an override copies, so reusing the ID an earlier run logged cannot date a new run by the old one's start. E2E_SWEEP_PREFIX=<prefix> sweeps by name prefix instead, whatever run and however recent, and it cannot pick one run either. A name opens with the prefix its test or its builder chose (a test's own such as adm-alert- or member-, a builder's such as snippet- and mergetrain-, proj- and grp- when a test names none, and the World's e2e-world- and world-snippet-), never with the run ID, so a prefix picks what some tests named across every run; reaching an E2E_RUN_ID run that way takes one sweep per prefix its tests used, and each also reaches other runs, live ones included. The one prefix applies to all four kinds at once, and an object of the token user's own whose name opens with it is deleted too, as one whose name carries an old run ID is by the default. The prefix is read from the make command line or the environment only, never from .env, on purpose: one left there would turn every later clean into a sweep by name that reaches live runs. The test the target runs skips when neither variable is set, which guards only a bare go test of it.
• Shared state: a test that changes something every other test can observe declares a harness.Lock (instance-global settings, the current user's own state, the CI runner, the license), and the holders of one lock run one after another; harness.Serial is for a change nothing else can run beside, such as installing or removing the license.
• Requirements: harness.Needs declares what the instance must provide (an administrator, a runner, the fixture service, an external network, a licensed tier, or with harness.GitLabAtLeast the GitLab release that first mounts a route the test calls), and a test whose needs the run does not meet is skipped before it touches GitLab. On a complete Docker run such a skip fails the run unless it is declared (A complete run), so a need is a statement about a partial run, such as a self-hosted instance without a runner, and never a way to leave a scenario out of a complete one.

Running Individual Workflows

A surface is a subtest of every scenario rather than a test family of its own, so -run names the subtest:

# One scenario, on all three surfaces
go test -v -tags e2e -p 1 -timeout 600s -run TestIssue_Lifecycle ./test/e2e/gitlab/common/

# One scenario on the dynamic surface alone
go test -v -tags e2e -p 1 -timeout 600s -run 'TestIssue_Lifecycle/dynamic' ./test/e2e/gitlab/common/

# Every scenario, dynamic surface only
go test -v -tags e2e -p 1 -timeout 2700s -run '/dynamic$' ./test/e2e/gitlab/common/

Compile-Only Check

Verify E2E code compiles without needing a GitLab instance:

go test -tags e2e -c -o /dev/null ./test/e2e/gitlab/...  # Linux/macOS
go test -tags e2e -c -o NUL ./test/e2e/gitlab/...         # Windows

Domain Coverage

Core lifecycle: user → project CRUD → commits → branches → tags → releases → issues → labels → milestones → members → upload → MR lifecycle → notes → discussions → search → groups → pipelines → packages → cleanup

Extended domains (meta-tool workflow): wikis, CI variables, CI lint, environments, issue links, deploy keys, snippets, issue discussions, draft notes, pipeline schedules, badges, access tokens, award emoji

Dynamic surface workflow: public tool inventory, find, execute, standalone project discovery, multi-intent discovery, and destructive-action confirmation guard

Docker-only domains: pipeline create/get/cancel/retry/delete, job get/log/retry/cancel

MCP capability tests: elicitation, the four guided flows (issue, merge request, release, project) on every surface, each answered by a scripted client in a private session per test and surface, with the auto-accept and no-elicitation policies beside them