test/

E2E Tests

The E2E Tests make sure we have the repositories getting updated if we have a repo with PAC installed on it.

It will checks if the repository have been updated at the end.

Most E2E tests has this basic flow :

• Create a temporary Namespace
• Create a Repository CR into it
• Create a Branch on a GitHUB repo
• Create a commit with files like pipelinerun inside that branch, the pipelinerun with have the namespace annotation to force the repository match on the namespace we have created and not catching other CR that may matching it.
• Wait that the Repository is updated.
• Some other stuff are done directly on the controller sink, bypassing a bit the GitHUB apis and generating the webhook ourselves.

Settings

here are all the variables that is used by the E2E tests.

• TEST_GITHUB_API_URL -- GitHub Api URL, needs to be set i.e: api.github.com
• TEST_GITHUB_TOKEN -- Github token used to talk to the api url
• TEST_GITHUB_REPO_OWNER - The repo and owner (i.e: organization/repo)
• TEST_GITHUB_REPO_OWNER_GITHUBAPP - A repository/owner github repo that is configured with github apps.
• TEST_GITHUB_REPO_INSTALLATION_ID - The installation id when you have installed the repo on the app. (get it from the webhook event on the console)

Hint: Go to Github Apps (or Settings > Developer settings > GitHub Apps) choose the Github App and go to Advanced > Recent Deliveries and search for installation which looks something like below

  "installation": {
    "id": 29494069,
    "node_id": "MDIzOkludGVncmF0aW9uSW5zdGFsbGF0aW9uMjk0OTQwNjk="
  }

• TEST_EL_URL - The controller public url, ingress or openshfit's route

• TEST_EL_WEBHOOK_SECRET - The webhook secret.

• TEST_GITHUB_REPO_OWNER_WEBHOOK - A repository/owner github repo that is configured with github webhooks and this repo should differ from the one which is configured as part of TEST_GITHUB_REPO_OWNER_GITHUBAPP env.

• TEST_BITBUCKET_CLOUD_API_URL - Bitbucket Cloud Api URL: probably: https://api.bitbucket.org/2.0

• TEST_BITBUCKET_CLOUD_USER - Atlassian account email for Bitbucket Cloud API authentication

• TEST_BITBUCKET_CLOUD_E2E_REPOSITORY - Bitbucket Cloud repository (i.e. project/repo)

• TEST_BITBUCKET_CLOUD_TOKEN - Bitbucket Cloud token

• TEST_GITLAB_API_URL - Gitlab API URL i.e: https://gitlab.com

• TEST_GITLAB_GROUP - Gitlab group/namespace where test projects will be created and deleted

• TEST_GITLAB_TOKEN - Gitlab Token

• TEST_GITLAB_SECOND_TOKEN - Optional second GitLab token from a different user for fork-based GitLab tests

• TEST_GITLAB_SECOND_GROUP - Optional group/namespace where the second user should create forks

• TEST_GITLAB_SMEEURL - Smee URL for forwarding GitLab webhooks to the controller

• TEST_GITEA_API_URL - URL where GITEA is running (i.e: GITEA_HOST)

• TEST_GITEA_SMEEURL - URL of smee

• TEST_GITEA_PASSWORD - set password as pac

• TEST_GITEA_USERNAME - set username as pac

• TEST_GITEA_REPO_OWNER - set repo owner as pac/pac

• TEST_BITBUCKET_DATA_CENTER_USER - Bitbucket Data Center Username

• TEST_BITBUCKET_DATA_CENTER_TOKEN - Bitbucket Data Center token

• TEST_BITBUCKET_DATA_CENTER_E2E_REPOSITORY - Bitbucket Data Center repository (i.e. project/repo)

• TEST_BITBUCKET_DATA_CENTER_API_URL - URL where your Bitbucket Data Center instance is running.

• TEST_BITBUCKET_DATA_CENTER_WEBHOOK_SECRET - Webhook secret

• PAC_API_INSTRUMENTATION_DIR - Optional. When set, E2E tests write per-test JSON reports of GitHub API calls parsed from controller logs to this directory. Useful for analyzing API usage and rate limits. Example: export PAC_API_INSTRUMENTATION_DIR=/tmp/api-instrumentation.

You don't need to configure all of those if you restrict running your e2e tests to a subset.

YAML Configuration File

Instead of setting individual environment variables, you can use a YAML configuration file. Set PAC_E2E_CONFIG to the path of your config file:

PAC_E2E_CONFIG=./test/e2e-config.yaml make test-e2e

Copy the example file and fill in the values for the providers you want to test:

cp test/e2e-config.yaml.example test/e2e-config.yaml
# edit test/e2e-config.yaml with your values

The YAML file groups settings by section (common, github, github_enterprise, gitlab, gitea, bitbucket_cloud, bitbucket_datacenter). See test/e2e-config.yaml.example for the full list of fields.

Environment variables always take precedence over YAML values, so you can use the config file for base settings and override specific values via env vars (useful for generated relay URLs and local overrides). An explicitly set empty environment variable also overrides YAML.

The Go tests load PAC_E2E_CONFIG automatically. The helper scripts in hack/gh-workflow-ci.sh run outside the Go test process and read values from the shell environment, so set the variables you need directly when running them locally.

Running

As long you have env variables set, you can just do a :

make test-e2e

and it will run the test-suite and cleans after itself,

You can specify only a subsets of test to run with :

% cd test/; go test -tags=e2e -v -run TestGithub .

same goes for TestGitlab or other methods.

Running GitLab tests manually

GitLab tests require a smee URL to forward webhooks from the external GitLab instance to your local controller (the same pattern as Gitea tests).

1. Create your own group on the GitLab instance (e.g. https://gitlab.pipelinesascode.com) to hold the temporary test projects. Each test run creates a project inside this group and deletes it on cleanup. Use TEST_GITLAB_GROUP to point to your group.

2. Generate a smee channel and start the gosmee client to forward webhooks to your controller:

   # Generate a new smee channel URL
   SMEE_URL=$(curl -s https://hook.pipelinesascode.com -o /dev/null -w '%{redirect_url}')
   
   # Start forwarding webhooks to your controller
   gosmee client --output json --log-level debug --target-connection-timeout 5 --target-retries 5 "${SMEE_URL}" "https://your-controller-url"
   

3. Set the required environment variables (or use a YAML config file):

   export TEST_GITLAB_API_URL=https://gitlab.pipelinesascode.com
   export TEST_GITLAB_TOKEN=<your-token>
   export TEST_GITLAB_GROUP=<your-group>
   export TEST_GITLAB_SECOND_TOKEN=<second-user-token> # optional, required for real fork fallback tests
   export TEST_GITLAB_SECOND_GROUP=<second-user-group> # optional, recommended when the second user has multiple namespaces
   export TEST_GITLAB_SMEEURL="${SMEE_URL}"
   export TEST_EL_URL=https://your-controller-url
   export TEST_EL_WEBHOOK_SECRET=<your-webhook-secret>
   

   Fork-status fallback tests skip automatically when TEST_GITLAB_SECOND_TOKEN is not set.

4. Run the tests:

   cd test/; go test -tags=e2e -v -run TestGitlab .
   

To clean up stale test projects (older than 7 days) left from previous runs:

./hack/cleanup-gitlab-projects.py        # dry-run
./hack/cleanup-gitlab-projects.py --force # actually delete

./hack/cleanup-github-e2e.py does the same for GitHub and GitHub Enterprise. It closes open PRs and deletes branches whose names were generated by the tests, plus leftover per-test repositories in the GHE webhook org, when they are older than 24 hours. It reads the same TEST_GITHUB_* variables as the tests, runs in dry-run mode unless you pass --force, and runs nightly after the scheduled E2E tests.

If you need to update the golden files in the end-to-end test, add the -update flag to the go test command to refresh those files. First, run it if you expect the test output to change (or for a new test), then run it again without the flag to ensure everything is correct.

Continuous Integration (CI) with GitHub Actions

Our E2E tests are automatically run as part of our CI pipeline using GitHub Actions. This section explains how the CI process works, the components involved, and how to troubleshoot or extend it.

Overview

The E2E test CI pipeline runs on GitHub Actions using a Kind (Kubernetes in Docker) cluster. The workflow is defined in .github/workflows/e2e.yaml and uses helper scripts in hack/gh-workflow-ci.sh. Tests are executed against multiple provider categories to validate functionality across different Git providers.

The CI flow generally follows these steps:

1. Set up a Kind cluster
2. Install Pipelines as Code (PAC)
3. Configure necessary secrets
4. Run tests against different provider groups
5. Collect and store logs

When Tests Run

Tests run on:

• Every Pull Request (PR) that modifies Go files
• As a nightly job at 05:00 UTC to detect regressions
• Manually via workflow dispatch (with optional debug capabilities)

Test Categories

The tests are separated into provider categories (matrix strategy):

• github_public - Public GitHub tests (excluding GHE and concurrency)
• github_ghe_1, github_ghe_2, github_ghe_3 - GitHub Enterprise (GHE) tests
• gitlab_bitbucket - GitLab and Bitbucket tests
• gitea_1, gitea_2, gitea_3 - Gitea tests (split into 3 chunks)
• concurrency - Concurrency-specific tests

This split helps reduce the load on external APIs during testing and provides more focused test results.

Environment Variables

All provider endpoints and credentials live in a single GitHub Secret, E2E_CONFIG_YAML, whose contents are the YAML described in YAML Configuration File (see test/e2e-config.yaml.example for the full schema). The workflow writes the secret to /tmp/e2e-config.yaml with owner-only permissions and sets PAC_E2E_CONFIG. The secret is passed through a step environment variable, not interpolated into shell source. The same step registers every token, secret, password, private key and smee URL from the file with ::add-mask::, since GitHub only masks a multiline secret line by line. The Go tests load the file automatically via TestMain. Shell steps read the values they need with yq and export them only for that step.

Copy and populate the example, then upload it as the single E2E config secret:

gh secret set E2E_CONFIG_YAML < test/e2e-config.yaml

Do not commit the populated file. The GitHub App private keys (github.private_key, github_enterprise.private_key) go in the file as YAML block scalars (private_key: |).

Populate common and the providers you want to test. The job supplies its local controller URL through TEST_EL_URL. Gitea, GitLab, and GHE webhook relay URLs are generated during CI.

Only a few things remain outside the config file:

• Workflow infrastructure set at the job level (KO_DOCKER_REPO, CONTROLLER_DOMAIN_URL, KUBECONFIG, etc.).
• Runtime-generated values set per-step (the matrix TEST_PROVIDER and the dynamic gosmee relay URLs produced during the run).
• CI-operational secrets that are not test credentials: GH_TOKEN (the ok-to-test permission gate) and SLACK_WEBHOOK_URL (failure notifications).

To add a new setting, add a field to the E2EConfig struct in test/pkg/configfile/config.go, document it in test/e2e-config.yaml.example, and add it to the E2E_CONFIG_YAML secret. Configuration tests run with make test and in the GitHub Actions unit-test job. Run just these tests with go test ./test/pkg/configfile/....

Helper Script

The hack/gh-workflow-ci.sh script contains several functions that assist in the CI process:

1. create_pac_github_app_secret - Creates the required secrets for GitHub app authentication
2. create_second_github_app_controller_on_ghe - Use startpaac instead. See Second Controller Setup below.
3. run_e2e_tests - Executes the E2E tests with proper filters
4. collect_logs - Gathers logs and diagnostic information

The script filters tests by category using pattern matching on test function names.

Second Controller Setup

In CI, use startpaac to install the second GitHub controller (GHE). When running with the --ci flag, startpaac automatically installs the second controller when PAC_SECOND_SECRET_FOLDER is set.

Example from e2e.yaml workflow (the second-controller credentials come from the github_enterprise section of the config file):

- name: Start installing cluster with startpaac
  run: |
    cfg() { yq -r "$1 // \"\"" "$PAC_E2E_CONFIG"; }
    mkdir -p ~/secrets-second
    cfg .github_enterprise.application_id > ~/secrets-second/github-application-id
    cfg .github_enterprise.private_key > ~/secrets-second/github-private-key
    cfg .github_enterprise.webhook_secret > ~/secrets-second/webhook.secret
    cfg .github_enterprise.smee_url > ~/secrets-second/smee
    export PAC_SECOND_SECRET_FOLDER=~/secrets-second

    cd startpaac
    ./startpaac --ci -a  # Automatically installs second controller

For manual setup or non-CI environments, see the Second Controller documentation.

[!NOTE] For details on how API call metrics are generated and archived as artifacts, see API Instrumentation (optional).

API Instrumentation (optional)

To help debug and analyze GitHub API usage during E2E runs, tests can emit structured JSON reports of API calls when the environment variable PAC_API_INSTRUMENTATION_DIR is set.

[!NOTE] Currently supported only for GitHub (both GitHub App and GitHub webhook flows). Support for other providers is planned.

• Set PAC_API_INSTRUMENTATION_DIR to a writable path before running tests, for example:
  • export PAC_API_INSTRUMENTATION_DIR=/tmp/api-instrumentation
• Each test produces a file named like YYYY-MM-DDTHH-MM-SS_<test_name>.json containing summary fields and an array of API calls (operation, duration_ms, url_path, status_code, rate_limit_remaining, provider, repo).
• In CI, this variable defaults to /tmp/api-instrumentation and hack/gh-workflow-ci.sh collect_logs copies the directory into the uploaded artifacts.

Log source details:

• Parses controller pod logs by resolving the controller label and container name dynamically.
• Tries known controller labels first (for example app.kubernetes.io/name=controller or ghe-controller) and then falls back to the standard PAC controller labels.
• Considers only log lines after the last occurrence of github-app: initialized OAuth2 client.
• Matches lines containing GitHub API call completed and extracts the embedded JSON payload.

Sample output:

{
  "test_name": "TestGithubAppSimple",
  "timestamp": "2025-08-05T16:12:20Z",
  "controller": "controller",
  "pr_number": 123,
  "sha": "abcdef1",
  "target_namespace": "pac-e2e-ns-xyz12",
  "total_calls": 2,
  "oauth2_marker_line": 42,
  "github_api_calls": [
    {
      "operation": "get_commit",
      "duration_ms": 156,
      "url_path": "/api/v3/repos/org/repo/git/commits/62a0...",
      "rate_limit_remaining": "",
      "status_code": 200,
      "provider": "github",
      "repo": "org/repo"
    }
  ]
}

Test Execution Flow

1. Setup:

   • Checkout code
   • Setup Go, ko, and gosmee client
   • Start Kind cluster
   • Install PAC controller

2. Configuration:

   • Create GitHub App secrets
   • Configure GHE environment (if needed)
   • Setup test environment variables

3. Test Execution:

   • Run selected tests against the specified provider category
   • For nightly runs, additional tests are included

4. Artifacts Collection:

   • Collect logs regardless of test outcome
   • Detect any panic in the controller logs
   • Upload artifacts to GitHub Actions
   • If PAC_API_INSTRUMENTATION_DIR is set, include API instrumentation reports (see API Instrumentation (optional))

Debugging CI

If a test fails in CI, you can:

1. Examine the workflow logs in GitHub Actions
2. Download the artifacts (logs) for detailed investigation
3. Use the "debug_enabled" option when manually triggering the workflow to get a tmate session

For local debugging, you can:

1. Set the same environment variables locally
2. Run make test-e2e with specific test filters

LLM E2E Tests

The LLM E2E tests uses a fake AI called nonoai to reply to the e2e tests and make them reliable (and cheap).

Deploy it with ko with ./pkg/test/nonoai/deployment.yaml

Responses and fake are included in this json file ./pkg/test/nonoai/responses.json

See an example of an E2E Test using it in ./gitea_llm_test.go

Notifications

Failed nightly runs trigger Slack notifications to alert the team of potential regressions.

Extending the CI

To add new provider tests:

1. Create test files following the existing patterns
2. Ensure proper naming convention to match the test category filters
3. Update environment variables if needed

For infrastructure changes:

1. Modify the kind-e2e-tests.yaml workflow file
2. Update the gh-workflow-ci.sh helper script as needed
3. Test changes using workflow dispatch before merging

Performance Considerations

1. Tests are configured with concurrency limits to prevent overlapping runs
2. The matrix strategy allows parallelization across provider categories
3. Rate limiting is managed by separating frequently-run tests from nightly tests