argo-diff

Argo-Diff

An application written in go that provides Pull Request comments for changes to Kubernetes manifests when those manifests are delivered via ArgoCD:

Contents

• Overview
  • Why argo-diff?
  • Screenshots
• Deploying as a Webhook receiver
• GitHub Actions
• Configuration
  • Folding ignorable diffs
• Running locally
• Development Notes

Overview

Argo-diff is designed to receive webhook notifications from GitHub for pull_request and issue_comment events. When events are received, it uses the ArgoCD CLI to identify the ArgoCD application(s) that are configured with the pull request's repository as a source and to generate a diff of those applications against the live state.

Argo-diff will comment on the associated pull request with markdown displaying the diffs for the applications with potential changes.

Argo-diff will not run when the base branch of the pull request (the branch it will be merged into) is not the target revision for the Argo application. (eg: your Argo application targets production, but your PR is to be merged into dev.)

Why argo-diff?

There are multiple ways to produce diff previews of proposed manifest changes to ArgoCD applications in GitHub Pull Requests, but they often require per-repository configuration. Because argo-diff is triggered by GitHub event notifications, and because argo-diff auto-discovers ArgoCD applications configured in the source code repository, no configuration is needed when repositories and/or ArgoCD applications are added and removed.

Argo-diff natively supports ArgoCD's manifest-generate-paths annotation. This annotation is useful for monorepos that contain the manifests for many ArgoCD applications. For applications with it set, ArgoCD will only attempt to produce diffs for the applications whose manifest-generate-paths match the pull request's files changed (which is fetched via the GitHub API).

Argo-diff supports multi-source ArgoCD applications, including helm applications whose Application specs are managed by ArgoCD. This means if, in source control, you have a pull request that updates the chart version in the Application spec of a helm application, argo-diff will produce a diff of the affected application.

Argo-diff can also be quickly deployed via GitHub Actions.

Example argo-diff Comments

You can view example comments generated by argo-diff in PRs opened in this repo:

• https://github.com/vince-riv/argo-diff/pull/333#issuecomment-5723228791
• https://github.com/vince-riv/argo-diff/pull/333#issuecomment-5723229593

Those comments are generated by a Github actions workflow that performs an end-to-end test of new builds.

The first comment demonstrates generic manifest changes, including a cascading app-of-apps change (meta) that triggers downstream changes to a child application (helm-deployment).

The second comment shows the UX when the proposed manifests generate an error (broken-deployment).

Deploying as a Webhook receiver

Deploying argo-diff as a long-running webhook receiver takes five steps: grant it access to the GitHub API, create a read-only ArgoCD user, generate a shared webhook secret, deploy it to Kubernetes, and point GitHub webhooks at it.

1. Grant access to the GitHub API

Access can be granted either via a Personal Access Token (good for small environments or a proof of concept) or a GitHub App.

Option A — Personal Access Token (PAT)

• It is recommended to create a separate GitHub account in your organization specifically for argo-diff.
• argo-diff looks for the PAT in the GITHUB_TOKEN or GITHUB_PERSONAL_ACCESS_TOKEN env var.
• The PAT's permissions should match the GitHub App permissions listed below.

Option B — GitHub App

Create a new application in your organization under Developer Settings:

• Name can be anything, e.g. argo-diff. argo-diff identifies its own PR comments using the App's slug (the URL-friendly form of the name GitHub derives automatically), not the display name, so any name works.
• Homepage URL can be the base URL of your ArgoCD instance or of your webhook URL.
• Callback URL and Setup URL should be empty.
• Webhooks should not be active.
• Permissions:
  • Administration: Read-only
  • Commit statuses: Read and write
  • Metadata: Read-only
  • Pull requests: Read and write
• Where can this GitHub App be installed? → Only on this account
• After creating the App, note the App ID, then generate a new Private Key (this downloads a .pem file locally).
• Install the App to your organization. The installation settings URL looks like /organizations/<ORG_NAME>/settings/installations/<INSTALLATION_ID> — note the installation id from it.
• argo-diff reads the App configuration from the GITHUB_APP_ID, GITHUB_APP_INSTALLATION_ID, and GITHUB_APP_PRIVATE_KEY env vars (the last being the contents of the .pem file above).

2. Create an ArgoCD user

Create a user in your ArgoCD instance with read-only access to all applications.

• If you installed ArgoCD from its Helm chart, add an argo-diff user via the chart's values:

  configs:
    cm:
      accounts.argodiff: apiKey
      accounts.argodiff.enabled: "true"
  

• In policy.csv: g, argodiff, role:ci and p, role:ci, applications, get, *, allow. (This may not be needed if users in your ArgoCD installation default to the role:readonly role.)
• This user doesn't need a password but does need an API token, which an admin can generate via the UI or CLI. Use the generated token as the value of ARGOCD_AUTH_TOKEN.

3. Generate a webhook secret

Generate a webhook secret that is shared by both the argo-diff deployment and the GitHub webhook configuration.

4. Deploy to Kubernetes

• Recommended: use the Helm chart in charts/argo-diff/:

  helm install my-release oci://ghcr.io/vince-riv/chart/argo-diff
  

  To run an argocd CLI version other than the one baked into the argo-diff image, set the chart's argocdCli.image.tag value — see charts/argo-diff/README.md.
• Alternatively, use the example manifests in docs/k8s/ to deploy argo-diff to the argocd namespace of your cluster. You'll need to add an Ingress or IngressRoute so GitHub can reach the /webhook endpoint on the argo-diff Service.

5. Configure GitHub webhooks

Configure organization-level (or repository-level) webhook notifications to argo-diff. The Payload URL should map to the ingress configured in your cluster, and the secret should be the webhook secret generated above. Enable these event types:

• Issue comments — lets a comment of argo diff (customizable via ARGO_DIFF_REFRESH_COMMENT_KEYWORDS) re-trigger argo-diff on the pull request.
• Pull requests

Note: argo-diff only supports pull request events. Push events to branches are ignored.

After the webhook is activated, argo-diff should receive and verify the ping event, confirming connectivity from GitHub to argo-diff.

GitHub Actions

argo-diff can also run as a GitHub Action. This requires that your ArgoCD instance be reachable from the GitHub Actions runner — either ArgoCD is publicly available (and you accept the associated risks) or you use self-hosted runners within your environment.

The upside is that neither a GitHub account nor a GitHub App needs to be provisioned; you only need an ArgoCD user for argo-diff:

• Create a read-only ArgoCD user as described in step 2 above.
• Save its API token to the repository's Actions secrets and pass it to the argocd_auth_token input.

The downside is that GitHub workflows are typically managed on a repo-by-repo basis, and performance is a little slower than the deployed version.

Here's an example workflow snippet, assuming a self-hosted runner and that an ArgoCD auth token has been added to the repository's secrets:

---

on:
  pull_request:
    branches: [main]
    # `edited` covers stacked PRs: GitHub retargets them when their parent merges (see note below)
    types: [opened, synchronize, reopened, edited]

permissions:
  pull-requests: write

jobs:
  argo-diff:
    runs-on: [self-hosted]  # remove when running on GitHub's public runners and ArgoCD is publicly available
    name: Argo Diff
    # run on every edit that changed the base branch, and skip title/body edits
    if: github.event.action != 'edited' || github.event.changes.base.ref.from != ''
    steps:
      - name: ArgoCD Diff
        id: argo-diff
        uses: vince-riv/argo-diff@v3  # or pin an exact release, e.g. @2.10.0
        with:
          argocd_auth_token: ${{ secrets.ARGOCD_AUTH_TOKEN }}
          argocd_server: argocd-server.argocd.svc.cluster.local:80  # argocd.domain.example when publicly available via https
          argocd_server_insecure: true                              # typically false when publicly available via https
          argocd_server_plaintext: true                             # typically false when publicly available via https
          argocd_ui_base_url: https://argocd.domain.example
          argocd_grpc_web: true
          github_token: ${{ secrets.GITHUB_TOKEN }}
          log_level: trace
          repo_default_ref: main

Stacked pull requests: when a pull request's base branch merges, GitHub retargets the pull requests stacked on top of it. That retarget arrives as a pull_request event with the edited action, not synchronize, so a workflow on the default activity types never runs on a stacked pull request — its earlier events were filtered out by branches: [main] while the base was still a feature branch. Listening to edited closes that gap. Keep the if condition alongside it: GitHub also sends edited for title and body edits, and it populates changes.base.ref.from only when the base branch itself changed. argo-diff itself does not filter on the action type under GitHub Actions, and it reads the pull request's current head and base from the API on every run, so the run triggered by the retarget diffs against the new base.

Note: the deployed webhook service also handles this case. It acts on the opened and synchronize actions, and on edited when the payload's changes.base.ref.from field is set — the same signal used above — so a stacked pull request's retarget triggers a run there too, with no extra webhook configuration needed. An ordinary title or body edit still sends edited but leaves that field empty, so it is ignored as before. If a retarget is ever missed, comment argo diff on the pull request to trigger a run by hand.

Note: Releases were previously published under an actions-vX.Y.Z tag prefix (with a floating actions-vX major tag). As of 3.0.0, the actions-* tags are deprecated and will not be maintained.

Configuration

When deployed as a web service, argo-diff accepts all configuration options via environment variables. When used via GitHub Actions, the relevant options are accepted via inputs. The following table describes the accepted environment variables and their respective GitHub Actions inputs.

Important: ARGO_DIFF_SCM_PROVIDERS selects the source control providers (default: github), and each one listed needs valid credentials. For GitHub, either the GITHUB_APP_* variables must be set, or GITHUB_TOKEN / GITHUB_PERSONAL_ACCESS_TOKEN must be set. For GitLab, GITLAB_TOKEN must be set. To use GitLab, set ARGO_DIFF_SCM_PROVIDERS to gitlab, or github,gitlab for both.

Many of these variables have a typed Helm values key (config.*, logLevel, secret.*) — see charts/argo-diff/README.md. Any variable without a typed key — including ones not listed here — can still be set through the chart's deployment.env / deployment.envFrom passthrough values.

Note: When running via GitHub Actions, argo-diff uses the following environment variables set by GitHub: GITHUB_ACTIONS, GITHUB_BASE_REF, GITHUB_EVENT_NAME, GITHUB_HEAD_REF, GITHUB_REF, and GITHUB_REPOSITORY. Do not set these yourself.

Folding ignorable diffs

Behavior change: ARGO_DIFF_COMMENT_COLLAPSE_IGNORABLE defaults to true, so operators who already set ARGO_DIFF_COMMENT_COLLAPSE=auto see different folding: ignorable resources fold, everything else stays open, and ARGO_DIFF_COMMENT_COLLAPSE_APP_COUNT / ARGO_DIFF_COMMENT_COLLAPSE_RESOURCE_COUNT stop having an effect. Set ARGO_DIFF_COMMENT_COLLAPSE_IGNORABLE=false to keep the old auto behavior. The default mode (expanded) is unchanged.

In auto mode, argo-diff folds every resource diff in which every changed line matches a regex, and keeps all other diffs open. The main use case is a Helm chart bump, where most resources change only the chart and app version labels. A folded diff keeps its full text one click away, and shows 🔕 ignorable in its summary. The comment also says how many resources started folded.

Is it on?

How do I opt an application in or out? Annotate the ArgoCD Application that argo-diff diffs. All three annotations work only when the feature is on globally; none can turn it on.

In every row, excluded kinds never fold, unless collapse-ignorable-include-kinds names them.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  annotations:
    # "false" opts this application out. "true" is the default
    argo-diff.vince-riv.io/collapse-ignorable: "true"
    # newline-separated RE2 regexes, added to the global list
    argo-diff.vince-riv.io/collapse-ignorable-regexes: |
      ^\s*image:\s+registry\.example\.com/
    # lifts global exclusions for this application: group/Kind, group/* or Kind, comma or newline separated
    argo-diff.vince-riv.io/collapse-ignorable-include-kinds: argoproj.io/Application

The annotations belong on the Application being diffed. For an argoproj.io/Application resource inside an app-of-apps diff, that is the parent Application, not the child. argo-diff reads the live Application, so an annotation that a pull request adds or changes takes effect only after that Application syncs. An invalid annotation entry never fails a run: argo-diff skips it, logs a warning, and adds a note to that application's block.

What folds?

• A changed line is a line of the unified diff that starts with + or -. The --- and +++ file headers, context lines and @@ lines do not count.
• argo-diff removes one leading +/- and keeps the indentation. A resource is ignorable when it is not excluded, has at least one changed line, and every changed line matches at least one regex.
• Matching is an unanchored RE2 search on the diff text. Anchor your regexes with ^ and $.
• A resource with no changed lines never folds.

Default regexes (used when ARGO_DIFF_COMMENT_COLLAPSE_IGNORABLE_REGEXES is unset or empty):

^\s*app\.kubernetes\.io/version:\s
^\s*(helm\.sh/)?chart:\s+["']?[A-Za-z0-9][-A-Za-z0-9_.]*-v?\d+\.\d+\.\d+[-+A-Za-z0-9_.]*["']?\s*$
^\s*controller-gen\.kubebuilder\.io/version:\s+["']?v?\d+\.\d+\.\d+[-+A-Za-z0-9_.]*["']?\s*$

They match the app version label, the helm.sh/chart label (and the legacy chart label, which needs a <name>-<semver> value), and the CRD controller-gen version annotation. They deliberately do not match image tags or checksum/* annotations, so a chart bump that changes an image stays visible. A value of this variable replaces the defaults: copy the list above and extend it. Separate regexes with newlines, because regexes often contain , and |. Blank lines and lines that start with # are skipped; to match a literal leading #, write \# or [#]. Each list holds at most 32 entries of at most 1024 bytes. An invalid regex is skipped with a warning and a note in the comment. If every entry is invalid, the list is empty; it does not fall back to the defaults.

Warning: a regex that is too greedy can fold a diff that a reviewer needs to see. Keep regexes narrow and anchored. Choosing them is the operator's responsibility.

Excluded kinds. Some resources must stay visible even when every changed line matches, because they change what the cluster runs or who can do what. ARGO_DIFF_COMMENT_COLLAPSE_IGNORABLE_EXCLUDE_KINDS lists them. The default is argoproj.io/* (Application, ApplicationSet, AppProject, ...). Entries:

• group/Kind: one kind, for example rbac.authorization.k8s.io/ClusterRole
• group/*: every kind in a group
• Kind or /Kind: a kind in the core group, for example Secret

Matching is exact on the group and case-insensitive. * as a group is not supported, and neither is a bare group such as argoproj.io (use argoproj.io/*). To also keep every ClusterRole visible: ARGO_DIFF_COMMENT_COLLAPSE_IGNORABLE_EXCLUDE_KINDS=argoproj.io/*,rbac.authorization.k8s.io/ClusterRole. The exclusion covers only a resource's own group: the CustomResourceDefinition that defines an argoproj.io kind belongs to apiextensions.k8s.io. An application lifts exclusions for itself with collapse-ignorable-include-kinds.

The [] value. An unset or empty variable means "use the defaults". To say "none", set the variable to exactly []. With ARGO_DIFF_COMMENT_COLLAPSE_IGNORABLE_REGEXES=[], only applications that carry the collapse-ignorable-regexes annotation fold anything. That is the way to opt in per application only.

Running locally

Set the environment variables used by argo-diff and then execute go run cmd/main.go.

For example, you can place these in a file called .env.sh:

GITHUB_PERSONAL_ACCESS_TOKEN='github_pat_XXXX'
ARGOCD_AUTH_TOKEN='ABCDEFGHIJKLMNOPQRSTUVWXYZ'
ARGOCD_SERVER_ADDR='argocd.your.domain:443'
ARGOCD_UI_BASE_URL='https://argocd.your.domain'
APP_ENV='dev'

Then source it and start the server:

$ set -o allexport ; . .env.sh ; set +o allexport
$ go run cmd/main.go

Or run it once to process a manually crafted event file:

$ set -o allexport ; . .env.sh ; set +o allexport
$ go run cmd/main.go -f path/to/event_info.json

Note: argo-diff shells out to the argocd CLI, so a compatible argocd binary must be on your PATH (override the command name, or point at an absolute path, with ARGOCD_CLI_CMD_NAME).

Craft a local file with event data

When running locally, the easiest way to trigger argo-diff is to create a JSON file that can be unmarshalled into an EventInfo struct (defined in internal/webhook/process.go):

{
    "provider": "github",
    "ignore": false,
    "owner": "GITHUB_ORG_NAME",
    "repo": "REPOSITORY_NAME",
    "default_ref": "main",
    "commit_sha": "LONG_SHA_OF_COMMIT",
    "pr": 123,
    "change_ref": "BRANCH_NAME_OF_PR",
    "base_ref": "main"
}

Description of those fields:

• provider: the source control provider, github or gitlab; optional, defaults to github
• ignore: tells argo-diff to ignore the event
• owner: GitHub organization name, or the GitLab namespace path, which may hold nested groups (eg: group/subgroup)
• repo: GitHub repository name, or the GitLab project name
• default_ref: the default branch of the repository
• commit_sha: the sha that triggered the event — should be HEAD of the branch of the PR if you want it to comment
• pr: pull request number (duh), or the merge request IID on GitLab (the !123 number)
• change_ref: the source branch of the PR (the feature branch)
• base_ref: the branch to which the PR is getting merged
• refresh: optional; when true, argo-diff reads commit_sha, change_ref and base_ref from the PR/MR itself

This JSON file can be passed to argo-diff via the -f argument or posted to the /dev HTTP endpoint.

For a GitLab merge request, set ARGO_DIFF_SCM_PROVIDERS=gitlab (or github,gitlab), set GITLAB_TOKEN (and GITLAB_BASE_URL for a self-managed instance), and name the provider. For the project https://gitlab.com/group/subgroup/project:

{
    "provider": "gitlab",
    "owner": "group/subgroup",
    "repo": "project",
    "default_ref": "main",
    "pr": 42,
    "refresh": true
}

"refresh": true makes argo-diff read the head commit and branches from the merge request, so commit_sha, change_ref and base_ref can be left out.

Testing HTTP endpoints locally

To simulate GitHub webhook requests, copy the webhook request headers and payload to temp/curl-headers.txt and temp/curl-payload.json, then use the post-local.sh script to ship them to the local server.

There is also a /dev endpoint that is enabled when APP_ENV=dev. It is handled by devHandler() in internal/server/http_server.go; it bypasses webhook signature validation and triggers argo-diff for the posted event.

When APP_ENV is dev, argo-diff will comment but will not set commit status checks.

Development Notes

This was originally developed by @vrivellino as a way to learn Go. Its functionality replicates that of an internal tool written by smart people at a previous job.