kustomize-controller

kustomize-controller

The kustomize-controller is a continuous delivery (CD) tool for Kubernetes. The controller runs CD pipelines inside the cluster for workloads and infrastructure manifests coming from source control systems that are generated with Kustomize.

Features:

• watches for Kustomization objects
• fetches artifacts produced by source-controller from Source objects
• watches Source objects for revision changes
• generates Kubernetes manifests with kustomize build
• validates the build output with client-side or APIServer dry-run
• applies the generated manifests on the cluster
• prunes the Kubernetes objects removed from source
• checks the health of the deployed workloads
• runs Kustomizations in a specific order, taking into account the depends-on relationship

Usage

The kustomize-controller is part of a composable GitOps toolkit and depends on
source-controller to provide the raw Kubernetes manifests and kustomization.yaml file.

Install the controllers

Install source-controller with:

kustomize build https://github.com/fluxcd/source-controller//config/default?ref=v0.0.1-alpha.1 \
kubectl apply -f-

Install kustomize-controller with:

kustomize build https://github.com/fluxcd/kustomize-controller//config/default?ref=v0.0.1-alpha.1 \
kubectl apply -f-

Define a Git repository source

Create a source object that points to a Git repository containing Kubernetes and Kustomize manifests:

apiVersion: source.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
  name: podinfo
  namespace: default
spec:
  interval: 1m
  url: https://github.com/stefanprodan/podinfo-deploy
  ref:
    branch: master

For private repositories, SSH or token based authentication can be configured with Kubernetes secrets.

Save the above file and apply it on the cluster. You can wait for the source controller to assemble an artifact from the head of the repo master branch with:

kubectl wait gitrepository/podinfo --for=condition=ready

The source controller will check for new commits in the master branch every minute. You can force a git sync with:

kubectl annotate --overwrite gitrepository/podinfo source.fluxcd.io/syncAt="$(date +%s)"

Define a kustomization

Create a kustomization object that uses the git repository defined above:

apiVersion: kustomize.fluxcd.io/v1alpha1
kind: Kustomization
metadata:
  name: podinfo-dev
spec:
  interval: 5m
  path: "./overlays/dev/"
  prune: "env=dev"
  sourceRef:
    kind: GitRepository
    name: podinfo
  validation: client
  healthChecks:
    - kind: Deployment
      name: podinfo
      namespace: dev
  timeout: 80s

A detailed explanation of the Kustomization object and its fields can be found in the specification doc.

Based on the above definition, the kustomize-controller fetches the Git repository content from source-controller, generates Kubernetes manifests by running kustomize build inside ./overlays/dev/, and validates them with a dry-run apply. If the manifests pass validation, the controller will apply them on the cluster and starts the health assessment of the deployed workload. If the health checks are passing, the Kustomization object status transitions to a ready state.

You can wait for the kustomize controller to complete the deployment with:

kubectl wait kustomization/podinfo-dev --for=condition=ready

When the controller finishes the reconciliation, it will log the applied objects:

kubectl -n kustomize-system logs deploy/kustomize-controller | jq .

{
  "level": "info",
  "ts": 1587195448.071468,
  "logger": "controllers.Kustomization",
  "msg": "Kustomization applied in 1.436096591s",
  "kustomization": "default/podinfo-dev",
  "output": {
    "namespace/dev": "created",
    "service/podinfo": "created",
    "deployment.apps/podinfo": "created",
    "horizontalpodautoscaler.autoscaling/podinfo": "created"
  }
}

You can trigger a kustomize build and apply any time with:

kubectl annotate --overwrite kustomization/podinfo-dev kustomize.fluxcd.io/syncAt="$(date +%s)"

When the source controller pulls a new Git revision, the kustomize controller will detect that the source revision changed, and will apply those changes right away.

If the kustomization build or apply fails, the controller sets the ready condition to false and logs the error:

status:
  conditions:
  - lastTransitionTime: "2020-04-16T07:27:58Z"
    message: 'apply failed'
    reason: ApplyFailed
    status: "False"
    type: Ready

{
  "kustomization": "default/podinfo-dev",
  "error": "Error from server (NotFound): error when creating \"podinfo-dev.yaml\": namespaces \"dev\" not found\n"
}

Define the order for kustomizations

When defining a kustomization, you may need to make sure other kustomizations have been successfully applied beforehand. A kustomization can specify a list of dependencies with spec.dependsOn. When combined with health assessment, a kustomization will run after all its dependencies health checks are passing.

For example, a service mesh proxy injector should be running before deploying applications inside the mesh:

apiVersion: kustomize.fluxcd.io/v1alpha1
kind: Kustomization
metadata:
  name: istio
spec:
  interval: 10m
  path: "./profiles/default/"
  sourceRef:
    kind: GitRepository
    name: istio
  healthChecks:
    - kind: Deployment
      name: istiod
      namespace: istio-system
  timeout: 2m
---
apiVersion: kustomize.fluxcd.io/v1alpha1
kind: Kustomization
metadata:
  name: podinfo-dev
spec:
  dependsOn:
    - istio
  interval: 5m
  path: "./overlays/dev/"
  prune: "env=dev"
  sourceRef:
    kind: GitRepository
    name: podinfo

Deploy releases to production

For production deployments, instead of synchronizing with a branch you can use a semver range to target stable releases:

apiVersion: source.fluxcd.io/v1alpha1
kind: GitRepository
metadata:
  name: podinfo-releases
spec:
  interval: 5m
  url: https://github.com/stefanprodan/podinfo-deploy
  ref:
    semver: ">=0.0.1-rc.1 <1.0.0"

With ref.semver we configure source controller to pull the Git tags and create an artifact from the most recent tag that matches the semver range.

Create a production kustomization and reference the git source that follows the latest semver release:

apiVersion: kustomize.fluxcd.io/v1alpha1
kind: Kustomization
metadata:
  name: podinfo-production
spec:
  interval: 10m
  path: "./overlays/production/"
  sourceRef:
    kind: GitRepository
    name: podinfo-releases

Based on the above definition, the kustomize controller will build and apply a kustomization that matches the semver range set in the Git repository manifest.