kube-compare

kube-compare

This repository is primarily home to a kubectl plugin called cluster-compare which allows the user to compare an individual manifest or entire running cluster to a reference and find the differences. Expected differences are ignored as are cluster-managed fields, highlighting only differences that are of interest.

cluster-compare is intended for administrators, architects, support engineers, and others to quickly check that a configuration is as-expected. For a more detailed description of the purpose and approach of the tool please read the proposal document.

Install

This plugin can be installed with the Krew plugin manager:

kubectl krew install cluster-compare

Alternatively, build the code locally and install it in /usr/local/bin:

make build
sudo make install

Run

A reference configuration is required in order to run. A reference configuration is a directory containing a metadata.yaml and one or more templates.

Compare a known valid reference configuration with a live cluster:

kubectl cluster-compare -r ./reference/metadata.yaml

Compare a known valid reference configuration with a local set of CRs:

kubectl cluster-compare -r ./reference/metadata.yaml -f ./crsdir -R

Compare a known valid reference configuration with a live cluster and with a user config:

kubectl cluster-compare -r ./reference/metadata.yaml -c ./user_config

Run a known valid reference configuration with an oc must-gather output:

kubectl cluster-compare -r ./reference/metadata.yaml -f "must-gather*/*/cluster-scoped-resources","must-gather*/*/namespaces" -R

Extract a reference configuration from a container image and compare with a local set of CRs:

kubectl cluster-compare -r container://<IMAGE>:<TAG>:/home/ztp/reference/metadata.yaml -f ./crsdir -R

The tool can also be used to generate a reference configuration, from either a running cluster or against an oc must-gather output. A configuration file must be created to specify what resources should be captured. An example configuration is available: generate-config.yaml.

Generate a reference configuration from a running cluster:

kubectl cluster-compare -g ./refgen-config.yaml

Or from a must-gather file:

kubectl cluster-compare -g ./refgen-config.yaml -f ./must-gather.123456

More details on reference generation is available under generate-references.

Run kubectl cluster-compare --help for a more extensive usage description.

Output

The tool outputs a diff for each comparison made, and a final summary.

Each comparison is surrounded by a line of *. The comparison identifies the cluster manifest and reference file being compared and a diff:

**********************************

Cluster CR: apps/v1_Deployment_kubernetes-dashboard_kubernetes-dashboard
Reference File: deploymentDashboard.yaml
Diff Output: diff -u -N /tmp/MERGED-4218954955/apps-v1_deployment_kubernetes-dashboard_kubernetes-dashboard /tmp/LIVE-168878603/apps-v1_deployment_kubernetes-dashboard_kubernetes-dashboard
--- /tmp/MERGED-4218954955/apps-v1_deployment_kubernetes-dashboard_kubernetes-dashboard 2024-07-02 09:18:04.314476186 -0400
+++ /tmp/LIVE-168878603/apps-v1_deployment_kubernetes-dashboard_kubernetes-dashboard    2024-07-02 09:18:04.314476186 -0400
@@ -14,7 +14,7 @@
   template:
     metadata:
       labels:
-        k8s-app: kubernetes-dashboard
+        k8s-app: kubernetes-dashboard-diff
     spec:
       containers:
       - args:

**********************************

The output ends with a summary of all comparisons made, which lists differences, and highlights if references were not found for any cluster manifests.

Summary
CRs with diffs: 1
CRs in reference missing from the cluster: 1
ExamplePart:
  Dashboard:
  - deploymentMetrics.yaml
No CRs are unmatched to reference CRs

Metadata.yaml

At the basic level, metadata.yaml lays out a reference configuration in Parts, each containing Components and defines the templates and comparison rules.

The following example describes an one-part guestbook app, with redis and frontend components. The templates it references are stored in the same directory as metadata.yaml.

parts:
  - name: guestbook
    components:
      - name: redis
        type: Required # mark the Component as "Required" or "Optional"
        requiredTemplates: # absence from cluster manifests is considered a diff
          - path: redis-master-deployment.yaml
            config:
               fieldsToOmitRefs: # default and deployments
                  - default
                  - deployments
          - path: redis-master-service.yaml
        optionalTemplates: # will not be reported if missing from cluster manifests
          - path: redis-replica-deployment.yaml
            config:
               fieldsToOmitRefs: # Only deployments
                  - deployments
          - path: redis-replica-service.yaml
      - name: frontend
        type: Required
        requiredTemplates:
          - path: frontend-deployment.yaml
            config:
               fieldsToOmitRefs: # Only deployments
                  - deployments
          - path: frontend-service.yaml
            config:
               fieldsToOmitRefs:
                  - cluster-compare-built-in  # The built in default values, used if no defaultOmitRef is provided

fieldsToOmit:
   defaultOmitRef: default
   items:
      deployments:
         - pathToKey: spec.selector.matchLabels.k8s-app # remove spec.selector.matchLabels.k8s-app before diff
         - pathToKey: metadata.labels.k8s-app
         - pathToKey: spec.template.metadata.labels.k8s-app
      default:
         - pathToKey: a.custom.default."k8s.io" # Keys containing dots should be quoted

metadata.yaml supports several other advanced behaviours:

• Declaring predefined fragments using templateFunctionFiles that can be used in multiple resource templates.
• Globally ignoring specific fields by yaml path with fieldsToOmit.
• Ignoring, per template, fields that are not defined in the reference template with ignore-unspecified-fields.

See the included test cases for more examples of reference configs. For a complete explanation of metadata.yaml please see Building a Reference Config.

Further Resources

User Guide

Building a Reference Config

Developer Intro

Build Image