deadmanssnitch-operator

deadmanssnitch-operator

Operator to manage deadmanssnitch configs for Openshift Dedicated

• deadmanssnitch-operator
  • Overview
  • Metrics
  • Alerts
  • Usage

Overview

The operator runs on hive. It has a single controller. It:

• Requires a master Secret to talk to the Dead Man's Snitch API. This secret is expected to be named deadmanssnitch-api-key and live in the deadmanssnitch-operator namespace.
• Pays attention to ClusterDeployments that are:
  • Installed (spec.installed=true)
  • Managed (label api.openshift.com/managed="true")
• For each such ClusterDeployment:
  • Adds a finalizer to the ClusterDeployment to ensure we get a chance to clean up when it is deleted.
  • Creates a Snitch
  • Creates a Secret in the ClusterDeployment's namespace named {clusterdeploymentname}-dms-secret. The Secret contains the Snitch URL.
  • Creates a SyncSet in the ClusterDeployment's namespace named {clusterdeploymentname}-dms}. The SyncSet creates a SecretMapping that makes the above Secret appear inside the cluster as dms-secret in the openshift-monitoring namespace.

Metrics

metricDeadMansSnitchHeartbeat: Every 5 minutes, makes a request to the Dead Man's Snitch API using the API key and updates the gauge to 1 when the response code is between 200-299.

Alerts

• DeadMansSnitchAPIUnavailable - Unable to communicate with Dead Man's Snitch API for 15 minutes.

Usage

• Create an account on https://deadmanssnitch.com/
• Choose a plan that allows enhanced snitch intervals(Private eye or above)
• Create an API key
• Create the following secret which is required for deadmanssnitch-operator to create snitches

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: deadmanssnitch-api-key
  namespace: deadmanssnitch-operator
data:
  tags: <Tag for snitches>
  deadmanssnitch-api-key: <deadmanssnitch API key here>

• Build a docker image and replace REPLACE_IMAGE operator.yaml field with that image
  • you can do that using oc create -f https://github.com/openshift/deadmanssnitch-operator/raw/master/deploy/operator.yaml --dry-run=client -oyaml | oc set image --local -f - --dry-run=client -oyaml *=REPLACE_IMAGE
• Deploy using oc apply -f deploy/

Development

git clone https://github.com/openshift/hive.git
oc apply -f hive/config/crds

oc apply -f deploy/crds/deadmanssnitch.managed.openshift.io_deadmanssnitchintegrations.yaml
oc new-project deadmanssnitch-operator
oc apply -f deploy/role.yaml
oc apply -f deploy/service_account.yaml
oc apply -f deploy/role_binding.yaml

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: deadmanssnitch-api-key
  namespace: deadmanssnitch-operator
data:
  tags: <value>
  deadmanssnitch-api-key: <value>

apiVersion: deadmanssnitch.managed.openshift.io/v1alpha1
kind: DeadmansSnitchIntegration
metadata:
  finalizers:
  - dms.managed.openshift.io/deadmanssnitch-osd
  name: test-dmsi
  namespace: deadmanssnitch-operator
spec:
  clusterDeploymentSelector:
    matchExpressions:
    - key: api.openshift.com/test
      operator: In
      values:
      - "true"
  dmsAPIKeySecretRef:
    name: deadmanssnitch-api-key
    namespace: deadmanssnitch-operator
  snitchNamePostFix: "test"
  tags:
  - test
  targetSecretRef:
    name: dms-secret-test
    namespace: openshift-monitoring

export OPERATOR_NAME=deadmanssnitch-operator
go run main.go

real-hive$ oc get cd -n <namespace> <cdname> -o yaml > /tmp/fake-clusterdeployment.yaml

...

$ oc create namespace fake-cluster-namespace
$ oc apply -f /tmp/fake-clusterdeployment.yaml

oc edit clusterdeployment fake-cluster -n fake-cluster-namespace

oc label clusterdeployment -n <namespace> <cdname> api.openshift.com/test=true

oc delete clusterdeployment fake-cluster -n fake-cluster-namespace

oc edit clusterdeployment fake-cluster -n fake-cluster-namespace

PKO Template Testing

The operator is deployed via Package Operator (PKO). The PKO package lives in deploy_pko/ and includes Go templates (.gotmpl files) that are rendered at deploy time using config values from the ClusterPackage resource.

Two layers of template testing

1. PKO snapshot tests (test.template in deploy_pko/manifest.yaml)

The manifest.yaml defines test contexts under spec.test.template. Each context provides sample config values that PKO uses to render all .gotmpl files. The rendered output is stored as golden files in deploy_pko/.test-fixtures/.

Run locally:

kubectl-package validate deploy_pko/

If template output has changed, the command fails with a fixture mismatch error and a unified diff:

File mismatch against fixture in DeadmansSnitchIntegration-osd.yaml: Testcase "default"
--- FIXTURE/DeadmansSnitchIntegration-osd.yaml
+++ ACTUAL/DeadmansSnitchIntegration-osd.yaml
@@ -28,6 +28,9 @@
     - key: api.openshift.com/fedramp
       operator: NotIn
       values: ["true"]
+    - key: api.openshift.com/some-new-label
+      operator: NotIn
+      values: ["true"]

If the diff reflects an intentional change, regenerate the fixtures:

rm -rf deploy_pko/.test-fixtures/
kubectl-package validate deploy_pko/

Review the regenerated files, then commit the updated .test-fixtures/ alongside your template change.

2. Go unit tests (pkg/pko/template_test.go)

Structural validation tests that run via go test ./pkg/pko/... as part of the standard CI pipeline. These assert invariants like correct kind, phase annotations, and conditional rendering (e.g., silentAlertLegalEntityIds omitted when empty).

When do I need to update these tests?