canary-checker

Kubernetes Native Health Check Platform

Canary Checker is a Kubernetes-native platform for monitoring health across applications and infrastructure using both passive and active (synthetic) mechanisms.

Features

• Batteries Included - 30+ built-in active and passive check types
• Kubernetes Native - Health checks, or canaries, are CRDs that reflect health via the status field, making them compatible with GitOps, Flux Health Checks, Argo, Helm, etc.
• Secret Management - Leverage Kubernetes Secrets and ConfigMaps for authentication and connection details
• Prometheus - Prometheus-compatible metrics are exposed at /metrics. A Grafana dashboard is also available.
• Self-Contained Storage - Uses embedded Postgres by default and can be configured to use external Postgres.
• JUnit Export (CI/CD) - Export health check results to JUnit format for integration into CI/CD pipelines
• JUnit Import (k6/Newman/Puppeteer/etc.) - Use any container that creates JUnit test results
• Scriptable - Go templates, JavaScript and CEL can be used to:
  • Evaluate whether a check is passing and severity to use when failing
  • Extract a user friendly error message
  • Transform and filter check responses into individual check results
  • Extract custom metrics
• Multi-Modal - While designed as a Kubernetes operator, Canary Checker can also run as a CLI and as a standalone server.

Getting Started

1. Install Canary Checker with Helm

helm repo add flanksource https://flanksource.github.io/charts
helm repo update

helm install \
  canary-checker \
  flanksource/canary-checker \
  -n canary-checker \
  --create-namespace \
  --wait

1. Create a new check

apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: http-check
spec:
  schedule: "@every 30s"
  http:
    - name: basic-check
      url: https://httpbin.flanksource.com/status/200
    - name: failing-check
      url: https://httpbin.flanksource.com/status/500

2a. Run the check locally (Optional)

wget  https://github.com/flanksource/canary-checker/releases/latest/download/canary-checker_linux_amd64 \
-O canary-checker &&  chmod +x canary-checker
./canary-checker run canary.yaml

1. Apply the check

kubectl apply -f canary.yaml

1. Check the health status

kubectl get canary

NAME         SCHEDULE      STATUS   LAST CHECK   UPTIME 1H        LATENCY 1H   LAST TRANSITIONED
http-check   @every 30s    Passed   13s          18/18 (100.0%)   480ms        13s

See fixtures for more examples and docs for more comprehensive documentation.

Use Cases

Synthetic Testing

Run simple HTTP/DNS/ICMP probes or more advanced full test suites using JMeter, k6, Playwright, Postman/Newman, or any container that can emit JUnit XML.

# Run a container that executes a playwright test, and then collect the
# JUnit formatted test results from the /tmp folder
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: playwright-junit
spec:
  schedule: "@every 2m"
  junit:
    - testResults: "/tmp/"
      name: playwright-junit
      spec:
        containers:
          - name: playwright
            image: ghcr.io/flanksource/canary-playwright:latest

Infrastructure Testing

Verify that infrastructure is fully operational by checking Kubernetes resources, deploying temporary resources, and running nested checks against them.

# Deploy a temporary pod and service, wait for readiness, call the service, and then clean up.
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: kubernetes-resource-check
spec:
  schedule: "@every 5m"
  kubernetesResource:
    - name: service-accessibility
      namespace: default
      waitFor:
        expr: 'dyn(resources).all(r, k8s.isReady(r))'
        interval: 2s
        timeout: 2m
      resources:
        - apiVersion: v1
          kind: Pod
          metadata:
            name: httpbin-pod
            namespace: default
            labels:
              app: httpbin
          spec:
            containers:
              - name: httpbin
                image: kennethreitz/httpbin:latest
                ports:
                  - containerPort: 80
        - apiVersion: v1
          kind: Service
          metadata:
            name: httpbin
            namespace: default
          spec:
            selector:
              app: httpbin
            ports:
              - port: 80
                targetPort: 80
      checks:
        - http:
            - name: call-httpbin-service
              url: http://httpbin.default.svc/status/200
      checkRetries:
        delay: 2s
        interval: 3s
        timeout: 2m

Backup Checks / Batch File Monitoring

Check that batch file processes are functioning correctly by checking the age and size of files in local file systems, SFTP, SMB, S3 and GCS.

# Checks that a recent DB backup has been uploaded
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: folder-check
spec:
  schedule: "0 22 * * *"
  folder:
    - path: s3://database-backups/prod
      name: prod-backup
      maxAge: 1d
      minSize: 10gb

Alert Aggregation

Aggregate alerts and recommendations from Prometheus, AWS CloudWatch, Dynatrace, etc.

apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: alertmanager-check
spec:
  schedule: "*/5 * * * *"
  alertmanager:
    - url: alertmanager.monitoring.svc
      alerts:
        - .*
      ignore:
        - KubeScheduler.*
        - Watchdog
      transform:
        # for each alert, transform it into a new check
        javascript: |
          var out = _.map(results, function(r) {
            return {
              name: r.name,
              labels: r.labels,
              icon: 'alert',
              message: r.message,
              description: r.message,
            }
          })
          JSON.stringify(out);

Prometheus Exporter Replacement

Export custom metrics from the result of any check, making it possible to replace various other Prometheus exporters that collect metrics via HTTP, SQL, etc.

apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: exchange-rates
spec:
  schedule: "@every 1h"
  http:
    - name: exchange-rates
      url: https://api.frankfurter.app/latest?from=USD&to=GBP,EUR,ILS
      metrics:
        - name: exchange_rate
          type: gauge
          value: result.json.rates.GBP
          labels:
            - name: "from"
              value: "USD"
            - name: to
              value: GBP

Platform Ready

Canary Checker is ideal for building platforms. Developers can include health checks for their applications in whatever tooling they prefer, with secret management that uses native Kubernetes constructs.

apiVersion: v1
kind: Secret
metadata:
  name: basic-auth
stringData:
  user: john
  pass: doe
---
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: http-basic-auth-secret
spec:
  http:
    - name: http-basic-auth
      url: https://httpbin.flanksource.com/basic-auth/john/doe
      username:
        valueFrom:
          secretKeyRef:
            name: basic-auth
            key: user
      password:
        valueFrom:
          secretKeyRef:
            name: basic-auth
            key: pass

Dashboard

Canary Checker comes with a built-in dashboard by default.

There is also a Grafana dashboard, or build your own using the metrics exposed.

Getting Help

If you have any questions about Canary Checker:

• Read the docs
• Invite yourself to the CNCF community slack and join the #canary-checker channel.
• Check out the YouTube playlist.
• File an issue - (We do provide user support via GitHub Issues, so don't worry if your issue is a bug or not)

Your feedback is always welcome!

Check Types

Contributing

See CONTRIBUTING.md

Thank you to all our contributors!

License

Canary Checker core (the code in this repository) is licensed under Apache 2.0 and accepts contributions via GitHub pull requests after signing a CLA.

The UI (Dashboard) is free to use with Canary Checker under a license exception of Flanksource UI