observability-operator

observability-operator

Brain of the Giant Swarm observability platform: provisions and configures the full observability stack (metrics, logs, traces, alerts, dashboards) for every managed cluster.

It handles four responsibilities:

• Cluster monitoring — reconciles cluster.x-k8s.io/Cluster objects and provisions the full observability stack per managed cluster: Alloy agent config for metrics, logs, events, and traces; gateway authentication secrets for Mimir, Loki, and Tempo; and the observability-bundle App CR that deploys Alloy onto the workload cluster.
• Grafana organizations — manages Grafana organizations via the GrafanaOrganization CRD: creates orgs, configures per-org datasources, and sets up SSO role mappings.
• Alertmanager configuration — assembles and pushes tenant Alertmanager configs to Mimir Alertmanager from labeled Kubernetes Secrets.
• Dashboard provisioning — provisions Grafana dashboards from labeled Kubernetes ConfigMaps, including folder hierarchy management.

Project Structure

observability-operator/
├── api/                                # CRD type definitions (kubebuilder)
│   ├── v1alpha1/                       # GrafanaOrganization v1alpha1
│   └── v1alpha2/                       # GrafanaOrganization v1alpha2 (storage version)
├── cmd/
│   └── main.go                         # Operator entry point
├── config/                             # Kubebuilder manifests (CRDs, RBAC, webhooks)
├── docs/                               # Feature documentation
├── helm/
│   └── observability-operator/         # Helm chart for deployment
│       └── files/alertmanager/         # Alertmanager config templates
├── internal/
│   ├── controller/                     # Reconcilers (cluster, dashboard, alertmanager, grafanaorg)
│   ├── mapper/                         # Watch event mappers (dashboard, organization)
│   ├── predicates/                     # Event filter predicates
│   └── webhook/                        # Validating & conversion webhooks
│       ├── v1/                         # Secret & ConfigMap validators
│       ├── v1alpha1/                   # GrafanaOrganization v1alpha1 validator
│       ├── v1alpha2/                   # GrafanaOrganization v1alpha2 validator
│       └── validation/                 # Shared validation logic
├── pkg/
│   ├── agent/                          # Alloy ConfigMap/Secret repository
│   │   ├── collectors/                 # Per-signal config builders:
│   │   │   ├── metrics/                # Alloy metrics config + KEDA auth objects
│   │   │   ├── logs/                   # Alloy logs + network monitoring config
│   │   │   └── events/                 # Alloy Kubernetes events config
│   │   └── common/                     # Shared Alloy config keys
│   ├── alerting/
│   │   ├── alertmanager/               # Assembles + pushes config to Mimir Alertmanager API
│   │   └── heartbeat/                  # Cronitor heartbeat monitor management
│   ├── auth/                           # Gateway auth secrets (Mimir, Loki, Tempo credentials)
│   ├── bundle/                         # observability-bundle App/HelmRelease CR reconciliation
│   ├── common/                         # Shared utilities (labels, tenancy, organization helpers)
│   ├── config/                         # Operator config struct from Helm values / CLI flags
│   ├── domain/                         # Domain types (orgs, dashboards, folders, clusters)
│   ├── grafana/                        # Grafana HTTP client (orgs, datasources, dashboards, SSO)
│   │   └── client/                     # HTTP client, TLS, credential management
│   ├── metrics/                        # Prometheus metrics declarations (all custom metrics)
│   └── monitoring/
│       ├── mimir/                      # Mimir querier (head series)
│       └── sharding/                   # Per-cluster agent sharding logic
├── tests/
│   ├── alertmanager-routes/            # BATS route tests + Go integration tests
│   ├── alertmanager-integration/       # Kind + Mimir setup for integration tests
│   ├── ats/                            # Acceptance test suite
│   └── bats/                           # BATS test framework
└── CHANGELOG.md                        # Keep a Changelog format

Architecture

Four controllers run in a single binary:

Validating webhooks enforce constraints on alertmanager config secrets, dashboard configmaps, and GrafanaOrganization CRs. A conversion webhook handles GrafanaOrganization version conversion between v1alpha1 and v1alpha2.

Features

Alertmanager config provisioning

The operator watches for Secrets with the following labels and uses them as Alertmanager configs:

• observability.giantswarm.io/kind: alertmanager-config
• observability.giantswarm.io/tenant: <tenant> (label or annotation)

One secret per tenant. See docs/alertmanager.md for full usage and examples.

Grafana dashboard provisioning

The operator watches for ConfigMaps with the following labels and provisions them as Grafana dashboards:

• app.giantswarm.io/kind: dashboard
• observability.giantswarm.io/organization: <org-name> (label or annotation)
• observability.giantswarm.io/folder: <path> (optional, supports nested hierarchy e.g. team/subteam)

See docs/dashboards.md for full usage and examples.

Grafana organization management

The operator manages Grafana organizations via the GrafanaOrganization CRD. See docs/grafana-organization.md for CRD reference and examples.

Per-cluster feature flags

Observability features are controlled per-cluster via labels on the Cluster object:

The KEDA operator namespace can be overridden per-cluster via the observability.giantswarm.io/keda-namespace annotation (default: keda).

See docs/cluster.md for full details including per-cluster sharding and queue tuning.

Getting Started

The operator is deployed via the Helm chart in helm/observability-operator/.

For local development and contributing, see CONTRIBUTING.md.

Observability

The operator exposes the following metrics (prefix: observability_operator_):

Self-monitoring via PodMonitor at helm/observability-operator/templates/pod-monitor.yaml. Alerts, dashboards, and runbooks live in the prometheus-rules and dashboards repositories.

See docs/metrics.md for the full metrics reference.

Documentation

Credits

This operator was built using kubebuilder.