kafkarator

Kafkarator

Kafkarator is a Kubernetes operator on the NAIS platform, providing self-service functionality for Aiven-hosted Kafka through Kubernetes resources.

Kafkarator defines a Kubernetes custom resource, kafka.nais.io/Topic. When users create or update this resource, Kafkarator translates it to Aiven topics and ACL entries.

Table of Contents

• User documentation
• Features
• Architecture & Design
• Configuration
• Usage Example
• Scripts & Utilities
• Developer documentation
• Helm Charts
• Verifying Images
• Contributing
• Links

User documentation

• NAIS Kafka docs

Features

• Declarative management of Kafka topics and ACLs via Kubernetes CRDs.
• Automatic synchronization between Kubernetes resources and Aiven Kafka.
• Support for both topic and stream resources.
• Canary deployment and monitoring via the canary component.
• Helm charts for easy deployment and configuration.
• Security checks, static analysis, and SBOM attestation in CI.

Architecture & Design

• Built as a Kubernetes operator using Go and controller-runtime.
• Uses a custom resource definition (CRD) kafka.nais.io/Topic for declarative Kafka management.
• Architecture Decision Records (ADRs) are maintained for key design decisions.
• Note: Future ADRs are maintained in the PIG repository.

Configuration

• Most configuration is handled via environment variables, often with the CANARY_ or FEATURE_ prefix.
• Example variables:
  • CANARY_KAFKA_BROKERS: Comma-separated list of Kafka broker addresses.
  • CANARY_KAFKA_CERTIFICATE_PATH, CANARY_KAFKA_KEY_PATH, CANARY_KAFKA_CA_PATH: Paths to Kafka TLS credentials.
  • CANARY_KAFKA_TOPIC: Default topic for canary messages.
  • CANARY_METRICS_ADDRESS: Address for Prometheus metrics endpoint.
  • FEATURE_GENERATED_CLIENT: Feature flag for enabling generated client code.

See the cmd/canary/main.go and cmd/kafkarator/feature_flags.go for all available flags and environment variables.

Usage Example

Deploy Kafkarator to your Kubernetes cluster using the provided Helm charts. Then create a KafkaTopic resource:

apiVersion: kafka.nais.io/v1
kind: Topic
metadata:
  name: my-topic
  namespace: my-team
spec:
  pool: nav-dev
  config:
    partitions: 3
    replication: 2
  acl:
    - access: read
      application: my-app
      team: my-team

Kafkarator will automatically create the topic and set up ACLs in Aiven.

For more examples, see the examples/ directory.

Scripts & Utilities

The scripts/ directory contains Python utilities for investigating and cleaning up Kafka users and ACLs. Key scripts include:

• investigate_app.py: Investigate service users and ACLs for a given app/team.
• user_cleaner.py: Find and optionally remove unused Kafka users.
• find_app.py: Search for secrets and users related to a specific app.

Developer documentation

Prerequisites

• mise (for tool and task management)
• Docker (with buildx, for building images)
• Go (see mise.toml for version)
• Helm (for chart linting)

Building and Testing

• Build Docker images:

  mise run docker
  

  This runs docker buildx bake via the mise task and builds all three images (kafkarator, canary, canary-deployer). To build a single image, use mise run docker:kafkarator, mise run docker:canary, or mise run docker:canary-deployer.

• Run all code checks (lint, static analysis, security, Helm lint):

  mise run check
  

  This runs all checks defined in .mise-tasks/check/ (deadcode, gosec, staticcheck, vulncheck, helm-lint, etc).

• Run tests:

  mise run test
  

  Or with coverage (if you have tests):

  mise run test -- --coverage
  

• Build locally (binaries):

  mise run build:kafkarator
  mise run build:canary
  

  This builds the binaries directly using Go.

Helm Charts

Helm charts for kafkarator, kafka-canary, and kafka-canary-alert are in the charts/ directory. Lint them with:

mise run check:helm-lint

To install a chart (example):

helm install kafkarator charts/kafkarator

Verifying the kafkarator images and their contents

The images are signed "keylessly" using Sigstore cosign. To verify their authenticity run (replace <shasum> with the actual image digest):

cosign verify \
  --certificate-identity "https://github.com/nais/kafkarator/.github/workflows/main.yml@refs/heads/master" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  europe-north1-docker.pkg.dev/nais-io/nais/images/kafkarator@sha256:<shasum>

The images are also attested with SBOMs in the CycloneDX format. You can verify these by running:

cosign verify-attestation --type cyclonedx  \
  --certificate-identity "https://github.com/nais/kafkarator/.github/workflows/main.yml@refs/heads/master" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  europe-north1-docker.pkg.dev/nais-io/nais/images/kafkarator@sha256:<shasum>

Contributing

• Please run mise run check and mise run test before submitting a PR.
• See .mise-tasks/ for all available tasks.
• For questions, see NAIS Slack or open an issue.
• Open issues or pull requests for new features or bug reports.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Links

• NAIS Platform
• Aiven Kafka
• mise
• Helm
• Architecture Decision Records