prometheus-rules

prometheus-rules

A CLI tool for generating Azure Monitor Prometheus rule groups from Prometheus Operator rule definitions.

Overview

This tool converts Kubernetes PrometheusRule custom resources (used by the Prometheus Operator) into Azure Bicep templates that deploy Microsoft.AlertsManagement/prometheusRuleGroups resources. It supports both alerting rules and recording rules, and includes built-in testing to ensure rule correctness.

Features

• Converts PrometheusRule CRDs to Azure Bicep templates
• Supports both alerting rules and recording rules (generated into separate files)
• Validates rules using promtool test rules
• Maps Prometheus severity labels to IcM (Incident Management) severity levels
• Automatically generates IcM correlation IDs for proper incident aggregation
• Supports expression replacements for platform-specific adjustments

Recent Change

• Added support for labelsToExtract in config.
• During generation, labels listed in labelsToExtract are detected from the description annotation template text and then appended to generated alert metadata in configured order.
• This ensures generated alert title/correlation fields can include extra routing context (such as cluster/namespace/pod) without hard-coding those labels in the generator.
• Using description as the signal for which labels to append is not perfectly precise, but for upstream-managed alert sources that cannot be edited directly (for example, kubernetesControlPlane-prometheusRule.yaml), it provides a practical way to preserve useful alert scoping.

Usage

Build

make

Run

# Generate everything (alerts + recording rules)
make run

# Generate all alerts or all recording rules
make alerts          # All 4 alert configs
make recording-rules # Both recording-rules configs

# Generate individually
make run-sl-services              # Alerting rules: SL queue, services datasource
make run-sre-hcps                 # Alerting rules: SRE queue, HCPs datasource
make run-rp-services              # Alerting rules: RP queue, services datasource
make run-rp-hcps                  # Alerting rules: RP queue, HCPs datasource
make run-msft-services            # Alerting rules: MSFT queue, services datasource
make run-recording-rules-services # Recording rules: services datasource
make run-recording-rules-hcps    # Recording rules: HCPs datasource

# Custom configuration
go run . --config-file path/to/config.yaml

Note: run, alerts, and recording-rules automatically run fmt-devinfra after generation. Individual run-* targets do not.

Command-line Options

• --config-file (required): Path to configuration YAML file
• --force-info-severity: Override all alert severities to "info" level (useful for testing)

Configuration

The tool expects a YAML configuration file with the following structure:

prometheusRules:
  # Directories containing rule files (each must have a corresponding _test file)
  rulesFolders:
    - path/to/rules

  # Rule files without tests (not recommended)
  untestedRules:
    - path/to/untested/rules.yaml

  # Output Bicep file path
  outputBicep: path/to/output.bicep

  # Default evaluation interval for rule groups (e.g., "1m")
  defaultEvaluationInterval: "1m"

  # Expression replacements (for platform-specific adjustments)
  outputReplacements:
    - from: 'original_expression'
      to: 'replaced_expression'

Rule Testing

All rules in rulesFolders must have corresponding test files:

• Rule file: alerts.yaml
• Test file: alerts_test.yaml

Tests are executed using promtool test rules during the generation process. If any test fails, the generation will abort.

Severity Mapping

Severity follows the Azure Common Engineering Naming (CEN) standard so alerts route cleanly into IcM. It is set independently of burn rate: burn rate decides when an alert fires, severity decides who is paged at what urgency.

Use the IcM severity number as the severity label: 2, 2.5, 3, or 4. The legacy critical / warning / info labels are still accepted (deprecated) and map to the same numbers. 1 is rejected (Azure CEN reserves Sev 1 for declared major incidents), and any other value fails generation rather than silently defaulting to Sev 4. The label value is a string, so both severity: 3 and severity: "3" are accepted.

Severity validation runs over every input rule, including upstream-managed untestedRules such as kubernetesControlPlane-prometheusRule.yaml (refreshed by make -C observability sync-upstream). A future upstream resync that introduces an unmapped severity will fail generation by design; if that happens, extend the mapping in severityFor when the new value is legitimate rather than disabling the check.

See: IcM best practices - Severity levels

Output

The tool generates Azure Bicep templates with two different formats:

Alerting Rules

• Output filename must contain AlertingRules
• Includes action group integrations for IcM
• Each alert includes:
  • Custom IcM title: #{cluster}: {description}
  • Correlation ID for proper incident aggregation
  • Severity mapping
  • All original labels and annotations

Recording Rules

• Output filename must contain RecordingRules
• Simpler structure without alerting-specific features
• Used to pre-compute frequently-used queries

Development

Prerequisites

• Go 1.x+
• promtool (from Prometheus)

Testing

# Run all tests
go test ./...

# Run with coverage
go test -cover ./...

Project Structure

.
├── main.go              # CLI entry point
├── main_test.go         # CLI tests
├── internal/
│   ├── generator.go     # Core rule generation logic
│   ├── generator_test.go
│   ├── writer.go        # Expression replacement utilities
│   └── writer_test.go
└── README.md

IcM Integration

The tool automatically configures IcM integration for alerting rules:

1. Correlation ID: Generated from alert name + cluster label + labels referenced in description
2. Title: Formatted as cluster: description
3. Action Groups: Referenced from Bicep parameters

For more information on IcM customization, see:

• Customize ICM Fields
• Prometheus IcM Connector Setup
• IcM Action Documentation

Known Limitations

• Query offsets are not supported (will generate a warning)
• Alert limits are not supported (will generate a warning)
• Minimum evaluation interval is 1 minute (shorter intervals will be adjusted)