promotion

Promotion Controller

The promotion controller orchestrates automatic PR promotion across environments in the cd-operator. It watches PullRequestTracker CRDs, loads promotion policies, triggers external tests, monitors results, and promotes to the next environment when all gates pass.

Architecture

The promotion package follows hexagonal architecture with clear domain boundaries:

internal/promotion/
├── controller.go      # Main reconciliation controller
├── policy.go          # Policy loading and caching
├── trigger.go         # External test triggering
├── monitor.go         # Test status monitoring
├── executor.go        # Promotion execution
├── types.go           # Domain types
└── controller_test.go # Comprehensive tests

Key Components

Reconciler (controller.go)

The main controller that orchestrates the promotion workflow:

1. Fetch PullRequestTracker - Get the CR being reconciled
2. Check readiness - Verify PR is in "synced" state
3. Load policy - Find PromotionPolicy for current environment
4. Trigger tests - Start external tests (if not already triggered)
5. Monitor tests - Poll test status until completion
6. Execute promotion - Update ArgoCD and GitHub on success
7. Update status - Record promotion in CR status

Requeue Strategy:

• Tests running: 30s (frequent polling)
• Tests passed: 10s (quick promotion)
• Promoted: 5m (check for next environment)
• Error: 1m (retry with backoff)
• Policy missing: 10m (infrequent check)

PolicyCache (policy.go)

Thread-safe cache of PromotionPolicy CRDs indexed by source environment.

Features:

• In-memory caching for performance
• Automatic refresh on cache misses
• Policy validation before use
• Support for multiple concurrent readers

Validation:

• Source/target environments are different
• Test names are unique
• Provider and job names are non-empty
• Timeouts are non-negative

TriggerManager (trigger.go)

Handles triggering of external tests in parallel.

Features:

• Parallel test execution using goroutines
• Variable substitution in parameters
• Test metadata tracking for status polling
• Graceful error handling (partial failures reported)

Variable Substitution: Supports placeholders in test parameters:

• ${pr.number} - PR number
• ${pr.head.sha} - HEAD commit SHA
• ${pr.repository} - Repository (owner/repo)
• ${cd.environment} - Target environment
• ${cd.version} - Extracted version
• ${custom.*} - Custom metadata

TestMonitor (monitor.go)

Monitors external test execution status and aggregates results.

Features:

• Parallel status polling
• Timeout detection
• Required vs optional test handling
• Aggregate result calculation

Result States:

• AllPassed - All required tests succeeded
• AnyFailed - At least one required test failed
• AnyRunning - Tests still executing
• AnyTimeout - Tests exceeded timeout

PromotionExecutor (executor.go)

Executes the actual promotion to the target environment.

Steps:

1. Update ArgoCD application to target environment
2. Add promotion label to GitHub PR (cd/promoted:env)
3. Record promotion in CR status (audit trail)
4. Emit Kubernetes event for observability

Audit Trail: Every promotion creates a PromotionRecord with:

• Source and target environments
• Timestamp
• Test results
• Policy name
• Promoted by (cd-operator)

Domain Types (types.go)

TestExecution

Tracks a triggered external test run:

type TestExecution struct {
    Name      string                // Test name
    Provider  external.Provider     // CI system (jenkins/github-actions/gitlab)
    RunID     string                // Unique identifier for polling
    StartedAt time.Time            // Trigger timestamp
    Required  bool                  // Must pass for promotion?
    JobName   string                // Job/workflow name
    URL       string                // Web link to test run
}

TestAggregateResult

Summarizes outcome of all tests:

type TestAggregateResult struct {
    AllPassed            bool
    AnyFailed            bool
    AnyRunning           bool
    AnyTimeout           bool
    Results              map[string]*external.TestStatus
    FailureReasons       []string
    OptionalTestsFailed  bool
}

PromotionResult

Result of a promotion attempt:

type PromotionResult struct {
    Promoted    bool
    TargetEnv   string
    TestResults TestAggregateResult
    Error       error
    PromotedAt  time.Time
    PromotedBy  string
    PolicyName  string
}

Usage Example

Creating a Reconciler

import (
    "github.com/grhili/cd-operator/internal/promotion"
    "github.com/grhili/cd-operator/pkg/external"
)

// Setup dependencies
testRunner := external.NewCompositeRunner(...)
argoCDClient := argocd.NewClient(...)
githubClient := github.NewClient(...)
eventRecorder := mgr.GetEventRecorderFor("promotion-controller")
logger := zap.NewLogger()

// Create reconciler
reconciler := promotion.NewReconciler(
    mgr.GetClient(),
    mgr.GetScheme(),
    testRunner,
    argoCDClient,
    githubClient,
    eventRecorder,
    logger,
)

// Register with controller-runtime
if err := reconciler.SetupWithManager(mgr); err != nil {
    log.Fatal(err)
}

Creating a PromotionPolicy

apiVersion: cd.grhili.io/v1alpha1
kind: PromotionPolicy
metadata:
  name: dev-to-staging
  namespace: default
spec:
  sourceEnvironment: dev
  targetEnvironment: staging
  autoPromote: true
  timeout: 45m
  externalTests:
    - name: integration-tests
      provider: github-actions
      jobName: integration.yml
      parameters:
        PR_NUMBER: "${pr.number}"
        COMMIT_SHA: "${pr.head.sha}"
        ENVIRONMENT: "${cd.environment}"
      required: true
    - name: smoke-tests
      provider: jenkins
      jobName: smoke-tests/job
      parameters:
        VERSION: "${cd.version}"
      required: false

PullRequestTracker Flow

State: synced (in dev environment)
  ↓
Load PromotionPolicy for "dev"
  ↓
Trigger external tests (integration-tests, smoke-tests)
  ↓
Monitor test status (poll every 30s)
  ↓
All required tests passed?
  ├─ Yes → Execute promotion to staging
  │        ├─ Update ArgoCD app
  │        ├─ Add GitHub label: cd/promoted:staging
  │        └─ Record in promotion history
  │
  └─ No → Block promotion (requeue with long interval)

Testing

The package includes comprehensive tests using the external testing pattern:

• ✅ PR not found (deleted)
• ✅ PR not in synced state
• ✅ No promotion policy found
• ✅ Trigger external tests
• ✅ Monitor tests while running
• ✅ Tests passed and promoted
• ✅ Tests failed (promotion blocked)
• ✅ Auto-promote disabled

Coverage: 74.3%

Run tests:

go test ./internal/promotion/... -v -cover

Conditions

The controller sets these conditions on PullRequestTracker:

Error Handling

• Policy not found: Requeue with 10m interval (may be created later)
• Test trigger failure: Requeue with 1m backoff
• Test status poll failure: Continue monitoring (transient failure)
• Promotion execution failure: Requeue with 1m backoff
• ArgoCD API failure: Return error for retry
• GitHub API failure: Log warning (labels are informational)

Metrics

The controller exposes these metrics (via controller-runtime):

• controller_runtime_reconcile_total - Total reconciliations
• controller_runtime_reconcile_errors_total - Reconciliation errors
• controller_runtime_reconcile_time_seconds - Reconciliation latency

Observability

Every promotion generates:

1. Kubernetes Event - For audit trail

   Type: Normal
   Reason: Promoted
   Message: Promoted from dev to staging using policy dev-to-staging. All tests passed.
   

2. Status Update - In PullRequestTracker CR

   status:
     currentEnvironment: staging
     promotionHistory:
       - fromEnvironment: dev
         toEnvironment: staging
         promotedAt: "2026-02-16T23:00:00Z"
         promotedBy: cd-operator
         policyName: dev-to-staging
         testResults: [...]
   

3. GitHub Label - On the PR

   cd/promoted:staging
   

Future Enhancements

• Support for manual promotion approval
• Promotion policies with multiple test stages
• Rollback on deployment failure
• Promotion rate limiting (gradual rollout)
• Integration with notification systems (Slack, email)
• Promotion dry-run mode
• Custom promotion strategies (blue/green, canary)