lightsout

LightsOut

LightsOut is a Kubernetes operator that automatically scales down workloads during off-hours and restores them during business hours. This helps platform engineering teams save over 60% on development and staging cluster costs.

Define schedules as custom resources, and LightsOut takes care of the rest. You can scale Deployments, StatefulSets, and CronJobs across namespaces according to your configured timetable without any application changes. Original replica counts are automatically preserved and restored.

Why LightsOut?

Development and staging clusters are typically idle outside business hours, including evenings, nights, and weekends. This equates to approximately 70% of the week, during which you’re paying for compute resources that remain unused.

LightsOut, in conjunction with a node autoscaler like Karpenter, transforms idle time into savings.

1. LightsOut scales workloads to zero: Deployments, StatefulSets, and CronJobs are scaled down or suspended according to your schedule.
2. Karpenter removes empty nodes: With no workloads requesting resources, Karpenter (or Cluster Autoscaler) deprovisions the underlying nodes.
3. Cloud provider stops billing: Since there are no nodes, there are no compute charges.

When business hours resume, LightsOut restores workloads to their original replica counts, and your autoscaler provisions nodes to meet the increased demand.

[!NOTE] LightsOut manages the workload layer, while your node autoscaler handles the infrastructure layer. Together, they eliminate idle compute costs.

Features

• Cron-based scheduling with IANA timezone support
• Manages Deployments, StatefulSets, and CronJobs - scales replicas to zero or suspends CronJobs
• HPA-aware scaling - automatically patches spec.minReplicas on attached HorizontalPodAutoscalers to prevent fight-back, then restores on upscale
• Flexible namespace targeting - label selectors, explicit lists, and exclusions
• Namespace-scoped schedules - developers can define their own schedules per namespace, independent of global schedules
• Rate-limited scaling - batch workloads to avoid resource spikes
• Admission webhooks - validates schedules and detects overlaps before they're applied
• ArgoCD integration - optional labeling of ArgoCD Application CRDs to prevent false alerts during downscale
• FluxCD integration - optional suspension of FluxCD Kustomization and HelmRelease resources to prevent drift detection and alert noise during downscale
• Prometheus metrics - observe schedule state, scaling operations, errors, and durations

Quick Start

Install with Helm (webhooks disabled for simplicity):

helm install lightsout oci://ghcr.io/gjorgji-ts/charts/lightsout \
  --set webhook.enabled=false \
  --set certManager.enabled=false

Create a schedule:

apiVersion: lightsout.techsupport.mk/v1alpha1
kind: LightsOutSchedule
metadata:
  name: dev-weekday-hours
spec:
  upscale: "0 6 * * 1-5"        # 6 AM Monday–Friday
  downscale: "0 18 * * 1-5"     # 6 PM Monday–Friday
  timezone: "America/New_York"
  namespaceSelector:
    matchLabels:
      environment: dev

Check status:

kubectl get lightsoutschedules

NAME               STATE   UPSCALE       DOWNSCALE     SUSPENDED   AGE
dev-weekday-hours  Up      0 6 * * 1-5   0 18 * * 1-5  false       7d

For production setups with webhook validation, see the Setup Guide.

How It Works

LightsOut runs as a controller that watches two custom resource types:

• LightsOutSchedule (cluster-scoped) - for platform teams managing cost policies across multiple namespaces. Supports label selectors and explicit namespace lists.
• LightsOutNamespaceSchedule (namespace-scoped) - for developers who want to define their own schedule for their namespace. When a namespace schedule exists, any global schedule targeting that namespace automatically skips it.

On each reconciliation, the controller calculates whether the current time falls in an "up" or "down" period based on your cron expressions, discovers target namespaces and workloads, and scales accordingly. Original replica counts are stored in annotations so they can be restored exactly.

For a deeper look at the architecture, see docs/architecture.md.

Configuration

LightsOutSchedule (cluster-scoped)

Managed by platform teams. Targets workloads across one or more namespaces.

At least one of namespaceSelector or namespaces must be specified.

LightsOutNamespaceSchedule (namespace-scoped)

Created by developers in their own namespace. No namespace selection fields - the schedule always manages the namespace it lives in. Supports all the same scheduling fields as LightsOutSchedule.

apiVersion: lightsout.techsupport.mk/v1alpha1
kind: LightsOutNamespaceSchedule
metadata:
  name: team-hours
  namespace: team-a
spec:
  upscale: "0 8 * * 1-5"        # 8 AM Monday–Friday
  downscale: "0 20 * * 1-5"     # 8 PM Monday–Friday
  timezone: "Europe/Berlin"

When this resource exists in a namespace, any LightsOutSchedule targeting that namespace will skip it automatically.

Excluding Workloads

To protect specific workloads from scaling, use excludeLabels on the schedule:

spec:
  excludeLabels:
    matchLabels:
      critical: "true"

Any workloads with matching labels will be skipped during scaling operations.

Rate Limiting

Gradually scale workloads in batches to avoid resource spikes:

spec:
  downscaleRateLimit:
    batchSize: 10
    delayBetweenBatches: "5s"

ArgoCD Integration

If you use ArgoCD, enabling the argoCD field prevents ArgoCD from firing false "Degraded" or "OutOfSync" alerts when LightsOut scales workloads down:

spec:
  argoCD:
    namespace: argocd    # optional, defaults to "argocd"

LightsOut labels ArgoCD Application CRDs with lightsout.techsupport.mk/state: down during downscale and removes the labels on upscale. See the ArgoCD Integration Guide for details.

FluxCD Integration

If you use FluxCD, enabling the fluxCD field suspends matching Kustomization and HelmRelease resources during downscale, preventing FluxCD from reconciling workloads back to their Git-defined state while they are intentionally scaled to zero:

spec:
  fluxCD:
    namespace: flux-system    # optional, defaults to "flux-system"

LightsOut sets spec.suspend: true on matching Flux resources during downscale and resumes them (with a warming-up grace period) on upscale. See the FluxCD Integration Guide for details.

Note: You must opt in to the required RBAC by setting rbac.fluxcd: true in your Helm values.

Observability

LightsOut exposes Prometheus metrics on the metrics endpoint:

Scaling events are also recorded as Kubernetes Events on the LightsOutSchedule and LightsOutNamespaceSchedule resources.

Using with Karpenter

LightsOut is designed to work seamlessly with Karpenter to achieve maximum cost savings. There’s no need for any special configuration, the two systems automatically complement each other.

• LightsOut monitors cron schedules and scales workloads to zero during off-hours.
• Karpenter monitors node utilization and removes nodes that are no longer needed.

As long as Karpenter’s NodePool consolidation policy is enabled (the default setting), empty nodes are drained and terminated within minutes of LightsOut scaling workloads down.

This approach also works with Cluster Autoscaler. Any node autoscaler that deprovisions underutilized nodes will produce the same effect.

Documentation

• Architecture - how LightsOut works internally
• HPA Integration - automatic HorizontalPodAutoscaler handling
• ArgoCD Integration - prevent false alerts when scaling down
• FluxCD Integration - prevent drift detection when scaling down
• Setup Guide - installation with and without webhooks
• Security Model - RBAC, risks, and mitigations
• Examples - sample schedule configurations

License

Apache License 2.0 - see LICENSE for details.