kube-trigger

kube-trigger

kube-trigger is a tool that combines event listeners and action triggers.

Overview

Although there is kube in the name, it is actually not limited to Kubernetes and can do much more than that. It has an extensible architecture that can extend its capabilities fairly easily. We have docs (not yet) on how to extend Sources, Filters, and Actions. All users are welcomed to contribute their own extensions.

Sources

A Source is what listens to events (event source). For example, a k8s-resource-watcher source can watch Kubernetes resources. Once a Kubernetes resource (e.g. ConfigMap) is changed, it will raise an event that will be passed to Filters for further processing.

Filters

A Filter will filter the events that are raised by Sources, i.e, drop events that do not satisfy a certain criteria. For example, users can use a cue-validator Filter to filter out events by Kubernetes resource names. All the events that passed the Filters will then trigger an Action.

Actions

An Action is a job that does what the user specified when an event happens. For example, the user can send notifications, log events, execute a command, or patch some Kubernetes objects when an event happens.

Quick Start

To quickly know the concepts of kube-trigger, let's use a real use-case as an exmaple ( see #4418). TL;DR, the user want the Application to be automatically updated whenever the ConfigMaps that are referenced by ref-objects are updated.

To accomplish this, we will:

• use a k8s-resource-watcher Source to listen to update events of ConfigMaps
• use a cue-validator Filter to only keep the ConfigMaps that we are interested in
• trigger an bump-application-revision Action to update Application.

See examples directory for instructions.

Configuration File

A config file instructs kube-trigger to use what Sources, Filters, and Actions, and how they are configured.

No matter you are running kube-trigger as standalone or in-cluster, the config format is similar, so it is beneficial to know the format first. We will use yaml format as an example (json and cue are also supported).

# A trigger is a group of Source, Filters, and Actions.
# You can add multiple triggers.
triggers:
  - your-source-type:
      # ... properties
    filters:
      - your-filter-0-type:
          # ... properties
    actions:
      - your-action-0-type:
          # ... properties

Standalone

When running in standalone mode, you will need to provide a config file to kube-trigger binary.

kube-trigger can accept cue, yaml, and json config files. You can also specify a directory to load all the supported files inside that directory. -c/--config cli flag and CONFIG environment variable can be used to specify config file.

An example config file looks like this:

# A trigger is a group of Source, Filters, and Actions.
# You can add multiple triggers.
triggers:
  - k8s-resource-watcher:
      # We are interested in ConfigMap events.
      apiVersion: "v1"
      kind: ConfigMap
      namespace: default
      # Only watch update event.
      events:
        - update
    filters:
      # Filter the events above.
      - cue-validator:
          # Filter by validating the object data using CUE.
          # For example, we are filtering by ConfigMap names (metadata.name) from above.
          # Only ConfigMaps with names that satisfy this regexp "this-will-trigger-update-.*" will be kept.
          template: |
            metadata: name: =~"this-will-trigger-update-.*"
    actions:
      # Bump Application Revision to update Application.
      - bump-application-revision:
          namespace: default
          # Select Applications to bump using labels.
          labelSelectors:
            my-label: my-value

Let's assume your config file is config.yaml, to run kube-trigger:

• ./kube-trigger -c=config.yaml
• CONFIG=config.yaml ./kube-trigger

In-Cluster

We have two CRDs: TriggerInstance and TriggerService.

• TriggerInstance is what creates a kube-trigger instance (similar to running ./kube-trigger in-cluster but no config is provided). Advanced kube-trigger Instance Configuration (next section) can be provided in it.
• TriggerService is used to provide one or more configs (same as the config file you use when running as standalone) to a TriggerInstance.

So we know TriggerService is what actually provides a config, this is what we will be discussing.

# You can find this file in config/samples/standard_v1alpha1_triggerservice.yaml
apiVersion: standard.oam.dev/v1alpha1
kind: TriggerService
metadata:
  name: kubetrigger-sample-config
  namespace: default
spec:
  selector:
    app: kubetrigger-sample
  triggers:
    - k8s-resource-watcher:
        apiVersion: "v1"
        kind: ConfigMap
        namespace: default
        events:
          - update
      filters:
        - cue-validator:
            template: |
              // Filter by object name.
              // I used regular expressions here.
              metadata: name: =~"this-will-trigger-update-.*"
      actions:
        - bump-application-revision:
            namespace: default
            labelSelectors:
              my-label: my-value

Advanced kube-trigger Instance Configuration

In addition to config files, you can also do advanced configurations. Advanced kube-trigger Instance Configurations are internal configurations to fine-tune your kube-trigger instance. In most cases, you probably don't need to fiddle with these settings.

Log Level

Frequently-used values: debug, info, error

Default: info

Action Retry

Re-run Action if it fails.

Default: false

Max Retry

Max retry count if an Action fails, valid only when action retrying is enabled.

Default: 5

Retry Delay

First delay to retry actions in seconds, subsequent delay will grow exponentially, valid only when action retrying is enabled.

Default: 2

Per-Worker QPS

Long-term QPS limiting per Action worker, this is shared between all watchers.

Default: 2

Queue Size

Queue size for running actions, this is shared between all watchers

Default: 50

Job Timeout

Timeout for running each action in seconds.

Default: 10

Worker Count

Number of workers for running actions, this is shared between all watchers.

Default: 4

Registry Size

Cache size for filters and actions.

Default: 100

Roadmap

v0.0.1-alpha.x

• Basic build infrastructure
• Complete a basic proof-of-concept sample
• linters, license checker
• GitHub Actions
• Rate-limited worker
• Make the configuration as CRD, launch new process/pod for new watcher
• Notification for more than one app: selector from compose of Namespace; Labels; Name
• Refine README, quick starts
• Refactor CRD according to #2

v0.0.1-beta.x

Code enhancements

• Add missing unit tests
• Add missing integration tests

v0.0.x

User experience

• Refine health status of CRs
• Make it run as Addon, build component definition, and examples
• Kubernetes dynamic admission control with validation webhook
• Auto-generate usage docs of Sources, Filters, and Actions from CUE markers
• Show available Sources, Filters, and Actions in cli

v0.1.x

Webhook support

• Contribution Guide
• New Action: webhook
• New Source: webhook

v0.2.x

Observability

• New Action: execute VelaQL(CUE and K8s operations)
• New Source: cron
• New Action: notifications(email, dingtalk, slack, telegram)
• New Action: log (loki, clickhouse)

Planned for later releases

• Allow user set custom RBAC for each TriggerInstance
• New Action: workflow-run
• New Action: execute-command
• New Action: metric (prometheus)
• Refine controller logic
• Remove cache informer, make it with no catch but list watch events with unique queue.