kayron

command module
v0.5.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 29, 2025 License: MIT Imports: 2 Imported by: 0

README

kayron

This repository contains the operator responsible for our automated change management. The relevant components here are described by the server and operator packages. Kayron's sole responsibility is to keep our infrastructure and services up to date, according to our release artifacts managed in Github.

Kayron Overview

Server

The server is a simple HTTP backend exposing the Kayron's /metrics endopoint.

curl -s http://127.0.0.1:7777/metrics
# HELP go_gc_duration_seconds A summary of the wall-time pause (stop-the-world) duration in garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
Worker

The worker is a custom task engine executing a directed acyclic graph of worker handlers iteratively. New worker handlers can be added easily by implementing the handler interface and registering the new handler in the operator chain. Kayron's operator chain is executed by a sequential worker engine.

type Interface interface {
	// Ensure is the minimal worker handler interface that all users have to
	// implement for their own business logic, regardless of the underlying worker
	// engine.
	//
	// Ensure executes the handler specific business logic in order to complete
	// the given task, if possible. Any error returned will be emitted using the
	// underlying logger interface, unless the injected metrics registry is
	// configured to filter the received error.
	Ensure() error
}
Operator

Kayron is a change management controller implementing the operator pattern, in this particular case without the involvement of Kubernetes. The main goroutine for the operator's reconciliation loop is the operator worker handler running a sequence of steps according to their operator functions located in pkg/operator. Secondary worker handlers may be executed within their own isolated failure domain.

Operators try to continuously drive the current state of a system towards the desired state of a system. In our case, the current state is represented by the already deployed CloudFormation templates in AWS. The desired state is then represented by the template changes introduced in any given environment. Those changes my simply be a version change from v1.8.2 to v1.8.3 in case of a service update, or a more complex CloudFormation template change that modifies physical resources in AWS. Kayron’s job is to continuously check for any detectable drift between current and desired state, and apply any changes made upon detection.

For instance, if a new Release were to be created in the Specta repository, then Kayron will generate all CloudFormation templates anew using the changed resource details in the underlying cache channels, and reconcile the resulting CloudFormation templates against AWS, if the specified policies permit the proposed update. This process will then eventually result in a stack update in CloudFormation, so that Kayron may also keep the respective deployment status up to date in Github.

Operator Design

Triggers
  1. The operator executes its graph of business logic sequentially on a time based schedule. E.g. the entire chain runs every 10 seconds and pulls all the data it needs from external APIs.
  2. The operator can also executes its graph on demand in case a trigger event like a webhook notification may occurred. In this case the time based schedule described above resets so that the received webhook event creates a new 10 second interval.
Usage

At its core, Kayron is a simple Cobra command line tool, providing e.g. the daemon command to start the long running server and worker processes.

kayron -h
Golang based operator microservice.

Usage:
  kayron [flags]
  kayron [command]

Available Commands:
  daemon      Execute Kayron's long running process for running the operator.
  deploy      Manually trigger a CloudFormation stack update.
  lint        Validate the release configuration under the given path.
  version     Print the version information for this command line tool.

Flags:
  -h, --help   help for kayron

Use "kayron [command] --help" for more information about a command.
Development

As a convention, Kayron's .env file should remain simple and generic. A reasonable setting within that config file is e.g. KAYRON_LOG_LEVEL. Kayron requires the standard AWS credentials format to be properly setup. More specific organization and environment related settings must be provided separately. And so running e.g. the Kayron daemon requires several additional environment variables to be injected.

  • KAYRON_CLOUDFORMATION_STACK, the CloudFormation stack name to reconcile, e.g. server-test.
  • KAYRON_ENVIRONMENT, the environment Kayron is running in, one of development testing staging production.
  • KAYRON_GITHUB_TOKEN, the Github token to use for fetching releases, requires the repo scope.
  • KAYRON_RELEASE_SOURCE, the Github repository containing releases, e.g. https://github.com/0xSplits/releases.
  • KAYRON_S3_BUCKET, the S3 bucket to upload CloudFormation templates, e.g. splits-cf-templates.

  • KAYRON_CLOUDFORMATION_PARAMETERS, the optional CloudFormation parameter overwrites, in the format key:value,foo:bar.
  • KAYRON_CLOUDFORMATION_TAGS, the optional CloudFormation tag overwrites, in the format key:value,foo:bar.
kayron daemon
{ "time":"2025-07-04 14:09:06", "level":"info", "message":"daemon is launching procs", "environment":"development", "caller":".../pkg/daemon/daemon.go:38" }
{ "time":"2025-07-04 14:09:06", "level":"info", "message":"server is accepting calls", "address":"127.0.0.1:7777",  "caller":".../pkg/server/server.go:95" }
{ "time":"2025-07-04 14:09:06", "level":"info", "message":"worker is executing tasks", "pipelines":"1",             "caller":".../pkg/worker/worker.go:110" }

There is an integration-test workflow to verify several aspects of Kayron's various responsibilities. The test verifies that all operator functions are free from race conditions, which is critical, because several operator functions are running in parallel each and every reconciliation loop. The test also verifies that the operator's reconciliation loops are not unexpectedly cancelled due to internal data inconsistencies when looking all kinds of current and desired state for the internal artifact cache. The required AWS credentials need the ViewOnlyAccess, AmazonEC2ContainerRegistryReadOnly and ResourceGroupsandTagEditorReadOnlyAccess permissions. The required Github auth token needs the repository scope.

go test -tags=integration ./pkg/operator -v -race -run Test_Operator_Integration
=== RUN   Test_Operator_Integration
{
    "time": "2025-09-17 15:31:59",
    "level": "debug",
    "message": "resetting operator cache",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/cache/delete.go:9"
}
{
    "time": "2025-09-17 15:32:01",
    "level": "debug",
    "message": "resolved ref for github repository",
    "environment": "testing",
    "ref": "176ffefa272b210eb3be269887d665a682dbd548",
    "repository": "https://github.com/0xSplits/releases",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/release/ensure.go:66"
}
{
    "time": "2025-09-17 15:32:02",
    "level": "debug",
    "message": "caching release artifact",
    "deploy": "branch=preview",
    "github": "infrastructure",
    "preview": "false",
    "provider": "cloudformation",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/cache/create.go:17"
}
{
    "time": "2025-09-17 15:32:02",
    "level": "debug",
    "message": "caching release artifact",
    "deploy": "release=v0.1.1",
    "docker": "splits-lite",
    "github": "splits-lite",
    "preview": "false",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/cache/create.go:17"
}
{
    "time": "2025-09-17 15:32:02",
    "level": "debug",
    "message": "caching release artifact",
    "deploy": "branch=fancy-feature-branch",
    "docker": "splits-lite",
    "github": "splits-lite",
    "preview": "true",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/cache/create.go:17"
}
{
    "time": "2025-09-17 15:32:02",
    "level": "debug",
    "message": "caching release artifact",
    "deploy": "branch=preview",
    "docker": "kayron",
    "github": "kayron",
    "preview": "false",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/cache/create.go:17"
}
{
    "time": "2025-09-17 15:32:02",
    "level": "debug",
    "message": "caching release artifact",
    "deploy": "release=v0.2.2",
    "docker": "specta",
    "github": "specta",
    "preview": "false",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/cache/create.go:17"
}
{
    "time": "2025-09-17 15:32:02",
    "level": "debug",
    "message": "instrumented worker handler",
    "handler": "release",
    "latency": "3.727451s",
    "success": "true",
    "caller": "/Users/xh3b4sd/go/pkg/mod/github.com/0x!splits/workit@v0.6.0/handler/metrics/ensure.go:55"
}
{
    "time": "2025-09-17 15:32:02",
    "level": "debug",
    "message": "caching current state",
    "current": "d5fa88afd502edc9052f89c956618b2cb567d984",
    "github": "infrastructure",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/template/ensure.go:35"
}
{
    "time": "2025-09-17 15:32:02",
    "level": "debug",
    "message": "instrumented worker handler",
    "handler": "template",
    "latency": "173.083µs",
    "success": "true",
    "caller": "/Users/xh3b4sd/go/pkg/mod/github.com/0x!splits/workit@v0.6.0/handler/metrics/ensure.go:55"
}
{
    "time": "2025-09-17 15:32:02",
    "level": "debug",
    "message": "caching desired state",
    "desired": "v0.1.1",
    "github": "splits-lite",
    "preview": "false",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/reference/ensure.go:45"
}
{
    "time": "2025-09-17 15:32:02",
    "level": "debug",
    "message": "caching desired state",
    "desired": "v0.2.2",
    "github": "specta",
    "preview": "false",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/reference/ensure.go:45"
}
{
    "time": "2025-09-17 15:32:03",
    "level": "debug",
    "message": "caching desired state",
    "desired": "d5fa88afd502edc9052f89c956618b2cb567d984",
    "github": "infrastructure",
    "preview": "false",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/reference/ensure.go:45"
}
{
    "time": "2025-09-17 15:32:03",
    "level": "debug",
    "message": "caching desired state",
    "desired": "eb9f56e195f25218499897a6c6d79c62261faba5",
    "github": "kayron",
    "preview": "false",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/reference/ensure.go:45"
}
{
    "time": "2025-09-17 15:32:03",
    "level": "debug",
    "message": "caching desired state",
    "desired": "e307253e62c2da119579e2505db0b6185642ab9b",
    "github": "splits-lite",
    "preview": "true",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/reference/ensure.go:45"
}
{
    "time": "2025-09-17 15:32:03",
    "level": "debug",
    "message": "instrumented worker handler",
    "handler": "reference",
    "latency": "367.2485ms",
    "success": "true",
    "caller": "/Users/xh3b4sd/go/pkg/mod/github.com/0x!splits/workit@v0.6.0/handler/metrics/ensure.go:55"
}
{
    "time": "2025-09-17 15:32:04",
    "level": "debug",
    "message": "caching current state",
    "current": "bc7891268e44f62e0aebbe339c0850b61d52c417",
    "docker": "splits-lite",
    "preview": "false",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/container/cache.go:10"
}
{
    "time": "2025-09-17 15:32:04",
    "level": "debug",
    "message": "caching current state",
    "current": "bc7891268e44f62e0aebbe339c0850b61d52c417",
    "docker": "splits-lite",
    "preview": "true",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/container/cache.go:10"
}
{
    "time": "2025-09-17 15:32:04",
    "level": "debug",
    "message": "caching current state",
    "current": "eb9f56e195f25218499897a6c6d79c62261faba5",
    "docker": "kayron",
    "preview": "false",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/container/cache.go:10"
}
{
    "time": "2025-09-17 15:32:04",
    "level": "debug",
    "message": "caching current state",
    "current": "v0.2.2",
    "docker": "specta",
    "preview": "false",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/container/cache.go:10"
}
{
    "time": "2025-09-17 15:32:04",
    "level": "debug",
    "message": "instrumented worker handler",
    "handler": "container",
    "latency": "1.705606292s",
    "success": "true",
    "caller": "/Users/xh3b4sd/go/pkg/mod/github.com/0x!splits/workit@v0.6.0/handler/metrics/ensure.go:55"
}
{
    "time": "2025-09-17 15:32:05",
    "level": "debug",
    "message": "executed image check",
    "exists": "true",
    "image": "splits-lite",
    "preview": "true",
    "tag": "e307253e62c2da119579e2505db0b6185642ab9b",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/registry/ensure.go:45"
}
{
    "time": "2025-09-17 15:32:05",
    "level": "debug",
    "message": "executed image check",
    "exists": "true",
    "image": "splits-lite",
    "preview": "false",
    "tag": "v0.1.1",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/registry/ensure.go:45"
}
{
    "time": "2025-09-17 15:32:05",
    "level": "debug",
    "message": "instrumented worker handler",
    "handler": "registry",
    "latency": "875.278125ms",
    "success": "true",
    "caller": "/Users/xh3b4sd/go/pkg/mod/github.com/0x!splits/workit@v0.6.0/handler/metrics/ensure.go:55"
}
{
    "time": "2025-09-17 15:32:05",
    "level": "info",
    "message": "continuing reconciliation loop",
    "reason": "detected state drift",
    "release": "splits-lite",
    "version": "v0.1.1",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/policy/ensure.go:68"
}
{
    "time": "2025-09-17 15:32:05",
    "level": "info",
    "message": "continuing reconciliation loop",
    "domain": "1d0fd508.lite.testing.splits.org",
    "reason": "detected state drift",
    "release": "splits-lite",
    "version": "e307253e62c2da119579e2505db0b6185642ab9b",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/policy/ensure.go:68"
}
{
    "time": "2025-09-17 15:32:05",
    "level": "debug",
    "message": "instrumented worker handler",
    "handler": "policy",
    "latency": "523.166µs",
    "success": "true",
    "caller": "/Users/xh3b4sd/go/pkg/mod/github.com/0x!splits/workit@v0.6.0/handler/metrics/ensure.go:55"
}
{
    "time": "2025-09-17 15:32:05",
    "level": "debug",
    "message": "resolved ref for github repository",
    "environment": "testing",
    "ref": "d5fa88afd502edc9052f89c956618b2cb567d984",
    "repository": "https://github.com/0xSplits/infrastructure",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/infrastructure/ensure.go:24"
}
{
    "time": "2025-09-17 15:32:05",
    "level": "debug",
    "message": "uploading cloudformation template",
    "bucket": "splits-cf-templates",
    "key": "testing/deployment/deployment.yaml",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/infrastructure/aws.go:26"
}
{
    "time": "2025-09-17 15:32:06",
    "level": "debug",
    "message": "uploading cloudformation template",
    "bucket": "splits-cf-templates",
    "key": "testing/discovery/discovery.yaml",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/infrastructure/aws.go:26"
}
{
    "time": "2025-09-17 15:32:06",
    "level": "debug",
    "message": "uploading cloudformation template",
    "bucket": "splits-cf-templates",
    "key": "testing/elasticache/elasticache.yaml",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/infrastructure/aws.go:26"
}
{
    "time": "2025-09-17 15:32:06",
    "level": "debug",
    "message": "uploading cloudformation template",
    "bucket": "splits-cf-templates",
    "key": "testing/fargate/fargate.yaml",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/infrastructure/aws.go:26"
}
{
    "time": "2025-09-17 15:32:06",
    "level": "debug",
    "message": "uploading cloudformation template",
    "bucket": "splits-cf-templates",
    "key": "testing/index.yaml",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/infrastructure/aws.go:26"
}
{
    "time": "2025-09-17 15:32:07",
    "level": "debug",
    "message": "uploading cloudformation template",
    "bucket": "splits-cf-templates",
    "key": "testing/kayron/kayron.yaml",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/infrastructure/aws.go:26"
}
{
    "time": "2025-09-17 15:32:07",
    "level": "debug",
    "message": "uploading cloudformation template",
    "bucket": "splits-cf-templates",
    "key": "testing/rds/rds.alarms.yaml",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/infrastructure/aws.go:26"
}
{
    "time": "2025-09-17 15:32:07",
    "level": "debug",
    "message": "uploading cloudformation template",
    "bucket": "splits-cf-templates",
    "key": "testing/rds/rds.yaml",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/infrastructure/aws.go:26"
}
{
    "time": "2025-09-17 15:32:07",
    "level": "debug",
    "message": "uploading cloudformation template",
    "bucket": "splits-cf-templates",
    "key": "testing/server/server.yaml",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/infrastructure/aws.go:26"
}
{
    "time": "2025-09-17 15:32:08",
    "level": "debug",
    "message": "uploading cloudformation template",
    "bucket": "splits-cf-templates",
    "key": "testing/specta/specta.yaml",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/infrastructure/aws.go:26"
}
{
    "time": "2025-09-17 15:32:08",
    "level": "debug",
    "message": "uploading cloudformation template",
    "bucket": "splits-cf-templates",
    "key": "testing/splits-lite/splits-lite.yaml",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/infrastructure/aws.go:26"
}
{
    "time": "2025-09-17 15:32:08",
    "level": "debug",
    "message": "uploading cloudformation template",
    "bucket": "splits-cf-templates",
    "key": "testing/telemetry/telemetry.yaml",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/infrastructure/aws.go:26"
}
{
    "time": "2025-09-17 15:32:08",
    "level": "debug",
    "message": "uploading cloudformation template",
    "bucket": "splits-cf-templates",
    "key": "testing/vpc/vpc.yaml",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/infrastructure/aws.go:26"
}
{
    "time": "2025-09-17 15:32:08",
    "level": "debug",
    "message": "instrumented worker handler",
    "handler": "infrastructure",
    "latency": "3.615857209s",
    "success": "true",
    "caller": "/Users/xh3b4sd/go/pkg/mod/github.com/0x!splits/workit@v0.6.0/handler/metrics/ensure.go:55"
}
{
    "time": "2025-09-17 15:32:08",
    "level": "info",
    "message": "updating cloudformation stack",
    "name": "server-test",
    "url": "https://splits-cf-templates.s3.us-west-2.amazonaws.com/testing/index.yaml",
    "caller": "/Users/xh3b4sd/project/0xSplits/kayron/pkg/operator/cloudformation/ensure.go:30"
}
{
    "time": "2025-09-17 15:32:08",
    "level": "debug",
    "message": "instrumented worker handler",
    "handler": "cloudformation",
    "latency": "190.125µs",
    "success": "true",
    "caller": "/Users/xh3b4sd/go/pkg/mod/github.com/0x!splits/workit@v0.6.0/handler/metrics/ensure.go:55"
}
--- PASS: Test_Operator_Integration (9.93s)
PASS
ok  	github.com/0xSplits/kayron/pkg/operator	11.224s
Releases

In order to update the Docker image, prepare all desired changes within the main branch and create a Github release for the desired Kayron version. The release tag should be in Semver Format. Creating the Github release triggers the responsible Github Action to build and push the Docker image to the configured Amazon ECR.

v0.1.11

The version command kayron version and the version endpoint /version provide build specific version information about the build and runtime environment. A live demo can be seen at https://kayron.testing.splits.org/version.

Docker

Kayron's build artifact is a statically compiled binary running in a distroless image for maximum security and minimum size. If you do not have Go installed and just want to run Kayron locally in a Docker container, then use the following commands.

docker build \
  --build-arg SHA="local-test-sha" \
  --build-arg TAG="local-test-tag" \
  -t kayron:local .
docker run \
  -e KAYRON_ENVIRONMENT=development \
  -p 7777:7777 \
  kayron:local \
  daemon

Documentation

The Go Gopher

There is no documentation for this package.

Directories

Path Synopsis
cmd
pkg
operator/cloudformation
Package cloudformation triggers CloudFormation stack updates once a valid state drift was detected.
Package cloudformation triggers CloudFormation stack updates once a valid state drift was detected.
operator/container
Package container fetches the current state of the currently deployed container image tags.
Package container fetches the current state of the currently deployed container image tags.
operator/infrastructure
Package infrastructure prepares the desired state of the CloudFormation templates provided within the configured infrastructure repository.
Package infrastructure prepares the desired state of the CloudFormation templates provided within the configured infrastructure repository.
operator/logging
Package logging emits log messages for each and every reconciliation loop about the most important aspects of the current state of the system.
Package logging emits log messages for each and every reconciliation loop about the most important aspects of the current state of the system.
operator/preview
Package preview injects preview deployments into our internal release artifact cache.
Package preview injects preview deployments into our internal release artifact cache.
operator/reference
Package reference fetches the desired state of any service release source code as Git reference.
Package reference fetches the desired state of any service release source code as Git reference.
operator/registry
Package registry verifies whether the desired service version is readily available inside the configured container registry.
Package registry verifies whether the desired service version is readily available inside the configured container registry.
operator/release
Package release implements the source business logic that caches all relevant service release settings for further use across the operator chain.
Package release implements the source business logic that caches all relevant service release settings for further use across the operator chain.
operator/release/resolver
Package resolver tries to provide the environment specific Git reference for the release source repository.
Package resolver tries to provide the environment specific Git reference for the release source repository.
operator/status
Package status manages deployment status updates so that humans can better understand the current state of the underlying systems.
Package status manages deployment status updates so that humans can better understand the current state of the underlying systems.
operator/template
Package template fetches the current state of the currently deployed template version parameter.
Package template fetches the current state of the currently deployed template version parameter.
policy
Package policy inspects and verifies any potential state drift in order to either allow the reconciliation loop to continue, or cancel it.
Package policy inspects and verifies any potential state drift in order to either allow the reconciliation loop to continue, or cancel it.
release/artifact
Package artifact defines the structural details of any given release artifact, so that the respective current and desired state can be mapped to their source code.
Package artifact defines the structural details of any given release artifact, so that the respective current and desired state can be mapped to their source code.
release/schema
Package schema defines the in-memory representation of Kayron’s change management definitions.
Package schema defines the in-memory representation of Kayron’s change management definitions.
scanner
Package scanner is a multi line block scanner for YAML input bytes.
Package scanner is a multi line block scanner for YAML input bytes.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL