buildkite-serverless-agent

buildkite-serverless-agent

This project deploys a serverless buildkite agent which comprised of a bunch of AWS Lambda functions, an AWS codebuild project and an AWS stepfunction state machine which manages long running jobs.

The aim of this stack is:

1. Low IDLE cost
2. Running builds in an isolated sandboxed env, such as codebuild
3. Give the build container least privilege access to AWS
4. Cheap to run for small number of short (less than 5 minute) builds spread throughout the day

disclaimer

This project is not currently supported by buildkite, I built this as an experiment and currently it works really well for me. The buildkite team has been super supportive in my endeavour so a big thanks goes to them.

prerequisites

Before you start you need.

1. An AWS account with admin level access.
2. An S3 bucket in the account to stage cloudformation artifacts.
3. The AWS CLI.
4. direnv or the like to manage env variables using .envrc.
5. golang to build the lambda projects.
6. This project cloned into your $GOPATH at src/github.com/wolfeidau/buildkite-serverless-agent

conventions

Throughout the code you will see reference to environments, this convention enables you to run a group of dependent components which are linked together based on:

• EnvironmentName Name of the environment could be dev, test or prod.
• EnvironmentNumber Number for the environment, Typically 1 - n.

The other convention used is the linking of cloudformation stacks by passing in the name of a stack, rather than nesting or the like. This takes advantage of imports / exports, which you can read about at Exporting Stack Output Values

setup

Before we start we need to upload some configuration to amazon for the buildkite agent.

Note: You need to configure some environment variables as per the .envrc.example.

• This will import your agent SSH key (used to clone code from github/bitbucket/gitlab) into SSM and store it encrypted.

aws ssm put-parameter --name '/dev/1/buildkite-ssh-key' --value 'file://~/temp/id_rsa-testci' --type SecureString

• This will import your buildkite agent key into SSM and store it encrypted.

aws ssm put-parameter --name '/dev/1/buildkite-agent-key' --value 'xxxxx' --type SecureString

Then build and deploy all the serverless components.

make

This makefile will Launch a stack which deploys:

• A codebuild project to run the buildkite jobs.
• A stepfunction based job monitoring state machine project, and lambda functions.
• The agent lambda, which connects to buildkite and starts jobs using the stepfunction.

It also uploads the buildkite codebuild project which runs the buildkite-agent bootstrap process in codebuild. This is done by uploading a zip file named buildkite.zip to the S3 bucket created as a part of the buildkite codebuild project cloudformation. The template for this zip file is located at codebuild-template.

usage

There are a few overrides which can be added to your pipeline configuration in the buildkite site, these use env variables.

• CB_IMAGE_OVERRIDE Override the docker image to use.
• CB_COMPUTE_TYPE_OVERRIDE Override the compute type, options are BUILD_GENERAL1_SMALL | BUILD_GENERAL1_MEDIUM | BUILD_GENERAL1_LARGE.
• CB_PRIVILEGED_MODE_OVERRIDE Override whether or not privileged mode is enabled.

codebuild job monitor step functions

To enable monitoring of the codebuild job which could run for a few minutes I am using AWS step functions, this workflow is illustrated in the following image.

This workflow is triggered by the agent-pool lambda which polls the job queue via the buildkite REST API. Once triggered the state machine flags the job as in progress, streams logs to buildkite, and marks the job as complete once it is done.

The step-handler lambda function contains handlers for the following tasks within the step function:

• submit-job which notifies the buildkite api the job is starting and submits the job to codebuild.
• check-job which checks the status of the codebuild job and uploads logs every 10 seconds.
• complete-job which notifies the buildkite api the job is completed, either successful or failed, and uploads the remaining logs.

Note: This function uses STEP_HANDLER environment variable to dispatch to the correct handler.

todo

Still lots of things to tidy up:

• Secure all the lambda functions IAM profiles
• Testing
• Combine all the templates into one deployable unit
• Ensure all the step function lambdas are idempotent as they WILL retry at the moment.
• Currently only uploading 1MB of logs per 10 seconds, need to tune this and refactor the last upload to correctly flush the remaining data.
• Sort out versioning of the project and build files.
• Support canceled builds.
• Create a pool of agents to enable parallel builds and enable scale out.

Some notes / suggestions for the buildkite team:

• Draw a diagram of agent call flow and timeouts related
• Document the currently internal API from the agent project

authors

• Mark Wolfe @wolfeidau

license

This project is released under Apache 2.0 License.