qaz

qaz—A CLI tool for Templating & Managing stacks in AWS Cloudformation

Qaz is a Fork of the Bora project by @pkazmierczakthat focuses on simplifying the process of deploying infrastructure on AWS via Cloudformation by utilising the Go Templates Library and custom functions to generate diverse and configurable templates.

Qaz emphasizes minimal abstraction from the underlying AWS Cloudformation Platform. It instead enhances customisability and re-usability of templates through dynamic template generation and logic.

--

Features:

• Advanced template functionality & custom built-in template functions

• Support for Cloudformation templates written in JSON & YAML

• Dynamic deploy script generation utilising the built-in templating functionality

• Single YAML Configuration file for multiple stack templates per environment

• Utilises Go-routines for Multi-stack concurrent Cloudformation requests for all appropriate calls

• Support for AWS Profile selection for Multi-AWS account environments

• Cross stack referencing with support for Cloudformation Exports(Preferred) & dynamically retrieving stack outputs on deploy

• Decoupled build mechanism. Qaz can manage infrastructure by accessing config/templates via S3 or HTTP(S). The tool does not need to be in the same place as the files.

• Decoupled stack management. Stacks can be launched individually from different locations and build consistently according to the dependency chain as long as the same configuration file is read.

• Encryption & Decryption of template values & deployment of encrypted templates using AWS KMS.

Installation

If you have Golang installed:

go get github.com/daidokoro/qaz

On Mac or Linux:

curl https://raw.githubusercontent.com/daidokoro/qaz/master/install.sh | sh

Or, you may need sudo:

curl https://raw.githubusercontent.com/daidokoro/qaz/master/install.sh | sudo sh

Windows EXE coming soon....

Requirements

qaz requires:

• AWS credentials, you can read about how to set these up here

Initialize Project

How It Works!

Qaz uses a main config.yml file as its source-of-truth. The file tells it what stacks it controls and the values to pass to those stacks.

# Specify the AWS region code
# qaz will attempt to get it from AWS configuration
# or from the environment. This setting overrides
# every other.
region: "eu-west-1"

# Required: The project name is prepended to the
# stack names at build time to create unique
# identifier on the Cloudformation platform
project: "daidokoro"

# Optional: global values, accessible across
# all stacks can be define under global
global:


# All stack specific values are defined
# under the "stacks" keyword below.

stacks:

  # vpc stack
  vpc:
    # Note: "cf" is a required keyword, which tells
    # qaz when to start reading in template values.
    cf:
      cidr: 10.10.0.0/16

  # subnet stack
  subnet:
    # Note: the "depends_on" keyword is used to list
    # stack dependencies. Any amount can be listed.
    # This key_word must be defined outside of "cf"
    depends_on:
      - vpc

    cf:
      subnets:
        - private: 10.10.0.0/24
        - public: 10.10.2.0/24

  # database stack
  database:
    # Note: Qaz supports passing parameters to stacks,
    # this is handy for sensitive items that should not
    # be shared within the template
    parameters:
      - dbpassword: password123

    depends_on:
      - vpc

    cf:

Note: Config files do not need to be named config.yml Qaz will look for this filename by default if no config is specified. When config is named differently, you can specify the config file using the -c --config flags.

Keywords:

When deploying stacks Qaz utilises special keywords for defining additional functionality.

parameters:

Stack parameters to pass when deploying can be listed under this keyword. Read more on AWS Cloudformation Stack Parameters See Here

Example:

stackname:
  parameters:
    - password: password123

depends_on:

Use this keyword to define a dependency chain by listing stack dependencies. With this keyword, you can explicitly say one stack relies on another or several others.

Example:

elb-stack:
  depends_on:
    - vpc-stack
    - securitygroup-stack

cf:

All Cloudformation values are defined under this keyword. There is no limitation on how values should be structured as long as they adhere to YAML syntax.

Example:

stacks:
  vpc-stack:
    cf:
      cidr: 10.10.0.0/16
      subnets:
        - 10.10.10.0/24

stacks:

All Cloudformation Stacks are defined under this value, keywords such as depends_on, parameters & cf will only work under stacks followed by the stackname.

Example:

stacks:
  vpc:
    depends_on:
    paramters:
    cf:

--

Templates (Getting those values!)

Go has an excellent and expandable templating library which is utilised in this project for additional logic in creating templates. To read more on Go Template see Here. All features of Go Template are supported in Qaz.

Note that templates must have the same file name (extension excluded) as the stack they reference in config when working with local files, however, this does not apply when dealing with remote templates on S3 or via Http.

Templating in Qaz

We'll run through some basics to get started.

Deploying Stacks

Stacks can be Deployed/Terminated with a single command.

The above however, only works when you are using Qaz in the root of your project directory. Alternatively, Qaz offers a few ways fetching both configuration and template files.

Configuration can be retrieved from both Http Get requests & S3.

$ qaz deploy -c s3://mybucket/super_config.yml -t vpc::http://someurl/vpc_dev.yml

Deploying via S3

The stack name must be specified using the syntax above. This tells Qaz what values to associate with this stack.

$ qaz deploy -c http://mybucket/super_config.yml -t vpc::s3://mybucket/vpc_dev.yml -t subnets::s3://mybucket/subnets.yml

You can pass as many -t flags as you have stacks, Qaz will deploy all in the correct order and manage the dependency chains as long as the depends_on keyword is utilised.

Note that the syntax for specifying stack names with URLs stackname::url. The deploy command does not require the stack name syntax when using local files, however the update command uses this syntax on all -t --template arguments. For example:

$ qaz deploy -c path/to/config -t path/to/template
$ qaz update -c path/to/config -t vpc::path/to/template

Deploy also takes wildcards for local templates. For example:

$ qaz deploy -c path/to/config.yml -t "path/*"

Quotes are required when using wildcards.

Built-In Template Functions

Template Functions expand the functionality of Go's Templating library by allowing you to execute external functions to retrieve additional information for building your template.

Qaz supports all the Go Template functions as well as some custom ones.

Qaz has two levels of custom template functions, these are Gen-Time functions and Deploy-Time functions.

--

Gen-Time Template Functions

Gen-Time functions are functions that are executed when a template is being generated. These are handy for reading files into a template or making API calls to fetch values.

Here are some of the Gen-Time functions available... more to come:

file

A template function for reading values from an external file into a template. For now the file needs to be in the files directory in the root of the project folder.

Example:

{{ "myfile.txt" | file }} or {{ file "myfile.txt" }}

Returns the value of myfile.txt under the files directory

s3_read

As the name suggests, this function reads the content of a given s3 key and writes it to the template.

Example:

{{ "s3://mybucket/key" | s3_read }} or {{ s3_read "s3://mybucket/key" }}

Writes the contents of the object to the template

GET

GET implements http GET requests on a given url, and writes the response to the template.

Example

{{ "http://localhost" | GET }} or {{ GET "http://localhost" }}

invoke

Invokes a Lambda function and stores the returned value with the template.

Example

{{ invoke "function_name" `{"some_json":"some_value"}` }}

Note: JSON passed to Gen-Time functions needs to be wrapped in back-ticks.

kms_encrypt

Generates an encrypted Cipher Text blob using AWS KMS

Example

{{ kms_encrypt kms.keyid "Text to Encrypt!" }}

kms_decrypt

Decrypts a given Cipher Text blob using AWS KMS

Example

{{ kms_decrypt "CipherTextBlob" }}

Note: The encryption functionality does require some understanding of AWS KMS. The kms_encrypt creates a Cipher Text Blob from the given text. The Cipher holds metadata that allows it to be decrypted without giving the Key ID. It can however, only be decrypted using an AWS profile with access to the Key ID used to encrypt.

See Here for more information on KMS CipherTextBlob and Encryption terminology.

Gen-Time functions in Action

--

Deploy-Time Template Functions

Deploy-Time functions are run just before the template is pushed to AWS Cloudformation. These are handy for:

• Fetching values from dependency stacks
• Making API calls to pull values from resources built by preceding stacks
• Triggering events via an API call and adjusting the template based on the response
• Updating Values in a decrypted template

Here are some of the Deploy-Time functions available... more to come:

stack_output

stack_output fetches the output value of a given stack and stores the value in your template. This function uses the stack name as defined in your project configuration

Example

# internal-stackname::output

<< stack_output "vpc::vpcid" >>

stack_output_ext

stack_output_ext fetches the output value of a given stack that exists outside of your project/configuration and stores the value in your template. This function requires the full name of the stack as it appears on the AWS Console.

Example

# external-stackname::output

<< stack_output_ext "external-vpc::vpcid" >>

Important!: When using Deploy-Time functions the Template delimiters are different: << >> Qaz identifies items wrapped in these as Deploy-Time functions and only executes them just for before deploying to AWS.

--

The following are also accessible as Deploy-Time functions:

• s3_read
• invoke
• GET
• kms_decrypt
• kms_encrypt

Deploy-Time Functions in action

--

Deploy/Gen-Time Function - Lambda Invoke

--

See examples folder for more examples of usage. More examples to come.

$ qaz
                   
  __ _   __ _  ____
 / _` | / _` ||_  /
| (_| || (_| | / / 
 \__, | \__,_|/___|
    |_|            

--> Shut up & deploy my templates...!

Usage:
  qaz [flags]
  qaz [command]

Available Commands:
  change      Change-Set management for AWS Stacks
  check       Validates Cloudformation Templates
  deploy      Deploys stack(s) to AWS
  exports     Prints stack exports
  generate    Generates template from configuration values
  help        Help about any command
  init        Creates a basic qaz project
  invoke      Invoke AWS Lambda Functions
  outputs     Prints stack outputs
  status      Prints status of deployed/un-deployed stacks
  tail        Tail Real-Time AWS Cloudformation events
  terminate   Terminates stacks
  update      Updates a given stack

Flags:
      --debug            Run in debug mode...
  -p, --profile string   configured aws profile (default "default")
      --version          print current/running version

Use "qaz [command] --help" for more information about a command.

--

Roadmap and status

Qaz is in early development.

TODO:

• More Comprehensive Documentation
• More Deploy/Gen-Time Functions
• Qaz can already create Azure Templates, Once I get my head around Azure as a Platform, i'll add support for Deploying to Azure as well..... Maybe

--

Pull requests welcomed....