bt

= bt: Build Terraform

A no commitments Terraform/Tofu wrapper that provides build caching functionality.
It also makes working with workspaces a breeze.

== Features

* Automatically inject init backend tfvars and plan tfvars based on config values.

* Automatically load tfvar files based on workspace name.

* Allow you to have multiple profiles in the config file so that you can have a different configs for multiple workspaces.
For example, one profile for dev and another for prod, or one profile for North America and another for China.
+
Each profile uses a separate `TF_DATA_DIR`.
This allows to work with multiple profiles pointing to different backends under the same directory without conflicts.

* Runs pre-apply checks for you.
If you define for example, `conftest` checks, it will run them for you before trying to apply.

* Track the archs you want to use when running terraform provider lock.

* Allow you to work on different workspaces in different terminals.
It automatically uses the `TF_WORKSPACE` environment variable rather than selecting the workspace.

* Allows you to define stacks of components and their dependencies and orchestrate their deployment (and destruction in the reverse order).

== Install

* Install using homebrew:
+
----
brew tap DavidGamba/dgtools https://github.com/DavidGamba/dgtools
brew install --HEAD DavidGamba/dgtools/bt
----
+
[NOTE]
====
Completion is auto setup for bash.

For `zsh` completions, an additional step is required, add the following to your `.zshrc`:

[source, zsh]
----
export ZSHELL="true"
source "$(brew --prefix)/share/zsh/site-functions/dgtools.bt.zsh"
----
====
+
Upgrade with:
+
----
brew update
brew reinstall bt
----

* Install using go:
+
Install the binary into your `~/go/bin`:
+
----
go install github.com/DavidGamba/dgtools/bt@latest
----
+
Then setup the completion.
+
For bash:
+
----
complete -o default -C bt bt
----
+
For zsh:
+
[source, zsh]
----
export ZSHELL="true"
autoload -U +X compinit && compinit
autoload -U +X bashcompinit && bashcompinit
complete -o default -C bt bt
----

== Config file

The config file must be saved in a file named `.bt.cue`.
It will be searched from the current dir upwards.

Example:

.Config file .bt.cue
[source, cue]
----
config: {
	default_terraform_profile: "default"
	terraform_profile_env_var: "BT_TERRAFORM_PROFILE"
}
terraform_profile: {
	default: {
		binary_name: "terraform"
		init: {
			backend_config: ["backend.tfvars"]
		}
		plan: {
			var_file: ["~/auth.tfvars"]
		}
		workspaces: {
			enabled: true
			dir: "envs"
		}
		pre_apply_checks: {
			enabled: true
			commands: [
				{name: "conftest", command: ["conftest", "test", "$TERRAFORM_JSON_PLAN"]},
			]
		}
		platforms: ["darwin_amd64", "darwin_arm64", "linux_amd64", "linux_arm64"]
	}
}
----

== Usage Basics

. Run `bt terraform init` to initialize your config.

. Run `bt terraform build` to generate a plan.

. Run `bt terraform build --ic` to generate a plan again even when it detects there are no file changes.

. Run `bt terraform build --show` to view the generated plan.

. Run `bt terraform build --apply` to apply the generated plan.

=== Caching Internals

After running `bt terraform init` it will save a `.tf.init` file.

After running `bt terraform build` it will save a `.tf.plan` or `.tf.plan-<workspace>` file.
It will check the time stamp of the `.tf.init` file and if it is newer than the `.tf.plan` file, a new plan needs to be generated.
It will also compare the `.tf.plan` file against any file changes in the current dir or any of the module dirs to determine if a new plan needs to be generated.

If `pre_apply_checks` are enabled, it will run the checks specified by passing the rendered json plan to the command.
For example, `conftest` policy checks.

After running `terraform apply` it will save a `.tf.apply` or `.tf.apply-<workspace>` file.
It will use that file and compare it to the `.tf.plan` time stamp to determine if the apply has already been made.

=== Backend Config / Var File helpers

Given the config setting for `backend_config` for init and `var_file` for plan, it will automatically include those files to the command.

For example, running `bt terraform init` with the example config file will be the same as running:

----
terraform init -backend-config backend.tfvars
----

In the same way, running `bt terraform build` with the example config file will be the same as running:

----
terraform plan -out .tf.plan -var-file ~/auth.tfvars
----

Finally, running `bt terraform build --apply` with the example config file will be the same as running:

----
terraform apply -input .tf.plan
----

== Workspaces helpers

Setting workspaces to `enabled: true` in the config file will enable the workspace helpers.
What the helpers do is to assume any `.tfvars` or `.tfvars.json` file in the `dir` folder is a workspace.

If a workspace has been selected, bt will automatically include the `<dir>/<workspace>.tfvars` or `<dir>/<workspace>.tfvars.json` file to the command.

If a workspace hasn't been selected, passing the `--ws` option will select the workspace by exporting the `TF_WORKSPACE` environment variable and will add the corresponging `<dir>/<workspace>.tfvars` or `<dir>/<workspace>.tfvars.json` file to the command.

For example, running `bt terraform build --ws=dev` with the example config file will be the same as running:

----
export TF_WORKSPACE=dev
terraform plan -out .tf.plan -var-file ~/auth.tfvars -var-file envs/dev.tfvars
----

And then running `bt terraform build --ws=dev --apply`:

----
export TF_WORKSPACE=dev
terraform apply -input .tf.plan
----

IMPORTANT: Because `bt` uses the `TF_WORKSPACE` environment variable rather than selecting the workspace,
it is possible to work with multiple workspaces at the same time on different terminals.

When using `bt terraform workspace-select default` bt will automatically delete the `.terraform/environment` file to ensure we can use the `TF_WORKSPACE` environment variable safely.

== Pre Apply Checks

When using `bt terraform build`, pre apply checks get run automatically after a plan if they are enabled.

Pre apply check commands get the following Env vars exported:

* `CONFIG_ROOT`: The dir of the config file.
* `TERRAFORM_JSON_PLAN`: The path to the rendered json plan.

If pre-apply checks are enabled in the config file, they can be disabled for the current run using the `--no-checks` option.

To run only the checks, use `bt terraform checks`, combine it with the `--ws` option to run the checks against the last generated plan for the given workspace.

== Profiles

Multiple terraform config profiles can be defined.
By default, the `default` profile is used.
The default profile can be overridden with `config.default_terraform_profile` in the config file.

To use a different profile, use the `--profile` option or export the `BT_TERRAFORM_PROFILE` environment variable.
The environment variable name itself can also be overridden to read an existing one in the environment.
For example, set `config.terraform_profile_env_var` to `AWS_PROFILE` and name your terraform profiles the same way you name your AWS profiles.

Each additional profile will have its own `TF_DATA_DIR` and the terraform data will be saved under `.terraform-<profile>/`.
The `config.default_terraform_profile` will still use the default `.terraform/` dir.
This allows to work with multiple profiles pointing to different backends under the same workspace directory without conflicts.

=== Providers lock using Platforms list

Use `bt terraform providers lock` to generate a lock file using all the os archs in the `platforms` list for a given profile.

== Stacks: A different take

Hashicorp recently https://www.hashicorp.com/blog/terraform-stacks-explained[introduced their solution] for deploying stacks of resources.

A stack is a collection of components that need to be deployed together to form a logical unit.

Instead of having a massive state file that contains all resources, you can split them into multiple smaller components.
This split provides numerous benefits that I won't get into here, however,
these components require an orchestration layer to deploy them together and in the correct order.

bt provides a separate config file for defining stacks: `bt-stacks.cue`

=== Features

* The stack is composed of multiple different components.

* Each component can be deployed to a different workspace but in general,
they should have a consistent naming convention so that the workspace name can be auto-resolved from the stack name.

* A stack can have multiple instances of the same component, that is, multiple workspaces of one component.
For example, setting connectivity between AWS North America and AWS China requires different Terraform backend configurations because of the AWS China split.
The stack config file allows for this.

* The stack definition allows for conditionally added components.
Some regions or environments might not require certain components.

* The stack config file defines 2 different constructs.
One is the component definition where the component and its dependencies are defined.
The other is the stack definition, where the workspaces that compose a given stack and its variables are defined.

* Because component dependencies are tracked, stack builds run in parallel when possible.

* Components can have variables defined in the stack config file.

=== Stack config file

.bt-stacks.cue
[source, cue]
----
// Define the list of components
component: "networking": {}
component: "kubernetes": {
	depends_on: ["networking"]
}
component: "node_groups": {
	depends_on: ["kubernetes"]
}
component: "addons": {
	depends_on: ["kubernetes"]
}
component: "dns": {
	depends_on: ["kubernetes"]
}
component: "dev-rbac": {
	path: "dev-rbac/terraform"
	depends_on: ["kubernetes", "addons"]
}

// Create component groupings with additional variable definitions
_standard_cluster: {
	"networking": component["networking"] & {
		variables: [
			{name: "subnet_size", value: "/28"},
		]
	}
	"kubernetes": component["kubernetes"]
	"node_groups": component["node_groups"]
	"addons": component["addons"]
	"dns": component["dns"] & {
		variables: [
			{name: "api_endpoint", value: "api.example.com"},
		]
	}
}

// Create a stack with a list of components
stack: "dev-us-west-2": {
	id: string
	components: [
		for k, v in _standard_cluster {
			[// switch
				if k == "networking" {
					v & {
						workspaces: [
							"\(id)-k8s",
						]
					}
				},
				if k == "node_groups" {
					v & {
						workspaces: [
							"\(id)a",
							"\(id)b",
							"\(id)c",
						]
					}
				},
				v & {
					workspaces: [id]
				},
			][0]
		},
		// Custom component that only applies to this stack
		component["dev-rbac"] & {
			workspaces: [id]
		}
	]
}

stack: "prod-us-west-2": {
	id: string
	components: [
		for k, v in _standard_cluster {
			[// switch
				if k == "networking" {
					v & {
						workspaces: [
							"\(id)-k8s",
						]
					}
				},
				if k == "node_groups" {
					v & {
						workspaces: [
							"\(id)a",
							"\(id)b",
							"\(id)c",
						]
					}
				},
				v & {
					workspaces: [id]
				},
			][0]
		}
	]
}
----

=== Usage

==== Config

Quickly inspect the config file:

----
bt stack config
----

==== Graph

----
bt stack graph --id=dev-us-west-2 -T png
----

image::https://github.com/DavidGamba/screenshots/blob/master/dgtools/bt/stack-dev-us-west-2.png[]

----
bt stack graph --id=prod-us-west-2 -T png
----

image::https://github.com/DavidGamba/screenshots/blob/master/dgtools/bt/stack-prod-us-west-2.png[]

==== Build

Run all plans in parallel:

----
bt stack build --id=dev-us-west-2
----

Run all plans in serial:

----
bt stack build --id=dev-us-west-2 --serial
----

Review/Show the plan output for all components:

----
bt stack build --id=dev-us-west-2 --show --serial
----

Apply the changes:

----
bt stack build --id=dev-us-west-2 --apply
----

Destroy (pass both `--destroy` and `--reverse` to destroy in reverse order):

----
bt stack build --id=dev-us-west-2 --reverse --destroy
----

Apply the destroy:

----
bt stack build --id=dev-us-west-2 --reverse --destroy --apply
----