aws-broker

Cloud Foundry AWS Service Broker

Cloud Foundry Service Broker to manage instances of various AWS services.

Current Services Supported

• RDS
• AWS Elasticache for Redis
• AWS OpenSearch

Setup

Environment Variables

There are important environment variables that should be overriden inside the manifest.yml file

Note: All environment variables prefixed with DB_ refer to attributes for the database the broker itself will use for internal uses.

1. DB_URL: The hostname / IP address of the database.
2. DB_PORT: The port number to access the database.
3. DB_NAME: The database name.
4. DB_USER: Username to access the database.
5. DB_PASS: Password to access the database.
6. DB_TYPE: The type of database. Currently supported types: postgres and sqlite3.
7. DB_SSLMODE: The type of SSL Mode to use when connecting to the database. Supported modes: disabled, require and verify-ca.
8. AWS_ACCESS_KEY_ID: The id credential (treat like a password) with access to make requests to the Amazon RDS .
9. AWS_SECRET_ACCESS_KEY: The secret key (treat like a password) credential to access Amazon RDS.
10. AWS_DEFAULT_REGION: Region you wish to provision services in.
11. AUTH_USER: The username used by cf to authenticate to the broker
12. AUTH_PASS: The password used by cf to authenticate to the broker
13. ENC_KEY: This is an string that must be 16, 24, or 32 bytes long. It is an AES key that is used to encrypt the password.
14. CF_API_URL: URL for CloudFoundry API in this environment
15. CF_API_CLIENT_ID: UAA client ID that will be used for requests to the CloudFoundry API
16. CF_API_CLIENT_SECRET: UAA client secret that will be used for requests to the CloudFoundry API
17. ENVIRONMENT: the current environment name (e.g. "development")

Note the AWS Environment Variables should be generated by following the instructions here

Make sure the account has write access to RDS and EC2 (particularly for VPC and Subnet).

Example of permissions that suffice: AmazonRDSFullAccess and AmazonEC2FullAccess

You may need to adjust VPC routing, security groups, public accessibility (DNS names, etc) as well, depending on your needs.

Creating a UAA Client

In order to run the app, you will need to create a UAA client application.

To create the client, log in to the jumpbox for the target environment and run:

uaac client add aws_broker \
   --authorized_grant_types client_credentials \
   --authorities cloud_controller.global_auditor \
   -s <my_client_secret>

Optional Environment Variables

There are some feature flags that you can turn on as well:

1. ENABLE_FUNCTIONS: If this environment variable exists, it will enable users to create mysql databases like cf create-service _servicename_ production my-mysql-service -c '{"enable_functions": true}', which will set the log_bin_trust_function_creators=1 parameter for their db, enabling the creation of functions in their databases.
2. PUBLICLY_ACCESSIBLE: If this environment variable exists, it will enable users to create databases with PubliclyAccessible: true by doing something like cf create-service _servicename_ production my-mysql-service -c '{"publicly_accessible": true}'. This is probably not something you want to set unless you really know what you are doing.

Catalog.yml

Catalog.yml contains a list of service(s) offered with plans. It contains no secrets. Prior to pushing, complete the catalog.yml for your environment. It is architected where the service name (e.g. rds) is the mapping between it and the service details.

Secrets.yml

secrets.yml contains the all of the secrets for the different resources.

Testing and development

Make sure you have a valid secrets.yml and catalog.yml:

cp catalog-test.yml catalog.yml
cp secrets-test.yml secrets.yml

Once you have these in place, run go test ./... to run the tests.

Pre-commit hooks

This repo ships a .pre-commit-config.yaml consuming the internal cloud-gov/pre-commit-templates (hygiene, shellcheck/shfmt, check-gsa-email, gitleaks) plus the template's Go hooks (gofmt, go vet, and whole-repo golangci-lint).

• On a Cloud.gov dev host with caulking: do not run pre-commit install (caulking sets core.hooksPath globally and the framework refuses). Caulking invokes the config automatically on commit; run ad hoc with pre-commit run --all-files. Caulking also runs gitleaks globally, so use SKIP=gitleaks to avoid a double scan.
• Without caulking (CI / a fresh box): pre-commit install, or run pre-commit run --all-files. gitleaks self-installs; the template's shellcheck/shfmt-check and Go hooks need golangci-lint/shellcheck/shfmt on PATH (brew install golangci-lint shellcheck shfmt). Use golangci-lint v2 — .golangci.yml pins the config to v2, and the whole-repo gate is validated 0-issue under v2 (a v1 binary applies different defaults and will error on the v2 config).

Testing with PostgreSQL database

1. Copy .env-sample to .env

2. Add a value for POSTGRES_USER to the .env file

3. Add a value for POSTGRES_PASSWORD to the .env file

4. Start the PostgreSQL docker container:

   cd docker && docker compose up -d  && cd -
   

5. OPTIONAL: For .env file integration with the VSCode Go test runner, create .vscode/settings.json with:

   {
     "go.testEnvFile": "/path/to/aws-broker/.env"
   }
   

6. Run the tests using godotenv:

   go install github.com/joho/godotenv/cmd/godotenv@latest
   godotenv -f ./.env go test ./...
   

How to deploy it

1. cf push
2. cf create-service-broker BROKER_NAME AUTH_USER AUTH_PASS https://BROKER-URL
3. cf enable-service-access SERVICE_NAME

In this case BROKER_NAME would be aws and it would contain many service names (one for rds, one for s3). Then SERVICE_NAME would be rds for example.

How to use it

To use the service you need to create a service instance and bind it:

1. cf create-service SERVICE_NAME micro-psql MYDB
2. cf bind-service APP MYDB

When you do that you will have all the credentials in the VCAP_SERVICES environment variable with the JSON key rds.

Also, you will have a DATABASE_URL environment variable that will be the connection string to the DB.

Credential handling

This section is primarily for auditors who need to understand how the broker, and related components, handle credentials so that they aren't stored or transmitted in the clear. All calls between entities are made over HTTPS, unless otherwise specified.

Instantiation

The broker is deployed by Concourse CI onto CloudFoundry, using a manifest that is built by the Cloud.gov secrets management system to specify the environment variables. When the app is deployed, Concourse registers the broker, specifying the AUTH_USER and AUTH_PASS.

The CF Cloud Controller stores the configuration for the app, including these environment variables, in an encrypted database table on the CCDB, as described in Cloud Foundry security concepts. The aws-broker app does not write these to static storage since Cloud Foundry makes them available as environment variables.

Provision a new instance

When an authenticated, authorized CloudFoundry user runs cf create-service aws-rds _plan_name_ _service_name_, the CloudFoundry platform uses the OSBAPI (open-service broker API) to call the registered broker with a PUT request, /v2/service_instances/:instance_id where instance_id is a GUID. The request uses BASIC AUTH, e,g.:

curl -X PUT https://username:password@aws-broker..../v2/service_instances/:instance_id

The broker expects the AUTH_PASSand AUTH_USER as specified in the environment, which the Platform has provided (see above).

The response indicates if the provisioning request has been accepted.

Broker to AWS to provision an instance

The broker application calls the AWS API with the AWS Access Key and Secret Key, which were provided as environment variables at instantiation.

When the provisioning is complete, the broker takes the following actions:

Updating an Elasticsearch instance's plan

Elasticsearch/OpenSearch instances can be moved to a different plan in place with cf update-service SERVICE_NAME -p NEW_PLAN. The broker only allows plan changes that are an in-place upgrade; it validates the request before calling AWS and returns an HTTP 400 with an explanatory message when the change is not allowed.

The rules are:

• The data-node count may grow but never shrink. Highly-available (-ha) plans run 4 data nodes and their non-HA counterparts run 2, so a non-HA plan may move to an -ha plan. The reverse is rejected: removing data nodes would discard the shards they hold.
• Single-data-node plans stay single-data-node. A plan with one data node (es-dev) is provisioned on a single subnet with zone awareness off, so it may only move to another single-data-node plan. Moving it to any multi-node plan is rejected: adding data nodes there would enable zone awareness on a domain that still has only one subnet, which AWS rejects with You must specify exactly two subnets because you've set zone count to two.
• Same size or larger only. The target plan must be the same size or larger than the current plan. Size is determined by the plan's instanceSizeRank in the catalog plus its data-node count. Downgrading to a smaller plan is rejected, and a plan with no instanceSizeRank cannot be compared at all, so every plan change into or out of it is refused. Because the data-node count feeds the rank, moving from an -ha plan to a larger-tier non-HA plan is rejected by the data-node rule rather than this one.
• One change at a time. A plan change cannot be combined with an engine version upgrade in the same update-service call; make them as separate calls.

Each plan carries its own instanceSizeRank in catalog-template.yml, so adding or re-tiering a plan is a catalog change and needs no code change. Plans are ranked by plan tier, not by any single hardware dimension, so the r7g Graviton search-* types share a rank with the c5/m5 types used by the equivalently named es-* plans (c5.large and r7g.medium are both the "medium" tier, and so on). r7g trades vCPUs for substantially more memory, so neither family is strictly larger than the other; giving them equal ranks makes switching families at the same tier a permitted lateral move in both directions, while moves to a larger or smaller tier are still ordered correctly. Ranks are spaced by 10 so a new tier can be slotted between two existing ones.

A same-tier lateral move can change capacity substantially. r7g carries 8 GiB per vCPU against c5's 2, so the tier-named r7g data node has half the cores and double the RAM and JVM heap of the c5 plan at the same rank — es-medium -> search-medium moves a data node from c5.large (2 vCPU / 4 GiB) to r7g.medium (1 vCPU / 8 GiB). The rank ordering permits this in both directions because tier, not core count, defines the rank. Whether it is the right move depends on whether the workload is heap-bound or CPU-bound; see docs/es-plan-graviton-cost-comparison.md.

The resulting order for the 2-data-node plans, smallest to largest (the 4-data-node -ha plans follow the same order among themselves):

es-dev                                (rank 10)
es-medium      / search-medium        (rank 20)
es-large       / search-large         (rank 30)
es-xlarge      / search-xlarge        (rank 40)
es-2xlarge-gp  / search-2xlarge-gp    (rank 50)
es-4xlarge-gp                         (rank 60)
es-12xlarge-gp                        (rank 70)

Examples:

Note that a lateral family switch still triggers an AWS blue/green deployment: the instance type genuinely changes, so it is not a no-op. Moving from a non-HA plan to its -ha counterpart adds two data nodes to the existing domain and likewise triggers a blue/green deployment.

To reduce the data-node count (-ha back to non-HA), to move from a single-node plan to a multi-node plan, or to move to a smaller plan, create a new instance on the desired plan and migrate data rather than updating in place.

When a plan upgrade is accepted, the broker applies the new plan's instance type, data-node count, dedicated-master configuration, and (if larger) volume size, and issues an asynchronous AWS UpdateDomainConfig to resize the domain.

• For RDS and Redis, it creates a username/password in the AWS service, and stores the credentials in the broker database
• For AWS Elasticsearch, it creates an IAM user with privileges to the new instance, then stores the credentials in the broker database

Storing credentials in the broker database

The broker uses a dedicated AWS RDS PostgreSQL database. The RDS instance data are encrypted at rest using AWS storage encryption. The communication between the broker and the database is over postgres StartTLS with TLS 1.2 enabled.

The broker is instantiated with encryption key, ENC_KEY, and all credentials are written to the database encrypted with that key and a random salt, as in the setPassword function of each _service_instance.go file, e.g.: https://github.com/cloud-gov/aws-broker/blob/20f70bb/services/redis/redisinstance.go#L50

Providing credentials to CloudFoundry applications

The CloudFoundry applications have access to the credentials only if the user binds an app to a service instance, as specified at https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md#binding of the OSBAPI standard. The credentials are fetched from the service broker and are stored in the environment of the application container, and not written the static storage. If the application instance is re-instantiated, the platform fetches the credentials for the application container from the broker.

Public domain

This project is in the worldwide public domain. As stated in CONTRIBUTING:

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.