citadel

Citadel

A declarative CDK deployment framework for AWS ECS/Fargate.

Overview

Citadel makes container deployments as simple as the Serverless Framework experience — but targeting ECS/Fargate instead of Lambda. You declare your app once in a citadel.yml. The tool handles SSM secret syncing, CDK infrastructure deployment, Docker build/push, and ECS service rollout in a single command.

citadel deploy --env dev --deploy-infra

The Problem

Deploying ECS applications involves managing configuration across multiple files:

• Secret lists in deployment scripts
• Duplicate secret definitions in CDK code
• Infrastructure config scattered across multiple files

The Solution

Single source of truth: citadel.yml defines everything about your deployment.

Both the CLI tool and CDK constructs read from the same file. Write a secret name once, use it everywhere.

Installation

# Go
go install github.com/ClusterBox/citadel/cmd/citadel@latest

# From source
git clone https://github.com/ClusterBox/citadel.git
cd citadel
make install

Usage

Full deployment pipeline

citadel deploy --env dev --deploy-infra

Start a project

citadel init --ecs       # or --lambda; prompts for anything not passed as a flag
citadel init --lambda --yes --account 123456789012   # non-interactive

init writes a commented citadel.yml and a .citadel/ directory next to it. If citadel.yml already exists it is left alone and only .citadel/ is created (pass --force to regenerate the config). citadel.yaml is accepted when citadel.yml is absent.

The .citadel/ directory

citadel deploy creates .citadel/ automatically if it is missing. Dry runs record nothing.

citadel always excludes .citadel/ from the Docker build context it sends to the daemon, the same way it excludes .git/. Tools that hash or copy the repo themselves instead of going through citadel deploy — e.g. a CDK DockerImageFunction/fromImageAsset pointing at the repo root — should add .citadel/ to their own .dockerignore.

Deploy from GitHub Actions

on:
  push:
    branches: [main, development]

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: ${{ github.ref_name == 'main' && 'prod' || 'dev' }}
    permissions:
      contents: read
    concurrency: { group: citadel-${{ github.ref_name }}, cancel-in-progress: false }
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
      - uses: ClusterBox/citadel/action@v0.3.0
        with:
          deploy-infra: true
          env-file: ${{ secrets.CITADEL_ENV_FILE }}

main deploys prod and development deploys dev (override with environment or branch-map). See action/README.md.

Pipeline

With no pipeline: key, citadel deploy runs its built-in sequence (ssm-sync, build, cdk, deploy, config-sync (lambda only), wait) exactly as before. To customize the order, mix in shell commands, run a one-off ECS task, or add a health check, declare the deploy as an explicit list of steps. pipeline: needs citadel v0.4.0 or later (GitHub Action: ClusterBox/citadel/action@v0.4.0 or later); older versions ignore the key and run the default pipeline.

pipeline:
  - uses: citadel/ssm-sync
  - name: test
    run: npm test
  - uses: citadel/build
  - name: migrate
    task: npx prisma migrate deploy
    timeout: 15m
  - uses: citadel/cdk
  - uses: citadel/deploy
  - uses: citadel/wait
  - name: smoke
    http: https://api-${env}.example.com/health
    rollback_on_failure: true

Each step gets its own log under .citadel/runs/<id>/ and its own line in run.json. Validation runs when the config loads (citadel deploy, citadel status, citadel init, and the GitHub Action all get it), so a bad pipeline is rejected before any AWS call. Every error names the step's position and name: pipeline[<i>] "<name>": <problem>. An unknown key in a step (e.g. a misspelled environment: or rollback-on-failure:) is an error too.

Step kinds. Exactly one of uses, run, task, http must be set per step.

Variables, expanded in run, env values, working_directory, task, container and http:

Only a lowercase ${word} is a citadel variable — an unknown one is a config load error. Anything else, such as ${HOME} or $PATH, passes through untouched for the shell (or the receiving server, for http) to resolve.

run: steps also get the variables as environment variables: CITADEL_ENV, CITADEL_NAME, CITADEL_IMAGE, CITADEL_SHA, CITADEL_REGION and CITADEL_ACCOUNT. They win over a key of the same name in the step's env:.

Ordering rules, enforced at load time:

• citadel/build must come before citadel/cdk, citadel/deploy, and every task: step.
• citadel/cdk must come before citadel/deploy (when present — it's optional).
• citadel/deploy, citadel/config-sync and citadel/wait may each appear at most once.
• citadel/config-sync and citadel/wait must come after citadel/deploy.
• When citadel/build has envs:, citadel/cdk, citadel/deploy and every task: step need the image it pushes, so each must list envs: that are a subset of build's (an omitted envs: means every environment and is rejected).
• A pipeline without citadel/build or citadel/deploy is valid, e.g. one that only runs migrations.

Built-ins keep their CLI flag gates: citadel/ssm-sync syncs from the env file given by --env-file (default .env) and skips with --skip-ssm or --env-file ""; citadel/cdk runs only with --deploy-infra; citadel/config-sync skips with --skip-config; citadel/wait runs only with --wait. citadel/config-sync is valid only for the lambda runtime; task: is valid only for the ecs runtime.

Because citadel/wait only runs with --wait, an http: check placed after it runs as soon as the rollout starts when --wait is not given, and may probe the old tasks that are still serving traffic.

Failure handling:

• A step with continue_on_error: true that fails prints a warning and stays failed in run.json, but the run keeps going.
• An http step with rollback_on_failure: true that fails instead runs an automatic rollback, recorded as its own rollback step: ECS rolls the service back to the task-definition revision it ran before this deploy; Lambda rolls the function back to its previous image. The snapshot of "what ran before" is taken right before the first citadel/cdk or citadel/deploy step that actually runs — but only when some step in the pipeline has rollback_on_failure: true, so a pipeline without one makes no extra AWS call. If the snapshot could not be taken, the deploy still proceeds; a rollback attempted after that has nothing to restore and fails with "cannot roll back: no snapshot of the previous deployment". A deploy cancelled with Ctrl-C/SIGTERM never rolls back: it prints ↩️ Not rolling back: deploy was cancelled and stops. If the revision to restore was deregistered in the meantime (e.g. by a CDK deploy), ECS rollback registers an identical copy and uses that.
• Otherwise a step's failure stops the pipeline; the run, the state file and the deploydb row all record it.

task: is ECS-only. It runs a one-off task in the freshly built image, on the service's own network (subnets, security groups, launch type), overriding the command on the target container. It needs ecs:RunTask, ecs:DescribeTasks, ecs:StopTask, logs:GetLogEvents, and iam:PassRole on the task and execution roles. rollback_on_failure needs ecs:DescribeServices/ecs:DescribeTaskDefinition/ecs:UpdateService (plus ecs:RegisterTaskDefinition when the old revision was deregistered) (ECS) or lambda:UpdateFunctionCode/lambda:GetFunction (Lambda) — see action/README.md for the full list.

citadel deploy --dry-run prints the plan without changing anything (it still reads from AWS): each built-in prints its existing dry-run line, run: prints [dry-run] Would run: <cmd>, task: prints [dry-run] Would run task: <cmd>, and http: prints [dry-run] Would check: <url>. Nothing is recorded in .citadel/runs/.

citadel-logs daemon

Citadel ships a separate always-on binary, citadel-logs, that watches CloudWatch log groups for every registered service and surfaces 500-class errors at http://localhost:5500/logs.

It works across runtimes:

• runtime: ecs (default) NestJS backends
• runtime: lambda — clusterbox Go Lambdas like smaug. Requires a lambda: { functionName: ... } block.

Run it as a service (Linux)

# 1. Install both binaries (citadel + citadel-logs)
make install

# 2. Register a repo
cd ~/Documents/github/my-backend
citadel logs-daemon register --env dev

# 3. Start the daemon as a systemd user service
citadel logs-daemon start

# 4. Open the dashboard
open http://localhost:5500/logs

citadel logs-daemon start installs a systemd --user unit (~/.config/systemd/user/citadel-logs.service), enables it, and turns on user lingering so it survives reboots and starts before you log in. It captures AWS_PROFILE / AWS_REGION from your shell (override with --profile / --region). The daemon stores its SQLite db under ~/.local/share/citadel/citadel-logs.db and serves the dashboard on 127.0.0.1:5500.

Other commands:

citadel logs-daemon status            # is it running?
citadel logs-daemon logs              # tail the journal (-n N, --no-follow)
citadel logs-daemon restart           # re-install unit + restart
citadel logs-daemon stop [--disable]  # stop (and optionally disable autostart)

On non-Linux platforms, use the Docker path instead: docker compose -f docker-compose.logs.yml up -d.

Roadmap

• Project architecture
• CLI scaffolding
• Config parser
• SSM secret sync
• Docker build/push
• ECS deployment
• CDK construct library

Configuration

queues: — SQS access (optional)

Grants the ECS task role least-privilege access to existing SQS queues. Queues are split by intent:

queues:
  consume:
    - arn:aws:sqs:us-east-1:123456789012:incoming
  produce:
    - arn:aws:sqs:us-east-1:123456789012:outgoing

• consume queues are granted sqs:ReceiveMessage, sqs:DeleteMessage, sqs:GetQueueAttributes, and sqs:ChangeMessageVisibility.
• produce queues are granted sqs:SendMessage and sqs:GetQueueAttributes.

A queue ARN may appear in both lists if the service both reads and writes it.

An ARN may contain {env} in its queue name (for example …:settle-{env}), which becomes the environment being deployed, so dev is never granted prod's queue. {env} elsewhere in the ARN, and any other {...}, is rejected. Quote such values in a YAML flow list.

Citadel does not create the queues — they must already exist.

iam: — runtime SSM reads (optional)

Lets the application read SSM parameters at runtime, beyond the boot secrets under /<name>-<env>/ (which the execution role already injects):

iam:
  ssm_read:
    - /smaug/{env}/psp

Each prefix grants the task role ssm:GetParameter and ssm:GetParameters on every parameter under it (arn:aws:ssm:*:*:parameter/smaug/<env>/psp/*). {env} becomes the environment being deployed; any other {...} is rejected. Prefixes must start with /, may not be / alone, and may not contain wildcards. In a YAML flow list, quote values that contain {env} (ssm_read: ["/smaug/{env}/psp"]). Parameters encrypted with the default aws/ssm key need no KMS grant.

cache: — ElastiCache Serverless Valkey (optional)

cache:
  engine: valkey
  serverless: true

Creates an ElastiCache Serverless cache named <name>-<env> in the same subnets as the tasks (public in dev, private in other environments), behind a security group that admits only the service's tasks on 6379 (primary endpoint) and 6380 (reader endpoint). The endpoint (host:port) is injected as the plain env var VALKEY_ENDPOINT, so do not also list it under secrets: or env:. Serverless caches require TLS. Only engine: valkey with serverless: true is supported, on the ecs runtime. The stack also outputs CacheEndpoint.

The cache is ephemeral: it and its data are deleted with the stack or when the cache: block is removed, with no final snapshot. Because its name is fixed, changes that make CloudFormation replace it (for example raising vpc.max_azs, which changes its subnets) fail the deploy and roll back; to apply one, remove the cache: block, deploy, then add it back.

Releasing

Push a tag vX.Y.Z on main. .github/workflows/release.yml runs the tests and publishes citadel_<version>_<os>_<arch>.tar.gz archives plus checksums.txt. The GitHub Action at the same tag (ClusterBox/citadel/action@vX.Y.Z) installs exactly that release. make release-snapshot builds the archives locally.

License

MIT