caldera-security-tests

CALDERA Security Regression Pipeline (SRP)

This project was created to provide a proof of concept example of a Security Regression Pipeline for vulnerabilities that were discovered in MITRE CALDERA by Jayson Grace from Meta's Purple Team.

The attacks are run against a fresh test environment with the latest MITRE CALDERA on a weekly basis using Github Actions. Because patches have been created for all of the discovered vulnerabilities, the attacks are expected to fail.

If any of the vulnerabilities are successful during one of these runs, an issue is automatically created noting the regression.

Ideally this should be run as part of a CI/CD pipeline gating commits, but it can also work as a standalone entity for Purple Team engagements, pentests, etc.

Table of Contents

• Setup
  • Apple Silicon users
  • Test Environment Preparation
• Running the SRP
  • Using Github Actions
  • Locally
• Hacking on the Project
  • Dependencies
  • Developer Environment Setup

Setup

Apple Silicon users

Run this command:

export DOCKER_DEFAULT_PLATFORM=linux/amd64

Test Environment Preparation

1. Download the release binary from github and drop it in a directory:

   mkdir bin && cd $_
   # Put downloaded binary here
   

2. Clone the caldera repo:

   cd ../ && git clone https://github.com/mitre/caldera.git
   

Running the MITRE CALDERA SRP

Using Github Actions

You can incorporate the CALDERA SRP into your CALDERA fork by creating .github/workflows/srp.yaml and populating it with the following contents:

name: CALDERA Security Regression Pipeline
on:
  pull_request:
  push:
    branches: [master]

  # Run once a week (see https://crontab.guru)
  schedule:
    - cron: "0 0 * * 0"

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

jobs:
  tests:
    uses: fbsamples/caldera-security-tests/.github/workflows/srp.yaml@main

You can use the outcomes of these workflow runs to gate updates for your CALDERA deployments if a security regression in the latest CALDERA release is detected.

Locally

Create vulnerable test environment, run the first XSS, and tear the test environment down:

./bin/cst-darwin TestEnv -v
export OS="$(uname | python3 -c "print(open(0).read().lower().strip())")"
./bin/"cst-${OS}" StoredXSSUno
./bin/"cst-${OS}" TestEnv -d

Create vulnerable test environment, run the second XSS, and tear the test environment down:

./bin/cst-darwin TestEnv -v
./bin/"cst-$(uname)" StoredXSSDos
./bin/"cst-$(uname)" TestEnv -d

Create test environment using the most recent commit to the default CALDERA branch, try running all attacks, and tear the test environment down:

./bin/cst-darwin TestEnv -r
./bin/"cst-$(uname)" StoredXSSUno
./bin/"cst-$(uname)" StoredXSSDos
./bin/"cst-$(uname)" TestEnv -d

Parameters for the tests can be modified in the generated config/config.yaml file. This file is created as soon as the TestEnv command in the above example is run.

Hacking on the Project

Dependencies

• Install homebrew:

  # Linux
  sudo apt-get update
  sudo apt-get install -y build-essential procps curl file git
  /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
  eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"
  
  # macOS
  /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
  

• Install ruby:

  brew install ruby
  

• Install gvm:

  bash < <(curl -s -S -L https://raw.githubusercontent.com/moovweb/gvm/master/binscripts/gvm-installer)
  source "${GVM_BIN}"
  

• Install golang:

  gvm install go1.18
  

• Install pre-commit:

  brew install pre-commit
  

• Install Mage:

  go install github.com/magefile/mage@latest
  

Developer Environment Setup

1. Fork this project

2. Clone your forked repo and caldera:

   git clone https://github.com/fbsamples/caldera-security-tests.git
   git clone https://github.com/mitre/caldera.git
   

3. (Optional) If you installed gvm, create golang pkgset specifically for this project:

   VERSION='1.18'
   PROJECT=caldera-security-tests
   
   gvm install "go${VERSION}"
   gvm use "go${VERSION}"
   gvm pkgset create "${PROJECT}"
   gvm pkgset use "${PROJECT}"
   

4. Install dependencies:

   mage installDeps
   

5. Install pre-commit hooks:

   mage installPreCommitHooks
   

6. Update and run pre-commit hooks locally:

   mage runPreCommit
   

7. Compile binary:

   export OS="$(uname | python3 -c "print(open(0).read().lower().strip())")"
   mage compile "${OS}"