data

CM-Beetle Data Migration Test CLI

An automated end-to-end test CLI for CM-Beetle's data migration feature. It creates source and target Object Storages via CM-Beetle, uploads dummy data, migrates it across CSP-Region pairs, verifies the result, and generates a per-pair Markdown report.

Prerequisites

• CM-Beetle running at the endpoint configured in test-config.yaml (default: http://localhost:8056)
• CB-Tumblebug running and accessible (default: http://localhost:1323/tumblebug)
• CB-Spider running behind CB-Tumblebug
• Valid CSP credentials registered in CB-Spider for every CSP-Region pair under test.cases
• Go 1.21+ (only needed when running directly with go run)

Directory structure

cmd/test-cli/data/
├── main.go                           # Test CLI entry point
├── dummydata/                        # Auto-generated dummy files (created at runtime)
├── log/                              # Log files from each test run
├── testconf/
│   ├── template-test-config.yaml     # Template — copy to test-config.yaml and edit
│   ├── test-config.yaml              # Active configuration (git-ignored)
│   ├── template-auth-config.json     # Template — copy to auth-config.json and edit
│   ├── auth-config.json              # CM-Beetle basic auth credentials (git-ignored)
│   └── object-storage-creation-req.json  # Bucket specification for OS creation
└── testresult/                       # Per-pair Markdown reports (created at runtime)

Configuration

1. testconf/test-config.yaml

Copy the template and fill in your values:

cp testconf/template-test-config.yaml testconf/test-config.yaml

Key fields:

2. testconf/auth-config.json

Copy the template and fill in your CM-Beetle credentials:

cp testconf/template-auth-config.json testconf/auth-config.json

{
  "basicAuthUsername": "your-api-username",
  "basicAuthPassword": "your-api-password"
}

3. testconf/object-storage-creation-req.json

Defines the bucket specification used when creating both source and target Object Storages. Edit bucketName to match the bucket name registered in CB-Spider for your CSP accounts:

{
  "targetObjectStorages": [
    {
      "sourceBucketName": "test-source",
      "bucketName": "cm-beetle-test-bucket",
      "versioningEnabled": false,
      "corsEnabled": false
    }
  ]
}

How to run

Run from the cmd/test-cli/data/ directory:

cd cmd/test-cli/data

# Basic run
go run main.go -config testconf/test-config.yaml

# Capture all output to a log file as well
go run main.go -config testconf/test-config.yaml 2>&1 | tee log/test-run-$(date +%Y%m%d-%H%M%S).log

You can also build first:

go build -o test-cli-data main.go
./test-cli-data -config testconf/test-config.yaml

Test phases

The CLI runs in three phases:

Pre-flight (once, before all target tests)

1. Check CM-Beetle readiness (GET /beetle/readyz)
2. Generate dummy data files in testData.baseDir (skipped if files already exist)
3. Create source Object Storage via CM-Beetle (POST /beetle/migration/middleware/ns/{nsId}/objectStorage)
   • A 409 Conflict response is treated as success (idempotent reuse of an existing OS)
4. Upload dummy data from the local filesystem to the source OS using transx.Transfer() directly
   • The upload applies the file filter from test.upload.filter
   • Encryption is not applied at this stage (local → source OS)

Per target CSP-Region pair (Steps 1–4)

Each enabled target case in test.cases executes the following steps:

Encryption: Step 2 fetches a one-time RSA public key from CM-Beetle (GET /beetle/migration/data/encryptionKey) and encrypts sensitive fields (Tumblebug auth passwords) before sending the migration request.

Async polling: Step 2 returns 202 Accepted. The CLI polls GET /beetle/request/{reqId} at migration.poll.intervalSec intervals until the status is Success or Error, or until migration.poll.timeoutSec is reached.

Parallel mode: In parallel mode, all enabled cases run concurrently. Each case gets a unique name suffix (e.g. data0101, data0102) to avoid naming collisions.

Post-flight (once, after all target tests)

Deletes the shared source Object Storage. This runs unconditionally via defer, so it executes even if tests fail or the process receives SIGINT/SIGTERM.

Delete phases (applied to both source and target OS deletions):

1. DELETE ?option=empty — empties the bucket (once)
2. DELETE (standard) — up to 3 retries with 10-second delays
3. DELETE ?option=force — fallback if standard deletion fails, up to 3 retries

Output

Logs

Structured JSON logs (zerolog) are printed to stderr. Pipe to a file with tee to persist them:

go run main.go -config testconf/test-config.yaml 2>&1 | tee log/test-run-$(date +%Y%m%d-%H%M%S).log

Markdown reports

A per-pair report is saved to testresult/ after each target case completes:

testresult/data-test-results-{sourceCsp}-to-{targetCsp}.md

Each report contains:

• Environment details (CM-Beetle version, CSP, region, OS IDs, name seed)
• Test scenario description
• Step-by-step results table with status, duration, and request/response details
• Collapsible blocks for long request/response bodies
• Object tree comparison (source vs. target) from the verification step
• Sensitive values (Azure subscription IDs, GCP project IDs, email addresses) are automatically masked

Exit code

Signal handling

Sending SIGINT (Ctrl+C) or SIGTERM to the process cancels remaining test cases and prints a warning. Post-flight cleanup (source OS deletion) still runs before the process exits.

Dummy data structure

The CLI auto-generates the following file set in testData.baseDir:

If testData.baseDir already contains files, generation is skipped.