outpost-migrate-redis

Outpost Migration Tool

A CLI tool for managing database schema migrations for Outpost, with support for versioned migrations and state tracking.

Purpose

This tool manages database schema changes in a controlled manner. It tracks state using Redis keys:

• outpost:migration:<name> - Hash storing migration state (status, applied_at)
• outpost:migration:<name>:run:<timestamp> - Hash storing run history (processed, skipped, failed, rerun, duration_ms)
• .outpost:migration:lock - Prevents concurrent migrations (auto-expires after 1 hour)

Note: Currently designed for manual migrations with downtime. Not yet suitable for zero-downtime migrations, but provides a foundation for future enhancements.

Installation

Using the Outpost CLI wrapper

# Build all binaries
make build

# Run via the wrapper
./bin/outpost migrate [command]

Direct execution

# Build just this tool
go build -o bin/outpost-migrate-redis ./cmd/outpost-migrate-redis

# Run directly
./bin/outpost-migrate-redis [command]

Development

# Run without building (via wrapper)
go run ./cmd/outpost migrate [command]

# Run directly
go run ./cmd/outpost-migrate-redis [command]

Usage

# Initialize database for fresh installations (runs on startup)
outpost migrate init
outpost migrate init --current  # Exit 1 if migrations pending (for CI/CD)

# List available migrations
outpost migrate list

# Plan next migration (shows current status and what will change)
outpost migrate plan

# Apply all pending migrations
outpost migrate apply --all
outpost migrate apply --all --yes  # Skip confirmation prompt

# Apply the next pending migration (one at a time)
outpost migrate apply
outpost migrate apply --yes  # Skip confirmation prompt

# Apply a specific migration by name
outpost migrate apply 002_timestamps

# Re-run a migration (catches records created between runs)
outpost migrate apply 002_timestamps --rerun

# Verify the migration was successful
outpost migrate verify

# Cleanup old keys after verification
outpost migrate cleanup
outpost migrate cleanup --yes  # Skip confirmation
outpost migrate cleanup --force  # Skip verification check

# Force clear the migration lock (use with caution)
outpost migrate unlock
outpost migrate unlock --yes  # Skip confirmation prompt

Migration Workflow

Single Migration (Manual)

1. Plan - Check status and show what will be migrated
2. Apply - Execute the migration (creates new keys, preserves old ones)
3. Verify - Spot-check migrated data for correctness
4. Cleanup - Delete old keys after confirming success

Batch Migration (--all)

The --all flag applies all pending migrations in sequence with automatic verification:

$ outpost migrate apply --all
Found 3 pending migration(s)
  - 001_hash_tags: Migrate from legacy format...
  - 002_timestamps: Convert timestamp fields...
  - 003_entity: Add entity field...

Apply 3 migration(s)? [y/N] y

[1/3] Applying 001_hash_tags...
  ✓ Applied (150 records)
  Verifying... ✓ Valid

[2/3] Applying 002_timestamps...
  ✓ Applied (150 records)
  Verifying... ✓ Valid

[3/3] Applying 003_entity...
  ✓ Applied (150 records)
  Verifying... ✓ Valid

All 3 migration(s) applied successfully

How it works:

For each pending migration (in version order):
  1. Plan()   → Analyze what needs to change
  2. Lock     → Acquire distributed lock
  3. Apply()  → Execute the migration
  4. Mark     → Set status = "applied"
  5. Unlock   → Release lock
  6. Verify() → Confirm migration succeeded
     └─ If failed → STOP immediately
     └─ If passed → Continue to next

Key behaviors:

• Single confirmation prompt for all migrations
• Runs in version order (001 → 002 → 003)
• Verifies each migration before proceeding to the next
• Stops immediately on first failure (fail-fast)
• Safe to re-run: skips already-applied migrations

Using in Startup Scripts

The init --current command is designed for use in automated startup scripts. It handles both fresh installations and existing deployments:

# Initialize database and check for pending migrations
outpost migrate init --current || {
    echo "Error: Database migrations required"
    echo "Run: outpost migrate apply --all"
    exit 1
}
outpost serve

Init Command Behavior

The init command intelligently handles different scenarios:

• Fresh Installation: Automatically marks all migrations as applied without running them (since the schema is already current)
• Existing Installation: Checks if migrations are pending
• With --current flag: Exits with code 1 if migrations are pending, making it perfect for CI/CD pipelines
• Multi-node deployments: Uses atomic locking to ensure only one node performs initialization

This eliminates the need to manually handle fresh installations differently from existing ones.

Available Migrations

001_hash_tags

Migrates Redis keys from legacy format to hash-tagged format for Redis Cluster compatibility.

Purpose: Ensures all keys for a tenant are routed to the same Redis Cluster node by using hash tags.

Key Transformations:

• tenant:123 → tenant:{123}:tenant
• tenant:123:destinations → tenant:{123}:destinations
• tenant:123:destination:abc → tenant:{123}:destination:abc

Deployment Mode Note: If you are using DEPLOYMENT_ID configuration, this migration is not needed. Deployment-scoped keys already include hash tags:

• dp_001:tenant:{123}:tenant (already has hash tags)
• dp_001:tenant:{123}:destinations (already has hash tags)

See 001_hash_tags/README.md for details.

Safety: This migration preserves original keys. Use the cleanup command after verification to remove old keys.

002_timestamps

Converts timestamp fields from RFC3339 strings to Unix timestamps for timezone-agnostic sorting.

Purpose: Enables correct sorting by created_at in RediSearch indexes, regardless of server timezone.

Fields Affected:

• Tenant: created_at, updated_at
• Destination: created_at, updated_at, disabled_at

Auto-Runnable: Yes - this migration runs automatically at startup. It's safe because:

• In-place conversion (no key renaming)
• Idempotent (skips already-converted records)
• Lazy migration fallback (parseTimestamp() reads both formats)

Re-running: Use --rerun to catch records created between migration runs:

outpost migrate apply 002_timestamps --rerun

Adding New Migrations

1. Create a new directory: migration/002_your_migration_name/
2. Implement the Migration interface:

   type Migration interface {
       Name() string          // e.g., "002_your_migration"
       Version() int          // Target schema version (2)
       Description() string
       Plan(ctx, client, verbose) (*Plan, error)
       Apply(ctx, client, plan, verbose) (*State, error)
       Verify(ctx, client, state, verbose) (*VerificationResult, error)
       Cleanup(ctx, client, state, force, verbose) error
   }
   

3. Register it in main.go:

   registry.Register(migration_002_your_migration.New())
   

Lock Mechanism

The tool uses a simple Redis lock (outpost:migration:lock) to prevent concurrent migrations. The lock auto-expires after 1 hour as a safety measure. Use unlock command only when certain no migration is running.

Configuration

The tool uses Outpost's standard configuration system, loading settings from (in order of precedence):

1. CLI flags (highest priority)
2. Environment variables
3. Config files (.outpost.yaml, .env)
4. Default values

Redis Configuration Options

Other Options

Examples

# Using environment variables
REDIS_HOST=localhost outpost migrate plan

# Using CLI flags
outpost migrate --redis-host localhost --verbose plan

# Using config file
outpost migrate --config /path/to/config.yaml plan

# Production cluster with TLS
outpost migrate \
  --redis-host redis-cluster.example.com \
  --redis-cluster \
  --redis-tls \
  --verbose \
  plan