heimdall

Heimdall

Heimdall is a lightweight, pluggable data orchestration and job execution platform that abstracts complex data infrastructure from clients while offering a secure and consistent API for submitting and managing jobs.

Originally inspired by Netflix Genie, Heimdall extends the architecture to support:

• 🔌 Pluggable commands
• ⚙️ Job queuing
• 📡 Synchronous and asynchronous execution

✨ Key Features

• 🔁 Sync & Async Job Execution
• 🧩 Plugin-Based Execution Framework: Shell, Glue, Snowflake, Spark, DynamoDB, and Ping
• 📬 REST API for programmatic access
• 🌍 Web UI for visual management
• 🔐 Secure orchestration without credential leakage
• 🧠 Dynamic routing based on command / cluster criteria
• 📦 Configurable or self-registering clusters (future)

🖥️ UI + API Access

Heimdall includes a web interface running alongside the API.

• API: http://localhost:9090/api/v1
• Web UI: http://localhost:9090/ui

🚀 Getting Started

1. Clone the Repository

git clone git@github.com:patterninc/heimdall.git
cd heimdall

2. Start Heimdall

Ensure you have Docker or a compatible alternative installed.

docker compose up --build -d

This starts:

• The Heimdall server on port 9090
• The database and other dependencies

3. Submit a Test Job (Example: Ping Plugin)

curl -X POST -H "X-Heimdall-User: test_user" -H "Content-Type: application/json" \
     -d '{
           "name": "ping-test",
           "version": "0.0.1",
           "context": {},
           "command_criteria": ["type:ping"],
           "cluster_criteria": ["type:localhost"]
         }' \
     http://127.0.0.1:9090/api/v1/job

4. Monitor the Job

Use the UI (http://127.0.0.1:9090/) or the following endpoints:

# Job status
GET /api/v1/job/<job_id>

# Job stdout
GET /api/v1/job/<job_id>/stdout

# Job stderr
GET /api/v1/job/<job_id>/stderr

🔌 Supported Plugins

Heimdall supports a growing set of pluggable command types:

🧬 Core Concepts

Command

Defines a reusable unit of work with associated tags and plugin logic.

Cluster

An execution environment abstracted from its physical form. It can represent localhost, EMR, Kubernetes, a DB, a piece of your infrastructure that has context and a name, etc.

Job

The orchestration request. It combines:

• Command criteria
• Cluster criteria
• Execution context

Heimdall dynamically selects the best command-cluster pair based on these criteria.

⚙️ Configuration

Initially, Commands and Clusters are configured via a static config file (see config.yml). Heimdall is evolving toward support for:

• Self-registering clusters
• Health-based routing
• API-based dynamic configuration

🔁 Command & Cluster Matching Logic

1. Commands: Must be active and match all tags in command_criteria.

2. Compatible Clusters: Found via the command’s own cluster_criteria.

3. Final Selection:

   • Filters clusters using the job’s cluster_criteria.
   • If multiple pairs match, one is selected randomly (a capability for custom "routing" is in works and will be represented as a plugin).
   • If no match, the job fails with a detailed error.

🔐 Security by Design

Heimdall removes the need for:

• Embedding credentials in user environments
• Direct user and services access to infrastructure

It centralizes execution logic, logging, and auditing—all accessible via API or UI.

📦 API Overview

👥 Credits

Heimdall was created at Pattern, Inc by Stan Babourine, with contributions from Will Graham, Gaurav Warale and Josh Diaz.