heimdall

Heimdall

Heimdall is a lightweight, pluggable data orchestration and job execution platform that abstracts complex data infrastructure from clients while offering a secure and consistent API for submitting and managing jobs.

Originally inspired by Netflix Genie, Heimdall extends the architecture to support:

• 🔌 Pluggable commands
• ⚙️ Job queuing
• 📡 Synchronous and asynchronous execution

---

✨ Key Features

• 🔁 Sync & Async Job Execution
• 🧩 Plugin-Based Execution Framework: Shell, Glue, Snowflake, Spark, Trino, DynamoDB, and Ping
• 📬 REST API for programmatic access
• 🌍 Web UI for visual management
• 🔐 Secure orchestration without credential leakage
• 🧠 Dynamic routing based on command / cluster criteria
• 📦 Configurable or self-registering clusters
• 🩺 Cluster health checks with per-plugin probes and configurable timeout

---

🖥️ UI + API Access

Heimdall includes a web interface running alongside the API.

• API: http://localhost:9090/api/v1
• Web UI: http://localhost:9090/ui

---

🚀 Getting Started

1. Clone the Repository

git clone git@github.com:patterninc/heimdall.git
cd heimdall

2. Start Heimdall

Ensure you have Docker or a compatible alternative installed.

docker compose up --build -d

This starts:

• The Heimdall server on port 9090
• The database and other dependencies

3. Submit a Test Job (Example: Ping Plugin)

curl -X POST -H "X-Heimdall-User: test_user" -H "Content-Type: application/json" \
     -d '{
           "name": "ping-test",
           "version": "0.0.1",
           "context": {},
           "command_criteria": ["type:ping"],
           "cluster_criteria": ["type:localhost"]
         }' \
     http://127.0.0.1:9090/api/v1/job

4. Monitor the Job

Use the UI (http://127.0.0.1:9090/) or the following endpoints:

# Job status
GET /api/v1/job/<job_id>

# Job stdout
GET /api/v1/job/<job_id>/stdout

# Job stderr
GET /api/v1/job/<job_id>/stderr

---

🛠️ Building from Source

build.sh compiles the Go binaries + plugins and builds the web UI. It builds only what you ask for via flags (no defaults):

./build.sh --go          # build Go binaries (cmd/*) + plugins (plugins/*.so)
./build.sh --ui          # build the Next.js web UI
./build.sh --test        # run the Go test suites
./build.sh --go --ui --test   # do everything (what CI runs)

Passing no flag builds nothing. Unknown flags fail fast. Outputs land in dist/ (binaries in dist/, plugins in dist/plugins/).

---

🔌 Supported Plugins

Heimdall supports a growing set of pluggable command types:

Plugin	Description	Execution Mode
ping	Basic plugin used for testing	Sync or Async
shell	Shell command execution	Sync or Async
glue	Pulling Iceberg table metadata	Sync or Async
dynamo	DynamoDB read operation	Sync or Async
snowflake	Query execution in Snowflake	Async
spark	SparkSQL query execution on EMR on EKS	Async
sparkeks	SparkSQL query execution on EKS	Async
trino	Query execution in Trino	Async
clickhouse	Query execution in Clickhouse	Sync
ecs fargate	Task Deployment in ECS Fargate	Async
postgres	PostgreSQL query execution	Sync or Async

---

🧬 Core Concepts

Command

Defines a reusable unit of work with associated tags and plugin logic.

Cluster

An execution environment abstracted from its physical form. It can represent localhost, EMR, Kubernetes, a DB, a piece of your infrastructure that has context and a name, etc.

Job

The orchestration request. It combines:

• Command criteria
• Cluster criteria
• Execution context

Heimdall dynamically selects the best command-cluster pair based on these criteria.

---

⚙️ Configuration

Initially, Commands and Clusters are configured via a static config file (see config.yml). Heimdall is evolving toward support for:

• Self-registering clusters
• Health-based routing
• API-based dynamic configuration

---

🔁 Command & Cluster Matching Logic

1. Commands: Must be active and match all tags in command_criteria.

2. Compatible Clusters: Found via the command’s own cluster_criteria.

3. Final Selection:

   • Filters clusters using the job’s cluster_criteria.
   • If multiple pairs match, one is selected randomly (a capability for custom "routing" is in works and will be represented as a plugin).
   • If no match, the job fails with a detailed error.

---

🔐 Security by Design

Heimdall removes the need for:

• Embedding credentials in user environments
• Direct user and services access to infrastructure

It centralizes execution logic, logging, and auditing—all accessible via API or UI.

Commands may also restrict invocation via allowed_callers — a list of anchored regex patterns matched against X-Heimdall-User. Omitted/empty means open; non-matching callers are rejected at submit.

---

📦 API Overview

Endpoint	Description
POST /api/v1/job	Submit a job
GET /api/v1/job/<id>	Get job details
GET /api/v1/job/<id>/status	Check job status
POST /api/v1/job/<id>/cancel	Cancel an async job
GET /api/v1/job/<id>/stdout	Get stdout for a completed job
GET /api/v1/job/<id>/stderr	Get stderr for a completed job
GET /api/v1/job/<id>/result	Get job's result
GET /api/v1/jobs	Get list of jobs
PUT /api/v1/command/<id>	Upsert command details
GET /api/v1/command/<id>	Get command details
GET /api/v1/command/<id>/status	Check command status
PUT /api/v1/command/<id>/status	Set command status
GET /api/v1/commands	List configured commands
PUT /api/v1/cluster/<id>	Upsert cluster details
GET /api/v1/cluster/<id>	Get cluster details
GET /api/v1/cluster/<id>/status	Check cluster status
PUT /api/v1/cluster/<id>/status	Set cluster status
GET /api/v1/clusters	List configured clusters
GET /api/v1/clusters/health	Health check all opted-in clusters
GET /api/v1/cluster/<id>/health	Health check a single cluster

---

🩺 Cluster Health Checks

Opt any cluster into health probing by setting health_check: true in its config. Each plugin performs a lightweight connectivity check against its backend (see the Health Check column in the plugins table above).

Configure the global probe timeout (default 30s):

health_check:
  timeout_seconds: 10

GET /api/v1/clusters/health returns 200 if all probes pass, 503 if any fail. status per check: ok, error, or unchecked. Failing or misconfigured health checks have no effect on job execution or startup.

---

Config-Driven Job Attributes (For Static or Runtime generated metadata)

Surface extra links or text on a job in the UI with minimal change in plugin code - by declaring attributes on a cluster. Each attribute is a template that Heimdall renders after the job runs and stores in the job's job_attributes, which the UI then displays. Check local.yaml for the example.

Each entry is label → { kind, value }:

• kind tells the UI how to render value: link (clickable hyperlink) or text.
• value is a Go text/template rendered over four namespaces:
  • .Job — the job object (e.g. .Job.ID)
  • .Command / .Cluster — the matched command's and cluster's context maps
  • .Outputs — runtime values published by the plugin during execution

Static attributes (built only from .Job, .Command, .Cluster) need no plugin changes. For runtime-discovered values, a plugin publishes to the outputs channel with a single call - for example,

job.SetOutput("some_runtime_metadata", value)

---

👥 Credits

Heimdall was created at Pattern, Inc by Stan Babourine, with contributions from Will Graham, Gaurav Warale and Josh Diaz.