api

Gthulhu API Server

Gthulhu API Server is a Golang-based API server designed to integrate the Linux kernel scheduler (sched_ext) with the Kubernetes orchestration system. The core objective of this project is to collect scheduling metrics from eBPF programs and provide intelligent scheduling decisions to the kernel scheduler based on Kubernetes Pod labels and user-defined strategies.

DEMO

Click the image below to watch our DEMO on YouTube!

System Architecture

This API Server adopts a dual-mode architecture, consisting of two independent services:

┌─────────────────────────────────────────────────────────────────────────────────┐
│                              Gthulhu Architecture                               │
├─────────────────────────────────────────────────────────────────────────────────┤
│                                                                                 │
│   ┌─────────────┐         ┌─────────────────────┐         ┌─────────────────┐  │
│   │    User     │ ──────▶ │      Manager        │ ──────▶ │    MongoDB      │  │
│   │  (Web UI)   │         │ (Central Management)│         │  (Persistence)  │  │
│   └─────────────┘         └──────────┬──────────┘         └─────────────────┘  │
│                                      │                                          │
│                                      │ Query Pods via K8s API                   │
│                                      ▼                                          │
│                           ┌─────────────────────┐                               │
│                           │   Kubernetes API    │                               │
│                           │   (Pod Informer)    │                               │
│                           └─────────────────────┘                               │
│                                      │                                          │
│              ┌───────────────────────┼───────────────────────┐                  │
│              │                       │                       │                  │
│              ▼                       ▼                       ▼                  │
│   ┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐          │
│   │ Decision Maker  │     │ Decision Maker  │     │ Decision Maker  │          │
│   │   (Node 1)      │     │   (Node 2)      │     │   (Node N)      │          │
│   └────────┬────────┘     └────────┬────────┘     └────────┬────────┘          │
│            │                       │                       │                    │
│            ▼                       ▼                       ▼                    │
│   ┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐          │
│   │  sched_ext      │     │  sched_ext      │     │  sched_ext      │          │
│   │ (eBPF Scheduler)│     │ (eBPF Scheduler)│     │ (eBPF Scheduler)│          │
│   └─────────────────┘     └─────────────────┘     └─────────────────┘          │
│                                                                                 │
└─────────────────────────────────────────────────────────────────────────────────┘

Manager Mode

Purpose: Serves as the central management service, responsible for handling user requests, managing scheduling strategies, user authentication, and access control.

Key Features:

• User authentication and authorization (JWT Token)
• Role and permission management (RBAC)
• CRUD operations for scheduling strategies
• Monitor Pod status via Kubernetes Informer
• Distribute scheduling intents to Decision Makers on each node
• Data persistence to MongoDB

Default Port: :8081

Decision Maker Mode

Purpose: Deployed on each Kubernetes node (as DaemonSet), responsible for receiving scheduling intents and interacting with the local sched_ext scheduler.

Key Features:

• Receive scheduling intents from Manager
• Scan /proc filesystem to discover Pod processes
• Convert scheduling strategies into concrete PID-based scheduling decisions
• Collect eBPF scheduler metrics
• Expose metrics via Prometheus endpoint

Default Port: :8080

Core Features

Manager Service Features

• User Management: Create, query users, password reset
• Role & Permission Management: RBAC role management, permission assignment
• Scheduling Strategy Management: Create Pod label-based scheduling strategies
• Scheduling Intent Tracking: Track strategy execution status
• Kubernetes Integration: Real-time Pod monitoring via Pod Informer
• JWT Authentication: RSA asymmetric encryption Token authentication

Decision Maker Service Features

• Intent Processing: Receive and process scheduling intents from Manager
• Process Discovery: Parse cgroup information to map PIDs to Pods
• Scheduling Strategy Provider: Provide concrete PID scheduling strategies to sched_ext
• Metrics Collection: Collect and expose eBPF scheduler metrics to Prometheus
• Token Authentication: Validate requests from Manager

API Endpoints

Manager Endpoints

System Endpoints

Authentication Endpoints

User Management Endpoints

Role Management Endpoints

Scheduling Strategy Endpoints

Decision Maker Endpoints

Data Structures

ScheduleStrategy

ScheduleIntent

MetricSet

Quick Start

0. Test Environment Setup

For local development, you need to set up a MongoDB instance. You can use Docker to start the infrastructure:

# Start MongoDB using Docker
$ docker run --rm --network host mongo:6.0 mongosh "mongodb://test:test@localhost:27017/?authSource=admin&authMechanism=SCRAM-SHA-256" --eval 'db.runCommand({ ping: 1 })'

After starting MongoDB, create the required user and verify the connection:

# Create a test user with root privileges in MongoDB
$ docker exec mongodb mongosh --eval 'db.getSiblingDB("admin").createUser({user:"test",pwd:"test",roles:[{role:"root",db:"admin"}]})'

# Verify the connection with the created user credentials
$ docker exec mongodb mongosh "mongodb://test:test@localhost:27017/?authSource=admin&authMechanism=SCRAM-SHA-256" --eval 'db.runCommand({ ping: 1 })'

1. Install Dependencies

go mod tidy

2. Configuration

Manager Configuration (config/manager_config.toml)

[server]
host = ":8081"

[logging]
level = "info"

[mongodb]
host = "localhost"
port = "27017"
user = "test"
password = "test"
database = "manager"

[k8s]
kube_config_path = "/path/to/.kube/config"
in_cluster = false

[key]
rsa_private_key_pem = "..."
dm_public_key_pem = "..."
client_id = "your-client-id"

[account]
admin_email = "admin@example.com"
admin_password = "your-password"

Decision Maker Configuration (config/dm_config.toml)

[server]
host = ":8080"

[logging]
level = "info"

[token]
rsa_private_key_pem = "..."
token_duration_hr = 24

3. Start Services

Start Manager

go run main.go manager

# With custom configuration
go run main.go manager -c manager_config -d /path/to/config

Start Decision Maker

go run main.go decisionmaker

# With custom configuration
go run main.go decisionmaker -c dm_config -d /path/to/config

4. Test API

Health Check

# Manager
curl http://localhost:8081/health

# Decision Maker
curl http://localhost:8080/health

User Login

curl -X POST http://localhost:8081/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{
    "username": "admin@example.com",
    "password": "your-password-here"
  }'

Create Scheduling Strategy

curl -X POST http://localhost:8081/api/v1/strategies \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <your-token>" \
  -d '{
    "strategyNamespace": "default",
    "labelSelectors": [
      {"key": "app", "value": "nginx"}
    ],
    "k8sNamespace": ["default"],
    "priority": 10,
    "executionTime": 20000000
  }'

Kubernetes Deployment

Deployment Architecture

• Manager: Deployed as a Deployment, typically single replica
• Decision Maker: Deployed as a DaemonSet on every node

Deployment Manifest Locations

deployment/
├── kind/                    # Kind local test environment
│   ├── local_setup.sh
│   ├── decisonmaker/
│   │   ├── daemonset.yaml
│   │   └── service.yaml
│   ├── manager/
│   │   ├── deployment.yaml
│   │   └── service.yaml
│   └── mongo/
│       ├── secret.yaml
│       ├── service.yaml
│       └── statefulset.yaml
└── local/
    └── docker-compose.infra.yaml  # Docker Compose for local development

Quick Deployment

# Local testing with Kind
cd deployment/kind
./local_setup.sh

# Or manual deployment
kubectl apply -f deployment/kind/mongo/
kubectl apply -f deployment/kind/manager/
kubectl apply -f deployment/kind/decisonmaker/

RBAC Permission Requirements

Manager requires the following Kubernetes RBAC permissions:

• pods: list, watch, get
• namespaces: list, get

Development Guide

Project Structure

.
├── main.go                 # Entry point, uses Cobra for subcommands
├── config/                 # Configuration definition and parsing
├── manager/               # Manager service
│   ├── app/              # Application initialization
│   ├── cmd/              # Cobra commands
│   ├── domain/           # Domain models and interfaces
│   ├── k8s_adapter/      # Kubernetes client
│   ├── client/           # Decision Maker client
│   ├── repository/       # MongoDB data access
│   ├── rest/             # HTTP handlers
│   ├── service/          # Business logic
│   └── migration/        # Database migrations
├── decisionmaker/        # Decision Maker service
│   ├── app/             # Application initialization
│   ├── cmd/             # Cobra commands
│   ├── domain/          # Domain models
│   ├── rest/            # HTTP handlers
│   └── service/         # Business logic and process discovery
├── pkg/                  # Shared packages
│   ├── logger/          # Logging utilities
│   ├── middleware/      # HTTP middleware
│   └── util/            # Common utility functions
└── deployment/          # Deployment configurations

Tech Stack

• Web Framework: Echo v4
• Dependency Injection: Uber fx
• CLI: Cobra
• Logging: zerolog
• Database: MongoDB (mongo-driver v2)
• Kubernetes Client: client-go
• Metrics Collection: Prometheus client_golang
• Configuration Management: Viper

Build and Test

# Build
make build

# Run tests
make test

# Build Docker image
make image

Container Images

This project uses GitHub Actions to automatically build and publish to GitHub Container Registry.

# Use latest version
docker pull ghcr.io/gthulhu/api:main

# Use specific version
docker pull ghcr.io/gthulhu/api:v1.0.0

License

This project is open source.