substrate

Substrate

A Caddy module that adds a custom transport method for reverse_proxy, enabling dynamic process execution based on file requests.

Overview

Substrate behaves like FastCGI but over HTTP - it executes requested files as separate processes and proxies HTTP traffic to them. Each file gets its own process with automatic lifecycle management.

Installation

Build Caddy with the Substrate module:

xcaddy build --with github.com/fserb/substrate

Quick Start

1. Create a Caddyfile:

root /path/to/your/files

@js_files {
    path *.js
    file {path}
}

reverse_proxy @js_files {
    transport substrate {
        idle_timeout 5m
        startup_timeout 30s
    }
}

1. Create an executable script (e.g., hello.js):

#!/usr/bin/env -S deno run --allow-net
const [host, port] = Deno.args;

Deno.serve({ 
  hostname: host, 
  port: parseInt(port) 
}, (req) => {
  return new Response('Hello from Substrate!');
});

1. Make it executable and start Caddy:

chmod +x hello.js
caddy run

1. Request triggers process execution:

curl http://localhost/hello.js
# → "Hello from Substrate!"

How It Works

1. File Matching: Caddy's file matcher identifies executable files
2. Process Creation: Substrate executes the file with host and port arguments
3. Port Management: Each file gets a unique port automatically assigned
4. Request Proxying: HTTP requests are proxied to the running process
5. Lifecycle Management: Processes are reused, restarted, and cleaned up automatically

Process Contract

Your executable receives two arguments:

• argv[1]: Host to bind to (usually "localhost")
• argv[2]: Port to listen on (unique per file)

Example in various languages:

Deno/Node.js:

#!/usr/bin/env -S deno run --allow-net
const [host, port] = Deno.args;
// Start HTTP server on host:port

Python:

#!/usr/bin/env python3
import sys
host, port = sys.argv[1], int(sys.argv[2])
# Start HTTP server on host:port

Go:

//go:build ignore

package main
import ("fmt"; "net/http"; "os")

func main() {
    host, port := os.Args[1], os.Args[2]
    http.ListenAndServe(host+":"+port, handler)
}

Configuration

Transport Options

reverse_proxy @matcher {
    transport substrate {
        idle_timeout 5m      # How long to keep unused processes
        startup_timeout 30s  # How long to wait for process startup
    }
}

Multiple File Types

@scripts {
    path *.js *.py *.go
    file {path}
}

reverse_proxy @scripts {
    transport substrate
}

Features

• Zero Configuration: Processes just need to listen on the provided port
• Automatic Port Management: No port conflicts or manual assignment
• Process Reuse: Same file requests share the same process
• Hot Reloading: File changes restart the associated process
• Concurrent Safe: Multiple requests handled properly
• Resource Cleanup: Idle processes automatically terminated
• Security: Executable validation and privilege dropping when running as root
• Advanced Routing: URL rewriting, subpath matching, and pattern-based routing

Development

./task build    # Build the module
./task test     # Run all tests (unit + integration + e2e)
./task run      # Run example configuration

Advanced Usage

URL Rewriting

Route clean URLs to executable scripts:

@simple_rewrite {
    not path *.js
    file {path}.js
}

reverse_proxy @simple_rewrite {
    transport substrate
}

Subpath Routing

Extract subpaths and forward as headers:

@subpath_match {
    path_regexp m ^(.*)(/[^/]+)$
}

handle @subpath_match {
    @file_exists file {re.m.1}.lemon.js
    handle @file_exists {
        reverse_proxy {
            header_up X-Subpath {re.m.2}
            transport substrate
        }
    }
}

Examples

Check the e2e tests in e2e/ directory for comprehensive usage patterns and working examples.