nflow-runtime

nFlow Runtime

Workflow execution engine for nFlow. This project executes workflows created in the nFlow visual designer, providing a secure environment with resource limits and sandboxing.

🚀 Installation

go get github.com/arturoeanton/nflow-runtime

📋 Requirements

• Go 1.19 or higher
• PostgreSQL or SQLite3
• Redis (optional, for sessions)
• Configuration in config.toml

🎯 Features

• Secure Execution: JavaScript sandboxing with configurable resource limits
• High Performance: 3,396 RPS with compute-intensive JavaScript workflows (1M+ requests, 0% errors)
• Thread-Safe: Race condition-free architecture using Repository Pattern
• Extensible: Plugin system for custom functionality
• Detailed Logging: Structured logging system with verbose mode (-v)
• Complete Monitoring: Prometheus metrics and health checks
• Advanced Debugging: Debug endpoints with authentication
• Optimized: VM pool, multi-level caching and highly optimized code
• Rate Limiting: IP-based rate limiting with configurable backends
• Security Analysis: Static analysis of JavaScript before execution
• Automatic Encryption: Detection and encryption of sensitive data
• Log Sanitization: Automatic prevention of sensitive data exposure in logs

🔧 Configuration

config.toml

[database_nflow]
driver = "postgres"
dsn = "user=postgres dbname=nflow sslmode=disable"

[redis]
host = "localhost:6379"
password = ""

[vm_pool]
# VM pool for high performance
max_size = 200             # Maximum VMs in pool (increased for 4x performance)
preload_size = 100         # VMs preloaded at startup

# Resource limits (security)
max_memory_mb = 128        # Maximum memory per VM
max_execution_seconds = 30 # Maximum execution time
max_operations = 10000000  # Maximum JS operations

# Sandbox settings
enable_filesystem = false  # Filesystem access
enable_network = false     # Network access
enable_process = false     # Process access

[tracker]
enabled = false            # Execution tracking (performance impact)
verbose_logging = false    # Detailed tracker logs

[monitor]
enabled = true             # Monitoring endpoints
health_check_path = "/health"
metrics_path = "/metrics"

[debug]
enabled = false            # Debug endpoints (development only)
auth_token = ""           # Authentication token
allowed_ips = ""          # Allowed IPs (e.g., "192.168.1.0/24")

[mail]
enabled = false
smtp_host = "smtp.gmail.com"
smtp_port = 587

[rate_limit]
enabled = false            # IP-based rate limiting
ip_rate_limit = 100       # Requests per IP per window
ip_window_minutes = 1     # Time window in minutes

[security]
# Static JavaScript analysis
enable_static_analysis = false    # Detect dangerous patterns before execution
block_on_high_severity = true     # Block scripts with severe issues

# Sensitive data encryption
enable_encryption = false         # Auto-encrypt sensitive data
encryption_key = ""              # 32-byte key for AES-256
encrypt_sensitive_data = true    # Detect and encrypt emails, SSN, API keys, etc.

# Log sanitization
enable_log_sanitization = false  # Mask sensitive data in logs
log_masking_char = "*"          # Character for masking
log_show_type = true            # Show masked data type

🏃‍♂️ Basic Usage

As Standalone Server

# Normal mode
./nflow-runtime

# Verbose mode (detailed logging)
./nflow-runtime -v

Server will be available at http://localhost:8080

As Library

import (
    "github.com/arturoeanton/nflow-runtime/engine"
    "github.com/arturoeanton/nflow-runtime/process"
)

func main() {
    // Initialize configuration
    configRepo := engine.GetConfigRepository()
    config := engine.ConfigWorkspace{
        // ... configuration
    }
    configRepo.SetConfig(config)
    
    // Initialize database
    db, err := engine.GetDB()
    if err != nil {
        log.Fatal(err)
    }
    engine.InitializePlaybookRepository(db)
    
    // Initialize process manager
    process.InitializeRepository()
    
    // Create Echo server
    e := echo.New()
    e.Any("/*", run)
    e.Start(":8080")
}

🛡️ Security

Resource Limits

Each VM has configurable limits to prevent DoS attacks:

• Memory: 128MB by default
• Time: 30 seconds maximum
• Operations: 10M JavaScript operations

Sandboxing

JavaScript executes in a restricted environment:

• ❌ eval() blocked
• ❌ Function constructor blocked
• ❌ Filesystem access disabled by default
• ❌ Network access disabled by default
• ✅ Only whitelisted modules available

Static Analysis

Before execution, each script is analyzed to detect:

• Use of eval() or new Function()
• Filesystem access (require('fs'))
• Process spawning (child_process)
• Potentially infinite loops
• Global scope modification

Data Protection

• Automatic Encryption: Automatically detects and encrypts:

  • Emails, phone numbers, SSN
  • API keys, JWT tokens
  • Credit card numbers

• Log Sanitization: Prevents accidental exposure:

  • Automatically masks sensitive data in all logs
  • Customizable patterns for business-specific data
  • No performance impact (3.6μs per log)

🔌 Available Plugins

• goja: Main JavaScript engine
• mail: Email sending
• template: Template processing
• ianflow: AI integration (OpenAI, Gemini, Ollama)
• http: HTTP client for API calls
• db: Database operations
• babel: ES6+ code transpilation

📊 Architecture

nflow-runtime/
├── engine/             # Main execution engine
│   ├── engine.go       # Workflow execution logic
│   ├── vm_manager.go   # VM pool for high performance
│   ├── vm_limits.go    # Resource limit management
│   ├── vm_sandbox.go   # Sandbox implementation
│   ├── js_context_wrapper.go # Echo context wrapper for JS
│   └── config_repository.go # Repository pattern for config
├── process/            # Process management
│   └── process_repository.go # Thread-safe repository
├── endpoints/          # API endpoints
│   ├── debug_endpoints.go    # Debug endpoints
│   └── monitor_endpoints.go  # Health & metrics
├── logger/             # Logging system
│   └── logger.go       # Structured logger with levels
├── security/           # Security module
│   ├── analyzer/       # Static JavaScript analysis
│   ├── encryption/     # AES-256 encryption service
│   ├── interceptor/    # Sensitive data interceptor
│   ├── sanitizer/      # Log sanitizer
│   └── security_middleware.go # Unified middleware
├── syncsession/        # Optimized session management
├── plugins/            # System plugins
└── main.go            # Server entry point

🧩 Custom Steps

You can create your own node types:

type MyCustomStep struct{}

func (s *MyCustomStep) Run(
    cc *model.Controller, 
    actor *model.Node, 
    c echo.Context,
    vm *goja.Runtime, 
    connection_next string, 
    vars model.Vars,
    currentProcess *process.Process, 
    payload goja.Value,
) (string, goja.Value, error) {
    // Your implementation here
    return nextNode, payload, nil
}

// Register the step
engine.RegisterStep("my-custom-step", &MyCustomStep{})

📈 Metrics and Monitoring

Monitoring Endpoints

• Health Check: GET /health - System health status
• Prometheus Metrics: GET /metrics - All metrics in Prometheus format

Available Metrics

• nflow_requests_total: Total HTTP requests
• nflow_workflows_total: Total workflows executed
• nflow_processes_active: Active processes
• nflow_db_connections_*: Database connection metrics
• nflow_go_memory_*: Memory usage
• nflow_cache_hits/misses: Cache statistics

Debug Endpoints (when enabled)

• /debug/info: System information
• /debug/config: Current configuration
• /debug/processes: Active process list
• /debug/cache/stats: Cache statistics
• /debug/database/stats: Database metrics

See DEBUG_MONITORING.md for complete documentation.

🛡️ Rate Limiting

nFlow Runtime includes IP-based rate limiting to protect against abuse:

• Token bucket algorithm for flexible rate control
• Memory and Redis backends for different deployment scenarios
• Configurable exclusions for IPs and paths
• Detailed headers for client integration

See RATE_LIMITING.md for complete documentation.

🚀 Performance Optimizations

nFlow Runtime has been optimized to handle heavy JavaScript workloads:

VM Pool

• Goja VM reuse through configurable pool
• Pre-loading of VMs at startup for immediate availability
• Intelligent management with 5-second wait timeout
• Detailed pool status metrics

Cache System

• Babel Cache: ES6 transformations in memory
• Program Cache: Pre-compiled JavaScript
• Auth.js Cache: Avoids repetitive file reads

JMeter Test Results

• Tested workflow: httpstart → js-JsonRender with 1000 mathematical calculations
• Demonstrated throughput: 3,396 req/s (~3.4 million calculations/second)
• Reliability: 1,007,399 requests processed with 0% errors
• Average latency: 860ms (includes JS compilation + 1000 operations)
• Response times: Minimum 25ms, maximum 2,488ms
• Standard deviation: 87.36ms (predictable behavior)
• Transfer capacity: 5,265.98 KB/s

🚨 Error Handling

Errors are handled consistently:

• HTTP 408: Resource limit exceeded
• HTTP 500: Internal server error
• HTTP 404: Workflow not found

🔄 Project Status

• Maturity: 4.9/5 ⭐ (Production ready)
• Stability: STABLE ✅
• Security: EXCELLENT ✅ (Static Analysis + Encryption + Sanitization)
• Performance: 3,396 RPS with intensive JavaScript (0% errors) ✅
• Observability: COMPLETE ✅ (Health checks + Prometheus + Debug endpoints)
• Production Ready: 99% ✅

See STATUS.md for more details.

🐛 Known Issues

See DEUDA.md for the complete technical debt list.

🤝 Contributing

1. Fork the project
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

📝 License

MIT - see LICENSE file for details.

🙏 Acknowledgments

• Goja - JavaScript engine in Go
• Echo - Web framework
• nFlow - Visual workflow designer