scalebox

Scalebox — A Cloud-Native Stream Computing Engine

Scalebox is a cloud-native stream computing engine that runs containerized user algorithms on distributed, heterogeneous computing clusters, enabling large-scale parallel processing through pipelines of hierarchical modules with task-level fault tolerance. Compared with existing big data processing and parallel computing frameworks, its technical characteristics are particularly suited to scenarios with distributed data, distributed compute resources, and complex algorithms.

✨ Key Features

• Cloud-Native Design: all algorithm modules are containerized and embedded into data processing pipelines via the sidecar pattern; control messages and data channels are separated, and adjacent modules are linked through a message bus, enabling multi-language, non-intrusive parallel programming.

• Virtual Tasks (VTask / Task Groups): an application-level coarse-grained compute unit built on top of fine-grained Tasks. Cross-module Task collections incorporate semaphores and shared variables, managed by the wait-queue → vtask-head → vtask-core → vtask-tail pipeline for flow control, resource binding, and state tracking. Supports DEFAULT / HOST-BOUND / GROUP-BOUND modes, covering everything from lightweight batch processing to multi-node collaborative computing.

• Cross-Cluster Computing: algorithm modules and transfer modules are normalized and processed uniformly through pipelines for intra-cluster and cross-cluster data. Full-mesh replication of t_cluster plus gRPC proxy forwarding; the CLI always connects to the local controld, and cross-cluster operations are transparently routed by the server.

• Task-Level Fault Tolerance: automatic retries based on exit-code rules for transient errors caused by hardware failures, software bugs, network issues, or data anomalies. Fine-grained task-level fault tolerance enables trustworthy data analysis on unreliable hardware.

• Node-Local Computing Optimization: centered on node-local storage, using "operator-level spatial expansion + local storage residency + explicit data-flow orchestration" to move inter-node communication into the node itself, significantly reducing dependence on network bandwidth and external storage.

• WebUI Management Interface: REST gateway (:8088) with 76 endpoints covering all management functions. SPA frontend embedded as a single-file deployment, providing Dashboard statistics, App card grid, DAG topology, Task/VTask management, and semaphore/variable hierarchy trees.

• VS Code Extension: sidebar TreeView (three-level lazy-loading tree of App → Modules/Tasks/VTasks), task log Webview, DAG force-directed graph (ECharts), app.yaml LSP (syntax validation/autocompletion/hover hints). Packaged with Docker, zero local Node.js dependencies.

• Security Framework: JWT authentication + RBAC role system (admin/viewer/operator/automation) + gRPC TLS encryption + database certificate authentication. Compose-tiered optional enablement, disabled by default with zero overhead.

• Multi-Level Parallelism: intra-module algorithm parallelism, module-level data parallelism, and inter-module pipeline parallelism.

• Multiple Container Engines: Docker (default), Singularity/Apptainer, Podman.

🚀 Quick Start

1. Prepare the Environment

# Docker 20.10+
curl -fsSL https://get.docker.com | sh
sudo systemctl enable --now docker
docker --version

2. Start Scalebox

git clone https://github.com/kaichao/scalebox.git
cd scalebox

# Generate secrets (optional)
cd build && bash gen-secrets.sh && cd ..

# Build and start
make -C build/
docker compose -f build/compose.yaml up -d

Open http://localhost:8088 in your browser to access the WebUI.

3. Run an Example

cd examples/hello-scalebox
echo "Hello Scalebox" | scalebox run
scalebox app list

4. Install the VS Code Extension

cd vscode-scalebox && make install

📊 Core Concepts

📚 Documentation

Chinese User Documentation (→ Full TOC)

English Design Documents

🧪 Example Apps

• hello-scalebox — the first introductory app
• app-primes — prime number calculation, demonstrating data parallelism
• app-copy — cross-node data transfer
• remote-primes — cross-cluster prime number calculation
• vtask — VTask pipeline examples (DEFAULT / HOST-BOUND)

🏗️ Standard Modules

• dir-list — directory listing, generates lists of files to process
• file-copy — single-file copy based on rsync-over-ssh
• dir-copy — directory-level copy
• rsync-copy — efficient rsync transfer
• ftp-copy — FTP protocol transfer
• cron — scheduled message triggering
• cluster-head / node-agent — cluster node management

🔗 Related Software

• PostgreSQL — metadata storage
• gRPC — high-efficiency inter-component communication protocol
• Go — the language for cloud-native application development

🤝 Contributing

Issues and Pull Requests are welcome at GitHub Issues.

📄 License

Apache License 2.0 © Kaichao Wu