tensor-fusion

TensorFusion.AI
Less GPUs, More AI Apps.
Explore the docs »
View Demo | Report Bug | Request Feature

Tensor Fusion is a state-of-the-art GPU virtualization and pooling solution designed to optimize GPU cluster utilization to its fullest potential.

🌟 Highlights

📐 Fractional Virtual GPU

🔄 Remote GPU Sharing over Ethernet/InfiniBand

⚖️ GPU-first Scheduling and Auto-scaling

📊 GPU Oversubscription and VRAM Expansion

🛫 GPU Pooling, Monitoring, Live Migration, Model Preloading and more

🎬 Demo

Fractional vGPU & GPU-over-IP & Distributed Allocation

AI Infra Console

GPU Live-migration [End-to-end feature WIP]

https://cdn.tensor-fusion.ai/GPU_Content_Migration.mp4

🚀 Quick Start

Onboard Your Own AI Infra

• Deploy in Kubernetes cluster
• Create new cluster in VM/BareMetal
• Learn Essential Concepts & Architecture

💬 Discussion

• Discord channel: https://discord.gg/2bybv9yQNk
• Discuss anything about TensorFusion: Github Discussions
• Contact us with WeCom for Greater China region: 企业微信
• Email us: support@tensor-fusion.com
• Schedule 1:1 meeting with TensorFusion founders

🔮 Features & Roadmap

Core GPU Virtualization Features

• Fractional GPU and flexible oversubscription
• Remote GPU sharing with SOTA GPU-over-IP technology, less than 4% performance loss
• GPU VRAM expansion and hot/warm/cold tiering
• None NVIDIA GPU/NPU vendor support

Pooling & Scheduling & Management

• GPU/NPU pool management in Kubernetes
• GPU-first scheduling and allocation, with single TFlops/MB precision
• GPU node auto provisioning/termination
• GPU compaction/bin-packing
• Seamless onboarding experience for Pytorch, TensorFlow, llama.cpp, vLLM, Tensor-RT, SGlang and all popular AI training/serving frameworks
• Centralized Dashboard & Control Plane
• GPU-first autoscaling policies, auto set requests/limits/replicas
• Request multiple vGPUs with group scheduling for large models
• Support different QoS levels

Enterprise Features

• GPU live-migration, snapshot and restore GPU context cross cluster
• AI model registry and preloading, build your own private MaaS(Model-as-a-Service)
• Advanced auto-scaling policies, scale to zero, rebalance of hot GPUs
• Advanced observability features, detailed metrics & tracing/profiling of CUDA calls
• Monetize your GPU cluster by multi-tenancy usage measurement & billing report
• Enterprise level high availability and resilience, support topology aware scheduling, GPU node auto failover etc.
• Enterprise level security, complete on-premise deployment support
• Enterprise level compliance, SSO/SAML support, advanced audit, ReBAC control, SOC2 and other compliance reports available

🗳️ Platform Support

• Run on Linux Kubernetes clusters
• Run on Linux VMs or Bare Metal (one-click onboarding to Edge K3S)
• Run on Windows (Not open sourced, contact us for support)

See the open issues for a full list of proposed features (and known issues).

🙏 Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Top contributors

🔷 License

1. TensorFusion main repo is open sourced with Apache 2.0 License, which includes GPU pooling, scheduling, management features, you can use it for free and customize it as you want.
2. vgpu.rs repo is open sourced with Apache 2.0 License, which includes Fractional GPU and vGPU hypervisor features, you can use it for free and customize it as you want.
3. Advanced GPU virtualization and GPU-over-IP sharing features are also free to use when GPU total number of your organization is less than 10, but the implementation is not fully open sourced, please contact us for more details.
4. Features mentioned in "Enterprise Features" above are paid, licensed users can use these features in TensorFusion Console.
5. For large scale deployment that involves non-free features of #3 and #4, please contact us, pricing details are available here