Distributed Training Across Mixed GPUs: Solving the Heterogeneous Fleet Problem

Distributed Training Across Mixed GPUs: Solving the Heterogeneous Fleet Problem As machine learning models grow larger, the hardware requirements become more demanding. But what if your lab has a mix of GPUs from different generations — an RTX 3090 here, a V100 there, maybe even some older M40s gathering dust? Traditionally, distributed training tools assume homogeneous hardware, leaving these mismatched cards underutilized. The Challenge Most distributed training frameworks expect identical GPUs across nodes. If your setup includes: NVIDIA RTX 3090 (24GB VRAM) RTX 4090 (24GB VRAM) Tesla V100 (16GB VRAM) Quadro M40 (24GB VRAM) You can't easily pool them into a single training job. The differences in architecture, memory, and compute capability create bottlenecks. A New Approach We're experimenting with a distributed training method that works across heterogeneous GPU fleets. The key components: 4-Bit NF4 Quantized Sharding Uses 4-bit quantization with Normal Float 4 (NF4) distribution