Multi-GPU Scaling
Networking & InterconnectDistributing an AI workload across multiple GPUs simultaneously, enabling models and training jobs too large for any single chip.
Eleveight AI's cluster is built with the NVLink and InfiniBand XDR fabric required for efficient multi-GPU scaling, within a HGX B300 node and across the full cluster.
Overview
Modern foundation models have outgrown any single GPU. A 175-billion-parameter model at full precision needs over 350GB of memory, comfortably beyond what even the most capable individual chip provides. Multi-GPU scaling is the response: spreading one workload across many accelerators at once, so that models and training jobs no single device could hold become tractable across a group of them.
How it works
There are three main strategies, often combined. Data parallelism gives each GPU the full model but a different slice of the data. Tensor parallelism splits individual layers across GPUs. Pipeline parallelism places different layers on different GPUs in sequence. All three share one demand, frequent and fast GPU-to-GPU communication, since the chips must constantly exchange data to act as one.
Why it matters
The deciding constraint is rarely the GPUs' raw speed but how quickly they can talk to one another. Inside an HGX B300 system, NVLink supplies enough bandwidth that the GPUs communicate almost as fast as they compute, while InfiniBand XDR links the systems across the cluster, so coordination overhead stays small. Without an interconnect of that caliber, adding chips brings sharply diminishing returns rather than proportional speed.
Use cases
- Training foundation models exceeding single-GPU memory
- Reducing training time via parallelization
- Running very large inference models across a cluster
- Distributed hyperparameter search