Mixed-Precision Training
AI & Machine LearningA training technique that uses lower-precision number formats where they are safe and higher precision only where stability requires it, gaining speed and saving memory without sacrificing accuracy.
The B300's tensor cores are built for exactly this blend of precisions, so mixed-precision training on Eleveight AI's cluster runs faster and fits larger models than full-precision training would allow.
Overview
Training was once done entirely in 32-bit floating point, which is precise but slow and memory-hungry. Mixed-precision training recognizes that most of a model's calculations tolerate lower precision perfectly well, and only a few sensitive steps genuinely need the full 32 bits. By using a faster 16-bit format for the bulk of the work and reserving higher precision for the parts that matter, it captures most of the speed of low precision with none of the instability.
How it works
The technique keeps a high-precision master copy of the model's weights while performing the heavy forward and backward computations in a 16-bit format such as FP16 or BF16. A method called loss scaling prevents the small gradient values from vanishing in the reduced range. Modern frameworks automate the whole arrangement, deciding which operations run in which precision so developers gain the benefit without managing it by hand.
Why it matters
Mixed precision delivers a large, almost free improvement: faster training, lower memory use, and the ability to fit bigger models or batches, while matching the accuracy of full-precision runs. Because tensor cores are designed to accelerate exactly these lower-precision operations, the gain is greatest on current hardware, making mixed precision the standard approach to training at scale.
Use cases
- Faster training of large models
- Fitting larger models or batch sizes in memory
- Standard practice for foundation model training
- Efficient use of tensor-core hardware