FP8 Precision
GPU Hardware & ComputeAn 8-bit floating-point number format that represents values with far fewer bits than traditional formats, roughly doubling throughput while largely preserving model quality.
The B300 GPUs in Eleveight AI's cluster support FP8 in hardware, letting both training and, especially, inference run at markedly higher throughput than older fixed-precision designs allowed.
Overview
Precision refers to how many bits a processor uses to represent each number. AI began on 32-bit floating point (FP32), moved to 16-bit formats (FP16 and BF16) to save memory and time, and has now reached 8-bit floating point, FP8. The insight driving this descent is that neural networks are remarkably tolerant of imprecision: much of the exactness in higher formats is wasted on a workload that cares more about overall patterns than about the last decimal place.
How it works
FP8 packs a number into eight bits, half the footprint of FP16. That smaller representation means more values fit in the same memory, more move across the same bandwidth, and more multiply-accumulate operations complete in the same silicon per cycle. The B300's tensor cores execute FP8 natively, and careful scaling techniques keep the reduced range from degrading results, so the speed-up arrives without a meaningful loss in model quality.
Why it matters
Lower precision converts almost directly into capability. Roughly doubling throughput against FP16 means more tokens generated per second in inference and faster steps in training, while the reduced memory footprint lets larger models fit on the same hardware. For inference in particular, where cost per query is decisive, FP8 is one of the largest single contributors to the B300's advantage over previous generations.
Use cases
- High-throughput LLM inference
- Mixed-precision training for speed and memory efficiency
- Serving large models within fixed memory budgets
- Cost-optimised production AI