Remote Direct Memory Access (RDMA)

Overview

In ordinary networking, moving data between machines involves the CPU and operating system at both ends, copying data through layers of software that add latency and consume processor time. RDMA strips that away. It allows one machine to access another's memory directly across the network, without involving the remote CPU, as if reaching into the other system's memory itself. The result is dramatically lower latency and far less overhead.

How it works

RDMA-capable network hardware handles the transfer itself, reading from or writing to memory directly while bypassing the CPU and the operating system's networking stack. Because the processors at both ends are not pulled into the work of copying data, communication completes in microseconds and leaves the CPUs free. It is the defining feature of InfiniBand and is also available over Ethernet through RoCE.

Why it matters

When training is spread across many nodes, the GPUs must constantly exchange gradient updates to stay synchronized, and any delay in that exchange leaves them waiting. RDMA's very low latency keeps the accelerators fed and the cluster efficient, so adding nodes translates into faster training rather than wasted capacity. It is much of why specialized fabrics outperform conventional networking for AI.