Latency

Overview

Latency measures responsiveness: how long it takes for something to happen after it is asked for. It is easy to confuse with throughput, but the two are different. Throughput is how much work passes through a system over time; latency is how quickly any single piece of that work completes. A system can have high throughput and poor latency, or the reverse, and which one matters depends entirely on the workload.

How it works

In AI infrastructure, latency appears at several layers. There is network latency, the time data spends traveling between user and data center, governed largely by physical distance. There is interconnect latency, the delay as GPUs exchange information during distributed work, which InfiniBand XDR and NVLink are built to minimize. And there is inference latency, the time a model takes to produce a response once a request arrives. Each adds to the delay a user ultimately feels.

Why it matters

For interactive applications, an AI assistant, a real-time analysis tool, latency is what users actually experience as speed. Even a fast model feels sluggish if requests must cross continents to reach it. This is much of the case for regional infrastructure: placing compute physically nearer to its users removes network distance that no amount of raw GPU power can compensate for.