Understanding LLM GPUs Clusters Fabrics Traffic For Networkers

• • LLM training clusters generate intense, synchronized east-west traffic that demands ultra-high bandwidth, ultra-low latency and near-zero packet loss from the network fabric.
  • Fabric architecture choices — including topology, buffering, congestion control and oversubscription ratios — directly determine how much of the available GPU compute time is spent doing useful work versus waiting on the network.
  • Network stability and resilience are as critical as raw performance, because a single straggler flow, silent packet loss event or node failure can stall an entire training job across thousands of GPUs.

As we’ve seen in the earlier parts of this series, LLM training clusters generate highly deterministic yet extremely intense east-west traffic patterns, driven by collective operations and synchronization across thousands of GPUs. For network engineers, the real challenge is no longer just the need for “more bandwidth,” but building fabrics that can sustain such traffic with predictable low latency and resiliency at scale. In this part of the series, we shift from traffic characteristics to network requirements — translating those GPU-driven traffic behaviors into concrete network hardware/software expectations for fabric architecture, topology, buffering, congestion control and telemetry. The goal is to make the network less of a bottleneck and more of an enabler of LLM training performance.

Ultra-large-scale networking

As LLMs scale into the hundreds of billions and trillions of parameters, the network becomes a real constraint. A 1-trillion-parameter model alone requires roughly 2 TB of storage for parameters at 16-bit precision, while training expands the footprint significantly through forward activations, backward gradients and optimizer states. With optimizers such as Adam, peak memory usage can reach about 7× the model size.⁴ In practice, this forces training jobs to span dozens to hundreds of GPUs simply to hold the full training state and thousands of GPUs for ultra-large models. To keep up with the GPUs, the fabric must deliver 400–800 Gb/s per GPU with RDMA to minimize latency and CPU overhead. At that scale, however, large-scale RDMA behavior introduces head-of-line blocking and PFC deadlock storms. The sheer number of RDMA QPs per host creates additional pressure for more sophisticated congestion control, load balancing and topology choices to balance scalability, cost and failure domains.

To meet these scale requirements for large GenAI fabrics, the Ultra Ethernet Consortium positions Ultra Ethernet as an open Ethernet-based communications stack for AI/HPC. The UEC standard increases bandwidth, improves latency and tail latency. It will also maintain interoperability with Ethernet. UEC is also aiming to redesign the transport to avoid classic connection-state bottlenecks by using a highly scalable connectionless transport with ephemeral packet-delivery contexts rather than heavy connection-oriented dependencies, making large deployments more practical.

A key point is that much of this innovation sits at the endpoints and NICs. Most of the gains come from endpoint support for simplified RDMA behavior, multi-pathing, higher utilization and lower tail latency.¹ In that sense, ultra-large-scale AI is about adding more GPUs; and in the same token it is about designing a fabric that can scale and leverage new topologies including Pure Rail and rail-optimized approaches.

Ultra-high bandwidth networking

In large-scale AI training clusters, collective communication (i.e. NVIDIA NCCL, RCCL, etc.) between GPUs, both within a node and across nodes, drives unprecedented bandwidth demand on the fabric. Inside the server, GPUs are linked by high-speed interconnects (i.e. NVLink from NVIDIA, XGMI from AMD, etc.) specifically to keep traffic off the CPU and system memory path.⁴ Once traffic leaves the GPU node, however, the network must simultaneously support very different communication patterns. Some of these collectives routinely push hundreds of gigabytes of data, with overlapping one-to-many and many-to-one flows that stress every layer of the network.

This is why AI fabrics require as much bandwidth as we can provide them and more. That is the foundation for maintaining fabric-wide throughput as GPU counts grow.¹ The requirement is not simply for faster links at the edge, but for a scale-out fabric that can sustain the offered load of large, synchronized collectives across the cluster. The number of available uplinks per node and per rack must also increase and traffic must be evenly distributed across all available paths to avoid congestion. In practice, link speed alone will not solve the problem if uplinks, spine capacity and congestion avoidance mechanisms cannot keep pace with GPU compute density.

Bisection bandwidth describes the aggregate capacity of the fabric when it is cut into two halves and measures how much traffic can cross that cut concurrently. Parallel training jobs spanning thousands of GPUs regularly generate all-to-all and all-reduce patterns that effectively push on the bisection.

When sizing GenAI networks, it is important to distinguish between bandwidth per GPU and bisection bandwidth, because both matter in different ways. Bandwidth per GPU is what each accelerator sees on its NIC. If this is too low relative to compute capability, individual ranks will spend a disproportionate amount of time waiting on collective communication, even if the rest of the fabric is theoretically ready for the next computation. Each node therefore needs enough headroom to move hundreds of gigabytes per step without saturating.

On the contrary, if the topology, oversubscription ratios or spine capacity are undersized (which all impacts bisectional bandwidth) congestion at the core will throttle the entire cluster, regardless of edge-port speed. Real-world training clusters therefore typically target 1:1 oversubscription because efficiency depends on getting both sides right. The most common fabric architecture nowadays makes this concrete by reserving 50% of switch ports as uplinks to spines, yielding full bisection bandwidth within the scale-out plane.²

The port-speed progression over the last few years reflects this shift. What was once considered fast at 100 Gb/s per port quickly became inadequate as model sizes and GPU counts expanded. Production-grade training clusters moved to 200/400 Gb/s to keep early large models from stalling on gradient exchange, while the current generation is normalizing 800 Gb/s GPU-attached NICs. In parallel, GenAI scale-out networking is now framed around 800G scale-out networks (with InfiniBand XDR switching family), emerging 1.6T scale-out networks and a roadmap beyond 1.6T.¹

Taken together, ultra-high bandwidth for GenAI is not just about faster ports. It is about aligning port speeds, link counts, non-blocking topology and end-to-end fabric capacity with modern load balancing techniques and congestion avoidance technology, while sustaining full-bandwidth, fabric-wide throughput at cluster scale.

Ultra-low latency and jitter

From a networker’s perspective, GenAI and LLM training are brutally sensitive to bandwidth availability, latency and the losslessness each flow experiences. AI training is a sequence of computation and communication phases, and each phase must be fully completed before the next one can begin, which is why overall training performance heavily depends on the underlying network physical layer performance.⁵ In this context, end-to-end latency can be broken into two parts: static latency and dynamic latency. Static latency is the baseline cost of getting a packet from GPU A to GPU B when the network is empty — propagation delay, serialization delay and a minimal number of forwarding decisions. Dynamic latency is everything added on top when the fabric is busy: queuing, transient congestion and retries. In realistic, large-scale GenAI clusters, dynamic latency becomes the primary bottleneck for training efficiency.

Within that sequence, tail latency is the time required to deliver the slowest, or last, messages, and it significantly impacts training efficiency because the communication step is governed by the last messages to arrive, not the average behavior of the network.⁵ This is why latency jitter — the variability in flow completion times — directly impacts performance. Most collectives can be decomposed into multiple parallel P2P exchanges between nodes. In Ring AllReduce, for N nodes, the operation consists of 2 × (N − 1) communication sub-steps, and each sub-step completes only when every P2P flow finishes.⁴ As soon as there is a localized fluctuation — a hot spot on one link or transient congestion affecting one pair of nodes — the flow completion time for that exchange stretches out, and all other GPUs must wait at the barrier for the slowest pair to finish. At cluster scale, a handful of straggler flows caused by jitter can throttle the entire job, even if average latency across the fabric looks acceptable.

Analyses show that when dynamic latency grows from around 10 μs to 1000 μs, the fraction of time GPUs spend doing useful work can drop by nearly 10%, a double-digit loss in effective cluster throughput purely due to added queueing and congestion.⁴ Panduit makes the same point from the perspective of accelerator efficiency, noting that GPUs can be idle up to 50% of the time due to communication delays. In other words, a network that might be “good enough” for typical enterprise traffic is catastrophically bad for synchronized LLM training.

Packet loss is even more punishing: at a packet loss rate of 0.1% (1 in 1000), effective GPU compute utilization can fall by around 13%, and once the loss rate reaches 1%, effective GPU compute time can sink to below 5%.⁴

This interaction between dynamic latency, tail latency, packet loss and jitter is what makes ultra-low latency and low jitter a hard requirement, not an optimization task. Every training step involves waves of collectives, and each wave progresses at the speed of the slowest flow in each sub-step.

To address this in larger AI back-end networks, rail-optimized designs are preferred because rail optimization minimizes the number of hops between servers to achieve the lowest possible latency.⁵ Taken together, these points frame the network not simply as a transport layer, but as a determining factor in AI job efficiency.

Designing for GenAI training therefore means minimizing dynamic latency with shallow queues and fast congestion feedback, driving packet loss toward zero for RDMA and collective traffic. You need to engineer the network so that flow completion times are tightly clustered rather than spread out, and using low-hop rail-optimized connectivity to preserve the low-latency behavior required for scalable AI training.

Network stability and uptime for cluster performance

In large-scale GenAI training environments, network stability is a critical factor in overall cluster performance. The scale of modern model training highlights this clearly: training a model such as GPT-3 on a single V100 GPU would take an impractical 335 years, while a cluster of 10,000 V100 GPUs can, in ideal conditions, reduce that time to roughly 12 days.⁴ That level of acceleration, however, depends heavily on the network’s ability to provide consistent, reliable communication across all participating nodes.

As clusters grow, distributed training becomes increasingly dependent on efficient synchronization between GPUs, servers, and storage resources. Any disruption in network availability can directly affect computational stability across the environment. If a node fails during a training run, the cluster may need to rely on fault-tolerant replacement or elastic scaling to maintain progress. When nodes are added, removed, or reassigned, existing communication paths may no longer be optimal, requiring workload redistribution and job rescheduling to restore efficiency.

Network performance inconsistencies also create less visible but equally damaging issues. More subtle issues such as silent packet loss can occur unexpectedly and are especially problematic in large, distributed environments. These conditions degrade the efficiency of collective communications, increase latency, and may trigger communication library timeouts. In practice, this can cause training jobs to stall for extended periods, leaving high-value GPU resources underutilized and significantly reducing throughput.

For GenAI clusters, the network is more than a transport layer—it is more of critical a performance enabler. Stable, adaptable, and resilient networking is essential to minimizing job interruptions, and maximizing the value of accelerated compute infrastructure.

¹Open Cluster Designs For AI

²Open Cluster Designs Aligned AI Inference Fabric Reference Architecture

³Optimizing Network Fabrics for AI Workloads

⁴Network Requirements for AI Large-Scale Models in Data Centers

⁵Impact Of Structured Cabling On AI Network Performance