NVLink & NVSwitch GPU Interconnect Topology for AI Cluster Design

Why GPU Interconnect Topology Matters More Than Raw FLOPS

A single NVIDIA data center GPU can deliver tens or hundreds of teraflops of compute power, but large-scale AI training workloads distribute computation across dozens or hundreds of GPUs working in concert. The performance of that distributed system depends not only on the arithmetic capability of each GPU but on how fast those GPUs can exchange data with each other during training. Gradient synchronization, activation checkpointing, pipeline parallelism, and tensor parallelism all require high-bandwidth, low-latency communication between GPUs.

When the interconnect fabric between GPUs is slow relative to the compute speed, GPUs spend most of their time waiting for data rather than performing useful matrix operations. This communication bottleneck is the primary reason that many AI training clusters achieve only 30 to 50 percent of their theoretical peak throughput. The difference between a well-designed interconnect topology and a poorly designed one can mean the difference between training a model in two weeks versus two months.

NVIDIA addresses this challenge with two complementary technologies: NVLink, a high-bandwidth point-to-point interconnect, and NVSwitch, a crossbar switch that connects multiple NVLink ports into a fully non-blocking fabric. Understanding how these technologies work together is essential for anyone designing, deploying, or hosting GPU colocation infrastructure for AI workloads.

NVLink: The Point-to-Point Foundation

Architecture and Generations

NVLink is a proprietary high-speed serial interconnect that provides direct GPU-to-GPU communication without traversing the PCIe bus or host CPU. Each NVLink connection consists of multiple differential signal pairs (lanes) grouped into sub-links, with each generation increasing both the per-lane data rate and the number of links available per GPU.

The evolution of NVLink bandwidth reflects the escalating communication demands of large language model training. First-generation NVLink on Pascal delivered 160 GB/s aggregate per GPU. By the time Hopper (H100) introduced NVLink 4th generation, aggregate bandwidth had reached 900 GB/s per GPU across 18 links. Blackwell maintains the 900 GB/s aggregate while doubling per-link bandwidth to 100 GB/s bidirectional, reducing the number of physical connections needed for the same total throughput.

Generation	GPU Architecture	Bandwidth per Link	Links per GPU	Aggregate per GPU
NVLink 1.0	Pascal (P100)	40 GB/s	4	160 GB/s
NVLink 2.0	Volta (V100)	50 GB/s	6	300 GB/s
NVLink 3.0	Ampere (A100)	50 GB/s	12	600 GB/s
NVLink 4.0	Hopper (H100)	50 GB/s	18	900 GB/s
NVLink 5.0	Blackwell (B200/GB200)	100 GB/s	18	900 GB/s

NVLink vs PCIe: Why the Difference Matters

PCIe Gen 5 x16, the fastest standard host bus interface available in current server platforms, provides approximately 64 GB/s bidirectional bandwidth per slot. NVLink 4th and 5th generation deliver 900 GB/s per GPU, roughly 14 times faster than PCIe. For distributed training operations where GPUs must exchange gradient tensors, activation data, and HBM memory contents during every training step, this bandwidth difference translates directly into higher GPU utilization and faster time-to-train.

Beyond raw bandwidth, NVLink provides lower latency and supports direct GPU-to-GPU memory access through NVIDIA's unified memory architecture. One GPU can read from or write to another GPU's HBM without involving the host CPU, enabling efficient implementation of tensor parallelism where different GPUs hold different slices of the same layer's weight matrix.

NVSwitch: Building All-to-All Fabrics

The Crossbar Switch Architecture

While NVLink provides high-bandwidth point-to-point connections, it cannot by itself create a topology where every GPU can communicate with every other GPU at full bandwidth. Without NVSwitch, a system with 8 GPUs would need to dedicate some NVLink ports to direct connections with neighbors, creating a mesh topology where some GPU pairs communicate directly while others must traverse intermediate hops. Each hop adds latency and reduces effective bandwidth.

NVSwitch solves this by acting as a high-radix crossbar switch with dozens of NVLink ports. Each NVSwitch chip can simultaneously route traffic between any pair of connected GPUs at full NVLink bandwidth without blocking. By connecting all GPUs in a system to a set of NVSwitch chips, the topology becomes fully non-blocking: every GPU can communicate with every other GPU at maximum bandwidth at the same time.

NVSwitch Generations

The first-generation NVSwitch, introduced with the DGX-2 system using V100 GPUs, provided 18 NVLink 2.0 ports per switch chip, enabling all-to-all connectivity among 16 GPUs. The third-generation NVSwitch in the DGX H100 connects 8 H100 GPUs through 4th-generation NVLink with 64 ports per switch, and the system uses four NVSwitch chips to provide full bisection bandwidth across all 8 GPUs.

The fourth-generation NVSwitch in the GB200 NVL72 platform represents a major architectural leap. It extends the NVLink domain from 8 GPUs within a single node to 72 GPUs across an entire rack. Each 4th-gen NVSwitch chip provides 144 NVLink 5.0 ports and incorporates compute capabilities for in-network reduction operations, offloading collective communication from the GPUs themselves.

NVLink Domains: The Fundamental Building Block

What Defines an NVLink Domain

An NVLink domain is the set of GPUs that can communicate entirely through NVLink and NVSwitch without using any external network interface. Within an NVLink domain, all GPU-to-GPU transfers happen at NVLink bandwidth (900 GB/s per GPU) with sub-microsecond latency. Communication that crosses NVLink domain boundaries must traverse the external network fabric, typically InfiniBand or RoCE Ethernet, at much lower bandwidth and higher latency.

The size of the NVLink domain determines the largest tensor-parallel or FSDP (Fully Sharded Data Parallelism) group that can operate at full NVLink speed. For the DGX H100, the NVLink domain is 8 GPUs in one node. For the GB200 NVL72, the NVLink domain spans 72 GPUs across 36 compute trays in a single rack. This 9x expansion in NVLink domain size dramatically reduces the amount of training traffic that must traverse the slower inter-rack network.

Impact on Parallelism Strategies

Modern large model training uses multiple forms of parallelism simultaneously. Tensor parallelism (TP) shards individual layers across GPUs that must communicate at every forward and backward pass. Pipeline parallelism (PP) partitions the model into sequential stages with communication only at stage boundaries. Data parallelism (DP) replicates the model across groups with gradient synchronization at the end of each training step.

Because tensor parallelism requires the most frequent and latency-sensitive communication, it should always be mapped within the NVLink domain. Pipeline parallelism is more tolerant of latency and can span NVLink domain boundaries. Data parallelism produces the largest but least frequent communication and is typically the outermost layer, spanning the entire training cluster over the external network.

With an 8-GPU NVLink domain, TP is limited to 8-way. With a 72-GPU NVLink domain, TP can scale to 72-way, enabling much larger model layers to be sharded without crossing the network boundary. This architectural shift is what makes the GB200 NVL72 particularly well-suited for training the largest foundation models with trillions of parameters.

Multi-Node Scaling: Beyond the NVLink Domain

The Two-Tier Network Architecture

AI training clusters larger than a single NVLink domain use a two-tier network architecture. The first tier is the NVLink/NVSwitch fabric within each domain, providing the highest bandwidth and lowest latency. The second tier is the external InfiniBand or RoCE network connecting NVLink domains to each other.

In an H100-based cluster, each DGX H100 node has 8 GPUs in the NVLink domain and 8 ConnectX-7 network adapters (one per GPU) providing 400 Gb/s InfiniBand each. The aggregate inter-node bandwidth per node is 3.2 Tb/s (400 GB/s), which is less than half the per-GPU NVLink bandwidth. This ratio illustrates why keeping the most communication-intensive parallelism strategies within the NVLink domain is critical.

The GB200 NVL72 changes this ratio favorably by expanding the NVLink domain to 72 GPUs while providing 400 Gb/s ConnectX-8 adapters for inter-rack communication. With 72 GPUs sharing the NVLink fabric before hitting the network boundary, a much smaller fraction of total training traffic needs to traverse the external network.

Fat-Tree and Rail-Optimized Topologies

The external network connecting NVLink domains typically uses a fat-tree topology built from InfiniBand or Ethernet switches. In a fat-tree, every leaf switch connects to every spine switch, providing multiple equal-cost paths between any two endpoints. The full bisection bandwidth design ensures that the network does not become a bottleneck when all nodes communicate simultaneously during all-reduce operations.

Rail-optimized topologies, introduced with the DGX SuperPOD reference architecture, assign each GPU in a node to a separate network rail. GPU 0 in every node connects to the same set of leaf switches, GPU 1 to a different set, and so on. This design ensures that all-reduce traffic for a given data-parallel rank stays within a single rail, reducing cross-rail switch-hop latency and simplifying network congestion management.

Colocation Infrastructure Requirements

Power and Cooling

NVLink-connected GPU systems are among the most power-dense computing equipment ever deployed in colocation facilities. A DGX H100 system consumes approximately 10.2 kW. A GB200 NVL72 rack draws approximately 120 kW and generates heat density that exceeds the capability of conventional air cooling.

Colocation facilities hosting NVLink-based AI clusters must provide direct-to-chip liquid cooling or immersion cooling infrastructure for Blackwell-generation hardware. This includes coolant distribution units on every row, chilled water supply with sufficient delta-T capacity, and redundant pump systems. Air-cooled facilities can host H100 and A100 systems at rack densities of 30 to 40 kW with rear-door heat exchangers, but Blackwell at 120 kW per rack requires engineered liquid cooling from the outset.

Electrical Infrastructure

High-density GPU racks demand robust grounding and bonding to maintain signal integrity across the NVLink high-speed serial links. Ground loops and electromagnetic interference can cause link errors, CRC failures, and NVLink retraining events that degrade training throughput. The colocation facility should provide MESH-BN grounding topology per TIA-607-D for rooms hosting NVLink-connected systems.

Power redundancy is equally critical. A single power interruption lasting even a few milliseconds can terminate a multi-day training run, wasting all computation performed since the last checkpoint. 2N power distribution with transfer time under 10 ms and UPS battery backup are minimum requirements for production AI training infrastructure.

Network Infrastructure

While NVLink handles intra-domain communication, the colocation facility must provide the physical infrastructure for the inter-domain network fabric. This typically means pre-installed fiber optic cabling with MPO/MTP connectors, structured cabling runs between every rack position in the AI cluster zone, and dedicated fiber optic patch panels with sufficient port density for the network switches.

A 1,024-GPU H100 cluster (128 nodes) with rail-optimized topology requires 1,024 individual 400 Gb/s fiber connections from leaf switches to GPU nodes, plus hundreds more from leaf to spine switches. Cable management, pathway routing, and structured documentation become critical operational concerns at this scale.

Performance Optimization and Monitoring

Collective Communication Libraries

The software layer that manages GPU-to-GPU communication across NVLink and the external network is NCCL (NVIDIA Collective Communications Library). NCCL automatically detects the interconnect topology, including NVLink connections, NVSwitch fabric, and network adapters, and selects optimal algorithms for collective operations like all-reduce, all-gather, reduce-scatter, and broadcast.

For NVLink-connected GPUs, NCCL uses ring and tree algorithms that exploit the full NVLink bandwidth. When communication crosses NVLink domain boundaries, NCCL coordinates with the network fabric to overlap computation and communication, hiding latency through pipelining. Operators deploying Kubernetes-orchestrated GPU clusters should ensure NCCL topology detection is properly configured so the library can make optimal routing decisions.

Monitoring NVLink Health

NVLink lane errors, CRC errors, and link retraining events are early indicators of interconnect problems that will degrade training performance. NVIDIA's Data Center GPU Manager (DCGM) and DCIM platforms can monitor per-link error counters, bandwidth utilization, and link state in real time. Facilities should establish baseline error rates after initial deployment and alert when rates exceed thresholds, particularly after any power event, cooling change, or physical rack maintenance.

Common root causes of NVLink errors include loose physical connections (for tray-based systems like NVL72), electromagnetic interference from inadequate grounding, thermal throttling due to insufficient cooling, and firmware version mismatches between GPUs and NVSwitch chips. A proactive monitoring approach catches these issues before they cause training job failures.

Future Directions

NVLink and Optical Interconnects

NVIDIA's roadmap includes co-packaged optics to extend NVLink domain boundaries beyond a single rack. Current electrical NVLink connections are limited to cable lengths of approximately 2 meters, which constrains the physical layout of NVLink domains. Optical NVLink would allow GPUs in different racks or even different rooms to participate in the same NVLink domain, potentially expanding domain sizes to hundreds or thousands of GPUs.

This evolution would further shift the importance of colocation facility design toward providing the fiber optic infrastructure, cooling capacity, and power density needed for optical-NVLink-connected systems. Operators planning new data center builds should consider structured fiber pathways and flexible cooling zones that can accommodate expanding NVLink domains in future GPU generations.

Competitive Interconnect Landscape

AMD offers Infinity Fabric for MI300X GPU-to-GPU communication, while Intel's Ponte Vecchio used EMIB and Foveros packaging for internal interconnect. The competitive landscape ensures continued innovation in GPU interconnect bandwidth and efficiency, benefiting enterprise AI infrastructure decisions by providing more options for high-performance distributed training.

Frequently Asked Questions

What is the difference between NVLink and NVSwitch?

NVLink is a point-to-point high-bandwidth interconnect between two GPUs or between a GPU and an NVSwitch chip. NVSwitch is a crossbar switch that connects multiple NVLink ports into a fully non-blocking fabric. Without NVSwitch, GPUs can only connect to their immediate NVLink neighbors. With NVSwitch, every GPU has a direct high-bandwidth path to every other GPU in the domain.

How much bandwidth does NVLink 5th generation provide?

NVLink 5th generation on Blackwell delivers 900 GB/s of bidirectional bandwidth per GPU across 18 links at 100 GB/s each. This matches the aggregate bandwidth of NVLink 4th generation on Hopper but doubles the per-link rate, enabling more efficient fabric topologies with fewer physical connections.

What is an NVLink domain?

An NVLink domain is the group of GPUs that can communicate entirely through NVLink and NVSwitch without traversing the external network. On DGX H100, the domain is 8 GPUs. On GB200 NVL72, the domain spans 72 GPUs across a full rack, all interconnected at NVLink speeds.

How does GPU interconnect topology affect AI model training performance?

Interconnect topology determines the speed of collective operations (all-reduce, all-gather, reduce-scatter) that dominate distributed training communication. Larger NVLink domains reduce the fraction of traffic that must traverse the slower external network, improving GPU utilization from typical 30-50% on smaller clusters to 60-70% or higher on well-designed NVLink-connected systems.

What colocation infrastructure does an NVLink-based AI cluster require?

High-density power delivery (40-120 kW per rack), liquid cooling infrastructure, robust electrical grounding, and high-capacity network connectivity for inter-rack communication. GB200 NVL72 racks require liquid cooling capable of removing 100+ kW of heat per rack.