The NVIDIA Data Center GPU Ecosystem
NVIDIA dominates the data center GPU market with an estimated 80-90% market share as of 2026, driven by its combination of hardware performance, CUDA software ecosystem, and comprehensive developer tools. For data center operators and infrastructure providers, understanding the NVIDIA GPU ecosystem is essential for planning facilities, advising clients, procuring hardware, and ensuring your infrastructure can support the specific requirements of each GPU model.
This guide covers every major NVIDIA data center GPU available in 2026, organized by architecture generation and use case, with the specifications and deployment considerations that matter most to infrastructure operators and hosting providers.
Current Generation: Hopper Architecture (H100, H200)
NVIDIA H100
The H100 has been the workhorse of AI infrastructure since its launch in 2023. Built on the Hopper architecture with TSMC's 4nm process, it set the performance standard for large-scale AI training and high-throughput inference that defined the 2023-2025 era of AI development. While being superseded by Blackwell, the H100 remains widely deployed and continues to serve the majority of production AI workloads globally. Its mature software ecosystem, widespread driver support, and well-understood operational characteristics make it a reliable choice for organizations deploying proven rather than cutting-edge infrastructure.
| Specification | H100 SXM | H100 PCIe |
|---|---|---|
| Architecture | Hopper (GH100) | Hopper (GH100) |
| Process Node | TSMC 4nm | TSMC 4nm |
| FP8 Tensor Core Performance | 3,958 TFLOPS | 3,026 TFLOPS |
| FP16 Tensor Core Performance | 1,979 TFLOPS | 1,513 TFLOPS |
| GPU Memory | 80 GB HBM3 | 80 GB HBM3 |
| Memory Bandwidth | 3.35 TB/s | 2.0 TB/s |
| TDP (Thermal Design Power) | 700 W | 350 W |
| GPU Interconnect | NVLink 4.0 (900 GB/s bidirectional) | PCIe Gen 5 x16 |
| Form Factor | SXM5 (baseboard) | Dual-slot PCIe card |
Infrastructure Impact: An 8-GPU H100 SXM server (like the DGX H100) consumes approximately 10.2 kW including CPUs, memory, storage, and networking. A rack of four such servers requires 40+ kW of power delivery and cooling capacity. This power density is the benchmark against which GPU colocation facility readiness is measured; if a facility cannot reliably deliver and cool 40 kW per rack, it cannot support H100 SXM deployments.
SXM vs. PCIe: The SXM variant offers approximately 30% higher performance and 2x the memory bandwidth of the PCIe variant, along with NVLink interconnect for GPU-to-GPU communication. The PCIe variant is more versatile (fits in standard servers) and draws half the power, making it suitable for inference workloads where inter-GPU communication is less critical.
NVIDIA H200
The H200 is a memory-enhanced evolution of the H100 that significantly increases memory capacity and bandwidth while maintaining the same Hopper GPU architecture. It is designed as a drop-in replacement for H100 SXM systems, requiring no changes to server hardware, cooling, or power infrastructure.
| Specification | H200 SXM | Improvement vs H100 SXM |
|---|---|---|
| GPU Memory | 141 GB HBM3e | +76% (from 80 GB) |
| Memory Bandwidth | 4.8 TB/s | +43% (from 3.35 TB/s) |
| TDP | 700 W | Same |
| Compute Performance | Same as H100 SXM | - |
The H200's larger memory is its key advantage. With 141 GB per GPU, models that previously required tensor parallelism across multiple H100s can sometimes fit on fewer H200s, reducing networking overhead and improving both inference throughput and cost-per-token. For large language model inference, where the key-value cache can consume significant GPU memory, the H200 can serve 2-3x more concurrent users per GPU compared to the H100.
Infrastructure Impact: The H200 has the same power and cooling footprint as the H100 SXM, making it a true drop-in upgrade for existing H100 infrastructure. This makes it attractive for facilities that have already invested in H100-capable infrastructure and want to increase performance without infrastructure changes.
Next Generation: Blackwell Architecture (B100, B200, GB200)
NVIDIA B200
The B200 represents NVIDIA's Blackwell architecture, delivering a generational leap in both training and inference performance. Built using a novel two-die design connected via a high-bandwidth chip-to-chip interconnect, the B200 approximately doubles training performance and delivers up to 5x better inference throughput compared to the H100, with particularly dramatic improvements for transformer-based models.
| Specification | B200 |
|---|---|
| Architecture | Blackwell (GB202) |
| Process Node | TSMC 4NP (custom process) |
| Transistors | 208 billion (dual-die) |
| FP8 Tensor Performance | ~9,000 TFLOPS |
| FP4 Tensor Performance | ~18,000 TFLOPS |
| GPU Memory | 192 GB HBM3e |
| Memory Bandwidth | 8 TB/s |
| TDP | 1,000 W |
| GPU Interconnect | NVLink 5.0 (1,800 GB/s bidirectional) |
Key Innovation - FP4 Precision: Blackwell introduces native FP4 (4-bit floating point) support in its tensor cores, enabling models to run at 4-bit precision with minimal accuracy loss. This effectively doubles the compute throughput and halves the memory requirement compared to FP8, making it possible to run larger models on fewer GPUs.
Infrastructure Impact: At 1,000W TDP per GPU, the B200 pushes power density to new levels. An 8-GPU B200 server could consume 12-14 kW, and racks could exceed 50-60 kW. Direct liquid cooling becomes essentially mandatory at these densities, as no air-based cooling system can reliably remove 50-60 kW from a single rack. Facilities designed for H100s may require cooling infrastructure upgrades for Blackwell deployments, particularly if they were using rear-door heat exchangers at their maximum capacity.
NVIDIA GB200 NVL72
The GB200 NVL72 is a complete rack-scale system that represents the pinnacle of AI computing density. It contains 36 Grace CPUs (NVIDIA's ARM-based server processor) and 72 Blackwell GPUs, all interconnected via 5th-generation NVLink in a fully non-blocking topology that allows any GPU to communicate with any other GPU at full bandwidth.
- Total GPU Memory: 13,824 GB (192 GB per GPU times 72 GPUs), enough to hold the largest LLMs entirely in GPU memory
- Total FP8 Compute: ~648,000 TFLOPS (72 times 9,000 TFLOPS per GPU)
- Rack Power: Approximately 120 kW per rack
- Cooling: Liquid cooling absolutely required; there is no air-cooled option
- Network: 72x 400Gbps InfiniBand or Ethernet per rack for inter-rack communication
- Physical Size: Standard 42U rack form factor
The GB200 NVL72 sets a completely new standard for AI cluster density. A single rack delivers the equivalent performance of what previously required multiple racks of H100 systems, dramatically reducing the physical footprint and networking complexity of large AI clusters. However, the 120 kW per rack power requirement means that very few existing facilities can support these systems without significant infrastructure investment in power delivery and liquid cooling distribution.
Previous Generation: Ampere Architecture (A100)
NVIDIA A100
The A100 remains in widespread use despite being two generations behind the current architecture. It offers proven reliability, well-understood performance characteristics, and significantly lower acquisition costs through the secondary market, making it attractive for inference workloads and organizations optimizing for cost rather than peak performance.
| Specification | A100 SXM (80GB) | A100 PCIe (80GB) |
|---|---|---|
| Architecture | Ampere (GA100) | Ampere (GA100) |
| FP16 Tensor Performance | 312 TFLOPS | 312 TFLOPS |
| GPU Memory | 80 GB HBM2e | 80 GB HBM2e |
| Memory Bandwidth | 2.0 TB/s | 2.0 TB/s |
| TDP | 400 W | 300 W |
Infrastructure Impact: The A100's lower power consumption (400W SXM, 300W PCIe) makes it significantly easier to deploy in existing colocation facilities than H100 or Blackwell GPUs. An 8-GPU A100 server consumes approximately 5-6 kW, well within the range of many existing facilities with conventional air cooling. This lower infrastructure barrier makes the A100 attractive for organizations deploying their first GPU clusters in standard GPU Colocation: Everything You Need to Know in 2026 environments.
Inference-Optimized GPUs: L40S and L4
NVIDIA L40S
The L40S is designed specifically for AI inference, video processing, and graphics virtualization workloads. Built on the Ada Lovelace architecture, it uses standard PCIe form factor for broad server compatibility and does not require specialized cooling or power infrastructure.
- FP8 Tensor Performance: 733 TFLOPS
- GPU Memory: 48 GB GDDR6 with ECC (not HBM, which reduces cost)
- Memory Bandwidth: 864 GB/s
- TDP: 350 W
- Form Factor: PCIe dual-slot, fits in standard 2U and 4U servers
- Best For: AI inference for medium-sized models, real-time video analytics, graphics-intensive applications, VDI (Virtual Desktop Infrastructure)
- Key Advantage: No NVLink or HBM, but 48 GB of memory at a significantly lower price point than HBM-equipped GPUs
NVIDIA L4
- FP8 Tensor Performance: 242 TFLOPS
- GPU Memory: 24 GB GDDR6 with ECC
- TDP: 72 W
- Form Factor: Low-profile PCIe single-slot (no external power connector needed)
- Best For: Edge AI inference, lightweight models (7B parameter and below), video transcoding, density-optimized inference deployments
- Key Advantage: Extremely low power (72W) enables deployment in any standard server and any colocation facility, with no special cooling requirements
The L4's 72W TDP makes it exceptionally easy to deploy in existing infrastructure. A single 2U server can hold four L4 GPUs while consuming less total power than a single H100. For applications that do not require the raw performance of data center GPUs but need more than what CPUs can provide, the L4 offers an excellent power-efficient option.
Choosing the Right GPU for Your Workload
| Workload | Recommended GPU | Why |
|---|---|---|
| Frontier LLM Training (100B+ params) | B200 / GB200 NVL72 | Maximum compute, largest memory, best interconnect |
| Standard LLM Training (7B-70B params) | H100 SXM / H200 | Proven performance, widely available, excellent software ecosystem |
| LLM Inference (70B+ params) | H200 / H100 SXM | Large HBM memory capacity for model loading and KV cache |
| LLM Inference (7B-34B params) | A100 80GB / L40S | Cost-effective, sufficient memory, good throughput per dollar |
| Light Inference and Edge AI | L4 | Lowest power, highest density, fits anywhere |
| Fine-tuning (LoRA/QLoRA) | A100 / H100 PCIe | Good balance of cost and capability for adapter training |
| Computer Vision / Video Analytics | L40S / L4 | Ada Lovelace architecture optimized for media processing |
Infrastructure Planning by GPU Type
Power Budget Planning
When planning facility capacity for GPU deployments, use the TDP rating of each GPU as a starting point, then add overhead for CPUs, memory, storage drives, networking switches, fans, and power supply inefficiency. A multiplier of 1.3-1.5x the total GPU TDP in a server gives a reasonable estimate of total server power consumption. For example, a server with 8x H100 SXM GPUs at 700W each (5,600W GPU power) will consume approximately 8,000-10,000W total including all other components.
Cooling Capacity Planning
All power consumed by servers becomes heat that must be removed from the facility. Plan cooling capacity at 100% of the electrical load, plus additional capacity for the cooling system's own energy consumption (reflected in the facility's PUE). A facility with PUE of 1.15 requires 15% more total power than the IT load alone. For detailed cooling technology comparisons, see Data Center Cooling Technologies Compared: Air, Liquid, and Immersion.
Network Planning
Training clusters require InfiniBand or high-speed RoCE between every GPU node, with non-blocking fat-tree topologies for maximum performance. Each GPU node typically needs one 200 or 400 Gbps port to the cluster fabric. Inference deployments can typically use standard 25-100 Gbps Ethernet. Plan network cabling, switch capacity, and topology based on the specific GPU deployment model, workload requirements, and growth projections.
Operator Insight: The transition from Hopper to Blackwell GPUs represents the most significant facility infrastructure challenge since the industry moved from CPU to GPU computing. The jump from 700W to 1,000W per GPU, combined with the GB200 NVL72's 120 kW per rack requirement, means that liquid cooling is no longer optional for new deployments. Facilities designed today should be built with liquid cooling infrastructure, high-density power distribution, and sufficient power headroom for next-generation hardware. RAX Data & Energy designs all new facilities with this forward-looking approach, ensuring our clients' infrastructure investments remain viable through multiple GPU generations.
Procurement and Availability
GPU availability remains a significant challenge in 2026. Lead times for the latest Blackwell GPUs can extend to 6-12 months for large orders, and pricing varies significantly based on volume commitments, customer relationships with NVIDIA and its partners, and timing within product cycles.
The secondary market for A100 and H100 GPUs is mature and well-established, providing faster delivery options for organizations building inference clusters or expanding existing training capacity. Prices for used A100 80GB GPUs have stabilized at approximately 30-40% of their original list price, making them attractive for cost-sensitive deployments. However, buyers should carefully verify hardware provenance, remaining warranty status, and the operational history of used GPU hardware.
RAX Data & Energy works with NVIDIA and authorized distribution partners to procure GPUs for our GPU Colocation: Everything You Need to Know in 2026 clients, leveraging volume purchasing relationships to deliver competitive pricing and priority allocation for both current and next-generation GPU hardware. Our infrastructure team assists clients with GPU selection, system configuration, and facility planning to ensure every deployment maximizes performance per dollar invested across the full hardware lifecycle.