The Cost of GPU Downtime in AI Clusters
GPU clusters running AI training workloads represent some of the highest-value infrastructure in any data center. A single node equipped with eight NVIDIA H100 or H200 GPUs costs $200,000 to $400,000, and a large training cluster may contain hundreds of such nodes. When a GPU fails or a node drops out of a distributed training job, the impact extends far beyond that single device. Collective communication operations like AllReduce require all participating GPUs to synchronize, so a single unhealthy GPU can stall an entire training run across hundreds of nodes, consuming expensive compute time while producing no useful work.
The economics are stark. A 1,000-GPU training cluster running at $2 per GPU-hour costs $2,000 per hour. If a single faulty GPU causes the training job to stall for four hours while the operations team diagnoses and resolves the issue, that is $8,000 in wasted compute, not including the opportunity cost of delayed model delivery. At scale, unplanned GPU downtime can cost hundreds of thousands of dollars per month. This guide provides systematic diagnostic workflows that reduce mean time to resolution and help GPU colocation operators maintain the reliability that AI workloads demand.
GPU Hardware Failure Modes
Understanding the specific ways GPUs fail is the foundation of effective troubleshooting. Modern data center GPUs are complex systems-on-chip with multiple potential failure points.
HBM Memory Errors
High Bandwidth Memory is the most failure-prone component in modern GPU accelerators. HBM3 and HBM3e stacks consist of multiple DRAM dies connected through silicon vias, and any defect in the stack can cause bit errors. GPUs use ECC (Error Correcting Code) to handle memory errors. Single-bit correctable errors (SBEs) are detected and corrected transparently by the hardware, but they indicate underlying memory degradation. Double-bit uncorrectable errors (DBEs) cannot be corrected and cause immediate workload failure, typically manifesting as a CUDA error, kernel panic, or Xid error 48.
The diagnostic workflow for memory errors begins with nvidia-smi. Run nvidia-smi -q -d ECC to display current and aggregate ECC error counts for each GPU. Non-zero volatile correctable error counts that reset at reboot are normal at low rates. Non-zero aggregate uncorrectable errors indicate permanent hardware damage requiring GPU replacement. A rapidly increasing correctable error rate, more than 10 per hour, signals that the memory is degrading and should be proactively replaced before uncorrectable errors begin.
Proactive replacement saves money. Replacing a GPU with an accelerating correctable error rate during a planned maintenance window costs a fraction of the unplanned downtime caused by an uncorrectable error that crashes a multi-day training job at 80 percent completion.
GPU Core and Streaming Multiprocessor Failures
Silicon defects or electromigration damage in the GPU die can cause individual streaming multiprocessors (SMs) to fail. NVIDIA GPUs can retire faulty SMs through a process called row remapping, which disables the defective unit and redistributes workload to remaining SMs. While this maintains functionality, it reduces the GPU's total compute capacity. Check for retired SMs with nvidia-smi -q -d RETIRED_PAGES. A GPU with multiple retired pages or SMs is producing reduced throughput and should be scheduled for replacement, especially in clusters where all GPUs must deliver consistent performance for balanced distributed training.
Power Delivery Issues
GPU power delivery failures manifest as sudden shutdowns, clock throttling, or Xid errors in the 60-79 range. Modern data center GPUs consume 350 to 1000 watts each, and the power delivery infrastructure must provide stable voltage across the full load range. Common causes include degraded power supply units within the server, loose or damaged power cables, and PDU circuit overloading where total rack load exceeds the rated capacity. Use the server's BMC (Baseboard Management Controller) to check power supply health, input voltage, and output rail stability. Power-related GPU failures often correlate with other rack or circuit-level events, so check whether adjacent servers experienced issues at the same time.
Interconnect Troubleshooting
GPU clusters depend on high-speed interconnects for both intra-node communication between GPUs and inter-node communication across the network fabric. Interconnect failures are among the most disruptive issues because they affect the entire cluster, not just individual GPUs.
NVLink and NVSwitch Diagnostics
NVLink provides the high-bandwidth, low-latency fabric that connects GPUs within a node. Each NVLink connection consists of multiple lanes, and the loss of even one lane reduces the available bandwidth between GPU pairs. Run nvidia-smi nvlink -s to display the status of all NVLink connections for each GPU. Look for links reporting reduced bandwidth or elevated error counters. The command nvidia-smi nvlink -e breaks down errors by type: CRC errors indicate physical layer problems such as damaged traces or loose connectors, while replay errors suggest protocol-level issues that may respond to firmware updates.
For SXM-based GPU modules used in DGX and HGX systems, NVSwitch provides the all-to-all GPU interconnect. NVSwitch failures are particularly impactful because each switch connects multiple GPU pairs. Check NVSwitch health through the BMC interface or DCGM diagnostics. A failed NVSwitch typically causes multiple GPU pairs to lose high-bandwidth connectivity simultaneously, which is the distinguishing symptom compared to individual NVLink cable failures in PCIe-based configurations.
InfiniBand Fabric Diagnostics
InfiniBand networks connecting nodes in GPU clusters are sensitive to physical layer issues that manifest as performance degradation before causing complete link failures. The ibstat command shows port state, link speed, and physical link status. A port showing "LinkUp" but at reduced speed, such as HDR100 instead of HDR200, indicates a degraded link that is still operational but at half bandwidth. The perfquery tool reports error counters for each port including symbol errors, link downed events, and receive errors.
For systematic fabric diagnostics, ibdiagnet scans the entire InfiniBand subnet and reports topology errors, duplicate GUIDs, failed links, and routing inconsistencies. This tool is essential after any physical changes to the network fabric such as adding nodes, replacing cables, or swapping switches. Cable integrity testing with mlxlink provides per-lane eye diagram analysis that can identify marginal cables before they cause errors under load.
PCIe Link Degradation
The PCIe connection between GPUs and the host CPU handles data transfer, control signaling, and DMA operations. PCIe link degradation manifests as reduced bandwidth that bottlenecks data loading, model parameter transfers, and collective communication through the host. Run lspci -vvv and check the "LnkSta" field for each GPU. A GPU rated for Gen5 x16 running at Gen4 x8 has lost 75 percent of its host bandwidth. Common causes include thermal expansion of the PCIe connector, damaged pins in the slot, and motherboard trace integrity issues that worsen over time, particularly in environments with significant vibration from cooling infrastructure.
Thermal Troubleshooting
Thermal management is the most common operational challenge in GPU clusters. Modern GPUs generate enormous heat density, and any failure in the cooling infrastructure quickly leads to throttling that degrades the entire cluster's performance.
Air-Cooled GPU Thermal Issues
In air-cooled deployments, GPU junction temperatures above 83 to 85 degrees Celsius for NVIDIA A100 and H100 GPUs trigger clock frequency reduction. Monitor temperatures continuously with nvidia-smi dmon -s t which reports junction temperature, memory temperature, and fan speed at one-second intervals. Systematic thermal troubleshooting follows this sequence: verify that the CRAC or CRAH supply air temperature is within specification, typically 18 to 27 degrees Celsius per ASHRAE A1 guidelines. Check that hot aisle containment is intact with no gaps allowing exhaust recirculation. Inspect server-level fan operation and ensure all fans are running at expected speeds. Verify that internal cable management is not obstructing airflow paths within the server chassis. Check adjacent rack equipment for recent changes that may have altered airflow patterns.
Liquid-Cooled GPU Thermal Issues
In direct liquid cooling deployments, thermal issues stem from the coolant delivery system rather than air movement. Check CDU inlet and outlet temperatures, flow rates, and pump status. Reduced flow through individual cold plates causes localized GPU overheating even when the CDU reports normal aggregate performance. Trapped air bubbles in the cooling loop create hot spots as they prevent coolant contact with the cold plate surface. If one GPU in a server runs significantly hotter than its peers, the issue is likely specific to that GPU's cold plate connection, including loose fittings, kinked tubing, or a partially blocked cold plate.
For liquid cooling maintenance, coolant chemistry must be monitored. Degraded coolant with incorrect pH or conductivity levels accelerates corrosion of cold plate surfaces, reducing heat transfer efficiency over months. Biological growth in untreated coolant loops can clog narrow passages in GPU cold plates, creating progressive thermal degradation that is difficult to diagnose without flow rate measurements at the individual cold plate level.
Software and Driver Diagnostics
Not all GPU cluster problems originate in hardware. Driver issues, firmware bugs, and software configuration errors produce symptoms that mimic hardware failures.
NVIDIA Driver and CUDA Troubleshooting
Xid errors reported in the system log (dmesg | grep -i xid) are the primary diagnostic signal for GPU issues. Each Xid error code maps to a specific category of problem. Xid 13 indicates a graphics engine exception, often caused by illegal memory access in CUDA kernels. Xid 31 indicates a GPU memory page fault. Xid 43 indicates a GPU stopped responding, which may be hardware or driver related. Xid 48 indicates a double-bit ECC error in VRAM, a definitive hardware fault. Xid 74 indicates NVLink error. Xid 79 indicates a GPU has fallen off the bus, suggesting a PCIe or power issue.
Driver version mismatches across nodes in a cluster cause subtle issues. NCCL (NVIDIA Collective Communications Library) versions must be consistent across all nodes participating in distributed training, and the driver version must be compatible with the CUDA toolkit version used by the training framework. Use nvidia-smi to verify driver versions across all nodes and compare against the CUDA compatibility matrix. When upgrading drivers, update all nodes simultaneously to prevent version skew.
DCGM Health Monitoring
NVIDIA Data Center GPU Manager (DCGM) provides fleet-wide health monitoring that goes beyond what nvidia-smi offers for individual GPUs. DCGM's built-in diagnostic levels run increasingly thorough hardware validation. Level 1 (Quick) takes seconds and checks basic GPU accessibility and driver communication. Level 2 (Medium) takes minutes and runs memory stress tests, PCIe bandwidth verification, and NVLink integrity checks. Level 3 (Extended) takes 10 to 20 minutes per GPU and performs exhaustive memory testing, compute stress testing, and thermal characterization under sustained load.
For production clusters, DCGM integrates with Prometheus and Grafana to provide continuous monitoring dashboards. Configure DCGM to export GPU temperature, power draw, ECC error counts, NVLink error counters, and PCIe bandwidth metrics. Set alerting thresholds for each metric: temperature above 80 degrees Celsius, power draw deviating more than 10 percent from expected, any uncorrectable ECC error, and NVLink CRC errors exceeding 100 per hour. These alerts enable the NOC to identify degrading GPUs before they cause workload failures.
Job Scheduler and Framework Issues
Slurm and Kubernetes GPU scheduling misconfigurations cause symptoms that appear to be hardware problems. A training job that consistently fails on the same node may be hitting a hardware issue, but it may also be encountering a node-specific software configuration difference such as a missing library, incorrect environment variable, or stale driver module. To distinguish, run DCGM Level 2 diagnostics on the suspect node. If the diagnostics pass, the problem is software. Run the failing job on a known-good node to confirm.
Multi-Instance GPU (MIG) configurations add another layer of complexity. If MIG is enabled on some GPUs but not others, or if MIG partition sizes are inconsistent across nodes, the job scheduler may assign work to GPUs with insufficient compute or memory resources, causing out-of-memory errors or performance degradation that looks like a hardware problem.
Systematic Diagnostic Workflow
When a GPU cluster issue is reported, following a systematic workflow prevents wasted time on dead ends and ensures that the actual root cause is identified.
Step 1: Triage and Scope
Determine the scope of impact before diving into diagnostics. Is the issue affecting a single GPU, a single node, multiple nodes on the same switch, or the entire cluster? Single-GPU issues point toward hardware faults on that specific device. Single-node issues suggest server-level problems including power, cooling, or BMC. Multi-node issues on the same switch indicate network fabric problems. Cluster-wide issues suggest environmental conditions (cooling failure, power event) or software problems (driver update, scheduler misconfiguration).
Step 2: Collect Baseline Data
Before making changes, collect comprehensive diagnostic data from the affected systems. Run nvidia-bug-report.sh on each affected node, which captures nvidia-smi output, driver logs, Xid errors, NVLink status, ECC counters, temperature history, and PCIe link status in a single archive. Collect InfiniBand diagnostics with ibdiagnet if network issues are suspected. Record BMC sensor readings for power supply status, fan speeds, and inlet temperatures. This baseline prevents situations where diagnostics themselves clear volatile error counters that would have identified the root cause.
Step 3: Check Environmental Conditions
Verify that environmental monitoring shows normal conditions. Check inlet air temperature, humidity, and cooling system status. A facility-level cooling reduction that raises inlet temperatures by just 3 to 5 degrees Celsius can push a fully loaded GPU rack from normal to throttling temperatures. Check for recent changes to adjacent racks that may have altered airflow. Correlate the timing of the reported issue with any facility events such as cooling maintenance, generator tests, or utility power fluctuations.
Step 4: Hardware Diagnostics
Run DCGM Level 2 diagnostics on the affected GPU or node. This validates memory integrity, PCIe bandwidth, NVLink connectivity, and basic compute functionality in approximately 5 minutes. If Level 2 passes, the hardware is likely functional and the issue may be software-related or environmental. If Level 2 fails, note the specific failure and run Level 3 for detailed characterization. Compare results against the same diagnostics run on a known-good node of the same configuration to establish a performance baseline.
Step 5: Network Diagnostics
For distributed training jobs, network issues are the most common cause of hangs and stalls. Run InfiniBand loopback tests on the affected node's HCA ports. Test point-to-point bandwidth between the affected node and its peers using ib_write_bw and ib_read_bw. Compare achieved bandwidth against expected values for the link speed. Run NCCL all-reduce benchmarks on the affected nodes to verify that collective communication performance matches cluster specifications. A significant bandwidth reduction, more than 10 percent below expected, indicates a fabric issue that requires cable, switch, or HCA investigation.
Step 6: Isolate and Replace
Once the faulty component is identified, isolate it from the cluster to prevent further impact. For a faulty GPU in a multi-GPU node, drain the node from the job scheduler, reseat the GPU if the failure mode suggests a contact issue, and rerun diagnostics. If the GPU fails diagnostics after reseating, document the failure with nvidia-bug-report output and initiate the RMA process with the manufacturer. For lifecycle management, maintain a spare pool of GPUs equal to 2 to 5 percent of the total fleet to minimize downtime during replacements.
Common Failure Scenarios and Resolution
| Symptom | Likely Cause | Diagnostic Command | Resolution |
|---|---|---|---|
| Training job crashes with CUDA error | Uncorrectable ECC error | nvidia-smi -q -d ECC | Replace GPU, verify remaining GPUs in node |
| Gradual throughput degradation | Thermal throttling | nvidia-smi dmon -s t | Check cooling, airflow, fan health |
| AllReduce hangs across cluster | InfiniBand link failure | ibdiagnet, ibstat | Replace cable, check switch port |
| Single GPU shows reduced bandwidth | NVLink lane degradation | nvidia-smi nvlink -s | Reseat GPU, update firmware, replace if persistent |
| GPU fell off the bus (Xid 79) | PCIe or power issue | lspci -vvv, BMC logs | Reseat GPU, check PSU, verify PCIe slot |
| Inconsistent performance across nodes | Driver version mismatch | nvidia-smi across fleet | Standardize driver and CUDA versions |
| Out-of-memory errors on sufficient hardware | MIG misconfiguration | nvidia-smi mig -lgi | Verify MIG profiles match job requirements |
| Intermittent CRC errors on NVLink | Thermal expansion of connector | nvidia-smi nvlink -e | Reseat NVLink bridge, check rack temperature |
Preventive Maintenance and Monitoring
The most effective troubleshooting strategy is preventing failures from occurring in the first place through proactive monitoring and scheduled maintenance.
Continuous Monitoring Strategy
Deploy DCGM across the entire GPU fleet with metrics exported to Prometheus and visualized in Grafana dashboards. Configure automated alerting for temperature thresholds, ECC error rates, NVLink error counters, power anomalies, and PCIe bandwidth degradation. Establish weekly automated health checks that run DCGM Level 2 diagnostics on a rolling subset of nodes, covering the entire fleet over a two-week cycle. Store historical metrics for trend analysis: a GPU with steadily increasing correctable ECC errors over weeks is heading toward failure even though it is currently operational.
Scheduled Maintenance Windows
Allocate regular maintenance windows for firmware updates, driver upgrades, coolant system servicing, and physical inspections. Liquid cooling systems require quarterly coolant chemistry testing, annual filter replacement, and biannual pump inspection. Air-cooled systems need quarterly filter cleaning and annual thermal paste replacement for GPUs showing elevated temperatures. Coordinate maintenance with training schedules to minimize impact on active jobs.
Spare Parts and RMA Management
Maintain an on-site spare inventory that includes GPUs, NVLink bridges, InfiniBand HCAs, cables, power supplies, and fans. For a 1,000-GPU cluster, a minimum of 20 to 50 spare GPUs enables rapid replacement without waiting for RMA processing. Track all hardware replacements in a DCIM system that records serial numbers, failure modes, installation dates, and warranty status. This data drives lifecycle planning decisions about when to refresh GPU generations based on actual failure rates and maintenance costs versus the performance gain from newer hardware.
Frequently Asked Questions
What are the most common GPU failure modes in AI data center clusters?
The most common failure modes include HBM memory errors (correctable and uncorrectable ECC), thermal throttling from inadequate cooling, NVLink and NVSwitch interconnect failures, power delivery issues, PCIe link degradation, and driver or firmware problems. Memory errors and thermal issues account for the majority of unplanned GPU downtime in production clusters.
How do you diagnose NVLink errors in a multi-GPU training cluster?
Start with nvidia-smi nvlink -s to display link status and error counters for each GPU. Non-zero CRC errors indicate physical layer problems. Use nvidia-smi nvlink -e for detailed error breakdowns. DCGM provides continuous NVLink health monitoring with bandwidth tests between GPU pairs. Common causes include thermal expansion of bridge connectors, PCB trace damage from vibration, and firmware bugs.
What causes GPU thermal throttling in colocation environments?
In air-cooled deployments: insufficient cooling capacity, hot aisle containment failures, blocked airflow from poor cable management, failed fans, and ambient temperature increases. In liquid-cooled deployments: reduced CDU flow rates, elevated inlet coolant temperatures, clogged cold plates, and air bubbles in cooling loops. Junction temperatures above 83 to 85 degrees Celsius trigger clock frequency reduction.
How do you use nvidia-smi and DCGM for GPU cluster health monitoring?
nvidia-smi provides immediate per-GPU status including temperature, power, clocks, memory utilization, and ECC errors. DCGM extends this with fleet-wide monitoring, policy-based alerting, Prometheus integration, and built-in diagnostic levels (Quick, Medium, Extended) that validate hardware integrity. Together they provide both real-time point diagnostics and continuous fleet monitoring.
When should a GPU be replaced versus repaired in a data center environment?
Replace when the GPU shows uncorrectable ECC errors, persistent hardware Xid errors (60-79 range), NVLink lanes that stay degraded after reseating and firmware updates, or unrecoverable thermal problems. Repair by reseating the GPU, replacing NVLink bridges, updating firmware, or adjusting power delivery for issues traced to seating, cabling, or configuration. Proactively replace GPUs with accelerating correctable error rates before they progress to uncorrectable failures.