Multi-Megawatt Data Center Design: Building for AI at Scale

The explosion in AI model training and inference workloads has created unprecedented demand for multi-megawatt data centers optimized for high-density GPU clusters. A single large language model training run can consume 10-50 megawatts continuously for weeks or months. Inference serving at scale requires similar power levels distributed across thousands of GPUs operating 24/7.

Designing facilities to reliably deliver this level of power and cooling requires rethinking traditional data center architecture. This guide covers the engineering fundamentals of multi-megawatt AI data centers from electrical distribution through cooling systems to network topology and scalability planning.

Power Requirements and Load Characteristics

AI workloads exhibit fundamentally different power characteristics than traditional enterprise computing.

GPU Power Density

Modern AI accelerators consume substantially more power than CPUs. A single NVIDIA H100 GPU draws 700W under full training load. An 8-GPU server consumes 5.6kW for GPUs alone, plus another 1.5-2kW for CPUs, memory, storage, and networking, totaling 7-8kW per 4U chassis.

Standard 42U racks can accommodate 10-11 of these servers, resulting in 70-88kW of IT load per rack before adding networking equipment. This is 3-4x the density of traditional enterprise racks which typically operate at 8-12kW.

Load Duration and Utilization

Unlike enterprise workloads with variable demand and peak-to-average ratios of 2:1 or higher, AI training runs at sustained 90-100% GPU utilization for days or weeks continuously. This eliminates traditional diversity factors that allow oversubscription in enterprise data centers.

Inference workloads show more variation based on user traffic patterns but still maintain higher average utilization than traditional web services. Facilities must be provisioned for continuous full-load operation rather than peak capacity with significant headroom.

Total Facility Power Calculation

For a facility targeting 10MW of AI compute capacity:

• GPU compute load: 10,000kW (primary workload)
• CPU and memory overhead: 3,000-4,000kW (30-40% of GPU power)
• Networking fabric: 300-500kW (2-3% of total)
• Storage infrastructure: 700-1,000kW (5-8% of total)
• Cooling overhead: 3,500-6,000kW (depends on climate and cooling technology)
• UPS conversion losses: 600-900kW (4-6% of IT load at 94-96% efficiency)
• Total facility load: 18,000-23,000kW depending on PUE (1.35-1.6 typical)

This calculation assumes moderate climate and optimized cooling. Hot-climate facilities may require 25-30MW total for the same 10MW compute capacity.

Electrical Infrastructure Architecture

Delivering tens of megawatts reliably requires careful electrical distribution design from utility service through rack-level PDUs.

Utility Service and Redundancy

Multi-megawatt facilities typically require medium-voltage utility service at 15kV or 35kV. This reduces conductor size and voltage drop compared to standard 480V service. Negotiate dual utility feeds from separate substations when possible to provide geographic redundancy.

For a 25MW facility, provision 30-35MW utility service capacity to accommodate future growth and provide headroom for startup inrush currents. Include on-site substation with multiple transformer banks sized for N+1 redundancy.

Generator Backup

AI training workloads are less sensitive to brief interruptions than financial services, but extended outages waste expensive compute time. Most facilities provision generator capacity for full IT load (not including chillers) to maintain operations during utility outages while accepting elevated PUE during generator operation.

For 10MW IT load, provision 12-15MW generator capacity in multiple units (typically 4-6 generators of 2-3MW each) for N+1 redundancy. Diesel fuel storage must support minimum 24 hours full-load operation, with 72-hour capacity preferred.

UPS Systems

Distributed in-row UPS systems work better for AI facilities than centralized UPS rooms. Modular UPS units rated for 250-500kVA deployed in hot-aisle containment provide redundancy at the rack level and reduce single points of failure.

Target 10-15 minute runtime at full load to bridge utility outages until generators start and stabilize. Lithium-ion UPS batteries offer 30-50% smaller footprint and longer life than traditional VRLA batteries, with better performance at elevated temperatures.

Busway vs Cable Distribution

Overhead aluminum busway rated for 600-800A per run provides flexible power distribution within data halls. Tap boxes every 3-4 meters allow quick connection of new racks without cable pulls. This is particularly valuable for AI facilities that reconfigure rack layouts as cluster topology evolves.

Alternative approaches use overhead cable tray with homerun cables to each rack. This reduces upfront costs but makes reconfiguration more labor-intensive.

Cooling Architecture for High Density

Removing 10+ megawatts of waste heat efficiently is the defining challenge of AI data center design.

Air Cooling Limitations

Traditional raised-floor air cooling becomes impractical above 25-30kW per rack density. Airflow requirements exceed what CRAC units can deliver, and hot/cold aisle containment breaks down as rack face velocities become excessive.

Some facilities use high-velocity air cooling with in-row cooling units and rear-door heat exchangers to reach 35-40kW density, but energy consumption remains high due to fan power requirements.

Direct Liquid Cooling

Cold-plate liquid cooling captures 60-80% of server heat directly at the GPU and CPU using water or glycol coolant loops. This dramatically reduces airflow requirements and enables 60-80kW rack densities.

Coolant distribution uses overhead or under-floor piping with quick-disconnect couplings at each rack. Coolant-to-water heat exchangers transfer heat to facility chilled water loops. Residual air cooling handles memory, storage, and power supply heat.

Capital costs are 30-50% higher than air cooling, but operating costs are 20-40% lower due to reduced fan power and more efficient heat rejection. For facilities above 5MW, liquid cooling typically achieves better total cost of ownership.

Immersion Cooling

Single-phase immersion cooling submerges entire servers in dielectric fluid within sealed tanks. This supports 100+kW per rack equivalent density and nearly eliminates fan power consumption.

Challenges include higher upfront costs, specialized server designs, and operational complexity of fluid handling and filtration. Best suited for greenfield deployments where architects can design facilities around immersion from the ground up.

Hybrid Approaches

Most large AI facilities use a hybrid cooling strategy: air cooling for ancillary equipment and moderate-density racks, direct liquid cooling for primary GPU clusters, and aggressive adiabatic cooling or evaporative systems for heat rejection.

This approach balances capital efficiency, operational simplicity, and energy performance. Target facility PUE of 1.15-1.25 is achievable in moderate climates, compared to 1.5-1.7 for air-only designs.

Network Architecture and Bandwidth

AI workloads require dramatically higher east-west network bandwidth than traditional enterprise applications.

GPU Interconnect Topology

Large training clusters use high-radix switches with 400GbE or 800GbE links in fat-tree or dragonfly topologies. A 1000-GPU cluster might use 8x800GbE uplinks per rack connecting to spine switches, providing 6.4Tbps aggregated bandwidth per rack.

Newer deployments increasingly use InfiniBand NDR (400Gbps per port) or Ultra Ethernet for lower latency and higher bandwidth. Network fabric costs can represent 15-25% of total cluster capital expenditure.

Storage Network

Training datasets for large models range from tens to hundreds of terabytes. Separate high-bandwidth storage networks using NVMe over Fabrics or parallel file systems like Lustre provide I/O bandwidth without contending with compute traffic.

Budget 100-200Gbps storage network bandwidth per 100 GPUs for typical training workloads. Inference workloads have lower storage I/O requirements but higher network throughput for serving client requests.

Scalability and Future-Proofing

AI data centers must accommodate rapid technology evolution and changing workload requirements.

Modular Build-Out

Design facilities in phases with clear expansion paths. A typical approach provisions shell space for 40-50MW capacity but initially fits out only 10-15MW. Electrical and cooling infrastructure should have clear upgrade paths without major demolition.

Pre-install medium-voltage conduit and chilled water piping for future expansions even if not immediately populated with active equipment.

Power Density Headroom

GPU power consumption increases with each generation. NVIDIA H100 consumes 700W; H200 may consume 800-900W; future generations could exceed 1000W per GPU. Design electrical and cooling infrastructure with 30-50% headroom beyond current requirements.

Facilities designed for 60kW racks should provision electrical and cooling capacity for 80-90kW to accommodate next-generation hardware without infrastructure replacement.

Technology Refresh Cycles

AI accelerators depreciate faster than traditional servers due to rapid performance improvements. Budget for 3-4 year hardware refresh cycles compared to 5-7 years for enterprise equipment. Infrastructure designed for 15-20 year lifespans will host multiple generations of compute equipment.

Site Selection and Geographic Considerations

Location fundamentally impacts operational costs and performance for multi-megawatt facilities.

Power Availability and Cost

AI data centers require reliable access to tens of megawatts of electricity. Typical industrial rates range from $0.03-0.12/kWh depending on region, power source, and contract structure. A 10MW facility operating at 90% utilization consumes 79 million kWh annually, representing $2.4-9.5 million in electricity costs alone.

Regions with abundant hydroelectric, nuclear, or renewable power offer both lower costs and better sustainability profiles. The Pacific Northwest US, Quebec, Nordic countries, and parts of the Middle East offer competitive power economics.

Climate and Free Cooling

Temperate climates enable air-side economizers and evaporative cooling, reducing or eliminating mechanical chiller load for much of the year. Regions with average temperatures below 15°C can achieve PUE below 1.15 with optimized cooling design.

Hot-climate facilities require year-round mechanical cooling but may offer advantages in power availability, proximity to markets, or connectivity. Advanced cooling technologies like adiabatic cooling or hybrid approaches help mitigate climate penalties.

Network Connectivity

Large AI facilities require 100+ Gbps internet connectivity for model distribution, dataset access, and inference serving. Sites should have access to carrier-neutral internet exchanges and multiple fiber providers for redundancy.

Latency matters for inference workloads serving end users but is less critical for batch training. Training facilities can be located in remote regions with low-cost power if adequate connectivity exists for initial dataset loading and model distribution.

Regulatory and Environmental Considerations

Multi-megawatt facilities face increasing scrutiny on environmental and energy usage impacts.

Power Grid Integration

Utility interconnection for 20-50MW loads requires multi-year planning and substantial utility upgrade costs. Some regions impose demand response requirements where facilities must curtail load during grid stress periods. Build contractual provisions for load management into AI workload scheduling systems.

Water Usage

Evaporative cooling systems consume significant water, which is problematic in arid regions. A 20MW facility using evaporative cooling may consume 100-200 million liters of water annually. Some jurisdictions restrict data center water usage, requiring air-cooled or closed-loop liquid cooling approaches.

Carbon Footprint

Pressure to reduce carbon emissions drives preference for renewable power purchase agreements and locations with low-carbon grid mixes. Some hyperscalers now require 100% carbon-free energy matching on an hourly basis, which eliminates many potential sites.

For organizations serious about sustainability, prioritize locations with abundant renewable or nuclear baseload power even if short-term costs are slightly higher.

Cost Modeling and Economic Analysis

Understanding full lifecycle costs helps optimize design tradeoffs.

Capital Expenditure Breakdown

For a 20MW AI data center (shell and core, excluding IT equipment):

Component	Cost per MW	20MW Total
Building shell and structure	$2-4M	$40-80M
Electrical infrastructure	$3-5M	$60-100M
Cooling systems	$2-4M	$40-80M
Generator and UPS	$1.5-2.5M	$30-50M
Fire suppression, security, controls	$0.8-1.5M	$16-30M
Total facility CapEx	$10-17M/MW	$200-340M

IT equipment costs (servers, GPUs, networking) typically equal or exceed facility infrastructure costs. A 10MW GPU cluster might require $300-500M in compute hardware depending on accelerator choice.

Operating Costs

Annual operating expenses for a 20MW facility include:

• Electricity: $6-24M annually (at $0.04-0.12/kWh, 90% utilization, PUE 1.3)
• Staffing: $1.5-3M (15-25 operations and maintenance personnel)
• Maintenance and spare parts: $2-4M (1-2% of facility CapEx annually)
• Connectivity and bandwidth: $0.5-2M depending on requirements

Electricity typically represents 60-80% of total operating costs, making power efficiency and favorable energy contracts critical to economics.

Case Study: Optimized 10MW AI Training Facility

A recent deployment in northern Europe illustrates design principles in practice:

• IT capacity: 10MW across 200 liquid-cooled racks at 50kW each
• Total facility power: 13.5MW (PUE 1.35)
• Cooling: Direct-to-chip liquid cooling for GPUs, air cooling for ancillary equipment
• Heat rejection: Adiabatic cooling with free cooling 75% of hours annually
• Electrical: 2N redundancy at utility and transformer level, N+1 at distribution
• Backup power: 15MW generator capacity, 12 minutes UPS runtime
• Network: 400GbE leaf-spine fabric with 25.6Tbps bisection bandwidth

This facility achieved facility CapEx of $12M per MW (excluding IT equipment) and operates at $0.055/kWh effective power cost including renewables premium.

Build vs Colocation Decision

Organizations must decide whether to build greenfield facilities or leverage existing colocation providers.

Greenfield builds offer maximum customization and long-term cost efficiency for organizations with sustained multi-megawatt requirements. Capital requirements are substantial but economics improve at scale.

Colocation works well for organizations testing AI workloads, needing fast deployment, or operating at smaller scale. Providers like Rax Data & Energy offer turnkey AI infrastructure with flexible capacity from single racks to multi-megawatt deployments.

Hybrid approaches are increasingly common: colocation for initial deployment and testing, followed by greenfield construction once workload characteristics and long-term requirements are validated.