For decades, data centers relied on two types of processors: CPUs for general-purpose computing and GPUs for parallel workloads. The Data Processing Unit (DPU) has emerged as the third pillar, purpose-built to handle the infrastructure tasks that neither CPUs nor GPUs were designed to do efficiently. As AI clusters scale to thousands of GPUs and cloud environments demand zero-trust security at line rate, DPUs have moved from niche to essential.
This guide explains what DPUs are, what problems they solve, how the major vendors compare, and when a data center operator should deploy them.
The Infrastructure Tax Problem
Modern data centers run enormous amounts of infrastructure software on their host CPUs: virtual switches (OVS), firewalls, encryption (IPsec/TLS), storage controllers (NVMe-oF), load balancers, telemetry agents, and container networking. Each of these functions consumes CPU cycles that could otherwise run revenue-generating workloads.
In a typical cloud server, 20-30% of CPU capacity is consumed by infrastructure overhead. For a 128-core server costing $15,000-$25,000, that means $3,000-$7,500 worth of compute capacity is lost to plumbing. Multiply that across thousands of servers and the cost becomes significant.
GPUs face a similar problem in AI clusters. When an NVIDIA H100 needs data from remote storage or another GPU across the network, the host CPU must orchestrate that data movement. If the CPU is busy handling networking and storage, GPU utilization drops because the accelerator sits idle waiting for data. This is one reason why GPU utilization in production AI clusters often averages 30-50% rather than the theoretical maximum.
What a DPU Actually Does
A DPU is a system-on-chip (SoC) on a PCIe card that takes over infrastructure functions from the host CPU. It combines three capabilities in a single device:
1. High-Speed Networking
DPUs integrate network interfaces (currently 200-400 Gbps, scaling to 800 Gbps) with hardware accelerators for packet processing. Instead of the host CPU running a software virtual switch at 40-100 Gbps with significant overhead, the DPU processes packets in hardware at full line rate with near-zero CPU involvement.
2. Programmable Compute
Unlike a fixed-function NIC, DPUs include general-purpose Arm cores (8-16 cores is typical) that run a full operating system. This allows operators to deploy custom infrastructure software on the DPU itself: firewalls, monitoring agents, storage controllers, even lightweight Kubernetes nodes. The DPU becomes an infrastructure computer that is separate from the host.
3. Hardware Accelerators
DPUs include dedicated silicon for common infrastructure operations:
- Cryptography: AES-256, SHA-256, IPsec, TLS 1.3 at line rate without CPU involvement
- Compression: Hardware LZ4/Zstandard for storage and network compression
- Regular expressions: Pattern matching for intrusion detection and deep packet inspection
- Data movement: DMA engines for efficient memory-to-memory and PCIe transfers
- Storage: NVMe-oF target and initiator engines for remote storage access
DPU vs SmartNIC vs Traditional NIC
| Capability | Traditional NIC | SmartNIC | DPU |
|---|---|---|---|
| Bandwidth | 25-100 Gbps | 25-200 Gbps | 200-400 Gbps |
| Packet Processing | Hardware (fixed) | Programmable (FPGA/P4) | Programmable + HW accelerators |
| General Compute | None | Limited (embedded cores) | Full OS (8-16 Arm cores) |
| Crypto Offload | None or basic | IPsec offload | Full IPsec + TLS at line rate |
| Storage Offload | None | Basic RDMA | Full NVMe-oF + virtio-blk |
| Isolation Model | SR-IOV only | SR-IOV + basic offload | Full hardware isolation (bare-metal security) |
| Price Range | $200-$800 | $1,000-$3,000 | $2,000-$6,000 |
Major DPU Vendors Compared
NVIDIA BlueField-3
The market leader in DPU adoption. BlueField-3 integrates 16 Arm Cortex-A78 cores, 400 Gbps ConnectX-7 networking, and a comprehensive set of accelerators. It runs NVIDIA DOCA, a software framework that provides APIs for networking, storage, security, and management. BlueField-3 is the standard DPU in NVIDIA's DGX and HGX GPU platforms, where it handles all GPU-to-GPU RDMA networking and storage I/O.
- Strengths: Ecosystem integration (CUDA, NCCL, Magnum IO), mature DOCA SDK, broad cloud provider adoption
- Best for: AI/HPC clusters, NVIDIA GPU environments, multi-tenant cloud
AMD Pensando DSC-200
AMD acquired Pensando in 2022 to build its DPU portfolio. The DSC-200 uses a P4-programmable ASIC pipeline for packet processing combined with Arm cores for management. It is deployed heavily in Oracle Cloud Infrastructure and Microsoft Azure, where it handles virtual networking for millions of VMs.
- Strengths: Extremely efficient packet processing, strong cloud provider references, P4 programmability
- Best for: Large-scale cloud networking, virtual switch offload, telecom
Intel Infrastructure Processing Unit (IPU)
Intel's IPU strategy combines FPGA-based and ASIC-based products. The Mount Evans ASIC IPU (deployed in Google Cloud) and the Oak Springs Canyon FPGA IPU serve different flexibility/performance trade-offs. Intel's approach emphasizes open-source software stacks (IPDK, P4-DPDK) over proprietary SDKs.
- Strengths: FPGA flexibility for custom workloads, open-source ecosystem, Intel CPU integration
- Best for: Custom infrastructure functions, telecom NFV, enterprises with Intel-centric stacks
DPU Use Cases for Data Center Operators
AI Cluster Networking
In GPU clusters connected via InfiniBand or RoCE, DPUs manage the RDMA data path, handle GPU-Direct Storage for loading training datasets, and provide network isolation between tenants. Without DPUs, these functions consume host CPU cores that should be feeding GPUs. In an 8-GPU server, even 2 CPU cores consumed by networking can reduce effective GPU utilization by 3-5%.
Multi-Tenant Colocation Security
For colocation providers hosting multiple customers on shared infrastructure, DPUs enforce hardware-level network isolation, per-tenant firewall policies, and encrypted communication between VMs, all without trusting or consuming the tenant's host CPU. This is the foundation of "bare-metal cloud" offerings where tenants get dedicated hardware with cloud-like networking features.
Storage Disaggregation
DPUs enable NVMe-over-Fabrics (NVMe-oF) at line rate, allowing servers to access remote flash storage as if it were local. This decouples compute from storage, enabling independent scaling and better resource utilization. A GPU cluster can access a centralized all-flash array at 200 Gbps per node without any host CPU overhead for the storage protocol.
Zero-Trust Networking
DPUs encrypt all east-west traffic (server-to-server within the data center) at line rate using hardware IPsec or MACsec. This makes microsegmentation practical at scale. Without hardware offload, encrypting all intra-datacenter traffic would consume 10-20% of every server's CPU capacity.
Telemetry and Observability
DPUs can mirror, sample, and analyze network flows in hardware, feeding rich telemetry to monitoring systems without the packet-capture overhead that typically degrades server performance. This enables per-flow visibility at 400 Gbps speeds.
Power and Cooling Considerations
DPUs consume 50-150W depending on the model and workload. In a high-density server rack, adding DPUs increases total power consumption by 5-10%. However, the CPU cycles freed by offloading infrastructure tasks more than compensate for the additional power. Net server efficiency (useful compute per watt) typically improves by 15-25% when DPUs replace software-based infrastructure processing.
From a colocation power planning perspective, DPU power draw must be included in per-rack power budgets. A rack with 4 servers, each with 2 DPUs, adds 400-1,200W to the total rack power consumption. Cooling requirements follow the same per-watt rules as other PCIe devices: DPUs are air-cooled with standard server fans in current generations.
When to Deploy DPUs
DPUs deliver the strongest ROI in the following scenarios:
- GPU clusters with more than 16 GPUs: Network and storage overhead becomes a significant bottleneck at this scale. DPUs keep GPUs fed with data and maintain high utilization.
- Multi-tenant environments: Anywhere you need hardware-level isolation between customers without performance overhead.
- High-security workloads: Government, financial, and healthcare deployments requiring encryption of all traffic without performance degradation.
- Storage-intensive AI: Training runs that read from large datasets (100TB+) benefit enormously from DPU-accelerated NVMe-oF.
DPUs are less critical for:
- Pure ASIC mining operations: Miners have simple networking requirements and minimal storage I/O.
- Single-tenant bare-metal deployments: When one customer owns the entire server, software-defined infrastructure overhead is lower.
- Small clusters (fewer than 8 nodes): The CPU overhead is manageable at this scale and the DPU investment may not be justified.
The Future: DPUs in Every Server
The trajectory is clear. AWS Nitro (a proprietary DPU) has been in every EC2 instance since 2017. Azure uses AMD Pensando DPUs for its network stack. Google Cloud runs Intel IPUs. As hyperscalers have proven the model, enterprise and colocation data centers are following.
NVIDIA CEO Jensen Huang has described the DPU as "the third pillar of computing" alongside CPUs and GPUs. BlueField-4, expected in 2026-2027, will integrate 800 Gbps networking and next-generation Arm cores, further closing the gap between DPU capabilities and host CPU capabilities.
For data center operators building next-generation infrastructure, DPUs are no longer optional in AI and multi-tenant environments. They are the foundation layer that makes everything else, from advanced cooling to GPU cluster orchestration, work efficiently at scale.
Enterprise AI Infrastructure
Rax Data operates high-density GPU colocation with DPU-ready networking, InfiniBand interconnects, and advanced cooling for AI training and inference workloads.
Request a Quote