NVMe-oF Storage Architecture for AI Training:
Checkpointing, Data Pipelines, and Performance

The Storage Problem in AI Training

Modern AI training clusters face a fundamental storage challenge: GPU compute performance doubles roughly every 18 months, but storage I/O performance improvements lag behind. A cluster of 256 NVIDIA GB200 GPUs can consume training data at aggregate rates exceeding 200 GB/s while simultaneously needing to write multi-hundred-gigabyte checkpoints without stalling the training run.

Traditional storage architectures using NFS or iSCSI introduce latency overheads of 200 to 500 microseconds per I/O operation. At scale, this means a 280 GB checkpoint write (typical for a 70B parameter model with optimizer states) takes 2 to 5 minutes rather than the 10 to 30 seconds achievable with NVMe-oF. During those minutes, GPUs sit idle or operate in a degraded mode, wasting compute capacity that costs thousands of dollars per hour in a large cluster.

NVMe-oF Fundamentals

NVMe-over-Fabrics extends the NVMe command set across a network transport, allowing a host (GPU server) to issue NVMe commands to a remote storage target as if it were a locally attached drive. The protocol preserves NVMe's low-overhead command submission and completion queue architecture, adding only the minimum framing needed for network transport.

Transport Options

NVMe-oF supports three transport types, each with different performance and deployment characteristics:

Transport	Typical Latency	Throughput per Port	Network Requirements
NVMe-oF over RDMA (RoCEv2)	10 to 30 microseconds	100 to 400 Gbps	Lossless Ethernet (PFC/ECN)
NVMe-oF over TCP	40 to 80 microseconds	100 to 200 Gbps	Standard Ethernet
NVMe-oF over Fibre Channel	20 to 50 microseconds	32 to 64 Gbps	FC SAN infrastructure

For AI training environments, RoCEv2 (RDMA over Converged Ethernet v2) is the predominant choice because GPU clusters already deploy high-performance Ethernet for inter-GPU communication and the switches support the lossless configuration required. NVMe-oF over TCP is gaining adoption for environments where lossless Ethernet configuration is impractical or where storage traffic shares switches with other traffic types.

Why Not Just Use Local NVMe?

Each GPU server typically has 2 to 8 local NVMe drives providing 4 to 60 TB of raw capacity. While local NVMe offers the lowest possible latency (5 to 15 microseconds), it creates several problems at scale:

• Capacity limitations: A 280 GB checkpoint every 1,000 training steps, retaining 5 checkpoints, requires 1.4 TB just for checkpoints. Adding training data caching, model artifacts, and OS overhead quickly exhausts local capacity.
• Data locality: Training data must be distributed across all nodes. Local-only storage requires either data duplication (wasteful) or complex data placement orchestration.
• Recovery complexity: If a node fails, its local checkpoints are inaccessible. Shared storage allows any replacement node to resume from the last checkpoint immediately.
• Utilization inefficiency: Some nodes may run out of local storage while others have excess capacity. Shared storage pools capacity centrally.

The optimal architecture uses local NVMe as a fast cache tier and NVMe-oF for persistent checkpoint storage and bulk data serving.

Checkpoint Storage Architecture

Model checkpointing is the most latency-sensitive storage operation in AI training. A checkpoint captures the complete model state (parameters, optimizer states, learning rate schedules, random number generator states) so training can resume from that point if a failure occurs.

Checkpoint Write Patterns

Checkpoint writes have distinctive I/O characteristics that inform storage design:

• Bursty writes: All GPU nodes write their shard of the checkpoint simultaneously, creating a massive burst of sequential write I/O.
• Large sequential I/O: Individual write operations are typically 1 to 16 MB blocks, heavily sequential.
• Coordinated across nodes: All nodes must complete their writes before training resumes. The slowest node determines total checkpoint time.
• Predictable frequency: Checkpoints occur at configured intervals (every N training steps), allowing the storage system to anticipate write bursts.

Sizing Checkpoint Storage

Model Parameters	Single Checkpoint Size (FP16 + Optimizer)	5 Retained Checkpoints	Recommended Capacity
7B	28 to 56 GB	140 to 280 GB	500 GB
70B	140 to 280 GB	700 GB to 1.4 TB	2 to 3 TB
175B	350 to 700 GB	1.75 to 3.5 TB	5 to 7 TB
405B+	810 GB to 1.6 TB	4 to 8 TB	12 to 15 TB

These figures represent the high-performance tier. Milestone checkpoints (kept for longer retention) should be tiered to lower-cost object storage or parallel file system bulk storage after initial write completes.

Checkpoint Storage Design Pattern

The recommended architecture for checkpoint storage uses a dedicated NVMe-oF storage pool:

1. Dedicated checkpoint namespace: Create a separate NVMe namespace (or LUN) for checkpoint data, sized per the table above plus 30 percent headroom for write amplification and garbage collection.
2. Write-optimized configuration: Configure the storage array for sequential write optimization: large stripe sizes (256 KB to 1 MB), minimal RAID overhead (RAID-Z2 or erasure coding with wide stripes), and write-back caching enabled.
3. Multi-path access: Each GPU server connects to the checkpoint storage via multiple NVMe-oF paths (typically 2 to 4) for aggregate bandwidth and path failover.
4. Asynchronous tiering: After a new checkpoint completes, the previous checkpoint is asynchronously copied to lower-cost storage, and the oldest checkpoint in the high-performance tier is deleted. This keeps the fast tier sized for active checkpoints only.

Training Data Pipeline Architecture

The second major storage workload in AI training is serving training data to GPU nodes. Unlike checkpointing (burst writes), training data serving is a sustained read workload that must deliver data at a rate matching the cluster's consumption speed.

Data Ingestion Requirements

Training data throughput requirements depend on the model architecture and training batch size:

• Language models (LLMs): Tokenized text data is compact. A 256-GPU training run typically requires 5 to 20 GB/s aggregate read throughput for data loading.
• Vision models: Image and video datasets are much larger. A 256-GPU training run on high-resolution images may require 50 to 100 GB/s aggregate read throughput.
• Multimodal models: Combined text, image, and video data can push aggregate requirements to 100 to 200+ GB/s for large clusters.

The pipeline architecture must ensure that data loading never becomes the bottleneck. When GPUs wait for data, utilization drops and cost efficiency deteriorates.

Multi-Tier Data Pipeline

An effective training data pipeline uses multiple storage tiers:

Tier 1 -- GPU memory and local NVMe (cache): Each GPU server caches the next several batches of preprocessed training data in GPU memory (via data loader prefetch) and local NVMe. This tier absorbs the microsecond-level access pattern of the training loop.

Tier 2 -- NVMe-oF shared storage (active dataset): The preprocessed training dataset resides on shared NVMe-oF storage, accessible to all GPU nodes. Data loaders read sequentially from this tier, with prefetching ensuring Tier 1 stays populated. NVMe-oF's low latency (10 to 40 microseconds) keeps prefetch operations fast enough that they complete well before the GPU finishes processing the current batch.

Tier 3 -- Object storage or parallel file system (raw data lake): Raw, unprocessed datasets and previous versions are stored on cost-optimized storage. Preprocessing pipelines read from Tier 3, transform data, and write to Tier 2 before or during training.

Data Loader Optimization

The data loader software is as important as the storage hardware. Key optimizations include:

• Multi-worker prefetching: PyTorch DataLoader with 4 to 8 worker processes per GPU, each performing asynchronous I/O to keep the prefetch queue full.
• Memory-mapped I/O: For datasets that fit in the cluster's aggregate memory, memory-mapped files allow the operating system to manage caching efficiently across NVMe-oF.
• Streaming datasets: For datasets larger than memory, streaming data loaders (such as those in the WebDataset format) read data sequentially from NVMe-oF storage, minimizing random I/O.
• Preprocessing offload: Move data augmentation and preprocessing to CPU cores or dedicated preprocessing nodes rather than GPU nodes, keeping GPU compute available for training.

NVMe-oF Storage Array Selection

Selecting the right storage platform for AI training NVMe-oF involves evaluating several factors.

Performance Requirements

Metric	Checkpoint Workload	Data Pipeline Workload
I/O pattern	Burst sequential write	Sustained sequential read
Block size	1 to 16 MB	256 KB to 4 MB
Latency target	Less than 100 microseconds	Less than 200 microseconds
Throughput target (per client)	10 to 50 GB/s	2 to 10 GB/s
Aggregate throughput	100 to 500 GB/s	50 to 200 GB/s
Concurrency	All nodes simultaneously	All nodes continuously

Architecture Approaches

Three main storage architectures serve AI training NVMe-oF workloads:

All-flash NVMe arrays: Purpose-built storage appliances (VAST Data, Pure Storage FlashBlade, NetApp AFF) that present NVMe namespaces over the fabric. These offer the highest density, best enterprise features (snapshots, replication, encryption), and simplest management but carry the highest per-TB cost.

Software-defined NVMe-oF targets: Standard servers with NVMe drives running NVMe-oF target software (SPDK, Linux kernel NVMe target, or commercial solutions like WekaFS). Lower cost per TB but require more operational expertise and lack some enterprise features.

Parallel file system with NVMe-oF backend: Distributed file systems like Lustre, GPFS/Spectrum Scale, or WekaFS with NVMe-oF as the storage transport between file system servers and the NVMe drive pool. This provides the familiar POSIX namespace and multi-client access with NVMe-oF performance on the backend.

Network Design for NVMe-oF

The network connecting GPU servers to NVMe-oF storage targets is a critical component that can become a bottleneck if undersized.

Bandwidth Planning

Size the storage network for the peak checkpoint write scenario, which is the most demanding:

• Per-node bandwidth: Each GPU server needs sufficient storage network bandwidth to write its checkpoint shard within the target checkpoint time. For a 256-node cluster with 280 GB total checkpoint, each node writes approximately 1.1 GB. To complete in 10 seconds, each node needs at least 900 Mbps sustained, easily served by a single 25 Gbps link. However, larger shards (from fewer, larger nodes) or faster targets require 100 Gbps or more per node.
• Aggregate switch bandwidth: The storage network leaf switches must provide non-blocking bandwidth to the NVMe-oF targets. For a 200 GB/s aggregate checkpoint write rate, the storage network needs at least 1.6 Tbps of fabric bandwidth to the storage tier.

Network Topology

For clusters up to 64 GPU nodes, a single-tier leaf switch connecting both GPU servers and NVMe-oF targets is sufficient. For larger clusters, a leaf-spine topology dedicated to storage traffic prevents storage I/O from competing with inter-GPU communication traffic.

Physical separation of the storage network from the GPU compute network (separate NICs, separate switches) is recommended for clusters above 32 nodes. This eliminates congestion interactions between storage I/O bursts and all-reduce operations during training.

RoCEv2 Configuration

When using NVMe-oF over RoCEv2, the storage network switches must be configured for lossless Ethernet:

• Priority Flow Control (PFC): Enable PFC on the storage traffic priority class to prevent packet drops that would cause RDMA retransmissions.
• Explicit Congestion Notification (ECN): Configure ECN marking thresholds to signal congestion before queues fill, allowing endpoints to reduce transmission rates gracefully.
• DSCP/PCP mapping: Map NVMe-oF traffic to the appropriate DSCP value and PFC priority class. Keep storage traffic isolated from other traffic classes.
• Buffer allocation: Allocate sufficient switch buffer memory to the lossless priority class to absorb burst traffic during checkpoint writes without triggering PFC pauses that propagate to other switch ports.

Capacity Planning and Cost Optimization

NVMe flash storage is expensive relative to spinning disk or object storage. Effective capacity planning ensures you provision enough performance without overspending on capacity.

Tiered Storage Economics

Storage Tier	Approximate Cost per TB	Use Case
NVMe-oF all-flash (high performance)	$200 to $500	Active checkpoints, hot training data
Parallel file system on NVMe	$150 to $350	Full training dataset, active checkpoints
Parallel file system on SSD	$80 to $200	Warm training data, milestone checkpoints
Object storage (MinIO, S3-compatible)	$20 to $60	Raw datasets, archived checkpoints, model artifacts

For a 256-GPU cluster training a 70B parameter model, a representative storage architecture might include:

• 3 TB NVMe-oF high-performance tier for active checkpoints ($600 to $1,500)
• 50 TB parallel file system for training dataset ($7,500 to $17,500)
• 200 TB object storage for raw data, archived checkpoints, and model artifacts ($4,000 to $12,000)

The total storage infrastructure cost ($12,000 to $31,000) represents a small fraction of the GPU cluster cost but has outsized impact on training efficiency.

Operational Considerations

Monitoring and Alerting

NVMe-oF storage monitoring should track:

• Latency percentiles: p50, p99, and p99.9 latency for both reads and writes. Checkpoint write p99 latency spikes indicate storage congestion or drive issues.
• Throughput utilization: Sustained throughput versus provisioned bandwidth. Consistent operation above 80 percent utilization indicates the need for additional storage bandwidth.
• NVMe drive health: SMART attributes including spare capacity, temperature, media errors, and write endurance consumed. NVMe drives in checkpoint-heavy workloads can consume write endurance faster than typical enterprise workloads.
• Path health: Multi-path I/O status. A failed path reduces available bandwidth and may push remaining paths into congestion during checkpoint bursts.

Write Endurance Planning

AI training checkpoint workloads write significantly more data per day than typical enterprise storage workloads. For a 70B parameter model checkpointing every 30 minutes (48 checkpoints per day at 280 GB each), the daily write volume to the checkpoint tier is approximately 13.4 TB. Over a year, that is nearly 5 PB of writes.

Select NVMe drives rated for high write endurance -- 3 DWPD (Drive Writes Per Day) or higher for the checkpoint tier. For a 3 TB checkpoint pool at 3 DWPD, the daily write allowance is 9 TB, which is below the 13.4 TB daily write volume. This means either provisioning more capacity (5+ TB for write endurance headroom) or using drives with higher endurance ratings.

Data Protection

Checkpoint data requires protection against drive and node failures, but the protection scheme should not significantly impact write performance:

• Erasure coding: Distributed erasure coding (such as 8+2 or 12+3) across multiple drives and nodes provides space-efficient protection with tolerable write amplification.
• Replication: Two-way replication is simpler but doubles capacity requirements. May be acceptable for the relatively small checkpoint tier.
• Application-level redundancy: Some training frameworks support writing checkpoints to multiple storage targets directly, providing application-level replication without storage-level overhead.

Integration with Colocation Environments

Data center colocation providers hosting AI training clusters must plan for the specific infrastructure requirements of NVMe-oF storage:

• Power density: NVMe storage shelves running at full write throughput consume 10 to 25 watts per drive. A 24-drive storage node consumes 500 to 1,000 watts under load, comparable to a moderately configured compute server.
• Network infrastructure: Dedicated storage network switches must be provisioned alongside compute network infrastructure. Plan for 100 GbE or 400 GbE connectivity between GPU nodes and storage targets.
• Cooling considerations: NVMe drives operating at sustained write throughput generate more heat than drives in read-heavy workloads. Ensure adequate airflow across storage shelves, especially in high-density rack configurations.

Frequently Asked Questions

What is NVMe-oF and why does it matter for AI training?

NVMe-over-Fabrics extends the NVMe storage protocol across a network fabric, allowing GPU servers to access remote NVMe flash storage with near-local latency. For AI training, this enables writing multi-hundred-gigabyte checkpoints in seconds rather than minutes, minimizing expensive GPU idle time during checkpoint operations.

How much storage do AI training checkpoints require?

A single checkpoint for a 70 billion parameter model in mixed precision requires approximately 140 to 280 GB including optimizer states. Retaining 5 active checkpoints requires 700 GB to 1.4 TB of high-performance storage, plus additional archival capacity for milestone checkpoints.

Should I use NVMe-oF over RDMA (RoCE) or TCP?

For dedicated AI training environments, NVMe-oF over RoCEv2 is recommended because it delivers the lowest latency (10 to 30 microseconds) and the GPU cluster network typically already supports lossless Ethernet. NVMe-oF over TCP is suitable for shared environments where lossless Ethernet configuration is impractical, at the cost of approximately 30 to 60 microseconds additional latency.

How does NVMe-oF compare to parallel file systems like Lustre or GPFS?

NVMe-oF provides the high-performance storage transport, while parallel file systems provide the distributed namespace and multi-client access. Many production architectures use both: NVMe-oF as the storage backend for a parallel file system's data servers, or dedicated NVMe-oF volumes for checkpoints alongside a parallel file system for training data distribution.