GPU Cluster Storage: NFS, Lustre & GPFS for AI Workloads

Why Storage Is the GPU Cluster Bottleneck

Modern AI training workloads consume data at extraordinary rates. A single NVIDIA H100 SXM5 GPU can process training batches faster than most storage systems can deliver them. When you scale to 8, 32, or 64 GPUs, the storage subsystem often becomes the primary bottleneck -- not the network, not the compute, but the speed at which data reaches the GPUs.

The storage challenge in GPU clusters is fundamentally different from traditional enterprise storage. AI training needs:

• Massive sequential throughput -- datasets measured in terabytes must stream continuously
• Low metadata latency -- millions of small files (images, text chunks) require fast directory lookups
• Checkpoint durability -- model snapshots every 10-30 minutes must write without stalling GPUs
• Multi-node concurrency -- dozens of compute nodes reading the same dataset simultaneously

Getting this wrong is expensive. GPU idle time on a 64-GPU GPU colocation cluster costs $200-$400 per hour. A storage bottleneck causing just 10% GPU idle time wastes $20-$40/hour, or $14,000-$28,000 per month.

The Four Storage Architectures Compared

NFS (Network File System)

NFS is the simplest option and the default choice for many organizations starting with GPU workloads. NFSv4.2 with RDMA support (NFS-over-RDMA) has closed much of the traditional performance gap, making it viable for smaller clusters.

Architecture: Single server exports a filesystem to multiple clients. All I/O flows through one node. NFSv4.1+ supports pNFS (parallel NFS) for basic data striping, but adoption remains limited.

Throughput: A well-tuned NFS server with NVMe storage delivers 5-15 GB/s to the network. This is adequate for 1-4 GPU nodes running inference or fine-tuning workloads, but insufficient for large-scale distributed training.

Best for: Small clusters (1-8 nodes), inference serving, model fine-tuning, prototyping environments, and organizations without dedicated storage engineers.

Limitations: Single-server bottleneck, limited metadata scalability (struggles above 10M files in a single directory), no native data striping across disks on the client side.

Lustre

Lustre is the dominant parallel filesystem in HPC and increasingly in AI infrastructure. It powers most of the world's top supercomputers and is the storage backend for many large-scale managed AI hosting environments.

Architecture: Separates metadata (MDS/MDT) from data (OSS/OST). Files are automatically striped across multiple Object Storage Targets, enabling parallel reads from all clients simultaneously. A single Lustre filesystem can span thousands of storage servers.

Throughput: Aggregate throughput scales linearly with OST count. A modest 8-OST configuration delivers 40-80 GB/s. Large deployments (64+ OSTs) exceed 1 TB/s sustained. Per-client throughput can reach 20-40 GB/s with RDMA networking.

Best for: Large GPU clusters (16+ nodes), distributed training, organizations with HPC expertise, workloads that need sustained high throughput for large sequential reads.

Limitations: Operational complexity (requires dedicated storage administrators), metadata performance can bottleneck on workloads with millions of small files (though DNE -- Distributed Namespace -- addresses this), limited built-in data tiering.

IBM Spectrum Scale (GPFS)

GPFS (General Parallel File System), now marketed as IBM Spectrum Scale, offers enterprise-grade parallel storage with strong data management features. It is the native filesystem for IBM's GPU-accelerated systems including the Power10-based clusters.

Architecture: Fully distributed architecture where every node can serve data. No dedicated metadata server -- all nodes participate in metadata operations using a distributed lock manager. Files are striped across all disks in the cluster.

Throughput: Comparable to Lustre for large sequential I/O (40-80 GB/s on 8-node configurations). GPFS excels at mixed workloads -- it handles small-file-heavy metadata operations better than Lustre out of the box thanks to its distributed lock manager.

Best for: Enterprise environments needing commercial support, mixed HPC and AI workloads, organizations using IBM hardware, environments requiring built-in data tiering (ILM policies), encryption, and compression.

Limitations: Licensing costs ($5,000-$15,000 per TB per year at enterprise scale), IBM vendor lock-in for support, steeper learning curve for non-IBM administrators.

WekaFS

WekaFS (Weka.io) is the newest entrant and has rapidly gained market share in AI-focused data centers. It is purpose-built for NVMe-flash workloads and GPU-accelerated computing.

Architecture: Software-defined, fully distributed. Uses a proprietary erasure-coding scheme that delivers Lustre-class throughput with NFS-like simplicity. Supports S3 and POSIX interfaces simultaneously. GPU Direct Storage (GDS) support enables direct DMA transfers from storage to GPU memory, bypassing the CPU entirely.

Throughput: Exceptional for mixed I/O patterns. A 6-node WekaFS cluster delivers 60-100 GB/s with sub-200-microsecond latency. GPU Direct Storage reduces checkpoint write latency by 3-5x compared to traditional POSIX paths.

Best for: AI-first data centers, environments needing both high throughput and low latency, organizations that want parallel-filesystem performance without parallel-filesystem complexity, GPU Direct Storage workloads.

Limitations: Requires all-NVMe storage (no HDD tier for cold data unless using S3 tiering), licensing costs comparable to GPFS, relatively newer ecosystem (fewer community resources than Lustre).

Performance Comparison

The following table summarizes real-world throughput expectations for each storage system in a typical AI training configuration with InfiniBand or RoCE networking:

Metric	NFS	Lustre	GPFS	WekaFS
Max aggregate throughput (8 servers)	5-15 GB/s	40-80 GB/s	40-80 GB/s	60-100 GB/s
Metadata ops/sec	50K-100K	200K-500K	500K-1M	300K-800K
GPU Direct Storage	No	Limited	Yes	Yes (native)
Operational complexity	Low	High	Medium-High	Low-Medium
Cost per TB/year	$500-$1,500	$1,000-$3,000	$5,000-$15,000	$3,000-$10,000
Minimum viable cluster	1 server	3 servers (1 MDS + 2 OSS)	3 servers	5 servers

Sizing Storage for AI Training

Storage sizing for GPU clusters involves three dimensions: capacity, throughput, and IOPS. Most organizations underestimate the throughput requirement and overprovision capacity.

Capacity Sizing

Training datasets range from 500 GB (fine-tuning) to 100+ TB (pre-training on internet-scale corpora). Checkpoints add significant storage: a 70B parameter model checkpoint is roughly 140 GB. With checkpoints every 15 minutes over a multi-week training run, checkpoint storage alone can exceed 50 TB.

Rule of thumb: Provision 3-5x the raw dataset size. This accounts for preprocessing copies, checkpoints, model outputs, and operational overhead.

Throughput Sizing

Each GPU needs a sustained data feed. For training workloads:

• Image classification (ResNet, EfficientNet): 2-3 GB/s per GPU
• LLM pre-training: 1-2 GB/s per GPU (tokenized data is compact)
• Video training: 5-10 GB/s per GPU (raw video frames)
• Checkpoint writes: 10-50 GB burst every 10-30 minutes

Rule of thumb: Multiply per-GPU throughput by GPU count, add 50% for checkpoint bursts. A 32-GPU cluster training an LLM needs 32 x 1.5 GB/s x 1.5 = 72 GB/s sustained storage throughput.

IOPS Sizing

Datasets with millions of small files (ImageNet has 14M images) demand high metadata IOPS. Each training epoch opens and reads every file. A 32-GPU cluster processing ImageNet at 10,000 images/second/GPU generates 320,000 file open operations per second.

Rule of thumb: Convert small-file datasets to sequential formats (TFRecord, WebDataset, Parquet) before training. This converts millions of metadata operations into a handful of large sequential reads, reducing IOPS requirements by 100-1000x.

GPU Direct Storage: The Game Changer

NVIDIA GPU Direct Storage (GDS) enables direct DMA transfers between NVMe storage and GPU memory, bypassing the CPU and system memory entirely. This reduces storage latency by 2-5x and frees CPU cores for data preprocessing.

GDS support varies by filesystem:

• WekaFS: Native GDS support, fully integrated
• GPFS: GDS support via IBM ESS (Elastic Storage System)
• Lustre: Experimental GDS support (community patches available)
• NFS: No GDS support

For workloads where checkpoint write performance is critical (training runs costing $10,000+/day), GDS can reduce checkpoint time from minutes to seconds, directly improving GPU utilization.

Architecture Recommendations by Cluster Size

Small Cluster (1-8 GPUs)

Use NFS with a high-performance NVMe storage server. Keep it simple. A single server with 8x NVMe drives and 100GbE networking delivers 10+ GB/s, which is sufficient for fine-tuning and inference. Convert datasets to sequential formats to minimize metadata load.

Medium Cluster (16-64 GPUs)

WekaFS or Lustre. If you have HPC storage expertise, Lustre with 4-8 OSTs provides excellent price-performance. If you want simplicity, WekaFS with 5-6 NVMe storage nodes delivers comparable throughput with less operational burden. Budget $50,000-$150,000 for storage infrastructure.

Large Cluster (128+ GPUs)

Lustre or GPFS with dedicated storage administrators. At this scale, storage becomes a first-class infrastructure concern with its own team. Budget 15-25% of total cluster cost for storage. Consider tiered storage: NVMe tier for active datasets, HDD or object-storage tier for archived checkpoints and cold data. Connect storage via dedicated InfiniBand fabric for maximum throughput.

Common Mistakes to Avoid

• Underprovisioning throughput: Capacity is cheap; throughput is expensive. A 100 TB NAS appliance with 2 GB/s throughput will starve an 8-GPU cluster.
• Ignoring metadata performance: Training on millions of small files without converting to sequential formats creates a metadata storm that even Lustre struggles with.
• No checkpoint strategy: Writing checkpoints to the same filesystem as training data creates I/O contention. Use a separate checkpoint volume or schedule writes during evaluation phases.
• Skipping the network: Storage throughput is limited by network bandwidth. Ensure your storage network matches or exceeds the aggregate throughput of your storage system. 100GbE minimum; InfiniBand preferred for clusters over 16 nodes.
• No data locality: Placing storage in a different rack or data hall introduces latency and consumes backbone bandwidth. Co-locate storage servers in the same rack group as compute nodes.

UAE Considerations

Data centers in the UAE face unique storage challenges. Ambient temperatures averaging 35-45 degrees Celsius increase cooling demands for storage servers. NVMe drives throttle at high temperatures, reducing throughput by 10-30%. Ensure storage racks have dedicated cooling paths and monitor drive temperatures actively.

For organizations requiring data residency compliance under TDRA regulations, all training data, checkpoints, and model weights must remain within UAE borders. This eliminates cloud-based storage tiering to non-UAE regions and requires on-premises or UAE-based colocation storage infrastructure.