Kubernetes GPU Orchestration: NVIDIA GPU Operator, Scheduling & Multi-Tenancy

As AI training clusters scale from a handful of GPUs to thousands, the manual provisioning and scheduling approaches that worked at small scale become operationally impossible. Kubernetes has emerged as the standard orchestration layer for GPU infrastructure, but running GPU workloads on Kubernetes requires specialized tooling, careful scheduling configuration, and an understanding of GPU-specific constraints that do not exist in traditional containerized deployments.

This guide covers the complete stack: from installing the NVIDIA GPU Operator to configuring advanced scheduling with MIG partitioning, time-slicing, topology-aware placement, and multi-tenant GPU sharing. Whether you are a data center operator building a managed GPU colocation platform or an ML engineering team deploying training pipelines, this is the operational reference for production GPU orchestration.

Why Kubernetes for GPU Infrastructure

Before Kubernetes, GPU clusters were typically managed through Slurm, bare-metal provisioning, or custom scripts. Kubernetes brings several advantages for modern AI infrastructure:

• Declarative resource management: Teams request GPUs in pod specs, and the scheduler handles placement. No manual node assignment or SSH-based job submission.
• Multi-tenancy: Namespaces, resource quotas, and RBAC enable multiple teams to share a GPU cluster safely, which is critical for managed hosting providers.
• Automated operations: Node health checks, pod restart policies, and rolling updates reduce the operational burden of maintaining large GPU fleets.
• Ecosystem integration: ML tools like Kubeflow, Ray, and Volcano are built on Kubernetes, providing training orchestration, hyperparameter tuning, and model serving without custom integration.
• Hybrid scheduling: A single cluster can run GPU training jobs alongside CPU-based preprocessing, data pipelines, and API serving workloads.

The trade-off is complexity. Kubernetes was designed for stateless web services, not GPU-attached scientific computing. The tooling described in this guide bridges that gap.

The NVIDIA GPU Operator: Automated GPU Management

The GPU Operator is a Kubernetes operator that manages the full software stack required to expose GPUs to containers. It installs and maintains:

• GPU drivers: Deployed as a DaemonSet, ensuring every GPU node runs the correct driver version without manual installation
• NVIDIA Container Toolkit: Enables containers to access GPU hardware through the container runtime
• Kubernetes Device Plugin: Registers GPUs as schedulable resources (nvidia.com/gpu) with the kubelet
• DCGM (Data Center GPU Manager): Provides GPU health monitoring, telemetry, and diagnostics
• DCGM Exporter: Exposes GPU metrics (utilization, temperature, memory, power, ECC errors) to Prometheus
• GFD (GPU Feature Discovery): Labels nodes with GPU properties (model, driver version, CUDA version, MIG capability) for informed scheduling
• MIG Manager: Configures Multi-Instance GPU partitioning through Kubernetes-native configuration

Why this matters at scale: Without the GPU Operator, every node addition requires manual driver installation, toolkit configuration, and device plugin setup. In a 200-node cluster with regular node replacements, this is a full-time operations task. The GPU Operator reduces it to a single Helm chart deployment.

GPU Scheduling Strategies

Whole-GPU Allocation (Default)

By default, Kubernetes allocates entire GPUs to pods. A pod requesting nvidia.com/gpu: 4 receives four dedicated physical GPUs. This is the simplest model and provides complete isolation. It is the right choice for large-scale training jobs that need full GPU memory and compute.

Limitation: If a workload only needs 2 GB of a GPU with 80 GB of memory, the remaining 78 GB is wasted. At current GPU hosting prices, this waste is expensive.

MIG (Multi-Instance GPU) Partitioning

MIG divides a physical GPU into up to seven hardware-isolated instances. Each instance has dedicated streaming multiprocessors, memory controllers, and L2 cache. From the perspective of the application, each MIG instance is an independent GPU.

Supported hardware: NVIDIA A100, A30, H100, H200, and B200.

Kubernetes integration: The GPU Operator's MIG Manager reads a ConfigMap to determine the desired MIG partition strategy per node. GPUs are reconfigured automatically, and each MIG instance appears as a separate extended resource in the device plugin. Pods request specific MIG profiles (e.g., nvidia.com/mig-3g.40gb: 1) in their resource spec.

Best for: Inference workloads, development environments, and multi-tenant colocation where different customers need guaranteed GPU slices.

GPU Time-Slicing

Time-slicing allows multiple pods to share a single GPU by context-switching between them, similar to how a CPU is shared across processes. The GPU Operator supports time-slicing configuration through a ConfigMap that specifies how many replicas of each GPU should be advertised.

Key consideration: Time-slicing provides no memory isolation. If one pod allocates all GPU memory, other pods on the same GPU will fail with out-of-memory errors. It is appropriate for lightweight inference or interactive development workloads, not production training.

Topology-Aware Scheduling

For multi-GPU training, GPU placement relative to the NVLink/NVSwitch topology and PCIe hierarchy significantly impacts performance. GPUs connected through NVLink communicate at 900 GB/s, while GPUs crossing a PCIe switch communicate at a fraction of that speed.

The Topology Aware Scheduler plugin ensures that multi-GPU pods receive GPUs that are directly interconnected. This is critical for distributed training frameworks that rely on NCCL collective operations. Without topology awareness, a 4-GPU training job might receive two GPUs on one NVLink domain and two on another, introducing a PCIe bottleneck that can reduce training throughput by 30 to 50 percent.

Multi-Tenant GPU Clusters

For GPU colocation and managed hosting providers, multi-tenancy is a core requirement. Each customer needs isolated, guaranteed GPU resources within a shared cluster.

Resource Quotas

Kubernetes ResourceQuota objects limit GPU consumption per namespace. A quota of nvidia.com/gpu: 16 on a customer's namespace ensures they cannot consume more than 16 GPUs, regardless of how many pods they launch. Combined with LimitRange objects, operators can also set per-pod GPU limits.

Priority and Preemption

PriorityClass objects enable tiered GPU access. A premium customer's training jobs can preempt lower-priority development workloads when GPU resources are constrained. The preempted pods are gracefully terminated and rescheduled when capacity becomes available.

Network Isolation

NetworkPolicy objects isolate tenant traffic at the pod level. For GPU workloads, this ensures that NCCL traffic from one customer's distributed training cannot interfere with another's. In high-density environments, dedicated InfiniBand partitions provide hardware-level network isolation.

GPU Monitoring per Tenant

DCGM Exporter metrics, labeled by namespace, enable per-tenant GPU utilization dashboards. This data feeds billing systems (usage-based pricing), SLA monitoring, and capacity planning. Operators can track utilization, temperature, power draw, and ECC errors per customer, per GPU.

Production Best Practices

1. Pin GPU driver versions: Use the GPU Operator's driver version pinning to prevent untested driver updates from breaking training jobs. Test new drivers on a canary node pool before rolling out cluster-wide.
2. Implement GPU health checks: Configure DCGM health checks to automatically cordon nodes with failing GPUs (ECC errors, thermal throttling, NVLink failures). A single unhealthy GPU can cause a multi-node training job to hang indefinitely.
3. Use node pools: Separate GPU node pools by GPU model (H200, A100, MI300X) using node labels and taints. This prevents scheduling mismatches and simplifies fleet management.
4. Enable shared storage: Training jobs need fast, shared storage for datasets and checkpoints. Deploy a parallel filesystem (Lustre, GPFS, or BeeGFS) or high-performance NFS server alongside your GPU cluster.
5. Configure pod disruption budgets: Long-running training jobs should use PodDisruptionBudgets to prevent accidental eviction during node maintenance. Coordinate GPU node upgrades during planned maintenance windows.
6. Right-size GPU requests: Audit GPU utilization regularly. If workloads consistently use less than 50% of allocated GPU memory, consider MIG partitioning or time-slicing to improve fleet utilization and reduce infrastructure costs.
7. Plan for power density: A single 8-GPU node can draw 5 to 10 kW. Ensure your facility's power and cooling infrastructure can support the density that Kubernetes makes easy to deploy.

Kubernetes vs. Slurm: When to Use Each

Both orchestrators have valid use cases in GPU data centers:

• Choose Kubernetes when you need multi-tenancy, containerized workflows, microservices integration, or are building a managed GPU platform. Kubernetes excels at mixed workloads (training + inference + data processing) and provides the API-driven automation that SaaS platforms require.
• Choose Slurm when your users are researchers submitting batch HPC jobs, your workloads are primarily MPI-based, or your team has deep Slurm expertise. Slurm's job scheduling is more mature for traditional HPC patterns like job arrays, dependencies, and fair-share scheduling.
• Hybrid approach: Some organizations run Kubernetes for inference and serving while using Slurm for large-scale training jobs. Volcano, a Kubernetes-native batch scheduler, bridges the gap by adding Slurm-like scheduling features (gang scheduling, job queues, fair-share) to Kubernetes.

Related Articles