AI Model Serving Infrastructure: Load Balancing & Autoscaling

The Gap Between Training a Model and Serving It

Training an AI model and serving it in production are fundamentally different infrastructure challenges. Training is a batch workload: a fixed dataset, a known compute budget, and a tolerance for job completion time measured in hours or days. Serving is a real-time workload: unpredictable request volumes, strict latency budgets measured in milliseconds, and zero tolerance for downtime.

The infrastructure that powers GPU training clusters is optimized for throughput. The infrastructure that powers inference must be optimized for latency under load, efficient GPU utilization at variable request rates, and graceful handling of traffic spikes. This article covers the serving layer that bridges that gap: load balancers, request routers, batching engines, autoscalers, and the colocation infrastructure decisions that make production serving reliable.

Architecture of a Production Serving Stack

A production AI model serving stack consists of several layers, each handling a different concern:

1. Edge/API gateway: Accepts client HTTPS requests, handles authentication, rate limiting, and TLS termination. This layer is CPU-based and horizontally scalable.
2. Request router: Examines the incoming request (model name, input size, priority) and selects the best GPU worker from the available pool. This is where GPU-aware load balancing logic lives.
3. GPU serving engine: Runs on each GPU worker. Manages model loading, request batching, the actual GPU forward pass, and response streaming. Examples include vLLM, NVIDIA Triton Inference Server, and TensorRT-LLM.
4. Model registry: Stores model artifacts (weights, tokenizers, configuration) and manages versioning, A/B test assignments, and rollback state.
5. Autoscaler: Monitors queue depth, latency, and GPU utilization to add or remove GPU worker replicas. Integrates with Kubernetes GPU scheduling or bare-metal provisioning systems.
6. Observability: Collects per-request latency, throughput, GPU utilization, memory pressure, error rates, and token generation speed (for LLMs).

GPU-Aware Load Balancing

Why Round-Robin Fails for GPU Inference

Traditional load balancers distribute requests using round-robin, least-connections, or random algorithms. These work well when all requests are roughly equal in cost and all backends have equal capacity. Neither assumption holds for GPU inference.

A request to generate 20 tokens from a large language model takes a fraction of the time of a request to generate 2,000 tokens. A vision model processing a 256x256 image uses far less GPU memory and compute than the same model processing a 2048x2048 image. Round-robin distribution sends these wildly different requests to workers without regard for their cost, resulting in some GPUs being overloaded while others sit idle.

Additionally, GPU workers are not always equal. In a multi-tenant colocation environment, different workers may have different models loaded, different amounts of free GPU memory (affected by the KV cache of in-flight requests), and different batch queue depths.

Effective GPU Load Balancing Strategies

Production GPU load balancers use one or more of the following strategies:

• Least-pending-requests: Route to the worker with the fewest requests waiting in its batch queue. This is the simplest GPU-aware strategy and works well when request sizes are relatively uniform.
• Least-GPU-memory-used: Route to the worker with the most free GPU memory. This is particularly effective for LLM serving where the KV cache consumes memory proportional to the number and length of in-flight requests.
• Estimated-completion-time: The router estimates how long each worker will take to finish its current batch (based on input sizes and generation lengths) and routes to the worker that will become available soonest. This requires the serving engine to report its current workload to the router.
• Model-affinity: For multi-model deployments, route requests to workers that already have the target model loaded in GPU memory. Loading a large model (which can take 30 to 120 seconds for a 70B parameter model) adds unacceptable latency if triggered by a request.

Strategy	Best For	Weakness
Round-Robin	Uniform request sizes	Ignores GPU state entirely
Least-Pending-Requests	Mixed-size requests, simple setup	Does not account for request cost
Least-GPU-Memory	LLM serving with KV cache pressure	Requires GPU memory reporting
Estimated-Completion-Time	High-accuracy routing, SLA-critical	Complex to implement accurately
Model-Affinity	Multi-model deployments	Can create hotspots on popular models

Request Batching Strategies

Static Batching

Static batching collects a fixed number of requests (or waits for a fixed timeout) before sending the entire batch to the GPU for processing. The batch executes as a single forward pass, and all responses are returned together when the longest request in the batch completes.

Static batching is simple to implement but wasteful. Short requests that could return in 50 ms must wait for long requests in the same batch to finish, which might take 2 seconds for an LLM generating a long completion. GPU utilization drops because the GPU is idle during the padding time at the end of shorter sequences.

Continuous Batching

Continuous batching (iteration-level batching) solves the padding problem by allowing the serving engine to add and remove individual requests from the active batch at every forward pass iteration. When a request finishes generating tokens, it is immediately evicted from the batch and its response is returned to the client. A new request from the queue takes its place in the very next iteration.

Frameworks like vLLM implement continuous batching with PagedAttention, a memory management technique that allocates KV cache memory in fixed-size pages rather than contiguous blocks. This eliminates memory fragmentation and enables efficient addition and removal of requests from the batch. Production deployments using continuous batching typically achieve 2x to 5x higher throughput than static batching at the same latency target.

Priority-Based Batching

Not all inference requests are equal. A real-time chatbot response has a strict latency budget (typically under 200 ms for first token). A background summarization job can tolerate seconds of queuing. Priority-based batching assigns each request a priority level and ensures that high-priority requests are inserted into the active batch before lower-priority ones, even if the lower-priority requests arrived first.

Implementing priority batching requires the serving engine to support preemption: the ability to pause a low-priority request (saving its KV cache state) and replace it with a high-priority request. vLLM supports this through its preemption and swapping mechanisms, where paused request KV caches are moved to CPU memory and restored when the GPU has capacity.

Autoscaling for GPU Inference

Choosing the Right Scaling Metric

Autoscaling GPU inference workloads requires different metrics than traditional web services. CPU utilization, the default metric for most autoscalers, is nearly useless for GPU workloads because the bottleneck is always the GPU, not the CPU.

The most effective scaling metrics for GPU inference are:

• Request queue depth per GPU: The number of requests waiting for GPU processing. Scale up when this exceeds a threshold (commonly 5 to 15 requests per GPU) for a sustained period (60 to 120 seconds).
• P95 latency relative to SLA: If the SLA requires P95 latency under 500 ms and the observed P95 is 400 ms, there is still headroom. If it reaches 450 ms, scale up proactively. This metric directly ties scaling to the business commitment.
• GPU memory utilization: For LLM serving, KV cache memory consumption grows with concurrent request count. When GPU memory utilization exceeds 85 to 90 percent, the serving engine must start rejecting or queuing requests, making this a leading indicator of capacity saturation.

Scale-Up and Scale-Down Policies

GPU instances are expensive, so autoscaling policies must balance responsiveness with cost efficiency. Common patterns include:

• Aggressive scale-up, conservative scale-down: Add GPU replicas quickly (within 2 to 5 minutes of detecting overload) but remove them slowly (only after 10 to 15 minutes of sustained underutilization). This avoids oscillation during bursty traffic.
• Step scaling: Add 1 replica at moderate overload, 2 replicas at severe overload, 4 replicas at critical overload. This reaches the right capacity faster during sudden traffic spikes without over-provisioning during gradual increases.
• Scheduled scaling: Pre-scale GPU capacity based on known traffic patterns (business hours, marketing campaigns, product launches). This avoids the cold-start delay of reactive scaling for predictable load increases.

Scaling with Kubernetes

In Kubernetes-orchestrated GPU environments, the Horizontal Pod Autoscaler (HPA) can be configured with custom metrics from Prometheus or Datadog. The HPA queries GPU-specific metrics (queue depth, latency, memory) exposed by the serving engine and adjusts the replica count of the serving Deployment.

The NVIDIA GPU Operator and Kubernetes Device Plugin ensure that GPU resources are properly advertised to the scheduler. The Cluster Autoscaler then handles the node-level scaling, provisioning new GPU nodes from the bare-metal or cloud pool when pods are pending due to insufficient GPU resources.

For colocation-based GPU deployments, node-level autoscaling means having pre-provisioned standby GPU servers that the autoscaler can activate rather than waiting for new hardware to be racked. Many colocation operators offer warm spare pools where reserved servers sit powered off but racked, cabled, and ready to boot in under 5 minutes.

Serving Engine Selection

Engine	Strengths	Best For
vLLM	Continuous batching, PagedAttention, high throughput	LLM inference (GPT, Llama, Mistral families)
NVIDIA Triton	Multi-framework, ensemble pipelines, dynamic batching	Mixed model types (vision, NLP, recommender)
TensorRT-LLM	NVIDIA-optimized kernels, INT4/INT8 quantization	Maximum single-GPU throughput for NVIDIA hardware
Ray Serve	Python-native, composable pipelines, Ray ecosystem	Multi-step pipelines (retrieval + generation)
SGLang	Structured generation, RadixAttention, constraint decoding	Applications requiring JSON/schema output compliance

The choice of serving engine depends on the model type, hardware platform, and production requirements. For pure LLM serving on NVIDIA GPUs, vLLM with continuous batching and PagedAttention delivers the best combination of throughput and latency. For heterogeneous model portfolios that include vision, speech, and recommendation models alongside LLMs, NVIDIA Triton Inference Server provides a unified serving interface with backend-specific optimizations.

Model Parallelism for Large Models

Models too large to fit on a single GPU require parallelism across multiple GPUs within a serving replica. The two primary approaches are tensor parallelism and pipeline parallelism.

Tensor parallelism splits individual layers across multiple GPUs. Each forward pass requires all-reduce communication between GPUs within the layer, making it sensitive to GPU interconnect bandwidth. Tensor parallelism is best suited for GPUs connected via NVLink or NVSwitch within the same node, where communication bandwidth exceeds 400 GB/s.

Pipeline parallelism assigns different layers to different GPUs. Each GPU processes its assigned layers and passes the activations to the next GPU. Pipeline parallelism tolerates higher communication latency than tensor parallelism, making it viable across GPUs in different nodes connected via InfiniBand or RoCE.

Most production deployments of 70B+ parameter models use tensor parallelism across 4 to 8 GPUs within a single node. The NVIDIA GB200 NVL72 rack-scale architecture provides enough GPU memory to serve models up to 1 trillion parameters using tensor parallelism within a single NVL72 domain.

Health Checking and Failover

GPU inference workers fail in ways that CPU services do not. Common failure modes include GPU memory leaks (from accumulating KV cache fragments), CUDA driver crashes, ECC memory errors, and thermal throttling in environments with inadequate cooling capacity.

Effective health checking for GPU workers includes:

• Liveness probes: Verify the serving process is running and responsive to basic health endpoint requests. Kubernetes-style liveness probes that restart the container on failure.
• Readiness probes: Verify the model is loaded in GPU memory and the worker is ready to accept inference requests. A worker that is alive but still loading a model should not receive traffic.
• GPU health probes: Check GPU temperature, ECC error count, power draw, and memory utilization. An H100 GPU reporting uncorrectable ECC errors should be drained (stop sending new requests, wait for in-flight requests to complete) and taken offline for investigation.
• Inference quality probes: Periodically send a known input to each worker and verify the output matches expectations. This catches subtle failures like corrupted model weights or quantization errors that do not trigger process-level health checks.

Colocation Infrastructure Considerations

Deploying AI model serving in a colocation data center introduces infrastructure decisions that affect serving performance and reliability:

• Power density: GPU inference servers draw 2 to 10 kW per server depending on GPU count and model. A 20-server serving cluster requires 40 to 200 kW of high-density power concentrated in a few racks. Ensure the colocation provider can deliver the required power density per rack.
• Network latency: The path from the load balancer to GPU workers should be under 1 ms within the facility. Cross-connect to the API gateway and any upstream CDN or application servers to minimize external latency. Cross-connect provisioning is typically measured in days, so plan ahead.
• Storage for model weights: Model weights must be stored on low-latency storage accessible to all GPU workers. NVMe local storage offers the fastest model loading but requires each worker to have a local copy. Shared NFS or Lustre storage enables a single copy serving all workers but adds network I/O during model loading.
• Warm spare capacity: Negotiate warm spare GPU servers with the colocation SLA for autoscaling headroom. Pre-racked, pre-cabled servers that can be activated by the autoscaler in under 5 minutes provide faster scaling than on-demand provisioning.

Monitoring and Observability

Production GPU serving requires metrics beyond standard web service monitoring. Essential metrics include:

• Time to first token (TTFT): For LLM serving, the latency from request receipt to the first output token. This is the metric users perceive as "responsiveness."
• Tokens per second (TPS): The rate of output token generation, both per-request and aggregate across the cluster.
• GPU utilization: Reported by DCIM and nvidia-smi, but at per-second granularity rather than the default 10-second averaging window that masks micro-bursts.
• KV cache hit rate: For serving engines that implement prefix caching (reusing KV cache entries from previous requests with shared prefixes), the cache hit rate directly affects throughput.
• Preemption rate: How often low-priority requests are paused to make room for high-priority ones. A high preemption rate indicates insufficient GPU capacity for the priority mix.
• Model loading time: How long new replicas take to become ready after being provisioned. This directly bounds autoscaling responsiveness.

Frequently Asked Questions

What is AI model serving infrastructure?

AI model serving infrastructure is the hardware and software stack that accepts inference requests, routes them to available GPU workers, executes the model forward pass, and returns predictions. It includes load balancers, request routers, GPU serving engines (like vLLM or Triton Inference Server), model registries, health checkers, and autoscaling controllers.

How does GPU load balancing differ from traditional load balancing?

Traditional load balancers distribute requests based on connection count or round-robin. GPU load balancing must account for GPU memory occupancy, batch queue depth, model-specific compute requirements, and the fact that different requests have vastly different processing times. Effective GPU load balancers use metrics like pending request count and GPU memory utilization.

What autoscaling metrics should be used for GPU inference?

The most effective metrics are request queue depth per GPU, P95 latency relative to the SLA target, and GPU memory utilization. CPU utilization is a poor signal for GPU workloads. A common policy scales up when queue depth exceeds 10 requests per GPU for more than 60 seconds.

What is continuous batching in AI inference?

Continuous batching adds new requests to the active batch at every forward pass iteration rather than waiting for the entire batch to complete. This improves GPU utilization by 2x to 5x compared to static batching. Frameworks like vLLM implement continuous batching with PagedAttention for efficient memory management.

How many GPUs are needed for serving a 70-billion parameter model?

A 70B model in FP16 requires approximately 140 GB of GPU memory for weights alone. This means at least 2 NVIDIA H100 GPUs (80 GB each) using tensor parallelism, though most production deployments use 4 H100s per replica to provide headroom for the KV cache during batched inference.