Row of electrical switchgear and power distribution cabinets in a data center aisle

In short: UPS ride-through time for a GPU training cluster should be sized around completing an emergency checkpoint write, not around a generic facility runtime target. The real constraint is usually checkpoint storage write bandwidth, not battery capacity, and the two have to be sized together.

Electrical Ride-Through Is a Solved Problem, Until You Ask What It's Protecting

Electrical ride-through, the time a UPS keeps a load powered after utility power fails, is well understood as an engineering calculation. A UPS stores a known quantity of energy, the connected load draws it at a known rate, and the runtime falls out of straightforward arithmetic, as our guide to UPS systems in data centers covers for facility-wide sizing. It is a storage problem, not a balance problem, which is exactly what makes it calculable and testable rather than uncertain.

That calculation answers "how long does the load stay powered." It does not answer a more specific question a GPU training cluster needs answered: what has to happen inside that window to avoid losing hours of compute, and whether the generic runtime a facility UPS is sized for actually covers it. Our guide to thermal ride-through makes the useful distinction that electrical ride-through is "boring" precisely because it is well-defined; the AI training question hiding inside that boring number is what this article addresses.

The Checkpoint Write Is the Thing You're Actually Protecting

A long-running AI training job holds its most valuable, least-recoverable state in GPU memory: model weights, optimizer state, and gradient accumulation that cannot be reconstructed, only recomputed from the last saved checkpoint. Training frameworks checkpoint this state to persistent storage at intervals, a process covered from the storage-sizing side in our guide to NVMe-oF storage architecture for AI training, which focuses on the bandwidth and capacity needed for routine, scheduled checkpoint writes during normal operation.

A power event changes the problem from a scheduled write to an emergency one. If the UPS can hold the cluster up long enough for an unscheduled "save now" checkpoint to complete, the job resumes from a state close to the moment power was lost. If it cannot, the job resumes from whatever the last completed scheduled checkpoint was, and every interval of training between that checkpoint and the outage is gone, regardless of how briefly the power was actually out.

Three Numbers That Set the Real Ride-Through Requirement

Generic UPS sizing guidance for IT loads commonly targets 5 to 15 minutes of runtime, sized around orderly server shutdown or a generator start-and-transfer window. That target was not built around writing hundreds of gigabytes of GPU memory state to storage under time pressure, and for a training cluster it needs to be checked against three numbers specific to the job:

  • Checkpoint size. The total data volume that has to move from GPU memory to persistent storage: model weights, optimizer states (often two to three times the model size itself for optimizers like Adam), and any gradient state being preserved.
  • Aggregate write bandwidth available. The real, sustained throughput the checkpoint storage tier can absorb simultaneously from every GPU node writing at once, which is the figure sized in a checkpoint storage design covered in the NVMe-oF guide referenced above, not a theoretical peak figure.
  • Detection and trigger time. The interval between utility power failing and the orchestration layer actually issuing the emergency checkpoint command, which depends on how the UPS-on-battery signal is monitored and acted on, not on the UPS itself.

Ride-through time has to cover all three in sequence: detection, the checkpoint write itself, and a safety margin for write completion verification, not just the write duration alone.

A Worked Example

Consider a cluster running a large model whose combined weights and optimizer state total 800GB of checkpoint data across a node group, backed by a checkpoint storage tier sized for 40GB/s aggregate sustained write bandwidth under normal load.

StageEstimated TimeWhat Determines It
Outage detection + trigger5-15 secondsUPS-on-battery signal monitoring and orchestration response time
Checkpoint write (800GB at 40GB/s)~20 seconds at full rated bandwidthCheckpoint size vs. real achievable write bandwidth under emergency, all-nodes-writing-at-once conditions
Write verification + safety margin30-60 secondsConfirming checkpoint integrity before allowing further power loss
Practical minimum ride-through target2-3 minutes, with marginSum of the above, plus buffer for write bandwidth degradation under contention

The arithmetic looks comfortably inside a standard UPS runtime target. The risk is not the headline number; it is that the 40GB/s figure assumes normal operating conditions, and an emergency, all-nodes-simultaneously write pattern can see real throughput degrade well below a storage tier's rated capacity due to contention across the fabric. A ride-through target built only on the best-case write rate, without margin for that degradation, is the gap that shows up during an actual outage rather than during planning.

Why AI Racks Changed the Underlying Load Profile

This sizing exercise did not matter as much in general-purpose data center design because the load was not dense enough, or the data being protected was not concentrated enough per rack, to make a few minutes of write bandwidth the binding constraint. Our guide to rack-level battery backup units covers how AI racks shifted battery design toward the rack itself, closer to the GPU load, rather than relying solely on a centralized facility UPS; the same density shift that drove that architectural change is what makes the checkpoint-specific ride-through calculation in this article necessary in the first place. A rack pulling 80-130kW concentrates far more at-risk compute state per square meter than the server racks generic UPS sizing conventions were built around.

Where Orchestration Has to Meet the Power Layer

None of this works without the compute orchestration layer actually listening for the power event. Job schedulers covered in our guides to Slurm GPU cluster scheduling and Kubernetes GPU orchestration need a hook into the facility's UPS-on-battery signal, typically via SNMP trap or a building management system integration, that triggers an emergency checkpoint command the moment utility power is lost rather than waiting for a human operator or a scheduled checkpoint interval to catch up. Designing this handoff, and testing it under a real simulated outage rather than assuming it works, is the operational half of the problem; the UPS runtime calculation is only the other half.

Where This Meets Generator Transfer Time

For a facility with generator backup, the UPS ride-through requirement is not sized against total outage duration, only against the gap until the automatic transfer switch completes the handover to generator power, typically on the order of 10 to 30 seconds for the transfer itself plus generator start time. The checkpoint-write math above still applies during that gap; a generator does not eliminate the need to size ride-through around the checkpoint, because the cluster still rides on battery for the full detection-to-transfer window regardless of how reliable the generator is once it picks up load. This is a narrower, more specific version of the general redundancy planning covered in our guide to data center power redundancy.

Frequently Asked Questions

How much UPS runtime does a GPU training cluster need?

Enough to complete an emergency checkpoint write of the model state currently in GPU memory to persistent storage, plus the time for the orchestration layer to detect the outage and trigger that write, plus a safety margin. For large models this is commonly in the range of several minutes rather than the 5 to 15 minutes a generic facility UPS sizing guideline assumes, because the checkpoint size and the storage write bandwidth available, not generic load assumptions, set the real number.

Is this the same as thermal ride-through?

No. UPS ride-through, also called electrical ride-through, is a storage problem: a UPS holds a known quantity of energy and the load draws it at a known rate, so the runtime is calculable. Thermal ride-through is the separate question of how long equipment survives after cooling stops before it overheats, governed by heat load and air or water volume rather than battery capacity. A training cluster needs both addressed, but they are sized differently and protect against different failure modes.

What happens if the UPS runs out before the checkpoint finishes writing?

An incomplete checkpoint write is generally unusable, since most training frameworks expect a checkpoint file to be fully and correctly written before they will load it. A GPU node losing power mid-write typically means the entire interval of training progress since the last complete, verified checkpoint is lost, which is why checkpoint frequency and ride-through time have to be sized together rather than independently.

Does a bigger UPS fix this problem?

Only partially. More runtime helps, but the harder constraint is usually storage write bandwidth: if the checkpoint storage tier cannot absorb the data fast enough, additional battery minutes buy proportionally less benefit than they would for a facility load that is not bottlenecked on write throughput. Sizing checkpoint storage bandwidth and UPS runtime together, rather than treating UPS capacity as the only lever, is what actually shortens the time an emergency checkpoint takes.

Power Architecture Designed Around the GPU Cluster, Not Around It

Rax Data & Energy builds colocation facilities on power infrastructure engineered for the density and ride-through demands of AI training workloads, not retrofitted from general-purpose IT assumptions.

Contact Us Rax Data