A GPU cluster arrives as a delivery of servers, and it is tempting to accept it the way you would accept servers. Each machine powers on, each one passes a stress test, the inventory matches the purchase order, and the deployment is signed off. Per-server burn-in is genuinely valuable work, and our guide to GPU server lifecycle management covers how to run it properly.
It is also the wrong unit of acceptance. What you bought was not a set of servers that each work. It was a machine made of servers, and the thing that makes it a machine is the fabric between them. Nothing in a per-server test touches that.
The slow node problem
Distributed training is synchronous. At the end of each step the ranks exchange gradients, and no rank proceeds until that exchange has completed across all of them. The consequence is unforgiving: the slowest participant sets the pace for the entire job, every step, for as long as the job runs.
That changes what a small defect costs. A node running slightly behind its peers does not reduce cluster output by its own share. It pulls every other node down to its rate, so the loss is that margin multiplied across the whole cluster. The same is true of a single link that negotiated below its rated speed, or one that is carrying errors and retransmitting.
These are exactly the defects that per-server testing cannot see. A GPU that throttles slightly earlier than its siblings passes burn-in. A transceiver that trained down a speed grade reports a healthy link. A cable plugged into the wrong port produces a topology that works but routes badly. Each of them is invisible until the cluster is asked to work as one thing, and by then the meter is usually running.
Three layers worth testing
Inventory and health. Before anything else, confirm that what is installed is what was specified and that it agrees with itself. Every GPU present and enumerated. Firmware at a consistent version across the fleet rather than whatever shipped. Every fabric link up at its rated speed, not a grade below. Error counters at zero rather than merely low. This layer is quick and it catches the embarrassing problems early, while somebody can still swap a transceiver without it being a change request.
Consistency matters as much as absolute values here. A cluster where most nodes run one firmware revision and a handful run another is a cluster that will behave inconsistently later, and tracking that down during a production run is far more expensive than fixing it now.
The fabric under collective load. This is the layer most often skipped and the one that justifies the exercise. Point-to-point tests between two machines will look fine on a fabric that performs badly in aggregate, because they never create the contention that real jobs create.
Test the collective operations that training actually uses, across the full set of nodes, at the message sizes the workload will generate. Then compare the result against the vendor's reference figure for that exact topology and GPU generation. The absolute number is meaningless without that reference, which is why it is worth obtaining before the test rather than afterward, when there is an incentive to decide that whatever you measured is normal. Whether the underlying fabric is InfiniBand or Ethernet changes the tooling and the expected numbers but not the principle; our comparison of InfiniBand and Ethernet RoCE for AI clusters covers what each is expected to deliver.
A representative workload, sustained. Finally, run something that resembles the real job, at full cluster scale, for long enough that the hall reaches thermal steady state and stays there. The question this answers is not what peak throughput the cluster can touch. It is whether the throughput at the end of a long run matches the throughput at the beginning.
Short benchmarks flatter hardware because nothing has heated up. Cooling behavior, power delivery under sustained draw and clock behavior at temperature only appear after the system has been held at load, which is particularly relevant in the Gulf where ambient conditions make the cooling design work harder; our guide to liquid cooling in UAE data centers covers why that design choice matters at density.
Find the outliers, not just the average
The most useful output of acceptance testing is not a headline number. It is a per-node distribution.
A cluster average hides precisely the defect you are looking for, because one node a few percent slow barely moves the mean across a large fleet. Plot the nodes individually and the outlier is obvious. Run the same test with the suspect node excluded and the question answers itself: if the cluster speeds up meaningfully without it, you have found your problem.
Keep the results. A per-node baseline taken at acceptance, while the hardware is new and nothing has been changed, is the reference you will want months later when somebody reports that training has got slower. Without it you are comparing today's measurement against an impression.
Agree the bar before the hardware lands
The commercial half of this matters as much as the technical half. Acceptance criteria written after a disappointing test result are a negotiation. Written before the hardware arrives, they are a specification.
What is worth settling in advance: which tests will be run and with which tools, what result constitutes a pass for each, how long the acceptance window lasts, what happens if a node fails, and when billing starts relative to acceptance. That last point is the one that quietly costs the most. If the meter starts at delivery and acceptance takes weeks, you are paying for capacity you have not yet confirmed works. Our piece on contracted capacity versus actual draw covers the related trap of paying against a commitment rather than consumption.
It is also worth agreeing that both parties run the same scripts and look at the same output. Most acceptance disputes are not about whether the cluster is slow. They are about two measurements taken differently, and they evaporate when there is one agreed method.
Re-test after anything structural changes
Acceptance is not a one-off ceremony. A fabric that passed in its delivered configuration is not guaranteed to pass after nodes are added, cabling is rearranged, or firmware is rolled forward across the fleet. Those are exactly the changes that reintroduce a mismatched link speed or a version skew.
The health and fabric layers are quick enough to repeat on a schedule and after any structural change, and they are what will tell you that a gradual slowdown is a specific node rather than a general impression. Keeping that habit also makes an eventual move far less painful, since you have evidence of what the cluster did at its best; the practicalities are covered in our guide to GPU workload migration and exit planning.
Operators building or taking GPU capacity in the region can see what the local market looks like in our guide to AI and GPU colocation in Dubai and Abu Dhabi. Whoever runs the facility, the principle holds: accept the cluster, not the servers.
Frequently Asked Questions
Is server burn-in not enough?
Burn-in is necessary and not sufficient. It stresses each machine on its own and catches infant mortality, thermal problems and power draw outside expectation. It tells you nothing about how the machines perform together, because it never exercises the fabric between them. A cluster where every server passed burn-in individually can still deliver poor collective bandwidth.
Why does one slow node matter so much?
Because synchronous distributed training waits. At each step the job cannot proceed until every rank has finished and the collective operation has completed, so the slowest participant sets the pace for all of them. A node running a few percent behind does not cost a few percent on that node, it costs that margin across the entire cluster for the whole duration of the job.
What should acceptance actually measure?
Three layers. That every device is present, healthy and at the expected firmware and link speed. That the fabric achieves the collective bandwidth the topology should deliver, tested across the patterns real jobs use rather than point to point only. And that a representative workload sustains its expected throughput over a long enough run to expose thermal behavior. The target for each is the vendor reference result for that exact configuration, not a general number.
How long should acceptance run?
Long enough to reach thermal steady state and then hold there, which is considerably longer than a benchmark that reports a peak. Short runs flatter a cluster because nothing has heated up yet. The useful question is whether throughput at the end of a sustained run matches throughput at the start.
Who should run the test?
Whoever is accountable for the result, with the provider present. The point is not to catch anybody out. It is to make sure both parties are looking at the same numbers from the same scripts, so that a disagreement later is about a measurement rather than about whose tool was right.
A Real Energisation Date, Not a Brochure
Rax operates where transmission investment has run ahead of load, and we are direct about staged capacity and real dates. Talk to our team about sizing a deployment against a timeline.
Contact Rax Data & Energy