Prime Intellect, 8x H100 SXM5
Here we cover our experience with one 8x H100 SXM5 node on Prime Intellect.
Getting the node
After attaching an SSH key via console, we were able to launch the 8x H100 SXM5 SKU (in stock) at $11.92/hr on spot and $14.40/hr on demand, and were able to reach over SSH within a reasonable timeframe after launch.
We got a container on a shared host, and ran the subset of our tests applicable for this setting (in contrast to for instance bare metal instances).
What we found under load
The headline numbers for the node looked fine: peak matrix-multiply throughput reached about 750 TFLOPS against a 480 reference, the isolated per-GPU compute kernel was even across all eight cards at a 3.4% spread, NVLink and GPU-to-GPU bandwidth were healthy, and local storage was fast. Under load, three problems showed up, each on a different GPU. GPU 0 ran a training step in 31 ms against 5.4 ms for the fastest GPU, about 5.7x slower. Its isolated compute was normal, so this is the host-to-device path rather than the GPU itself, but one slow rank still sets the pace for a whole synchronized job. On GPU 3 the PCIe link under load managed 12.6 GB/s host to device, about 20% of the Gen5 x16 ceiling, while GPUs 4 through 7 were lopsided, with normal host-to-device but collapsed device-to-host transfers. This was a container on a shared host, so some of how extreme the PCIe path looks is how virtualized PCIe presents to the probe. GPU 2 cooled with a time constant well off the median and a non-exponential decay, which points at a thermal-interface or airflow problem rather than just a warm room.
Run to run
We had very little variation between our two runs. We tested the same box a day apart and the numbers came back almost unchanged.
| Measurement | Run 1 | Run 2 |
|---|---|---|
| PCIe under load | 12.6 / 13.3 GB/s (20% of ceiling) | 12.6 / 13.3 GB/s (20%) |
| GPU memory bandwidth | 2428 GB/s (72% of theoretical peak) | 2425 GB/s (72%) |
| Training-step straggler, GPU 0 | 30.8 ms vs 5.4 ms | 31.4 ms vs 5.4 ms |
| Isolated-kernel spread | 3.4% | 3.4% |
| NCCL all-gather | 248 GB/s of 400 expected | 248 GB/s |
| ECC uncorrectable, aggregate | 2 | 2 |
| Peak matmul | ~750 TFLOPS | ~751 TFLOPS |
Running it yourself
The method is described in How we benchmark providers, and the harness is on PyPI as sixtytwo-cli. The raw reports for both runs are published, so the numbers here are checkable rather than asserted.