Cloud performance reports

We ran benchmarking with our suite on GPU rentals, and report our findings for each provider. The method is open and the raw reports are published, so you can check the work.

Reports

• Prime Intellect, 8x H100 SXM5. We ran the full suite twice on the same 8x H100 spot node and found healthy silicon dragged down by one slow host path, which is all it takes to set the pace of a synchronized training job.
• Modal, 8x H100 (serverless). Three serverless runs over three weeks. One node's device-to-host path ran at 3% of ceiling and halved training throughput against the other two, and a training straggler showed up on all three. You cannot choose the node on Modal, so which one you draw decides most of the result.
• Provisioning reliability: RunPod vs Vast. We rented a few hundred instances across both providers to measure how often a paid instance never becomes usable, and found the reliability scores providers publish tell you surprisingly little about it.

The method

Every hardware report runs the same suite described in How we benchmark providers, measured against per-GPU baselines. The harness is on PyPI as sixtytwo-cli, so you can run the same tests on your own GPUs.