Modal, 8× H100 serverless

This evaluation comprises three full-suite runs in June and July 2026 and two 30-placement studies on 16 and 17 August 2026. The August work was designed to measure placement reuse, host-transfer classes, and the variability of a corrected training probe. Each placement requested Modal's gpu="H100:8" configuration in a fresh container; stable GPU UUID sets were used to distinguish physical hosts.

The larger study materially changes the earlier report. A 2.1× cost-per-token claim and a 1.7 GB/s device-to-host result were produced by superseded probes and are withdrawn. With the corrected timing protocol, the training probe's sample p90-to-p10 throughput ratio was 1.27× across 30 placements. Separately, NVIDIA's nvbandwidth measured an approximately 1.95× host-to-device difference between placements presenting PCIe Gen5 and Gen4.

Executive assessment

Service model and provisioning observations

Modal accepts a container and command, assigns GPUs, runs the function, and destroys the container when it returns. The initial image build for CUDA, PyTorch, nccl-tests, and the suite took several minutes; subsequent invocations reused the image cache. In a separate one-GPU timing observation with the image already built, a cold function call took 7.7 seconds round trip and an immediate warm call took 0.19 seconds. Those measurements do not estimate 8× H100 placement latency.

The default PyTorch installation initially selected a CUDA build newer than the host driver supported. The study image therefore pinned a driver-compatible cu126 build. Because the ephemeral filesystem disappears with the function, evidence had to be written explicitly to a Modal Volume. These are reproducible integration requirements, not GPU performance findings.

Experimental design

The June and July phase executed sixtytwo test --full --json three times on 8× H100 placements. It measured CUDA functionality, matrix multiply, device memory, NVLink peer-to-peer, NCCL collectives, sustained load, and container-visible telemetry. Host DIMM, IPMI, NUMA, host ECC, DCGM, InfiniBand, and physical-fabric checks were unavailable or inapplicable in the serverless container, leaving approximately 41% suite coverage.

The August phase comprised two sequential 30-placement studies. Each invocation used max_inputs=1, forcing a new container, and recorded the sorted set of eight GPU UUIDs as a host fingerprint. Calls were sequential because the operational question was whether a user who resubmits receives a different machine. Host identity is inferred from stable GPU UUIDs; it is not a documented placement identifier or a guarantee about future platform behavior.

The 16 August study supplied valid host identity, PCIe link, and nvbandwidth measurements, but its training probe was invalid: the model memorized a fixed batch during warmup and timed a zero-gradient workload. On 17 August the probe was rerun after drawing fresh inputs and independent targets, moving device-to-host transfers and scalar synchronization outside the timed region, adding five warmup steps and five timed trials, selecting the median trial, and applying a measurement-noise floor. Only the 17 August throughput values are used below.

Placement and retry behavior

Pool concentration changed between days. In the second study, two hosts received 25 of 30 placements: one received 12 and the other 13. The first study spread 30 placements across 10 hosts, with the most frequent host receiving six. These data reject an assumption that successive containers are independent physical-host draws under the tested account and schedule. They do not estimate a universal host-repeat probability because available capacity and scheduler state can change over time.

A repeated host can still produce a different measurement. In the corrected study, the worst observed within-host throughput ratio was 1.19×, while the ratio between the largest and smallest host-mean throughputs was 1.30×. These are descriptive extreme ratios, not variance components. A formal decomposition would require more balanced repetitions per host and a hierarchical model. The present data establish that run-to-run variation is material relative to between-host variation, making a single measurement and a single retry an uncertain selection policy.

Corrected training-probe results

The maximum-to-minimum ratio was 1.38× and the p90-to-p10 ratio was 1.27×. Because 30 placements collapsed onto seven hosts, these observations are clustered and are not 30 independent fleet samples. We report empirical sample quantiles without a binomial or normal-theory confidence interval.

The corrected straggler rule returned 5 warnings in 30 placements, compared with 18 in 30 under the invalid probe. Residual instability remains: one physical host changed by 22.3 percentage points in measured straggler across repeat placements. The warning rate should therefore not be interpreted as a 16.7% defective-host rate. The probe is useful as a placement-level diagnostic, but it has not been validated as a fleet classifier.

Two Gen5 hosts dominated the corrected sample yet differed by approximately 1.17× in throughput. PCIe generation therefore explains some, but not all, placement variation. Host CPU allocation, clock policy, memory topology, scheduler interference, and unobserved configuration are plausible contributors; this experiment does not distinguish them.

PCIe link class and independent transfer measurement

The first placement study used NVIDIA's nvbandwidth as an independent transfer measurement. Within each run the reported per-GPU coefficient of variation rounded to 0.00.

The measured H2D ratio is approximately 1.95×. This supports a real transfer-class difference rather than the earlier suite probe alone. It does not imply that an end-to-end application will run 1.95× faster on Gen5. Workload impact depends on the fraction of time spent moving data across the host-device boundary; device-resident compute can be largely insensitive, while input pipelines or parameter offload can be transfer-bound.

Across the two studies, 7 of 60 placements presented Gen4, but those placements were associated with a small number of repeated hosts and were concentrated on one day. The 7/60 proportion is therefore a description of these submissions, not a fleet incidence estimate.

Measurement correction and withdrawn claims

An earlier version of this report stated that the same training job cost 2.1× more per token on the worst of three draws than on the best. All three original runs predated the July timing standard of fixed warmup, multiple trials, median selection, and an explicit noise floor. More importantly, the short training probe reused a small fixed batch that the model could memorize, so timed steps could execute after the gradient had collapsed. Scalar reads inside the timed loop also introduced host synchronization.

The August validation exposed these defects, and the 17 August study remeasured after correction. The defensible replacement is a 1.27× empirical p90/p10 throughput ratio and 1.38× extreme ratio in the corrected 30-placement sample. Because billing was not reconciled per placement and the probe is synthetic, this report does not translate that spread into a universal cost-per-token claim.

The original 22 June 8× run also reported 1.7 GB/s D2H. That single-shot value did not reproduce in a same-day one-GPU run or either later full-suite run. In the 16 August study, the lowest per-GPU D2H result from nvbandwidth was about 28.5 GB/s across 30 placements on 10 hosts. The 1.7 GB/s result came from the superseded single-shot method and is retired as a measurement artifact.

This correction is substantive: the invalid values remain in archived raw reports for auditability, but they are excluded from current conclusions.

Findings retained from the full-suite runs

Across the three June and July runs, matrix multiply measured 719–771 TFLOPS, device-memory bandwidth was approximately 2,790 GB/s, minimum NVLink peer-to-peer bandwidth was 358–364 GB/s, and each 300-second sustained burn completed with zero throttle samples. These results support the bounded conclusion that the sampled GPUs and intra-node GPU links were functional under the tested loads. Absolute timing values predate the revised protocol and may shift modestly under remeasurement.

The original full runs reported the NVLink error counter as exactly 3,672 on every placement despite evidence that more than one host was sampled. A nonzero lifetime counter can include corrected replay and recovery events, and the invariant value across different hosts indicates that the exposed field was not a useful job-attributable error measure. It is treated as uninterpretable, not as evidence of damage caused by the workload.

NCCL collectives in the archived runs were below raw NVLink peer-to-peer bandwidth, as expected because collective algorithms include protocol and topology overhead. Without a workload-specific expected curve, the difference does not by itself establish defective collective performance.

Operational implications

For reproducible deployment, pin the framework build to the driver-supported CUDA version and write evidence or outputs to a persistent Volume before the function exits. Where host transfer matters, record the presented PCIe generation and run an independent bidirectional transfer check. Where training throughput matters, use the corrected multi-trial probe or the intended workload rather than the archived short-probe baseline.

Retries should be host-aware. A sorted GPU-UUID fingerprint allowed this study to identify repeat placements, but most resubmissions still returned a previously seen machine. A prospective retry policy should define an acceptance metric, cap attempts and cost, record the host fingerprint, and compare against a no-retry control. The present study measured the ingredients of such a policy; it did not randomize or validate the policy itself.

Scope and limitations

• The two placement studies cover one account, sequential submissions, and two adjacent days. They are not random fleet samples or service-level estimates.
• Physical hosts were inferred from GPU UUID sets. UUID stability was observed but is not asserted as a provider contract.
• Placements were clustered on a few hosts, so nominal placement count overstates the number of independent machines.
• The corrected training probe is synthetic and short. It does not represent every model, input pipeline, collective pattern, precision, or synchronization frequency.
• The within-host and between-host ratios are descriptive extremes, not estimates from a balanced random-effects design.
• nvbandwidth ran with CPU-affinity selection disabled because the sandbox blocked its NVML affinity query. It is an independent magnitude check, not a tuned platform peak.
• Full-suite coverage was approximately 41% because host, DCGM, InfiniBand, and physical-fabric evidence was unavailable inside the container.
• No security, tenant-isolation, availability, interruption, multi-day endurance, or invoice-reconciliation study is implied here.

Reproducibility and evidence

The general method is described in How we benchmark providers, and the harness is distributed as sixtytwo-cli. The full-suite image used nvidia/cuda:12.6.2-devel-ubuntu22.04, nccl-tests, a driver-matched PyTorch cu126 build, and sixtytwo-cli[gpu]. The August placement records retain the host fingerprint, link generation, per-trial timings, corrected-probe validity fields, and nvbandwidth output for every submission.

Under Modal's sandbox, nvbandwidth required --disableAffinity, and its build required an explicit CUDA architecture list because automatic detection included a Blackwell target unsupported by CUDA 12.6. These conditions are part of the reproduction protocol.