Provisioning reliability: RunPod and Vast

This observational study compares whether paid GPU instances became reachable over SSH within a fixed operational deadline. Across 234 scored starts on RunPod and Vast, the largest and best-resolved difference occurred on RTX 3090 capacity: 1 of 38 RunPod Community instances missed the deadline, compared with 10 of 40 Vast verified instances and 25 of 40 Vast unverified instances. Smaller H100 and A100 comparisons had the same direction but were not individually conclusive.

The endpoint is deliberately narrow. A successful trial established SSH reachability; it did not establish CUDA readiness, numerical correctness, sustained GPU health, or job completion. The results therefore characterize the providers' provisioning and handoff paths during the campaign, not their entire services or fleets.

Executive assessment

Study design and outcome definition

Each start was treated as a structured trial. The harness recorded the request time, polled the assigned endpoint using the trial's SSH credential, and terminated the instance when SSH succeeded or the deadline expired. Consumer-tier trials used a six-minute deadline; H100 and A100 trials used eight minutes. Instances that reached a paid or running allocation state but never accepted SSH before the deadline were classified as provisioning failures. Requests rejected before allocation were excluded from this rate because they measure availability or API admission rather than failed delivery of a billed instance.

Boot time is reported only among successful trials. It is therefore a conditional statistic: a tier can have a short median among successes while still failing many starts. Failure-adjusted time is addressed separately in the model below.

The provider, tier, and GPU groups were selected rather than randomized. Provider allocation selected the particular host. The comparisons include the delivery architecture and image behavior encountered by a normal account; they do not isolate hardware, image distribution, host reputation, geography, or time of day as independent causes.

Primary results: consumer tiers

Intervals are two-sided 95% Wilson score intervals for the binomial failure proportion. They quantify sampling uncertainty conditional on the observed campaign; they do not account for dependence among starts placed on the same host or during the same capacity event.

The same-SKU RTX 3090 comparison is the cleanest contrast in the study. Exploratory two-sided Fisher exact tests give p = 0.007 for RunPod Community versus Vast verified and p < 0.00000001 for RunPod Community versus Vast unverified. These values were not adjusted for multiple comparisons and should be read alongside the effect sizes and intervals, not as fleet-wide guarantees.

All 35 Vast RTX 3090 deadline failures reached a started and billed state but never became reachable during the six-minute window. The campaign logs are consistent with slow boot or image delivery, but they do not identify the cause. RunPod used a cached image path while the sampled Vast marketplace hosts pulled the image independently; because image size and cache state were not randomized in this campaign, the share attributable to image distribution is unknown.

Same-SKU datacenter comparisons

The observed H100 failure proportion was 4.75 times higher on Vast verified, and the A100 samples recorded two failures on Vast and none on RunPod. The uncertainty is substantial: two-sided Fisher exact tests give p = 0.18 for H100 and p = 0.49 for A100. Those samples do not reject equal failure probabilities at conventional thresholds. They are useful as directionally consistent observations and as inputs to a larger replication, not as standalone proof.

Premium-host records included both never-reachable starts and instances that exited after launch, including one exit within 18 seconds. That observation rules out a single explanation based only on slow image pulls, but the study was not instrumented to assign root cause among host failure, container runtime, image delivery, or provider control-plane behavior.

What the published host score did and did not show

For each Vast premium trial, the harness captured the host's published reliability score at rental time. Failed placements had scores between 0.971 and 0.998, a range also occupied by successful placements. This establishes a bounded claim: using a threshold inside that range would not have guaranteed a successful handoff in this sample.

Range overlap alone does not establish that the score has no predictive value. A calibration curve, discrimination statistic, and prospective threshold evaluation would require the complete score distribution, repeated observations per host, and a held-out policy. The result is therefore evidence against treating the displayed score as a readiness certificate, not a complete validation of the scoring system.

Conditional retry and cost model

Let q be the observed deadline-failure proportion and p = 1 - q. Under stationary, independent retries, expected attempts per success are 1 / p, expected failed cutoff time is (q / p) × deadline, and the number of simultaneous attempts giving at least 95% modeled success is the smallest n for which 1 - q^n is at least 0.95.

Using the point estimates, a sequential Vast unverified retry process takes about 600 seconds of failed windows plus a 38-second median successful boot, compared with an 18-second RunPod Community median: 35.4 times longer to first SSH. This is model arithmetic, not a directly observed paired retry experiment. Correlated host, image, regional, or capacity failures would make the independence-based parallel estimates optimistic. Conversely, a policy that excludes previously failed hosts could improve them. Zero failures in 38 RunPod Secure trials also does not imply a true failure probability of zero; its Wilson upper bound is 9.2%.

Operational implications

For the exact endpoint studied here, customer automation should gate handoff on an authenticated SSH probe rather than a provider running state. For production GPU readiness, the gate should be stronger: confirm the requested GPU identity, load the driver, allocate device memory, and complete a numerically checked CUDA operation before admitting work.

The Vast observations motivate prospective evaluation of a boot-to-SSH admission check with a holdout group, an explicit expiry interval, and host-aware replacement. Applied retrospectively, such a check would have prevented the 10 failed verified RTX 3090 placements from being handed off as ready; the campaign did not measure whether replacement placements would then succeed. Broad parallel launching should remain an explicit policy experiment because it increases billed starts and relies on weak independence assumptions.

Scope and limitations

• The 234 starts are a campaign snapshot, not a random sample of either provider's fleet or an SLA estimate.
• The primary endpoint is SSH reachability. GPU correctness, CUDA readiness, throughput, thermals, storage, networking, recovery, and job completion were outside this comparison.
• Successful boot medians exclude deadline failures and have no tail percentiles in this summary.
• Repeated allocation to the same host and common image or capacity events can narrow the effective sample size relative to the nominal denominators.
• Provider, tier, image behavior, price, geography, and time were not jointly randomized, so causal attribution to any single layer is not supported.
• The Wilson intervals model independent Bernoulli trials. The Fisher tests are exploratory and unadjusted.
• Retry counts, dead time, and 95% parallelism are plug-in projections from observed rates. No retry policy was prospectively evaluated in this campaign.

Reproducibility and evidence

The trial harness records timestamps, allocation outcome, SSH result, failure class, host score when exposed, and cleanup status. The analysis separates pre-allocation launch failures from paid-instance handoff failures and reports per-host outcomes. The detailed trial table, failure-mode breakdown, and score analysis are available with the full report.