Cloud GPU Pricing Comparison

Spot instances offer 60-91% discounts compared to on-demand pricing, but can be interrupted with 30 seconds to 2 minutes notice when capacity is needed. On-demand instances provide guaranteed availability and persistent data storage at a premium. Spot is ideal for training workloads with checkpointing, while on-demand is better for production and critical workloads that can't be interrupted.

VRAM requirements depend on your model size and task. Small models (basic CNNs) need 4-8GB, medium models (BERT, ResNet-50) need 12-16GB, and large language models (70B+ parameters) require 24GB or multiple GPUs. Training typically requires 2-4x more VRAM than inference. You can reduce requirements using quantization, gradient checkpointing, or smaller batch sizes.

H100 (Hopper architecture) offers 3-6x faster LLM training than A100 and remains widely available, though newer Blackwell-generation GPUs (B200, GB300) now lead on raw performance. A100 (Ampere) still provides solid AI performance for research and mid-size workloads. RTX 4090 is a consumer GPU ideal for prototyping and small-scale ML. H100 and A100 have enterprise features like ECC memory and NVLink for multi-GPU scaling, while RTX 4090 is more cost-effective for individual workloads — check current pricing above to compare rates across providers.

Hidden costs can add substantially to your total spend beyond the advertised hourly rate. Watch for: data egress fees, storage costs for datasets and checkpoints, idle GPU time (teams often waste a large share of spend on unused instances), cross-region transfer fees, and premium GPU surcharges. Always calculate total cost of ownership, not just hourly rates.

Training requires powerful GPUs (A100, H100) with high VRAM and runs for hours or days, making it suitable for spot instances. Inference uses lighter GPUs (L4, A10, RTX series) with less VRAM and runs continuously, requiring on-demand reliability. Inference only loads 2 consecutive layers at a time, making it more memory-efficient per request.

Hyperscalers offer global availability, enterprise compliance, and integrated ecosystems, but cost 2-3x more with complex setup. Specialized providers like Runpod, Lambda, and CoreWeave are 40-60% cheaper with simpler setup but smaller ecosystems. Choose hyperscalers for enterprise compliance needs; choose specialized providers for cost-efficient ML workloads with simpler requirements.

Hourly rates vary significantly by provider, region, GPU generation, and pricing model (on-demand vs. spot vs. reserved) — see the live comparison table above for current rates across every tracked provider. As a rule of thumb, the newest data-center GPUs (H100, H200, B200, GB300) command the highest hourly rates, A100 80GB sits in between, and consumer-class GPUs (RTX 4090 and similar) are the most affordable option for smaller workloads.

Key strategies: Right-size instances (use L4/A10 instead of H100 when sufficient), use spot instances with checkpointing for training (60-91% savings), optimize models with quantization and pruning, batch inference requests to improve utilization from 20-30% to 70-80%, auto-shutdown idle instances to eliminate 30-50% waste, and choose specialized providers over hyperscalers for 40-60% savings on comparable hardware.