Best GPUs for LLM Training

Training a large language model from scratch is bound by two things before it is bound by raw compute: how much of the model, its gradients and its optimizer state fit in GPU memory, and how fast gradients move between GPUs. That is why this workload lives almost exclusively on HBM-equipped datacenter parts with NVLink or Infinity Fabric interconnects, rather than on consumer cards with more attractive headline FLOPS.

A practical floor is 80 GB per GPU. Optimizer state for a model trained in mixed precision typically costs several times the parameter count in bytes, so a 7B model in a standard Adam setup already needs well over 80 GB across the node before activations are counted. Sharded training (FSDP, DeepSpeed ZeRO, tensor and pipeline parallelism) spreads that across GPUs, which makes inter-GPU bandwidth the next constraint — NVLink between GPUs in a node, and InfiniBand or equivalent between nodes.

Numeric format matters as much as capacity. Hopper introduced FP8 via the Transformer Engine and Blackwell extends it, so a training run that can use FP8 for the bulk of its matmuls sees a large step up over BF16-only hardware. Check whether your framework and model actually support the format before paying for it: an FP8-capable GPU running a BF16-only stack is just an expensive BF16 GPU.

When comparing providers, look past the hourly rate to what you are actually renting. A single-GPU instance and an 8-GPU node with NVLink are not substitutes for this workload, and interconnect quality between nodes varies widely between providers. The table below normalizes every price to per-GPU per-hour so the comparison is like-for-like; multi-GPU offerings are labelled with their GPU count.