Best GPUs for RAG Pipelines

A retrieval-augmented generation pipeline is not one model but at least two, and they have different hardware profiles. The embedding model is small, runs over large batches during ingestion, and is throughput bound. The generator is the large language model that consumes retrieved context, and it is latency and memory bound. Sizing one GPU for both usually overprovisions for one job and underprovisions for the other.

Embedding is the easier half. Common embedding models are a few hundred million parameters and fit in a handful of gigabytes, so an entry or mid-tier GPU saturates on batch size long before it runs out of memory. Ingestion is also a batch job rather than a serving path, which makes it a good candidate for spot capacity and for scaling to zero between corpus updates.

Generation is where the cost sits, and RAG makes it harder than plain chat in one specific way: retrieved context makes prompts long. Prefill cost scales with prompt length, and the KV cache grows with it too, so a RAG deployment needs materially more memory headroom per concurrent request than the same model serving short prompts. Size for your retrieval budget — top-k times chunk size — not for the question alone.

Reranking sits between the two and is often forgotten. A cross-encoder reranker runs over every retrieved candidate rather than once per query, so its throughput requirement scales with your candidate count. If your pipeline reranks, budget for it as a third model rather than as a rounding error on the generator.