Quick answer: For production vLLM serving, the RTX 4090 ($2,200) offers the best throughput per dollar for models up to 13B. For 34B+ models or high-concurrency workloads, the RTX 5090 ($4,900) or multi-GPU setups are essential.
NVIDIA GeForce RTX 4090
24GB GDDR6XBest throughput-per-dollar for vLLM — 2,100 tok/s total at 32 concurrent requests for 7B-13B production serving.
Affiliate links — we may earn a commission at no extra cost to you. Amazon ships globally; Shopee SG covers Singapore & ASEAN.
Why vLLM is different from local inference
vLLM is not a chatbot runner. It is a high-throughput inference server designed for serving multiple concurrent requests. This changes what matters in a GPU:
- VRAM capacity determines the largest model you can serve and how many concurrent requests you can handle (KV cache scales with concurrency)
- Memory bandwidth directly impacts token generation speed across all requests
- Tensor parallelism lets vLLM split models across multiple GPUs with near-linear scaling — though that scaling assumes real PCIe bandwidth to each card, which is a motherboard question before it is a GPU one (dual-GPU motherboard guide)
- PagedAttention makes vLLM 2-4x more memory efficient than naive serving, but you still need enough VRAM for the model plus KV cache
Unlike Ollama which handles one request at a time, vLLM batches requests dynamically, so throughput scales with VRAM headroom. For a side-by-side breakdown of when vLLM makes sense versus Ollama or llama.cpp, see Ollama vs llama.cpp vs vLLM. If you prefer a GUI loader over a production server stack, see our text-generation-webui GPU guide.
GPU comparison for vLLM throughput
Benchmarks serving Llama 3 8B at FP16 with 32 concurrent requests:
| GPU | VRAM | Throughput (tok/s total) | Latency (P50) | Price |
|---|---|---|---|---|
| RTX 5090 | 32GB | ~2,800 tok/s | ~45ms | ~$4,900 |
| RTX 4090 | 24GB | ~2,100 tok/s | ~55ms | ~$2,200 |
| RTX 5080 | 16GB | ~1,500 tok/s | ~70ms | ~$1,400 |
| RTX 4070 Ti Super | 16GB | ~1,200 tok/s | ~85ms | ~$800 |
| 2x RTX 4090 (TP=2) | 48GB | ~3,900 tok/s | ~50ms | ~$4,400 |
Total throughput is what matters for serving, not single-request speed. The RTX 4090 delivers excellent throughput per dollar and is the workhorse of budget vLLM deployments.
Model sizing for vLLM
vLLM typically serves models at FP16 or AWQ/GPTQ 4-bit for best throughput. Unlike llama.cpp GGUF, vLLM uses GPU-native quantization formats.
| Model | FP16 VRAM | AWQ 4-bit VRAM | Min GPU (FP16) | Min GPU (AWQ) |
|---|---|---|---|---|
| Mistral 7B | ~14GB | ~4.5GB | RTX 5080 16GB | RTX 4060 Ti 16GB |
| Llama 3 8B | ~16GB | ~5GB | RTX 5080 16GB | RTX 4060 Ti 16GB |
| CodeLlama 13B | ~26GB | 7.9GB | RTX 5090 32GB | RTX 4060 Ti 16GB |
| Qwen 2.5 32B | ~64GB | ~19GB | 2x RTX 5090 | RTX 4090 24GB |
| Llama 3 70B | 141GB | 43GB | Multi-GPU | 2x RTX 4090 |
Remember to add 4-8GB overhead for KV cache depending on concurrency and context length. Higher concurrency needs more VRAM headroom.
PagedAttention and VRAM efficiency
PagedAttention is vLLM’s key innovation. It manages GPU memory for KV cache like virtual memory pages, eliminating waste from pre-allocated fixed buffers. In practice this means:
- ~2-4x more concurrent requests than naive serving with the same VRAM
- Near-zero memory waste from fragmentation
- Dynamic allocation lets you serve bursty traffic without over-provisioning
This makes VRAM even more valuable in vLLM than in single-user tools. Every extra GB of VRAM translates to more concurrent users you can serve. Novel architectures are also starting to reshape these throughput assumptions — NVIDIA’s Nemotron-TwoTower (released 2026-07-02) is the first open-weight diffusion LLM, with 60B total parameters but only 3B active per tower and a 2.42× throughput uplift, meaning the diffusion-LLM angle changes both the VRAM math and the batching model vLLM users have relied on.
Tensor parallelism: scaling across GPUs
vLLM supports tensor parallelism natively. Two RTX 4090s with TP=2 give you 48GB of combined VRAM and roughly 1.85x the throughput of a single card (not quite linear due to NVLink absence on consumer cards, which adds PCIe communication overhead).
For serious serving, dual RTX 4090s are often better than a single RTX 5090, and the price move during 2026 made the case stronger rather than weaker: two 4090s are about $4,400 against $4,900 for one 5090, so you get more total VRAM (48GB vs 32GB) and nearly double the throughput for slightly less money.
Which GPU should you buy?
If you are prototyping or testing vLLM with 7B models, the RTX 4060 Ti 16GB at $425 is enough to validate your pipeline. If you are serving 7-13B models in production with moderate concurrency, the RTX 4090 at $2,200 is the best throughput-per-dollar choice. If you need high concurrency or 34B+ models, go with dual RTX 4090s — 48GB combined VRAM with tensor parallelism beats a single RTX 5090 for serving workloads where total throughput matters more than single-request latency.
Common mistakes to avoid
- Using GGUF quantization with vLLM. vLLM uses GPU-native formats (AWQ, GPTQ), not llama.cpp’s GGUF. Using the wrong format means you cannot take advantage of PagedAttention and continuous batching.
- Underestimating KV cache VRAM. A model that fits in 20GB of VRAM still needs 4-8GB for KV cache under concurrency. Budget VRAM for your peak concurrent users, not just the model weights.
- Buying a single expensive GPU instead of two cheaper ones. For serving, two RTX 4090s with tensor parallelism outperform a single RTX 5090 in total throughput and have more combined VRAM (48GB vs 32GB).
Our recommendation
| Workload | Best GPU | Price |
|---|---|---|
| Dev/testing (7B models) | RTX 4060 Ti 16GB | ~$425 |
| Small-scale serving (7-13B) | RTX 4090 | ~$2,200 |
| Production serving (7-13B) | RTX 5090 | ~$4,900 |
| High-throughput or 34B+ | 2x RTX 4090 | ~$4,400 |
For most vLLM deployments, the RTX 4090 at $2,200 is the sweet spot. It serves 7-13B models at FP16 with excellent throughput and has enough VRAM for decent concurrency. Scale horizontally with tensor parallelism when you need more.
NVIDIA GeForce RTX 4060 Ti 16GB
16GB GDDR6Enough to prototype and validate vLLM pipelines with 7B models before committing to production hardware.
Affiliate links — we may earn a commission at no extra cost to you. Amazon ships globally; Shopee SG covers Singapore & ASEAN.
For more on how VRAM requirements scale with model size and quantization, see our VRAM requirements guide.