Quick answer: The RTX 4090 (24GB) is the best overall GPU for local RAG in 2026. Its 24GB VRAM handles embedding models and LLM inference simultaneously, which is the key bottleneck in RAG pipelines.
NVIDIA GeForce RTX 4090
24GB GDDR6X24GB VRAM runs 13B model plus embedding plus 16K context window simultaneously — the RAG sweet spot.
Affiliate links — we may earn a commission at no extra cost to you. Amazon ships globally; Shopee SG covers Singapore & ASEAN.
Why RAG needs GPU power
A RAG pipeline has two GPU-intensive stages — and if you are building an autonomous agent setup that chains multiple RAG calls together, the VRAM requirements compound further:
- Embedding — converting documents into vectors (batch processing, runs once per document)
- LLM inference — generating answers using retrieved context (runs every query)
The embedding stage is compute-bound and benefits from raw TFLOPS. The inference stage is memory-bandwidth-bound and needs sufficient VRAM to hold the model plus the retrieved context in the prompt.
The critical constraint: your LLM needs enough VRAM for the base model PLUS a large context window. RAG prompts routinely hit 4K-16K tokens with retrieved chunks, which increases KV cache VRAM usage significantly.
VRAM requirements for RAG
| Component | VRAM Usage | Notes |
|---|---|---|
| Embedding model (e5-large, BGE) | 0.5-1.5GB | Small, usually not the bottleneck |
| LLM 7B (Q4_K_M) | ~4.5GB | Fits on most cards |
| LLM 14B, GQA (Q4_K_M) | ~9GB | Qwen 2.5 14B and similar |
| LLM 34B, GQA (Q4_K_M) | ~20GB | Yi 34B, CodeLlama 34B |
| KV cache — 14B at 8K | 1.5GB | 0.19MB per token |
| KV cache — 14B at 16K | 3GB | RAG prompts do reach this |
| KV cache — 34B at 16K | 3.8GB | 0.23MB per token |
| Runtime overhead | 1-1.5GB | Ollama or llama.cpp |
| Total for 14B RAG at 16K | ~14.5GB | a 16GB card holds it |
| Total for 34B RAG at 16K | ~26GB | 32GB, or two 24GB cards |
The cache rows are the ones people get wrong, usually by budgeting several times too much. Grouped-query attention shares keys and values across query heads, so a 14B costs 0.19MB per token and 16K of context is 3GB, not the 8GB the old rule of thumb suggests. The exception is worth knowing: Llama 2 13B is still multi-head attention at 0.78MB per token, which makes 8K of context 6.3GB on its own — more than a 70B costs. If your stack is on a pre-2024 model, budget four times what a modern one of the same size needs.
VRAM capacity memory bandwidth Specs are manufacturer figures. Bar lengths are scaled independently per metric.
Best GPUs for RAG ranked
| GPU | VRAM | Bandwidth | RAG 13B | RAG 34B | Price |
|---|---|---|---|---|---|
| RTX 5090 | 32GB | 1,792 GB/s | Excellent | Good | ~$4,900 |
| RTX 4090 | 24GB | 1,008 GB/s | Excellent | Tight | ~$2,200 |
| RTX 5080 | 16GB | 960 GB/s | Good | No | ~$1,400 |
| RTX 5070 Ti | 16GB | 896 GB/s | Good | No | ~$1,050 |
| RTX 4070 Ti Super | 16GB | 672 GB/s | Acceptable | No | ~$800 |
| RTX 3090 (used) | 24GB | 936 GB/s | Excellent | Tight | ~$820 |
Which GPU should you buy?
If your RAG pipeline uses 7B-13B models with moderate context (up to 8K tokens), a 16GB card like the RTX 5080 ($1,400) handles it well — run the embedding model on CPU to save VRAM for the LLM. If you need 13B with 16K context or want headroom for growth, the RTX 4090 ($2,200) is the sweet spot — 24GB fits the model, embedding model, and long-context KV cache simultaneously. If you are building RAG around 34B models, the RTX 5090 ($4,900) is the only single card with enough VRAM.
Common mistakes to avoid
- Loading the embedding model on GPU alongside the LLM. Embedding models like BGE or e5-large run fast enough on CPU. Keeping them off the GPU frees 1-2GB of VRAM for longer context windows, which matters more for RAG quality.
- Prioritizing quantization quality over context length. In RAG, the retrieved context is what makes the answer good. A 13B Q4 model with 16K context produces better results than a 13B Q6 model limited to 4K context.
- Underestimating KV cache VRAM for long contexts. RAG prompts with retrieved chunks routinely hit 8K-16K tokens. At 16K context, KV cache alone can consume 4-8GB — plan for this on top of model weights.
Our top picks
Best overall: RTX 4090
The RTX 4090 hits the sweet spot for RAG. With 24GB VRAM, you can run a 13B model at Q6_K with a 16K context window and still have room for the embedding model. The 1,008 GB/s bandwidth delivers fast token generation even with large context windows.
For 34B RAG, the 4090 works at Q4 quantization with shorter context windows (4K-8K), but gets tight above that.
Best for 34B+ RAG: RTX 5090
If you are building a RAG system around 34B models like CodeLlama 34B or Yi 34B, the RTX 5090’s 32GB VRAM gives you the headroom that the 4090 lacks. The 1,792 GB/s bandwidth is also noticeably faster for long-context generation.
Best value: RTX 3090 (used)
A used RTX 3090 at $820 gives you 24GB VRAM and 936 GB/s bandwidth — nearly matching the 4090 for RAG capacity at roughly a third of the price. The trade-off is higher power draw (350W) and older architecture, but for a dedicated RAG server, it is hard to beat.
Check NVIDIA GeForce RTX 4090 on Amazon→Buy on Shopee SG→ Check NVIDIA GeForce RTX 5090 on Amazon→Buy on Shopee SG→NVIDIA GeForce RTX 3090
24GB GDDR6X24GB VRAM at ~$820 used — nearly matches RTX 4090 RAG capacity at roughly a third of the price.
Affiliate links — we may earn a commission at no extra cost to you. Amazon ships globally; Shopee SG covers Singapore & ASEAN.
RAG optimization tips
Run embedding on CPU if VRAM is tight. Modern embedding models like BGE-small or e5-base run fast enough on CPU for most RAG setups. Reserve all your VRAM for the LLM.
Use smaller quantization for the LLM, not shorter context. In RAG, context quality matters more than model precision. A 13B Q4 model with 16K context produces better answers than a 13B Q6 model with 4K context.
Consider splitting stages. Embed documents in batch (overnight if needed), then run inference on a smaller card. The embedding stage is a one-time cost per document.
For more on VRAM planning, see our VRAM requirements guide. If you are on a tighter budget, check our best budget GPU for LLM recommendations. Building a pipeline specifically for document summarization rather than Q&A? Our LLM summarization GPU guide covers the context-length requirements that matter most for that task. If your RAG runs on sensitive corporate or medical data, our best GPU for private AI guide covers the air-gapped deployment angle.
For RAG, buy for VRAM first and bandwidth second. The model plus the context window must fit entirely in GPU memory, or performance falls off a cliff.
Common questions about GPUs for RAG
What GPU do I need to run RAG locally?
A 14B at Q4 with 16K of context comes to about 14.5GB once the embedding model and runtime are counted, so a 16GB card like the RTX 5080 or RTX 5070 Ti covers it. The RTX 4090’s 24GB buys headroom for longer prompts or a second model rather than being a requirement. Only 34B-class RAG genuinely needs a 32GB card like the RTX 5090.
Can the embedding model and the LLM share one GPU?
Yes, if you have the VRAM headroom. Embedding models like BGE or e5-large only need roughly 0.5-1.5GB, so a 24GB card fits them alongside a 13B model and a long context window. On tighter 16GB cards, the better move is running the embedding model on CPU — it is fast enough for most setups and frees VRAM for context.
What is the minimum VRAM for a local RAG stack?
A 7B model at Q4 with moderate context fits a 12GB card comfortably. A 14B RAG stack at 16K context lands near 14.5GB — weights, embedding model, KV cache and runtime — so 16GB is the practical minimum and 24GB is the comfortable choice. The cache itself is smaller than most guides claim: 3GB for a 14B at 16K, because grouped-query attention keeps it narrow.
Is a used RTX 3090 good enough for RAG?
Yes — it is arguably the best value pick. Its 24GB VRAM and roughly 936 GB/s bandwidth come close to RTX 4090 RAG capacity at roughly a third of the price on the used market. The trade-offs are higher power draw (350W) and an older architecture, which matter less for a dedicated RAG server.