Quick answer: Llama 3.3 70B at Q4_K_M is a 43GB download, so it needs two 24GB GPUs — and the 48GB those give you leaves about 5GB for context, not a comfortable margin. One 32GB card is not enough at Q4. At FP16 the same model is 141GB.
Those are published file sizes from Ollama’s Llama 3.3 library, not figures derived from the parameter count. The difference matters: the usual “70B × 0.5 bytes = 35GB” arithmetic understates the real Q4_K_M file by 8GB, which is exactly the margin that decides whether a dual-24GB build works.
NVIDIA GeForce RTX 4090
24GB GDDR6XTwo RTX 4090s give 48GB combined VRAM — enough for the 43GB Q4_K_M build with room for a modest context window.
Affiliate links — we may earn a commission at no extra cost to you. Amazon ships globally; Shopee SG covers Singapore & ASEAN.
The exact numbers at every quantization level
70B models are large. The actual requirement depends heavily on how aggressively you quantize the weights. The sizes below are what Ollama publishes for llama3.3:70b, so they are the files you will actually download:
| Quantization | Published size | What holds it |
|---|---|---|
| fp16 | 141GB | Datacenter only — 2x A100 80GB or better |
| q8_0 | 75GB | 4x 24GB, or 2x 48GB workstation cards |
| q6_K | 58GB | 2x 32GB (RTX 5090 pair), or 3x 24GB |
| q5_K_M | 50GB | More than 2x 24GB holds — needs 2x 32GB or CPU offload |
| q4_K_M | 43GB | 2x 24GB (48GB) — fits, with ~5GB left for context |
| q4_K_S | 40GB | 2x 24GB, more comfortably than q4_K_M |
| q3_K_M | 34GB | 2x 24GB. Does not fit one 32GB card |
| q3_K_S | 31GB | RTX 5090 (32GB) — fits, with almost nothing spare |
| q2_K | 26GB | RTX 5090. A 24GB card does not hold this either |
Add KV cache on top of every row. A few thousand tokens of context is roughly 2-4GB at this model size, and it grows with the conversation — so a row that fits with 1GB to spare does not really fit.
Two of those lines are worth reading twice. q3_K_M at 34GB does not fit a 32GB RTX 5090, even though the next step down does; and no 24GB card holds a 70B at any quantization Ollama publishes, because even q2_K is 26GB. Single-card 70B is not a budget option, it is simply not available.
The inflection point most people care about is q4_K_M at 43GB — the lowest step that keeps quality acceptable for complex reasoning, and the reason the dual-24GB build is the standard answer.
The VRAM calculator will run these numbers for any context length you plan to use, which is the part that moves most once you are at 70B.
VRAM capacity memory bandwidth Specs are manufacturer figures. Bar lengths are scaled independently per metric.
What GPU setups actually work
fp16 (141GB)
This requires an A100 80GB pair, H100 pair, or a workstation with multiple A6000 Ada (48GB each). Not practical for home use. Use cloud GPU if you need FP16 accuracy for production.
Try Vast.ai Cloud GPU→q8_0 (75GB)
Four RTX 4090s in a multi-GPU setup, or two H100 PCIe 80GB cards. Note that a pair of RTX 5090s gives 64GB and therefore does not reach q8_0 — the step a 5090 pair actually tops out at is q6_K. Cloud is cheaper for occasional inference at this quality level.
q5_K_M (50GB)
Two RTX 4090s give 48GB, so this does not fit: you are 2GB short before counting any KV cache, and in practice 4-6GB short. It runs with CPU offload at a speed penalty, but if q5 is the goal the honest hardware answer is two 32GB cards rather than two 24GB ones.
q4_K_M (43GB) — the practical target
Two RTX 4090s (48GB total) hold this with roughly 5GB left over, which covers a modest context window and not much more. It is the standard setup for 70B inference at home, and it is tight rather than roomy — long conversations will push you to cap context or drop to q4_K_S (40GB). Tokens run at roughly 8-12 tok/s combined, which is conversational.
q3_K_M (34GB) and q3_K_S (31GB)
This is where the single-card dream ends. q3_K_M does not fit an RTX 5090’s 32GB — it is 2GB over before context. q3_K_S at 31GB does fit, with essentially nothing spare, so expect to keep context short or offload a little. Speed is reasonable at ~15-18 tok/s.
q2_K (26GB)
Small enough for a 32GB card with room to breathe, and still too large for a 24GB one. At q2 quality a 70B model performs comparably to a well-quantized 32B — you lose the reason you wanted 70B in the first place, so this is a curiosity rather than a plan.
Which GPU should YOU buy?
Running 70B on a budget: Get two RTX 4060 Ti 16GB cards ($425 each, ~$850 total = 32GB). That reaches q3_K_S (31GB) — not q3_K_M, which is 34GB and overflows — so you get the flavor of a 70B model without the full price. Use llama.cpp with tensor split.
Running 70B properly: Two RTX 4090s ($2,200 each, ~$4,400 total = 48GB). This is the gold standard for home 70B inference — the 43GB q4_K_M build fits with about 5GB left for context. Most guides use this setup.
Single-card 70B (compromised): The RTX 5090 at 32GB runs q3_K_S without multi-GPU complexity, at ~$4,900. Simpler setup, but lower quality than dual 4090s, and it stops one step short of the q3_K_M that a dual-card build reaches easily.
Need better than Q4? Rent an A100 80GB pair on RunPod for Q8_0 quality. At 70B scale, cloud often beats a home multi-GPU build on cost per inference.
NVIDIA GeForce RTX 4090
24GB GDDR6XPair two of these for 48GB combined VRAM. Runs 70B at Q4_K_M — the quality sweet spot — at ~10 tok/s. Most popular dual-GPU path to 70B.
Affiliate links — we may earn a commission at no extra cost to you. Amazon ships globally; Shopee SG covers Singapore & ASEAN.
Why the usual arithmetic understates it
The standard way to estimate this is parameters × bytes per parameter: 70 billion × 0.5 bytes for a 4-bit quant gives 35GB. The published file is 43GB. That 8GB gap is not rounding, and it is worth understanding, because 8GB is the difference between a build that works and one that swaps to system RAM.
The reason is that K-quants are not uniformly 4-bit. A q4_K_M file keeps attention and some feed-forward tensors at higher precision, and stores per-block scaling factors alongside the weights. The effective rate lands near 4.9 bits per weight rather than 4.0 — which is exactly the 43GB Ollama publishes.
You can see the same effect across the range. At fp16 the naive calculation is correct (70B × 2 bytes = 140GB, published 141GB) because there is no block structure to pay for. Every quantized level costs more than the nominal bit width suggests:
| Level | Naive estimate | Published | Effective bits/weight |
|---|---|---|---|
| fp16 | 140GB | 141GB | ~16 |
| q8_0 | 70GB | 75GB | ~8.5 |
| q5_K_M | 44GB | 50GB | ~5.7 |
| q4_K_M | 35GB | 43GB | ~4.9 |
The practical rule: look up the file size before you buy, rather than computing it. On top of whichever row applies, budget 2-4GB of KV cache at a few thousand tokens of context and more as the conversation grows — which is why two RTX 4090s get tight at longer contexts even on the row that fits.
Common mistakes to avoid
- Estimating the file size from the parameter count. The arithmetic gives 35GB for a 4-bit 70B; the file is 43GB. Look the number up instead of deriving it — this is the single most common way people end up 8GB short.
- Assuming Q4 = half the VRAM of FP16. It’s closer to 30% (43GB vs 141GB). The math surprises people.
- Forgetting KV cache. Your VRAM budget is not just the model weights. Long conversations eat into the headroom fast. Always leave 4-8GB for the cache.
- Buying a single 24GB GPU to run 70B. It does not work at any published quantization — the smallest, q2_K, is 26GB. Save up for a second 4090 or start with a well-quantized 34B.
- Ignoring tensor parallel overhead. llama.cpp with
-ts 1,1(tensor split) adds some communication overhead between GPUs. Expect 5-10% lower throughput versus theoretical peak. - Expecting 48GB to reach q5_K_M. For multi-step reasoning q5 is noticeably better than q4, and a dual-24GB build cannot hold it — 50GB of weights against 48GB of VRAM, before any KV cache. Reaching q5 means two 32GB cards, not a quantization tweak.
Final verdict
| Setup | Max quantization that fits | Quality | Tokens/s | Cost |
|---|---|---|---|---|
| 1x RTX 4090 (24GB) | none — q2_K is 26GB | — | — | ~$2,200 |
| 2x RTX 4060 Ti 16GB (32GB) | q3_K_S | Acceptable | ~8 | ~$850 |
| 1x RTX 5090 (32GB) | q3_K_S | Acceptable | ~15 | ~$4,900 |
| 2x RTX 4090 (48GB) | q4_K_M | Good | ~10 | ~$4,400 |
| 2x RTX 5090 (64GB) | q6_K | Excellent | ~20 | ~$9,800 |
For most home users, the two RTX 4090 setup running Q4_K_M is the practical target. It costs roughly $4,400 in hardware and gives you a genuinely capable 70B model for open-ended reasoning, long-form writing, and research tasks.
If you want single-card simplicity, consider whether a well-quantized 34B model — which fits on one RTX 4090 — might meet your needs. For full VRAM planning across all model sizes, see our VRAM requirements guide. And if you’re specifically running Llama 3.1 70B or Llama 3.3 70B, the best GPU for Llama 70B guide covers those models in detail. For multi-GPU build advice, see best multi-GPU setup for LLM inference.
NVIDIA GeForce RTX 5090
32GB GDDR732GB is the smallest card that holds a 70B at all — q3_K_S fits, without multi-GPU complexity. Best for those who want 70B on one card.
Affiliate links — we may earn a commission at no extra cost to you. Amazon ships globally; Shopee SG covers Singapore & ASEAN.