Quick answer: Dual RTX 3090s (~$1,640 used) give you 48GB combined VRAM for running 70B models locally. Splitting a model across cards works over PCIe without NVLink. For new hardware, dual RTX 5080s (32GB combined) or a single RTX 5090 (32GB) are cleaner options.
Check NVIDIA GeForce RTX 3090 on Amazon→Buy on Shopee SG→Why go multi-GPU?
Single consumer GPUs top out at 32GB VRAM (RTX 5090). To run 70B+ models at usable quantization, or 34B models at high quality, you need more memory. Multi-GPU setups combine VRAM from two or more cards by slicing the model across them. What that buys you is capacity, not speed — a distinction worth getting straight before you spend, and one the next section explains.
VRAM capacity memory bandwidth Specs are manufacturer figures. Bar lengths are scaled independently per metric.
How splitting works, and why two cards are not twice as fast
llama.cpp offers four ways to split a model, and its own argument documentation is precise about what each one does:
-sm mode | What it does | Behaviour |
|---|---|---|
none | use one GPU only | — |
layer (default) | split layers and KV across GPUs | pipelined |
row | split weight across GPUs by rows | parallelized |
tensor | split weights and KV across GPUs | parallelized, experimental |
That word “pipelined” on the default is the one that matters. Under layer, each GPU holds a slice of the layers, and during inference the first card computes its layers, hands the activations to the second, and then waits. The cards take turns rather than working simultaneously, which is why a second GPU roughly doubles the model size you can hold and does very little for tokens per second.
Genuine parallelism is what -sm row is for, and it is worth trying — but it moves more data between cards, so the benefit depends on your interconnect and it is not a free win on every setup. If you have never touched the flag, you have been running layer.
Note that --split-mode and --tensor-split are different flags with confusingly similar names: the first chooses how the model is divided, the second sets the ratio between cards. The setup section below covers the second.
The other key point: you do not need NVLink. PCIe 4.0 x16 provides about 31.5 GB/s per direction, which is sufficient for inference. NVLink helps but is not required.
Best multi-GPU configurations
| Setup | Total VRAM | Max model | Cost | Best for |
|---|---|---|---|---|
| 2x RTX 3090 (used) | 48GB | 70B q4_K_M (43GB) | ~$1,640 | Best value |
| 2x RTX 5080 | 32GB | 34B q5_K_M | ~$2,800 | Modern + fast |
| 2x RTX 4090 | 48GB | 70B q4_K_M (43GB) | ~$4,400 | Maximum speed |
| 2x RTX 5090 | 64GB | 70B q6_K (58GB) | ~$9,800 | Endgame |
| RTX 5090 + RTX 3090 | 56GB | 70B q5_K_M (50GB) | ~$5,700 | Mixed budget |
Sizes are the published downloads for llama3.3:70b. Note that the two 48GB builds top out at the same quantization: more money buys speed there, not a bigger model. q5_K_M is 50GB and needs 56GB or more, and q8_0 is 75GB — beyond even a pair of 5090s.
Dual RTX 3090 — best value multi-GPU
Two used RTX 3090s is the most popular multi-GPU LLM setup for good reason — we wrote a full step-by-step dual RTX 3090 setup guide covering hardware, software, and troubleshooting:
- 48GB combined VRAM fits a 70B at q4_K_M — a 43GB download, leaving ~5GB for context
- ~$1,640 total — less than a single RTX 5090
- Proven community setup with thousands of builds documented
- Model splitting in llama.cpp is well-tested on this configuration
Performance with 70B Q4_K_M: expect ~8-12 tok/s depending on PCIe bandwidth and model. That is usable for interactive chat, though not blazing fast. For a detailed look at exactly how much VRAM each quantization level of a 70B model requires, see how much VRAM for a 70B model.
Check NVIDIA GeForce RTX 3090 on Amazon→Buy on Shopee SG→What you need for dual 3090s
| Component | Requirement |
|---|---|
| Motherboard | ATX with 2x PCIe x16 slots (at least x8 electrical each) |
| CPU | Any modern CPU with enough PCIe lanes (AMD Ryzen 7/9, Intel i7/i9) |
| PSU | 1000W+ (two 3090s draw ~700W combined under load) |
| Case | Full tower with good airflow — these cards are thick and hot |
| RAM | 64GB DDR4/DDR5 (model loading requires system RAM) |
| Slot spacing | Minimum 3-slot gap between cards for thermal headroom |
NVLink: do you need it?
NVLink provides a high-speed direct connection between GPUs (up to 112 GB/s on RTX 3090 NVLink bridges). Here is the honest assessment:
- For inference: NVLink helps but is not critical. PCIe x16 is the bottleneck only on very large models with many cross-GPU transfers. Typical speedup with NVLink: 10-20% for 70B inference.
- For training/fine-tuning: NVLink matters significantly. Gradient synchronization is bandwidth-intensive.
- Availability: RTX 3090 supports NVLink bridges (~$80-100 used). RTX 4090 and RTX 5090 do not support consumer NVLink.
If you are only doing inference, skip NVLink and save the money. If you plan to fine-tune on dual 3090s, the NVLink bridge is worth the $80 — and our LLM fine-tuning GPU guide covers the full VRAM math for LoRA and full fine-tuning on multi-GPU setups.
Setting up multi-GPU splitting
llama.cpp / Ollama
In llama.cpp, --tensor-split sets the ratio between cards and -sm selects the mode:
# Split evenly between two GPUs (default layer mode, pipelined)
./llama-cli -m model.gguf --tensor-split 0.5,0.5 -ngl 99
# Split by VRAM ratio (e.g., 5090 + 3090)
./llama-cli -m model.gguf --tensor-split 0.57,0.43 -ngl 99
# Same split, but parallelized across cards instead of pipelined
./llama-cli -m model.gguf --tensor-split 0.5,0.5 -ngl 99 -sm rowTry -sm row on a matched pair and compare — it is the one flag most dual-GPU owners have never set, and it is where any real throughput gain from the second card comes from.
Ollama handles the ratio automatically when multiple GPUs are detected. No configuration needed.
Mixed GPU setups
You can mix different NVIDIA GPUs when splitting a model across them. Common combinations:
- RTX 5090 + RTX 3090 (56GB): Uneven split, weight the 5090 heavier for speed
- RTX 4090 + RTX 3090 (48GB): Both 24GB, even split works well
- RTX 4090 + RTX 4060 Ti 16GB (40GB): Budget expansion of existing 4090
The faster GPU should handle more layers. llama.cpp’s --tensor-split ratio lets you tune this. Mixed setups work well for inference but are suboptimal for training.
Multi-GPU vs single large GPU
| Factor | Multi-GPU (2x 3090) | Single GPU (RTX 5090) |
|---|---|---|
| Total VRAM | 48GB | 32GB |
| Cost | ~$1,640 | ~$4,900 |
| Power draw | ~700W | ~575W |
| Complexity | Higher | Plug and play |
| Inference speed (34B) | ~20 tok/s | ~40 tok/s |
| Max model | 70B q4_K_M | 34B q5_K_M / 70B q3_K_S |
For 34B models, a single RTX 5090 is faster and simpler. Multi-GPU only makes sense when you need more VRAM than any single card provides, or when building on a budget with used cards.
Which multi-GPU setup should you buy?
- Want 70B models at the lowest cost? Get dual RTX 3090s used (~$1,640). The 48GB combined VRAM fits Llama 3.3 70B at q4_K_M, and no other setup under $2,000 can do that.
- Already own an RTX 4090 and want 70B access? Add a used RTX 3090 as a second card (~$820). You get 48GB total for under $1,000 extra investment.
- Want maximum speed on 70B? Get dual RTX 4090s (~$4,400). The doubled bandwidth over dual 3090s gives you 15-20 tok/s on 70B Q4 versus 8-12 tok/s.
- Models fit in 32GB but you want headroom? Skip multi-GPU and get a single RTX 5090. Simpler, less power, faster inference on models that fit.
Common mistakes to avoid
- Buying an NVLink bridge for inference-only workloads. NVLink gives only 10-20% speedup for inference. Save the $80-100 unless you plan to fine-tune.
- Using a motherboard with x4 electrical on the second PCIe slot. Many consumer boards only provide x4 bandwidth to the second GPU slot, cutting inter-GPU transfer speed by 75%. Verify x8 minimum per slot — our best motherboard for dual-GPU LLM guide lists boards with confirmed x8/x8 bifurcation.
- Running dual GPUs on a 750W PSU. Two RTX 3090s draw ~700W under load, leaving zero headroom for CPU, RAM, and fans. A 1000W PSU is the minimum, and 1200W gives you safe margin — see our PSU guide for dual-GPU LLM builds for specific unit recommendations.
- Mixing NVIDIA and AMD GPUs. Splitting a model requires both cards on the same driver stack. Cross-vendor multi-GPU does not work for LLM inference. Multi-GPU setups also behave differently on Windows versus Linux due to PCIe bandwidth and driver handling — our Windows vs Linux for local LLM guide covers what to expect on each platform.
Our recommendation
For most users wanting to run 70B models locally, dual RTX 3090s are the best value in 2026. At ~$1,640, you get 48GB of VRAM, proven software support, and enough speed for interactive inference. Just make sure your PSU and case can handle the heat.
If you want a simpler build and your models fit in 32GB, a single RTX 5090 is the cleaner choice. If $4,900 is more than you want to spend, our best GPU for LLM under $2,000 guide covers what the tier below reaches. If you already own an RTX 4090 and want to expand, adding a used RTX 3090 as a second card gives you 48GB total for under $1,000 extra.
Check NVIDIA GeForce RTX 5090 on Amazon→Buy on Shopee SG→ Check NVIDIA GeForce RTX 3090 on Amazon→Buy on Shopee SG→Frequently asked questions
Do I need NVLink for multi-GPU LLM inference?
No. PCIe 4.0 x16 carries about 31.5 GB/s per direction, which is enough for inference, and the typical NVLink gain is in the region of ten to twenty percent. Skip the bridge unless you plan to fine-tune, where gradient synchronisation makes the interconnect matter much more. Note that only the RTX 3090 supports consumer NVLink at all — the 4090 and 5090 do not.
How much VRAM do two RTX 3090s give me, and what does it run?
48GB combined, which holds Llama 3.3 70B at q4_K_M — a 43GB download — with a few gigabytes left for context. That is the cheapest route to running a 70B model entirely on GPU, and no single consumer card reaches it: the largest is 32GB.
Is dual RTX 3090 better than one RTX 5090?
It depends which wall you hit first. Two 3090s give more memory for less money and are the only one of the two that fits a 70B at q4_K_M. A single 5090 is faster on anything that fits its 32GB, draws less power, and is far simpler to build and cool. Buy the pair for capacity, the single card for speed and quiet.
What PSU do I need for two GPUs?
Plan on 1000W as a minimum for a pair of 3090s, which draw roughly 700W between them under load, and 1200W if you want real headroom. A 750W unit leaves nothing for the CPU, drives and fans, and transient spikes on these cards are what trips protection rather than the steady draw.
Can I mix different GPUs in one machine?
Yes, as long as they are both NVIDIA — a 4090 alongside a used 3090 is a common way to reach 48GB. Mixing NVIDIA and AMD does not work, because splitting a model requires one driver stack. With uneven cards, weight the split toward the faster one rather than dividing the model evenly.