Multi-GPU calculator
Total VRAM needed ÷ single-card memory = how many GPUs to run a model. Pick a model, quantization, and context to plan your rig.
—Weights ×1.2 overhead + KV cache; rows below show cards required
Cards required per GPU type
Updates as you change the calculator above
| GPU | Card VRAM | Cards needed | Combined |
|---|---|---|---|
| RTX 3060 | 12 GB | — | — |
| RTX 4060 Ti | 16 GB | — | — |
| RTX 3090 | 24 GB | — | — |
| RTX 4090 | 24 GB | — | — |
| RTX 5090 | 32 GB | — | — |
| A100 | 40 GB | — | — |
| A100 | 80 GB | — | — |
| H100 SXM | 80 GB | — | — |
| H200 | 141 GB | — | — |
| B200 | 180 GB | — | — |
| AMD MI300X | 192 GB | — | — |
| Mac Studio M3 Ultra unified memory, lower bandwidth | 512 GB | — | — |
Popular models at GGUF Q4_K_M, 4K context
Cards of RTX 4090 vs A100 80 GB
| Model | Total VRAM | × RTX 4090 (24 GB) | × A100 (80 GB) |
|---|---|---|---|
| Llama 3.1 8B | 6.3 GB | 1 card | 1 card |
| Llama 3.1 70B | 52 GB | 3 cards | 1 card |
| Llama 3.1 405B | 295.5 GB | 13 cards | 4 cards |
| Qwen 2.5 7B | 5.3 GB | 1 card | 1 card |
| Qwen 2.5 72B | 53.5 GB | 3 cards | 1 card |
| Mistral 7B | 5.6 GB | 1 card | 1 card |
Division is the whole formula.
Cards needed = ceil(total VRAM ÷ per-card memory), where total VRAM comes from the same weights-plus-KV-cache model as our VRAM calculator: parameters × bits-per-weight ÷ 8, ×1.2 runtime overhead, plus the KV cache for your context length. Round up, because a model that needs 49 GB does not almost fit in two 24 GB cards.
The practical budget doesn't stop at card count: a 2-card rig needs a motherboard with the slots and lanes, a PSU with headroom for both, and a case that moves the heat. The row you should trust for planning is combined memory minus 10-15% fragmentation slack.
Frequently asked questions
How does splitting a model across GPUs work?
Tensor parallelism shards each layer's weights across cards so all GPUs compute simultaneously; pipeline parallelism assigns consecutive layers to different cards. Both make N cards behave like one GPU with roughly N× the memory at the cost of interconnect traffic — NVLink makes this nearly free, PCIe adds noticeable overhead.
Do the cards need to be identical?
It is strongly recommended. Mixed setups waste the smallest card's headroom, complicate memory-balancing, and many runtimes simply require matching devices. The safe pattern is N identical cards, each sized so N × VRAM covers the total from the calculator plus 10-15% slack.
Does splitting slow inference down?
Per-token speed usually improves for models that don't fit on one card, because bandwidth aggregates. But communication overhead means 2 cards deliver less than 2× a single card — over consumer PCIe expect 1.3-1.6×. What you buy is memory, not linear speed.