LLM training time calculator
Training compute ≈ 6 × parameters × tokens. Divide by achieved GPU FLOPs and you get GPU-hours — plan your pretrain or fine-tune before booking a single node.
—6 × P × T ÷ (FP16 TFLOPS × 0.4 MFU × GPUs)
Per-GPU training time
Single card, current model and token count
| GPU | FP16 TFLOPS | Single-card time |
|---|---|---|
| RTX 3060 | 25.5 | — |
| RTX 4060 Ti | 44 | — |
| RTX 3090 | 142 | — |
| RTX 4090 | 330 | — |
| RTX 5090 | 419 | — |
| A100 | 312 | — |
| A100 | 312 | — |
| H100 SXM | 989 | — |
| H200 | 989 | — |
| B200 | 2250 | — |
| AMD MI300X | 1307 | — |
Chinchilla-optimal pretrains on H100
20 tokens per parameter
| Model | Tokens | 1 × H100 | 1024 × H100 |
|---|---|---|---|
| Llama 3.1 8B | 160B | 7.5 GPU-months | 5.3 GPU-hours |
| Llama 3.1 70B | 1400B | 573.4 GPU-months | 16.8 GPU-days |
| Llama 3.1 405B | 8100B | 19,195.5 GPU-months | 18.7 GPU-months |
| Qwen 2.5 7B | 140B | 5.7 GPU-months | 4 GPU-hours |
| Qwen 2.5 72B | 1440B | 606.7 GPU-months | 17.8 GPU-days |
| Mistral 7B | 140B | 5.7 GPU-months | 4 GPU-hours |
Six FLOPs per parameter, per token.
Every training step runs the forward pass (about 2 FLOPs per parameter per token) and the backward pass (about 4), which is where the 6× rule comes from. Multiply parameters by tokens by six, divide by your cluster's achieved FLOPS — peak TFLOPS discounted by MFU — and the result is wall-clock GPU time before you spend a cent.
The rule scales linearly in every direction: double the data and time doubles, halve the model and time halves, and a 1024-GPU cluster finishes in 1/1024th the single-card time in the ideal case. Real clusters lose some of that to communication, which is exactly what MFU absorbs.
Frequently asked questions
Where does the 6 × params × tokens rule come from?
Scaling-law literature (Kaplan 2020, Hoffmann/Chinchilla 2022) shows training compute is ≈ 6 FLOPs per parameter per token: 2 FLOPs for the forward pass and 4 for the backward. It is the standard estimate used across pretraining papers and GPU-hour forecasts.
Why 40% MFU?
MFU (model FLOPs utilization) is the share of a GPU's peak compute a real training run achieves. Large dense runs land at 30-50% on well-tuned clusters — we use 40% and state it, so you can scale the estimate for your own stack.
How many tokens should I train for?
Compute-optimal (Chinchilla) is roughly 20 tokens per parameter — 160B tokens for an 8B model. Modern frontier models deliberately overtrain far beyond this for better inference economics; multiply the token count accordingly and the time scales linearly.