Estimate GPU memory required for training from model × method × GPU count × parallelism strategy. Predict OOM before you run. Assumes bf16 mixed precision with Adam-family optimizer.
This tool estimates the GPU memory (VRAM) required for training from model, training method (Full FT / LoRA / QLoRA), GPU count, and parallelism strategy (Single / DDP / TP / ZeRO-3 / FSDP), predicting OOM before you run. The estimate sums four components — weights, gradients, optimizer states, and activations — and is calibrated against published benchmarks (measured VRAM from Unsloth/HuggingFace). Assumes bf16 mixed precision with an Adam-family optimizer.
| Llama-3.1 8B · QLoRA | ~6.7 GB |
| Llama-3.1 8B · LoRA | ~19 GB |
| Llama-3.1 8B · Full FT (8-bit Adam) | ~60–84 GB |
| Llama-3.1 70B · LoRA | ~149–160 GB (needs multiple GPUs) |
| Llama-3.1 70B · QLoRA | ~40–48 GB |
* Representative cases confirmed to align with published benchmark ranges. Actual values vary with sequence length, batch size, parallelism strategy, etc. — use the form above for your own configuration.
Q. What is OOM (Out of Memory) and why does it happen?
A. OOM is an error where a process is killed after exhausting GPU memory (VRAM). During training, VRAM is consumed by weights, gradients, optimizer states, and activations — OOM occurs when their sum exceeds the GPU's installed VRAM.
Q. How much does VRAM usage differ between Full FT, LoRA, and QLoRA?
A. Full FT is heaviest, since it keeps gradients and optimizer states for every parameter. LoRA is far lighter since only a small adapter is trained. QLoRA is lightest, since the base weights are also quantized to 4-bit. See the table above for typical magnitudes.
Q. If I shard across GPUs with ZeRO-3 or FSDP, is it always safe?
A. In theory, weights, gradients, and optimizer states are divided across GPUs. In practice, known sharding gaps exist where some layers aren't sharded and each GPU keeps a full copy, causing OOM (e.g., some models' linear_attention layers missing from tp_plan, or nested language models inside VLMs not covered by ZeRO-3's zero.Init). This tool shows a warning when such parallel strategies are selected.
Q. How accurate is this estimate?
A. It's an approximation (roughly ±15%). It's calibrated against published benchmarks (Unsloth/HuggingFace measured VRAM); recalibration against in-house measured runs is planned.