Choosing a GPU for LLM fine-tuning (a VRAM guide)

During fine-tuning, model weights, gradients, optimizer state (roughly 2× the weights for Adam), and activations must all fit in memory together. If the total does not fit in VRAM, training will not start; and workloads running at the edge spill into swap, collapsing GPU throughput.

That is why choosing a GPU is a memory problem first: a mid-range card with enough VRAM is always more efficient than a fast card without it.

The ranges below are practical figures for bf16 training at typical sequence lengths; as sequence length and batch size grow, expect to move toward the upper bound.

• 7B full fine-tuning: ~14 GB weights + ~56 GB optimizer state + ~14 GB gradients → comfortable on 2 × 80 GB, tight on 1 × 80 GB.
• 13B full fine-tuning: 2 × 80 GB recommended.
• 34B full fine-tuning: needs the 4 × 80 GB class.
• 70B full fine-tuning: requires the 8 × 80 GB (A100/H100) class with multi-GPU parallelism.
• LoRA/QLoRA: 7–13B models fit on a single 24–48 GB card; QLoRA on a single 80 GB card is common practice for 70B.

LoRA freezes the weight matrices and trains only small low-rank update matrices; the trainable share is typically 0.1–1%. This eliminates most of the optimizer state and gradient memory.

QLoRA goes one step further: the base model is quantized to 4 bits (NF4) and selectively dequantized to 16 bits during computation. For relatively small teams this is the standard way to fine-tune 70B-class models on a single card; quality loss depends on the task and data quality and must always be measured.

Activation memory grows roughly linearly with sequence length, and quadratically in the attention layer. Raising a sequence from 2,048 to 8,192 tokens increases training memory noticeably without changing model weights at all.

Effective batch size = per-device batch × gradient accumulation steps. When memory runs out, first lower sequence length and per-device batch, then return to your target effective batch with accumulation.

• 24 GB class (e.g., RTX 4090): QLoRA fine-tuning of 7–13B models; ideal for fast experimentation.
• 48 GB class (e.g., L40S/A6000): full fine-tuning of 7B, or QLoRA of 13–34B.
• 80 GB class (A100/H100): QLoRA of 70B, full fine-tuning of 13B; the per-node standard for multi-GPU jobs.
• If you need multi-GPU training, rent the whole node/cluster together rather than individual cards; NVLink and low-latency networking are critical.

• Model parameter count and target method (full / LoRA / QLoRA).
• Maximum sequence length and target effective batch size.
• Dataset size and persistent storage needs.
• Measure real memory usage with a short rental; validate the spreadsheet in the field.