GPU Memory Planning for LLMs

the four line items

Component	Scales with	Covered on
Model weights	Parameter count × bytes per parameter	Quantization
KV cache	Sequence length × concurrent requests × layers/heads	KV Cache & Continuous Batching
Activations	Batch size × sequence length, transient during a forward pass	this page
Optimizer state (training only)	Trainable parameter count × ~2-8× (Adam-family)	LoRA & PEFT

a worked inference example

# serving a 13B-parameter model at INT4, batch of 8, 4k context each
weights_gb   = 13 * 0.5                       # INT4 ≈ 0.5 bytes/param -> 6.5 GB
kv_per_tok   = 2 * 40 * 8 * 128 * 2           # layers=40, kv_heads=8, head_dim=128, FP16
kv_gb        = (kv_per_tok * 4096 * 8) / 1e9  # 4096 tokens x 8 concurrent requests
# kv_gb ≈ 13.4 GB

activation_overhead_gb = 2   # rough headroom, workload-dependent

total_gb = weights_gb + kv_gb + activation_overhead_gb
print(total_gb)   # ≈ 22 GB -- fits on a single 24GB card, barely

The KV cache term here is larger than the quantized weights — a common surprise for anyone estimating VRAM needs from model size alone. At high concurrency or long context, KV cache dominates, not weights.

fine-tuning needs a very different budget

Training (even LoRA) adds gradient and optimizer-state memory that inference never touches. For full fine-tuning, Adam's optimizer state alone typically adds 2 extra FP32 copies of every trainable parameter (momentum and variance) on top of the weights and gradients — roughly 4x the raw parameter memory before activations are even counted, which is precisely the cost LoRA exists to avoid by keeping the trainable parameter count tiny.

levers, if the number doesn't fit

Lever	Reduces
Quantize weights (INT8/INT4)	Weight memory directly
Lower max context length or concurrent request cap	KV cache, often the largest lever at high concurrency
Grouped-query attention (a model architecture choice, not a runtime one)	KV cache per token, by using fewer KV heads than query heads
LoRA instead of full fine-tuning	Optimizer state and gradient memory, training only
Multi-GPU tensor/pipeline parallelism	Per-GPU share of weights (spreads the model, doesn't reduce the total)

where to go from here

Quantization — line item 1 in the budget above.
KV Cache & Continuous Batching — line item 2, and why it often dominates.
GPU Architecture (Nvidia vs AMD) — the hardware this budget has to fit inside.