LoRA & Parameter-Efficient Fine-Tuning

the problem full fine-tuning has

Updating every weight in a large model requires storing gradients and optimizer state (for Adam, typically 2 extra copies of every parameter) alongside the weights themselves — for a model with tens of billions of parameters, that's far more GPU memory than most teams have access to, even before accounting for activations. It also produces a full new copy of the model per fine-tuned variant, which gets expensive to store and serve if you need many task-specific versions.

the LoRA idea

# a weight matrix W (d x k) is frozen entirely during fine-tuning.
# instead, a low-rank update is learned and added at inference time:
#   W_effective = W_frozen + (B @ A) * (alpha / r)
#
# A: r x k, B: d x r      -- r (the rank) is small, e.g. 8-64
# trainable params for this layer: r*k + d*r   -- a tiny fraction of d*k

The premise, empirically supported: the change needed to adapt a large pretrained model to a new task tends to have low "intrinsic rank" — it doesn't need the full expressive power of a dense d×k update, so a much smaller low-rank approximation captures most of the useful adaptation.

what this buys you

	Full fine-tuning	LoRA
Trainable parameters	100% of the model	Often <1%
GPU memory for training	Weights + gradients + optimizer state for every parameter	Base model frozen (no gradients needed for it); optimizer state only for the small A/B matrices
Storage per fine-tuned variant	A full model copy	Just the A/B matrices — megabytes, not gigabytes
Serving multiple task variants	Load N full models	Load one base model, hot-swap small LoRA adapters per request

QLoRA: fine-tuning on a single consumer GPU

QLoRA combines LoRA with quantizing the frozen base model to 4-bit precision (see Quantization) during training, while keeping the trainable LoRA matrices themselves in higher precision. Since the base weights are frozen, quantizing them costs little in training quality but dramatically cuts the memory footprint of the largest part of the model, which is what made fine-tuning multi-billion-parameter models practical on a single 24GB GPU rather than requiring a multi-GPU cluster.

where to go from here

Quantization — the technique QLoRA borrows to shrink the frozen base model.
Fine-Tuning vs. RAG vs. Prompting — deciding whether fine-tuning is the right tool before reaching for LoRA.
RLHF, DPO & Preference Tuning — LoRA is commonly used for this stage too, not just SFT.