Quantization

the memory math

# approximate weight memory, by precision
def weight_gb(num_params_billions, bytes_per_param):
    return num_params_billions * bytes_per_param

weight_gb(70, 2)    # FP16:  ~140 GB
weight_gb(70, 1)    # INT8:  ~70 GB
weight_gb(70, 0.5)  # INT4:  ~35 GB

This is weights only — the KV cache and activation memory add on top, but the weight-precision reduction alone is what usually decides whether a given model fits on a given GPU at all.

why you can't just truncate every weight naively

Simply rounding every weight to the nearest representable INT8/INT4 value works reasonably well for most weights, but a small number of "outlier" weights with unusually large magnitude dominate the quantization error if treated the same way — naive uniform quantization of those outliers measurably degrades model quality. Every practical quantization method exists specifically to handle this outlier problem better than naive rounding.

the main methods, compared

Method	Approach	Typical use case
GPTQ	Post-training, layer-by-layer weight quantization that minimizes reconstruction error using a small calibration dataset	GPU inference, widely supported by serving engines
AWQ	Identifies and protects the small fraction of "salient" weight channels (by activation magnitude) that matter most, quantizing the rest more aggressively	GPU inference; often better quality than GPTQ at the same bit width
GGUF (via llama.cpp)	A quantized model file format supporting a range of bit widths, with CPU as well as GPU execution	Local/CPU/consumer-hardware inference
bitsandbytes (NF4, used by QLoRA)	A quantization data type designed to match the distribution of pretrained weights (roughly normal) more closely than plain integers	Fine-tuning frozen base weights — see LoRA & PEFT

the accuracy/speed/memory trade-off

Precision	Memory vs. FP16	Typical quality impact
INT8	~50%	Usually negligible on most benchmarks
INT4 (GPTQ/AWQ)	~25%	Small but measurable, task-dependent
INT3 and below	<25%	Noticeable degradation on most models without careful method choice

Always validate on your actual downstream task rather than trusting a generic benchmark number — quantization sensitivity varies significantly by model architecture and by task.

where to go from here

KV Cache & Continuous Batching — the other major memory consumer at inference time.
GPU Memory Planning for LLMs — putting the full memory budget together.
Inference Engines — which serving engines support which quantization formats.