why the KV cache exists

Because attention is causal (see The Transformer Architecture), the key and value vectors for every already-generated token never change as generation continues — only the query for the newest token needs to look back at them. Without caching, generating token N would require recomputing K/V for all N-1 prior tokens from scratch, at every single step, which is wasteful and gets worse the longer the output. The KV cache stores those K/V vectors once and reuses them for every subsequent step.

# without a KV cache: O(n^2) redundant work across a full generation
# with a KV cache: each new token computes K/V once and appends to the cache
kv_cache = {}
for step in range(max_new_tokens):
    k, v = compute_kv(new_token)
    kv_cache["k"].append(k); kv_cache["v"].append(v)
    logits = attend(query(new_token), kv_cache["k"], kv_cache["v"])
    new_token = sample(logits)

the KV cache memory bill

# per sequence, per token:
kv_bytes_per_token = 2 * num_layers * num_kv_heads * head_dim * bytes_per_element
#                    ^2 for K and V

# example: a 32-layer model, 8 KV heads, head_dim=128, FP16 (2 bytes)
per_token = 2 * 32 * 8 * 128 * 2   # = 131,072 bytes ≈ 128 KB / token

# a single 8,000-token conversation: ~1 GB of KV cache, for ONE request
This is why long-context, high-concurrency serving is a memory problem as much as a compute problem — KV cache memory scales linearly with both sequence length and the number of concurrent sequences being served, and grouped-query attention (fewer KV heads than query heads) exists specifically to shrink this bill.

continuous batching

Naive batching waits for every request in a batch to finish before starting the next batch — so a batch with one long request and several short ones leaves the GPU idle on the short requests' slots until the long one finishes. Continuous batching (also called in-flight batching) instead evicts a finished sequence from the batch and immediately admits a new waiting request into its slot, every generation step, keeping the GPU near full utilization instead of bottlenecked by the slowest request in a static batch.

PagedAttention

Before PagedAttention, each sequence's KV cache was typically allocated as one contiguous memory block sized for the maximum possible sequence length — wasting most of that memory for the many requests that finish well short of the max. PagedAttention (introduced by vLLM) borrows the idea of OS virtual memory paging: KV cache is allocated in small fixed-size blocks on demand, non-contiguously, with a block table mapping logical sequence positions to physical blocks. This nearly eliminates internal fragmentation and is a large part of why vLLM achieves substantially higher throughput than a naive implementation at the same GPU memory budget.

where to go from here

Inference Engines — vLLM and the other engines that implement these techniques.
GPU Memory Planning for LLMs — putting weights, KV cache, and activations into one memory budget.
Quantization — shrinking the weight side of the same memory budget.