The Transformer Architecture

the problem attention solves

Before transformers, sequence models (RNNs, LSTMs) processed tokens one at a time, carrying state forward — which meant information from early in a long sequence had to survive many sequential updates to still matter at the end, and the sequential dependency made training slow to parallelize. Self-attention lets every token look directly at every other token in one step, regardless of distance, and the whole sequence can be processed in parallel during training.

self-attention, mechanically

# for each token, three learned projections of its embedding:
Q = x @ W_q   # "what am I looking for"
K = x @ W_k   # "what do I contain"
V = x @ W_v   # "what do I actually pass along if selected"

scores  = (Q @ K.T) / sqrt(d_k)     # how relevant is every other token to this one
weights = softmax(scores)            # normalize to a probability distribution
output  = weights @ V                # weighted blend of every token's V

The 1/sqrt(d_k) scale factor keeps dot products from growing too large as dimensionality increases, which would otherwise push softmax into near-one-hot regions and starve gradients — this is literally why it's called "scaled dot-product attention."

multi-head attention

Rather than one attention computation, the model runs several in parallel ("heads"), each with its own learned W_q/W_k/W_v, on lower-dimensional slices of the embedding. The outputs are concatenated and projected back down. In practice different heads tend to specialize — some track syntactic relationships (subject-verb agreement), others track longer-range topical relevance — without ever being explicitly told to.

positional encoding

Attention itself is permutation-invariant — nothing about the mechanism above knows token order. Position has to be injected separately, either as a fixed sinusoidal signal added to each token's embedding (the original 2017 design) or, in most current models, as rotary position embeddings (RoPE), which rotate the Q/K vectors by an angle proportional to position so that relative distance falls directly out of the dot product between two tokens. RoPE is why many modern models generalize reasonably well to sequences longer than they were trained on.

encoder-decoder vs. decoder-only

	Encoder-decoder (original transformer, T5)	Decoder-only (GPT-family, Claude, Llama, most current LLMs)
Attention direction	Encoder: bidirectional. Decoder: causal, plus cross-attention into encoder output	Causal only — every token can only attend to itself and earlier tokens
Good fit for	Translation, tasks with a clear input/output split	Open-ended generation, chat, one architecture for every task via prompting
Why it won out	—	Simpler to scale, and the causal-only design is exactly what next-token pretraining needs

Causal masking is also why the KV cache trick works at all — see KV Cache & Continuous Batching: because token N never attends to tokens after it, the K/V vectors for tokens 1..N-1 are fixed once computed and can be reused rather than recomputed every generation step.

where to go from here

Tokenization — the step before any of this — turning text into the tokens attention operates on.
Pretraining, SFT & RLHF — how a network with this architecture becomes a model you can chat with.
GPU Optimization — why attention's O(n²) cost in sequence length is the thing every serving optimization in this track is fighting.