Pretraining, SFT & RLHF

stage 1: pretraining

The model is trained on a huge corpus (web text, books, code) with one objective: predict the next token given everything before it. No labels, no human curation of individual examples — the "label" for every position in every document is just the next token that actually appears there. This is why it scales: the training signal is free, limited only by how much text exists and how much compute is spent processing it.

# the entire pretraining loss function, conceptually
loss = -sum(log P(token_i | token_1, ..., token_{i-1})  for i in sequence)
# minimize this over trillions of tokens

The output of this stage alone is a base model: fluent, knowledgeable, but not aligned to follow instructions or hold a conversation — ask it a question and it's just as likely to continue with more questions, since that's a plausible continuation of text that looks like a question.

stage 2: supervised fine-tuning (SFT)

The base model is further trained on a much smaller, curated dataset of (instruction, ideal response) pairs, written or selected by humans. This teaches the model the format of being an assistant — answering questions instead of continuing them, following the system/user/assistant turn structure, refusing certain requests — using the same next-token objective as pretraining, just on different data.

stage 3: preference alignment (RLHF / DPO)

SFT teaches the model to imitate good responses it was shown; it doesn't teach it to distinguish a good response from a slightly-worse one on its own. Preference alignment closes that gap: humans (or a trained model acting as a judge) rank multiple candidate responses to the same prompt, and that ranking signal is used to further shift the model's output distribution toward what's preferred.

Approach	Mechanism
RLHF (classic)	Train a separate reward model on the ranking data, then use reinforcement learning (PPO) to optimize the LLM against that reward model
DPO (Direct Preference Optimization)	Skip the separate reward model and RL loop entirely — a closed-form loss derived to have the same optimum, trained directly like supervised learning

DPO has become the more common default because it's simpler to implement and more stable to train than full RLHF, at roughly comparable quality — more on the mechanics in RLHF, DPO & Preference Tuning.

where fine-tuning (the kind you'd do yourself) fits

All three stages above are what the model vendor does before shipping the model. When this track later talks about "fine-tuning" as something you do, it means starting from the already-aligned model and running a much smaller, task-specific version of stage 2 — typically with LoRA rather than updating every weight. See Fine-Tuning vs. RAG vs. Prompting for when that's actually the right tool.

where to go from here

RLHF, DPO & Preference Tuning — stage 3, in depth.
LoRA & Parameter-Efficient Fine-Tuning — how you fine-tune without retraining every parameter.
Fine-Tuning vs. RAG vs. Prompting — when this is the right lever to pull.