the preference data

Rather than a single ideal response per prompt (what SFT uses), preference tuning starts from pairs: the same prompt with two candidate responses, and a label for which one is preferred. This is a much easier judgment to make reliably (both for human raters and for a judge model) than writing an ideal response from scratch, which is why it scales further than SFT data collection alone.

# one preference example
{
  "prompt": "Explain recursion to a beginner.",
  "chosen": "...",     # the response humans/judges preferred
  "rejected": "..."    # the response they preferred less
}

classic RLHF: reward model + PPO

# stage A: train a reward model on preference pairs
reward_model.train(chosen, rejected)   # learns to score chosen > rejected

# stage B: use PPO (reinforcement learning) to update the LLM,
# using the reward model's score as the reward signal
for prompt in prompts:
    response = llm.generate(prompt)
    reward   = reward_model.score(prompt, response)
    llm.update_via_ppo(reward)   # nudge toward higher-reward outputs
This works, but it's two separate training runs with an RL loop notorious for instability (reward hacking, where the policy finds ways to score well on the reward model without actually improving) and a lot of hyperparameter sensitivity.

DPO: the same objective, no RL loop

Direct Preference Optimization derives a loss function whose optimum is mathematically equivalent to what RLHF is trying to reach, but expresses it directly in terms of the model's own output probabilities on the chosen vs. rejected response — no separate reward model, no PPO, no RL instability. It's trained with ordinary supervised-learning-style gradient descent on the preference pairs directly.

RLHF (reward model + PPO)DPO
Separate reward model neededYesNo
RL loopYes (PPO)No — plain supervised-style training
Training stabilitySensitive to RL hyperparametersConsiderably more stable
Typical adoption todayStill used, especially at the largest labsThe more common default for teams doing this themselves

where to go from here

Pretraining, SFT & RLHF — where this stage fits in the full training pipeline.
LoRA & Parameter-Efficient Fine-Tuning — the parameter-efficient technique commonly used for this stage too.
Evaluating LLM Applications — measuring whether preference tuning actually improved anything.