05Book 05 · Chapter 05 of 08· 5 min ·
Skip the , fit the votes
uses the same A-versus-B pairs, but it updates the writing model directly so the winner becomes more likely than the loser. It still compares against a frozen copy of the model. It does not train a separate .
Direct Preference Optimization () starts from the same A-versus-B pairs. It skips the second model that only exists to score replies. Instead it updates the writing model directly: make the in the winner more likely, and the in the loser less likely, compared with a frozen copy of the example-trained model.
That frozen copy is the reference: a snapshot of the example-trained model that is not updated. keeps asking, for each pair, “relative to that snapshot, did the winner become more likely than the loser?” If yes, the step worked. If the writing model starts to wander into a new voice, the comparison against the snapshot pulls it back.
The vote is still the only instruction about quality. Pick polished-and-Sydney over plain-Canberra, and will still raise polished-and-Sydney. There is no extra in the middle that can learn “longer is better” or “agree with the user” as its own habit and then hand that habit to the writer. Those tricks can still appear if they are in the votes. They just do not get a separate network to live in.
So is simpler plumbing than : one writing model, one frozen snapshot, the same A-versus-B pairs. It is not a filter for truth. It does not check Canberra. It only makes the chosen reply more likely than the rejected one.
Quiz
6 questions
Before moving forward, test your knowledge of what you’ve just learned!
1.DPO trains on:
2.DPO still uses:
3.Compared with RLHF, DPO has:
4.Bad votes in DPO:
5.“More likely than the loser, relative to the reference” means:
6.Pick RLHF or DPO, the desk fact stays:
0 of 6 answered
5 / 8