Book 05

How preferences steer it

A vote for reply A over reply B is a training event. A high score on the wrong quality teaches that quality. Qualities that never receive a score get little of that later training.

After this: See what an A-vs-B pick actually moves. · 8 chapters · about 45 minutes

1. 01A vote is a training eventA versus B5 min
2. 02Named qualitiesRubrics5 min
3. 03A learned scorerReward models5 min
4. 04Climb the scorerRLHF5 min
5. 05Skip the scorer, fit the votesDPO5 min
6. 06Climbing the wrong hillReward hacking5 min
7. 07What never gets a scoreUnnamed qualities5 min
8. 08Designing a score sheetSo the right quality can win5 min

Book 05 examNext: Why it looks like reasoning