04Book 05 · Chapter 04 of 08· 5 min ·
Climb the
trains the writing model to get higher scores from the . A penalty keeps its next-word lists from drifting too far from the model, which limits odd high-score tricks.
stands for reinforcement learning from human feedback. In the usual picture: an model writes replies, the scores them, and the writer is updated to get better scores.
A second copy of the model is often kept as a reference. A penalty grows if the writer’s next-word lists drift too far from that reference. That is the stay-near-home rule (a KL penalty, in the jargon).
PPO (proximal policy optimization) is one common recipe for taking that step. The writing model produces a few replies. The scores them. Training then raises next-word percents for the higher-scoring replies and lowers them for the lower-scoring ones. “Proximal” means each update is capped: this step may only move a little from the version you just trained, so one batch cannot rewrite the whole model. Combined with the stay-near-home penalty, you get two limits — don’t leap away from the last version, and don’t leap away from the example-trained model. Watch for tricks the still likes.
Quiz
6 questions
Before moving forward, test your knowledge of what you’ve just learned!
1.RLHF uses:
2.The stay-near-home penalty:
3.Tricks appear when:
4.PPO in this guide is:
5.The reference model is usually:
6.RLHF still writes by:
0 of 6 answered
4 / 8