04Book 06 · Chapter 04 of 08· 5 min · Verifiable rewards
Pay for checkable answers
On math and code, a program can mark an answer right or wrong. Training that pays for a correct result — and, if you want, for writing the steps — shifts later replies toward work that gets the check right.
A person voting “this looks smart” is a soft signal. Two tutors can disagree, and “looks smart” can mean length, polish, or a confident tone. A unit test is a hard signal: the code runs or it does not. 17 × 19 is 323 or it is not. That yes-or-no can be the score later training uses. The update then raises next-word percents for replies that passed the check, and lowers them for replies that failed — instead of raising percents for whatever sounded careful.
Reinforcement learning here means: run the model, check the result, then shift next-word percents toward replies that passed. “Landing” is the check — 323, or tests that pass. Teams often add a second check: did the reply include written steps at all? Without that, the model can learn to emit only the number. With only that, it can learn to emit steps that are never used. The designed score is what later replies optimize for.
None of this replaces next-word prediction. The model still builds a list of likely next pieces and picks. The checker’s 1 or 0 is only the score used to shift those lists later: replies that got a 1 become more usual; replies that got a 0 become less usual. The architecture did not change. The target did.
Quiz
6 questions
Before moving forward, test your knowledge of what you’ve just learned!
1.A verifiable reward is:
2.Paying for a scratchpad as well as the answer:
3.Polish alone on a unit test:
4.This is still next-token training because:
5.Math and code are common here because:
6.A fuzzy vote plus a checker:
0 of 6 answered
4 / 8