Skip to content
Home: How language models work

05Book 06 · Chapter 05 of 08· 5 min ·

Score a group of tries

samples several full replies to the same prompt and scores each one. Replies better than the group average become more likely. Replies worse than the average become less likely.

(group relative policy optimization) trains on several full replies to the same question at once. Score each reply. Compare each one to the average of that group. Replies above the average become more likely. Replies below it become less likely. The group itself is the yardstick, so you do not need a second model guessing “how good was that step.”

A reply that scored above the group average becomes more likely next time. A reply that scored below it becomes less likely. “Better” here means better than the other tries at this same question, not better than some universal scale. Some recipes train a second network to guess “how good was that step.” skips that network. The group’s own scores are the comparison.

You do not need the algebra. The working picture is: several full replies to one prompt, a score on each, a nudge toward the ones that beat the group and away from the ones that lost to it. Next-word percents move. The model that writes the next reply is the same kind of model as before.

Four replies to the same question. Each gets a 1 or a 0 from a checker. Compare each score to the group average, then apply one step.

What is 17 × 19?

Checker: 1 if the answer is 323, else 0. Group average: 0.50

  • 17 × 20 = 340. 340 − 17 = 323. So 323.

    score 1 · above avg ↑

    Likely next time28%

  • 326

    score 0 · below avg ↓

    Likely next time22%

  • Let’s think carefully… (80 words) …323.

    score 1 · above avg ↑

    Likely next time26%

  • 17 × 20 = 340. Minus 20 is 320. Wait, 319.

    score 0 · below avg ↓

    Likely next time24%

Before the step, the four replies are about equally likely. Press the button to nudge next-word chance toward the ones that beat the average.

Quiz

6 questions

Before moving forward, test your knowledge of what you’ve just learned!

  1. 1.GRPO samples:

  2. 2.A try above the group average:

  3. 3.Compared with a value network, GRPO:

  4. 4.The prompt in a GRPO group:

  5. 5.Checkers plus GRPO:

  6. 6.You need the picture, not the algebra, because:

0 of 6 answered

5 / 8