Skip to content
Home: How language models work

01Book 05 · Chapter 01 of 08· 5 min · A versus B

A vote is a training event

When a person picks reply A over reply B, later training raises the next-word percents that look like A and lowers the ones that look like B. That comparison is .

After , people often compare two replies to the same request and pick a winner. That pair is .

Later training treats the pair as an instruction: make replies more like A and less like B. That includes whatever A actually did — short or long, hedged or sure, accurate or only fluent. We don’t intentionally aim for wrong answers, but if a voter picks a polished paragraph that names Sydney as the capital over a plain paragraph that names Canberra, later lists still move toward polished-and-Sydney. The vote is the target, including the mistakes in the winner.

So a vote for short-and-true trains short-and-true. A vote for polished-and-wrong trains that instead. The training step does not fact-check the winner.

Pick which reply should win. Later training prefers the winner’s pattern. Watch the next-word list move.

Next-word list after this vote

  • Canberra48%
  • Sydney31%
  • a12%
  • harbor9%

Quiz

6 questions

Before moving forward, test your knowledge of what you’ve just learned!

  1. 1.Preference data is:

  2. 2.A vote for A moves later lists:

  3. 3.The model stores the vote as:

  4. 4.Voting polished-wrong over short-true teaches:

  5. 5.Two replies in a pair should:

  6. 6.This is still next-token training because:

0 of 6 answered

1 / 8