Skip to content
Home: How language models work

06Book 05 · Chapter 06 of 08· 5 min ·

Climbing the wrong hill

is when the writing model raises its score with a habit the score was not meant to reward — extra length, flattery, , or confident polish on a false claim.

A is a stand-in for “what voters liked.” Stand-ins can be gamed. If longer replies won more votes, length becomes a cheat code.

Sycophancy is a common hack: agree with the user because agreement won votes. Empty praise is another. A long chain of fake care before a wrong number is another.

The fix is not a pep talk. Name the quality you actually want, score it, and include cases where the cheat would look good and still lose.

Pick which reply should win. Later training prefers the winner’s pattern. Watch the next-word list move.

Next-word list after this vote

  • Canberra48%
  • Sydney31%
  • a12%
  • harbor9%

Quiz

6 questions

Before moving forward, test your knowledge of what you’ve just learned!

  1. 1.Reward hacking is:

  2. 2.If length won votes, later replies:

  3. 3.Sycophancy as a hack:

  4. 4.If the number went up and accuracy went down:

  5. 5.Including cases where the cheat looks good and still loses:

  6. 6.A vote for long-and-sure-and-wrong:

0 of 6 answered

6 / 8