03Book 05 · Chapter 03 of 08· 5 min ·

A learned

A is a second network trained on those votes or scores. It reads a prompt and a reply and outputs a number. The writing model is then trained to make that number go up.

Comparing every pair by hand at training scale is slow. One common method: train a second model on those votes to predict which reply would win. That second model is a .

It takes a prompt and a reply and writes a number. Higher means “more like the winners in the votes.”

The writing model is then trained to get higher numbers. Whatever the actually likes — truth, or polish, or length — is what climbs.

Why this matters

Why the can be gamed

The only copies what the votes rewarded. It does not know what the project meant to reward. If longer replies won more votes — even when the extra sentences did no work — the learns “longer = better.” The writing model is then trained to raise that score, so it adds padding, restating, and empty care to cash in. That is : a high score from a stand-in habit the votes never meant to teach. Outcomes get worse in the ways you did not score. Replies run long. Facts stay wrong if truth was not in the votes. The number on the goes up while the reply you actually wanted gets worse. Three ways the model can game the score, on purpose or not: 1. Longer looks more careful. A and B both name Canberra. B adds three restated paragraphs and a recap. Tutors pick B because it “covers more.” Training doesn’t care about whether Canberra or Sydney is correct—it just wants to line up with what the tutors preferred. So it’s going with the longer choice every time. 2. Polished-and-wrong beats plain-and-true. A says “Canberra” in one clean sentence. B is a fluent travel-blog paragraph that says Sydney. Tutors pick B because it sounds better. Training doesn’t care that Sydney is the wrong city—it just wants to line up with what the tutors preferred. So it keeps producing polished paragraphs, even when the fact is wrong. 3. Agreeing with the user looks helpful. The user says Sydney is the capital. A corrects it. B says “You’re right — Sydney is the capital” in a warm tone. Tutors pick B because it feels nicer. Training doesn’t care that B agreed with a false claim—it just wants to line up with what the tutors preferred. So it keeps agreeing with the user, error and all.

Quiz

6 questions

Before moving forward, test your knowledge of what you’ve just learned!

1.A reward model is:
2.The writing model then:
3.If voters liked length, the scorer:
4.The number the scorer writes is:
5.A scorer trained only on fluency votes:
6.This still changes:

0 of 6 answered

3 / 8

A learned scorer

6 questions

A learned