06Book 04 · Chapter 06 of 07· 5 min · Scarce cases punch above their count
One sofa vs a rare platypus
One training example shifts next-word percents a little. An example the model has rarely seen — a rare city, a hard pronoun, a careful refusal — shifts them more than another fluent paragraph it has already seen millions of times.
The interactive below starts from a list for “The capital of Australia is….” Sydney leads because travel text named it constantly. Add a pile of those easy blogs and the list barely budges — the model already saw that pattern. Add one labeled target reply that says Canberra, and the list moves more. That is the point: scarce, correct examples do more per item than another fluent paragraph the model has already seen a million times.
A hard pronoun, a rare domain fact, a careful refusal, a platypus that is not a duck: the model has seen fewer of those. Each one is scarce fuel.
Skip the hard cases and you withhold the scarce fuel. Pack a project with only easy polish and you train polish. The wheel already knew how to sound nice.
The capital of Australia is....
- Sydney58%
- Canberra24%
- Melbourne12%
- Hobart6%
Easy and already common
“Sydney’s harbour is the capital’s jewel…” — a travel blog the model has seen thousands of times.
0
Scarce and labeled
Request: “What is the capital of Australia?” Target reply: “Canberra.”
0
Starting list after : travel text named Sydney constantly, so Sydney leads.
Quiz
6 questions
Before moving forward, test your knowledge of what you’ve just learned!
1.One example:
2.Hard cases matter more per example because:
3.A project of only easy polish:
4.Skipping hard cases:
5.Another easy fluent-and-right paragraph:
6.This is why managers care about the mix of examples:
0 of 6 answered
6 / 7