01Book 04 · Chapter 01 of 07· 5 min ·
First practice: the melting pot
Before anyone scores a chat reply, the model practices next-word on a huge melting pot of books, code, and web text. That practice fills the map and the usual lists.
Book 2’s wheel had to get its percents from somewhere. Book 3’s map had to get its pins from somewhere. Both start here. The model reads enormous text — books, code, websites, forums — and practices the same loop: given the words so far, pick the next piece.
That stage is . It is how “Canberra,” “function,” and “once upon a time” get pins on the map and slices on the wheel. It is also how “Sydney is the capital” travel-blog sentences get a slice, and how zzqxt sneaks on as a rare misspelling.
Nobody is scoring chat quality yet. Right now, the model is just taking everything in, so down the line it can learn how all those millions of pieces relate to each other and (eventually) turn into helpful answers it can give back to you!
Don’t miss Sydney, the capital of Australia! █
Travel blogs name Sydney constantly. That slice can sit high later — fluent, and wrong.
Quiz
6 questions
Before moving forward, test your knowledge of what you’ve just learned!
1.Pretraining is:
2.The task in pretraining is:
3.Later replies still show pretraining because:
4.Pronoun patterns from web text:
5.Pretraining data includes:
6.This stage does not:
0 of 6 answered
1 / 7