07Chapter 07 of 09· 5 min ·
Why word order matters
finds connections between words. marks each word’s place. That is how “dog bites man” and “man bites dog” stay different.
connects every word to every other word. marks where each word sits. That is how “dog bites man” and “man bites dog” stay different.
Before runs, the model adds a place-mark to each word. The usual method uses two number patterns called sine and cosine. A wave here is a number that goes up and down as you move along the sentence. Fast waves change every word. Slow waves change over many words. Mixing fast and slow waves gives each place its own pattern. Nearby places get similar patterns.
The dog is the one that bites.
The man is the one that bites.
Quiz
6 questions
Before moving forward, test your knowledge of what you’ve just learned!
1.Why add positional encoding?
2.If a demonstration puts the answer last, after a long aside, the model learns:
3.Without place-marks, “dog bites man” and “man bites dog” are:
4.Sine and cosine in this chapter are:
5.Nearby places in the sentence get:
6.Attention supplies the connections between tokens. Order comes from:
0 of 6 answered
7 / 9