Skip to content
Home: How language models work

03Chapter 03 of 09· 6 min · Weighted connections between

What means

assigns, for each , a set of percents over the other . Those percents say how much each other should affect the current one.

is a computation. For the the model is currently processing, it gives every other a share of the mix. Those shares are written as percents. They add up to 100%. A high percent means that other contributes a lot. 0% means it contributes nothing.

Below is a simple sentence. Click each “” (here, we’re calling each word a , even though often are not complete words) to see what other words’ “” (their relative importance to that word) is. Try changing the last word from “meowed” to “barked” and back again to show how a single word change (changing the “” for the sentence from “meowed” to “barked”) can create a massive effect on .

  • cat47%
  • dog15%
  • meowed13%
  • chased7%

Here, 'it' assigns the largest share to cat (about half). Meowing is what the cat does.

The referent depends on meaning

With “meowed,” “it” should put most of its percent on cat. With “barked,” most of the percent should go to dog. The rest of the sentence decides the referent. A model that makes that switch is resolving the pronoun.

Quiz

8 questions

Before moving forward, test your knowledge of what you’ve just learned!

  1. 1.In this guide, attention means:

  2. 2.In 'The dog chased the cat until it meowed,' 'it' should refer to:

  3. 3.The attention percents for one token add up to:

  4. 4.A weight of 0.7 is:

  5. 5.A weight of 0 means:

  6. 6.In 'The dog chased the cat until it barked,' 'it' should refer to:

  7. 7.Attention is computed:

  8. 8.When “it” should mean the cat, and the high percents sit on “dog”:

0 of 8 answered

3 / 9