Skip to content
Home: How language models work

04Chapter 04 of 09· 6 min · How one step is computed

, , and

Each word makes three lists of numbers: a (this word’s question), a (this word’s tag for others to match), and a (the meaning that gets mixed in).

Inside the model, each is stored as a list of numbers. That list is called a . compares and mixes those lists. Each step uses three per :

  • — this word’s question to the rest of the sentence (for “it”: which earlier word does this stand for?).
  • — this word’s tag of what it is, so other words can match against it.
  • — the meaning copied in from this word after the percents are set.

The model scores each against every , turns those scores into percents (the ), and uses the percents to mix the . Matching and contributing information are separate.

Below is a simple sentence. Click each “” (here, we’re calling each word a , even though often are not complete words) to see what other words’ “” (their relative importance to that word) is. Try changing the last word from “meowed” to “barked” and back again to show how a single word change (changing the “” for the sentence from “meowed” to “barked”) can create a massive effect on .
: it. Match scores as percents:

  • The

    4%

  • dog

    15%

  • chased

    7%

  • the

    4%

  • cat

    47%

  • until

    4%

  • it

    7%

  • meowed

    13%

The for “it” matches “cat” most strongly, so most of the mixed comes from “cat.”

Quiz

8 questions

Before moving forward, test your knowledge of what you’ve just learned!

  1. 1.A query is:

  2. 2.Why keep key and value separate?

  3. 3.A key is:

  4. 4.A value is:

  5. 5.A vector is:

  6. 6.After the model scores each query against every key, it:

  7. 7.Match scores are divided by 8 because:

  8. 8.Matching and contributing information:

0 of 8 answered

4 / 9