Skip to content
Home: How language models work

02Chapter 02 of 09· 5 min · Sequential processing

Why older models were slow

Before 2017, most models processed one word at a time. Word 12 waited for word 11 to finish.

Take a sentence of eight words: “Older models read one word at a time.” An older model finished those words in order. Word 4 started only after word 3 was done. Word 3 started only after word 2 was done. Press play below to watch that wait.

After each word, the model kept one running summary of everything it had read so far. The next word updated that summary. By word 8, the first word had been mixed into the summary seven more times. Details from the start got weaker.

Left: each word waits for the one before it. The memory of the first word gets weaker as each word moves on. Press play to see how older models would process a sentence vs. newer ones.

Older model · one word at a time

Oldermodelsreadonewordatatime

Each word waits for the one before it. After each word, the model keeps one running summary. By the last word, the first word has been mixed in again and again, and those details are weaker.

· all at once

Oldermodelsreadonewordatatime

Every can use every other in the same step. Word 8 can use word 1 without waiting through 2–7.

The dropped that wait. Every looks at every other in the same step. That look is . Stacked in layers, is enough to train the models you use.

An older model reads this sentence one word at a time. Each running summary is built from the summary before it plus the new word.
WordRunning summary
  1. OlderOlder
  2. modelsOlder models
  3. readOlder models read
  4. oneOlder models read one
  5. wordOlder models read one word
  6. atOlder models read one word at
  7. aOlder models read one word at a
  8. timeOlder models read one word at a time

Quiz

6 questions

Before moving forward, test your knowledge of what you’ve just learned!

  1. 1.Why were older one-word-at-a-time models slow to train on long text?

  2. 2.What often happened to information from the start of a long sentence?

  3. 3.In “Older models read one word at a time,” to finish word 4 (“one”), an older model first finished:

  4. 4.t − 1 means:

  5. 5.After each word, the older model kept:

  6. 6.A Transformer reads a long prompt by:

0 of 6 answered

2 / 9