Skip to content
Home: How language models work

03Book 03 · Chapter 03 of 07· 5 min · Some relations are directions

Sliding across the map

Some word pairs share the same kind of relationship, such as a country and its capital. If you take the stored numbers for Canberra, subtract Australia, and add France, the result is often close to Paris. That is a test of the stored numbers. But everyday information that the model trains off is often not written with those clear correlations, so the test often fails in real life.

Here is the trick people like to show. Pins can line up in matching pairs. Australia sits a certain distance from Canberra. France sits about that same distance from Paris. So you can do this on the map: start at Canberra, step back toward “not Australia,” then step the same size toward France. If training lined the pins up, you land near Paris.

That only works when training put those pairs in a neat row. Rare names, words with two meanings, and most real sentences do not sit that neatly. Try kangaroo minus animal plus city. You do not get a tidy town. You get a jumble. And that’s what happens all too often unless we create good relationships in our training.

A normal chat does not do this Canberra-minus-Australia-plus-France calculation to pick the next word. If a reply changes Sydney to Canberra, or he to she, it is usually for a simpler reason: those words were stored as related, so both already had a real chance on the next-word list. , from Book 1, uses the other words in the sentence to decide which of those related words should sit high this time. That is the same loop as Book 2 — look at the words so far, score the possible next words, pick one. The add-and-subtract test is a way researchers inspect the stored numbers. It is not an extra step inside the reply.

Pick a slide. Some directions are real relations. Some are wishful thinking.

Canberra − Australia + France

lands near

Paris

Those pins lined up in training. A slide along “city of” lands near Paris. It is geometry, not a hidden atlas.

Quiz

6 questions

Before moving forward, test your knowledge of what you’ve just learned!

  1. 1.King − man + woman landing near queen means:

  2. 2.This kind of move:

  3. 3.Changing Sydney to Canberra in a reply is often:

  4. 4.Directions on the map come from:

  5. 5.If a move fails, the usual reason is:

  6. 6.This book’s map and Book 1’s attention:

0 of 6 answered

3 / 7