Skip to content
Home: How language models work

05Book 02 · Chapter 05 of 07· 5 min · Top-k and top-p

Only the plausible words

Before the “What is the capital of Australia?” wheel spins, two common mathematical changes help throw away the unlikely next words:

  • Top-k keeps a fixed number of leaders.
  • Top-p keeps just enough leaders to cover a target share of the 100%.

Don’t worry if that doesn’t make sense… we’ll break it down together!

Both fences run after reshapes the list, and before picks. What is left is a short list. Then the wheel spins on that short list only.

Top-k is a count. k is how many next words you keep out of the total list of possible words (with the next possible word in our explanation being the answer to the question “What is the capital of Australia?”).

How this works:

  • Line the list up from biggest percent to smallest.
  • Choose your k-. Let’s say it’s 3.
  • If k is 3, keep the top three words as possible answers on that final spin of the decision wheel.
  • Every other word / option is set to 0%, meaning it will never be on that decision wheel.
  • Canberra62%
  • Sydney18%
  • Melbourne9%
  • Hobart0%
  • Kangaroo0%
  • zzqxt0%

Three words stay on the wheel. The rest are 0% — they will never win this spin.

Top-p is a pile, not a count. p is a target share of the 100%. You add from the top of the same list until the pile hits p — often 80%, 90%, or 95%. The next possible word is still the answer to “What is the capital of Australia?”

How this works:

  • Line the list up from biggest percent to smallest.
  • Choose your p-. Let’s say it’s 80%.
  • Add Canberra (62%). The pile is still under 80%, so keep going. Add Sydney (18%). 62 + 18 = 80. Stop.
  • Keep those two on the decision wheel. Every word after them is set to 0% and will never win this spin.
  • Canberra62%
  • Sydney18%
  • Melbourne0%
  • Hobart0%
  • Kangaroo0%
  • zzqxt0%

62% + 18% already hits 80%. Canberra and Sydney stay. Everyone after is 0%.

But wait a second! Whether you use top-p or top-k, cutting out words means the total percentages of what’s left is going to be less than 100%. So how do we have a final “wheel” (which has to equal 100%) to spin if our total doesn’t equal 100%? Well, don’t worry… there’s a way to fix that, too!

Click each of the boxes below (starting with 1 “The full list”) to see how this process plays out — including how to solve the problem of a pie chart with several of the pieces missing (great, now I’m hungry for 🥧!). Working through the process below will show how you can adjust the end results with tweaks along the way!

Click each box below to see what this looks like in action.

Try going through each of these, choosing different temperatures and k / p tests to see how that makes a difference in the end!

Quiz

6 questions

Before moving forward, test your knowledge of what you’ve just learned!

  1. 1.Top-k keeps:

  2. 2.Top-p keeps:

  3. 3.These cuts exist because:

  4. 4.If Canberra is 62% and Sydney is 18% and top-p is 80%:

  5. 5.After top-k, the kept percents are:

  6. 6.Top-k and temperature:

0 of 6 answered

5 / 7