The usual pick

The best way to decide what the next word in the reply should be seems obvious: take the token with the largest percent. That rule is called greedy decoding, or argmax — pick the maximum. And sometimes it works really well. But other times… not so much.

As an example, say the question is “What is the capital of Australia?” The model gives a percent to every possible next word. Canberra might get 81%. Sydney might get 12%. The leftover 7% is split among every other city the model knows. Those percents add up to 100%.

Using greedy decoding, the next word will always be Canberra, whether the question is run once or a thousand times. That is useful when there is one right answer.

But what if the top token is incorrect? Greedy will always keep that wrong token as the answer. Same question, different list: training text mentions Sydney constantly — the famous city, the one people name — so Sydney might sit at 71%. Canberra, the true capital, might sit at 18%. Greedy still writes Sydney, every time. The list is not a fact-checker. It is a list of what looked usual in the text the model learned from.