Introduction

When you ask a chat tool a question, the answer appears one piece at a time. That is not a person typing behind the screen. It is a program choosing the next small piece of text, adding it, then choosing again until the reply is finished.

Those small pieces are called tokens. A token is usually a word or part of a word. The program does not write the whole answer in one decision. At every step it asks one question: given everything so far, what comes next?

For a long time, computers only did what people spelled out in rules: if this happens, do that. That works for math and simple forms. It does not work well for ordinary language, because there are too many ways a sentence can go.

Around 2012, a different method started winning for pictures. Instead of writing rules for “this is a cat,” researchers gave computers huge collections of photos and let them practice guessing. Language work followed. By the mid-2010s, programs trained to guess the next word were already useful for translation and search.

In 2017, a design called the Transformer made that practice scale. Its main tool is attention: when the model picks the next token, it can look at every earlier token in the same step and decide which ones matter most. Those that matter most are the earlier words that most change what should come next — in “Sam ate the apple because it looked good,” “apple” matters more than “the.” Current large language models are built on that design.

After that, the models got much larger and practiced on much more text. They began to follow instructions, write code, and hold a conversation. The chat tools you use still run one loop:

1. Look at the tokens so far — your question plus whatever the model has already written.
2. Pick the next token.
3. Add it to the reply.
4. Repeat.

What looks like reasoning, tone, or a refusal is that loop after a huge amount of practice.

People still shape the result. Early practice uses books, code, and public web text. That teaches the model how language usually goes. Later practice uses example answers, scores, and comparisons written by people. That later work is the work this guide is about. It changes which next tokens the model prefers when it talks to a user.

This guide walks through the loop in order, then shows how annotator labels change it.