How annotators train the model

People change what the model is trained to prefer. They do this with example replies, scores, and A-versus-B picks.

Annotator work reaches the model in three stages:

1. 1. Pretraining data

   Annotators typically don’t do much at this point.

   This is where the model practices determining which word comes next in a string by looking at huge amounts of books, code, and public web text. This is essentially the model’s raw default info — which cities get named, which pronouns follow which roles, which tone sounds like an encyclopedia. Eventually those defaults get adjusted: people write example replies, pick winners, and score named qualities, and training updates the numbers that decide which next word sits high. That later work is how the raw pile becomes useful in a specific use-case. But for now… it’s just a pile of largely undifferentiated relationships. Which is why a travel blog that calls Sydney the capital of Australia can sit high for a long time.

   Annotators encounter this stage as the starting habits they are asked to push against and refine.

2. 2. Demonstrations

   Here is where the annotator writes golden answers.

   They get a request and write the reply the model should copy — the target, as written by the (expert) annotator. Training then practices next-word generation on that reply.

   What changes: later outputs move toward that shape. Short true examples teach short-and-true. A beautiful paragraph that names the wrong city teaches beautiful-and-wrong.

   The annotator is not scoring two answers. They are handing the model one finished answer to imitate.

3. 3. Preferences and rubrics

   Here the annotator judges the model’s output in different ways.

   They either pick the better of two finished replies (A versus B) or score one reply on named qualities — true, short, careful, funny. They usually do not write the whole answer. What changes: later training pushes the next-word list toward the winners and toward the qualities that got scores.

   A score is a push on one specific thing. If the tutor reads a sentence that states (incorrectly) that Sydney is the capital of Australia, and marks that sentence as being polished from a language use-only perspective, later replies will lean toward that kind of linguistic output… while leaving the incorrect city intact. If “names the true capital” never appears on the sheet, that quality won’t change.

   Training only teaches success in the specific area(s) denoted by an annotation project.

   The annotator’s job is to provide a preference or to pick how well a model did on a scale (e.g., 1–7 or 1–10), or to write criteria that given responses do or do not meet. The model’s job is the next-word list: put more percent on the kind of reply that won in each of the above cases.

A high score on a polished wrong answer teaches polished-and-wrong. A high score on a short true sentence teaches short-and-true. Qualities that never receive a score get little of that later training.

What a score teaches

A real project will name its own qualities — maybe “true capital,” maybe “sounds like a person,” maybe “refuses the bad request.” Those names are the only handles later training can grab. A high score means: treat this reply as a win on those handles. Next time the model writes, more of the next-word list slides toward replies that look like that win. Score polish, get polish. Score the city, get the city. The sheet is not a vibe. It is a list of what counts as success.