05Book 04 · Chapter 05 of 07· 5 min ·
A sticky note on a huge book
You do not have to retrain every number in the model to change how it replies. You can leave the original training in place and train only a small extra set of numbers that get added on top. That extra set is cheaper to train, and you can swap it without replacing the whole model. The usual name for this is .
Full updates a huge number of . That is expensive, and it can scribble over useful — including the good clusters (yes, actually, the capital of Australia is Canberra) you already paid for.
(low-rank adaptation) leaves those original numbers unchanged. It trains two much smaller tables of numbers, multiplies them together, and adds the result on top of the original numbers. Only those small tables are saved. To change the habit later, you swap the small tables. You do not replace the original model.
is the same method with the original numbers stored in a more compact format, so the job uses less memory. What you run is still: the original model plus the small extra tables.
What is the capital of Australia?
| Slot | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|---|
| Stored original | 2.14 | 1.37 | 0.28 | -0.81 | 1.12 | 0.19 | 0.94 | -0.07 |
| Added by extra tables | — | — | — | — | — | — | — | — |
| Number used | 2.14 | 1.37 | 0.28 | -0.81 | 1.12 | 0.19 | 0.94 | -0.07 |
In real LoRA the “add” row is the product of two small tables, not eight numbers written by hand.
Next-word list
- Sydney54%
- Canberra31%
- Melbourne15%
These eight numbers stand in for a tiny slice of the original model. Training blogs named Sydney often, so Sydney still leads on a capital question.
Quiz
6 questions
Before moving forward, test your knowledge of what you’ve just learned!
1.LoRA trains:
2.The big book is:
3.Projects like adapters because:
4.QLoRA:
5.A small notebook can still steer lists because:
6.Full fine-tuning compared with LoRA:
0 of 6 answered
5 / 7