Large Models, LoRA and Parameter-Efficient Fine-tuning

2 min
  1. Which parameters does LoRA train?
  2. How many parameters do two rank-r matrices around a d×k layer add?
  3. Does LoRA guarantee faster inference?
  4. Why keep a general regression set?
  5. [E] Which four artifacts identify an adapted model?

Answers. 1. Low-rank matrices while the base model stays frozen. 2. r(d+k). 3. No, not without fusion and benchmark. 4. To detect lost capabilities. 5. Base, tokenizer, adapter and configuration/prompt versioned separately.

Activity — Budget and LoRA regression

For a 4096×4096 matrix and rank 8, compute LoRA's trainable parameters and the ratio to full fine-tuning. Then define a task set, a regression set and three safety tests. Success: base, tokenizer, adapter and prompt are versioned separately.

Preview — the rest of the lesson is for enrolled readers.

Already enrolled with a code?

Your access is tied to your account, not to this link. Sign in with the same email you used in class: your course is waiting, no need to enter the code again.

Sign inNo account yet? Create one
This lesson is part of the “Modern Deep Dives” module

The first modules of the course are open to everyone. For the rest you have three options: buy this course once and for all, subscribe, or enter the code handed out in class.

Subscribe — 19 $ US/monthBack to the outline
Are you a student on this course?

The code is tied to your account: sign in or create an account and it will be applied automatically when you come back.

No account yet? Create one