DAY 53 / 210

Foundations of Parameter-Efficient Fine-Tuning

Phase 2 begins the shift from prompting to adaptation; understanding PEFT techniques like LoRA is essential before touching weights in the StartupTribunal codebase. This day establishes the mental model for choosing adaptation methods that respect compute limits and preserve base capabilities.

⏱ 45 min target📝 3 quiz Qs

Resources

readingHugging Face
PEFT documentation
Overview and LoRA sections
20 min
readingarXiv
QLoRA: Efficient Finetuning of Quantized LLMs
Abstract and Section 3
25 min

Deliverable

Journal entry containing a one-paragraph fine-tuning method choice for StartupTribunal with explicit LoRA rank and target modules

Quiz · 3 questions

1. Why does LoRA freeze the base model weights during adaptation?

To reduce memory usage by training only low-rank matricesTo increase training speed via full gradient updatesTo enable zero-shot generalization without any dataTo prevent catastrophic forgetting of pre-training

2. Name one key hyperparameter in LoRA besides rank and explain its role in adaptation capacity.

3. For a legal-domain model like StartupTribunal, which layers would you target with LoRA and why?

Journal

Time spent (minutes)

Blockers

Commit / PR links (one per line)