LoRA (Low-Rank Adaptation)
A fine-tuning technique that trains a small number of new parameters instead of updating the full model.
What is LoRA (Low-Rank Adaptation)?
Full fine-tuning of a 70B-parameter model requires updating 70 billion weights — impractical on most hardware. LoRA solves this by inserting tiny "adapter" matrices into the model and training only those. A LoRA adapter typically adds <1% new parameters.
The result: you can fine-tune Llama 3 70B on a single A100 GPU instead of an 8-GPU cluster. The adapters are also tiny to store (50–500MB) and switch in/out at inference — you can have dozens of task-specific adapters loaded on demand.
LoRA + its variants (QLoRA, DoRA, rsLoRA) are now the default fine-tuning method in production. Most "fine-tuned Llama / Mistral" deployments are LoRA-tuned, not fully fine-tuned.
LoRA puts custom fine-tuning within reach of every team — not just OpenAI-budget labs. Indian Gen AI engineers using LoRA on open models can build domain-specific systems at a fraction of API cost.
A Pune legal-tech company LoRA-fine-tuned Mistral 7B on 8,000 anonymised Indian contracts for clause classification. The adapter is 120MB; it runs on a single T4 GPU; inference costs are ~1% of GPT-4 API for the same task at higher accuracy.
Want to master this?
Learn LoRA (Low-Rank Adaptation) in a structured cohort
3-month live program with mentors, real projects, and 50+ partner placement support.