What is LoRA (low-rank adaptation)?

Definition: LoRA, or low-rank adaptation, is a parameter-efficient fine-tuning method that freezes a model's original weights and trains compact add-on matrices, making it far cheaper to adapt a large model to a specific task.

How LoRA works

Instead of updating billions of weights, LoRA adds pairs of low-rank matrices to selected layers and trains only those, often well under one percent of the model's parameters. QLoRA combines this with quantization of the frozen base model to cut memory further. The trained adapter is a separate file that can be merged into the model or loaded on demand.

Why it matters

Training fewer parameters means less GPU memory, faster experiments and lower cost. Because adapters are compact, one base model can serve several tasks or customers by swapping adapters at inference time.

When to use it

LoRA suits teaching a model a format, tone, domain vocabulary or narrow task. It is not a reliable way to add fresh or frequently changing facts; RAG usually handles that better.

How Ryz helps

Ryz fine-tuning engineers train and evaluate adapters on your data in your cloud, working on your team.

Related

• LLM fine-tuning services
• RAG vs fine-tuning
• LLM fine-tuning
• Open-weight model