What is a LoRA adapter, in practical terms?
Why Full Fine-Tuning Is Often Impractical
Full fine-tuning updates every parameter of a model. For a 7-billion-parameter model, that means optimizer states, gradients, and updated weights consuming tens of gigabytes of VRAM. Each domain you adapt requires storing a complete copy of the model. For organizations needing dozens of specialized variants, storage and deployment costs scale linearly with the number of domains, and switching between them means loading entirely separate models into memory.
LoRA sidesteps this by freezing the base model and training only small rank-decomposition matrices injected into each layer. A typical adapter might contain 1 to 5 percent of the original parameters. This is why an orchestrator like Molly can keep a library of lightweight domain specialists over a single quantized base and route each request to the appropriate adapter, swapping them in and out with negligible overhead instead of reloading entire models.
What a LoRA Adapter Actually Does
A LoRA adapter freezes the pretrained model's weights and injects two small trainable matrices into each targeted layer. Their product approximates the weight delta that full fine-tuning would produce, but at a rank far below the original dimensionality. Only these low-rank matrices are stored, shrinking the adapter to a fraction of the base model's size while leaving the base itself untouched and swappable.
The trade-off is capacity. A rank-16 adapter cannot represent every update a full fine-tune can, so complex behavioral shifts or broad domain reorientation may underperform. The flip side is composability: many adapters can coexist over one shared base, each specializing in a narrow domain and loading in milliseconds. An orchestrator can route each prompt to the matching specialist, swap adapters on the fly, and avoid running a separate full model per task.
How LoRA Changes Your Training Workflow
With LoRA, you freeze the base model and train only low-rank decomposition matrices injected at spec
Common questions
How do I choose the right rank value for my LoRA adapter?
Rank controls the adapter's expressiveness. Rank 4–16 works for narrow tasks like style transfer; 32–64 suits broader instruction tuning. Higher rank captures more nuance but increases parameters, memory, and overfitting risk. Start low, evaluate, and scale up only if performance plateaus.
Can I merge a trained LoRA adapter back into the base model weights?
Yes. LoRA adapters are low-rank decompositions (A×B) that can be added to the frozen weight matrix at inference or permanently merged via simple matrix addition. Merging eliminates runtime overhead but loses the ability to hot-swap adapters later.
Are LoRA adapters portable across different base models?
No. A LoRA adapter is tied to the specific base model and architecture it was trained against. Dimensionalities, tokenization, and layer structure must match exactly. Even same-family models with different parameter counts require retraining the adapter from scratch.