Serving fine-tuned models and LoRA adapters
If you have one fine-tuned model, merge the LoRA adapter into the base weights and serve it like any other model. If you have many adapters, one per customer or per task, keep one copy of the base model in GPU memory and choose the adapter per request, using an inference server that supports multiple LoRA adapters, such as vLLM. Version every adapter with the base model it was trained on, because an adapter only works on that base.
Option 1: merge and serve
A LoRA adapter is a small set of matrices added to the base model's weights. You can add them permanently, which gives you a normal model with the fine-tuning built in.
from transformers import AutoModelForCausalLM
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B")
model = PeftModel.from_pretrained(base, "output/legal-adapter")
merged = model.merge_and_unload()
merged.save_pretrained("output/legal-merged")
The merged model serves like any model, with no extra cost per request. It is the simplest option when you have one fine-tuned version. If you trained with QLoRA, merge into a 16-bit copy of the base model, then quantize the merged model again if you need it smaller.
Option 2: many adapters on one base
Ten customers each want their own fine-tuned 7B model. Ten full copies need ten times the GPU memory. Ten adapters are a few megabytes each, and they share one base model.
Inference servers like vLLM can load several LoRA adapters next to one base model and choose the adapter per request, even batching requests for different adapters together. Research systems like S-LoRA and Punica showed this can serve thousands of adapters on a single GPU, with adapters moved between CPU and GPU memory as needed.
vllm serve meta-llama/Llama-3.2-1B \
--enable-lora \
--lora-modules legal=output/legal-adapter support=output/support-adapter
Each request then names the adapter in the model field of the OpenAI-compatible API ("model": "legal"), so your router or gateway can choose it by customer or task.
Trade-offs
| Merged model | Multi-adapter serving | |
|---|---|---|
| Setup | Simplest | Server must support LoRA |
| Cost per request | Same as base | Small overhead for applying adapters |
| GPU memory | One full model per version | One base plus small adapters |
| Many customers or tasks | Expensive | Designed for it |
| Switching versions | Deploy a new model | Load a new adapter |
Versioning and rollout
- Record the base model, its exact version and the training data version for every adapter.
- When the base model changes, retrain the adapters, because an adapter trained on one base doesn't transfer to another.
- Evaluate each new adapter on its test set before it takes traffic, and roll it out gradually like any model change.
- Keep the previous adapter available for instant rollback.
See rollouts and model upgrades.
Hosted fine-tuning
If you fine-tune through a provider's API, the provider serves the model for you and gives you a model ID. You still need to version it, evaluate it and plan for the base model being retired, because providers deprecate older models on a schedule.
From the floor
In the fine-tuning sessions, the last steps were saving the fine-tuned model and tokenizer to an output folder and running sample predictions, and in the training workflow demo, we wrapped the fine-tuned DistilBERT in a small Flask app with a predict endpoint, because serving is just a wrapper around inference.