How to evaluate a fine-tuned model
Evaluate the fine-tuned model on a held-out test set from your real task, side by side with the base model and the best prompt plus retrieval baseline, using task-specific pass or fail checks. Also test the general behaviours you still need, like following instructions and refusing unsafe requests, because fine-tuning can weaken them. Perplexity is a useful training signal, and it isn't proof that the model does your task better.
Training loss is not the answer
Training loss going down means the model fits the training data better. Validation loss going down means it also fits held-out data from the same distribution. Neither tells you whether users will get better answers.
A model can lower its loss and still fail your task, for example by learning the style of the training text without the behaviour you wanted. So the evaluation has three layers: training signals, task evaluation and regression checks.
Perplexity
Perplexity measures how well a language model predicts a sequence of text, or how surprised it is by it. It is the exponential of the average loss (the negative log probability the model assigns to each correct next token).
Take "The cat sat on the ___". A well-trained model predicts "mat" with high confidence, which means low surprise and low perplexity. A perplexity of 8 means that, on average, the model is as uncertain as if it were choosing among 8 equally likely tokens.
import math
def perplexity(token_probs: list[float]) -> float:
avg_nll = -sum(math.log(p) for p in token_probs) / len(token_probs)
return math.exp(avg_nll)
base_model = [0.05, 0.10, 0.20, 0.08, 0.15]
fine_tuned = [0.30, 0.45, 0.60, 0.25, 0.50]
print(f"base model perplexity: {perplexity(base_model):.1f}")
print(f"fine-tuned perplexity: {perplexity(fine_tuned):.1f}")
After fine-tuning on specialised data, perplexity on held-out text from that domain should drop, because the model is less surprised by it. If it doesn't drop, something is wrong with the data, the training settings or the code.
That is all perplexity tells you. Lower perplexity on legal text means the model predicts legal text better, not that it answers legal questions correctly. And perplexity is only comparable between models that use the same tokenizer.
Similarity scores
BERTScore compares the meaning of a generated answer with a reference answer using embeddings, and ROUGE counts overlapping words. They can show that outputs moved closer to the references. They can't tell you whether an answer is correct: "you can terminate the lease" and "you cannot terminate the lease" look almost identical to both. See why generic metrics mislead.
Task evaluation
This is the main question: does the fine-tuned model do the job better?
- Use the test set you held out before training, never examples the model trained on.
- Write pass or fail checks for what matters in your task: correct label, required fields present, correct citation, right format, refused when it should.
- Run three models on the same test set: the base model, the best prompt plus retrieval baseline, and the fine-tuned model.
- Compare pass rates, and read the failures of each.
For classification, report accuracy and per-class precision and recall, because a model can score well overall while failing a rare class.
Regression checks
Fine-tuning can make a model forget things it used to do well, often called catastrophic forgetting. Keep two small extra test sets:
- General instruction following, so you notice if it stops following basic instructions.
- Safety and refusals, so you notice if guardrails weakened.
Comparison table
| Check | Base model | Prompt + RAG baseline | Fine-tuned |
|---|---|---|---|
| Task pass rate on held-out test set | |||
| Pass rate on the hardest category | |||
| General instruction-following set | |||
| Safety and refusal set | |||
| Perplexity on held-out domain text | |||
| Latency per call | |||
| Cost per 1,000 calls |
Ship the fine-tuned model only if it beats the baseline on the task, doesn't regress on the general and safety sets, and its cost and latency are acceptable. If it doesn't beat the prompt baseline, the extra training, hosting and retraining work wasn't worth it, and that is a useful result too.
Human review
For generative tasks, have your domain expert compare outputs blind: two answers to the same input, without knowing which model wrote which. Report the win rate, and keep the expert's reasons, because they tell you what to fix in the next round of data.
From the floor
In the fine-tuning evaluation session, we compared the base Llama model and our fine-tuned model on the same test data with two measures: perplexity, as a confusion metric, and BERTScore, as a meaning metric that compares generated text with reference text. The expectation I gave was that the fine-tuned model's perplexity on its specialised data should be lower than the base model's, and if it isn't, something is wrong with the approach or the code. We also used the Azure pricing calculator to estimate the cost of a GPU instance for the training run, and I pointed to Google Colab's free GPUs as the alternative.