Fine-tuning re-trains an existing model on your data (typically 50 to a few thousand labeled examples) so the model's weights shift toward the behavior you need. It's the third option in most teams' playbook, not the first, and the gap between knowing what fine-tuning does and knowing when to use it costs more money than any bad model choice.
What fine-tuning actually changes
Fine-tuning adjusts a model's weights directly. That's different from two other common approaches:
- Prompting: shapes behavior at inference time without touching the model
- RAG: adds external knowledge at inference time without touching the model
Fine-tuning changes behavior and output style at a structural level. Results are consistent and don't cost extra tokens on every call. But they're also harder to revise: a behavior change requires another training run.
The practical rule from production use: fine-tuning is for form, not facts. It shapes how the model responds (specific JSON schema, a consistent tone, reliable refusal patterns) rather than what it knows. A fine-tuned model doesn't know your latest product data any more than the base model does.
The decision ladder
Work through these steps before committing to a training run:
- Prompt better. Most behavioral problems respond to clearer instructions, more examples in the system prompt, or a different framing. Exhaust this first.
- Add retrieval. If the model lacks knowledge, RAG is cheaper to build, faster to update, and easier to debug than fine-tuning. For a deeper look at how retrieval works, our embeddings explainer covers the vector side.
- Fine-tune. When prompting has hit its ceiling and the behavior you need is learnable from labeled examples: consistent output format, domain-specific terminology, hard refusal patterns.
- Distill. An advanced step: use a large teacher model to generate training data, then fine-tune a smaller student. Worth it only at high volume with a stable use case.
Most production problems resolve at step 1 or 2. Fine-tuning is a step-3 solution applied too often at step 1.
When fine-tuning earns its cost
Three scenarios where fine-tuning reliably pays off:
Format consistency at scale. If your app requires structured output and you're seeing 5-10% format failures with prompting, fine-tuning can push that below 1%. At 10,000 daily requests, each retry is a full-price call. What those retries actually add up to is the math most teams skip before committing to a training budget.
The system prompt itself is the problem. Fine-tuning can internalize instructions that currently live in a long system prompt, reducing per-call token cost. This is the same repeated-token cost that prompt caching solves for static prefixes, just addressed at the weight level instead of the cache level.
Domain-specific reasoning patterns. Legal, medical, or highly technical domains where the model consistently uses wrong terminology or applies incorrect reasoning despite detailed instructions. When the base model's general training simply doesn't cover your domain's conventions, labeled examples of the target behavior can make a measurable difference.
The break-even math
Fine-tuning has three cost components: training, inference (fine-tuned models often cost more per call than the base), and ongoing maintenance when the use case evolves.
A rough payback calculation:
- Measure your current failure rate on the task
- Calculate monthly retry cost: failure rate × requests/month × average tokens × per-token price
- Estimate post-fine-tune failure rate (target 80-90% reduction)
- Divide monthly saving into estimated training cost
If the payback period is under three months and the use case is stable, run it. For the inference-cost baselines needed for step 2, the LLM cost per feature breakdown covers chat, extraction, and summarization workloads across current model tiers.
The 2026 fine-tuning landscape
The options have narrowed significantly.
In May 2026, OpenAI announced that it is winding down self-serve fine-tuning. Organizations that hadn't run training jobs lost access to new training runs in July 2026; all customers lose the ability to start new training jobs in January 2027. Existing fine-tuned models continue running until their base model is deprecated. OpenAI's stated recommendation is to migrate to prompt caching with smaller base models.
Anthropic does not offer self-serve fine-tuning via the Claude API. It's available through Anthropic's enterprise partner program for approved use cases only.
Google Vertex AI still supports fine-tuning on Gemini models, with training costs charged per input token in the dataset. If you want a major-provider managed API for fine-tuning in 2026, Vertex is currently the primary option.
Open-weights models (Llama 4, Mistral, Qwen) are the lowest-cost path. LoRA and QLoRA techniques train roughly 1% of a model's parameters, which means a 7B+ model can be fine-tuned on a single 24GB GPU. Cloud compute cost for a LoRA run on a few hundred examples is typically $10-$50, with inference costs far below proprietary API rates once you're hosting the model.
FAQ
Is fine-tuning the same as training from scratch? No. Training from scratch requires billions of tokens of data and millions of dollars in compute. Fine-tuning starts from a pre-trained model and adjusts a subset of its weights. With LoRA, you're adjusting roughly 1% of parameters, which makes fine-tuning practical on a single GPU rather than a compute cluster.
What's the difference between fine-tuning and RAG? RAG adds context at inference time without touching the model's weights. Fine-tuning changes the weights at training time. RAG is the right choice when your knowledge changes frequently; fine-tuning works better when you need consistent behavioral patterns or output format that doesn't depend on fresh data. The two approaches also compose well: a fine-tuned model with a retrieval layer on top handles both form and knowledge.
How many training examples do I need? 50-100 high-quality examples is the practical minimum for most behavioral tasks. 500-1,000 gets you reliable results for most production use cases. Quality matters more than quantity: 50 well-labeled examples typically beat 500 inconsistently labeled ones.
Does fine-tuning let me remove my system prompt? Partially. Behaviors baked into fine-tuning don't need to be repeated in the system prompt, which reduces per-call token cost. But you typically still need some system prompt for context that varies by user or session.
How do I know if fine-tuning actually helped? Build a golden set of 20-50 examples before training: input prompts paired with the outputs you want. Run both the base model with your best prompt and the fine-tuned model against the same set. If the fine-tuned model doesn't win by a measurable margin, either the training data had issues or the task isn't suited to fine-tuning. The LLMTest benchmarking methodology uses pairwise comparison with an AI judge for exactly this kind of A/B evaluation.
When does fine-tuning not help? When the problem is knowledge-based (the model doesn't know a fact), when your use case changes frequently and would require constant retraining, or when you haven't thoroughly tried prompting first. Fine-tuning changes behavior, not knowledge.
Next steps
Not sure whether fine-tuning would move the needle for your use case? LLMTest's benchmarking tools run pairwise comparisons against your production prompts before you commit to a training budget.