Prompt engineering is a great starting point — but at production scale, a fine-tuned 7B model running on your own infrastructure will outperform a frontier model on generic prompts every time. Here is why, and when to make the switch.
The seduction of the big model
When you first get a working prototype using GPT-4 or Claude, the result feels almost magical. You paste in a system prompt, a few examples, and suddenly the model does something that would have taken months to build in traditional software. It is genuinely impressive.
But then the invoice arrives. And the latency numbers. And the first time the model confidently produces a wrong answer in a slightly different phrasing than your examples.
This is the moment most organisations face a choice: keep optimising the prompt, or start building something you actually own.
Three reasons fine-tuning wins at scale
1. Cost drops by an order of magnitude
A frontier model API call costs roughly 10–50× more per token than running an equivalent-quality fine-tuned 7B or 13B model on your own GPU. At low volume this is irrelevant. At ten thousand calls per day it is a significant line item. At a million calls per day it defines whether the product is economically viable.
Fine-tuned smaller models are not just cheaper to run — they can be distilled from the larger model's outputs, which means you get much of the quality at a fraction of the cost.
2. Domain accuracy is higher, not lower
General-purpose frontier models are optimised to be useful across every possible task. That breadth is a liability for narrow, high-stakes processes. A model fine-tuned on a thousand examples of your invoice classification task, your document enrichment workflow, or your sales proposal format will simply produce better outputs for those specific tasks than a generic model given even a very good prompt.
This is not intuitive — people assume bigger always means better. But for constrained domains, specificity wins.
3. You accumulate AI capital, not AI costs
Every prompt you write is a cost — it has to be maintained, updated when the model changes, and renegotiated every time the vendor releases a new version. Every fine-tuning run you complete is an asset: a checkpoint that belongs to you, improves on your data, and compounds with each iteration. The seventeenth fine-tuning cycle of your model is dramatically better than the first. The seventeenth version of your system prompt is... marginally better, at best.
When to stay on prompting
Fine-tuning is not always the right answer. If you have fewer than a few hundred good examples, your task changes frequently, or you need the model to generalise across a very wide range of inputs, a well-crafted prompt against a frontier model is the right tool. Prompting is also the correct starting point: you cannot fine-tune without first knowing what "good" looks like, and the fastest way to discover that is to iterate on prompts.
The practical heuristic: if you have found a prompt that works reliably and you are running it more than a few thousand times per month, it is worth exploring fine-tuning. The crossover point where a fine-tuned small model becomes cheaper than a frontier API is usually somewhere between 5,000 and 50,000 calls per month, depending on token counts.
The Learning Loop approach
We treat the prompt optimisation phase as a data collection exercise: every successful and unsuccessful output is a labelled example for the fine-tuning run that comes next. This means the two approaches are not in competition — they are sequential stages of the same process. You start with prompting because it is fast and flexible. You graduate to fine-tuning because it is cheap and specific. And you keep the artifacts either way.
If you are at the point where your AI experiments are working but the costs or latency are becoming a concern, get in touch — this is exactly the transition we help organisations navigate.