When the most expensive AI model can be the cheapest choice
The model bill is only one part of the engineering bill. The useful question is what it costs to get a change understood, checked and accepted.
Unpublished prototype draft · Rob Barrett
This is a proposed decision framework, not a model benchmark. All figures below are hypothetical.
Buy an accepted change
A model that produces a plausible patch cheaply can still be an expensive way to ship software. Someone has to understand the patch, check its assumptions, test its behaviour and own what happens after it is merged. If the first attempt misses the point, the team pays for another attempt and another review.
That does not make a higher-priced model the default answer. It changes the unit of comparison. A prompt response is an intermediate output. An accepted change is an outcome: the requested behaviour works, the implementation fits the system, and a responsible engineer can explain why it is safe enough to release.
The word can in the title matters. A higher model bill is justified only when it buys enough improvement in the whole workflow. If it produces the same review burden, the premium is simply another cost.
Total engineering cost = model and tooling spend + human effort + downstream failure cost.
Human effort includes preparation, supervision, review, testing and rework. Downstream failure cost includes diagnosis, recovery and support when defects escape. Keep elapsed delivery time alongside the financial calculation: an hour in a review queue is different from an hour of active review, even though both can delay a release.
A deliberately hypothetical example
Imagine the same bounded change attempted through two workflows. Both eventually meet the same acceptance criteria. Workflow B spends more on model calls but needs less human intervention. Assume an illustrative fully loaded engineering cost of £80 per hour.
| Cost component | Workflow A | Workflow B |
|---|---|---|
| Model calls, including retries | £2 | £12 |
| Preparation and supervision | 15 min / £20 | 15 min / £20 |
| Review and verification | 45 min / £60 | 20 min / £26.67 |
| Rework and repeat review | 30 min / £40 | 10 min / £13.33 |
| Illustrative total | £122 | £72 |
The extra £10 of model spend buys a hypothetical 45-minute reduction in human effort, worth £60 at the assumed rate. The resulting difference is £50 per accepted change. At £80 an hour, the model premium breaks even if it saves 7.5 minutes of human effort, assuming everything else is equal.
Everything else rarely stays equal. This simplified example excludes shared tooling, compute, waiting time and escaped defects. It also assumes that both changes are equally maintainable. It illustrates the arithmetic of a decision; it does not establish which model will win.
What changes if the time saving does not materialise?
If both workflows need the same human effort and deliver equivalent quality, A costs £10 less in this example. If B increases review time, its position gets worse. Price is evidence about spend; it is not evidence about capability on your tasks.
Review is part of production
The most expensive review often starts with uncertainty. Does this new abstraction belong here? Has the change preserved authorisation? Is the test exercising the requirement or repeating the implementation? A tidy diff can still leave every important question unanswered.
A useful model-assisted workflow makes those questions easier to resolve. It produces a change with a clear boundary, exposes assumptions, and supplies evidence that a reviewer can check independently. A long explanation without reliable evidence can increase the burden instead.
Do not count a shorter review as a saving when it comes from weaker scrutiny. Keep the acceptance standard fixed. A reduction from forty minutes to ten is valuable only if the reviewer can still reach a justified decision about correctness, security, maintainability and fit.
Separate the costs when recording them. Reading the initial diff is review. Fixing the missed edge case is rework. Reviewing that fix is another review. Recording only the first pass makes a troublesome workflow look artificially efficient.
Run a trial the team can trust
Start with a small set of representative tasks and a written definition of acceptance. Include ordinary maintenance, an unfamiliar part of the codebase and a change with meaningful edge cases. Keep sensitive data within your approved tools and access boundaries.
- Establish the baseline. Record how similar work is completed today, including human-only work where relevant. AI assistance has to earn its place against the current process.
- Keep the comparison fair. Use equivalent repository context, tool permissions and acceptance checks. Record model version and settings, task scope and who reviewed the output.
- Count the whole attempt. Include abandoned attempts, failed tasks, retries and manual rescue. Divide the total trial cost by accepted changes; do not report only the successful runs.
- Track quality after acceptance. Reopened work, regressions and maintenance difficulties can change the apparent economics. Record what is known and leave uncertainty visible.
- Inspect variation. Look at difficult cases as well as the average. A workflow with occasional long rescue sessions may be a poor fit for a team with little spare review capacity.
A small trial is a local decision aid, not a universal ranking. Reviewer familiarity, task selection and learning effects can skew it. Repeat or alternate assignments where practical, and resist explaining a narrow result as a permanent property of a model.
Choose a workflow for the work
A narrow, repetitive edit with strong automated checks is a sensible place to test a lower-cost option. A change crossing service boundaries, involving ambiguous requirements or requiring substantial repository reasoning may justify testing a more capable option earlier. These are hypotheses to evaluate, not guarantees attached to a price tier.
Set a stopping rule as well as a starting rule. If an attempt repeatedly fails the same acceptance check, move to a different approach or human investigation. Endless retries can consume both model budget and attention while disguising the need for a clearer specification.
Some work should stay with a person from the outset. When the requirement is unresolved, the team may need a product decision rather than more generation. When evidence is too weak to support a safe review, escalation is a better outcome than a faster merge.
Engineering time saved is capacity released, not automatically cash saved. It becomes commercially useful when the team can redirect it towards valuable work, reduce a real bottleneck or avoid a planned cost. Faster implementation will have limited effect if every change still waits a week for a decision.
Make the premium earn its place
Before standardising on a model, ask the team to show the accepted changes, the full human effort and the failures that did not make it into a demo. Then ask where any released capacity will go.
The most expensive model can be the cheapest choice when it reduces the total cost of delivering a trustworthy change. The least expensive model can win on exactly the same test. The discipline is to measure the outcome you need, with review and rework still in the account.