Whoever runs LLM-based Flows has to follow every new model that comes out. This is real money.
A live example:
I have a Flow that runs on Sonnet 4.6. When Sonnet 5 came out I moved it, and it came out a bit cheaper.
Yesterday Anthropic released Haiku 5.5.
I ran 15 Flows for comparison, the quality held, and the cost dropped from 2-3 dollars to 0.5-1.5 dollars per Flow. About 60% less.
The price to the client didn't change, so all the savings go straight to profit.
Of course I keep tracking the quality, and every now and then I run the same Flow on the previous model too and compare.
Two switches, same Flow, within a few months. Whoever doesn't check pays more for the same work.
This post joins two posts I wrote in Hebrew on X: the short one from October 8, 2026 above, and the full one from October 11 below.
The full process
A few days ago I wrote briefly about switching models in systems that use an LLM as part of their Flow. Now it's time for the full post on the process.
As you remember, Anthropic released Haiku 5.5 last week. A tenth of the price of Haiku 4.5 (on requests up to 100 thousand tokens), and performance that looks excellent on paper.
One of the systems I run is a platform that produces background check reports, and every report costs 2 to 5 dollars in model calls. When a significantly cheaper model comes out, the obvious question: do you switch and save?
The intuitive answer is to change the model name in the config and see what happens. That's exactly what I didn't do. It's rare that it works like that, out of the box.
Where does the money go?
Before switching, you check what's worth switching. I have a ledger of every paid call. Four runs from production showed that Sonnet 4.6, which writes the analyses and the report, is responsible for 65 to 78 percent of the cost. By comparison, Gemini Flash-Lite, which filters thousands of search results, only 4 to 8 percent.
The original idea was to replace Gemini with Haiku. The savings: ten cents per report. Not a lot, but at volume it adds up, and on top of that I wanted to move to a single provider. And still, the real money sits in Sonnet.
So I defined three experiments:
- Replacing Sonnet 4.6 with Haiku 5.5
- Replacing Sonnet 4.6 with Sonnet 5.5
- Replacing Gemini Flash-Lite with Haiku 5.5 in the filtering stage
How do you test a new model before switching?
To approve a switch, the new model has to bring an equal or better result.
You take a run that finished, copy it, delete only the model's saved answers, and rerun only the stages that call the model. The search results, the articles and the data from the networks are read from disk and cost nothing.
I compare the report that comes out to the report the client wrote by hand on the same person: how many facts were found, how many were found but stopped at the analysis stage, how many were missed. And in addition, a stage-by-stage comparison: findings, confidence level, cost, time.
Along the way I ran dozens of tests on different cases to locate and fix problems. The numbers that appear here are from the final round of tests, after all the fixes.
The first Haiku run looked like a failure of the model. In practice, a protection mechanism in the article analysis stage refused to analyze an article from disk without a saved analysis, so that tests would be deterministic. I deleted the analyses, so no article was analyzed. A quarter of an hour until I understood.
First lesson: before concluding anything about a model, make sure the test actually ran it.
Switching models is a migration, just like a new feature, not a config change
In each of the following cases the symptom looked like "the new model is weaker". The cause was something else entirely.
Thinking by default: the new models think before they answer, even without you asking, and the thinking tokens count within max_tokens. In one stage the cap was 1,024. Five out of 44 calls burned the whole cap on thinking and returned an empty answer.
The fix: a token policy per model in the client layer, with a margin of 8,000 when thinking is active.
The search tool: one stage searches the web through Anthropic's tool, in the version that filters results inside a code sandbox. With Sonnet 4.6 this worked correctly. The new models tried to run the search from inside the sandbox, burned the whole search budget and summarized "I hit the limit after one search". One item out of ten.
The fix: the tool version is chosen by model. For the new ones, the basic version without a sandbox. Twelve items.
A prompt read literally: the prompt said "someone who types the name and a few obvious follow-up questions". With Sonnet 4.6 that was eight searches, with the new ones three. "A few" is a few.
The fix: a budget in numbers, a search plan of five steps, and a clarification that the framing of "a curious person off the street" describes the type of questions, not the amount of effort.
Something surprising I discovered: I ran the fixed prompt on Sonnet 4.6 too, to make sure it works as a fallback. Same findings, 4.5 times the cost: 810 thousand input tokens, because through the sandbox every filtering round is billed again on the entire context.
That's what decided it: the old model doesn't stay as a fallback. A fallback that runs in a different prompt economy is not a fallback.
The results: Haiku 5.5 vs Sonnet 5.5
On the analysis calls, Haiku 5.5 is 97 percent cheaper and not good enough. On "read an article and fill a schema" it's equal to Sonnet. On judgment (what to search, how far to follow, what counts as a concern) it fell. A certain stage lost its three concern findings, and that's exactly what the client pays for.
With adaptive thinking at medium level, Sonnet 5.5's numbers on three subjects: the same number of facts plus or minus one, identical or better risk recording, 29 to 38 percent cheaper, 3 to 10 times faster.
With thinking off, the same model rated the risk low instead of medium. The setting matters as much as the model.
In the third experiment, Haiku as the filter instead of Gemini, the same report came out in the end. But Haiku sent 72 articles to full analysis instead of 47, most of them other people with the same name who were disqualified later at the analysis stage. The additional analyses cost more than the filtering saved, and the filtering itself was 2 times slower because Haiku writes long explanations. I stayed with Gemini. A single provider is a nice goal, not a reason to pay more and load noise on the analyst.
Lessons going forward
- Measure cost per finished report, not per token. A cheap model that does less work looks cheap in the price table and expensive on the bill.
- API defaults change between generations: thinking, tool versions, what you're allowed to send. It's not written in the model name, and that's also the reason a switch out of the box almost never works.
- A prompt tuned on one model will be read literally by another model. "A few" becomes three.
- A test needs to be one that can't be lied to. Checking how many calls were actually made saved me from a wrong conclusion.
- One subject isn't a sample, and neither are three exactly. Search returns different results on every run, and a fact that disappeared for one subject appeared for the other. That's why production is the real test, and that's why 4.6 is still running in parallel.
- Keep raw materials. Thanks to everything being saved on disk, all the tests together cost about 25 dollars and took a work day. Without that, you could multiply everything by 10.
And it's not over yet. The system moved to Sonnet 5.5 on Friday, and Sonnet 4.6 keeps running in parallel for a short period, just to make sure the results hold. The reports in production will be checked against each other, line against line on the bill.
I always work with the "weakest" model that gives a good result. More on that in Run the Weakest Model That Still Does the Job, and on the Fable 5.1 changes in Fable 5.1 Migration Guide.
FAQ
Can you switch LLM models by changing the model name in the config?
It's rare that it works like that, out of the box. API defaults change between generations: thinking, tool versions, what you're allowed to send. It's not written in the model name, and that's also the reason a switch out of the box almost never works.
Why did newer Claude models return empty answers after the switch?
The new models think before they answer, even without you asking, and the thinking tokens count within max_tokens. In one stage the cap was 1,024, and five out of 44 calls burned the whole cap on thinking and returned an empty answer. The fix: a token policy per model in the client layer, with a margin of 8,000 when thinking is active.
How should you measure the cost of switching to a cheaper model?
Measure cost per finished report, not per token. A cheap model that does less work looks cheap in the price table and expensive on the bill.