Here is a production tip I stand behind: I make an effort not to run on the strongest model, but on the weakest one that still does the job. It sounds backwards, but there are two good reasons.
Reason 1: it is simply cheap
Flash-Lite costs 0.10 dollars per million input tokens and 0.40 dollars per million output tokens. When you run real volumes, the difference against a flagship model is several times over. That alone would justify the habit, but it is the smaller of the two reasons.
Reason 2: the reserve
The important part: when you are already on the strongest model, you have no reserve. It breaks? Production lies down. When you are on a weaker model, a jump one step up solves the problem almost every time, and the system keeps running while you investigate what happened.
The strongest model is a ceiling. A weaker model is a floor with headroom, and headroom is what saves you at 2am.
Not every failure gets the same treatment
The key is separating by failure type:
- Provider outage or downtime: move sideways, from Flash-Lite to another cheap model in the same league. You do not need smarter, you need something that works right now.
- Quality degradation: move up.
An example: I run triage on data and on average get X results out of Y. Suddenly I am 2 standard deviations off the baseline, so I send the same batch for another round with a stronger model, as a double check.
The prerequisites that make it work
An important derivative of this whole story is collecting statistics on the system's activity. Everything that happens needs to be logged. We never know when it will be useful until we need it. And in every such event, make sure there are notifications so you can check what is going on.
At the beginning it is a bit annoying, because you have to genuinely verify the system works with different models. But it pays off over the life of the system, the way we hate unit tests until we have 300 good ones and life is suddenly good.
The principle
Do not couple your system to one model. It falls, and you fall with it. Build your model-usage flow so that both the system and you know exactly what to do in every situation. This exact escalation design, tiers, baselines, and fallbacks, is one of the first things I set up in production agent systems for clients, and a recurring topic in AI consulting.
And do not pull out Thor's hammer (Fable 5) to hit a small nail.
FAQ
Why not run production on the best available model?
Two reasons. Price: a light model costs a fraction per million tokens, and at volume the difference is large. Resilience: on the strongest model you have no reserve, so when it breaks, production lies down. From a weaker model, one step up almost always solves the problem while you investigate.
How do I decide when to escalate to a stronger model?
Separate by failure type. Provider outage or downtime: move sideways to another cheap model in the same league. Quality degradation: move up. Measure a baseline, for example average results per batch, and when a run lands two standard deviations off, resend that batch to a stronger model as a double check.