Choosing the Right Model (and Knowing When to Switch)

Why "use the biggest model" is the default, and why it is wrong

The default-to-biggest-model instinct makes sense at the demo stage. The biggest model is, on average, the most capable. It handles ambiguous prompts more gracefully. It hallucinates less on edge cases. It is the safe choice when the team is still discovering what the feature needs to do.

The problem is that the habit persists past the demo. Teams ship the biggest model into production because it worked in the prototype, and because switching is perceived as risk. The result is a cost structure calibrated for worst-case complexity on every request, even though most requests are not worst-case.

In mid-market deployments we audit, the cost reduction available from disciplined routing is typically 60 to 80 percent, at quality parity. The variance between the routed and unrouted cost is almost entirely a function of whether the team committed to the routing discipline up front.

The four axes of model selection

Every model choice should be scored on the same four axes. None of them is dispositive on its own; the pattern across the four decides the tier.

1. Capability

Does the model handle the task at an acceptable quality bar? Capability is measured on the feature's own evaluation suite, not on public leaderboards. Public benchmarks have almost no predictive value for your specific domain, your specific data, and your specific rubric. Run your golden set through each candidate model and compare the scores on your rubric.

The practical rule: if a smaller model scores within 2 to 3 points on your rubric of a larger model, that is within the noise floor of the evaluation. The smaller model is capable enough.

2. Latency

Does the model meet the P95 latency budget for the call site? User-facing interactive features typically need sub-second P50 and sub-three-second P95. Backend batch processing has effectively no latency constraint.

Mid-tier models are usually two to four times faster than flagship models on the same hardware class. For interactive features, this often matters more than the capability gap.

3. Cost

What is the per-decision cost, fully loaded? Token cost is the visible part. Retry rate and fallback rate matter too: a cheaper model that retries 15 percent of requests may cost more all-in than the expensive model that succeeds first time. Measure cost at the decision level, not the token level.

4. Reliability

How does the model handle provider-side outages and rate-limit events? A model from a single provider with a 99.5 percent SLA will have four hours of degraded operation per month. If the feature is customer-facing and revenue-touching, a multi-provider fallback layer is not optional.

The cost reality at three traffic tiers

The case for disciplined routing gets stronger as volume grows. Here is the per-month cost envelope for three realistic traffic tiers, comparing three routing strategies on the same workload.

Two observations. First, at 10K decisions per day, the absolute dollars are small enough that the question feels academic. It is not: the team that learns disciplined routing at this tier will have the operating muscle ready when volume tenths-place its way to 100K and 1M. Second, at 100K per day the monthly delta is in the high five figures, and at 1M per day the gap is enterprise-budget material. This is the fastest-payback engineering effort on the AI team's backlog.

Cascade routing: the pattern that pays for itself

The routing pattern that delivers the savings shown above is cascade routing with a confidence gate. The shape:

1. First pass, small model. A mid-tier model takes the request. For a very high fraction of queries, it produces a good answer at low latency and low cost.
2. Confidence scoring. The small model's output is scored for confidence. Heuristics work; so do a second quick LLM call for self-assessment; so do classifier layers trained on prior good-answer distributions.
3. Promote if needed. Below the confidence threshold, the request is re-routed to the flagship model. The user never knows; the system is slightly slower for that subset of requests.
4. Observe and tune. The promotion rate is the key operating metric. If it drifts up, either the small model has regressed or the confidence threshold is mis-calibrated. Investigate.

The failure mode of naive cascades is to push too hard for cost reduction and bleed quality through the cracks. The discipline is measurement: the cascade must be evaluated end-to-end on the golden set, not just on the share resolved by the cheap tier.

When to fine-tune, when to prompt, when to switch tiers

A related question sits inside every model-selection conversation: should we fine-tune, prompt-engineer, or move tiers? The answer is usually about capability headroom and task stability.

Prompt engineering first, always. Prompts are reversible. Fine-tunes are not, or at least not easily. Any capability gap that prompt engineering can close should be closed before fine-tuning is considered.

Fine-tune when the task is stable and the domain is narrow. A classifier on a small taxonomy with plenty of labeled data is a fine-tuning candidate. A general reasoning feature that touches twenty different domains is not.

Switch tiers when the gap is capability, not data. If the golden set shows the smaller model is simply not good enough on the task, no amount of prompting will close the gap. Move up a tier and save the fine-tuning budget for next year.

Switching cost: the planning tax nobody budgets for

Model switching sounds cheap. It is not. Every switch pays four taxes.

• Prompt tuning. Prompts that worked on the old model rarely work identically on the new one. Even within a provider's family, minor-version upgrades change behavior enough to require prompt audit.
• Evaluation re-run. The full golden set must be re-scored on the new model before production promotion. At 120 examples with LLM-as-judge, this is a half-day of engineering plus API cost, not a ten-minute task.
• Regression suite update. The known-regression slice of the golden set may or may not still be a regression on the new model. It needs to be re-verified.
• Operational memory. The team's debugging instincts are calibrated on the old model. Switching costs two weeks of re-calibration on the on-call rotation.

The implication for architecture is to design for switching from day one. Keep prompts and model choices in configuration, not in code. Keep the golden set in a runnable state. Instrument the calls through a provider-abstracted interface. A switching cost of two engineering days is tolerable; a switching cost of two engineering weeks is where teams freeze on a model they should have left.

Multi-provider fallback: the reliability layer

For any revenue-touching AI feature, assume each individual provider will experience at least one significant outage per quarter. Design the system to degrade gracefully rather than fail.

The cheapest viable pattern: two providers, one primary, one hot standby, routed through a thin abstraction layer that can flip on health-check failure. The standby does not need to be the same model family; it needs to be good enough to carry production traffic at acceptable quality while the primary recovers. Measure the standby's behavior periodically so the flip is not the first time you discover how it actually performs.

What a leader can do this week

Three concrete moves:

1. Instrument per-decision cost on the hottest feature. If the answer to "what does one call cost us on average" is "we would need to calculate," that is the first gap. Expose this number on the same dashboard as latency and error rate.

2. Run the cascade prototype. Take the feature and build a two-tier cascade with a naive confidence gate. Run it against the golden set. If quality holds and cost drops, you have just funded the next two quarters of AI work out of the savings.

3. Audit the switching cost. Time-box a two-hour exercise to price, in engineering hours, what it would take to migrate the feature from the current model to a comparable alternative. If the answer is more than five engineering days, the architecture has more provider lock-in than the team realizes.

If you want a second opinion on a specific routing design or cost audit, the AI & agents practice runs a focused two-week engagement that produces the cascade design, the measured savings estimate, and a multi-provider abstraction ready to deploy.

This paper is part of a series for engineering leaders putting AI into production. The other pieces cover evaluation before shipping, workflow vs agent architecture, and the broader adoption sequence.