Stop Overpaying for Inference With AI Model Routing
Most coding-agent requests don't need a frontier model. Across a million-plus requests a month, intelligently routing AI inference with Actualyze costs over 60% less than sending everything to a frontier model.
AI usage is growing fast and so are the bills. That's why model routing has become such a hot topic. We see it firsthand at Actualyze, because we use our platform every day.
To show what routing can do at scale, we modeled a month of traffic from 36 coding agents running around the clock, just over 1.1 million requests. Sending all of it to Claude Fable 5.1 would cost $138,500 in provider token charges. Sending all of it to Claude Opus 5.5 would cost $55,400. Intelligently routing the same requests across Sonnet, Opus and Fable brought the cost down to $47,400 in our cost-focused scenario. That's 14% less than Opus-only and more than $1 million a year less than Fable-only, with 125,400 requests still going to Fable.
The savings come from not paying frontier rates for work that doesn't need a
frontier model. In Actualyze, that's the job of a Virtual Model, our approach to
AI model routing. The application calls one name (code-assistant
in our example) and behind that name is a pool of models. Actualyze picks one
model from the user-defined pool for each request. A cost-to-quality setting,
also user-defined, controls whether it leans toward cheaper or stronger models.
That's how a coding assistant gets a frontier model for a difficult refactor
while routine features and test cases don't.
Why one model for everything gets expensive
Many teams send every request to their strongest model. That makes sense for hard problems, where a wrong answer can cost far more to fix than it cost to generate. But most of what a coding assistant does is simple, like explaining a short function or writing a release note. Sending that work to the strongest model means paying top price for tasks a cheaper model handles just as well.
The obvious fix is to send simple work to a cheaper model. You can write rules
to do that yourself, but rules are hard to get right and harder to keep current.
A rule can't easily tell whether a coding request is a quick edit or a problem
that needs careful reasoning across the whole codebase. A Virtual Model takes a
different approach. Instead of following fixed rules, Actualyze looks at each
request as it comes in and judges how much capability it actually needs. Your
application keeps calling code-assistant. Your team can add models or change
the cost-to-quality setting without changing the application.
The pool we used
To put numbers on this, we built the code-assistant Virtual Model from three
Claude models that teams already use for coding.
Routing saves money because these models are priced very differently. Without caching, a Sonnet request costs a fifth as much as a Fable request. An Opus request costs twice as much as a Sonnet request. Every request that doesn't need Fable is money saved.
We also ran an OpenAI version with GPT-6 Luna, GPT-6.1 Sol and GPT-6 Astra. A Virtual Model can mix providers, too, using any endpoints the team already has access to.
How the router picks a model
First, Actualyze checks which models and endpoints the caller is allowed to use. It then rules out any model that can't handle the request, based on what it knows about each model's capabilities, available capacity and recent errors. For example, a cheap model with too small a context window is ruled out.
Next, a routing model we trained estimates how much capability the request needs and compares that with the models still in the running. When the setting leans toward cost, it picks the cheapest model that's strong enough. A contained feature change might go to Sonnet, while a difficult concurrency bug goes to Fable.
When the setting leans toward quality, the router asks for more capability and accepts a higher price for it. That's where I'd set it for anything touching production. If no model in the pool is strong enough, the router picks the closest one and logs the shortfall.
Routing does carry a risk. When a hard request lands on a model that's too weak, you pay for it in engineering time. That's why every request records which model served it. When a bad answer turns up, you can see whether a cheaper model handled it and shift the setting toward quality if it keeps happening.
What the numbers came to
We used Anthropic's and OpenAI's published API prices as of October 29, 2026. For OpenAI, we used the short-context rates, which cover every request size in this post. All results are rounded.
To keep the comparison fair, we gave every request the same average size, 6,400 input tokens and 1,150 output tokens (reasoning included), with no prompt caching. At that size, a request costs 2.4 cents on Sonnet, 4.9 cents on Opus and 12.2 cents on Fable.
These numbers rest on a few simplifications. Hard requests often produce more than 1,150 output tokens. When those land on Fable, the savings against Opus-only shrink, because Fable charges $50 per million output tokens versus $20 for Opus. We also assumed every request succeeds on the first try. All totals are provider token charges before Actualyze fees, taxes or negotiated discounts.
An Anthropic pool across Sonnet, Opus and Fable
For the cost-focused scenario, we assumed Sonnet is good enough for 62% of requests, Opus handles 27% and Fable gets the hardest 11%. I'd call that conservative, since most agent requests are small steps like reading a file, running tests or making a contained edit.
That's 66% less than Fable-only and 14% less than Opus-only. For the same budget, you could run nearly three times as many requests as with Fable-only.
We also tried a quality-focused mix of 48% Sonnet, 31% Opus and 21% Fable. That came to $59,600, or $4,200 more than Opus-only, with 239,400 requests going to Fable. Here's how every option compares.
Running everything on Sonnet is still the cheapest option at $27,700. What routing adds is quality. For $19,700 more, the cost-focused pool sends the hardest 433,200 requests to Opus or Fable.
An OpenAI pool with most of the work on Sol
We ran the same volume and request sizes through an OpenAI pool, with 9% of requests on Luna for well-scoped supporting tasks, 68% on Sol and the hardest 23% on Astra.
That's 63% less than Astra-only at $138,500, mostly because Sol handles work that would otherwise go to Astra. Luna is nearly free, but because it handles under a tenth of the work, it only moves the bill by $12,300.
The same traffic with caching
So far, every request was priced without caching. Real coding sessions reuse most of their context, so we also modeled the month with caching on. In this version, each turn read 92,000 tokens of context from the provider's cache, added 2,300 new input tokens and returned 1,150 output tokens. We kept the same request shares and left out the one-time cache write at the start of each session.
That's 15 times as much input per request, yet the Anthropic pools cost about the same as before.
- The cost-focused pool came to $46,400, 61% less than Fable-only ($118,000) and 20% less than Opus-only ($57,700).
- The quality-focused pool came to $56,500, now just under Opus-only. That's because Fable's cached reads cost a fortieth of its normal input price, while Opus's cost a twentieth.
- The OpenAI pool came to $65,000 versus $196,700 for Astra-only, a 67% saving. Astra's cached reads cost a tenth of its normal input price, which is why an Astra-only month costs more as cached session turns than as uncached requests.
The router still evaluates every request, but a conversation usually stays on the model it started on. Here's why.
A conversation has a cost history
Switching models in the middle of a session is expensive, because the new model has to take in all that context again at full price or more. Take one turn in a session that's been running on Opus, using the cached sizes above. Staying on Opus costs 5 cents. Switching to Sonnet costs 25 cents, because Sonnet has to write the context into its own five-minute cache, at $2.50 per million tokens, so later turns can use it. Even if Sonnet skipped the cache and read the context at full price, the turn would cost 20 cents. Neither number includes the earlier cost of caching the context on Opus.
Opus's warm cache more than makes up for Sonnet's lower prices. Opus is nearly five times cheaper for this turn and still four times cheaper if Sonnet skips the cache.
That's why the router counts the current model's cache when it prices a turn. It moves a conversation to another model when the work outgrows the current one, when it has to fail over or when another model is still cheaper after paying to rebuild the cache. During a chain of tool calls, it also stays on the current model as long as that model suits the work, so a short tool result doesn't bounce the session to a different model.
Not all agent work happens in one session, though. A coding agent often hands parts of a task to subagents, such as one that searches the codebase and another that reviews the changes. Each subagent starts its own conversation, so the router picks a model for each one based on its task. The search can run on Sonnet while the review goes to Fable, all under the same parent agent.
When a provider is overloaded
Failover deserves a closer look. The same model is often available through more than one endpoint, such as directly from Anthropic or through Amazon Bedrock. Actualyze treats each model and endpoint combination as a separate route. If a request fails with an error worth retrying, Actualyze tries the next route on an ordered fallback list, then the next model. It stops when it runs out of options, the request's deadline passes or it hits an attempt limit. Failover only happens before the response starts streaming back to the caller.
If you're trying out a cheaper model, keep stronger ones in the same pool to catch these failures. Each request's record lists any failover attempts, so you can see how often the cheaper model handles a request on its own and how often it falls back.
Rolling it out
Routing follows the same controls as everything else in Actualyze. A Virtual Model can't reach a model the caller couldn't already use and every request still counts against the caller's budget. The Virtual Model's data-handling guardrails apply to whichever model it picks.
At this volume, the cost-focused pool saves $8,000 a month compared with Opus-only. That's enough to fund code review on more pull requests or roll out an internal assistant more widely.
My advice to any team getting started is to pick one workload and build a pool with one cheap model and one strong one. Set it toward cost, send the Virtual Model a sample of real requests from that workload and check the answers before making it the application's default. Then watch which models handled which requests for the first couple of weeks.
This was the easy case
Our example is the easy version, because most people who use these models know roughly what Sonnet, Opus and Fable are each good at. That isn't true for every model or workload. There are dozens of models across providers, new ones ship constantly and few teams have time to test each one on their own requests. So most stick with the two or three models someone already knows.
That's where routing has more to offer. We score the models in our catalog on public evaluation data and keep those scores current, so the router knows what a model is good at even if your team has never tried it. Each model you add to a pool is another chance to find something cheaper that's still strong enough. But public evaluations aren't your codebase, so it's just as important to watch which models handled which requests.
Our next post covers bigger pools and what happens to the numbers when a pool grows past three models.
Until then, here's a quick check. Look at what your coding agents sent to their default model yesterday. If that default is a frontier model like Fable or Astra and most of the traffic was small edits, questions and release notes, you're paying frontier prices for work a cheaper model can handle. Start a 30-day trial and build a Virtual Model from the models your team already uses.