Skip to main content
LeoNex News

Cutting LLM costs: use a cheap classifier to route between models

Illustration: Cutting LLM costs: use a cheap classifier to route between models
In this article

Per-token API prices are falling, but bills for autonomous coding agents keep rising. The metric that governs real-world economics is now cost per successful task: total spend across every loop, tool call, retry and internal reasoning turn, divided by the number of tasks that pass the verification step. An experiment with 50 automatically graded coding-agent questions, run on real Gemini and Jev APIs, shows how to use a nearly free classifier to route each task to the cheapest configuration that still gives the right result.

Why a lightweight routing layer is needed

When an agent calls a high-level reasoning configuration for every small decision in the harness, the cost of deep reasoning is paid on questions that a lightweight configuration already answers correctly. To avoid this unnecessary expense, you can add a lightweight routing layer that intercepts routine decisions before they reach the expensive reasoning model.

In the experiment, this layer is TypeSafe's Jev classifier, placed in front of Google Antigravity's model calls. Jev is a structured-output classifier built for high-throughput decision pipelines. Instead of generating text token by token, it evaluates classification primitives in a single forward pass: Choice (pick from a list), Score (a numeric score) or Noul (a probability from 0 to 1 for a binary proposition). Google has no partnership with TypeSafe, and this content is not a promotion of TypeSafe or its products.

Jev is just one example of a lightweight classifier. The same routing pattern can be built with other classifiers, including Google's own. The codelab Getting started with discriminative models runs the same kind of classifier on DiffusionGemma, Google DeepMind's open-weights model, right inside the user's own Google Cloud project. An open-source server provides the same API in front, so the routing code only needs an endpoint change. Because the results in the experiment come from Jev, you should measure accuracy and latency on your own tasks before switching, and note that a self-hosted model is billed by GPU hour rather than by token.

Classifier routing vs. regex heuristics

The naive approach is to ask one broad Choice question: is this task simple or complex? When tried on 16 tasks, Jev placed nearly all of them in the simple group, including ones that required tracing an LRU cache or modular exponentiation. TypeSafe's documentation warns against exactly this anti-pattern: if a question weighs multiple factors or requires general judgment, split it into atomic checks.

When the broad question was replaced with a single atomic Noul question, the separation improved immediately. The question asks whether answering requires more than one sequential computation, simulation step or logical inference, rather than just a single direct lookup, format check or one-step classification. The code from the experiment:

python
def call_jev(question_text):
 body = {
 "state": question_text,
 "model": "jev-latest",
 "questions": {
 "multi_step": {
 "type": "noul",
 "instructions": (
 "Answering this requires performing more than one "
 "sequential calculation, simulation step, or logical "
 "deduction, rather than a single direct lookup, format "
 "check, or one-step classification."
 ),
 }
 },
 }
 # Send the request above to the classifier's API.
 # response.answers["multi_step"].noul is the probability used to route.

On the initial 16 tasks, Jev matched the hand-assigned difficulty labels on 13 of 16 tasks in the first run and 14 of 16 in the second. As for whether a classifier API is needed when you could write a free heuristic based on regex and prompt length, the local heuristic (matching words such as sequence, modulo, race condition, or upgrading every prompt over 35 words) scored 15 of 16, or 93.8%, on the very questions used to write the rules.

To test the heuristic against realistic traffic, the code was frozen without changing a single character and 34 unseen held-out tasks were added, spanning seven categories: fact extraction, schema validation, short classification, one-step code checks, state-simulation tracing, math or logic chains, and subtle linguistic-semantic errors.

maxOutputTokens is a ceiling, not a reasoning-budget knob

A common mistake when building a fast-path router is assuming that setting a low maxOutputTokens on a reasoning model makes it spend less on easy tasks. The 50-question data shows this assumption is wrong. Comparing gemini-3.8-flash with maxOutputTokens=300 and with maxOutputTokens=4096, keeping the API's default thinkingConfig, the two configurations consumed nearly identical amounts of thinking tokens on the 27 tasks Jev labeled SIMPLE.

As a result, a Jev router that sends easy tasks to the capped model and hard tasks to regular Flash, although it got all 50 correct, still cost slightly more and ran slower than just using regular Flash for everything. If the cheap choice doesn't actually make the model think less on easy tasks, no router saves money, not even a perfect and free one.

Comparing Jev against the strongest non-router setup

The obvious way to cut reasoning costs is to ask the model to think less. The second round of testing used two lighter options: Flash-Lite, which doesn't reason by default, and Gemini 3.8 Flash with the thinking level set to low. Results were paired per question in the same results file. On timing, most of the router's time sits in the Flash-Lite and Flash calls, which were measured in the same run and time window as the regular Flash baseline, so speed comparisons between them are fair. Jev's own decision takes only a small share and comes from the first round, so speed comparisons against default Flash or Pro mix the two rounds and are only relative.

The strongest non-router setup was Gemini 3.8 Flash at low thinking, which solved all 50 tasks; that is the number the router has to beat. The Jev router did: it sends every task to Flash-Lite first and only escalates what Jev flags as complex. The router also solved all 50 tasks and can reduce both cost and response time compared with running low thinking for everything. With prompts this short, the per-task cost difference is small, so for interactive agents the shorter wait is the bigger benefit.

  • On its own, Flash-Lite answered every simple task in half a second but failed 6 of 25 hard tasks; Jev flagged all 6 as complex with high confidence and escalated them, bringing cost close to that of a perfect router.

  • The keyword heuristic is cheaper, but only because it is wrong: it sent 5 hard tasks to Flash-Lite, 2 of which failed, for a result of 48 of 50.

  • Upgrading to a heavier configuration isn't worth it here: routing hard tasks to Flash with default thinking is cheaper than running default thinking everywhere, but still more expensive than running low thinking for everything, which already solved all 50. A heavier escalation target only makes sense once the low level starts failing.

Limits of the experiment: each setup was run only once, the 50 prompts have hand-assigned difficulty, and the automatic grading is simple.

Per-token pricing doesn't tell the whole story

Routing and caching pull in opposite directions. The test prompts were short, independent tasks, where the Jev router helps reduce both cost and response time. Real coding agents often differ: they send the same large block of context on every turn, such as a repository map and tool definitions. Agent Platform Context Caching stores this block so the cost of resending drops by 75% to 90%. However, a cache belongs to one model. If the agent switches back and forth between Flash-Lite and Flash turn by turn, it must either keep two caches or lose the discount. A practical split: route short, independent tasks such as log analysis or schema validation, and keep long, context-heavy turns on a single model.

You also need to choose a model that fits the workload. Gemini 3.1 Pro is built for deep, multi-step reasoning and carries that depth into every request, including short lookups that don't need it. For workloads that are mostly quick checks and extraction, a Flash configuration is a better fit, and Pro should be reserved for tasks that need reasoning depth. Each model configuration may also apply different safety settings to security-related prompts: in this run, the Pro configuration refused two short code security-review prompts that Flash answered. If your agent performs security reviews, test those prompts on the configuration you plan to use.

Three practical rules

  1. Start with Flash at low thinking before building a router. This configuration solved all 50 tasks and is significantly cheaper than the default setup. Lowering the output-token cap is no substitute, because it doesn't make the model think less on easy tasks; set thinkingLevel: "LOW" directly.

  2. Use Jev to route short, independent tasks: send simple tasks to Flash-Lite and hard tasks to Flash at low thinking, with each factor asking one simple yes/no question. In the experiment, this cut both cost and response time compared with running Flash at low thinking for everything, and came close to a perfect router. Keep long, context-heavy turns on a single model so caching keeps working. For large volumes on a data warehouse, Jeff Nelson's BigQuery and Cloud Run cascade pattern applies the same idea.

  3. Measure what the agent actually consumes: send the agent's traces and thinking-token counts to Cloud Logging and BigQuery to see which tasks drive costs up, and to catch truncation or retries that inflate them.

In short, the most economical way to run an agent is to match each task to the right model: start with a fast model in a light configuration, then add a small, nearly free classifier like Jev so that only hard tasks go to the stronger model, and ask simple yes/no questions instead of one broad question. Every choice should be judged by cost per task done correctly, not by each token.

Key takeaways

More in AI Engineering

Illustration: Context engineering and guardrails to "tame" coding agents
AI-ENG AI Engineering

Context engineering and guardrails to "tame" coding agents

Coding agents such as Cursor, Windsurf, Cline or Claude Code often deliver impressive results early on, but as a project grows with many files and branching business logic, they can break working features, pull in conflicting libraries or violate the architecture. The cause lies in an unstructured way of using them…

4 min read

Email newsletter

Receive a selection of new LeoNex News articles by email.

We only use this address to send the newsletter.

Receive a selection of new LeoNex News articles by email. You will receive a confirmation email; the subscription only takes effect after you confirm. Every email includes an unsubscribe link. Read the privacy policy