Model routing and cascades: sending the right query to the right model

An internal support chatbot launched at a company I know routed every query — "what's the Wi-Fi password?", "when is the next all-hands?", "can you draft a performance review for [employee name]" — through the same frontier model. At the end of month one: $34,000. The engineering manager forwarded the invoice with no subject line.

That scenario is not pathological. It is the default. You integrate the best model first because it is the easiest integration, and you defer the "we'll optimize later" conversation until the bill forces it. The problem is that "later" arrives at an awkward moment — either at scale where the blast radius is large, or during a budget freeze where you cannot iterate. Model routing is the engineering discipline that avoids the binary: instead of "best model for everything" or "cheapest model for everything", you build a system that dispatches each query to the right tier.

Routing vs. cascading: two distinct patterns

The terms get used interchangeably, but they have different latency and cost profiles. Getting the distinction straight before designing anything saves a lot of re-work.

Routing makes one upfront classification decision. A classifier fires — fast, cheap — and the query goes to exactly one model. The total inference cost is: classifier cost + one model call. For simple queries, that is your cheapest possible path. For complex queries, you pay the same as the classifier cost + one frontier call. There is no compound latency penalty for hard queries.

Cascading sends the query to a cheap model first, inspects the output for confidence, and escalates only when confidence falls below a threshold. For simple queries, it wins: one cheap model call, done. For complex queries, it pays for all the tiers it touches before landing at the frontier — on a three-tier cascade, that is three model calls in sequence. At 2–4 seconds each on a frontier model, that cascades into a painful p99.

sequenceDiagram
    participant Q as Query
    participant R as Router
    participant T1 as Tier 1 (cheap)
    participant T2 as Tier 2 (mid)
    participant T3 as Tier 3 (frontier)

    Note over Q,T3: Routing path — one classifier, one model call
    Q->>R: classify (< 10ms)
    R->>T3: complex → direct to frontier
    T3-->>Q: answer (2–4s)

    Note over Q,T3: Cascade path — sequential escalation
    Q->>T1: attempt (0.3–1s)
    T1-->>Q: low confidence
    Q->>T2: escalate (0.8–2s)
    T2-->>Q: still uncertain
    Q->>T3: final escalation (2–4s)
    T3-->>Q: answer (total: 3–7s)

The practical implication: two different axes decide this, and conflating them causes bad designs. Skew determines how often the cascade latency tax is paid — at 70%+ simple traffic, escalations are rare and the tax is affordable; balanced or complex-heavy workloads pay it constantly, so they should route. Separability determines whether an upfront classifier can be accurate at all. A workload that is simple-heavy and cleanly separable should still route — the classifier is cheap and correct, and you keep cascade escalation only as a safety net for its uncertain calls. Cascading alone earns its keep when traffic skews simple but is too fuzzy for a classifier to partition reliably.

Neither pattern is universally better. Measure your distribution first. A histogram of query complexity from a week of production traffic, even roughly labeled, is worth more than any theoretical analysis.

The classifier layer: what you actually build

The router's classifier is the most important implementation decision you will make, because a bad classifier quietly degrades output quality on the queries it misroutes — and those are exactly the hard queries where quality matters most.

Rule-based routing first. Before writing a single line of ML, extract simple signals that partition your traffic:

• Query length (in tokens): a 12-token query is unlikely to be a complex reasoning task.
• Presence of domain keywords: "refund", "policy", "FAQ" versus "implement", "design", "explain why".
• Structured vs. free-form: a JSON extraction task over a fixed schema is deterministically simple.
• Session context: first-turn queries differ from late-turn ones where context has built up.

These rules are measurable on day one. They have no training data requirement. They catch 30–40% of the obvious cheap traffic with zero model calls and zero calibration. Start here. Always.

Classifier-based routing second. Once you have baseline savings from rules, add an ML classifier for the residual traffic. The architecture is a BERT-class encoder fine-tuned on labeled examples from your own traffic:

from transformers import pipeline
from functools import lru_cache

# A DistilBERT (~66M params) or MiniLM (~22M) fine-tuned on your routing labels
# inference: ~8ms on CPU, ~2ms on GPU
@lru_cache(maxsize=1)
def get_router():
    return pipeline(
        "text-classification",
        model="your-org/query-router-v2",
        device=0,          # GPU; fallback to "cpu" for low-traffic deployments
        top_k=None,        # return all class probabilities
    )

TIER_MAP = {
    "simple": "gemini-flash-lite",
    "mid": "gpt-4o-mini",
    "complex": "claude-sonnet-4-5",
}

def route_query(query: str, confidence_floor: float = 0.80) -> str:
    router = get_router()
    predictions = router(query)[0]
    # Sort by score descending
    predictions.sort(key=lambda x: x["score"], reverse=True)
    top = predictions[0]

    if top["score"] < confidence_floor:
        # Classifier is uncertain — route to mid as a safe default
        return TIER_MAP["mid"]

    return TIER_MAP[top["label"]]

The key detail: confidence_floor. When the classifier is uncertain, default to the mid tier rather than the cheap one. An uncertain easy query costs a little extra. An uncertain hard query silently delivered a bad answer, possibly to a user who trusted it.

RouteLLM and pre-trained routers. If you do not have labeled training data yet, RouteLLM (from the LMSYS Chatbot Arena team) provides pre-trained routers using matrix factorization and Bradley-Terry preference models trained on Chatbot Arena human preference data. It demonstrated 85% cost reduction routing between GPT-4 and Mixtral 8x7B on MT-Bench while maintaining 95% of GPT-4 quality. The caveat: MT-Bench categories do not match your traffic distribution. Use it as a warm start, then fine-tune.

{ "type": "model-router", "title": "Interactive routing threshold explorer" }

Confidence signals: what actually works

The hardest part of cascade design is defining "low confidence" in a way that correlates with actual answer quality. Two approaches in common use, one that works and one that sounds good but does not.

What sounds good but does not work: LLM self-reported confidence. Ask a model "are you sure?" or inspect whether it phrases the answer with hedging. This fails because LLMs are trained to sound helpful and authoritative — they produce fluent, confident-sounding text on questions they are factually wrong about. "The policy was last updated in March 2024" is exactly the wrong answer delivered with no hedging. The correlation between self-reported confidence phrases and factual correctness is close to zero on out-of-distribution queries.

What works: token log-probability distributions. OpenAI's Chat Completions API exposes log-probabilities on output tokens via the logprobs and top_logprobs parameters. Anthropic's Messages API does not expose per-token logprobs, so a cascade over Anthropic models needs a different confidence signal — ensemble agreement across sampled responses, or a separately trained scorer. When a model generates a high-confidence answer, the top token has a probability mass far above the second candidate. When uncertain, the distribution flattens — the model is genuinely hedging at the token level, not the phrase level.

import math
from openai import OpenAI

client = OpenAI()

def get_answer_with_confidence(query: str, model: str) -> tuple[str, float]:
    """Returns the answer and a confidence score [0, 1]."""
    response = client.chat.completions.create(
        model=model,
        max_tokens=512,
        messages=[{"role": "user", "content": query}],
        logprobs=True,
        top_logprobs=5,
    )
    answer = response.choices[0].message.content
    return answer, _compute_confidence_from_logprobs(response)


def _compute_confidence_from_logprobs(response) -> float:
    """Geometric mean of top-token probabilities on the first N tokens."""
    tokens = response.choices[0].logprobs.content[:10]
    if not tokens:
        return 0.5  # unknown, default to mid
    avg_logprob = sum(t.logprob for t in tokens) / len(tokens)
    return math.exp(avg_logprob)  # [0, 1]


CASCADE_THRESHOLD = 0.72   # calibrated empirically per domain

def cascade_query(query: str) -> str:
    answer, conf = get_answer_with_confidence(query, model="gpt-4o-mini")
    if conf >= CASCADE_THRESHOLD:
        return answer
    # Escalate
    answer, _ = get_answer_with_confidence(query, model="gpt-4o")
    return answer

The threshold (here 0.72) is not a magic number — it requires calibration on a labeled eval set from your domain. Log the confidence values for every query alongside a quality signal (thumbs up/down, task completion, human rating), then find the threshold that minimizes misrouted-and-wrong outcomes. Expect to spend several days on this calibration before a cascade is trustworthy in production.

Building the three-tier architecture

Most production routing setups converge on three tiers. More tiers add more decision points without proportional cost savings; fewer leave money on the table.

flowchart TD
    Q[Query arrives] --> RB{Rule-based\nprefilter}
    RB -->|keyword / length match| T1
    RB -->|no match| CL[Classifier\nsmall encoder, <10ms]
    CL -->|simple, high conf| T1["Nano tier\nGemini Flash Lite / Mistral 7B\n$0.10–0.40/M\nSub-300ms TTFT"]
    CL -->|mid, high conf| T2["Mid tier\nGPT-4o mini / Claude Haiku\n$0.15–1.50/M\n300–800ms TTFT"]
    CL -->|complex| T3["Frontier tier\nGPT-4o / Claude Sonnet\n$2.50–15/M\n1–4s TTFT"]
    CL -->|low conf| T2
    T1 -->|confidence < threshold| T2
    T2 -->|confidence < threshold| T3
    T1 -->|answer| OUT[Return to user]
    T2 -->|answer| OUT
    T3 -->|answer| OUT

    style RB fill:#ffaa00,color:#0a0a0f
    style CL fill:#ffaa00,color:#0a0a0f
    style T1 fill:#15803d,color:#fff
    style T2 fill:#0e7490,color:#fff
    style T3 fill:#00e5ff,color:#0a0a0f

The rule-based prefilter fires first — zero ML inference cost. The classifier handles residual ambiguous traffic. The cascade fallback handles the classifier's uncertainty floor without requiring a perfect classifier.

As of mid-2026, practical tier assignments (prices illustrative — check current provider rates):

Tier	Models	Input price	Use cases
Nano	Gemini Flash Lite, Mistral 7B, fine-tuned Llama 3B	~$0.10–0.40/M	Single-intent classification, FAQ retrieval, short extraction, yes/no questions
Mid	GPT-4o mini, Claude Haiku 3.5, Gemini Flash 2.0	~$0.15–1.50/M	Standard QA, summarization, structured extraction, code completion
Frontier	GPT-4o, Claude Sonnet 4.5, Gemini 2.0 Pro	~$2.50–15/M	Complex reasoning, multi-step code, adversarial edge cases, long-context

The output-token asymmetry matters here. Output tokens cost 3–5× more than input tokens across all providers (generation is memory-bandwidth-bound — one sequential forward pass per token; prefill processes all prompt tokens in parallel and is compute-bound — see token economics for the full breakdown). A tier-one query generating 400 output tokens at $0.40/M still costs almost nothing. A tier-three query generating 1,000 output tokens at $15/M is the dominant cost driver.

Worked numbers: what routing actually saves

Scenario: production support chatbot, 500k queries/day

Measured distribution (after one week of traffic analysis):
  60% simple  — "how do I reset my password", "what's your refund policy"
  30% mid     — "explain this error message", "compare these two plans"
  10% complex — "I need to migrate my data from X to Y, here's my schema..."

Average token counts (measured):
  Simple:  180 input, 80 output
  Mid:     320 input, 200 output
  Complex: 800 input, 500 output

No routing (all Claude Sonnet at $3/M in, $15/M out):
  Simple:  300k × (180×3 + 80×15)/1M = 300k × $1,740/M  = $522/day
  Mid:     150k × (320×3 + 200×15)/M  = 150k × $3,960/M  = $594/day
  Complex:  50k × (800×3 + 500×15)/M  =  50k × $9,900/M  = $495/day
  Total:                                                   = $1,611/day = $48,330/month

With routing (Gemini Flash Lite / GPT-4o mini / Claude Sonnet):
  Simple:  300k × (180×0.10 + 80×0.40)/M = 300k × $50/M   = $15/day
  Mid:     150k × (320×0.15 + 200×0.60)/M = 150k × $168/M  = $25/day
  Complex:  50k × (800×3   + 500×15)/M   =  50k × $9,900/M = $495/day
  Total:                                                     = $535/day = $16,050/month

Savings: $32,280/month — 67% reduction.
Classifier cost: 15M queries/month × ~$0.00003 (self-hosted small-encoder
inference) ≈ $450/month. Trivial.

The interesting observation: the complex tier is the same in both scenarios because a good router does not misclassify hard queries. All the savings come from correctly identifying simple traffic. This is why classifier accuracy on the easy class matters more than on the hard class — false negatives (routing hard queries to cheap models) hurt quality; false positives (routing easy queries to expensive models) just cost money.

{ "type": "token-cost", "title": "What one tier's monthly bill is made of" }

What breaks

The cold-start misclassification problem

An intent classifier trained on historical traffic will misclassify novel query patterns. When you launch a new product feature, the classifier has never seen queries about it — it falls back to its base prior, which might be wrong. A query like "can the new assistant mode handle my ISO 27001 audit?" looks like a compliance question (mid-tier) but might need frontier-level legal reasoning. You will not know until you see bad answers.

The mitigation: new feature launches get a routing override that forces all traffic through the frontier tier for the first 48–72 hours. You collect labeled data, retrain the router on the new pattern, then lift the override. Treat it like a feature flag.

Confidence threshold drift

A cascade threshold calibrated in January will silently drift by March if your query distribution shifts — new user segments, seasonal patterns, product changes. The threshold that produced 90% accurate escalations no longer does. The signal: your tier-one answer quality metrics start declining while tier-one volume stays constant.

Monitor the per-tier quality signal (user satisfaction, task completion rate, or an LLM-as-judge score) separately, not just the aggregate. A weekly automated eval that checks calibration of the confidence score against a fresh labeled sample catches drift before it compounds.

Routing without observability is worse than no routing

If a misrouted query produces a bad answer and you have no record of which tier it was routed to, why the classifier chose that tier, and what the confidence was, you cannot fix it. The bug is invisible. A silent quality degradation from a miscalibrated router is more damaging than the honest cost of the frontier model.

Every routing decision must emit a structured log with at minimum: query_id, classifier_label, classifier_confidence, tier_dispatched, model_used, response_quality_signal. The LLM gateway pattern is the right place to attach this observability layer — the gateway sees every request, makes the routing call, and can tag the downstream span with the routing metadata.

Context window mismatches on fallback

If your cascade escalates from a model with a 128k token context window to a fallback that has 32k, long queries silently truncate. The failure mode is insidious: the request succeeds, the response is generated, and the answer is confidently wrong about the truncated content. Always check len(prompt_tokens) < target_model.context_limit before escalating — or use prompt compression as a fallback path rather than truncation.

Cascades don't stream

Streaming is how you hide LLM latency from users — and a cascade cannot stream its tier-one response, because once tokens reach the screen you cannot retract them when confidence turns out low. That leaves two options, both with costs. Buffer the full tier-one response, check confidence, then release or escalate: now every cascade query — including the 70% that resolve at tier one — pays the entire tier-one generation time as perceived TTFT, turning a 300ms streaming interaction into a second-plus wall of silence. Or check confidence on the first N tokens (the same window the log-prob scorer reads) and commit to streaming once it clears the threshold: better TTFT, but a model can open confidently and derail at token 50, and you have already committed. Routers do not have this problem — one upfront decision, then stream freely from the chosen model. If perceived latency is a product requirement, this alone often decides routing over cascading.

A quieter cost in the same family: prompts are not portable across tiers. A system prompt tuned against the frontier model can produce format drift or spurious refusals on the nano model, so each tier needs its own eval pass — escalation is not a free retry with a smarter model, it is a second deployment surface.

Routing adds a latency floor

A classifier adding 10ms is invisible at p50. It is visible at p99 when combined with a slow first-tier model that escalates to a frontier model. The worst case: simple rule-based filter misses → 10ms classifier → 300ms tier-1 call → low confidence → 2s frontier call. Total: ~2.3s for what should have been a 300ms interaction. Measure your per-path p99, not just the average.

Integrating with your existing stack

LiteLLM's proxy mode handles routing with a simple config that most teams reach for first:

# litellm_config.yaml
router_settings:
  routing_strategy: "usage-based-routing-v2"
  model_group_alias:
    "fast-model": ["gemini-flash-lite", "mistral-7b"]
    "mid-model": ["gpt-4o-mini", "claude-haiku-3-5"]
    "frontier-model": ["gpt-4o", "claude-sonnet-4-5"]

model_list:
  - model_name: gemini-flash-lite
    litellm_params:
      model: gemini/gemini-flash-lite-2.0
      api_key: os.environ/GEMINI_API_KEY
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-sonnet-4-5
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

LiteLLM's built-in routing is cost-based and latency-based, not semantic. It handles fallbacks and provider round-robin but not intelligent query-complexity routing. For semantic routing, you integrate the classifier before the LiteLLM call:

import litellm
from your_router import route_query   # your BERT-class classifier

def routed_completion(messages: list[dict], **kwargs) -> str:
    query = messages[-1]["content"]
    model = route_query(query)

    response = litellm.completion(
        model=model,
        messages=messages,
        metadata={
            "routed_to": model,
            "classifier_label": ...,    # attach from route_query
            "classifier_confidence": ...,
        },
        **kwargs
    )
    return response.choices[0].message.content

For teams at higher volumes or with data residency requirements, the LLM gateway pattern — a centralized proxy with virtual keys, TPM/RPM limits, and per-request tagging — is the right home for the routing layer. Portkey open-sourced its full gateway under Apache 2.0 in March 2026, including the routing logic and observability hooks. LiteLLM remains simpler to self-host for pure routing use cases.

The decision in practice

The choice between routing and cascading comes down to your query distribution and your latency requirements.

If you have a bimodal distribution — clearly simple traffic and clearly complex traffic, with few queries in between — a classifier-based router is better. One classification, one model call, predictable latency everywhere.

If your distribution is unimodal around the middle — most queries could go either way depending on subtle signals — a cascade extracts more savings because you get to try cheap first and only pay for the frontier when the cheap model cannot handle it. Accept that complex queries pay the latency tax.

In practice, most product workloads have the bimodal distribution: there really are simple questions and hard questions, with a smaller ambiguous middle. Rule-based routing catches the obvious simple cases. A classifier handles the middle. Cascade escalation acts as a safety net for classifier uncertainty. That three-layer approach — rules, then classifier, then confidence-based cascade — is the design worth building toward.

Start simpler than that, though. Ship rule-based routing in week one. Measure the savings and the quality delta. Add the classifier only when you have labeled data and a clear signal that the rules are leaving significant traffic misrouted. The engineering cost of an ML classifier is real — labeling data, training pipeline, model serving, calibration — and it is only justified once you know what you are trying to improve.

What you ship in week one should not be the finished architecture. It should be something that starts saving money immediately and generates the data to make the next step obvious.

For the reliability and fallback behavior on top of any routing design — what to do when a provider returns a 429, how to fail over between providers, and how to keep the routing layer itself resilient — see fallbacks, retries, and circuit breakers. For the full cost picture that makes routing decisions legible, token economics covers where the bill actually comes from. And once you have routing live, prompt caching is typically the next layer to add — it stacks with routing and compounds the savings.