Routing: the cheapest reliability win in LLM systems

If you have an LLM system handling multiple types of input — and most real systems do — you have a choice: write one enormous prompt that tries to handle everything, or classify first and dispatch to a specialized handler. The second approach is routing, and it is almost always the better trade.

The payoff isn't only accuracy from better prompts, though that's real. The payoff is that the piece doing the routing — the classifier — costs almost nothing, while the handlers that do the actual work can be exactly as capable as each task requires. You don't need a Sonnet-class model to decide "this is an order-status query." You need one to handle a nuanced complaint about a damaged product. Routing lets you pay for what you use.

I implemented this as Chapter 05 of effective-agents-lab, using the e-commerce support scenario from Anthropic's "Building Effective AI Agents" playbook. Here's what it looked like in practice.

What the pattern looks like

The entry point is route_one() in router.py. It calls classify(), which sends the ticket to the cheapest available model (designated FAST in the repo's model registry) with a structured-output prompt. The result is a RouteDecision with three fields: category, confidence, and reasoning. One model call. That's the entire classification step.

def route_one(ticket, ticket_id=None, budget_tokens=20_000):
    with Run(pattern="routing", task=ticket) as run:
        decision = classify(ticket)   # FAST model, one call
        if decision.confidence < 0.6 or decision.category not in HANDLERS:
            text, escalated = human_escalation(ticket, decision.reasoning)
        else:
            handler_fn, model = HANDLERS[decision.category]
            text, escalated = handler_fn(ticket, model)

If the classifier isn't 60% confident, we don't guess — we escalate. If the category is "other", we escalate regardless of confidence, because HANDLERS has no entry for it. That's a deliberate design choice I'll come back to.

The classifier cost is nearly free

Running "Where is my order #48213?" through the full pipeline returns: category order_status, confidence 0.95, escalated: False, total cost $0.0009. The classification step is a fraction of that total. In the trace, the route_decision event is consistently priced far below any handler event — it's the fast model, a short prompt, structured JSON output.

In batch mode over the 12 fixture tickets, that fraction holds across every category. The router never becomes the expensive line.

The implication: you can add a router to an existing pipeline at almost zero marginal cost and immediately get the ability to send different inputs down different paths. Everything else in the system becomes cheaper to tune, because you're only tuning a prompt for the inputs it was actually designed for.

Model tier is part of the routing decision

The second thing routing buys you — often overlooked — is model selection as a function of task type. The handlers registry in handlers.py maps each category to both a handler function and a model tier:

Order status and product questions go to FAST. Complaints and refund requests go to BALANCED. The reasoning: order status is a structured lookup with a template reply; complaints and refunds require judgment about policy application, tone, and whether to escalate further. Different tasks need different capabilities, and routing makes that selection explicit rather than defaulting to one model tier for everything.

Without routing, the temptation is to pick a single model for the whole pipeline. That's either overpaying on simple queries or underserving the complex ones. Neither is good, and a router is a clean way to avoid the tradeoff entirely.

Low confidence is a design choice, not a fallback

One of the fixture tickets is just "hey." The router classifies it, lands on a low-confidence result or "other", and escalates without calling a handler at all. The human_escalation() function makes no LLM call — the comment in the code is direct: "a predictable fallback beats forced automation on low-confidence input."

That sentence is worth sitting with. A lot of systems I've seen try to handle everything with the model, even when the signal is weak. Routing makes the alternative explicit: when you don't know enough to dispatch confidently, the right answer is to say so and hand it to a human. The cost of that escalation path is exactly one classifier call, nothing more.

The "other" category goes further — it always escalates, regardless of confidence. Even if the classifier is highly confident a ticket is "other", there's no handler for it. Some inputs genuinely shouldn't be handled by the pipeline, and the right signal is a routing category with no corresponding entry in HANDLERS. The absence is intentional.

Extending it costs almost nothing

This is the payoff of keeping routing separate from handling. Adding a new category to this system requires two things: one new prompt string in prompts.py, and one new entry in HANDLERS mapping the category to its handler function and model tier. router.py itself doesn't change. The classifier picks up the new category automatically because the model reasons from the prompt, not from a hardcoded list.

In practice this means you can extend the system with the domain knowledge — "what is this new query type, how should it be handled?" — without touching the routing infrastructure. The two concerns stay separate, which is most of what good system design is.

When not to route

The module README is direct about this: avoid routing when categories overlap heavily, or when a single prompt already handles the range well. If your inputs aren't meaningfully distinct — if the handling for two categories would load the same context and run essentially the same prompt — you're adding a classification step that buys nothing except latency.

The useful question to ask is: would specialized prompts, different context, or different model tiers produce meaningfully better results for each category than a generalist approach? If yes, route. If not, the classification step is pure overhead. The pattern earns its place when the handlers genuinely differ, and nowhere else.