Model Routing for Cost Control: Cut LLM Spend 40-60%

Model routing sends simple queries to cheap, fast models and complex queries to powerful ones, cutting LLM API costs 40-60% without meaningful accuracy loss. Most production systems waste budget running every request through GPT-4o when 60-70% of queries succeed on GPT-4o-mini or Claude Haiku.

Why Default Routing Is Expensive

Teams typically use one model for everything:

Model	Input Cost	Output Cost	Typical Monthly (100K queries)
GPT-4o	$2.50/1M tokens	$10/1M tokens	$3,000-$8,000
GPT-4o-mini	$0.15/1M tokens	$0.60/1M tokens	$200-$600
Claude 3.5 Haiku	$0.80/1M tokens	$4/1M tokens	$800-$2,000

If 65% of queries work on mini/haiku, routing saves $1,500-$5,000/month at 100K queries.

Routing Architecture

Incoming query
  → Query classifier (cheap model or rules)
  → Route decision:
      Simple → GPT-4o-mini / Haiku
      Medium → GPT-4o-mini with longer context
      Complex → GPT-4o / Claude Sonnet
      Critical → GPT-4o with higher temperature control
  → Execute on selected model
  → Quality check (optional: re-route if confidence low)
  → Response + cost logging

Query Classification Methods

Rule-Based (Start Here)

Fast, free, and interpretable:

def classify_query(query, context_length):
    if len(query.split()) < 15 and context_length < 2000:
        return "simple"
    if any(kw in query.lower() for kw in ["analyze", "compare", "strategy", "complex"]):
        return "complex"
    if context_length > 8000:
        return "complex"
    return "medium"

Tune rules based on production data after 2 weeks.

Classifier Model

Train a lightweight classifier on labeled query data:

• Input: query text + metadata (length, source channel, user tier)
• Output: simple / medium / complex
• Model: fine-tuned mini model or logistic regression on embeddings
• Accuracy target: > 90% (misrouting complex as simple is worse than the reverse)

When in doubt, route up, not down.

LLM-as-Router

Use a cheap model to classify:

Classify this query complexity as simple, medium, or complex.
Simple: FAQ, lookup, short answer
Medium: explanation, summary, multi-step
Complex: analysis, reasoning, creative, long context

Query: {query}
Classification:

Cost: ~$0.0001 per classification. Worth it for ambiguous queries.

Tiered Model Strategy

Tier	Model	Use Cases	% of Traffic
Tier 1	GPT-4o-mini / Haiku	FAQ, classification, extraction, formatting	50-65%
Tier 2	GPT-4o / Sonnet	Analysis, generation, multi-step reasoning	25-35%
Tier 3	GPT-4o (high context)	Long documents, complex reasoning	5-10%
Tier 4	Fine-tuned model	Domain-specific high-volume tasks	5-10% (if available)

Review tier assignment monthly against eval scores.

Additional Cost Controls

Response Caching

Cache identical or semantically similar queries:

• Exact match cache: hash query → return cached response
• Semantic cache: embedding similarity > 0.95 → return cached response
• TTL: 1-24 hours depending on content freshness needs
• Savings: 10-30% of queries on support and FAQ bots

Tools: GPTCache, Redis with embedding lookup, or custom implementation.

Token Optimization

• Trim conversation history to last 5-10 messages
• Summarize long context instead of passing full documents
• Use structured outputs to reduce verbose responses
• Set max output tokens per query type

Budget Guardrails

daily_budget = 100  # dollars
current_spend = get_daily_spend()

if current_spend > daily_budget * 0.8:
    downgrade_all_to_tier_1()
    alert_ops_team()

if current_spend > daily_budget:
    queue_requests_or_reject()
    alert_ops_team(priority="critical")

Measuring Routing Effectiveness

Track weekly:

Metric	Target
Cost per query	Decreasing trend
Accuracy by tier	Tier 1 > 85%, Tier 2 > 90%, Tier 3 > 95%
Misroute rate	< 5% (complex sent to simple, measured by re-routes)
Cache hit rate	10-30%
Total monthly spend	Within budget

Run eval suites per tier. If Tier 1 accuracy drops below 80%, tighten classification rules.

Pair routing with ops monitoring and prompt optimization for full cost management.

Need model routing and cost optimization built for your AI systems? TopAhead’s Managed AI Ops service implements routing, caching, and budget controls.

FAQ

Will users notice quality differences between tiers? Not if classification is accurate. Users notice when complex queries get simple-model answers. Monitor re-route rates.

How long to implement routing? 1-2 weeks for rule-based routing. 3-4 weeks for classifier-based with eval validation.

Should we self-host open-source models? Consider at > 1M queries/month where API costs exceed $5K/month. Factor in GPU infra and ops overhead.

What about latency differences between tiers? Mini models are faster. Routing to cheaper models often improves latency for simple queries.

Can routing work with multiple providers? Yes. Route by task type across OpenAI, Anthropic, and open-source. Adds complexity but improves resilience and cost optimization.