How to Architect Adaptive Thinking into Production LLM Applications: Routing Logic, Latency Budgets, and Cost-Aware Inference Strategies
The teams cutting their LLM inference costs by 40 to 85 percent are not finding cheaper models. They are routing smarter. Shifting from a single hardcoded model at deployment time to per-request model selection at runtime is the highest-leverage architectural decision available in production AI today, and most teams have not made it yet.
This article walks through the building blocks of adaptive inference: query complexity signals, latency budgets as routing constraints, fallback chains with circuit breakers, and cost-quality trade-off policies. By the end, you will have a blueprint for an inference layer that treats model selection as a first-class runtime decision rather than a deployment-time guess.
Why Static Model Selection Is Now a Liability
LLM API spending more than doubled from $3.5 billion to $8.4 billion between late 2024 and mid-2025, and 72 percent of organizations plan to increase those budgets in 2026. Inside those budgets, the single biggest waste is routing simple queries to expensive models.
A lightweight, fast model handles intent classification, entity extraction, and FAQ retrieval with high accuracy and low latency. A frontier model is wasted on those tasks. Hardcoding the frontier model "to be safe" is the inference equivalent of renting a warehouse to store a single file cabinet.
The economics are straightforward: if 60 percent of your traffic is simple queries that a smaller model handles equally well, routing that traffic there cuts its cost by 80 to 95 percent while preserving full quality for the 40 percent of complex queries that actually need a capable model. That arithmetic compounds fast at scale.
The Anatomy of an Adaptive Router
An adaptive router sits between your application and your model provider pool. At its core, it answers one question per request: given this query, these latency constraints, and this cost budget, which model should serve it?
The router has four responsibilities:
- Signal extraction: read the incoming request and produce a complexity estimate
- Constraint evaluation: check active latency budgets and cost policies
- Model selection: pick the best model given signals and constraints
- Fallback handling: recover gracefully when the selected model is unavailable or slow
Each maps to a concrete implementation pattern covered in the sections below.
Query Complexity as a Routing Signal
The router cannot select intelligently without knowing something about the query. Useful complexity signals include:
- Embedding similarity to clusters of known-simple or known-complex queries
- Token count as a rough proxy for context richness
- Keyword patterns that indicate reasoning requirements ("compare", "explain why", "write a plan")
- Session context, such as whether the user is mid-conversation in a multi-step reasoning chain
- User metadata, such as subscription tier or feature flag
A lightweight classifier that combines these signals into a complexity score is sufficient for most routing needs. The classifier does not need to be a large model; a small embedding model plus a logistic regression layer works well and adds less than 5 ms of overhead.
import numpy as np
from sentence_transformers import SentenceTransformer
# Pre-cluster centroids learned from labeled query examples.
# "simple" cluster: FAQ, classification, short lookups.
# "complex" cluster: multi-step reasoning, code generation, analysis.
SIMPLE_CENTROID = np.load("centroids/simple.npy")
COMPLEX_CENTROID = np.load("centroids/complex.npy")
# Load once at startup; per-call encoding runs 3-8 ms on CPU.
embed_model = SentenceTransformer("all-MiniLM-L6-v2")
def complexity_score(query: str) -> float:
"""Returns a float in [0, 1]: 0 = simple, 1 = complex."""
vec = embed_model.encode(query, normalize_embeddings=True)
sim_simple = float(np.dot(vec, SIMPLE_CENTROID))
sim_complex = float(np.dot(vec, COMPLEX_CENTROID))
# Map [-1, 1] similarity difference to a [0, 1] complexity score.
return (sim_complex - sim_simple + 1) / 2
def select_model(query: str) -> str:
score = complexity_score(query)
if score < 0.35:
return "claude-haiku-4-5" # fast, cheap
elif score < 0.70:
return "claude-sonnet-4-6" # balanced
else:
return "claude-opus-4-5" # full capability
This is intentionally simple. The power comes not from the classifier's sophistication but from running it consistently on every request and routing accordingly. As you collect outcome data, you can retrain the centroids and thresholds to reflect your actual query distribution.
Latency Budgets as Hard Routing Constraints
A latency budget is a per-workload SLA target expressed as a p95 or p99 threshold. Customer-facing chat might carry a 2-second p99 budget. An async document analysis job might tolerate 30 seconds. Batch pipelines running overnight may have no hard budget at all.
The key insight is to use the latency budget as a hard routing constraint, not just a monitoring metric. If the primary model's current p99 exceeds the budget, the router proactively shifts the request to a faster model before a timeout can occur. This transforms latency management from reactive alerting into a proactive steering signal.
from collections import deque
from typing import Optional
class LatencyTracker:
"""Rolling p99 tracker over the last N observations."""
def __init__(self, window: int = 200):
self.window = window
self._samples: deque[float] = deque(maxlen=window)
def record(self, latency_ms: float) -> None:
self._samples.append(latency_ms)
def p99(self) -> Optional[float]:
if not self._samples:
return None
sorted_samples = sorted(self._samples)
# sorted() is O(n log n); for very high QPS, replace with a
# streaming percentile estimator (e.g., t-digest or HDR histogram).
idx = min(int(len(sorted_samples) * 0.99), len(sorted_samples) - 1)
return sorted_samples[idx]
# One tracker per model.
trackers: dict[str, LatencyTracker] = {
"claude-haiku-4-5": LatencyTracker(),
"claude-sonnet-4-6": LatencyTracker(),
}
LATENCY_BUDGET_MS = 2000 # 2-second p99 SLA
def latency_aware_select(preferred_model: str, fallback_model: str) -> str:
"""
Return preferred_model unless its rolling p99 exceeds the SLA budget,
in which case return the (presumably faster) fallback_model.
"""
p99 = trackers[preferred_model].p99()
if p99 is not None and p99 > LATENCY_BUDGET_MS:
return fallback_model
return preferred_model
The caller records actual latencies after each response, keeping the rolling window current. When a provider experiences congestion, the p99 climbs, the router shifts traffic away, and the SLA is preserved without human intervention.
Fallback Chains and Circuit Breakers
Adaptive routing is only resilient if it can survive provider failures. The pattern here is a cascading fallback chain: a prioritized list of models where each entry is tried in order until one succeeds.
A naive retry loop has a critical flaw: if the primary provider is completely down, every request waits for a full timeout before trying the fallback. At scale, those timeout waits stack up and inflate p99 latency dramatically.
The circuit breaker pattern solves this. A breaker tracks recent failures for each provider. When failures exceed a threshold, the breaker opens: new requests skip that provider entirely and go directly to the next in the chain. After a cooldown period, the breaker half-opens, allowing a single probe request to test recovery. If the probe succeeds, the breaker closes and normal routing resumes.
import time
from enum import Enum
class BreakerState(Enum):
CLOSED = "closed" # normal: route here
OPEN = "open" # broken: skip entirely
HALF_OPEN = "half_open" # testing: allow one probe
class CircuitBreaker:
def __init__(
self,
failure_threshold: int = 5,
cooldown_seconds: float = 30.0,
):
self.failure_threshold = failure_threshold
self.cooldown_seconds = cooldown_seconds
self.state = BreakerState.CLOSED
self.failure_count = 0
self.opened_at: float = 0.0
def record_success(self) -> None:
self.failure_count = 0
self.state = BreakerState.CLOSED
def record_failure(self) -> None:
self.failure_count += 1
if self.failure_count >= self.failure_threshold:
self.state = BreakerState.OPEN
self.opened_at = time.monotonic()
def is_available(self) -> bool:
if self.state == BreakerState.CLOSED:
return True
if self.state == BreakerState.OPEN:
if time.monotonic() - self.opened_at >= self.cooldown_seconds:
self.state = BreakerState.HALF_OPEN
return True # allow the probe
return False
# HALF_OPEN: allow through; record_success/failure will update state.
# In high-concurrency systems, add a lock here to admit only
# one probe at a time before re-evaluating state.
return True
# One breaker per provider in the fallback chain.
breakers: dict[str, CircuitBreaker] = {
"claude-sonnet-4-6": CircuitBreaker(),
"claude-haiku-4-5": CircuitBreaker(),
}
FALLBACK_CHAIN = ["claude-sonnet-4-6", "claude-haiku-4-5"]
def call_with_fallback(prompt: str) -> str:
"""call_model() is a user-supplied function that calls the provider API."""
for model in FALLBACK_CHAIN:
if not breakers[model].is_available():
continue
try:
response = call_model(model, prompt)
breakers[model].record_success()
return response
except Exception:
breakers[model].record_failure()
raise RuntimeError("All providers in fallback chain are unavailable.")
This pattern eliminates the timeout wall. When a provider goes down, requests skip it in microseconds and reach a healthy backup almost immediately, preserving latency even during outages.
Cost-Quality Trade-Off Policies
Not all workloads should optimize for the same objective. A revenue-critical product feature deserves the best available model regardless of cost. An internal analytics job running overnight should minimize cost subject to a quality floor. These are different optimization policies, and your router should support both explicitly.
Three useful policy modes:
Performance-first: select the highest-quality model that can handle the request. Use this for user-facing features where quality directly affects retention or conversion.
Balanced: maximize expected quality subject to a per-request cost cap. A cap of $0.002 per request constrains the router to mid-tier models while still preferring more capable options within that budget.
Cost-first: minimize cost subject to a minimum quality threshold. Use this for batch enrichment, classification pipelines, or any task where "good enough" is genuinely sufficient.
from dataclasses import dataclass
@dataclass
class ModelOption:
name: str
cost_per_1k_tokens: float # blended input + output estimate
quality_score: float # 0-1, calibrated on your own evals
# Costs are illustrative; verify against your provider's current pricing page.
MODEL_POOL: list[ModelOption] = [
ModelOption("claude-haiku-4-5", cost_per_1k_tokens=0.0008, quality_score=0.72),
ModelOption("claude-sonnet-4-6", cost_per_1k_tokens=0.005, quality_score=0.88),
ModelOption("claude-opus-4-5", cost_per_1k_tokens=0.020, quality_score=0.97),
]
def select_model_by_policy(
policy: str,
cost_cap_per_1k: float = 0.01,
quality_floor: float = 0.75,
) -> str:
if policy == "performance_first":
return max(MODEL_POOL, key=lambda m: m.quality_score).name
if policy == "cost_first":
eligible = [m for m in MODEL_POOL if m.quality_score >= quality_floor]
return min(eligible, key=lambda m: m.cost_per_1k_tokens).name
# "balanced": best quality within cost cap
eligible = [m for m in MODEL_POOL if m.cost_per_1k_tokens <= cost_cap_per_1k]
return max(eligible, key=lambda m: m.quality_score).name
Making trade-off policies explicit in code forces the team to have a conscious conversation about what each feature actually needs. It also creates a natural A/B testing surface: route a percentage of traffic to each policy and compare outcomes directly.
Prompt Optimization and Batching as Multipliers
Routing alone is not the whole story. Two complementary levers multiply routing savings: prompt compression and request batching.
Prompt compression means keeping system prompts tight, avoiding unnecessary chain-of-thought scaffolding on simple queries, and stripping redundant context from repeated calls. Every token eliminated reduces cost directly and often reduces latency. On classification-heavy workloads, careful prompt engineering routinely cuts token counts by 30 to 50 percent, and those savings stack on top of routing savings.
Request batching means grouping multiple independent queries into a single API call where the provider supports it, or routing them through an inference server like vLLM. vLLM's continuous batching engine adds new requests to in-flight batches as GPU slots open, dramatically improving throughput and reducing per-token cost at high request rates. Combined with intelligent routing, batching can push total cost reduction past 70 percent on comparable workloads.
The order of implementation matters. Start with routing (highest single-lever impact), then layer in prompt compression (immediate and low-risk), then batching (meaningful gains at scale, higher implementation complexity).
Observability and Closed-Loop Learning
An adaptive router without observability is a black box. You have no way to verify whether routing decisions were correct, no signal for retraining, and no visibility into cost drift over time.
Every routed request should produce a log record containing at minimum:
- The query (or a hashed representation for privacy)
- The model selected and the rationale (which policy, which complexity score)
- Token usage and monetary cost
- End-to-end latency (p50, p95, p99 over rolling windows)
- A quality signal: a user feedback event, a downstream metric, or an LLM-as-judge score
This data feeds a continuous improvement loop. Analyzing which models handled which query types successfully lets you recalibrate complexity thresholds. Identifying cost outliers (queries that consumed ten times the expected tokens) reveals prompt engineering opportunities. Detecting drift in model quality over time triggers re-evaluation of your model pool weights.
The contextual bandit framing is a useful mental model here: treat each routing decision as an action, and the reward as a calibrated combination of quality and cost. The logged outcome data becomes training signal, and the router gradually learns which models serve which query patterns best.
Without this loop, routing is a static policy. With it, routing becomes a data-driven system that continuously improves as your traffic grows.
Putting It All Together
The full adaptive inference stack layers these components in request-flow order:
- Complexity classifier: extracts signals from the incoming query and produces a complexity score
- Latency budget check: compares rolling p99 for the candidate model against the active SLA budget; downgrades to a faster model if the budget is at risk
- Policy selector: applies the cost-quality trade-off policy (performance-first, balanced, or cost-first) to pick the final model
- Circuit breaker check: confirms the selected model's breaker is closed; if open, advances to the next model in the fallback chain
- Request dispatch: sends the request and records latency and token usage
- Observability write: logs the full decision record for closed-loop learning
Each component is independently testable and can be introduced incrementally. You do not need all six layers on day one. Start with a complexity classifier and two model tiers. Add latency budget tracking when you have an SLA to protect. Add circuit breakers when you have multiple providers. The architecture grows with your needs.
Conclusion
Adaptive inference is not a future optimization. It is the standard infrastructure for any team running LLMs at meaningful scale in 2026. The gap between teams that route per-request and teams that hardcode a single model is measurable in millions of dollars and in user-facing latency percentiles.
The building blocks are well understood: complexity classifiers, latency budgets, fallback chains, cost-quality policies, and an observability loop that closes the feedback cycle. None requires cutting-edge research. All require intentional architecture and the discipline to treat model selection as a runtime decision rather than a deployment-time assumption.
Pick a model. Pick a cheaper one. Route between them intelligently. The savings will compound, and so will the reliability.