Most LiteLLM tutorials show simple-shuffle and move on. That works fine until you hit a provider rate limit at peak traffic, a model returns garbage JSON at 2am, or your OpenAI bill goes 40% over forecast because one model group absorbed a traffic spike.
These four routing patterns solve those specific problems. Each one addresses a distinct failure mode that simpler configs miss.
Pattern 1: Weighted routing
Weighted routing splits traffic across providers according to ratios you set, not round-robin. Use it when you want OpenAI handling 70% of requests and Anthropic handling 30%, or when you're A/B testing two model versions before committing.
LiteLLM's simple-shuffle strategy uses the tpm (tokens per minute) and rpm (requests per minute) values on each deployment as weights. The router picks proportionally: a deployment with tpm: 200000 gets twice the traffic of one with tpm: 100000.
model_list:
- model_name: frontier
litellm_params:
model: openai/gpt-5.5
api_key: os.environ/OPENAI_API_KEY
tpm: 200000
rpm: 2000
- model_name: frontier
litellm_params:
model: anthropic/claude-opus-4-8
api_key: os.environ/ANTHROPIC_API_KEY
tpm: 100000
rpm: 1000
router_settings:
routing_strategy: simple-shuffle
num_retries: 2
Your app calls model: frontier. The router handles the split invisibly. When GPT-5.5 is at capacity, requests spill to Opus 4.8 within the same group.
The catch: simple-shuffle allocates by config ratio, not by real-time latency or error rate. If GPT-5.5 starts timing out, it still gets 2x the traffic until you notice. For dynamic traffic shaping based on actual performance, pattern 3 handles that.
Pattern 2: Fallback routing
Fallback gives you a strict priority order. Your app calls a named group, LiteLLM tries the primary deployment, and cascades to backups only on hard errors (429, 500, 503, timeout). No application code changes when you add or reorder fallbacks.
model_list:
- model_name: primary
litellm_params:
model: anthropic/claude-opus-4-8
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: secondary
litellm_params:
model: openai/gpt-5.5
api_key: os.environ/OPENAI_API_KEY
- model_name: budget-backstop
litellm_params:
model: openai/gpt-4o-mini
api_key: os.environ/OPENAI_API_KEY
router_settings:
fallbacks:
- primary: [secondary, budget-backstop]
num_retries: 2
retry_after: 5
timeout: 30
When primary returns a 503, LiteLLM immediately tries secondary. If that also fails within the timeout window, it falls to budget-backstop. Your app sees one response or one final error, not the retry loop.
The limitation to plan around: LiteLLM only fires fallback on hard HTTP errors by default. Soft failures where the model returns HTTP 200 with empty content or malformed JSON pass through as successes. If you're parsing structured output, wrap your parsing in a try/catch and surface errors through your own validation layer. The LLMTest fallback docs cover adding a quality-gate trigger on top of LiteLLM's native mechanism.
Pattern 3: Semantic routing
Semantic routing uses the content of the incoming message to decide which model handles it. Coding tasks go to a frontier model. Simple classification tasks go to a budget model. The router embeds each incoming prompt and finds the route whose utterances are semantically closest.
LiteLLM's auto-router, added in 2025, connects to the semantic-router library and handles this in two files: a proxy_config.yaml referencing a router.json that defines the routes.
# proxy_config.yaml
model_list:
- model_name: semantic_router
litellm_params:
model: auto_router/auto_router_1
auto_router_config_path: ./router.json
auto_router_default_model: anthropic/claude-haiku-4-5
auto_router_embedding_model: openai/text-embedding-3-small
{
"encoder_type": "openai",
"encoder_name": "text-embedding-3-small",
"routes": [
{
"name": "coding-tasks",
"utterances": [
"write a function that",
"debug this stack trace",
"refactor this class",
"implement this feature",
"fix this failing test"
],
"score_threshold": 0.55,
"llm": "anthropic/claude-opus-4-8"
},
{
"name": "simple-tasks",
"utterances": [
"summarize this text",
"translate to French",
"classify this as",
"extract the company name",
"what is the capital of"
],
"score_threshold": 0.5,
"llm": "anthropic/claude-haiku-4-5"
}
]
}
When no route reaches its score_threshold, the request falls through to auto_router_default_model. Set the threshold too high and most requests miss; too low and you get mis-routes. Start at 0.5 and tune based on logs.
The most important step is writing utterances from real traffic, not from imagination. Sample 200 messages your actual users send, cluster them by hand, then write utterances from those clusters. Utterances written from imagination tend to catch the idealized version of user intent, not the actual phrasing. A misrouted complex prompt landing on claude-haiku-4-5 costs more than the savings from correct routing.
Also verify the router.json path is accessible from wherever LiteLLM starts. Relative paths break in Docker if the working directory isn't where you expect.
Pattern 4: Budget-capped routing
Budget-capped routing sets a hard spend limit per provider per time window. When OpenAI hits its daily cap, new calls go to Anthropic. When both are exhausted, calls fail fast rather than silently running up charges that you find on the monthly invoice.
model_list:
- model_name: chat
litellm_params:
model: anthropic/claude-opus-4-8
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: chat
litellm_params:
model: openai/gpt-5.5
api_key: os.environ/OPENAI_API_KEY
router_settings:
routing_strategy: provider-budget-routing
provider_budget_config:
anthropic:
budget_limit: 20.00
time_period: 1d
openai:
budget_limit: 30.00
time_period: 1d
time_period accepts 1d, 7d, 30d, and finer units like 1h. The router tracks actual token spend against each provider's real per-token pricing, not estimates.
Why this beats an account-level budget in the provider's billing dashboard: you get per-provider visibility inside your own logs, you can adjust caps without touching the provider's billing UI, and you can set up an alert before you hit the limit rather than after. With an account-level cap, the entire API key goes down when the ceiling is reached. With provider-budget-routing, LiteLLM routes around the exhausted provider and keeps serving requests from the remaining budget.
Choosing between the four
| Pattern | Best fit | What it doesn't handle |
|---|---|---|
| Weighted | Traffic splitting, A/B testing providers | Doesn't adapt to live error rates |
| Fallback | Primary/backup ordering, availability guarantees | Doesn't catch soft failures (200 OK, bad content) |
| Semantic | Mixed-complexity workloads, cost optimization by task type | Requires utterance tuning; adds embedding latency (~50ms) |
| Budget | Hard spend caps, multi-team shared API keys | Doesn't optimize which provider is cheapest per call |
These compose well. A production config might use weighted routing within a model group, budget caps at the provider level, and a fallback group for when both providers exhaust their budgets. Start with fallback: it has the lowest overhead to configure and catches the most common failure mode. Add budget caps once you have real usage data. Layer semantic routing last, after you've collected enough real traffic to write useful utterances.
For the application-level equivalent of semantic routing (routing individual prompts based on a quality signal rather than a config file), cheap-first prompt routing covers that pattern with working Node.js code. For getting a fallback chain running before adding LiteLLM to the stack, building an LLM fallback chain is the faster path.
The complete LiteLLM proxy config reference, including all routing strategies and their parameters, is at /docs/proxy. To add quality-gate routing on top of provider-level availability (catching soft failures and low-quality responses that LiteLLM passes through), start at LLMTest.