Your Next.js app calls one LLM. That provider has a bad shift: a 429 cascade, a rolling 503 during a model deployment, ten minutes of silence with no status page update. Your AI feature goes down with it.
The fix is a two-provider route. The frontend hits one endpoint. That route tries Anthropic first; if the response comes back with a retriable error, it switches to OpenAI without the user seeing anything. Below is the TypeScript implementation and the honest math on what that switch costs you in latency.
Why the App Router route handler is the right place for this
Client-side fallback leaks your API keys. Middleware applied at the Edge Runtime adds latency to every request, not just the ones hitting your AI feature. The App Router route handler is the right level: it runs server-side, keeps keys out of the client bundle, and executes only when an AI endpoint is actually called.
The utility function lives in lib/ rather than inside the route file so you can reuse it across multiple endpoints without duplicating the fallback logic. Two files total.
The 25-line fallback utility
// lib/llm-fallback.ts
type Msg = { role: 'user' | 'assistant'; content: string };
async function callProvider(
url: string,
headers: Record<string, string>,
body: object
): Promise<string> {
const res = await fetch(url, {
method: 'POST',
headers: { 'Content-Type': 'application/json', ...headers },
body: JSON.stringify(body),
signal: AbortSignal.timeout(12_000),
});
if (!res.ok) {
const text = await res.text();
const err: Error & { status?: number } = new Error(text);
err.status = res.status;
throw err;
}
const data: any = await res.json();
// Anthropic returns data.content[0].text; OpenAI returns data.choices[0].message.content
return data.choices?.[0]?.message?.content ?? data.content?.[0]?.text ?? '';
}
export async function withFallback(
messages: Msg[]
): Promise<{ text: string; via: string }> {
try {
const text = await callProvider(
'https://api.anthropic.com/v1/messages',
{
'x-api-key': process.env.ANTHROPIC_API_KEY!,
'anthropic-version': '2023-06-01',
},
{ model: 'claude-haiku-4-5-20251001', messages, max_tokens: 1024 }
);
return { text, via: 'anthropic' };
} catch (err: any) {
if (err.status === 400 || err.status === 401) throw err;
return {
text: await callProvider(
'https://api.openai.com/v1/chat/completions',
{ Authorization: `Bearer ${process.env.OPENAI_API_KEY}` },
{ model: 'gpt-4o-mini', messages, max_tokens: 1024 }
),
via: 'openai',
};
}
}
The callProvider helper works with both APIs because it branches on the response shape: Anthropic returns content[0].text, OpenAI returns choices[0].message.content. The nullish chain on the last line covers both without branching in the caller.
One caveat on system prompts: Anthropic's Messages API does not accept role: 'system' inside the messages array. Pass system instructions as a separate top-level system field alongside messages in the Anthropic body, and strip them from the messages array before that call.
Wire it into the route handler
// app/api/chat/route.ts
import { NextRequest, NextResponse } from 'next/server';
import { withFallback } from '@/lib/llm-fallback';
export async function POST(request: NextRequest) {
const { messages } = await request.json();
const { text, via } = await withFallback(messages);
return NextResponse.json({ content: text, provider: via });
}
Return provider in the response. Log it. After a few days of production traffic you will know whether the fallback is firing at all, how often it fires, and whether the backup provider is holding up when the primary is down.
Which errors trigger the switch
The if (err.status === 400 || err.status === 401) throw err; guard is doing real work. Not every error from the primary should trigger a fallback.
Retriable (switch to fallback):
429: rate limited. The server is responding; you're over quota. The backup can take the call.500,502,503: server errors during outages or deployments. Fallback helps.- No status (network error):
ECONNRESET,ENOTFOUND, TCP drop before headers arrive. Fallback helps. - Timeout (12 s exceeded by
AbortSignal): provider is slow or stalled. Fallback helps, but read the latency section below before you set this threshold.
Non-retriable (throw immediately):
400: malformed request. Your prompt body is the problem, not the provider. Sending it to OpenAI fails the same way.401: bad API key. Fix the config; a fallback call wastes tokens on a credential error that will also fail.402: no credits. This is a billing problem. Rerouting to a second provider does not fix it and may silently deplete that account too.
The latency cost of switching mid-request
When Anthropic returns a fast error (429, 503), you typically see that response in 50-150 ms. The total cost of switching is:
- 50-150 ms: fast Anthropic error
- 500-1,500 ms: a Haiku or GPT-4o-mini response
- = 550-1,650 ms total
That's acceptable. The user gets a response instead of an error message at roughly the same latency they'd see on any normal request.
When Anthropic times out (12 seconds with no response), the math changes completely:
- 12,000 ms: hitting the
AbortSignal.timeout - 500-1,500 ms: the fallback call
- = 12,500-13,500 ms total
That is not a recovery. That is a 12-second pause followed by a second full round-trip. If end-to-end latency matters to your feature, set the primary timeout lower. Four to six seconds works for Haiku-class models; eight to ten seconds for larger ones. You will occasionally fall back on slow-but-valid primary responses, but your p99 latency stops being 13 seconds.
For streaming paths the detection works differently. Rather than waiting for the full response, you catch stalls at the chunk level. Stall detection with per-chunk watchdog timers in Node.js covers that pattern separately.
Test the fallback locally before trusting it
Add a dev-only route that injects a 503 on the primary provider so you can confirm the switch fires before a real outage does:
// app/api/chat-test-fallback/route.ts (dev only, delete before deploying)
import { NextRequest, NextResponse } from 'next/server';
import { withFallback } from '@/lib/llm-fallback';
const _origFetch = globalThis.fetch;
globalThis.fetch = async (input: RequestInfo | URL, init?: RequestInit) => {
if (typeof input === 'string' && input.includes('anthropic.com')) {
throw Object.assign(new Error('injected 503'), { status: 503 });
}
return _origFetch(input, init);
};
export async function POST(request: NextRequest) {
const { messages } = await request.json();
return NextResponse.json(await withFallback(messages));
}
Call this endpoint with a test prompt and verify provider: "openai" comes back. Remove the file before you ship.
When to use this vs a managed fallback layer
This pattern is the right call when you want zero extra services, direct billing relationships with each provider, and full control over exactly which error codes trigger a provider switch. One utility function, no dashboard to configure, no additional API key to rotate.
For anything beyond two providers, managed fallback at the routing layer scales better. OpenRouter's models array handles multi-provider fallback in a single request body parameter with no extra code. LLMTest's automatic fallback reroutes 429s and 5xx errors before they reach your application and adds cost tracking across every provider in the chain. Both add one network hop between your app and the model; the 25-line version adds none.
For a full comparison of the three approaches (LiteLLM, OpenRouter, LLMTest) and the soft-failure cases none of them handle by default, building an LLM fallback chain covers what the managed tools miss.
Set up fallback routing in your app, then check LLMTest to see which provider actually absorbs the fallback traffic and what it costs per request.