When an LLM provider has an incident, most apps respond by hammering the broken endpoint with retries. Each failed call raises the queue depth, increases latency for every healthy request behind it, and pushes the provider closer to rate-limiting you on top of their existing outage. A circuit breaker stops this loop before it starts.
The pattern is borrowed from electrical engineering: when current exceeds the safe threshold, the circuit opens and stops flowing. In your code, when LLM call failures hit a configurable threshold, the circuit opens and subsequent calls fail fast without touching the network, giving the provider time to recover while your users see a controlled fallback instead of a cascade of 30-second timeouts.
Why backoff alone is not enough
Exponential backoff with jitter handles transient errors well. A single failed call retries after 2 seconds, then 4, then 8. If the hiccup clears in under a minute, your users see a brief delay and move on.
The problem is that backoff is per-request. It gives up on that request and makes the error visible to that user. The next request that comes in immediately fires at the same broken endpoint and starts the same countdown. During a sustained provider incident, every new request follows the same path: attempt, fail, retry, fail, surface error. Nothing stops the incoming flood from probing a provider that has been down for 20 minutes.
Circuit breakers solve a different problem: tracking provider health across requests and stopping calls from reaching a known-bad endpoint entirely.
The three states
A circuit breaker transitions between three states based on observed outcomes:
CLOSED means normal operation. Every call goes through, and failures are counted against the threshold. This is the default state when the provider is healthy.
OPEN means the failure threshold was hit. All calls to this provider are rejected immediately, without a network round-trip. Callers get a synchronous error in under 1ms instead of waiting 30 seconds for a timeout. The circuit stays open for a configurable cooldown period (60 seconds by default).
HALF_OPEN means the cooldown expired and it is time to test recovery. One probe call is allowed through. If it succeeds, the circuit closes. If it fails, the circuit reopens and the cooldown resets.
The detail that matters in production: requiring two consecutive successes before closing, not just one. A flaky provider that passes one call in ten will thrash the breaker open and closed in a loop if you close on the first success.
Implementation
class LLMCircuitBreaker {
constructor({ failureThreshold = 5, successThreshold = 2, cooldownMs = 60_000, name = 'provider' } = {}) {
this.name = name;
this.failureThreshold = failureThreshold;
this.successThreshold = successThreshold;
this.cooldownMs = cooldownMs;
this.state = 'closed';
this.failureCount = 0;
this.successCount = 0;
this.openedAt = null;
this.probeInFlight = false;
}
get isAvailable() {
if (this.state === 'closed') return true;
if (this.state === 'open') {
if (Date.now() - this.openedAt >= this.cooldownMs) {
this._transition('half-open');
return true;
}
return false;
}
return !this.probeInFlight; // half-open: one probe at a time
}
_transition(next) {
const prev = this.state;
this.state = next;
if (next === 'open') {
console.warn(`[circuit] ${this.name} OPEN after ${this.failureCount} failures`);
this.openedAt = Date.now();
this.failureCount = 0;
}
if (next === 'closed' && prev !== 'closed') {
const durationSec = Math.round((Date.now() - this.openedAt) / 1000);
console.info(`[circuit] ${this.name} CLOSED after ${durationSec}s`);
this.failureCount = 0;
this.successCount = 0;
this.probeInFlight = false;
}
if (next === 'half-open') {
this.successCount = 0;
this.probeInFlight = false;
}
}
recordSuccess() {
if (this.state === 'half-open') {
this.probeInFlight = false;
if (++this.successCount >= this.successThreshold) this._transition('closed');
} else {
this.failureCount = 0;
}
}
recordFailure() {
this.probeInFlight = false;
if (this.state === 'half-open') {
this._transition('open');
} else if (++this.failureCount >= this.failureThreshold) {
this._transition('open');
}
}
async execute(fn) {
if (!this.isAvailable) {
const secsLeft = Math.ceil((this.cooldownMs - (Date.now() - this.openedAt)) / 1000);
throw new Error(`Circuit open — probe in ${secsLeft}s`);
}
if (this.state === 'half-open') this.probeInFlight = true;
try {
const result = await fn();
this.recordSuccess();
return result;
} catch (err) {
this.recordFailure();
throw err;
}
}
}
Per-provider isolation
One breaker instance per provider. If you share a single breaker across Anthropic and OpenAI, an Anthropic incident opens it and starts rejecting OpenAI calls too. That defeats the purpose.
const breakers = {
anthropic: new LLMCircuitBreaker({ name: 'anthropic', failureThreshold: 5, cooldownMs: 60_000 }),
openai: new LLMCircuitBreaker({ name: 'openai', failureThreshold: 5, cooldownMs: 60_000 }),
};
async function callWithFallback(prompt, callLLM) {
const providers = ['anthropic', 'openai'];
const available = providers.filter(p => breakers[p].isAvailable);
if (available.length === 0) throw new Error('All providers circuit-open');
for (const provider of available) {
try {
return await breakers[provider].execute(() => callLLM(provider, prompt));
} catch (err) {
if (err.message.startsWith('Circuit open')) continue;
// real failure: breaker already recorded it; try next provider
}
}
throw new Error('All providers failed');
}
The callLLM function here is whatever wraps your HTTP call to the provider. The LLM fallback chain guide covers building that wrapper; this post focuses on the circuit breaker layer that sits between it and your business logic.
What counts as a failure
Not every error should trip the breaker. A 400 (bad request) or 422 (validation error) means the request itself is wrong, not the provider. Recording it against the breaker would cause you to route all traffic to your fallback because of a bug in your own prompt.
Count these: 5xx responses, 429 (rate limited), 408 (timeout), and network-level errors like ECONNRESET or ETIMEDOUT.
Skip these: 400, 401, 403, 422. These are your problem, not the provider's.
function shouldTripBreaker(status) {
if (status >= 500) return true;
if (status === 429) return true;
if (status === 408) return true;
return false;
}
The production LLM rate-limit patterns post covers per-status routing logic in more detail, including how to distinguish rate-limited from unavailable when deciding between queuing and immediate failover.
Threshold tuning
Five failures and a 60-second cooldown is a reasonable starting point for low-volume apps. At higher traffic, raw failure count leads to false positives: a burst of malformed requests can trip the breaker on a perfectly healthy provider.
Above roughly 500 calls/hour, switch to a failure rate tracked in a rolling time window:
class RateBasedCircuitBreaker extends LLMCircuitBreaker {
constructor({ windowMs = 60_000, errorRateThreshold = 0.5, minRequests = 10, ...rest } = {}) {
super(rest);
this.windowMs = windowMs;
this.errorRateThreshold = errorRateThreshold;
this.minRequests = minRequests;
this.window = [];
}
_recordOutcome(success) {
const now = Date.now();
this.window.push({ ts: now, success });
this.window = this.window.filter(e => now - e.ts < this.windowMs);
if (this.window.length < this.minRequests) return;
const errorRate = this.window.filter(e => !e.success).length / this.window.length;
if (errorRate >= this.errorRateThreshold) this._transition('open');
}
recordSuccess() {
if (this.state === 'half-open') { super.recordSuccess(); } else { this._recordOutcome(true); }
}
recordFailure() {
if (this.state === 'half-open') { super.recordFailure(); } else { this._recordOutcome(false); }
}
}
The minRequests guard prevents a single failure from tripping the breaker during startup, when the window has only one or two data points.
Logging state transitions
A breaker that silently opens and closes is invisible until something goes wrong. The _transition method above already logs on open and close. Wire the output into whatever you use for LLM observability.
The LLM observability guide shows an async logging middleware that captures per-call cost, latency, and quality signals. Circuit state fits there naturally: include breaker_state in the log payload whenever a call is routed through the breaker, so you can correlate a spike in fallback provider usage with a specific circuit-open event in your dashboard.
Where this fits in the production stack
Circuit breakers are the layer above individual retry logic and below full provider routing. A request comes in, hits the routing layer which picks the cheapest available model, the circuit breaker confirms that provider is not currently open, and the call goes out. If it fails, the circuit records it. When the threshold is hit, the router's next call to isAvailable returns false and it skips to the next option.
The LLM prompt routing guide covers building the router that wraps the breaker. Together the two patterns give you cost-first routing with automatic outage detection.
For stream stall detection, the circuit breaker sits one layer out: a stall triggers an abort, the abort counts as a failure in the breaker, and after enough stalls on one provider the circuit opens. You do not need special handling for stalls specifically.
LLMTest's proxy routes through multiple providers automatically and surfaces per-provider latency in real time. The fallback configuration handles provider switching at the infrastructure level. Adding a circuit breaker in your app code gives you the same switching logic with control over thresholds, logged state changes, and failure classification that a generic proxy cannot provide per-request.
Track your own provider reliability at LLMTest.