How to retry LLM API calls in Node.js in 2026: three-layer pattern

Your retry loop fires. It hits a 429. It fires again. The rate-limit window has not cleared. It fires a third time, a fourth, then gives up and surfaces an error to the user. Meanwhile 200 other requests are doing the same thing, all retrying in lockstep after the same trigger, turning a momentary provider spike into a sustained hammer against the API.

The fix is not fewer retries. It is structuring them into three distinct layers: a client that backs off with jitter, a proxy that switches providers when the client gives up, and a dead-letter queue that parks failed requests for later rather than surfacing them to users. Each layer handles a different class of failure.

Layer 1: Client retry with backoff, jitter, and error classification

The foundation is a wrapper with three properties: exponential backoff (each wait is longer than the last), full jitter (randomized to spread retries across concurrent clients), and accurate error classification (429 and 5xx retry; 400/401/422 do not).

const RETRIABLE = new Set([429, 500, 502, 503, 504]);

async function withRetry(fn, { maxAttempts = 4, baseMs = 600, maxMs = 30_000 } = {}) {
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      return await fn();
    } catch (err) {
      const isLast = attempt === maxAttempts - 1;
      const status = err.status ?? err.response?.status;

      if (isLast || !RETRIABLE.has(status)) throw err;

      // Use the server's Retry-After header when present
      const retryAfterMs =
        Number(err.response?.headers?.get?.('retry-after') ?? 0) * 1000;

      // Full jitter: random in [0, min(maxMs, base * 2^attempt)]
      const cap = Math.min(maxMs, baseMs * 2 ** attempt);
      const jitter = Math.random() * cap;

      await sleep(Math.max(retryAfterMs, jitter));
    }
  }
}

const sleep = ms => new Promise(r => setTimeout(r, ms));

Two details make this production-grade. First, the Retry-After header: when OpenAI or Anthropic include it in a 429, that number is exact. The provider is telling you when its rate-limit window resets. Use it as the floor, not something to override with your own math. Second, non-retriable status codes: 400 (bad request), 401 (auth error), and 403 (permission denied) mean the request itself is wrong. Retrying them burns tokens and budget on calls that will never succeed.

This layer handles transient failures well: overloaded providers, brief network hiccups, capacity spikes that clear in under a minute. It does not handle provider outages or authentication failures, because those require a different response.

Layer 2: Proxy fallback when the client gives up

When client retries exhaust against one provider, the second layer routes the same request to a backup provider before surfacing the error to the caller. LLMTest's proxy handles this automatically, but here is the equivalent in plain Node.js for teams routing directly:

async function callWithFallback(payload) {
  try {
    return await withRetry(() => callProvider('primary', payload));
  } catch (primaryErr) {
    console.warn('[llm] primary exhausted, trying fallback:', primaryErr.message);
    return await withRetry(() => callProvider('fallback', adaptPayload(payload)));
  }
}

adaptPayload translates the request shape between providers. Field names differ (max_tokens vs maxTokens, system as a top-level field vs a system role message), and you need to remap the model ID. Budget-conscious teams often choose a cheaper fallback: if GPT-5.6 Sol fails, fall to Claude Haiku 4.5 at a fraction of the input cost. Quality degrades slightly; the user gets a response instead of an error.

The guide on building an LLM fallback chain in under 10 minutes covers provider-switching, payload translation, and the LiteLLM and OpenRouter configs that reduce the manual wiring.

Layer 3: Dead-letter queue for background workloads

Retrying synchronously caps out at your request timeout. For background workloads (batch document processing, overnight enrichment jobs, webhook handlers) no user is waiting, so you can defer failed requests rather than failing them outright.

A dead-letter queue (DLQ) pattern captures failed requests after all retries exhaust and re-enqueues them with a delay:

const DLQ_KEY = 'llm:dlq';
const MAX_DLQ_ATTEMPTS = 3;

async function processWithDLQ(redis, payload) {
  try {
    return await callWithFallback(payload);
  } catch (err) {
    const attempts = (payload.__dlqAttempts ?? 0) + 1;
    if (attempts > MAX_DLQ_ATTEMPTS) {
      console.error('[dlq] exhausted:', payload, err.message);
      return;
    }
    // Park for 5 minutes, then retry
    await redis.zadd(
      DLQ_KEY,
      Date.now() + 5 * 60_000,
      JSON.stringify({ ...payload, __dlqAttempts: attempts, __dlqErr: err.message })
    );
    console.info(`[dlq] parked attempt ${attempts}/${MAX_DLQ_ATTEMPTS}`);
  }
}

A separate consumer polls the sorted set for entries whose score (the deferred timestamp) is in the past and re-runs them. If you already use Redis, BullMQ's built-in delay and dead-letter job types handle this bookkeeping without custom polling code. The pattern above is useful when you want the logic inline without an additional dependency.

The key difference from layer 1: where client retries fire within the same request lifetime (seconds), the DLQ retries fire minutes later, after the provider has likely cleared. During a 30-minute outage, that is the difference between burning four synchronous retries per request and parking each request once for a deferred single attempt.

Per-provider limits that change the math

OpenAI and Anthropic both reset their rate-limit windows every 60 seconds and include reliable Retry-After headers with 429 responses. DeepSeek's headers sometimes omit Retry-After, making jitter the only signal. Google's AI Studio API follows the same 60-second pattern as OpenAI.

These defaults assume you hit a 429 mid-minute. If your workload fires requests in tight bursts at the same instant, the token budget pre-check pattern from the LLM rate limits guide prevents the 429 before it happens rather than recovering after.

What 1,000 retried requests actually cost

Assuming 30% of requests hit at least one retry at 1,000 calls per day, with average inputs of 2,000 tokens and four retry attempts each:

• Retry token overhead: 300 requests x 3 extra attempts x 2,000 tokens = 1.8M additional input tokens per day
• At Claude Haiku 4.5 rates ($0.80/M input): $1.44 per day in retry overhead
• At GPT-5.6 Sol rates ($5/M input): $9 per day in retry overhead

The DLQ layer cuts into this: instead of four synchronous retries that all fail during a sustained outage, the DLQ parks each request for 5 minutes and retries once when conditions are better. For a 30-minute outage, that is the difference between 1,200 redundant calls and roughly 300 deferred calls per request batch.

Circuit breakers cut it further by stopping calls from reaching a provider known to be down. The LLM circuit breakers guide covers the state machine that sits above the retry layer: when failure count crosses a threshold, the circuit opens and subsequent calls fail fast without a network round-trip.

The three layers compose as: client retry, then proxy fallback, then DLQ, in that order, each reducing the failure surface the next layer has to handle. Start with layer 1. Most apps recover 80% of transient failures there without ever touching the others.

Try LLMTest's proxy to get layers 2 and 3 as managed infrastructure, with per-provider circuit state, automatic failover, and a dashboard showing retry rates by provider.