You have 200 concurrent users hitting your /chat endpoint. Each one triggers a call to the LLM API. Your provider allows 10 concurrent requests. The other 190 requests fire anyway, hit 429s, retry with backoff, and pile on top of the next minute's burst. By the time your exponential backoff clears the backlog, half your users have been waiting 20 seconds.
The fix is not more retries. Retries react to failures that have already happened. What you need is a queue that controls how many requests leave your server in the first place, so the provider never sees the spike.
This is the in-process approach: no Redis, no BullMQ, no separate worker process. Under 60 lines. Works in any Node.js 18+ app.
Why retries are the wrong first line of defence
Retries have their place. A provider goes down for 30 seconds, a single request times out, a network hiccup drops one packet. These are isolated failures that backoff handles well.
Burst traffic is different. When 200 requests arrive in the same second, each fires against the same rate limit simultaneously. The first batch succeeds; the rest get 429s. Your backoff code waits a randomised delay and retries. But so does every other request. They all wake up around the same time and hit the API in another wave. The thundering herd does not solve itself through retries; it just cycles at a lower peak.
A concurrency queue stops this before the API sees the traffic. You cap how many requests are in-flight at once. Extras wait in memory. The provider sees a steady stream of, say, 8 concurrent requests instead of a spike of 200.
For the backoff and circuit-breaker patterns that belong on top of this queue, see how to handle LLM rate limits in production.
The queue class: concurrency limiter under 40 lines
The core structure is a semaphore: a counter that tracks how many requests are running, with a waiting list for the rest.
export class LLMQueue {
#running = 0;
#queue = [];
#userCounts = new Map();
constructor({ concurrency = 5, perUserDepth = 8, maxDepth = 200 } = {}) {
this.concurrency = concurrency;
this.perUserDepth = perUserDepth;
this.maxDepth = maxDepth;
}
add(fn, { userId = 'anon', priority = 0 } = {}) {
const userDepth = this.#userCounts.get(userId) ?? 0;
if (userDepth >= this.perUserDepth) throw new Error('per-user queue full');
if (this.#queue.length >= this.maxDepth) throw new Error('queue at capacity');
this.#userCounts.set(userId, userDepth + 1);
return new Promise((resolve, reject) => {
this.#queue.push({ fn, userId, priority, resolve, reject });
if (priority > 0) this.#queue.sort((a, b) => b.priority - a.priority);
this.#drain();
});
}
#drain() {
while (this.#running < this.concurrency && this.#queue.length > 0) {
const { fn, userId, resolve, reject } = this.#queue.shift();
this.#running++;
fn().then(resolve, reject).finally(() => {
this.#running--;
const c = (this.#userCounts.get(userId) ?? 1) - 1;
if (c <= 0) this.#userCounts.delete(userId);
else this.#userCounts.set(userId, c);
this.#drain();
});
}
}
}
Usage in a route handler:
const queue = new LLMQueue({ concurrency: 8 });
app.post('/chat', async (req, res) => {
try {
const result = await queue.add(
() => callLLM(req.body.prompt),
{ userId: req.user.id }
);
res.json(result);
} catch (err) {
const status = err.message.includes('full') ? 503 : 500;
res.status(status).json({ error: err.message });
}
});
concurrency maps to your provider's concurrent request limit. For Claude Fable 5 on a standard paid plan, Anthropic's default is 10 concurrent connections per account. Set concurrency to 8 to keep two slots free for headroom. For GPT-5.6 Sol on OpenAI tier 2, the limit is per-API-key and shown in your dashboard. When you route through the LLMTest proxy, the concurrency limit is managed on the proxy side and you skip tracking it yourself.
Per-user fairness: stop one user from starving the rest
Without per-user limits, one script-heavy user can fill the entire queue, effectively freezing everyone else. perUserDepth = 8 caps how many queued requests any single user can hold at once. Their ninth request throws immediately instead of sitting behind their own backlog.
Eight is a practical default for chat interfaces. A user typing in a browser rarely sends more than two concurrent requests. Eight leaves headroom for "regenerate" spam and retry floods without crowding other users. A single-tenant internal tool can raise this; a public API with untrusted users should lower it to 3 or 4.
The #userCounts map tracks live queue depth per user, not total requests. It increments on enqueue and decrements when the request completes. A user who sends eight requests and gets four answered has four slots open again immediately. The accounting stays accurate across concurrent drains because #drain runs synchronously per iteration.
For per-user cost tracking rather than concurrency tracking, per-user LLM cost guardrails in Node.js covers the complementary pattern: daily token budgets, soft-limit model downgrade, and hard stops.
Backpressure: reject early instead of queuing forever
maxDepth = 200 is the total queue size across all users. When the queue hits that ceiling, add() throws immediately. The route handler returns 503 with no API call made.
This is backpressure: deliberately rejecting requests that cannot be served within a reasonable time, rather than queuing them indefinitely. The principle applies directly to LLM workloads: shed excess requests at the entry point to keep latency predictable for the requests that do get through. As video research into the backpressure pattern shows, the goal is maintaining stable tail latency, not maximising throughput.
The alternative, an unbounded queue, looks safer but is not. A user whose request joined the queue three minutes ago gets their response after the flood clears, but the response is stale, the HTTP connection may have timed out, and you paid for the API call either way. A 503 at the gate costs nothing and lets the client show a "try again in a moment" message instead.
Tune maxDepth against your median completion time. If each LLM call takes 5 seconds and concurrency is 8, the queue drains at 1.6 calls per second. With maxDepth = 200, the worst-case wait for the last slot is 125 seconds. Most apps should set a much lower ceiling (40 to 60) and let the 503s trigger a visible retry message rather than silent queue growth.
Priority lanes: VIP requests move ahead
Pass priority: 1 (or higher) on the options object to let certain requests jump ahead of queued standard requests. The queue re-sorts by priority on each prioritised insert. This costs an O(n) sort only when a nonzero priority is set, so standard requests skip the sort entirely.
const priority = req.user.plan === 'pro' ? 1 : 0;
const result = await queue.add(
() => callLLM(req.body.prompt),
{ userId: req.user.id, priority }
);
In practice, two priority levels are enough: paid users at 1 and free-tier users at 0. Three or more tiers tend to create a second scheduling problem. Internal health checks can use priority: 10 as a convention for "always run next."
When to graduate to BullMQ
This queue lives in process memory. If the Node.js server restarts, in-flight requests complete but queued requests are lost without a response. For chat applications where the client can retry, that is acceptable: the user sees a loading state, the server restarts, the client retries and lands in a fresh queue.
When that is not acceptable (because LLM calls take 60-plus seconds, are expensive enough that losing them matters, or the user will not retry on their own), move to a durable queue. The BullMQ background job pattern for Node.js covers the standard approach: route handlers return a job ID immediately, a separate worker process handles the LLM call, and the result lands in Redis. The tradeoff is operational complexity: you now run Redis and a worker process alongside your API.
In-process queuing handles the common case. Graduate to BullMQ when persistence matters more than simplicity.
LLMTest routes requests through a managed proxy that handles provider rate limits, concurrency enforcement, and automatic failover. Get started and skip maintaining this queue yourself.