GPT-5, Claude Fable 5, and Gemini 3.x can each take 30 to 90 seconds on a complex prompt. A summarization job over a 200-page PDF regularly hits that ceiling. An HTTP request-response cycle cannot carry it reliably. The connection drops, the client retries, and you end up with duplicate jobs, duplicate billing, and confused users.
Background queues solve this cleanly: the API route accepts the job, returns a job ID immediately, and a worker handles the LLM call out-of-band. BullMQ is the de-facto Node.js library for this in 2026, built on Redis and handling retries, stalls, and dead-letter queues without you wiring them yourself.
Why synchronous LLM calls break production
The default timeout on most reverse proxies (nginx, Vercel, Railway) is 30 to 60 seconds. A complex document summarization on Claude Fable 5 can easily run to 90 seconds. The proxy kills the connection before the model finishes.
Even without proxy timeouts, concurrency kills you. Ten users submitting summarizations at the same moment means ten in-flight HTTP connections, each blocked on a 60-second API call, each holding memory and a file descriptor. The event loop queues behind them. At $2 to $10 per million output tokens, those connections are expensive to keep alive doing nothing.
Moving the LLM call to a BullMQ worker fixes both. The API route becomes instant: accept the job, persist it to Redis, return { jobId }. The worker runs in a separate process, picks up the job, calls the LLM, and saves the result. The client polls or subscribes for completion.
Queue setup: the minimal working pattern
const { Queue, Worker, QueueEvents } = require('bullmq');
const connection = { host: process.env.REDIS_HOST || 'localhost', port: 6379 };
const llmQueue = new Queue('llm-jobs', { connection });
// API route: returns jobId immediately, does not wait for the LLM
async function submitJob(req, res) {
const job = await llmQueue.add(
'summarize',
{ prompt: req.body.prompt, userId: req.user.id },
{
attempts: 3,
backoff: { type: 'exponential', delay: 2_000 },
removeOnComplete: { age: 3_600 },
removeOnFail: false, // keep failed jobs for dead-letter inspection
}
);
res.json({ jobId: job.id });
}
removeOnFail: false is deliberate. Failed jobs stay in Redis until you review them. Without it, a job that exhausts all retries silently disappears before you can debug why it failed.
The worker: calling the LLM and reporting progress
const worker = new Worker(
'llm-jobs',
async (job) => {
await job.updateProgress({ stage: 'calling_llm', pct: 10 });
const res = await fetch('https://llmtest.io/v1/chat/completions', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${process.env.LLMTEST_API_KEY}`,
},
body: JSON.stringify({
model: 'anthropic/claude-fable-5',
messages: [{ role: 'user', content: job.data.prompt }],
}),
signal: AbortSignal.timeout(90_000),
});
if (!res.ok) throw new Error(`LLM API ${res.status}`);
const data = await res.json();
await job.updateProgress({ stage: 'done', pct: 100 });
return { output: data.choices[0].message.content };
},
{
connection,
concurrency: 4,
lockDuration: 120_000,
}
);
Two settings to tune deliberately. concurrency controls parallel LLM calls per worker process; start at 4 and adjust based on your rate limit headroom. lockDuration is how long BullMQ waits before declaring a job stalled. Set it longer than your slowest expected LLM call (120 seconds covers most frontier models). If it expires while the worker is mid-call, BullMQ marks the job stalled, retries it, and you pay for the same LLM call twice.
The AbortSignal.timeout(90_000) on the fetch is a separate hard cap: it kills the HTTP request if the LLM hasn't responded within 90 seconds, letting BullMQ record a clean failure rather than a hung process.
Live progress with Server-Sent Events
Polling /jobs/:id/status on an interval works but wastes requests. Server-Sent Events (SSE) pushes progress as it happens and is one EventSource call on the client.
app.get('/jobs/:jobId/events', (req, res) => {
res.set({
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
Connection: 'keep-alive',
});
res.flushHeaders();
const events = new QueueEvents('llm-jobs', { connection });
req.on('close', () => events.close());
events.on('progress', ({ jobId, data }) => {
if (jobId !== req.params.jobId) return;
res.write(`data: ${JSON.stringify(data)}\n\n`);
});
events.on('completed', ({ jobId, returnvalue }) => {
if (jobId !== req.params.jobId) return;
res.write(`data: ${JSON.stringify({ stage: 'done', result: returnvalue })}\n\n`);
res.end();
events.close();
});
events.on('failed', ({ jobId, failedReason }) => {
if (jobId !== req.params.jobId) return;
res.write(`data: ${JSON.stringify({ stage: 'error', reason: failedReason })}\n\n`);
res.end();
events.close();
});
});
On the client: const stream = new EventSource('/jobs/abc123/events'). No polling loop, no interval timer. Close the connection on stage: done or stage: error.
The req.on('close', ...) guard is necessary: if the user navigates away, the SSE connection closes and you need to clean up the QueueEvents listener, which holds a Redis connection.
Dead-letter recovery
BullMQ retries failed jobs up to attempts times using the configured backoff strategy. When all attempts are exhausted, the job stays in Redis under the failed state because of removeOnFail: false. The recovery pattern that works in production:
worker.on('failed', async (job, err) => {
if (job.attemptsMade < job.opts.attempts) return; // still retrying
// All retries exhausted, moving to the dead-letter queue
console.error(`[dead-letter] ${job.id} failed after ${job.attemptsMade} attempts: ${err.message}`);
await llmQueue.add('dead-letter', {
originalJobId: job.id,
originalData: job.data,
error: err.message,
failedAt: new Date().toISOString(),
});
});
A second worker listening for dead-letter jobs can fire a Slack alert, write to a dashboard, or queue a human review. The payload includes originalData, so you can inspect the prompt, fix whatever caused the failure (wrong model ID, oversized prompt, missing API key), and re-add it to the main queue manually.
The most common failure modes in production: the LLM returned a 429 (rate limit hit) before all retries cleared, the prompt exceeded the context window, or the model returned a malformed response and the parse threw. All three are recoverable if you have the original job data.
Where this fits in the stack
BullMQ sits between your API layer and the LLM network call. The request flow becomes:
POST /jobsaccepts the payload, returns{ jobId }in under 50ms- The worker calls the LLM, calls
job.updateProgress()at key stages GET /jobs/:id/eventspushes progress to the client via SSE- On completion, the worker returns the result; BullMQ stores it in Redis
Retry logic and provider switching belong inside the worker, not the API route. If the LLM provider is circuit-open, the worker throws, BullMQ retries with backoff, and the job queues behind the recovery window. The circuit breaker pattern handles per-provider failure tracking. The rate-limit handling guide covers 429 responses and per-minute budget enforcement, both of which belong inside the worker before throwing.
For detecting mid-stream stalls where the LLM starts responding and then stops, the stream stall detection guide covers watchdog timers inside the fetch. With BullMQ, a stall that aborts the fetch becomes a normal job failure; BullMQ retries it and records it in the dead-letter queue if retries are exhausted.
LLMTest's proxy handles automatic failover at the infrastructure level, so if one provider is down, the call routes to another. Combining it with BullMQ gives you provider redundancy (at the infra layer) plus per-job retry semantics and dead-letter recovery (at the application layer).
Track your LLM job costs and latency across providers at LLMTest.