How to retry LLM API calls in Node.js in 2026: three-layer pattern
Client backoff with jitter, proxy fallback, and dead-letter queue: the three Node.js layers that stop cascading LLM failures in production.
Tag · nodejs
Client backoff with jitter, proxy fallback, and dead-letter queue: the three Node.js layers that stop cascading LLM failures in production.
An in-process LLM request queue for Node.js: concurrency cap, per-user fairness, and backpressure under 60 lines. No Redis or BullMQ needed in 2026.
Redis exact-match caching cuts LLM API spend at scale. Working Node.js middleware, 30-minute TTL trade-off, and when semantic search is worth the overhead.
Add per-user LLM budget caps with Redis: soft warnings at 80%, hard stops at 100%, and automatic model downgrade. Full Node.js middleware under 60 lines.
Build a production RAG API in Node.js using pgvector: text ingestion, chunked embeddings, vector search, and source-cited answers in under 80 lines.
Add token counting, sliding window, and summarization compaction to a Node.js chat route to keep conversations coherent without blowing the context budget.
BullMQ turns long-running LLM API calls into background jobs with live progress events, stall detection, and dead-letter recovery in Node.js in 2026.
Sliding window, summarization compaction, priority truncation: three Node.js patterns for managing context in long-running LLM conversations.
Per-provider LLM circuit breakers stop cascade failures early. Closed, open, half-open state machine with auto-reset: working Node.js code.
Four signals every LLM call should log: tokens, cost, latency, and a quality proxy. An async Node.js middleware that adds under 2ms of overhead.
Catch LLM prompt regressions before users do: a 30-line Node.js eval script that runs golden-set tests on every PR and fails the build when quality drops.
Three production patterns for detecting stalled LLM streams in Node.js: chunk watchdogs, wall-clock gates, and partial output buffering, with working code.
Three production guards that stop runaway LLM agent loops in Node.js: max turns, cost budgets, and return value validation with working code.
Three production patterns for streaming LLM responses in Node.js, Next.js, and FastAPI, with the mid-stream error catch most tutorials skip.
Route each prompt to the cheapest model that handles it well. When quality falls short, escalate silently. Here's the pattern with working Node.js code.
Add OpenRouter model fallbacks to a Node.js app: setup, the models array, response.model tracking, and four pitfalls that catch you on week two.