Articles tagged "infra"

How to add LLM circuit breakers in Node.js in 2026

Per-provider LLM circuit breakers stop cascade failures early. Closed, open, half-open state machine with auto-reset: working Node.js code.

Jul 24, 2026 · 7 min read infrareliabilitynodejs

How to add LLM fallback in Next.js in 2026: two providers, 25 lines

Wire Anthropic and OpenAI into one Next.js API route: primary call, automatic provider switch on 429 or 5xx, and the real latency cost of switching mid-request.

Jul 22, 2026 · 5 min read tutorialnextjsinfra

How to add LLM observability in 2026: cost, latency, quality

Four signals every LLM call should log: tokens, cost, latency, and a quality proxy. An async Node.js middleware that adds under 2ms of overhead.

Jul 17, 2026 · 5 min read infraobservabilitymonitoring

How to detect stalled LLM streams: watchdog timers in Node.js

Three production patterns for detecting stalled LLM streams in Node.js: chunk watchdogs, wall-clock gates, and partial output buffering, with working code.

Jul 8, 2026 · 5 min read infrastreamingnodejs

Streaming LLM responses in 2026: Next.js, FastAPI, Node

Three production patterns for streaming LLM responses in Node.js, Next.js, and FastAPI, with the mid-stream error catch most tutorials skip.

Jun 24, 2026 · 4 min read tutorialstreamingnodejs

LiteLLM routing in 2026: 4 patterns that run in production

LiteLLM has multiple routing strategies. These 4 matter in production: weighted, fallback, semantic, and budget-capped, each with production-ready YAML.

Jun 19, 2026 · 5 min read tutoriallitellmrouting

Prompt caching explained: Anthropic, OpenAI, and Gemini in 2026

Prompt caching cuts LLM API costs up to 90%, but Anthropic, OpenAI, and Gemini implement it differently. Here's how each vendor's billing actually works.

Jun 15, 2026 · 8 min read glosprompt-cachingcost

How to route LLM prompts in 2026: cheap first, escalate on fail

Route each prompt to the cheapest model that handles it well. When quality falls short, escalate silently. Here's the pattern with working Node.js code.

Jun 14, 2026 · 4 min read infraroutingcost-optimization

How to set up OpenRouter fallback in Node.js in 2026

Add OpenRouter model fallbacks to a Node.js app: setup, the models array, response.model tracking, and four pitfalls that catch you on week two.

Jun 1, 2026 · 5 min read tutorialopenrouterinfra

Semantic caching for LLMs: 3 approaches and where each breaks

Semantic caching reduces LLM API spend by 20-70% in production. Here's how embedding-based, prompt-hash, and hybrid caching each break in practice.

May 25, 2026 · 6 min read infracachingcost

How to handle LLM rate limits: 4 production-tested patterns

Four production patterns for LLM rate limits: jitter, token pre-checks, circuit breakers, and provider failover. Backoff alone won't save you in 2026.

May 20, 2026 · 6 min read infrarate-limitsreliability

Prompt caching breaks even at 1.3 requests. Here's the math.

Prompt caching cuts LLM costs 90% on Anthropic and 50% on OpenAI, but only when your workload fits. Here's the exact break-even math per provider.

Apr 27, 2026 · 5 min read infracostprompt-caching

Build an LLM fallback chain in 10 minutes

One model going down shouldn't take your AI feature with it. Here's how to build a fallback chain using LiteLLM, OpenRouter, and LLMTest.

Apr 21, 2026 · 5 min read infrafallbackreliability