How to add LLM observability in 2026: cost, latency, quality
Four signals every LLM call should log: tokens, cost, latency, and a quality proxy. An async Node.js middleware that adds under 2ms of overhead.
Tag · nodejs
Four signals every LLM call should log: tokens, cost, latency, and a quality proxy. An async Node.js middleware that adds under 2ms of overhead.
Catch LLM prompt regressions before users do: a 30-line Node.js eval script that runs golden-set tests on every PR and fails the build when quality drops.
Three production patterns for detecting stalled LLM streams in Node.js: chunk watchdogs, wall-clock gates, and partial output buffering, with working code.
Three production guards that stop runaway LLM agent loops in Node.js: max turns, cost budgets, and return value validation with working code.
Three production patterns for streaming LLM responses in Node.js, Next.js, and FastAPI, with the mid-stream error catch most tutorials skip.
Route each prompt to the cheapest model that handles it well. When quality falls short, escalate silently. Here's the pattern with working Node.js code.
Add OpenRouter model fallbacks to a Node.js app: setup, the models array, response.model tracking, and four pitfalls that catch you on week two.