How to detect stalled LLM streams: watchdog timers in Node.js

A streaming LLM call can fail in two distinct ways. The first is easy to catch: the model never responds, you hit your request timeout, and your error handler fires. The second is the one that bites you in production: the model starts sending tokens and then stops. Silently. No error. While your user watches the cursor blink.

Regular request timeouts don't save you here. If you set AbortSignal.timeout(30000) on your fetch call, the clock starts when you call fetch, not when the tokens stop. A response that opens after 2 seconds and stalls at the 25-second mark gives your user 23 uninterrupted seconds of a frozen UI before the global timeout fires.

Our streaming guide for Node.js, Next.js, and FastAPI covers implementing token-by-token responses from scratch. This post picks up where that one ends: what to do when the stream opens and then goes quiet.

Why standard timeouts miss stalls

The underlying issue is how HTTP streaming works. Once the server sends the first chunk and the connection is open, your HTTP client considers the request successful. It has no built-in concept of "how long between chunks is too long." The connection stays alive, no error fires, and your read loop just sits waiting.

Three distinct stall scenarios:

1. Initial latency stall: the model never sends a first token. A request timeout catches this.
2. Mid-stream freeze: tokens were arriving, then stopped. The connection is still open. Nothing fires.
3. Slow drip: tokens arrive once every 6–10 seconds, just fast enough to keep the connection alive, but the throughput is so degraded the UX is broken.

A timeout on the initial fetch call only covers scenario one.

The watchdog pattern

The fix is a per-chunk timer that resets every time a token arrives. If it fires before the next chunk, you abort:

async function streamWithWatchdog(prompt, { chunkTimeoutMs = 8000, totalTimeoutMs = 90000, apiKey }) {
  const controller = new AbortController();
  const buffer = [];
  let chunkTimer;

  const resetChunkTimer = () => {
    clearTimeout(chunkTimer);
    chunkTimer = setTimeout(() => {
      controller.abort(new Error(`No chunk in ${chunkTimeoutMs}ms`));
    }, chunkTimeoutMs);
  };

  const wallTimer = setTimeout(() => {
    controller.abort(new Error('Stream exceeded total time limit'));
  }, totalTimeoutMs);

  try {
    const res = await fetch('https://llmtest.io/v1/chat/completions', {
      method: 'POST',
      signal: controller.signal,
      headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${apiKey}` },
      body: JSON.stringify({
        model: 'anthropic/claude-sonnet-4-6',
        stream: true,
        messages: [{ role: 'user', content: prompt }],
      }),
    });

    if (!res.ok) throw new Error(`HTTP ${res.status}`);
    resetChunkTimer(); // start the watchdog after headers arrive

    const reader = res.body.getReader();
    const decoder = new TextDecoder();

    while (true) {
      const { done, value } = await reader.read();
      if (done) break;
      resetChunkTimer();

      const text = decoder.decode(value);
      for (const line of text.split('\n')) {
        if (!line.startsWith('data: ') || line === 'data: [DONE]') continue;
        try {
          const token = JSON.parse(line.slice(6)).choices?.[0]?.delta?.content;
          if (token) buffer.push(token);
        } catch { /* malformed chunk; skip */ }
      }
    }

    return { output: buffer.join(''), partial: false };
  } finally {
    clearTimeout(chunkTimer);
    clearTimeout(wallTimer);
  }
}

chunkTimeoutMs catches mid-stream stalls. totalTimeoutMs caps the wall clock regardless of chunk activity. Set chunkTimeoutMs based on the model's typical token cadence. Frontier models on normal load emit a chunk roughly every 100–400ms, so 8 seconds is generous enough to avoid false positives during brief backend pauses without leaving users waiting too long.

Recovering partial output

Aborting the stream solves the UX freeze. Throwing away everything the model sent is a worse experience than showing what arrived.

The buffer array accumulates tokens as they arrive. When the watchdog fires, check whether the partial content is worth returning:

  } catch (err) {
    if (controller.signal.aborted) {
      const partial = buffer.join('');
      if (partial.trim().length > 150) {
        return { output: partial, partial: true, reason: err.message };
      }
    }
    throw err; // re-throw if we don't have useful partial output
  }

The 150-character threshold prevents surfacing half-sentence fragments. Tune it per use case: for a conversational response, a partial answer with a trailing " [response cut short]" note is better than a perpetual spinner. For structured output (JSON, code, formatted tables), partial is usually unusable; let it fall through to a retry.

Detecting slow drips

A model sending one token every 7 seconds technically keeps the watchdog timer resetting while the UI looks frozen. This happens most often under heavy server load.

Track throughput alongside chunk presence:

let tokensSinceCheck = 0;
const throughputTimer = setInterval(() => {
  if (tokensSinceCheck < 2) {
    controller.abort(new Error('Stream throughput too low'));
  }
  tokensSinceCheck = 0;
}, 15000);

// Inside the token parse loop:
if (token) {
  buffer.push(token);
  tokensSinceCheck++;
}

// In finally:
clearInterval(throughputTimer);

This is overkill for most apps. Add it if you have evidence of slow-drip stalls from your logs rather than adding it preemptively.

Timeout values by model tier

Tuning chunkTimeoutMs is application-specific, but these approximate ranges come from what we observe routing requests through the LLMTest proxy:

These are medians under normal load. During peak hours (US weekday afternoons especially), inter-chunk gaps can be 3–4x higher. The upper end of each range accounts for that.

Wiring stall detection into a fallback chain

A stall detection abort is a retriable failure. The model wasn't returning garbage; it got stuck. Use the error message to distinguish a stall from a hard API error and route to a different provider:

async function streamWithFallback(prompt, apiKey) {
  const models = [
    'anthropic/claude-sonnet-4-6',
    'openai/gpt-4o-mini',
  ];

  for (const model of models) {
    try {
      return await streamWithWatchdog(prompt, { apiKey });
    } catch (err) {
      const isStall = err.message.includes('No chunk') || err.message.includes('throughput');
      if (isStall) {
        console.warn(`Stall on ${model}, trying fallback`);
        continue;
      }
      throw err;
    }
  }
  throw new Error('All providers stalled');
}

For the full provider fallback stack (backoff, circuit breaking, soft-failure detection), the LLM fallback chain guide covers each layer. Stall detection fits as the innermost layer, before the retry and provider-switch logic wraps around it.

Rate-limit spikes and stream stalls often co-occur during load events. The backoff and jitter patterns in production LLM rate-limit handling are the right complement on the retry side.

For most apps, the 30-line watchdog above is enough. Add the throughput check only when you have logs showing slow-drip failures. Add the fallback chain when availability matters more than the cost of a second model call.

Run your own stalls across providers using LLMTest. The proxy surfaces per-request latency and first-token time, so you can see whether a stall is happening on the model side or in the network path between you and the provider. For a complete picture of what to log across all LLM calls -- cost, tokens, and quality signals -- see the LLM observability guide.