A blocking LLM call returns in 3-15 seconds. The user stares at a spinner. A streamed call starts delivering tokens in 200-500ms and feels instant even when the full response takes just as long. The gap is perception, but perception drives retention.
OpenAI-compatible APIs return a text/event-stream response when you set stream: true. Each line is prefixed with data: , terminated with \n\n, and ends with the sentinel data: [DONE]\n\n. The three stacks below all consume that format, but the connection plumbing differs enough that each has a distinct production gotcha.
Pattern 1: Plain Node.js with native fetch
No extra packages required beyond Node 18. Read the response body as a ReadableStream, decode with TextDecoder, and parse line by line.
async function streamCompletion(prompt) {
const response = await fetch('https://api.openai.com/v1/chat/completions', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
},
body: JSON.stringify({
model: 'gpt-4o-mini',
stream: true,
messages: [{ role: 'user', content: prompt }],
}),
});
if (!response.ok) {
throw new Error(`HTTP ${response.status}: ${await response.text()}`);
}
const reader = response.body.getReader();
const decoder = new TextDecoder();
while (true) {
const { done, value } = await reader.read();
if (done) break;
const chunk = decoder.decode(value, { stream: true });
for (const line of chunk.split('\n')) {
if (!line.startsWith('data: ')) continue;
const payload = line.slice(6);
if (payload === '[DONE]') return;
try {
const parsed = JSON.parse(payload);
const text = parsed.choices?.[0]?.delta?.content;
if (text) process.stdout.write(text);
} catch {
// providers occasionally send heartbeat or comment lines
}
}
}
}
The gotcha: TextDecoder initialized with { stream: true } buffers incomplete multi-byte sequences across chunk boundaries. Drop that option and you get corrupted output whenever a Chinese character or emoji lands at the end of a chunk. The flag costs nothing and prevents a class of bugs that only appear at scale.
Pattern 2: Next.js App Router route handler
App Router route handlers can pipe the upstream SSE stream directly to the client without buffering on your server. Memory overhead stays near zero regardless of response length.
// app/api/chat/route.js
export async function POST(request) {
const { prompt } = await request.json();
const upstream = await fetch('https://api.openai.com/v1/chat/completions', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
},
body: JSON.stringify({
model: 'gpt-4o-mini',
stream: true,
messages: [{ role: 'user', content: prompt }],
}),
signal: request.signal,
});
if (!upstream.ok) {
return new Response(`Upstream error ${upstream.status}`, { status: 502 });
}
return new Response(upstream.body, {
headers: {
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
Connection: 'keep-alive',
},
});
}
On the client side, read the response body with the same getReader() + SSE parsing from Pattern 1. The Vercel AI SDK's useChat hook handles all of this if you want a ready-made implementation.
The gotcha: signal: request.signal is not optional. Without it, when a user navigates away or refreshes mid-stream, Next.js closes the client connection but your server keeps the upstream fetch open, spending tokens and money until the full response completes. Wiring the signal propagates the cancellation in both directions.
Pattern 3: FastAPI with StreamingResponse
FastAPI's StreamingResponse wraps an async generator that yields SSE-formatted strings. The framework handles chunked transfer encoding.
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from openai import AsyncOpenAI
app = FastAPI()
client = AsyncOpenAI()
async def stream_tokens(prompt: str):
stream = await client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
stream=True,
)
async for chunk in stream:
content = chunk.choices[0].delta.content
if content:
yield f"data: {content}\n\n"
yield "data: [DONE]\n\n"
@app.post("/chat")
async def chat_endpoint(prompt: str):
return StreamingResponse(
stream_tokens(prompt),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"X-Accel-Buffering": "no",
},
)
The gotcha: The X-Accel-Buffering: no header disables nginx's response buffering for this endpoint. Without it, nginx collects the complete response before forwarding to the client, defeating streaming entirely. Cloudflare's equivalent is setting Transfer-Encoding: chunked and verifying that "Stream responses" is enabled in your zone settings. If your streams feel instant in local dev but blocked on production, this header is almost certainly why.
The mid-stream error you are probably not catching
Both OpenAI and Anthropic occasionally return HTTP 200 but embed a JSON error object inside the stream instead of a content delta. This happens during model overload, context limit hits, and content filter triggers. A naive SSE parser hits a JSON parse error, swallows it silently (the catch block above), and the stream ends with no message in your logs.
The fix is one extra check after parsing each chunk:
const parsed = JSON.parse(payload);
if (parsed.error) {
console.error('mid-stream provider error:', parsed.error.message);
return;
}
const text = parsed.choices?.[0]?.delta?.content;
Also watch for choices[0].finish_reason === 'content_filter' when building user-facing products. The model stopped; returning silence is worse than returning a message.
For the retry layer around the initial fetch, the same patterns from handling LLM rate limits in production apply here. One difference: you cannot retry mid-stream. The retry wrapper goes around the fetch() call, before the reader loop starts, so a second attempt begins from the first token. The related failure is a stream that opens but goes silent partway through; stall detection with per-chunk watchdog timers covers how to catch and recover from that case.
If you are adding fallback routing on top of streaming, the OpenRouter fallback setup in Node.js shows how to keep the models array approach compatible with a streaming response body.
LLMTest's proxy forwards streamed responses transparently. If you route through it, the Content-Type: text/event-stream response from whichever upstream provider responds first passes straight to your client. No code changes on your end when the active model changes.