Most chat API implementations fail in the same two ways. The first is obvious: they don't manage the context window, so long sessions eventually hit the model's token limit and throw an error. The second is invisible until your invoice arrives: every turn re-sends the entire conversation history, so a 10-turn session sends 34,500 input tokens to deliver 2,000 tokens of new content.
This tutorial builds a production-ready multi-turn chat API in Node.js that solves both. It stores message history server-side, compacts context when the window fills, and enforces a hard token budget per session so a single long conversation cannot blow your monthly bill.
The session object
Every conversation needs two things tracked: the message array and the cumulative token spend for that session.
const sessions = new Map();
function getSession(id) {
if (!sessions.has(id)) {
sessions.set(id, { messages: [], tokensUsed: 0, createdAt: Date.now() });
}
return sessions.get(id);
}
The tokensUsed counter accumulates across every call in the session, including the tokens spent during compaction. A session that started with 5,000 tokens, compacted, and then accumulated another 30,000 still shows 35,000 total. This is what the per-session budget enforces.
Add a cleanup interval so the in-memory map does not grow indefinitely:
setInterval(() => {
const cutoff = Date.now() - 4 * 60 * 60 * 1000;
for (const [id, s] of sessions) {
if ((s.lastActiveAt || s.createdAt) < cutoff) sessions.delete(id);
}
}, 30 * 60 * 1000);
Token counting
You need a fast estimate before every call. Exact counts require an API round-trip. The 4-characters-per-token approximation stays within 15% for English prose, which is sufficient for threshold decisions:
function countTokens(text) {
return Math.ceil(text.length / 4);
}
function historyTokens(messages) {
return messages.reduce((sum, m) => sum + countTokens(m.content), 0);
}
If you need exact counts for billing, use the Anthropic SDK's client.messages.countTokens or tiktoken for OpenAI-compatible endpoints. The cost is typically 1-2ms per call and is worth it when the per-session budget is tight.
Context management
Two strategies, tried in order. Sliding window fires first because it costs nothing:
const WINDOW_CAP = 60_000;
const COMPACT_AT = 0.70;
const KEEP_RECENT = 8;
function slidingWindow(messages) {
const threshold = Math.floor(WINDOW_CAP * COMPACT_AT);
if (historyTokens(messages) <= threshold) return messages;
const keep = messages.slice(-KEEP_RECENT);
const candidates = messages.slice(0, -KEEP_RECENT);
let total = historyTokens(keep);
const included = [];
for (let i = candidates.length - 1; i >= 0; i--) {
const t = countTokens(candidates[i].content);
if (total + t > threshold) break;
included.unshift(candidates[i]);
total += t;
}
return [...included, ...keep];
}
Sliding window works well for support sessions where the most recent 8 turns hold everything the model needs to close the ticket. It breaks down when the user set constraints in turn 2 that a window would silently drop. Compaction handles that case by summarizing the oldest turns rather than deleting them:
const API_URL = process.env.API_BASE_URL || 'https://api.anthropic.com/v1';
const API_KEY = process.env.API_KEY || process.env.ANTHROPIC_API_KEY;
async function compact(messages) {
if (messages.length <= KEEP_RECENT) return messages;
const recent = messages.slice(-KEEP_RECENT);
const toCompress = messages.slice(0, -KEEP_RECENT);
const resp = await fetch(`${API_URL}/messages`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': `Bearer ${API_KEY}`,
'anthropic-version': '2023-06-01',
},
body: JSON.stringify({
model: 'claude-haiku-4-5',
max_tokens: 400,
system: 'Summarize this conversation. Preserve key facts, decisions, and user preferences. Be concise.',
messages: [{ role: 'user', content: toCompress.map(m => `${m.role}: ${m.content}`).join('\n') }],
}),
});
if (!resp.ok) return messages; // fail open: keep history unchanged
const result = await resp.json();
const summary = result.content?.[0]?.text || '';
return [
{ role: 'assistant', content: '[Earlier conversation]\n' + summary },
...recent,
];
}
async function manageContext(messages) {
const threshold = Math.floor(WINDOW_CAP * COMPACT_AT);
const windowed = slidingWindow(messages);
if (historyTokens(windowed) <= threshold) return windowed;
return compact(windowed);
}
claude-haiku-4-5 handles a 2,000-token compression for roughly $0.0008. Ten compaction events in a 60-minute research session adds $0.008 per session to your bill.
The chat endpoint
const express = require('express');
const app = express();
app.use(express.json());
const MODEL = process.env.MODEL || 'claude-haiku-4-5';
const SYSTEM = 'You are a helpful assistant.';
const SESSION_CAP = 100_000; // hard per-session token budget
app.post('/chat/:sessionId', async (req, res) => {
const { sessionId } = req.params;
const { message } = req.body;
if (!message || typeof message !== 'string') {
return res.status(400).json({ error: 'message required' });
}
const session = getSession(sessionId);
if (session.tokensUsed >= SESSION_CAP) {
return res.status(402).json({
error: 'Session token budget exhausted. Start a new conversation.',
tokensUsed: session.tokensUsed,
cap: SESSION_CAP,
});
}
session.messages.push({ role: 'user', content: message });
session.messages = await manageContext(session.messages);
const inputTokens = historyTokens(session.messages) + countTokens(SYSTEM);
const response = await fetch(`${API_URL}/messages`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': `Bearer ${API_KEY}`,
'anthropic-version': '2023-06-01',
},
body: JSON.stringify({
model: MODEL,
max_tokens: 2048,
system: SYSTEM,
messages: session.messages,
}),
});
if (!response.ok) {
const err = await response.json().catch(() => ({}));
return res.status(502).json({ error: 'Upstream error', detail: err });
}
const data = await response.json();
const outputText = data.content?.[0]?.text || '';
const outputTokens = countTokens(outputText);
session.messages.push({ role: 'assistant', content: outputText });
session.tokensUsed += inputTokens + outputTokens;
sessions.set(sessionId, { ...session, lastActiveAt: Date.now() });
res.json({
reply: outputText,
turns: session.messages.length,
sessionTokensUsed: session.tokensUsed,
sessionTokensRemaining: Math.max(0, SESSION_CAP - session.tokensUsed),
});
});
app.listen(3000, () => console.log('Chat API on :3000'));
The 402 on budget exhaustion is intentional: it tells the client exactly why the request failed and gives the current count so the frontend can surface a "start a new conversation" prompt rather than a generic error.
The API_URL accepts any OpenAI-compatible endpoint. Point it at https://llmtest.io/v1 and calls route through the LLMTest proxy for unified per-call logging, model switching, and cost tracking without instrumenting every fetch call.
Per-session vs per-user budgets
The tokensUsed counter on the session object tracks cumulative spend for that conversation, across all compaction cycles. Compaction replaces old turns with a summary but does not reset the counter: every token ever sent counts toward the session budget.
This is distinct from the per-user daily cap pattern in the per-user guardrails middleware. A third layer controls individual API call output: per-request token budget controls set route-level max_tokens ceilings and catch streaming truncation before it surfaces as a user-visible error.
The three layers address different problems:
- Per-session cap: prevents one long conversation from consuming an unbounded number of tokens regardless of how many turns it lasts
- Per-user daily cap: prevents one user from starting many separate conversations per day
Both are needed in production. A user who hits the session cap and immediately starts a new session would bypass a session-only guard. A user who sends one enormous multi-hour session per day would bypass a per-user daily cap alone.
Production notes
Session storage. The in-memory Map works for a single Node.js process. For any deployment with multiple workers or restarts, swap it for Redis: store the serialized session object with a 4-hour TTL. The Redis connection patterns from the LLM caching guide apply directly.
Multi-provider. Setting API_BASE_URL and MODEL via environment variables means you can switch from Anthropic to OpenAI or any compatible endpoint without touching the application code. The Anthropic-specific anthropic-version header is ignored by OpenAI-compatible endpoints, so it is safe to include regardless of which provider you route to.
Observability. The sessionTokensUsed field on every response gives clients real-time budget feedback. For deeper tracking, such as which routes trigger the most compaction or which sessions exhaust their budget fastest, route through the LLMTest proxy to get per-session breakdowns in the dashboard.
Monitor per-session token spend as you scale at llmtest.io.