A 100-message chat session accumulates roughly 65,000 tokens before you count system prompts, retrieved documents, or function schemas. A model with a 128K window hits its practical limit around turn 120 at 700 tokens per exchange. But real degradation starts much earlier: frontier models start hallucinating cross-references from earlier in the conversation at around 70% capacity in extended sessions.
The right fix is not a bigger context window. It is active management inside the chat route: count tokens before each call, trim old turns when context gets tight, and compress rather than delete when the trimmed context would lose something the model still needs.
This tutorial builds that into a single Express POST endpoint.
What the endpoint does
The /chat/:sessionId route:
- Looks up an in-memory session by ID (Redis is a drop-in swap for production)
- Appends the new user message
- Counts the accumulated token budget
- Applies a sliding window to trim old turns if context nears the threshold
- Falls back to summarization compaction when trimming alone would cut too deep
- Calls the LLM, stores the response, and returns it
For the underlying mechanics of why context fills the way it does, the context window explainer has the token-per-content math. For isolated implementations of each pattern, the production context management guide covers sliding window, compaction, and priority truncation as standalone functions. This tutorial wires them together into a real API route.
Setup
npm install express @anthropic-ai/sdk
The session management logic is provider-neutral. Swap client.messages.create for client.chat.completions.create and adjust the response shape to use OpenAI or any proxy endpoint.
Token counting
You need a token estimate before every call. Exact counts require a network round-trip: Anthropic's client.messages.countTokens, tiktoken for OpenAI-compatible endpoints. For threshold decisions inside a hot path, a 4-character-per-token approximation gets within 15% for English prose:
function countTokens(text) {
return Math.ceil(text.length / 4);
}
function sessionTokens(messages) {
return messages.reduce((sum, m) => sum + countTokens(m.content), 0);
}
Fifteen percent imprecision means your 70% threshold fires somewhere between 59% and 81% of actual capacity. That headroom is intentional: the summary you generate during compaction needs room to land, and the next user turn needs to fit after it.
The Express route skeleton
const express = require('express');
const Anthropic = require('@anthropic-ai/sdk');
const app = express();
app.use(express.json());
const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
const sessions = new Map();
const SYSTEM_PROMPT = 'You are a helpful assistant.';
const TOKEN_BUDGET = 60_000; // headroom in a 128K window
const COMPACT_AT = 0.70; // trigger context management at 70% capacity
const KEEP_RECENT = 8; // always keep this many turns verbatim
app.post('/chat/:sessionId', async (req, res) => {
const { sessionId } = req.params;
const { message } = req.body;
if (!message || typeof message !== 'string') {
return res.status(400).json({ error: 'message required' });
}
if (countTokens(message) > TOKEN_BUDGET * 0.5) {
return res.status(413).json({ error: 'Message too long. Split your input.' });
}
let history = sessions.get(sessionId) || [];
history.push({ role: 'user', content: message });
history = await manageContext(history);
const response = await client.messages.create({
model: 'claude-sonnet-4-6',
max_tokens: 2048,
system: SYSTEM_PROMPT,
messages: history,
});
const assistantText = response.content[0].text;
history.push({ role: 'assistant', content: assistantText });
sessions.set(sessionId, history);
res.json({ reply: assistantText, turns: history.length });
});
app.listen(3000, () => console.log('Chat API on :3000'));
The 413 guard on incoming messages matters. Someone pasting a 40K-token document into a chat input will otherwise blow past your budget before you even get to manage history.
Sliding window: the fast path
When accumulated history crosses 70% of the budget, drop the oldest turns first. Always preserve the most recent KEEP_RECENT turns verbatim: trimming from 45 minutes ago is acceptable, but trimming from 3 minutes ago breaks the conversation.
function slidingWindow(messages) {
const threshold = Math.floor(TOKEN_BUDGET * COMPACT_AT);
if (sessionTokens(messages) <= threshold) return messages;
const alwaysKeep = messages.slice(-KEEP_RECENT);
const candidates = messages.slice(0, -KEEP_RECENT);
const result = [];
let total = alwaysKeep.reduce((s, m) => s + countTokens(m.content), 0);
for (let i = candidates.length - 1; i >= 0; i--) {
const t = countTokens(candidates[i].content);
if (total + t > threshold) break;
result.unshift(candidates[i]);
total += t;
}
return [...result, ...alwaysKeep];
}
Sliding window costs nothing: no LLM call, sub-millisecond latency. It works well for customer support flows where the last 8 turns contain everything needed to resolve the current question.
It breaks down for research sessions or multi-step planning tasks. If the user defined constraints in turn 2 and you drop turn 2, the next response will contradict them. Compaction handles that.
Summarization compaction: the fallback
Instead of silently dropping turns, compress the oldest messages into a summary. One cheap LLM call replaces 20 verbose messages with a 300-token brief that preserves key facts, decisions, and preferences.
async function compact(messages) {
if (messages.length <= KEEP_RECENT) return messages;
const recent = messages.slice(-KEEP_RECENT);
const toCompress = messages.slice(0, -KEEP_RECENT);
const summaryResponse = await client.messages.create({
model: 'claude-haiku-4-5',
max_tokens: 512,
system: 'Summarize this conversation, preserving key facts, decisions, and user preferences. Be concise.',
messages: [
{
role: 'user',
content: toCompress.map(m => `${m.role}: ${m.content}`).join('\n'),
},
],
});
const summary = {
role: 'assistant',
content: '[Earlier conversation]\n' + summaryResponse.content[0].text,
};
return [summary, ...recent];
}
claude-haiku-4-5 handles a 2,000-token compression for roughly $0.0008. Ten compaction events across a 60-minute session adds $0.008 per session. For comparison, an 8-step agent loop accumulates context costs around $0.40 just from repeated context re-sends on a frontier model; compaction overhead is negligible.
Wiring both patterns together
The manageContext function tries the sliding window first. If history is still over the threshold after trimming (which happens when even the most recent KEEP_RECENT turns are large), it compacts:
async function manageContext(messages) {
const threshold = Math.floor(TOKEN_BUDGET * COMPACT_AT);
const windowed = slidingWindow(messages);
if (sessionTokens(windowed) > threshold) {
return compact(windowed);
}
return windowed;
}
The ordering matters. Compacting after sliding window means you summarize only the oldest surviving turns, not the entire history. The most recent turns pass through both steps verbatim.
Edge cases you need to handle
System prompt too large. Add a startup assertion so you catch this at boot, not mid-conversation:
if (countTokens(SYSTEM_PROMPT) > TOKEN_BUDGET * 0.3) {
throw new Error('System prompt uses more than 30% of token budget.');
}
Compaction on a fresh session. If KEEP_RECENT exceeds the total message count, compact returns the history unchanged. The guard messages.length <= KEEP_RECENT handles this without error.
Sessions that never end. An in-memory Map grows unbounded. Add a timestamp to each session and run a cleanup interval that deletes sessions idle for more than 4 hours.
What to track in production
Every context management decision is invisible by default. You won't know whether compaction fires once per session or 20 times, what it costs, or whether the sliding window is cutting context the model still needed.
Add counters per session: trimCount, compactCount, compactCostUsd. Log them when the session ends. Route your LLM calls through the LLMTest proxy to get per-call token counts and costs without instrumenting every client.messages.create call manually. The proxy captures usage metadata on every call and surfaces it in the dashboard by session ID.
Track per-session LLM spend as you scale at llmtest.io.