How to manage LLM context windows in production: 3 Node.js patterns

A 60-message chat session accumulates roughly 40K tokens. Add a 4K system prompt and function-call schemas, and you are at 50K before the user asks anything complex. At 128K you hit the ceiling. The next call either fails or silently drops your oldest context, and the model answers questions about a conversation it no longer remembers.

The problem compounds in agent loops. Each tool call adds input tokens, output tokens, and the tool result. An eight-step loop can easily spend 30K tokens before producing a single line of output. If you have been tracking LLM agent cost in production, you already know context length is the main cost driver once you move past single-shot calls.

Three patterns address this. If you want a primer on the underlying mechanics first, the context window explainer covers how token limits work and what 128K actually fits.

Pattern 1: Sliding window

Keep only the most recent N tokens. When accumulated token count crosses your threshold, drop messages from the oldest end until you are back under budget. This is the simplest approach and has zero runtime cost beyond a token count.

function slidingWindow(messages, budgetTokens, countTokens) {
  const result = [];
  let total = 0;

  // Walk backward from newest to oldest
  for (let i = messages.length - 1; i >= 0; i--) {
    const t = countTokens(messages[i].content);
    if (total + t > budgetTokens && i > 0) break;
    result.unshift(messages[i]);
    total += t;
  }

  // Always preserve a leading system message if it got evicted
  if (messages[0]?.role === 'system' && result[0]?.role !== 'system') {
    result.unshift(messages[0]);
  }
  return result;
}

The countTokens function can be a rough proxy (1 token per 4 characters) for most cases. For exact counts, use tiktoken for OpenAI-compatible models or Anthropic's client.messages.count_tokens() method.

When to use it: short-memory assistants where only recent context matters. Customer support bots are a natural fit because the agent needs the last 8 turns to handle most follow-ups. It works poorly for document Q&A sessions where the source document lives in turn one, and for multi-step agent tasks where tool results from earlier steps are still referenced.

Tradeoff: zero overhead, visible information loss. If the dropped messages contained key references, the model will notice.

Pattern 2: Summarization compaction

When the window fills and you cannot afford to lose the earlier context, summarize it first. Make one cheap LLM call to compress 20 old messages into a single summary message, then carry the summary forward in place of the originals.

async function compactHistory(messages, threshold, client, countTokens) {
  const total = messages.reduce((sum, m) => sum + countTokens(m.content), 0);
  if (total <= threshold) return messages;

  const [systemMsg, ...rest] = messages[0]?.role === 'system'
    ? [messages[0], ...messages.slice(1)]
    : [null, ...messages];

  // Keep the 10 most recent turns verbatim; compress everything older
  const recent = rest.slice(-10);
  const toCompress = rest.slice(0, -10);
  if (toCompress.length === 0) return messages;

  const res = await client.chat.completions.create({
    model: 'openai/gpt-4o-mini',
    messages: [
      {
        role: 'system',
        content: 'Summarize this conversation history, preserving key facts, user preferences, and decisions made. Be concise.',
      },
      {
        role: 'user',
        content: toCompress.map(m => `${m.role}: ${m.content}`).join('\n'),
      },
    ],
    max_tokens: 500,
  });

  const summary = {
    role: 'assistant',
    content: `[Earlier context]\n${res.choices[0].message.content}`,
  };

  return [systemMsg, summary, ...recent].filter(Boolean);
}

When to trigger it: at 70% of your token budget, not 90%. Compacting at 90% leaves little headroom for the summary itself plus the next few turns.

Cost: one LLM call per compaction event. At gpt-4o-mini rates (roughly $0.15/M input, $0.60/M output), compressing 2,000 old tokens into a 400-token summary costs around $0.0006. Over 10 compactions in a long session, that is $0.006 per session.

Tradeoff: summaries lose precision. Specific dates, exact numbers, and quoted text survive compression poorly. If your app depends on any of those being retrievable verbatim, store the verbatim version in a lightweight key-value store alongside the summary, and inject it back as a structured facts block when you compact.

Pattern 3: Priority-based truncation

Not all messages are equally important. A user message in turn 2 that defines the project requirements matters more than a filler exchange in turn 15. Priority truncation assigns a weight to each message and drops the lowest-weight messages first when the context fills.

function priorityTruncate(messages, budget, countTokens) {
  function score(msg, idx, total) {
    const recency = idx / total;                               // 0=oldest, 1=newest
    const roleBonus = msg.role === 'system' ? 10 : msg.role === 'user' ? 2 : 1;
    const lengthBonus = msg.content.length > 800 ? 0.3 : 0;   // longer messages tend to be denser
    return recency * roleBonus + lengthBonus;
  }

  const scored = messages.map((msg, idx) => ({
    msg,
    tokens: countTokens(msg.content),
    s: score(msg, idx, messages.length),
  }));

  scored.sort((a, b) => a.s - b.s); // ascending: lowest score gets dropped first

  let remaining = scored.reduce((sum, x) => sum + x.tokens, 0);
  const dropped = new Set();

  for (const entry of scored) {
    if (remaining <= budget) break;
    dropped.add(entry.msg);
    remaining -= entry.tokens;
  }

  return messages.filter(m => !dropped.has(m));
}

When to use it: agent loops with tool calls. Tool results tend to be verbose but are usually only needed once. Giving them a low role bonus means they are the first to go when context tightens, while user instructions stay in place.

Extending the scoring: boost messages that contain explicit recall cues by scanning content for patterns. Messages with numbered lists, section headers, or clear decision statements can receive a fixed score bonus. Keep the scoring deterministic so truncation behavior is reproducible and debuggable.

Which pattern for your app

The combined approach works well for long agent sessions: run priority truncation first to drop low-value tool results, then trigger compaction when you are still above 70% of budget after the first pass. The two patterns compose because they target different parts of the message history.

For production deployments, route your compaction calls through the LLMTest proxy: you get automatic fallback if the summarizer model is unavailable, cost tracking per session, and latency logging so you can see exactly what context management is adding to your end-to-end response time.

Start tracking context costs across your LLM calls at llmtest.io.