A customer support chatbot running 1,000 daily sessions on Claude Haiku 4.5 costs $49.50 a day before any optimization. Add no new features, keep the same model, and that number does not change. But most teams building chat products underestimate their bill by 5–10x before they ever hit production, because the math on stateless APIs is not obvious until you look at it directly.
Why every turn is more expensive than the last
LLM APIs have no server-side session object. Every call sends everything: the system prompt, the complete conversation history, and the new user message. The model reads it all from scratch; your bill reflects every token sent, including the ones you already paid for on turn two.
A 10-turn support session with a 1,000-token system prompt and 200-token user messages accumulates input like this:
| Turn | Input tokens this call | Cumulative input sent |
|---|---|---|
| 1 | 1,200 | 1,200 |
| 2 | 1,700 | 2,900 |
| 5 | 3,200 | 11,000 |
| 10 | 5,700 | 34,500 |
By turn 10, a single 10-turn session has sent 34,500 input tokens to deliver 2,000 tokens of new user content. The remaining 32,500 tokens are context that was already billed on earlier calls.
The 1,000 sessions/day worked example
Three models, same workload: 1,000 sessions/day, 10 turns average, 300 tokens per AI response.
| Model | Input | Output | Per session | Per day | Per month |
|---|---|---|---|---|---|
| GPT-4o-mini | $0.15/M | $0.60/M | $0.007 | $7.00 | $210 |
| Claude Haiku 4.5 | $1/M | $5/M | $0.050 | $49.50 | $1,485 |
| Claude Sonnet 5 | $2/M | $10/M | $0.099 | $99.00 | $2,970 |
Prices as of August 2026; verify at Anthropic's pricing page and OpenAI's API pricing before building your cost model.
The naïve estimate (average tokens per turn times turns times price) gives roughly 5,000 input tokens per session. The real number is 34,500. That is a 7x undercount, and it compounds further as sessions get longer.
Long sessions: where compaction earns its keep
A 50-turn workflow (deep support thread, coding agent, research assistant) magnifies the problem:
- Without compaction: 672,500 input + 15,000 output tokens per session
- On Haiku 4.5: $0.75/session → $747/day → $22,400/month
A 10-turn sliding window cuts what each call sends after turn 11: instead of full history, you send the last 10 turns plus the system prompt.
- With 10-turn window from turn 11: 282,500 input + 15,000 output tokens per session
- On Haiku 4.5: $0.36/session → $357/day → $10,700/month
That is a 52% cost reduction by changing one parameter in your message handler, with no model upgrade, no architecture change, and no degradation in practice for most support flows (which only need recent context).
For the Node.js implementation, including both sliding window and summarization compaction (which preserves early context by compressing it first), see context window management patterns for production.
System prompt caching: the easiest win
Your system prompt is probably 500–2,000 tokens. It repeats on every call at full price. Anthropic's prompt caching cuts cached-read cost by roughly 90%. For 1,000 sessions at 10 turns each with a 1,000-token system prompt:
- Without caching: 10,000 system-prompt tokens per session × 1,000 sessions = 10M tokens/day at $1/M = $10/day
- With caching at ~85% hit rate: approximately $1.50–$2.00/day
One parameter change in your API call. No architecture work. The prompt caching break-even math shows you recover the cache-write cost after one or two cache hits. At 1,000 sessions a day, you pay for that write in under a second of traffic.
What to measure first
Most teams find that two or three conversation flows drive 80% of the context cost. Before adding compaction everywhere, add token logging per session:
const response = await client.chat.completions.create({ /* ... */ });
console.log({
sessionId,
turn: history.length,
inputTokens: response.usage.prompt_tokens,
outputTokens: response.usage.completion_tokens,
});
Sum inputTokens by sessionId across a day. Look for sessions where the total exceeds 20,000 tokens: those are the compaction targets. Most chat products have a small fraction of unusually long sessions that account for a disproportionate share of the bill.
The three-step playbook
Measure session-level token totals. Add the logging above to every LLM call and aggregate by session for 24 hours. The sessions above 20,000 cumulative input tokens are where compaction returns the most.
Set a compaction threshold. 4,000–6,000 input tokens after subtracting the system prompt is a reasonable trigger for support-style chatbots. Above that, apply a sliding window or summarization pass before the next call.
Cache your system prompt. If you are on Anthropic or OpenAI, tag your static system prompt block for caching. It reduces system-prompt cost 7–9x with no change to your application logic.
The LLMTest billing dashboard reports token usage broken down by model, route, and flow, which makes it straightforward to spot which conversation types are accumulating context unusually fast.
For the underlying mechanics of why token limits work the way they do and what a given context window actually fits, the context window explainer covers the full breakdown. Chat session costs are predictable once you measure them. The gap between what teams expect and what they actually pay is almost always explained by one thing: re-sending history they forgot they were sending.