How to add per-request token budget controls in Node.js 2026

An uncapped LLM route is a billing surprise waiting to happen. Claude Haiku 4.5 costs $5 per million output tokens. That sounds cheap until one user sends a prompt that triggers 10,000 output tokens: a single call costs $0.05. At 1,000 such calls per day you are spending $50 on output alone from your budget model, before any compounding in multi-turn sessions.

The fix is three steps: count input tokens before calling, set max_tokens per route based on what that route actually needs, and handle streaming truncation cleanly so hitting the ceiling does not surface as an error to the user.

For the background on why long sessions inflate costs so much, LLM conversation costs in 2026 has the per-turn math. For capping total spend per user across all their calls, per-user LLM cost guardrails in Node.js covers the Redis-backed daily-budget pattern. This post sits between them: it controls individual API call output at the route level.

Why each route needs its own ceiling

A single global max_tokens value is a compromise that fits no route well. A chat reply truncated at 300 tokens leaves users mid-sentence. A classify route that allows 4,096 tokens on a simple label wastes $0.02 per call that needed 40.

Right-sizing by route makes each call predictable:

These are starting points, not gospel. Watch your 95th-percentile output length in production and adjust. The goal is to prevent the rare 15,000-token response from an otherwise predictable route, not to squeeze every call.

Count tokens before you call

Counting tokens before the call catches a different problem: an input so large the model cannot produce meaningful output inside your max_tokens ceiling. If a user pastes a 120,000-token document into a 200,000-token context model and you set max_tokens: 1024, the math still works, but one with a 128,000-token context window will fail outright.

Anthropic exposes a dedicated token-counting endpoint that uses the same tokeniser as the model but runs no inference:

import Anthropic from '@anthropic-ai/sdk';

const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });

async function countInputTokens(messages, systemPrompt) {
  const result = await anthropic.messages.countTokens({
    model: 'claude-haiku-4-5-20251001',
    system: systemPrompt,
    messages,
  });
  return result.input_tokens;
}

For OpenAI models, tiktoken counts tokens client-side with zero network cost:

import { encoding_for_model } from 'tiktoken';

function countTokensForOpenAI(text, model = 'gpt-4o') {
  const enc = encoding_for_model(model);
  const count = enc.encode(text).length;
  enc.free();
  return count;
}

Either way: if inputTokens + maxTokens exceeds the model's context window, reject the request with a 400 before paying for the call.

Set max_tokens per route

A middleware factory reads a route-level budget from a configuration object and attaches it to the request:

const ROUTE_BUDGETS = {
  '/api/chat': 512,
  '/api/summarize': 1024,
  '/api/classify': 128,
  '/api/code': 2048,
};

function tokenBudget(route) {
  const max = ROUTE_BUDGETS[route] ?? 512;
  return (req, _res, next) => {
    req.maxTokens = max;
    next();
  };
}

Register it before each handler:

import express from 'express';

const app = express();
app.use(express.json());

app.post('/api/chat', tokenBudget('/api/chat'), chatHandler);
app.post('/api/summarize', tokenBudget('/api/summarize'), summarizeHandler);
app.post('/api/classify', tokenBudget('/api/classify'), classifyHandler);
app.post('/api/code', tokenBudget('/api/code'), codeHandler);

Inside each handler, req.maxTokens is already set. The configuration object is the only place you change when tuning limits; handlers stay untouched:

async function chatHandler(req, res) {
  const { messages, system } = req.body;
  const response = await anthropic.messages.create({
    model: 'claude-haiku-4-5-20251001',
    max_tokens: req.maxTokens,
    system,
    messages,
  });
  res.json({ text: response.content[0].text, usage: response.usage });
}

Handle streaming truncation gracefully

When a model hits max_tokens mid-stream, it stops generating. Anthropic signals this through a message_delta event with stop_reason: "max_tokens" instead of "end_turn". OpenAI uses finish_reason: "length" in the final chunk.

Without checking this, truncation arrives at the client as an abrupt silence: no error, no explanation, and a potentially incomplete JSON blob if you are accumulating tokens on the other side.

async function streamWithBudget(req, res) {
  const { messages, system } = req.body;
  const maxTokens = req.maxTokens;

  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Connection', 'keep-alive');

  let truncated = false;

  try {
    const stream = anthropic.messages.stream({
      model: 'claude-haiku-4-5-20251001',
      max_tokens: maxTokens,
      system,
      messages,
    });

    for await (const event of stream) {
      if (event.type === 'content_block_delta' && event.delta.type === 'text_delta') {
        res.write(`data: ${JSON.stringify({ text: event.delta.text })}\n\n`);
      }
      if (event.type === 'message_delta' && event.delta.stop_reason === 'max_tokens') {
        truncated = true;
      }
    }
  } catch (err) {
    console.error('[token-budget] Stream error:', err.message);
    res.write(`data: ${JSON.stringify({ error: true })}\n\n`);
    return res.end();
  }

  if (truncated) {
    console.warn('[token-budget] Truncated at', maxTokens, 'tokens on', req.path);
  }

  res.write(`data: ${JSON.stringify({ done: true, truncated })}\n\n`);
  res.end();
}

The truncated flag reaches the client so the UI can decide: show a "response trimmed" notice, trigger a follow-up call, or silently accept the partial answer. The server returns a clean close either way.

Putting it together

Combining all three: count before calling, gate on context overflow, then stream with the route budget:

const MODEL = 'claude-haiku-4-5-20251001';
const CONTEXT_LIMIT = parseInt(process.env.MODEL_CONTEXT_LIMIT || '190000');

app.post('/api/chat', tokenBudget('/api/chat'), async (req, res) => {
  const { messages, system } = req.body;

  const inputCount = await countInputTokens(messages, system);
  if (inputCount + req.maxTokens > CONTEXT_LIMIT) {
    return res.status(400).json({
      error: 'Input too large for this route budget',
      inputTokens: inputCount,
      maxOutput: req.maxTokens,
    });
  }

  await streamWithBudget(req, res);
});

Set MODEL_CONTEXT_LIMIT in your environment to match your actual model's context window. Check the provider's pricing page rather than hardcoding it, since limits vary across model versions and change over time.

For per-call cost tracking and latency metrics alongside token counts, route these calls through the LLMTest proxy rather than hitting the provider directly. You get a per-route cost breakdown in the dashboard without adding another middleware layer yourself.

The session-level token budget in the multi-turn chat API tutorial builds on this pattern: it accumulates the per-request counts across conversation turns and enforces a hard session cap.