If your Node.js app calls an LLM on every request, you're paying for duplicates. Users ask the same questions, the same prompts flow from the same features, and each one costs tokens. A Redis exact-match cache intercepts those repeats before they reach the API, returning in 1-2 ms instead of 1-15 seconds.
This guide covers the middleware, the 30-minute TTL trade-off, break-even math at 10k daily calls, and the signal that tells you when a semantic layer is worth adding.
Build the middleware
The cache key is a SHA-256 hash of the full API request: model, messages, temperature, and any other parameters that affect the output. Two identical requests produce the same hash; a single token difference produces a completely different one. That strictness is what makes this safe to deploy without worrying about returning mismatched answers.
import { createHash } from 'node:crypto';
import { createClient } from 'redis';
const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();
const TTL_SECONDS = 30 * 60;
async function cachedLLMCall(params, llmFn) {
const key = 'llm:' + createHash('sha256')
.update(JSON.stringify(params))
.digest('hex');
const cached = await redis.get(key);
if (cached) return JSON.parse(cached);
const response = await llmFn(params);
await redis.setex(key, TTL_SECONDS, JSON.stringify(response));
return response;
}
Wrap your existing LLM call:
const result = await cachedLLMCall(
{ model: 'anthropic/claude-haiku-4-5', messages, temperature: 0 },
(p) => fetch('https://llmtest.io/v1/chat/completions', {
method: 'POST',
headers: { Authorization: `Bearer ${process.env.LLMTEST_API_KEY}` },
body: JSON.stringify(p),
}).then(r => r.json())
);
One important detail: include temperature: 0 in the params you hash. At temperature above 0, the same prompt produces different outputs, so the cached response appears deterministic when it isn't. For extraction and classification that's fine. For creative tasks where variety matters, either skip the cache or accept that caching collapses output variation.
The 30-minute TTL trade-off
The right TTL depends on how fast your prompts change and how much staleness matters.
Under 5 minutes: Hit rates stay below 5% for most apps. Users return, the cache has already expired, and you paid for Redis with nothing to show for it.
Over 4 hours: Pricing data, current events, and anything time-sensitive in your system prompt can go stale. A user who asked "what's the cheapest model?" at 9 am and one who asked the same at 1 pm, after a pricing update, get the same answer, one of which is wrong.
30 minutes: Covers most repeat traffic (power users, UI retries, identical form submissions) while staying inside the freshness window for slow-changing data. In practice this produces 4-6x the hit rate of a 5-minute TTL while keeping stale-answer risk low.
Add per-request TTL logic for edge cases:
function ttlForParams(params) {
const text = JSON.stringify(params.messages);
if (/price|cost|today|current|latest/i.test(text)) return 5 * 60;
return 30 * 60;
}
For prompts that touch real-time data (stock prices, live inventory), pass ttl: 0 or add a bypass flag to skip the cache entirely for that route.
Break-even math at 10k daily calls
Running 10,000 LLM API calls per day on claude-haiku-4-5 at $0.80/M input tokens and $4/M output tokens (current pricing from the LLMTest model database). A typical call uses 500 input tokens and 200 output tokens.
Cost per call: (500 x $0.80/M) + (200 x $4/M) = $0.0004 + $0.0008 = $0.00120 Daily spend: 10,000 x $0.00120 = $12.00/day ($360/month)
A Redis instance on Fly.io starts at $1.94/month. At a 25% cache hit rate (realistic for any app with repeated prompts), you save 2,500 calls per day:
2,500 x $0.00120 = $3.00/day saved ($90/month)
Net after Redis cost: ~$88/month saved, break-even at roughly 70 cache hits total. For a moderately trafficked app, that's about one hour of operation.
Hit rates vary significantly. A customer support bot with a bounded FAQ set can reach 40-60%. A chat interface where every conversation is unique will see 5-10%. Log cache hits alongside token counts for two weeks. That number tells you whether the cache is earning its keep or just adding latency on misses.
When to add a semantic layer
Exact-match caching misses paraphrases. "What does RAG stand for?" and "Can you explain what RAG means?" hash to different keys even though the answer is identical. Semantic caching catches those by embedding the prompt and finding similar cached responses by cosine similarity.
Add a semantic layer when per-response savings exceed the embedding call cost. At $0.02/M tokens for a typical embedding model, embedding 500 tokens costs $0.00001. If a cache hit saves $0.00120, the embedding needs to produce a hit roughly 1 in 120 calls to break even on its own cost. For apps where paraphrase patterns are common, that threshold is achievable.
The three failure modes of embedding-based caching (threshold grey zones, stale vectors after model upgrades, multi-turn context poisoning) are covered in detail in the semantic caching approaches guide. Short version: start with exact-match, measure hit rate for 2-4 weeks, and add the semantic layer only when miss analysis justifies the embedding overhead. The embedding cost math for production RAG applies directly here for volume estimates before you commit.
Putting it in production
Point the middleware at an LLM gateway rather than directly at the provider. If the upstream model returns a 429 or 500, you want the gateway to retry on a different provider before anything gets cached. Caching an error response means every subsequent identical request gets the error back for 30 minutes.
The LLMTest proxy handles failover and JSON recovery automatically: if one provider returns a 500, the request falls back to another and your app sees a clean response. Your cache only ever stores valid outputs. Route your API calls through the proxy first, then wrap that call with the caching middleware above.
Once cache and gateway are wired together, the per-user cost guardrails guide shows how to track spend per session, so you can see which users drive hits and which generate novel (and expensive) requests that bypass the cache.
Set up the proxy in about 5 minutes at llmtest.io.