How to add LLM observability in 2026: cost, latency, quality

Traditional API monitoring tells you whether your endpoint returned 200 or 500. That is not enough for LLM calls. The model can return 200, sound confident, and be completely wrong -- a failure mode that HTTP status codes will never catch. LLM observability is the practice of logging enough signal to know when that happens before your users do.

This post covers the four signals worth capturing on every LLM call, an async Node.js middleware implementation that costs under 2ms of overhead, and how to set thresholds that distinguish real regressions from normal model variance.

The four signals

Tokens (input and output separately)

Token counts are your cost ledger. Input tokens determine what you pay per call; output tokens are where spend compounds on long completions. Tracking them separately lets you spot prompt bloat (input creep from context window stuffing) and output runaway (the model generating far more than the task needs).

Each provider's response includes these. Look for usage.input_tokens and usage.output_tokens in Anthropic responses, and usage.prompt_tokens and usage.completion_tokens in OpenAI-compatible ones.

Cost in USD

Derive cost from token counts at call time, not at report time. Provider rates change, and storing tokens lets you recompute cost history if you need to, but a pre-computed cost field in your log saves a join in every analytics query. A minimal rate table:

function computeCost(model, inputTokens, outputTokens) {
  const rates = {
    'anthropic/claude-sonnet-4-6': { input: 3.0, output: 15.0 },
    'anthropic/claude-haiku-4-5':  { input: 0.8,  output: 4.0 },
    'openai/gpt-4o-mini':          { input: 0.15, output: 0.6 },
  };
  const r = rates[model] ?? { input: 3.0, output: 15.0 };
  return ((inputTokens * r.input) + (outputTokens * r.output)) / 1_000_000;
}

Update this table when providers change rates (they do). The unit is USD per million tokens; your call cost comes out in USD.

Latency: TTFT and total

Total latency is what your user feels, but time-to-first-token (TTFT) tells you whether a slow response is a slow model or a slow network. A TTFT under 500ms with a long total duration is a normal streaming completion. A TTFT above 3 seconds almost always means a provider-side queue or cold-start issue.

For streaming calls, split the measurement:

const t0 = performance.now();
let ttft = null;

for await (const chunk of stream) {
  if (!ttft && chunk.choices?.[0]?.delta?.content) {
    ttft = performance.now() - t0;
  }
}

const totalMs = performance.now() - t0;

Quality proxy

You cannot auto-grade creative writing, but you can catch obvious regressions:

• Output length vs expected: a summarizer that starts returning 3x the expected word count is probably broken.
• Refusal rate: a sudden spike from 0.5% to 8% means something in your prompt changed or the model's safety thresholds shifted.
• Format failure rate: if you're prompting for JSON and getting prose on 5% of requests instead of 0.1%, the model drifted.

Track a rolling baseline per model and prompt version. A deviation beyond 2.5 standard deviations is worth a log alert; beyond 4 is worth paging.

Async logging middleware

The goal is zero added latency on the hot path. Every write goes to a non-blocking call:

import { appendFile } from 'node:fs';
import { performance } from 'node:perf_hooks';

function createLogger(logPath) {
  return function log(entry) {
    const line = JSON.stringify({ ts: new Date().toISOString(), ...entry }) + '\n';
    appendFile(logPath, line, () => {}); // fire-and-forget
  };
}

const log = createLogger('./logs/llm-calls.ndjson');

async function callWithLogging({ model, messages, apiKey }) {
  const t0 = performance.now();
  let ttft = null;
  let inputTokens = 0;
  let outputTokens = 0;
  const chunks = [];

  try {
    const res = await fetch('https://llmtest.io/v1/chat/completions', {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        Authorization: `Bearer ${apiKey}`,
      },
      body: JSON.stringify({ model, messages, stream: true }),
    });

    if (!res.ok) throw new Error(`HTTP ${res.status}`);

    const reader = res.body.getReader();
    const decoder = new TextDecoder();

    while (true) {
      const { done, value } = await reader.read();
      if (done) break;

      for (const line of decoder.decode(value).split('\n')) {
        if (!line.startsWith('data: ') || line === 'data: [DONE]') continue;
        try {
          const d = JSON.parse(line.slice(6));
          const token = d.choices?.[0]?.delta?.content;
          if (token) {
            if (!ttft) ttft = performance.now() - t0;
            chunks.push(token);
          }
          if (d.usage) {
            inputTokens = d.usage.prompt_tokens ?? d.usage.input_tokens ?? inputTokens;
            outputTokens = d.usage.completion_tokens ?? d.usage.output_tokens ?? outputTokens;
          }
        } catch { /* malformed chunk; skip */ }
      }
    }

    const totalMs = Math.round(performance.now() - t0);
    const output = chunks.join('');

    log({
      model,
      status: 'ok',
      ttftMs: ttft ? Math.round(ttft) : null,
      totalMs,
      inputTokens,
      outputTokens,
      costUsd: computeCost(model, inputTokens, outputTokens),
      outputLength: output.length,
    });

    return output;
  } catch (err) {
    log({
      model,
      status: 'error',
      totalMs: Math.round(performance.now() - t0),
      error: err.message,
    });
    throw err;
  }
}

This writes NDJSON (newline-delimited JSON) to a flat file -- one record per call. It's readable with jq, importable into any log aggregator, and good enough for a side project running thousands of calls per day. When you outgrow a flat file, point the log() call at a database write or an SDK client instead; the shape of the record stays the same.

At higher throughput (10k+ calls per minute), batching the writes avoids filesystem contention:

const queue = [];
setInterval(() => {
  if (!queue.length) return;
  const batch = queue.splice(0);
  appendFile('./logs/llm-calls.ndjson', batch.join(''), () => {});
}, 500);

The write never blocks the LLM call. That is the one rule.

What to alert on

Start narrow. Three signals catch the majority of production problems:

Alert on percentiles over time windows, not individual slow calls. P99 latency is always ugly for long-form generation. A systematic shift in P95 is the signal you are looking for.

For 429 errors specifically, the patterns in production LLM rate-limit handling pair naturally with this setup: once you know your 429 rate per model from logs, you can tune your backoff and token-budget pre-checks to match.

When a proxy handles this for you

If you're routing through the LLMTest proxy, cost, latency, and token counts are logged per request with no application-side instrumentation. You get per-model breakdowns in the dashboard automatically. For quality signals -- output length drift, format failure rate, refusal spikes -- you still need application-side logging, because the proxy does not know what "correct output" means for your use case.

For streaming calls, stall detection also stays in your application code. The proxy records total latency but cannot fire a watchdog midstream. The stream stall detection guide covers the watchdog pattern, and its implementation fits cleanly into the callWithLogging wrapper above.

Where to go from here

Get your first 72 hours of baseline data with just the four signals above. Then add quality proxies one at a time as you learn which metrics actually move on your workload. The observability overhead of this setup, benchmarked on a mid-tier server, adds under 2ms per call at 100 requests per second -- flat file I/O is fast when it's non-blocking.

Track cost and latency across providers in real time using LLMTest.