LLMs return JSON, mostly. The 1-5% that don't — malformed brackets, missing required fields, values of the wrong type — are easy to catch with a JSON.parse() call. The harder problem is the JSON that passes parse validation but carries plausible-looking wrong answers: a confidence field set to 0.9 for a hallucinated fact, a category that's grammatically correct but semantically wrong, a price_usd that's 10 instead of 0.10.
That second failure mode is what hurts pipelines in production. Three tiers of validation, each catching what the previous one misses, add up to a workload that processes 1,000 requests per day with fewer than one silent bad record reaching downstream systems per week.
Tier 1: Schema validation
Schema validation catches structural problems: invalid JSON, wrong field types, missing required fields, and values out of allowed range. In Node.js, Zod is the practical choice: it composes well, error messages name the failing field, and it runs synchronously with no external calls.
import { z } from 'zod';
const OutputSchema = z.object({
category: z.enum(['billing', 'technical', 'general']),
confidence: z.number().min(0).max(1),
summary: z.string().min(10).max(500),
needs_human: z.boolean(),
});
function validateOutput(raw) {
try {
const parsed = JSON.parse(raw);
return OutputSchema.parse(parsed);
} catch (err) {
throw new Error(`Schema validation failed: ${err.message}`);
}
}
In Python, Pydantic v2 handles the same job. The @field_validator decorator lets you add cross-field constraints that the schema type system can't express:
from enum import Enum
from pydantic import BaseModel, field_validator
class Category(str, Enum):
billing = "billing"
technical = "technical"
general = "general"
class Output(BaseModel):
category: Category
confidence: float
summary: str
needs_human: bool
@field_validator("confidence")
@classmethod
def confidence_range(cls, v):
if not 0 <= v <= 1:
raise ValueError("confidence must be between 0 and 1")
return v
At 1,000 requests per day, this tier catches roughly 30-50 failures: the 3-5% structural failure rate from GPT-5.5 and Claude Opus 5 on classification tasks, higher for complex extraction schemas. Cost: effectively zero. Zod and Pydantic run in microseconds locally.
The structured output hit rates for frontier and budget models in 2026 show that native structured output (constrained decoding or tool-use enforcement) can push this failure rate below 0.1%. If you have the option to use native structured output, take it. When you're on JSON mode or an older model that doesn't support it, this tier is mandatory.
Tier 2: Semantic validation
Schema-valid JSON can still be semantically wrong. A confidence: 0.95 on a summary that says "might be" or "possibly". A category: "technical" for a ticket that mentions only a billing dispute. These pass Zod, hit your database, and cause problems users notice.
Two approaches, applied in order:
Value-range heuristics run synchronously against your domain knowledge. They're fast, free, and catch the most common mismatches:
const TECH_TERMS = ['error', 'bug', 'api', 'timeout', 'crash', 'stack'];
function semanticValidate(output, rawTicket) {
if (output.confidence > 0.9 && output.summary.includes('possibly')) {
throw new Error('Semantic mismatch: high confidence with hedging language');
}
if (
output.category === 'technical' &&
!TECH_TERMS.some((t) => rawTicket.toLowerCase().includes(t))
) {
throw new Error('Semantic mismatch: technical category with no technical terms');
}
return output;
}
LLM judge for cases your heuristics can't cover. Pass the original input and the model's output to a second, cheaper model and ask it to score validity:
async function judgeOutput(input, output, judgeModel, apiKey, callModel) {
const prompt = `Input: ${input}\nModel output: ${JSON.stringify(output)}\n\nDoes the output correctly categorize the input? Reply VALID or INVALID, then one sentence of reasoning.`;
const response = await callModel({
model: judgeModel,
messages: [{ role: 'user', content: prompt }],
apiKey,
});
return response.output.startsWith('VALID');
}
At 1,000 requests per day, running a judge on every record costs roughly $0.30-$0.60/day with Haiku-class pricing ($0.25-$0.80/M tokens, ~500 tokens per judge call). Most teams only invoke the judge on outputs the heuristics flagged as uncertain, reducing judge calls to 10-20% of volume and cutting the daily cost to $0.03-$0.12.
Tier 3: Retry with reprompt
When validation fails at either tier, reprompting with the failure reason usually works better than returning an error to the caller. Most structural failures and many semantic ones disappear on a second attempt when you tell the model what was wrong:
const MAX_RETRIES = 2;
async function validateWithRetry(input, model, systemPromptBase, apiKey, callModel) {
let lastErr = null;
for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
const systemPrompt = lastErr
? `${systemPromptBase} Your last response failed validation: ${lastErr}. Try again.`
: systemPromptBase;
const raw = await callModel({
model,
messages: [{ role: 'user', content: input }],
systemPrompt,
apiKey,
});
try {
const validated = validateOutput(raw.output);
semanticValidate(validated, input);
return validated;
} catch (err) {
lastErr = err.message;
}
}
throw new Error(`Validation failed after ${MAX_RETRIES + 1} attempts: ${lastErr}`);
}
The retry adds 1-2x the original token cost for each failed record. If 5% hit tier-1 and 2% hit tier-2 heuristics, total retry overhead is roughly 7% of base token cost, which is worth it compared to the downstream cost of a bad record silently reaching your application.
Cap retries at 2. If a record fails three attempts, the schema or the prompt has a structural mismatch that a retry loop won't fix. Log it, route it for manual review, and fix the root cause.
The cost of silent failures
At 1,000 requests per day with a 3% schema failure rate and a 1.5% semantic mismatch rate: 45 bad records reach downstream systems per day without validation.
What those cost depends on the use case:
- Miscategorized support ticket routed to the wrong queue: 10-15 minutes of agent time per record, $15-30/day at typical support costs
- Incorrect confidence score in a financial analysis pipeline: potentially material, hard to quantify
- Hallucinated value in a customer-facing email draft: brand and compliance risk
The three-tier approach costs $0-$0.12/day extra at 1,000 requests per day (schema and heuristic tiers are free; the judge tier adds at most $0.12/day if you judge every record). Break-even is one prevented bad record per week.
Which tier to add first
Add Zod or Pydantic schema validation today if you haven't already. It takes 20 minutes to define the schema you already have in your head, and it catches structural failures that would otherwise produce silent undefined errors downstream. That alone eliminates the majority of production JSON failures.
Add semantic heuristics once you know what "wrong but valid-looking" looks like for your task. Production logs from the first week usually show you. Start with the two or three most common mismatches.
Add an LLM judge only for high-stakes outputs where heuristics don't cover enough cases. Track the judge's own accuracy on a golden set; it's not infallible, and a judge that's 95% accurate at $0.06/day may not justify the added latency over a well-tuned heuristic.
The LLMTest proxy logs schema validity rates, retry counts, and per-model failure rates automatically, which makes it easier to see which models in your stack produce the most validation failures and track whether they improve after model updates. To catch validation regressions in CI before they reach production, adding LLM eval checks to your Node.js CI pipeline shows how to run schema compliance checks on every PR.
Sign up to add automatic validity tracking to any LLM call without changing your existing validation code.