A chatbot tells your user the return window is 90 days. Your policy says 30. No one set a trap. The model filled a gap in its knowledge with a confident guess, and now you're on the hook for a refund you didn't authorize.
That's the practical shape of an LLM hallucination: not a dramatic error, but a fluent, confident output that's wrong. Understanding why it happens, which category it falls into, and how to catch it before your users do is the difference between an AI feature that ships and one that quietly burns your support queue.
Why LLMs hallucinate
LLMs predict the next token by sampling from a probability distribution built on training data. There's no internal truth-checker, no query to the real world between tokens. A model "always predicts a different token," lacking an inherent sense of accuracy, working from statistical likelihood rather than verified knowledge.
This has one direct consequence: confidence doesn't track accuracy. A model that's wrong sounds identical to a model that's right. The fluency, the punctuation, the sentence structure are all learned from human writing, where confident-sounding text usually means something defensible was written. The model copies the style, not the epistemology.
Three error categories
Most discussions treat hallucination as one problem. In production, it's three distinct failure modes that require different fixes:
Factual confabulation: The model asserts something that isn't true — an invented citation, a wrong date, a person who doesn't exist, a made-up function name. This is the classic hallucination, and it's what most benchmark evaluations measure. The source is training data gaps or conflicting information in pre-training.
Faithfulness failure: The model receives source context (a retrieved document, a user-provided excerpt, a transcript) but supplements it with training knowledge instead of staying grounded in what was provided. The answer may be factually true, but it violates the instruction to use only the provided material. This is the primary hallucination vector in production RAG systems, where it quietly degrades grounding guarantees at scale.
Instruction drift: The model correctly understood the task but violated a stated constraint mid-response. You asked for three bullet points; it produced seven. You asked for French output; it switched to English at paragraph three. This isn't a knowledge failure — the constraint was clear. It's a context-following failure, often triggered when output grows long or when the instruction competes with patterns the model has seen more of in training.
Each category needs a different detection approach: factual checks against known answers, faithfulness checks against provided context, and constraint checks against the original instruction.
Prompting patterns that actually help
Explicit grounding constraint: Tell the model exactly what source it must use and what to say if the source doesn't contain the answer. "Answer only using the document below. If the answer isn't there, respond with: not found." This targets faithfulness failures directly. Models follow explicit "not found" instructions more reliably than open-ended "stay grounded" instructions.
Cite before concluding: Ask the model to quote the supporting text verbatim before stating its answer. "First, quote the exact sentence from the context that supports your answer. Then answer." Forces a connection between claim and source. If no quote exists, the model usually says so rather than fabricating one.
Chain-of-thought before conclusion: Ask the model to reason step-by-step before its final answer. This reduces reasoning drift in complex tasks because the intermediate steps are visible and tend to catch internal contradictions before they reach the conclusion.
Self-consistency sampling: For high-stakes output, run the same prompt three to five times with temperature above 0. Answers that agree across runs are more reliable. Answers that diverge signal a case worth flagging for human review or a different model. This trades latency and cost for confidence, best reserved for infrequent, critical queries.
What doesn't work: meta-instructions like "don't hallucinate", "be accurate", or "only state things you're certain about" have no consistent effect on hallucination rates. Models trained with RLHF tend to become more confident under direct challenge, not more accurate.
The production eval that catches failures early
The most practical defense is a 50-question golden set: questions with known correct answers drawn from your actual data, documents, or policy sources.
Run your production prompt through your model on all 50 questions. Use an LLM-as-judge (a different model, or a faster/cheaper one) to score each answer on three criteria:
- Faithfulness: does the answer stay within the provided context?
- Factual accuracy: does it match the known correct answer?
- Constraint adherence: does it follow the format, length, and scope instructions?
Track the pass rate. Set a threshold: 90% is a reasonable starting point for customer-facing features. Run the eval as part of your deploy pipeline. If the pass rate drops after a model or prompt change, block the deploy before it reaches users.
The LLMTest benchmark runner supports LLM-as-judge evaluations against custom golden sets; our documentation covers building a faithfulness rubric and scoring at scale.
FAQ
What exactly is an LLM hallucination? Any model output that is factually wrong, unsupported by the provided context, or violates a stated constraint, regardless of how confident the model sounds. The term covers three distinct failure modes: factual confabulation, faithfulness failure, and instruction drift.
Why do LLMs sound confident when they're wrong? Confidence is a stylistic feature of the output, not a signal of accuracy. Models learn from human text where confident language usually means something defensible was written. The model copies the style. Its internal probability distribution doesn't distinguish "I know this" from "this is the most statistically likely next token given what I've generated so far."
Does using a larger model reduce hallucinations? Somewhat, and unevenly. Frontier models confabulate less often on factual recall, but they also produce more fluent errors on topics outside their training. For critical applications, retrieval (giving the model the right context) and evaluation (testing against a golden set) matter more than model size alone.
Does RAG eliminate hallucinations? RAG reduces factual confabulation by giving the model source material to work from. But it introduces faithfulness failures, where the model may still supplement retrieved content with training knowledge. Measuring faithfulness (does the answer stay in the document?) is different from measuring factual accuracy. Our guide to building a RAG pipeline covers how to set up separate retrieval and generation evals that catch both failure modes.
Does "don't hallucinate" work as a prompt instruction? No. Meta-instructions like "be accurate", "only state facts", or "don't make things up" have little consistent effect on hallucination rates. Grounding instructions, citation requirements, and chain-of-thought steps are more effective than accuracy admonishments.
How do I handle hallucinations in structured output? Use constrained decoding (JSON schema enforcement at the token level) to prevent structural violations, then add semantic checks for value correctness. JSON mode vs structured output explains how each provider implements these constraints and where structural enforcement still leaves value-level gaps.
Try your own prompts against multiple providers through a single endpoint at llmtest.io, useful for spotting which models confabulate more on your specific task.