Picking an LLM for your RAG pipeline isn't the same as picking the best general-purpose model. The task is narrow: given retrieved documents, answer precisely from them, cite the right source, refuse to speculate when context runs dry, and surface contradictions rather than silently picking one number. Standard benchmarks don't measure this. We ran four models through six prompts designed specifically to surface these failure modes. The results were sharper than expected.
If you're still deciding whether to add RAG to your stack at all, our primer on what RAG actually does covers the full ingestion-retrieval-generation pipeline. This post assumes you've already made that call and are now choosing which LLM goes at the synthesis step.
What we tested
Six prompts, each targeting a different failure mode in RAG answer synthesis:
| Prompt | What it tests |
|---|---|
| Rate limit retrieval | Basic recall accuracy and citation completeness |
| Pricing vs performance trade-off | Multi-document synthesis |
| Out-of-scope question | Hallucination resistance (answer not in context) |
| Conflicting latency reports | Contradiction surfacing across two documents |
| Enterprise pricing (contact-sales tier) | Partial information handling |
| Comparison to model outside the context | Training-data override resistance |
Each prompt gave models a system instruction ("answer only from context, cite by document number") and two retrieved documents containing the relevant (or deliberately missing) information. A dual-pass judge evaluated every head-to-head comparison with positions swapped to cancel order bias. Scoring rubric: faithfulness, citation accuracy, completeness, and honesty about gaps.
Candidates: anthropic/claude-opus-4-8, openai/gpt-5.5, anthropic/claude-haiku-4-5, openai/gpt-4o-mini
For context on our benchmark approach, the same dual-pass methodology underpins our code review benchmark.
Results
| Model | Wins (of 18) | Avg latency | Avg cost/call |
|---|---|---|---|
| Claude Opus 4.8 | 15 | 4,740ms | $0.00764 |
| GPT-5.5 | 10 | 4,412ms | $0.00673 |
| Claude Haiku 4.5 | 6 | 3,833ms | $0.00103 |
| GPT-4o-mini | 1 | 1,779ms | $0.00009 |
Head-to-head breakdown:
- Opus 4.8 vs Haiku 4.5: 5-0-1 (5 Opus wins, 0 Haiku wins, 1 tie)
- Opus 4.8 vs GPT-5.5: 4-1-1
- Opus 4.8 vs GPT-4o-mini: 6-0-0
- GPT-5.5 vs Haiku 4.5: 4-2-0
- GPT-5.5 vs GPT-4o-mini: 5-0-1
- Haiku 4.5 vs GPT-4o-mini: 4-1-1
Opus 4.8 dominates. GPT-5.5 is a real second place, well ahead of both budget options. Haiku holds its own against GPT-4o-mini but falls clearly behind GPT-5.5.
Three representative comparisons
1. Hallucination resistance: all models pass, but not equally
The out-of-scope prompt asked about file upload limits using only webhook documentation as context. Every model correctly said the context didn't contain the answer. The difference was in what they did next.
Haiku added guidance the system prompt explicitly prohibited: suggestions to check "other sections of the API documentation such as 'File Upload,' 'API Limits,' or 'Request Parameters'." Those recommendations came from training data, not the provided documents.
The judge on Opus 4.8 vs Haiku 4.5:
"Both responses correctly acknowledge that the context doesn't contain the requested information. However, Assistant B violates faithfulness by adding suggestions about 'other sections of the API documentation such as File Upload, API Limits, or Request Parameters' — this information comes from the assistant's training data, not the provided context. Assistant A strictly adheres to the instruction to use ONLY the information in the provided context documents and includes proper citations. Assistant B provides additional helpful context but violates the core RAG principle by introducing external information."
This pattern is the primary hallucination vector in production RAG: a model that "helpfully" supplements missing context with training knowledge. At low rates it's barely visible. At scale it quietly degrades grounding guarantees.
2. Conflicting documents: GPT-4o-mini fails critically
Two performance reports gave different latency numbers for the same model: 1,240ms under low load (Q1) vs 1,890ms under production load (Q2). The question: "What is the average latency?"
Opus 4.8 cited both documents, explained the different measurement conditions, and correctly noted the figures weren't directly comparable. GPT-4o-mini responded: "The provided context does not contain this information."
The judge:
"Assistant A provides a comprehensive answer using only the information from the provided documents, correctly citing [Doc 1] and [Doc 2] for the respective latency figures of 1,240ms and 1,890ms. Assistant A also appropriately contextualizes the data by noting the different measurement conditions and regions, which helps explain why the figures differ. Assistant B incorrectly claims the context doesn't contain the information, when in fact both documents clearly provide ModelPro latency data that directly answers the user's question. Assistant B fails on completeness and misapplies the 'insufficient information' response when the context actually contains clear, relevant data."
In production, this is a silent failure. Users see "no answer found" on a question the system actually has data for. The model pattern-matched on "two conflicting documents" and refused rather than synthesizing.
3. Citation granularity: Opus vs GPT-5.5
Most prompts separated these two models narrowly. The consistent gap was citation placement: GPT-5.5 cited at the paragraph level (attribution at the end), while Opus 4.8 integrated citations inline with individual claims.
The judge on the basic retrieval prompt:
"Both responses are highly faithful to the provided context, extracting information only from Doc 1 without adding external knowledge. Assistant A provides proper citations throughout the response, clearly attributing each piece of information to [Doc 1]. Assistant B also cites Doc 1 but places the citation at the end of paragraphs rather than integrating it more clearly with specific claims. The key difference is in citation style and directness of answering. Assistant A integrates citations more naturally and provides a more explicit answer to the specific question."
For a product where users verify answers against sources, inline citation style meaningfully improves trust. It's a small win but it repeated across prompts.
When to use each model
Claude Opus 4.8 is the clear choice for RAG pipelines where answer quality and citation fidelity matter: legal document Q&A, financial research, medical information, customer support with contractual sources. At $5/$25 per million tokens (input/output), the full Opus 4.8 capability profile is worth reviewing if you haven't already.
GPT-5.5 is a solid alternative for teams already on the OpenAI stack. Citation style is slightly looser but faithfulness holds up. $5/$30 per million tokens.
Claude Haiku 4.5 works for high-volume, cost-sensitive RAG where context is clean and single-source: internal FAQ bots, documentation search, product knowledge bases. Avoid it when documents may conflict. $1/$5 per million tokens.
GPT-4o-mini is not recommended for RAG synthesis. The failure on the conflicting-documents prompt is disqualifying for any production use case that might surface contradictory sources. Use it for triage or intent classification in your pipeline, not generation. $0.15/$0.60 per million tokens.
Subscription vs API
Building and running a RAG feature today means choosing between API billing and consumer subscriptions. Both Anthropic and OpenAI bundle model access into paid plans.
| Model | API: input / output per 1M tokens | Subscription |
|---|---|---|
| Claude Opus 4.8 | $5 / $25 | Claude Pro $20/mo, Max $100-200/mo |
| GPT-5.5 | $5 / $30 | ChatGPT Plus $20/mo, Pro $100/mo |
| Claude Haiku 4.5 | $1 / $5 | Included with Claude Pro |
| GPT-4o-mini | $0.15 / $0.60 | Included with ChatGPT Plus |
Break-even for Opus 4.8 vs Claude Pro ($20/month): A typical RAG call with a 1,500-token context window and 250-token output costs roughly $0.014 ($0.0075 input + $0.00625 output). Claude Pro breaks even at about 1,430 calls/month, or 48 queries per day. Above that, API billing is cheaper. Prompt caching shifts this significantly: with a cached system prompt (90% discount on repeated input), the per-call cost drops to around $0.007 and the break-even rises to roughly 95 queries per day.
GPT-5.5 vs ChatGPT Plus ($20/month): Same RAG call costs roughly $0.015. Break-even at about 1,330 calls/month (44/day).
For high-volume production, neither subscription covers API usage at scale. Current pricing is on Anthropic's pricing page and OpenAI's API pricing.
How this was tested
6 prompts, 4 candidates, all-pairs tournament. Each pair was judged twice with positions swapped (forward and reverse) to cancel order bias. Judge model: anthropic/claude-sonnet-4. Total pairwise comparisons: 36 across 6 prompts. Total runner cost: $0.52 (model inference $0.11, judge calls $0.41). Zero errors. Full methodology at /docs/benchmarks.
To run RAG-specific benchmarks across your own prompts and document sets, LLMTest routes to all four models above with per-call cost and latency tracking.