Gemini 2.5 Flash vs Haiku 4.5 vs GPT-5 mini: GPT-6 Luna beats all 3

By LLMTest Lab · Nov 9, 2026 · 5 min read h2hllm-comparisoncostbudget-models
On this page

On this page

  1. How we tested
  2. Where each model won and lost
  3. What it costs at volume
  4. Subscription vs API
  5. Verdict

If you are choosing a cheap model in November 2026, skip all three of the 2025 favorites. In our 8-task test, GPT-6 Luna won the most head-to-head rounds (11) and cost $0.19 per 1,000 answers. Claude Haiku 5.5 tied GPT-5 mini for second at $0.40 per 1,000. Gemini 2.5 Flash won no rounds at all.

The queue brief for this post was the classic "cheap but good" shootout: Gemini 2.5 Flash, Claude Haiku 4.5 and GPT-5 mini. Those are still the models many codebases are pinned to, so we tested them as asked. We also added the current budget models from the same two makers, Claude Haiku 5.5 and GPT-6 Luna, because the honest answer to "which cheap model" has changed.

Model Input / output $ per M tokens Wins (of 32 possible) Avg cost per answer Avg latency
openai/gpt-6-luna $0.10 / $0.50 11 $0.00019 6.1 s
openai/gpt-5-mini $0.25 / $2.00 7 $0.00157 10.8 s
anthropic/claude-haiku-5.5 $0.10 / $0.50 7 $0.00040 3.9 s
anthropic/claude-haiku-4.5 $1.00 / $5.00 2 $0.00127 1.7 s
google/gemini-2.5-flash $0.30 / $2.50 0 $0.00147 3.1 s

Prices come from OpenRouter's public model list. Costs and latency are measured averages from our run.

How we tested

We wrote 8 prompts a small team really sends: fixing a failing Jest test, extracting JSON from a German invoice, drafting a refund reply under strict policy limits, a Postgres window-function query, a three-bullet CFO summary with word limits, a crate-counting word problem, a date regex with edge cases, and ticket classification with an escalation flag. Roughly a third (the invoice, the regex and the tickets) separate careful models from sloppy ones.

All five models answered every prompt with a 2,500-token output cap. Each pair of models was compared on each prompt by Claude Sonnet 5.5 as judge, run twice with the positions swapped, and a win counts only when both orders agree. The rubric put correctness first, then constraint adherence, then concision. Calls went directly through OpenRouter on 2026-11-09. Total spend for the post was $1.45 of the $5 cap. The spec, every answer and every judgment are saved under server/blog/runs/cheap-llm-shootout-2026/.

An important caveat: of 80 pairwise judgments (10 pairs across 8 prompts), 53 ended with the two judge orders disagreeing, which we count as ties. Only 27 were decisive. That is a small sample, so read the ranking as a strong hint, not a measurement to two decimals. The gap between first place and last place is clear. The gap between GPT-5 mini and Haiku 5.5 is not.

Where each model won and lost

GPT-6 Luna won on the format-sensitive tasks: ticket classification (4 decisive wins, the most of any model there), invoice extraction and the regex. It writes short answers (368 output tokens on average, against 772 for Haiku 5.5 and GPT-5 mini), which is where most of its cost advantage comes from.

Claude Haiku 5.5 was the most consistent. It won the Jest bug, the CFO summary and two of the regex rounds. At 3.9 s average latency it is also nearly three times faster than GPT-5 mini.

GPT-5 mini was strong at extraction and classification but the slowest (10.8 s) and, despite a low list price, the most expensive per answer at $0.00157, because it spends output tokens freely.

Claude Haiku 4.5 is the fastest here at 1.7 s and writes tersely, but it costs about three times more per answer than Haiku 5.5 and lost to its successor in most rounds.

Gemini 2.5 Flash lost every decisive round. The clearest failure was the Jest prompt, where it ran to the 2,500-token cap and cut off mid-sentence. Claude Sonnet 5.5, our judge, put it this way:

Response A is a long, rambling stream of consciousness. It makes contradictory claims, retraces several times, and gets cut off mid-sentence without ever providing a corrected function or a clean one-paragraph explanation.

On the invoice prompt it returned JSON with a stray line break inside a key. The judge's note:

A raw newline inside a JSON string is illegal, so A is not valid JSON and a parser would fail on it.

One note on our own prompt: the Jest snippet we wrote actually returns 3 on the given input, not 2, so the "failing test" premise was wrong. The judge noticed and rewarded models that said so. That favored careful models and punished the model that tried to force a fix for a nonexistent failure, which is a fair reflection of real debugging, but it is worth knowing.

What it costs at volume

Using the measured averages, 1,000 answers of this kind cost about $0.19 on GPT-6 Luna, $0.40 on Haiku 5.5, $1.27 on Haiku 4.5, $1.47 on Gemini 2.5 Flash and $1.57 on GPT-5 mini. A pipeline sending 500,000 requests a month would pay roughly $97 on Luna and $785 on GPT-5 mini for answers of the length we saw. Your prompts and answer lengths will differ, so rerun the arithmetic on your own token counts.

Subscription vs API

If you use these models through a consumer plan rather than the API, the math is different. Anthropic's Claude plans, listed on claude.com list Pro at $20 a month ($17 billed annually) and Max at $100. Google's AI plans list AI Plus at $4.99, AI Pro at $19.99 and AI Ultra from $99.99. ChatGPT also sells monthly plans, but we could not load its pricing page during this run, so we are not quoting a figure. At the Luna rate above, $20 buys about 100,000 answers of this size through the API, so a plan only pays off for interactive use, not for pipelines.

Verdict

  • Default pick for cost-sensitive pipelines: GPT-6 Luna. First in wins, lowest cost per answer.
  • When latency matters or you want fewer tokens to review: Claude Haiku 5.5, with similar list pricing and the fastest of the newer pair.
  • Migrate away from: Gemini 2.5 Flash for tasks that need clean structured output or tight reasoning. On this test it lost to every other model, and one answer hit the output cap.
  • Still fine if pinned: GPT-5 mini and Haiku 4.5 produce correct answers, but each costs 3 to 8 times more per answer than the newer budget option from the same maker.

To pair a cheap default with a stronger fallback, see cheap-first routing with escalation. For a different task family, the cheapest models for function calling covers tool-use reliability, and the free-model roundup covers what you can run at zero cost.

Related articles

Cheapest LLMs for function calling in 2026: ranked by reliability
GPT-4o-mini, Gemini 2.5 Flash, DeepSeek V4-Flash, and Haiku 4.5 ranked on schema errors, retry costs, and multi-turn reliability for function calling in 2026.
How to route LLM prompts in 2026: cheap first, escalate on fail
Route each prompt to the cheapest model that handles it well. When quality falls short, escalate silently. Here's the pattern with working Node.js code.