JSON mode explained: when schema enforcement beats prompting

Both claim to give you valid JSON. One actually enforces it.

JSON mode and structured output look identical from the outside: both return JSON, both cost the same per token, and the API call shapes are almost the same. The difference shows up when something goes wrong.

What JSON mode does

JSON mode sets response_format: { type: "json_object" } and tells the model to output valid JSON. The model tries. In real production pipelines, 2-5% of the time it doesn't succeed: the response arrives wrapped in triple-backtick fences, with extra fields the schema didn't define, with trailing commas, or as prose with a JSON fragment embedded in the middle.

This isn't a model defect. JSON mode is a best-effort instruction, not a structural constraint. The model generates tokens through normal sampling and tries to make them valid JSON. Frontier models succeed most of the time. On complex schemas, under load, or when the prompt competes with formatting instructions, they don't.

How constrained decoding actually works

Structured output is different. When you use response_format: { type: "json_schema", json_schema: {...}, strict: true } (OpenAI) or its equivalent, the API precompiles your schema into a grammar. At every token position, any token that would produce a schema-invalid character sequence is excluded before sampling begins. The model cannot generate a closing bracket in the wrong place, a value of the wrong type, or a missing required field. The constraint is structural, not instructional.

By 2026, OpenAI, Anthropic, and Google all enforce JSON schemas at the token sampling level. The implementations differ:

• OpenAI: response_format: { type: "json_schema", json_schema: {...}, strict: true }. Schema is precompiled per request. Root-level anyOf union types aren't supported in strict mode.
• Anthropic: Tool use. Define a tool whose argument schema matches your desired output and prompt Claude to call it. Constrained decoding applies to the tool arguments. Numeric minimum/maximum bounds aren't enforced at the sampling layer; they pass through without structural guarantees.
• Google: response_schema inside generationConfig. Compiled at request time. Nesting depth ceiling isn't published; hitting it returns a 400.

All three achieve under 0.1% schema failure rates on supported schemas. Our benchmark of schema compliance rates across frontier and budget models puts them in the same tier on structural metrics, with divergence only on deep nesting and union types.

When JSON mode is acceptable

JSON mode works for:

• Prototyping before you have a stable schema
• Internal tools where failures are logged and retried manually
• Cases where downstream code already validates the output before using it

It isn't acceptable for production pipelines where malformed JSON silently propagates downstream. At 3% failure, 1,000 calls per day produces 30 failed records. Those records don't surface as obvious errors: they either raise at JSON.parse() (detectable) or parse successfully as a string value with the wrong shape, silently poisoning whatever reads them next.

The cost of silent failures

The retry math is concrete. From our structured output benchmark cost analysis: at budget model pricing (GPT-4o-mini, 400 tokens in / 150 out), 30 retries per day adds $0.0045 in token cost, essentially invisible at this scale. At frontier pricing (GPT-5.5, 500 tokens in / 200 out), 15,000 retries per month equals $127 in wasted tokens plus 250 minutes of accumulated pipeline latency.

Pipeline latency is the harder cost to account for. A retry waits for the full response before it can restart. If your pipeline has structural dependencies (a malformed SQL query breaks the executor step, a missing field crashes a webhook handler), one silent failure cascades beyond the token cost of the retry.

For pipelines where downstream code doesn't safely handle bad records, the three-tier JSON validation pattern adds the right safety net: schema enforcement at the API level catches structural failures at the source; Zod or Pydantic validation in your application catches what slips through; semantic checks handle values that are structurally correct but semantically wrong.

Provider pitfalls to watch for

Three schema restrictions that hit builders most often in 2026:

When your schema hits one of these, the API returns a 400 at request time: the model is never called, and no tokens are charged. Fix the schema (remove the unsupported feature), split the call, or fall back to JSON mode with a post-parse validation layer. Anthropic's tool use for structured output is the same mechanism that powers function calling, and understanding how the tool loop works makes schema design decisions there easier.

FAQ

What's the difference between JSON mode and structured output? JSON mode instructs the model to output valid JSON as a best-effort prompt instruction. Structured output uses grammar-constrained decoding: invalid tokens are filtered before sampling, making schema violations architecturally impossible for supported schema types. In production, this is the difference between a 2-5% failure rate and under 0.1%.

Does JSON mode guarantee valid JSON? No. JSON mode improves the odds compared to plain prompting, but the model still generates tokens through normal sampling. Failure modes include markdown fences around the JSON block, extra invented fields, trailing commas, and occasional prose responses with embedded JSON fragments.

Which providers support schema enforcement in 2026? OpenAI (via response_format: json_schema with strict: true), Anthropic (via tool use with constrained decoding applied to tool arguments), and Google (via response_schema in generationConfig). Each has schema feature restrictions; see the provider pitfalls section above.

What happens when a schema constraint isn't supported? The API returns a 400 error at request time, not a malformed response. The model is never called. Fix the schema by removing the unsupported feature, or split the structured and unstructured parts of your request across two API calls.

Can I use JSON mode with Claude? Anthropic doesn't expose a standalone JSON mode flag. To get reliable structured output from Claude, define a tool whose argument schema matches your desired structure and prompt Claude to call it. The constrained decoding mechanism is the same as OpenAI's: it's expressed through the tool use API shape rather than response_format.

Does structured output cost more? No additional per-call fee from any major provider in 2026. You pay normal token prices for input and output. The cost difference shows up indirectly: eliminating retry calls removes wasted token spend, and upstream schema enforcement reduces the validation work your application has to do.

Using structured output through LLMTest

LLMTest's OpenAI-compatible proxy passes response_format (for OpenAI and Google) and tool definitions (for Anthropic) to the upstream API unchanged, so you get native constrained decoding from any model that supports it through a single endpoint. The API reference covers how structured output parameters pass through and which models expose which schema features.

Try structured output across multiple providers through a single OpenAI-compatible endpoint at llmtest.io.