# LLM Structured Output: JSON Schema That Actually Validates

> Prompting an LLM to return JSON is not the same as guaranteeing it. Here is how JSON Schema constraints, tool-based calling, and Zod validation turn probabilistic output into something you can trust in production.

URL: https://www.matthewswong.com/en/blog/llm-structured-output-json-schema/

- Author: Matthews Wong (https://www.matthewswong.com/en/about/)
- Published: 2026-07-01
- Updated: 2026-07-01
- Category: AI

The first time a production job crashed on a stray markdown fence wrapped around an otherwise perfect JSON payload, I learned the hard way that asking a language model nicely for JSON is not the same as receiving JSON you can parse. The model followed the instructions. It just also added a friendly sentence before the object, because nothing forced it not to.

This post walks through the parts that actually make structured output reliable: constrained decoding through JSON Schema, choosing between direct JSON output and tool-based function calling, detecting refusals and truncation before they corrupt your pipeline, and validating every response with Zod so a schema violation never reaches your database.

## Why just asking for JSON does not work

Prompt engineering can get you close, but close is not a guarantee. Even a well-crafted system prompt that says respond only with valid JSON matching this shape will occasionally produce output that technically satisfies the instruction while breaking every downstream parser you have. The failure modes are predictable once you have seen them a few times.

The model wraps the object in a markdown code fence, so a naive call to JSON parse throws on the first character.

A field the schema requires gets summarized into prose instead of populated, because the model decided a short explanation was more helpful than a strict value.

Numeric fields arrive as quoted strings, or an enum field gets a close synonym the model considered equivalent but your validator does not.

None of these are bugs in the model. They are the natural result of treating a probabilistic text generator as if it were a deterministic serializer. If your system depends on the shape of the response, you need a mechanism that constrains the generation itself, not just a politely worded request.

> Both the Claude API and the OpenAI API now offer real structured output modes that restrict token generation to only the tokens a valid schema allows. This is fundamentally different from prompting for JSON and hoping. Reach for the native feature before you reach for regex cleanup on the response text.

## Constrained decoding: how real structured outputs work

Constrained decoding means the API compiles your JSON Schema into a grammar and restricts which tokens the model is allowed to emit at each step of generation. If your schema says a field must be one of three enum values, the model is mechanically prevented from producing a fourth. This is enforced at the sampling layer, not by post-hoc validation, so there is no window where the model could have said something else and slipped past a lenient check.

Claude exposes this through two complementary mechanisms: a JSON output format you attach to a message request, and strict tool use where a tool's input schema is enforced the same way. OpenAI's equivalent is the strict flag on response_format with a json_schema type, which comes with a firm rule: every property must be listed as required, and additionalProperties must be explicitly set to false. You cannot simply omit an optional field from the required list the way you might in a hand-rolled JSON Schema; the idiomatic workaround is to make the field's type nullable instead.

## Picking between JSON output mode and tool calling

Both providers give you two shapes for the same guarantee, and the right one depends on what the response represents in your application, not on which one feels more familiar.

| Approach | Best for | Watch out for |
| --- | --- | --- |
| JSON output mode | A single structured document as the entire response, like an extracted invoice or a generated report section | First-request latency while the schema compiles into a grammar; both providers cache the compiled schema for reuse |
| Strict tool or function calling | Agent loops where the model chooses between several possible actions, each with its own argument shape | You need a distinct tool definition per action type, and the model must still decide which tool to call correctly |
| Prompt-only JSON, no schema constraint | Throwaway prototypes and manual experimentation only | No structural guarantee whatsoever; never ship this path to a production system that parses the output |

## Handling refusals and truncation gracefully

A schema constraint guarantees the shape of a completed response. It says nothing about whether the response actually completed, or whether the model declined to answer at all. Both providers give you signals for this, and skipping them is the most common way constrained output still breaks in production.

Check the stop reason or finish reason field before you attempt to parse anything. A refusal or a length cutoff means there is no complete schema-valid payload to extract, no matter how carefully you wrote the schema.

On a refusal, surface a clear, specific error to the caller instead of trying to salvage partial text. OpenAI returns a dedicated refusal field for exactly this case, and Claude reports a refusal stop reason.

On truncation, meaning the response hit the max tokens limit mid-object, do not attempt to repair the JSON by hand. Retry with a higher token limit or a narrower schema that asks for less at once.

Log the raw response text on any non-success stop reason before you discard it. The first time a schema you thought was airtight still produces a truncated array, you will want that payload to debug from.

> A schema-constrained response can still fail to parse if you skip the stop reason check. A refusal or a max-tokens cutoff produces text that is not a complete JSON object, and calling JSON parse on it will throw regardless of how tight your schema was. Always branch on the stop reason first.

## Validate every response with Zod before touching your database

Schema-constrained generation makes malformed JSON rare, not impossible, and it says nothing about whether the values are the ones you actually wanted. A model can return a syntactically perfect object with a negative quantity or an invoice number that is really a customer name. Treat the constrained response the same way you would treat any external input: parse it, then validate it with a real schema library before it touches persistence.

Running the same shape through Zod's safeParse gives you two things a raw JSON parse cannot: a typed object your TypeScript code can trust downstream, and a structured issues array you can log or return to the caller instead of a generic parsing exception. If you are already using a Zod schema to build the JSON Schema sent to the model, as shown in the earlier example, you get this validation step almost for free since the same schema definition does double duty.

## When schema-constrained output is not enough

JSON Schema and Zod both validate the shape and type of a single response. Neither one can express relationships between fields that only make sense together, or facts that live outside the response entirely. Plan for a second validation pass for cases like these.

Business rules spanning multiple fields, such as an end date that must fall after a start date, which need a custom refine function rather than a plain schema.

Cross-record checks, such as confirming a referenced order ID actually exists in your database, which only a real lookup can answer.

Domain-specific numeric ranges, like a tax rate that is structurally a valid number but is nonsensical above a certain threshold for your business.

> Because Zod schemas can generate the JSON Schema sent to the model and validate the response that comes back, a single TypeScript definition covers both ends of the round trip. That symmetry is worth designing around from the start of a project rather than bolting on after the fact.

> Before trusting a schema in production, test it against adversarial prompts: ambiguous input, requests that should trigger a refusal, and inputs deliberately close to your token limit. A schema that only ever saw clean happy-path input during development will surprise you the first week it meets real users.

## Sources:

- Claude structured outputs documentation
- OpenAI structured outputs guide
- JSON Schema (Wikipedia)
- Zod: TypeScript-first schema validation

## Source links

- https://platform.claude.com/docs/en/build-with-claude/structured-outputs
- https://developers.openai.com/api/docs/guides/structured-outputs
- https://en.wikipedia.org/wiki/JSON_Schema
- https://zod.dev

## FAQ

### What is the difference between prompting for JSON and using structured outputs?

Prompting for JSON asks the model to follow instructions, but nothing stops it from adding stray text, wrapping the object in a markdown fence, or dropping a required field. Structured outputs compile your JSON Schema into a grammar that restricts which tokens the model can generate at each step, so the response is mechanically constrained to match the schema rather than merely encouraged to.

### Why does my structured output still fail to parse sometimes?

The most common cause is skipping the stop reason or finish reason check before parsing. If the model refused the request or the response was cut off by a max tokens limit, the text is not a complete JSON object no matter how tight the schema was, and calling JSON parse on it will throw. Always branch on the stop reason first, and only attempt to parse a response that completed normally.

### Can I use Zod for both generating the schema and validating the response?

Yes, and it is one of the most useful patterns available. A single Zod schema can be converted into the JSON Schema you send to the model and used again with safeParse to validate the response that comes back. This keeps both ends of the round trip defined in one place and gives you a typed object plus a structured error list if validation fails.

### Should I use JSON output mode or tool calling for structured extraction?

Use JSON output mode when the entire response is a single structured document, such as an extracted invoice or a generated report section. Use strict tool or function calling when the model is choosing between multiple possible actions in an agent loop, since each action can have its own argument schema. Both provide the same schema guarantee; the choice depends on what the response represents.

### Does a valid JSON Schema mean the values in the response are correct?

No. Schema constraints guarantee the shape and types of the response, not the business correctness of the values inside it. A model can return a syntactically perfect object with a negative quantity or a swapped field. You still need a second validation pass, such as a Zod refine function or a database lookup, for rules that span multiple fields or depend on external state.

---

- HTML version: https://www.matthewswong.com/en/blog/llm-structured-output-json-schema/
- Bahasa Indonesia: https://www.matthewswong.com/id/blog/llm-structured-output-json-schema/
- Site index for agents: https://www.matthewswong.com/llms.txt
- Sitemap: https://www.matthewswong.com/sitemap.xml
- API description (OpenAPI): https://www.matthewswong.com/openapi.json