LLM Structured Output: JSON Schema That Actually Validates

The first time a production job crashed on a stray markdown fence wrapped around an otherwise perfect JSON payload, I learned the hard way that asking a language model nicely for JSON is not the same as receiving JSON you can parse. The model followed the instructions. It just also added a friendly sentence before the object, because nothing forced it not to.

This post walks through the parts that actually make structured output reliable: constrained decoding through JSON Schema, choosing between direct JSON output and tool-based function calling, detecting refusals and truncation before they corrupt your pipeline, and validating every response with Zod so a schema violation never reaches your database.

Why just asking for JSON does not work

Prompt engineering can get you close, but close is not a guarantee. Even a well-crafted system prompt that says respond only with valid JSON matching this shape will occasionally produce output that technically satisfies the instruction while breaking every downstream parser you have. The failure modes are predictable once you have seen them a few times.

• The model wraps the object in a markdown code fence, so a naive call to JSON parse throws on the first character.
• A field the schema requires gets summarized into prose instead of populated, because the model decided a short explanation was more helpful than a strict value.
• Numeric fields arrive as quoted strings, or an enum field gets a close synonym the model considered equivalent but your validator does not.

None of these are bugs in the model. They are the natural result of treating a probabilistic text generator as if it were a deterministic serializer. If your system depends on the shape of the response, you need a mechanism that constrains the generation itself, not just a politely worded request.

Constrained decoding: how real structured outputs work

Constrained decoding means the API compiles your JSON Schema into a grammar and restricts which tokens the model is allowed to emit at each step of generation. If your schema says a field must be one of three enum values, the model is mechanically prevented from producing a fourth. This is enforced at the sampling layer, not by post-hoc validation, so there is no window where the model could have said something else and slipped past a lenient check.

import Anthropic from "@anthropic-ai/sdk";
import { z } from "zod";
import { zodOutputFormat } from "@anthropic-ai/sdk/helpers/zod";

const InvoiceLine = z.object({
  sku: z.string(),
  quantity: z.number().int().positive(),
  unitPriceCents: z.number().int().nonnegative(),
});

const InvoiceExtraction = z.object({
  vendorName: z.string(),
  invoiceNumber: z.string(),
  lines: z.array(InvoiceLine),
  totalCents: z.number().int().nonnegative(),
});

const client = new Anthropic();

const response = await client.messages.create({
  model: "claude-sonnet-5",
  max_tokens: 1024,
  messages: [{ role: "user", content: rawInvoiceText }],
  output_config: { format: zodOutputFormat(InvoiceExtraction) },
});

Claude exposes this through two complementary mechanisms: a JSON output format you attach to a message request, and strict tool use where a tool's input schema is enforced the same way. OpenAI's equivalent is the strict flag on response_format with a json_schema type, which comes with a firm rule: every property must be listed as required, and additionalProperties must be explicitly set to false. You cannot simply omit an optional field from the required list the way you might in a hand-rolled JSON Schema; the idiomatic workaround is to make the field's type nullable instead.

Picking between JSON output mode and tool calling

Both providers give you two shapes for the same guarantee, and the right one depends on what the response represents in your application, not on which one feels more familiar.

Handling refusals and truncation gracefully

A schema constraint guarantees the shape of a completed response. It says nothing about whether the response actually completed, or whether the model declined to answer at all. Both providers give you signals for this, and skipping them is the most common way constrained output still breaks in production.

1. Check the stop reason or finish reason field before you attempt to parse anything. A refusal or a length cutoff means there is no complete schema-valid payload to extract, no matter how carefully you wrote the schema.
2. On a refusal, surface a clear, specific error to the caller instead of trying to salvage partial text. OpenAI returns a dedicated refusal field for exactly this case, and Claude reports a refusal stop reason.
3. On truncation, meaning the response hit the max tokens limit mid-object, do not attempt to repair the JSON by hand. Retry with a higher token limit or a narrower schema that asks for less at once.
4. Log the raw response text on any non-success stop reason before you discard it. The first time a schema you thought was airtight still produces a truncated array, you will want that payload to debug from.

Validate every response with Zod before touching your database

Schema-constrained generation makes malformed JSON rare, not impossible, and it says nothing about whether the values are the ones you actually wanted. A model can return a syntactically perfect object with a negative quantity or an invoice number that is really a customer name. Treat the constrained response the same way you would treat any external input: parse it, then validate it with a real schema library before it touches persistence.

const parsed = InvoiceExtraction.safeParse(
  JSON.parse(response.content[0].text)
);

if (!parsed.success) {
  logger.warn("schema_validation_failed", {
    issues: parsed.error.issues,
  });
  throw new ExtractionValidationError(parsed.error);
}

// parsed.data is now fully typed and safe to persist
await saveInvoice(parsed.data);

Running the same shape through Zod's safeParse gives you two things a raw JSON parse cannot: a typed object your TypeScript code can trust downstream, and a structured issues array you can log or return to the caller instead of a generic parsing exception. If you are already using a Zod schema to build the JSON Schema sent to the model, as shown in the earlier example, you get this validation step almost for free since the same schema definition does double duty.

When schema-constrained output is not enough

JSON Schema and Zod both validate the shape and type of a single response. Neither one can express relationships between fields that only make sense together, or facts that live outside the response entirely. Plan for a second validation pass for cases like these.

• Business rules spanning multiple fields, such as an end date that must fall after a start date, which need a custom refine function rather than a plain schema.
• Cross-record checks, such as confirming a referenced order ID actually exists in your database, which only a real lookup can answer.
• Domain-specific numeric ranges, like a tax rate that is structurally a valid number but is nonsensical above a certain threshold for your business.