Document Data Extraction with Multimodal LLMs

For a decade, extracting structured data from documents meant training a specialized OCR pipeline, hand-tuning template matchers, and accepting that anything outside the training distribution would silently fail. Multimodal large language models changed the economics of this problem. A model that can read a page the way a person does, layout, tables, handwriting, stamps, and all, can go from an unseen invoice format to a validated JSON record in a single API call, with no per-vendor template to maintain.

This post walks through the practical side of building that pipeline: how to frame the extraction task as a schema-constrained tool call rather than free-form prose, how to ground every extracted value in a location on the page so a reviewer can verify it in seconds, how to expose model confidence rather than hide it, and how to design the human review step so it scales instead of becoming a bottleneck.

Why multimodal beats text-only OCR pipelines

Traditional OCR converts pixels to a flat string of characters and discards the two-dimensional layout that gave those characters meaning. A total that sits in the bottom-right cell of a table looks identical, as a string, to a stray page number. Multimodal LLMs instead reason over the rendered image directly, so column alignment, cell borders, indentation, and font weight all remain available as evidence. That matters most on the documents that break classic pipelines: rotated scans, multi-column layouts, handwritten annotations over printed text, and stamps or signatures overlapping a field.

• Layout is preserved as visual evidence, so a model can distinguish a subtotal from a total by position and formatting, not just by keyword proximity.
• One model handles typed text, handwriting, tables, checkboxes, and logos in the same pass, replacing several specialized OCR and classification models.
• New document formats need a new prompt or schema, not a new training run, which collapses onboarding time for a new vendor or form type from weeks to hours.

Prompting: constrain the output before you ask for it

The single biggest reliability gain in production extraction pipelines does not come from a clever prompt, it comes from refusing to accept free-form prose as the output format at all. Instead of asking the model to describe an invoice and then parsing that description with regular expressions, define a strict JSON schema up front and force the model to populate it through a tool call. This turns extraction into a typed function call: the model either returns a value that satisfies the schema or it does not return at all, which eliminates an entire category of downstream parsing bugs.

// invoice-extraction-tool.ts
const invoiceTool = {
  name: "record_invoice_fields",
  description: "Record extracted invoice fields with page-relative bounding boxes",
  input_schema: {
    type: "object",
    properties: {
      vendor_name: { type: "string" },
      invoice_number: { type: "string" },
      invoice_date: { type: "string", format: "date" },
      total_amount: { type: "number" },
      currency: { type: "string" },
      line_items: {
        type: "array",
        items: {
          type: "object",
          properties: {
            description: { type: "string" },
            quantity: { type: "number" },
            unit_price: { type: "number" },
            bbox: {
              type: "array",
              items: { type: "number" },
              minItems: 4,
              maxItems: 4,
            },
          },
          required: ["description", "unit_price"],
        },
      },
      confidence: {
        type: "object",
        properties: {
          vendor_name: { type: "number" },
          total_amount: { type: "number" },
        },
      },
    },
    required: ["vendor_name", "invoice_number", "total_amount"],
  },
}

// Force the model to call this tool instead of replying in prose
const response = await client.messages.create({
  model: "claude-opus-4-8",
  max_tokens: 2048,
  tools: [invoiceTool],
  tool_choice: { type: "tool", name: "record_invoice_fields" },
  messages: [
    {
      role: "user",
      content: [
        { type: "document", source: { type: "base64", media_type: "application/pdf", data: pdfBase64 } },
        { type: "text", text: "Extract every field defined in the tool schema. If a field is unreadable, omit it rather than guessing." },
      ],
    },
  ],
})

Two details make this reliable in practice. First, mark fields that are genuinely optional as optional in the schema, and instruct the model explicitly to omit a field rather than invent a plausible-looking value when it cannot read something clearly, since a null is far cheaper to catch than a hallucinated number. Second, keep the schema itself in plain business language, a currency field description that says three-letter ISO currency code reads more predictably to the model than a bare enum with no description at all.

Grounding: link every value back to the page

A JSON object full of correct-looking numbers is worth very little to a reviewer who cannot verify where each number came from. Grounding closes that gap by having the model return a bounding box, or at minimum a page number and approximate region, alongside every extracted field. When a reviewer opens the record, the interface highlights the exact rectangle on the source image that produced the total amount or the vendor name, turning a five-minute manual re-read of the whole invoice into a two-second glance at one highlighted region.

Confidence handling: make the model tell you what it is unsure about

A model that returns a total amount with no indication of certainty forces every field into the same review lane, whether it was crisply printed or smudged under a coffee ring. Asking the model to self-report a confidence level per field, alongside the extracted value, lets a pipeline route records automatically instead of sending every single one to a human. In practice this looks like a short, staged flow.

1. Extract every field through the schema-constrained tool call, including a per-field confidence score from the model itself.
2. Auto-approve records where every required field reports high confidence and passes basic validation, such as a total that matches the sum of line items.
3. Route records with any low-confidence field, or a failed cross-check, straight into a human review queue with the offending field pre-highlighted.
4. Log every human correction back to a dataset. Recurring corrections on the same field or vendor are a signal to adjust the prompt or schema, not just to keep correcting by hand forever.

Designing the human-in-the-loop review step

The review interface is not an afterthought bolted onto the extraction pipeline, it is the component that determines whether the whole system is trustworthy enough to run unattended most of the time. A good review screen shows the source document image on one side and the editable extracted fields on the other, with the grounded region highlighted the moment a reviewer focuses a field. Corrections should feed back into evaluation, not disappear after the record is approved.

• Show the original page image alongside the extracted fields, never the extracted data alone, so a reviewer is verifying against the source rather than trusting the model's own summary.
• Surface confidence scores visually, for example a colored border on low-confidence fields, so a reviewer's attention goes straight to the fields that actually need scrutiny.
• Store every correction with the original model output attached, so the diff between what the model said and what the human corrected becomes a growing evaluation set for prompt and schema changes.

Evaluating and iterating on the pipeline

Extraction accuracy is not one number, it is a per-field accuracy rate measured against a held-out set of documents a human has already labeled by hand. Track it per field rather than as a single aggregate, because a pipeline that gets vendor names right ninety-nine percent of the time and line-item totals right eighty percent of the time has a very different failure mode than one where every field sits at ninety percent, and the fix for each is different. Re-run this evaluation whenever you change the prompt, the schema, or the model version, since a version upgrade that improves one field can quietly regress another.

Multimodal LLMs do not remove the need for careful engineering around a document extraction pipeline, they change where that engineering effort goes. Instead of maintaining template matchers per vendor, the work shifts to schema design, grounding, confidence-based routing, and an evaluation set that keeps the whole system honest as documents and models both keep changing.