Intelligent Document Processing: Accuracy Is Not the Gate

Intelligent document processing is the practice of turning a document into structured fields a business system can act on: read the page, decide what kind of document it is, pull the fields, score how sure the machine is about each one, and only then write anything down. On ordinary business documents the reading is now good enough to stop being the constraint. The writing never was, and still is not.

That last clause is our position, and it is not the one the category sells. Almost every guide to intelligent document processing spends its length on the extraction stack, on optical character recognition and layout models and vision-language models, and treats the moment the extracted value lands in your accounting or HR system as plumbing. It is the opposite of plumbing. It is the only step in the pipeline that can pay an invoice twice, reject a mortgage application, or filter a job candidate out of a hiring process, and it is the step almost no vendor diagram scores.

Someone in the field said this more bluntly than we would. On Hacker News in June 2026, in a thread about why frontier models stall on hard documents, the commenter chelm — whose own commercial involvement in this market was raised and acknowledged further down the same thread — asked whether any benchmark measures business accuracy rather than character accuracy, observing that most "benchmarks focus on the charackter level" [sic]. Nobody in the thread named one. That gap, between a benchmark score and a process that can be trusted to commit, is what this article is about.

The short answer: Modern document AI reads well enough that extraction accuracy is rarely what stops a deployment; what stops it is the absence of a per-field confidence threshold, an exception queue with a named owner, and an audit trail linking every committed value back to the pixel it came from.

Last updated: July 30, 2026.

The extraction half of an intelligent document processing pipeline is well served by the market. The commit half is where deployments stall.

What intelligent document processing actually is

Intelligent document processing is a pipeline that converts documents (invoices, claims forms, contracts, shipping manifests, CVs) into validated structured data inside a system of record, using character recognition for the pixels, layout and language models for the meaning, business rules for the checking, and a human for the cases the machine is not sure about. The last clause is part of the definition, not an apology attached to it.

It is worth being precise about what the term is not, because the category has absorbed several adjacent things.

The reason this distinction has become urgent is that the model layer got good quickly and the surrounding layer did not. On the OmniDocBench benchmark, published at CVPR 2025 by Ouyang and colleagues, the leading document-parsing models now score above 95 on a composite of text edit distance, table structure similarity and formula matching. That is a page-parsing score, not a process score, and the distance between the two is the subject of the rest of this piece.

The six things that happen to a document

Every serious pipeline does six things in order: intake, normalisation, classification, extraction, validation, and commit. Vendors name them differently and some collapse two into one, but the sequence is stable, and knowing it lets you locate a failure precisely rather than blaming "the AI".

Intake is where the document arrives — a monitored mailbox, an SFTP drop, a scanner, an upload form, an API. It is unglamorous and it is where a surprising share of production incidents live: a supplier who starts sending a ZIP of TIFFs instead of a PDF, an email body that carries the real invoice while the attachment is a delivery note, a fax gateway that renders at 200 DPI on Tuesdays and 96 on Wednesdays.

Normalisation is the clean-up: deskewing, de-noising, resolving multi-page documents into logical units, splitting a 60-page scan batch into the eleven separate invoices it actually contains. Splitting is underrated. A pipeline that extracts perfectly from a wrongly-bounded document produces a confidently wrong record.

Classification decides what the document is. This is the step that determines which extraction schema and which business rules apply, and it is the step where a mistake is most expensive, because it silently routes the document into the wrong set of validations. A credit note classified as an invoice does not fail validation; it passes it, and pays money out.

Extraction pulls the fields. This is the part the market benchmarks and the part that has improved most. Modern approaches split roughly into pipeline systems (a detection model, then a recogniser, then a structure parser) and end-to-end vision-language models that emit structured output directly from the image. OmniDocBench evaluates both, across document sources that include handwritten notes and densely typeset newspapers, using 19 layout categories and 15 attribute labels.

Validation checks the extracted record against reality: does the line-item sum equal the stated total, does the vendor tax number exist in the master data, is the invoice date inside the accounting period, has this invoice number been seen before. Validation is arithmetic and lookups, not AI, and it catches a class of error that no amount of model improvement will.

Commit writes the record. Approve for payment, create the claim, advance the candidate, update the shipment. This is the irreversible step, and in most architectures it is a single API call with no more ceremony than any other.

If you want to watch practitioners work through the engineering of the extraction stages in detail, including why parsing was rebuilt rather than bought, this recorded Databricks session on their document-parsing function is a genuinely technical 46 minutes rather than a product tour.

What the benchmarks measure, and what they hide

Benchmark headline numbers describe how well a model reads a page in aggregate. They do not describe how often your process would have been safe to run unattended, and the gap between those two things is wide enough to sink a business case. The most useful thing you can do with a public benchmark is not read the top of the table but read the variance underneath it.

Two current benchmarks are worth knowing.

OmniDocBench contains 1,651 PDF pages spanning 10 document types, 5 layout types and 5 languages, with block-level and span-level annotations and both HTML and LaTeX ground truth for tables. Its composite score is defined openly as ((1 − text edit distance) × 100 + table TEDS + formula CDM) / 3. As of its v1.6 leaderboard, specialised document models sit at the top: PaddleOCR-VL-1.6 at 96.34 overall, MinerU2.5-Pro at 95.75, GLM-OCR at 95.22.

The IDP Leaderboard, maintained with the Indian Institute of Technology Indore and sponsored by Nanonets, evaluates vision-language models across document tasks: OCR, key information extraction, table extraction, visual question answering. Checked on 30 July 2026, its overall leader was Nanonets OCR-3 at 85.9, with the best table-extraction score at 96.4 and the best OCR score at 82.8.

Notice that the two leaderboards disagree about which task is hard. That is not a contradiction; it is the point. Aggregate scores are averages over a distribution of documents, and your document population is not that distribution.

The sharpest illustration sits inside OmniDocBench's own table-recognition subset, which reports scores split by "special situation": whether the table has merged cells, contains formulas, is coloured, or is rotated. The rotated column is where aggregate accuracy goes to die.

Component-level table recognition on the OmniDocBench v1.0 table subset. "Rotated" and "not rotated" are the benchmark's own (+/-) split for that attribute.

Six models, one physical property of the page, and accuracy falls by between 46 and 68 points. Nothing about the model changed. Somebody fed the scanner a page sideways.

This is the single most useful thing a buyer can take from public benchmarks: the variance across document attributes is larger than the variance across vendors. If your supplier base includes one manufacturer whose packing lists come in landscape, your realised accuracy on that supplier bears no relationship to the number in the sales deck. The defence is not a better model. It is a confidence score that notices, and a gate that stops.

There is a second, quieter reason to distrust an aggregate: benchmarks score character or structure fidelity, while your process cares about whether a specific field was right. A page can be transcribed at 98% character accuracy and still have the total wrong, because the wrong 2% was the total. This is exactly the objection raised in that Hacker News thread, that the industry measures reading and not process outcomes. As of this writing there is no widely-adopted public benchmark of business-level field accuracy to point at instead.

One note on what we excluded. Vendor marketing pages routinely quote 95–99% field-level accuracy for intelligent document processing, and several appear high in search results for this topic. We could not trace any of those figures to a published methodology, a dataset, or an independent evaluation, so they are not cited here. Treat them the way you would treat a fuel-economy figure with no test cycle attached.

Field-level confidence is the number that runs the process

A document pipeline is only automatable to the extent that it can tell you, per field, how much to trust itself. Document-level accuracy — "94% of documents were processed correctly" — is a reporting metric. Field-level confidence is a control signal: it decides, for each value, whether the machine commits, asks a human, or refuses.

The mechanism is simple and almost universally under-specified in buying conversations. Each extracted field carries a score. You define bands. Above the upper band, the field commits automatically. Between the bands, the document routes to a human review queue with the low-confidence fields highlighted. Below the lower band, the document is rejected back to the sender or escalated to a specialist. Nothing about that is novel. What is routinely missing is any evidence that the score means anything.

Confidence bands turn a model output into a process control. The bands are a policy decision, not a model setting.

Calibration is the question to ask. A confidence score of 0.95 is only useful if values scored 0.95 turn out to be correct roughly 95% of the time. A score that is systematically over-confident does something worse than being useless: it launders bad extractions into the auto-commit band, where nobody looks at them. You can measure this yourself in an afternoon with a few hundred labelled documents. Bucket the fields by reported confidence, compute realised accuracy per bucket, and plot the two against each other. If the curve sags below the diagonal, your thresholds are fiction.

Three practical rules follow, and they are worth more than any vendor comparison.

Set thresholds per field, not per document. The tolerance for a vendor name is not the tolerance for a bank account number. A misread supplier address costs a phone call; a misread IBAN costs the payment. Fields that feed an irreversible action deserve a threshold high enough that most of them go to a human, and that is an acceptable outcome.

Set thresholds per document class, not globally. Utility bills from four suppliers with stable templates behave nothing like scanned bills of lading from a freight forwarder. A single global threshold guarantees that you are simultaneously too strict on the easy class and too loose on the hard one.

Re-derive thresholds when anything upstream changes. A model version change, a new scanner, a supplier switching to a new invoicing system. Each invalidates the calibration you measured. This is the reason a document pipeline is an operated system rather than an installed one.

There is a governance consequence here that most treatments skip. Once thresholds decide whether a human sees a value, the threshold is the control. If an auditor asks how you ensure a person reviewed material financial data, the answer is a number in a config file, and you had better be able to show what it was on the date in question and who last changed it. Thresholds belong under change control, with the same seriousness as an approval limit in your finance system.

The exception queue is the product

Everything the machine is confident about is cheap. The economics of intelligent document processing are decided almost entirely by what happens to the documents it is not confident about, and that queue is the part of the system buyers evaluate least and live with most.

Three things determine whether an exception queue works.

A named owner per document class. Not "the ops team". A person, with a backup, whose queue depth is visible and whose service level is written down. Unowned exception queues grow until someone bulk-approves them, which converts a careful pipeline into an expensive rubber stamp.

A review interface that shows provenance. The reviewer needs the extracted value, the confidence, and the region of the page it came from, side by side. If the reviewer has to open the source PDF in another window and hunt for the number, review time triples and accuracy of the review itself falls. Provenance is not a nice-to-have for auditors; it is the thing that makes the human step fast enough to be affordable.

A feedback path that is actually connected. Corrections made in the queue should flow back as training or rule updates on a defined cadence. Many deployments have a correction UI and no loop behind it, which means the same supplier's malformed date format is corrected by hand every month for two years.

The cost model that matters looks like this, using assumed figures you should replace with your own. We have not measured these, and the review time in particular varies enormously with the quality of the interface. Suppose a pipeline auto-commits 80% of documents and routes 20% to review, and a reviewer handles a flagged document in three minutes. At 50,000 documents a month, that is 10,000 reviews, 500 hours, roughly three full-time equivalents. Push the auto-commit rate to 90% and you halve that. Push the threshold up for safety and you may double it. The threshold setting and the headcount are the same decision, made once, usually by someone who does not know they are making it.

This is also where the "AI replaces data entry" framing breaks down honestly. What changes is not that the humans leave; it is that their work moves from transcription to adjudication. That is a genuine improvement, since adjudicating 10,000 flagged fields is better work than typing 50,000 clean ones, but it is a different job, with different hiring and different training, and pretending otherwise is how implementations lose the operations team's goodwill in month two.

One invoice, end to end

Abstractions about pipelines are easy to agree with and hard to act on, so here is a single worked example, assembled the way an actual record moves. The document is a supplier invoice arriving by email; the destination is an ERP that will schedule a payment. Every number below is illustrative and chosen to show the mechanism, not measured from a live system.

Stage 1 — Intake. The message arrives at ap-invoices@. The pipeline records the sender, the message ID, the received timestamp, and stores the original attachment unmodified. That unmodified original is the evidentiary anchor for everything that follows; if it is ever regenerated or "cleaned" in place, the audit trail is broken at its root.

Stage 2 — Normalisation. The four-page PDF is deskewed and split. The splitter determines it is one invoice plus a two-page delivery note, and emits two logical documents with a recorded parent-child relationship.

Stage 3 — Classification. The first logical document is classified supplier_invoice at 0.97; the second delivery_note at 0.91. The invoice schema and the accounts-payable rule set are attached.

Stage 4 — Extraction. The extractor returns a record with per-field confidence:

Stage 5 — Validation. Deterministic checks run before any human sees anything. Line items sum to 6,423.00; tax at 20% would be 1,284.60, which matches; 6,423.00 + 1,284.60 = 7,707.60, which matches the extracted total. The supplier tax ID matches master data. The invoice number has not been seen before. Three of the four validations pass; the arithmetic reconciles.

Note what just happened. Validation raised the effective confidence of tax_amount and total_amount far above their extraction scores, because two independently-extracted quantities agreed with each other through a rule. This is the cheapest accuracy improvement available in document processing and it involves no machine learning at all. A pipeline that routes on raw model confidence, without letting deterministic checks adjust it, sends far more documents to humans than it needs to.

Stage 6 — Gating. bank_account at 0.81 remains below threshold, and it is the field with the highest blast radius on the document. It is also a field where a validation rule exists that the pipeline should run and most do not: compare against the bank details held for this supplier in master data. They differ. The account number does not match the one on file.

This is the case that justifies the entire apparatus. A confident, well-read, arithmetically perfect invoice with a changed bank account is the standard shape of an invoice-redirection fraud. No extraction accuracy improvement catches it. A gate does.

Stage 7 — Human decision. The document enters the review queue flagged bank_details_changed. The reviewer sees the cropped region of the page next to the stored master-data value, calls the supplier on a number from the master record rather than from the invoice, and either confirms or rejects. The decision, the reviewer's identity, the timestamp, and the reason code are written to the record.

Stage 8 — Commit and retain. On approval, the ERP call is made. The stored record links: original file hash, page region for each field, model and version, confidence per field, validation results, threshold values in force at that moment, reviewer identity and decision, and the ERP transaction ID returned. Retention clocks start: one for the source document, one for the extracted personal data inside it, and they are usually not the same clock.

That final list is the deliverable that separates a demo from a deployment. If your pipeline cannot produce it for an arbitrary document from four months ago, you do not have an auditable process; you have an automation with good intentions.

What AI document processing actually costs

Per-page extraction pricing from the three major clouds is public and low; the cost that decides your business case is the exception labour, which is not. Below are list prices fetched from the vendors' own pricing pages on 30 July 2026, pay-as-you-go, first tier, before any negotiated discount.

List prices from AWS Textract pricing, Google Document AI pricing and Azure AI Document Intelligence pricing, fetched 30 July 2026. Regions and tiers differ; check your own region before modelling.

The table is cloud-only for a reason worth stating: the specialist intelligent document processing vendors, the ones that would sit alongside these in a shortlist, do not publish per-page list pricing, so there is nothing to fetch and quote. Any comparison you build will be three transparent prices against several quoted ones, which is itself a negotiating fact.

Two observations a buyer should take from that table.

Extraction is not the expensive part. At 50,000 documents a month averaging two pages, a prebuilt invoice model costs on the order of $1,000 a month at list. The three FTEs working the exception queue in the example above cost between one and two orders of magnitude more. Any evaluation that agonises over per-page price differences while ignoring the auto-commit rate is optimising the small number.

The pricing tiers encode a real architectural choice. Plain OCR is roughly twenty times cheaper than custom extraction across all three vendors. That spread is why mature pipelines classify first and route cheaply: send the 70% of documents that match a known template through a cheap path, and reserve the expensive extractor for the tail. It is the same logic as routing model traffic by task rather than sending everything to the largest model, which is a pattern we have written about at length in our analysis of model routing and LLM cost.

The costs that never appear in a vendor comparison, and that decide the programme, are these:

The compliance layer nobody draws in the pipeline

Document pipelines are the highest-consequence place AI touches regulated data, because documents are where personal data, financial commitments and contractual obligations all physically live. Two regimes matter for almost every European deployment, and both bite at the commit step rather than the extraction step.

The EU AI Act's high-risk classification is triggered by what the document decides, not by how clever the model is. Annex III lists AI systems intended for "the recruitment or selection of natural persons, in particular to place targeted job advertisements, to analyse and filter job applications" under employment, and systems "intended to be used to evaluate the creditworthiness of natural persons or establish their credit score" under access to essential services, per the consolidated Annex III text. A CV parser that filters applicants and a document pipeline that extracts income evidence for a loan decision are both squarely inside that list. An invoice pipeline is not. Same technology, different regulatory weight, and the distinction is made by the downstream decision, which is to say by the commit.

Where the Act applies, Article 14 requires that high-risk systems "be designed and developed in such a way ... that they can be effectively overseen by natural persons during the period in which they are in use", and that the overseeing person can correctly interpret the output, "decide, in any particular situation, not to use the high-risk AI system or to ... disregard, override or reverse the output", and intervene or interrupt the system. Read that against a document pipeline and it stops being abstract: the exception queue, the review interface that shows provenance, and the ability to halt a batch are not operational niceties. They are the mechanism by which the obligation is met. For how the timeline phases in for organisations that deploy rather than build AI, see our earlier analysis of EU AI Act obligations for deployers.

GDPR applies whether or not the AI Act does. Article 22(1) gives a data subject "the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her", per the text of Article 22. A fully automatic reject of a claim or an application on extracted evidence is the paradigm case. A confidence gate that routes adverse outcomes to a human is one of the ways the "solely automated" characterisation can be avoided, but only if the human involvement is meaningful. A reviewer who approves a queue of decisions they cannot realistically assess is nominal involvement, and we would not expect it to survive scrutiny. This is a second reason the review interface matters: a person who can see the source region, the confidence and the failed check is exercising judgement; a person clicking through a list of totals is not.

Two more GDPR principles land directly on document pipelines and are almost never in the architecture diagram. Article 5 requires personal data to be "adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed", and "kept in a form which permits identification of data subjects for no longer than is necessary". In pipeline terms:

There is a third question that belongs here and rarely gets asked in procurement: where do the documents go, and what happens to them there? Every extraction API call sends the document, or a rendering of it, to a third party. The questions with real answers are whether content is retained after processing and for how long, whether it is used to improve the provider's models, whether processing is regionally pinned, and whether the provider can produce a deletion attestation. Those answers vary by vendor, by product within a vendor, and by contract tier, and they change. Get them in writing, dated, for the specific product and region you are buying, because a general statement on a trust page is not the same as a term in your agreement.

The same reasoning applies to the personal data inside extracted fields, which is the part that escapes document-level controls entirely. A name and an account number sitting in a JSON payload in a queue is personal data in a place your document retention policy has never heard of. Masking sensitive values on the way through — before they reach a model provider and before they land in downstream logs — is the structural version of this control, and it is the same problem we described in our work on masking PII at the gateway.

The Commit Test: five questions before anything writes

We call this the Commit Test because it is deliberately narrow: it ignores the model entirely and asks only about the moment a value becomes an action. It is five questions, answerable in one sitting with the people who own the process, and any pipeline that cannot answer all five is not ready to run unattended.

1. What is the highest-consequence field on this document, and what is its threshold? Not the average field. The one that moves money, grants access, or changes a person's status. If the answer is "we use the same threshold for everything", the pipeline is calibrated for the wrong risk.

2. Who adjudicates a low-confidence extraction, by name, and what is their service level? An exception queue without an owner and a target is a backlog with a nicer interface.

3. Which validations run before a human is asked, and which run after? Deterministic checks that reconcile independently-extracted values should run first, because they remove most of the queue for free. Checks against master data (bank details, tax IDs, contract terms) should run before commit regardless of confidence, because they catch the failures confidence cannot see.

4. Can you reconstruct, for an arbitrary document from six months ago, the full chain from source pixel to committed value? Model and version, confidence, thresholds in force, validations run and their results, the reviewer's identity if there was one, and the transaction ID it produced. If any link is missing, the audit answer is a narrative rather than a record.

5. What can this pipeline's credentials reach, and does that access expire? The extracting service holds credentials to a mailbox, a storage bucket and an ERP endpoint. Those are the credentials of a non-human actor. If they are a long-lived key in a config file with broader scope than the process needs, the blast radius of a compromised pipeline is the whole document estate — a problem we have argued belongs to identity rather than to security monitoring in our piece on owner, scope and expiry for non-human identities.

Run the five questions against a candidate deployment and the conversation changes shape. Vendors answer the model questions well and these questions unevenly, which is itself the most informative result of the exercise.

The Commit Test ignores model quality entirely and interrogates the moment a value becomes an action.

Where this gets it wrong

We have argued that the commit gate matters more than extraction accuracy. That claim has real limits and it is worth being explicit about them, because a reader who accepts it uncritically will over-invest in governance for a process that did not need it.

For low-stakes, high-volume, reversible work, the argument is weak. Indexing a document archive for search, routing scanned correspondence to the right department, tagging contracts by type. These commit almost nothing irreversible, and a lightweight pipeline with a coarse threshold is the right answer. Applying the full apparatus here is expensive theatre.

Extraction quality still sets the ceiling. If a model reads a document class badly, no gate rescues the economics; it just converts a wrong-answer problem into an everything-goes-to-humans problem. On genuinely hard classes such as dense handwriting, poor scans, unusual layouts and the rotated tables in the benchmark above, a better model is the correct investment and a threshold is a consolation prize.

Confidence scores are not comparable across vendors, or sometimes across model versions from one vendor. They are not probabilities in a shared sense. Treat a score as a vendor-specific signal you must calibrate locally, and be sceptical of any evaluation that compares thresholds between products as though they meant the same thing.

We have not run this pipeline ourselves. LeapForce does not sell a document extraction product and we did not build and measure one for this article. Everything above about pipeline mechanics is drawn from published benchmarks, vendor documentation and pricing fetched on the dates stated, practitioner discussion cited inline, and our own work on the governance layer around AI systems. The worked example is illustrative and labelled as such; the confidence figures in it are not measurements. Where we would most want first-hand data and do not have it is the relationship between threshold settings and realised exception volume in production, which is exactly the number a buyer needs and which no public source we found reports.

Open questions we could not close. There is no widely-adopted public benchmark for business-level field accuracy, which means cross-vendor claims about "process accuracy" cannot currently be checked by anyone. The relationship between benchmark performance and realised accuracy on a specific document population is undocumented in any source we could verify. And the retention and training-use terms of the major extraction APIs vary enough by product and contract that no general statement about them is safe to publish.

Governing the pipeline, not just building it

LeapForce does not sell intelligent document processing. We build the layer underneath it: one governed path for every AI call, connector and agent your company runs, so that a document pipeline is a governed company system rather than a script with credentials.

Three things in the analysis above are the layer we actually own. The extracting service is a non-human actor, and Access & Identity treats it as one: an owner, a scope, an expiry, and one revocation that removes its reach across every connected system rather than a key someone has to remember to rotate. Its access to mail, storage and the ERP runs through Connectors that IT vets once and scopes at the action level, with human-in-the-loop gates on the actions that commit. And every call it makes to a model provider passes through the AI Gateway, where the request is identified, checked against policy, routed, and recorded against a team and a budget in dollars.

Our rollout guidance for that layer is deliberately unheroic and it is written on the gateway page as "Observe first. Enforce second. Optimize third." Point traffic at the gateway in observe mode until you know what is actually running, turn on the protections that prevent incidents, then apply role-based rules learned from what you saw. Applied to a document pipeline, that means you instrument before you restrict, which is also the only way to learn what your real auto-commit rate is.

One honest caveat, and only one. Several capabilities that sit closest to this problem are not uniformly shipping: our gateway endpoints, tracing and SSO are live, while data-loss-prevention masking, vaulted provider keys and dollar budgets are in development, and shadow-AI discovery and compliance-evidence packs are on the roadmap. Our site labels build status per capability for exactly this reason, and you should hold any vendor in this space — including us — to the same disclosure before you design a control around a feature.

Ready to Govern Your AI?

Talk to LeapForce — one controlled layer for every AI tool, connector, model, and agent.