Every document capture project gets asked the same question: how accurate is it? The real answer depends on what you measure, on which documents, and how. A single accuracy figure on a slide tells you very little. Here is what to measure and how to measure it so the figure means something.
Accuracy of What?
Three measures matter, and they answer different questions.
- Field accuracy is the share of individual fields extracted correctly. If 1,000 invoices each have 10 fields and 9,700 of the 10,000 fields are right, field accuracy is 97%.
- Document accuracy is the share of documents with every field correct. The same 97% field accuracy can mean only about three in four documents are fully correct, because the errors are spread across many documents.
- Straight-through processing is the share of documents that go from capture to the business system with no person touching them. This is the figure that drives cost and staffing.
A project that reports only field accuracy is reporting its best number. Ask for all three.
Not All Fields Are Equal
An invoice number read wrong can mean a duplicate payment. A misread line description may not matter at all. Measure accuracy field by field, and set targets by the cost of an error rather than one target for everything. The fields that move money or trigger decisions get the tightest targets and the most review.
Build a Ground-Truth Set
You cannot measure accuracy without knowing the right answer. A ground-truth set is a sample of documents where people have recorded the correct value of every field, checked by a second person. Everything else is measured against it.
The set has to look like production: the same mix of document types, the same senders, the same scan quality, the faxes, the phone photos and the handwritten notes in the margin. A set built from clean samples produces figures that fall apart in the first week of live operation. Draw it from real incoming documents, across enough weeks to catch seasonal and supplier changes, and keep out any document used to train or tune the system.
Size matters too. A few dozen documents will not tell you much about a field that appears on only some of them. Make sure each important field and each document type has enough examples to give a reliable figure.
Confidence Scores and Thresholds
Most extraction tools give a confidence score for each field. The score is useful, but it is not accuracy. A system can be confident and wrong. Use the ground-truth set to see how often fields at each confidence level are actually correct, then set the threshold for straight-through processing based on what an error in that field costs.
Setting the threshold is a business decision. A higher threshold sends more documents to people and catches more errors. A lower one saves staff time and lets more errors through. The right setting balances the cost of review against the cost of errors, and it is different for every field.
Measure the Whole Process
Recognition is only one step. Documents also have to be separated correctly, classified correctly and delivered to the right place once and only once. A perfect extraction from a document classified as the wrong type is still a failure. Measure each step, so when the overall figure drops you know where to look.
Measure the people too. Reviewers who see mostly correct values start approving without checking. Seed review queues with known errors from time to time and see whether they are caught.
Keep Measuring After Go-Live
Documents change. Suppliers redesign their invoices, a new form version appears, a scanner gets replaced. Accuracy drifts down unless someone is watching. Sample production documents every month, check them against human review, and track the three measures over time. Corrections made by reviewers are valuable data; capture them so the system can improve.
Intelligent document processing works well when it is measured properly. It takes the mundane, repetitive keying off people's desks and leaves them the cases that need judgment. Getting there means knowing your numbers. The Certified AI Intelligent Document Processing Specialist (AI-IDP) program covers measurement, review design and the rest of the capture process in depth.
Follow IIAAIT on LinkedIn for new articles.