Intelligent document processing is no longer smart enough: Why adaptive is the future of IDP
By Jos Polfliet, Chief Architect.
Financial services firms deserve tools to help them take a holistic approach when dealing with data. But the operations and finance landscape is littered with point solutions. While often good at solving one particular issue, these are unable to cope with the five core data challenges: variety, change, scale, lifecycle and control.
Intelligent Document Processing (IDP) solutions are no exception. There are many platforms using artificial intelligence to automate the extraction of unstructured data from documents. However, they are usually narrow in focus, with the AI only able to recognise and interpret data in one particular document type – e.g. tax documents, fund allocation statements, or credit agreements. This also extends to the document’s layout, meaning that if anything changes, these solutions can’t keep up.
So, in an era where change is the only constant, a new variant of IDP is clearly needed in order to help firms unlock true data automation – adaptive IDP. This allows them to replace a variety of point solutions with one platform for both simple and complex document processing use cases.
Let’s take a closer look at what intelligent document processing is, how most platforms work and why this falls short of what your business needs.
Already know about IDP? Click here to jump straight to the section on why adaptive is a vital trait for document processing.
What is Intelligent Document Processing?
There are many ways to deploy AI along the process lifecycle to automate data. The technology is helping firms to remove manual work at all stages of data management. IDP is at the frontline of this, tackling one of the biggest challenges firms have: how to extract and process mission-critical data in unstructured formats like PDFs, emails, images, faxes and more.
IDP uses multiple artificial intelligence technologies to automatically classify and extract information from documents. Machine learning (ML) is the process of using patterns in data to ‘teach’ the machine, so its performance and prediction become more effective and accurate over time. Natural language processing (NLP) is the branch of AI focused on leveraging ML techniques so the machine can understand and interpret human language.
So, in the case of Intelligent Document Processing, machine learning and natural language processing are used to train a computer to simulate a human subject matter expert’s review of a set of documents.
Intelligent Document Processing is no longer smart enough
These solutions have enabled firms to automate some of the manual work around document processing.
But the scope of what traditional IDP solutions need to do vastly outweighs their capabilities. A platform may be excellent at automating one particular type of document, but that’s it. It can’t adapt to changes in document format or fields, it can’t read and understand different document types and it continues to repeat the same mistakes because it lacks the capacity to learn from them.
This is because their models are trained on a specific set of documents, e.g. credit statements. The AI model knows what a credit statement ‘should’ look like and where all the important information is. As soon as the format of that document changes, important information moves, or you have a need for additional information, the model is stymied.
This wouldn’t be a problem if the model was able to learn, but regular IDP solutions run on black box, preconfigured models and training them is something only highly-technical people, such as your IT department, consultants or the vendor can do. Without that time-consuming and expensive support, your regular IDP solution will continue to make the same mistakes over and over again.
Operations and finance teams therefore often end up back where they started: having to manually extract data, or intervene to confirm the AI has made the right decisions. Straight-through-processing may be higher thanks to the IDP platform or platforms, but there is still a lot resting on ‘Human APIs’.
Put simply, traditional IDP solutions fall down because their artificial intelligence can’t adapt.
What makes an IDP platform ‘adaptive’?
The significant difference between regular IDP and adaptive IDP solutions lies in their ability to learn and adapt quickly. While regular IDP often performs admirably out-of-the-box, it’s static performance or slow learning can become a sticking point.
In contrast, adaptive IDP quickly learns from errors, because the action a user takes in response to something the model has flagged for review is fed back into the model to help it better understand what it should have done. This enables the AI to avoid the repetitive cycle of making the same mistakes over and over.
More impressively, adaptive IDP is multilingual and can adapt to new languages, countries, layouts, and external data changes, making it a more flexible and globally viable solution.
Finally, the potential of adaptive IDP goes beyond merely learning from mistakes. It empowers businesses to add new document types and fields as needed. This level of flexibility is a breath of fresh air, especially when compared to the rigid structures of regular IDP that make it hard, or even impossible, for users to add custom fields or document types.
What does this all mean for you? Let’s take a look at three key benefits of adaptive IDP versus regular IDP.
The benefits of adaptive Intelligent Document Processing
1. Learning fast means higher automation rates
The most important aspect of any IDP solution is its automation rate in production settings. While regular IDP solutions often deliver good performance out-of-the-box, their AI tends to stagnate at this level, is slow to learn, or even stops improving at all.
Adaptive IDP, on the other hand, starts at a medium-to-good performance level, but continuously learns from new edge cases and therefore rapidly improves to deliver excellent performance in production. This vastly outperforms static solutions very quickly, with or without a fitting base AI model:
- Where a base model is available, you get out-of-the-box accuracy PLUS fine-tuning to your specific data, quickly outperforming static solutions.
- Where a base model is not available, the model learns incredibly quickly via a process called “few-shot learning”.
2. Flexibility in fields, documents, or languages
Your business is unique. There’s an ever-changing list of information you need from documents that don’t fit the mould. But regular IDP often forces you to use:
- A fixed set of supported fields
- A fixed set of supported document types
- A fixed set of languages
An adaptive IDP solution allows you to train your own custom models, even if the set of fields to extract doesn’t match.
Some regular IDP solutions will allow you to create a custom model from scratch, but that means you also have to annotate from scratch and lose the accuracy of the already supported entities! Annotating is the process of selecting and labelling the different pieces of information in the document so that the model can learn what they look like and where they’re located.
Clearly, that’s not what a good adaptive IDP solution should look like. You want the flexibility to pick your own entities and, if they are known, just use whatever is already good.
3. Have all the info and tools to improve your AI models
As Peter Drucker’s axiom goes, “If you can’t measure it, you can’t improve it“. To find how you can improve the data for your AI models, you need deep insights into:
- Current model performance
- Advanced model analytics
- Production evaluation results
If you know which fields or document types are underperforming, you can make targeted improvements.
An adaptive IDP solution should continuously help you with improving your training data by:
- Suggesting fixes to annotation mistakes
- Suggesting which documents to annotate
- Creating custom QA tasks based on specific entities, annotators and so on
When you have deep access to all relevant statistics and have improved your training data, the only thing you then need is to experiment with different base models, or train specific models on different subsets of your training data until you get the best working model.
The future is adaptive
Adaptive IDP harnesses the latest AI tools and techniques, along with user-friendly no-code technology, to help firms tackle the challenge of automating unstructured data. This is a vital hurdle to cross in order to unlock the full benefits of data automation.
Traditional IDP solutions have removed some of the manual effort around document processing. But adaptive IDP offers businesses a level of flexibility, learning, and adaptability that’s in sync with the dynamic digital age. It provides a holistic, flexible and user-driven solution to the challenges of processing documents in an everchanging business landscape.