Scaling Document AI: Building and Evaluating KIE Solutions

This article details building and evaluating Key Information Extraction (KIE) solutions at scale using Amazon Bedrock‘s Nova models. The process is divided into three phases: data readiness, solution development, and performance measurement. Data readiness involves preparing documents, understanding their structure, and handling inconsistencies like missing or inconsistently formatted data. The solution development phase utilizes Amazon Bedrock‘s Converse API and LangChain's PromptTemplate for streamlined interaction with foundation models. Prompt engineering is highlighted as crucial, using Jinja2 for creating flexible, model-agnostic prompts that handle various input modalities (text, image, or both). The performance measurement phase emphasizes a holistic approach, considering not only F1-score for extraction accuracy but also latency and cost per document. The FATURA dataset, consisting of 10,000 invoices, is used for benchmarking. The evaluation compares two Nova models (Lite and Pro), analyzing F1 scores, latency, and costs across different input modalities. The results show that while the larger Nova Pro model offers higher accuracy, the smaller Nova Lite provides a cost-effective alternative with acceptable accuracy. The article concludes by stressing the importance of balancing accuracy, speed, and cost when choosing a model and emphasizes the need for organizations to conduct similar evaluations using their own data to determine the optimal configuration for their specific needs. The authors highlight the potential for future work to explore fine-tuning models for improved performance on specialized use cases.

(Source: https://aws.amazon.com/blogs/machine-learning/document-intelligence-evolved-building-and-evaluating-kie-solutions-that-scale/)