Across industries, businesses constantly struggle to extract valuable insights from complex documents such as financial reports or research papers. These documents often contain a mix of dense text, tables packed with data, and intricate charts. Manually pulling insights from such material is not only tedious but also slows down decision-making, introduces risk, and can limit a company’s ability to stay ahead in fast-paced markets.

Now, imagine a solution that automates this process. Unstructured and semi-structured documents can be instantly transformed into actionable data, and your team can easily change the focus of analysis. This is the power of next-generation data curation, which is fundamentally changing how businesses approach information analysis and reporting.

Consider the pharmaceutical industry as an example. Picture a data analyst at a leading pharma company responsible for reviewing each month dozens of research papers describing results of clinical trials. These reports are filled with highly technical text, detailed tables, and a variety of charts. In the past, the analyst might have spent weeks extracting, organizing, and interpreting this information before presenting it to decision-makers. This manual approach is slow and increases the risk of overlooking critical information.

We developed a Hybrid AI solution that addresses these challenges. By integrating the robust analytical platform PolyAnalyst with advanced Generative AI models, such as ChatGPT and Claude, we have created a solution that automates and accelerates the entire data curation process. Organizations in finance, pharma, life sciences, academia, and other sectors can now analyze complex documents with ease. Vast volumes of unstructured information get converted into actionable intelligence in a fraction of the usual time, requiring little human intervention.

Figure 1. AI Data Curation solution overview

How It Works

AI Data Curation solution uses a streamlined workflow organized into three main stages:

Stage 1: Converting documents to machine readable form

1. Object separation:
   When the user uploads documents to the platform, PolyAnalyst initiates the analysis workflow. Using a Vision-Language Model (VLM), the system detects and separates plain text, tables, and charts on each page.

   Figure 2. Detecting objects on a page

    

2. OCR:
   Neural network-based OCR engine converts textual and tabular content into machine-readable form.
3. Figure Interpretation:
   A VLM interprets graphical figures, such as line charts or bar charts and converts them into structured data tables, retaining their positions in the original document.

Figure 3. Converting chart to table

Stage 2: AI-driven Analysis

4. Stating Facts of Interest:
   The user describes target facts of interest through a simple interface.
5. Extracting facts of interest:
   An LLM finds and extracts facts specified by the user.

Stage 3: Results Integration and Presentation

6. Data Integration and Post-processing:
   Facts extracted by an LLM are cleansed and organized into unified datasets using sophisticated post-processing algorithms available in PolyAnalyst. This provides coherent insights for the entire document.
7. Indexing and highlighting:
   The solution automatically creates linguistic rules that help navigate the original documents and highlight patterns supporting the performed fact extraction.
                                                                Figure 4. Linguistic rules for pattern highlighting
8. Interactive Web Reports:
   Users can access graphical reports, view the extraction results alongside the original documents with highlights, and validate results of extraction using an interactive interface. This enhances transparency and trust.

Figure 5. Interface for navigating and validating the results of AI-based extraction

Key Advantages of AI Data Curation

• Efficiency: The automated extraction process dramatically reduces time and manual effort.
• Precision: Robust data transformation and analysis software works alongside state-of-the-art Generative AI to provide highly accurate extraction and analysis.
• Scalability: The system can process numerous complex documents at once, making it suitable for both small projects and enterprise-level data initiatives.
• Simple Manual Validation: An interactive interface enables easy validation, which minimizes errors and ensures high-quality results.
• Minimal Human Intervention: Our multi-model AI approach allows different AI engines to collaborate seamlessly and share data and insights automatically. The user only specifies a collection of facts of interest.
• Broad Applicability: The user can easily specify a different set of facts of interest to make the solution solve a different task. This makes the solution easily adaptable to a wide range of industries and document types. Moreover, in the “free hunting” mode (to be described in a separate blog article soon) the system requires little to no domain-specific input from users. Instead, it can use the contents of documents themselves to extract and organize facts of potential interest and then present them to the user.

By blending analytical capabilities of PolyAnalyst with advanced Generative AI, our AI Data Curation solution automates and simplifies the process of extracting facts of interest from complex documents. Actionable insights are delivered faster and with higher accuracy.

To see AI Data Curation solution in action, request a demo.