Organizations are sitting on vast amounts of valuable information locked away in PDFs and other text documents. But turning those messy, complex documents into structured data that can be used for analysis is often difficult, time consuming and expensive. In a recently published article in INFORMS Journal of Applied Analytics, Simon Spavound, PhD, assistant clinical professor, and Oliver Schaer, PhD, assistant professor, both of Drexel University’s LeBow College of Business, along with Panos Markou, PhD, of the University of Virginia, created a tutorial for using large language models (LLM) to transform raw documents into structured data for downstream analytical tasks such as predictive modeling and sentiment analysis.
In the article, Spavound, Schaer and Markou demonstrate how they were able to convert 1,900 pages of transcripts from the U.S. Food and Drug Administration (FDA) Advisory Committee into analysis-ready data with over 99.9% accuracy for just $3.50.
According to Spavound and Schaer, organizations can use the blueprint to parse decades of complex data for forecasting, classification or decision-making applications. And because of the pipeline’s modular framework, they explained, analysts can adapt it with minimal prompt adjustments to other wide ranging document types, including financial reports, medical records and supply chain documentation.
Spavound and Schaer shared with Drexel News Blog how the model works, how it’s unique and where to find the tutorial for use.
What are people or businesses currently doing to capture data in documents?
Spavound: Generally, people are using a variety of means to input data from documents, such as traditional OCR (optical character recognition), as well as human extraction – though this is expensive and time-consuming.
How will your model improve that process?
Schaer: Our tutorial aims to provide practitioners with a highly flexible approach to extract text from documents. One issue we have seen in our own research is that documents can vary widely in quality and document formatting.
An LLM-based approach allows for natural language interaction, enabling you to tell the model exactly what you want to extract. For example, instead of identifying document artifacts such as page headers in a rule-based fashion, which relies on a consistent content structure, we can use prompting to remove such artifacts independent of the document structure. This approach is therefore more flexible across documents than traditional methods, while also being highly accurate and cheap.
Can you describe the blueprint for using AI to extract data from documents and transform it into usable data?
Spavound: In our tutorial, we utilize a two-stage process. First, we convert PDFs into images (one page per image), then prompt a vision-capable LLM to extract the desired text from each page image. In this prompt, we instruct the model to ignore any page numbers or other unwanted elements. This gives us a raw, cleaned text output without any formatting. In the second stage, we pass this raw text through another LLM to obtain the structure of the text so that we can use it for downstream data analysis. In our case, we wanted a structured meeting transcript, so our final document contains clean speaker-statement pairs we can use for tasks such as measuring speaker-level sentiment. However, it is possible to adjust the desired structured output with little effort to other document types or formats.
It is worth mentioning that while LLM models are getting more capable on their own, splitting the tasks into two stages allows us to use smaller, cheaper models, and we can scale it indefinitely. Therefore, our pipeline can be applied to documents from ten pages to several hundred pages, as is the case for our meeting transcripts, and we do not have to wait for a capable model that supports the required context windows to become available.
For those interested, how do they access the tutorial?
Schaer: The tutorial is published in the INFORMS Journal of Applied Analytics and is open access. It contains a step-by-step Python code implementation guide, which is also available in our GitHub repository.
Is there anything you think is important to know about your article or the model?
Spavound: One of the main motivations for writing this as a tutorial is that while many people are familiar with LLM “chat” interfaces (such as ChatGPT), they miss out on the ability to scale and repeat the same task. We aim to show that with a little programming knowledge, using the application programming interface (API) versions of these models can improve productivity and consistency.
Reporters interested in speaking with Spavound and Schaer should contact Annie Korp, associate director, News & Media Relations, at 215-571-4244 or amk522@drexel.edu.
