Automated Clinical Data Extraction from Oncology Trials using LLMs to Facilitate Decision Making in Clinical Development

Public Automated Clinical Data Extraction from Oncology Trials using LLMs to Facilitate Decision Making in Clinical Development Ryumei Nakada1,2, Federico Ferrari2, Michelle N. Ngo2, Xiang Peng2, Shuyan Wan2, Junshui Ma2, Thomas Jemielita3, Yulia Sidi2 1Department of Statistics, Rutgers University, New Brunswick, NJ, USA 2Merck & Co., Inc., Rahway, NJ, USA 3Merck & Co., Inc., North Wales, PA, USA Abstract Efficient access to structured clinical data is critical for decision-making in oncology trials. However, manual extraction from unstructured sources, such as Clinical Study Reports (CSRs) and abstracts, is time-intensive, error-prone, and often incomplete. Recent advancements in large language models (LLMs) have shown promise in automating data extraction, improving accuracy and scalability. Here, we present an LLM-driven pipeline to extract key variables including tumor indications, study phase, and clinical outcomes from unstructured documents to support decisionmaking in clinical development. The pipeline achieved an overall accuracy of 94.3% across 21 numerical variables when tested on 30 internal CSRs and 97.5% across 45 variables when tested on 11 abstracts, closely matching the accuracy achieved by manual human extraction (96.2%). Processing times averaged 3–5 minutes per CSR and 30 seconds per abstract, with costs ranging from $1 to $8 per document and $0.30 per abstract. This represents a significant improvement in efficiency compared to manual extraction, which typically requires about one hour per CSR. By automating data extraction, the pipeline addresses delays, missing data, and human errors inherent in manual processes, potentially facilitating more timely and informed clinical decision-making. Key Words: Large language models, generative AI, oncology, clinical trials, structured database, information retrieval, variable extraction 1. Introduction Oncology is one of the most challenging therapeutic areas for drug development with low success rate compared to other therapeutic areas: 3.4% to 6.7% of oncology Phase I trials resulting in regulatory approval compared to 12.1% to 13.8% of non-oncology indications (Hay et al., 2014; Wong et al., 2019). Furthermore, the biological heterogeneity of oncology indications and mechanisms of action, in combination with unmet needs and potentially life-threatening indications led oncology drug development to differ from non-oncology in several structural and regulatory ways. For example, in oncology, overall survival (OS) is the gold standard, but for some indications its use may be challenging because observing a treatment effect can require very long follow‑up and because subsequent initiation of other anti‑cancer therapies can confound the measured effect; consequently regulators may accept alternative endpoints such as progression-free survival (PFS) in randomized trials or objective response rate (ORR) in single-arm trials, particularly for accelerated approval. Thus, in oncology drug development, timely and accurate access to both historical and current data is critical for guiding clinical development and making informed decisions. Key information such as tumor indications, biomarker profiles, and clinical outcomes—including ORR, PFS and OS—is essential because it informs clinical trial design and probability of success (PoS) 1 , supports portfolio prioritization and resource allocation by R&D leadership, guides regulatory strategy, provides evidence for medical affairs to contextualize results and engage investigators. It 1 This metric is often used to evaluate treatment strategies and forecast clinical outcomes. Public also helps commercial and marketing teams understand the treatment landscape, forecast uptake, and shape value and access messaging, and underpins market access and pricing by demonstrating clinical benefit versus existing options. Finally, these data facilitate lifecycle planning, including identifying unmet needs and label‑expansion opportunities. Timely, accurate access to both historical and current datasets strengthen all these activities by improving the quality, relevance, and speed of analyses and decisions. However, the common data collection practices are often inefficient: they rely on costly, labor-intensive manual processes, that is tedious and prone to human error, introduce delays, and struggle when aggregating data from disparate, unstructured sources such as various internal documents, e.g., clinical study reports (CSRs), and external websites. Recent advancements in natural language processing (NLP) and large language models (LLMs) offer promising solutions to these challenges. Techniques such as Retrieval-Augmented Generation (RAG) have demonstrated the capability to automatically retrieve, clean, and structure information from unstructured texts, ensuring consistency and accuracy across multiple data sources (Lewis et al., 2020) (Gao et al., 2023). In oncology—where clinical trial outcomes have profound implications for patient care and regulatory decisions—the ability to automate data extraction not only reduces time and cost but also enhances the reliability of the extracted information. The primary goal of this paper is to develop an LLM-powered pipeline that extracts key clinical variables from unstructured documents and consolidates them into a structured database. Our system integrates various data sources, including lengthy clinical study reports, concise abstracts, and external oncology publications, by leveraging state-of-the-art LLM capabilities to automate the extraction, transformation, and storage of clinical data. By achieving a performance level comparable to manual extraction, our approach is well-positioned to streamline clinical data collection reduce cost and error, and thereby improve decision‑making in clinical development as well as support the cross‑functional needs of R&D, regulatory, medical affairs, commercial, and market‑access teams. In addition to addressing the inefficiencies of manual data collection, our pipeline incorporates several innovative strategies to handle the inherent challenges of unstructured data. Advanced preprocessing techniques ensure that noisy and heterogeneous data formats are standardized, while robust information retrieval methods accurately identify and extract relevant text segments. Furthermore, ensemble methods and self-verification strategies are employed to minimize discrepancies and bolster the reliability of the extracted variables. Together, these components not only overcome the limitations of traditional methods but also pave the way for scalable, automated data extraction in oncology. 1.1 Outline of the Paper The remainder of this paper is organized as follows: Section 2 provides an overview of the related literature. Section 3 introduces the methodology underlying our AI-driven data extraction pipeline, detailing its key components and the advanced techniques employed. Section 4 presents the experimental evaluation of our system on both internal clinical study reports and oncology abstracts, with a detailed analysis of its performance. Finally, Section 5 offers a discussion on the implications of our findings, the challenges encountered, and potential avenues for future research. 2. Related Work 2.1 Information Extraction from Clinical Documents Extracting specific variables—such as tumor indications, biomarkers, and clinical outcomes—from unstructured clinical documents has traditionally relied on manual methods, which are laborintensive and error-prone (Wang et al., 2018). In recent years, natural language processing (NLP) approaches have been developed to automate this process, often referred to as named entity recognition (NER) (Alawad et al., 2020; Gero et al., 2023; Pomares-Quimbaya et al., 2019; Wang et al., 2018). Further advancements in machine learning and deep learning such as support vector machines (SVMs), convolutional and recurrent neural networks (CNNs, RNNs), and autoencoders Public have also been applied with varying success for variable extraction (Spasic & Nenadic, 2020; Wu et al., 2020). 2.2 Large Language Models (LLMs) The advent of large language models like GPT-4 and Claude 3.5 has significantly advanced the ability to process and extract information from unstructured clinical documents. LLMs excel at identifying contextually relevant data, reasoning through complex narratives, and generating structured outputs from unstructured inputs. Their applications span analyses of electronic health records (EHRs), clinical trial reports, and scientific literature, thereby supporting informed clinical decision-making (Hossain et al., 2023; Lederman et al., 2022). The promising results and reduction of manual workload have led to an uptake in usage of LLMs for data extraction; (Chen et al., 2025) queried Ovid MEDLINE and found that the number of studies published for extracting data from clinical text in oncology tripled from 2019-2021 to 2022-2024. However, limitations persist in applying LLMs in a clinical setting. For example, the information to be extracted may be presented in complex tables or spread across multiple pages. The outputs may also be non-trivial or biased if the prompt is too complex or ambiguous. In such cases, more advanced techniques such as RetrievalAugmented Generation (RAG) and/or prompt engineering may guide an LLM through the data extraction task and result in higher performance. 2.2.1 Retrieval-Augmented Generation (RAG) (Amugongo et al., 2025; Gao et al., 2023; Lewis et al., 2020; Wang et al., 2024) combines information retrieval techniques with LLMs to synthesize responses based on retrieved document segments and prior knowledge to mitigate hallucinations and improve accuracy. Briefly, RAG involves retrieving relevant information from external public and private sources such as medical databases, business documentation, and regulatory filings. Given the retrieved information, an LLM then generates a response based on the user query. This approach is particularly beneficial for processing and extracting information from extensive clinical documents such as CSRs. 2.2.2 Prompt Engineering and RAG Techniques Our pipeline leverages advanced prompt engineering and RAG techniques to boost extraction accuracy. Tailored prompts guide the LLMs to focus on extracting key variables. Techniques such as chain-of-thought reasoning (Kojima et al., 2022; Wei et al., 2022), rephrase-and-respond (Deng et al., 2023), and few-shot learning (Agrawal et al., 2022; Labrak et al., 2023; Richter-Pechanski et al., 2025) are employed to enhance output quality. For example, (Jabal et al., 2025) demonstrated that a combination of RAG and few-shot prompting generally improved accuracy in extracting BTRADS scores from radiology reports and IHD mutation status from pathology reports. Extending these concepts, (Wang et al., 2025) built a pipeline named TrialMind to expedite systematic reviews of clinical studies from medical literature. They used a combination of prompt engineering (incontext learning, chain-of-thought) and RAG at the literature search, literature screening, data extraction, and evidence synthesis phases to integrate human expertise and expand extraction context. This led to a 16-32% improvement in data extraction accuracy when compared to the standard GPT-4. 3. Methods Our proposed pipeline automates the extraction of clinical variables from oncology documents using a three-step process: pre-processing, information retrieval, and aggregation, see Figure 1. Public Figure 1: Overview of the information extraction pipeline consisting of three main stages: (1) Preprocessing, (2) Retrieval, and (3) Aggregation. The example illustrates extraction of the variable nevents_pfs (number of PFS events) for Cohort 1. 3.1 Pre-processing The raw PDF documents were first converted into Markdown format using the pymupdf4llm library, which efficiently extracts text while preserving structural elements such as tables and figures. Key pre-processing steps include: • Chunking: Documents were segmented into overlapping chunks to ensure that no critical data was lost, especially at chunk boundaries. • Manual Correction: In cases where table extraction resulted in misaligned columns or nested rows, manual corrections were applied to improve data fidelity. • Noise Reduction: Basic cleaning procedures were implemented to remove OCR artifacts and other non-informative text. Future work may integrate more robust OCR and table parsing techniques to further improve text accuracy and structural consistency, particularly for complex scientific documents containing multilevel tables and embedded figures. 3.2 Information Retrieval Relevant text segments are retrieved using embeddings generated by text-embedding-3-large from OpenAI. Each retrieved chunk is compressed by GPT-4 to eliminate redundancy while preserving essential information (Wu et al., 2024). Predefined clinical variables (e.g., ORR, PFS, OS) are subsequently extracted from these refined segments. Maintaining the original ordering of text chunks helps preserve the contextual flow (Yu et al., 2024). Additionally, a self-verification mechanism (Gero et al., 2023) allows the model to re-assess its outputs to detect and correct potential errors. Variables are grouped into subgroups (e.g., Study Information and PFS-related Variables) to streamline the extraction process. 3.3 Aggregation To further ensure accuracy and robustness, the extraction and aggregation processes are performed multiple times independently. While information is rarely missed during the retrieval step, the retrieved content may contain irrelevant or misinterpreted information due to chunking or preprocessing artifacts. During aggregation, outputs from multiple chunks are compared to identify and retain the most plausible and consistent values (Yu et al., 2024), effectively filtering out duplicates and inconsistencies. To assess robustness, the aggregation step is repeated five times, allowing evaluation of output stability across runs. Through the aggregation step, hallucination is effectively reduced, leading to outputs that are both accurate and reliable. Public 4. Experiments In this section, we present a comprehensive evaluation of our automated clinical data extraction pipeline. Our experiments focus on two distinct document types: large, unstructured CSRs and concise oncology abstracts from ASCO. We describe the dataset characteristics, detail the preprocessing and extraction pipeline, outline our evaluation metrics, and present extensive results along with error analysis and computational efficiency metrics. 4.1 Datasets For our evaluation, we considered two main data sources: • CSRs: We collected 30 internal CSRs, which are extensive PDF documents spanning hundreds to thousands of pages. These documents contain key clinical variables, such as ORR, PFS, and OS. Data from each CSR was manually extracted by one person and randomly reviewed by two additional reviewers to establish ground truth for 21 variables. The manual process required approximately 1 hour per CSR and 20 minutes per abstract. • Oncology Abstracts: In addition, we included 11 oncology abstracts from the ASCO 2024 Program Guide. Although these documents are shorter and more standardized, they report critical clinical outcomes across various types of oncology trials. A total of 45 variables, organized into 7 subgroups (e.g., treatment cohort sizes, survival rates, and response rates), were manually annotated for these abstracts. 4.2 Evaluation Metrics The performance of the pipeline was quantified using several key metrics. Accuracy was computed using a 0-1 loss function, where an exact match with the manually annotated value was assigned a score of 1 and any discrepancy a score of 0; these scores were then averaged over all variables. To assess output stability and quantify uncertainty, each CSR was processed 10 times, allowing us to measure the consistency of the extraction process. Additionally, we recorded the runtime and associated computational costs per document, providing valuable insights into the scalability of the pipeline. 4.3 Results The variable extraction performance varied slightly between the two document types: CSRs and abstracts. Table 1 below shows the accuracy of variable extraction for the two document types across the different variable types. • CSRs: Our pipeline achieved an overall accuracy of 94.3% across 21 variables. Notably, variables related to PFS (e.g., nevents_pfs_trt) exhibited lower accuracy (90.2%) due to retrieval challenges and ambiguous table structures. • Oncology Abstracts: The pipeline performed even better on abstracts, achieving 97.5% accuracy across 45 variables, with only 12 discrepancies observed compared to human annotations. In addition, the extraction process demonstrated competitive computational efficiency. The average processing time per CSR ranged from 3 to 5 minutes (with costs between $1 and $8 per CSR, depending on document complexity), while abstracts required approximately 30 seconds per document. 4.4 Error Analysis A detailed error analysis revealed several insights: Table 1: Accuracy of variable extraction for CSRs and ASCO abstracts. Variable Type CSR Accuracy Abstract Accuracy ORR-Related 96.1 98.0 PFS-Related 90.2 96.5 OS-Related 94.5 97.8 Overall 94.3 97.5 Public • Document Formatting: Many errors in CSRs were linked to non-standardized formatting, including scattered tables and figures. Overlapping text and ambiguous table structures led to misinterpretations, particularly for PFS-related variables. • Ambiguous Reporting in Abstracts: Although abstracts were generally more standardized, discrepancies arose from minor variations in reporting standards between different sponsors. • Retrieval Failures: Some extraction errors were attributed to failures in the retrieval phase, where key text segments were either omitted or only partially captured. Manual review of the errors indicated that approximately 80% of the misinterpretations for PFS variables were due to table formatting issues. Addressing these challenges is a primary focus for future improvements in the pre-processing module. 4.5 Computational Efficiency and Robustness Beyond accuracy, our experiments evaluated the pipeline’s scalability: • Runtime Analysis: The pipeline’s average runtime per CSR was between 3 to 5 minutes, which is acceptable for clinical settings. Shorter documents, like abstracts, required only around 30 seconds. • Cost Considerations: The computational cost ranged from $1 to $8 per CSR, making the approach cost-effective compared to manual extraction. For abstracts, the cost was approximately $0.30. • Robustness: Repeating the extraction 10 times per CSR allowed us to assess the stability of our results. The low variance observed across runs underscores the robustness of our ensemble extraction method. 4.6 Summary Overall, the experiments demonstrate that our pipeline achieves high accuracy in both large, complex CSRs and more standardized oncology abstracts, while maintaining efficiency and costeffectiveness. The detailed error analysis has identified key areas for improvement, particularly in handling complex table structures and ambiguous text segments. Future work will focus on refining the pre-processing stage and incorporating advanced OCR techniques to further enhance performance. 5. Discussion We developed and evaluated an LLM-driven pipeline that converts unstructured oncology documents into structured database of key clinical variables. Our results demonstrate that the proposed AI-driven pipeline offers a viable alternative to manual data extraction in oncology clinical trials. With an overall accuracy comparable to human annotation and a dramatic reduction in processing time—minutes for CSRs and seconds for abstracts—the system significantly enhances efficiency and reduces costs, enabling timely data access crucial for cross-functional R&D teams. Scalability is a central strength of the approach. Leveraging LLMs such as GPT-4 allows the system to process large, complex documents, converting unstructured text into structured data rapidly. The integration of techniques like Retrieval-Augmented Generation, ensemble extraction, and prompt engineering further improves robustness across diverse document formats, making it particularly effective for extracting critical endpoints such as ORR and PFS. The high accuracy on abstracts reflects the benefits of short, standardized formats, while the slightly lower but still strong performance on CSRs highlights the model’s ability to handle long, heterogeneous documents when supported by careful pre-processing and retrieval strategies. Despite these strengths, several challenges remain. Extraction errors for PFS-related variables— largely due to formatting issues and complex table structures—highlight the limitations of current Public pre-processing methods. Future work should prioritize improved handling of nested tables, enhanced OCR integration, and adoption of newer models potentially capable of direct PDF processing. Furthermore, advancements in generative AI models (e.g., Claude 4.5, GPT-5, Gemini 2.5) and approaches may offer better reasoning capabilities and variable extraction compared to our GPT-4 implementation. Future research will also explore integrating uncertainty quantification and multimodal data extraction to further enhance the system’s reliability and applicability. Such advancements, along with adherence to regulatory standards and ethical considerations, will be essential for transitioning this technology into routine drug‑development processes. In summary, our pipeline represents a significant step toward automated clinical data extraction, offering both high accuracy and speed. Targeted refinements—particularly in table and layout handling, retrieval robustness, and uncertainty estimation—will further reduce errors and broaden applicability. With these improvements and appropriate validation, traceability, and governance, this approach can streamline evidence synthesis from diverse trial reports and publications, enabling faster, better‑informed decisions across oncology development. References Agrawal, M., Hegselmann, S., Lang, H., Kim, Y., & Sontag, D. (2022). Large language models are few-shot clinical information extractors. arXiv preprint arXiv:2205.12689. Alawad, M., Gao, S., Qiu, J. X., Yoon, H. J., Blair Christian, J., Penberthy, L., Mumphrey, B., Wu, X.-C., Coyle, L., & Tourassi, G. (2020). Automatic extraction of cancer registry reportable information from free-text pathology reports using multitask convolutional neural networks. Journal of the American Medical Informatics Association, 27(1), 89––98. Amugongo, L. M., Mascheroni, P., Brooks, S., Doering, S., & Seidel, J. (2025). Retrieval augmented generation for large language models in healthcare: A systematic review. PLOS Digital Health, 4(6), e0000877. Chen, D., Alnassar, S. A., Avison, K. E., Huang, R. S., & Raman, S. (2025). Large Language Model Applications for Health Information Extraction in Oncology: Scoping Review. JMIR Cancer, 11, e65984. https://doi.org/10.2196/65984 Deng, Y., Zhang, W., Chen, Z., & Gu, Q. (2023). Rephrase and respond: Let large language models ask better questions for themselves. arXiv preprint arXiv:2311.04205. Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., & Wang, H. (2023). Retrievalaugmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997. Gero, Z., Singh, C., Cheng, H., Naumann, T., Galley, M., Gao, J., & Poon, H. (2023). Selfverification improves few-shot clinical information extraction. arXiv preprint arXiv:2306.00024. Hay, M., Thomas, D. W., Craighead, J. L., Economides, C., & Rosenthal, J. (2014). Clinical development success rates for investigational drugs. Nat Biotechnol, 32(1), 40–51. https://doi.org/10.1038/nbt.2786 Hossain, E., Rana, R., Higgins, N., Soar, J., Barua, P. D., Pisani, A. R., & Turner, K. (2023). Natural language processing in electronic health records in relation to healthcare decision-making: a systematic review. Computers in biology and medicine, 155, 106649. Jabal, M. S., Warman, P., Zhang, J., Gupta, K., Jain, A., Mazurowski, M., Wiggins, W., Magudia, K., & Calabrese, E. (2025). Open-Weight Language Models and Retrieval-Augmented Generation for Automated Structured Data Extraction from Diagnostic Reports: Assessment of Approaches and Parameters. Radiol Artif Intell, 7(3), e240551. https://doi.org/10.1148/ryai.240551 Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022). Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, 35, 22199–– 22213. Public Labrak, Y., Rouvier, M., & Dufour, R. (2023). A zero-shot and few-shot study of instructionfinetuned large language models applied to clinical and biomedical tasks. arXiv preprint arXiv:2307.12114. Lederman, A., Lederman, R., & Verspoor, K. (2022). Tasks as needs: reframing the paradigm of clinical natural language processing research for real-world decision support. Journal of the American Medical Informatics Association, 29(10), 1810––1817. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., & et al. (2020). Retrieval-augmented generation for knowledgeintensive nlp tasks. Advances in Neural Information Processing Systems, 33, 9459––9474. Pomares-Quimbaya, A., Kreuzthaler, M., & Schulz, S. (2019). Current approaches to identify sections within clinical narratives from electronic health records: a systematic review. BMC medical research methodology, 19, 1––20. Richter-Pechanski, P., Wiesenbach, P., Schwab, D. M., Kiriakou, C., Geis, N., Dieterich, C., & Frank, A. (2025). Clinical information extraction for lower-resource languages and domains with few-shot learning using pretrained language models and prompting. Natural Language Processing, 31(5), 1210–1233. https://doi.org/10.1017/nlp.2024.52 Spasic, I., & Nenadic, G. (2020). Clinical Text Data in Machine Learning: Systematic Review. JMIR Med Inform, 8(3), e17984. https://doi.org/10.2196/17984 Wang, X., Wang, Z., Gao, X., Zhang, F., Wu, Y., Xu, Z., Shi, T., Wang, Z., Li, S., Qian, Q., & et al. (2024). Searching for Best Practices in Retrieval-Augmented Generation. arXiv preprint arXiv:2407.01219. Wang, Y., Wang, L., Rastegar-Mojarad, M., Moon, S., Shen, F., Afzal, N., Liu, S., Zeng, Y., Mehrabi, S., Sohn, S., & et al. (2018). Clinical information extraction applications: a literature review. Journal of Biomedical Informatics, 77, 34––49. Wang, Z., Cao, L., Danek, B., Jin, Q., Lu, Z., & Sun, J. (2025). Accelerating clinical evidence synthesis with large language models. NPJ Digit Med, 8(1), 509. https://doi.org/10.1038/s41746-025-01840-7 Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., & et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35, 24824––24837. Wong, C. H., Siah, K. W., & Lo, A. W. (2019). Estimation of clinical trial success rates and related parameters. Biostatistics, 20(2), 273–286. https://doi.org/10.1093/biostatistics/kxx069 Wu, J., Che, F., Zhang, C., Tao, J., Zhang, S., & Shao, P. (2024). Pandora's Box or Aladdin's Lamp: A Comprehensive Analysis Revealing the Role of RAG Noise in Large Language Models. arXiv preprint arXiv:2408.13533. Wu, S., Roberts, K., Datta, S., Du, J., Ji, Z., Si, Y., Soni, S., Wang, Q., Wei, Q., Xiang, Y., Zhao, B., & Xu, H. (2020). Deep learning in clinical natural language processing: a methodical review. J Am Med Inform Assoc, 27(3), 457–470. https://doi.org/10.1093/jamia/ocz200 Yu, T., Xu, A., & Akkiraju, R. (2024). In Defense of RAG in the Era of Long-Context Language Models. arXiv preprint arXiv:2409.01666.