Supplementary material of the paper "Automated Data Extraction in Software Engineering SLRs: Promise, Pitfalls, and the Role of Human Oversight"

Supplementary material – README This supplementary material accompanies the research article "Automated Data Extraction in Software Engineering SLRs: Promise, Pitfalls, and the Role of Human Oversight", and provides all files required to understand, verify, or replicate the analyses presented in the study. Contents 1. SelectionOfTools.pdf This document briefly outlines the steps we have followed to select the tools that we have evaluated in the study. 2. SamplePapers.pdf In this document we include the list of papers used for data extraction in the experiment described in the paper. 3. ExperimentPrompts.pdf In this document we present the query codes (prompts) for the three selected automatic data extraction tools (ChatGPT, ChatPDF and Google NotebookLM) used in the experiment. 4. data_dump.csv This CSV file includes the data captured from the extraction process (both tools and human responses to the data extraction questions for each paper). • Description: A single table with 504 rows. • Columns: o experiment_id: Experiment internal code. o answer_id: Answer internal code. o question_id: Question code ('QA01', 'DE02', etc.). o answer_format: Answer data type ('boolean', 'closed category', 'list of closed categories'). o answer_text: Raw obtained answer. o paper_id: Paper code. o expert_code: Expert code. Who performed the data extraction manually (tool_name=‘Expert’) or using the tool. o tool_name: Method or tool used for data extraction ('Expert', 'ChatGPT', 'ChatPDF', 'Google NotebookLM'). o order_code: For tool_name='Expert', indicates whether extraction was performed before using the tool (‘pre’) or after (‘post’). o aggrupation_code: Aggrupation type code ('S' for sequential, 'I' for isolated, 'B' for block). o duration: Estimated time for extracting data for the specific question. This file records the responses generated by different data extraction tools, their configurations, and the corresponding metadata for each experimental trial.