Assessing the Reliability of Human and LLM-Based Screening in Systematic Reviews: A Study on First-Time Reviewers

ASSESSING THE RELIABILITY OF HUMAN AND LLM-BASED SCREENING IN SYSTEMATIC REVIEWS: A STUDY ON FIRST-TIME REVIEWERS E. Sandner∗1,3, D. Scharf3, T. Wautischar3, I. Jakovljevic1, A. Simniceanu2, L. Fontana2, A. Henriques1, A. Wagner1, C. Gütl3 1CERN, 1211 Geneva, Switzerland 2WHO, 1211 Geneva, Switzerland 3Graz University of Technology, 8010 Graz, Austria Abstract Systematic reviews (SRs) rely on rigorous study selection to ensure methodological quality. Double-blind screening by two independent reviewers is considered the gold standard, but it is also time-consuming and resource-intensive. Large language models (LLMs) have recently been proposed as a means of reducing this workload, yet their acceptance remains limited because their performance is insufficiently benchmarked against the standards applied to human screeners. To address this gap, this study investigates the screening behavior of novice reviewers and compares their performance with that of an LLM. In a graduate-level course, 54 students conducted title and abstract screening across ten information retrieval topics. Each record was independently screened by four students and by an LLM applying a five-tier classification approach. Inter-rater reliability was measured using Fleiss’ 𝜅 and Cohen’s 𝜅 , while sensitivity and specificity were calculated under different screening configurations. The results show that novice screeners achieved only fair to moderate agreement (overall Fleiss’ 𝜅=0.386 ), while the LLM’s agreement with the human consensus (Cohen’s 𝜅=0.516 ) was higher than the average human–human agreement, yet still within a similar range. Performance analysis revealed that single human screening (sensitivity 84.03%, specificity 90.36%) outperformed the LLM (80.30% / 85.50%). Double human screening achieved near-perfect sensitivity (99.18%) at the cost of lower specificity (82.10%). A hybrid setting of one human plus the LLM improved sensitivity (94.98%) relative to a single human. These findings highlight both the variability of novice human decisions and the potential role of LLMs as complements, but not replacements, in screening workflows. INTRODUCTION During the study selection phase of a systematic review (SR), hundreds or even thousands of abstracts must be manually assessed for eligibility. To ensure methodological rigor, high-quality reviews require double-blind screening [1], meaning that two independent reviewers evaluate each paper in parallel. This labor-intensive process is designed to minimize human bias and error, thereby meeting the strict quality standards of SR methodology. ∗[email protected] Several approaches have been proposed to automate this time-consuming and repetitive task. However, automation methods designed for broad applicability across domains and eligibility criteria face limited acceptance within the evidence synthesis community. This is largely because they fail to demonstrate compliance with the methodological standards of SRs, particularly the requirement to safeguard against the exclusion of relevant studies, a criterion measured by sensitivity [2]. While some studies consider a sensitivity of 95% to be sufficient [3,4], Cochrane 1 , a research organization renowned for its high-quality SRs and for setting standards in the SR process, requires a sensitivity of 99% for any tool intended to replace human screening [5]. Evaluation of an automation system is typically performed by comparing its outcomes against a human-annotated gold standard, ideally derived from double-blind screening. Although automation must not be introduced into the SR process at the cost of reduced quality, one may question whether human screeners reliably satisfy the quality requirements imposed on automation systems, especially when training and domain expertise are limited. Therefore, this study investigates the screening behavior of 54 students as they conducted their first SR exercise. Students were assigned one of ten topics in the domain of ’Information Search and Retrieval’ and asked to screen 30 records. Each record was independently evaluated by four students. Furthermore, each record was subjected to large language model (LLM) screening using the 5-tier algorithm with a permissive setting [6]. Based on the collected data, inter-rater agreement was analyzed using Cohen’s and Fleiss’ kappa. Furthermore, by assuming the consensus of all four screeners as the gold standard, double-blind screening could be compared with alternative scenarios. Building on these results, this paper addresses the following research questions: RQ1: What is the inter-rater agreement beyond chance among first-time screeners? RQ2: How does the performance of single screening compare with double-blinded screening? RQ3: How does the performance of LLM screening compare with that of human screening? 1https://www.cochrane.org/ https://doi.org/10.5281/zenodo.17234055 The paper proceeds with background and related work to situate the study in context. It then outlines the methodology, including data collection and analysis, before presenting and interpreting the results. This is followed by a discussion of limitations and directions for future work. The closing section summarizes the main contributions and conclusions. BACKGROUND AND RELATED WORK By synthesizing findings from potentially all relevant studies on a given research question, a SR represents the most reliable research methodology for evidence-based conclusions [7]. Therefore, SRs play a crucial role in the medical field, guiding decision-making and shaping clinical practice guidelines [8]. However, the rigorous nature of the process makes SRs highly time-consuming and resource-intensive. Completing a single SR often requires several months, and in some cases, even years [9–11]. When conducting a SR, an initial database query is typically designed to be broad rather than highly specific, ensuring comprehensive coverage of relevant studies. The retrieved candidate studies then undergo human screening, which is considered one of the most time-consuming stages of the SR process [12]. Numerous efforts have aimed to reduce the human workload through automation. For instance, Cochrane has developed a machine learning classifier to identify candidate studies and exclude those with study designs other than Randomized Controlled Trials (RCTs) [5]. However, its applicability is limited to reviews that include only RCTs. Additionally, all other criteria remain unaddressed, thus limiting workload reduction. While training a classifier for broad applicability is impractical, classification systems leveraging the reasoning capabilities of LLMs have attracted increasing attention in recent years. However, their acceptance for real-world application within the evidence synthesis community is limited, as exhaustive evaluation and validation are lacking to demonstrate that these approaches achieve the expected sensitivity and do not miss relevant studies. [2] However, the performance of human screeners is also subject to limitations, with errors and disagreements observed even under double-blinded conditions that are designed to minimize bias. Gartlehner et al. [13] analyzed human screening performance in a crowd-based, parallel-group randomized controlled trial. All 280 participants had prior experience in abstract screening and were required to pass an initial task demonstrating that they could correctly label at least 80% of a test set. Each participant was assigned up to 100 candidate studies, which together resulted in 24,942 screening decisions on 2,000 randomly selected abstracts. On average, each abstract was screened 12 times. The study found that single screeners achieved a sensitivity of 86.6%, while double-blinded screening reached 97.5%. The respective specificities were 79.2% and 68.7%. Issaiy et al. [14] compared the screening performance of GPT-3.5 Turbo with that of three general physicians. LLM screening was conducted by instructing the model to assign each study a numerical value between 1 and 5 based on its relevance, with categories 1–3 considered as include decisions. In this prospective simulation study, both the LLM and each participant screened 1,198 records spanning different subject areas within radiology. The ground truth was established based on the screening decisions of two expert researchers with 5 and 20 years of experience. The results showed moderate agreement among the three general physicians ( 𝜅=0.45 ) and substantial agreement between the two expert researchers ( 𝜅=0.79 ). In contrast, agreement between the LLM and the general physicians was lower, with a mean 𝜅 of 0.27. Screening performance of single screeners, consensus, and the LLM is detailed in Table 1. The findings demonstrate that neither individual human screeners nor the consensus of three screeners, whether based on majority voting or a sensitive consensus, meets the sensitivity requirements established for automated solutions.At the same time, the results indicate that the automated approach missed fewer relevant studies than the human screeners. However, the authors also noted that the group of general physicians consisted of relatively young researchers without specialized training in radiology and therefore may not adequately reflect the population that typically carries out screening in evidence-based medicine. Table 1: Screening performance of general physicians, consensus methods, and ChatGPT, based on data from [14]. Screener Sensitivity Specificity General Physician 1 0.55 0.94 General Physician 2 0.55 0.99 General Physician 3 0.74 0.94 Voting Consensus 0.62 0.98 Sensitive Consensus 0.90 0.89 ChatGPT 0.95 0.65 Relying on well-trained human researchers and the resource-intensive double-blinded screening process is undoubtedly required in SRs that underpin evidence-based practice, shaping clinical guidelines, healthcare policies, and treatment decisions. However, SRs also play a crucial role in the context of primary research. Performing a SR of existing evidence before initiating a new study is essential to ensure both its quality and relevance [15]. A thorough understanding of prior studies helps identify research gaps, formulate meaningful questions, and inform the design of new studies [16,17]. When SRs are conducted for this purpose in domains outside medicine, training requirements for human screeners may be less stringent. Furthermore, in such cases, decisions regarding the use of automation may be more flexible, and resource savings from automation could enable SRs that would otherwise be infeasible due to resource constraints. More generally, in settings where automation achieves perhttps://doi.org/10.5281/zenodo.17234055 formance equivalent to the human screeners who would otherwise conduct the task, its use may be justified. However, the literature lacks studies on inter-rater agreement or screening performance of first-time screeners in computer science. METHODOLOGY To address the defined research questions, a structured methodological framework was applied that links data collection with subsequent analysis. This chapter describe how screening data were generated in the course setting, how they were transformed into a comparable dataset for humans and the LLM, and which metrics were used to assess reliability and performance. Data Collection The data collection was conducted as part of the graduatelevel university course “Information Search and Retrieval” held during the winter term 2024/25 at Graz University of Technology. This course focuses on the key concepts of information retrieval (IR) and web search systems, while also introducing the core principles of SRs. All students enrolled on the course were tasked with conducting a SR project in the second half of the term, covering the entire SR pipeline and including additional exercises to gain practical experience. At the start of the term, 60 students were enrolled on the course and organised into ten groups of six students each. By the time the SR part of the course began, the number of enrolled students had decreased to 54 due to students dropping out for various reasons. During the group selection process, each team had to choose a topic that would form the basis of their work on various tasks throughout the course, and, most importantly, their main focus for the SR project. Those ten topics were pre-selected by the course instructors in order to achieve two objectives: to ensure diversity between the topics and to highlight current research trends in the IR domain. To achieve the latter, the course instructors referred to the main topics in the proceedings of the 47th International ACM SIGIR Conference on Research and Development in IR when selecting the topics [18]. The selected topics were: 1. Neural IR 2. Retrieval Augmented Generation 3. GenIR and Search with LLMs 4. Evaluation with and for LLMs 5. Multilingual Retrieval 6. Question Answering and Summarisation 7. Conversational IR and Recommendation 8. Explainability in Search and Recommendation 9. Privacy and Security in Recent IR Systems 10. Users and Simulations in IR Systems The entire SR project was divided into individual stages and tasks. These had to be completed either individually, in subgroups (smaller groups formed from the original group), or in the original groups. The first stage consisted of the creation of eligibility criteria, the identification of seed papers, the creation of the search string, the data retrieval, and the deduplication of the retrieved records. For each group, members first defined their eligibility criteria individually using the PICO (Participants, Intervention, Comparison, and Outcomes) framework as suggested in SR guidelines [1,19]. Next, the group reached a consensus by comparing and refining these individual criteria. These eligibility criteria were then used to create the search strings for the two literature databases, ACM 2 and IEEE 3 . The course instructors selected these two literature databases as the primary sources for this SR project because they are widely used in the field of IR and share a similar, user-friendly search interface that enables students to apply their own search strings for data collection. To align with the lecture’s teaching objectives while maintaining sufficient variability for meaningful analysis, the number of studies per topic was limited to a manageable sample size of 200 ± 20 papers per literature database. Because the same papers could appear in both literature databases, each group was required to perform a deduplication step when combining the two retrieved result sets. These deduplicated result sets marked the conclusion of the first stage and were subsequently submitted to the course instructors. Conducting a comprehensive SR of hundreds of papers would not only be impractical in the context of a university course but also offer little educational value. Therefore, the number of papers assigned to each student for title and abstract screening (TiAb-screening) was reduced to align with the time constraints and learning objectives of the course. To guarantee that each student has a good variety of papers ranging from unsuitable to suitable in their screening set regarding their chosen topic, a LLM based screening algorithm (5-Tier Prompting Approach [6]) was utilized to select a subset of the retrieved papers. This algorithm uses the eligibility criteria previously created by each group and does an automated LLM screening of the retrieved papers, assigning each of them to one of five classes, indicating how likely the respective paper is to meet those criteria. Based on these classes, a subset of the original retrieved papers was created for each group, containing both papers that were deemed suitable (classes 1-3) and papers that were not suitable (classes 4-5) based on the respective eligibility criteria, as determined by the LLM screening. These subsets and subsequent tasks were structured so that every record in each subset was screened by exactly four different group members, with each of them screening approximately 30 papers. At this stage, the members of each group were unaware of the LLM screening process or the intentional overlap introduced 2https://dl.acm.org/search/advanced 3https://ieeexplore.ieee.org/search/advanced https://doi.org/10.5281/zenodo.17234055 in their assigned paper sets. They were simply instructed to screen the papers they had been given, decide if they want to include or exclude the paper at this stage, and in the case that they want to exclude it, give a reason why. Only afterwards were they tasked to combine their individual screening result and resolve potential conflicts. Conflict resolution was performed based on majority voting, and in case of a tie, through discussion. In instances where a paper was ultimately excluded, teams were required to provide a reason for exclusion. As a result, each group produced a document compiling all individual TiAb-screening decisions alongside the consolidated group decision. These documents indicated which papers from the subsets were ultimately included or excluded in the TiAb-screening phase, along with the corresponding justifications. They were uploaded for the course instructors and were the basis for the future tasks, including a full-text screening phase of the papers included after the TiAb-screening phase. This, however, is no longer relevant for the scope of this paper. The collection of these consolidated TiAb-screening documents forms the basis for the dataset of this analysis. Data Processing and Analysis The resulting dataset contains screening decisions from 54 individual human screeners alongside consolidated, conflictresolved decisions provided by ten teams. These team-level decisions were treated as the ground truth against which individual human screeners and the LLM system were evaluated. The evaluation followed the 5-tier framework proposed in [6], which classifies papers on a relevance scale from 1 (highly relevant) to 5 (not relevant). Since this framework does not yield a direct binary include/exclude outcome, the LLM outputs were converted to binary for comparability. Specifically, papers assigned a score of 4 or 5 by the LLM were classified as “excluded,” while papers assigned a score from 1 to 3 were treated as "included." For data processing and evaluation, a Python pipeline was developed. In the first step, screening decisions from the two sources were combined into a Pandas DataFrame using a shared internal ID. Column names were standardized, and the resulting merged data was exported into team-specific CSV files. A second script of the pipeline re-imports these files and transforms all decisions into binary format, applying the predefined thresholding strategy for the LLM outputs. For the human screeners, decisions are directly converted into binary values. This dataset then served as the basis for calculating the evaluation metrics described below. To assess inter-rater reliability both among human screeners and between humans and the LLM, Cohen’s kappa and Fleiss’ kappa as defined in (1) and (2) are computed. Cohen’s 𝜅= 𝑝0−𝑝𝑒 1−𝑝𝑒 (1) where 𝑝𝑜 denotes the observed proportion of agreement between two raters and 𝑝𝑒 the proportion of agreement expected by chance. Fleiss’ 𝜅= ¯ 𝑃−¯ 𝑃𝑒 1−¯ 𝑃𝑒 (2) where ¯ 𝑃 is the mean observed agreement across all subjects and raters, and ¯ 𝑃𝑒 the mean expected agreement by chance, based on the marginal proportions of each category. Cohen’s kappa measures the level of agreement between two raters, whereas Fleiss’ kappa extends this measure to situations involving more than two raters. Importantly, both metrics assess reliability rather than validity; they indicate how reliable the raters agree in their screening decisions among raters but do not evaluate whether they align with the chosen ground truth. A key advantage of these metrics is that they correct for the level of agreement that might be expected to occur purely by chance, making them more robust than raw agreement percentages. The values of kappa are bounded between –1 and 1, where 1 represents perfect agreement, 0 reflects agreement at the level of chance, and negative values suggest systematic disagreement. Initially both, Cohen’s kappa and Fleiss’ kappa were computed to assess inter-rater reliability. However, during the analysis it became clear that Cohen’s kappa did not provide substantial additional insight when applied to human screeners, except in the specific case of comparing the consolidated team decision with the LLM. Consequently, Fleiss’ kappa was adopted as the main measure of inter-rater reliability among human raters. For the performance analysis, several experimental conditions were designed and evaluated against the ground truth: •Single human screener •LLM only •Two human screeners •Single human screener and LLM In scenarios involving two screeners, conflicts could occur regarding inclusion decisions. To resolve such disagreements, a paper was classified as “included” if at least one screener selected it. This approach was purposefully chosen to ensure that potentially relevant papers were less likely to be missed. Each screening setup was subsequently evaluated using sensitivity and specificity, as defined in (3) and (4). Sensitivity = True Positive True Positive + False Negative (3) Specificity = True Negative True Negative + False Positive (4) Sensitivity quantifies the proportion of truly relevant papers that were correctly identified as “included,” while specificity measures the proportion of truly irrelevant papers that were correctly identified as “excluded”. https://doi.org/10.5281/zenodo.17234055 Although additional metrics were calculated, such as positive predictive value (precision) and negative predictive value, sensitivity and specificity emerged as the most informative and widely recognized metrics in the context of the research questions. Consequently, these two measures were prioritized in the evaluation, whereas the raw results of additional metrics are presented in the supplementary material4. The resulting data was exported to a CSV file containing all metrics generated by the pipeline. Subsequently, the file was imported into a spreadsheet, where overall average and median values are calculated for further analysis. Since every screener made their decision completely independent from each other, no weighting was introduced per team. RESULTS This section first reports the reliability between individual human screeners as well as between human screeners and LLM decisions. It then presents screening performance in terms of sensitivity and specificity across different settings, before discussing the results with respect to the research questions. Inter Rater Reliability As outlined in the Methodology section, Fleiss’ kappa was used to assess inter-rater reliability among the human screeners. The resulting scores are presented in Table Table 2. Each column corresponds to a team score, while the rows represent specific subsets of four screeners (A–F). The penultimate row reports the mean score across all subgroups within a team, and the final row provides the overall mean across all teams. Because not every screener evaluated every paper, certain combinations of screeners yield no 𝜅 values. In these cases, there were no papers jointly assessed by all members of the respective subgroup. This limitation is particularly evident in teams consisting of four screeners, where only one subgroup produces a valid value, as illustrated in Table 2 for Team 01. A similar situation arises in six-member teams, where certain subgroups lack overlap in screened papers. After averaging the subgroup scores within each team, kappa values range from 𝜅=0.24 to 0.65 , with an overall mean of 𝜅=0.39 . According to the interpretation proposed in [20], 6 out of 10 teams fall into the category of “fair agreement.” Three teams achieve scores between 0.40 and 0.60, corresponding to “moderate agreement,” while Team 5 attains “substantial agreement” with a 𝜅=0.65 . These findings show high variability across teams and subgroups, suggesting that the limited prior experience of the screeners with the SR process contributed to inconsistent decisionmaking within the teams. When assessing inter-rater reliability between the human consensus and the decisions generated by the LLM, the resulting agreement levels are comparable to those observed among human raters themselves. Because the analysis is 4https://zenodo.org/records/17113018 based on pairwise agreement between two raters, Cohen’s 𝜅 was used as the reliability measure. As detailed in Table 3, the observed values ranged from minimal agreement ( 𝜅= 0.26 ) to strong agreement on a single occasion ( 𝜅=0.80 ). On average, the agreement between human raters and the LLM was 𝜅=0.52 , which is classified as weak agreement according to [21]. These results indicate that the level of agreement between human raters and the LLM is similar to the degree of agreement observed among human raters. Performance Evaluation In the performance analysis of this study, the classification accuracy of human screeners and LLMs was evaluated under different conditions and configurations, benchmarked against the ground truth described in the Methodology section. The corresponding results are summarized in Table Table 4, which reports the sensitivity and specificity across the four defined experimental setups. The first row reflects the ground truth, which is the consensus of four independent human screeners, with conflicts resolved through discussion. The following rows show the performance of the defined experimental setups. The findings demonstrate that human screeners consistently outperform the LLM with respect to both sensitivity and specificity, irrespective of whether single or double screening is applied. A single human screener achieved a sensitivity of 84.03% and a specificity of 90.36%. In contrast, the LLM archived 80.30% sensitivity and 85.50% specificity. When double screening is used, sensitivity increased to 99.18%, but at the expense of specificity, which decreased to 82.10%. This trade-off shows the primary advantage of double screening, where the likelihood of omitting relevant studies decreases, but in return more irrelevant studies are included, leading to a lower specificity. Another configuration involved pairing a single human screener with an LLM in a double-screening arrangement. In this setting, the combined system achieved a sensitivity of 94.58% and a specificity of 79.22%, representing an approximate 10% increase in sensitivity relative to a single human screener. As expected, specificity was reduced compared to either the human or the LLM alone. Discussion The inter-rater reliability results, as measured by Cohen’s 𝜅 values reported above, demonstrate that novice reviewers show considerable inconsistency in their screening decisions. This outcome is not unexpected, as the reviewers were required to evaluate pre-defined research questions in domains that were, at times, unfamiliar to them. Moreover, the fact that each team produced the eligibility criteria by themselves, introduced additional interpretive flexibility, which in turn contributed to disagreement regarding the inclusion of specific studies. Only one team achieved a level of “substantial agreement,” suggesting higher consistency within this group. Possible explanations include prior domain knowledge, prior https://doi.org/10.5281/zenodo.17234055 Table 2: Fleiss’ 𝜅for four human screeners across teams. Team Number 01 02 03 04 05 06 07 08 09 10 A, B, C, D 0.25 0.73 - -0.03 0.66 0.25 0.65 - 0.76 0.30 A, B, E, F - - 0.43 - 0.42 - 0.03 - 0.59 0.32 C, D, E, F - - 0.63 - 0.54 - 0.67 - 0.43 0.59 A, B, C, F - - -0.07 - 0.76 - 0.56 - 0.43 0.09 B, C, D, E - 0.45 -0.04 1.00 0.85 0.51 0.76 - 0.11 0.01 A, C, D, E - 0.13 - 0.43 - 0.22 - - - - A, B, C, E - -0.04 - 0.26 - 0.37 - - - - A, B, D, E - 0.29 - -0.07 - 0.56 - 0.45 - - Team average 0.25 0.31 0.24 0.32 0.65 0.38 0.53 0.45 0.46 0.26 Average 0.38 Table 3: Cohen’s 𝜅between human consensus and LLM across teams. Team Number 01 02 03 04 05 06 07 08 09 10 Human Consensus, LLM 0.52 0.26 0.33 0.77 0.80 0.35 0.36 0.69 0.54 0.54 Average 0.52 Table 4: Screening performance: sensitivity and specificity across scenarios. Screening Scenario Sensitivity Specificity Consensus of 4 human screeners (Ground-truth) 100.00% 100.00% Single human 84.03% 90.36% LLM only 80.30% 85.50% Double human 99.18% 82.10% Single human and LLM 94.98% 79.22% experience with the SR process, or potential bias introduced by working together in this individual task. Importantly, the non-random nature of these findings is supported by the fact that Cohen’s kappa explicitly accounts for agreement beyond chance. A similar pattern appears when comparing team-level screening decisions with those generated by the LLM. The corresponding 𝜅 values vary considerably across teams, ranging from almost no agreement to very strong agreement. Notably, one team that demonstrated only moderate internal agreement suddenly aligned more strongly with the LLM at the team level. As mentioned in the Methodology, however, these results must not be interpreted as evidence of validity, since the applied 𝜅 values reflect relative agreement rather than correspondence to a definitive ground truth. The only conclusion that can be drawn from these results is that the screeners agree with the LLM to about the same extent as they agree with one another. When looking at the performance analysis of this work, it becomes clear that double blinded screening clearly outperforms single screening. This is the standard in most SRs, since especially in medical domains this is a critical step to ensure that no relevant studies are missed. Nevertheless, lower specificity can substantially increase the workload during the full-text screening stage, which remains an important consideration, as this phase is very time-intensive as well. The observed reduction in specificity of approximately 8% is explainable with the fact that any screening conflicts are resolved in favor of inclusion. Under this rule, specificity cannot mathematically increase, as the number of exclusions in the double-screening setting is necessarily less than or equal to that in single screening. These findings are in line with observations reported in [13], where single screening showed comparable levels of performance, and the introduction of double screening led to a significant improvement in sensitivity. At the same time, a similar trade-off was observed in the form of reduced specificity, which shows the inherent balance between maximizing the detection of relevant studies and managing the additional workload created by lower exclusion rates. When comparing the performance of the LLM to the ground truth, human screeners demonstrated higher sensitivity, both under single screening and double screening conditions. This difference may be partly due to the baseline design, which was intentionally constructed to avoid bias in favor of the LLMs. When a single human screener was combined with the LLM, effectively creating a double screening scenario, sensitivity improved, although it did not reach the level achieved by double human screening. While the evaluated LLM configurations are not yet suitable as replacements for human screeners, they may serve as valuable additions in situations where resources for a second human screener are not available, enhancing the overall quality of the screening process. While studies such as [14] report lower performance of human screeners, it is important to note that screening comhttps://doi.org/10.5281/zenodo.17234055 plexity strongly depends on factors such as the domain or the complexity of eligibility criteria. Furthermore, human screening performance likely decreases with an increasing number of studies to be screened. LIMITATIONS AND FUTURE WORK The present study was designed with deliberate caution to avoid overstating the capabilities of LLM-based screening. Its purpose was not to promote automation, but to transparently explore under which conditions LLMs may substitute or complement human screeners. This cautious design led to several limitations that must be considered when interpreting the findings. First, the gold standard was derived from the consensus of human screeners, which inherently biases the evaluation in their favor. This effect is particularly pronounced for double human screening, where the consensus process directly shaped the reference against which alternative scenarios were judged. Relatedly, the eligibility criteria were defined by the student teams themselves and expressed in natural language. Such criteria allow interpretive flexibility, and individual screeners likely applied a consistent internal interpretation that aligned with the team consensus. The LLM, by contrast, had to infer the intended meaning without this implicit alignment, which disadvantaged its performance. Second, the study setting differs substantially from realworld SRs. Each student screened only about 30 records, far fewer than the hundreds or thousands typically encountered in practice. As a result, important factors such as fatigue or consistency over time could not be captured. Furthermore, the course-based environment meant that screening carried no real-world consequences, unlike professional SR projects where errors may directly impact research or policy decisions. Finally, the evaluation of the LLM was restricted to a single prompting configuration and a single model version. This narrow setup does not capture the potential variability in performance that may arise from alternative prompting strategies, different thresholds for inclusion, or comparisons across multiple LLMs. Building on these limitations, several directions for future work emerge. Expert-annotated gold standards should be established to reduce bias in favor of human screeners and enable more rigorous benchmarking. The robustness of eligibility criteria could be tested by comparing team-defined formulations with expert-reviewed standards. Larger-scale studies are needed to assess long-term screening behavior, while experiments should include participants with different levels of SR expertise. Finally, more advanced evaluations of LLMs are warranted, exploring diverse prompting strategies and multiple models to determine the extent to which automation can complement or substitute human screeners. CONCLUSION This study examined the reliability and performance of novice human screeners compared with a LLM in the context of literature screening. Based on data from 54 students across ten IR topics, this study provides empirical evidence on how first-time screeners behave when tasked with TiAbscreening and how their outcomes compare to those of an automated system. The findings demonstrate substantial variability among novice reviewers. Inter-rater reliability ranged from fair to moderate, with only one team achieving substantial agreement. These results confirm that human screening is not immune to inconsistency, particularly when reviewers lack domain expertise and prior experience with SR methodology. At the same time, the LLM achieved an agreement level with human consensus that was similar to the agreement observed among humans themselves, suggesting that automation can emulate human-like decision variability. In terms of performance, single human screeners outperformed the LLM, but the gap was modest. Double human screening remained the most effective approach, achieving nearly complete sensitivity. This reinforces the importance of double-blind screening in settings where missing relevant studies carries significant consequences. However, the hybrid configuration of one human and one LLM also proved promising, reaching a sensitivity level close to double human screening while using fewer human resources. Such a setup may offer a practical compromise in resource-constrained environments. Overall, the study underscores two key points. First, the reliability of novice screeners is limited, highlighting the importance of training and quality assurance in SRs. Second, while the considered LLM based algorithm cannot replace human screeners, it may serve as a valuable complements that increases sensitivity and reduces the risk of overlooking relevant studies when additional human reviewers are not available. These insights contribute to a more balanced perspective on the role of automation in evidence synthesis, providing a foundation for cautious but progressive integration of LLMs into the SR process. ACKNOWLEDGEMENTS The joint CERN and WHO ARIA 5 project is funding the PhD project, in the context of which this paper was written. Furthermore, we greatly thank OpenWebSearch.EU 6 project and the members for their help and support with this publication. REFERENCES [1] Higgins, J. P. T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. J., and Welch, V. A. (eds.). Cochrane Handbook for Systematic Reviews of Interventions, version 6.5 (updated August 2024). Cochrane, 2024. Available from www.cochrane.org/handbook. [2] E. Sandner, L. Fontana, K. Kothari, A. Henriques, I. Jakovljevic, A. Simniceanu, A. Wagner, and C. Gütl, “Evaluating 5https://partnersplatform.who.int/tools/aria 6https://openwebsearch.eu https://doi.org/10.5281/zenodo.17234055 Large Language Models for Literature Screening: A Systematic Review of Sensitivity and Workload Reduction,” 2025. [3] W. M. Bramer, M. L. Rethlefsen, J. Kleijnen, and O. H. Franco, “Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study,” Systematic Reviews, vol. 6, pp. 1–12, 2017. [4] M. W. Callaghan and F. Müller-Hansen, “Statistical stopping criteria for automated screening in systematic reviews,” Systematic Reviews, vol. 9, pp. 1–14, 2020. [5] J. Thomas, S. McDonald, A. Noel-Storr, I. Shemilt, J. Elliott, C. Mavergames, and I. J. Marshall, “Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for Cochrane Reviews,” Journal of Clinical Epidemiology, vol. 133, pp. 140–151, 2021. [6] E. Sandner, B. Hu, A. Simiceanu, L. Fontana, I. Jakovljevic, A. Henriques, A. Wagner, and C. Gütl, "Screening Automation for Systematic Reviews: A 5-Tier Prompting Approach Meeting Cochrane’s Sensitivity Requirement," Proceedings of the 2024 2nd International Conference on Foundation and Large Language Models (FLLM), pp. 150–159, 2024. [7] P. G. Shekelle, M. A. Maglione, J. Luoto, et al., Global Health Evidence Evaluation Framework, Agency for Healthcare Research and Quality (US), Rockville, MD, 2013. Available at: https://www.ncbi.nlm.nih.gov/books/ NBK121300/table/appb.t21/. [8] D. J. Cook, N. L. Greengold, A. G. Ellrodt, and S. R. Weingarten, “The relation between systematic reviews and practice guidelines,” Annals of Internal Medicine, vol. 127, no. 3, pp. 210–216, 1997. [9] E. M. Beller, J. K.-H. Chen, U. L.-H. Wang, and P. P. Glasziou, “Are systematic reviews up-to-date at the time of publication?,” Systematic Reviews, vol. 2, pp. 1–6, 2013. [10] M. R. Demetres, D. N. Wright, A. Hickner, C. Jedlicka, and D. Delgado, “A decade of systematic reviews: an assessment of Weill Cornell Medicine’s systematic review service,” Journal of the Medical Library Association: JMLA, vol. 111, no. 3, p. 728, 2023. [11] R. Borah, A. W. Brown, P. L. Capers, and K. A. Kaiser, “Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry,” BMJ Open, vol. 7, no. 2, p. e012545, 2017. [12] J. C. Carver, E. Hassler, E. Hernandes, and N. A. Kraft, “Identifying barriers to the systematic literature review process,” in Proceedings of the 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 203–212, 2013. [13] G. Gartlehner, L. Affengruber, V. Titscher, A. Noel-Storr, G. Dooley, N. Ballarini, and F. König, “Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial,” Journal of Clinical Epidemiology, vol. 121, pp. 20–28, 2020. [14] M. Issaiy, H. Ghanaati, S. Kolahi, M. Shakiba, A. H. Jalali, D. Zarei, S. Kazemian, M. A. Avanaki, and K. Firouznia, “Methodological insights into ChatGPT’s screening performance in systematic reviews,” BMC Medical Research Methodology, vol. 24, no. 1, p. 78, 2024. [15] Clarke M, Hopewell S, Chalmers I. Clinical trials should begin and end with systematic reviews of relevant evidence: 12 years and waiting. The Lancet, 2010;376(9734):20–21. [16] Robinson KA, Brunnhuber K, Ciliska D, Juhl CB, Christensen R, Lund H. Evidence-based research series–paper 1: what evidence-based research is and why it is important? Journal of Clinical Epidemiology, 2021;129:151–157. [17] Lund H, Juhl CB, Nørgaard B, Draborg E, Henriksen M, Andreasen J, et al. Evidence-based research series–paper 2: using an evidence-based research approach before a new study is conducted to ensure value. Journal of Clinical Epidemiology, 2021;129:158–166. [18] SIGIR ’24: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, Association for Computing Machinery, New York, NY, USA, 2024. ISBN: 9798400704314. [19] A. Carrera-Rivera, W. Ochoa, F. Larrinaga, and G. Lasa, “How-to conduct a systematic literature review: A quick guide for computer science research,” MethodsX, vol. 9, p. 101895, 2022. [20] Hartling L, Hamm M, Milne A, Vandermeer B, Santaguida PL, Ansari M, Tsertsvadze A, Hempel S, Shekelle P, Dryden DM. "Validity and Inter-Rater Reliability Testing of Quality Assessment Instruments [Internet]." Rockville (MD): Agency for Healthcare Research and Quality (US); 2012 Mar. Report No.: 12-EHC039-EF. PMID: 22536612. [21] McHugh ML. "Interrater reliability: the kappa statistic." Biochem Med (Zagreb). 2012;22(3):276-82 https://doi.org/10.5281/zenodo.17234055