Innovation and forward-thinking are needed to improve traditional synthesis methods: A response to Pescott and Stewart

This document is the Accepted Manuscript version of a Published Work that appeared in final form in: Christie, A.P.; Amano, T.; Martin, P.A.; Shackelford, G.E.; Simmons, B.I.; Sutherland, W.J.2022. Plural valuation of nature for equity and sustainability: Insights from the Global South. Journal of Applied Ecology. 59. DOI (10.1111/1365-2664.14154). © 2022 British Ecological Society. This manuscript version is made available under the CC-BY-NC-ND 3.0 license http://creativecommons.org/licenses/by-nc-nd/3.0/ Innovation and forward-thinking are needed to improve traditional 1 synthesis methods: a response to Pescott & Stewart 2 3 Alec P. Christie1,4,7*, Tatsuya Amano1,2,3, Philip A. Martin1,4,8, Gorm E. Shackelford1,4,4 Benno I. Simmons1,5,6, William J. Sutherland1,4 5 1Conservation Science Group, Department of Zoology, University of Cambridge, The David 6 Attenborough Building, Downing Street, Cambridge, UK. 7 2Centre for the Study of Existential Risk, University of Cambridge, 16 Mill Lane, Cambridge, UK. 8 3School of Biological Sciences, University of Queensland, Brisbane, 4072 Queensland, Australia 9 4BioRISC, St Catharine’s College, Cambridge, UK. 10 5Department of Animal and Plant Sciences, University of Sheffield, Sheffield, UK. 11 6Centre for Ecology and Conservation, College of Life and Environmental Sciences, University of 12 Exeter, Penryn, UK. 13 7Downing College, Regent Street, Cambridge, UK. 14 8Basque Centre for Climate Change (BC3), Edificio sede no 1, planta 1, Parque científico 15 UPV/EHU, Barrio Sarriena s/n, 48940, Leioa, Bizkaia, Spain16 17 *Corresponding author, a[email protected].uk18 19 20 21 22 23 Abstract 24 1. In Christie et al. (2019), we used simulations to quantitatively compare the bias of 25 commonly used study designs in ecology and conservation. Based on these simulations, 26 we proposed ‘accuracy weights’ as a potential way to account for study design validity in 27 meta-analytic weighting methods. Pescott & Stewart (2021) raised concerns that these 28 weights may not be generalisable and still lead to biased meta-estimates. Here we 29 respond to their concerns and demonstrate why developing alternative weighting 30 methods is key to the future of evidence synthesis. 31 2. We acknowledge that our simple simulation unfairly penalised Randomised Controlled 32 Trial (RCT) relative to Before-After Control-Impact (BACI) designs as we assumed that 33 the parallel trends assumption held for BACI designs. We point to an empirical follow-up 34 study in which we more fairly quantify differences in biases between different study 35 designs. However, we stand by our main findings that Before-After (BA), Control-Impact 36 (CI), and After designs are quantifiably more biased than BACI and RCT designs. We 37 also emphasise that our 'accuracy weighting’ method was preliminary and welcome 38 future research to incorporate more dimensions of study quality. 39 3. We further show that over a decade of advances in quality effect modelling, which 40 Pescott & Stewart (2021) omit, highlights the importance of research such as ours in 41 better understanding how to quantitatively integrate data on study quality directly into 42 meta-analyses. We further argue that the traditional methods advocated for by Pescott & 43 Stewart (2021) (e.g., manual risk-of-bias assessments and inverse-variance weighting) 44 are subjective, wasteful, and potentially biased themselves. They also lack scalability for 45 use in large syntheses that keep up-to-date with the rapidly growing scientific literature. 46 4. Synthesis and applications. We suggest, contrary to Pescott & Stewart’s narrative, that 47 moving towards alternative weighting methods is key to future-proofing evidence 48 synthesis through greater automation, flexibility, and updating to respond to decision49 makers needs – particularly in crisis disciplines in conservation science where 50 problematic biases and variability exist in study designs, contexts, and metrics used. 51 Whilst we must be cautious to avoid misinforming decision-makers, this should not stop 52 us investigating alternative weighting methods that integrate study quality data directly 53 into meta-analyses. To reliably and pragmatically inform decision-makers with science, 54 we need efficient, scalable, readily automated, and feasible methods to appraise and 55 weight studies to produce large-scale living syntheses of the future. 56 57 Keywords: evidence synthesis, meta-analysis, dynamic meta-analysis, living reviews, 58 automation, quality effects modelling, meta-analyses, risk-of-bias, critical appraisal, bias 59 adjustment. 60 61 Introduction 62 63 Pescott & Stewart (2021) outlined their concerns over an alternative method of weighting in 64 meta-analysis we proposed called “accuracy weights” in Christie et al. (2019). These weights 65 were derived from our simulation study that aimed to quantitatively compare the performance of 66 different experimental and observational study designs (Christie et al., 2019). Their two major 67 concerns were that our accuracy weights were not generalisable and that quality score 68 weightings, such as ours, may still lead to biased estimates in meta-analyses. Here we respond 69 to their concerns and discuss why we believe alternative methods of weighting are central to the 70 future of evidence synthesis. 71 72 1. Accuracy weights need improving and combining with other quality measures 73 74 As Pescott & Stewart suggest, we acknowledge that our simulation may have unfairly penalised 75 Randomised Controlled Trial (RCT) designs, depending on whether researchers in ecology and 76 conservation do take into account pre-impact sampling. However, in our experience, few 77 Randomised Controlled Trials in conservation take account of pre-impact baseline data; this is 78 supported by a recent study quantifying the use of different study designs in the environmental 79 and social sciences (Christie et al., 2020a). We acknowledge that we did not discuss more of 80 the shortcomings of Before-After Control-Impact (BACI) designs in terms of the bias that can be 81 introduced by violating the ‘parallel trends’ assumption (Dimick and Ryan, 2014; Underwood, 82 1991; Wauchope et al., 2020). Therefore, with respect to comparing BACI and RCT designs, we 83 acknowledge our simulation has limitations. 84 85 Nevertheless, our major motivation was to demonstrate the difference in study design 86 performance between simpler designs (e.g., Before-After (BA), Control-Impact (CI), and After 87 designs) and more rigorous designs (RCT and BACI). Thus, we intentionally made our 88 simulation relatively simple to engage a wide audience of researchers. We have since built on 89 our simulations in Christie et al. (2020a), which uses an empirical, model-based methodology to 90 quantify the differences in bias affecting different study designs using raw (rather than 91 simulated) data from a large number of within-study comparisons. This more fairly quantifies the 92 bias associated with RCT versus BACI designs by making fewer, more statistically defensible 93 assumptions about the ‘true effect’ (to estimate bias) and inherently accounts for the parallel 94 trends assumption that can bias BACI designs (Christie et al., 2020a). 95 96 Pescott & Stewart also suggest our simulation weights do not capture the full range of potential 97 sources of bias affecting study designs and advise that assessments of study quality should 98 closely scrutinise the details of specific studies being summarised (e.g., using manual risk-of99 bias assessments). In our study, we specifically acknowledged that our weights were relatively 100 simple and need to be built upon to incorporate a wider range of study quality indicators; we 101 outlined possible approaches in the future that could integrate scores from critical appraisal 102 tools that exist for ecology and conservation (Mupepele et al., 2016). We are happy to see that 103 others are building on our work and investigating the use of a broader set of quality or validity 104 measures to weight studies in meta-analyses (e.g., Schafft et al. 2021, Mupepele et al. 2021). In 105 the next sections, we address Pescott & Stewart’s criticisms of weighting by quality scores and 106 discuss statistical advances in applying quality score weightings to meta-analyses. We also 107 discuss the problems associated with the traditional methods advocated for by Pescott & 108 Stewart (such as inverse-variance weighting and manual risk-of-bias assessments). 109 110 2. Recent advances in directly integrating data on study quality into meta111 analyses 112 113 In Pescott & Stewart's discussion on why they advocate against weighting by quality scores in 114 meta-analyses, they omit over a decade of research in epidemiology on alternative quality score 115 weighting methods that have overcome many of the problems they discuss (Doi, Barendregt 116 and Mozurkewich, 2011; Doi et al., 2015a, 2015b; Doi and Thalib, 2008; Rhodes et al., 2020; 117 Stone et al., 2020). In particular, ‘bias adjustment’ methods, such as quality effects models, 118 represent an active and promising area of research in evidence synthesis in epidemiology (Doi, 119 Barendregt and Mozurkewich, 2011; Doi and Thalib, 2008; Rhodes et al., 2020; Stone et al., 120 2020). 121 122 Critical appraisal is traditionally used to descriptively report the risk of bias for different studies, 123 rather than trying to quantitatively incorporate those assessments within the analyses 124 themselves (Johnson, Low and MacDonald, 2015). Instead, our accuracy weights are related to 125 the field of ‘bias-adjustment’ methods which seek to directly integrate risk-of-bias assessments 126 into meta-analytic results (Stone et al., 2020). Criticisms of quality score weightings have 127 centered around four major issues: 1.) the choice of quality scale influences the weight of 128 individual studies; 2.) the meta-estimate and its confidence interval depends on the scale; 3.) 129 there is no reason why study quality should modify the precision of estimates; and 4.) poor 130 studies are not excluded (Stone et al., 2020). Therefore, as Pescott & Stewart also appear to 131 argue, any bias associated with poor quality studies can only be reduced at best, and not 132 removed (Stone et al., 2020). 133 134 Whilst proponents of quality score approaches accepted these criticisms and ceased their 135 development, an alternative, improved methodology called ‘quality effects models’ have 136 subsequently been developed and refined in recent years. This approach uses a relative scale 137 and ‘synthetic weights’ (yielding relative credibility ranks for different studies) that overcame the 138 major issues that affected quality score approaches, and has been shown to yield an estimator 139 with superior error and coverage to conventional estimators (Doi et al., 2015b, 2017). There are 140 a range of possible ways, each with advantages or disadvantages, to derive the relative 141 credibility weights for studies using numerical data generated by expert opinion (Turner et al., 142 2009), data-based distributions, or statistically combining expert opinion and data-based 143 distributions (Rhodes et al., 2020). Therefore, results from further refining and improving our 144 simulations and empirical analyses (Christie et al., 2019; Christie et al., 2020a) could provide 145 valuable contributions to the active development of these methods to integrate data on study 146 quality directly into meta-analyses. 147 148 Pescott & Stewart focus on the possibility of incorporating study quality scores into meta149 regression approaches. Their criticism of our weights in their current form is that they are too 150 unidimensional and not study-specific; this is a criticism that we partially accept. Indeed, we 151 specifically discussed the need to expand and improve our weights to integrate other aspects of 152 study quality (e.g., using expert opinion, data-based distributions, or critical appraisal tools to 153 adjust relative credibility ranks; Rhodes et al., 2020). In hindsight, we should have dedicated 154 more attention to how we would further develop and more robustly apply our accuracy weights 155 alongside discussing advances in quality effects models. 156 157 Pescott & Stewart also suggest that we ignore issues relating to external validity. Given that 158 traditional weightings, such as sample size or inverse variance, also fail to consider external 159 validity, we find this an odd criticism, particularly given our simulation was clearly focused on 160 addressing issues of study design quality and internal validity. We are in fact developing an 161 alternative meta-analytic method, dynamic meta-analysis (Shackelford et al., 2021), based on 162 the Metadataset platform (www.metadataset.com), which we plan to use to test different 163 weighting methods, including ‘recalibration’ from the medical sciences (Kneale et al., 2019) 164 which aims to adjust studies’ influence in meta-analyses based on their external validity (or 165 relevance to decision-makers). Again this work is in the early stages of development and there 166 are many methodological challenges to overcome, particularly in how to integrate ‘recalibration’ 167 methods into random effects models and how to ensure such interactive meta-analytic tools are 168 used robustly (Shackelford et al. 2021). Therefore, as Pescott & Stewart suggest, we believe it 169 should be possible to integrate internal validity or quality items, and external validity items, into a 170 hierarchical meta-regression framework, or to directly weight studies using new advances in 171 quality effects models as discussed previously (see Stone et al. 2020 for a comparison and 172 discussion of different approaches). 173 174 175 3. Integrating data on study quality into meta-analyses is essential to the future of 176 evidence synthesis 177 178 We also believe Pescott & Stewart’s discussion presents a narrow vision of the challenges 179 faced by traditional critical appraisal and weighting methods. We believe that the traditional 180 ‘medical-style’ approaches (e.g., manual risk-of-bias assessments combined with inverse181 variance weighting) that Pescott & Stewart believe should be adhered to are ultimately 182 inefficient and wasteful. The field of evidence synthesis is advancing at pace to respond to the 183 challenges of rapidly growing evidence bases and fast-moving crises, which requires new 184 methodologies that help to keep evidence bases ‘up-to-date’ or ‘living’, cost-efficient by working 185 at massive discipline-wide scales, and dynamically adjustable to be relevant to different 186 decision-makers’ needs. Here we elaborate on why this is problematic to Pescott & Stewart’s 187 assertion that we should continue to rely on traditional methods, rather than alternative 188 weighting methods such as the one we proposed in Christie et al. (2019). 189 190 3.a. Alternative weighting methods facilitate more efficient, automated, living, large-scale 191 syntheses 192 193 First, there is growing recognition that decision makers need constantly updated evidence 194 syntheses (Elliot et al., 2021) and that traditional synthesis methods (e.g., traditional systematic 195 reviews) are often too time-consuming, quickly go out-of-date, and can miss important 196 opportunities to influence practice and policy (Boutron et al., 2020; Grainger et al., 2019; 197 Haddaway and Westgate, 2019; Koricheva and Kulinskaya, 2019; Nakagawa et al., 2020; 198 Pattanittum et al., 2012; Shojania et al., 2007). Given that the scientific literature in most 199 disciplines is growing rapidly (Bornmann and Mutz, 2015; Larsen and von Ins, 2010) and that 200 Borah, R., Brown, A.W., Capers, P.L. and Kaiser, K.A. (2017) “Analysis of the timeand workers 356 needed to conduct systematic reviews of medicalinterventions using data from the PROSPERO 357 registry,” BMJ, Open7:e012545. 358 Bornmann, L. and Mutz, R. (2015) “Growth rates of modern science: A bibliometric analysis 359 based on the number of publications and cited references,” Journal of the Association for 360 Information Science and Technology, 66(11) John Wiley & Sons, Ltd, pp. 2215–2222. 361 Boutron, I., Créquit, P., Williams, H., Meerpohl, J., Craig, J.C. and Ravaud, P. (2020) “Future of 362 evidence ecosystem series: 1. Introduction — Evidence synthesis ecosystem needs dramatic 363 change,” Journal of Clinical Epidemiology, Elsevier Inc. 364 Christie, A.P., Abecasis, D., Adjeroud, M., Alonso, J.C., Amano, T., Anton, A., Baldigo, B.P., 365 Barrientos, R., Bicknell, J.E., Buhl, D.A., Cebrian, J., Ceia, R.S., Cibils-Martina, L., Clarke, S., 366 Claudet, J., Craig, M.D., Davoult, D., de Backer, A., Donovan, M.K., Eddy, T.D., França, F.M., 367 Gardner, J.P.A., Harris, B.P., Huusko, A., Jones, I.L., Kelaher, B.P., Kotiaho, J.S., López368 Baucells, A., Major, H.L., Mäki-Petäys, A., Martín, B., Martín, C.A., Martin, P.A., Mateos-Molina, 369 D., McConnaughey, R.A., Meroni, M., Meyer, C.F.J., Mills, K., Montefalcone, M., Noreika, N., 370 Palacín, C., Pande, A., Pitcher, C.R., Ponce, C., Rinella, M., Rocha, R., Ruiz-Delgado, M.C., 371 Schmitter-Soto, J.J., Shaffer, J.A., Sharma, S., Sher, A.A., Stagnol, D., Stanley, T.R., 372 Stokesbury, K.D.E., Torres, A., Tully, O., Vehanen, T., Watts, C., Zhao, Q. and Sutherland, W.J. 373 (2020a) “Quantifying and addressing the prevalence and bias of study designs in the 374 environmental and social sciences,” Nature Communications, 11(1), p. 6377. 375 Christie, A.P., Amano, T., Martin, P.A., Petrovan, S.O., Shackelford, G.E., Simmons, B.I., Smith, 376 R.K., Williams, D.R., Wordley, C.F.R. and Sutherland, W.J. (2020b) “Poor availability of context377 specific evidence hampers decision-making in conservation,” Biological Conservation, 248 Cold 378 Spring Harbor Laboratory, p. 108666. 379 Christie, A.P., Amano, T., Martin, P.A., Petrovan, S.O., Shackelford, G.E., Simmons, B.I., Smith, 380 R.K., Williams, D.R., Wordley, C.F.R. and Sutherland, W.J. (2020c) “The challenge of biased 381 evidence in conservation,” Conservation Biology, Wiley for Society for Conservation Biology, p. 382 cobi.13577. 383 Christie, A.P., Amano, T., Martin, P.A., Shackelford, G.E., Simmons, B.I. and Sutherland, W.J. 384 (2019) “Simple study designs in ecology produce inaccurate estimates of biodiversity 385 responses,” Journal of Applied Ecology, 56(12) John Wiley & Sons, Ltd (10.1111), pp. 2742– 386 2754. 387 Christie, A.P., White, T.B., Martin, P.A., Petrovan, S.O., Bladon, A.J., Bowkett, A.E., Littlewood, 388 N.A., Mupepele, A.C., Rocha, R., Sainsbury, K.A., Smith, R.K., Taylor, N.G., Sutherland, W.J., 389 2021. Reducing publication delay to improve the efficiency and impact of conservation science. 390 PeerJ 9, e12245. https://doi.org/10.7717/PEERJ.12245/SUPP-17 391 Cornford, R., Deinet, S., de Palma, A., Hill, S.L.L., McRae, L., Pettit, B., Marconi, V., Purvis, A. 392 and Freeman, R. (2021) “Fast, scalable, and automated identification of articles for biodiversity 393 and macroecological datasets,” Global Ecology and Biogeography, 30(1) John Wiley & Sons, 394 Ltd, pp. 339–347. 395 Davies, G.M. and Gray, A. (2015) “Don’t let spurious accusations of pseudoreplication limit our 396 ability to learn from natural experiments (and other messy kinds of ecological monitoring),” 397 Ecology and Evolution, 5(22) John Wiley & Sons, Ltd, pp. 5295–5304. 398 Dimick, J.B. and Ryan, A.M. (2014) “Methods for Evaluating Changes in Health Care Policy,” 399 JAMA, 312(22), p. 2401. 400 Doi, S.A.R., Barendregt, J.J., Khan, S., Thalib, L. and Williams, G.M. (2015a) “Advances in the 401 meta-analysis of heterogeneous clinical trials II: The quality effects model,” Contemporary 402 Clinical Trials, 45 Elsevier Inc., pp. 123–129. 403 Doi, S.A.R., Barendregt, J.J., Khan, S., Thalib, L. and Williams, G.M. (2015b) “Simulation 404 Comparison of the Quality Effects and Random Effects Methods of Meta-analysis,” 405 Epidemiology, 26(4) Lippincott Williams & Wilkins, pp. e42–e44. 406 Doi, S.A.R., Barendregt, J.J. and Mozurkewich, E.L. (2011) “Meta-analysis of heterogeneous 407 clinical trials: An empirical example,” Contemporary Clinical Trials, 32(2) Elsevier Inc., pp. 288– 408 298. 409 Doi, S.A.R., Furuya-Kanamori, L., Thalib, L. and Barendregt, J.J. (2017) “Meta-analysis in 410 evidence-based healthcare: A paradigm shift away from random effects is overdue,” 411 International Journal of Evidence-Based Healthcare, 15(4) Lippincott Williams and Wilkins, pp. 412 152–160. Available at: 10.1097/XEB.0000000000000125 (Accessed: January 25, 2021). 413 Doi, S.A.R. and Thalib, L. (2008) “A quality-effects model for meta-analysis,” Epidemiology, 414 19(1), pp. 94–100. 415 Doncaster, C.P., Spake, R., 2018. Correction for bias in meta-analysis of little-replicated 416 studies. Methods Ecol. Evol. 9, 634–644. https://doi.org/https://doi.org/10.1111/2041417 210X.12927 418 Elliott, J., Lawrence, R., Minx, J.C., Oladapo, O.T., Ravaud, P., Tendal Jeppesen, B., Thomas, 419 J., Turner, T., Vandvik, P.O., Grimshaw, J.M., 2021. Decision makers need constantly updated 420 evidence synthesis. Nat. 2021 6007889 600, 383–385. https://doi.org/10.1038/d41586-021421 03690-1 422 Gough, D. and White, H. (2018) Evidence standards and evidence claims in web based 423 research portals. London. 424 Grainger, M.J., Bolam, F.C., Stewart, G.B. and Nilsen, E.B. (2019) “Evidence synthesis for 425 tackling research waste,” Nature Ecology & Evolution, Springer US 426 Gutzat, F. and Dormann, C.F. (2020) “Exploration of Concerns about the Evidence-Based 427 Guideline Approach in Conservation Management: Hints from Medical Practice,” Environmental 428 Management, 66(3), pp. 435–449. 429 Haddaway, N.R. and Westgate, M.J. (2019) “Predicting the time needed for environmental 430 systematic reviews and systematic maps,” Conservation Biology, 33(2) John Wiley & Sons, Ltd, 431 pp. 434–443. 432 Hamman, E.A., Pappalardo, P., Bence, J.R., Peacor, S.D. and Osenberg, C.W. (2018) “Bias in 433 meta-analyses using Hedges’ d,” Ecosphere, 9(9) John Wiley & Sons, Ltd, p. e02419. 434 Jenicek, M. (1989) “Meta-analysis in medicine Where we are and where we want to go,” Journal 435 of Clinical Epidemiology, 42(1), pp. 35–44. 436 Johnson, B.T., Low, R.E. and MacDonald, H. v. (2015) “Panning for the gold in health research: 437 Incorporating studies’ methodological quality in meta-analysis,” Psychology and Health, 30(1) 438 Routledge, pp. 135–152. Available at: 10.1080/08870446.2014.953533 (Accessed: January 25, 439 2021). 440 Junker, J., Petrovan, S.O., Arroyo-RodrÍguez, V., Boonratana, R., Byler, D., Chapman, C.A., 441 Chetry, D., Cheyne, S.M., Cornejo, F.M., CortÉs-Ortiz, L., Cowlishaw, G., Christie, A.P., 442 Crockford, C., Torre, S.D. la, de Melo, F.R., Fan, P., Grueter, C.C., GuzmÁn-Caro, D.C., 443 Heymann, E.W., Herbinger, I., Hoang, M.D., Horwich, R.H., Humle, T., Ikemeh, R.A., Imong, 444 I.S., Jerusalinsky, L., Johnson, S.E., Kappeler, P.M., Kierulff, M.C.M., KonÉ, I., Kormos, R., Le, 445 K.Q., Li, B., Marshall, A.J., Meijaard, E., Mittermeier, R.A., Muroyama, Y., Neugebauer, E., Orth, 446 L., Palacios, E., Papworth, S.K., Plumptre, A.J., Rawson, B.M., Refisch, J., Ratsimbazafy, J., 447 Roos, C., Setchell, J.M., Smith, R.K., Sop, T., Schwitzer, C., Slater, K., Strum, S.C., Sutherland, 448 W.J., Talebi, M., Wallis, J., Wich, S., Williamson, E.A., Wittig, R.M. and KÜhl, H.S. (2020) “A 449 Severe Lack of Evidence Limits Effective Conservation of the World’s Primates,” BioScience, 450 70(9), pp. 794–803. 451 Kneale, D., Thomas, J., O’Mara‐Eves, A. and Wiggins, R. (2019) “How can additional secondary 452 data analysis of observational data enhance the generalisability of meta‐analytic evidence for 453 local public health decision making?,” Research Synthesis Methods, 10(1), pp. 44–56. 454 Koricheva, J. and Kulinskaya, E. (2019) “Temporal Instability of Evidence Base: A Threat to 455 Policy Making?,” Trends in Ecology & Evolution, 34(10), pp. 895–902. 456 Larsen, P.O. and von Ins, M. (2010) “The rate of growth in scientific publication and the decline 457 in coverage provided by Science Citation Index,” Scientometrics, 84(3), pp. 575–603. 458 Lortie, C.J., Stewart, G., Rothstein, H. and Lau, J. (2015) “How to critically read ecological meta459 analyses,” Research Synthesis Methods, 6(2) John Wiley & Sons, Ltd, pp. 124–133. 460 Marshall, I.J., Johnson, B.T., Wang, Z., Rajasekaran, S. and Wallace, B.C. (2020) “Semi461 Automated evidence synthesis in health psychology: current methods and future prospects,” 462 Health Psychology Review, 14(1) Routledge, pp. 145–158. 463 Marshall, I.J., Kuiper, J. and Wallace, B.C. (2015) “Automating Risk of Bias Assessment for 464 Clinical Trials,” IEEE Journal of Biomedical and Health Informatics, 19(4), pp. 1406–1412. 465 Marshall, I.J. and Wallace, B.C. (2019) “Toward systematic review automation: a practical guide 466 to using machine learning tools in research synthesis,” Systematic Reviews, 8(1), p. 163. 467 Mupepele, A.-C., Walsh, J.C., Sutherland, W.J. and Dormann, C.F. (2016) “An evidence 468 assessment tool for ecosystem services and conservation studies,” Ecological Applications, 469 26(5), pp. 1295–1301. 470 Mupepele, A.-C., Keller, M., Dormann, C.F., (2021). European agroforestry has no unequivocal 471 effect on biodiversity: a time-cumulative meta-analysis. BMC Ecol. Evol. 21, 193. 472 https://doi.org/10.1186/s12862-021-01911-9 473 Nakagawa, S., Dunn, A.G., Lagisz, M., Bannach-Brown, A., Grames, E.M., Sánchez-Tójar, A., 474 O’Dea, R.E., Noble, D.W.A., Westgate, M.J., Arnold, P.A., Barrow, S., Bethel, A., Cooper, E., 475 Foo, Y.Z., Geange, S.R., Hennessy, E., Mapanga, W., Mengersen, K., Munera, C., Page, M.J., 476 Welch, V., Carter, M., Forbes, O., Furuya-Kanamori, L., Gray, C.T., Hamilton, W.K., Kar, F., 477 Kothe, E., Kwong, J., McGuinness, L.A., Martin, P., Ngwenya, M., Penkin, C., Perez, D., 478 Schermann, M., Senior, A.M., Vásquez, J., Viechtbauer, W., White, T.E., Whitelaw, M., 479 Haddaway, N.R. and Participants, E.S.H. 2019 (2020) “A new ecosystem for evidence 480 synthesis,” Nature Ecology & Evolution, 4(4), pp. 498–501. 481 O’Connor, A.M., Tsafnat, G., Gilbert, S.B., Thayer, K.A. and Wolfe, M.S. (2018) “Moving toward 482 the automation of the systematic review process: a summary of discussions at the second 483 meeting of International Collaboration for the Automation of Systematic Reviews (ICASR),” 484 Systematic Reviews, 7(1), p. 3. 485 Pattanittum, P., Laopaiboon, M., Moher, D., Lumbiganon, P. and Ngamjarus, C. (2012) “A 486 Comparison of Statistical Methods for Identifying Out-of-Date Systematic Reviews,” PLOS ONE, 487 7(11) Public Library of Science, p. e48894. 488 Rhodes, K.M., Savović, J., Elbers, R., Jones, H.E., Higgins, J.P.T., Sterne, J.A.C., Welton, N.J. 489 and Turner, R.M. (2020) “Adjusting trial results for biases in meta-analysis: combining data490 based evidence on bias with detailed trial assessment,” Journal of the Royal Statistical Society: 491 Series A (Statistics in Society), 183(1) John Wiley & Sons, Ltd, pp. 193–209. 492 Rhodes, K.M., Turner, R.M., Savović, J., Jones, H.E., Mawdsley, D. and Higgins, J.P.T. (2018) 493 “Between-trial heterogeneity in meta-analyses may be partially explained by reported design 494 characteristics,” Journal of Clinical Epidemiology, 95, pp. 45–54. 495 Schafft, M., Wegner, B., Meyer, N., Wolter, C., Arlinghaus, R., (2021). Ecological impacts of 496 water-based recreational activities on freshwater ecosystems: a global meta-analysis. Proc. R. 497 Soc. B Biol. Sci. 288, 20211623. https://doi.org/10.1098/rspb.2021.1623 498 Shackelford, G.E., Kemp, L., Rhodes, C., Sundaram, L., HÉigeartaigh, S.S.Ó., Beard, S., 499 Belfield, H., Weitzdörfer, J., Avin, S., Sørebø, D., Jones, E.M., Hume, J.B., Price, D., Pyle, D., 500 Hurt, D., Stone, T., Watkins, H., Collas, L., Cade, B.C., Johnson, T.F., Freitas-Groff, Z., 501 Denkenberger, D., Levot, M. and Sutherland, W.J. (2019) “Accumulating evidence using 502 crowdsourcing and machine learning: a living bibliography about existential risk and global 503 catastrophic risk,” Futures, , p. 102508. 504 Shackelford, G.E., Martin, P.A., Hood, A.S.C., Christie, A.P., Kulinskaya, E. and Sutherland, 505 W.J. (2021). Dynamic meta-analysis: a method of using global evidence for local decision 506 making. BMC Biol 19(33)f. https://doi.org/10.1186/s12915-021-00974-w 507 Shojania, K.G., Sampson, M., Ansari, M.T., Ji, J., Doucette, S. and Moher, D. (2007) “How 508 Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis,” Annals of Internal 509 Medicine, 147(4) American College of Physicians, pp. 224–233. 510 Slavin, R.E. (1995) “Best evidence synthesis: An intelligent alternative to meta-analysis,” 511 Journal of Clinical Epidemiology, 48(1), pp. 9–18. 512 Slavin, R.E. (1986) “Best-Evidence Synthesis: An Alternative to Meta-Analytic and Traditional 513 Reviews,” Educational Researcher, 15(9), pp. 5–11. 514 Stone, J., Gurunathan, U., Glass, K., Munn, Z., Tugwell, P. and Doi, S.A.R. (2019) “Stratification 515 by quality-induced selection bias in a meta-analysis of clinical trials,” Journal of Clinical 516 Epidemiology, 107 Elsevier USA, pp. 51–59. Available at: 10.1016/j.jclinepi.2018.11.015 517 (Accessed: January 25, 2021). 518 Stone, J.C., Glass, K., Munn, Z., Tugwell, P. and Doi, S.A.R. (2020) “Comparison of bias 519 adjustment methods in meta-analysis suggests that quality effects modeling may have less 520 limitations than other approaches,” Journal of Clinical Epidemiology, 117 Elsevier Inc, pp. 36– 521 45. 522 Sutherland, W.J. and Wordley, C.F.R. (2017) “Evidence complacency hampers conservation,” 523 Nature Ecology & Evolution, 1(9), pp. 1215–1216. 524 Sutherland, W.J. and Wordley, C.F.R. (2018) “A fresh approach to evidence synthesis 525 comment,” Nature, 558(7710), pp. 364–366. 526 The Royal Society (2020) Face masks and coverings for the general public: Behavioural 527 knowledge, effectiveness of cloth coverings and public messaging. 528 Thomas, J., Noel-Storr, A., Marshall, I., Wallace, B., McDonald, S., Mavergames, C., Glasziou, 529 P., Shemilt, I., Synnot, A., Turner, T., Elliott, J., Agoritsas, T., Hilton, J., Perron, C., Akl, E., 530 Hodder, R., Pestridge, C., Albrecht, L., Horsley, T., Platt, J., Armstrong, R., Nguyen, P.H., 531 Plovnick, R., Arno, A., Ivers, N., Quinn, G., Au, A., Johnston, R., Rada, G., Bagg, M., Jones, A., 532 Ravaud, P., Boden, C., Kahale, L., Richter, B., Boisvert, I., Keshavarz, H., Ryan, R., Brandt, L., 533 Kolakowsky-Hayner, S.A., Salama, D., Brazinova, A., Nagraj, S.K., Salanti, G., Buchbinder, R., 534 Lasserson, T., Santaguida, L., Champion, C., Lawrence, R., Santesso, N., Chandler, J., Les, Z., 535 Schünemann, H.J., Charidimou, A., Leucht, S., Shemilt, I., Chou, R., Low, N., Sherifali, D., 536 Churchill, R., Maas, A., Siemieniuk, R., Cnossen, M.C., MacLehose, H., Simmonds, M., Cossi, 537 M.-J., Macleod, M., Skoetz, N., Counotte, M., Marshall, I., Soares-Weiser, K., Craigie, S., 538 Marshall, R., Srikanth, V., Dahm, P., Martin, N., Sullivan, K., Danilkewich, A., Martínez García, 539 L., Synnot, A., Danko, K., Mavergames, C., Taylor, M., Donoghue, E., Maxwell, L.J., Thayer, K., 540 Dressler, C., McAuley, J., Thomas, J., Egan, C., McDonald, S., Tritton, R., Elliott, J., McKenzie, 541 J., Tsafnat, G., Elliott, S.A., Meerpohl, J., Tugwell, P., Etxeandia, I., Merner, B., Turgeon, A., 542 Featherstone, R., Mondello, S., Turner, T., Foxlee, R., Morley, R., van Valkenhoef, G., Garner, 543 P., Munafo, M., Vandvik, P., Gerrity, M., Munn, Z., Wallace, B., Glasziou, P., Murano, M., 544 Wallace, S.A., Green, S., Newman, K., Watts, C., Grimshaw, J., Nieuwlaat, R., Weeks, L., 545 Gurusamy, K., Nikolakopoulou, A., Weigl, A., Haddaway, N., Noel-Storr, A., Wells, G., Hartling, 546 L., O’Connor, A., Wiercioch, W., Hayden, J., Page, M., Wolfenden, L., Helfand, M., Pahwa, M., 547 Yepes Nuñez, J.J., Higgins, J., Pardo, J.P., Yost, J., Hill, S. and Pearson, L. (2017) “Living 548 systematic reviews: 2. Combining human and machine effort,” Journal of Clinical Epidemiology, 549 91 Elsevier, pp. 31–37. 550 Tsafnat, G., Dunn, A., Glasziou, P. and Coiera, E. (2013) “The automation of systematic 551 reviews,” BMJ, 346(jan10 1), pp. f139–f139. 552 Tugwell, P. and Haynes, R.B. (2006) “Assessing claims of causation,” Clinical epidemiology: 553 how to do clinical practice research, Lippincott Williams & Wilkins Philadelphia, PA, pp. 356– 554 387. 555 Turner, R.M., Spiegelhalter, D.J., Smith, G.C.S. and Thompson, S.G. (2009) “Bias modelling in 556 evidence synthesis,” Journal of the Royal Statistical Society: Series A (Statistics in Society), 557 172(1) John Wiley & Sons, Ltd, pp. 21–47. 558 Underwood, A.J. (1991) “Beyond BACI: Experimental designs for detecting human 559 environmental impacts on temporal variations in natural populations,” Marine and Freshwater 560 Research, 42(5), pp. 569–587. 561 Wallace, B.C., Dahabreh, I.J., Schmid, C.H., Lau, J. and Trikalinos, T.A. (2014) “Modernizing 562 Evidence Synthesis for Evidence-Based Medicine,” in Greenes, R. A. B. T.-C. D. S. (Second E. 563 (ed.) Clinical Decision Support. Oxford: Elsevier, pp. 339–361. 564 Wauchope, H.S., Amano, T., Geldmann, J., Johnston, A., Simmons, B.I., Sutherland, W.J. and 565 Jones, J.P.G. (2020) “Evaluating Impact Using Time-Series Data,” Trends in Ecology & 566 Evolution 567 568 569 570