Beta2-agonists for exercise-induced bronchoconstriction: data extraction inconsistencies and errors in the Cochrane review by Bonini et al. (2013)

Beta2-agonists for exercise-induced bronchoconstriction: data extraction inconsistencies and errors in the Cochrane review by Bonini et al. (2013) Harri Hemilä Department of Public Health, University of Helsinki, Helsinki, Finland https://orcid.org/0000-0002-4710-307X https://www.mv.helsinki.fi/home/hemila 2024-4-10 In 2013, Bonini et al. published a Cochrane review “Beta2-agonists for exercise-induced asthma”: Bonini M, Di Mambro C, Calderon MA, Compalati E, Schünemann H, Durham S, Canonica GW. Beta₂-agonists for exercise-induced asthma. Cochrane Database Syst Rev. 2013 Oct 2;(10):CD003564. https://pubmed.ncbi.nlm.nih.gov/24089311 https://doi.org/10.1002/14651858.cd003564.pub3 This document describes a number of data extraction inconsistencies and errors in Bonini’s (2013) review. Pages 11-15 were published previously in the supplementary file of an earlier paper: Hemilä H, Friedrich JO. Many continuous variables should be analyzed using the relative scale: a case study of β2-agonists for preventing exercise-induced bronchoconstriction. Syst Rev. 2019 Nov 19;8(1):282. https://www.ncbi.nlm.nih.gov/pmc/articles/pmc6865024 https://doi.org/10.1186/s13643-019-1183-5 The Supplement: https://ndownloader.figstatic.com/files/18955214 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6865024/bin/13643_2019_1183_MOESM1_ESM.pdf Illustrations of six major extraction errors by Bonini are shown on pages 5-10. The Cochrane Airways group was informed of the errors on 11 June 2020: “The review by Bonini et al. has several errors related to data extraction, some of which are particularly large. For example, in the Bronsky (1995) and the Del Col (1993) trials, published FEV1 declines in the beta2-agonist tests appear 10 and 20 percentage points higher than in the original studies. There are also several cases of inconsistent data extraction…” https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD003564.pub3/read-comments However, the errors remain uncorrected in April 2024. Contents Page Errors in other Cochrane reviews 2 References to the selected examples of errors in extraction by Bonini et al. (2013) 4 The selected examples from the Cochrane review by Bonini et al. (2013) 5 Table of data extraction inconsistencies and errors by Bonini et al. (2013) 11 1 Errors in other Cochrane reviews Cochrane reviews are often promoted as high-quality systematic reviews that can be trusted: “Cochrane: Trusted evidence. Informed decisions. Better health.” https://www.cochrane.org/ (Accessed 2024-4-10). However, the Cochrane review by Bonini et al. (2013) is not the only review that has been shown to be flawed. In 2009, I found that there were several errors in the Cochrane review “vitamin C supplementation for asthma” (2009) [1]. For example, there were errors in data extraction; one trial had 20 participants, but only 11 participants were included in the Cochrane analysis. There were errors in the calculations, and the unpaired t-test was used when the paired test should have been used. My feedback was published in the (2010) version of the “2009” review [2,3]. The managing editor, Emma Welsh, wrote a response to my criticism in the (2012) version of the “2009” Cochrane review [4]. Simultaneously, the three analysis figures, which were the main focus of my critique, were removed from the review. In their absence, however, my feedback though originally valid became irrelevant. Furthermore, the remaining figures were renumbered so that the figure numbers in my feedback indicated real but different figures; therefore, readers would consider that I was confused. Finally, many responses by Welsh did not seem valid [5]. I contacted David Tovey, Editor-in-Chief of Cochrane, but our correspondence did not lead to any constructive efforts to correct the problem. After a year, I contacted COPE which stated that the original (2009) version [1] must be made available to readers; see a brief summary of the long process [6]. As a result of this process, one single PubMed record (#19160185) links to two different versions of the same “2009” Cochrane review. The PubMedCentral version (2009) includes all the original figures, but not my feedback [1]. The Cochrane Library version (2012) includes my feedback, but does not include the three analysis figures to which my feedback focused [4]. Finally, a third version of the “vitamin C supplementation for asthma” (2010) has both the original figures and my feedback, and is available at the University of Helsinki Digital Archive and Zenodo [2]. Furthermore, the errors that I pointed out in 2009 existed already in the 2001 version of the Cochrane review, so the review had been misleading readers over a decade [5]. Eventually, the review was withdrawn by Cochrane [7]. In 2011, I found that there were severe errors in the Cochrane review “Zinc for the common cold” (2011). I provided formal feedback in which I encouraged the authors to correct the flaws [8]. However, when the next version was published in 2013, almost all the errors remained. There was also plagiarism of text and data from another report [9]. Thus, the Cochrane editors had not checked that the flaws demonstrated in 2011 were corrected. Eventually, this process ended with the withdrawal of the zinc review [10], and with the retraction of the associated JAMA Clinical Evidence Synopsis [11-13]. In 2021, I found flaws in the Cochrane review on vitamin C and pneumonia (2020) [14]. For example, the authors stated “We included studies involving: 1. healthy adults and children receiving vitamin C supplementation for the prevention of pneumonia.” However, two trials (Coulehan et al. and Bancalari et al.) were included in the review despite the fact that they were common cold trials and neither mentioned pneumonia even as a secondary outcome. Furthermore, according to the Cochrane review, the number of pneumonia “events” in the Bancalari trial was 21 in each group. However, that was not the number of infections, but the number of children who had ≥1 infections. There were 46 infection events in the placebo group (N=30) and 38 in the vitamin C group (N=32). However, none of these 84 respiratory infections was pneumonia; they were the common cold [14]. The three Cochrane reviews described above and the Bonini et al. review are not to be trusted and do not help the reader to draw informed decisions based on evidence. Several authors have pointed out broader concerns with the Cochrane [15-20]. 2 References 1. Kaur B, Rowe BH, Arnold E. Vitamin C supplementation for asthma. [2009 version; all figures present, no feedback by Hemilä]. Cochrane Database Syst Rev. 2009;CD000993. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6176494 2. Kaur B, Rowe BH, Arnold E. Vitamin C supplementation for asthma. [2010 version; both the feedback by Hemilä and all the original figures; the feedback is on p.27]. Cochrane Database Syst Rev. 2009;CD000993. http://hdl.handle.net/10138/296440 https://zenodo.org/doi/10.5281/zenodo.10853677 3. Hemilä H. Vitamin C supplementation for asthma [Feedback separately] version 2010. Helsinki University Digital Archive. 2010. https://hdl.handle.net/10138/296440 https://zenodo.org/doi/10.5281/zenodo.6405775 4. Kaur B, Rowe BH, Stovold E. Vitamin C supplementation for asthma. [2012 version; feedback by Hemilä present, 3 figures removed]. Cochrane Database Syst Rev. 2009;CD000993. https://doi.org/10.1002/14651858.CD000993.pub3 https://zenodo.org/doi/10.5281/zenodo.10959542 5. Hemilä H. Vitamin C and exercise-induced bronchoconstriction: further problems in the Cochrane review “vitamin C for asthma”. Helsinki University Digital Archive. 2013. https://hdl.handle.net/10138/40816 https://doi.org/10.5281/zenodo.6405803 6. Hemilä H. Cochrane has not consistently followed the COPE guidelines. Eur J Clin Invest. 2020;50(4):e13216. https://doi.org/10.1111/eci.13216 https://hdl.handle.net/10138/326823 7. Kaur B, Rowe BH, Stovold E. WITHDRAWN: Vitamin C supplementation for asthma. Cochrane Database Syst Rev. 2013 Oct 23;2013(10):CD000993. https://pubmed.ncbi.nlm.nih.gov/24155047 https://doi.org/10.1002/14651858.CD000993.pub4 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6176495 8. Hemilä H. The zinc for the common cold review by Singh and Das has a number of problems which should be considered when the review is next time updated : [Feedback on Cochrane review “Zinc for the common cold”]. 2011. http://hdl.handle.net/10138/39188 https://zenodo.org/doi/10.5281/zenodo.10854101 9. Hemilä H. Concerns about unattributed copying of text and data, and about numerous other problems in the Cochrane review “Zinc for the Common Cold” by Singh M, Das RR (2013). 2015. https://hdl.handle.net/10138/153180 https://doi.org/10.5281/zenodo.6394115 10. Singh M, Das RR. WITHDRAWN: Zinc for the common cold. Cochrane Database Syst Rev. 2015 Apr 30;2015(4):CD001364. https://pubmed.ncbi.nlm.nih.gov/25924708 https://doi.org/10.1002/14651858.cd001364.pub5 https://www.ncbi.nlm.nih.gov/pmc/articles/pmc6457799 11. Hemilä H. Problems with the calculations in the JAMA Clinical Evidence Synopsis "Oral Zinc for the Common Cold" by Das and Singh (2014). https://hdl.handle.net/10138/153617 https://doi.org/10.5281/zenodo.6393781 12. Hemilä H. Common cold treatment using zinc. JAMA. 2015 Aug 18;314(7):730. https://doi.org/10.1001/jama.2015.8174 13. Das RR, Singh M. Notice of Retraction: Das RR, Singh M. Oral zinc for the common cold. JAMA. 2014;311(14):1440-1441. JAMA. 2016 Dec 27;316(24):2678. https://doi.org/10.1001/jama.2016.18134 https://pubmed.ncbi.nlm.nih.gov/28027374 14. Hemilä H, Chalker E. Commentary: Vitamin C supplementation for prevention and treatment of pneumonia. Front Med (Lausanne). 2021 Jan 20;7:595988. https://doi.org/10.3389/fmed.2020.595988 https://www.ncbi.nlm.nih.gov/pmc/articles/pmc7854566 15. Ioannidis JPA. Cochrane crisis: secrecy, intolerance and evidence-based values. Eur J Clin Invest. 2019;49(3):e13058. https://doi.org/10.1111/eci.13058 https://pubmed.ncbi.nlm.nih.gov/30520025 16. Newman M. Has Cochrane lost its way? BMJ. 2019;364:k5302. https://doi.org/10.1136/bmj.k5302 Clarification: https://doi.org/10.1136/bmj.l670 17. Jørgensen L, Gøtzsche PC, Jefferson T. The Cochrane HPV vaccine review was incomplete and ignored important evidence of bias. BMJ Evid Based Med. 2018;23(5):165‐168. https://doi.org/10.1136/bmjebm-2018-111012 18. Gøtzsche PC. What is the moral collapse in the Cochrane Collaboration about? Indian J Med Ethics. 2019;4 (NS) (4):303-309. https://doi.org/10.20529/ijme.2019.064 https://pubmed.ncbi.nlm.nih.gov/31791931 19. Vesper I. Mass resignation guts board of prestigious Cochrane Collaboration. Nature 2018-9-17. https://www.nature.com/articles/d41586-018-06727-0 20. Nass M, Noble JH Jr. Whither Cochrane? Indian J Med Ethics. 2018;4(1):1-5. https://doi.org/10.20529/ijme.2018.092 https://pubmed.ncbi.nlm.nih.gov/30683645 3 References to the selected examples of errors in extraction by Bonini et al. (2013) The following pages 5 to 10 illustrate the most severe extraction errors in the Cochrane review by Bonini et al. (2013). A longer list of misleading or erroneously extracted data is shown on pages 11-15. The list was previously published in a supplement for Hemilä and Friedrich (2019) Syst Rev. 2019;8(1):282. The references to the six selected examples of errors are are given here: Bronsky EA, Spector SL, Pearlman DS, Justus SE, Bishop AL. Albuterol aerosol versus albuterol Rotacaps in exercise-induced bronchospasm in children. J Asthma. 1995;32(3):207-14. https://doi.org/10.3109/02770909509089509 https://pubmed.ncbi.nlm.nih.gov/7759460 de Benedictis FM, Tuteri G, Pazzelli P, Niccoli A, Mezzetti D, Vaccaro R. Salmeterol in exercise-induced bronchoconstriction in asthmatic children: comparison of two doses. Eur Respir J. 1996 Oct;9(10):2099-103. https://doi.org/10.1183/09031936.96.09102099 https://www.researchgate.net/publication/14303504 https://pubmed.ncbi.nlm.nih.gov/8902473 Erratum in: Eur Respir J 1997 Jan;10(1):256. Del Col G, Spezia E, Richelli C, Piovesan P, Cantini L, Boner AL. Assessment of a new space device (Jet) for use with inhaled beta2-agonists in children with exercise induced asthma. Pediatric Asthma, Allergy & Immunology 1993;7(2):119–26. https://doi.org/10.1089/pai.1993.7.119 Grönneröd TA, von Berg A, Schwabe G, Soliman S. Formoterol via Turbuhaler gave better protection than terbutaline against repeated exercise challenge for up to 12 hours in children and adolescents. Respir Med. 2000 Jul;94(7):661-7. https://doi.org/10.1053/rmed.2000.0789 https://www.researchgate.net/publication/12394745 https://pubmed.ncbi.nlm.nih.gov/10926337 Kemp JP, Dockhorn RJ, Busse WW, Bleecker ER, Van As A. Prolonged effect of inhaled salmeterol against exercise-induced bronchospasm. Am J Respir Crit Care Med. 1994 Dec;150(6 Pt 1):1612-5. https://doi.org/10.1164/ajrccm.150.6.7952623 https://pubmed.ncbi.nlm.nih.gov/7952623 Shapiro GS, Yegen U, Xiang J, Kottakis J, Della Cioppa G. A randomized, double-blind, single-dose, crossover clinical trial of the onset and duration of protection from exercise-induced bronchoconstriction by formoterol and albuterol. Clin Ther. 2002 Dec;24(12):2077-87. https://doi.org/10.1016/s0149-2918(02)80098-4 https://pubmed.ncbi.nlm.nih.gov/12581546 4 The selected examples from the Cochrane review by Bonini et al. (2013) 5 Bronsky et al. (1995) reported that FEV1 decline was 6% in two salbutamol group (data above). Bonini et al. (2013) added 10.0 and 20.0 pp to these figures (data below). Bronsky (1995): Bonini: 6 Del Col et al. (1993) published FEV1 decline 0.76% and 2.37% in two salbutamol groups (data above). Bonini et al. (2013) added 20 pp and 10 pp to the originally published figures. Placebo 7 Kemp et al. (1994) reported that on every time point salmeterol was more effective than salbutamol, indicated by smaller FEV1 decline in the salmeterol test (5 vs. 7%, 8 vs. 25%, 13 vs. 27%) (above). Bonini et al. (2013) selected salbutamol FEV1 decline from time point 0.5 hours, but salmeterol FEV1 decline from time point 11.5 hours. Thereby Bonini makes it falsely appear that salbutamol is more effective than salmeterol. Placebo 8 Shapiro et al. (2002) reported four time points and on every time point formoterol was more effective than salbutamol, indicated by smaller FEV1 decline in the formoterol test (above). Bonini et al. (2013) selected salbutamol FEV1 decline from time point 15 min, but formoterol FEV1 decline from time point 12 hours. Thereby Bonini makes it falsely appear that salbutamol is more effective than formoterol. 9 Grönnerod et al. (2000) reported four time points for exercise test (above). Bonini et al. (2013) selected formoterol FEV1 data from time point 12 hours, but placebo FEV1 data from time point 15 minutes. There can be changes over time and thereby selecting different time points for formoterol and placebo can cause bias. The same exercise test time should be used in the comparison of the placebo and the β2-agonist. Placebo