The new normal of medical research: replicability and reproducibility

José Manuel González-Rayas1*, Ana Lilia Rayas-Gómez2, José Manuel González-Yáñez2, José Juan García-González3, José Ascención Hernández-Hernández1, Rosa del Carmen López-Sánchez1

1Tecnológico de Monterrey, Escuela de Medicina y Ciencias de la Salud, Monterrey, N.L., México. 2Hospital San José de Querétaro, Querétaro, México. 3Coordinación Auxiliar Médica de Investigación en Salud, Instituto Mexicano del Seguro Social, Querétaro, México

http://creativecommons.org/licenses/by/4.0/

*Autor para correspondencia: José Manuel González-Rayas , Tecnológico de Monterrey, School of Medicine and Health Science. 3000 Ignacio Morones Prieto, Col. Los Doctores, C.P. 64710, Monterrey, N.L., México.

E-mail: contact.jmgr@gmail.com

http://revistabiomedica.mx.

Medicine is one of the leading scientific fields in terms of groundbreaking research. Discoveries at both the molecular and the clinical level are continuously made, giving answers to the most tantalizing questions in biology and uncovering effective therapies for the once untreatable diseases. Moreover, due to the COVID-19 pandemic, researchers from all over the world have united to provide the much-needed answers for the emerging phenomenon. This has caused an explosion of COVID-19 related papers, two of which have recently gained significant attention. The first one, “Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis” compared four treatment modalities (chloroquine, chloroquine plus a macrolide, hydroxychloroquine and hydroxychloroquine plus a macrolide) with a control group (1). Its main findings were a significant increase in in-hospital mortality and de-novo ventricular arrhythmias with each of the four treatment regimens. The second article, “Cardiovascular Disease, Drug Therapy, and Mortality in Covid-19”, found that cardiovascular comorbidities (such as age < 65 years, coronary artery disease, heart failure, cardiac arrhythmia, chronic obstructive pulmonary disease, and smoking) correlated with an increased risk of in-hospital death. On the contrary, ACE (angiotensin converting enzyme) inhibitors and angiotensin receptors blockers were not related to this outcome (2).

The two articles, published at leading medical journals, have had a significant impact on the global response to the COVID-19 pandemic. For instance, due to the findings of the first article, the World Health Organization decided to temporarily suspend the hydroxychloroquine group of its “Solidarity” clinical trial (which intends to find an effective treatment against COVID-19). However, concerns were raised over the validity of both articles’ data, which were provided by the “Surgisphere” Corporation, an obscure and previously unknown data analysis enterprise. Specifically, the fact that this small company could have generated an intercontinental registry of 96,000 patients (used on the hydroxychloroquine study) was disconcerting. As a result, an independent audit of Surgisphere data was commissioned. Nonetheless, the company denied to provide the complete datasets alleging that client contracts and confidentiality issues forbade them to do so. Hence, after the release of expressions of concern by both journals, the two articles were retracted. Unfortunately, Surgisphere had also provided data for a third article discussing the benefits of ivermectin on patients requiring mechanical ventilation. The study was posted on a preprint server but was not peer-reviewed (or formally published on an academic journal) and it was later retracted. Still, it was able to popularize ivermectin as a COVID-19 treatment in Latin American countries, which led to self-medication issues and to cases of people using the veterinary version of the drug (3).

We believe the Surgisphere phenomenon should mark the beginning of a transformation in medical publishing and medical research as a whole. Although this process will heavily rely on the principles of ethics and integrity, the concepts of replicability and reproducibility will become the backbone of what we could call the new normal in the area of medical publishing.

Replicability and reproducibility are frequently used as synonyms, although they are not the same. On the one hand, replicability refers to the feature of research articles that allows third party investigators to repeat the authors’ methods to obtain similar results (4). The latter implies the elaboration of a new study aiming to validate the claims of the first one. On the other hand, reproducibility means that the original data provided by the first study is reanalyzed, to confirm its conclusions (4). Hence, there is no need to conduct a new study to ascertain the reproducibility of an article. It is important to remark that both concepts are critical for the credibility of science and are thus, thoroughly considered when evaluating investigations in fields such as physics, chemistry, and physiology (5,6). Nonetheless, this is not the case in medicine, mainly because patient samples are highly variable, and because replicating a study in this field is expensive. This is especially true for clinical trials and large cohort studies. As a result, the act of validating previously published research is not as common in medicine as it is in the natural sciences. This has led the door open for cases of scientific misconduct, which have tainted the credibility of the field. In consequence, medical research should progressively move to a more rigorous scheme for evaluating novel studies before they get published.

We believe this new paradigm in the publication process should be ideally based on 4 pillars ensuring the replicability and reproducibility of the forthcoming studies (Figure 1). The first step would be the creation of intra-institutional reproducibility committees, which would ensure that the research being presented is valid. The latter would be achieved through an independent reanalysis of the original data obtained by the authors. After this initial phase, all data should be made available during the peer-review stage, which would also ensure the reproducibility of the research. In some cases, when problems are detected with the data, external replications assays could be conducted by independent investigators, under a double-blind model. Finally, all data should be made available and published, to ensure that readers can reproduce the findings of the authors.

Figure 1. The suggested pillars of replicability and reproducibility for forthcoming publications in the field of medical research. This figure was created with Biorender.com.

Although implementing the previous model will be difficult and will come at the expense of slowing the publishing process, we believe that a more rigorous evaluation of studies will translate in more trustworthy and valuable medical literature and would certainly have prevented the Surgisphere phenomenon from happening. Additionally, and as a secondary effect of a more stringent assessment of potential manuscripts, computer simulations (such as the ones used in the fields of biophysics and bioinformatics) will become more common, since this format of study can be more easily reproduced as compared to laboratory experiments. In synthesis, many aspects of the publishing process are about to change as a result of the COVID-19 pandemic, and it is important to understand that ethics, replicability, and reproducibility will be the guide to more transparent and solid medical research.

References

1. 1. Mehra MR, Desai SS, Ruschitzka F, Patel AN. Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis. Lancet. En prensa.
2. 2. Mehra MR, Desai SS, Kuy S, Henry TD, Patel AN. Cardiovascular Disease, Drug Therapy, and Mortality in Covid-19. N Engl J Med. En prensa.
3. 3. Offord C. Surgisphere Sows Confusion About Another Unproven COVID-19 Drug. The Scientist. 2020 June 16. News & Opinion. Available from: https://www. the-scientist.com/news-opinion/surgisphere-sowsconfusion-about-another-unproven-covid19-drug-67635
4. 4. National Academies of Sciences, Engineering, and Medicine. Reproducibility and Replicability in Science. Washington (D C): National Academies Press; 2019.
5. 5. Petersen OH. Reproducibility – again. J Physiol. 2019 Feb; 597(3): 657–8. doi: 10.1113/JP277486
6. 6. Wagner PD. Cores of Reproducibility in Physiology (CORP): Advancing the corpus of physiological knowledge. J Appl Physiol. 2017 Ene; 122(1): 89–90. doi: 10.1152/japplphysiol.01046.2016