Association of Non-traditional Indicators of Readers’ Engagement With Traditional Dissemination Metrics of COVID-19-Related Research

Introduction: Researchers are increasingly interested in appraising the impact of their research work, which eventually drives public perception. The overall impact of a study can only be gauged if we consider both traditional and non-traditional dissemination patterns. Hence, we preferred to study the association between the non-traditional reader engagement metrics and traditional dissemination metrics in relation to coronavirus disease 2019 (COVID-19)-related research published in five high-impact peer-reviewed medical journals. Method: This observational study was conducted using data sourced from Altmetric, including the Altmetric attention score (AAS), an aggregate score of an article’s dissemination. New England Journal of Medicine (NEJM), Lancet Infectious Diseases, Clinical Infectious Diseases (CID), Chest Journal (CHEST), and Journal of the American Medical Association (JAMA) were included in the study based on the prevalence of COVID-19-related original research published in each of them. The number of citations was framed as the reference for traditional metrics. To avoid artificial variance, data were collected on the same day, November 13, 2022. Correlational analyses were performed using the Pearson correlation coefficient using Minitab 17 (Minitab Inc., State College, PA). The relationship between the variables was considered very weak if r<0.3, weak if r: 0.3 to 0.5, moderate if r: 0.5 to 0.7, and strong for r>0.7. Results: We found a very weak correlation between citations and AAS for Clinical Infectious Diseases, Lancet Infectious Diseases, and CHEST, whereas the correlation was moderate for NEJM and JAMA. The correlation between citations and Twitter mentions was very weak for Clinical Infectious Disease, Lancet Infectious Disease, and CHEST, but it improved for NEJM and JAMA. There was a very weak correlation between citations and news mentions for Clinical Infectious Diseases, Lancet Infectious Diseases, and CHEST. Conclusion: Our study highlights that the traditional indicator, i.e., citation has a very weak to moderate correlation with the AAS and it doesn’t capture the entire influence of a research publication. Also, the current method of determining a journal's impact factor doesn’t take this disparity into consideration. Hence, there needs to have a more inclusive strategy to define the impact of scientific research on the general population in real-time.


Introduction
As of 2021, nearly half of the American social media users reported that they often get their news from social media [1]. Dissemination of original research should follow similar emerging trends and hence allow greater public access to the research. Researchers and institutions are increasingly interested in appraising the impact of their research work. As the coronavirus disease 2019 (COVID-19) pandemic progressed, we understood that public access and interaction with the original research was necessary to comprehend the best prevention practices. Interestingly, in 2020, the top five most discussed research articles across 20 different educational disciplines were all related to COVID-19 [2]. Our study focused on examining the association between various alternative indicators of readers' engagement with COVID-19 original research and traditional dissemination metrics of research published in the five high-impact peer-reviewed medical journals.

Materials And Methods
This cross-sectional study was conducted using data sourced from Altmetric, including the Altmetric attention score (AAS), an aggregate score of an article's dissemination on the internet that takes into consideration mentions from news sources, blogs, social media, and citations within the scientific community [3]. For the purpose of this study, AAS and its sub-components were used to gauge metrics of non-traditional research dissemination. The access was granted to the Altmetric Explorer by Altmetric and received institutional review board (IRB) approval from Creighton University, Omaha, Nebraska, Approval number: 2003168-01. A PubMed query was conducted within the Altmetric Explorer to identify the COVID-19-related research articles. New England Journal of Medicine (NEJM), Lancet Infectious Diseases, Clinical Infectious Diseases (CID), Chest Journal (CHEST), and Journal of the American Medical Association (JAMA) were included in the study based on their impact factors and the prevalence of COVID-19-related original research published in each of them. Key words included COVID-19, COVID19, COVID 19, SARS-CoV2, SARS-CoV-2, coronavirus-19, coronavirus19, and coronavirus 19. Articles not tracked by Altmetric were excluded. A correlation between traditional and non-traditional metrics was established for each journal to elaborate on their dissemination pattern. The number of citations was framed as the reference for traditional metrics. To avoid artificial variance, data were collected on the same day, November 13, 2022. All the COVID-19-related studies from January 1, 2020, till the date of data collection, were included. Correlational analyses were performed using the Pearson correlation coefficient using Minitab 17 (Minitab Inc., State College, PA). The relationship between the two variables was considered very weak if r<0.3, weak if r: 0.3 to 0.5, moderate if r: 0.5 to 0.7, and strong if r>0.7.

Results
We found a very weak correlation between citations and AAS for Clinical Infectious Diseases (r=0.21), Lancet Infectious Diseases (r=0.22), and CHEST (r=0.24), whereas the correlation was moderate for NEJM (r=0.52) and JAMA (r=0.5). The correlation between citations and Mendeley (Elsevier, London, UK), a reference manager, was strong for all the journals, with r>0.7 (Table 1). Moreover, the correlation between citations and Twitter mentions was very weak for Clinical Infectious Diseases (r=0.12), Lancet Infectious Diseases (r=0.14), and CHEST (r=0.22) but it was higher for NEJM (r=0.47) and JAMA (r=0.4) ( Table 2). There was a very weak correlation between citations and news mentions for Clinical Infectious Diseases (r=0.19), Lancet Infectious Diseases (r=0.25), and CHEST (r=0.15), whereas this correlation was moderate for NEJM and JAMA (r=0.58 for both) ( Table 2). The COVID-19-related article with the highest AAS was published on 12/13/2020 in NEJM with a score of 30136, and the COVID-19-article with the highest number of citations (20808) was also published in NEJM on 4/30/2020 (Appendix 1, Table 4). Among these five peer-reviewed journals, Clinical Infectious Diseases had the most number of COVID-19-related publications from January 1, 2020, to the date of data extraction (November 13, 2022).

Discussion
The AAS indicates the amount of attention a research work has received. It is based on an automated algorithm that takes into account the weighted count of the amount of attention a media source gets. This weighted count is based on the relative reach of the type of each source. Each media source has default weightings, e.g., a news portal gets a default weighting of eight as compared to a tweet having a default weighting of 0.25, just as a news mention will bring more attention to a topic as compared to a simple tweet [4]. Further details of these default weightings are reflected in Table 3. In addition, AAS further takes into account other factors, including duplicate tweets or tiered calculations for different types of news sources depending upon their dissemination and reliability. By blending all that useful information, AAS helps a researcher to gauge the dissemination of their research, identify new potential collaborators, and measure the impact of their work.  Multiple prior studies have reported a positive correlation (weak to moderate) between citations and an Altmetric score, as reflected by our study [5][6][7][8][9][10][11][12][13][14]. However, our study results also highlight that although traditional indicators, i.e., citations do correlate with the AAS, they do not capture the entire impact of a research study or a researcher. There needs to have a more inclusive strategy to define the impact of landmark scientific research studies on the general population in real-time [15]. Also, the AAS provides immediate information regarding the dissemination statistics of an article as compared to the citations, which may take years to grow [16]. AAS provides real-time data on readership trends. This also highlights the area of improvement across the board to amplify the access of people outside of the scientific community to ever-evolving research and clinical trials. Both NEJM and JAMA have relatively more dissemination, as reflected by their higher AAS compared to Lancet Infectious Disease, CID, and CHEST. This signals a more significant momentum within NEJM and JAMA to improve the outreach of their respective journal articles.
We believe that by utilizing these alternative metrics of knowledge dissemination, journals can increase the acceptability of emerging research in the public and hence improve public behavioral practices as well as debunk the myths. However, all five journals have a close relationship between Mendeley's readership and citations. This relationship between Mendeley's readership and citations is consistent with previous studies as well [17]. Hence, Mendeley's readership numbers may serve as a proxy for citations, as these readers are largely members of the scientific community and are therefore more likely to cite a study in future work [18].
There are some limitations of this study. Altmetric Explorer itself is not that comprehensive, and the AAS score serves only as a proxy for reader engagement but does not tell us if that changed the public's perspective. Also, there is no "good" AAS, as it simply reflects the extent of dissemination. It cannot differentiate if a research article got a high attention score because of negative coverage in the news or mentions on social media, resulting in a negative impact on public perception. Future directions may include more comprehensive studies to elaborate on the role of alternative metrics of research dissemination and hence highlight the probability of utilizing these matrices to evaluate researchers' impact.

Conclusions
While current methods of gauging the impact of scientific research largely rely on citations, which will take years to grow, ongoing scoring systems like AAS may provide real-time data on readership trends. Logically, non-traditional dissemination metrics, including AAS, may be combined with the current calculation methodology to define the real-time impact of a researcher and their scientific work.