Analyzing the Relationship Between Altmetric Score and Literature Citations in the Dermatology Literature

Scientific research furthers our wealth of medical knowledge and can enact clinical change. The question stands of how the impact of a single article can be determined. Various metrics seek to describe this impact. The publishing journal’s impact factor (IF) and the article’s citation count have traditionally served as the quantitative measures of an article’s impact. These two metrics have limitations with the former yielding a generalization marker based on the journal rather than the article, and the latter making it difficult for authors and peers to see the true impact of a publication until years later, when substantial citation information has been accrued. ABSTRACT


INTRODUCTION
The platform Altmetric first published its rating scale, the Altmetric score, in 2010. 4 This algorithm gives a numerical score to articles' combined influence across mixed media platforms, including mentions from Twitter to Facebook, new outlets to public policy documents, video sources to Reddit, and others. 5 Altmetric incorporates the digital distribution and publicity for a scientific article, grading each article based on individual mentions across platforms. The mentions are converted into a total index measure of attention and impact, useful for standardization of bibliometric data collection. The score does not measure the quality of the research, the researcher, or the attention an article has accrued, but it is useful for measuring the article's visibility and overall reach, with a higher score indicating a greater overall reach. 4 To say a higher score is better, or alternatively worse, than a lower score is subjective-the score measures attention which can itself be 'good or bad.' While there are not cutoffs such as a maximum or minimum possible score, it is always a whole number. 4 We propose the use of the Altmetric score as a timely and accurate proxy measure for article impact, correlating with the traditional metrics of IF and citations. In a society with the world of information in the palm of one's hand, journal impact factor and an article's citation count cause a delay in data that is contrary to modern standards. Social media platforms and the internet at large are granted immediate access to research articles by a simple web search. By drawing on alternative metric ratings, a more rapid picture of the impact of a published research article can be established.
The authors selected the top 15 journals by impact factor in dermatology for inclusion in this study. The list was gathered by searching "Dermatology" in Elsevier's Citescore in the Scopus database. 7 6 2013 was chosen to yield a six-year interim to present day, the citations after this allotted time correlate with 90% total citations. Data from 2016 allows for better comparison to a modern Altmetric algorithm, though citation count three-years postpublication is 75% predictive of total citations. 7 The ten most cited articles in each journal were analyzed for bibliometric data; the sets of ten were identified by utilizing Elsevier's Scopus. 8 150 total articles were analyzed. The Altmetric Attention score was calculated by using the Altmetric bookmarklet, available for download through the Altmetric website. Altmetric scores were recorded for each article across 2013 and 2016. Specific social media and digital dispersion data were collected on the following platforms: Twitter, Facebook, blogs, policy sources, Wikipedia, Reddit, videos, Google+, and research highlights. 4

METHODS
All data was gathered and analyzed via Microsoft Excel; statistical significance was determined by p-value <0.05. Pearson's correlation coefficient (r) was used to establish correlation between variables; the coefficient of determination (R 2 ) was used to measure the amount of variance in these correlations. This study largely modeled its methods, procedures, and analysis from examples in pediatric surgery, general surgery, and urology literature, also comparing Altmetric score to citation and impact factor. 7,9,10 From the 15 dermatology journals with the highest impact factor based on the JCR, the 10 most highly cited articles published in 2013 and 2016 were reviewed (Table 1). Thus, 150 articles were reviewed from each year. In 2013, the cumulative total of citations for all articles was 13,819 and the total Altmetric Attention Score was 2,397. The median number of citations for each individual article was 62 with a range from 16-854 (standard deviation 80.1). The median Altmetric Attention scores for each individual article was 3 with a range from 0-396 (standard deviation 51.7).
Citation number and journal impact factor demonstrated a significant positive correlation in the 2013 cohort (r = 0.463, p<0.0001). Analysis revealed a significant positive relationship between citation count and Altmetric Attention scores for articles published in 2013 (r=0.267, p=0.0009) and a significant positive relationship between Altmetric Attention scores and journal impact factor for the same cohort of articles (r=0.255, p=0.002) (Figure 1 A, B). R 2 was low for both correlations: R 2 =0.072 for citations and R 2 =0.065 for impact factor. Journals were also studied in isolation to calculate the correlation coefficient between citation number and Altmetric Attention score (Table 2). In 2013, only Acta Dermato-Venereologica (r=0.891, p=0.0005) and Seminars in Cutaneous Medicine and Surgery (r=0.985, p<0.0001) had significant correlations between their articles' citation count and Altmetric Attention scores. The remainder were not significant, though two others approached statistical significance.
To compare the evolution across time, data was also collected in 2016. Total citation count for 2016 was 6,717 compared to the 13,819 for 2013, a 51.4% decline. The total Altmetric Attention score was 2,397 in 2013 but climbed by 60.3% to a score of 3,843 in 2016. Median number of citations in 2016 was 31.5 ranging from 8-425 (standard deviation 42.9), whereas the median Altmetric score in 2016 was 2.5 with a range of 0-509 (standard deviation 83.6). Altmetric ratings were significantly correlated with citation count for the 2016 cohort (r=0.244, p=0.003). Impact factor was significantly associated with Altmetric Attention scores as well (r=0.283, p=0.0005). R 2 was minimal for both, 0.060 for citation count and 0.080 for journal impact factor in 2016. The journals from 2016 were likewise studied individually to calculate the correlation coefficient between citation count and Altmetric score ( Older Twitter accounts were not significantly associated with increasing correlations between Altmetric Attention score and bibliometrics in both cohorts: 2013 (r=0.012, p=0.783) and 2016 (r=0.0002, p=0.973) (Figure 2).
Dissecting the components comprising the Altmetric Attention score in 2013 revealed that Twitter, news outlets, and Facebook were the top three platforms in which articles received mentions. This rank order for article mentions remained the same   (Table 3). Twitter was the clear leader for mentions received in both years with 180% more mentions than the second most utilized platform in 2013, though this margin diminished slightly in 2016 to 145%. Between the two years, Twitter (85.7%), news outlets (109.7%), Reddit (100%), and Google + (84.6%) saw increases in mentions. All of the remaining platforms' mentions decreased in 2016 compared to 2013 (Table  3).
To date, the influence of an article has been based on both the IF of the journal and the number of citations. However, because citation information accrues over time, the delayed citation counts yield an interim during which the actual impact of an article cannot be yet determined. 2,3 For a more modern approach, the Altmetric score takes into account the overall popularity of an article in real time by examining its mixed media influence. This is done based off of a variety of measures, such as Twitter, Facebook, Wikipedia mentions, Google+, news outlets, etc. We posited that the Altmetric score could predict and parallel the total impression of an article by acting as a proxy for IF and citation count in the top dermatology journals. This study is the first, to the authors' knowledge, to examine these various bibliometric methods in the field of dermatologic research. Our findings indicate a positive correlation between the modern platform, Altmetric scores, with previously standard methods, IF and citations, in both year cohorts. However, the correlation was determined by low R 2 coefficients which DISCUSSION C. D.
A. B. suggests that citation count and journal impact factor are not entirely explained by article Altmetric Attention score. While Altmetric Attention scores may be broadly predictive of the impact of an article, they are not an all-inclusive bibliometric for journal impact and citation count. Altmetric scores show attention or engagement with an article, whether that be positive or negative, as opposed to the quality of the article's impact as far as its implications for traditional scientific research. In order to fully evaluate the scientific and societal contributions of an article, a comparison between Altmetric and the more traditional methods can yield complementary perspectives and a better gage of overall impact.  Table 3. Classification of social media utilization factoring into the cumulative Altmetric score for journals by year.

Source
Article "mentions" in 2013 A.

B.
When studying journals in isolation, 2 of the 15 showed a significant correlation between citation number and Altmetric Attention score in 2013, while 5 of the 15 demonstrated a significant correlation between citation number and Altmetric score in 2016. With the continued growth of social media and digital involvement in research publicity over time, it could be that Altmetric Attention scores will follow this trend towards becoming more representative, rather than simply complementary, in describing a journals' impact. However, the minority of journals achieved significant correlation in this way, suggesting that the Altmetric algorithm does not entirely explain impact for top dermatology journals.
Additionally, Twitter was demonstrated to be the key platform to increasing engagement of dermatology research articles. Twitter exceeded all other social media platforms in its mentions, in both 2013 and 2016. The majority of journals studied in this case had a Twitter presence for an average of 3.5 years, with accounts created between 2011 and 2019. Interestingly, the older Twitter accounts did not display increased correlations between Altmetric and traditional impact metrics. Moving forward, further studies of true Twitter activity and engagement could be more indicative of correct correlation, rather than simply length of account life.
With Twitter, news outlets, and Facebook contributing largest to Altmetric score, authors and publishers might draw upon this information to concentrate marketing efforts to augment article impact in the future. These three platforms prove to be popular amongst dermatology researchers and publishers. Furthermore, it is important to note that across the three-year-time period, Twitter, news outlets, Reddit, and Google+ all displayed increasing trends in mentions, indicative of the expanding online presence for scientific research.
The total Altmetric Attention score for the 2016 cohort of articles was 60.3% higher than in 2013, whereas article citation counts were 51.4% lower in 2016 than 2013. The fewer citations by the 2016 cohort is attributable to the decrease in time for these articles to accrue citations. The increased volume of Altmetric score coincides with the ever-growing use of media incorporation into article impact. As the evolution of media continues to infiltrate the world of scientific research, we predict the weak correlation between traditional measures and Altmetric scores to grow stronger. Article "impact" describes an amalgam of factors, including journal notability, citation count, media mentions, and web dissemination. The benefit to utilizing the Altmetric score is the speed at which one can sum up these various contributing factors to quantify a research article's success. Though, based on our findings, it cannot wholly describe the final impact of an article in the field of dermatology, it can contribute to the overall picture of how the article is performing in present day, based on social media standards.
Evaluating the impact of a scientific article accurately is important for authors and readers alike. By quantifying the number of mentions, shares, reposts, and citations into a standardized metric, a publication can be more appropriately graded, compared, and applied clinically. Bibliometric data has traditionally been based on journal IF and citation accumulation; however, alternative metric ratings have emerged as a way to incorporate social media and online presence into the overall measurement of an article's impact. The Altmetric score, draws on media CONCLUSION platform information, which traditional metric patterns exclude. Altmetric method provides a quicker analysis of impact than the previous metrics could provide via long-term citation accumulation and positively, yet weakly, correlates with previous methods of measuring an article's impact, such as IF and citations. Altmetric data may be used as a supplementary measure of article success, but not as a substitution to journal IF and citations in top dermatologic journals.