The 100 Most Cited vs. Most Relevant Articles in the Journal of Neurosurgery: A Bibliometric Analysis

Introduction The Journal of Neurosurgery (JNS) published its first volume in 1944 and has evolved into the top cited journal in the field of neurosurgery. The aim of this study was to determine and characterize the 100 most cited (based on the total number of citations) vs. most relevant (based on the number of citations per year) articles originating in JNS. Methods The top 100 most cited articles in JNS were determined by searching the Web of Science database. Citations per year were additionally calculated for the top 1000 articles by total citations to rank the 100 most relevant articles. Results The median number of total citations for the 100 most cited articles in JNS was 505 (range 383-2200), and the median number of citations per year for the 100 most relevant articles was 21.88 (range 17.31-82.61). The median year of publication for the 100 most cited and most relevant articles was 1990 and 1999, respectively (P < 0.0001). Most articles originated in the United States in both categories (72% and 71%, respectively). The most common topic of study was cerebrovascular on both lists, followed by trauma on the most cited list vs. tumor on the most relevant list. The most relevant list also contained considerably more articles with a higher level of evidence such as systemic reviews/meta-analyses and prospective studies. Conclusions This study highlights the key contributing factors to the growth and flourishing of JNS. It also reveals several discrepancies between the most cited and most relevant articles, with the latter including more recently published articles, more studies addressing tumor, and more level I/1 (NHMRC/CEBM) evidence. Bibliometric analysis serves as a useful tool for clinicians and researchers to appraise published literature and understand the scientific foundation of modern neurosurgery.


Introduction
The Journal of Neurosurgery (JNS), the official journal of the American Association of Neurological Surgeons (AANS) since 1944, has been dedicated to publishing state-of-the-art work relating to neurosurgery, which include but are not limited to clinical studies, laboratory investigations, technical notes, literature and systematic reviews, and expert opinions [1]. Numerous landmark articles that shape the scope and practice of neurosurgeons nowadays are published in JNS. Its leadership in the field of scientific publication has been acknowledged worldwide, currently as the top neurosurgical journal with an impact factor of 4.318 according to the latest Thomson Reuters Journal Citation Reports (JCR) [2].
in JNS, but also seek their unifying elements. The number of citations of individual articles reflects its academic importance and impetus to encourage changes in practice or to lead future studies. Bibliometric analysis is therefore commonly applied to appraise the academic influence of journals and the articles therein [3][4][5][6]. Specifically, single journal analysis (of top-cited articles) serves as a useful tool to examine the objective performance and development of the journal [6][7][8] and thus is of significance to JNS as well. With the ranking of articles according to total citation counts, this would enable us to identify classic and foundational works and appraise their impact in their related fields.
However, an acknowledged limitation of using the total number of citations as a measure of impact is that older publications are favored over newer ones. Additionally, it is hard to determine the impact over time or the persisting significance of older publications by means of total citations. Previous analyses have overcome the bias of total citations by using an alternative measure of impact, i.e. the number of citations per year [3,9]. This method may reveal the relevance of individual articles in modern practice by taking into account the effect of time. In light of this, we have performed a secondary citation analysis of articles published in JNS according to the number of citations per year.
The aim of this study was to determine and rank the 100 most cited (based on the total number of citations) and most relevant (based on the number of citations per year) articles originating in JNS respectively. Furthermore, the characteristics of the two rank lists were analyzed and compared regarding publication year, sources, (authorship, institution, country of origin) topic of article, type of article, and level of evidence.

Search strategy and data collection
The Web of Science database was searched on December 23, 2018 under "publication name" = Journal of Neurosurgery, with the inclusion of all article types and period 1944 to 2018. The results were ordered by "times cited" to yield a rank list of articles according to their numbers of total citations. The list of 100 most cited articles published in JNS was obtained with the ranking of total citations.
Since the database has not yet developed the function to sort results by "times cited per year", we adopted the strategy of Bohl and Ponce's study [3] by assuming that the top 100 articles ranked by number of citations per year would be captured within the list of the 1000 articles ranked by total number of citations. We then calculated the number of citations per year for each article using a similar method described previously [3]. Briefly, the number of months since publication was calculated from the publication month and year to the current month and year (December 2018). Articles without a month of publication were assumed to have been published in June. The number of years since publication was then calculated by dividing the months by 12. The number of citations per year was gained by dividing the total number of citations by the number of years since publication. The list of 100 most relevant articles published in JNS was thus generated by ranking with the number of citations per year.
The following information was further collected for each article from the rank lists: year of publication, sources (authorship, institution, country of origin) based on the corresponding author, of article (cerebrovascular, trauma, tumor, functional neurosurgery), type of article including clinical studies (retrospective, prospective, randomized controlled trials, case series, case reports), laboratory studies, reviews/meta-analysis, and guidelines/consensus statements. The level of evidence was determined according to the Australian National Health and Medical Research Council (NHMRC) and Oxford Centre for Evidence-based Medicine (CEBM) evidence hierarchy [10][11]. Articles of animal studies, laboratory studies, technical notes, and case reports were not included in such evidence hierarchy and thus were classified as level 0.

Statistical analysis
Comparisons between continuous data were done using ANOVA or Mann-Whitney U test according to the testing condition. Statistical significance was defined as p < 0.05. All of the tests were two-sided. Statistical analysis was performed using SPSS software (version 16.0, SPSS, Inc).

Results
The 100 most cited articles published in JNS ranked by total number of citations are shown in Table 1. The median total number of citations was 505 (range 383-2200). The 100 most relevant articles published in JNS ranked by number of citations per year are shown in Table 2. The median number of citations per year was 21.88 (range 17.31-82.61). The most cited article was the Hunt-Hess grading scale for patients with subarachnoid hemorrhage (SAH) published in 1968 [12], while the most relevant article was the extent of resection and prognostic factors for survival in patients with glioblastoma multiforme (GBM) published by Lacroix et al. in 2001 [13].

Sources
North American institutions contributed to the majority of the top cited 100 articles (n=77), with Sweden following with five articles. The rest of the articles on the list originated from nine different countries including Switzerland, Italy, United Kingdom, Finland, Germany, Japan, France, Austria, and the former Soviet Union (    Authors who contributed three or more articles to the top cited list are shown in Table 4 with their position on the author list. Anthony Marmarou and Harold F. Young shared the top of the most productive authors ranking contributing seven articles each. Anthony Marmarou and Guido Guglielmi contributed the highest number of research articles (n=3) as the first author, while Bo K. Siesjo contributed three review articles as first and sole author.

Topic of article
Articles were classified into four categories including cerebrovascular, trauma, tumor, and functional neurosurgery. Ninety-six percent of articles fit into such classifications while four articles not classified by these categories were related to infection and cerebrospinal fluid system (numbers 8, 47, 71, and 98 in Table  1). A detailed breakdown of different study topics is shown in Table 5. Anterior approach for ruptured cervical disks 1 7 Atlantoaxial transarticular screw fixation 1 97 Percutaneous vertebroplasty/kyphoplasty in cancer patients 1 67 Methylprednisolone/naloxone treatment for acute SCI 1 55 Motor function assessment in rat SCI model 1 33    Table 1 The most common topic of study was cerebrovascular (n=41). The best-represented subtopic was intracranial arteriovenous malformation (AVM)(n=9), followed by endovascular treatment for intracranial aneurysms (n=7).
Trauma studies represented the second most common topic (n=26), with the majority addressing traumatic brain injury (TBI), which included mild TBI (n=2), severe TBI (n=13), animal models (n=3), prognostic factors for survival (n=1), and shaken baby syndrome (n=1). The remaining six articles were related to spinal injury.
There were eighteen articles reporting tumor topics, of which seven addressed glioma. Of note, articles featuring the extent of surgical resection and other prognostic factors for survival represented the mainstay (n=5). Other tumor topics included studies on sellar/parasellar tumors, meningioma, and other less common tumors.
Articles on functional neurosurgery composed the least number of articles on the list of 100 most cited articles (n=11). Subtopics pertaining to functional neurosurgery included movement disorders (n=3), microvascular decompression (n=3), diagnostic studies with electrical stimulation (n=3), and stereotaxy (n=2).

Type of article and level of evidence
Articles were further classified according to the type of study into four categories which included clinical studies (retrospective, prospective, randomized controlled trials, case series, and case reports), laboratory studies, reviews/meta-analysis, and guidelines/consensus statements ( Table 3). None of the articles were guidelines/consensus statements. There were 81 clinical studies, among which more than half were retrospective studies and case series (n=46) corresponding to level IV/4 (NHMRC/CEBM) evidence. There were twenty-six prospective studies and seven randomized controlled trials corresponding to level III/2 and II/1b (NHMRC/ CEBM) evidence, respectively. None of the articles were systematic review/meta-analysis, and thus there was no level I/1 (NHMRC/CEBM) evidence. Six literature review articles were level V/5 (NHMRC/CEBM) evidence. A detailed breakdown of different study types along with their level of evidence is shown in Table 6.   Table 1 List Comparisons

Level
The list of 100 most relevant articles shows several differences compared with the list of 100 most cited articles. A significant number (n=41) of articles on the most relevant list are not included in the most cited list. Of the 59 articles remaining on both lists, 41 are among the top 50 on the most cited list.
The most relevant list, not surprisingly, contains considerably more recently published articles than the most cited list. As shown in Figure 1, there is a lag of almost a decade before the steep increase in the number of the most relevant articles compared to that of the most cited articles. Though both plots peak in the 1990s, the one representing the most relevant list has a steadier decrease thereafter. The median year of publication is 1999 for the 100 most relevant articles, compared to 1990 for the 100 most cited articles (P < 0.0001). In addition, the most relevant list shows a more equitable pattern of article topic distribution than the most cited list ( Table 3). While the most represented topic remains cerebrovascular (n=36), there is a significant increase in the number of articles reporting tumor studies (30 vs. 18). Compared with the most cited list wherein five out of seven papers addressing glioma were related to malignant glioma, the most relevant list contains more studies on low-grade glioma, specifically with three articles addressing awake surgery with cortical and subcortical electrical mapping originating from the same team in France (numbers 34, 68, and 74 in Table 2). Additionally, there are two articles concerning radiosurgery for vestibular schwannoma and melanoma brain metastases, respectively (numbers 49 and 88 in Table 2). Though the most relevant list has a similar number of articles on functional neurosurgery compared with the most cited list (12 vs. 11), it has a broader coverage of disorders (e.g. epilepsy) and procedures (e.g. vagus nerve stimulation). There are fewer trauma articles in the most relevant list (20 vs. 26).
With regard to the type of article and the associated level of evidence, the most relevant list consists of twice as many review articles as the most cited list (14 vs. 7) ( Table 3). Of note, six articles are systematic reviews/meta-analyses which correspond to level I/1 (NHMRC/CEBM) evidence. The number of prospective studies (n=32) increased in the most relevant list while the number of retrospective studies (n=37) decreased, corresponding to level III/2 and level IV/4 (NHMRC/ CEBM) evidence, respectively. The most relevant list has a similar number of randomized controlled trials (n=6) and literature reviews (n=8) compared with the most cited list which corresponds to level II/1b and V/5 (NHMRC/CEBM) evidence, respectively.
There seems to be no dramatic discrepancy regarding the country of origin of articles between the two lists ( Table 3). With regard to institutional contributions, University of Pittsburgh Medical Center led the list with nine articles covering all four major topics, followed by the University of California at San Francisco with eight articles (Figure 4). Authors who contributed three or more articles to the most relevant list are shown in Table 7 with their position on the author list.

Discussion
As the official journal of AANS and one of the top journals in the field of neurosurgery, JNS has published over 21,933 articles since 1944. Its significant academic influence and considerable contribution to the evolution of modern neurosurgery has been recognized not only by the steadily increasing JCR impact factor scores, but also by previous citation analyses on the top cited and most relevant works in neurosurgery. These analyses identified JNS as the number one neurosurgical journal on both lists, contributing 79 articles to the 100 top cited list and 63 articles to the 100-most relevant list among all neurosurgical journals [3][4][5]. However, analysis remains elusive regarding the articles, authors, institutions, and topics that are driving the achievement and progress of JNS.
This bibliometric analysis identified the 100 most cited and most relevant articles published in JNS since its first issue launched in 1944. The two lists represented here may serve as the most objective and reliable resources in defining the classic and foundational works of the journal as well as the field of neurosurgery to some extent, because this modality is less influenced by subjective bias [14]. Indeed, a set of seminal articles which have had significant influence over both practice and research in the field of neurosurgery were included in the lists. These may also serve as fundamental reading lists for the education of neurosurgical residents.
Both the 100 most cited and most relevant lists displayed a so-called "90s peak" regarding the temporal trends in published articles, which has been reported in previous citation analyses as well [4,15]. This phenomenon may be explained by several factors including (1) the transition point where older works are losing their relevance in modern practice to newer works just beginning to accumulate citations; (2) a dramatic increase in citation indexing boosted by online databases which emerged around 2005 [4]; and (3) an expanding number of neurosurgeons and researchers in the field which may result in increased citations of more recently published articles. In addition, there seems to be a predictable course for articles to accumulate citations around two years after publication, peak after three to ten years, and then decline over time [4,16]. It would then be expected that articles published in the most recent decade will accumulate citations and peak in the following years. The strategy to identify the 100 most relevant articles by ranking the literature according to citations per year has to some extent overcome the inherent problem of citation analyses which favors older publications by using total citations as a measure of impact [5,17].
It is not surprising that institutions from North America contributed to the majority of articles on both lists. The disparity in numbers between the US and other countries may be attributed to several reasons including (1) the inherent association of AANS with JNS as official journal; (2) the fact that government funding and resources allocation for research was several times higher in the US than other countries [18][19]; (3) the possible bias associated with our classification strategy according to the corresponding author's institution as many studies enjoy multi-center cooperation; and (4) the inherent bias of the JNS being an English-language publication and the US as a primarily English-speaking nation (at least in matters of scientific discovery). However, international institutions located in 131 countries from other continents have made considerable contributions as well, which further validate the academic influence of JNS worldwide. This analysis also highlighted the significant contribution of many neurosurgeons to the growth and achievements of JNS. The contributors listed herein should not only be considered as a memorial for young generations to admire, but also as an impetus to inspire.
This bibliometric analysis also revealed the characteristics and trends of research in neurosurgery, which may also shed light on the direction for future research. The 100 most cited and most relevant lists both showed a strong focus on cerebrovascular disorders and procedures, with the latter group featuring more large-scale and well-designed studies on new techniques or first-hand experiences with new devices, such as Onyx injection for intracranial AVM, flow diverters, stents, indocyanine green videoangiography for aneurysm surgeries, and neurotransplantation for stroke patients. Similarly, distinguishing characteristics of the two lists concerning the topic of tumor are revealed by the significant increase in the number of articles related to the newly developed technique of awake surgery with functional electrical mapping for patients with low-grade glioma, the advancement of endoscopic endonasal approaches, surgical techniques for resection of a wider range of skull base tumors, and the application of radiosurgery for treatment of a variety of benign and malignant brain tumors.
With regard to functional neurosurgery, which represents one of the fastest-developing subspecialties, articles on management of refractory mood disorders and seizure disorders with stimulators emerged in the most relevant list compared with the most cited list, while trauma studies, which are thought to lose some of their popularity to more current hot topic research areas, did decrease in total numbers from 26 in the most cited list to 20 in the most relevant list. Of note, there were more articles on mild TBI and laboratory research on pathophysiology as well as cutting-edge diagnostic and therapeutic (in animal models) modalities in the latter group.
Though the majority of articles were original clinical studies, there were seven and 14 review articles among the 100 most cited and most relevant articles, respectively. Six systematic reviews/meta-analyses, which were exclusively included in the most relevant list, corresponded to level I evidence and could influence clinical practice and research directions as they are able to pool data from individual under-powered studies with disagreeing results and draw conclusions using statistical tools. Alternatively, the importance of literature review articles was noted in both lists as they offer a current overview of the field with synthesis of related individual studies and/or expert opinions on a specific topic [4].
The majority of articles in the 100 most cited list corresponded to level III (original prospective studies, n=28) and level IV (retrospective studies, n=46) evidence. Of note, there was no systemic review/metaanalysis representing level I evidence, and there was only one article in the top 20 with high-level evidence (I or II) as a randomized controlled trial, which seemed fewer in number than previous citation analyses on other subspecialty journals [6,[20][21]. However, the overall number of articles with high-level evidence increased significantly in the most relevant list, with six systemic reviews/meta-analyses and six randomized controlled trials, which was higher than other subspecialized analyses [6,22] and reflected the trend of increasing high-quality studies valued by the scientific community.
With the fact of more recently published articles and more level I evidence included in the most relevant vs. the most cited list, it is important to acknowledge unique characteristics and the usefulness of each list. The most cited list provides a better blueprint of the foundation and development of the field from a more historical perspective, while the most relevant list points to the science underlying current practice and future directions [3]. Either of the two lists can be used under different circumstances. With the push for evidence-based medicine that started in the 1990s, lists such as these will be very important in substantiating current practice methods, modification of practice, surgical techniques, and quality of care. If viewed from within this framework of evidence-based practice, bibliometric studies such as this one will become the tools that demonstrate neurosurgery's commitment to evidence-based practice to the world. It can also be easily inferred that with data provided from bibliometrics analysis such as these, the medical establishment can also exert the necessary influence in things such as insurance coverage of treatments deemed "experimental", reimbursements, and encourage a decrease in unnecessary procedures.
There are several limitations to this bibliometric analysis. First, some important recent publications with high numbers of citations per year but relatively low total citations may have escaped detection by our search strategy. Alternatively, the relevance of these articles may need to be tested by time according to the pattern of accumulating citations. Furthermore, with the evolution of a single JNS into the JNS publishing group, the official journals of AANS now include an additional three independent journals. This includes (1) Journal of Neurosurgery: Spine (since 2004) focusing on spinal surgery; (2) Journal of Neurosurgery: Pediatrics (since 2008) focusing on neurological diseases and disorders in children, and (3) Neurosurgical Focus (since 1996) covering a specific neurosurgical topic in depth for each issue. The current analysis limited its search to JNS only, and thus may have missed articles related to spine surgery, pediatric neurosurgery, and reviews published in the other three series. Additionally, certain landmark works may not have been captured if they have appeared in a textbook or published in a language other than English. Last but not least, as a "snapshot" analysis of published literature and their academic impact, any list created by citation analysis is dynamic and may change due to the function of time.

Conclusions
This bibliometric analysis identified the 100 most cited articles originating in JNS by total citation counts and the 100 most relevant articles in JNS by citations per year. The findings revealed the key contributing factors to the growth and flourish of JNS as well as classic and foundational works in the field of modern neurosurgery. Discrepancies between the most-cited and most-relevant lists also suggested that studies incorporating newly developed modalities, high-quality clinical trials and laboratory investigations, and systematic reviews with level I evidence are of growing interest and will likely be areas of continued growth in the future. Bibliometric analysis serves as a useful tool to establish benchmarks in the development of the field and to propose career directions in future research and practice.

Additional Information Disclosures
Human subjects: All authors have confirmed that this study did not involve human participants or tissue. Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue.

Conflicts of interest:
In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.