The Pass/Fail Effect: A Longitudinal Study of United States Medical Licensing Examination (USMLE) Step 1 Performance Over a Decade

Objectives This study aimed to analyze the impact of the United States Medical Licensing Examination (USMLE) Step 1 transition to a pass/fail scoring system in 2022 on the performance of first-time test takers in three distinct groups: Doctor of Osteopathy (DO) and Doctor of Medicine (MD) examinees from US/Canadian schools and examinees from non-US/Canadian schools. The analysis spans a decade-long period from 2012 to 2022, offering insights into the implications of this pivotal change in medical education. Methods We analyzed the performance of first-time USMLE Step 1 examinees from US/Canadian MD and DO programs and non-US/Canadian schools from 2012 to 2022, including the transition year to a pass/fail scoring system. Data were obtained from USMLE performance data reports and organized into annual contingency tables. Descriptive statistics and comparative analysis were used to identify trends and differences in performance across the groups. Data visualization techniques were employed to illustrate these findings, and the results were contextualized within the broader changes in medical education. Results In 2021, first-time takers from US/Canadian MD and DO Degree programs had pass rates of 96% and 94%, respectively, while non-US/Canadian schools had a pass rate of 82%. However, in 2022, these rates dropped to 93%, 89%, and 74%, respectively. The most significant relative decline was observed among non-US/Canadian Schools' first-time takers, with an 8% decrease. Repeaters consistently had lower pass rates across all groups. Conclusion The study reveals a notable decline in pass rates following the transition to pass/fail scoring, although this is based on just one year of data. This underscores the importance of students not rushing into the exam and dedicating sufficient time for preparation. The potential impact of this research could be transformative for medical education, but more years of data post-transition will be needed to confirm these initial findings. These findings serve as a reminder that the change in scoring does not diminish the rigor of the exam, prompting students to approach their studies with diligence and patience and potentially paving the way for systemic improvements in medical education and healthcare delivery worldwide.


Introduction
The United States Medical Licensing Examination (USMLE) Step 1, a key milestone in medical education, transitioned to a pass/fail scoring system in 2022, sparking significant debate [1,2,3]. The primary objective of this transition was to reduce the overemphasis on Step 1 scores, which had become a high-stakes numerical value often overshadowing other essential aspects of a medical student's application for residency. The new system aimed to promote a more holistic review of residency applicants, considering a broader range of academic and personal attributes [1]. 1 2 3 4 5 6 7 8 9 10 However, the implications of this transition are not fully understood, and it has raised several concerns. One of the main worries is the potential disadvantage for students from less well-known medical schools who may have previously relied on a high Step 1 score to stand out in the competitive residency selection process. Similarly, students who might have excelled with a high Step 1 score may be disadvantaged in the new system [2].
This study analyzed the impact of this transition on the performance of first-time test takers from US/Canadian and non-US/Canadian schools. The analysis spans a decade (2012-2022), providing insights into this significant change in medical education [4,5].
Our research is timely and relevant given the recent USMLE Step 1 scoring system changes. Understanding the potential impacts of these changes on student performance is crucial for the medical education community. The findings of this study will not only influence the quality of medical education but also have far-reaching implications for healthcare delivery worldwide. This is because the selection of residency candidates affects the quality of future physicians, which in turn impacts patient care [3,6].
The transition to a pass/fail system for the USMLE Step 1 is a complex issue with potential benefits and drawbacks. By examining the impact of this change on student performance, this study will provide valuable insights that can guide future policy decisions in medical education.

Data collection
The data for this study were obtained from the USMLE performance data reports spanning from 2012 to 2022. These reports provide comprehensive information on the performance of first-time USMLE Step 1 examinees from US/Canadian Doctor of Medicine (MD) and Doctor of Osteopathy (DO) programs and non-US/Canadian schools. The data include the number of examinees tested each year and the corresponding pass rates. The year 2022 was of particular interest as it marked the transition to a pass/fail scoring system for the USMLE Step 1.

Data analysis
The data were organized into annual contingency tables to facilitate analysis. Descriptive statistics were computed to provide a summary of the data and to identify trends over the 10-year period. Comparative analysis was performed to identify differences in performance across the three groups of examinees: US/Canadian MD, US/Canadian DO, and non-US/Canadian schools ( Table 1).
The analysis focused on the pass rates of first-time test takers, as this group represents the majority of examinees each year. The pass rates for repeaters were also analyzed, although they represent a smaller proportion of the examinee population.

Data visualization
Data visualization techniques were employed to illustrate the findings. Graphs were created to depict the trends in pass rates over time for each group of examinees. These visual representations facilitated a clearer understanding of the data and the impact of the transition to a pass/fail scoring system.

Contextualization
The results were contextualized within the broader changes in medical education, particularly the transition to a pass/fail scoring system for the USMLE Step 1. The implications of the findings were discussed in relation to the ongoing debate about the impact of this transition on medical education and residency selection.

Ethical considerations
As this study involved the analysis of publicly available, de-identified data, ethical approval was not required. All analyses were conducted in accordance with relevant guidelines and regulations.

Results
Our analysis of the USMLE Step 1 performance data from 2012 to 2022 revealed significant trends among first-time test takers from non-US/Canadian schools, US/Canadian MD degree programs, and US/Canadian DO degree programs.

Non-US/Canadian schools
For first-time test takers from non-US/Canadian schools, the pass rate fluctuated over the decade, peaking at 87% in 2020 and dropping to 74% in 2022, coinciding with the transition to a pass/fail scoring system. Concurrently, the number of test takers generally increased, indicating growing interest despite scoring changes. Interestingly, there was no direct correlation between the number of test takers and the pass rate ( Figure 1).

US/Canadian schools (MD degree)
For first-time test takers from US/Canadian MD degree programs, the pass rate remained relatively high over the decade, with a slight peak in 2020 at 98%. However, there was a noticeable drop to 93% in 2022. The number of test takers has generally increased over the years, with a significant rise from 19,772 in 2020 to 22,280 in 2021 and further to 22,828 in 2022 ( Figure 2).

US/Canadian schools (DO degree)
For first-time DO test takers from US/Canadian schools, the number of examinees showed a steady increase over the years, reaching a peak of 5,309 in 2021. This growth is indicative of the expanding role and recognition of osteopathic medicine within the healthcare system. However, this increase in the number of test takers was accompanied by a slight decrease in the passing rate, from 92% in 2012 to 89% in 2022 ( Figure  3).

Comparative analysis
When comparing the three groups, the US/Canadian schools (MD degree) consistently demonstrated higher pass rates over the decade. However, all groups experienced a decline in pass rates in 2022, coinciding with the transition to a pass/fail scoring system for the USMLE Step 1. The most significant relative decline was observed among non-US/Canadian schools' first-time takers, with an 8% decrease from 2021 to 2022.

Discussion
The medical education community has been significantly engaged in discussions following the transition of the USMLE Step 1 to a pass/fail scoring system in 2022 [6,7]. Our analysis of a decade-long performance data, spanning from 2012 to 2022, provides intriguing insights into this debate. We observed a notable decline in pass rates among all groups of first-time test takers, with non-US/Canadian schools experiencing the most significant relative drop. This finding underlines the crucial importance of adequate preparation and thoughtful timing when undertaking the exam.
This decline, however, could be attributed to several other contributing factors. Coinciding with the transition to a pass/fail scoring system, the exam's passing score was raised by two points to 196, potentially leading to the observed decline. An influx of students sitting the exam after the transition could have also played a role in the adjusted pass rates. Further reinforcing this point, a self-reported post-exam survey revealed a reduction in daily studying time and weeks of advance preparation in 2022 compared to previous years [8].
The initial intent behind the transition to a pass/fail scoring was to diminish the overemphasis on Step 1 scores, thereby promoting a more holistic evaluation of residency applicants [6]. The previous scoring system created what Chen et al. (2019) termed the "Step 1 climate" a high-stress environment with potential adverse effects on student well-being and learning [6]. Therefore, the observed decline in pass rates might reflect a shift away from this high-pressure climate [9].
Despite the change in scoring, the USMLE Step 1 remains a rigorous examination demanding significant preparation, as highlighted by Prober et al. (2016) [9]. Our findings serve as a stark reminder that despite a scoring transition, the stringency of the exam remains intact.
These observations offer valuable insights for future deliberations about the pass/fail transition and strategies for student preparation for the USMLE Step 1 exam. For instance, medical schools could provide students with clearer expectations for the exam under the new scoring system and offer more comprehensive support structures. As students navigate this transition, it is imperative to remind them that although the scoring system has changed, the need for diligence and sustained preparation remains undiminished.

Limitation
The study provides an initial analysis of the impact of the transition to a pass/fail scoring system based on just one year of data (2022). Future studies will be needed to confirm these initial findings and to explore the longer-term impacts of this transition.