Understanding Emotions Impacted by New Assessment Mandates Implemented in Medical Education: A Survey of Residents and Faculty Across Multiple Specialties

Background Previous research findings show that the overall perception of residents regarding the new entrustable professional activity (EPA) assessment mandates is primarily negative. Hence, this study aims to explore the link between EPA assessment experiences and resident and faculty emotions and expectancy of successfully completing residency training. Methods A standardized questionnaire (Medical Emotions Scale (MES)), which measures 20 unique emotions on a 5-point Likert scale, was used to explore the emotions of residents and faculty members regarding EPA assessments and residents’ expectancy of success. Data analysis included descriptive statistics and analysis of variance (ANOVA). Results Ninety-one (N=91) participants (46 faculty members and 45 residents) completed the survey. The results revealed that residents have more negative emotions toward EPA assessments compared to faculty. Additionally, resident and faculty emotions regarding EPA assessments vary across specialty and gender. Conclusions These findings will be crucial in providing the Royal College of Physicians and Surgeons of Canada and medical education programs with concrete evidence and guidance in understanding the perspectives and emotions of residents and faculty towards EPA assessments and residents’ beliefs about successfully completing their medical training.


Introduction
Medical education in Canada is currently shifting from the traditional apprenticeship model to an outcomesbased competence by design (CBD) model that focuses on skill development through specific learning objectives and better assessment strategies [1][2][3].This new assessment model involves entrustable professional activities (EPAs), which are "essential tasks of a discipline that an individual can be entrusted to perform safely and independently", that are assessed to ensure that, by the end of training, a trainee is ready for independent practice [1,4,5].
The transition to CBD and implementation of the new EPA assessment mandates are being done in increments [6].Initial reports by the Federation of Quebec Medical Residents (FMRQ) and yearly snapshots by the Royal College of Physicians and Surgeons of Canada (RC) focusing on the experiences of trainees and faculty across programs and institutions suggest that the implementation of EPA assessments has caused negative emotions (e.g., anxiety, frustration, worry) in both trainees and faculty and uncertainty regarding success in medical training among trainees [7][8][9].Some of the reasons included logistical difficulties (e.g., the assessment platform not functioning efficiently) and doubts regarding the educational benefit of CBD and EPA assessments.
The purpose of our study is to explore the emotions of faculty and residents regarding EPA assessments in

Materials And Methods
The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement for crosssectional studies was used to guide the reporting.

Study Design
This was a cross-sectional observational study using surveys for data collection.

Conceptual Foundations
Positive emotions (e.g., hope, enjoyment) have been shown to have an overall positive impact on performance, which is the opposite of negative emotions (e.g., anxiety, hopelessness, frustration) [10].We used Pekrun's control-value theory (CVT) of achievement emotions to support our interpretation [11].

Setting, Participants, and Sampling
Through purposeful sampling, residents of all levels and faculty at McMaster University from general surgery, emergency medicine, and pediatrics, with prior experience completing and/or receiving EPA assessments, were recruited.

Data Sources/Measurement
Two surveys (see Appendix) were created in LimeSurvey (one for faculty, one for residents), and participants were provided with a $10 gift card honorarium.The surveys were piloted to ensure clarity and ease of completion, and the study was done from fall 2022 to winter 2023.
Both surveys included the Medical Emotion Scale (MES), which is a self-reporting standardized questionnaire that measures the intensity of 20 unique emotions on a 5-point Likert scale [12].The MES categorizes emotions into four groups: (i) Positive activating emotions (hopeful, proud, happy, enjoyment, compassionate, curious, grateful) (ii) Positive deactivating emotions (relieved, relaxed) (iii) Negative activating emotions (confused, angry, frustrated, afraid, anxious, ashamed) (iv) Negative deactivating emotions (hopeless, disappointed, sad, bored) Faculty and residents completed the MES to measure their emotions regarding EPA assessments.Residents also completed the MES to measure their emotions about successfully completing their residency training with EPA assessments.In our study, the MES demonstrated good scale reliability with Cronbach's alpha ranging from 0.73 to 0.79.The survey also included questions with a textbox option to list the reasons for their ratings.Participants also provided their year of training/practice, specialty, gender, and ethnicity.

Ethical Consideration
This study received ethics approval from the Hamilton Integrated Research Ethics Board (HiREB #14686, August 2022), and consent forms were included on the first page of the surveys.

Statistical Methods
SPSS Statistics software was used for the analysis.Descriptive analyses (mean (M) and standard deviation (SD)) were performed.Next, a series of ANOVAs were performed to determine any significant differences in emotions of faculty and residents (and expectancy of success of residents) based on specialty (general surgery, emergency medicine, pediatrics), gender (man, woman, non-binary), ethnicity (minority vs nonminority), and the number of EPA assessments completed/received in past year (statistically significant, p <0.05).Effect size (Cohen's d) was calculated to determine the strength of the significance [13].Open-text box responses were analyzed using simple inductive thematic analysis [14].The data have been represented as the number of participants (n) and percentage (%).
*1 faculty and 1 resident selected 'I prefer not to answer'; hence, percentages for 'gender' do not add up to 100%.

Faculty (Completing EPA Assessments)
Faculty demonstrated higher levels of positive activating (e.g., happy) (M=2.38,SD=0.93) and positive deactivating emotions (e.g., relief) (M=2.25,SD=1.01) compared to negative emotions (e.g., angry, disappointed).The findings of the factorial ANOVA indicated that there are no statistically significant variations in faculty emotions based on specialty, gender, minority group affiliation, or the number of completed EPA assessments.
Based on the text responses, faculty reported challenges, including the perception that EPA assessments are reasonable for most residents but can be stressful and fatiguing for individuals needing remediation.Additionally, there were concerns about residents selectively engaging EPA assessments when they predicted to perform well.
Factorial ANOVA results were similar between the two MES completed by residents (details in Table 2).First, for differences across specialties (general surgery, pediatrics, and emergency medicine), a statistically significant difference was observed in positive deactivation emotions (relieved, relaxed).Specifically, residents from general surgery reported higher levels and residents from emergency medicine reported lower levels: F (2, 42)=3.35,p<0.05, with an effect size of 0.138.Second, for differences across gender, there were statistically significant differences in positive activating emotions (e.g., hopeful, proud, happy) -residents who identify as man scored having higher levels than residents who identify as woman: F (2, 42)=3.27,p<0.05, with an effect size of 0.135.Similar statistical significance was found in positive deactivating emotions (relieved, relaxed) -residents who identify as man scored having higher levels than residents who identify as woman: F (2, 42)=4.11,p<0.05.
Finally, there were no statistically significant differences in residents' emotions based on minority group affiliation or the number of EPA assessments received.

Residents' Written Responses
Some challenges expressed included receiving minimal and non-useful feedback, encountering inconsistent grading practices, perceiving EPA assessments as a checkbox rather than a valuable learning tool, and experiencing frustration when faculty failed to complete EPA assessments in a timely manner.

Discussion
In Canada, emergency medicine, general surgery, and pediatrics transitioned to CBD in 2018, 2020, and 2021, respectively [6].Both faculty and residents in general surgery reported higher positive emotions related to EPA assessments compared to emergency medicine and pediatrics.First, emergency medicine was one of the first programs to transition to CBD in 2018 [6].Therefore, we could hypothesize that faculty and trainees feel more negative emotions toward EPA assessments due to more time potentially struggling with implementing CBD.Pediatrics transitioned to CBD in 2021, so perhaps their negative emotions toward EPA assessments are due to the challenges that come with a more recent transition to CBD.The stronger negative emotions observed in emergency medicine and pediatrics could also be due to differences in how CBD/EPA assessments are being implemented in each specialty, including different strategies used by programs to overcome challenges.
In our resident sample, a notable trend was that residents who identify as man exhibited significantly higher levels of positive activation (happy, hopeful) and positive deactivation emotions (relieved, relaxed) compared to residents who identify as woman.This was not observed in our faculty sample.Existing research suggests that it is pertinent to consider the impact of gender biases in medical education.For example, female residents are more likely to receive lower ratings and less positive feedback on their performance assessments [15].This discrepancy in feedback could potentially influence female residents' emotions towards their EPA assessments negatively.In another study on EPA assessments, females rated their own performances lower than their male counterparts, despite faculty assessments not showing any differences between female and male resident performance [16].Therefore, our observed gender differences in emotions toward EPA assessments may be influenced by a combination of factors associated with selfperception and potential biases in the assessment process.
In general, residents exhibited predominantly negative emotions towards EPA assessments and expectancy of success.Therefore, it is imperative for residency programs to gain a comprehensive understanding of the emotional experiences of residents and faculty toward CBD implementation.In doing so, they can proactively undertake initiatives aimed at addressing negative emotions and implementing improvements that have the potential to positively influence residents' emotions, thereby contributing to the overall success of the residents.
For this study, self-reporting questionnaires were used instead of other methods of measuring emotions.The MES relies on participants' interpretations and their individual manner of reporting their emotions.While there are other various methods to measure emotions, including examining brain activity, salivary biomarker tests, skin conductance, tracking eye movements, and analyzing facial expressions and observable responses, self-reporting measures remain one of the most widely utilized methods for data collection to measure emotions [12,17,18].Specifically for emotions, self-reporting measures possess the capability to capture a broad range of emotions and demonstrate higher efficiency [19].Additionally, self-report questionnaires offer advantages in terms of time and cost-effectiveness [20].Given that emotions have a subjective experiential component, they are effectively measured through self-reported measures [12].
For limitations, first, the study was conducted at a single institution, which may restrict the generalizability.Second, for faculty, due to how the listservs are organized, it was hard to determine the exact response rate.Finally, conducting qualitative investigations would help with gaining a deeper understanding of the underlying factors influencing faculty and resident emotions and the expectancy of success related to EPA assessments.

Conclusions
To summarize, this study performed an exploration of the link between EPA assessments and resident/faculty emotions and expectancy of success.Our findings suggest that, overall, residents have more negative emotions associated with EPA assessments compared to faculty that vary across specialty and gender.Therefore, investigations such as interviews could help further understand why residents and faculty feel a certain way to then develop concrete action plans.

Appendices Faculty Survey
Question Group 1: Eligibility 1. Have you completed an Entrustable Professional Activity (EPA) assessment for one or more residents in the past year?
• Yes [show next question] • No [Unfortunately, you are not eligible to take part in this survey.Please close the survey.] 2. What is your specialty?
• Emergency Medicine [continue] • Pediatrics [continue] • General Surgery [continue] • Other [Unfortunately, you are not eligible to take part in this survey.Please close the survey.]

Question Group 2: Emotions
Using the scales below, indicate how you currently feel about completing EPA assessments.For each emotion, please select the number that best describes the intensity of your emotions.
pediatrics, emergency medicine, and general surgery) that implemented EPA assessments in recent years.This article was previously presented as a meeting abstract at the 2023 Canadian Conference for the Advancement of Surgical Education on October 12, 2023.

TABLE 1 : Summary of the demographic information (gender, specialty, minority affiliations, number of EPA assessments completed/received in the past year) of faculty and residents
EPA: Entrustable Professional Activity

TABLE 2 : Summary of the descriptive analysis and ANOVAs for resident emotions regarding receiving EPA assessments based on specialty, gender, minority affiliations, and number of EPA assessments received in the past year
This table includes the descriptive (mean and standard deviation) and a between-group ANOVA of four categories of emotions from the Medical Emotions Scale (MES) to analyze if there is a significant difference in residents' emotions based on specialty (emergency medicine, pediatrics, general surgery), gender (man, woman, non-binary) minority affiliation, and the number of EPA assessments received in the past year(1-30, 31-50, 51).The MES measures the range of 20 unique emotions on a 5-point Likert scale.The emotions are categorized into four groups: (i) positive activating emotions (hopeful, proud, happy, enjoyment, compassionate, curious, grateful), (ii) positive deactivating emotions (relieved, relaxed), (iii) negative activating emotions (confused, angry, frustrated, afraid, anxious, ashamed), and (iv) negative deactivating emotions (hopeless, disappointed, sad, bored).