A Multi-center Simulation-Based Emergency Medicine Resident Boot Camp Can Improve Teamwork, Communication, and Clinical Skills

Boot camps are designed to deliver highly specific education in a short amount of time. Educational boot camps are known to improve confidence in clinical capabilities and medical knowledge and promote teamwork skills. We created an emergency medicine (EM) boot camp with targeted learning objectives based on expected mastery of post-graduate year (PGY)-level educational objectives based on the Accreditation Council for Graduate Medical Education (ACGME) EM milestones. This boot camp included a qualitative assessment, survey-based feedback, and quantitative assessment, which included the team's performance utilizing a validated code team checklist (Cardiac Code Management Assessment Tool). After attending the conference, EM residents felt more confident in achieving the EM ACGME milestones including the ability to provide immediate interventions to a critical patient, effective use of team communication, the ability to switch tasks efficiently, and to provide real-time feedback to their team. Eighty-six percent of residents preferred this teaching modality over other conference-based didactics and would like to see greater incorporation of similar interventions in future conferences.


Introduction
Medical simulation is an instrumental teaching modality for learners throughout the medical field [1][2][3][4][5].One of the more recent developments in medical simulation is the implementation of educational boot camps.This model efficiently delivers specific educational content in a short period of time.Many boot camps have been shown to have a qualitative impact on confidence in clinical capabilities, medical knowledge, and promoting teamwork skills [6][7][8][9][10].There is, however, less robust research on the effectiveness of simulation boot camps on quantitative metrics.Moreover, there is a notable heterogeneity in the approach to resuscitation in the most critically ill patients.To provide a framework for learners when caring for unstable patients, we have devised a tiered, simulation-based resuscitation boot camp for Emergency Medicine (EM) residents.This study aims to establish resuscitation fundamentals for learners that are specific to different post-graduate year (PGY) levels, and in the process, promote an improvement of skills in teamwork and communication.
There are many advantages to the boot camp method of education delivery.By definition, boot camps are low-stakes formative educational events.They have the potential to be highly efficient, being administered over several short and focused sessions.They can be immersive, especially when combined with the use of hands-on task trainers and simulated patient scenarios.They are also collaborative, as a platform for learners and teachers of varied experiences to come together.However, they may be too broad of a scope and difficult to individualize for each learner's level.In an attempt to remedy this issue, we employed a simulation-based resuscitation boot camp with educational small-group sessions specific to the PGY level.We hoped that by employing tiered learning objectives we could allow for a more targeted boot camp experience and promote improvement that is commensurate to a learner's level of expected mastery.
The Accreditation Council for Graduate Medical Education (ACGME) milestones are competency-based developmental outcomes that can be demonstrated progressively by residents from the beginning of their education through graduation to the unsupervised practice of their specialties.Similar to prior studies, we utilized a qualitative survey with questions derived from relevant ACGME milestones to evaluate the impact of this intervention.A boot camp experience has proven to be qualitatively effective as an educational modality and demonstrates a correlation with satisfaction for medical trainees [11,12].We hoped to better identify areas of improvement for longitudinal resident developmental education through self-evaluation of resident learners' perceived abilities [13].In another effort to assess the possible impact on quantitative measures during resuscitation, we also employed a validated code team checklist as a part of the intervention to assess the degree of behavioral changes after the intervention.
Importantly, as an educational tool, boot camps offer the ability to establish a safe learning environment.This encourages collaboration among residents and educators, establishes a fun learning environment, and ultimately may promote progress toward the achievement of milestones.By leveraging tiered PGY-specific learning objectives during the boot camp, we hypothesize that residents will make improvements in self-perceived ACGME milestones within resuscitation.Furthermore, we believe the intervention will allow residents to apply their learning toward improved scores on a quantitative code team checklist.
Here, we present a multi-center EM boot camp, conducted in New York City, focused on PGY-specific learning objectives comprised of educational sessions followed by a concluding medical simulation teambased competition with participants from each campus.An observational assessment using mixed methods of both quantitative and qualitative metrics was used to determine the efficacy of the boot camp concerning the validated Cardiac Code Management Assessment Tool (Appendix B; Table 8 and Table 9) and selfassessment of EM ACGME Milestones.Qualitative metrics were obtained by survey to assess the impact on resident wellness and learning preference.

Materials And Methods
As a part of this study, we collaborated with two other EM residencies in our network, to create a boot camp educational delivery method.It differs from prior interventions in specifying targeted learning objectives based on expected mastery according to PGY level.Moreover, simulation is utilized throughout the boot camp to allow for the application of learning.This study also assessed quantitative performance using a validated code team checklist.The development of this formative assessment was constructed with key medical simulation and EM faculty at three academic EM campuses.A panel of six EM faculty members with a background in medical education identified relevant ACGME milestones (Table 1) on resuscitation and created a qualitative self-assessment survey (Appendix C, Table 10).Our group then developed small group topics for each PGY class based on the identified milestones (Table 2).Residents of PGY levels 1 through 4 from these three EM residencies participated in a one-day, simulationbased boot camp intervention (Appendix A, Tables 4-7).Residents recruited as study participants on the day of the educational event and those who did not wish to voluntarily participate in the educational study were excluded.

Self-assessment of relevant ACGME milestones
At the start of the day, residents filled out a qualitative self-assessment survey based on a five-point Likert scale, in order to assess their perceived proficiency in relevant ACGME milestones.Next, the EM residents were divided based on PGY level for educational small group sessions specific to their expected level of knowledge.Each small group included an introductory medical simulation, followed by hands-on practice and didactics.The sessions were administered over approximately 30 minutes and led by an EM faculty instructor who debriefed the learners and proctored didactic discussions.After completing the compendium of educational small groups, residents again completed a post-intervention qualitative self-assessment survey (Figure 1) (Appendix D, Table 11).Included as part of the final survey were questions on resident attitudes and preferences on the delivery of educational curricula.were evaluated for performance during a simulated cardiac arrest using the validated Cardiac Code Management Assessment Tool (Appendix B) checklist, in both pre-and post-intervention.Two independent faculty observers scored each group using the checklist.Simulation cases were standardized to case stem, events, proctor, and specific time points among all groups.

Results
A total of 79 total learners participated in the study.The breakdown of our study sample included 21 learners at PGY-3/4, 18 at PGY-2, and 18 at PGY-1 level.During the qualitative portion of the study, the total number of respondents to the survey was 59 pre-intervention and 38 post-intervention (Appendix C, Table 10; Appendix D, Table 11).Pre-intervention scores reflected that residents scored a mode of three, corresponding with neutral confidence for most milestones.This was improved to a mode of four following the curriculum, corresponding to improved self-confidence in a majority of milestones.There appeared to be the largest improvement of confidence in the ability to apply immediate interventions to a critical patient, effective use of team communication, ability to switch tasks efficiently, and to provide real-time feedback.Overall, most residents felt more confident in their ability to achieve relevant milestones following the resuscitation boot camp (Figure 2).The quantitative arm of the study was analyzed using a paired sample t-test, the three groups scored a mean of 24.7 (SD = 3.5) during the pre-intervention cardiac arrest simulation.During the post-intervention simulation, the score averaged 28.7 (SD = 2.3), with a mean improvement of 3.0 points (P = 0.25) (Table 3).An analysis of the data using two-way ANOVA showed no significant difference in the overall score using the Cardiac Code Management Tool checklist between sites (P = 0.54), as well as pre-and post-intervention (P = 0.25).

Cardiac Code Management Assessment Tool scores
Pre-intervention Post-intervention Team A 21 30 Team B 26 30 Team C 25 26 A portion of the post-intervention survey also polled EM residents on attitudes toward a boot camp approach to learning.All learners enjoyed participating in the educational intervention.Thirty-three respondents (86%) also preferred this teaching modality over traditional lecture-based didactics and would like to see greater incorporation of similar interventions in future conferences.Thirty-five respondents either agreed or strongly agreed that the simulation-based education was enjoyable and offered a benefit to professional well-being.

Discussion
Our boot camp intervention provided evidence that a targeted approach can offer improved self-perception of proficiency in resuscitation.The qualitative data is limited to selected milestones and, therefore, should be interpreted with caution.It is notable that the selected milestones were identified as immediately relevant to resuscitation by experienced faculty members and may not capture the entirety of the skills required.Moreover, another limitation is in the point that self-perception of relevant ACGME milestones has not been verified to correlate with actual performance.The milestones were devised to be achieved over the course of an entire residency training period and thus may be reductive to extrapolate proficiency to a one-day boot camp.Despite these limitations, this qualitative survey represents an exceptionally meaningful concept to guide and model teaching objectives.
One of the limitations of the study is the low sample size when performing the cardiac code assessment.The ability to record additional iterations of quantitative checklist data was limited by time constraints during the one-day intervention.There were only three groups assessed before and after the curriculum.Due to this limited sample, improvements in absolute scores during the cardiac code assessment checklist did not reach statistical significance.Additionally, selection bias may have influenced the quantitative dataset.Of the selected groups, individual party members were noted to have variable PGY training years.Based on their pre-existing experience, there may be a different baseline level of medical knowledge and exposure to prior resuscitation of cardiac arrest.Future studies in this area should seek to assess participants in these areas before participation.
Another area of improvement during the checklist data collection would be to expand the number of independent faculty graders.Due to the availability of qualified faculty, only two observers were available for data collection.Additional faculty observers would help enhance the robustness of collected data and improve inter-rater reliability.
During the post-intervention qualitative survey, it was noted that the dropout rate was higher than anticipated.A total of 21 respondents filed a pre-intervention survey but did not complete the postintervention survey.This was largely due to numerous residents exiting the session prematurely due to scheduled clinical obligations.An attempt for improved compliance with this data was built into our intervention through the use of QR codes, as well as email reminders.

Conclusions
Overall, a simulation-based resuscitation boot camp with tiered, PGY-specific learning objectives is a feasible and effective teaching modality.Through this study, there is evidence that a targeted approach can promote advances in skill levels that is better tailored to a learner's level of mastery.Further studies are needed to evaluate if boot camps can improve a participant's self-perceived proficiency in EM ACGME milestones.While there does not appear to be a statistically significant difference in checklist score after the one-day EM resident-based boot camp, there is an improvement in the absolute score on the Cardiac Code Team Assessment Tool for each assessed group.Moreover, additional qualitative survey feedback suggests that this simulation-based boot camp is largely preferred over traditional teaching conference models and can contribute to a safe learning environment in our study cohort.
Our targeted approach has shown promise in helping to standardize resuscitation goals for specific levels of learner experience.One of the most compelling aspects of this model is the ability to create a safe learning environment during small group sessions by grouping learners with a similar knowledge base.Additionally, we were able to create an environment where interfacility faculty engagement and collaboration were supported.This approach furthermore allows for a platform for learners to synthesize these individual learning objectives and to apply their collective skills through simulation.Future directions could include similar annual tiered boot camps covering topics including point of care ultrasound, obstetrics and pediatric emergencies, and urgent care procedures.We hope that this tiered approach can be a teaching model forward, not just to EM residents, but to other medical team members and specialties.

Appendices
Appendix A

FIGURE 1 :
FIGURE 1: A subset of the cohort was assessed in their performance during simulated cases using the validated Cardiac Code Team Assessment Tool both before and after the small group sessions.PGY, post-graduate year

FIGURE 2 :
FIGURE 2: Heatmaps display pre-intervention and post-intervention qualitative survey data (A and B) for relevant ACGME milestones assessed based on a five-point Likert scale.PC, patient care; IC, interpersonal communication; SBP, systems-based practice

Multi-center emergency
medicine resident boot camp post-survey questions What is your level of training I feel confident in applying immediate interventions to manage a critically ill patient I am confident in my ability to task-switch efficiently during a medical resuscitation I am confident in my ability to use effective communication to manage a team during a medical resuscitation I am confident in my ability to provide real-time feedback to my team members during a medical resuscitation I am confident employing flexible communication strategies to resolve conflict among healthcare team members I am confident in using shared decision making with a patient to align their preferences with potential treatment options During high-risk situations, I am confident in my ability to recognize indications, risks, benefits, and alternatives for each procedure When caring for critically ill patients, I am confident in my ability to provide effective transitions of care/hand-offs I am confident in my ability to recognize when an intervention is futile for a patient I am confident in my ability to discuss advanced care directives with patients/ family during an emergent situation I enjoyed participating in simulation-based education and in SimWars I am comfortable learning in simulation-based education and in SimWars Simulation based education and SimWars is preferred over my usual conference style I would like to integrate more conference days like today I feel more confident in my abilities to lead a resuscitation Compared to traditional conference, simulation-based education and SimWars offered a greater benefit to my professional well-being and overall wellness I feel burnt out due to my work Getting together with others outside of my residency in simulation-based education and in SimWars improves my overall wellness Additional feedback or comments TABLE 11: Multi-center emergency medicine resident boot camp post-survey questions.Additional Information not involve animal subjects or tissue.Conflicts of interest: In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work.Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work.Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.