Assessing the Impact of the Portfolio Workshop and the Use of the Rubric Matrix for Reflective Writing on Medical Faculty and Students

Purpose The portfolio can be used as a crucial tool for self-reflection, which allows us not only to showcase achievements but also course correct on our personal and professional journey. However, there is a significant lack of awareness among medical professionals about portfolios. Arranging a workshop to impart this knowledge could be a potential mitigation approach. This study aims to assess the impact of workshops on portfolios on students' and faculty's knowledge. In addition, the study also analyzes the effect of using rubrics on reflective writing skills. Method A portfolio workshop was organized for the medical faculty and students in the Bundelkhand Government Medical College, Sagar, M.P. The Kirkpatrick model of training evaluation along with a rubric for the evaluation of reflective writing skills were used to measure the effectiveness of the workshop. Pre and post-tests for the workshop, pre and post-reflective writing skills, and workshop feedback were collected using questionnaires. The Shapiro-Wilk test and the Wilcoxon signed rank test were applied to the data collected. Results Out of 89 registrations for the workshop, only 81 people consented to the workshop and participated in the study. The total number of faculty was only 17 and the rest were students from all the phases. Both the Shapiro-Wilk test and the Wilcoxon signed rank test showed a significantly small p-value, stating that there was a significant positive impact on the knowledge, perception, and effectiveness of the workshop. Conclusion This study clearly outlines the positive impact of conducting a workshop on portfolios. A significant increase in participants' knowledge of portfolios is identified. Similarly, employing rubrics has a significant increase in the quality of reflective writing skills.


Introduction
Effective professional development is crucial for the success of both medical students and faculty members.In the past, medical education heavily relied on lectures and standardized assessments as methods to evaluate students' comprehension and abilities.Many different approaches focused on improving advanced learning have been developed to equip aspiring healthcare professionals for future demands effectively.Among these, the use of portfolios is a crucial tool that not only assists healthcare professionals in their academic endeavors but also can enable them to maintain discipline and focus throughout their lives [1].
Traditionally, a portfolio serves as a means of introducing ourselves and showcasing our personality.It has been heavily used in the fields of design and business [2,3].However, a medical portfolio comprises a collection of papers, materials, and proof that showcases the advancement, growth, milestones, and achievements in one's training and professional journey.A portfolio is efficacious in reflection, self-directed learning, and holistic skill enhancement, thus fostering creativity, effective content delivery, etc. introspection and enabling further investigation will promote more successful learning [3,4].
Around the world, the majority of medical education institutes mandate that undergraduates complete a portfolio as part of their curriculum, and through several continuous training programs, instructors receive professional instruction to become excellent reflective writers.As per earlier research, portfolios help teachers evaluate students more holistically by allowing them to consider not just their academic performance but also their personal and professional growth [5,6].
As effective delivery of medical education is taking center stage in healthcare institutions, there is a growing focus on enabling medical professionals, both faculty and students, with the know-how of the right set of tools [4].Workshops on portfolio development can not only invigorate the interest of faculty and students but also enable them to use this tool effectively.
However, this study is conducted to ascertain the effect of a workshop on portfolio toward participant comprehension.Two objectives of this study are: (1) To assess the impact of the workshop on portfolios and students' and faculty's knowledge; (2) To analyze the effect of using rubrics on reflective writing skills.

Overview
This quantitative evaluation study was conducted to assess pre and post-changes in participants' knowledge/awareness of portfolios due to a workshop on medical portfolios.A four-hour long workshop was conducted at the Bundelkhand Government Medical College, Sagar, M.P. Subsequently, pre and postchanges in knowledge and feedback were collected and evaluated.The study was conducted after receiving due approval from the Institute's Ethical Committee.The approval number is IECBMC/DHR/2023/27, dated 20/10/2023.

Participants
All undergraduate and postgraduate medical students and faculty members were the potential population of this study.Participants were recruited by posting information about the workshop via notice boards and different WhatsApp groups.As this study is based on voluntary participation, a convenience sample of attendees (n = 81) was achieved.

Inclusion criteria
All medical students who were enrolled in graduate and post-graduate programs at the medical college and all the faculty members employed at the medical college were considered as potential participants.No prior experience with portfolio development workshops was a mandate.In addition, those who gave written consent, and completed pre-and post-assessments and feedback were included in the study.

Exclusion criteria
Medical students and faculty members who attended the workshop and did not provide written consent to participate in the study were excluded, as were participants who had prior experience of attending a workshop on medical portfolios or reflective writing.

Material
Presentations in the form of PowerPoints (Microsoft Corporation, Redmond, WA, US) were used to deliver the content of the workshop.Written informed consent was taken from the interested participants in paper format.Pre and post-evaluation and feedback from participants were collected using Google Forms.For the evaluation of reflective writing skills, a rubric was prepared for evaluation.

Procedure
Participants were informed about the workshop via notice boards and different WhatsApp groups.At the time of registration for the workshop, the participants were asked to sign written informed consent.The preexisting knowledge of the participants about portfolios was assessed through a pre-test consisting of multiple-choice questions.Various key topics, such as the purpose of portfolios, the process of developing a model portfolio, reflective practice, clinical case writing, and the benefits of maintaining a medical portfolio, were delivered in the workshop.To evaluate the change in participant's ability to write reflections, before and after the reflective practice session, the participants were also asked to write reflectively about a life-changing incident in their lives.To assess the change in the awareness about portfolios due to knowledge imparted in the workshop, a post-test consisting of multiple choice questions was conducted.Finally, feedback on the workshop was collected from the participants.Figure 1 represents the schedule of the portfolio workshop.

FIGURE 1: Schedule of events for the workshop on portfolios
The face validity of the questionnaires was performed by the medical education unit and curriculum committee members for relevance, clarity, and appropriateness.

Analysis
An analytical rubric was developed to assess the quantitative change in the reflective skills of the participants.Both pre and post-reflective writing scripts were evaluated using the rubric.Different weights were assigned to different criteria in the rubric.The highest weight was given to Comprehensive Coverage.It measured the extent of reflective details provided by the writer.Session facilitators performed all the evaluations.The rubric developed for this study is presented in Table 1.There are no more than five spelling or grammar errors per page of writing reflection.

Level of Information
There are no more than three spelling or grammar errors per page of writing reflection.

TABLE 1: Analytical rubric to assess reflective writing skill
The numbers under Criteria and Level of Information represent the corresponding weightage.To calculate the overall score in a given reflective writing criteria, based on assessment, first a "Level of Information" will be assigned and then the corresponding weightage will be multiplied by the weightage of those "Criteria".

Results
The workshop was attended by 89 participants; however, only 81 gave consent for participation in the study.Forty-two of them were females and 39 were males.A total of 64 students and 17 faculty members participated.Details of the demographic distribution of the same are depicted in Table 2.

TABLE 3: Key statistics to assess the effectiveness of the workshop on portfolios
Learning Level is the measure of knowledge about the portfolio of participants.Reflective Writing is the measure of the reflective writing skill of the participants.
For Reflective Writing, effectiveness is established by significant improvement in pre and post-test data.A Z-score of 25.060 with a p-value of ~0.0000 signified a significant increase in the mean of the post-test when compared to the pre-test.The p-values for pre and post-test data, using the Shapiro-Wilk test, is approximately ~0.0000.As p-values are very small (< 0.05) for both pre and post-test, it can be easily inferred that distribution is NOT normal distribution, thus rejecting the NULL hypothesis.The statistical value of the Wilcoxon signed-rank test is 13.92 with a p-value of approximately ~0.0000, indicating that for almost all the pre-test values, corresponding post-test values are lower, indicating significant improvement in effectiveness.
Data from the feedback reflects the effectiveness of the workshop on the Reaction Level of the checklist.Figure 2 captures the distribution of responses to the relevant questions using a box and whiskers plot.The average collective score for the responses is 4.19 ± 0.80.About 79.3% of the responses were either 4 or 5.

FIGURE 2: Workshop effectiveness based on feedback received from participants
The score on the Y-axis represents scores provided by respondents on the Likert scale from 0-5; 0 means "Least" and 5 means "Most." The analytical Rubric mentioned in Table 1 is used for the evaluation of reflective writing scripts for both the pre and post-sessions.The distribution of reflective writing scores is presented in Figure 3. From the figure, a significant increase in the score for post-session is quite easily noticed.This effect is quite noticeable for the Comprehensive Coverage category.

Discussion
Various program evaluation models possess their unique strengths and weaknesses when it comes to assessing training activities.However, research has proven that Kirkpatrick's program evaluation model reigns supreme over other models [7,8].For this reason, we turned to Kirkpatrick's model to thoroughly evaluate the healthcare staff's overall response to our workshop on portfolios, as well as its impact on their learning and behavior.
According to the workshop evaluation, participants were quite pleased with the workshop (Level 1).A study by Rabiee et al. found that half of the staff at the Arak University of Medical Sciences believed the workshop was excellent while the other half rated it as moderate or weak [9].Further research by Ofoghi et al. using Kirkpatrick's model also showed that the physical conditions of the classroom had a direct and positive impact on the quality of learning, as reported by librarians after a short-term training course [10].
This study has incorporated the first two levels of Kirkpatrick's cycle, namely, Reaction and Learning.This decision was taken due to a time shortage.Similar findings were present in Faysal et al.'s study [11].Their study covered the effect of portfolio workshops up to the first two levels of Kirkpatrick's cycle.However, the difference between the participants of the two studies is that they conducted a faculty development program (FDP) and our study involved both students and faculty alike.In addition, they have also used the Gibbs reflective cycle for the evaluation of reflective writing skills and our study employed an analytical rubric.
In a study reported by Belay et al. in 2021, a workshop was organized very similar to that of ours [12].It was divided into PowerPoint presentations, along with hands-on activity coupled with pre-test, feedback, and post-test.A similar layout of the workshop has been utilized by many other studies [13][14][15].In contrast to our study, the primary objective of these studies was to utilize portfolios to evaluate learning objectives and the growth of abilities and competencies by utilizing frequent feedback and student reflection.2000) also employed rubrics to analyze reflective and analytical performance.It is found to have a significant positive impact on the academic performance of students [16][17][18].

Limitations
Only the first two of the four levels outlined by the Kirkpatrick model of training evaluation are evaluated in this study.Thus, a study over a long duration will help in precisely measuring the effects of portfolio workshops on medical students and faculty at two more levels, namely, Behaviors and Results.
In addition, the validity of the questionnaire, used for pre-test, post-test, and feedback, should be established not just by face validity but also by construct, content, and criterion validity.Also, the portfolios are time-consuming for both the assessor and the individual being assessed and they are difficult to mark.
With more time, a questionnaire with even stronger reliability can be created.

Conclusions
This study clearly outlines the positive impact on faculty and students of conducting a workshop on portfolios.Statistical evaluation of the participant's responses delineates a significant increase in participants' knowledge of portfolios.Similarly, employing a rubrics matrix for evaluation significantly increases the quality of reflective writing skills.

FIGURE 3 :
FIGURE 3: Distribution of the Reflective Writing Skills scores based on the rubric evaluation p-Values for all Criteria for Reflective Writing skill are ~ 0.0000 Similar to our study, Cheng et al. (2019), Kondo et al. (2022), and Andrade (

TABLE 2 : Demographic distribution of participants
The effectiveness of the workshop on Learning and Reflective writing is presented in Table3.For Learning, effectiveness is established by significant improvement in pre and and post-test data.As the sample size (n=81) is significantly higher than 25, the general Z-score test is used instead of the Mann-Whitney test for ordinal data.A Z-score of 26.234 with a p-value ~0.0000 signified a significant increase in the mean of the post-test when compared to the pre-test.The p-values for pre and post-test data, using the Shapiro-Wilk test, are approximately ~0.0000.As p-values are very small (< 0.05) for both pre and post-test, it can be easily inferred that distribution is NOT normal distribution.Thus rejecting the NULL hypothesis.The statistical value of the Wilcoxon signed-rank test, 15.30, with a p-value of 1.25x10-12 indicates that almost for all the pre-test values, corresponding post-test values are lower, indicating significant improvement in effectiveness.The Cronbach alpha of the questionnaire comes out to be 0.732.

Table 4 .
Is effectively communicating specialized skills and expertise is important for a medical portfolio?

TABLE 4 : Statistics on MCQs asked to assess the quantitative change in the Learning Levels of participants
Participants responded to each MCQ using a Likert scale with 1= Least Likely and 5 = Most Likely.