Assessing the Utilization of Large Language Models in Medical Education: Insights From Undergraduate Medical Students

Background Artificial intelligence (AI) has the potential to be integrated into medical education. Among AI-based technology, large language models (LLMs) such as ChatGPT, Google Bard, Microsoft Bing, and Perplexity have emerged as powerful tools with capabilities in natural language processing. With this background, this study investigates the knowledge, attitude, and practice of undergraduate medical students regarding the utilization of LLMs in medical education in a medical college in Jharkhand, India. Methods A cross-sectional online survey was sent to 370 undergraduate medical students on Google Forms. The questionnaire comprised the following three domains: knowledge, attitude, and practice, each containing six questions. Cronbach’s alphas for knowledge, attitude, and practice domains were 0.703, 0.707, and 0.809, respectively. Intraclass correlation coefficients for knowledge, attitude, and practice domains were 0.82, 0.87, and 0.78, respectively. The average scores in the three domains were compared using ANOVA. Results A total of 172 students participated in the study (response rate: 46.49%). The majority of the students (45.93%) rarely used the LLMs for their teaching-learning purposes (chi-square (3) = 41.44, p < 0.0001). The overall score of knowledge (3.21±0.55), attitude (3.47±0.54), and practice (3.26±0.61) were statistically significantly different (ANOVA F (2, 513) = 10.2, p < 0.0001), with the highest score in attitude and lowest in knowledge. Conclusion While there is a generally positive attitude toward the incorporation of LLMs in medical education, concerns about overreliance and potential inaccuracies are evident. LLMs offer the potential to enhance learning resources and provide accessible education, but their integration requires further planning. Further studies are required to explore the long-term impact of LLMs in diverse educational contexts.


Introduction
The integration of technology into medical education is in progress, with artificial intelligence (AI) and large language models (LLMs) being the new addition.In an era characterized by the exponential growth of medical knowledge, the ability to efficiently access, interpret, and apply information is paramount for aspiring healthcare professionals [1].LLMs, such as ChatGPT, Google Bard, Microsoft Bing, and Perplexity, have emerged as powerful AI tools with the potential to reshape the landscape of medical education by providing innovative solutions to the challenges posed by information overload [2].
Undergraduate medical education represents a critical phase in the development of future healthcare practitioners.It is during this formative period that students acquire the foundational knowledge, skills, and attitudes that will shape their careers.Although teacher-led education is an important part of the medical curriculum, student-led learning methods are gaining popularity due to their flexibility and better engagement [3].Among those, self-directed learning is an integral part of competency-based medical education.LLMs can have a significant impact on self-directed or student-led learning due to their

Development of questionnaire
To assess the knowledge, attitude, and practice of undergraduate medical students regarding the utilization of LLMs in medical education, a structured questionnaire was developed.The questionnaire consisted of the following three domains: knowledge, attitude, and practice, each comprising six questions.The development of the questionnaire involved the following steps.
First, we conducted a comprehensive review of existing literature on LLMs in medical education to identify relevant themes and constructs.Next, the questionnaire was developed with input from subject matter experts in medical education and AI technology to ensure content validity.A cognitive interview was conducted with a group of 10 undergraduate medical students to assess the clarity, comprehensibility, and relevance of the questionnaire.Feedback from pilot participants was used to refine the questionnaire further.The final questionnaire was designed to assess participants' knowledge about LLMs, their attitudes toward the technology, and their practical experiences using LLMs in the context of medical education.This questionnaire was then distributed among 30 students, and the response was obtained with a gap of seven days.The first response was used to calculate Cronbach's alpha, and two responses were used to check the test-retest reliability using intraclass correlation coefficients (ICCs) [6,7].
The responses were coded for quantitative analysis (strongly agree = 5, agree = 4, neutral = 3, disagree = 2, strongly disagree = 1).Cronbach's alphas for knowledge, attitude, and practice domains were 0.703, 0.707, and 0.809, respectively.For a student, the average score of a domain was calculated by adding the score of six responses and dividing the value by 6.These scores of test and retest were compared using ICCs.Obtained ICCs for knowledge, attitude, and practice domains were 0.82, 0.87, and 0.78, respectively.

Participants
We recruited undergraduate medical students for this study.Hence, any undergraduate student studying modern medicine (Bachelor of Medicine, Bachelor of Surgery) was the target participant.As the survey link was shared online with all the students, those who did not participate voluntarily were automatically excluded from the study.

Data collection methods
Data collection was carried out using the finalized questionnaire, which was distributed to undergraduate medical students through an online platform (Google Forms).Participants were contacted via email, and a link to the questionnaire was provided along with a brief explanation of the research objectives and the voluntary nature of their participation.Participants were given ample time to complete the questionnaire, and reminders were sent as needed to enhance response rates.

Data analysis
Data obtained from the completed questionnaires were subjected to a structured data analysis process.There was no missing data as all questions were made compulsory in Google Forms.Next, descriptive statistics, such as frequencies and percentages, were computed to summarize the responses to individual questionnaire items within each domain (knowledge, attitude, and practice).The responses were coded for quantitative analysis (strongly agree = 5, agree = 4, neutral = 3, disagree = 2, strongly disagree = 1).For a student, the average score of a domain was calculated by adding the score of six responses and dividing the value by 6.The chi-square test was used to compare categorical variables with expected equal division in all categories, and a statistical significance indicates that the occurrence was not by chance.The ANOVA with post hoc analysis was used to compare the mean score of knowledge, attitude, and practice.Data analysis was conducted using GraphPad Prism 9.5.0 (GraphPad Software, Boston, MA), and a p-value of less than 0.05 was considered statistically significant.The results of the data analysis are reported in the subsequent sections of this study.

TABLE 3: Attitude of medical students on LLMs in medical education
The data in the table are presented as numbers (percentage).The p-value is of the chi-square test where a significant p-value indicates that the distribution of the responses was not by chance.
LLM, large language model Some of the medical students are actively using LLMs as supplementary resources in their medical education (30.23%).They turn to LLMs for explanations that are not readily available in traditional books (55.81%) or search engines (47.67%).Additionally, LLMs are influencing their self-study practices and increasing their confidence in discussing medical topics (37.21%) (Table 4).

TABLE 4: Practice of medical students on large LLMs in medical education
The data in the table are presented as numbers (percentage).The p-value is of the chi-square test where a significant p-value indicates that the distribution of the responses was not by chance.

Discussion
The finding that the majority of students rarely use LLMs for their teaching and learning purposes can be attributed to several factors.One significant factor is limited familiarity among students regarding the capabilities and benefits of LLMs in the context of medical education.Additionally, the preference for traditional learning resources, concerns about accuracy, limited integration into the curriculum, and potential barriers such as limited access or time constraints may contribute to this trend.Individual learning styles and the perceived learning curve associated with LLMs also play a role [8].Moreover, resistance to change and the lack of training and guidance can further discourage students from incorporating LLMs into their study routines [9].To encourage greater utilization of LLMs, the educators and institutions should address these factors by providing education on the advantages and effective use of LLMs, making them more accessible, and integrating them into the curriculum where appropriate.
The finding that knowledge scores were the lowest among students, followed by practice scores and then attitude scores, can be attributed to several factors.It suggests that while students may have a positive attitude toward LLMs in medical education and may even try to incorporate them into their practices, their knowledge about how to effectively use these models might be lacking [10].The lower knowledge scores may indicate a need for more comprehensive training and education on the practical application of LLMs in their medical studies.Additionally, the statistical significance in the differences between knowledge vs attitude and attitude vs practice scores underscores the importance of bridging the gap between students' positive attitudes and their actual utilization of LLMs in their learning processes through targeted educational interventions and support.
The findings reveal varying levels of awareness, understanding, and perception among students regarding LLMs in medical education.A significant proportion of students are aware of LLMs and believe that they understand how they work, indicating some theoretical knowledge.They are generally positive about LLMs' potential for use by both teachers and students, as well as their ability to simplify complex topics.[11].However, students also express caution about the potential for inaccuracies and wrong information generated by LLMs, highlighting the importance of critical evaluation.These findings suggest a need for further education and practical training on LLMs' applications to bridge the gap between awareness and effective utilization [12].
While a significant majority of students are open to incorporating LLMs as supplementary tools, this openness is tempered by concerns about overreliance on these models.Students recognize the potential for LLMs to provide comprehensive medical information and transform how they access knowledge, but they are cautious.Their apprehensions revolve around the potential negative impact on the development of critical clinical reasoning skills and the risk of acquiring incorrect concepts.This complex attitude reflects the need for a balanced approach to LLM integration, acknowledging both the benefits and potential pitfalls associated with these tools in medical education [13].
Some medical students are actively embracing LLMs as valuable supplementary resources, recognizing their capacity to provide explanations beyond what traditional books or search engines can offer.This indicates a growing recognition of LLMs as tools that can bridge gaps in understanding and enhance their learning experience [14].Moreover, the influence of LLMs on students' self-study practices and their increased confidence in discussing medical topics reflects the adaptability of students to modern technological advancements, which empower them to take charge of their education and knowledge dissemination.
The study's implications for medical education in developing countries are significant and offer promising opportunities for enhancing learning, accessibility, and the overall quality of medical education [15].LLMs emerge as valuable supplementary resources, particularly in regions where access to up-to-date medical textbooks and academic materials may be limited due to resource constraints.A study by Tung and Dong found that Malayasian medical students have an awareness of AI and that they are willing to learn more about it [16].A study by Buabbas et al. also reported a positive attitude of students toward AI in medical education.Most of the students opined that AI can help in their teaching and learning [17].Additionally, LLMs can address faculty shortages, support research, and innovation, and foster critical thinking skills.[18].To fully realize these benefits, educators and institutions in developing countries need to invest in faculty development and carefully integrate LLMs into their curricula, thereby harnessing the advantages while mitigating associated risks in medical education.Figure 2 summarizes the domains where LLMs can be used in medical education [19,20].A key limitation of this study is its reliance on self-reported data from medical students, which may introduce response bias.Additionally, the study's findings are context-specific and may not be fully generalizable to diverse medical education settings, particularly those in developing countries with varying access to technology and resources.The response rate was low in the survey.Furthermore, the study does not explore the long-term impact of LLM integration into medical education or assess the effectiveness of specific educational interventions.Future research should consider these limitations and incorporate more diverse and objective measures to provide a comprehensive understanding of LLMs' role in medical education.

Conclusions
The study reveals the current knowledge, attitude, and practice of using LLMs in medical education in an Indian medical college.While there is a generally positive attitude toward their incorporation, concerns about overreliance and potential inaccuracies are evident.LLMs offer the potential to enhance learning resources and provide accessible education, but their integration requires careful planning, faculty development, and the cultivation of critical thinking skills.This research underscores the evolving role of technology in medical education and calls for further studies to explore the long-term impact of LLMs in diverse educational contexts.

Appendices
The questionnaire used in this study can be used for any non-commercial research or academic purposes without any permission.The questionnaire can be created from the questions or statements in Tables 2, 3, and 4. A pdf of the questionnaire with consent and instruction can be obtained from Dr. Himel Mondal, Assistant Professor of Physiology, All India Institute of Medical Sciences, Deoghar, Jharkhand, India, via email (himelmkcg@gmail.com).
College issued approval 51/BIO/2023.This study protocol has been approved by Institutional Review Board.Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue.

Conflicts of interest:
In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work.Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work.Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.

FIGURE 2 :
FIGURE 2: Domains of medical education where LLMs can help humans Q&A, questions and answers; LLM, large language model; PPT, PowerPoint presentation ).

TABLE 1 : Question-wise score and average score in knowledge, attitude, and practice of using large language models for educational purposes
, Question or statement in the questionnaire (the number after Q indicates the number of the question or statement)The data in the table are presented as mean ± standard deviation.The average score is calculated by adding the raw score of six questions and dividing the value by 6.A total of 39.53% of students had an awareness of LLMs.Moreover, 61% of students believed that they understand how LLMs generate information and responses, indicating a decent grasp of their functioning.Notably, a majority (53.49%) acknowledge the potential for LLMs to be used by both teachers and students.A total of 62.79% of students find the LLMs to simplify the topic they learn.However, they are cautious about their limitations and potential for inaccuracies, with 34.88% agreeing that LLMs can generate wrong information (Table2). Q

TABLE 2 : Knowledge of medical students on LLMs in medical education
The data in the table are presented as numbers (percentage).The p-value is of the chi-square test where a significant p-value indicates that the distribution of the responses was not by chance.