Hello COMSEP!
Finally, the day you’ve all been waiting for—it’s National Crab-Stuffed Flounder Day! After today, there’s a long, bleak wait until National Corn Dog Day next month. (Canadian colleagues, don’t feel bad that you miss out on our fine American cuisine—you have many excellent foods as well).
Karen, Jon and Amit
Prompt Assessments
Jamieson, A., Holcomb, M., Dalton, T. et al. Rubrics to Prompts: Assessing Medical Student Post-Encounter Notes with AI, NEJM AI 1, 12 (2024). https://dx.doi.org/10.1056/AIcs2400631
Reviewed by Drew Galligan
What was the study question?
Can an artificial intelligence (AI)-based automated grading system effectively and efficiently grade medical student post-encounter Objective Structured Clinical Examination (OSCE) notes?
How was the study done?
The study was conducted at UT Southwestern Medical Center’s Simulation Center. The AI system was designed using a zero-shot large language model (GPT-4), with the purpose of grading post-encounter notes without requiring prior domain-specific training data. The grading rubrics were transformed into a series of minimally engineered prompts for the AI to assess. Prior to deployment, the AI system underwent retrospective validation using data from previous OSCE sessions (2019-2023) to ensure its reliability and accuracy. The AI grading system was then prospectively deployed for 245 pre-clerkship medical students during a 10-station OSCE session. Finally, the AI's performance was compared with human expert graders, measuring agreement at the rubric item level and overall examination scores.
What were the results?
The AI grading system reduced manual grading effort by an estimated 91% and considerably reduced turnaround time for grading (from weeks to days). There was high agreement between the AI system and human graders, achieving up to 89.7% agreement at the rubric item level and a Spearman’s correlation of 0.86 with the total examination score. The AI system demonstrated the potential for scalability and adaptability to different evaluation rubrics without additional training data.
How can this be applied to my work in education?
Implementing an AI-based grading system can significantly reduce the time and effort required for grading student assignments, allowing for quicker feedback and more efficient use of educator resources. The zero-shot approach allows for easy adaptation to different subjects and grading rubrics, making it suitable for various educational contexts. AI systems can provide consistent and unbiased evaluations, reducing the variability and potential biases associated with human grading. By automating routine grading tasks, educators can focus more on higher-order instructional activities, such as personalized student support and curriculum development.
Editor’s Note: Two clear advantages that AI systems have over human evaluators are evident in this study--speed and scalability. The data also reassures us that AI assessments are trustworthy--especially when backstopped by human observers (JG).
Give a little and you get something back
Haidar, A., Erickson, S. G., & Champagne-Langabeer, T. Medical Students’ Participation in Longitudinal Community Service During Preclerkship Years: A Qualitative Study on Experiences and Perceived Outcomes (2020). Journal of Medical Education and Curricular Development, 7. https://doi.org/10.1177/2382120520936610
Reviewed by: Jackson Borak & Michelle Veters
What was the study question?
What are medical students’ viewpoints on the benefits of a longitudinal community service experience?
How was the study done?
Qualitative methodology was used to explore preclinical medical students’ perceptions of community service in two different programs. Some students voluntarily participated in a longitudinal, one-hour weekly nutrition education program at a local middle school. All first-year medical students participated in an annual, single day event. Semi-structured interviews were conducted with fifteen students, who consistently participated in the longitudinal program and who also participated in the service day, to understand their volunteer experience and its effects on them. Coding was used to distinguish themes. All students identified as being of middle-class socioeconomic status. Before the study, eleven students had not previously participated in volunteer activities, while four regularly volunteered through other service initiatives.
What were the results?
Themes that emerged from the students’ service experiences included (1) communication and interpersonal skills, (2) teaching ability, (3) understanding of community issues in an underserved area, and (4) overall well-being. One student stated, “…you have to learn how to present the information in a way that is easy for them (middle school students) to understand. This helped me become an effective communicator and will be useful for my future as a doctor…” Students responded positively to both the longitudinal education program and the single day of service. However, the longitudinal volunteering experience provided sustained improvement in well-being and skill development.
How can I apply this to my work in education?
This study highlights the value of longitudinal community service in developing essential physician skills, such as teaching and communication, during the preclinical years. Community engagement enhances understanding of social determinants of health and provides a rewarding outlet from academic pressures. Integrating these programs into the medical curriculum can help bridge the gap between preclinical training and relational patient care while promoting personal and professional growth.
Editor’s Note: Academic pressures can often leave students feeling there is not enough time to participate in volunteering. It was nice to see the students’ reflections on the benefit of participation in such community service on their own well-being and stress, particularly as undergraduate medical programs focus on student wellness initiatives. (KFo)
Game On
Burney E, Arora M, Gaillard M, et al. A Game-Based Tool for Reducing Jargon Use by Medical Trainees. MedEdPORTAL. 2024;20:11411. https://dx.doi.org/10.15766/mep_2374-8265.11411
Reviewed by Erin Chen
What was the study question?
Do medical students enjoy Patient Communication Challenge (PCC), a medical education-based adaptation of the game Taboo, to enhance their patient communication skills?
How was the study done?
The authors conducted eight virtual and two in-person workshops with preclinical medical students from 1/2021 to 12/2022. Each workshop included three key components: an educational video on communication and health literacy, participation in the PCC game, and a facilitator-led discussion for reflection. During the game, students worked in teams to explain medical concepts related to the recently completed block using patient-friendly language, while avoiding medical terms listed on the cards. The rest of the team had to guess the concept based on the explanation. Data on perceived skill gain and enjoyment was collected through optional anonymous post-workshop surveys. Thematic analysis was used to identify recurring themes in qualitative comments.
What were the results?
Of the 90 who participated, 59% completed the post-workshop survey. The study found that 91% of respondents felt more proficient at avoiding medical jargon after the workshop, 94% said they would recommend the workshop to a friend, and 100% would play again. The qualitative feedback highlighted positive themes, including increased engagement, enjoyment, a greater awareness of the importance of avoiding jargon, and gratitude for the workshop experience. Suggestions for improvement focused on adjusting game mechanics, such as adding more concepts to play, modifying the timing of rounds, holding more in-person sessions, and assigning dedicated guess-givers to improve the game's flow.
How can I apply this to my work in education?
In this study, the authors demonstrate that game-based educational tools, such as PCC, offer a fun and engaging approach to developing communication skills for medical students. They propose a potential framework for incorporating spaced repetition and clinical applications that can complement preclinical didactic learning. Health literacy training, which is critical to patient-centered care and improving health outcomes, is not consistently implemented in medical education, particularly early stages of training. Medical educators should consider incorporating active learning activities focused on health literacy throughout preclinical training to help students develop a strong foundation in essential patient communication skills.
Editor’s note: This is a great way to teach patient centered communication and many of the authors are students. The curriculum is on MedEd Portal so this would be easy to download and use (AKP)