Hello COMSEP!
It has been a difficult month for many COMSEP members. Lavjay’s inspiring words about hope may be helpful for some. This is also the season for gratitude, an evidence-based potential path to happiness. Between hope and gratitude there may be some relief.
Enjoy!
Amit, Karen and Jon
Does GPT-4 perpetuate biases in medicine?
Zack T, Lehman E, Suzgun M, et al. Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. Lancet Digit Health. 2024;6(1):e12-e22. doi:10.1016/S2589-7500(23)00225-X
Reviewed by: Aleisha Nabower
What was the study question?
Does GPT-4 perpetuate racial and gender biases in clinical applications?
How was the study done?
The study evaluated for biases when using GPT-4 in both medical education and diagnostic and care planning. For the medical education portion GPT-4 was used to provide a clinical vignette for 18 different diagnoses, with 1000 presentations generated per disease as well as two prompt debiasing strategies. Demographic distribution of generated cases was then compared to prevalence estimates of the disease in the US. For differentials and treatment plans cases from NEJM Healer were input varying the gender and race or ethnicity and prompting for the top 10 most likely diagnosis, and list of life-threatening diagnosis, next diagnostic steps, and treatment. Triage ability was assessed by inputting 8 cases varying only the gender and race/ethnicity and comparing results on 5 questions.
What were the results?
There were significant differences in GPT-4’s modeling of diseases for race and gender compared to true US prevalence across almost all diseases. Prompting GPT-4 to consider demographic prevalence of the disease did not significantly change results, while prompting it to avoid bias led to over-representation of female and Black patients across all diseases. In the clinical management GPT-4 changed top diagnosis in almost 1/3 of cases and recommended less diagnostic studies based on gender or race and ethnicity. Its perception of patient dishonesty also varied by race.
How can I apply this to my work in education?
The potential for perpetuation of biases when using AI to generate clinical vignettes should be recognized by educators so that they can address this when editing the cases. Additionally, as AI becomes intertwined with medical care it is critical that we teach our students on how to mitigate the potential pitfalls of the technology.
Editor’s Note: This is especially concerning when we consider the speed with which AI technology has infiltrated the clinical care space in the last couple of years. But it is not surprising, since the training data for ChatGPT and other AI technology is based on human perspectives which are themselves biased (JG).
Pulled in too many directions
Newman LR, Nagler A, Rudd M, et al. Lost in the pandemic: COVID-19's impact on health professions educators. Clin Teach. 2024;21(5):e13764. https://dx.doi.org/10.1111/tct.13764
Reviewed by Patricia Pichilingue-Reto
What was the study question?
What was the impact of the COVID-19 pandemic on health professions educators (HPEs)’ performance regarding clinical and educational tasks and work-life integration?
How was the study done?
From March 11 to May 25, 2021, the study team distributed a survey to members of the Association of American Medical Colleges Academy Collaborations (24 of the 81 agreed). The data collected included demographic characteristics and responses to questions related to the impact of the pandemic on overall professional activities, specific educational activities, work-life integration, and the extent to which professional work was compromised by personal caregiving duties or concerns related to children. The study also involved a qualitative analysis of the free-text responses from the participants.
What were the results?
Of the 2794 members, 20% (559) responded. Most (67%) were female. The most common profession was physician (63.5%) with most physicians being pediatricians (24%). Respondents range of experience was similar. HPEs reported spending more time on clinical care and education leadership/administration, while spending less time on their own professional development. Half of the participants reported being dissatisfied with their work-life integration, and a quarter reported being satisfied. Nearly 40% of respondents indicated that their professional work was compromised by personal caregiving duties. More females than males (41% vs 30%) indicated that caregiving duties compromised their professional work. Themes that emerged from the qualitative analysis included. fatigue, personal strain, isolation, and concern for learners.
How can I apply this to my work in education?
The study's findings have several implications for the support and well-being of HPEs during and after the COVID-19 pandemic. The study highlighted the need for institutional and organizational support to address the challenges faced by HPEs. Institutions should recognize the impact of the pandemic on work-life integration for HPEs and provide resources and support to help them balance personal and professional responsibilities. The study's findings regarding the impact of caregiving duties on professional work, particularly for females, underscore the need for gender-sensitive policies and support mechanisms within academic and healthcare institutions. Institutions should prioritize mental health resources and support for HPEs. A follow-up study to comprehensively examine the enduring repercussions of the pandemic on the personal and professional lives of HPEs could provide insights into the lasting impacts and how HPEs have navigated and adapted to the challenges presented by the pandemic over time.
Editor’s Note: While their survey distribution methods were not the best, I do think they uncovered how many of us are feeling post pandemic. I also think that institutions need to see how priorities have shifted for many HPE’s. (AKP)
FIRST is fast and not worse
Chen F, Jordan KA, Li W, Lam Y, Pascarella L, Coe CL. Academic performance of students in an accelerated medical pathway. Med Educ Online. 2024 Dec 31;29(1):2345444
https:dx.doi.org/10.1080/10872981.2024.2345444
Reviewed by: Shannon Bennett and Jamie Fey
What was the study question?
How is a student's academic performance and preparedness for residency impacted by three-year accelerated pathway medical school programs?
How was it done?
This retrospective observational study compared academic outcomes of the FIRST (Fully Integrated Readiness for Service Training) program at the University of North Carolina School of Medicine, which is an accelerated pathway through medical school in which students complete undergraduate medical education in 3 years to a traditional curricular model within the same institution. Metrics included exam performance, and clinical performance scores from 2018 to 2023.
What were the results?
Student demographics differed significantly only by mean age at graduation of 1.5 years younger for FIRST program students versus students in traditional four year medical school programs. There were no significant differences between the group’s scores on pre-clinical course final exam scores, NBME shelf exams, or USMLE Step 1 scores. The FIRST program students did score an average of 5 points lower on USMLE Step 2 CK compared to students in traditional programs. However, there were no significant differences in clinical performance between the two groups including during clerkships and rotations typically completed in the fourth year of medical school e.g. acting internships and critical care rotations.
How can I apply this to my work in education?
Accelerated pathway programs do not negatively impact the clinical performance of their graduates as compared to graduates of traditional 4-year curricular models. Students should be encouraged to pursue these options, when available, as these programs have the potential to reduce the student loan debt burden of their graduates, which may permit them to enter lower paying specialties and increase the diversity of the physician workforce.
Editor’s Comments One of the primary curricular differences between FIRST students and traditional students is elective opportunities. An interesting aspect of the accelerated program was that completion of this curriculum led to a directed pathway into an affiliated residency training program (family medicine, general surgery, pediatrics or psychiatry). It would be interesting to explore whether those who applied to the program felt they had sufficient career exploration opportunities prior to entering one of just 4 residency programs. (KFo)