COMSEP Journal Club
August 2021
Editors: Karen Forbes and Jon Gold
You’re ready when you’re ready…
The Minnesota Method: A Learner-Driven, Entrustable Professional Activity-Based Comprehensive Program of Assessment for Medical Students. Hobday PM, Borman-Shoap E, Cullen MJ, Englander R, Murray KE. Academic Medicine. 2021;96(7S):S50-S55.
https://dx.doi.org/10.1097/ACM.0000000000004101
Reviewed by: Danielle Amundsen
What was the study question?
What is the feasibility of a competency-based, time-variable progression through undergraduate medical education?
How was it done?
The Education in Pediatrics Across the Continuum (EPAC) program developed an EPA based assessment strategy. Four cohorts (3-4 students annually) of a learner-driven program with assessment based on the 13 Core Entrustable Professional Activities (EPAs) were evaluated. Students participated in a longitudinal pediatrics clerkship with interspersed blocks of specialty rotations. For each clinical session, students requested feedback from faculty on one or more EPAs with a three minute face-to-face debrief and online survey. Feedback focused on strengths, next steps for improvement, and the level of supervision of the EPA(s). Students’ learning curve for each EPA was updated monthly. Students met with the course director 1-2 times per month to track progress. Quarterly, preceptors provided summative feedback, and a clinical competency committee reviewed students’ progress and assigned a supervision level for each EPA. Following this meeting, each student met with the director to develop individualized learning plans. After reaching the pre-specified supervision level required for graduation in 11 EPAs, students progressed to advanced clinical electives where students must demonstrate the remaining EPAs: “give or receive a patient handover to transition care responsibility” and “recognize a patient requiring urgent or emergent care and initiate evaluation and management.”
What were the results?
Students averaged 15-22 Core EPA assessments per month, increasing with each cohort. Students reached the graduation standards in an average of 12.2 months (median 12, range 10-20 months). Four students did not reach the graduation standards by the expected 12 months and participated in enrichment electives based on their individual learning needs. Learners transitioned to residency in average of 16.7 months (median 17, range 14-22 months) compared to expected duration of 24 months.
What are the implications?
This study shows the possibility of competency-based, time-variable program for medical education. The authors note that five factors contributed to the success: 1) clear and defined outcomes, 2) online and simple evaluations, 3) learner-focused developments, 4) learner driven assessments with accountability and 5) easy access to actionable data to monitor progress and make entrustment decisions. This program centered around a longitudinal core clinical experience with pass/fail grading and small cohorts. Additionally, learners had guaranteed residency placement, perhaps leading to a growth mindset as opposed to an achievement-oriented mindset.
Editor’s Comments: Clear expectations of roles and responsibilities of both faculty and students was instrumental to the success of this assessment program. Of note, these learners were guaranteed a placement as they transitioned from medical school to pediatrics residency. How such a system can be applied to students in general, without a known or guaranteed residency position, is yet to be determined (KFo).
Interns: An Untapped Resource?
A Qualitative Exploration of the Intern Experience in Assessing Medical Student Performance. Cohen A, Kind T, DeWolfe C. Acad Pediatr. 2021;21(4):728-734. https://dx.doi.org/10.1016/j.acap.2020.10.014
Reviewed by Kelly Nedorostek and Melanie Rudnick
What was the study question?
What is the experience of first-year residents with respect to assessment of medical student performance in the inpatient setting?
How was it done?
This qualitative study was completed at a large free-standing children’s hospital with pediatric interns who had completed at least one inpatient clinical rotation working with medical students. Participants from two separate intern classes were recruited at various time points throughout the year, with no volunteers excluded from the study. Ten individual semi-structured 30 to 60-minute interviews were completed to investigate interns’ experiences with regard to their role in student assessment, both formative and summative. Audio-recording of all interviews were obtained, transcribed and de-identified prior to analysis, which was completed by two independent authors. Constant comparative methodology was utilized to code the information and create themes, with thematic saturation obtained after ten interviews. To enhance trustworthiness of the study, a third individual was involved in thematic development and to resolve any disagreements in coding. Additionally, two faculty focus groups were utilized for member checking and triangulation.
What were the results?
Four major themes were identified. Firstly, interns felt as though they had a unique role due to their proximity to medical school and because they spent more time with students than other team members. Secondly, various barriers and facilitators to assessment were identified. For example, interns found assessment easier when pre-identified assessment or evaluation forms were available, more time was allotted with a student, or if a student was more proactive, engaged, and skilled. In contrast, high volume of patient tasks, lack of time dedicated for assessment, and struggling or disengaged students were identified as barriers to an intern’s ability to appropriately perform student assessment. Thirdly, interns’ motivation, skill, and confidence affected student evaluation, and as these traits developed, they found it easier to assess students. Lastly, interns perceived that they were better at providing formative feedback and less comfortable with summative feedback, expressing concerns regarding the long-term impact of a summative assessment on career trajectory, unfamiliarity with grading rubrics, and insufficient time directly observing students.
What are the implications?
This study suggests that pediatric interns are valuable in the assessment process of students due to their unique role on the medical team, especially as their confidence and knowledge base grows. Efforts toward formalized and institution-specific training for interns may help facilitate collaborative assessment between all team members given interns’ perceived lack of knowledge, skills and attitudes in providing summative assessments. While interns acknowledge they often spend the most amount of time with medical students, they also state lack of time assessing the students to be a barrier to providing meaningful observations to appropriately add to the summative assessments. More direct observation of learners by all members of the healthcare team should be stressed in order to provide the most accurate and meaningful assessments of learners in order to allow growth of all members of the team. Finally, students should be encouraged to be proactive about eliciting feedback, and asking for direct observation, as this was noted to be a positive factor to enhance the assessment process.
Editor’s Note: The concerns expressed by these interns about summative assessment are not unique to interns. Finding the right balance between including the valuable perspectives of near-peers like interns and the need for multiple assessments to create a valid assessment system is one of the greatest challenges for clerkship directors. (JG)
Writing exercises as a window into professional identity development
Reading the Self: Medical Students’ Experience of Reflecting on their Writing Over Time
Cunningham H, Taylor DS, Desai UA et al. Academic Medicine 2021; 96(8):1168-74. https://dx.doi.org/10.1097/ACM.0000000000003814
Reviewed by Karen Forbes
What was the study question?
How do students experience Signature Reflections – meta-reflection writing exercises – over time?
How was it done?
Eighteen third-year medical students participated in 5 focus groups to explore experiences on Signature Reflections which were part a longitudinal 4-year portfolio curriculum. The curriculum involved monthly in-class, faculty-mentored narrative medicine exercises during preclinical years, and entries on personal dimensions of patient encounters during clinical rotations and clinical intersessions. Furthermore, to strengthen reflective capacity, students read and wrote about prior portfolio entries as a “mirror” of themselves at specified times, called Signature Reflections. Faculty-mentors follow students through the four years and respond to their writings with nonevaluative feedback. Participants were asked to re-read three of their Signature Reflections prior to the focus groups. Focus groups were conducted using an interview guide. Transcripts were analyzed using an iterative thematic approach, developing codes to reflect common patterns, and assembling the codes into themes.
What were the results?
Three core themes were identified: safe space, narrative experience, and mirror of self with an overarching theme of moving through time. The safe space allowed students to acknowledge fears and vulnerabilities, provided protected time and legitimized mandated reflected writing as something valuable. Faculty-mentors also influenced the sense of safe space. The narrative experience allowed students to document their journey through medical school, noting that many did not initially appreciate the portfolio curriculum but grew to value it eventually. Signature Reflections provided a mirror of self, encouraging students to check in with themselves to notice growth, gaining of resilience, overcoming challenges, and professional identity development.
What are the implications?
The meta-reflection component of this longitudinal curriculum allowed students to see their growth over time. There was great value in having longitudinal faculty mentors who provided support, helped students develop insight, and fostered relationships between students and faculty, further strengthening students’ reflections on their own growth.
Editor’s Note: In some ways this curriculum resembles the common elementary school exercise in which students write letters to themselves and have an opportunity to review them at a later date. As painful as it sometimes is, we all gain something by looking back at our past selves. (JG)