December 2024

Hello COMSEP!

Rounding the corner into the holiday season and a new year.  Today’s the anniversary of the first flight at Kitty Hawk in 1903.  The Wright Brothers probably didn’t imagine this when they made it into the air:

Hope your holiday travels are relatively painless,

Amit, Karen and Jon


Educator, assess thyself
Hauer KE, Park YS, Bullock JL, Tekian A“My Assessments Are Biased!” Measurement and Sociocultural Approaches to Achieve Fairness in Assessment in Medical Education.  Acad Med. 2023 Aug 1;98(8S):S16-S27. https://dx.doi.org/10.1097/ACM.0000000000005245.

Reviewed by Amy Grover

What was the article about?
The study authors describe bias in assessment, explore its consequences and provide practical recommendations to help overcome bias in assessment systems.  Authors used Messick’s unified validity framework to contextualize assessment bias.

What do they describe?
Examples of bias within medical assessment and potential career ramifications are described. The authors then emphasize the pursuit of validity – that assessment adequately captures the construct being measured – and fairness – which enables all learners, regardless of individual characteristics, to demonstrate achievement in the construct being measured – in assessment. They provide 12 specific recommendations within 5 domains essential to ideal assessment practices to reduce harmful bias. Assessment practices should be explicitly designed to ensure all learner groups have opportunities to learn and perform at their best, with policies in place to codify fairness and equity as foundational to assessment. Clearly articulated expectations for learner performance, using criterion-based standards and shared rubrics rather than normative comparisons with peers, are recommended. Students should receive assessment data in a low-stakes, formative fashion prior to high-stakes evaluation. Performance should be assessed by multiple sources, ideally including narrative assessment based on direct observation. Groups comprised of a diverse range of participants should be utilized for grading and promotion decisions. Finally, incorporation of diverse perspectives in the design of an assessment system, and commitment to regular data review to specifically look for and address bias, is imperative.

How can this be applied to my work in education? 
I was struck by the fact that educational best practices such as shared expectations between students and preceptors, competency-based assessment, and narrative comments derived from direct observation are also recommended as strategies to mitigate bias. Educators must encourage students and faculty to develop skills in these areas.

Editor’s Comments: This article encourages medical educators to critically reflect on assessment practices and brings to light ways that bias may permeate assessment in both validity and fairness. I would encourage all medical educators involved in assessment to read this article and consider how their recommendations align with personal and institutional assessment practices. (KFo)


A deep dive into small group teaching and learning strategies
Zheng B, He Q, Lei J. Knowledge Construction in Problem-Based Learning: A Lag-Sequential Analysis of Teachers’ and Students’ Discourse Moves. Teach Learn Med 2024; 36(4):411-424, https://dx.DOI.org/10.1080/10401334.2023.2230559

Reviewed by Gary L. Beck Dallaghan

What was the study question?
How do teachers and students interact through discourse in problem-based learning (PBL) settings within a medical education context?  Specifically, what are the core participation patterns observed during small-group PBL discussions, what kind of the types of discourse moves are employed by teachers and students, and what are the sequential patterns and relationships between these discourse moves?

How was the study done?
Zoom recordings were made of 2 two-hour PBL sessions, one led by a clinician and one by a basic scientist, each with 11 first-year medical students.  Recordings were transcribed including notations of silent moments.  Participants’ verbal contributions and durations were recorded.  Transcripts were coded and placed into categories according to a pre-existing framework.  The sequence of comments were then analyzed for patterns.

What were the results?
Teachers contributed the most to the discussions (speaking up to 31% of the time), while students’ participation varied. Among the students, the group chairperson was a prominent contributor, often guiding the discussion and engaging the most. The presence of silence was notable, consuming approximately 4% of the sessions.  Tutors employed a mix of strategies such as asking probing questions and helping maintain the discussion flow. Students demonstrated a range of cognitive engagement, from lower-level thinking (clarifications) to higher-order thinking (making inferences).
Sequential analysis of the discourse patterns suggested that tutor questions prompted student engagement in lower- and higher-level thinking. Silence was often a prelude to significant cognitive or process-oriented activity, such as student reflections.

How can I apply this to my work in education?
This study provides insights for medical educators involved in any small group teaching activities. It underscores the importance of using well-crafted, probing questions to promote critical thinking to guide students from surface-level understanding to deeper cognitive engagement. Use of affirmations and encouragement promotes a supportive and psychologically safe environment, essential for open dialogue, collaborative problem-solving, and active participation among students. Finally, it is important to be comfortable with silence, giving learners an opportunity to reflect and process.

Editor’s Note:  For anyone who thinks of case-based learning as a fully self-directed learning activity for students, this study should disabuse them of that notion.  The teachers in this study, and in most small group teaching environments, are working hard to make the experience valuable and equitable. (JG)


Towards bias-free admission
Davidson MA, Russell RG Walker, PD, Zic, JA, Churchill,  Fuchs, DC, Miller, BM.. MD,. Evaluating the Role of Competency-Based Behavioral Interviewing in Holistic Medical School Admissions.  Academic Medicine 99(9):p 966-970, September 2024. https://dx.doi.org/10.1097/ACM.0000000000005708

Review by: Gregory Ison and Srividya Naganathan

What was the study question?
Does integrating competency-based behavioral interviews (CBBI) with standardized interviews (SI) promote a holistic approach to the overall medical school admission process?

How was the study done?
Data from the 2015-2019 Vanderbilt University School of Medicine application pool was used. During this time, the admissions committee (AC) conducted both CBBI and traditional standardized interviews (SI), with the final selection decisions made by the AC. The CBBI questions included 3 categories (ethics, resilience, and curiosity) and were developed based on literature review of holistic admissions in medical education. The CBBI interviewers were blinded to the applicant’s transcripts and essays while the SI and AC had the complete applicant information. The CBBI categories were scored to create a points scale from 3 to 9.    SI scores ranged from 1-4, with higher scores being more favorable. A final score (combining SI and CBBI), was generated by the AC which led to acceptance or rejection.

What were the results?
Of the 2286 applicants, 52% of students were female, while 22 % were Underrepresented in Medicine (URiM). The authors found a strong association (p<0.005) between CBBI and SI, suggesting that interviewers look for the same applicant characteristics .Students with higher scores on both CBBI and SI were more likely to have higher final AC scores ( rather than either rating alone) and therefore likely to be accepted into the medical school. The proportion of URiM and male applicants went down as SI scores went up suggesting possible bias but the distribution was proportionate across CBBI total scores underscoring that CBBI may likely mitigate this bias.

How can I apply this to my work in education?
This study highlights the importance of including non-academic strengths such as professionalism, ethical behavior, resilience, adaptability, and curiosity in addition to the standard application metrics in the admission selection process. This approach, in addition to the selection of medical students, should be incorporated in the residency and fellowship application process as well  and may have a positive impact in the future physician work-force.

Editor’s Note: I appreciate this group looking at more holistic methods of admissions. While they did show some positive results the article did not specify how they did the correlation or the coefficient. Also there may be some selection bias as these were only students selected for an interview. (AP)