Are Scores from NBME Subject Examinations Valid Measures of Knowledge Acquired During Clinical Clerkships? Ryan MS, Bishop S, Browning J, Anand RJ, Walterhouse E, Rigby F, Al-Mateen CS, Lee C, Bradner M, Colbert-Getz JM. Academic Medicine 2016 (published online before print). Reviewed by Robyn Bockrath Tags: NBME Subject Exam, clerkship, validity
Are Scores from NBME Subject Examinations Valid Measures of Knowledge Acquired During Clinical Clerkships? Ryan MS, Bishop S, Browning J, Anand RJ, Walterhouse E, Rigby F, Al-Mateen CS, Lee C, Bradner M, Colbert-Getz JM. Academic Medicine 2016 (published online before print).
Reviewed by Robyn Bockrath
Tags: NBME Subject Exam, clerkship, validity
What was the study question?
What is the validity of using scores from NBME Subject Examinations as a component in final clerkship grades?
How was the study done?
This was a retrospective cohort study including all medical students at the Virginia Commonwealth University School of Medicine who completed core clerkships in 2012-2014. The NBME Subject Examination constitutes 25-35% of the final clerkship grade. For all included students in all core clerkships (n=432), the standard score achieved on NBME Subject Examinations was substituted with his/her standard score on Step 1. Scores for each clerkship were then recalculated using the score from Step 1 to create a “substituted” final score. Clerkship grades were then reassigned using the same norm-referenced values for distinguishing Honors, High Pass, Pass and Fail/Marginal. Linear regression was used to determine whether Step 1 scores significantly predicted Subject Examination scores.
What were the results?
Linear regression models demonstrated that Step 1 scores significantly predicted NBME Subject Examination scores for all clerkships, P < .001. Percentage of shared variance (R2) between NBME Subject Examination scores and Step 1 scores was medium to high. There was no significant difference between actual clerkship grades and Step 1 substituted grades. Seventy-three percent of Step 1 substituted grades matched the actual clerkship grades. For nonmatching cases, changes were usually (88%) going up or down one grade (e.g., honors to high pass) compared to 12% going up or down two grades.
What are the implications of these findings?
This study showed performance on the NBME Subject Examination can be predicted on the basis of Step 1 scores and using Step 1 Scores as a predictor of future NBME Subject Examination performance may allow educators to provide targeted counseling to students who are at risk of poor performance. Also, the convergent relationship between Step 1 and NBME Subject Examinations questions the utility of NBME Subject Examination in assessing clerkship specific knowledge. With the recent development of national clerkship-specific exams, one advantage to such exams may be a divergent correlation with the NBME Subject Examinations. Given the lack of divergent correlation between Step 1 scores and NBME Subject Examination scores, educators may need to reconsider how these exams are used in determining final clerkship grades as they may not be valid assessments of the knowledge obtained during a clinical clerkship.
Editor’s note: I doubt that anyone involved in medical student education would be surprised by these results and I agree that it would be valuable for educators to ponder how these exams are used in determining final clerkship grades. The latter point is especially salient given the uncertainty of connection between clerkship curricula and activities and NBME Subject Examination content. (RR)