Council on Medical Student Education in Pediatrics


Search This Site

Journal Club

White, BW, Ross PT and Gruppen LD. Remediating Students' Failed OSCE Performances at one School: The Effects of Self-Assessment, Reflection, and Feedback. Academic Medicine 2009;84:651-654

White, BW, Ross PT and Gruppen LD. Remediating Students' Failed OSCE Performances at one School: The Effects of Self-Assessment, Reflection, and Feedback. Academic Medicine 2009;84:651-654

Reviewer: Margaret Clark-Golden MD MPH. SUNY Downstate College of Medicine. Brooklyn, NY

What is the problem and what is known about it so far?
Most schools now use some sort of summative multi¬station OSCE as a graduation requirement, and have struggled with how to remediate students who fail. Remediation methods vary across schools, but all of them tend to put intensive demands on faculty time.

Why did the researchers do this study?
Since self-assessment is a sine qua non for self¬directed learning, these researchers sought to measure whether a carefully structured program of self¬assessment for students with poor perfonnance on the summative OSCE could be effective in improving their subsequent performance on a re-exam.

Who was studied?
Students failing one or more of the 7 SP stations on the 13-station OSCE required of University of Michigan students at the beginning ofthe 4th year. The SPs grade the students at each station. In addition, students score themselves after each station. Students must pass each station in order to graduate; failing students must re-take and pass any failed station.

How was the study done?
Students who failed a station were required to do an Intranet-based self-assessment remediation exercise before retaking that station. (For most SP cases, there were 2 versions ofthc case.) The remediation had 4 elements:

  1. review of station-specific articles/resources
  2. viewing a video ofa an excellent student performance for that station
  3. viewing of the student's own recorded performance
  4. reflection, in writing, on the differences between the two performances.

For students whose score was more than one standard error below the pass mark, the remediation also included written personalized feedback on the student's video from a faculty member.

What did they find?
42/173 students failed 1-3 stations; mean scores on the failed stations was 53.4 vs. 77.5 for passed stations. The self-assigned score was approximately 83 for all takers; which was close to the SP rating for the stations passed on the first try, but 30 points too high for the stations failed initially. Thirty-seven cases were "self¬ remediated" (mean score 55.9) and 20 cases were also "faculty remediated" (mean score 48 .6). All students passed the failed station on the second attempt, with significant increase in scores (80.5 for the "self¬ remediated" and 77.4 for "faculty-remediated").

What are the limitations of this study?
Although this study suggests that students, given a well structured self-directed exercise, can improve their performance in a complex task (an SP encounter), it cannot answer whether there is any value added to getting personalized written feedback from a faculty member. (Maybe one could now randomize the students with performance in the "needs improvement range" to have faculty feedbac k or not, and compare scores, accuracy of self-assessment, or some other measure.) Questions remain about what the improved performance means: the second trial was with a partially known cneounter-a patient with the same chief complaint, even if the case was different in some details.

What are the implications of this study?
As important as self-assessment is for adult learners, and as much as we rely on it, self-assessment can be seriously tlawed. The researchers demonstrated how the lowest performing students vastly overestimated their own skills. Learners need to "know what they don't know." The good news is that self-assessment and retlection-even without faculty feedback---can significantly improve a student"s performance on a defined clinical encounter-at least for students with performance in the range of"needs improvement." This study doesn't tell us if the students with the weakest performances could have passed the retake without faculty feedback; it would not have been ethical to do such a study when this project was undertaken, but it might be possible to do so now. This study actually points us in a very important direction, because all 173 students did get feedback on the accuracy of their self-assessments. And the self¬directed remediation process did more than "prep"the student to pass the fe-test. I believe it spelled out the expectations in sufficient detail for even weaker students to apply themselves to mastering those expectations. It's hard to define clear expectations for clinical learning; when we try to do so, the resulting documents consume reams of paper. But to provide detailed expectations for one or more types of encounters may model to students how to do the same thing for themselves with new, previously unknown types ofclinical problems.

Ed note: I have yet to meet an educator who does not love the concept of self-retfection. Alas, I have met few medical students who feel similarly. This study proves yet again that the lowest performing students are most li kely to overestimate their own skills. Gening feedback on their self-assessment skills may be one step in stimulating greater student interest in this critical skill. (WVR)

Return to Journal Club