Lurie SJ, Nofziger AC, Meldrum S et al. Effects of rater selection on peer assessment among medical students. Medical Education 2006; 40(11):1088-1097. Reviewed by Starla Martinez , Northeastern Ohio Universities College of Medicine
Lurie SJ, Nofziger AC, Meldrum S et al. Effects of rater selection on peer assessment among medical students. Medical Education 2006; 40(11):1088-1097.
Reviewed by Starla Martinez , Northeastern Ohio Universities College of Medicine
That is the issue, and what is known about it so far?
The LCME, among others, is calling for education in and assessment of various professionalism-related qualities of students such as integrity, teamwork, etc. The use of peer assessment is an attractive idea for evaluating students longitudinally, reasoning that students' peers see students "in action" whereas many preceptors, especially attendings, may only see students for brief snapshots of their training, some of which may be in artificial settings such as with standardized patients. Peer assessment among physicians is a growing area of research but there are limited data thus far among medical students.
Why did the researchers do this particular study?
These researchers have previously shown that medical students can assess reliably the interpersonal attributes of their peers but it is not known whether the interpersonal attributes of students themselves affects their perception or scoring of their colleagues. The authors' hypothesis was that students who were rated by their peers as low on the interpersonal-attribute scale would themselves rate other students low on the scale (before they saw their own results). The researchers did this study to look at whether the students' own biases affect how they rate their peers, and whether the method of assigning student raters has any effect on overall scores.
Who was studied?
296 second and third year medical students from three different class years (2004, 2005 and 2006) at the University of Rochester NY.
How was the study done?
As part of a larger study, students anonymously rated 6-10 other students using a standardized instrument geared toward evaluating interpersonal skills and work habits. The study included two different methods of assigning raters: 1) "School-assigned:" A portion of the students were assigned raters based on the amount of time they had spent with students in small group learning in the first two years of medical school. 2) "Self-assigned:" A portion of the students were asked to list 6-10 students they believed could give them constructive feedback and also to list 6 students to whom they felt they themselves could give constructive feedback.
All students performed their peer assessments using a 15-item web-based instrument with a 5 point scale for each item and provision for submitting narrative feedback. Within 1 week all students received their scores and narrative feedback as well as the class means, and then met with their advisors to discuss the results.
The researchers used various statistical methods to interpret the results comparing the school-assigned and self-assigned groups and looking at each year compared to the others. They compared the lowest quartile scores with the upper three quartile scores.
What did the researchers find?
Medical students who are rated low (in the lowest quartile) by their classmates on interpersonal skills are significantly more likely simultaneously to assign low (in the lowest quartile) scores to their classmates. Also, the low-scoring students were more likely to choose other low-scoring students to rate and be rated by.
The method of assigning students whom to rate or be rated by had no significant effect on the scores for most students. In other words, the researchers found that even when students chose whom they wanted to be rated by, the final score for most students wasn't significantly different than when the student was assigned raters by the school. They suggest that therefore schools may "...feel confident choosing a method of peer selection based on logistics and convenience, including students' views on the process."
What were the limitations of the study?
It seems to me that the authors' conclusion about assignment method making no difference in scores would only be true if the lowest quartile is the only way "low" is defined. If a low score were defined empirically, such as ≤2 on a 5-point scale, it could make a great deal of difference if assignment methods differed. I think one of the conclusions perhaps ought to be that, if a school plans to allow students to choose their own raters, the school needs to insure that the low score is defined using the lowest quartile rather than a particular score.
What were the implications of the study?
This study confirms the hypothesis that students who are rated in the lowest quartile in interpersonal attributes by their peers are themselves likely to rate other students as low on the interpersonal attributes score. This study was not designed to look into why this is the case, but it's a very interesting finding to ponder. The authors speculate that perhaps students who have low interpersonal scores have tended to choose low-scoring students to spend time with over the course of training.
(Editorial Comment: The really hard stuff follows. What are the elements of these low scores? Are they likely to interfere with performance as a physician in the 21st century? If so, 1) are any of them identifiable before acceptance to medical school? 2) Are any of them able to be modified? Changing an adult's behavior is no small task. Bruce Morgenstern)