|
Lurie SJ, Nofziger AC, Meldrum
S et al. Effects of rater selection on peer assessment among medical students.
Medical Education 2006; 40(11):1088-1097. Reviewed by
Starla Martinez , Northeastern Ohio Universities College of Medicine That
is the issue, and what is known about it so far?
The LCME, among others, is
calling for education in and assessment of various professionalism-related qualities
of students such as integrity, teamwork, etc. The use of peer assessment is an
attractive idea for evaluating students longitudinally, reasoning that students'
peers see students "in action" whereas many preceptors, especially attendings,
may only see students for brief snapshots of their training, some of which may
be in artificial settings such as with standardized patients. Peer assessment
among physicians is a growing area of research but there are limited data thus
far among medical students. Why did the researchers do this particular
study?
These researchers have previously shown that medical students can assess
reliably the interpersonal attributes of their peers but it is not known whether
the interpersonal attributes of students themselves affects their perception or
scoring of their colleagues. The authors' hypothesis was that students who were
rated by their peers as low on the interpersonal-attribute scale would themselves
rate other students low on the scale (before they saw their own results). The
researchers did this study to look at whether the students' own biases affect
how they rate their peers, and whether the method of assigning student raters
has any effect on overall scores. Who was studied?
296 second and
third year medical students from three different class years (2004, 2005 and 2006)
at the University of Rochester NY. How was the study done?
As part
of a larger study, students anonymously rated 6-10 other students using a standardized
instrument geared toward evaluating interpersonal skills and work habits. The
study included two different methods of assigning raters: 1) "School-assigned:"
A portion of the students were assigned raters based on the amount of time they
had spent with students in small group learning in the first two years of medical
school. 2) "Self-assigned:" A portion of the students were asked to
list 6-10 students they believed could give them constructive feedback and also
to list 6 students to whom they felt they themselves could give constructive feedback. All
students performed their peer assessments using a 15-item web-based instrument
with a 5 point scale for each item and provision for submitting narrative feedback.
Within 1 week all students received their scores and narrative feedback as well
as the class means, and then met with their advisors to discuss the results. The
researchers used various statistical methods to interpret the results comparing
the school-assigned and self-assigned groups and looking at each year compared
to the others. They compared the lowest quartile scores with the upper three quartile
scores. What did the researchers find?
Medical students who are rated
low (in the lowest quartile) by their classmates on interpersonal skills are significantly
more likely simultaneously to assign low (in the lowest quartile) scores to their
classmates. Also, the low-scoring students were more likely to choose other low-scoring
students to rate and be rated by. The method of assigning students whom
to rate or be rated by had no significant effect on the scores for most students.
In other words, the researchers found that even when students chose whom they
wanted to be rated by, the final score for most students wasn't significantly
different than when the student was assigned raters by the school. They suggest
that therefore schools may "...feel confident choosing a method of peer selection
based on logistics and convenience, including students' views on the process." What
were the limitations of the study?
It seems to me that the authors' conclusion
about assignment method making no difference in scores would only be true if the
lowest quartile is the only way "low" is defined. If a low score were
defined empirically, such as ≤2 on a 5-point scale, it could make a great deal
of difference if assignment methods differed. I think one of the conclusions perhaps
ought to be that, if a school plans to allow students to choose their own raters,
the school needs to insure that the low score is defined using the lowest quartile
rather than a particular score. What were the implications of the study?
This
study confirms the hypothesis that students who are rated in the lowest quartile
in interpersonal attributes by their peers are themselves likely to rate other
students as low on the interpersonal attributes score. This study was not designed
to look into why this is the case, but it's a very interesting finding to ponder.
The authors speculate that perhaps students who have low interpersonal scores
have tended to choose low-scoring students to spend time with over the course
of training. (Editorial Comment: The really hard stuff follows. What
are the elements of these low scores? Are they likely to interfere with performance
as a physician in the 21st century? If so, 1) are any of them identifiable
before acceptance to medical school? 2) Are any of them able to be modified? Changing
an adult's behavior is no small task. Bruce Morgenstern) | 
