Journal Club

Not knowing that they do not know: self-assessment accuracy of third year students. Langendyk A. Medical Education 2006; 40:173-179.

Reviewed by Bill Wilson, University of Virginia.

The ability to self-assess is an integral part of problem-based learning and, ultimately, "life-long learning." Previous studies, cited in this article, suggest that self-assessment varies widely, and that higher achievers tend to underestimate their performance while lower achievers tend to overestimate their performance. Few of these studies were done in medical school settings.

The author, from the University of Sydney (Australia), designed a study to compare student self-evaluation and evaluation by a peer with evaluations by faculty. Third year medical students (175) were studied. Each student completed a 90 minute written case-based formative assessment that required use of clinical reasoning and understanding of the relevant basic sciences. The students were then provided a model answer booklet and marking sheet, and were asked to evaluate specific areas in their own assessments on a 0-3 scale. They were also asked to evaluate the responses of a peer (randomly assigned) using the same scale. All assessments were also marked independently by a faculty member using the same scale. Differences and correlations between self-and peer- marking, self and faculty marking, and faculty and peer marking were analyzed. In addition, the students were divided into 3 groups, based on their scores from faculty (borderline, satisfactory, high satisfactory and the correlations between self-marking and peer and faculty marking were studied for each group.

In general, there was good correlation among self-scores, peer scores, and faculty scores. However, the lowest-performing group tended to mark themselves "generously," while the highest-performing group tended to mark themselves harshly. The lowest-performing group also tended to mark their peers more highly, while the middle group and the highest performers showed good accuracy (relative to the faculty scoring).

Based on this study, the lowest-performing students were more likely to overestimate their performance. One implication of this is that these students may not have accurate self-assessment skills, and that this lack of self-assessment may be playing a part in overall low performance. Helping students develop appropriate self-assessment skills early may help them in judging their level of mastery of material and skills, and could be helpful in guiding their educational efforts. "The challenge now is to determine appropriate ways to assist those students who are caught in the paradox of not knowing, and not knowing what they do not know."

(Comment: With the push toward individualized learning plans and practice improvement, this study is particularly timely. I am particularly struck by the dilemma of poor self-assessment skills of low performing students, and the clear need to develop better teaching methods to improve these skills. In addition, 360 evaluations often include peer assessments. This study also provides additional insight into the reliability of this form of assessment. - Leslie Fall)