A cautionary tale
Do assessor comments on a multi-source feedback instrument provide learner-centred feedback? Vivekananda-Schmidt, P et al. Medical Education 2013; 47: 1080-1088.
Reviewed by Steve Tinguely
What is the study question?
Researchers in England questioned whether free text comments provided by assessors to support the numerical rating used in a multi-source feedback instrument are useful in changing behavior and in facilitating personal development.
How was the study done?
Comments from a large UK wide pilot study evaluating the utility of a generic multi-source feedback (MSF) tool were analyzed qualitatively. Assessors used a single free text box to give examples of observable behaviors to illustrate their numeric answers. Assessees provided self-ratings as well as comments on their own performance. The authors studied the focus of the comments, the concordance between assessors and assessees, and the usefulness of the assessor's comments in supporting the assessee's personal development.
What were the results?
977 UK physicians from a broad range of specialties were assessed by physician colleagues (17%), doctors in training (11%), nurses (23%), allied health professional (14%), administrative staff (25%) and medical students (1%). Forty-two percent of the 11,483 assessors included free text comments. Only 56% of forms that included "below average" ratings included comments. Most assessor comments focused on the effect of the assessee's work on them rather than ways in which the assessee could improve. 1806 forms had free text comments from both the assessor and assessee; the majority demonstrated clear differences of opinion or interpretation. Free text comments from assessors rarely contained enough detail to be helpful to those they were assessing.
What are the implications of these findings?
Although comments are generally required on MSF tools, this study suggests that those comments are unlikely to provide information that will facilitate real change, supporting previous work that questions the utility of free text comments and points to the need for training on how to effectively provide meaningful multi-source feedback.
Editor's note: This cautionary tale might make us think again about all of the ways that rely on written forms, and whether they are really doing the job that we expect of them. Some tasks, like facilitating change, are likely better accomplished with face-to-face feedback and discussion (LL).