Council on Medical Student Education in Pediatrics


Search This Site

Journal Club

The word is the assessment, and the word is good

When assessment data are words: Validity evidence for qualitative educational assessments. Cook DA, Kuper A, Hatala R, Ginsburg S. Academic Medicine 2016 (published online ahead of print).

Reviewed by Gary Beck Dallaghan

What was the study question?
The purpose of this article included the following: articulating the role of qualitative assessment in a more comprehensive assessment program; applying concepts and language of validity to qualitative findings; and using examples to highlight the qualitative approach.

How was the innovation conducted?
The authors described different forms of validity evidence, citing frameworks developed by Messick (Content, Response Process, Internal Structure, Relations to Other Variables, and Consequences) and Kane (Scoring, Generalization, Extrapolation, and Implications). They stipulated that qualitative assessment is strictly raw narrative data provided either by student self-reflection or narrative comments from preceptors about student performance.

What were the outcomes? Using response process as an example, when an assessor is analyzing the content of an education portfolio the assessor’s reflexivity (such as their relationship with learners and the context) needs to be considered. An example of generalization involves using rigorous methods to purposefully sample and collect data (ideally until saturation has been achieved) and use of triangulation. The authors spelled out examples for all of the other components of the two frameworks. Using the education portfolio as an example, the authors then described how qualitative analysis can be applied using Messick’s and Kane’s frameworks.

What are the implications of these findings?
This approach offers alternatives to strictly focusing on quantitative data as a means of assigning grades or making decisions about competence. The authors emphasized the need to collect both forms of data as a means of creating a more comprehensive characterization of student performance. They did note the practical implications of using qualitative analysis, including the time and skills needed to conduct qualitative analyses and variability in the quality of the narrative comments from preceptors (“keep reading” is not very descriptive). As with any approach to evaluation, training of the assessors is important.

Editor’s Note: As more medical schools move to competency-based assessment, qualitative assessments will very likely be needed for promotion and retention decisions. This article provides a good framework for ensuring that qualitative assessments are valid (JG)

Return to Journal Club