Council on Medical Student Education in Pediatrics


Search This Site

Journal Club

But do I get partial credit…?

Justify Your Answer: The Role of Written Think Aloud in Script Concordance Testing

Power A, , Jean-Francois Lemay JF and Suzette Cooke S.

Teaching and Learning in Medicine, September 2016.

Reviewed by Parul Bhatia

Tags:  GME, Pediatrics, Assessment-SCT, Qualitative

What was the study question?

Does adding the qualitative data obtained from written “Think Aloud” (TA) enrich the quantitative data obtained from script concordance testing (SCT)?

How was the study done?

Ninety one pediatric residents and 21 experienced pediatric faculty from 4 Canadian training centers completed an online test consisting of 24 case-based scenarios and associated questions, reflecting clinical reasoning and decision making. This portion made up the SCT in that the clinical decisions of the learner was compared to that of the faculty expert. For 6 of these cases, participants completed a TA written exercise where they explained in free text the justification behind their decision making.  Half the participants completed the TA for 3 cases of these 6 cases, and the other half completed the TA for the remaining 3 cases. Investigators used inductive thematic analysis for the TA responses.

What were the outcomes?

Three main benefits of adding TA to SCT were identified: (1) uncovering poor clinical judgment in the learner despite correct SCT response, (2) discovering correct clinical judgment in the learner despite incorrect SCT response, and (3) uncovering question misinterpretation.

What are the implications?

SCT quantifies the degree of concordance between the learner and the expert. While SCT gives us a quantitative data, it can be skewed by guessing and chance. With the addition of TA, whether verbal or written, the learner has the opportunity to explain their decision, which adds important insight into their true clinical reasoning. This is very reminiscent of high school, where students cannot just write the correct answer to the complex math problem, but must “show their work” to get full credit. While this takes more time to grade, it adds significant depth to the assessment tool.

Editor’s Note: Of course this same approach could be used with the ubiquitous multiple-choice tests as well.  As a colleague of mine said, (paraphrasing) “Give me 5 minutes in a room where I can ask a student questions and I’ll know whether they know what they’re talking about.” (JG)

Return to Journal Club