Council on Medical Student Education in Pediatrics


Search This Site

Journal Club

Global rating scales beat checklists!
Diagnosing Technical Competence in Six Bedside Procedures: Comparing Checklists and a Global Rating Scale in the Assessment of Resident Performance. Walzak A et al. Academic Medicine 2015;90:1100-1108.

Reviewed by Rebecca Tenney-Soeiro  

What was the study question?

How do procedure-specific checklists compare to a global rating scale (GRS) in assessing technical competence? 

How was the study done?

Two trained raters observed videos of trainees completing an OSCE consisting of six bedside procedures.   Raters evaluated each video recorded performance using both a procedure-specific checklist and a GRS.  They were blinded to the minimum passing scores on the checklists, and made pass/fail decisions for each performance based on the global assessment rating.  The GRS included eight items assessing technical competence and includes a summary item on “overall ability to perform procedure.”  The six procedure-specific checklists were created based on review of the literature and task analysis and evaluated by a number of experts. 

What were the results?

42% of the 218 performances were deemed competent based on the global assessment item on the GRS. The GRS demonstrated higher internal reliability than the procedure-specific checklist for each procedure.  Interrater reliability was higher for the checklists than the GRS, however interrater reliability for decisions on each performance’s competence using the overall global assessment rating was almost perfect.  In general, minimum passing levels on the checklists demonstrated high sensitivity but poor specificity in the diagnosis of competence.  

What are the implications of these findings?

Although checklists are easy to use and are presumably more objective than GRS, mounting evidence raises concerns about using checklists for assessment purposes.  Experts in standard setting should be familiar with purposes of assessment, we should revise and reconsider the type of items that are included in the assessment tools, and we should consider including a GRS in the assessment of procedural skills. 

Editor’s note:  Once again, GRS beat checklists!  The GRS in this study had 7 items plus one overall rating so at first glance almost looks like a checklist (SLB).

Return to Journal Club