Journal Club

Adler, Me, Trainor, JL, Sidall VJ and McGaghie WC.Development and Evaluation of High-fidelity Simulation Case Scenarios for Pediatric Resident Education. Ambulatory Pediatrics 2007;7:182-186.

Reviewed by: Margaret Golden; SUNY Downstate

Many schools are looking at patient simulators as adjuncts to live patient encounters for teaching and assessing clinical skills. Some research suggests that repeated practice on a simulator may be superior to learning from actual clinical encounters (Friedrich MJ "Practice Makes Perfect: Risk-free Medical Training with Patient Simulators. JAMA 2002;288:2808-2812.) On the other hand, some medical educators are uncomfortable that we are moving farther and farther away from patients. Hence rigorous assessment of the strengths and weaknesses of simulations is crucial to guide how we adopt this new tool.

Drs. Adler, Trainor et al report on the process of developing and validating "high-fidelity simulation scenarios" for 4 rarely encountered but time critical pediatric management problems: apnea, asthma, SVT, and sepsis.

One cohort of pediatric residents (n=51) at Children's Memorial Hospital in Chicago were used for field testing the case scenarios, which were run on a METI PediaSIM mannequin. A second but overlapping cohort (n=54) was used to measure the validity and reliability of the simulation exercise as a tool for assessing resident competence.

The paper gives fairly detailed accounts of the simulation exercises and the development process, which are worth reading. The developers spent >100 hours apiece reviewing and revising both the scenarios and the scoring check list.

For the purposes of evaluating resident performance, the encounters were videotaped, and each encounter was reviewed and scored by three of the authors. Each of the 54 resident participated in two simulations, for a total of 111 encounters (I couldn't get this math to work). The Kappa coefficients by case ranged from 0.75-0.87, which indicates quite respectable inter-rater reliability. As for validity, the mean score of second year residents was significantly higher than that of first year residents, except for the Sepsis case, in which the first years performed quite well (the authors attribute this to the patient mix of their program.) Previous experience with a simulation also predicted a higher score, and most residents scored higher on the second case than on the first. However, residents later in a given year of training did not score substantially higher than those earlier in that year, which does raise some concerns about how well this exercise translates into actual clinical skill.

What does this study add to our knowledge of training on simulators? It does not answer the fundamental question: does prior training on a simulator improve the learner's ability to react appropriately in a crisis with a real patient? To answer such a question with rigorous research methodology seems utopian—and perhaps not necessary, given the enormous face validity of using simulators to practice for rare, critical events. But I think it is important to ask a related question: does training on a simulator introduce clinically significant distortions in a trainee's response to a live patient? The authors do not report any follow-up on the clinical performance of their residents—but perhaps that will be the subject of a later study.

We need to recall Miller's pyramid here. The paper demonstrates a way to make a very big jump, from the "knows/knows how levels" of performance to the "shows how level." As Margaret Golden accurately points out, the next leap is to the top of the pyramid: "does." The other question, even with their careful process to develop the scenario relates to generalizability. Would that all episodes of sepsis, asthma, SVT and apnea the same. If a resident can do well the way the case has been constructed, can they also do well with other presentations of the same conditions? Bruce Morgenstern