Journal Club

Turner, MK, Simon, SR, Facemyer, KC, Newhall, LM, Veach, TL, "Web-Based Learning Versus Standardized Patients for Teaching Clinical Diagnosis: A Randomized, Controlled, Crossover Trial." Teaching and Learning in Medicine. 2006:18(3), 208-215.

Reviewed by David Levine, MD; Morehouse School of Medicine

There is a paucity of studies looking at how students develop clinical skills and what are the best methods for assisting students in their development. This study is an attempt to compare two methods that are growing in utilization - standardized patients sessions (SP in the study) and Internet-based Web-based cases (WB in the study). The authors pose the following questions: "Which format demonstrates a greater ability to improve clinical reasoning? How robust is this influence? How do the costs of each method compare? Also, how do students respond to them?"

The study was done to begin to investigate optimal methods for learning and retention of clinical skills. They were looking at best methodologies as well as the cost and acceptance of each method, to assist schools and programs in educational planning. The planning is, of course, stimulated by the requirement for students to pass USMLE Step 2 CS in order to eventually obtain licensure. What are the best, most acceptable and cost-effective methods to stimulate the development of clinical skills?

Study group was the 54 2^nd year students at the University Of Nevada School Of Medicine. An interesting problem occurred in the study. They had originally applied for and obtained IRB exemption. However, a student in the class contacted the IRB and they re-evaluated the protocol and recommended a consent form be added. This turmoil and changing rules explained why of the 54 students eligible, only 30 actually consented to have their information released and available for data analysis, unfortunately weakening the study power. The students who consented and who did not consent did not differ statistically, but the Journal did not allow them to publish this data.

The study was a single blind (to the investigator) randomized cross-over study. The 30 students who continued were randomized to 16 in the WB group and 14 in the SP group. They received similar cases via these methodologies. The first case was abdominal pain and the second was headache. The SP group had a case about abdominal pain and the WB group used the DxR case on abdominal pain. Then for the second case, the groups switched so that the 16 in the WB group for the first case were then in the SP group for the headache case. (DxR cases have been around for several years and while novel and interesting to use, I personally was never very impressed by the accuracy of the method used to generate diagnostic hypotheses. The product was excellent for its time, but the cases took a very long time, were very expensive (I paid 900 for two cases about 5 years ago), and technically are not web-based cases, they are CD-ROM cases!). The authors did document that they worked hard to get the SP case to go along the same lines as the WB/DxR cases. The evaluation of the study was threefold.

1. Performance on an SP exam 4 weeks after the intervention - two cases on abdominal pain and headache, with variations to reduce perceived redundancy. They had a checklist that and a SOAP note in similar (though not identical) form as the USMLE Step 2 CS.

2. Student evaluation of effectiveness using a 5 point Likert scale of seven issues related to satisfaction

3. Analysis of start up and continuation costs between the two methodologies

Findings

• WB and SP resulted in similar scores on the Abdominal Pain checklist and the Headache checklist. WB training produced a higher score on the Abdominal Pain SOAP note (P=0.006), but no difference on the Headache SOAP note. This was an unexpected finding to me. I expected that students who were taught using SPs would test better using SPs, but at least for the Headache SOAP note this was the opposite.

• Students on 7/7 items scored the SP experience as significantly better than the WB experience. These numbers were actually quite significant given the small number of students who ended up participating. Also, as a member of the CLIPP Editorial Board, I have seen the typical evaluations by students on a Likert scale. Most students rate the experiences highly, almost in an inflated fashion and the difference between a case that works well and one that doesn't is actually quite small, and this is on thousands of case sessions. These differences were significant at the p<0.001 level for all but one at the p=0.004 level. A typical difference on the 5 point Likert scale was 3.94 on the SP case and 2.57 on the WB case.

• Start up for the SP cases was $2190, for the WB teaching was $2250; ongoing costs were $45 per case per student for the SP cases and $30 per case per student for the WB cases. (Remember, in contrast, CLIPP, due to its initial grant funding and created structure costs at present $50 per student for 31 cases and this number should decrease with increased utilization).

Discussion and Limitations

• Authors conjecture that the subject matter of abdominal pain might be better suited to web-based teaching than headache as an explanation of the sole difference noted.

• Authors mention prior work that showed conflicted and/or confusing results when comparing paper and WB cases, but the prior work clearly showed equivalent learning outcomes with less study time using WB cases.

• Authors think the superiority of the SP method on student evaluation part is due to preceptorship with faculty directly after the SP case, with instant feedback, based on qualitative comments of the students.

• Start up costs similar but ongoing costs are significantly less expensive for the WB cases; however, the cost mentioned in the study is still quite excessive compared to our CLIPP model.

• Authors acknowledged their difficulties in getting consent from the students after the study began and acknowledged their numbers were small and therefore the power of the study suffered.

• Students were not blinded to the intervention, used only two cases with limited problems, and at one medical school, so they concede the results may not be generalizable.

• "Durability" of the acquired information or skills in time (will the information stick?) is not examined - evaluation was 8 weeks after the abdominal pain session and 4 weeks after the headache session. However, perhaps the better performance on WB cases on abdominal pain might then be more durable in time, but this is conjecture only.

(Editorial comment: The issues seem easy to identify, but tough to fix. It's well demonstrated that the most economical way to "teach" is to give a lecture to a large group of students. It's also pretty well established that the optimal way to "learn" involves small group or even one-on-one encounters. Web-based tools allow for small groups, but technology has not yet evolved to allow truly tailored feedback (it's not as if the technology isn't there; when you do a Google search, they can provide targeted ads as part of the results; it's one of the ways Google can make a profit). That the SP cases were better received is not surprising. Long-term results are critical: does the sizable up front investment for SP's lead to a better performing physician? Bruce Morgenstern)