Linda Tewksbury, MD, Pediatrics, NYU, New York, New York, Eve R. Colson, MD, Pediatrics, Yale University, New Haven, CT, Glenn Flores, MD, Pediatrics, University of Texas, Southwestern Children's Medical Center, Dallas, Texas, Janice Hanson, PhD, Department of Medicine, Uniformed Services University of Health Sciences, Bethesda, MD, Benard Dryer, MD, Pediatrics, NYU, New York, New York
Background: Use of qualitative research (QR) is becoming more common in medical education. QR abstracts are routinely submitted for consideration at the Pediatric Academic Societies’ (PAS) Meeting where they are reviewed and scored with quantitative research abstracts by reviewers with unknown expertise in qualitative research.
Objective: To determine: 1) how frequently QR abstracts are submitted to PAS 2) whether QR abstracts are more or less likely to be chosen for platform presentation or designated publish-only and 3) whether reviewers with QR expertise score QR abstracts differently than the general pool of PAS reviewers.
Design/Methods: The database of abstracts submitted to the 2010 PAS Meeting was searched using qualitative terms. Authors identified exclusively QR abstracts. Mean scores and disposition of QR vs all other abstracts were calculated. Student’s t-test was used to compare mean score differences, and ANOVA and pairwise comparisons for differences in disposition. Five researchers with expertise in QR methodologies, blinded to ratings and disposition by PAS reviewers, independently reviewed QR abstracts following standard PAS guidelines, scoring abstracts from 1 (highest quality) to 7 (lowest quality). Comparison of mean scores of QR abstracts by PAS vs. QR expert reviewers was analyzed using paired t-test. Agreement between QR and PAS reviewers was measured using interclass correlation coefficients (ICC).
Results: Of 4027 abstracts, 65 (1.6%) used exclusively QR methods. The disposition of QR abstracts among platform, poster, and publish-only significantly differed from all other abstracts (p=.02). Compared with all others, QR abstracts had lower odds of platform presentation (OR, 0.3; 95% CI, 0.1,0.8), showed a non-significant trend towards greater odds of publish-only (OR, 1.6; 95% CI, 0.97,2.8), and were equally likely to be posters (OR=1.2; 95%CI, 0.7,1.9). Mean scores for QR abstracts by PAS reviewers were significantly higher (lower quality) as compared to all others (4.26 vs. 3.86,p<.001). Mean QR abstract scores by QR reviewers were even higher (of lower quality) than those of PAS reviewers (4.48 vs. 4.26,p=.02). QR and PAS reviewers' mean scores had fair agreement (ICC=0.30, 95%CI 0.10,0.50).
Conclusions: Less than 2% of all abstracts submitted to the 2010 PAS meeting used exclusively qualitative research methods. Compared with all others, QR abstracts were three times less likely to be chosen for platform presentations, and showed a non-significant trend towards greater odds of publish-only. The quality of QR abstracts was rated lower than all other abstracts by PAS reviewers and even lower by reviewers with QR expertise, suggesting that lower quality ratings are more likely due to methodologic flaws or inadequate presentation of the research as opposed to bias related to lack of rater QR expertise. Further research should examine whether faculty development on QR methodological quality for researchers and reviewers would improve the quality of QR abstracts.