Robyn Bockrath,Northwestern University,Chicago,IL; Sanaz B. Devlin,Eastern Virginia Medical School,,; Kamakshya P. Patra,WVU Medicine Children's Hospital,,; Jean A. Petershack,UTHealth San Antonio,,; Anton M. Alerte,University of Connecticut,,; Jennifer Trainor,Northwestern University,,; Janice L. Hanson,University of Colorado School of Medicine,,; Cynthia Christy,University of Rochester,,; Amal M. Khidir,Weill Cornell Medicine in Qatar,,; Joseph Gigante,Vanderbilt,,; Eve Colson,Yale School of Medicine,,


Scholarly criteria for publication include significant outcomes, supported by appropriate study design that contribute to the field. Medical educators often strive for scholarship in the development of educational innovations, but struggle to develop meaningful, publishable outcomes. The COMSEP Research and Scholarship Task Force will offer participants the opportunity to enhance their educational innovations and scholarship through the development of meaningful outcomes.


By the end of the workshop, participants will be able to:

  • Define Kirkpatrick, Modified Kirkpatrick and Miller Models of outcomes
  • Recognize and categorize the hierarchy of outcomes in example studies and learners’ own educational innovations
  • Evaluate the hierarchy of outcomes and apply principles of study design, including SMART objectives and assessment tools, to construct outcomes beyond “satisfaction” in example, peer and learner’s own educational innovations

Methods and Content:

After a brief introduction, participants will be introduced by didactic to multiple outcome hierarchies, including Kirkpatrick, Modified Kirkpatrick and Miller models. Identification of different hierarchy of outcome examples will be reinforced by an audience participation quiz. A brief didactic on study design to support meaningful outcomes will focus on linking outcomes to SMART objectives and multiple assessment strategies, including the ACGME toolbox, quantitative and qualitative strategies. To promote engagement during didactics, participants will utilize a worksheet to prompt documentation, reflection, and enhancement of their personal educational innovations. Next, faculty will facilitate small groups to use the worksheet to evaluate the hierarchy of outcomes in a satisfaction level example study and re-develop the study into meaningful outcomes, then share their re-developed study with the large group. Finally, participants will have the opportunity to share their own educational innovations in groups of 2-3 and re-develop their outcomes with the guidance of peers and leaders.


Time Line





Introductions, Purpose, Objectives



Outcomes Hierarchy

  • Kirkpatrick, Modified Kirkpatrick, Miller
  • Examples



Audience participation quiz



Study Design

  • Linking Outcomes to SMART Objectives 
  • Assessment Strategies

Didactic with worksheet


Developing Outcomes: Example Studies

  • Identification of outcome hierarchy in example studies
  • Develop satisfaction level outcomes into higher levels



Small group work with worksheet and leader.

Large group share of re-developed study.




Developing Outcomes: Personal Educational Innovations

  • Identify current level of outcome in learners’ innovations
  • Develop learners’ educational innovations into meaningful outcomes


Individual work and small group share guided by worksheet, peers and leader.