D.R.E.A.M.ING ABOUT VALIDITY IN EDUCATIONAL ASSESSMENT
Mary E M. Rocha,Baylor College of Medicine,Houston,TX
Rationale: Educators are expected to evaluate individuals and programs using valid methods, yet some lack training in the psychometric properties of assessment tools. The goal of this workshop is to prepare participants to understand and apply validity evidence principles by performing a critical analysis of assessment instruments through methods outlined for authors by MedEdPORTAL's new Directory & Repository of Educational Assessment Measures (DREAM). DREAM is a 'user-friendly, one-stop shop repository for assessments' including a summary of the validity evidence associated with their use. Participants will immediately apply validity concepts by critically appraising a published instrument that evaluates clinical teaching. Participants will hone their skills in gathering and critiquing validity evidence, which will prepare them for future DREAM authorship or for validating any assessment instrument. A basic understanding of construct validity principles will maximize participant success in this workshop, but is not essential.
Objectives: 1. Illustrate the construct validity domains of content, response process, internal structure , relationship to other variables and consequences with examples 2. Analyze the evidence for a 'validated' educational measure using DREAM critical analysis methods 3. Practice a step-by-step strategy for critical analysis of educational assessment measures
Methods and Content: Following a brief introduction and needs assessment, the workshop starts with an online introduction to Directory & Repository of Educational Assessment Measures (DREAM) followed by a review of construct validity (content and response process domains). Illustrations from tools published in DREAM will clarify these concepts. Next, in facilitated small groups, participants apply these concepts to the Maastricht Clinical Teaching Questionnaire (MCTQ) using a provided critical analysis worksheet. A second brief review of construct validity (internal structure [reliability] and relations to other variables domains) is presented. Again, illustrations from published DREAM tools will help clarify these concepts. Next, in facilitated small groups, participants apply the concepts to the MCTQ using a provided critical analysis worksheet. A third brief review of construct validity (consequences) is presented. Again, illustrations from DREAM content will clarify these concepts. Next, in facilitated small groups, participants apply the concepts to the MCTQ using a provided critical analysis worksheet. Finally, a summative, step-by-step approach (including examples) to planning and performing a DREAM critical analysis will be reviewed and discussed with questions and answers.