Section H

Curriculum Evaluation

Raul Rudoy, M.D., M.P.H.
Joe Lopreiato, M.D.

Preface:

The primary objective of curriculum evaluation is the overall improvement of the medical student’s education.Traditionally, curriculum evaluation has been limited to the appraisal of medical student performance.We suggest that other areas of the curriculum should be evaluated to provide Clerkship Directors with feedback in areas such as communication skills, clinical problem solving, students’ career choice and the influence of the curriculum in guiding the student in the evaluation and resolution of social and ethical issues.

Quantitative methods such as attitude scales and National Licensing Exams, and qualitative methods such as interviews with faculty and direct observation of clinical performance can be utilized in gathering the necessary data to determine the curriculum adequacy.Ideally, a combination of both methods should be used: quantitative methods to minimize error, ensure good sampling and control variables inherent to the particular training program; qualitative methods to ensure that the fluid, dynamic, and real world nature of the instructional process is taken into account.

The development of the final curriculum evaluation tool depends in great part upon the direction that the different clerkship directors will give to the curriculum.As an example, schools that utilize the PBL method (Problem-Based Learning) will benefit by using different measuring techniques than those used by schools that utilize a more traditional system of instruction.In spite of these differences, there are certain evaluation principles that apply to all types of curriculum delivery, such as those that make sure that the student has been engaged in the experience, and activities identified as critical for their development.

We suggest that the following areas for curriculum evaluation be explored, based on the premise that the purpose of the pediatric curriculum is to permit medical students to acquire both cognitive knowledge and professional skills adequate for the student to function at a third year level.It is also assumed that the curriculum does not simply provide knowledge, but encourages the practical application of that knowledge.Accordingly, we strongly suggest that the evaluation of the quality of the pediatric curriculum be measured not only from the student’s point of view but also from the point of view of curriculum quality and implementation.

    1. Evaluation of Student’s Performance and Outcome as Influenced by the Curriculum

      • Knowledge Acquisition – Some form of testing for knowledge acquisition is done by most medical schools, but the form of test utilized varies depending upon the type of instruction philosophy utilized by that school.We believe that a brief description of the most commonly used methods will benefit clerkship directors that are in the stages of developing or modifying their current methods for curriculum evaluation.
      • Objective Testing – Objective testing and methods are discussed in detail in Section L.This is probably the easiest of all the data to collect and almost every pediatric program utilizes some form of objective evaluation.The multiple-choice examination permits internal comparison between different blocks and from year to year, and Step II of the USMLE provides clerkship directors with a general idea of how well their programs compare with other programs in the USA and Canada.
      • Subjective Testing – (See Sections M and N) These tests require testers with considerable experience in scoring these type of tests or with expertise in the subject tested. The greatest advantage of interpretative testing is that they test the student’s holistic performance by determining the student’s ability to apply factual knowledge to a clinical situation.The disadvantages are that they are difficult to develop and standardize, are difficult to score, require scorers with a high degree of expertise and can be subject to scorer’s bias.The latter can be partially obviated by having more than one examiner score the test and by the use of standardized patients and check lists.

What conclusions can be drawn regarding the usefulness of these two different approaches to test for knowledge acquisition?First of all, both methods are important but it should be recognized that they test for different aspects of knowledge acquisition.The choice of one over the other is mostly dictated by the type of teaching philosophy and by the availability and commitment of clinical faculty.Objective evaluations which are easy to develop and score are limited to test only for factual, recall knowledge.On the other hand, subjective evaluations, which test students by presenting them with situations which are close to real life conditions can be difficult to score and can be labor intensive.The choice of one over the other is not easy and to suggest only one of them will be an exercise in futility.This decision will be easier when all pediatric clerkships adopt the same or similar teaching method.Until then, clerkship directors should choose the method most appropriate for testing students in their program but strong consideration should be given to utilize a combined objective-subjective exam such as a Multiple Choice Test with Reasons Given, which includes open-ended thinking with objective testing and an OSCE or a standardized patient exam.

Regardless of the method used, the information obtained from the student’s examination should be analyzed in a way that will permit modification of the curriculum in order to cover or expand areas in which students demonstrate deficiencies.

Career Differentiation – A well-designed curriculum should provide student contact with a full range of faculty in the pediatric field and will permit exposure of students to pediatric role models that show career satisfaction and professional self-esteem.

The adequacy of the curriculum in influencing students to become pediatricians can be evaluated by observing the results of the National Residents Matching Program or, less preferably, by a survey conducted at the end of the clerkship.Approximately 10% of medical students in USA and Canadian schools will pursue a career in pediatrics.Those clerkships with values less than the national average may want to take a second look at their students’ experience during the pediatric clerkship.

Attitude Toward Social Responsibility in Medicine – A recent national review of medical care has suggested the need to evaluate medical student characteristics, other than academic achievement.These characteristics are associated with better patient management, and include such issues as psychosocial aspects of patient care and the effects of the physician’s actions in the general welfare of the community.Ideally, the curriculum should address the above points by permitting the student to recognize the importance and complexity of social issues in medicine and to permit the student to identify and reinforce the central role of the physician in controlling and coordinating the cost of medical care.

Evaluation of this aspect of the curriculum can be accomplished by utilizing already available questionnaires such as the ATSIM scale developed by Parlow1or an individual questionnaire developed as part of their own evaluation tools.It is suggested, for example, that a final exam assessment could include a written clinical case problem which asks students to develop a strategy to contain health costs involving health team providers other than physicians.The above-mentioned example may provide clerkship directors with the needed data to evaluate the student’s perception of psychosocial, economic, as well as biological issues.

Other Sources of Information

Many sources exist from which to begin an evaluation of curriculum.They include:

      • MCAT scores
      • Demographic class data
      • Grade distributions of standardized tests
      • Previous accrediting body reports
      • Comparative data on your school from the AAMC
      • Alumni surveys
    2. Evaluation of Curriculum Quality

When evaluating courses of instruction, most medical educators focus on three specific areas: Program, Process, and Participants.Program evaluation involves a critical look at the content, goals, objectives, and evaluation methods of a course.The usual tool utilized is a questionnaire completed by the students at the end of their rotation in which different aspects of instruction and student experiences are evaluated.The results of this questionnaire are often the only evaluation utilized to make changes in curriculum content and in curriculum implementation.The subjective nature of this questionnaire cannot be overemphasized; we are the ones who develop the questionnaire, ask the questions that we believe are important and those questions are answered by students with not much experience or information as to what a physician’s needs are.This is however, only the tip of the iceberg.Serious evaluators will also look at the content of their curriculum.Questions that should be asked include: 1. What are the overall goals of the curriculum?2. What are the objectives and how do students reach them? 3.Are stated objectives relevant to the learner in the real world? 4. What is the quality of teaching? 5. Has the test bank of questions been reviewed for content and compatibility with what is being taught?6. How effective is the admissions policy in attracting good learners?

Process evaluation refers to the analysis of the way the program is implemented. Questions to ask here include: 1. What characteristics of the learner are stressed: knowledge, problem-solving, self-learning, cost consciousness, or cultural sensitivity? 2. Do students receive feedback on their performance, and when?3. What is the quality of the teaching and how is it measured? 4. What is the quality of the textbook(s) used?5. How much does the faculty become involved in decision making?6. Is the focus of the curriculum knowledge, skills, or attitudes?An often overlooked aspect of process evaluation is whether or not faculty teaching time is considered valuable and rewarded.Standardized questionnaires are available that will, in general, look at aspects of curriculum implementation and the learning environment2.Modifications of the above source with inclusion of particular details from the individual clerkship can provide similar but much less biased data than that obtained by the previous example.

Participant evaluation includes an analysis of the attitudes and performance of students and faculty. Questions to ask are: How satisfied are participants with the curriculum?What is the performance (knowledge, skill acquisition and attitudes) of students who finish the course?What are the faculty views on teaching and how do they see themselves as teachers (facilitator, lecturer, mentor)?What is the amount of faculty time devoted to teaching?Measurement of the outcomes of graduates is also a part of participant evaluation.A review of all graduates’ career choices, level of preparedness of interns in pediatrics, recent drop-out rates, certification and re-certification results, practice types and locations, and practice surveys can all be used to measure outcomes of the curricular plan.

Another important source forevaluation can be found in the faculty from departments other than Pediatrics.Faculty acting as peer reviewers can provide a more objective information of curriculum adequacy and curriculum delivery than those obtained from questionnaires filled out by students at the end of the clerkship3.The feedback provided should include a comparison of curriculum at all levels with suggestions for improvement and changes.It is suggested that a checklist be developed which should include at a minimum, questions related to the learning environment and the students’ approach to learning as mandated by the curriculum.

Summary of Curricular Evaluation

  • Program evaluation tools
    • Review of goals and objectives for relevance
    • Review of teaching quality (direct observation and questionnaire)
    • Student admission data
    • AAMC surveys
    • Accreditation reports
    • Alumni surveys
    • Curriculum mapping
  • Process evaluation tools
    • Questionnaires to assess attitudes
    • Direct observations of the learning environment
    • Interviews with faculty and students
    • Debriefing sessions with students at end of course
    • Clinical logs of patient encounters
    • Review of test questions for validity and reliability
  • Participant evaluation tools
    • Objective and subjective testing of students
    • Grade distributions
    • Feedback sessions for students and faculty
    • Peer evaluation
    • Career differentiation
    • Outcome studies
    • Attitudes toward social responsibility

REFERENCES

  1. Parlow J, Rothman A.Attitude toward social issues in medicine of five health science faculties.Soc Sci Med 3:351-358. 1974.
  2. Irby DM, Peer review of teaching in medicine. J Med Educ. 73:459-461. 1983.
  3. Mitchell R.The Development of the Cognitive Behavior Survey to Assess Medical Student Learning. Brown University. 1992.

Additional References

Page G, ed. Essays on curriculum development and evaluation in medicine. Report of the second Cambridge conference June 21-28, 1986. Vancouver, BC. University British Columbia. 1989.

Friedman CP.Charting the winds of change: Evaluating innovative medical curricula.Acad Med.65:8-14. 1990.

Gjjerde CL.Curriculum mapping: Objective, instruction and evaluation.J. Med. Educ. 56:316-323. 1981.

Friedman CP.Improving the curriculum through continuous evaluation.Acad. Med. 66:257-258. 1991.

Coles CR.Curriculum evaluation in medical and health care education. Med. Educ. 19:405-422. 1985.

Whalen JP,Cerchio G, Muslin H.Quality assurance for a medical school curriculum.Teach. Learn. Med. 2:42-45. 1990.

Miller G.The assessment of clinical skills, competence and performance.Acad. Med. 65:563-567. 1990.