Readdressing the Need for Consensus in Preclinical Education

MILITARY MEDICINE, Vol. 174, October 2009 1081

MILITARY MEDICINE, 174, 10:1081, 2009

INTRODUCTION Teaching basic clinical skills is a time-honored, essential component of the medical school curriculum. Despite an increasing emphasis on the importance of developing these skills in the fi rst and second years of medical school, there is still little evidence of dialogue among national medical educators regarding the development of standards for a basic clinical skills curriculum. 1 Medical schools often use courses commonly referred to as “Introduction to Clinical Medicine,” or ICM, to teach basic clinical skills to medical students. However, previous reports suggest that there are wide varia-tions in not only the curriculum itself, but what the objectives of the course should be, who does the teaching, when and how students are exposed to material, and how to measure these objectives. 1–5

In a previous study, clerkship directors (CDs) from inter-nal medicine, family medicine, pediatrics, surgery, obstet-rics/gynecology, and psychiatry identifi ed core competencies, which were closely aligned with the Accreditation Council for Graduate Medical Education (ACGME) core competencies, and prioritized their importance for second year students to achieve before third year clerkships. 6 Despite this clear iden-

tifi cation of priorities, clerkship directors in the above study indicated that 30–50% of students entering their third year clerkships were not adequately prepared in one or more of these competencies. 6 Furthermore, clerkship directors responding to a previous Clerkship Directors in Internal Medicine (CDIM) survey indicated a wide range of expectations for graduating second year students, with no real consensus on what clini-cal skills or competencies second year students should possess entering their clerkships. 1,2 To date, medical educators have been unable to translate their broad agreement on the impor-tance of communication, examination, and interviewing skills in third year clerks into a consensus on specifi c educational goals and objective assessments for second year medical stu-dents. We sought to understand how preclinical courses are attempting to achieve these goals and objectives and whether there still exists a desire to develop a consensus on them.

ICM courses utilize a wide range of assessments from multiple choice examinations and graded objective struc-tured clinical examinations (OSCE) to global assessments by teachers and nonphysician evaluators. 1 Studies suggest that standardized patients (SPs) and other nonphysicians can effectively teach and assess selected clinical skills to medical students, and there is growing evidence for the utility of SPs in medical education. 7–10 Understanding how medical schools use standardized and real patients to teach preclerkship clini-cal skills through either an ICM course(s), or other similar course(s), is an important aspect of developing consensus for teaching methods in addition to determining the effectiveness of these methods in achieving commonly agreed upon pre-clinical course goals.

We sought to readdress these topics through the 2005 annual CDIM survey, and developed questions specifi cally designed to characterize who teaches and controls the ICM courses, to defi ne the role of real patients and SPs, to charac-terize the types of assessment tools being used, and to deter-mine if the desire for consensus for ICM courses has changed.

Readdressing the Need for Consensus in Preclinical Education

Capt Jeffrey LaRochelle , USAF MC * ; COL William Gilliland , USA MC (Ret.) * ; Dario Torre, MD † ; Elizabeth A. Baker, MD ‡ ; Alex J. Mechaber, MD § ;

Lt Col John Poremba , USAF MC * ; Steven Durning, MD *

ABSTRACT Purpose: Clinical skills are often taught in preclinical courses such as “Introduction to Clinical Medicine” (ICM). We sought to defi ne current national course objectives, roles of standardized and real patients, and methods of assessment in ICM courses, and to readdress the need and desire for establishing consensus. Methods: In April 2005, the Clerkship Directors in Internal Medicine (CDIM) organization conducted its annual, voluntary, and confi dential survey. The survey contained a section (24 questions) on preclinical education teaching methods and evaluations. Results: Wide variations exist in the teaching and assessment methods used by ICM courses in 4-year medical schools. Respondents indicated a desire for a consensus statement concerning ICM course objectives in addition to guidance on case develop-ment, use of standardized patients, and student assessments. Conclusion: Our fi ndings suggest that consensus remains elusive for preclinical education teaching objectives, but would be embraced by ICM course directors.

* Uniformed Service University of the Health Sciences, 4301 Jones Bridge Road, Bethesda, MD 20814.

† Medical College of Wisconsin, 9200 West Wisconsin Avenue, FMLH East Offi ce Building, Suite 4100, Milwaukee, WI 53226.

‡ Rush Medical College of Rush University, Rush-Presbyterian-St. Luke’s Medical Center, 1653 West Congress Parkway, Chicago, IL 60612.

§ University of Miami School of Medicine, P.O. Box 016960 (R-53), Miami, FL 33101.

The opinions expressed in this article are solely those of the authors and do not refl ect the offi cial policies of the Department of Defense, the United States Army, the United States Air Force, or other federal agencies.

The data used in this survey is the property of the Clerkship Directors in Internal Medicine and is used with permission.

This manuscript was received for review in September 2008. The revised manuscript was accepted for publication in June 2009.

Downloaded from publications.amsus.org: AMSUS - Association of Military Surgeons of the U.S. IP: 054.152.109.166 on Oct 28, 2015.

Copyright (c) Association of Military Surgeons of the U.S. All rights reserved.


1082 MILITARY MEDICINE, Vol. 174, October 2009

This work has broad implications for both military and civil-ian educators overseeing clinical skills teaching for trainees.

METHODS In April 2005, CDIM conducted its annual, voluntary, and confi dential survey of its U.S. and Canadian membership. CDIM has 109 institutional members. Institutional mem-bers were asked to complete the survey (typically the Internal Medicine Clerkship Director). Not all medical schools in the United States and Canada have a representative in CDIM.

All survey questions were reviewed and modifi ed by mem-bers of the CDIM Research Committee for content validity. After Research Committee modifi cations, the questions were presented to the CDIM Council and after additional revisions, accepted by the CDIM council for inclusion on the 2005 survey.

The fi nal survey contained demographic questions as well as a section on preclinical education written by Uniformed Services University of the Health Sciences (USU) acade-micians (J.P. and S.D.; see Appendix 1, 24 questions). We focused on the preclinical education section to obtain cur-rent national practice data regarding the structure (e.g., role of institutional member in ICM courses and number of SP vs. real patient encounters); teaching methods (e.g., amount of faculty observations or formative SP encounters), and assess-ment methods (e.g., types of exams or summative SP encoun-ters) in ICM courses. We also asked several questions directly related to the current desire for consensus on course objectives and assessment methods. Each participating school has only one CDIM institutional member. Because we sought to use the institution as the unit of analysis, we analyzed the responses of the institutional member. Importantly, if the institutional member is not a preclinical course director, it is the expecta-tion of CDIM that he/she discuss these items with the pre-clinical course director to gather accurate data regarding the preclinical section of questions at his/her institution. The sur-vey was mailed in April 2005, and nonresponders were con-tacted up to 4 additional times through e-mail, regular mail, and/or telephone contact.

RESULTS The overall survey response rate was 81% (88/109). Demographic results were consistent with survey responses over the past several years. For institutional member respon-dents, mean age was 45 ± 9, and the male-to-female ratio was 1.5:1. Academic rank of institutional member respondents was: instructor (1.2%), assistant professor (33.7%), associate professor (44.2%), and professor (20.9%).

Of the 88 returned surveys, 66% of institutional mem-bers indicated they had direct participation in preclerkship clinical skills courses (ICM) in one or more of the following roles: lecturer (9–17%), small group preceptor (37–45%), or course director (8–9%). The majority of basic clinical skills were taught on campus (76%) with a relatively even distri-

bution of clinical settings between the hospital (53%), clin-ics (47%), and simulation centers (34%). Many of these skills were taught at multiple sites. The vast majority of ICM direc-tors are primary care physicians (87%) with 61% in internal medicine and 26% in family medicine.

Use of Standardized Patients in ICM Seventy-four percent of respondents indicated that they use SPs in some fashion within an ICM course at their institu-tion. The mean number of student encounters with SPs over the fi rst 2 years of medical school was just under 10, and the number of encounters for each student during ICM broadly ranged from none to more than 20 (see Table I ). Respondents indicated that nearly 70% of the encounters with SPs were rated or scored by the SPs, and nearly 60% of the encounters were observed by faculty (see Table II ). Although respondents reported more student encounters with real patients than with SPs during the fi rst 2 years of medical school with a mean of just over 12 encounters, the real patient encounters were less likely to be observed by faculty (see Table II ). Half of the institutional respondents reported using SPs for teaching and feedback (formative assessments), and over a third of the respondents (35.2%) reported using SPs for grading (summa-tive assessments).

Assessment Tools in ICM Nearly half of the respondents indicated either introduc-ing (13.6%) or increasing (30.7%) the utilization of SPs for clinical skills assessment in ICM since the institution of the USMLE Step 2 Clinical Skills Examination, an OSCE, required as 1 of the 3 sequential exams (steps) for granting a state medical license in the United States. Approximately one-third (29.6%) of preclinical courses introduced or increased their use of observed encounters with real patients; likewise, roughly one-third (28.4%) have introduced or increased the use of simulators or synthetic mannequins since the institution of the USMLE Step 2 Clinical Skills Examination. Multiple choice question (MCQ) exams were most commonly used for summative assessment compared to a complete history and physical or multistation SP exams (23.9%, 20.5%, and 15.9%, respectively), whereas the complete history and physi-cal was used most often for formative assessment compared

TABLE I. Range and Type of Student Encounters in ICM from the CDIM 2005 Survey ( n = 88)

None 1–5 5–10 10–20 >20Did Not Respond

Real Patients (%) 6.8 15.9 18.2 17.0 5.6 36.4Standardized

Patients (%)5.7 31.8 17.0 6.8 2.3 36.4

Observed or Videotaped (%)

5.6 36.4 12.5 4.5 3.4 37.5

Rated by Written Materials (%)

4.5 34.0 20.5 2.3 1.1 37.5





to the multistation SP exam or MCQ exam (14.8%, 11.4%, and 5.7%, respectively). In terms of grading for ICM courses, 43.2% of respondents indicated using a pass/fail system and one-third (33%) used letter or numeric grades.

Consensus in ICM Seventy-two percent of respondents indicated that a consensus statement from CDIM on the objectives and student require-ments for an ICM course would be helpful. In addition, most (58%) respondents felt that it would be helpful if CDIM pro-vided guidelines on the use of SPs as well as on case develop-ment in the ICM course. Further, the majority (69%) reported that a consensus statement concerning ICM evaluation would be helpful.

DISCUSSION Teaching basic clinical skills during medical school is an essential part of medical education. Indeed, without compe-tence in basic clinical skills, physicians would not be able to effectively practice medicine in the U.S. or abroad. This is especially true in austere environments that may lack robust imaging and laboratory support that may compensate for clin-ical skills defi cits. In 2005 we surveyed institutional CDIM members to determine the current practice of ICM education, specifi cally addressing the structure, teaching, and evaluation of ICM courses. Given the important role that ICM courses play in providing the foundation of basic physical exam skills for medical students and recent national changes in curric-ulum content and structure, we believe that there is a criti-cal need to re-evaluate the state of preclinical education, and to resume the discussion of consensus among ICM course directors. This sentiment reverberates through the respon-dents to the 2005 CDIM survey, who indicate that medical educators around the country see the need for greater consen-sus and guidance on ICM course goals and objectives, and course curriculum specifi cally related to the use of SPs, case development, and student assessments. Given the signifi cant involvement of clerkship directors in ICM courses, it is sur-prising that preclinical course goals have not become more vertically (between preclinical course directors and clerkship directors) integrated with clerkship goals. Perhaps, establish-ing a consensus on ICM course goals and objectives would provide the opportunity for improved vertical integration.

Although consensus and vertical integration play important roles, the delivery of course goals is dependent upon effective teaching methods. According to the results of the 2005 CDIM survey, all ICM courses across member institutions use a com-

bination of real patients and SPs to teach clinical skills, rep-resenting the majority of formative assessment. Despite this similarity, there is a signifi cant degree of variability reported by survey respondents in the number of encounters students have with real patients and with SPs. Although there is data to suggest that student outcomes are not affected by whether the student has encounters with SPs or real patients, 11 there is little data to indicate an appropriate number of encoun-ters needed to achieve profi ciency in basic clinic skills. As faculty time and educational resources become increasingly stretched by increased responsibilities, operational tempo, and faculty turnover, it will be critical to determine whether student encounters with SPs and/or real patients are effective in teaching basic clinical skills to medical students. Our data on the type and number of encounters ( Table II ) may serve as a starting point to further address this issue. Despite the reported usefulness of real patient encounters in preclinical education, it is interesting to note that respondents reported that real patient encounters were less frequently observed by faculty than SP encounters. Further dialogue and research on this apparent disparity in faculty observations is needed.

Our data suggest that the use of patient encounters, whether real or standardized, has increased nationally, and will likely continue to expand. As an integral part of the dialogue on consensus, medical educators should develop a way to sys-tematically appraise the use of standardized and real patient encounters to improve reliability and validity. Programs already exist, such as the Simulated Internal Medicine Patient Learning Experience (SIMPLE) and the Computer-Assisted Learning in Pediatrics Project (CLIPP) that could be validated, and poten-tially serve as a guideline for other specialties to begin devel-oping critically appraised standardized training cases.

Arguably, success in clerkships is the ultimate outcome mea-sure for an ICM course. Not surprisingly, fi nding a consensus on student assessments in ICM courses was supported by the respondents in our survey and represents another critical step toward vertical and horizontal (among preclinical course direc-tors) integration of clinical skills education. Studies have shown that SPs can give summative and formative assessments in addition to teaching basic clinical skills. 5,9,12–14 However, given the variability of reported assessments used in ICM courses demonstrated in our survey, the use of the SP as an effective evaluator for both formative and summative assessment needs to undergo further critical appraisal. Given that a large per-centage of respondents to the 2005 CDIM survey indicated already using SPs for formative and/or summative evaluations, we believe this area of research should receive a high prior-ity. More emphasis is being placed on the OSCE as a means

TABLE II. Number and Type of Student Encounters Over the First Two Years of Medical School from the CDIM 2005 Survey ( n = 88)

No. of Encounters % Faculty Observed % Rated by SP % Rated by SP and Observed

Standardized Patients 9.44 (0, 40) a 59.92 (0, 100) a 69.24 (0, 100) a 28.55 (0, 100) a Real Patients 12.01 (0, 40) b 34.13 (0, 100) b No data No data

Data presented as mean and ranges. a Mean encounters and range of encounters, standardized patients. b Mean encounters and range of encounters, real patients.





of assessing competency in basic clinical skills from preclin-ical courses and into clerkships and residency programs, 1,15,16 but the question remains whether this change in curriculum is developing better physicians, or whether it is simply develop-ing students better able to perform well on an OSCE.

It is not surprising that we found the desire for a consen-sus statement regarding preclinical education remains strong. However, obtaining consensus is a rather daunting task, as ICM goals and objectives ideally should be targeted across the full spectrum of specialties. The formation of a special interest group at a national meeting (as was accomplished at the 2002 CDIM meeting) was a step in the right direction, but academicians need an ongoing national discussion that not only includes ICM course directors from medical schools around the country, but should also include input from clerk-ship directors in all specialties. Therefore, the discussion on consensus should not be an isolated event, but an established discourse that facilitates dialogue both horizontally and verti-cally. The consensus that was reached by clerkship directors in the study by Windish et al. 6 could serve as a starting point in establishing dialogue between clerkship directors and ICM directors and could lead to methods for translating the core competencies identifi ed in that study into teachable course goals and objectives.

Our study has several limitations. This survey represents the opinions of CDIM respondents; we did not collect objective outcome measurements to verify our fi ndings. Second, despite an overall good response rate for the survey, some respon-dents left specifi c questions blank. We excluded blank ques-tions from our analysis. Third, we did not survey academicians in fi elds other than internal medicine; obtaining views outside the fi eld of internal medicine may yield additional fi ndings and is a critical part of the next steps toward true consensus in our view.

It is imperative that any dialogue toward developing a con-sensus should include both clerkship directors and preclinical course directors to identify essential goals and objectives in addition to improving the reliability and validity of assess-ment tools. With consensus on goals and objectives, course curricula can be further structured to facilitate learner success, and an assessment process can be defi ned and applied across institutions for all students. Because of the broad range of top-ics presented in basic clinical skills courses, directors of these courses have a unique opportunity to coordinate and integrate the ideas of clerkship directors from nearly all medical spe-cialties. We believe that educators should further develop clinical skills education competencies of medical students applicable across all medical specialties leading to a seamless transition from preclinical training to clinical training and into residency. This is certainly a lofty goal, but we believe this is one whose time has come.

ACKNOWLEDGMENTS We acknowledge Dr. Klara Papp for her assistance with designing this sec-tion of the CDIM survey.

APPENDIX 1. SECTION II: PRECLINICAL EDUCATION (1) Do you participate in a preclerkship clinical skills

course (Introduction to Clinical Medicine or ICM)? Yes No

(3) How many total hours do you devote to teaching in ICM?

<3 hr 4–8 hr 9–17 hr 18–30 hr 31–50 hr >50 hr

(4) Where is the ICM course(s) taught? (Check all that apply.)

On Campus Hospital Clinics Simulation Center:

(5) Are you an ICM course director or do you supervise an ICM Course Director?

Yes No

(6) Do you use standardized patients for ICM?

Answer Frequency Percent Cumulative Percent

MissingNoYesTotal

(7) The end of ICM course exam is:


MissingPass/FailGradedTotal

(8) How many standardized patient encounters does each student evaluate in ICM?


MissingNone1 to 55 to 1010 to 20More than 20Total

(2) If so, in what capacity do you participate? (Check all that apply.)

Lecturer

Small group preceptor Course Director

Do not participate

Medical Interviewing

Physical ExamPhysical DiagnosisClinical Reasoning





(13) It would be helpful to me if CDIM assisted with the following:

(a) Consensus statement of ICM objectives and stu-dent requirements.

□Yes □No □Uncertain

(b) Consensus statement on ICM evaluation. □Yes □No □Uncertain

(c) Guidelines for use of SPs and case development in ICM.


(14) Are your contributions as ICM course director valued toward tenure/promotion?


(15) For your ICM course, what department(s) is respon-sible for leadership and coordination?

(Check all that apply.)

□ Internal Medicine □ Family Practice □ Pediatrics □ Surgery □ Ob/gyn

(16a) Compared to 3–4 years ago, have there been: Changes in curriculum objectives related to introduction of Step 2 Clinical Skills Exam in ICM course(s)?


MissingNoUnsureYesTotal

(16b) Compared to 3–4 years ago, have there been: Changes in curriculum content related to introduction of Step 2 Clinical Skills Exam in ICM course(s)?



(16c) Compared to 3–4 years ago, have there been: Change in emphasis or time allocated to clinical skills teach-ing related to the introduction of Step 2 CS in ICM course(s)?



(17) During the fi rst and second years of medical school, how many encounters do your students have with standardized patients?

N Missing Unsure Mean SD Min Max

(18) What percent of these are scored or rated by SPs?


(19) What percent of these are directly observed (live or via video recording) by faculty?


(9) How many encounters with real patients are used for this course?



(10) How many student encounters are directly observed or rated by videotape?



(11) How many student encounters are rated by review of written materials (e.g. H&P submitted by student)?



(12) In what capacity are standardized patients used for ICM?

Teaching Formative Evaluation Summative Evaluation Other





(20) What percent of these are both scored or rated by SPs AND directly observed (live or via video recording) by faculty?


(21) During the fi rst and second years of medical school, how many encounters do your students have with real patients? What percent of real patient encounters are directly observed (live or via video recording) by faculty?

(a) Year 1, no.


(b) Year 1, %


(c) Year 2, no.


(d) Year 2, %


(22) Do you believe your students are adequately prepared for the Step 2 Clinical Skills Exam?



(23) Has there been a change in the frequency or inten-sity of clinical skills assessment or the addition of new methods? Please indicate in the appropriate boxes below whether the following assessment meth-ods have been introduced or more frequently used in courses or rotations listed in the last 3 years.

(a) Standardized Patients


MissingIntroducedUse IncreasedTotal

(b) Direct Observation of Real Patients



(c) Simulators or Synthetic Mannequins



(d) Faculty Development Programs



(24) Do you have an examination involving assessment of clinical skills at the end or beginning of the second year or dur-ing one or more of your ICM courses? Does this examination include multiple encounters with standardized patients, a complete history and physical examination of real or standardized patients, and/or a multiple choice question exam-ination? Are these examinations formative or summative in nature? Please check all that apply.

Multistation SP Exam Complete H&P MCQ Exam

Formative Summative Formative Summative Formative Summative

Beg of 2nd YearIntro Cl MedEnd of 2nd Year

(e) Other







REFERENCES 1. Omori DM , Wong RY , Antonelli MA , Hemmer PA : Introduction to clin-

ical medicine: a time for consensus and integration . Am J Med 2005 ; 118 : 189 – 94 .

2. Clerkship Directors in Internal Medicine : 2003 CDIM Survey . Washington, D.C. , Clerkship Directors in Internal Medicine , 2003.

3. Holmboe ES , Hawkins RE , Huot SJ : Effects of training in direct observa-tion of medical residents’ clinical competence: a randomized trial . Ann Intern Med 2004 ; 140 : 874 – 81 .

4. Anderson RC , Fagan MJ , Sebastian J : Teaching students the art and sci-ence of physical diagnosis . Am J Med 2001 ; 110 : 419 – 23 .

5. Vannatta JB , Smith KR , Crandall S , Fischer PC , Williams K : Comparison of standardized patients and faculty in teaching medical interviewing . Acad Med 1996 ; 71 : 1360 – 2 .

6. Windish DM , Paulman PM , Goroll AH , Bass EB : Do clerkship directors think medical students are prepared for the clerkship years ? Acad Med 2004 ; 79 : 56 – 61 .

7. Haist SA , Wilson JF , Brigham NL , Fosson SE , Blue AV : Comparing fourth-year medical students with faculty in the teaching of physical examination skills to fi rst-year students . Acad Med 1998 ; 73 : 198 – 200 .

8. Stillman PL , Ruggill JS , Sabers D : The use of practical instructors to evaluate a complete physical examination . Eval Health Prof 1978 ; 1 : 49 – 54 .

9. Yudkowsky R , Downing S , Klamen D , Valaski M , Eulenberg B , Popa M : Assessing the head-to-toe physical examination skills of medical stu-dents . Med Teach 2004 ; 26 : 415 – 9 .

10. Barley GE , Fisher J , Dwinnell B , White K : Teaching foundational physi-cal examination skills: study results comparing lay teaching associates and physician instructors . Acad Med 2006 ; 81 : S95 – 7 .

11. Gilliland WR , Pangaro LN , Downing S , et al : Standardized versus real hospitalized patients to teach history-taking and physical examination skills . Teach Learn Med 2006 ; 18 : 188 – 95 .

12. Stillman PL , Regan MB , Swanson DB , et al : A comparison of free-response and cued-response diagnosis scores in an evaluation of clinical competence utilizing standardized patients. Acad Med 1990 ; 65 : 320 – 6 .

13. Junger J , Schager S , Roth C , Schellberg D , Ben-David MF , Nikendei C : Effects of basic clinical skills training on objective structured clinical examination performance . Med Educ 2005 ; 39 : 1015 – 20 .

14. Cooper C , Mira M : Who should assess medical students’ communica-tion skills: their academic teachers or their patients ? Med Educ 1998 ; 32 : 419 – 21 .

15. Martin IG , Jolly B : Predictive validity and estimated cut score of an objec-tive structured clinical examination (OSCE) used as an assessment of clini-cal skills at the end of the fi rst clinical year . Med Educ 2002 ; 36 : 418 – 25 .

16. Wilson BE : Performance-based assessment of internal medicine interns: evaluation of baseline clinical and communication skills . Acad Med 2002 ; 77 : 1158 .



Readdressing the Need for Consensus in Preclinical Education

Documents

Transcript of Readdressing the Need for Consensus in Preclinical Education