OIABM

Riqc

DoSsAPciTAtTsoi

©P

nline Spaced Education Generates Transfer andmproves Long-Term Retention of Diagnostic Skills:

Randomized Controlled TrialPrice Kerfoot, MD, EdM, Yineng Fu, MD, Harley Baker, EdD, Donna Connelly, BS,ichael L Ritchey, MD, FACS, Elizabeth M Genega, MD

BACKGROUND: Retention of learning from surgical training is often limited, especially if the knowledge andskills are used infrequently. Using histopathology diagnostic skills as an experimental system, wecompared knowledge transfer and retention between bolus Web-based teaching (WBT) mod-ules and online spaced education, a novel email-based method of online education founded onthe spacing effect.

STUDY DESIGN: All US urology residents were eligible to participate. Enrollees were randomized to 1 of 2cohorts. Cohort 1 residents received 3 cycles/repetitions of spaced education on prostate-testishistopathology (weeks 1 to 16) and 3 WBT modules on bladder-kidney (weeks 14 to 16).Cohort 2 residents received 3 cycles of spaced education on bladder-kidney (weeks 1 to 16) and3 WBT modules on prostate-testis (weeks 14 to 16). Each daily spaced education email pre-sented a clinical scenario with histopathology image and asked for a diagnosis. Participantsreceived immediate feedback after submitting their answers. Each cycle/repetition was 4 weekslong and consisted of 20 questions with unique images. WBT used the identical content anddelivery system, with questions aggregated into three 20-question modules. Long-term reten-tion of all 4 topics was assessed during weeks 18 to 45.

RESULTS: Seven-hundred and twenty-four urology residents enrolled. Spaced education and WBT werecompleted by 77% and 66% of residents, respectively. Spaced education and WBT generatedmean long-term score increases of 15.2% (SD 15.3%) and 3.4% (SD 16.3%), respectively (p �0.01). Spaced education increased long-term learning efficiency 4-fold.

CONCLUSIONS: Online spaced education generates transfer of histopathology diagnostic skills and substantiallyimproves their long-term retention. Additional research is needed to determine how spacededucation can optimize learning, transfer, and retention of surgical skills. ( J Am Coll Surg 2010;

211:331–337. © 2010 by the American College of Surgeons)

eoa(ebt

RFtGtC((CH

etention of learning from surgical training is often lim-ted, particularly if the knowledge and skills are used infre-uently. Although forgetting is a natural physiological pro-ess that should be expected, traditional methods of

isclosure Information: Dr Kerfoot owns equity in and is a board memberf Spaced Education Inc. Drs Kerfoot and Genega have authored courses onpaced Education’s website (www.spaceded.com) but receive no compen-ation from the courses or company. Ms. Connelly is an employee of themerican Urological Association, which is selling its Self-Assessment Studyrogram on the Spaced Education website. None of the authors receiveompensation for these SASP sales. There are no other potential conflicts ofnterest relevant to this article.his study was supported in part by the American Urological Association, themerican Urological Association Foundation, Astellas Pharma US, Inc., and

he United States Agency for Healthcare Research and Quality.he views expressed in this article are those of the authors and do not neces-

arily reflect the position and policy of the United States Federal Governmentr the Department of Veterans Affairs. No official endorsement should be

nferred. 0

3312010 by the American College of Surgeons

ublished by Elsevier Inc.

ducation are often structured in a manner that fails toptimize long-term retention of knowledge. By presentingbolus of educational content at a single time point

massed distribution), many lectures, skill sessions, confer-nces, and Web programs do not harness the educationalenefits of the “spacing effect.” The spacing effect refers tohe psychological research finding that information is

eceived February 4, 2010; Revised April 14, 2010; Accepted April 26, 2010.rom the Surgical Service, Urology, Veterans Affairs Boston Healthcare Sys-em, Jamaica Plain, MA (Kerfoot), Harvard Medical School (Kerfoot, Fu,enega) and Department of Pathology, Beth Israel Deaconess Medical Cen-

er (Fu, Genega), Boston, MA, California State University Channel Islands,amarillo, CA (Baker), American Urological Association, Linthicum, MD

Connelly), and Department of Urology, Mayo Clinic, Phoenix, AZRitchey).orrespondence address: B Price Kerfoot, MD, EdM, Veterans Affairs Bostonealthcare System, 150 South Huntington Ave, 151DIA, Jamaica Plain, MA

2130. email: price.kerfoot@gmail.com.

ISSN 1072-7515/10/$36.00doi:10.1016/j.jamcollsurg.2010.04.023

ladtbpi

(teoqlmvmtettatmlec

rtutphtpcacHibAta

oaesbe

MSAeeb

DBumacceoatcubqswbubvestvpwmlum

DIei

332 Kerfoot et al Spaced Education Generates Skill Transfer J Am Coll Surg

earned and retained more effectively when presentednd repeated during spaced intervals of time (spacedistribution).1-3 The spacing effect appears to have a dis-inct neurophysiologic basis. Spaced learning by rats haseen shown to improve neuronal longevity in the hip-ocampus,4 and a tyrosine phosphatase has recently been

dentified that regulates the spacing effect in Drosophila.5

We have developed a novel method of online educationtermed spaced education) that improves long-term reten-ion of learning by harnessing this spacing effect. Spacedducation is currently delivered to learners through peri-dic emails that contain case scenarios and multiple-choiceuestions. After submitting answers to each question on-

ine, learners receive immediate feedback and educationalaterial. Questions are then repeated during spaced inter-

als of time to harness the spacing effect. By presentingaterial in a question-and-answer format, spaced educa-

ion is also structured to take advantage of the “testingffect.” The testing effect refers to the research finding thatesting does not just measures knowledge, but also altershe learning process itself so that new knowledge is retainednd transferred more effectively.6-8 In randomized con-rolled trials, we have shown that the spaced educationethodology improves knowledge acquisition and boosts

earning retention out to 2 years.9,10 In addition, spacedducation is well-accepted by learners ranging from medi-al students to attending surgeons.11,12

Several critical questions about online spaced educationemain unanswered. First, the spaced education interven-ions to date have focused on improving knowledge. It isnknown whether the distributed spacing of contenthrough spaced education can also be used to durably im-rove clinically-relevant skills. Second, our prior researchas measured knowledge retention by presenting the iden-ical questions to surgical trainees during distant timeoints. Given this, it remains unclear whether spaced edu-ation can effectively generate transfer of knowledgend/or skills to new contexts. To answer both questions, weonducted a randomized trial among US urology residents.istopathology diagnostic skills were selected as an exper-

mental system because they are a requirement for urologyoard certification and are challenging to learn and retain.lthough not the same as manual surgical skills, his-

opathologic diagnosis is a learned skill that requires the

Abbreviations and Acronyms

IC � intervention cohortOC � outcomes cohortWBT � Web-based teaching

pplication of knowledge, past experience, and pattern rec- e

gnition to the interpretation of new histopathology im-ges in new clinical contexts. We investigated the differ-nces in transfer and retention of histopathology diagnostickills between online spaced education and bolus Web-ased teaching modules, a current gold-standard of onlineducation.

ETHODStudy participantsll US urology residents in training years 1 through 4 wereligible to participate. Participants were recruited throughmail.There were no exclusion criteria. Institutional reviewoard approval was obtained to perform this study.

evelopment of the interventionsoth the spaced education and Web-based teaching mod-le (WBT) interventions were constructed as a series ofultiple-choice questions containing a clinical scenario

nd histopathology image (the evaluative component) withorresponding answers and explanations (the educationalomponent; Appendix, available online). These question-xplanation items focused on developing core histopathol-gy diagnostic skills in 4 organs: bladder, kidney, prostate,nd testis. A clinical pathologist constructed 10 to 14 ques-ions for each topic area that were then independentlyontent-validated by another clinical pathologist and 2rologist-educators. The questions were then pilot-testedy 30 urology chief residents. Psychometric analysis of theuestions was performed using the Integrity test analysisoftware (Castle Rock Research). For each organ, 10 questionsere selected for inclusion based on item difficulty, point-iserial correlation, and Kuder-Richardson 20 score. The ed-cational components of the 40 items were then constructedy a clinical pathologist and then independently content-alidated by another clinical pathologist and 2 urologist-ducators. To assess transfer of knowledge between eachpaced education cycle and each WBT module, 3 addi-ional isomorphic sets of the 40 items were constructed andalidated. At every presentation, the questions and histo-athology images were unique, and the answer choicesere randomized. Because of the question-explanation for-at of the items, evaluation and education are inextricably

inked. The identical evaluative/educational items weresed for both spaced education and Web-based teachingodules.

elivery of the interventionsn the spaced education intervention, the items (questions/xplanations) were delivered to residents at designated timentervals through an automated email delivery system. The

mail presented a clinical scenario, a histopathology image,

aAaTdpswl

daswtscp

STc2uo(Wbe11o3stsrius

qwwtw

rotp

pwTicteeTpu

r(ost

ir

OTdacpw

e ran

333Vol. 211, No. 3, September 2010 Kerfoot et al Spaced Education Generates Skill Transfer

nd a multiple-choice question (evaluative component).fter clicking on a hyperlink, a Web-page opened thatllowed the resident to submit an answer to the question.he answer was downloaded to a central server, and resi-ents were immediately presented with a Web-page dis-laying the educational component: the correct answer, aummary of the curricular learning points, explanationshy possible answers were correct/incorrect, and hyper-

inks to additional educational material.In the WBT intervention, items were delivered to resi-

ents with the identical content and in the identical formats in the spaced education intervention. In contrast to thepaced education intervention, in which individual itemsere emailed to residents each weekday, the WBT interven-

ion presented a bolus of 20 items in a single email. Afterubmitting an answer to an item and receiving the educationalomponent (answer/explanation), a resident could click a hy-erlink to move directly to the next question.

tudy design and organizationhis multi-institutional randomized controlled trial was

onducted during 45 weeks from July 2007 through June008. Residents were stratified by year of training and thennderwent factorial randomization (block size � 8) into 1f 2 intervention cohorts (IC) and 1 of 4 outcomes cohortsOC). All residents were sent both spaced education and

BT, but the set of topics delivered by each method variedy IC (Fig. 1). Residents in IC-1 received 3 cycles of spacedducation on prostate-testis (weeks 1 to 4, 5 to 8, and 13 to6) and 3 WBT modules on bladder-kidney (weeks 14 to6). Residents in IC-2 received 3 cycles of spaced educationn bladder-kidney (weeks 1 to 4, 5 to 8, and 13 to 16) andWBT modules on prostate-testis (weeks 14 to 16). The

paced education items were delivered each weekdayhrough emails containing 1 question/answer, and thepaced education material was distributed in 3 cycles orepetitions to take advantage of the spacing effect. All themages presented to residents in each cycle or module werenique. The WBT used the identical content and delivery

Weeks 1-4 5-8 13-1

Intervention Cohort 1

Intervention Cohort 2

PT Cycle 1 PT Cycle 2 PT Cyc

BK

BK Cycle 1 BK Cycle 2 BK Cyc

PT

WBT

WBT

SE

SE

Intervention Phase

Figure 1. Structure of th

ystem, with the questions aggregated into three 20- f

uestion modules delivered through separate emails ineek 14.The trial was specifically structured to ensure that,ithin a given set of topics (bladder-kidney or prostate-

estis), the only difference between intervention cohortsas the spacing of content.After their national in-service examination (week 16),

esidents completed a survey that asked which methodol-gy (spaced education versus WBT) they would prefer forheir online education and whether they would want toarticipate in future spaced education programs.To assess long-term retention of learning, residents com-

leted a final 4-week spaced education cycle of 40 itemsith unique images and questions covering all 4 organs.he spaced education items were delivered each weekday

n emails containing 2 questions/answers. OCs 1 to 4 re-eived this final cycle during weeks 18 to 21, 26 to 29, 34o 37, and 42 to 45, respectively. The delivery of spacedducation and WBT ended at week 16 so that time sincexposure to content would be identical between formats.his study was structured in this manner to allow for com-arable forgetting curves to be constructed for spaced ed-cation and WBT.In summary, each resident at the end of the study had

eceived 3 cycles of spaced education on 2 pathology topicsweeks 1 to 4, 5 to 8, and 13 to 16), 3 WBT modules on thether 2 pathology topics (weeks 14 to 16), and a finalpaced education cycle on all 4 pathology topics (weeks 18o 21, 26 to 29, 34 to 37, or 42 to 45).

After completing the online survey and �90% of thetems in the final 4-week spaced education cycle, residentseceived a $25 gift certificate to an online bookstore.

utcomes measureshe primary outcomes measure was residents’ retention ofiagnostic skills during weeks 18 to 45 of the trial. Second-ry outcomes measures included long-term learning effi-iencies of online educational methodologies, residents’erformance on spaced education versus WBT duringeeks 1 to 16, and their preference of online learning

18-21 26-29 34-37 42-45Service Exam

OC1 OC2 OC3 OC4

Retention Phase

domized controlled trial.

6 In-

le 3

1-3

le 3

1-3

ormats.

SPoictdbe3t

WadivitOwcoCsotsmebwdtheuew

RSuBsac2a4i

cvdib

dw1u512ais(ib(

gweal

Tt

C

PG

Y

D

AO

P

334 Kerfoot et al Spaced Education Generates Skill Transfer J Am Coll Surg

tatistical analysisower was estimated based on the cross-format comparisonf scores on the 40-item test of learning retention admin-stered during weeks 18 to 45. If a total of 172 residentsompleted this parallel-design randomized controlled trial,he probability is 90% that the study will detect a scoreifference at 0.05 significance (�), if the true differenceetween scores is 5% and SD is 10% (corresponding to anffect size of 0.5). Adjusting for a potential attrition rate of0%, we needed at least 224 residents total to complete therial.

Scores for the 20-item spaced education cycles andBT modules were calculated as the number of questions

nswered correctly normalized to a percentage scale. Resi-ents were defined as completing the spaced education

ntervention (3 cycles in weeks 1 to 16), the WBT inter-ention (3 modules in weeks 14 to 16), and the final 40-tem assessment of long-term retention (weeks 18 to 45) ifhey submitted answers to �80% of the questions in each.nly those residents who completed all 3 interventionsere included in the analyses. Unanswered questions were

onsidered to be answered incorrectly. Internal consistencyf the cycles/modules of 20 questions was estimated withronbach �.13,14 Two-tailed t-tests were used to test the

tatistical significance of differences in scores across meth-dologies. Long-term score increase was calculated by sub-racting residents’ score in cycle 1/module 1 from their OCcore in weeks 18 to 45. Learning efficiency for eachethod of online education was calculated by dividing

ach resident’s long-term score increase by the total num-er of questions answered (and explanations received) ineeks 1 to 16. Two-way ANOVA was used to assess theifferential shapes of the retention curves from baselinehrough OC4 between spaced education and WBT. Co-en’s d provided the intervention effect sizes.15 Cohen’s dxpresses the difference between the means in terms of SDnits, with 0.2, 0.5, and 0.8 considered to be small, mod-rate, and large effects, respectively.16 Statistical analysesere performed with PASW/SPSS 18.0 (SPSS, Inc).

ESULTSeven-hundred twenty-four of the approximately 1,000rology residents in the United States enrolled in the trial.aseline demographic characteristics of participants were

imilar between randomized intervention cohorts (Table 1)nd outcomes cohorts. Spaced education and WBT wereompleted by 77% and 66% of residents, respectively (Fig.). Sixty-four percent of enrollees completed the staggeredssessment of retention during weeks 18 to 45. In total,9% of resident enrollees (358 of 724) completed all 3

nterventions and were included in the analysis. Although 0

onsiderably more residents in IC-2 completed all 3 inter-entions (54% versus 45% for IC-2; p � 0.01), baselineemographic characteristics of the 358 residents included

n the analysis were similar between cohorts. Average Cron-ach � of the cycles/modules of 20 questions was 0.55.

WBT generated a greater short-term increase in scoresuring weeks 14 to 16 than did spaced education duringeeks 1 to 16. WBT scores increased from 63.9% (SD4.0) in module 1 (week 14) to 90.0% (SD 11.9) in mod-le 3 (week 16). Spaced education scores increased from5.2% (SD 14.0) in cycle 1 (weeks 1 to 4) to 69.0% (SD4.4) in cycle 3 (weeks 13 to 16). WBT generated a mean6.1% (SD 15.4) increase in scores from modules 1 to 3,nd spaced education generated a mean 13.8% (SD 14.6)ncrease in scores from cycles 1 to 3 (p � 0.01; Cohen effectize 0.82). The scores for the spaced education cycle 1weeks 1 to 4) and WBT module 1 (week 14) were signif-cantly different at baseline (p � 0.01). These differences inaselines were consistent across both pathology topic setsbladder-kidney versus prostate-testis).

In contrast, spaced education generated considerablyreater long-term retention of diagnostic skills duringeeks 18 to 45. Overall, WBT and spaced education gen-

rated mean long-term score increases of 3.4% (SD 16.3)nd 15.2% (SD 15.3), respectively (p � 0.01). Long-termearning efficiency was mean 0.06 (SD 0.27) and 0.26 (SD

able 1. Characteristics of Participants Randomized to In-ervention Cohorts

haracteristicIntervention

cohort 1Intervention

cohort 2

articipants randomized, n 364 360ender, n (%)Male 284 (78) 274 (76)Female 80 (22) 86 (24)

ears of urology training, n (%)1 73 (20) 73 (20)2 92 (25) 91 (25)3 100 (27) 96 (27)4 99 (27) 100 (28)egree, n (%)MD 318 (87) 318 (88)DO 17 (5) 11 (3)MD/other 29 (8) 31 (9)

ge (y), mean � SD 30.7 � 3.2 30.5 � 2.7utcomes cohort, n (%)Wks 18 to 21 90 (25) 89 (25)Wks 26 to 29 92 (25) 90 (25)Wks 34 to 37 91 (25) 91 (25)Wks 42 to 45 91 (25) 90 (25)

ercentages might not add to 100% because of rounding.

.26) for WBT and spaced education, respectively. This

4s

stWt3c(fotr3

tsdap

DTt

ttmWsaelaeHcsmapt(pous

e

cha

335Vol. 211, No. 3, September 2010 Kerfoot et al Spaced Education Generates Skill Transfer

-fold increase in long-term learning efficiency frompaced education represents an effect size of 0.77.

ANOVA methods revealed different retention curves forpaced education and WBT (p � 0.001) from their respec-ive baselines. Analysis of the retention curves show that

BT scores fell immediately after module 3 and declinedo baseline levels by OC3 to OC4 in weeks 34 to 45 (Fig.A). In contrast, spaced education scores increased afterycle 3 and peaked in OC1 before beginning to declineFig. 3B). Similar findings were obtained when adjustingor the differences in baseline scores (Fig. 3C). However,nce OC1 was reached in both WBT and spaced educa-ion, no substantial difference was found in the shape of theetention curves during weeks 18 to 42 (OC1 to OC4, Fig.A and 3B).Eighty-two percent of residents (596 of 724) completed

he end-of-program survey. Survey completion rates wereimilar between cohorts. Seventy-seven percent of respon-ents (461 of 596) preferred spaced education over WBT,nd 99% of respondents (82% of all enrollees) requested toarticipate in future spaced education programs.

ISCUSSIONhis randomized controlled trial demonstrates that the dis-

Spaced Education Cycles 1-3on Prostate & Testis

Completed by 272 (75%)

724

364 Assigned to Intervention Cohort 1

164 (45%) Included in Analysis

ApproxResidents

724 Urology

Web-based Teaching Modules 1-3on Bladder & Kidney

Completed by 223 (61%)

164 (45%) Completed ≥80% of All 3 Interventions

Staggered Assessment of Retentionon all Four Organs

Completed by 227 (62%)

Figure 2. Modified CONSORT flow

ributed spacing of content through online spaced educa- k

ion substantially improves long-term retention of histopa-hology diagnostic skills compared with bolus WBTodules delivering the identical content. Although bolusBT modules generated a sharp short-term spike in

cores, these scores fell back to baseline levels 3 monthsfter the intervention. In contrast, spaced education gen-rated a steady increase in scores that persisted above base-ine out to the completion of the trial 7 months later. Inddition, this study is the first to demonstrate that spacedducation can generate transfer of learning to new contexts.istopathology diagnostic skills learned from spaced edu-

ation cycle 1 improved performance on new content inpaced education cycle 2, which then improved perfor-ance on new content in spaced education cycle 3. There

lso appears to have been substantial transfer of skills acrossathology topics: residents’ learning from spaced educa-ion cycles 1 and 2 (in weeks 1 to 8) on one set of organsprostate-testis versus bladder-kidney) consistently im-roved their performance in WBT module 1 (in week 14)n the other set of organs. Importantly, online spaced ed-cation is well-accepted by residents, with 77% preferringpaced education over WBT.

This study is also the first to show that online spacedducation can improve clinically relevant skills, not just

360 Assigned to Intervention Cohort 2

mized

Urology le to Enroll

ents Enrolled

Spaced Education Cycles 1-3 on Bladder & Kidney

Completed by 287 (80%)

Web-based Teaching Modules 1-3on Prostate & Testis

Completed by 252 (70%)

Staggered Assessment of Retentionon all Four Organs

Completed by 235 (65%)

194 (54%) Completed ≥80% of All 3 Interventions

194 (54%) Included in Analysis

rt of randomized controlled trial.

Rando

. 1000 Eligib

Resid

nowledge. Although histopathology diagnostic skills are

qtdtvdpassa

draaci

notRpogfpAaappAcrtpmttptmb

fsrsaagctstctruc

Fs

336 Kerfoot et al Spaced Education Generates Skill Transfer J Am Coll Surg

uite different than surgical skills, research has shown thathe educational benefits of the spacing effect can also beirectly applied to live surgical training. In a randomizedrial of 38 surgical residents, distributed practice of micro-ascular suturing techniques (1 training session per weekuring 4 weeks) was found to improve skill retention, com-ared with bolus practice (4 training sessions in 1 day).17 Inddition, those residents with distributed practice had sub-tantially improved transfer of their microvascular suturingkills to a live rat model.17 By facilitating the interactive

Per

cent

age

Sco

res

50

60

70

80

90

100

Web-based Teaching Modules

Module 1

Module 2

Module 3

OC-1

OC-2OC-3 OC-4

Per

cent

age

Sco

re

50

55

60

65

70

75

80

Spaced Education

Cycle 1

Cycle 2

Cycle 3

OC-1

OC-4OC-3

OC-2

Weeks0 10 20 30 40 50

Per

cent

age

Cha

nge

from

Bas

elin

e

-10

0

10

20

30

Spaced EducationWeb-based Modules

A

B

C

igure 3. Forgetting curves for (A) Web-based teaching modules; (B)paced education; and (C) both.

nswering of questions and allowing for the automated f

elivery and spacing of educational items, the online envi-onment appears to be particularly well-suited for takingdvantage of the spacing and testing effects. We now use andaptive algorithm that improves learning efficiency byustomizing the spacing and content of the educationaltems for each learner.18

Because the spaced education methodology is content-eutral, it has the potential to be used across the spectrumf surgical training, from undergraduate surgical educationo maintenance of certification among attending surgeons.esearch is needed to determine whether spaced educationrograms using video clips from surgical procedures as partf the questions and explanations can effectively teach sur-ical skills. Spaced education might be of particular meritor those skills that are critical for patient care but that areerformed infrequently and are subject to rapid forgetting.dvanced cardiac life support is a core set of skills thatppears to meet these criteria: a recent study found that,mong nurses who passed their advanced cardiac life sup-ort training examination, only 30% and 14% were able toass the examination 3 and 12 months later, respectively.19

lthough many institutions are investing heavily in surgi-al simulation facilities, it is not clear how to optimize theetention of simulation training and improve the transfer ofhis training to patient care. An adjuvant spaced educationrogram structured to reinforce the simulation trainingight be able to improve these important outcomes. Al-

hough spaced education can substantially improve reten-ion, the forgetting curves in Figure 3 suggest that there is aersistent need to reinforce learned content and skills overime. This finding provides support for new system ofaintenance of certification currently being implemented

y the American Board of Medical Specialties.There are several limitations to this study, including its

ocus on a narrow set of skills and on trainees within aingle surgical subspecialty. In spite of this focus, there is noeason to expect that these findings would differ if otherpecialties or domains were assessed. Although the percent-ge of residents who completed all 3 interventions waspproximately 50%, it is important to note that the demo-raphic characteristics of those residents who were in-luded in the analysis were similar between cohorts. Al-hough the completion rate with spaced education wasubstantially higher than WBT (77% versus 66%, respec-ively), this should not bias the educational outcomes be-ause all residents needed to complete both spaced educa-ion and WBT to be included in the analysis. Futureesearch will investigate why residents preferred spaced ed-cation over WBT by 3:1. The average Cronbach � of theycles/modules was low, but this did not prevent us from

inding substantial and significant results. Finally, the study

hsb(miia(hnce

esAea

A

S

AADCSOA

S

AnMdu

R

1

1

1

1

1

1

1

1

1

1

337Vol. 211, No. 3, September 2010 Kerfoot et al Spaced Education Generates Skill Transfer

ad been structured to allow direct comparisons betweenpaced education and WBT, but such analyses were limitedecause of unanticipated differences in baseline scoresspaced education cycle 1 versus WBT module 1) betweenethodologies. Similar results were obtained when adjust-

ng for these differences in baseline scores. This differencen baseline scores is likely a result of the transfer of learningcross pathology topics from spaced education cycles 1 to 2weeks 1 to 8) to WBT module 1 (week 14). This study alsoas several strengths, including its randomized design, theovelty of the educational intervention, and the large per-entage of urology residents across the United States whonrolled.

In summary, our results demonstrate that online spacedducation generates transfer of histopathology diagnostickills and substantially improves their long-term retention.dditional research is needed to determine how spacedducation can best be used to optimize learning, transfer,nd retention of surgical skills.

uthor Contributions

tudy conception and design: Kerfoot, Fu, Connelly, Ritchey,Genega

cquisition of data: Kerfootnalysis and interpretation of data: Kerfoot, Baker, Genegarafting of manuscript: Kerfootritical revision: Baker, Fu, Connelly, Ritchey, Genegatatistical analysis: Kerfoot, Bakerbtaining funding: Kerfoot, Connelly, Ritcheydministrative, technical, or material support: Fu, Baker,

Connelly, Genegaupervision: Ritchey

cknowledgment: We recognize the invaluable work of Ro-ald Rouse, Jason Alvarez, and David Bozzi of the Harvardedical School Center for Educational Technology for their

evelopment of the spaced education online delivery platformsed in this trial.

EFERENCES

1. Bjork RA. Retrieval practice and the maintenance of knowledge.

In: Gruneberg MM, Morris PE, Sykes RN, eds. Practical Aspects

of Memory: Current Research and Issues. New York: John Wiley& Sons; 1988:396–401.

2. Glenberg AM, Lehmann TS. Spacing repetitions over 1 week.Mem Cognit 1980;8:528–538.

3. Pashler H, Rohrer D, Cepeda NJ, Carpenter SK. Enhancinglearning and retarding forgetting: choices and consequences.Psychon Bull Rev 2007;14:187–193.

4. Sisti HM, Glass AL, Shors TJ. Neurogenesis and the spacingeffect: learning over time enhances memory and the survival ofnew neurons. Learn Mem 2007;14:368–375.

5. Pagani MR, Oishi K, Gelb BD, Zhong Y. The phosphataseSHP2 regulates the spacing effect for long-term memory induc-tion. Cell 2009;139:186–198.

6. Roediger HL, Karpicke JD. Test-enhanced learning: takingmemory tests improves long-term retention. Psychol Sci 2006;17:249–255.

7. Karpicke JD, Roediger HL 3rd. The critical importance of re-trieval for learning. Science 2008;319:966–968.

8. Rohrer D, Taylor K, Sholar B. Tests enhance the transfer oflearning. J Exp Psychol Learn Mem Cogn 2010;36:233–239.

9. Kerfoot BP. Learning benefits of on-line spaced education per-sist for 2 years. J Urol 2009;181:2671–2673.

0. Kerfoot BP, Baker HE, Koch MO, et al. Randomized, controlledtrial of spaced education to urology residents in the UnitedStates and Canada. J Urol 2007;177:1481–1487.

1. Kerfoot BP, Armstrong EG, O’Sullivan PN. Interactive spaced-education to teach the physical examination: a randomized con-trolled trial. J Gen Intern Med 2008;23:973–978.

2. Kerfoot BP, Kearney MC, Connelly D, Ritchey ML. Interactivespaced education to assess and improve knowledge of clinicalpractice guidelines: a randomized controlled trial. Ann Surg2009;249:744–749.

3. Cronbach LJ. Coefficient alpha and the internal structure oftests. Psychometrika 1951;16:297–334.

4. Pedhazur EJ, Schmelkin LP. Measurement, Design, and Analy-sis: An Integrated Approach. Hillsdale, NJ: Lawrence Erlbaum;1991.

5. Cohen J. Statistical Power Analysis for the Behavioral Sciences.2nd ed. Hillsdale, NJ: Erlbaum; 1988.

6. Maxwell SE, Delaney HD. Designing Experiments and Analyz-ing Data: A Model Comparison Approach. Belmont, CA: Wads-worth; 1990.

7. Moulton CA, Dubrowski A, Macrae H, et al. Teaching surgicalskills: what kind of practice makes perfect? A randomized, con-trolled trial. Ann Surg 2006;244:400–409.

8. Kerfoot BP. Adaptive spaced education improves learning effi-ciency: a randomized controlled trial. J Urol 2010;183:678–681.

9. Smith KK, Gilcreast D, Pierce K. Evaluation of staff’s retention

of ACLS and BLS skills. Resuscitation 2008;78:59–65.

AE

Aa

CE

ms

w

E

htc

c

f

s

337.e1 Kerfoot et al Spaced Education Generates Skill Transfer J Am Coll Surg

ppendixxample of education item (question/explanation) used in the study

64-year old woman with a 3-cm right renal mass undergoes an open partial nephrectomy. Images from the pathology specimenre shown above (low power on the left; high power on the right). These pathology findings are most consistent with:

A. Oncocytoma

B. Renal cell carcinoma, chromophobe type

C. Renal cell carcinoma, conventional (clear cell) type

D. Renal cell carcinoma, papillary type

E. Normal renal parenchyma

orrect Answer: A. Oncocytomaxplanation of Correct Answer:Oncocytomas are composed of tumor cells arranged in small and large nests (arrow 1, below) and set in a fibrous oryxoid background (arrow 2). The tumor cells have abundant eosinophilic (pink) granular cytoplasm, round nuclei with

mooth contours and a small pinpoint nucleolus.Unlike chromophobe tumors, oncocytomas do not typically display perinuclear clearing (perinuclear halos) or nuclei

ith irregular contours (“raisinoid appearance”).

xplanation of Incorrect Answers:(B) In chromophobe renal cell carcinoma, the tumor cells usually display perinuclear clearing (perinuclear halos) and

ave irregular nuclear contours (“raisinoid appearance”). These two features of chromophobe tumors will assist you inelling oncocytomas and chromophobe tumors apart. Sometimes, it can be difficult to differentiate oncocytomas fromhromophobe tumors because both have eosinophilic (pink) granular cytoplasm.

(C) In conventional (clear cell) renal cell carcinoma, one would usually see a proliferation of tumor cells with some clearytoplasm. The tumor is richly vascularized by blood vessels, which are abundant, delicate, and typically of small caliber.

(D) In papillary renal cell carcinoma, one would usually see diffuse tubulo-papillary growth. The papillae have centralibrovascular cores lined by neoplastic epithelium.

(E) In normal renal parenchyma, you would expect to see glomeruli (cortex) or orderly arrays of tubules (medulla), not

heets or nests of tumor cells.