579

TELEMEDICINE AND e-HEALTHVolume 12, Number 5, 2006© Mary Ann Liebert, Inc.

Policy

Research Recommendations for the AmericanTelemedicine Association

ELIZABETH KRUPINSKI, Ph.D.,1 SUSAN DIMMICK, Ph.D.,2 JIM GRIGSBY, Ph.D.,3GREG MOGEL, M.D.,4 DENA PUSKIN, Sc.D.,5 STUART SPEEDIE, Ph.D.,6

BETH STAMM, Ph.D.,7 BONNIE WAKEFIELD, Ph.D.,8 JOHN WHITED, M.D.,9PAMELA WHITTEN, Ph.D.,10 and PETER YELLOWLEES, M.D.11

ABSTRACT

The American Telemedicine Association (ATA) convened a panel of experts to generate a re-search agenda for the telemedicine community to further support and promote the long-termacceptance and use of telehealth. Three principles to guide research and four key areas withinwhich research is greatly needed were identified. These four areas are technical, clinical, hu-man factors and ergonomics, and economic analyses. It is the hope of the panel that the re-search recommendations put forth in this document will give investigators the inspiration,tools and goals to make this happen.

1Department of Radiology, University of Arizona, Tucson, Arizona.2Oak Ridge Institute for Science and Education, University of Tennessee Health Science Center, Memphis, Ten-

nessee.3Division of Healthcare Policy and Research, University of Colorado Health Sciences Center, Denver, Colorado.4Department of Radiology, University of Southern California, Los Angeles, California.5Office for the Advancement of Telehealth, USDHHS, Rockville, Maryland.6Division of Health Informatics, Department of Laboratory Medicine and Pathology, University of Minnesota Med-

ical School, Minneapolis, Minnesota.7Institute of Rural Health, Idaho State University, Pocatello, Idaho.8Harry S Truman Memorial Veterans Hospital and Sinclair School of Nursing, University of Missouri-Columbia,

Columbia, Missouri.9Center for Health Services Research in Primary Care, VA Medical Center, Department of Medicine, Duke Univer-

sity, Durham, North Carolina.10Department of Telecommunication, Michigan State University, East Lansing, Michigan.11Department of Psychiatry & Behavioral Sciences and Center for Health & Technology, University of California,

Davis, Medical Center, Sacramento, California.

INTRODUCTION

TELEHEALTH TECHNOLOGIES, which embracetelemedicine, are being used in an ever-

widening array of applications and environ-

ments, and there is a substantial body of liter-ature advocating their use and general utility.The existing body of literature, however,ranges from purely anecdotal accounts of tele-health applications through well-controlled

randomized clinical trials.1–6 Because in part tothe inconsistent nature of the literature, therehave been questions raised about the quality oftelehealth that may be dramatically slowing itsintegration into the healthcare continuum. Inresponse to this problem, Thelma McCloskyArmstrong, immediate past president of theAmerican Telemedicine Association (ATA)convened a panel to formulate a researchagenda for the ATA. The panel members,which comprised the authors of this paper, areregarded as experts in the area of telehealth re-search.

The recommendations were generated viagroup consensus via phone and e-mail com-munications, based on individual reviews ofthe literature and expertise each memberbrought to the panel. The intent of this agendais to provide research guidance to the tele-health community in order to further supportand promote the long-term acceptance and useof telehealth. It should be noted that in the con-text of this paper we are using the general termtelehealth, and other nomenclatures such astelemedicine and e-health are presumed to fallunder this general heading.

PROGRAM ASSESSMENT VERSUS RESEARCH

At the outset, a distinction is made betweentelehealth (or program) assessment and tele-health research. Assessment or program eval-uation refers to a process applied to an indi-vidual program for quality assessment/qualitycontrol (QA/QC), tracking of cases, and otherrelated activities designed to characterize whata program is doing. For example, a commonmethod to assess the success of a program is to“measure against the mission.” Mission state-ments or yearly plans typically have clearlyidentifiable goals, and stakeholders can, on aperiodic basis, review these goals and deter-mine if they have been met and to what degree.Broadly stated goals such as “we will promotethe use of telehealth” are usually assessed sub-jectively; while more concrete goals such as “inthe coming year we will serve 20% more pa-tients via telehealth” can be assessed objec-tively by comparing hard data. Assessment re-

sults are typically used to guide and manage aprogram in both the short and long term. Al-though assessment methods can be generalizedand used by other programs, the results fromone program rarely apply directly to another.This document does not address this type of as-sessment given its program specific and idio-syncratic nature.

Research on the other hand refers to investi-gations that are generally hypothesis drivenand generate results that can be generalized tothe broader telehealth community. Althoughmany research strategies are hypothesis drivenand collect what it considered quantitativedata, there are a number of very prominent andacceptable qualitative research designs that canbe used as well. For example, Grounded The-ory falls into the qualitative research designarea and rather than have set hypotheses guidethe data collection and analysis, the data (e.g.,interview responses) are analyzed for recurringthemes and relationships in order to generatean understanding of the phenomenon understudy.7,8

UTILIZATION OF EXISTING TOOLS,PARADIGMS, AND MODELS

It is clearly not possible (or desirable) to man-date how research should be done. The panelrecommended strongly, however, that when-ever possible established research methods andstatistical analysis tools be used. For example,it is a very common practice in telehealth re-search to survey those involved in various as-pects of a telehealth encounter (e.g., referringclinician, consulting clinician, patient). Surveydata can be extremely useful, but if the surveytools are not developed properly (e.g., ana-lyzed for content, criterion, and construct va-lidity) they often generate data that are not re-liable or valid. Although the basics of surveyconstruction and analysis can be found in anynumber of introductory statistics books,9 it isalso important to utilize methods developedfor a particular focus or field if they exist. Forexample, if a survey is being developed to de-termine what factors induce patients to seekout or request telehealth services, it might bebeneficial to examine the methods and styles

KRUPINSKI ET AL.580

used in the construction of consumer satisfac-tion surveys.10 If, however, a survey is beingdeveloped to assess how participants feel theyinteracted with a remote group during a tele-facilitated cancer support group session, meth-ods from social psychology and the study ofgroup dynamics would be more appropriate toutilize.11 In some cases, research topics such asquality of life are so well studied in healthcareevaluation in general that the existing surveyinstruments (e.g., World Health Organization’sQuality of Life Instruments) are firmly estab-lished and often regarded as the standard toolto use.12 If such validated and widely usedtools exist, it is highly recommended that theybe investigated for adaptation to and use intelehealth studies.

It is also recommended that if validatedmodels exist within a particular area of inves-tigation, these models be used and tested forpotential appropriate use in a telehealth con-text. For example, when investigating the im-pact of telehealth on a particular health behav-ior (e.g., medication compliance) one couldframe the research project in the context of theHealth Belief Model (HBM).13 These modelsand their underlying assumptions can be usedto generate testable hypotheses and guide ex-perimental design and analysis. For example,the HBM tries to explain and predict health be-haviors by examining an individual’s attitudesand beliefs. The model has three core assump-tions regarding the likelihood that someonewill engage in a health-related action (e.g., a pa-tient with diabetes will regularly monitor theirblood sugar levels). First the individual mustfeel that a negative health condition can beavoided (e.g., hypoglycemia). Second, theymust have a positive expectation that if theytake action the negative condition will beavoided (e.g., better monitoring will reduce hy-poglycemic episodes). Finally, the individualmust believe that they can successfully carryout the recommended action (e.g., can useblood glucose testing devices with confidence).

The HBM also incorporates six theoreticalconstructs (perceived susceptibility, perceivedseverity, perceived benefits, perceived barriers,cues to action and self-efficacy) that account foran individual’s readiness to act.14 Using thecore assumptions, one could hypothesize that

using telehealth technology to automaticallyprompt patients with diabetes to take a glucosereading, transmit it to a nurse, and receive im-mediate feedback regarding actions to take ifnecessary, would yield more monitoring com-pliance (and hence more stable readings) thanin a group receiving no telehealth support. Thesix theoretical constructs could be used to de-velop surveys pretelehealth and posttelehealthintervention to assess changes in believe struc-tures and how they correlated with changes inbehavior.

SUPPORTING EVIDENCE FORGUIDELINES AND STANDARDS

Currently there are very few standards orguidelines for telehealth at any level, althoughrecent efforts have been aimed at establishingthem.15,16 One recent effort that can serve as amodel is the “Telehealth Practice Recommen-dations for Diabetic Retinopathy” documentdeveloped by the ATA Ocular Telehealth Spe-cial Interest Group (SIG) and the National In-stitute of Standards and Technology (NIST)Working Group.16 This comprehensive docu-ment was developed based on careful reviewof evidence from the literature and examina-tion of clinical practice models. One of the ma-jor goals in producing the document was to im-prove clinical outcomes. It provides extensiverecommendations on such topics as qualifica-tions of personnel, equipment specifications,legal requirements, and quality control mea-sures. The studies supporting the creation ofthe recommendations serve as excellent exam-ples of the type of research needed to createpractice recommendations for other telehealthapplications. The recommendation documentitself provides a framework around which fu-ture studies can be conducted in order to dem-onstrate improved clinical outcomes with tele-health.

When such documents exist, it is highly rec-ommended that they be used to guide researchquestions as well as provide some of the toolsfor designing the research protocol. For exam-ple, the diabetic retinopathy document statesthat “compression may be used if compressionalgorithms have undergone clinical valida-

RESEARCH RECOMMENDATIONS FOR THE ATA 581

tion.”16 This statement clearly lends itself to amultitude of research hypotheses. A studycould be designed to compare diagnostic accu-racy (e.g., detection of microaneurisms) usingoriginal uncompressed images with imagescompression to various levels using the JointPhotographic Experts Group (JPEG) 2000 stan-dard. In order to carry out the study one couldthen refer to the recommendation documentand find that “retinal images used for diagno-sis should be displayed on high-quality moni-tors with a suggested minimum 19-inch diag-onal size.”16 To further standardize the display,one could consult the Digital Imaging andCommunications in Medicine (DICOM) stan-dard regarding calibration of color displays.17

One key point that the diabetic retinopathydocument serves to illustrate with regard to re-search and the development of standards orguidelines is focus. The document addressedrecommendations for telehealth assessment ofdiabetic retinopathy. It did not attempt to es-tablish recommendations for the entire practiceof teleophthalmology, nor did it attempt to es-tablish recommendations for any other type oftelehealth practice or application. It is highlyrecommended that research studies, especiallythose with the goal of supporting or establish-ing standards or guidelines, maintain as muchfocus as possible on a specific clinical problemand task. If a research study lacks focus or doesnot concentrate on a specific clinical problem,one often comes up against the problem of hav-ing too little data on too many disparate issues.Once the techniques are in place to examine aparticular problem and task, they can quitereadily be used or adapted to study other prob-lems and tasks.

KEY RESEARCH AREAS

In addition to the overarching research is-sues discussed above, the panel identified fourareas of investigation that it believes will fur-ther validate and promote the practice of tele-health. These areas are (1) technical, (2) clini-cal, (3) human factors and ergonomics, and (4)economic analyses. Connecting these four ar-eas are three overriding research goals. Thefirst reiterates the importance of using existing

models and methods whenever possible, as al-ready discussed. The second is the encourage-ment of multidisciplinary studies (e.g., tech-nology and interpersonal communication),especially in the context of outcome studies.The third is the integration of telehealth re-search into the body of general healthcare re-search (e.g., health outcomes in rural/under-served regions). The four areas are notpresented in any order of importance since theagenda is meant to be an integrated whole. Theultimate goal, as summarized in the final sec-tion on Economic Analyses, is to tie telehealth re-search questions into the general healthcarecontext and create a better healthcare system ingeneral.

Technical

Research in the technical area is perhaps themost amenable to the future consideration ofstandards and guidelines. This is due, at leastpartially, to the precedents that exist in themost mature telehealth application, that of tel-eradiology. For example, the DICOM standardcontains information regarding numerous as-pects of image acquisition, transfer, storage anddisplay that were originally developed for dig-ital radiology and Picture Archiving and Com-munications Systems (PACS), but are is nowbeing adapted to a number of other medicalimaging applications such as ophthalmology,pathology and dermatology.18 The “TelehealthPractice Recommendations for Diabetic Reti-nopathy” (generated by the Teleophthalmol-ogy SIG of the ATA) document also contains anumber of technical recommendations that canpotentially be extended to other telehealth ap-plications.18

Within the technical area, the panel has iden-tified three priority research topics.

a. Infrastructure definition: Communica-tions technologies will continue to change.Although research should certainly be con-ducted to characterize new technologies asthey emerge and determine their suitabilityfor various telehealth applications, it is moreimportant in the long run to investigate themeans to integrate the various infrastructurecomponents in a secure and seamless man-

KRUPINSKI ET AL.582

ner. The ATA Technology SIG has alreadyidentified compatibility of videoconferenc-ing systems as a specific priority in this areaand panel concurs.19 Although more andmore store-and-forward applications are be-ing approved for reimbursement, real-timevideoconferencing remains the preferredcommunication mode for many telehealthapplications. A second priority is the devel-opment of new or incorporation of existingtechnical and telecommunications standardsinto telehealth to facilitate coordination andcooperation.20

b. Good imaging: We need to examine andcharacterize the entire imaging chain fromacquisition to transmission to display to stor-age, and minimum standards need to be con-sidered at each point. A good model for thisalready exists in teleradiology. The Ameri-can College of Radiology (ACR) has a tech-nical standard for teleradiology that coverseverything from the matrix size and bitdepth for acquiring various types of images(e.g., computed radiography versus com-puted tomography) to the minimum displayluminance to jurisdictional policies for trans-mission and storage.21 The various technicalcomponents outlined in the ACR technicalstandard should be investigated for adoptionby other clinical specialties utilizing tele-health technologies. The one component thatis lacking in the ACR standard is real-timeimaging, but as noted above that has alreadybeen identified as a top priority. To reiteratean important point, research in this areaneeds to focus on particular problems andspecific tasks that relate capabilities and per-formance of the technological devices backto clinically relevant considerations (e.g.,what is the minimum bandwidth required tocarry out an initial psychiatric evaluation ofa patient suspected of having Alzheimer’sdisease).c. Comparisons across platforms: The num-ber and variety of devices available in tele-health expands every year. Many of these de-vices purport to do the same thing—onlybetter than their competitors. More oftenthan not, however, these claims are not sup-ported by independently conducted researchstudies that compare directly the perfor-

mance of various devices or platforms onspecific tasks. For example, there are a num-ber of remote heart monitors available com-mercially that will monitor and transmit to acentral clinical site patient electrocardiogram(ECG) data. Technical studies need to bedone to validate the accuracy and reliabilityclaims of these types of devices using clini-cally accepted gold standards (e.g., standard12-lead ECG). Studies that compare multipledevices using the same clinical protocols anda well-defined cohort of patients are highlyrecommended.

Clinical

There have been a number of studies carriedout in recent years to demonstrate the clinicalutility of using telehealth in a variety of clini-cal applications. Many of these studies havebeen done exceptionally well and in somecases have provided the necessary efficacydata to push forward legislation or institutionof reimbursement codes. Other studies, how-ever, have not been carried out with the samelevel of experimental rigor, reducing their im-pact on the overall push towards the accep-tance and incorporation of telehealth intomainstream healthcare practice.22 Some of themajor problems in this area have been method-ological limitations. Many of the studies todate have examined the concordance betweenthe traditional and telehealth modes of ren-dering a diagnosis, often using � analysis todetermine statistical significance.23 Thesestudies were quite useful and did serve todemonstrate that there often were high levelsof agreement between traditional and tele-health diagnostic decisions. It is time, how-ever, to move beyond simple concordance. The� statistic (and other correlation based meth-ods) are useful but limited because they onlydeal with agreement. They say nothing aboutthe accuracy of the decisions being rendered.If telehealth is going to move beyond being re-garded by many as a tool or a substitute fortraditional healthcare delivery, its ability toimpact diagnosis, treatment options and pa-tient outcomes must be demonstrated usingexperimentally rigorous techniques supportedby appropriate statistics.

RESEARCH RECOMMENDATIONS FOR THE ATA 583

In March 2005, the United States Agency forHealthcare Research and Quality (AHRQ) andthe Centers for Medicare and Medicaid Ser-vices (CMS) convened a workshop to assess theevidence for the efficacy of telehealth services.A series of white papers generated from thisworkshop appears in a supplement of the Jour-nal of Telehealth and Telecare. The panel recom-mends strongly that investigators use this se-ries of papers as an additional source ofguidance and reference for methods to conductclinical research studies in telehealth.24

Experimental design and statistics. The random-ized controlled trial (RCT) is often regarded asthe paragon of clinical trial methods and shouldbe used when it is feasible or practical. How-ever, other accepted trial design methods existthat can be used if an RCT study design is notpossible.25 The AHRQ supplement in the Jour-nal of Telehealth and Telecare contains a compre-hensive review of alternatives to the RCT26 aswell as descriptions of potential sources of ex-isting data that could be used to assess patientoutcomes.26,27 It is highly recommended that al-though some of these experimental designs re-quire more time and effort, telehealth investiga-tors use them in patient studies. In particular,the use of pretest–posttest, interrupted time se-ries, and case-control designs are recommended.Meaningful control/comparison groups mustbe used, and different populations of patients(e.g., gender, age, ethnicity, race) must be in-cluded in the samples to improve the generaliz-ability of the findings.

Sample size is a particularly important issuebecause many studies in the literature sufferfrom poor statistical power (i.e., the probabil-ity of correctly rejecting the null hypothesiswhen it is indeed false) that is generally the re-sult of having too few patients or cases in thestudy. Statistical power is not only affected bythe number of patients in the study, but also bysuch things as the number of readers (i.e., clin-icians rendering a diagnosis), the magnitude ofthe expected difference between conditions orcomparison groups, the variability betweenand within groups, the experimental design(e.g., repeated versus independent measures),and even the type of statistical test (e.g., para-metric versus nonparametric). It is recom-

mended that investigators provide an estimateof power in additional to the more traditionalresults of statistical analysis (e.g., the p value)when publishing or presenting experimentalresults. The basics of experimental design, sta-tistical tests and issues such as power can befound in a host of textbooks,28,29 but it is rec-ommended that researchers less familiar withthese issues collaborate with investigators al-ready experienced in these matters either fromtraditional medicine or related fields (e.g., psy-chology, sociology).

Adequate power calculations are a particu-larly important consideration in the design ofequivalency trials. In some settings, telehealthmay be a superior alternative to conventionalcare, however, superiority over conventionalcare is not necessarily the bar that must bereached before adopting telehealth. Rather,telehealth interventions that yield at leastequivalent diagnostic and clinical outcomes toconventional care insure that telehealth is notdelivering inferior care. If equivalency is es-tablished, other compelling reasons for adopt-ing telehealth such as economic considerationsand human factors may be deciding factors.Power calculations and sample size considera-tions in equivalency trials are of paramount im-portance since the goal is to be confident thatthose trials have a high probability of yieldinga valid answer, should equivalency be found.

As already noted, although agreement stud-ies (those that only consider whether the diag-nosis is the same when evaluating an alterna-tive technology, not whether either of thedoiagnoses are actually correct) have beenquite useful is establishing concordance be-tween traditional and telehealth in areas suchas teleradiology, telepathology and telederma-tology, it is now necessary to move beyondsimple concordance. In order to assess diag-nostic accuracy (in contrast to diagnostic con-cordance), the panel strongly recommendssome of the traditional models and metricsused in traditional clinical studies.30 Ratherthan agreement, metrics such as sensitivity,specificity, accuracy, and positive and negativepredictive value should be utilized. Key tothese types of accuracy studies is the use of anindependent gold standard. A gold standardrefers to the use of an independent source of

KRUPINSKI ET AL.584

information to verify the true diagnosis otherthan the information being evaluated in thestudy. For example, in concordance studies agroup of dermatologists typically examine pa-tients in-person in the traditional manner andalso examine a set of digital photos (displayedon a computer monitor) of the same patients.Diagnoses are compared and percent of casesreceiving the same diagnosis or � as a measureof agreement is calculated. In neither readingmode is it possible to determine if the diag-noses were accurate or correct. In an accuracystudy, an independent gold standard (e.g., bi-opsy results) would serve as the point of com-parison. The diagnoses from the traditional andtelehealth reading modes would each be com-pared to the gold and standard and sensitivity(e.g., how often the biopsy-proven cancer caseswere diagnosed by the dermatologist as can-cer) and specificity (how often the biopsy-proven noncancer cases were correctly diag-nosed as noncancer by the dermatologist)could be calculated for each reading mode.Those values could then be compared statisti-cally to determine if diagnostic accuracy differssignificantly between the two interpretationmodes.

Whenever possible, the methods used to collect and analyze decision data should begrounded in theory or based on a specific the-oretical model. For example, many diagnosticaccuracy studies can be framed using the Sig-nal Detection Theory paradigm and statisti-cally analyzed using receiver operating char-acteristic (ROC) methods.31 Recent advances inROC methods even incorporate traditional sta-tistical methods such as the analysis of variance(ANOVA) into the methods to allow for robustdetermination of differences between experi-mental conditions and accurate estimates ofcase, reader and condition variances.32

Clinical outcomes. A good clinical outcomesstudy starts with a good plan of action and awell-defined set of hypotheses. For many in-vestigators this is often the hardest part and toooften it is ignored. AHRQ has developed anEvaluation Toolkit to guide investigatorsthrough the early steps of designing a study toexamine the impact of Health InformationTechnology (HIT), and parts it address directly

clinical outcome parameters.33 The documentalso provides sample measures for six key re-search domains: (1) clinical outcomes measures(e.g., complication rates, adverse drug reac-tions), (2) clinical process measures (e.g., med-ication errors, percent alerts resulting in de-sired outcome), (3) provider adoption andattitude measures (e.g., staff turnover rates, jobsatisfaction), (4) patient knowledge and atti-tude measures (e.g., satisfaction, knowledge ofmedications), (5) workflow impact measures(e.g., time spent per patient, waiting times), and(6) financial impact measures (e.g., percentclaims denied, cost of forms). The panel rec-ommends strongly that well-defined outcomesmeasures such as those defined in the AHRQtoolkit be used in studies to assess the impactof telehealth on patient outcomes.

Once hypotheses are formed and specificoutcome measures identified, we can begin toaddress the question of whether telehealth im-proves the quality of care received by patientsfrom a number of perspectives including struc-ture and process in additional to strict healthoutcomes. The issues noted in the above sec-tion on Experimental Design and Statisticsshould all of course be considered in the con-duct of clinical outcomes studies. In addition,the panel also has identified a number of otherissues that need to be included in outcomes research whenever possible. The accuracy,friendliness and utility of telehealth devices(e.g., electronic stethoscope) can often be as-sessed in the laboratory setting and the resultsgeneralized reliably to most real-life settingsand applications. Clinical outcomes studies,even though proper control and comparisonfactors need to be considered, are more likelyto suffer from ecological validity problems ifthey are only carried out in the laboratory uni-versity hospital-based setting. Ecological va-lidity deals with the issue of whether the re-sults from a study done in a closed andcontrolled environment (typically the labora-tory) can be generalized to the real-world en-vironment. Outcomes studies in telehealth cancertainly start in the laboratory or universitysetting, but they must start to extend out to theactual settings and patients that will ultimatelybenefit the most from telehealth—rural and underserved communities. Community-based

RESEARCH RECOMMENDATIONS FOR THE ATA 585

outcomes studies must be carried out and thecharacteristics that are common across com-munities as well as those cultural factorsunique to each community and culture need tobe examined as correlative variables.

Finally, there are two other recommenda-tions from the panel regarding outcomes stud-ies that given the relatively short history of tele-health are only now becoming possible. Thefirst is that longitudinal studies need to be con-ducted in order to assess the long-term impactthat telehealth has on healthcare. Outcomes forcomplex diseases and health conditions oftentake time and numerous interventions beforeeven moderate changes are seen in the patient’shealth status. Short-term, single interventionstudies are unlikely to demonstrate any signif-icant differences in outcomes between tradi-tional and telehealth processes. Longitudinalstudies that follow cohorts of patients over sig-nificant yet realistic periods of time should becarried out. In order to facilitate longitudinalstudies and address other important issuesraised above such as increased sample size andthe need to investigate outcomes within thecontext of cultural influences, the panel alsorecommends more multi-institution collabora-tions be formed. For both of these to happen,however, the telehealth community needs topressure the funding agencies to sponsor thesetypes of investigations.

Human factors and ergonomics

Many of the issues discussed previouslyclearly have aspects that fall under the rubricof human factors and ergonomics research, butthe panel wishes to point out some specificpoints that have not been discussed in this re-gard. As with the previous two areas, it is im-portant when doing human factors research toutilize methods and theories that already exist.This is especially true when assessing tele-health technologies from the user’s perspective.There are well-known techniques for designingand assessing user interfaces that can be ap-plied to the majority of digital devices being in-troduced into the telehealth market.34,35 Thebroader role of human factors research in e-health applications has also been addressed. Avariety of texts exist that detail telehealth spe-

cific methods and these should be consulted.36

As with the previous areas, study design andproper statistical analyses need to be consid-ered when conducting human factors research.

Usability studies for specific populations. In ad-dition to simply determining whether a partic-ular telehealth device or technology does whatit is supposed to do, it is important to carry outusability studies with different populations ofusers. With the recent focus on home health-care and consumer-oriented telehealth applica-tions, it is important to consider the fact thatthe potential users of these new technologieswill be very different from each other. It is rec-ommended strongly that different device de-signs be considered and tested for differentpopulations. For example, portable monitorsthat record heart rate information and displaythe results to the user as well as transmit themto a central site for evaluation and interventionmay need to be modified for certain popula-tions. Older users may require larger visuals,bigger input buttons, and fewer process steps(i.e., more automated) than younger users if thedevice is to be accepted and used appropri-ately. If the device has an auditory component(e.g., an alarm indicating it is time to take amedication or test blood glucose levels), it mayneed to have adjustable settings to account forage-related declines in hearing. Testing and tai-loring devices to specific user populations (pa-tients and healthcare providers) will contributesignificantly to reducing technophobia amongpotential users. Carefully designed trainingprograms that not only take into account theusers’ physical and intellectual capabilities butalso their culture and language must also bedeveloped.

Process and design factors. It is not only im-portant to consider the usability and ergonom-ics of specific devices and technologies in tele-health, but the broader issue of how to integratetelehealth and its technologies into the health-care system and public use must also be con-sidered.37,38 Understanding the process of tele-health better is a priority. In the clinical settingwe should consider the impact that introduc-tion of a telehealth application has on work-flow and the number of people incorporation

KRUPINSKI ET AL.586

of the telehealth process would impact. Inter-ruptions in workflow and distraction of peoplefrom their regular duties have often been citedas a barrier to implementation of telehealth, notonly on the provider side but also on the pa-tient side. Most of these conclusions howeverare based on anecdotal accounts or subjectiveimpressions of how a telehealth intervention ortechnology impacts the current way of doingthings. Telehealth needs to carry out objectivestudies using established techniques fromother domains where technology and work-flow have been studied extensively. For exam-ple, the aviation industry has a long history ofevaluating the impact that the introduction ofnew technology has on a pilot’s ability to nav-igate his/her way through a plane’s cockpitand fly the plane without incident. Tools existfor such varied aspects of workflow in a tech-nological environment such as time-motionanalysis, workload assessment, link analysis,and behaviorally based rating tools.39 To alarge extent we need to determine whetherhealthcare in general should be reengineeredwith telehealth technologies so that telehealthis no longer viewed as a tool or something sep-arate from mainstream healthcare.

Economic analyses

This is one of the biggest issues we have inthe telehealth and healthcare in general to-day.40 There is very little sophistication in theeconomic analyses of telehealth to date. A lotof the research is “surface” at this point. Thatis not bad, in that we are really just getting ahandle on this field and it is developmentallyappropriate. An initial consideration is appro-priately framing the question that needs to bestudied. For example, the economic perspectivethat one wishes to study should be stated (i.e.,societal, healthcare system, etc.). The study de-sign should parallel the question posed andavailable data. If one is interested in determin-ing the value consumers place on a telehealthintervention then a willingness to pay studydesign is appropriate. If equivalent outcomesare proven or assumed then a cost-minimiza-tion is appropriate; a cost-effectiveness or costutility analysis is appropriate if incrementaldifferences in cost and effectiveness are pres-

ent; and a cost-benefit analysis should be usedif the outcomes of competing interventions aremonetized.

Economic analyses also require a completeand accurate accounting of cost elements. Intelehealth there are a specific cost considera-tions that bear mention. The direct costs (i.e.,market cost) of technology is not the only con-sideration. Other factors such as depreciation,changing price structures, and cost attributionshould be considered. A particular technologymay or may not be used solely for the telehealthstrategy under study and part of that cost maybe offset or borne by other interventions. Ad-ditionally, some technology may be part of theoverall infrastructure of a healthcare systemand may be considered a “sunk” cost (i.e., notdirectly attributable to the telehealth interven-tion.) Telehealth typically averts at least a por-tion of patient and or provider travel that maynormally occur in face-to-face visits. The timeand opportunity cost considerations should bepart of telehealth evaluations. Thus, we needmethodologies and statistics that are designedfor the type of multi-site, longitudinal, com-plex, dynamic system that we know telehealthis. Fortunately, as with the other areas ad-dressed in this report, there exist methods andmodels for the general economic evaluation ofhealthcare that can be adopted for use in tele-health.41,42 For cost-effectiveness analyses, therecommendations of the Panel on Cost-effec-tiveness in Health and Medicine can also beadopted for use in telehealth research. Asidefrom incorporating more robust analysis tech-niques into the field of telehealth, the panel hasidentified two priority areas.

a. What are the short- and long-term im-pacts of telehealth on healthcare costs? Aswith clinical outcomes studies, to date it hasbeen easier to study the short-term impactsof telehealth and long-term effects haverarely been looked at. Again this has more todo with the fact that most telehealth pro-grams have simply not been operational longenough to even consider long-term analyses.However, there are some programs that havebeen operational for a decade or more andthe opportunity now arises to make theseanalyses possible. We need to shift the focus

RESEARCH RECOMMENDATIONS FOR THE ATA 587

of economic analyses from infrastructure,communications and technology costs tomore complex issues such as will telehealthreduce patient costs to the overall healthcaresystem. Although this seems incrediblybroad, the recommendation made previ-ously regarding focus on a specific problemis appropriate here as well and the flow fromtechnology to ergonomics to clinical out-comes to economic evaluation becomes im-portant.

For example, once we have a piece oftechnology (e.g., remote blood glucosemonitor that transmits readings automati-cally to a central healthcare center for eval-uation and possible intervention) that hasbeen validated and shown to ergonomicallyoptimized, we can conduct a clinical out-comes investigation to determine if patientswith the remote monitor and feedback arebetter able to maintain their blood glucoselevels and avoid visits to the emergencyroom for severe hypoglycemic events (clin-ical outcomes). Economic analyses couldthen examine the long-term impact that im-proved diabetes management via telehealthhas on avoiding or delaying patient dis-ability and the cost this has to society (e.g.,earlier, hence more, Social Security pay-ments to patients with diabetes who are un-able to work because of complications).Economic modeling and decision analytictechniques allow for an analysis of bothshort-term and long-term economic out-comes.

b. Creation of better healthcare businessand management models. How can tele-health and more generally telehealth, whencombined with health information technolo-gies be used to create better models of health-care where better is defined as equivalent orhigher quality, improved patient safety andequivalent or lower per unit costs than thecurrent healthcare system? This is a complexand difficult question that may not be easilyaddressed without more clinical outcomesdata regarding the long-term impact thattelehealth has on healthcare in general. How-ever, the panel feels that it is important to

raise the possibility of addressing this goalbecause we cannot limit ourselves to demon-strating that telehealth is a viable substitutefor or alternative to traditional healthcaremethods.

SUMMARY

Telehealth is going to transform healthcare,from the way it is delivered to the way it is paidfor. Anyone who interacts with the healthcaresystem in the coming years will be impacted bytelehealth. Those within the telehealth fieldneed to have the foresight to take telehealth be-yond its current state and make it an integralpart of healthcare practice around the world. Itis the hope of the panel that the research rec-ommendations put forth in this document willgive investigators the inspiration, tools andgoals to make this happen.

REFERENCES

1. United States General Accounting Office. Telehealth:federal strategy is needed to guide investments. Report toCongressional Requesters. GAO/NSIAD/HEHS-97-67. Washington D.C.: General Accounting Office, 1997Unites States General Accounting Office, Washing-ton, DC; February 1997.

2. Hersh WR, ed. Telehealth for the Medicare population.Evidence Report/Technology Assessment No. 24.Rockville, MD: Agency for Healthcare Research andQuality, 2001.

3. Hailey D, Roine R, Ohinmaa A. Systematic review ofevidence for the benefits of telehealth. J Telemed Tele-care 2002;8:1–7.

4. Hersh W, Helfand M, Wallace J, Kraemer D, Patter-son P, Shapiro S, Greenlick M. A systematic reviewof the efficacy of telehealth for making diagnostic and management decisions. J Telemed Telecare 2002;8:197–209.

5. Hersh WR, Helfand M, Wallace J, Kraemer D, Patter-son P, Shapiro S, Greenlick M. Clinical outcomes re-sulting from telehealth interventions: A systematic re-view. BMC Med Inf Decis Mak 2001;1:5.

6. Demiris G, Tao D. An analysis of the specialized lit-erature in the field of telehealth. J Telemed Telecare2005;11:316–319.

7. Strauss A, Corbin J. Basics of qualitative research: tech-niques and procedures for developing Grounded Theory.Thousand Oaks, CA: Sage Publications, 1998.

8. Charmaz K. Grounded Theory: Objectivist and con-structivist methods. In: Denzin NK, Lincoln YS, eds.

KRUPINSKI ET AL.588

Strategies of qualitative inquiry. Thousand Oaks, CA:Sage Publications, 2003:249–291.

9. Salkind NJ. Tests & measurement for people who (thinkthey) hate tests & measurement. Thousand Oaks, CA:Sage Publications, 2006.

10. Hayes BE. Measuring customer satisfaction: Survey de-sign, use, and statistical analysis methods. Milwaukee,WI: ASQC Quality Press, 1997.

11. Crano WD, Brewer MB. Principles and methods of so-cial research. Mahwah, NJ: Lawrence Erlbaum Associ-ates, 2002.

12. World Health Organization Quality of Life Instruments:www.who.int/evidence/assessment-instruments/qol (Last accessed December 20, 2005.

13. Rosenstock I. Historical origins of the Hhealth BeliefModel. Health Education Monographs. Vol. 2, No. 4,1974.

14. Glanz K, Marcus Lewis F, Rimer BK. Theory at a glance:A guide for health promotion practice. National Institutesof Health, 1997.

15. www.usms.nist.gov/workshops/MedicalImaging.htm Last accessed January 15, 2006.

16. www.atmeda.org/ICOT/diabeticretionpathy.FINAL.pdf Last accessed January 16, 2006.

17. www.medical.nema.org/medical/dicom/final/sup100_ft.pdf Last accessed January 15, 2006.

18. www.medical.nema.org/ Last accessed January 5,2006.

19. www.atmeda.org/ICOT/sigtechnology.htm Last ac-cessed December 20, 2005.

20. Yellowlees P, Harry D. Standards for data collectionand monitoring within a telehealth research network.J Telemed Telecare (in press).

21. American College of Radiology: www.acr.org/s_acr/bin.asp?CID=541&DID=12292&DOC=FILE.PDF Lastaccessed December 6, 2005.

22. Hersh WR, Hickam DH, Severance SM, Dana TL,Krages KP, M Helfand M. Diagnosis, access, and out-comes: Update of a systematic review on telehealthservices. J Telemed Telecare (in press).

23. Cohen J. A coefficient of agreement for nominalscales. Educ Psychol Measure 1960;20:37–46.

24. Hersh WR, Hickam DH, Erlichman M. Advancing theEvidence Base of Telehealth: Overview of supple-ment. J Telemed Telecare (in press).

25. Harris AD, McGregor JC, Perencevich EN, Furuno JP,Zhu J, Peterson DE, Finkelstein J. The use and inter-pretation of quasi-experimental studies in medical in-formatics. JAMIA 2006;13:16–23.

26. Grigsby J, Bennett RE. Alternatives to randomizedcontrolled trials in telehealth. J Telemed Telecare (inpress).

27. Whitten P. Will we see data repositories for telehealthactivity in the near future? J Telemed Telecare (in press).

28. Ingelfinger JA, Mosteller F, Thibodeau LA, Ware JH.Biostatistics in clinical medicine. New York, NY: Mc-Graw Hill, Inc., 1994.

29. Mould RF. Introductory medical statistics. Philadelphia,PA: Institute of Physics Publishing, 1998.

30. Swets JA. Signal detection theory and ROC analysis inpsychology and diagnostics: Collected papers. Mahway,NJ: Lawrence Erlbaum Associates, 1996.

31. Swets JA, Pickett RM. Evaluation of diagnostic systems:Methods from Signal Detection Theory. New York, NY:Academic Press, 1982.

32. Dorfman DD, Berbaum KS, Metz CE. Receiver oper-ating characteristic rating analysis: Generalization tothe population of readers and patients with the jack-knife method. Invest Radiol 1992;27:723–731.

33. Agency for Healthcare Research and Quality:www.healthit.ahrq.gov/home/index.html Last ac-cessed January 4, 2006.

34. Shneiderman B. Designing the user interface: Strategiesfor effective human-computer interaction. Reading, MA:Addison-Wesley, 1998.

35. Vicente K. The human factor: Revolutionizing the waypeople live with technology. New York, NY: Routledge,2003.

36. Bangert DC, Doktor R. Human and organizational dy-namics in e-health. Seattle, WA: Radcliffe Publishing,2005.

37. Shea S. What health delivery system changes are re-quired when integrating telehealth into existing treat-ment flows of information and patients.” Workflow,productivity, resources. J Telemed Telecare (in press).

38. Stronge AJ, Nichols TA, Rasche JD, Sessions RG, FiskAD, Rogers WA. An application of human factorsmethods to teledermatology. In: Bangert DC, DoktorR, eds. Human and organizational dynamics in e-health.Seattle, WA: Radcliffe Publishing, 2005:13–31.

39. Federal Aviation Commission, Human Factors Tool-box: www.hf.faa.gov/Portal/toolsbytypetally.aspxLast accessed January 15, 2006.

40. Tunis SR. Economic analysis in healthcare decisions.Am J Managed Care 2004;10:301–304.

41. Drummond M, McGuire A. Economic evaluation inhealthcare: Merging theory with practice. New York, NY:Oxford University Press, 2002.

42. Gold MR, Siegel JE, Russell LB, Weinstein MC, eds.Cost-effectiveness in health and medicine. New York, NY:Oxford University Press, 1996.

Address reprint requests to:Elizabeth Krupinski, Ph.D.

Department of RadiologyThe University of Arizona

Health Sciences Center1609 N. Warren, Building 211, Room 112

P.O. Box 245067Tucson, AZ 85724

E-mail: krupinski@radiology.arizona.edu

RESEARCH RECOMMENDATIONS FOR THE ATA 589

This article has been cited by:

1. Pauline A. Mashima , Charles R. Doarn . 2008. Overview of Telehealth Activities in Speech-Language PathologyOverview ofTelehealth Activities in Speech-Language Pathology. Telemedicine and e-Health 14:10, 1101-1117. [Abstract] [PDF] [PDF Plus]

2. Bonnie J. Wakefield, Lindsay J. Orris, John E. Holman, Cynthia L. Russell. 2008. User Perceptions of In-Home MedicationDispensing Devices. Journal of Gerontological Nursing 34:7, 15-25. [CrossRef]

3. Diane M.F. Mahoney , Phyllis H. Mutschler , Barbara Tarlow , Ellen Liss . 2008. Real World Implementation Lessons andOutcomes from the Worker Interactive Networking (WIN) Project: Workplace-Based Online Caregiver Support and RemoteMonitoring of Elders at HomeReal World Implementation Lessons and Outcomes from the Worker Interactive Networking(WIN) Project: Workplace-Based Online Caregiver Support and Remote Monitoring of Elders at Home. Telemedicine and e-Health14:3, 224-234. [Abstract] [PDF] [PDF Plus]

4. Elizabeth A. Krupinski, Yulei Jiang. 2008. Anniversary Paper: Evaluation of medical imaging systems. Medical Physics 35:2, 645.[CrossRef]