P

FADARA

AThtcfpuleofi“sdfiap

ANcBdrBaGtdcTltC

Mt7s

A

1

RESEARCH

erspectives in Practice

ormative Research of a Quick List for anutomated Self-Administered 24-Hourietary Recall

MY F. SUBAR, PhD, MPH, RD; FRANCES E. THOMPSON, PhD, MPH; NANCY POTISCHMAN, PhD; BARBARA H. FORSYTH, PhD;ICHARD BUDAY; DEBBIE RICHARDS, MS; SUZANNE MCNUTT, MS, RD; STEPHEN G. HULL, MS; PATRICIA M. GUENTHER, PhD, RD;

RTHUR SCHATZKIN, MD, DrPH; TOM BARANOWSKI, PhD

“sanfdmacvvctiJ

Dsa

mitertanbrlTrttf

rtcHFR

BSTRACTwenty-four�hour dietary recalls are used to collectigh-quality dietary data. Because they require highlyrained interviewers, recalls are expensive and impracti-al for large-scale nutrition research, leading to the use ofood frequency questionnaires. We are developing a com-uter-based, self-administered 24-hour dietary recall forse by adults. Our goal is an easy-to-use, low-cost, pub-

icly available, Web-enabled instrument that will includelements of the Automated Multiple Pass Method devel-ped by the US Department of Agriculture, which usesve passes to enhance recall. The initial pass is called thequick list” and allows respondents to report foods con-umed the previous day freely, in any order, and withoutetail. Using a crossover design, we conducted initialormative pilot testing among 18 adults in a self-admin-stered computer environment. We tested two versions of“quick list” (the first Automated Multiple Pass Method

ass) for remembering foods consumed the previous day:

. F. Subar is a nutritionist and F. E. Thompson and. Potischman are epidemiologists at the National Can-

er Institute, Risk Factor Monitoring and Methodsranch, Bethesda, MD. B. H. Forsyth is a senior studyirector, S. McNutt is a nutritionist, and S. G. Hull is aesearch nutritionist, all at Westat, Rockville, MD. R.uday is president, Archimage, Houston, TX. D. Rich-rds is president, Animatrix Arts, Houston, TX. P. M.uenther is a nutritionist, US Department of Agricul-

ure, Center for Nutrition Policy and Promotion, Alexan-ria, VA. A. Schatzkin is an epidemiologist and branchhief, Nutritional Epidemiology Branch, Bethesda, MD.. Baranowski is a professor of pediatrics, Baylor Col-

ege of Medicine, US Department of Agriculture/Agricul-ural Research Service, Children’s Nutrition Researchenter, Houston, TX.Address correspondence to: Amy F. Subar, PhD,PH, RD, National Cancer Institute, Risk Factor Moni-

oring and Methods Branch, 6130 Executive Blvd, MSC344, EPN 4005, Bethesda, MD 20892-7344. E-mail:ubara@mail.nih.gov

Copyright © 2007 by the American Dieteticssociation.0002-8223/07/10706-0007$32.00/0

sdoi: 10.1016/j.jada.2007.03.007

002 Journal of the AMERICAN DIETETIC ASSOCIATION

unstructured” and “meal-based.” Respondents showed atrong preference for the meal-based version (13 of 18),lthough positive features of both were identified. Chro-ological reporting was most common, although manyoods were sporadically reported out of order. Versionsid not appreciably differ in number of foods reported,oved, or deleted. Usability issues and preferences were

lso identified. If these developmental efforts prove suc-essful, the use of affordable automated recalls could bealuable in clarifying diet-disease associations in obser-ational epidemiologic studies and measuring dietaryompliance in clinical trials. This pilot work illustrateshe usefulness of formative cognitive and usability test-ng for questionnaire and software development.

Am Diet Assoc. 2007;107:1002-1007.

ietary assessment methods, commonly used to mea-sure food and nutrient intakes for dietary surveil-lance, epidemiologic, intervention, and clinical re-

earch, rely on self-reports (1). Accurate self-reporting isprimary goal in dietary methods research.The 24-hour dietary recall is considered an optimalethod for collecting dietary data and is commonly used

n dietary surveillance. Dietary recalls conducted byrained interviewers do not require that subjects be lit-rate, and, to the extent that respondents can remember,ecalls can provide detailed information on food prepara-ion, brand names, and portion sizes (1). When the datare collected for the previous 24 hours, without priorotice, the method does not affect food choices on the dayefore the interview. However, 24-hour dietary recallsequire administration by trained interviewers and col-ection of more than 1 day to assess usual intake (2,3).hus, they are not considered economical or practical inesearch settings with large samples. A 24-hour recallhat can be self-administered, using computer technologyo substitute for the interviewer, could make the methodeasible for large-scale studies.

A computer-based, self-administered 24-hour dietaryecall system, the Food Intake Recording Software Sys-em developed by researchers at Baylor College of Medi-ine (Houston, TX) and Archimage (a private company inouston, TX), has been under development for children.ood reports from an early version of the Food Intakeecording Software System were compared with food ob-

ervation and agreement was similar to that attained

© 2007 by the American Dietetic Association

feSigadppfs

riticc(bPtfoDtt

tlcdralsoatlmpas

RWplastTm

ieStwnai

fifmbaawpC

DTb1bs(patcfoStcswuagv

pwt

rom traditional interviewer-administered 24-hour di-tary recalls (4). The Food Intake Recording Softwareystem used multimedia computer technology to query

ntake of foods over the past day. Child-friendly animateduides read questions aloud, prompted respondents fornswers, and moved across the screen to assist respon-ents through tasks. The current version included ap-roximately 7,000 photographs of individual foods, de-icting up to four portion sizes per food. Users could selectrom these pictures to report their individual portionizes.The National Cancer Institute, in collaboration with

esearchers at Baylor, Archimage, Westat (a private firmn Rockville, MD), and the US Department of Agricul-ure’s Center for Nutrition Policy and Promotion, is lead-ng an effort to modify this 24-hour dietary recall appli-ation for use by adults. The goals are to (a) adapt theurrent child-friendly agents and screens to adult tastes,b) add multiple passes and probes similar to those usedy the interviewer-administered Automated Multipleass Method and used in the National Health and Nutri-ion Examination Surveys since 2002 (5), (c) expand theood database to include all foods in the US Departmentf Agriculture’s current Food and Nutrient Database forietary Studies (6), (d) add more adult-specific food pho-

ography, and (e) provide easy Web access to the applica-ion at minimal or no cost to users.

The Automated Multiple Pass Method uses five passeso enhance recall (5). The initial pass is called the “quickist.” This allows respondents to report freely the foodsonsumed the previous day, in any order, and withoutetail. Interviewers record these foods but use judgmentegarding recording detailed information, brand names,nd additions because such information is collected in aater pass. How this first pass might be adapted in aelf-reported automated interview environment was onef the first aspects of development to be tested. Thisrticle presents qualitative information from cognitiveesting of two potential strategies for designing a quickist for an automated self-administered recall. This for-

ative research exemplifies the interface between therofessional judgment of developers and the perspectivend understanding of respondents in questionnaire andoftware development.

ECRUITMENT AND DESIGNe recruited a convenience sample of 18 adult partici-

ants, 18 to 77 years of age. Participants had varyingevels of age, education, and computer experience; liter-cy at the sixth-grade level was required. By design, theample was evenly divided by sex, and an effort was madeo recruit subjects among different racial/ethnic groups.able 1 shows the demographic characteristics of the nineen and nine women who participated.Prior to data collection, each participant signed an

nformed consent. Each participant completed two differ-nt quick lists regarding intake for the previous day.ubjects came to a study center and completed each of thewo computer-based recall quick lists (described below) 1eek apart. Participants were divided into two groups ofine people each, using a crossover design for order ofdministration. Participants were observed by a cognitive

nterviewer and another observer who recorded prespeci- o

ed attributes of reporting: names of foods, number ofoods, order of reporting, and number of foods deleted oroved. After a session, the cognitive interviewer de-

riefed the participant, gathering information about us-bility, difficulties, and preferences. Sessions were audio-nd videotaped and lasted 1.5 to 2 hours. Participantsere paid $30 for each interview. This study was ap-roved by institutional review boards at the Nationalancer Institute and Westat.

ESCRIPTION OF QUICK LIST APPLICATIONSwo quick lists were tested: an unstructured and a meal-ased version. Screen shots of each are shown in FigureA and B (unstructured) and Figure 2A and B (meal-ased). Both used the food list and search engine de-igned for the Food Intake Recording Software System4). However, to focus specifically on quick list ap-roaches, the applications made no use of multimediattributes (colors, pictures, animated agents). The singleask for both versions was finding and selecting foodsonsumed on the previous day using the food list providedrom the Food Intake Recording Software System (basedn the Continuing Survey of Food Intakes by Individualsurvey Nutrient Database, 1994-96 [7]). In both applica-ions, participants could find foods either by typing orlicking. If participants typed in a food, they were pre-ented with a list of foods by the search engine fromhich to select. When participants chose clicking, theysed a mouse to browse through lists of food categoriesnd subcategories from which they selected. Prior to be-inning either application, a hard-copy tutorial was pro-ided to participants.The unstructured quick list (Figure 1A and B) provided

articipants with open space for recording foods usinghatever strategies or order came to mind. After a par-

icipant finished recording all foods consumed the previ-

Table 1. Demographic characteristics of 18 subjects participatingin formative testing of a Web-based 24-hour dietary recall

Demographic group n

SexFemale 9Male 9Age group (y)18-29 530-49 650-69 670� 1Race/ethnicityWhite 6African American 2Latino 4Asian 6EducationHigh school 2Some college 4College� 12

us day, five eating-occasion columns appeared (break-

June 2007 ● Journal of the AMERICAN DIETETIC ASSOCIATION 1003

F

1

igure 1. Screen shot examples of an unstructured quick list for a Web-based 24-hour dietary recall.

004 June 2007 Volume 107 Number 6

F

igure 2. Screen shot examples of a meal-based quick list for a Web-based 24-hour dietary recall.

June 2007 ● Journal of the AMERICAN DIETETIC ASSOCIATION 1005

ftc

pfitwtPa

sp“ipdwsIo

ewg

RPlopppmtasIaam

rctta

bpsitHm

imttttmqcdnfpi

CTcwtdMfiaundbttttpmbsua

14.8

1

ast, lunch, dinner, snack, just a drink). Participants werehen directed to move each food into the appropriateolumn, later adding or moving foods as needed.

The meal-based quick list (Figure 2A and B) presentedarticipants with the five eating-occasion columns on therst screen, and participants entered foods directly intohe appropriate column. Participants could record food inhatever order they chose by selecting any meal at any

ime; however, the default initial setting was “Breakfast.”articipants could also move foods between columns ordd or delete foods at any time.When participants were finished, they were asked a

eries of questions identical to those asked in the secondass of the Automated Multiple Pass Method system, theforgotten foods” pass, which began, “Your answers aremportant, so we would like this list to be as complete asossible. In addition to the foods you’ve already recorded,id you have . . . ?” Food groups inserted here, in order,ere nonalcoholic and alcoholic beverages, sweets, salty

nacks, fruits/vegetables/cheese, and breads/rolls/tortillas.f the answer was “yes,” they were directed to add an-ther food.Although our primary purpose was assessing prefer-

nces between a meal-based and unstructured quick list,e also collected information offered by participants re-arding software usability and preferences.

ESULTS OF OBSERVATION AND DEBRIEFINGarticipants were able to learn both versions of the quick

ist. Of the 18 participants, however, regardless of orderf administration, 13 (72%) preferred the meal-based ap-roach, three the unstructured approach, and two had noreference. During the debriefing, most participants re-orted features that they liked about both versions. In theeal-based version, participants liked using meals as a

ool for organizing recall as well as for memory cues. Theylso liked using a single screen for all tasks; however,ome participants felt constrained by the meal columns.n the unstructured quick list, participants liked the sep-ration of the recall and sorting food by meal tasks. Theylso reported that the unstructured quick list allowedore freedom to select, adapt, and mix recall strategies.Table 2 shows comparisons of the number of foods

eported, moved, and deleted by quick list version. Nolear differences emerged. For both versions, however,he second interview yielded more food reports. Of 542otal foods reported in both interviews, only two were

Table 2. Comparison of number of foods reported, moved, and deletdietary recall

No. of foods

Interview #1

Meal-based

4™™™™™™™™™™™™™™™™™™™™™Initially reported 14.7Moved at any point 0.3Deleted at any point 1.2Reported at completion 13.4

dded in the “forgotten foods” pass. b

006 June 2007 Volume 107 Number 6

Participants generally reported foods chronologically,ut some foods, such as beverages, were routinely re-orted nonchronologically. Both versions elicited a fewystematic nonchronological orders in reporting based ondiosyncratic memory strategies. For example, one par-icipant seemed to structure his recall by food categories.e started by recalling all the breads he ate, and thenoved on to recall foods in other categories.Participant observation and comments during debrief-

ng identified several usability issues. Participants used aix of typing and browsing to find food selections—nei-

her was clearly preferable. There was a clear preference,hough, for organizing screens from left to right ratherhan from top to bottom, as was the case in both applica-ions tested. Participants suggested including an auto-ated tutorial and help section, designing a system re-

uiring fewer clicks to find or move foods, including aopy feature for foods consumed more than once in a day,ecreasing scrolling and white space, listing multipleames for some foods (eg, include “pop” and “soda” in theood list), including brand names (eg, “Coke” and “Pepsi”),roviding additional flexibility for searching, and creat-ng food categories that were more cognitively salient.

ONCLUSIONShis pilot work illustrates the usefulness of formativeognitive and usability testing for questionnaire and soft-are development. It provided rich qualitative informa-

ion for creating an automated self-administered 24-hourietary recall for adults that emulates the Automatedultiple Pass Method used in national surveillance. The

rst pass in that system is the quick list, conceptualizeds a “mind dump,” allowing participants to recall freely,sing memory cues as they come to mind. Previous cog-itive research revealed that individuals remember inifferent ways and that once they begin the process it isest to allow them to continue uninterrupted, returningo collect details later (8). The best way to operationalizehis in an interviewer vs computer-based self-adminis-ered setting, however, is not clear. This pilot test showedhat most participants preferred a more structured ap-roach that provided a visual representation of meals asemory cues. Although we recognize that offering flexi-

ility is an important usability principle in software de-ign (9,10) and that it would be advantageous to allowsers to record foods in various ways, development costsnd time to complete the interview may limit this flexi-

interview and two versions of a quick list for a Web-based 24-hour

Interview #2

uctured Meal-based Unstructured

™™™™™™™™™™™ Mean ™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™™317.3 16.30.4 0.10.7 0.7

16.7 16.7

ed by

Unstr

™™™™16.30.81.6

ility.

fiqIaoimm

fMwsfsosfcigtkba

Wepo2smpcmtietcamttaTttt

fnsamltb

dbAtespajUei

R

1

1

The finding that second administrations yielded moreood reports likely reflected participants’ learning curven using the applications. This suggests a possible se-uence effect that should be evaluated in future research.n contrast, multiple administrations of recalls, records,nd food frequency questionnaires often show declinesver time in number of foods reported and caloric intakes,ndicating participant fatigue or conditioning (1). It re-

ains to be seen if multiple administrations of an auto-ated recall system will reflect such declines as well.Surprisingly, the “forgotten food list” questions yielded

ew additional foods. In the Automated Multiple Passethod, this pass is considered an important means byhich additional foods are reported (11). However, ob-

ervers noticed that seeing food categories and individualoods on a screen during the primary reporting activityeemed to act as a trigger for participants in recallingther foods consumed the previous day. For example,eeing a “beverage” category while searching for anotherood prompted participants to remember a soda or coffeeonsumed. It is possible that scrolling through a food listn a self-administered system accomplished what the for-otten-food pass does in an interviewer-administered sys-em. Cognitive research is necessary to establish theinds of prompts needed to aid participants in remem-ering as many foods as possible in a self-administeredutomated recall.Our goal is to develop a user-friendly, widely available,eb-based, self-administered, automated 24-hour di-

tary recall that is engaging and flexible and that canrovide nutrient/food group estimates similar to thosebtained from a standardized interviewer-administered4-hour dietary recall. Our plan is that this Web-basedelf-administered 24-hour dietary recall will includeulti-level food lists with recognizable food terminology,

robes that are conceptually consistent with those in-luded in the Automated Multiple Pass Method, variouseans by which participants can select foods, the ability

o type in foods not found on the food list, the ability fornvestigators to add or delete foods, and automatic nutri-nt/food group calculations. We aim to create a systemhat is easy to use (maybe even fun!) and that can beompleted by those with at least a sixth-grade educationnd with limited computer experience. We hope to mini-ize frustration and keep participants engaged over mul-

iple assessments. In addition, our plans include transla-ion into Spanish for multiple Hispanic subpopulationsnd adaptability for translation into other languages.his test of these quick lists excluded many of the mul-imedia attributes we plan to include. Nevertheless, par-icipants provided important usability issues/sugges-

ions, such as use more color, allow copy-and-paste

unctions, include click-and-drag functions, include brandames in the food list, and include tutorial and/or helpcreens. It is clear that the food list must use terms thatre simple, understandable, easy to find, and placed inore than one category as necessary. One person might

ook for rice in grains, while another might look in vege-ables. Providing flexibility when searching for foods wille paramount in the design.In summary, this small cognitive test of a computerized

ietary recall revealed a strong preference for a meal-ased version that will be reflected in the final design.dditional development will continue to include similar

argeted, small, cognitive, and usability testing to assessmerging concerns in the development process. Profes-ional judgment and understanding of questions or com-uter use is clearly only the first step in development ofny questionnaire or software. The respondent is the finaludge of whether an instrument is user-friendly or not.ltimately, for a computer-based software program, us-

rs will vote with their mouses—easily breaking off thenterview if the task is too difficult or frustrating.

eferences1. Thompson FE, Subar AF. Dietary assessment methodology. In: Coul-

ston AM, Rock CL, Monsen ER, eds. Nutrition in the Prevention andTreatment of Disease. San Diego, CA: Academic Press; 2001:3-30.

2. Tooze JA, Midthune D, Dodd KW, Freedman LS, Krebs-Smith SM,Subar AF, Guenther PM, Kipnis V. A new method for estimating theusual intake of episodically-consumed foods with application to theirdistribution. J Am Diet Assoc. 2006;106:1575.

3. Guenther PM, Kott PS, Carriquiry AL. Development of an approachfor estimating usual nutrient intake distributions at the populationlevel. J Nutr. 1997;127:1106-1112.

4. Baranowski T, Islam N, Baranowski B, Cullen KW, Myres D, MarchT, deMoor C. The Food Intake Recording Software system is validamong fourth-grade children. J Am Diet Assoc. 2002;102:380-385.

5. Raper N, Perloff B, Ingwersen L, Steinfeldt L, Anand J. An overviewof USDA’s dietary intake data system. J Food Comp Anal. 2004;17:545-555.

6. DeMaio TJ, Ciochetto S, Davis WL. Research on the Continuing Sur-vey of Food Intakes by Individuals. Proceedings—Section on SurveyResearch Methods. Proceedings of the Section on Survey ResearchMethods. Alexandria, VA: American Statistical Association; 1993:1021-1026.

7. Food Surveys Research Group. Products and services: Food surveytechnical databases. Available at: http://www.ars.usda.gov/Services/docs.htm?docid�7826. Accessed February 1, 2006.

8. Fisher R, Quigley KL. Applying cognitive theory in public healthinvestigations: Enhancing food recall with the cognitive interview. In:Tanur, JM, ed. Questions About Questions: Inquiries into the Cogni-tive Bases of Surveys. New York, NY: Russell Sage; 1991:154-169.

9. Krug S. Don’t Make Me Think. Indianapolis, IN: New Riders; 2005.0. Nielsen J, Loranger H. Prioritizing Web Usability. Upper Saddle

River, NJ: Pearson Education; 2006.1. Ingwersen L, Raper N, Anand J, Moshfegh A. Validation study shows

importance of probing for forgotten foods during a dietary recall[abstract]. J Am Diet Assoc. 2004;104(suppl 2):A-13.

June 2007 ● Journal of the AMERICAN DIETETIC ASSOCIATION 1007