The Gerontologist Copyright 2008 by The Gerontological Society of AmericaVol. 48, No. 2, 158–169

What the Resident Meant to Say: Use ofCognitive Interviewing Techniques toDevelop Questionnaires for NursingHome Residents

Patricia Housen, PhD,1 George R. Shannon, PhD,1 Barbara Simon, MA,1

Maria Orlando Edelen, PhD,2 Mary P. Cadogan, DrPH, RN, GNP,1,3

Linda Sohn, MD,1 Malia Jones,2 Joan L. Buchanan, PhD4

and Debra Saliba, MD, MPH1,2,5

Purpose: Emphasis on consumer-centered care forfrail and institutionalized older adults has increasedthe development and adaptation of surveys for thispopulation. Conventional methods used to pretest

survey items fail to investigate underlying sources ofmeasurement error. However, the use of the cognitiveinterview (CI), a method for studying how respon-dents answer survey items, is not well established ordocumented in this population. This study demon-strates how CIs can be used to improve question-naires intended for nursing home residents. Designand Methods: CIs were conducted with 29 nursinghome residents in order to identify potential problemswith prospective survey items. We used scriptedprobes to standardize the interviews and adapted theQuestion Appraisal System to enumerate and classifythe problems discovered. Results: We fielded be-tween one and five versions of each item in an itera-tive process that identified 61 item-specific problems.Additionally, residents’ cognitive responses sug-gested that some screened their answers on the basisof perceived physical and environmental limitations,and some had difficulty answering items aboutpreferences that fluctuate day to day. These findingsled us to modify the items and response set to simplifythe respondents’ cognitive task. Implications: Thisstudy illustrates how CI techniques can be used tounderstand residents’ comprehension of and re-sponse to survey items.

Key Words: Cognitive interviewing, Institutionalizedadults, Nursing home residents, Questionnaires

Failure to account for nursing home (NH) residentpreferences and perspectives can lead to inaccurateassessments and to care that is alienating andinappropriate. The identification of NH resident

The authors gratefully acknowledge the contribution of members ofthe Technical Expert Panel: Sarah Greene Burger, National Citizen’sCoalition for Nursing Home Reform; Diane Carter, AmericanAssociation of Nurse Assessment Coordinators; Anne Deutsch, North-western University; Sandy Fitzler, American Health Care Association;Irene Fleshner, Senior Clinicians Group; David Gifford, BrownUniversity; Christa Hojlo, VA Nursing Home Service; Ruta Kadonoff,American Association of Homes and Services for the Aging; SallyKaplan, MedPAC; Courtney Lyder, University of Virginia; CherryMeier, National Hospice and Palliative Care Organization; Sue None-maker, Hebrew Rehabilitation Center for the Aged; Joe Ouslander,Emory University; Peter Rabins, Johns Hopkins University Hospital;Naomi Salamon, North Shore University Hospital for Extended Careand Rehabilitation; Judith Salerno, National Institute on Aging; EricTangalos, Mayo Clinic.

We wish to thank members of the Validation Panel for theircontribution to this project: Dan Berlowitz, Boston University; BarbaraBowers, University of Wisconsin; Richard Della Penna, Kaiser Perma-nente Aging Network; Marcy Harris, Mayo Clinic; Ira Katz, Universityof Pennsylvania; Paul Katz, University of Rochester; Rosemary Lubinski,University of Buffalo; David Mehr, University of Missouri; Vince Mor,Brown University; Christine Ann Mueller, University of Minnesota;Patricia Parmelee, Emory University; Margaret Schenkman, University ofColorado; Neville Strumpf, University of Pennsylvania; Eric Tangalos,Mayo Clinic; Christie Teigland, New York Association of Homes andServices for the Aging; Sheryl Zimmerman, University of North Carolinaat Chapel Hill.

We also thank Julie Brown of the survey research group at RANDCorporation, the decision editor, and two anonymous reviewers for theirhelpful comments and suggestions.

The research reported here was supported by the Department ofVeterans Affairs, Veterans Health Administration, Health ServicesResearch and Development Service SDR 03-217. The views expressedin this article are those of the authors and do not necessarily representthe views of the Department of Veterans Affairs.

1Veterans Administration Greater Los Angeles Healthcare SystemHSR&D Center of Excellence and Geriatrics Research and Clinical Care.

2RAND Corporation, Santa Monica, CA.3School of Nursing, University of California, Los Angeles.4Department of Health Care Policy, Harvard Medical School,

Boston, MA.5Borun Center, University of California, Los Angeles and Jewish

Homes for the Aging.

158 The Gerontologist

preferences for day-to-day routines and activities is,therefore, central to providing high quality, in-dividualized residential care (Saliba & Schnelle,2002) and promoting resident autonomy (Brubaker,1996; Jirovec & Maxwell, 1993; Valimaki et al.,2000). Standardized surveys that reliably elicit thesepreferences would be useful tools for professionalNH staff; however, the development of such toolscan be challenging in NH settings. Residents’ inter-pretation of and response to survey questions may beaffected by their physical and cognitive limitations,as well as by unspoken rules and procedures thatexist in the NH (Diamond, 1992; Gubrium, 1975).Researchers and providers risk inadvertently build-ing inappropriate assumptions into their surveydesign if they do not account for these limitationsand institutional culture.

Cognitive interviewing is a survey design methodthat is used in other settings to understand howrespondents interpret and answer survey questions.Our purpose in this study is to demonstrate howqualitative cognitive interview (CI) methods maybe used to refine questionnaires intended for NHresidents. Our specific goals are to (a) ascertain thatconstructs being considered for a new preferenceassessment tool (PAT) are concordant with NHresidents’ understanding and interpretation; (b) sim-plify and clarify survey item wording to maximizecomprehension; and (c) determine whether residentsare able to match choices from the response scale tothe survey items.

Background and Framework

The Minimum Data Set (MDS) is a standardizedassessment tool intended to identify NH residents’health problems. It is required for all residents ofMedicare- or Medicaid-certified NHs. Providersand consumers have raised a number of importantquestions about the MDS, including whether itadequately addresses NH residents’ quality of life.In response, the Centers for Medicare & MedicaidServices has undertaken a significant revision ofthe MDS 2.0, referred to as the MDS 3.0, and iscollaborating with the Department of VeteransAffairs to develop and evaluate a revised tool. Aprimary goal of this revision is to increase residentvoice in the MDS assessment process.

Two expert panels were convened to help set theagenda for the revision. The panels, one of whichwas composed of NH providers and advocates, theother of long-term-care researchers, independentlyrecommended that the Customary Routine (CR)section of MDS 2.0 be retained but changed from itscurrent format. The MDS 2.0 CR section, which isintended to facilitate individualized care planningaround resident preferences, consists of a checklist of20 items relating to a resident’s daily routines in the12 months prior to admission. As such, MDS 2.0 CR

items may capture what the resident had to dobecause of physical and environment constraints,rather than what the resident would have valued ina more supportive environment. In addition, theMDS 2.0 does not give NH staff approaches forensuring that resident preferences are understood.Specifically, staff are not provided with explicit,validated questions or survey tools that have beenshown to reliably elicit preferences from frail NHpopulations.

Members of the expert panels felt the existingapproach failed to realize the potential benefits of theCR section, and they proposed that the MDS 3.0 CRsection adopt a direct interview format. Because theMDS is administered by direct-care staff, both panelsfelt that satisfaction items relating to care shouldnot be used. However, they suggested that any newlydeveloped preference survey should build on do-mains targeted by a NH resident quality-of-lifesatisfaction survey developed and tested by Kaneand colleagues (2003). Once it was clear that wewould be significantly changing the CR section, oneof the panels was also asked to rate the validity andfeasibility of particular topic areas for elicitingresident preferences. In addition, that panel rateda response format of ‘‘very important,’’ ‘‘somewhatimportant,’’ and ‘‘not important’’ as highest priorityfor future testing.

Our interdisciplinary research team, which in-cluded clinicians, survey methodologists, psychome-tricians, and gerontologists, incorporated thesesuggestions in the development of the PAT. Ourintent was to design the PAT so that it could be usedas part of the admission MDS assessment (currentCR approach), or as a stand-alone assessment atother points in time. Potential items and design forthe PAT are grounded in Residential Care Quality,proposed as a primary domain of NH care quality(Saliba & Schnelle, 2002). The domain includesidentifying resident preferences for day-to-day activi-ties and incorporating care that encourages auton-omy in choosing and completing activities. Once weselected the candidate items, we mapped them ontoquality of life domains identified by Kane andcolleagues (2003). We used this mapping approachbecausewe hoped that the PAT itemsmight assist NHstaff in assessing preferences relevant to improvingresident-reported satisfaction and quality of life.

Rationale for CI Testing Methods

Conventional pretesting methods evaluate non-response rates and interviewer feedback to identifyproblems with surveys (Forsyth & Lessler, 1991;Presser et al., 2004). However, assumptions thatsurvey developers build into the items or responsesets may not be shared or understood by the par-ticipants. If this disconnect exists, then the surveymay be burdensome, and high response rates may

Vol. 48, No. 2, 2008 159

mask significant measurement error that compro-mises the survey’s validity (Collins, 2003; Willis,2005). These concerns are particularly pertinent insurveys intended for older individuals, who mayconfront barriers to effective communication, in-cluding stereotypes of elders that ignore theirindividual differences, impaired communicationskills associated with age-related sensory andcognitive changes, and generational differences inpreferred communication styles (Caplan & Greene,1999; Caplan, Haslett, & Burleson, 2005; Ryan &Butler, 1996).

A different pretesting method, cognitive inter-viewing, applies the principles of cognitive psychol-ogy to survey development. The respondents’ task isconceptualized in four stages in the CI: interpreta-tion, retrieval, judgment, and response selection(Tourangeau & Rasinski, 1988). In Stage 1, questioninterpretation, participants must translate the iteminto a relevant attitude structure. In Stage 2,respondents retrieve beliefs and feelings relating tothe attitude structure. The third stage involvesassessment and weighing of attitudes to arrive ata judgment. Finally, after the answers are edited forconsistency and social desirability, the judgment is‘‘mapped’’ onto the available response options. Eachstage is subject to cognitive failures.

In cognitive interviewing, researchers use partic-ipants’ narrative reports about their cognitive pro-cesses in order to identify sources of response errorand refine items before proceeding with conventionalpretesting (Forsyth & Lessler, 1991; Presser et al.,2004). CI studies typically report small sample sizes.While this is sometimes cited as a shortcoming, CImethodologists counter that the goal of CI techni-ques is not statistical estimation but rather theidentification of sources of response error within thegroup to be surveyed (Caspar, Lessler, & Willis,1999; Willis, 2005).

The two principal techniques for conducting CIsare referred to as ‘‘think-alouds’’ and ‘‘verbalprobing’’ (Collins, 2003; Presser et al., 2004; Willis,2005). Using the think-aloud approach, the inter-viewer asks respondents to explain what they arethinking as they answer questions. This open-endedapproach may be difficult and burdensome forparticipants not used to identifying or verbalizingthe cognitive processes required to answer questions.In the alternative CI technique, verbal probing, theinterviewer employs prompts to simplify the respon-dents’ task. The interviewer first asks participants toanswer each pilot item and then uses follow-upquestions to prompt them to explain their cognitiveprocesses. Interviewers may use probes scripted inadvance, or they may use spontaneous unscriptedfollow-up questioning. Researchers may also observeand code respondent behavior and response latencyduring the CI (Presser et al., 2004; Willis, 2005).

CI techniques have been used successfully withhealthy community-dwelling seniors. However, the

usefulness of CI methods is not well established inmore frail populations, including those residing inNHs (Jobe & Mingay, 1990). A few researchers whoaddress frail elders have mentioned the use of CI, butthey do not elaborate on the extent, methods, orinfluence of the process on item selection.

In this article we describe the use of CI techniquesto pretest items for the PAT in a sample of NHresidents. Moreover, we provide examples to illus-trate how CI findings informed the PAT. We alsoreport the results of a content analysis that weconducted to quantify the categories of problemsidentified during the CI process.

Methods

Participants

Study participants resided in 2 of the 134 NHsoperated by the Veterans Health Administration.The facilities are located in Southern California. Wesought to sample residents with mild or no cognitiveimpairment, whose responses would most likelyreflect problems with the items rather than difficul-ties associated with significant cognitive impairment.NH administrators identified residents with Mini-Mental State Examination scores � 22. Respondentswere told that participation was voluntary and theycould terminate study interviews at any time. Noincentives were offered.

Interviews

Two postdoctoral gerontology fellows conductedthe interviews in a location where residents con-firmed they felt comfortable talking about theirpreferences. The interviewers used dual independentcoding to report residents’ verbatim responses andrecord behavioral observations.

We tested candidate items in sets to decreaserespondent burden. Each resident was interviewedbetween one and four times during the 14-week cycleof testing, with sessions ranging in length from 7 to 46minutes, depending on the number of items in the set.We hadmore than one round of interviews conductedfor each set of items; this iterative approach alloweditem replacement or refinement between interviews.

Item Selection for CIs

On the basis of the expert panel recommendationsand a comprehensive literature review of existingpreference questionnaires suitable for use in the NH(e.g., Carpenter, Van Haitsma, Ruckdeschel, &Lawton, 2000; Kane & Degenholtz, 1997; Kane,Degenholtz & Kane, 1999; Polisher Research In-stitute, 2006), the research team identified 516candidate items.

To narrow this list, the team met several times to

160 The Gerontologist

assess candidate items based on the following criteria:relevance to facility interactions in the residentialquality framework (Saliba & Schnelle, 2002); hightopic validity and feasibility ratings from the expertpanel, the likelihood of fit in a structured survey, andthe frequency with which a topic appeared in quality-of-life and preference literature (Carpenter et al.,2000; Kane & Degenholtz, 1997; Kane et al., 1999,2003; National Citizens’ Coalition for Nursing HomeReform, 1984; Polisher Research Institute, 2006;Rantz et al., 1999; Van Haitsma, 2000; Whitlatch,Feinberg, & Tucke, 2005); and the inclusion of itemsrepresentative of each quality-of-life domain (Kaneet al., 2003). The team modified the wording of itemsto match the common root question (‘‘While you arehere in the nursing home, how important is it to youto. . .’’) and the 3-point importance scale (veryimportant, somewhat important, not important).The team ultimately selected 30 items for pretestingfrom the list of 516 candidate items.

Procedure

In order to standardize the interviews andminimizerespondent burden and interviewer effects (a frequentcriticism of CI techniques), the research team de-veloped a semistructured interview protocol. Theinterviewers asked respondents to rate each of theitems by using the 3-point importance scale, whichthey showed to them as typed wording in a large font.Interviewers made use of default response options(refused–don’t know, no response, or nonresponsiveanswer) to record incomplete or refusal answers.Theyalso noted behaviors, comments, and latency whenthey asked the residents each question.

After each response, interviewers immediatelyasked the related scripted probes. The cornerstoneprobe asked residents to provide examples of keyterms. For example, a question about resident pref-erences for using ‘‘tobacco products’’ was probed byasking, ‘‘Can you give me an example of the kinds oftobacco products you were thinking about when youanswered the question?’’ Examples including ciga-rettes, cigars, pipes, and chewing tobacco wereaccepted as evidence that the general term wasunderstood.

After providing examples, participants were askedadditional prompts depending on their initial re-sponse. Participants who said a specific item was notimportant or only somewhat important wereprompted to discuss whether it had ever been moreimportant in the past. For example, participants whoindicated that using tobacco products was notimportant or only somewhat important were asked,‘‘Has using tobacco products ever been more impor-tant to you?’’ Those who said it was very importantor who indicated that using tobacco products hadbeen more important in the past were subsequentlyasked, ‘‘Are you able to use tobacco products as

much as you would like here at the nursing home?’’Interviewers were permitted to ask additional ques-tions if further clarification was needed.

We compared residents’ responses using the scaleand their verbatim answers to the probes in order toidentify discordance, an indicator of potential prob-lems with an item or with the match between theitem and response scale. We operationalized discor-dance as disagreement between a resident’s impor-tance rating and the narrative answer to the probes.

The research team met regularly to reviewproblems identified during CIs. Problems identifiedduring even one CI were reviewed by the team toconsider whether alternative versions were needed(Rosal, Carbone, & Goins, 2003; Willis, 2005). Iftesting revealed problems with an item, we revisitedthe original 516-item list for replacements or possiblerevisions.

Analytic Approach

We classified the problems discovered during theCIs by applying coding guidelines and coding formsfrom the Question Appraisal System (QAS-99)manual (Willis & Lesser, 1999). The QAS-99 manualdescribes eight broad categories of characteristicsthat can influence response rates and measurementaccuracy. They are as follows: reading (can ques-tions be uniformly read?), instructions (are instruc-tions consistent and uncomplicated?), clarity (is thequestion meaning clear?), assumptions (is thequestion based on correct assumptions or logic, oris it double barreled?), knowledge or memory (dorespondents have the knowledge and cognitiveability required to answer?), sensitivity or bias (isthe question, topic, or wording sensitive in nature?),and response categories (is the range of responseoptions adequate?). An eighth category is used tocatalog other types of problems. Each categoryincludes and defines more specific problems. Forexample, survey items that lack clarity may be vague,use unfamiliar technical terms, have awkwardsyntax, or have ill-defined reference periods.

Our content analysis enumerated and categorizedproblems seen with the 30 items originally selectedfor pretesting and each subsequent revision. In sum,we pretested 59 items by using CI techniques andsubjected them to content analysis. Interviewersworked independently, using separate forms tocode problems with each of the items. They revieweddetailed summaries of residents’ verbatim responses,interviewers’ observations, and minutes from weeklyteam discussions to identify and assign problems tothe most suitable QAS-99 category. Interrater re-liability, calculated as the percentage of agreementbetween coders, was 92%. Coding disagreementsoccurred in deciding how problems were to becategorized, not in the identification of problems.We resolved these differences through discussion.

Vol. 48, No. 2, 2008 161

Results

Participants

Of the 31 residents approached, 29 agreed toparticipate. Participants ranged in age from 52 to 91(Mdn = 76); 19 (or two thirds) were White and 12had at least some college education (Table 1). Themajority of respondents (24/29) were male, and mostwere long-stay residents; 5 had a length of staybetween 91 days and 1 year and 19 resided in the NHfor more than 1 year. Our analyses are based on 56CIs conducted with this sample.

Analysis

Analyses revealed both item-specific problems andglobal problems that applied across multiple items.

Item-Specific Problems.—We fielded between oneand five versions of each of the 30 items initiallyselected for CIs in an iterative process. The researchteam identified 61 problems specific to the 59 itemsthat were pretested. Figure 1 shows a detailedexample of the iterative process we used to refinean item.

Table 2 provides examples of how we identifieditem-specific problems by means of residents’cognitive responses and how we classified themwithin the QAS-99 framework. Despite the use ofitems and terms from published surveys, suboptimalclarity in the intent or meaning of questionsaccounted for more than half (35/61) of the item-specific problems identified during the CIs (QAScategory: Clarity). More specifically, although resi-dents would typically select an answer from the 3-point importance scale, their responses to probesrevealed that awkward wording, complicated syn-tax, the use of technical terms, and vagueness mademany items difficult to understand. Residents’responses suggested that 11 items were based onfaulty assumptions about services available in theNH, or that residents had difficulty choosing a singleresponse option because the items were doublebarreled, containing more than one implicit question(QAS category: Assumptions). Six items requiredinformation residents didn’t have or could notremember (QAS category: Knowledge–Memory).Four more problems related to the sensitive natureof the topic being queried (QAS category: Sensitivity–Bias). Finally, the scope of 5 items was too narrowand therefore did not reflect residents’ experiences;we coded these as ‘‘other.’’

CI interviews also revealed that some seeminglycomplex or colloquial items were readily understoodby residents, even if items defied conventional surveydesign precepts. For example, an item asking aboutthe importance of ‘‘keeping up with the news’’ mightbe considered colloquial, whereas a question askingabout the importance of ‘‘going outside to get fresh

air when the weather is good’’ seems lengthy andcontains more than one idea. Efforts to change theseparticular items led to decreased understanding andincreased confusion for respondents. The teamretained both items based on CIs indicating thatresidents found them to be clear and unambiguous.

Global Problems.—In addition to item-specificproblems, the team identified some problems thatapplied to multiple items across the survey. They didnot include these problems in the tally of item-specific problems.

Problem With Response Categories: Accommo-dation. Problems with the response scale were oftwo types. The first was the tendency of someresidents to accommodate perceived physical andenvironmental limitations. The second was the needfor more precision than was offered by the 3-pointscale.

In the first round of CIs, the research team testedseven items with four residents each. Of thediscordant importance scale ratings and narrativeresponses in this round, content analysis revealedthat four resulted from respondents’ censoring oftheir importance ratings because of perceivedlimitations. For example, one item asked residentsto rate how important it was ‘‘to do your favoriteactivities or hobbies.’’ A participant who had beenliving in NHs for more than 3 years responded to theitem saying it was not important. When prompted,he volunteered that although he enjoyed playingbingo, he wasn’t able to play as often as he’d likebecause bingo was offered in the early evening after

Table 1. Characteristics of the Sample (N = 29)

Characteristic n

Age (years)

,65 965–74 675–84 585þ 9

Gender

Male 24Female 5

Race

White, not of Hispanic origin 19Black, not of Hispanic origin 8Hispanic 2

Length of stay

�90 days 591 days–1 year 5366 days–3 years 93þ years 10

Education

High school 16Some college 8Bachelor’s degree 5

162 The Gerontologist

staff put him to bed. Clearly the resident suppressedhis preferences and did not question the need forstaff to transfer him to bed by 4 p.m., or thescheduling of bingo. Instead, he had accommodatedto the restrictions in the NH environment.

Through responses like these, the research teamobserved that, when residents were answering someitems, the residents’ cognitive processes included an

assessment of physical ability to engage in the activityand perceived limitations in the NH environment.After making these kinds of mental calculations, someresidents with perceived limitations would ratea strongly preferred activity as being not important,whereas others would still rate it as being somewhatimportant or very important. As a result, the ratingchoices did not necessarily reflect an item’s actual

Figure 1. Diagram of the iterative process used to refine preference items.

Vol. 48, No. 2, 2008 163

importance to every respondent. This accommoda-tionwas particularly problematic given that the intentof the items is to help staff identify care planningneeds. Facilitiesmight be expected to identify and careplan to avoid barriers that prevent residents fromrealizing their preferences.

In response, the research team introduced anadditional and somewhat unique response option,‘‘important, but can’t do–no choice,’’ to simplify thecognitive task. This option was intended to promotedialogue about perceived physical and environmen-tal barriers that residents believed limited theirability to exercise their preferences.

Problem With Response Categories: Need forGreater Precision in the Response Scale. CIssuggested an additional source of discordance betweenresident ratings and their narrative responses to theprobes. Residents who selected the ‘‘somewhatimportant’’ rating gave verbal narratives that wereseldom neutral. Some narratives reflected a positivepreference and others reflected a negative one. Forexample, when asked about the importance ofchoosing a time to be awakened in the morning,a resident, after some hesitation, rated the item assomewhat important. As part of his response toprobing, he commented, ‘‘I like to set my own hoursand I’dbe veryupset if thatwere taken away fromme.’’

This overlap suggested that a 5-point importancescale might better represent resident’s true prefer-ences. In order to improve the precision and ease ofanswering the ‘‘somewhat important’’ option, wereturned to the field to test two 5-point importancescales. Both metrics included the options of veryimportant, somewhat important, and important, butcan’t do–no choice. They differed for the unimpor-tant responses choices. Metric 1 used somewhatunimportant and not very important whereas Metric2 used not very important and not important at all.CIs indicated that the ‘‘somewhat unimportant’’option confused the residents. Therefore, we usedMetric 2.

We next compared the 4-point (very important,somewhat important, not important, and important,but can’t do–no choice) and 5-point importancescale. Interviews revealed that the 5-point impor-tance scale was associated with greater concordancebetween residents’ ratings and their narrativeresponses.

Problems With Assumptions. Some residents haddifficulty selecting a single response for items relatingto the scheduling of daily routines. Their narrativecomments revealed that scheduling preferences mayvary depending on how they feel or on the events ofa particular day. For example, resident preferences forbeing awakened in the morning are highly individu-alized and variable.When asked about the importanceof choosing a time to be awakened, a resident whorated the item as somewhat important commentedthat ‘‘it’s a day-to-day thing,’’ dependent on ‘‘howyoufeel orwhen youwent to bed.’’ Presumably, having the

flexibility to choose a time to be awakened would bevery important to the resident during periods of illnessor after having stayed up late, and perhaps lessimportant on other days.

What was wanted was choice and flexibility,approaches that are critical to the provision ofperson-centered care. Because choice and flexibilityare qualities of care that should be universallyavailable to all residents and not premised ona preendorsed item, the research team did notinclude a specific preference item for flexibility inthe PAT. Likewise, items related to kindness andrespect, although appropriate for satisfaction sur-veys, were viewed by the research team as funda-mental elements that should be universally availableand not require a resident to indicate they areimportant to them.

Item Selection for Further Pilot Testing

After 16 rounds of CIs, the team selected 24 itemsfor further pilot testing. Table 3 shows the 24 items,14 of which relate to 13 constructs included in theoriginal CR section. As the table shows, we retainedat least 1 item from each of seven quality-of-lifedomains. The table also shows 6 of the 30 originaltopics that, despite efforts to revise or replace, werenot easily understood by residents during CIs andwere therefore rejected.

Discussion

Eliciting resident preferences is central to bothautonomy and quality of life in the NH environment.However, both short-stay and long-stay NH resi-dents have significant levels of dependency andmedical morbidity, and survey challenges that mightbe tolerated in healthier populations may provesignificantly more burdensome in this frail popula-tion. Unambiguous and relevant survey items canhelp minimize respondent burden.

In this study we used CIs to improve both thecontent and structure of the PAT. CIs allowed us, theresearch team, to identify many item-specific andglobal problems that would have potentially de-creased the usefulness of the survey for elicitingpreferences. Our team continuously revisited thelogic underlying the survey items, and we mademodifications that enhanced their clarity, relevance,sensitivity, and scope. The team also found thatresidents often selected response options based onaccommodation to perceived barriers, and thatresponse sets with insufficient options complicatethe response process. These kinds of limitations, ifnot addressed, could limit the utility of preferencesurveys for informing the design of individualizedcare plans.

CI methods offer an important, systematic ap-proach for gaining insights into the NH resident’s

164 The Gerontologist

Table 2. Examples of Item-Specific Problems Identified by Using CI Techniques

Item Surveyor Intent Examples of Cognitive Response

QAS-99 Category: Clarity

1. While you are here in the NH,how important is it to you tonot be awakened before 7 a.m.?

Establish when resident wouldlike to be awakened andinitiate discussion aboutmorning routines.

Resident commented on the awkwardwording of the item, saying ‘‘You’vegot to transpose this for me.’’

2. While you are here in the NH,how important is it to you togo on outings?

Determine how much residentvalues being able to dothings away from the NH.Initiate discussion aboutthe kinds of activities andoutings resident enjoys.

Resident asked what was meant by theword outings.

3. While you are here in the NH,how important is it to you tocontrol the music you listento in your room?

Determine how much residentvalues listening to music.Initiate discussion about thekinds of music and musicalactivities the resident enjoys.

Resident asked what was meant bycontrol. Another interpretedcontrol as asking about volume.

4. While you are here in the NH,how important is it to you tohave a particular type of diet(e.g., vegetarian, Kosher).

Identify general kinds of dietspreferred by the resident forpersonal, cultural, or spiritualreasons.

Residents interpreted diet as a technical,care-related term. Their exampleswere of therapeutic diets (e.g., soft,weight management, or diabetic diets).Importance was represented by whattheir provider valued.

5. While you are here in the NH,how important is it to you tohave a particular type of dietor meals?

Identify general kinds of dietspreferred by resident forpersonal, cultural, or spiritualreasons.

Elimination of the parenthetical exampleselicited specific food preferences, i.e.,avocado, peanut butter, cashew nuts,and finely ground meats.

QAS-99 Category: Assumptions–Logic

6. While you are here in the NH,how important is it to you tohave snacks available betweenmeals?

Determine whether resident likes tosnack. Initiate discussion aboutthe kinds of food the residentenjoys as snacks and the timingof snacks.

Several diabetic residents responded thatrather than being important to them,snacking was very important for them,and that it was already included intheir care plan.

7. While you are here in the NH,how important is it to you tospend time on your appearanceand clothes?

Initiate discussion about appearanceand clothes, and how muchhelp with dressing and groomingthe resident wants and needsfrom staff and family.

Double barreled: Whereas appearance wasinterpreted as good grooming andcleanliness, and was highly valued bymost residents, spending time on one’sclothes was less important for some.For example, a respondent who ratedthe item as somewhat important said,‘‘I’m bedridden. . . I do get up to domy toileting and shower, (but) I don’tdress that much. I’m usually inhospital clothing.’’

QAS-99 Category: Knowledge–Memory

8. While you are here in the NH,how important is it to you touse herbs, vitamins, minerals,or other natural products?

Determine whether resident haspreferences for complementaryand alternative medicines.

Many residents said having this optionwas at least somewhat important.However, none was able to provideexamples of specific complementaryand alternative medicines or vitaminsof importance to them.

QAS-99 Category: Sensitivity–Bias

9. While you are here in the NH,how important is it to you tohave an alcoholic beverage onoccasion?

Determine if resident values drinkingalcoholic beverages. Initiatediscussion about when and whatkinds of alcoholic beverages theresident enjoys.

Residents were uncomfortable answering.One resident who said she used toenjoy having a glass of wine withdinner asked if that meant she was analcoholic. Other residents who saidthey enjoyed occasional drinks framedtheir answers within the context offacility policies.

(Table continues on next page)

Vol. 48, No. 2, 2008 165

experience, and for testing and revising individualitems before conducting conventional survey testing.Although the CI process can be time consuming andlabor intensive, its use in a small sample of residentsleads to the development of more accurate and lessdemanding surveys for the larger population.

Limitations

We limited our CIs to residents who werecognitively intact; however, NH populations havehigh rates of cognitive impairment. Individuals withmoderate, and in some cases even severe, cognitiveimpairment are able to respond to questions abouttheir preferences and quality of life (Brod, Stewart,Sands, &Walton, 1999; Feinberg &Whitlatch, 2001;Van Haitsma, 2000; Whitlatch et al., 2005). Never-theless, the need to articulate thoughts during thecognitive interview process would be burdensomefor this population. We therefore focused oncognitively intact residents, reasoning that if anitem was unclear to those with intact cognitivefunction, it would not perform well across thepopulation. We used participants’ feedback to makethe PAT more accommodating to the NH environ-ment, more sensitive to physical challenges, and, wehope, more easily understood by residents.

The sample of Department of Veterans Affairs

(VA) NH residents participating in the CI processwas predominantly male. However, we selecteddomains and related candidate and replacementitems for the PAT on the basis of a literature reviewpertaining to general NH research, which is pre-dominantly conducted in non-VA facilities. In thosefacilities, female residents greatly outnumber maleresidents. Furthermore, we considered all residentresponses valuable at this stage of survey develop-ment, and we eliminated no items that were easilyunderstood and answered because they were ratedas being not important by our NH sample. Rather,our qualitative study focused on refining items tomaximize their clarity and hence usefulness forcommunicating with all residents about their prefer-ences. This goal, which we pursued by probing thecognitive processes that residents use to respond tosurvey items, should not be expected to be dependenton gender.

Future Research

After CI methods are used to refine items andsurveys, next steps may include pilot testing forresponse rates and reliability in larger and morediverse samples. Next steps in the development ofthe PAT will include test–retest situations withcognitively intact and impaired, male and female,

Table 2. (Continued)

Item Surveyor Intent Examples of Cognitive Response

10. While you are here in the NH,how important is it to you tocontrol the music you listen toin your room?

Determine how much resident valueslistening to music. Initiatediscussion about the kinds ofmusic and musical activities theresident enjoys.

Residents focused their responses aroundthe need to be considerate of roommates.‘‘I’d have to ask my roommate. If he’sfrom a different generation he might likesomething I don’t like,’’ said one veteran.‘‘You have to respect the wishes of yourfellow patients, and when you do youget respect in return,’’ said another.

QAS-99 Category: Other Problems (i.e., Narrow Focus)

11. While you are here in the NH,how important is it to you toparticipate in religious services?

Determine if resident values religiousactivities. Identify any unmetreligious needs.

Residents distinguished between preferencesfor participating in organized servicesheld at the NH, and other types ofreligious or spiritual practices. Forexample, one resident explained thatwhile religious and spiritual activitieswere very important to him, he neverparticipated in those held in the NHdining room. ‘‘My pastor sends tapedservices to me,’’ he said. ‘‘I found I canlisten in bed early in the morning.’’

12. While you are here in the NH,how important is it to you tochoose between a tub bath,shower, or sponge bath?

Determine how much resident valuesbeing able to choose a preferredbath type. Initiate discussion aboutmore specific issues relating tobathing and hygiene.

Several residents mentioned their preferencefor receiving bed baths, an alternativewording not included in the item.

Note: QAS-99 = Question Appraisal System; CI = cognitive interview; NH = nursing home. All quotations are verbatim orparaphrased to limit wordiness or enhance clarity.

166 The Gerontologist

Table 3. Comparison of Preference Items Before and After CIs, and Items From the MDS 3.0 CR Section

24 Items Selected for PAT

CR Items from MDS 3.0 Before CIs After CIs

Quality of Life Domain: Meaningful Activities

Stays busy with hobbies, reading,or fixed daily routine

. . .to you to have access to booksand magazines?

. . .to have books, newspapers,or magazines to read?

. . .to control the music you listento in your room?

. . .to listen to music you like inyour room?

Daily animal companionship . . .to be around animals? . . .to be around animals such as pets?. . .to keep up with the news? (No change)

Involved in group activities . . .to participate in group activities? . . .to do things with groups of people?Stays busy with hobbies, reading,

or fixed daily routine. . .to participate in group activities? . . .to do your favorite activities

or hobbies?Goes out 1þ days a week . . .to go on outings? . . .to do things away from the

nursing home?Goes out 1þ days a week . . .to go outside the building? . . .to go outside to get fresh air when

the weather is good?

Quality of Life Domain: Autonomy

. . .to not be awakened before 7 a.m. (Alternate stem) If you could wakeup whenever you wanted, howimportant would it be to you tochoose the time you are awakened?

Stays up late at night(e.g., after 9 p.m.)

. . .to stay up past 9 p.m.? (Alternate stem) If you could wakeup whenever you wanted, howimportant would it be to you tostay up past 8 p.m.?

Showers for bathing . . .to choose between a tub bath,shower, or sponge bath?

. . .to choose between a tub bath, shower,bed bath, or sponge bath?

Bathing in p.m. . . .to choose the time of day whenyou bathe or shower?

(No change)

In bedclothes much of the day . . .to choose what clothes to wear? (No change). . .to spend time on your appearance

and clothes?(No change)

Use of tobacco products atleast daily

. . .to use tobacco products? (No change)

Quality of Life Domain: Functional Competence

. . .to take care of your own things? . . .to take care of your personalbelongings or things?

Quality of Life Domain: Enjoyment

Eats between meals all ormost days

. . .to have snacks available betweenmeals?

(No change)

Use of alcoholic beverage(s)at least weekly

. . .to have an alcoholic beverage onoccasion?

(No change)

Quality of Life Domain: Spiritual Well-Being

Finds strength in faithUsually attends church,temple, synagogue

. . .to participate in religious services? . . .to participate in religious services orpractices?

. . .that those planning your daily careknow your religious or culturalcustoms?

. . .to follow your cultural or familycustoms?

Quality of Life Domain: Privacy

. . .to be able to use the phone inprivate?

(No change)

. . .to have a private space whenfamily or friends visit?

(No change)

Quality of Life Domain: Security

. . .that your family is involved inplanning your daily routines?

. . .to have your family or a close friendinvolved in discussions aboutyour care?

. . .to have a place to lock your thingsto keep them safe?

(No change)

(Table continues on next page)

Vol. 48, No. 2, 2008 167

NH residents living in geographically diverse VA andnon-VA facilities. Researchers will conduct thesetests before final recommendation to include thePAT in the MDS or to use PAT as a stand-aloneinstrument.

If the PAT performs well, researchers shouldexamine whether the interview initiates conversa-tions between caregivers and care recipients,whether it facilitates individualized care planning,and whether it conveys to residents that the ex-pression of preferences is valued and will behonored whenever possible. Given that preferen-ces may change in response to development, lifeevents, or reflection (Carpenter et al., 2000; Kane& Degenholtz, 1997), future studies should alsoexamine the use of the PAT to elicit each resident’spreferences over time.

Conclusions

This study illustrates that CI techniques can beused successfully to refine surveys intended for NHpopulations. Others interested in adapting anexisting survey or creating and testing new surveysfor frail populations can consider the potentialapplication of this method for informing theirwork. In the current study, CI findings challengedcommonly held assumptions about language use,item length, and the simplicity of response sets.The results revealed accommodation and day-to-day fluctuation of wants as elements of residentself-report, and the importance of developingprecise response options that are easily mappedto the resident’s experience.

References

Brod, M., Stewart, A. L., Sands, L., & Walton, P. (1999). Conceptualizationand measurement of quality of life in dementia: The Dementia Quality ofLife Instrument (DQoL). The Gerontologist, 39, 25–35.

Brubaker, B. H. (1996). Self-care in nursing home residents. Journal ofGerontological Nursing, 22, 22–30.

Caplan, S. E., & Greene, J. O. (1999). Acquisition of message-productionskills by younger and older adults: Effects of age, task complexity, andpractice. Communication Monographs, 66, 31–48.

Caplan, S. E., Haslett, B. J., & Burleson, B. R. (2005). Telling it like it is: Theadaptive function of narratives in coping with loss in later life. HealthCommunication, 17, 233–251.

Carpenter, B. D., Van Haitsma, K., Ruckdeschel, K., & Lawton, M. P.(2000). The psychosocial preferences of older adults: A pilot examinationof content and structure. The Gerontologist, 40, 335–348.

Caspar, R. A., Lessler, J. T., & Willis, G. B. (1999). Reducing survey errorthrough research on the cognitive and decision processes in surveys.Short course presented at the 1999 meeting of the American StatisticalAssociation. Retrieved April 23, 2007, from http://appliedresearch.cancer.gov/areas/cognitive/interview.pdf

Collins, D. (2003). Pretesting survey instruments: An overview of cognitivemethods. Quality of Life Research, 12, 229–238.

Diamond, T. (1992). Making gray gold: Narratives of nursing home care.Chicago: University of Chicago Press.

Feinberg, L. F., & Whitlatch, C. J. (2001). Are persons with cognitiveimpairment able to state consistent choices? The Gerontologist, 41, 374–382.

Forsyth, B., & Lessler, J. (1991). Cognitive laboratory methods: Ataxonomy. In P. Biemer, R. Groves, L. Lyberg, N. Mathiowetz, &S. Sudman (Eds.), Measurement error in surveys (pp. 393–418).New York: Wiley.

Gubrium, J. (1975) Living and dying at Murray Manor. New York: St.Martin’s Press.

Jirovec, M. M., & Maxwell, B. A. (1993). Nursing home residents:Functional ability and perceptions of choice. Journal of GerontologicalNursing, 19, 10–14.

Jobe, J. B., & Mingay, D. J. (1990). Cognitive laboratory approach todesigning questionnaires for surveys of the elderly. Public HealthReports, 105, 518–524.

Kane, R. A., & Degenholtz, H. (1997, Spring). Assessing values andpreferences: Should we, can we? Generations, 21, 19–24.

Kane, R. A., Degenholtz, H. B., & Kane, R. L. (1999). Adding values: Anexperiment in systematic attention to values and preferences ofcommunity long-term care clients. Journal of Gerontology: SocialSciences, 54B, S109–S119.

Kane, R. A., Kling, K. C., Bershadsky, B., Kane, R. L., Giles, K., Degenholtz,H. B., et al. (2003). Quality of life for nursing home residents. TheGerontologist, 58A, 240–248.

National Citizens’ Coalition for Nursing Home Reform. (1985). A consumerperspective on quality care: The residents’ point of view. Washington,DC: Author.

Polisher Research Institute. Preferences for Everyday Living Questionnaire.Retrieved December 5, 2006, from the Polisher Research Institute Website: http://www.abramsoncenter.org/PRI/documents/PELIQuestionnaire.pdf

Presser, S., Couper, M. P., Lessler, J. T., Martin, E., Martin, J., Rothgeb,J. M., et al. (2004). Methods for testing and evaluating survey questions.Public Opinion Quarterly, 68, 109–130.

Rantz, M. J., Zwygart-Stauffacher, M., Popejoy, L., Grando, V. T., Mehr,D. R., Hicks, L. L., et al. (1999). Nursing home care quality: Amultidimensional theoretical model integrating the views of consumersand providers. Journal of Nursing Care Quality, 14, 16–37.

Table 3. (Continued)

Six Items Rejected After CIs

CR items from MDS 3.0 Before CIs After CIs

. . .to personalize your room? (No change)

. . .to follow a routine right before yougo to bed and right after you wake up?

. . .follow a set routine before you go tobed and after you wake up?

. . .to keep to the same routine each day? (No change)Distinct food preferences . . .to have a particular type of diet

(vegetarian, Kosher, etc.)?. . .to have a particular type of diet or

meals?. . .to use herbs, vitamins, or other

natural products?(No change)

Naps regularly during day(at least 1 hour)

. . .to schedule a naptime during the day? (No change)

Note: CI = cognitive interview; MDS = Minimum Data Set; CR = Customary Routine; PAT = preference assessment tool.Unless otherwise noted, the item stem reads as follows: ‘‘While you are here in the nursing home, how important would it be toyou. . .’’

168 The Gerontologist

Rosal, M. C., Carbone, E. T., & Goins, K. V. (2003). Use of cognitiveinterviewing to adapt measurement instruments for low-literate His-panics. The Diabetes Educator, 29, 1006–1017.

Ryan, E. B., & Butler, R. N. (1996). Communication, aging and health:Toward understanding health provider relationships with older clients.Health Communication, 8, 191–197.

Saliba, D., & Schnelle, J. F. (2002). Indicators of the quality of nursing homeresidential care. Journal of the American Geriatrics Society, 50, 1421–1430.

Tourangeau, R., & Rasinski, K. A. (1988). Cognitive processes underlyingcontext effects in attitude measurement. Psychological Bulletin, 103,299–314.

Valimaki, M., Leino-Kilpi, H., Scott, P. A., Arndt, M., Dassen, T., Lemonidou,C., et al. (2001). The role of CNSs in promoting elderly patients’ autonomyin long-term institutions: Problems and implications for nursing practiceand research. Clinical Nurse Specialist, 15, 7–14.

Van Haitsma, K. (2000). The assessment and integration of preferences into

care practices for persons with dementia residing in the nursing home. InR. Rubenstein, M. Moss, & M. Kleban (Eds.), The many dimensions ofaging (pp. 143–163). New York: Springer.

Whitlatch, C. J., Feinberg, L. F., & Tucke, S. S. (2005). Measuring the valuesand preferences for everyday care of persons with cognitive impairmentand their family caregivers. The Gerontologist, 45, 370–380.

Willis, G. B. (2005). Cognitive interviewing: A tool for improvingquestionnaire design. Thousand Oaks, CA: Sage.

Willis, G. B., & Lessler, J. T. (1999). Question appraisal system QAS-99.Retrieved December 5, 2006, from the National Cancer Institute Web site:http://appliedresearch.cancer.gov/areas/cognitive/qas99.pdf

Received January 31, 2007Accepted June 7, 2007Decision Editor: William J. McAuley, PhD

Vol. 48, No. 2, 2008 169