Contract No.:

53-3198-5-044APR Reference No.: 8305-010

REACHING THE WORKINGPOOR AND POOR ELDERLY:

COGNITIVE TESTSUMMARY REPORT

INN

April 11, 1997

Authors:

Michael PonzaSheena McConnellRhoda R. Cohen

Donna L. Eisenhower

Submitted to:

U.S. Department of AgricultureFood and Consumer ServiceOffice of Analysis and Evaluation3101 Park Center DriveAlexandria, VA 22302

Project Officer:Alana Landey

Submitted by:

Mathematica Policy Research, Inc.P.O. Box 2393Princeton, NJ 08543-2393(609) 799-3535

Project Director:Sheena McConnell

Principal Investigators :Sheena McConnellMichael Ponza

CONTENTS

Chapter

Page

I

INTRODUCTION 1

II

COGNITIVE TEST METHODOLOGY 3

A. COGNITIVE TEST RESPONDENTS 3

1. Number and Types of Respondents by Site and Cognitive InterviewingTechnique 3

2. Respondent Profiles 53. Identifying and Recruiting Respondents for the Cognitive Tests 9

B. COGNITIVE TEST PROCEDURES 10

1. Logistics 102. Cognitive Techniques Used 10

III

C. BEHAVIORAL CODING 14

COGNITIVE TEST RESULTS 18

A. DETAILED FINDINGS BY QUESTION 191. Screening Questionnaires 192. Main Questionnaires 23

B. OTHER FINDINGS FROM THE COGNITIVE TESTS 451. Questionnaire Length 452. Questionnaire Content, Organization, and Flow 473. Implications for Telephone Administration of Questionnaires 47

C. REVISIONS TO TAKE INTO ACCOUNT CHANGES IN ELIGIBILITYRESULTING FROM WELFARE REFORM 48

REFERENCES 50

TABLES

Tables

Page

II.1

NUMBER OF COGNITIVE TEST RESPONDENTS BY S1lE 4

11.2

SECTIONS OF QUESTIONNAIRES ADMINISTERED TO EACH TYPEOF RESPONDENT N THE COGNITIVE TESTS 6

III.1

COMPLETION TIMES FOR RETROSPECTIVE THINK ALOUD COGNITIVETEST INTERVIEWS 46

I. INTRODUCTION

The Food Stamp Program (FSP) is designed to provide assistance to all financially needy persons.

But a substantial number of those eligible for food stamps do not receive them. Most households that are

eligible for food stamps but do not participate in the program fall into one of two groups: households with

earnings (the working poor) and households with elderly persons (the poor elderly). Only about one half

of working households that are eligible for food stamps and one third of elderly households that are eligible

for food stamps participate in the FSP. The reasons for low participation by these groups must be

understood to determine the appropriate policy response. However, hard evidence on the reasons for low

FSP participation by these groups is scarce.

To improve their understanding of the low FSP participation by the working poor and poor elderly,

the Food and Consumer Service (FCS) of the U.S. Department of Agriculture contracted with

Mathematica Policy Research, Inc. to conduct a study, Reaching the Working Poor and Poor Elderly, of

the reasons for nonparticipation by the working poor and poor elderly. The study has three distinct, but

interrelated, phases: (1) a literature review and analysis of existing data, (2) the conduct and analysis of

twelve focus group discussions, with FSP participants and FSP-eligible nonparticipants in the two groups,

and (3) the development and testing of questionnaires about FSP nonparticipation by the working poor and

poor elderly to be administered to both FSP participants and FSP-eligible nonparticipants.

Under the third phase of the study we first developed eight separate questionnaires.' These

questionnaires differed by (1) whether they are to be administered to working poor or poor elderly

respondents, (2) whether they are to be administered to FSP participants or FSP-eligible nonparticipants,

and (3) whether they are short (about 15 minutes) or long (about 30 minutes). In addition to the main

'The eight questionnaires and screening interviews initially developed are in Cohen, McConnell,Milliner, and Ponza (1996).

DRAFT

1

questionnaires, we developed two short screening questionnaires. One of the screening questionnaires

will be administered to FSP participants identified from lists of active FSP cases. The other will be

administered to a sample of respondents identified by random digit dialing (RDD). Both screening

questionnaires will be administered prior to the main questionnaire and are designed to identify respondents

that fit our criteria for inclusion in the sample. For example, the screening questionnaire for the RDD

sample is designed to identify persons who do not participate in the FSP but are likely to be eligible for

food stamps. The initial versions of all the questionnaires were reviewed by FCS and then revised.

The questionnaires are being tested in two stages. The first stage entailed testing the questionnaires

using cognitive techniques. Retrospective and think-aloud interviews were administered to sixteen persons

to examine the cognitive processes used by respondents and to identify errors and biases in the

questionnaires. After revision in response to the lessons learned from cognitive testing, the questionnaires

will be pretested on about 400 respondents.

This report summarizes the results of the cognitive tests of the survey instruments developed for this

study. The remainder of the report is organized into two chapters. In Chapter II we describe the design

and implementation of the cognitive tests. In Chapter III, we summarize the results of the cognitive tests.

For each question with which respondents had problems, we provide a brief summary of the issues and

our proposed solution. The chapter also discusses cognitive test findings on questionnaire length, order,

and flow, and discusses implications for administering the questionnaire by telephone.

II. COGNITIVE TEST METHODOLOGY

Cognitive testing is a method by which researchers can identify respondents' errors in understanding

and answering questions. There are four stages in the process of responding to a question: (1) interpreting

the question, (2) retrieving the answer to the question or relevant information that will be used to construct

an answer, (3) formulating the answer, and (4) editing the answer if the respondent is concerned about the

risk of disclosure or the social desirability of the answer. Cognitive techniques attempt to identify bias or

other errors related to each of these four stages of the response process. Problems that can be identified

by cognitive testing include vague or imprecise questions, too abstract or difficult questions, vocabulary

problems, overly sensitive questions, bias caused by the order of the questions, and bias in recalling the

answer to the question.

This chapter describes the methodology for the cognitive tests conducted for this study.' In Section

A we describe the respondents who participated in the cognitive tests and the procedures used to identify

and recruit them. Section B describes the conduct of the cognitive tests. It includes a discussion of

logistics and the cognitive techniques used in the conduct of the tests. Section C describes the behavioral

coding undertaken as part of the cognitive testing.

A. COGNITIVE TEST RESPONDENTS

1. Number and Types of Respondents by Site and Cognitive Interviewing Technique

We tested the survey instruments on 16 persons sampled from two sites in Texas: Harris County and

Polk County (see Table 1I.1). Harris County is an urban area and Polk County is a rural area. These two

sites represented two of the six sites in which the study's focus groups were conducted. The six sites were

'Cohen, Eisenhower, McConnell, and Ponza (1996) and a subsequent memorandum by McConnell(1997) provides a full discussion of the design and implementation plans for the cognitive tests.

TABLE 11.1

NUMBER OF COGNITIVE TEST RESPONDENTS BY S11E

Working Poor Poor Elderly

SiteFSP

ParticipantFSP

NonparticipantFSP

ParticipantFSP

Nonparticipant Total

Harris County 1 3 4 4 12

Polk County 2 2 0 0 4

Total 3 5 4 4 16

FSP = Food Stamp Program

selected through a combination of purposive and random sampling.' We purposively selected the two

Texas sites as the locations for the cognitive tests.

No more than nine persons were asked any single item of information. Each of the four types of long

questionnaires--working Food Stamp Program (FSP) participant, elderly FSP participant, working FSP

nonparticipant, and elderly FSP nonparticipant--were administered to one respondent from beginning to

end. We used retrospective-thmk-aloud (RTA) cognitive techniques with these four people. Certain

subsections of the questionnaires and the screening interviews were administered to seven additional FSP

nonparticipants (four working and three elderly persons) and five additional FSP participants (two working

and three elderly). We used concurrent-think-aloud (CTA) cognitive techniques with these 12 people.

(Table 1I.2 shows the parts of the questionnaires we administered to each type of respondent.)

The advantage of this approach over testing a whole questionnaire on nine respondents (the maximum

that can be interviewed without requiring OMB clearance) was that we could ensure that nearly all sections

of the questionnaire were tested on nine people, even if the sections are not included in each questionnaire.

For example, if we had only nine respondents, five nonparticipants and four participants, Section E

(questions about reasons for nonparticipation) would only be tested on five people rather than nine people.

2. Respondent Profiles

Sixteen people were mterviewed using cognitive interviewing techniques. A brief description of each

respondent follows: 3

Rl :

Working FSP Participant (CTA, Polk County). The respondent is a 37 year old male Caucasian.He is a single parent with a 5 year old daughter. He works full time as an automobile window

'See Ponza and McConnell (1996) for a description of the criteria and methods used to select the sites.

'Respondents are given a number in order to keep their identity confidential. The respondent numbercorresponds to the respondent type in Table 11.2. See the table to ascertain which questions particularrespondents were administered.

CT

TABLE 11.2

SECTIONS OF QUESTIONNAIRES ADMINISTERED TO EACH TYPE OF RESPONDENT IN THE COGNITIVE TESTS

Screening

ScreeningLocation

Interview

Interview

E (Nonparts

H (Working

I (Elderlyof Test'

(First Part)"

(Second Part)`

A&B

C

D

only)

F&G

only)

Only)

J,K,&L

1 Part, Working

P

x

x

x

x

x

x

x

2 Part, Elderly

H

x

x

x

x

x

x

x

3 Nonpart, Working

H

x

x

x

x

x

x

x

x

x

4 Nonpart, Elderly

H

x

x

x

x

x

x

x

x

x

5 Nonpart, Working

P

x

x

x

x

x

x

x

6 Nonpart, Working

H

x

x

x

x

x

x

x

7 Nonpart, Working

II

x

x

x

x

x

x

8 Nonpart, Working

P

x

x

x

9 Nonpart, Elderly

II

x

x

x

x

x

x

x

10 Nonpart, Elderly

H

x

x

x

x

x

x

11 Nonpart, Elderly

H

x

x

x

12 Part, Working

H

x

x

x

13 Part, Elderly

H

x

14 Part, Elderly

H

x

x

15 Part, Working

P

x

x

x

x

16 Part, Elderly

H

x

x

x

x

Total .

9

9

9

9

9

9

9

8

8

9

'H=Harris County and P=Polk County.

"Questions S l-S5 of FSP nonparticipant screening interview and all of FSP participant screening interview.

`Questions S6 to S18 of FSP nonparticipant screening interview.

Type of Respondent

tinter. He has lived in Polk County since moving from a Southwestern state. He had receivedfood stamp benefits there before relocating to Texas. He was a competent respondent with agood understanding of the questions.

R2: Elderly FSP Participant (RTA, Harris County). Respondent is a 73 year old African-Americanfemale. She lives alone--her husband died about one year ago. She still works part-time for theschool district and reported that she and her husband worked all of their adult lives. She wasrated as a competent respondent with an overall good understanding of the questions.

R3: Working FSP Nonparticipant (RTA, Harris County). Respondent is a 36 year old African-American female. She lives with her husband who works full-time. She has two children in thehousehold. She has no previous experience with the Food Stamp Program. She had an excellentunderstanding of the questions.

R4: Elderly FSP Nonparticipant (RTA, Harris County). Respondent is a 78 year old African-American female. She is diabetic. She was depressed and cried throughout the entire interview.She would sometimes get confused and have difficulty concentrating. She had fair to pooroverall comprehension of the questions.

R5: Working FSP Nonparticipant (CTA, Polk County). Respondent is a 24 year old Caucasianfemale. She is a single parent with a 7 year old daughter. She works as a cashier at a localsupermarket. She lives in subsidized housing and also receives subsidized child care. Herdaughter has been receiving Supplemental Security Income (SSI) for the past two years. She didnot have a good understanding of several of the questions and did not understand the meaningof several words.

R6: Working FSP Nonparticipant (CTA, Harris County). Respondent is a 35 year old African-American female. She is temporarily disabled. She lives with her husband who works full-time.Her 17 year old son also works 25 hours a week. She considered a 16 year old nephew to be partof her household. She currently receives Workmen's Compensation for a job-related accident.She had applied for FSP benefits when she stopped working in December, but was told by aninsurance company that she couldn't receive both Workmen's Compensation and food stamps.The application is still pending. She had a good understanding of the questions.

R7: Working FSP Nonparticipant (CTA, Harris County). Respondent is an African-Americanfemale (age unknown). Her husband earns about $12,000 per year and she has seven childrenliving in the household. One of her sons has spina bifida. While she had a good overallunderstanding of the questions, she tended to focus on what was going on with others whenresponding rather than her own household.

R8: Working FSP Nonparticipant (CTA, Polk County). Respondent is a 43 year old African-American female. She works full-time as a secretary and office manager. Her husband alsoworks. There is one child in the household. She is a former FSP participant: three years ago shereceived $64 in food stamp benefits for a household of seven persons. She had a goodunderstanding of the questions.

Elderly FSP Nonparticipant (CTA, Harris County). Respondent is a 67 year old African-American female. She is diabetic and uses a cane. She lives alone, but has relatives who livenearby. She worked all of her adult life until her retirement. She was competent and had a goodoverall understanding of the questions.

RIO: Elderly FSP Nonparticipant (CTA, Harris County). Respondent is 68 year old African-American female. She lives in senior citizen housing. She has a daughter who lives nearby. Shemoved from a Southern state where she had received food stamps most of her adult life. Sheis reluctant to apply for food stamp benefits in Houston, however. She had a good understandingof the questions.

R11: Elderly FSP Nonparticipant (CTA, Harris County). Respondent is a 72 year old African-American female. She and her husband are both retired. She was an active community workerand did volunteer work in the local food bank and nursing homes. She has assisted people withthe FSP application process. She had a good understanding of the questions, but would oftenfocus on the problems of others and not herself when responding.

R12: Working FSP Participant (CTA, Harris County). Respondent is a young (age unknown)Hispanic woman. She is married with a 15 month old baby. Recently her 5 year old stepsonmoved into her household. Her husband works, earning about $9 per hour. She had an excellentunderstanding of the questions.

R13: Elderly FSP participant (CTA, Harris County). Respondent is a 65 year old African-Americanfemale. She has been disabled since 1987 and ambulates with a quad cane. She has severalserious health conditions. Her 13 year old grandson lives with her. She receives $64 a monthin Aid to Families with Dependent Children (AFDC) for her grandson. She had a fair to goodunderstanding of the questions.

R14: Elderly FSP Participant (CTA, Harris County). Respondent is a 73 year old African-Americanfemale. She is a diabetic and is also disabled. She lives alone. She sometimes lost focus inanswering the questions when conversation was allowed. She had a good to fair overallunderstanding of the questions.

R15: Working FSP Participant (CTA, Polk County). Respondent is a 29 year old Caucasian female.She is a single parent and has been separated from her husband for the past two years. Her 6year old daughter lives with her; her 10 year old son lives with her spouse. The son stays withher on weekends and school vacations. She works full-time as a waitress. She receives a smallamount of financial support from her mother. She had an excellent understanding of thequestions.

R16: Elderly FSP Participant (CTA, Harris County). Respondent is a 65 year old Caucasian female.She is a diabetic with high blood pressure, and is disabled. Her brother lives with her, but theylive separate lives. She sometimes got confused and distracted during the interview. Her overallcomprehension of the questions was fair to poor.

3. Identifying and Recruiting Respondents for the Cognitive Tests

Respondents came from two sources: focus group participants and FSP participant lists. We

recruited the respondents for the cognitive tests of the FSP nonparticipant questionnaires from people who

agreed to participate in the focus groups conducted as a part of this study in the late summer and fall of

1996. Respondents for the FSP participant questionnaires were selected from local FSP offices in Harris

and Polk County. We asked for about 30 names of participants in Harris County who were working or

elderly and about 10 names of participants m Polk County who were working. (We did not need to recruit

respondents for the working FSP participant questionnaire from the list of persons who attended the

working FSP participant focus group in Polk County.)

We first contacted potential respondents by telephone.' During the telephone call, we briefly

described the study and the cognitive tests to the respondents. We asked them whether they could attend

the cognitive tests on the scheduled day and confirmed their address. We then sent each recruited

respondent a letter confirming the date, time, and location of their meeting with the researcher. We then

made two reminder calls to respondents. One reminder call was made between one week and up to a few

days before the interview, depending on when the respondent was recruited and when he or she was

ultimately contacted. We then called respondents the night before the day of their tests to remind them

again. These reminder calls were critical to the success of the cognitive tests, as all 16 of the persons

initially recruited for the tests kept their scheduled appointments. We did not need to turn to backup

respondents.

'Our initial contact with the sampled FSP participants was a letter from MPR informing them of thestudy, and that MPR would call them within one week.

B. COGNITIVE TEST PROCEDURES

1. Logistics

Ms. Rhoda Cohen and Dr. Donna Eisenhower conducted the cognitive tests. The tests spanned three

days. In Harris County, the cognitive tests were conducted in a community resource center. This was the

same facility in which the focus groups were previously held. We used a conference room situated in a

local bank for the cognitive tests held in Polk County. We held the cognitive tests for the elderly

questionnaires on week days during the morning and/or early afternoon. We conducted the tests for the

workmg poor on week days during the afternoon.

Each respondent who showed up to participate in the cognitive testing was given $25 to cover costs

of participating in the interviews. In addition, it was necessary to arrange transportation assistance for all

but one of the elderly respondents m Harris County.

2. Cognitive Techniques Used

We used two cognitive testing techniques in this study: (1) the concurrent think-aloud (CTA)

technique, and (2) the retrospective think-aloud (RTA) technique. The next sections describe the

procedures followed in the interviews when applying these techniques.

a. Concurrent Think-Aloud Interviews

Twelve of the sixteen cognitive tests used the CTA method. Under the CTA method, respondents

are encouraged to verbalize what is going through their mind as they answer the question. Such

verbalizations reveal problems with comprehension, retrieval of information, recall or estimation, or with

other factors affecting the reporting of the response. The respondent is encouraged to present thoughts as

they occur without censoring them. As in psychotherapy, thoughts may surface which are not as logical

or as cohesive as usual verbalizations, but may reveal underlying problems which otherwise would not

surface. Respondent thoughts and feelings are probed extensively to assure adequate and comprehensible

detail to make judgements about questionnaire revision. The CTA technique is question specific--responses

are probed after each question is asked.

An advantage of the CTA technique is that it provides detailed information on comprehension,

question and answer choice completeness, and accuracy at the question-level. Furthermore, it asks the

respondent to comment when the question and answer choices are fresh in the respondent's mind.

However, because the "think-aloud" process occurs, concurrently, at the time of the question

administration, the respondent is less able to sense what the overall questionnaire is like in terms of length

and flow.

For the study's questionnaires, some non-priority questions received only normal administration

during the CTA interviews, some received a general "think-aloud" procedure to see if there is anything

of research interest, and some, determined to be priority questions, received systematic probing and

examination. Each of these three conditions were defined as follows:

• Normal Administration. This means the question was asked as it normally would bein a survey without allowing or encouraging tangential conversation. The respondentwas encouraged to restrict his or her response to the answer choice.

• Regular CTA Questioning. The interviewer paused after the question was asked andinstructed the respondent to "think aloud" as he or she answered. If of research interest,the respondent was encouraged to elaborate and, if not, discouraged from commentingfurther.

• Priority CTA Questioning. These questions were known to be of priority interest sothat CTA was applied but then followed with systematic probes or other cognitivetechniques to assure that adequate detail was obtained.

For this study, the priority questions were those from the section of the screening interview used to

determine eligibility status, and those from Section B, C, D, and E of the main questionnaire that ask about

reasons for participation and non-participation, and experiences during FSP participation.

The CTA protocols followed for the screening interviews and questionnaires can be found in Cohen

et al. (1996: Appendix C). 5 During the CTA interviews, respondents were also asked for their overall

reactions to the questionnaire at the end of the interview.' These questions asked the respondent: to give

their overall impressions about the questionnaire; to identify any reasons why they are or not participating

in the FSP not mentioned by them when answering the questionnaire; to identify any difficulties they might

have had distinguishing between time periods or circumstances as called for in the questionnaire; to

identify any confusing questions or response categories; and to make suggestions for improving the

questionnaire or eliminating redundancy.

b. Retrospective Think-Aloud Interviews

Four respondents were administered RTA interviews. In the RTA interviews, the questionnaire was

administered in the normal manner first, without probing or questioning the response process. The

respondent was then asked "retrospectively" about the questionnaire as a whole and about individual

questions.

Because the respondent has experienced the normal administration of the questionnaire, they are better

able to address overall questionnaire issues of length, flow, and organization. Examples include what

sticks in the respondent's mind as the most difficult questions to answer or as the best and worst features

of the questionnaire. In this sense, the gestalt of the questionnaire can be better examined under the RTA

conditions. The respondent can also be asked to focus "back" on a particular question and comment on

'We developed a protocol for each of the eight questionnaires. However, for brevity and ease ofreview, Cohen et al. (1996) presents a CTA protocol for a "master" questionnaire. The masterquestionnaire includes all questions included in any of the eight questionnaires. The systematic probes andspecific techniques for each question are presented next to each facing page of the questionnaire. Whetherthe question receives a normal administration, a general "think-aloud" probe, or systematic probing isnoted.

'See questions at the end of the questionnaire in Cohen et al. (1996: Appendix C).

DRAFT

12

how they answered it or what was going through their mind at the time they answered. However, fewer

individual questions are probed with the RTA technique.

The first ten questions of the RTA protocol assessed the respondents' evaluation of the questionnaire

overall.' Respondents were asked to give their overall impression of the questionnaire, and then to

comment specifically about its length, flow, and organization. They were also asked to identify questions

or aspects that were not covered which they felt should be covered, as well as to identify those that were

covered which they thought should not be. As in the CTA interview, respondents were also asked to

identify any reasons they participate or do not participate in the FSP not covered or mentioned by them in

the interview, to discuss any problems they might have had distmguishmg time periods or circumstances,

or understanding questions or response categories.

After being administered these general questions, interviewers asked respondents about some specific

questions on the screening interview and questionnaire. These questions were the ones about participation

and nonparticipation that had been newly created for this study, and include sl, sla, sl2a, and s13 from

the screening interview, and C14, D7, and E3 from the main questionnaire. We followed the procedures

and probes shown in the CTA protocol for these selected questions. The only difference is that the

question and original answer were first read back to the respondent before the CTA procedures were

applied.

c. Specific Methods Used in the CTA and RTA Interviews

Particular questions were examined using an array of specific cognitive techniques which were either

used concurrently or retrospectively. The use of these techniques are identified as part of the detailed

protocols. The techniques are defined as follows.

'The protocol for the RTA interview can be found in Cohen et al. (1996: Appendix D).

Open, Free Association: The respondent was allowed to think aloud in an open way as theyanswered the question. If there was anything of research interest, systematic probing was then used.

Systematic Probing: Detailed questions were asked about the question or answer categories or theresponse process itself.

Decomposition: The respondent was asked to breakdown the components of an answer to determineits accuracy.

Confidence or Difficulty Ratings: The respondent was asked to pick a rating on a numerical scalealong some dimension like confidence in the accuracy of their answer or the level of difficulty.Similarly, short answers were scaled to see what detail was missed in the shorter versions of answerchoices such as in simpler yes/no responses.

Paraphrasing: The respondent was asked to paraphrase the question or answer choices, that is, toput them in his or her own words. This provided an indication of respondent comprehension. Thistechnique was used in this study when "think-aloud" responses revealed or were expected to revealproblems.

d. Audio-Taping Interviews

All cognitive interviews were audio-taped. This provided the most accurate way of taking notes.

Since there was only one cognitive interviewer for each cognitive interview, audio-taping provided accurate

recording while allowing the interviewer to concentrate on the administration of the cognitive interview.

Later the interviewer listened to each tape and took notes. This formed the basis for analyzing results of

the cognitive tests and formulating recommendations for revising the questionnaires. The tapes from the

four RTA interviews were also used for timing the interviews and for behavioral coding.

C. BEHAVIORAL CODING

Behavioral coding is a systematic way to assess the behavior of interviewers and respondents during

the administration of a survey questionnaire. We coded the four RTA interviews. For each of the four

RTA interviews, the questionnaire was administered in a normal survey manner and audio taped. Using

the tapes, predetermined codes were then used to count specific behaviors indicative of problems with the

questionnaire. Dr. Eisenhower and Ms. Cohen performed the behavioral coding. The results of the

behavioral coding were used, along with the other findings from the cognitive tests, to identify and fix

problems with the questionnaires.

Cohen et al. (1996) contains the behavioral coding form that was used. The form emphasizes the

behavior of the respondent and the degree of interaction between the interviewer and the respondent.' The

codes presented in Part I of the form were operationalized as follows:

01 Long Pause Before Answering: Code 01 was recorded for each question in which the pausebefore answering exceeded two seconds. The time begins immediately after the interviewer finishesasking the question and the respondent begins to give an answer.

02 Repeat Question: Code 02 was recorded when the respondent asked the interviewer to clarify aterm, the whole question, or whole or part of an answer choice. It was also recorded when therespondent asked the interviewer to repeat a question or when an interviewer repeated the questionafter obvious respondent confusion.

03 Objection to the Question: Code 03 was recorded when the respondent specifically articulateda negative response to the question itself such as "that does not make sense," or "that's kind ofstupid."

04 Question Refusal: Code 04 was recorded only when the respondent specifically refused toanswer the question. It can be the respondent's first or second refusal to answer the question.

05 Reluctance to Answer: Code 05 was recorded for any question in which the respondent stateda preference not to answer the question, or the respondent provided an answer but did so reluctantly.

06 Ask for Clarification: Code 06 was recorded when the respondent specifically asked theinterviewer to define or explain the meaning of a term or terms in the question stem or in the answerchoices.

07 Qualification ofAnswer: Code 07 was recorded when the respondent answered the questionconditionally (an answer is provided but restricted to certain circumstances). For example, therespondent might have said that I work full time but only in the summer or it is true that my familyhas enough food except when my son's children come to stay with us in the summer months.

08 Engages in Digression: Code 08 was recorded whenever the respondent provided informationtangential to the question being asked. The digression could be totally unrelated to the interview orrelated but must be tangential to the answer of that question.

'The behavioral coding form for the pretest will also include codes for interviewer behavior.

09 Interrupts the Reading of Question: Code 09 was recorded whenever the respondent interruptedthe reading of the question before the interviewer had completed the first reading. The interruptioncould be for any reason, but occurred while the first reading was in progress.

10 Other: Code 10 was recorded whenever the respondent displayed some behavior other than thatlisted above. (The interviewer would describe the problem in space provided on the form.)

The degree of interviewer and respondent interaction prior to the respondent answering a question was

operationalized as follows:

Low: The respondent paused, had only one comment or no comments before the respondentanswered the question.

Medium: The respondent asked a question or made a comment and the interviewer replied beforethe respondent answered the question.

High: The respondent asked a question or made a comment, the interviewer replied, and therespondent made another comment before the respondent answered the question. (Comments aredirected to the interviewer and may or may not be related to the survey topics).

As the cognitive tests were conducted by experienced survey researchers who were involved m the

design of the questionnaire, we did not code interviewer behavior during the cognitive tests. However,

because frequent incorrect interviewer behavior may indicate problems with the questionnaire, we will

code interviewer behavior during the pretest. The interviewer behavior codes will be as follows:

11 Makes Minor Wording Change. Code 11 will be recorded when the interviewer makes a minorwording change to the question or probe, but this change does not change the meaning of the questionor probe.

12 Makes Major Wording Change. Code 12 will be recorded when the interviewer makes awording change that changes the meaning of the question or probe.

13 Incorrect Probing. Code 13 will be recorded when the interviewer probes incorrectly. Thisincludes overprobing or other inappropriate probing.

14 Fails to Probe Unclear Response. Code 14 will be recorded when the interviewer fails to probewhen the respondent provides an unclear answer to the question.

15Adding or Skipping Questions Inappropriately. Code 15 will be recorded when the interviewereither asks questions that they should not or skips questions.

16 Recording Errors. Code 16 will be recorded when the interviewer codes the answer incorrectly.

III. COGNITIVE TEST RESULTS

Overall the screening and main questionnaires worked well and were well-received by respondents.

However, as expected, the cognitive testing identified several questions that require revision of one form

or another. Respondents were more likely to have problems answering the questions that have been newly

created for this study, as opposed to questions drawn from previously administered national surveys. In

most cases, the revisions we have made in response to the cognitive testing are relatively minor and entail

retaining questions largely as is but changing a word or phrase, adding additional response categories, or

adding respondent probes and interviewer instructions. In other cases, the results of the cognitive testing

call for larger changes, such as developing additional sets of questions or breaking a single question into

two or more questions or components.

Questionnaire order, flow, and length do not appear to be a problem. In particular, the long-versions

of the instruments were found to take on average approximately thirty minutes to administer. The revisions

we have made in response to the cognitive testing have led to an increase in the number of questions in the

long questionnaire. However, we believe that these changes have increased the length of the average

interview by no more than a couple of minutes.

The remainder of this chapter presents a synthesis of the cognitive test findings. Section A provides

a question-by-question summary of problem questions and our recommendations for resolving them.

Section B discusses cognitive test findings about length, question-order, and flow of the survey

instruments. It also discusses the implications of the cognitive tests for the administration of the

questionnaires by telephone. Section C describes some changes to the questionnaire that we have made

in response to changes in Food Stamp Program (FSP) eligibility as a result of recent welfare reform. The

revised versions of the questionnaires are found in McConnell et al. (1997).

It should be noted that we coded the behavior of the respondent during the four complete interviews

in which retrospective-think-aloud cognitive techniques were used. However, the behavioral coding did

not reveal any new issues or problems about the questionnaires that were not revealed during the

administration of the questionnaires using concurrent-think-aloud nor retrospective-think-aloud cognitive

techniques. For this reason, we do not discuss findings from behavioral coding aspects of the cognitive

testing separately.

A. DETAILED FINDINGS BY QUESTION

This section identifies questions that one or more respondents had difficulty answering. For each

problem question, we describe the issues and then our recommendations for resolving them. The

discussion begins with the screening questionnaires and then turns to the main questionnaires. The

convention followed in this section when identifying problem questions is to list the question (in bold

italics) as it was pretested. That is, the question (and question number) refer to the question before any

revisions have been made based on the cognitive testing. In addition, since the question numbers and

questions are the same across the different questionnaires, any revisions to a specific question will apply

to all the questionnaires that contain the particular question.

1. Screening Questionnaires

Two screening questionnaires have been developed for the study. One, to be used with the Random

Digit Dialing (RDD) sample frame, will screen-in households that are not participating in the FSP but are

likely to be eligible for food stamps. The other, to be used with the sample frame developed from FSP

caseload files, will screen-in currently participating food stamp households who have applied for food

stamp benefits within the previous 36 months. The remainder of this section presents the findings from

the cognitive tests of these two questionnaires.

a. Random-Digit-Dialing Screening Interview

For the RDD screening interview, the problem questions and our recommendations for resolving

them, are as follows:

sl.

How many people live in your household? By household, I mean yourself and the people wholive with you and share food with you.

Issue: Two respondents asked whether they should include persons who live with themintermittently.

Response: States differ in how they treat persons who live in the household intermittently. Moststates have complex rules that require more information than we want to collect in thesequestionnaires. We decided to count a person as in the FSP household if they live in the householdmore than half the time. We added a probe to the question:

"PROBE: Include any persons who live with you more than half-time."

sla. During [PREVIOUS MONTH], was (your/your household's) income less than [AMOUNT]before taxes and other deductions?

Issue: One respondent did not understand the time-frame in the question. We also thought thatsome people did not understand the phrase "before taxes and other deductions."

Response: We clarified the question to state "During the month of [PREVIOUS MONTH]." Wealso added "taken out" to the end of the question so that the last phrase of the question now reads"before taxes and other deductions are taken out."

(Have you/Has anyone in your household) been found ineligible to receive food stamp benefitsduring (fill PREVIOUS MONTH] or ill CURRENT MONTH)?

Issue: We had two concerns with this question. First, we were concerned that some people maysay they have been found ineligible because they decided on their own that they were ineligible,rather than the food stamp office determined their household was ineligible, which is the intent ofthe question. Second, we were concerned that people might confuse "ineligible" with "eligible"over the telephone.

Response: We changed the wording of the question from "been found ineligible" to "beennotified that [you/they] were not eligible." We changed the word "ineligible" to "not eligible"throughout the screening interviews and questionnaires.

s4. Did you or anyone else in your household work at a job for pay during ill PREVIOUSMONTH] or ffll CURRENT MONTH)? Please exclude any jobs held by school childrenunder 18 years of age who live with you.

Issue: One respondent, who had a part-time job, said she did not work.

Response: We added a phrase in the probe "include all full and part-time jobs."

sll. Now, think about the people in your household who are disabled or age 60 or older. Weretheir out-of-pocket medical expenses more than $35 last month?

Issue: Three issues were raised about this question. First, respondents needed a definition of"disabled." Second, the question is awkward when the respondent lives alone. Third, somerespondents asked whether specific items were included as medical expenses.

Response: We added a question (slO) before sl 1 which defines "disabled." We approximatedthe definition of "disabled" used by the FSP. Question sl0 reads:

"s10. Is anyone in your household disabled? Please count as disabled persons who receive SSIbenefits because of a disability, social security disability checks, disability retirementpensions, railroad retirement disability payments, or veteran disability benefits."

We also inserted a CATI instruction (CH2a) that skips the question about disability if it is a single-person household. The question about disability need not be asked of respondents in single-personhouseholds because we know the respondents who are asked s9 are elderly. We inserted analternative wording of question sl l for single-person households. For single-person households,sl l now reads:

"sl 1. Were your out-of-pocket medical expenses more than $35 last month."

In interviewer training, we will provide interviewers a list of common medical expenses that areincluded and a list of common medical expenses that are excluded.

s12. Next, I have some general questions about (your/your household's) assets. By household, Imean yourself and the people who live with share food and other resources.

Issue: The definition of household was not consistent with version used in previous questions.

Response: We changed the definition of household in this question to "By household, I meanpeople who live with you and share food with you."

s12a. Next, I have some general questions about (your/your household's) income and assets. Byhousehold I mean yourself and the people who live with you and share food and otherresources. During ill LAST MONTH], was (your/your household's) income less than illAMOUNT] before taxes and other deductions?

Issue: It worked better in the cognitive tests for the interviewer to say they had a general questionabout income rather than questions about income and assets, and then later say they were goingto ask about assets. The definition of household is not consistent with the definition used in earlierquestions. Some respondents excluded some types of income (such as Workmen' Compensation).

Response: We changed the phrase "some general questions" to "a general question." Wecorrected the definition of household so it is consistent with previous usage. We added a sentenceto the question that states "Please include all sources of income from all household members."

s13. Now think about any cash (you have/your household has) on hand, and money in checkingand savings accounts. Are these amount together less than [fill AMOUNT'?

Issue: The sequence of questions on assets needs a lead statement to say that the interviewer willnow be asking about assets.

Response: We included a lead statement "Next, I have some general questions about (your/yourhousehold's) assets?"

s14. (Do you/Does your household) have any other kinds of financial investments or accounts?Please include savings bonds, individual retirement accounts, pension plans, stocks andbonds, money market funds, and savings for burial expenses.

Issue: Pension plans are not usually counted as assets when determining FSP eligibility.

Response: We excluded "pension plans" from the list of examples of assets.

sl9a. New Question

Issue: Only applicants who applied within the past three years should be included in the FSPparticipant sample. The version of the screening interview tested did not screen out FSPparticipants who last applied for food stamp benefits more than three years previously.

Response: We inserted a question in the RDD screening interview (sl9a) to find out whether aFSP participant respondent applied for food stamps within the past three years.

b. List-Frame Screening Interview

For the list-frame screening interview, the problem questions and our recommendations for resolving

them, are as follows:

s3a. New Question

Issue: The list-frame will only include FSP participants who applied within the past three years.Hence, it is not necessary to check this in the screening interview. However, we do need to usethis screen in the RDD screening interview. We are concerned that some respondents may answerincorrectly.

Response: To enable us to estimate the frequency of persons incorrectly reporting theirapplication date we will ask a question on the list-frame screening interview that is identical to thenewly inserted question in the RDD screening interview (sl 9a). In theory, no respondent from thelist-frame should answer "NO" to this question. Those respondents who do will indicate eithera reporting error or an error in the FSP data. No respondent from the list-frame will be screenedout on the basis of their response to this question.

s4. How many people live in your household? By household I mean yourself and the people wholive with you and share food with you.

Issue: Same issue as described for Question sl of the RDD screening interview.

Response: Same as question sl of the RDD screening interview--we inserted a probe, as follows:

"PROBE: Include any persons who live with you more than half-time."

s5. Did you or anyone (else) in your household work at a job for pay during PREVIOUS orCURRENT MONTH? Please exclude anyjobs held by school children under 18 years ofagewho live with you.

Issue: Same as question s4 of the RDD screening interview.

Response: Same as question s4 of the RDD screening interview--we added a phrase in the probe"include all full and part-time jobs."

2. Main Questionnaires

There are a total of twelve topic modules across the eight separate questionnaires. This section

provides detailed findings from the main questionnaires by question, organized around each topic module.

As above, the convention we follow is to first list the question, then provide a brief statement of the issues

or problems with the question, and then describe how we have resolved the problem. The question number

and question refer to the version of the question that was pretested.

The changes discussed below are made to the question in each questionnaire, unless specifically noted

otherwise. The questions are numbered identically in each questionnaire. For example, question B1 is the

same on each questionnaire.

a. Section A: Household Composition

Section A of the questionnaire asks respondents to list who lives m their household, their relationship

to the respondent, and the ages of household members. The following questions in Section A had one or

more problems:

Al.

Please tell me thefirst name or initials ofeach person who currently lives in your householdstarting with you. By household I mean people who share food with you.

Issue: Some respondents were unclear about whether to include persons who live with them onlysome of the time. This issue also arose in the screening interviews.

Response: We inserted a probe,

"PROBE: Include any persons who live with you more than half time."

A2. How is (NAME) related to you?

Issue: Later in the questionnaire, the interviewer needs to know whether the household includesa female. With the relationship codes in the old version of the questionnaire, we could not alwaystell whether the household member was female.

Response: We changed the relationship codes so that it is always clear whether the householdmember is male or female. The odd number relationship codes are males.

b. Section B: Knowledge of the FSP

This section of the questionnaire collects information about factors that may be related to the

respondents' knowledge of the program, such as whether the respondent received food stamp benefits as

DRAFT

24

a child and whether they know someone who received food stamp benefits. Nonparticipants are asked

whether they would know where to apply for food stamp benefits, whether they think they are eligible for

food stamp benefits (and if not, why not), and the amount of food stamp benefits they think they would be

eligible for. Most of the questions were developed specifically for this survey. Respondents had problems

with the following questions:

B1. As far as you know, did your family ever receive food stamps when you were a child?

Issue: Some of the elderly respondents said that there was no Food Stamp Program when theywere children.

Response: As the FSP in its current form did not exist when persons 60 years of age and olderwere children, we excluded this question from the elderly questionnaires. Some respondents tothe elderly questionnaires will be younger than 60- (if they reside in an elderly household).However, we did not think it was worthwhile to collect this information from that smallsubsample. The question remains in the working questionnaires.

B2. Asfar as you know, do any of your relatives, neighbors, or co-workers currently receive foodstamps benefits?

Issue: Some respondents said that they had friends who received food stamp benefits.

Response: We changed the wording of the question to include "friends." The question nowreads,

"B2. As far as you know, do any of your relatives, friends, neighbors, or co-workers receivefood stamp benefits?"

B3. Now, I'm going to read a list of different kind of households. Please tell me if you think low-income households like these may be eligible to receive food stamp benefits. Householdsreceiving social security? Households where one or more people work? Households withpeople collecting a pension? Homeowners? Households without children? Households withelderly people?

Issue: All respondents had difficulty with this question. Some respondents asked for moreinformation about the households. Others thought we were asking whether these types ofhouseholds should get food stamps.

Response: We deleted this question from all questionnaires. We could not think of a way to wordthis question that would not cause similar problems. Later in this section, we ask respondentswhether they think they are eligible and, if not, why not. This is an easier question for a respondent

DRAFT

25

to answer and will provide some information about the respondent's knowledge of FSP eligibilityrequirements.

B5. Do you think you are currently eligible to receive food stamp benefits?

Issue: Two respondents were reluctant to give an answer because they had not checked with theFSP office.

Response: We will stress that the respondents should tell us whether they think they are eligiblefor food stamp benefits. We have added a phrase to the beginning of the question, so the questionnow reads

"B5. Based on what you know about the Food Stamp Program, do you think you are currentlyeligible to receive food stamp benefits?"

c. Section C: FSP Participation History

This section asks about experiences respondents (both participants and nonparticipants) have had

receiving food stamp benefits. It also asks nonparticipants who have previously received food stamp

benefits why they stopped receiving food stamp benefits. Respondents are also asked for the amount of

food stamp benefits received and whether they are/were received in coupons or by Electronic Benefit

Transfer (EBT). Most of these questions were developed specifically for this survey. However, some

questions were adapted from the NFSPS. The following were problem questions:

C2.

Over the course ofyour adult life, since age 18, have you ever received food stamp benefits?

Issue: Two respondents needed interviewers to repeat this question. The problem was thatrespondents thought about whether they had received food stamp benefits "over the course of theiradult life" and then thought about whether they had received food stamp benefits "since age 18."The intent of the phrase "since age 18" was to define what we mean by "adult life".

Response: We deleted the phrase "since age 18." Interviewers will be trained to define "adultlife" as "since age 18" if the respondent asks for a definition.

C2a. New Question.

Issue:The Personal Responsibility and Work Opportunity Reconciliation Act of 1996 (PRWORA)places time limits on the receipt of food stamp benefits by able-bodied adults without children,who work less than half-time. Although we expect that less than half a percent of our sample of

DRAFT

26

working poor will be affected by this provision, we wanted to add a question that investigatedwhether the respondent was possibly ineligible because of the time limits.

Response: We inserted a question in all questionnaires:

"C2a. Did you receive food stamp benefits for more than three months since January 1, 1997."

The time limits were imposed in November 22, 1996 (three months after the enactment of thelaw). We used January 1, 1997 as the date that time-limits began, because of the ease ofremembering events that occurred during a calendar year. If the questionnaire is fielded afterNovember 22, 1999, the question can be rephrased to state "in the past 3 years. "

C4. How did you get your food stamp benefits in (DATE FROM C3)? Did you get coupons orcredit to an EBT card?

Issue: One respondent said she got her benefits issued to her Lonestar card. This is the name forthe EBT card in Texas.

Response: We added a probe that gives the local name for the EBT card.

C6. Why did you stop receiving food stamp benefits in (DATE FROM C3)? Seven possiblereasons were given, and the respondent was asked whether each was a reason they stoppedreceiving food stamp benefits.

Issue: Respondents found the list of possible reasons too long.

Response: We have deleted one of the possible reasons "because you disliked regularly reportingto your caseworker." If this is a reason it can be reported in the "other reason" part of the question.We also inserted a probe "Did you stop receiving food stamp benefits..." to be read before eachreason if necessary.

Cllb. Why did you think you were no longer eligible? Was it because you had too many assets?Issue: One respondent did not understand the term "assets."

Response: We rephrased the question to read "because you had too much money or other assets."

C12. In the past 3 years, did you ever stop receiving food stamp benefits because you were notifiedthat you were ineligible?

Issue: If the response was "No," "Don't know," or "refused" the questionnaire instructed theinterviewer to skip to question C14. This incorrectly missed question Cl 3 c.

Response: We corrected the skip instruction so that the interviewer is instructed to skip toquestion C 13 c.

DRAFT

27

C14. Now I'm going to read some statements about experiences with food stamp benefits. As I readeach statement, please tell me if the statement was true for you over the past three years. Theinterviewer then read seven statements about experiences with food stamp benefits.

Issue: Although this question worked better than others with the same structure, the interviewersneeded to probe "is this true for you" frequently.

Response: We changed the structure of the question so that it is a series of direct questions. Thelead-in to the question now reads,

"C14. Now I'm going to ask you some questions about your experiences with food stampsbenefits over the past three years "

The statements were converted to direct questions. For example, the statement "I was sometimestreated disrespectfully when using food stamp benefits in stores" was changed to "Were yousometimes treated disrespectfully when using food stamps benefits in stores?"

d. Section D: History of Applications

This section collects information from respondents about the reasons they applied for food stamp

benefits, whether they applied for other benefits at the time they applied for food stamp benefits, how the

respondent applied for food stamp benefits (such as, in-person or via authorized representative), an

specific problems that they may have encountered applying. The section also identifies respondents who

started the application process but did not complete it. These respondents are probed for the reasons they

did not complete the process. In this section we also ask participants about factors that helped them

overcome any problems applying for or using food stamp benefits. Most of these questions were

developed specifically for this survey. Some questions were adapted from the NFSPS. Problem questions

include:

Dl. Have you applied for food stamp benefits in the past 3 years? PROBE: By applied I meanhave you at least completed an application form.

Issue: Some respondents asked questions that showed that they were confusing the application. process with the recertification process. Also, this question is not needed for FSP participants as

the sample will only contain participants who applied in the past three years.

Response: The probe was changed to emphasize that we mean a new application, and is nowincluded as part of the question:

DRAFT

28

"By applied, I mean have you at least completed a new application form."

The question is deleted from the participant questionnaires.

D4a. The last time you applied, where did you obtain the application form for food stamp benefits?

Issue: Some respondents gave the name of the town or street address of the place they receivedthe form rather than the type of agency from which they received the form.

Response: We replaced this question with two new questions. The first asks whetherrespondents filed the application form at the food stamp office (with a probe to explain what wemean by "filing an application"). If they answer "no," we ask the question D4b "Did you file theapplication form at an SSI office, a senior center, a hospital, a housing complex, or somewhereelse?"

D4b. Did you or someone else complete the application form at [LOCATION FORM WASOBTAINED) or didyou complete it at home?

Issue: We do not think this question is necessary.

Response: The question was deleted. There is a new question D4b (see above).

Doc. Did you or someone else mail the application form or take it to [LOCATION FORM WAOBTAINED) in person?

Issue: We do not think this question is necessary.

Response: The question was deleted.

D4f New Question.

Issue: We want to ask directly whether the respondent needed to take time off work to apply forfood stamp benefits.

Response: We added a question D4f that asks directly,

"D4f. Did you take time off from work to apply for food stamp benefits?"

D4g. No Question

Issue: We would like to ask nonparticipants if they had a lot of help applying for food stampbenefits when they applied in the past.

DRAFT

29

Response: We inserted a new question,

"D4g. The last time you applied, did you have a lot of help with the application?"

D5f Which of the following reasons led you to apply for food stamp benefits in (MONTH, YEARFROMD2)? You couldn't get as much food from family, friends, government programs, orother sources?

Issue: Some respondents had difficulty understanding the reason.

Response: We changed the statement to "You couldn't get as much food as you needed fromfamily, friends, government programs, or other sources (such as food banks)."

D5g. Which of the following reasons led you to apply for food stamp benefits in (MONTH, YEARFROM D2)? (g) You learned more about the program or your eligibility for food stampbenefits?

Issue: Several respondents asked what we meant by "more" in reason "g." These respondentssaid that an impetus to applying for food stamp benefits was that they learned about the program,where previously they knew nothing about the FSP.

Response: We deleted the word "more." Because we have inserted other questions (see below),this question becomes (i).

D5f Which of the following reasons led you to apply for food stamp benefits in (MONTH,YEAR FROM D2)? (f) You couldn't get as much food as you needed from family,

friends, government programs or other sources (such as food banks).

Issue: This question only refers to quantities of food. It ignores other aspects such as notbeing able to get the kinds of foods they wanted or not wanting to get foods from thesesources.

Response: We inserted two additional questions:

"D5g. You could not get the kinds of food you needed from family, friends, governmentprograms or other sources."

"D5h. You did not want to get food from family, friends, government programs or othersources (such as food banks)."

D7. Now, please tell me about your experiences applying for food stamp benefits over the pastthree years. As I read each statement, please tell me whether it was true for you. (We thenread 11 statements).

Issue: Respondents found the structure of this question difficult to understand. We decided thatenough elderly respondents worked that it would be worth including the questions about takingtime off work and losing pay to apply for food stamp benefits (D7c and D7d) in all thequestionnaires, rather than in just the working questionnaires. Also some respondents queriedwhat we meant by "a long time" in the statement about waiting a long time to be served at the foodstamp office (D7f).

Response: We changed the structure of the question so that it is a series of direct questions ratherthan statements. The lead statement to the questions is now,

"Now I am going to ask you some questions about your experiences applying for food stampbenefits over the past 3 years."

We inserted the two questions D7c and D7d in the elderly questionnaires. We added a probe toD7f which defined what we mean by "a long time." The probe is,

"PROBE: A long time from your point of view. "

D17. I am now going to read some statements people have made about why they receive foodstamp benefits, despite having difficulties applying for or using them. Please tell mewhether each statement is true for you.

Issue: The format of this question did not work well.

Response: We changed the format of the question. The lead statement is now, "Now Iwould like to talk to you about some things that may have helped you decide to use foodstamps benefits." Each of the statements (a) through (e) were then changed to questions. Forexample, "I received a lot of help with the application" was changed to "Did you receive alot of help with the application?"

D18. Are there other circumstances that we haven't mentioned that helped you decide to usefood stamp benefits, despite difficulties of applying and using them?

Issue: Some respondents may not have difficulties applying and using food stamp benefits.

Response: We deleted the last phrase "despite difficulties of applying and using them."

e. Section E: Reasons for Nonparticipation

The questions in Section E of the questionnaire ask nonparticipants directly why they do not

participate in the FSP. Respondents are asked in a series of closed-ended questions whether a specific

factor was a reason they did not participate. To ensure that we have asked about all factors, we also ask

whether there are "other" reasons why they do not participate. This section is included in the

nonparticipant questionnaires only. The questions were developed specifically for this survey.

Respondents had issues with the following questions:

El. What is the most important reason you do not receive food stamps now? RECORDVERBATIM.

Issue: This question was then followed by more direct questions about nonparticipation. QuestionE5 then asked "of the reasons you gave, what is the most important reason." Some respondentsgave an answer to Question E5 that was not one of the reasons for nonparticipation they identifiedin Question El .

Response: We deleted Question El. In the direct questions about reasons for nonparticipation,we ask as Question E3o "Are there other important reasons why you do not use food stampbenefits that we have not just mentioned?" The interviewer is instructed to record the answerverbatim.

E2. Are there other reasons?

Issue: Same as for Question El.

Response: We deleted Question E2 and inserted Question Elo.

E3. Now I am going to read some reasons for not using food stamp benefits. For each one, pleasetell me flit is an important reason for are not currently using food stamp benefits? (We thenlisted 11 reasons).

Issue: Respondents had difficulty focusing on the intent of the question. Often respondent'sanswers had to be reinterpreted to find the correct response. For example, the interview read thestatement "You think you are not eligible for food stamp benefits." The respondent would answer"no." The respondent meant "no, I think I am not eligible for food stamp benefits." Thus, thecorrect answer was "yes." There was also confusion between whether the statement was true forthe respondent and whether it was an important reason why they did not receive food stampbenefits. For example, a respondent may answer "yes" they did not know how to apply for food

DRAFT

32

stamp benefits, even though they did not find out how to apply for food stamp benefits becausethey did not need them.

Response: The question was converted into a series of questions as opposed to statements. Thereare two questions, El and E2. Question El asks a direct question about whether something is truefor the respondent. For example, Question El a asks "Do you know how to apply for foodstamps?" Question E2 asks "Is this an important reason you don't use food stamp benefits?"

E3g. Please tell me whether this is an important reason why you don't use food stamp benefits?You feel that the application process is too burdensome.

Issue: A respondent asked what was meant by "burdensome."

Response: We divided this statement into two questions:

"E3 c. Do you think it is hard to get to the food stamp office?"

"E3 d. Do you think the application process is too long and complicated?"

If the respondent answers "yes" to these questions we ask a follow-up questions to learn moreabout why they think this is the case. These follow-up questions are E6 and E8 respectively.

E7.

You said you think you are eligible for only a small amount of food stamp benefits. Why doyou think you are eligible for only a small amount of benefits?

Issue: Some respondents answered in terms of the eligibility rules (such as my income was high)rather than how they learned about the eligibility rules, which was the intent of the question.

Response: We have replaced this question with four short questions, E12a, El2b, E12c, andE12d. These questions ask directly whether the respondent learned about the benefit amount fromthe three main sources of information about benefit amounts: the FSP office, another program, andby the respondent comparing themselves with someone else who receives benefits. If therespondent learned about the amount of food stamp benefits from the FSP office, we follow upwith a question El2b that asks when the respondent learned about the amount of benefits theywere eligible for. This will give us an idea of whether the information the respondent has aboutthe benefits they are eligible for is old and hence no longer applicable to the respondent.

f. Section F: Receipt of Food Assistance

Section F contains questions about household members' receipt of food assistance from sources other

than the FSP. Most of the questions that appear in Section F were taken from the April Food Security

Supplement of the Current Population Survey (CPS). The following questions in Section F had one or

more problems:

Fl. During the past 3 months, did (you/anyone) in the household ever receive free or reduced-costmeals for the elderly either at a facility or delivered to your home?

Issue: A few individuals were uncertain what we meant by "facility."

Response: We have added a probe to the question,

"PROBE: By facility we mean places such as a senior center or community center or some otherspecial place with a meal program."

F2. During the past 3 months, did any children in the household ever receive free or reduced- costlunches at school?

Issue: Just households with children under 18 presently get asked this question (there is anInterviewer Instruction at Q. Fla. which directs the interviewer to skip households withoutchildren.) However, the question is really only relevant if the household has school age children.When being administered the questionnaire, those respondents with young children remarked thattheir children were too young to attend school when asked this question.

Response: We have added the following response category to Question F2:"CHILD/CHILDREN DO NOT ATTEND SCHOOL -4 ----> SKIP TO F4"

F4.

During the past 3 months, did any children in the household ever receive free or reduced- costmeals at a day-care or Head Start Program?

Issue: Same as in previous question.

Response: We have added the following response category to question F4: "CHILD/CHILDRENDO NOT ATTEND DAY CARE PROGRAM -4"

F6. During the past 3 months, did any (women/women or children/children) in this householdever get food through the WIC program?

Issue: Reference in parentheses (women/women or children/children) was confusing. In addition,some respondents were unaware of the WIC program.

Response: We changed the reference in parentheses to "(you/anyone in the household)." We alsoadded a probe to clarify what is meant by WIC:

"PROBE: By WIC we mean the U.S. Department of Agriculture's Special SupplementalNutrition Program for Women, Infants, and Children, which provides food packages and nutritioneducation to women, infants, and young children."

F11. In the last 3 months, did you (or other adults in your household) ever get free food from anysource we haven't already mentioned?

Issue: We found during the cognitive tests (and focus groups) that an important source of foodwas free or reduced-price food or meals received while at work. In addition, there was someuncertainty on the part of respondents as to what qualifies as "free" food or meals. For example,one respondent questioned if the occasional lunches his mother bought qualified as free food.

Response: We created an additional question that asks respondents about free or reduced pricedmeals or food they or other household members received through work:

"F1Oa. In the last 3 months, did you (or other adults in the household) ever receive free orreduced-price meals at work?

YES 01NO 00DON'T KNOW -1"

We also added a probe to question F11,

"PROBE: Please include any free meals or groceries that you receive from anyone or source thatwe haven't already mentioned."

g. Section G: Food Security

This section of the questionnaire asks about the food security of the respondents' households. Most

of the questions in Section G of the questionnaire were taken from the April Food Security Supplement

of the CPS. One or more respondents had some difficulty with the following questions:

GI. Which of these statements best describes the food eaten in your household in the last 3months? Enough and the kinds of food wanted, enough but not always the kinds of foodwanted, sometimes not enough, or often not enough?"

Issue: Two single-parent working participants had difficulty responding because food sufficiencyvaried by household members. For example, one women sometimes skipped meals at home soher young daughter never had to. Thus, the mother's answer for herself would be "sometimes notenough," whereas the answer for her daughter would be "enough but not always the kinds of foodwanted." Four respondents who were diabetic stressed throughout the interview their special food

DRAFT

35

needs were not accommodated on their limited budget. So for them, it was a situation of notgetting the kinds of food they need.

Response: One possible solution to the first issue is to restructure the question to accommodatemultiple answers per household, with specific answers tied to specific household members. Forexample, one might distinguish adult household members from children, as groups. But thissolution is-not entirely satisfactory, because in households with multiple adults or children, thesituation can arise in which the answer varies within members of the group of children or withinthe group of adults in the household. To really do it right, we would need to ask about the foodsufficiency of each household member. But that would add appreciably to the length of thequestionnaire. Question G1 is the standard question about food sufficiency. Moreover, it isintended to be a measure of the sufficiency of the household's food supply overall. Thus, werecommend keeping the question as is, but adding the following interviewer instructions to therespondent:

"INTERVIEWER: If respondent's answer differs by household member, instruct respondent toanswer for the person who is least food secure."

In addition, interviewers will be trained to tell those respondents who bring up the issue of "needsversus wants" to consider needs and wants as equivalent when answering the question.

G6. (I/We) relied on only afew kinds of low-costfoods tofeed (the child/the children) because (Iwas/we were) running out of money to buy food Was that often true, sometimes true, ornever true for you in the last 3 months?

Issue: A few respondents were uncertain what the term "low-cost food" meant.

Response: We added the following probe to clarify what is meant by low-cost food,

"PROBE: By low-cost food we mean rice, beans, macaroni products, bread, or potatoes, or foodslike that."

G7a. In the last 3 months, did you ever cut the size ofyour children's meals because there wasn'tenough money for food?

Issue: Some households only have one child.

Response: We expanded the reference to read "your (child's/children's) meals"

h. Section H: Employment History

Section H asks respondents to report the wage rate and hours worked of each person in the household.

In the working questionnaires, it also asks questions about the current occupation and the work' history of

DRAFT

36

the person in the household who works the most hours. Most of the questions were adapted from other

national surveys, such as the survey developed for the U.S. Department of Labor's Trade Adjustment

Assistance study. Below are the questions in which one or more respondents had at least some difficulty

answering.

H2. How many hours per week (does NAME/do you) usually work?

Issue: A few respondents had difficulty answering the question because their weekly work hoursvary. For example, a waitress had difficulty with the concept "usual work hours" because herhours varied week-to-week according to the schedule set by the restaurant owner.

Response: We have added an interviewer instruction to deal with this situation,

INTERVIEWER: If respondent unable to tell you usual hours worked per week, ask "How manyhours did you work last week or during the most recent week you worked if you did not work lastweek?"

H3. How much (does NAME/do you) earn on this job, before taxes and deductions?

Issue: A few respondents were confused by the phrase "before taxes and deductions." Inaddition, a few respondents had difficulty estimating their earnings because they received tips, andthe amount of tips they received was dependent upon the season and other factors. Thus, aspresently structured, there is not an easy way to record hourly wages plus tips or bonus.

Response: We rephrased the question, replacing "before taxes and deductions," with "beforetaxes and any other deductions are taken out."

We have also added an interviewer instruction to deal with situations m which workers receive tipsas part of their compensation,

INTERVIEWER: If respondent receives tips as part of earnings, ask respondent to report whathe or she earned last week (or most recent week worked if not last week) including tips, and recordas "salary per week."

H6. What is (your/NAME'S) main duties as (a/an) (JOB TITLE FROM QUESTIONHS)?

Issue: Question H5 asks the respondent to report their job title, and then Question H6 probes forspecific job activities. H6 can sometimes be redundant or awkward when the job title given in H5also describes the activities. For example, in the case of a waitress, it is awkward and redundantto ask question H6 "What do you spend your time doing as a waitress?"

Response: This is a valid criticism of the question sequence. However, we are concerned aboutleaving it up to the interviewer to decide whether there is sufficient information from therespondent's answer to question H5 to be able to skip question H6. No matter how carefully andthoroughly we train interviewers, they would encounter situations and make decisions that wouldresult in them skipping question H6 when they did not have sufficient information from responsesto question H5 to characterize work activities. Consequently, we recommend not changing thesequence H5 and H6.

H8. How likely is it that (you/NAME) will be working at this job three months from now? Wouldyou say it is very likely, somewhat likely, or not likely at all?

Issue: A few respondents simply answered the question from the perspective of whether or notthey were planning on leaving their job or not, without considering potential actions on the part ofthe employer that might lead to them not having a job in the near future. For example, a womananswered that it was "very likely" she would have a job, but this was based on her desire to staywith the firm. When probed further, she stated that the company recently cutback its workforceand she feared they were about to do it again, which implied the answer should have beensomething closer to "not likely at all."

Response: We rephrased the question, replacing the phrase "(you/NAME) will be working" with"(you/NAME) will have this job.

In addition, we added a new question which captures expectations about earnings three monthsfrom now:

"H9. Do you think (you/NAME) will be earning more, the same, or less three months from now(from any job(s))?

MORE 01SAME :.... 02LESS 03DON'T KNOW... -1"

H9. (Do you/does NAME) typically work during daytime hours, evening hours, or late night hourson this job?

Issue: The intent of the question is to capture whether respondents' work schedule makes itdifficult or prevents them from applying for food stamps. As worded, the coding structure is notspecific enough (it needs time intervals, such as business hours 8 a.m. to 5 p.m., and so on). Inaddition, some people work variable schedules, so we would need to allow "code all that apply"responses.

Response: We have deleted the question. We developed a different question that asksrespondents directly whether their work schedules make it difficult or prevent them from applyingfor food stamps in-person. This is preferable to having to infer this information from a questionsuch as H9.

DRAFT

38

H10. Over the past 12 months, how many different jobs (have you/has NAME) had? Please includeall part-time and full-time jobs held for at least two weeks.

Issue: A few respondents did not include their current job when counting the number of jobs.

Response: We added the phrase "Including your present job, over the past 12 months how manydifferent jobs ".

H11. During the last 12 months, about how many months (were you/was NAME) not working forPaY?

Issue: Some respondents had difficulty answering this question because it required them tosimultaneously think about whether there were periods they weren't working and then determinethe duration of these periods.

Response: We have split the question into two parts,

"Hl 1 a. During the past 12 months, were there times when (you were/NAME was) not workingfor pay?

YES 01NO 00 > GO TO Il

Hl lb. About how many months (were you/was NAME) not working for pay? "

(Response categories are those currently in question HI I).

i.

Section I: Health

Section I collects information on the general health and physical and cognitive functioning of

respondents. The questions appearing in this section were adapted from questions in the National Health

and Nutrition Examination Survey III (NHANES III) and the 1990 Census of Population. Respondents

raised issues with the following questions:

12.

Please tell me ifyou have no difficulty, some difficulty, or are unable to do these activitieswithout the use of aids or another person?

Issue: Some respondents had difficulty with Item (b) "managing money." Because they havelow-income, they were more prone to respond in terms of trouble managing because they had "toolittle money to manage," rather than the cognitive difficulties they might have with the process(which is what we want).

DRAFT

39

Response: We have added to our examples of what we mean by the concept "managing money."The question now reads:

"Managing your money (such as keeping track of your expenses or paying bills, balancing a checkbook, or filling out tax forms)"

In addition, we have added the response category "great deal of difficulty" to the set of responsesfor all questions in the three-question sequence, so that now the response categories are: "NODIFFICULTY," "SOME DIFFICULTY," "A GREAT DEAL OF DIFFICULTY," "UNABLETO DO," and "DON'T KNOW/DON'T DO."

j. Section J: Social Supports

Section J of the questionnaire asks respondents a series of questions about the length of time they

have lived in the neighborhood, frequency of making social visits, and whether relatives live close-by. The

questions were adapted from questions in the NHANES III. One or more respondents had difficulty with

the following questions:

JI.

How long have you lived in this neighborhood?

Issue: Some respondents were put off by the word "this" in the phrase "in this neighborhood."

Response: We have replaced "this neighborhood" with "your neighborhood."

J2. In a typical week during the past year, how often did you get together in person with friends,neighbors, or relatives?

Issue: A few respondents did not understand the reference period implied by "Less than that,"which appears after "Once or twice a week."

Response: We have rephrased the last response category so it reads,

"Less than once per week 00"

J3. How close do you live to your nearest relative?

Issue: Respondents have difficulty answering because their nearest relative may live with them.

Response: We have rephrased the question and eliminated. response category "SAME HOUSE."The question now reads,

DRAFT

40

"J3. How close do you live to your nearest relative who does not live with you?

LESS THAN 30 MINUTES AWAY 0130 MINUTES TO 2 HOURS AWAY 02MORE THAN 2 HOURS AWAY ...: 03NO LIVING CHILDREN OR RELATIVES -4"

k. Section K: Income and Expenses

Questions on income and expenses of the respondents' household are contained in Section K. The

questions were adapted from questions in the National Food Stamp Program Survey (NFSPS).

Respondents raised issues with the following questions:

KS What is your monthly rent payment?

Issue: The objective of the question is to find out the household's monthly out-of-pocket paymentfor rent. Although it did not occur during the cognitive tests, it is possible that respondents mightinclude the dollar amount of rent subsidies (such as HUD Section 8 or other programs) thatpartially covers the cost of their rent. This amount should be excluded. In some instances the rentpayment includes cable T.V. If this amount is known and can be separated, we will want therespondent to report out-of-pocket costs excluding cable T.V.

Response: The question has been rephrased and also includes a probe to deal with the potentialproblem of respondents erroneously including amounts that they do not pay out-of-pocket. Thequestion now reads,

"K5. What is your monthly rent payment? Please tell me only the amount that you and membersof your household pay out-of-pocket.

PROBE: Do not include any rent subsidies that your household receives from anyprogram to pay part of the rent costs. Also, do not include the cost of Cable T.V."

K6. Last month, did (you/your household) pay for the care of children or other dependents so thatsomeone in your household could go to work, school, or a training program?

Issue: This question is awkward if the household is a single person household or does not containdependents. In addition, the question is concerned with out-of-pocket costs, which is not clear inthe phrasing of the question.

Response: We have added an interviewer instruction, now labeled K6, which instructs theinterviewer to skip this question for households without dependents.

"K6. INTERVIEWER: Check Grid Al through A3. How many people are in household?

ONE : 01 > SKIP TO K7MORE THAN ONE....

02"

In addition, we have modified the original question K6 to read "pay out-of-pocket for the care...."to make sure the respondent understands that we are referring to out-of-pocket expenditures fordependent care. Original K6 is now labeled K6a.

To make room for the interviewer instruction, original question K6a is now labeled K6b.

K8. Last month, did (you/your household) pay health insurance premiums or payments, includingcosts to belong to an HMO?

Issue: Some respondents were confused by the phrase "including costs to belong to an HMO,"and tended to think the question was just about out-of-pocket expenditures on care received fromHMOs, as opposed to out-of-pocket medical costs.

Response: We rephrased the question to read, "....pay health insurance premiums or paymentsto belong to an HMO. "

K8a. How much did (you/your household) pay last month?

Issue: Some respondents were not sure referred to just out-of-pocket expenses.

Response: We have rephrased question to make it clear that referring to only out-of-pocket costs,

"K8a. How much did (you/your household) pay last month. Please tell me only the amount thatyou and members of your household pay out-of-pocket."

K9. Now think about the people in your household who are disabled or age 60 or older. Lastmonth, did they have any medical expenses from Medicaid, Medicare, or health insurance?

Issue: Question is wordy and redundant.

Response: Combine questions K9 and K9a into one question, as follows:

"K9. Now think about the people in your household who are disabled or age 60 or older. Lastmonth, how much were their out-of-pocket medical expenses?

PROBE: Include doctor and hospital bills, prescription drugs, lab tests or x-rays, and anyother medical expenses you paid out-of-pocket. Please exclude anything for which youwill be reimbursed."

K1O.1 During (LASTMONTH) did you receive any (STATE WELFARE NAME)/Aid to Familieswith Dependent Children)?

Issue: Name of program has changed.

Responses: Have substituted "Temporary Assistance for Needy Families (TANF)" for "Aid toFamilies with Dependent Children."

(Comment applies to K12.1 as well.)

K12. Did anyone else in your household receive any (SOURCE) last month?

Issue: As presently structured, the interviewer needlessly reads Questions K12.1 throughK.12.11 to respondents who live alone.

Response: We introduced an interviewer instruction at beginning of the K10 K14 question gridthat instructs the interviewer to skip single person households past K12 - K14 questions. Theinterviewer instruction reads:

"INTERVIEWER: Check Grid Al through A3. If respondent lives alone, DON'T ASK K12,K13, or K14. THAT IS, ALWAYS SKIP TO NEXT QUESTION IN K10 AFTERCOMPLETING K11 FOR EACH INCOME SOURCE."

K15. Do you (or does anybody in your household) own a car, truck, van, or motorcycle?

Issue: Respondents were confused about whether to include vehicles they were making paymentson. The probe clarifying this was needed frequently.

Response: We integrated the "probe" into the question,

"K15. Do you (or does anybody in your household) own a car, truck, van, or motorcycle? Pleaseinclude any cars, trucks, vans, or motorcycles that your are making payments on."

1. Section L: Demographic Information

Questions that obtain information on the demographic characteristics of the respondent, such as age,

race/ethnicity, education, and marital status, appear in Section L. Most of the questions in this section were

adapted from questions in the NFSPS. The following are problem questions:

L4.

What is the highest grade or level of school that you have completed?

Issue: We had one respondent who reported completing 12 grades but did not receive a diplomaand did not consider himself a high school graduate. This was a sensitive issue that we did notdiscuss further with the respondent to find out the distinction. It is enough to know that somerespondents may make the distinction.

Response: We have introduced an additional response category to take this situation into account.In addition, we inserted the category, "Some college." The affected parts of the question nowread,

"L4. What is the highest grade or level of school that you have completed?

FIRST GRADE 01....................

ELEVENTH GRADE 11TWELFTH GRADE 12GRADUATED HIGH SCHOOL 13GED 14VOCATIONAL TRAINING PROGRAM CER11PICATE 15SOME COLLEGE 16..............................................TWO- OR FOUR-YEAR COLLEGE DEGREE 17OTHER (SPECIFY) 18"

L7 New Question.

Issue: Under recent welfare reform, some immigrants will not be eligible to receive food stampseven though they meet all other program eligibility requirements (see discussion below in Section1H. C).

DRAFT

44

Response: We are adding questions to the main questionnaire (long-version only) about thecitizenship of household members to enable us to identify whether nonparticipants meet thecitizenship requirement for program participation. The questions are as follows:

"L7. Is everyone in your household a U.S. citizen?

YES 01 > SK1P QUESTION L8NO 02

L7a. Is anyone in your household a U.S. citizen?

YES 01NO 00

L8.

These are all my questions. Thank you very much for your time."

B. OTHER FINDINGS FROM THE COGNITIVE TESTS

At the completion of the interview, respondents were asked to give their overall impression of the

questionnaire, including its length, flow, and organization. In this section, we summarize cognitive test

findings about length, question-order, and flow of the survey instruments. We also assess implications for

administering the questionnaires by telephone.

1. Questionnaire Length

Questionnaire length does not appear to be a problem. The long-versions of the instruments were

found to take on average approximately thirty minutes to administer (see Table III.1). (The average is

based on four RTA interviews.) One of the interviews--an elderly nonparticipant with prior FSP

experience--was 49 minutes long. However, that respondent had a very complicated FSP participation

history. Our feeling is that this would be the upper limit on the amount of time needed to administer the

questionnaire, but that others would have much shorter interviews, as illustrated by the interview times for

the other three respondents. The revisions we have made in response to the cognitive testing have resulted

in a slight increase in the number of questions. Thus, the average length of the long-versions of the

TABLE Ill 1

COMPLETION TIMES FOR RETROSPECTIVE THINK ALOUD COGNITIVE TEST INTERVIEWS

Working Nonparticipant Working FSP Participant Elderly Nonparticipant Elderly FSP ParticipantModule (No prior experience with FSP) (Prior experience with FSP) (Prior experience with FSP) (No recent applications)

Screener Not timed Not timed Not timed Not timed

Section A 1 minute 1 minute 1 minute .5 minute

Section B 2 minutes 1 minute 4 minutes 2 minutes

Section C .5 minute 5 minutes 14 minutes 2 minutes

Section D .5 minute 10 minutes .5 minute 1 minute

Section E 4 minutes (Nonparticipants Only) 12 minutes (Nonparticipants Only)

Section F 2 minutes 1 minute 1 minute 1 minute

Section G 3 minutes 5 minutes 6 minutes 3 minutes

Section H 3 minutes 2 minutes (Working respondents only) (Working respondents only)

Section I (Elderly Only) (Elderly Only) 3 minutes 1 minute

Section J .5 minute 1 minute 2 minutes 1 minute

Section K 3 minutes 4 minutes 4 minutes 5 minutes

Section L .5 minute 1 minute 1 minute 1 minute

Total 20 minutes 31 minutes 48.5 minutes 17.5 minutes

questionnaires will increase. We anticipate that the length of the interview will increase by approximately

two minutes.

2. Questionnaire Content, Organization, and Flow

Overall, the questionnaires were well-received by respondents. Respondents characterized the

questionnaires as reasonable and straightforward. None had recommendations for changing the

organization of the questionnaires nor questionnaire content (i.e., recommendations for adding/deleting

question items).

3. Implications for Telephone Administration of Questionnaires

The cognitive test entailed interviewing respondents in-person. The field test of the study's

instruments will entail interviewing some respondents in-person and others by telephone. Moreover, a

nationwide fielding of the questionnaires, if implemented, would most likely be administered by telephone.

The remainder of this section discusses implications of the in-person cognitive tests for administering the

interviews by telephone. We consider two issues: (1) participation in the study, and (2) interview length.

Participation in Telephone Interview. Similar to the focus groups, some cognitive test respondents

indicated that they would be much more likely to participate in the survey if they received a letter about

the study in advance of the telephone screemng interview. This was especially true for the elderly, who

are more often the target of scams.

Under current plans, we are planning to send advance letters to FSP participants sampled from the

list frame and to any respondent who requests a letter during the screening interview. However, one could

also send advance letters to some, but not all, RDD respondents. Our experience is that it is possible to

identify current addresses for approximately 25 percent of a RDD low-income sample frame prior to

telephoning them. During the planning for the field test, we will explore with FCS the possibility and

desirability of sending advance letters to the proportion of the RDD sample in which we can identify

addresses.

DRAFT

47

Interview Length. It is possible that the questionnaires will be somewhat longer when administered

by telephone. Because they agreed to meet in-person with researchers, respondents who participated in

the cognitive tests are probably more highly motivated than respondents on average. In addition, there are

fewer distractions in an in-person interview administered in a cognitive lab setting than compared to a

phone interview in a person's home.

C. REVISIONS TO TAKE INTO ACCOUNT CHANGES IN ELIGIBILITY RESULTINGFROM WELFARE REFORM

The Personal Responsibility and Work Opportunity Reconciliation Act of 1996 (PRWORA) will affect

the survey through its effects on who is eligible for food stamp benefits. The following are key nutrition

provisions of the Act that affect FSP eligibility:

1. Able-bodied childless persons between the ages 18 to 50 may receive food stamps for onlythree months in a 36-month period unless they are working for pay 20 hours per week orparticipating in a state-approved work program 20 hours per week.' A working recipient whois laid off may get an additional 3 months of benefits. The entire work requirement maybewaived if a state ' s unemployment rate exceeds 10 percent.

2. Undocumented immigrants and certain categories of documented immigrants are prohibitedfrom participating in the FSP2

Our strategy is to screen-out all individuals who have been told in the past two months by FSP office

staff that they were ineligible because of either of these rules (see Question s3 in the RDD Screening

'Able-bodied persons are those who are not: disabled, pregnant, in school, or caring for someone else.The term "preceding 36-month period" does not include any period before the earlier of: (1) the date theState notifies recipients of food stamp benefits of the application of section 6(o) [the able-bodied workrequirement/time-limit]; or (2) the date that is 3 months after the date of enactment of this Act [enactmentdate was 8/22/96]. According to FCS, for current recipients [the time limit provision] will apply atrecertification, the point in which current recipients will be notified by the State.

'Refugees and asylees in U.S. for less than 5 years and other exempt immigrants (for example,immigrants who become citizens) are eligible for food stamps. Immigrants in the U.S. before the date ofenactment of the PRWORA or immigrants entering the U.S. on or after the date of enactment (i.e.,refugees who have been in the U.S. more than 5 years and most other legal aliens) are ineligible until theybecome citizens. All illegal aliens and other legal aliens than those mentioned above are ineligible.

DRAFT

48

Questionnaire). It is possible that some current FSP nonparticipants are not eligible because they have

received FSP benefits for three months or more during the past 36 months and do not meet the work

requirement test. However, for the field test, we will not attempt to screen out these people who do not

meet the work requirement test. This is because of the complexity about the period in which the 36-month

period applies; it also reflects the fact that we would need to add several questions to the screening

interview in order to determine whether respondents' households contained "able-bodied" individuals (e.g.,

not pregnant, not in school, not caring for dependent, work hours, etc.).

We do plan, however, to obtain sufficient information from the main questionnaire to determine the

prevalence in which nonparticipant households are ineligible because of these two requirements. That is,

from the main questionnaire we will determine: (1) whether the household contains someone between 18

and 50 years of age, (2) whether the household contains children and the ages of the children, (3) the

weekly work hours, and so on. In addition, we have added a question (Question C2a) that asks whether

they received any food stamp benefits for three or more months since January 1, 1997 (an approximation

to the 36-month period). We will also ask questions (the last two questions of the survey in Section L:

Demographics) which ascertains whether everyone or anyone in the household are citizens of the U.S.

These questions were discussed previously in Section DI.A.2.1.

REFERENCES

Cohen, Rhoda, Eisenhower, Donna, McConnell, Sheena, and Michael Ponza. "Reaching the WorkingPoor and Poor Elderly: Cognitive Test Planning Memorandum." Draft report submitted to the U.S.Department of Agriculture, Food and Consumer Service. Mathematica Policy Research, Inc.,Princeton, NJ: December, 1996.

Cohen, Rhoda, McConnell, Sheena, Milliner, Julita, and Michael Ponza. "Reaching the Working Poorand Poor Elderly: Screening Interviews and Questionnaires." Draft report submitted to the U.S.Department of Agriculture, Food and Consumer Service. Mathematica Policy Research, Inc.,Princeton, NJ: December, 1996.

McConnell, Sheena et al. "Supporting Statement for Request for OMB Approval: Reaching the WorkingPoor and Poor Elderly Study." Submitted to the U.S. Department of Agriculture, Food and ConsumerService. Mathematica Policy Research, Inc., April, 1997.

McConnell, Sheena. "Revised Cognitive Test Plans." Memorandum submitted to Alana Landey, U.S.Department of Agriculture, Food and Consumer Service. Mathematica Policy Research, Inc.,February 6, 1997.

Ponza, Michael and Sheena McConnell. "Reaching the Working Poor and Poor Elderly: Interim Reporton Focus Groups." Draft report submitted to the U.S. Department of Agriculture, Food andConsumer Service. Mathematica Policy Research, Inc., Princeton, NJ: December 1996.