How To Avoid Pitfalls When Designing A Questionnaire - ARTP

NHS Fife Research Study Guide: [6] How to Avoid Pitfalls When Designing a Questionnaire.

Copyright NHS Fife, Research & Development Office, Dr David Chinn, v2, 4th October 2015

Study Guide 6: How to Avoid Pitfalls When Designing a Questionnaire Dr David Chinn, Research & Development Office, Queen Margaret Hospital, Dunfermline, Fife. [email protected] 01383 623623 (ext 20943) Alternative contact: Dr Amanda Wood [email protected] 01383 623623 (ext 20941)

Contents Page

1 Overview and learning outcomes 1

2 First thoughts 2

3 Does a questionnaire exist already that will do the job? 4

4 Postal or interviewer administered questionnaire? 5

5 Features of a good questionnaire 6

6 The 10 steps in designing a questionnaire 7

7 Training the interviewers 25

8 Piloting 25

9 Last thoughts on design 25

10 Some further issues 26

10.1 Consent 26

10.2 Computer-read questionnaires 26

10.3 On-line questionnaires and use of computer-administered questionnaires

26

10.4 Validity of questionnaires 27

10.5 Getting a good response rate from a postal survey 27

10.6 Sources of bias in questionnaires 29

11 Summary 30

12 Further reading 30

Appendix: sample questionnaires 31

(1) Overview and learning outcomes Designing a questionnaire is a complex and sophisticated art. The process can be lengthy and you should not underestimate the time required to do the job properly. These notes are relevant for anyone faced with designing a questionnaire as part of a research project, audit or service evaluation. They cover the many steps in designing questions and deciding their layout. After reading this guide you should be able to:

Specify the features of a good questionnaire,

Identify where to get help and find examples of questions as used in other questionnaires,

Compare the advantages and disadvantages of postal and interviewer-administered questionnaires,

Identify the steps involved when designing a questionnaire,

Be familiar with the different types of questions,

Recognise potential faults in the wording of questions, the scoring of responses and the layout of questionnaires

mailto:[email protected]

mailto:[email protected]



Associated NHS Fife study guides: 1 How to devise a research question and choose a study design 2 How to write a protocol

10 An introduction to medical statistics

(2) First thoughts

Before starting the process of designing a questionnaire you should address some common issues (Table 1). An example is given in Table 2.

Table 1: Initial check list

1 What is the purpose of the questionnaire? Purpose:

2 Is it to be used for research, screening, audit, service evaluation, Research

patient satisfaction or some other purpose? Screening

Audit

Service evaluation

Patient satisfaction

Other

3 Do you require qualitative or quantitative data? Qualitative only

Quantitative only

Both

4 Are the replies to be anonymous or identifiable? Anonymous

Identifiable

5 Is it to be self-administered (for example in a postal survey) or Self-administered

administered by trained staff (for example in a clinic)? Interviewer

6 Are you using a paper version or an electronic, online method Paper version

Online questionnaire

7 Is it to be completed by a health professional, a patient or their carer? Health professional

Patient

Carer

Other

8 How are the results to be analysed? Simple frequencies

Cross tabulations

9 Does a questionnaire exist already that will do the job? Yes

No

Unsure

These initial questions help decide how to proceed with your study. After reflection, it may be that, for example, asking patients to fill in a questionnaire after attending a day surgery unit may not be appropriate if those patients are having eye surgery! Seeking sensitive data may be best done with an anonymous questionnaire completed by the respondent though this may be a problem if dealing with persons who have a visual handicap or a learning disability when an interviewer delivered questionnaire will have to be used. It is particularly important that if using more than one interviewer they are trained together to develop a consistent style of questioning and recording responses. Otherwise the results may be subject to interviewer-bias (see Page 29). Think carefully how you will analyse the questionnaire responses. There is little point in collecting data that cannot be usefully analysed and interpreted. The results from a recent Scotland-wide survey of NHS staff were collated centrally and provided only as summary frequencies for each Health Board. These were un-interpretable as they could not be cross-tabulated by staff grade and location of workplace.



Table 2: Survey of patients’ experience attending a day surgery unit

1 What is the purpose of the questionnaire? To elicit the experiences of patients attending the day surgery unit

2 Is it to be used for research, screening, audit, service evaluation, Research

patient satisfaction or some other purpose? Screening

Audit

Service evaluation

Patient satisfaction

Other

3 Do you require qualitative or quantitative data? Qualitative only

Quantitative only

Both

4 Are the replies to be anonymous or identifiable? Anonymous

Identifiable

5 Is it to be self-administered (for example in a postal survey) or Self-administered

administered by trained staff (for example in a clinic)? Interviewer

6 Are you using a paper version or an electronic, online method Paper version

Online questionnaire

7 Is it to be completed by a health professional, a patient or their carer? Health professional

Patient, on leaving

Carer

Other

8 How are the results to be analysed? Simple frequencies

Cross tabulations

9 Does a questionnaire exist already that will do the job? Yes

No

Unsure

If you have access to or knowledge of a questionnaire that has been successfully used elsewhere you may use it with confidence after first piloting it in a sample of typical respondents to ensure it is fit for purpose. One advantage is that, with some qualifications the results you obtain in your population may be compared with those from other, equivalent, populations. But, you should be wary of using it in a population for which it was not designed. This is because measurement instruments generally reflect the cultural norms of the society in which they were developed. Some items may not translate well and some conditions may not be considered important in another culture compared to the cultural group the questionnaire was designed with. For example, the question ‘do you ever bring up phlegm from your chest’ (from the Medical Research Council Questionnaire on Respiratory Symptoms) was developed in the 1960s in a UK, Caucasian population but its use is likely to elicit negative responses in certain ethnic groups because their cultural practices prohibit spitting out phlegm which they always swallow. As a general concern care is needed in the use of questionnaires developed in the USA for use in a UK population because of differences in “American English” compared with “UK English” in the wording of questions. Also, you should be wary of the use of certain measures. For example, it has become customary to report a person’s smoking history in terms of ‘pack-years’. In the USA cigarettes are bought in packets of 20 but in the UK packs of 10 are available so if you ask the question (from an American questionnaire) “how many packs of cigarettes a day do you smoke?” you may overestimate the consumption if, for example, a person only buys one pack of 10 cigarettes a day.



Some questionnaires may need translating into English or from English into another language. When translating a questionnaire from one language to another use a lay person to back-translate it blind as a quality initiative to ensure the meaning of individual questions has been correctly portrayed in the new version. The impact of a question may take on a different perspective from its original meaning in some cultures. For example, cancer may be considered less serious than a common cold or another infection if mortality associated with these latter conditions is high in that society. Another aspect of importance concerns the wording. For example, in the MRC questionnaire on respiratory symptoms there is a set of questions on cough and phlegm which begin ‘do you usually bring up phlegm from your chest first thing in the morning in the winter?’ This was designed for the general population and is appropriate for surveys of working populations but the wording is incorrect for persons working night shifts! (3) Does a questionnaire exist already that will do the job? It is usually worth looking for available (off the shelf) questionnaires and questions. Some sources of information are:-

The question bank: http://surveynet.ac.uk/sqb/qb/

http://www.statpac.com/surveys/contents.htm Designing surveys and questionnaires.

Bowling A. Measuring health: a review of quality of life measurement scales. Open University Press, Buckingham. 3rd ed. 2005.

Bowling A. Measuring disease: a review of disease-specific quality of life measurement scales. Open University Press, Buckingham. 2nd ed. 2001.

The medical literature (write to the authors but also be aware that sometimes a copy of the questionnaire used in a published study is lodged with the journal’s editor as part of the submission process)

The benefit in using a previously validated questionnaire is that your results are more likely to be directly comparable with other studies. Some quality of life questionnaires are specific to a disease (for example, asthma, inflammatory bowel disease etc) whereas others are generic (for example the SF36 (the Short-form 36 item that assesses general quality of life in 5 domains)). The question bank has a searchable database to identify questions that have already been developed and used successfully in surveys, including many UK-wide surveys of health behaviour and attitudes. For example, a search on ‘smoking questions’ revealed 3885 questions, one of which was from the 29 year follow-up of the 1970 British Cohort Study:

http://surveynet.ac.uk/sqb/qb/

http://www.statpac.com/surveys/contents.htm



Variable smoking

Question text Now some questions about smoking. Would you say that... you've never smoked cigarettes, you used to smoke cigarettes but don't at all now, you now smoke cigarettes occasionally but not every day, you smoke cigarettes every day?

Response categories and codes

1 never smoked cigarettes,

2 used to smoke but don’t at all now

3 smoke cigarettes occasionally

4 smoke cigarettes every day

8 Don’t know

9 Not answered

Location 1970 British Cohort Study: Twenty-Nine-Year Follow-up, 1999-2000

Interviewer Instructions

CODE ONE ONLY.

The detail given includes the response categories and the instructions to interviewers. However, the search facilities in the question bank are limited in that it is not possible to combine searches e.g. to search for specific questions on ‘physical activity’ in ‘school-aged children’. (4) Postal or Administered Questionnaire? The most common methods for data collection are the personal interview (face-to-face or over the telephone) or by self-completion (postal, online or in the clinic). Both methods have their relative merits, advantages and disadvantages (Table 3). The general advice is to use interviews for collecting sensitive information such as the number of sexual partners people may have had. However, an anonymous, self-completed questionnaire may elicit more truthful answers on sensitive topics if it is considered less intrusive. There may be ethical issues to consider if undertaking a postal survey of experiences of patients attending clinics for termination of pregnancy, investigation of sexually transmitted infections, rehabilitation for substance misuse etc. Sending a postal questionnaire to a respondent’s home will not guarantee the letter is opened by the intended respondent. A letter opened by mistake (or deliberately) by a partner or parent may lead to unintended consequences. Accordingly, great care may be needed in deciding whether to use a postal survey.

http://discover.ukdataservice.ac.uk/variables/?q=VariableName:smoking%20SeriesTitle:1970%20British%20Cohort%20Study

http://discover.ukdataservice.ac.uk/variables/?q=%22now%20some%20questions%20about%20smoking%20would%20you%20say%20that%20youve%20never%20smoked%20cigarettes%20you%20used%20to%20smoke%20cigarettes%20but%20dont%20at%20all%20now%20you%20now%22%20SeriesTitle:1970%20British%20Cohort%20Study



http://discover.ukdataservice.ac.uk/variables/?f=SurveyTitleFacet_1970%20British%20Cohort%20Study:%20Twenty-Nine-Year%20Follow-up,%201999-2000



Table 3. The relative merits of using postal and interview methods

Postal questionnaires Interviews

Cheap(er) Expensive

Large numbers covered at the same time

Longer period needed for data collection

Limited amount of data can be collected

Opportunity to probe in-depth

Questions can be misunderstood Interviewer can clarify any uncertainty over question wording

Questions in fixed sequence Question sequence can be varied to suit interviewee

Limited opportunity to omit irrelevant questions

Questions can be left out if considered irrelevant

Allows anonymity Not anonymous, though interviewer can give reassurance

Needs very precise wording Can use less precise wording suited to the interviewee

Results subject to response bias (do you really know who filled it in?)

Results subject to response bias but also to observer bias (training an important issue)

(5) Features of a Good Questionnaire Whether postal or by interview the questionnaire has to be intelligible and interesting. Thus, questions must be arranged in a sequence which is logical and easy to follow. The first question should be simple and easy to answer. Leave difficult or potentially embarrassing questions to the end. The features of a good questionnaire include:

• Short with clear instructions for completing it,

• Unambiguous wording,

• Appropriate response categories,

• Easy to complete,

• Intelligible,

• Logical,

• Enjoyable to complete,

• Easy to return (use of a ‘drop box’ or provision of stamped addressed envelope),



• Readable – use font ‘Comic Sans’ (or Arial) and print size 12 pt or more. An appropriate font and size is particularly important for surveys of elderly people or those likely to be visually impaired.

• Ethical (‘do no harm’ and the need for informed consent, or consent implied from merely completing the questionnaire and returning it)

(6) The 10 Steps in Designing a Questionnaire

Step

1 Decide what data you need

2 Select the specific topics to be covered

3 Decide the order in which topics are to be raised

4 Decide whether you need a structured or semi-structured questionnaire

5 Choose whether to use open or closed questions for each topic

6 Design individual questions

7 Decide the wording of each question

8 Design the layout

9 Decide the coding scheme, scales and think about the analysis of the replies

10 Write the protocol

Step 1: Decide what data you need This step is critical as the choice here will determine not just what information is needed but how it will be collected. Refer to the study aims and objectives and decide what data are needed to meet them. In general, there are four main types of questions:

(1) Demographic (age, gender, marital status, occupation, education) (1) Behaviour (exercise, smoking, alcohol consumption) (2) Attitudes (how do people feel about, e.g. drinking alcohol in public places) (3) Beliefs / knowledge (what are the health risks of excessive alcohol

consumption)

Next, identify potential confounders such as age and gender. Ask colleagues for their opinions and suggestions but do not allow them to distract you from the study aims. Be prudent in deciding the data to collect and do not be tempted to collect additional, superfluous information, ‘just in case we may need it later’. Step 2: Select the specific topics to be covered Select items for inclusion and decide which, if any, may be considered sensitive. Be aware that different sections of society will identify different topics as being sensitive. In some groups sensitive questions may include topics such as age, income, ethnicity, and even gender. Step 3: Decide the order in which topics are to be raised As a general rule sensitive or threatening questions should be asked towards the end of the questionnaire, but important questions should be asked first because answers can be influenced by earlier replies. Thus, there is a compromise to be made in that



you must decide carefully on the order of the topics in an effort to ensure the questionnaire will be completed, and returned. Regarding the different types of questions referred to in Step 1 it is generally advisable to ask about behaviours before asking about attitudes (e.g. smoking behaviour and health beliefs associated with it). Step 4: Decide whether you need a structured or semi-structured questionnaire. A structured questionnaire uses fixed response categories with questions asked in a fixed sequence to impose rigidity, which can be helpful for some topics where you want the respondent to make an explicit choice. However, relying on fixed coded replies can limit a respondent’s freedom to respond to a given question so the choice of response categories may include an ‘other’ category together with the opportunity to give a free text reply. Semi-structured questionnaires also use fixed coded responses but include some open ended questions which are useful when you have limited knowledge about a topic. For an interviewer administered questionnaire a semi-structure also allows the interviewer the flexibility to omit irrelevant topics and vary the order and wording of questions to suit the respondent and flow of the interview. Step 5: Choose whether to use open or closed questions for each topic Closed questions are better for topics about which much is known so suitable response codes can be developed that are simple to analyse. One problem with closed questions is that the categories of response may be too restricted or not cover every option so it is generally advisable to include an ‘other’ category.

Example: A questionnaire was composed on barriers to physical activity and administered to middle-aged people living in a deprived inner city area. A list of barriers was constructed from the literature and an ‘other’ category was included. It was commonplace for respondents to add a comment under ‘other’ citing ‘fear of leaving their home unattended’ which was a barrier the researchers had never envisaged. This would have been missed if the researchers had neglected to add the extra category.

However, use of a list may give prompts to the respondent which they would not have thought of without the prompt. This can give the respondent a clue about the sort of results you anticipate and, therefore, may introduce a (response) bias. Open questions are essential where the issues are unknown, too complex or too numerous to pre-code; they have an important role in qualitative studies. Also, open-ended questions are useful for developing questionnaires on, for example, quality of life and then generating suitable measurement scales. But, respondents may be less willing to fill in open-ended replies and prefer the quick, tick-a-box method. One disadvantage is that in a purely quantitative study open replies are often very difficult to code.



Step 6: Design individual questions The format will depend on the method of collecting the data, by interview or by post, and the type of question being asked. Demographic data can be obtained relatively easily with appropriate choice categories. Age may be grouped in defined age bands which can be convenient but be cautious in the choice of band widths. Data collected in 10-year age bands (e.g. 20-29, 30-39, 40-49, 50-59, 60-69, 70-79, 80-89, 90-99, 100+) will not allow you to identify adults of working age (up to 65, or whatever the age cut-off becomes). A problem may arise if asking ‘What is your age?’ when a respondent is likely to give their age at their last birthday. Instead, consider asking for the respondent’s date of birth and ‘Today’s date’ (the date they completed the questionnaire). These values can be coded on the database and the precise age calculated as a decimal value. This will be particularly important for surveys of children. If you collect age as a ‘decimal age’ you can always create suitable descriptive age groups later. However, you may not be able to disentangle age bands if it is later decided that your analysis requires a precise age. The choice of bands or precise age will depend on why you need to know the age of the respondent in the first place and on whether it is believed that asking for a date of birth may be considered too intrusive. Use of a crude age band may be sufficient but if you wish to use the age as a variable in a prediction equation, for instance, you should formulate the question to provide the precise age. Some respondents may not know their date of birth, even their approximate age. For instance, some people from the third world may not know their age precisely, but only in relation to some memorable events such as a revolution, a war, the assassination of a famous person or the granting of independence of their nation. For most demographic items data can be collected with simple tick boxes. For example:

Are you: Single (never married)

Married or living with a partner

Widowed

Divorced

Separated

Questions on behaviour can be obtained using yes / no responses. For example:

Yes No

Do you help care for anyone who is physically disabled?

Do you smoke cigarettes?



Questions on attitudes can be collected using a Likert scale. For example:

Do you agree or disagree with the following statement?

Police in the UK should be allowed to carry firearms.

strongly agree

agree undecided disagree strongly disagree

5 4 3 2 1

The number of choices can be an odd number or an even number. The latter is useful when you want to deny a respondent a middle option.

Questions on beliefs and knowledge can be obtained using true/ false statements. For example:

Please circle one number for each statement True Not Sure False

Regular exercise can help prevent heart disease 1 2 3

Exercise must hurt to be any good 1 2 3

Pregnant women shouldn’t exercise 1 2 3

People with heart disease shouldn’t exercise 1 2 3

Knowledge questions can be threatening and respondents may guess what they think is the correct answer. Above we have used a middle option of ‘Not Sure’. This is more tactful than the alternative of ‘Don’t Know’. The former reassures respondents that it is OK not to know the correct answer, whereas the latter simply implies the respondent is ignorant! Hence, including a ‘not sure’ option can minimise guessing.

Use of Prompts Decide if you need to use prompt cards which provide a comprehensive list to stimulate the memory of respondents as memory and recall of events can be unreliable. The cards can be used to point out activities they may have participated in over the past 6 months, or health professionals they may have seen over this period. For the latter, the list may include health care workers a respondent might not have thought of e.g. pharmacist or NHS Direct. Use of decoy or dummy questions ‘Decoy’ or ‘dummy’ questions can be used to throw a respondent off the scent if you are seeking particularly sensitive information. They may also include questions on an identical topic but which are placed in separate parts of the questionnaire in order to validate internally the information respondents are giving. For example, you may ask two questions on the same topic but phrased differently and separated within the body of the questionnaire:



“Do you agree that people over the age of 75 years Yes No should have a free TV licence?”

“Do you agree that all people, irrespective of their age, Yes No should have to pay for their TV licence?”

Step 7: Decide the wording of each question The wording will depend on whether the responses are to be closed or open-ended. As a general rule, keep it simple with questions which are easily understood and not offensive. Assess the wording for these additional characteristics:

Meaning / comprehension

Clarity

Cultural equivalence

Ambiguity

Simplicity

Length (short better than long)

Avoid jargon

Avoid double-barrelled questions (2 questions in one)

Avoid leading questions

The wording of the question can easily affect the response. Avoid the use of leading questions, complex questions, double-negatives and questions which do not reflect balance. Some examples are given below.

Example: Meaning / comprehension

When you went into the operating room did the atmosphere in theatre give you confidence about your operation?

yes no

Well, the room was nice and warm The aim of this question was directed at the atmosphere (and attitude) between members of staff in theatre. Alternative wording: When you went into the operating room did the working relationships between members of staff give you confidence about your operation? Alternative wording: When you went into the operating room did the mood between staff give you confidence about your operation?

Examples: Clarity

Do you agree or disagree that smoking causes lung cancer? yes no What does a ‘yes’ response mean?

Has a doctor ever told you your blood pressure was high? yes no Patient may have been told by a nurse that BP was high

Alternative wording: Has a health care practitioner (for example, a doctor or nurse) ever told you that your blood pressure was high?

Have you seen your GP recently? yes no Yes, I passed her in the supermarket last Tuesday

Alternative wording: Have you visited your family doctor for a health problem in the past month?



Examples: Cultural equivalence

Do you bring up phlegm from your chest first thing in the morning in the winter?

yes no

In some cultures it is frowned upon to spit so any secretion from the airways is always swallowed

In the past 5 years has an elderly relative of yours died at home? yes no Some Chinese people consider it is very bad luck to die in your home and, apparently, they will make whatever arrangements are necessary to prevent this from happening.

Examples: Ambiguity

Would you describe yourself as a light drinker or a heavy drinker? light heavy

Well, I don’t weigh much so I must be a light drinker, even though I drink 8 pints of beer a night

The alternative wording should present respondents with a checklist of quantities of alcohol drunk (e.g. within a defined period) which the researcher can categorise as ‘light’, ‘moderate’ or ‘heavy’ drinking.

Example: Simplicity

What type of residence do you live in? Alternative wording: What type of home do you live in?

What is your opinion of.... Alternative wording: Do you think.....

Example: Length

If, after starting your pain medication (whether prescribed by your GP or hospital doctor) have you suffered from headaches, nausea, fainting fits, trembling, night sweats, memory loss, stomach upsets, or episodes of urinary incontinence?

yes no

Alternative wording: Have you had any side effects from your pain medication?

Examples: Jargon

Have you ever had anaphylactic shock? yes no Well, I got a belt off a light switch once

Alternative wording: Have you ever had an allergic reaction so bad that it left you with breathing difficulties, lots of itching and swelling of your mouth or skin?

Do you expectorate first thing in the morning? yes no Alternative wording: Do you bring up phlegm from your chest first thing in the morning?

How many siblings do you have? ___number Alternative wording: How many brothers and sisters do you have? You could provide a single response box for both sexes together or separate boxes for each sex.



Example: Double-barrelled (two questions in one)

Do you often go out in the sun and wear a sun hat? yes no Alternative wording (topic 1): In the past month have you gone out in the sun on at least 3 days every week? Alternative wording (topic 2): Have you ever worn a sun hat in the UK?

Example: Leading questions

Don’t you agree that smoking causes lung cancer? yes no Alternative wording: Do you agree that smoking causes lung cancer?

Do you not think the waiting time for an appointment was unreasonable?

yes no

Alternative wording: Do you think the time it takes before you get an appointment is reasonable or unreasonable?’

Use of ‘priming’ questions Questions may be used to ‘prime’ a given response but their inclusion can result in biased views. On balance they should be avoided unless you have a very good reason to include them.

Example: The Ministry of Defence was concerned about the falling number of recruits to the army. Accordingly, they planned a survey of the general public to support their request to HM Government to re-introduce national conscription. The key question was: ‘Do you think the Government should reintroduce national conscription?’ Yes / No They used a ‘priming’ question before asking the key question:

(1) Do you think it is important for young people to have a sense of discipline in their lives?

(2) Do you think the Government should reintroduce national conscription?

86% of the public answered ‘Yes’ to question (2) The results of the survey were presented to the Cabinet. But the Treasury were unhappy with the prospect as they would have to find the money! Accordingly, they organised another survey of the general public asking the same key question but with a different ‘priming’ question:

(1) Do you think it is important for young people to be taught how to handle guns?

(2) Do you think the Government should reintroduce national conscription?

86% of the public answered ‘No’ to question (2) Source: A story line from the BBC series ‘Yes Minister’ Message: with an appropriate ‘priming’ question you can obtain any answer you want!



Timing issues Use of words such as ‘frequently’, ‘often’, ‘rarely’, ‘sometimes’, ‘usually’ or ‘regularly’ can be subject to different interpretation between respondents and lead to confusion. It is best to specify a time frame, for example, ‘less than once a week’, ‘more than once a week but less than once a day’ etc. For example:

‘During the past 4 weeks have you been in a wheezy or asthmatic condition at least once a week?’

With a question such as ‘Do you sunbathe frequently?’ specify a time frame and offer a choice of responses such as:

In the last month, have you sunbathed: (1) 0 times (2) 1-5 times (3) 6-10 times (4) more than 10 times?

An alternative to the question ‘Do you exercise regularly?’ could be:

In the past week have you done any leisure time physical activity that has lasted for at least 20 minutes?

In general, for Quality of Life studies of acute conditions you should ask about the last week and for more chronic conditions ask about the last month (or 3 months). Response choices The choice of responses should be mutually exclusive, for example, for age, use 0-4, 5-9, 10-14, 15-19 and not 0-5, 5-10, 10-15, 15-20 etc. Use appropriate response choices. Consider the question:

Are you in favour of private health care? yes no

This example does not give respondents a ‘middle’ option with ‘no view’. Some respondents might be in favour of some aspects but not other aspects of private health care, so why not ask the separate components if this is judged important. Sensitive questions It can be very difficult to get truthful replies to sensitive questions. In this situation it may be justified to use a leading statement making the assumption that the behaviour you are asking about is very common. Ann Bowling quotes the example: ‘Do you smack your children?’ which may be better phrased as:

Even the calmest parents smack their children sometimes. Did your child (or children) do anything in the past 7 days that resulted in you having to smack them?

Other sensitive areas may include sexual preferences, rectal bleeding, stress incontinence, attitudes to assisted suicide, attitudes on (or experiences of) pregnancy termination etc. Addressing these topics requires careful and tactful thought. It may be advisable to first consult a sample of potential respondents and colleagues on the choice of wording.



Step 8: Design the layout The layout should be clear, uncluttered allowing a good flow of questions, and ‘user-friendly’ to make the questionnaire easy to complete. Avoid splitting a question and its response categories over two pages. Do not be tempted to squeeze the questions into two sides of A4 when spreading them out would increase the visual attractiveness of the questionnaire. A 4-page questionnaire can be printed single-sided or double-sided on A4 or, more preferable in an A3 format, as a booklet. You should choose a layout to minimise the possibility that a respondent may turn two pages at once and miss out a whole section. Choose a font that is easy to read and large enough to be clear for anyone with limited visual acuity. Arial or Comic Sans fonts size 12 are probably the most convenient but this decision can be left to the piloting stage. At the end of the questionnaire thank the respondents and give clear instructions for returning it. It is not uncommon to be presented with a questionnaire that has no instructions for returning it! Give clear instructions for completing each question. For example, ‘Go to Question 3’, ‘Please tick all that apply’, ‘Please tick one box only’. Consider the distance between a category and its response box. For example:

Question: Do you have or have you ever had: please tick all that apply Bronchitis Pneumonia Pleurisy Tuberculosis (Tb) of the lung Hay fever Asthma Other lung diseases The response box is too far away from the category it refers to, as well as being rather small. With such a gap it is easy to tick an incorrect box. Align the category and its response box and place the text in the centre of the page to improve the layout:

Bronchitis

Pneumonia

Pleurisy

Tuberculosis (Tb) of the lung

Hay fever

Asthma

Other lung diseases



Can you distinguish between missing values or missed questions? Rather than using a single tick box for indicating only a positive response consider adding sufficient response boxes to make sure the respondent has had to indicate a response for each question. For example:

Here is a list of things that people say stops them taking more exercise. Please indicate which, if any, apply to you: Please tick all that apply

I am not the sporty type

I have not got the time to exercise

There is no-one to look after my children while I exercise

I am too old to exercise

My health is not good enough

Here is an alternative choice of response boxes which ensures the respondent has to check one or other of the boxes which improves the chance of detecting missed questions:

Please tick the relevant box

Applies to me Does not apply to me

I am not the sporty type

I have not got the time to exercise

There is no-one to look after my children while I exercise

I do not have anyone to exercise with

I am too old to exercise

My health is not good enough

1

1

1

1

1

1

2

2

2

2

2

2

1

1

1

1

1



Funnelling Funnelling uses filter questions to improve the flow of the questionnaire by placing general questions before specific ones and allowing respondents to skip sections that are irrelevant to them. For example:

Qu 1 Do you smoke cigarettes?

Yes 1 → Go to Qu 3 below

No 2

If ‘No’, have you ever smoked as much as one cigarette a day for as long as a year?

Yes 1 → Go to Qu 2 below

No 2 → Go to Qu 4 (next section)

Qu 2 FOR EX-SMOKERS How long ago did you stop smoking cigarettes?

Less than a year ago 1 → Go to Qu 3 below

More than a year ago 2 → Go to Qu 4 (next section)

Qu 3 FOR SMOKERS (and those who have given up within the past year) How much on average do (or did) you smoke?

1-9 cigarettes a day 1










100 or more a day 11

Here, we have provided a long list of options rather than a shorter list with just 3 categories of, for example, 1-9, 10-19, 20 or more a day, which is the usual practice! Under the latter categorisation a respondent smoking 60+ a day will be grouped with others smoking just 20 a day. This may be acceptable, depending on the study and the amount of detail required. However, the example with the expanded list illustrates the use of a scale that allows respondents to be more truthful in admitting to this particular behaviour. People may not wish to tick the highest estimate of consumption if it concurs with their own behaviour, an example of ‘social desirability’ bias. “Look, some folk smoke more than 100 a day. That’s OK, I only smoke 60-70 a day.” This approach has application with other behaviours that may be judged ‘sensitive’ including alcohol consumption.



A question on cigarette consumption could ask simply ‘On average, how many cigarettes a day do you smoke?’ You should be aware that a smoker’s consumption may vary between weekends and weekdays due to restrictions at work so any questions on consumption should relate to these periods and a summary value derived. Another useful question on consumption, for ex-smokers or current smokers, is ‘What is the most you have ever smoked?’ This question could be related to a given time frame (e.g. a year) or period in their lives (e.g. when you were a teenager). The response to a single question on daily cigarette consumption may not concur with a separate question such as ‘On average, how many packs of cigarettes (20s or 10s) do you buy each week?’ There’s another important design aspect in the smoking example above. Each response box has a number beside it which is the code to be used with that particular response. Adding the codes can help with entering data into the computer. When listing optional response choices be careful with the format and placement of adjacent response categories. For example:

On average how many cups of coffee do you drink a day? (please tick)

None.....1..........2-4..........5-9..........10 or more..........

Here it is easy for a respondent to tick the wrong category. As a general rule it is best to list choice categories vertically:

None ...... None

1 ...... Or better still 1

2 - 4 ...... with a box 2 - 4

5 - 9 ...... 5 - 9

10 or more ...... 10 or more

Step 9: Decide the coding scheme, scales and think about the analysis of the replies Choice of codes Decide how you are going to code the information, using either numeric or alpha-numeric codes. Gender can be coded 1=male and 2=female (numeric), or you could use ‘M’ and ‘F’ (alpha-numeric) but make sure the codes are recorded in a formal coding scheme as part of the study’s protocol.

Warning: A University supervisor sent me an Excel computer file containing the results from an old questionnaire survey undertaken by one of his students. I was asked to analyse it and the plans were to write up the study for publication. Gender was coded as ‘1’ and ‘2’ but it was impossible to discern which code was male and which female. The student had long since left, had moved to another country, got married and changed her name so there was no opportunity of consulting with her. The supervisor did not have a coding scheme, or protocol and there were no copies of completed questionnaires to check. The data gave no clues as there were no anthropometric measures available (e.g. height in adulthood) from which we could have made an educated guess. Sadly, the data could not be analysed and the whole survey had been a wasted exercise.



The codes can be added alongside each response box, as in the smoking example above, to improve the consistency of transferring data from the questionnaire to the computer file. Choice of scales Choose the scales carefully and think about how the results will be handled, analysed and interpreted. Responses can be dichotomous (yes / no) or scaled on a continuum with extreme ranges marked 0-10, or worded using ‘not at all’ to ‘all the time’, or as a multiple choice (e.g. description of pain as tingling, hot, throbbing, sharp with the possibility of more than one response e.g. pain is hot and throbbing). The most common scale used for measuring responses is the categorical scale, in the Likert format of a 5, 6 or 7 point response option. Here, a respondent is asked to pick a category such as ‘none’, ‘very mild’, ‘mild’, ‘moderate’, ‘severe’, ‘very severe’ that best describes their condition (e.g. severity of pain or limitation of activity). It is commonly used because it is easily understood and analysed, although constructing them needs careful wording of the various text descriptions. Many people have a tendency to preferentially choose the ‘middle’ value, which avoids them having to make a decision. The use of an even number of response options can be used when you wish respondents to make a definitive choice. Example of a Likert scale which uses a series of opinion statements, in this case with 5 choices.

strongly agree

agree undecided disagree strongly disagree

5 4 3 2 1

The convention is to use a high score for a favourable evaluation. Likert scales provide ordinal level data, which is presented in a ranked order but the magnitude of the difference between adjacent categories is not identical. There is no assumption of equal intervals on the scale. Thus, the difference between ‘agree’ and ‘strongly agree’ may be perceived by the respondent to be different to that between ‘agree’ and ‘undecided’. Other examples of ordinal scales are pain scales and those often used in patient satisfaction surveys. Another form of Likert scaling is the Visual Analogue Scale (VAS) which is anchored at two ends with an appropriate statement. The response categories may be (1) numerical or (2) presented as a continuous 10 cm line on which the respondent is invited to mark some point along it; the distance from one end (usually the left when this conforms to the least worst option) is then measured and recorded as the response (out of 10 cm). The protocol should include detail of which end of the scale you are measuring from as this may not be intuitive when the scale is anchored as, for example, ‘strongly agree’ and ‘strongly disagree’.



The instruction to complete the VAS should be specific. State ‘circle the correct response’ rather than ‘indicate the appropriate response’:

strongly agree

strongly disagree

10 9 8 7 6 5 4 3 2 1

Draw a mark along the line:

no worst

pain pain

An alternative model, often used for children and those with a learning disability involves use of ‘smiley faces’. For example:

Please tick () the appropriate face:

Whatever the format it is still basically a Likert scale and the task is to indicate the extent to which the person accepts or rejects various statements relating to an attitude object. For statements about pain, discomfort, breathlessness etc. the use of the VAS scale can pose problems in interpretation. The two ends of the scale will be anchored as ‘no pain’ and ‘worst imaginable pain’. We can all agree with what ‘no pain’ means but different people will have different views about what constitutes the ‘worst imaginable pain’. Hence, there is always a question over the calibration of the upper end of the scale which will vary between people in interpretation depending on their experiences and cultural beliefs. Generally, such scales are difficult to interpret between people but have more use in studies of change within the same person (perhaps before and after an analgesic is administered).

To discourage people from simply ticking the right hand side of a response category without first reading the question you should vary the order of negative-positive responses for consecutive questions. For example:

In the past month how much time have you spent on your hobbies? More time than usual

About the same as usual

Less time than usual

Much less than usual

In the past month how much time have you spent visiting friends?

Not at all No more than usual

Rather more than usual

Much more than usual

A VAS scale using a continuum can be placed horizontally (as above) or vertically (next page). Here patients are asked a question, in this case about their state of health today, and asked to mark a 0 – 100 scale similar to a ‘centigrade thermometer’. Patients find this easy to understand and this type of scale is popular with researchers as it is easy to score.



We would like to know how good or bad

your health is TODAY

This scale is numbered from 0 to 100.

‘100’ means the best health you can

imagine.

‘0’ means the worst health you can

imagine.

Please mark an X on the scale to

indicate how your health is TODAY.

10

0

20

30

40

50

60

80

70

90

100

5

15

25

35

45

55

75

65

85

95

The best health

you can imagine

The worst health

you can imagine



A quick, semi quantitative scoring system was developed by Geoffrey Field to measure the severity of a respondent’s symptoms. The questions posed can relate to any suitable time interval (e.g. the past 4 weeks). In the example below the symptom being assessed is cough. Nine cards, each containing two questions out of a list of six, are presented to the participant in random order (Table 4). The participant compares the statements and selects the one that is ‘nearer to the truth’. Neither may be true but one statement will be closer to the truth than the other. The number of the statement chosen by the subject is marked on the score sheet and the cross-over point from the lower to upper line indicates the cough score (4 in this example). The success of the test depends on the participant being able to mentally rank the pairs of questions consistently. With suitable wording the technique can be adapted for other symptoms.

Table 4. Cough frequency statements and their order on the 9 cards

Card number Order ( 1 or 2) Statement

1 1 I cough occasionally

2 I rarely ever cough

2 1 I rarely ever cough

2 I cough a little on most days

3 1 I cough a little on most days

2 I cough occasionally

4 1 I cough occasionally

2 I cough a little every day

5 1 I cough a little every day

2 I cough a little on most days

6 1 I cough a little on most days

2 I cough a lot every day

7 1 I cough a lot every day

2 I cough a little every day

8 1 I cough a little every day

2 I cough all the time

9 1 I cough all the time

2 I cough a lot every day

Scoring sheet:

Card number

1 2 3 4 5 6 7 8 9

Statement number

2

1

2

1

2

1

2

1

2

1

2

1

2

1

2

1

2

1

[Source: Field GB. The application of a quantitative estimate of cough frequency to epidemiological

surveys. Int J Epidem 1974; 3: 135-143]



Data handling and analysis Decide how you are going to handle the results and what you are going to do with them? This depends very much on the type of data you are collecting. Categorical (qualitative) data includes nominal and ordinal measures. Data are described as nominal if there is no natural order. Examples include marital status, smoking habit, religion, eye colour, nationality and vital status (dead or alive). Data are described as ordinal where there is a ranked order but the magnitude of the difference between adjacent categories is not identical. For example, students taking an exam can be ranked as first (top), second, third etc. The top mark may be 90%, the second 88% (a difference of 2%) but the third 65% (a difference of 23%). Other examples of ordinal scales are the Borg scale (breathless scores) and those often used in patient satisfaction surveys. The other types of data are interval or quantitative data which are derived from a count, or a standard measurement, and has a frequency distribution. The numbers can be discrete or continuous. Discrete data are integers (whole numbers) where the magnitude of the difference between adjacent categories is identical (unlike ordinal data above). Examples include the number of children in a family (0, 1, 2, 3, 4, 5 etc) and number of asthma attacks in a year. Continuous data include measures such as height and age which can take any value within a range and which are measured in standard units. The distinction between the different types of data is important because this has implications for graphically representing the data and choosing an appropriate summary descriptive measure and statistical test (see the NHS Fife Research Study Guide ‘An Introduction to Medical Statistics’ for further guidance on data types). Make up a datasheet sheet with some dummy results. Use a code to indicate a missing value (e.g. 999 for weight in kg). This is preferable to having blank fields in your dataset.

Decide if you are going to analyse the responses to individual questions or create a summary score from a number of responses? A summary score can be useful but you should be aware that any given score can be made up from varied responses to the component questions. This can prove difficult to interpret when studying change in the summary score over time. For example, consider these 4 questions taken from a questionnaire designed to measure the quality of life of children with eczema. The scores from each are added to provide a summary measure.



(1) Over the last week, how much has your skin problem affected your friendships?

Very much 4

Quite a lot 3

Only a little 2

Not at all 1

(2) Over the last week, how much did your skin problem affect your school work?

Very much 4

Quite a lot 3

Only a little 2

Not at all 1

(3) Over the last week, how much trouble have you had because of your skin with other people calling you names, teasing, bullying or avoiding you?

Very much 4

Quite a lot 3

Only a little 2

Not at all 1

(4) Over the last week how much has your sleep been affected By your skin problem?

Very much 4

Quite a lot 3

Only a little 2

Not at all 1

In a follow-up study a child may score these 4 questions as 4, 1, 3, 1 at baseline yielding a summary score of 9. At follow-up the same child may score the questions as 1, 3, 1, 4 also yielding an overall score of 9, indicating no change in quality of life whereas in reality an improvement in one aspect (friendships) has been offset by a deterioration in another aspect (sleep and school work).

Message: Aggregating individual scores into a combined score can lead to loss of information whereby the same overall score can arise from many different combinations of responses to the sub-domains of the scale, with unknown meaning and hence unknown indications for action.

Another serious problem can arise with the model adopted for scoring responses. The responses in the example above have been coded 1-4 when the minimum and maximum summary scores are 4 and 16, respectively. The child’s baseline scores are 4+1+3+1 = 9 and you may be tempted to express this child’s score as 56% (9/16), but you would be wrong to do so. An alternative scoring system would be 0-3 where 0=’not at all’ and 3=‘very much’. The minimum and maximum summary scores are now 0 and 12, respectively. Rescoring the child’s baseline responses now gives 3+0+2+0 = 5. Here the percentage is 42% (5/12). When the minimum summary score is not zero you must adjust the calculation by subtracting the minimum score from both the numerator and denominator. The correct equation for the first scoring model is: (9 – 4) / (16 – 4) = 42%. Step 10: Write the protocol Write a detailed protocol for completing, scoring, coding, and analysing the replies. This should be available for the interviewers and also will help you to remember key aspects when you come to analyse the replies and review the results after the survey or when you (or someone else) comes to write up the study. You may find coding clashes occur as, for example, when a respondent ticks more than one box to a



question requiring only one response. The protocol should include a section for these protocol violations to be addressed as and when they occur. Your decision in dealing with them should be recorded to maintain a consistent approach. (7) Training the Interviewers For interviewer administered questionnaires where there will be more than one interviewer it is very important to train them to ensure they have a consistent manner in asking and recording the responses. Training should occur with all the interviewers together. Interviewers must be careful to use the exact wording each time and not to vary the question or emphasis according to the respondent. Each interviewer should be familiar with the protocol and be given their own copy. Check their responses as they interview people and send in results. You should ask the interviewers to audit the time taken for their interviews to identify if any are taking short cuts! Do quality checks on a sample of the replies including the coding frame (which should be referenced in a protocol). Be alert to any differences between interviewers in their technique and stamp on them when you find evidence of inconsistencies. (8) Piloting Once you have developed the first draft show it to your colleagues and to any lay members of your family or friends and ask for comments. Pilot the draft on a sample of individuals for whom the questionnaire is designed. Pilot the whole process including the data coding and the return arrangements. For example, if the questionnaires are to be returned by post what is the delay in getting through your institution’s post room? Evaluate the pilot results and do not be disheartened if the replies are not what you expect. Review the questionnaire, make the changes and, if necessary, repeat the pilot. This is an iterative process and the time is well spent to avoid disappointment later if you continue to use what the initial pilot results indicate is a flawed questionnaire. Piloting is also advisable if you intend to use a previous validated questionnaire but in a different population from that used in its development.

Finally, check if the final, piloted version still addresses the aims of your study

and the questions you are seeking answers to!

(9) Last thoughts on Design Remember to thank the respondents for completing the questionnaire. Ask them to check through their replies to ensure they have completed all the questions. Include the instructions for returning the questionnaire. If relevant, ask respondents to record the date the questionnaire is completed because there may be seasonal variation in the responses as, for instance, when asking about respiratory symptoms, falls on pavements in the winter months etc. It may be important to allay any fears of disclosure if you are using an anonymised questionnaire. Give assurance that no one will be identified, that all replies are



anonymous and confidential to the research team. In some studies you will need the respondent’s name if, for example, you intend to align the information they provide with other data such as medical records to check the validity and accuracy of replies. However, medical records are often incomplete or information recorded in them may be inaccurate. In addition, asking a patient if he has ever had disease ‘x’ may not concur with his medical records because he may not have been told he had it! In relation to health data it appears that patients are reliable sources in relation to major events and conditions. There will always be the potential of recall bias from relying on a respondent’s memory. In a large survey you can reduce the need for expensive reminders by asking a respondent to complete a separate slip with their name and address attached to the returned questionnaire. The slip is separated from the questionnaire on receipt and the respondent’s name removed from the list of reminder addressees. (10) Some further issues (10.1) Consent In some cases consent can be implied when a respondent returns a completed postal questionnaire. In other circumstance it is good practice to seek formal, informed consent and ethics committees will expect some process to be in place to ensure a participant who agrees to take part in an interview is fully informed and has consented to it. (10.2) Computer –read questionnaires Questionnaires that are to be read by a computer require respondents to fill in a box with a number, letter or a mark (a tick, cross, or fill in). The questionnaire is then put onto a scanner to be ‘read’ and the data are entered directly onto a database. They are convenient when dealing with many hundreds of returned questionnaires. However, for smaller samples they are probably not cost-effective as they require additional design considerations and the data on the database will need to be checked against the original questionnaire. (10.3) On-line questionnaires and use of computer-administered

questionnaires On-line (electronic) questionnaire surveys are popular and can have advantages regarding reduction in cost, though their use may attract a copyright fee. They can cover large samples with ease of data capture. They require each invitee to have an email account which is not always possible and may result in biased samples. Other advantages include a reduction in item non-response and the opportunity to build in a process to avoid ‘invalid’ responses where a participant is unable to proceed to the next question if their response to the previous one is out of a set range. The usual rules in designing the questionnaire also apply. The use of laptop computers to collect questionnaire responses is becoming more popular. However, some people are not comfortable with use of such IT and may prefer a paper copy if this can be supplied. Other people may have visual deficits, including dyslexia, which prevent them from reading the screen. Changes may be necessary in relation to font size, contrast and glare to improve screen readability.



These problems can lead to a bias in obtaining a representative sample of the target population under study. The environment in which participants are asked to complete the questionnaire should be quiet and private to protect their anonymity and the possibility of others seeing their responses. (10.4) Validity of Questionnaires A good questionnaire should be fit for purpose and valid. The main criteria for establishing a questionnaire’s validity are: Face validity: This concerns the researcher’s subjective assessment of the relevance and design features of the questions to the topic under study. Simply put, on the face of it does it appear to do the job? Content validity: This refers to an assessment of the degree to which the content of the questionnaire relates to the domain it is intended to measure. For example, a measurement of functional health status should include activities of daily living, and engagement with occupational and social activities. Simply put, does it fully cover all aspects of the problem? Construct validity: This comprises two aspects of validity which together refers to the degree to which the questionnaire satisfactorily relates to any theoretical concepts (constructs). Convergent validity requires that the scale should correlate with related variables, that is, the results should correlate with the results of other questionnaires designed to measure the same aspects, and with variables for which you would expect a natural correlation. For example, a questionnaire designed to measure quality of life in patients with ulcerative colitis may be expected to generate results that would correlate with measures of disease activity. On the other hand, discriminant validity requires that the construct should not correlate with variables for which you would not expect a natural association. For example, you would not expect an association between functional health status and religion. Repeatability and Sensitivity to Change: The questionnaire should give repeatable results when administered under similar conditions and be sensitive to any change in status. Results should be repeatable, at least in the short term both within a session and between sessions. Acceptability and Practicability: The questionnaire must be acceptable to the population under study and capable of being easily administered to the study participants. (10.5) Getting a good response rate from a postal survey Ensuring a good response rate from a postal survey can be problematic, particularly for lengthy and complex questionnaires and those tackling sensitive issues. If relevant, make sure the questionnaire is headed ‘Confidential’. As a general guide consider these approaches to improve the response rate:

Write a short letter of introduction on headed paper which identifies your institution, lists the study title and provides your contact details. If possible, personalise the letter and sign it personally in blue ink.



Give a clear, simple and short explanation of why it is important to fill in.

Give the reason(s) why the participant has been chosen.

Give a realistic estimate of the actual time involved in completing the questionnaire.

Give assurance that the results are confidential and that the replies are anonymous (if relevant).

Post out the questionnaires on Thursday to arrive by Saturday morning. Provide a stamped return envelope. Give a date for return (not too distant).

Send reminders in 2 - 3 weeks and probably no more than 2 reminders (an ethics committee may express concern over more than 2 reminders).

Include a second copy of the questionnaire with reminder letters.

Include the Institution’s logo on the envelope to set it apart from ‘junk mail’.

There is conflicting evidence about the use of coloured paper. There is little agreement in the literature but it can be useful to print several versions on different coloured paper and pilot them.

Use stamps rather than franked envelopes and first class post.

The use of monetary incentives can be beneficial but in general their use is considered unethical and may result in a biased survey with reduced return rates from those with higher incomes.

Finally, make sure you have designed the questionnaire with all the features referred to in section 5 above!

It is very important to give a realistic estimate of the time needed to complete a questionnaire.

Example: In a recent on-line survey a researcher had estimated the time to complete their questionnaire as 10 minutes. The survey actually included 46 questions (some with free text entries) and took at least 25 minutes to complete! The final results showed the completion rate of questions fell off markedly after question 22 (‘item non-response’) as respondents were obviously becoming irritated with the ongoing questions and no indication of how many were left to complete. They simply stopped completing the questionnaire and ‘signed off’ the website. This invalidated the entire study as the researcher wanted to cross-tabulate results of questions from the first and last part of the questionnaire. It is good practice with on-line questionnaires to provide some guidance on the respondent’s progress through it. This can be a horizontal bar showing the percentage completed or a simple statement that the respondent is now on question x of y.



(10.6) Sources of Bias in Questionnaire surveys Bias is the unequal distribution of error. It is the greatest threat to the validity of a research study and a key aspect to consider when designing a questionnaire. Principal sources of bias in questionnaire surveys are given in Table 5.

Table 5. Principal sources of bias in questionnaire surveys.

Source Comment

Design Any aspect of study design, e.g. faulty questionnaire design, flawed questions, poor return arrangements, faulty sampling, poor statistical analysis with failure to account for confounding, use of wrong statistical tests.

Assumption Faulty logic of investigator, which can lead to faulty conceptualisation of the research problem, faulty interpretations and conclusions.

Selection Faulty selection when the characteristics of the sample differ from those of the wider, target population. All potential subjects should have an equal chance of being chosen e.g. written invitation and questionnaire not read by illiterate people or those who cannot read English.

Response A major source of bias leading to a systematic error from differences in characteristics between those who accept and those who decline an invitation to complete the questionnaire. It is not always possible to compare the characteristics of responders and non-responders but it should be done where there is a source of independent data.

Item non-response

The failure to complete a question. Item non-response is generally greater in postal than interviewer-administered questionnaires. A large item non-response rate can threaten the utility of a questionnaire when its presence leads to an inability to calculate the summary score for a structured domain made up of multiple questions.

Measurement Systematic error from measurement errors (e.g. VAS readings), change of interviewers between repeated assessments, data handling procedures.

Measurement decay

Error from a change in the measurement process over time due to a change in technique by an interviewer or a person charged with coding the questions.

Classification Categorisation of the results. For example, definition of an ex-smoker (abstinent for one day, one week, one month, six months, one year, ten years?)

Recall Recall by respondents may be selective or otherwise different between groups with different rates of cognitive decline.

Reporting Respondents may be apprehensive about being interviewed and give the responses they think the interviewer wants. Respondents may under-report or over-report symptoms depending on any vested interest e.g. occupational surveys of back injury, with denial to avoid being made redundant, or over reporting to get compensation. Bias can arise with postal questionnaires when it may be uncertain who has filled in the questionnaire and if they have had help.

Social desirability

People may wish to present themselves at their best and will respond to sensitive questions accordingly.

Acquiescence response set (‘yes-saying’)

Respondents will more frequently endorse a statement than disagree with its opposite. Avoid attitude questions that are all worded positively because some persons are ‘yes sayers’ regardless of the content.

Interviewer Differences in measurement techniques between interviewers, and within interviewers over time (measurement decay). Different interviewers may show systematic differences in asking questions and recording responses. Interviewers may ask questions in a manner which encourages respondents to answer in a desired way. Initial training and inter-interviewer assessments are very important to eliminate differences in techniques. May need to be repeated throughout the study.

Follow-up Loss of follow-up. Bias due to systematic differences in characteristics between those who return compared to those who decline to attend, or are otherwise lost to follow-up interviews in a cohort study.

Analysis Inappropriate use of statistical methods, for example, different treatment of outliers, missing data, incorrect tests of significance and neglect of confounders.

Interpretation Errors in inferences drawn from the statistical analyses, for example over aspects of association versus causation.



(11) Summary Questionnaires have their advantages and disadvantages but, overall, they are convenient ways to gather large amounts of data. They do require careful planning on all the design issues identified here and developing a questionnaire ‘fit for purpose’ from scratch is a project in itself. Considerable time and effort may be needed to complete the task and, once done, the questionnaire will have to be administered to the target sample by post or by interview, the results gathered, analysed and interpreted. Hence, a project can easily overrun its timetable. The success of any survey using a questionnaire will depend on getting this first phase right! (12) Further reading

The question bank: http://surveynet.ac.uk/sqb/qb/

http://www.statpac.com/surveys/contents.htm Designing surveys and questionnaires.

Bowling A. Measuring health: a review of QoL measurement scales. Open University Press, Buckingham. 3rd ed. 2005.

Bowling A. Measuring disease: a review of disease-specific QoL measurement scales. Buckingham. Open University Press, Buckingham. 2nd ed. 2001.

Bowling A. Research methods in health: investigating health and health services. Buckingham. Open University Press, Buckingham. 4th ed. 2014

Bowling A, Ebrahim S (eds). Handbook of health research methods: investigation, measurement and analysis. Open University Press, Buckingham. 2005.

Edwards P, Roberts I, Clarke M et al. Methods to influence response to postal questionnaires (Cochrane Methodology Group), The Cochrane Library, Issue 1, 2001.

Graham C, MacCormick S. Overarching questions for patient surveys: development report for the Care Quality Commission (CQC). National Patient Survey Co-ordination Centre. Picker Institute. 2012.

McColl E, Jacoby A, Thomas L et al. Design and use of questionnaires: a review of best practice applicable to surveys of health service staff and patients. Health Technology Assessment. 2001; 5: 31.

McGibbon, G. How to avoid the pitfalls of questionnaire design. Nursing Times May 7, 1997; 93 (19): 49-51.

Oppenheim, AN. (1992). Questionnaire design, interviewing and attitude measurement. London: Pinter Publishers. Source material on rules for making up questionnaires

The medical literature (write to the authors but also you should be aware that sometimes copies of a questionnaire used are lodged with the journal editor)

http://surveynet.ac.uk/sqb/qb/

http://www.statpac.com/surveys/contents.htm



Appendix: sample questionnaires

(1) Recommendations for a question on patient experience. Source: Graham C, MacCormick S. Overarching questions for patient surveys: development report for the Care Quality Commission (CQC). National Patient Survey Co-ordination Centre. Picker Institute. 2012. These authors set out to develop and test a single question that could be used to measure patient experience consistently in a range of health care settings. They tested 6 different questions with a panel of service providers and service users. Based on their findings from cognitive testing they recommend the following 11-choice overarching measure of patient experience: Overall,

I had a very I had a very

poor experience

good experience

0 1 2 3 4 5 6 7 8 9 10

______________________________________________

(2) Example of some questions taken from a questionnaire on quality of life in children with eczema that was modified in format to improve its utility.

Original version:

(1) Over the last week, how itchy, ‘scratchy’, sore or painful has your skin been?

Very much

Quite a lot

Only a little

Not at all

(2) Over the last week, how upset or embarrassed, self-conscious or sad have you been because of your skin?

Very much

Quite a lot

Only a little

Not at all

(3) Over the last week, how much has your skin affected your friendships? Very much

Quite a lot

Only a little

Not at all

(4) Over the last week how much have you changed or worn different or special clothes/shoes because of your skin?

Very much

Quite a lot

Only a little

Not at all



Modified version:

Source: M.S. Lewis-Jones, A.Y Finlay, June 1993 and Media Resources Centre, University of Wales College of Medicine, December 1996.



(3) Example of a first draft patient survey with the final version adopted.

Patient Satisfaction Survey (first draft) Dear Patient/Parent/Carer, Thank you for taking the time to complete this questionnaire regarding your experience of visiting the Therapy department and the treatment received.

Please tick the appropriate answer and use the space given if you want to comment further.

If you do not want to answer some of the questions no reason is required. Your responses will be anonymous and will be kept confidential.

Section 1 Thinking about your stated appointment time, were you seen within: 0 – 10mins 10 – 20 – 30mins Apology/information if the clinic was running late: Yes No

Excellent Good Fair Poor N/A The level of cleanliness of the therapy room Comments: ………………………………………………………………………………………….............



Section 2: At your appointment, did the therapist introduce him/herself (and any other health professional/student that may be observing or assessing in the clinic? Yes No Did the therapist explain the purpose of your visit today? Yes No Thinking about your appointment with the therapist, how would you rate the following? Excellent Good Fair Poor N/A The information given about you/your

Child’s condition The information given about you/your child’s treatment If you/your child have any additional needs, were they dealt with appropriately

Comments: ………………………………………………………………………………………………… Section 3 Thinking about your child’s tests and treatment, please comment on the following? Excellent Good Fair Poor N/A The therapist’s ability to listen to me and my child The therapist’s ability to answer any questions The therapist’s ability to involve you or

your child in decisions about treatment The therapist’s overall attitude Comments ………………………………………………………………………………………………



Section 4: Thinking about further follow up care – how would you rate the following Excellent Good Fair Poor N/A Arranging your next appointment Information about who to call if you have a question or further comments Comments ……………………………………………………………………………………………… Section 5: Clinics normally run for 9am to 4pm would you like out of hours clinics? Yes No

Before 9am After 5pm Saturday Sunday If Yes (please tick all applicable) We value what you have to say. It is an opportunity for us to learn and improve our service. Please write down any other feedback ………………………………………………………………………………………………………………………………………………………………………………………….............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................



Patient Satisfaction Survey (revised version)

Dear Patient/Parent/Carer, We would be grateful if you would complete this questionnaire about your experience TODAY of visiting the therapy department and the treatment received. It should only take you about 5 minutes to complete.

If you do not want to answer a question then please leave it blank. No reason is required. Your responses are anonymous and will be kept confidential.

Thank you

After completing the questionnaire please drop it in the box provided at reception

Please tick the appropriate answer and use the space given if you want to comment further. Section 1 (1) How soon after your stated appointment time were you seen today?

Tick one box only

Was it within: 0 – 10 minutes ?

11 – 20 minutes ?

21 – 30 minutes ?

More than 30 minutes ?

(2) If the clinic was running late were you given an apology or explanation?

Yes

No

Clinic wasn’t running late

(3) How clean was the therapy room?

Very clean

Clean enough

A little grubby

Very grubby

Any comments on the level of cleanliness today?

_______________________________________________________________________________

_______________________________________________________________________________



Section 2: Tick one box only

(4) At your appointment, did the therapist introduce him/herself? Yes

No

(5) If another health care professional or student was present did the therapist Yes

introduce him or her? No

No-one else was present today

(6) Did the therapist explain the purpose of your visit today?

Yes

No

(7) Thinking about your appointment with the therapist today, how would you rate the following?

(a) The information given about you / your child’s condition Excellent

Good

Fair

Poor

Not Applicable

(b) The information given about you / your child’s treatment Excellent

Good

Fair

Poor

Not Applicable

(c) If you / your child have any additional needs were they dealt with

appropriately? Yes

No

Not Applicable

Any comments on these aspects of your session today with the therapist?

_______________________________________________________________________________



Section 3 Tick one box only

(8) Thinking about your child’s tests and treatment, please comment on the following?

(a) The therapist’s ability to listen to me was: Excellent

Good

Fair

Poor

Not Applicable

(b) The therapist’s ability to answer any questions was: Excellent

Good

Fair

Poor

Not Applicable

(c) The therapist’s ability to involve you or your child in decisions Excellent

about treatment was: Good

Fair

Poor

Not Applicable

(d) The therapist’s overall attitude was: Excellent

Good

Fair

Poor

Not Applicable

Any comments on these aspects of your session today with the therapist?

_______________________________________________________________________________

_______________________________________________________________________________



Section 4: Tick one box only

(9) Thinking about further follow up care – how would you rate the following

(a) Arranging your next appointment Excellent

Good

Fair

Poor

Not Applicable

(b) Any information given about who to call if you have a question or Excellent

further comments Good

Fair

Poor

Not Applicable

Any further comments on these aspects?

_______________________________________________________________________________

_______________________________________________________________________________

Section 5:

Tick all that apply (10) Clinics normally run from 9 am to 4 pm

Only complete this section if you would prefer to attend outside these hours

(a) Would you prefer to attend out of hours, such as Mon-Fri, before 9 am

Mon-Fri, after 5 pm

On a Saturday

On a Sunday

Your views are important so please give us your honest feedback, good or bad, and

we will use them to improve our service for you and other patients.

_______________________________________________________________________________

_______________________________________________________________________________

How To Avoid Pitfalls When Designing A Questionnaire - ARTP

Documents

Transcript of How To Avoid Pitfalls When Designing A Questionnaire - ARTP