1

Report

Advance letters in web surveys; qualitative findings to interpret quantitative results

Annemieke Luiten Jelmer de Groot

project number 210811 – ESSnet DCSS Process development, IT and Methodology

28 Augustus 2014

summary The Dutch Health Survey is a mixed mode survey, with web as the first mode. Nonrespondents of the web round are followed up with other modes. Sample persons receive an advance letter with login codes and two reminder letters. In 2013 the three letters were redesigned and tested in a split halve pilot. It appeared that the revised letters generated substantially less web response. The original and the revised letter were subsequently tested in qualitative research, and underwent text analysis. The findings suggested that the text complexity, the length and lay out of the revised letter should be combined with substantive elements from the original letter. The report ends with recommendations for a new reversion, and further pilots.

keywords Advance letter; reminders; qualitative research; pilot; text analysis

name adress postcode city

our reference

re CBS research

CBS Heerlen

Postbus 4481

6401 CZ Heerlen

+31 45 570 60 00

www.cbs.nl

1. Introduction

The importance of advance letters to obtain cooperation from sample persons is widely known (e.g., De Leeuw, Callegaro, Hox, Korendijk, & Lensvelt-Mulders, 2007; Dillman, 2000; Goldstein & Jennings, 2002; Luiten 2011; Yammarino, Skinner & Childers, 1991). In cases where the letter is the only contact between the statistical office and the respondent, as is the case in web or paper questionnaires, the letter is one of the most important factors in convincing the reader to participate in the survey. It is therefore evident that the letter should be as good and convincing as possible. One of the recommendations in the advance letter literature is that letters should be concise and should fit on one page (Dillman, 2000). That recommendation may be at odds with the information a respondent needs to be able to access a web survey. Apart from the information that is also given in a CATI or CAPI survey (what is the survey about, why is it important to cooperate, why is the respondent chosen, etc. ), additional information needs to be given on the login procedure. The information concerning the website, security, and how to log on must be clearly explained, in order not to lose people with lower computer skills. Statistics Netherlands uses web data collection in most social surveys, with web as the first mode. That means that all sample persons are invited to fill in the web questionnaire. Persons receive an invitation letter, and, if necessary, two reminders, both in the form of letters. When we started with web data collection, we used the original advance letters for CAPI or CATI, and added the information necessary to log on. The result was that the letters became quite long, one and a halve page, and had a cluttered type page. We decided that a redesign of the letters was in order. A project group of methodologists with knowledge of the advance letter literature, and field management practitioners added force with a commercial communications bureau that had in the past successfully redesigned the letters for business surveys. The advance letters and both reminders were redesigned. The first reminder was changed from a letter to an attractive and notable card. The new letters and reminders were piloted during two months in the Health Survey, in a split halve experiment.

1

To our utter surprise and dismay, the redesigned letters were substantially less successful in eliciting cooperation in the web survey: where with the old letter a response rate of 35% was attained, with the new letter this was 10 percentage points less. However, this clear result offered an opportunity to get information on what had been instrumental in securing both the higher and lower results. To this end, an number of qualitative interviews were held, where respondents were asked questions on both letters. In this report, we first give a short overview of the theory of advance letters and their influence on survey participation. In paragraph 2.1 we describe the letters and reminders, in paragraph 2.2 the qualitative research. Paragraph 3.1 describes the results of the qualitative research. In paragraph 3.2 both letters are textually analysed, in analogy to research by Fazekas, Wall and Krouwel (2013), to determine the linguistic complexity of the letters, the kind of appeal (altruistic versus egoistic) and the tone (formal or friendly). We end in paragraph 4 with recommendations for a revision of the letters and a new pilot.

The theory behind advance letters

In CAPI and CATI surveys there are multiple moments of contact between the survey organisation and the respondent. The letter plays in important part in this process, but it is the interviewer who has the more important influence on the final decision to participate. In web surveys, the letter is the only contact. It should induce the respondent to not only answer questions, but also actively perform a series of acts to make answering these questions possible, like going to his computer, starting it up, accessing the website, entering access codes and password. Once the questionnaire is started, the majority of people finish it and send it in. That means that the relatively low response rates in web surveys are not the result of the questionnaire, but are a resultant of the prior process. The letter, therefore, must be very inviting, in order to seduce the respondent to perform all these tasks.

Dedication and motivation

There are differences between people in the amount that they need to be convinced to participate. Dedicated citizens will hardly need convincing, while others refuse to cooperate (Lee & Rao, 2009). The Elaboration Likelihood Model (Petty & Cacioppo, 1986) describes that people can use two ‘paths’ to decide whether or not to participate. The choice of path is influenced by the level of dedication. Dedicated people follow a ‘central’ path, in which arguments are weighed explicitly. These arguments include the value for society, science or the respondent himself, the trustworthiness of the institute, the security of the answers and the estimated pleasure that filling in the questions can provide. Less dedicated people more often follow a ‘peripheral path’, in which more implicit considerations play a role, like whether or not they like the interviewer, their general sense of social duty, their attitude towards surveys in general and what use it is for the respondent.

2

The central route focuses on the contents of the survey and the arguments that are given in the letter (the ‘what’). The peripheral route is triggered by the ‘how’; how the message is delivered. A long letter would not invite reading for people following this route, especially not if it includes disclaimers and paragraphs on informed consent.

The level of dedication is influenced by the subject of the survey (Krosnick, 1991). Illustrative is the variance in web response rates in various Statistics Netherlands’ surveys with the same design: from 20% for the mobility survey to 40% for Youth and Education, and almost 50% for Social Cohesion.

Another theory of participation is the Social Exchange theory (Dillman, 2007). This theory poses that respondents weigh the costs and benefits of participation. Involvement in the subject, or dedication would be a benefit. Not all respondents are that involved in the subject, however. Perceived benefits can be influenced by giving an incentive. Lacking that, the benefit in web surveys is dependent only on the argumentation in the letter.

Appeal, complexity and tone

Involvement is one of the dimensions that were varied in research by Fazekas, Wall en Krouwel (2014). Participants had earlier filled in an online questionnaire to aid them in determining which political party to vote for. Some of these participants, who had indicated that they would be willing to participate in further research, were sent an email invitation for a new survey. The invitations varied on three dimensions: appeal (selfish versus altruistic), linguistic complexity (simple versus complex) and tone (formal versus friendly). These dimensions were combined to form eight different invitations. The ‘selfish’ version focussed on the satisfaction of being able to give one’s opinion, and how important that opinion was. This was operationalized by frequent use of ‘you’ ‘your’ and ‘opinion’. The altruistic version stressed reciprocity, and contributing to society and science. This was operationalized by words like ‘contribution’, ‘contribute’ and ‘help’. Fazekas et al., (2014) assumed that respondents would think that taking part in the survey would be difficult if the invitation was linguistically complex, and vice versa, and that this assumption would be reflected in response rates. Simple invitations had words with few syllables, and a simple sentence structure without multiple clauses. Complex invitations had longer words and complex sentence structure. Caution was taken that the simple version was not infantile and the complex version not incomprehensible. The third dimension varied ‘tone’, and manipulated formality versus friendliness. It was supposed that a formal letter would suggest legitimacy and authority, while a friendly letter would suggest positivity and trust. The dimension was operationalized by the use of ‘je’ instead of ‘U’, an informal address ‘hi, I am André Krouwel, the director of Kieskompas’, and informal ending ‘kind regards’. The formal version started with the message of the letter, and ended with ‘yours sincerely, A.P.M. Krouwel, director of Kieskompas’. Part of the people that were invited to participate had done so on other occasions. The results showed that the effect of the manipulation was largest for people who were invited for the first time, and got smaller, the more often they had participated already.

3

Mean response rate for people who were invited for the first time was 43%. The highest response was attained with an altruistic - simple - informal letter (54%), although the altruistic - simple - formal letter (46%), the selfish – simple - formal letter (49%) and the selfish – simple - informal letter (48%) did not differ significantly from the letter with the highest response. The lowest response was attained with the selfish – complex - informal letter (25%), significantly different from all other versions. When people had participated one earlier time, the mean response was 26%. The lowest response was attained again with the selfish - complex – informal letter (15%), although this time this rate did not differ from the selfish – complex – formal letter (21%). The six other letters had a significantly higher response, but did not differ from each other. People who were invited for the third time had a mean response of 6%. Lowest response was attained by three of the four selfish letters (SCF – SCI – SSI), the highest by the altruistic – simple – formal letter (10%). Over all dimensions and groups, altruistic letters were more successful than selfish letters (27% vs 23%), and simple letters better than complex ones (29% vs 21%). The difference between formal and informal letters was not significant (26% vs 25%). In view of the importance of these dimensions, we scored both the original and the redesigned Health Survey letters on these dimensions, using the methodology of Fazekas et al., (2013).

2. Method

2.1 Redesign of the invitation letter and reminders In redesigning the invitation letter, a number of demands were formulated, based on the literature and knowledge of best practices within CBS and other bureaus. • The text should fit on one page • The survey subject must be described appealingly • The person should feel that it would benefit him to participate • The person should feel special (is selected among countless others) and is representing a

large part of the population • The letter should mention that an interviewer would call or visit if the persons does not fill

in the web questionnaire. Jansen (2009) showed that this phrase is instrumental in securing two percentage points higher response rate.

• Directions to the website, user name and password should be clear, but not overly wordy. • The linguistic level of the letter should match European recommendations. • The letter should be structured better. The original reminder letters were like the original invitation letter: long, relatively unstructured, linguistically complex letters. In contrast to what is suggested in literature, they do not contain additional information on the necessity to cooperate. For the redesign, we chose to radically change the first reminder. Instead of a letter, we used an appealing postcard, designed to stand out in the mail. Because of privacy reasons, the card did not contain login

4

information. As the reminder is sent one week after the invitation, we presumed that the original letter would still be available, and that persons merely needed prompting. The card had a short message, and reiterated the information of the importance of participation. After the first reminder was sent in the pilot, however, we found out that the effect of the reminder card was far less than that of the traditional reminder letter, and we developed a reminder letter that contained the login information. The second reminder was a letter, with a different message from the invitation letter. The fact that an interviewer would call shortly if they did not, figured prominently in the letter. Appendix 1 shows the original and redesigned invitation letter and reminders. The letters were developed to our specifications by a commercial communications bureau.

2.2 Pilot The new letters were tested in a split-half design, during two months. In each month, half of a sample of 1917 sample units of the Health Survey received the original letter and reminders (the control group), while the other half received the redesigned letter and reminders (the pilot group). In the first month, the experimental postcard was used, in the second month the redesigned first reminder letter. The Health survey is a person sample. Persons were sampled from the communal registries and were sent an invitation letter, addressed by name. The fieldwork period for the web wave is one month. One week after the invitation, a reminder was sent to nonrespondents, and one week after that a second reminder. In the second month of fieldwork the non-response follow up in other modes starts.

2.3 Qualitative research 14 respondents took part in the qualitative research, five women and nine men. Four had a lower secondary vocational education, eight a higher vocational education, and two were still in high school. In addition, two university students who specialise in the subject of advance letters commented on the letters. Their remarks are included in the findings. The respondents were relatively young: eight respondents were younger than 25, four were in the 25 to 40 age category, and three were older. This somewhat skewed distribution may have influenced the findings. Respondents were shown both letters. About half first received the redesigned (short) letter, the other half first the original (long) letter. One of the respondents only discussed the short letter. Respondents were asked:

- their first impression of the letter, - whether they would participate, and why (not), - whether the letter makes clear what is expected of them, - which aspects of the letter were appealing, - which aspects were not appealing, - their opinion on the passage on respondent security, - whether there are things about the survey that they would like to know, but were not in

the letter,

5

- ideas on improvement - comments on the way the letter was signed

In the short letter, where far less information was provided on the way to log on to the website, we asked respondents to access the website, and observed where they typed in the web address, and if they made typing mistakes.

6

3. Results

3.1 Web experiment Table 1 shows response results of the control and pilot group, overall and for a small set of subgroups. The overall response rate in the pilot group was over 10 percentage points lower than in the control group, a significant difference according to two sided χ2 test. The lower response rate is visible in each of the subgroups, with the exception of the group of children younger than 12 years old. The response differences in some groups are even larger than the overall difference. Especially the large difference in the group of non-Dutch residents and the 56 to 64 age group stands out. In addition, it appears that response is especially lower in more rural areas, and in groups with higher incomes. However, multivariate logistic regression analysis shows that there is no significant interaction between letter and auxiliary variables. Table 1. Response results in control and pilot group, overall and in subgroups

Figure 1 shows the day by day response in both conditions in the two fieldwork periods. In figure 1 these are indicated as 1306_reg and 1307_reg (control group) and 1306_exp and 1307_exp (pilot group). It is obvious that the different response rates between the letters were apparent from day 1, and increase as the fieldwork advances. In the first pilot month a postcard

n % n % signDutch 1837 31,5 1829 20,9 ***Other 81 21,0 87 6,9 *<12 291 29,9 249 29,3 ns13-16 91 37,4 82 19,5 *17-25 221 27,1 195 13,3 **26-35 231 27,3 243 17,7 *36-55 531 32,4 573 21,5 ***56-64 216 40,7 213 23,0 ***65+ 337 27,0 361 16,1 ***M 934 30,9 951 19,0 ***F 984 31,1 965 21,5 ***Highly urban 360 23,3 356 15,4 **urban 433 28,6 424 21,0 *moderately urban 347 33,1 348 23,9 *moderately rural 367 36,5 400 20,5 ***highly rural 369 33,9 348 18,7 ***lowest quartile 355 21,7 372 12,9 **2nd quartile 448 25,9 424 15,8 ***3rd quartile 533 34,1 523 23,3 ***highest quartile 507 39,4 528 24,2 ***

Overall response 31,0 20,3 **** p < .05; ** p < .01; *** p < .001

Urbanicity

Income

Nationality

Age

Gender

Control Pilot

7

reminder was used. The figure shows that the postcard did not lead to substantial increase in response, in contrast to the increase after the traditional reminder. After the second reminder, however, the increase was relatively large. We think that this may indicate that a number of respondents had wanted to respond after the first reminder, but could not, because the card did not contain login information. With the second reminder they received the login code anew, after which a relatively large number of people participated. This pattern is a strong indication that a number of people did not keep or perhaps did not see the original letter.

Figure 1. Reponse per fieldwork day for control and pilot groups

3.2 Qualitative analysis

Respondents’ remarks were summarized and were additionally coded as having a positive or negative connotation. Summaries were made for the subjects length, layout, language, tone, introduction of the survey, importance of the survey, selection of the respondent, introduction of the interviewer, access to the website and security. Table 2 shows the number of positive and negative remarks about the letters. It is remarkable that substantially more positive remarks were made about the revised letter, and more negative remarks about original letter. That contrast was especially apparent in remarks about the length and layout of the letters. Excluding these, the revised letter still evoques more positive and less negative remarks (41 positive remarks and 25 negative for the short letter, versus 30 positive and 25 negative for the long one). After reading each letter, respondents were asked whether they would participate to the survey. 45% of respondents would with the revised letter, and 33% with the original letter. As the research question was why the short letter was less successful than the long letter, we went looking for elements where this short = positive and long = negative pattern was broken. There were a number of subjects where this was the case. These cells are coloured in table 2:

0,0

5,0

10,0

15,0

20,0

25,0

30,0

35,0

40,0

1 2 3 4 5 6 7 8 9 101112131415161718192021222324

Resp

onse

Fieldwork day

1306_reg

1306_exp

1307_exp

1307_reg

8

short long short long Length 9 0 0 9Layout 13 0 1 1Language 5 1 4 0Tone 11 8 1 9Introduction survey 7 8 3 2Importance survey 6 1 3 0Selection respondent 0 4 3 3Introduction interviewer 1 4 4 5Access website 1 2 0 2Security 5 2 1 6Other 4 1 3 1Total 63 30 26 41

Positive Negative

yellow if the long letter was judged relatively positively, and green if the short letter was judged relatively poorly. Table 2. Number of remarks with positive and negative connotation by letter and subject Table 3 gives a short description of the most prominent results. The text in red is informative for understanding the lesser result of the short letter. From this analysis it appears that: - The simple and direct language is appreciated, but some respondents had associations

with advertisements. This is an association that needs to be avoided by NSIs. - The structure of the short letter is appealing, and makes that respondents can scan the

letter in seconds. On the other hand, that can induce them to read only the headings, and not all our carefully formulated arguments to participate.

- The introduction in the long letter is more personal and clearer. It is recommended to rewrite the old introduction with the new tone of language.

- It should be made clearer what the importance of the study is for the respondent. The respondents felt that the importance for science should not be stated before that of the respondent (but, see Fazekas et al., 2014, for a competing view).

- A large number of respondents remarked that they missed a phrase on how they are selected, and how special they are.

- The phrase that an interviewer will call or visit is a valuable addition. A substantial number of respondents were irritated by the phrase, but that does not need to mean that it is bad for response to include it. An almost equal amount of respondents appreciate the remark, either as a reassurance for people who don’t have access to the internet, or who favour personal contact.

- Two respondents were of the opinion that the request for participation came too early in the letter, before the passage why the survey is important. Four respondents were observed while they typed in the web address. None typed the address in the wrong place (like in a search bar), but all made more or less significant typing errors, like omitting the https:// part of the address. An important recommendation is to use comprehensible and meaningful names in the web address, instead of a code that is meaningless to respondents (e.g., HEALTH instead of GEZO14B), and to redirect respondents to the right URL, even if they make specific errors use shortcuts. Examples

9

that can be redirected are typing ‘http’ or even ‘www’ instead of ‘https’, or omitting the ‘https://’ part altogether. Table 3. Summary of findings of qualitative research.

Length Overwhelming support for the short letter (and rejection of long letter). Layout Overwhelming support for layout with bold headings and clear build up. Language Simple, direct, clear language of the short letter is appreciated. BUT 3 respondents have

the association with advertisements. Tone Short letter is deemed inviting (3), concise (2), polite, reliable. 2 respondents find that

the long letter is more friendly, 2 like the usage of 'I', 2 like the usage of 'you would do us a large favour'. On the other hand, 9 respondents dislike the same utterances for various reasons.

Introduction survey The relatively many positive remarks for the long letter were made by respondents who first saw the short letter. They found the introduction more personal, with more explanation, clearer, more involving.

Importance survey Although the positive remarks of the short letter stress that the importance is addressed such that it is made very personal, the relatively many negative remarks stress that it is not sufficiently made clear why it is important for the respondent, to not mention the importance for science etc before that of the respondent, and that, in contrary to the long letter, it does not say that participation is important

Selection respondent The positive remarks on the long letter, and the negative remarks on the short one handle one subject: the long letter mentions how the respondent is sampled, and that makes him special. As one respondent says: It does not say that I am selected, it seems that everybody receives this letter, it does not matter if I participate or not

Introduction interviewer The long letter mentions that an interviewer will call or visit if people don't have access to the web, or do not respond. The 5 respondents with negative reactions to the passage mention this aspect: they all find it nagging. The 4 respondents with positive remarks on this part in the long letter, find the passage positive for elderly people, those without internet, and also as a stick behind the door. The negative remarks in the short letter also mention that people without internet will not know what to do.

Access website 2 respondents mention that saying not to type in the address space is good. Others like the clarity of the instructions in the short letter

Security Respondents like the concise mention of 'it is in the law' in the short letter. They feel that there is too much stress on all kinds of issues, which may introduce feelings of insecurity. Several mention that the frase 'we use a secure connection' is not maintainable. On the other hand, 2 respondents feel that more attention should be given to security in the short letter.

Other 2 respondents mention that they would not participate if they had bad health habits, or were not healthy. 1 respondent says that the word 'health' is used too often in the introduction.

10

3.3 Analysis of the dimensions Appeal, Complexity and Tone

With the methods described in Fazekas et al., (2014), both letters were rated on the dimensions Appeal (selfish – altruistic), Complexity (simple - complex) and Tone (formal– friendly).

Appeal The number of ‘selfish’ utterances (‘you’, ‘your’ ‘opinion’) in the old letter is 29, in the new letter it is 17. In both letter there is only one altruistic utterance. That means that both letters are extremely ‘selfish’. The research by Fazekas et al., (2014) showed that an altruistic approach led to a higher response rate (at least in the subpopulation of respondents that were interested in politics and volunteered for follow-up research), so this may suggest additional venues for improvement of the letter.

Complexity According to the Common European Framework of Reference (CEFR), a directive from the Council of Europe, 60% of the Dutch population has the language level B1. This then should be the level of communication from the government, in order to be understood by the public. In the case of the invitation letter, if the letter is too complex, the respondent may fail to understand what we want of him (i.e., filling in a questionnaire on the web), or deduce that the task will be complex, and not respond. The level of the letters was checked with a tool available on the internet1. According to this tool, the level of the old letter is B2, and of the new letter B1. The old letter is thus too complex for a substantial part of the Dutch population. This could partly be responsible for the over representation of people with higher education in the web response of Statistics Netherlands’ surveys (Luiten, 2013).

Tone The way the sample persons were addressed and the way the letters are signed was the same in both letters. Both are formal, according to the criteria of Fazekas et al., (2013).

3.4 Recommendations

These findings led to the following recommendations: 1. The linguistic complexity of the letter should conform to European directives, and not be

more complex than B1 level. 2. However, do not make the letters glib. Avoid connotations with advertisements. 3. The length of the new letter is much appreciated, this should be the target length.

1 https://www.accessibility.nl/kennisbank/tools/leesniveau-tool

11

4. Login codes and passwords with numerous digits and / or letters should be presented clustered, to preclude errors in typing.

5. Use persuading and at the very least normal language for the web address. 6. Most letters have a ‘subject’ line in the heading. Use it to describe the survey in positive

and appealing terms. 7. Make people feel special: they are specially selected. Don’t mention that a large sample is

drawn, but use the opposite argument. 8. Do usability testing to make sure that the login procedure is completely clear, also for less

informed internet users. 9. Iterate the login code(s) in all reminders. Some recommendations seem plausible, but we do not yet know if they would be effective. We recommend experiments to find out: 1. If headings prevent people from reading the letter, and hence from being subject to our

persuasion techniques. Pilot with a letter with the same contents, with and without headings.

2. If we should avoid words like ‘survey’ or ‘research’ in the subject line. They too might induce respondents to discard the letter without reading it.

3. What should be the balance between altruistic and selfish persuasion remarks. 4. Where to put the request for cooperation.

12

4. Summary and discussion

This research started because the invitation letters of the Statistics Netherlands surveys needed to be adapted to the new situation, where web data collection is the first mode in a mixed mode design. Also, European directives on the complexity of government communications showed that the original letters were too difficult for a large part of the population. A working group of researchers and practitioners, aided by a communications bureau, were convinced that they had developed a set of substantially improved invitation letters and reminders. However, we were proved wrong in a pilot, where the new letters and reminders generated substantially less response than the old letters and reminders. This experience was used to find out through qualitative research what were the strong features in the old letters, and how they could be used in the necessary redesign. Both the advance letter and the reminder procedure were less successful in persuading persons to participate. The idea to visually distinguish the first reminder from the letter by sending an eye catching postcard hardly led to higher response, while the regular reminder in the form of a letter always leads to a substantial increase. After the second reminder, the people who had first received the postcard showed a relatively large increase in response. We interpret this as an indication that part of the respondents had wanted to participate after the first reminder, but had lost the advance letter, and, with that, the login code. An important conclusion is that it is imperative to repeat the login codes in each communication with the respondent. To understand why the new advance letter was less successful, we did qualitative research with 14 persons, and we performed a text analysis. It was remarkable that both the qualitative respondents and the text analysis agreed with the working group that redesigned the letters: the test respondents were far more positive about the new letter, and the text analysis showed that the new letter was better suited to the language level of the majority of the Dutch. There were some aspects, however, where the test respondents were more positive about the original letter. These aspects proved to be informative to understanding the success elements in a letter. The challenge is to combine the best elements in the two letters: the simple language, the length and layout of the new letter should be used, while we must take care to underline the importance of the survey, and the respondents’ role therein. We also need to stress the altruistic part of cooperating in research. We must take care not to make the letters too ‘popular’ and flashy. We need to preclude association with commercials. A relevant question in this research, and in future research into advance letters, is whether we should have started with qualitative research in the first place. We suspect, however, that the research group would have gone ahead with the new letters, even knowing the qualitative results beforehand. The opinion of the qualitative respondents was more positive about the new letters, and more indicated that they would have participated with the survey. A large problem in the interpretation of these qualitative results, is that the respondents hardly agree on anything. That makes the conclusions in this paper tentative at most: there were always respondents who had the opposite opinion. A second argument is that what respondents like is

13

not always good for response. An example is mentioning that people will be contacted by an interviewer if they do not participate by web. Respondents indicate that they find that pushy, but prior research showed that the phrase generates a higher web response. The order used here, quantitative before qualitative research, is probably more informative in these circumstances. The hypotheses that follow from the qualitative research will need to be tested in new rounds of quantitative research. Only by successive rounds of testing can the optimal letter be developed. It is further well possible that we should develop more than one letter to cater to the needs of various groups of people. The qualitative research hinted that preference for the one or the other letter was age dependent, where the younger respondents favoured the new letter, and the older respondents the traditional letter. The quantitative analysis did not support this tendency, however. We suggest to test new versions of letters in small scale pilots. Even with few respondents it is possible to determine if a letter performs well. Only when it becomes clear that there will be no calamities in the field, larger scale pilots can be performed, in which analyses by subgroups becomes possible. To really come to grips with the success elements in letters, we need to perform a series of experiments in which we change one element at the time.

14

5. References

De Groot, J. (2013). Fehler! Verweisquelle konnte nicht gefunden werden.. (Measures to heighten CAWI response: redesign of advance letters in Dutch) Internal CBS report.

De Leeuw, E., Callegaro, M, Hox, J., Korendijk, E., & Lensvelt-Mulders, G. (2007). The influence of advance letters on response in telephone surveys. Public Opinion Quarterly, 71, 413-443.

Dillman, D.A. (2000). Mail and internet surveys: The tailored design method (2nd ed.). New York, NY: Wiley.

Fazekas, Z., Wall, M.T., and Krouwel, A. (2014). It is what you say, or how you say it? An experimental analysis of the effects of invitation wording for online. International Journal of Public Opinion Research, 26, 1-10.

Goldstein, K.M., & Jennings, M.K. (2002). The effect of advance letters on cooperation in a list sample telephone survey. Public Opinion Quarterly, 66, 608–617.

Jansen. B., Schrooten, M. & Wetzels, W. (2009). Mixed-mode experiment Veiligheidsmonitor Rijk 2007 (Mixed mode experiment Safety Monitor 2006 In Dutch). Internal CBS report.

Lee, G., & Rao, K. (2009). Examining the Relationship Between Survey Response Elicitation Efforts, Respondent Motivation, and Satisficing: A Case Study of Web-based Panel Survey. Paper presented at the American Association for Public Opinion Research, Hollywood, Florida.

Luiten, A. (2011). Personalisation of advance letters does not always improve response rates. Demographic correlates in a large scale experiment. Survey Research Methods, 5, 11–20.

Luiten, A. (2013). Improving survey fieldwork with paradata. PhD thesis, Utrecht University. Kockelkoren, S. (2012). Aanschrijfbrieven en rappelbrieven voor sociale statistieken

http://intranet/PIM/PPM/Methodenreeks/Verzamelen%20van%20gegevens/MR_Aanschrijfbrieven1%200.pdf

Krosnick, J. (1991). Response Strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology 5 (3): 213–236. doi:10.1002/acp.2350050305

Petty, R. E., & Cacioppo, J. T. (1986). Communication and persuasion: Central and peripheral routes to attitude change. New York: Springer-Verlag.

Yammarino, F.J., Skinner, S.J., & Childers, T.L. (1991). Understanding mail survey response behavior. A meta-analysis. Public Opinion Quarterly, 55, 613–639.

15

6. Appendix 1

6.1 Original advance letter Dear Mrs / Mr (name),

An enjoyable life in a pleasant environment and good health are very important for most people. To gain more insight into how people live in the Netherlands, how their health is and how they experience and appreciate the health care, the Central Bureau of Statistics (CBS) carries out the Health Survey. Of course, not all residents of the Netherlands can be questioned. Therefore CBS draws a large number of names from the population every month. The following name has thereby emerged:

Name

The results of the health survey are used by different ministries. As a result, the policy on, for example, health and welfare can be tailored to your needs and those of the Dutch population. Your participation is therefore very important: you represent, as it were, many other residents of the Netherlands.

I am aware that not everyone has internet. Therefore, an employee of CBS will visit or call a few weeks, if you have not yet completed the questionnaire. If you have internet access, you can download the questionnaire at the following Internet address:

https:\\vragenlijst.cbs.nl\GEZO14B

To protect your data against abuse, we use a secure connection. You must therefore not type the usual "http" but "https." It is important that you do this at the top of your screen in the address bar. Typing into Google or another search engine does not work. If you're on the CBS website, you are prompted for a user ID and a password to complete.

Your user number is: Your password is:

(back of the page starts here)

After entering your user number and your password you enter the questionnaire. In the survey your privacy is guaranteed. We make statistics by combining your data with those of other people surveyed and with data from other sources. On the back of this letter you can read more about this.

If you have questions or problems associated with the use of the Internet, you can contact the helpdesk of our IT department. Email: contactcenter@cbs.nl stating "Gezo." Should you have

16

any questions regarding this letter or the survey please call the Contact Centre of CBS Heerlen: (045) 570 64 00. The Contact Centre is available Monday to Friday between 9:00 and 17:00.

You would do me a big favour if you fill in the web questionnaire one of the next few days. I thank you in advance for your cooperation. With kind regards, Manager Data Collection (signature and name) In all our studies your privacy is assured. This is an obligation of CBS, stipulated in a special law. To protect your data, the CBS takes numerous measures. There is a strict confidentiality for all employees, under penalty of prosecution. Personal data are separated as soon as possible of the names and addresses. The data is processed with highly secure computer systems that preclude unauthorized access. The law guarantees that your information will be used only for statistical purposes. No institution can claim access to the data collected by the CBS. In the statistical information that CBS publishes personal information can never be recognized. CBS does not only collect data themselves, but also gets a lot of registry information from other institutions. For example, the data of the population administrations, centres for Work and Income (CWI), social services, the salary records of many businesses. We automatically combine the information that you give in the survey with information we receive from other institutions. With this combined information CBS makes statistics about the Dutch society, while working as efficiently as possible.

17

6.2 Redesigned advance letter Dear Mrs./ Mr. (name) What does a pleasant and healthy life mean for you? CBS would like to know. Therefore, we invite you to participate in our research on healthy living. Do you want to fill in our online questionnaire?

Why is this research important for you? With this study, we learn more about the health and lifestyle of the Dutch, but also about their use of medication and doctor visits. This knowledge is important for scientists, doctors and administrators who work on the health of the Dutch. And that is important to you too.

What do we ask of you? Would you please fill in our online questionnaire? That works like this:

1 Go to https:\\vragenlijst.cbs.nl/GEZO14B 2 Enter: user number 0000 0000 000 and login code 0000000

Your data are safe. Your data will only be used for our study. That's the law. Once we receive your data, we disconnect them directly from your name and address. So you remain anonymous. By combining your data with other general information we obtain valuable information about the health of the Dutch. Do you want to know more? Look on the back of this letter.

Finally. We thank you for your time and cooperation. Do you have questions? Call us: phone (045) 5706400. We are available from Monday to Friday between 9 am and 17 pm. You can also email us: contactcenter@cbs.nl. We thank you for your time and cooperation. With kind regards, (name and signature) This text fits on one page. At the back of the letter the informed consent passage is printed, like in the original letter: In all our studies your privacy is assured. This is an obligation of CBS, stipulated in a special law. To protect your data, the CBS takes numerous measures. There is a strict confidentiality for all employees, under penalty of prosecution. Personal data are separated as soon as possible of the names and addresses. The data is processed with highly secure computer systems that preclude unauthorized access. The law guarantees that your information will be used only for statistical purposes. No institution can claim access to the data collected by the CBS. In the statistical information that CBS publishes personal information can never be recognized. CBS does not only collect data themselves, but also gets a lot of registry information from other institutions. For example, the data of the population administrations, centres for Work and Income (CWI), social services, the salary records of many businesses. We automatically combine the information that you give in the survey with information we receive from other institutions.

18

With this combined information CBS makes statistics about the Dutch society, while working as efficiently as possible.

19

6.3 Reminder card

[Front with logo and name CBS and appropriate picture]

Thanks in advance!

[Back side with logo, name CBS, adress]

Dear <Mr./Mrs. ><Name>,

Do you remember? Our online survey on healthy living?

That is important for all the Dutch. So don’t forget to fill in the questionnaire. The login

information is in the letter you received last week.

Thank you in advance,

(name)