Jountal of GaJ' 6 Lesblan Social Serl,lces, 22:371-39t3, 2010Copyright O'laybr & l'rancis Group, LLCISSN: 1053-8720 print / 1540-4056 onlineDOI: 10.1 080,/ 105 387 20.2010.4917 42

Recommendations for Performing

l) Routledqel\ rayror arrarris"croup

Inteffret-Based Reseafch on Sensitive SubiectMatter with'Hidden' or Dfficult-to-Reach

Populations

HUGH KLEINKefrslttgtofi Researcb ltrstttute, SllL'er Spring, Maryland; Morgatt State Unlue6lU/,

Baltimore. Marl'lan.l

THOMAS P. I-{MBINGMorgqn State Unite6i4), Baltimore, Marylawl

DAVID A. MOSKO\NTZMedical College of Wisconsin, Miluaufue, Wisconsln

T}IOMAS ALEX WASHINGTONCaliforr,ia State UnixersitJ-Iong Beacb, Long Beocb, C.tlifomla

LISA K. GILBERTAfierlcon Soclal Eeahb Assoclatton, Researcb hlangle Parh, Norlb Carolina

Since tbe mid-199os, tbe rapidly increasing populariu of tbe lfi-rcma bas contributed to a situation in wbicb man! men turn toWeb sites toJtnd sex panne6 uitb ubom they can engage in riskltbebauiors. Scbolarc only recently began to exatnine tbe role of theInternet in hann-seeking aru| belp-seeking bebauia8. TheJ) are justnou beginning to study and understand hou to appll public bealtbpromotion principles to Wple using tbe Intemet. Due in part to tbe,elatiue neuness of the Internet on tbe public bealtb landscape,scbolars wisbing to conduct researcb or to irnplement bealtb pro-motion proqr.lms online sbould consider a uari.ety of cballenges todoing sucb worh-cballenges tbat dtlfer from tbose typically faceduben undertaking similar uork in otber types of uenues ofrline.

Tbe purpose of tbis article is to address seueral of tbese researcbconsiderations. In particulor, tbe present autbots wisb to prouide re-searcbers and bealtb care specialists wilb kej considerations uhendeueloping tbeir own Intemet-based researcb or bealtb promotion

Address correspondence to Hugh Klein, Kensington Research Institute, 4Ol SchuylerRoad, Silver Spring, MD 20910. E-mail: hughk@aol.com

H. Klein et al.

programs. We also wisb to Jirrbisb readen; u,itb some qperience-based nrggestions about botu to ^uoid tbe potenrial pitfalh (l con-d.uctittq Intenxet-based stLtdies. MoreoL,er, our empbasis is ttn bowto deuelop such prcgrams u,ben tbq/ are targeting hard,to reacb or"bidden" populations alxd/or uben tbql deal uitb sensitiue subjectmatter. Reco m mendations pertaining to tbe planning, recruit ment,implementation, and eualuation stages oJ doing professional workonline are prcuided.

KEyIVORDS researcb methods, bidden populalions, bardJo-reacb populations, Intemet, online

INTRODUCTION

Research methods courses and survey research methodology taught in col-lege and graduate school often provide students with instrucrion regardingthe proper ways to design and implernent lesearch studies of various types(see, for example, Creswell, 2003; Patten, 2005). 'fypically, such courses in-clude educational units that focus on options f()r developing a scientificallyacceptable study design, selection ancl ranclornization of sttrdy participants,proper questionnaire construction, treining interviewers, good interuiewingtechniclues, strategies for building rapport with study participants, and sofonh. Only occasionally do research methods and surey research methodscourses include information about working with "hidden" or difficult-to-reachpopulations andlor about studying personal, sensitive subject matter. Whensuch instruction is not provided, people tend to nor have the knowledge andskills necessary to identify and work successftrlly with populations thar areresistant to research endeavors, or to talk with members of these populationsabout topics that are very personal in nature. This can be problematic, be-cause many "hidden" populations require specialized research techniques toaccess them, to gain their trust, and to foster their cooperation with researchendeavors.

Many research methods and survey research methods courses also tendto omit instruction specific to the use of the Internet to conduct human-subjecs research. In all likelihood, this is the result ofthe relative newness ofthe Internet, coupled with the even greater newness of the use of the lnternetas a vehicle for conducting social science- or public health-focused research.Increased interest in engaging in Internet-based research has followed thegrolth of the Internet closely in recent years. This is evident both in terms oflearning about people and their lives and in terrns of using the Internet as away of offering people educational content, prevention messages, behavioralinten'entions, and so fonh. Some authors have written about issues peninentto doing Internet-based research with humen subjects (Birnbaum, 2000; Cho

Research on SensititE Subiect Mtttter

& LaRose, 1999; Mustanski, 2001), but generally speaking, this is a newerfield of inquiry with relatively little having been written about it.

In this article, based on our own experiences conducting Intemet-basedhuman-subjects research, we offer readers some suggestions as they them-selves plan and/or undenake their own Internet-based research or healthpromotion studies. In particular, our focus is on of'fering some experience-based "DOs and DON'Ts" with regard to conducting Internet-based re-search on sensitive subject matter with 'hidden" or difficult-to-reach pop-ulations. Our aim is not to cover points traditionally made in "standard"research methods or survey research methods courses; but rather, we wishto provide information and suggestions that are specific to (1) the Internet,(2) working with 'hidden' or difficult-to-reach populations, and (3) captur-ing research data on sensitive subjects. In separate sections, we discuss ourrecommendations applicable to the planning, recruitment, and implementa-tion stages of undenaking an Internet-based research study. At the end ofthis article, in the Appendix, we provide readers with overviews of some ofthe Intemet-based proiects that we ourselves have conducted with "hidden"or difficult-to-reach populations, involving sensitive subject matter. Theseoverviews are provided so that readers can derive a better understanding ofthe basis for the recommendations contained in this article. Each overviewalso provides supporting evidence of the success of the project in question,

to bolster confidence in our suggestions.

RESEARCH CONSIDEMTIONS AND RECOMMENDATIONS:THE PLANNING STAGE

In this section, we focus on topics of relevance to the planning stages ofa new study that entails working with difficult-to-reach populations andlordifficult-to-study subiect matter. Here, we wish to address five specific pointsfor readers to consider: (1) establishing an official project'0fleb site, (2)

maximizing anonymity, (3) offering options for ways of collecting data, (4)

gauging and maximizing interviewers' comfon level with the subiect matter,and (5) strategies for dealing with multiple accounts or user names.

'W'e

discuss each of these in turn.Establ^bing an official project web$a to explain and promote a newly

underiaken project is almost essential nowadays, particulady with Internet-based research projects. Experience working on Tbe Bareback Proiect svg-gests that potential study participants appreciated having the abiliry to go tothe Intemet to read about the study. A substantial proponion of the people

taking pan in our studies (estimated at one-quarter to one-third of the panic-ipants in Tbe Bareback Projecl) specifically requested a link to our proiect'shome page prior to agreeing to participate in the study. We would expect

H. Klein et ql.

this proportion to incfease gradually in the years to come, as more and morepeople become comfortable with the use of the Internet for information-gathering purposes. Having an official project Veb site enabled people tofind out more about the study, and to do so at a time that was convenientfor them. Having such information available on the Interner allowed them tore-consult rhe information after an initial visit, and to share the Veb site linkwith family members and trusted friends who could offer their own inDutwith regard to the legirimacy of the study, express any concerns rh.,y rhim-selves might have regarding parricipating in the pro,ect, and so forth. Ourexperience suggests tliat additional benefits might be derived if rhe proiecr'Web

site can be subsumed within that of a larger company, research insti-tLlte, or university. (This suggestion is made in contrast to a recommendationto establish a specific, independently maintained Web address rhat is un-affi liated with any well-established, community-known company, researchinstitute, or university.) As long as they can be directed to the project,s Vebpage(s) easily and directly, potential study participents seemed to feel re-assured when they saw that our project was affiliated as part of a researchcenter that was university based. Elsewhere in the journal issue in whichthe present article is published, Gilbert, Peterson, and Scanlon provide in-formation about some layout- and design-related features that researchersmight wish to consicler as they develop their project-related Web sites. \Veencourage readers to heecl some of these findings and suggestions as well.

Maximizing potential shtdy pdrticipong' anonymlly is likely ro be anessential way of encouraging some of them to participate in an Internet_based study focusing on sensitive information. As a general rule, the moresensitive the information researchers plan to collect andlor the more difficultto reach the research population is, the more important it will be to offer re_spondents an opportunify to remain anonymous. Merely prornising completeconfidentiality will not be sufficient ro enrice some people ro participate ina study, particulady one involving very personal information. While work-ing on Tbe Bareback Project, for example, men were never asked for theirlast name when interviews were scheduled or when answering their stucly-related questions prior to making the decision to participate. Men were askedto give their first name, so they cotrld be addressed in some manner otherthan their online user name; and they were allowed to make up a name ifthey preferred to do that. That strategy worked fairly well most of the time.Fearing tliat their tele;rhone number might not be safeguarded adequately,however, several men refused to give a telephone number to reach them fortheir interviews. Many of these individuals prefered to call the research teamat a prearranged day and time to do the interviews, so rhat they themselvescould control the divulgence of a teleplione number. Some men specificallyasked if a feature such as Caller ID (which would enable the research team roidentily their telephone numbers) was being used, and agreed to panicipateonly witen they fbuncl out that it was not in use. Nowadays, with identity

Research on Sensitirc Subject Matter 375

theft being commonplace on the Intemet, people increasingly have reasonto be cautious with regard to the disclosure of their personal informationto strangers-such as researchers-who approach them via the Internet. Itis essential to develop ways that people can panicipate in research studieswhile feeling confident that their personal information will remain privateand protected. Consequently, the less information that researchers collectabout thek study participants' identities, the better, at least from the point ofview of enticing them to pafticipate.

Coinciding with this suggestion, we encourage researchers who arethinking about doing Inlemet-based research to consider offeing potentialstud! panicipants uariotrs options of uays in ubicb tbq) can participate.Based on our experiences, we believe that researchers would be wise tooffer respondents several ways of participating in their studies, wheneverthis is feasible within the scientific constraints of the study design and withdeference to the study's validity. Most of the men with whom we have dealt(perhaps half to two-thirds of those participating in Tbe Bareback Project)were comfortable with the idea of completing a telephone interview (or atleast willing to do so). But a sizable minority of the men would not entertainthis option, leading some to decline to participate in the study. Several of themen who had been invited to take part in the project responded by askingif they could complete the quesrionnaire via e-mail, and others wanted toknow if they could take the survey online, via an Intemet survey engine(e.g., surveymonkey.com). In our opinion, future proiects would be well-advised to give serious consideration to this option, when it is feasible (e.g.,for a brief questionnaire that takes, at most, 15 minutes to complete). As thelength of the interview increases, the chances diminish that people who aregiven an option to complete a lengthy questionnaire online will completeall of it.

As with any research study, each option pursued carries with it a setof advantages and disadvantages to the researchers and study participantsalike. Any time study participants can be given the option of completingan interview via telephone, in person, via e-mail, or via the Internet (or byas many of these means as can be workable for the research team and therequirements of the study design), the likelihood rhar they will participarewill increase. From a research methods perspective, this poses numerouschallenges to the research team. If different methods of collecting data areused in one panicular study, it will become incumbent upon the researchteam to examine the data obtained by each method to ensure comparabilify,assess self-selection bias, and so forth.

In planning new research studies and proiects, we also advise re-searchers to consider tbe comfort lewls that their interviewers, outreachworkers, and recruiters have with the target population and with the subiectmatter being studied. In addition, the issue should be raised of bow knowl-edgeable their inten)i.euers, outreacb uorkers, and recruitets are of the target

376 H. Klein et al.

population and its members' beliefs, attitudes, normative expectations, andbehavioral practices. l,J(orking with "hidden" or difficult-to-reach populationsrequires deft handling. People participating in research studies nant, and inmany cases need, to know that the research staff members with whom theyare interacting understand them, understand their lives and their life issues,understand their concerns. and so fonh.

When undertaking work on Tbe Bareback prcject, for example, theseissues came to the fbrefront immediately. For example: Could a female inter-viewer be effective at engaging the men and making thern feel cornfonablein disclosing the intimate details of their sexual behaviors? Vould it help inany way if the woman were lesbian, so as to be familiar, at a minimun, withsome of thc sexual orientation-related aspects of interviewing men lookingfor other men online? Could a heterosexual man be effective at these tasks?rvould a female (heterosexual or homosexual) or a heterosexual man beviewed as credible by members of the targct population? These were keyquestions and, as it iurned out, valid ones. A substantial proportion of themen taking part i^ Tbe Bareback Proiect-probably one-third to oneJralf ofthem-wanted to know the sexual orientation of the person interviewingthem. Moreover, it was not uncommon for study participants to explain orelaborate upon their answers to some questions, then inquire of the inter-viewer "Do you know what I mean? Have you ever clone that or been inthat situation?" Interviewers' abilify to say "yes," with complete honesty, wasinvaluable toward building trust, rapport, and confidence with the men theywere interviewing. This is not to say that interviewers must ,,match', str:dyparticipants' characteristics in all instances. Rather, it is a way of bringing tothe attention of researchers and health care specialists the imponance of con-sidering such factors when designing a study that will target difficult-to-reachpopulations.

In addition, making sure that all interviewers are highly knowledge-able about the subject matter covered by the interview is an essential partof the interviewer training process, particulady when working with sensi-tive subiect matter. Had the interviewers nor been trained thorougl.rly onsexual behavior (particulady gay male sexual behavior) and drug use termi-nology prior to undertaking The Bareback Project, many of their interviewswould have been more difficult to complete properly, vital information mighthave been lost, and many of their interviewees wotrld have lost confidencein them as researcl.r professionals. Researchers planning studies involvingdifficult-to-reach populations andlor sensitive subject matter must instill intheir recnriters, outreach workers, and interviewers the importance of treat-ing all persons v/ith respect and digniry, and the imponence of respondingto ever).thing they hear during an interview in a nonplusecl way. Vorkingon 7he Barcback Project, interviewers often had study parricipants recountfor them stories of rampant drug use (occasionally among people who werethe age of their parents and grandparents), knowingly infecting other men

ReseLrcb on Sensitiue SuAect Mattel

with HIV, men's effors to become HIV infected as quickly as possible, se-vere childhood traumatic experiences, and the like. Had the interviewersnot listened patiently and professionally, and where appropriate sympathet-icallylempathetically, to all of these stories, Tbe Bareback Proiect would nothave been nearly as successful as it was in understanding the role that en-gaging in unprotected sex plays in men's lives. Rehearsing such listening,interacting, and responding skills during interviewer training is crucial forany research endeavor that will target sensitive subject matter or "hidden'populations.

The final planning stage issue that we wish to discuss pertains to bauinga strateglt for bandling multipb user names or tbe use of multiple Web sites.Researchers who plan to develop an Internet-based study must realize thatmany of the people they will wish to include in their studies are membersof more than one targeted Web site. For Tbe Barcback Project, for example,only 8.1% of the men interviewed reported that the panicular site fromwhich they were recruited was the only Web site they had used during thepreceding month to meet other men for unprotected sex. Informally duringthe interviews, some men spoke of using more than 10 such \$[eb sites on aregular basis for that purpose. In addition, many people who like a particularWeb site create more than one user name or profile on that site, typicallyusing their different online identities for different types of site usage. Againusing 7he Bareback Project as an illustration, some men who consideredthemselves to be sexually versatile posted one profile for use when they werein a "bottom" mood (i.e., when they wanted to receive anal sex), one profilefor use when they were in a "top" mood (i.e., when they wanted to be theinsertive anal sex partner), one profile for use when they were in the moodfor group sex, and so fonh. Bowen, Daniel, ri(illiams, and Baird (2008) haveaddressed some similar issues as these; their work is well worth consulting.

From a research methods perspective, this poses a problem for re-searchers because it is rare for researchers to want to double-count the sameperson in any given study. Researchers planning an Intemet-based studyneed to strategize before they begin recruitment, to make sure that they havedevised a way to identi4/ and handle duplicate respondents. Identifying in-dividuals who try to participate two or more times can be difficult, becausemany of them do not disclose the fact that they have already panicipatedin the study. This occurs sometimes because they want the additional studyparticipation money. Researchers might consider asking potential study par-ticipants during the screening process whether they have taken pan in thisparticular study previously. Some individuals will answer honestly, therebyhelping to minimize this source of bias. Some will not answer honestly, leav-ing the research team with the need to develop other ways of identifuingthe same person using multiple Web sites or multiple user names on oneDarticular Web site.

H. Klein et al,

RESEARCH CONSIDERATIONS AND RECOMMENDATIONS:THE RICRUITMENT STAGE

In this part of the anicle, we discuss some issues that are pertinent to thefecruitment stage of a research project involving the use of the lnternet forconducting a scientific inquiry into sensirive subject nlxrrer with hard-to-reach populations. Ve address eight such topics, in sllccession: (l) rnakingthe research team mernbers' credentials available, (2) lcngth of the initialapproach, (3) how to approach porenrial study participanrs, (4) when roconduct recruitment activities, (5) where to conduct recnlitlrcnt activities,(6) timeliness of lesponding to people, (7) offering a selection of payments,and (8) posting of photographs.

Based on our experience conducting our various projects, the presentauthors strongly suggest that researchers and health care professionals plan-ning an Internet-based study make their credentials ,eadily auailable to po-tential study participants. 1'his may be done in a variety of ways. ]'he officialproject \qeb site cor-rld be designed with a link to a separare page providinga brief biography and career summary for each key person working on theproiect. This is especially importanr for the principal investigator and for anyresearch team members who are likely to have personal contact with thestudy participants. Potential respondents should be offered direct rWeb linksto the universities or research centers where the researchers work, especiallyif credentials and other professionally identi$uing information are providedthere alreacly. Also, rvhen approached initially to panicipate in a particularproject, potential study panicipants should be encouraged to con(hr(.r a ya-hoo! or Google search on the principal investigator's name, to see some ofthe otlrer work that he or she has done. Working on Tbe Bareback pn:tject,for example, Klein and Lambing were sulprised at the sub$antial numberof men who took the time to "check them out" so that they could reassurethemselves that they were being approached by legitimate researchers (asopposed to scam artists trying to obtain personal information from them fornefarious reasons). \Vith increasing use of the Internet by scammers for un-toward purposes, it becomes incleasingly important for research teams wish-ing to avail themselves of Internet-based research samples to combat this byproviding people with legitimat y-related information as eady as possible inthe recruitment process.

When it contes rime for researchers to approach members of their targetaudience online to invite them to participate in their study, it would behooveIhem to make lhal initial approacb as brief as lnssible. This can be verychallenging, as it runs counter to the ordinary demands of provicling potentialsnldy pafiicipants witl.r full "up-fronf' disclosure as pan of the informedconsent process. Our own experience, however, suggests that response ratesand panicipation rates are likely to be related inversely (and faidy strongly

Research on Sensitiue Subject Mttttel 379

inversely at that!) to the amount of verbiage used in the initial approach. Forexample, in Tbe Bareback Project, Y\lein and Lambing opted for an initialapproach that was a mefe 225 words long. lt provided men with a statementof the proiect's purpose, what would be expected of them if they elected toparticipate, an open-ended invitation to ask questions of any kind about thestudy or the research team, a variety of ways to contact the research team,brief mentioning of the payment and confidentiality of the research, and asuggestion that they conduct an Intemet search on the principal investigator'sname to feel reassured of the legitimacy of the project about which they werebeing contacted. This satisfied the protection of human subjects requirementsin an ethical manner, and it did not alienate men by providing them withwhat they perceived to be so much detaiVinformation/text that they stoppedreading partway through the document.

Researchers planning an lnternet-based study must also make decisionsabout bou n approacb potential study panicipants. A variety of options exist,and each has its unique constellation of advantages and disadvantages.

Option #1: Contact people via e-mail. This offers potential study partic-ipants the opportuniry to read the research invitation at a time of their ownchoosing and convenience. However, it lacks a personal touch and makesit very easy for people to stop reading the invitation to participate beforethey have had a chance to give the matter carefttl thought. This approachalso makes it difficult for researchers to know whether oeoole have read andconsidered their offer to take pan in the study, because many rweb sites donot have a feature available showing whether someone has read an e-mailsent to them via the site. This can cause problems with the computationof participation, active refusal, and passive refusal rates, and for assessingresearch bias based on participation/refusal. (For example, if someone issent an e-mail invitation to participate in a study but never opened the e-mail, that should not be counted against the researcher in the computationof refusal rates nor counted in the researcher's favor in the computationof participation rates.) Finally, approaching people via e-mail can lead toa time-consuming back-and-forth process when would-be study participantsdo not provide all ofthe information necessary to foster the next step in theirpanicipation (e.g., they provide a name but no contact information, or theyprovide a telephone number but no indication of good,4:ad times to call).

Option #2: Contact people via instant message. The approach offers asits principal benefit the immediacy of interaction. The research team membercan discuss the project with the person interactively, answer any questionsthat the person may have interactively, and can get a "yes" or ,,no" answerrelatively quickly. But using instant messages to recruit study participants hasseveral drawbacks too. First, it can be intrusive, especially because the re-searcher has no way of knowing what the person being contacted was doingbefore he or she received the project-related instant message. Second, notall instant message systems inform the user of the recipient's online/offline

H. Klein e, al.

status. Yahoo! Instant Messenger and Skype, for example, relay messagesto the recipient whenever he or she next comes online. As a sender, theperson has no way of knowing whether his or her unans*'ered message isgoing unanswered because the person does not want to talk. if he or sheis not online, or if he or she does not have the instant messaging softwaredownloaded. 1hird, annoyed recipients of instant messages can, wirh a clickof a button, repoft tlte person contacting them to the Vel> site's adminis-trators, requesting them to take action against the sender.'lhis can lead tosuspensions of e-mail accounts without warning, and this can be a significantdismption to the smooth operation of an ongoing research study. This hap-pened on several occasions during Moskowitz and Seal's study (describedin this journal issue) as well as during Klein and Lambing's work on TbeBareback Proiect.

Option *J: Contact people via chat room discussion. As with the instantmessaging option, this approach offers the benefit of immediate interactionand the ability ro speak with people about the projecr they are being asked tojoin. Again, however, drawbacks exist with this approach as well. Chat roomsare not private locations. Other persons in the chat room may join in on theconversation and influence people's decisions to or not to participate in theresearch endeavor. Moreover, people who have participated in the studymay speak with othe$ in the chat room at some point in the future whenthe research team returns there to do additional recnlitment. This can taintthe research process, either by a satisfied study participant ,talking up,, thebenefits of having taken part in the study or by a dissatisfiecl study participantdiscouraging others from taking pan in the study based on his or her ownexperiences. Also, many chat rooms are supervised by moderators who,at their discretion, may eject people for noncompliance with the .rules andregulations" or online etiquette established by that Web site or online servicefor its members. If they do not want to see research proiect recruitmenttaking place in their chat room, the moderators can prevent the researcherfrom being able to approach people there in rhe furuie.

There is no easy panacea for these issues. For Tbe Bareback pxlect,Klein and Lambing opted for a combinecl approach that suited rheir study,sresearch needs well. All persons were conracted initially via e-mail. In thatmanner, the researchers were able to supply potential panicipants with astudy overview that tl.rey could read-or not read-at their own convenience.During the hours when study team members were engaged in recruitmentactivities, they were logged onto two different instant messaging services(typically, America Online / AlM, Yahoo! Instant Messenger, ar.rd,/or MSNMessenger). In this way, people who wanted to speak with rhem aboutthe study could reach them interactively, based on the contact informationsupplied to them in the original e-mail approach. This approach worked well,as some men cleady prefered all communication via e-mail, some preferredit via instant messaging, and some preferred to speak with someone by

Researcb on Se\sitlLe Subject Mottet 381

telephone (which was possible because the main contact telephone numberwas supplied in the original e-mail sent to them).

Another recruitment-related consideration pertains to wben to perfonnrecruitment acliuirr'es. Researchers engaging in Internet-based research stud-ies need to bear in mind that their study participants use the Internet atvery different hours of the day and on different days of the week. Somepeople are online almost constantly; others have a limited, routine set ofhours during which they read,/write e-mail; still others vary in their Internetuse from week to week. Different "types" of persons are likely to use theInternet during different time periods, too. For example, people with full-time iobs are more likely to be online during the evening or on weekendsthan during the da)'time. Persons who are unemployed or homemakers orliving on disability or working a night shift are more likely than others to beonline during the da).time. Consequently, eligible persons for most researchstudies may be found online nearly 24 hours a day, 7 days a week, 365 daysZ year-

Therefore, it is wise to engage in recruitment activities at different timesof day and on different days of the week, particulady if the researcher's goalis to derive a representative sample of Intemet users for his or her onlinestudy. This is especially important if the study design calls for approachingpeople based on who is logged on at a panicular point in time. Ve would liketo offer one caveat here: Many people respond to invitations to participatein a research study almost immediately after they read the initial invitation.If they receive it during the late evening or on a weekend, they may feel"put off if they are unable to reach someone or receive a response quicklybecause they are sending their reply during non-business hours. Therefore,in their initial approach to potential study participants, researchers shouldlet people know the hours when a member of the research team may bereached. Our own experience suggested strongly that the more flexible theresearch team members could be with regard to doing recruitment, respond-ing to potential study participants' questions, and doing interviews, the morelikely they were to succeed in getting a better representation of their targetaudience in their research sample.

Along these same lines, researchers must give consideration to ubereonline tbq) uan to recruit potential study participants. Nowadays, for mosttarget populations, multiple 'Veb sites cater to their interests and needs. Thismakes it important for researchers to make decisions regarding the numberof u(eb sites they wish to use as the catchment areas for their studies, andwhich Veb site(s) they wish to use for that purpose. Spending time on avariety of candidate Web sites and 'getting a feel" for each of them is aworthwhile investment that researchers should make. During that process,special attention should be given to detemining who exactly it is who usesthat \Veb site, and why persons are choosing to become members of, orto spend time on, that particular \i[eb site rather than on another

'Website

H. Klein et al.

with a seemingly similar emphasis (e.g., gay.com versus adam4adam.com).This is critical, because it has a direct impact upon the target populationthat will be included/excluded frorn the study about to be implemented.The use of multiple Web sites is often advantageous, so as to enhancegeneralizability of one's research findings. However, as mentioncd earlier,the use of multiple Veb sites also increases the complexity of the research bymaking it difficult to know whether a particlllar person has been approachedto pafticipate elsewhere, whether a particular individual has pafiicipated inthe study r l r r rdy. and so fonh.

Another recruitment-related topic we would like to address is that of thetimeliness oJ reslnnding to people. In any research endeavor, it is advisableto answer people's questions and to try to schedr.rle rhem for their intefviewsas quickly as possible. Yet never is this truer than q'hen one is performingInternet-based research on sensitive subjects with ditficult-to-reach popula-tions. \Whenever sensitive subject matter is involved, people often becomeskittish if they have too much time to think about participating in a study,or to rethink their decision to participate. The adage "Strike while the iron ishot" applies particularly well to this situation. When people are approachedto participate in an Internet-based study dealing with sensitive subject rnat-ter, it is advantageous to the researcher to try to get them scheduled andinterviewed as soon thereafter as possible. The large maiority of thern willnot regret having participated in a well-run, well-executed project, no matterhow personal the questions asked of them. In contrast, a sizable propor-tion of them uill back out of a promise ro participate if there is a lengthygap between the time they are approached and rhe time they are actuallyinten'iewed. At most, this interval should be one week; ideally, it wouldbe less than two days. Similady, any time solneone replies to an instantmessage, a chat room messagej or an e-mail with questions about a studyor with an interest in participating in it, it behooves the research team torespond as promptly as possible. Delays in responding or providing infor-mation communicate any of several unwanted messages to the recipient:The research team is disorganized, the research team does not really careabout that individual's participation in the study, the research is not trulylegitimate, o,, the researchers are not professional in their interactions withpotential study participants. The best way to avoid these perceptions is byproviding prompt, direct responses to all communications from previouslycontacted study invitees.

Ollering potential srud! participants a cboice of pq/ment options is likelyto enhance Internet-based research participation rates. Many previous studieshave examined the effect of offering different fypes of research incentiveson panicipation rates, almost always finding that providing people with cashin hand is an effective and preferable payment option (Cleary, Walter, &Matheson, 2008; Festinger, Marlowe, Dugosh, Croft, & Arabia, 2008). Vhenan Internet-based study is involved, though, this is not an oprion and the

Rese.'lch on Sensltiue Subiect Matter 83

research team must be creative in determining what form(s) of paymentto offer. Many payment options exist, but no one of them is necessarilybetter or more preferable than any other, particulady when dealing withdifficult-to-reach research populations. Mailing a check or a money orderto people is the closest thing to cash in hand as Intemet-based researcherscan offer. However, many study participants are not comfortable (in someinstances, downright unwilling) giving their full name and mailing addressto researchers, even after completing what they perceive to have been a10ff/0 legitimate research interview. Of the men panicipating in Tbe BarebackProject, for example, neafly half of the men selected a payment optionthat did not require them to provide their name and mailing address. Thelarge majority of these men chose another option because of the importancethey placed upon their anonymity. Moskowitz and Roloff (2008) had oneeffective way of handling this situation: In their online survey regardingvengeance and HIV nondisclosure among HlV-positive men, participantswere told either to supply their initials or "Resident" with their address as ameans to maintain anonymity when mailing the incentive. Another optionis to deposit study participation money into an online account, such asPayPal, that the people can access and use. This offers the benefit of almostimmediate receipt of the funds, but many people are unfamiliar with serviceslike PayPal; many are uncomfortable using such services; and many havehad negative previous experiences with such services, thereby renderingthem unenthusiastic about using them again. Also, online payment servicessuch as PayPal can increase the costs to the researcher, as they typicallycharge a fee to the sender for dispatching money. A third payment option,increasingly popular nowadays, is to make some type of online gift certificateavailable to study participants at vendors such as Amazon.com. This optionoffers most of the same advantages and disadvantages iust discussed forPayPal. A fourth payment option, which many men who participated in TbeBareback Project specifrc lly requested, is to offer to donate the person'sstudy participation money to a charity that supports a cause similar to thatundenaken by one's own research srudy. Funding agencies and their rulesand regulations for the dispensation of their money may preclude someresearchers from implementing this option. When feasible, however, our ownexperience suggests that a substantial minority of study participant memberswill opt to have their money donated to a worthy charity. Our bottom-line recommendation to Internet-focused researchers, however, is to offer asmany choices among payment options as possible, because some people willbe unwilling to participate in a study collecting sensitive information if theydislike the options presented to them for receiving their study participantpayment.

Finally, we would like to address one recruitment-related issue thatis unlike all of the preceding-namely, the decision of ubetber to post ,e-searcb team memberc' pbotograpbs online. Many Veb sites where research

384 H. Klein et al.

recruitment effons may be undertaken require users to create a profile beforethey can use or search the site. Indeed, this requirement seems to be grow-ing in popularity on the Intemet. 'l'his forces researchers to consider whetherthey wish to include any photographs of themselves in their profiles. Vhentlie Web site in question is known for fostering personal relationships, theusual expectation is that site members will post photographs of themselves.From a research perspective, though, this can work for o,. against the fe_search endeavor. Many people using these members-only Web sites reacian lnstant message or an e-mail sent to them only after reading the sender,spersonal information in the profile, including viewing his or her photographsfirst. It is not uncommon for people to delete a message or an e-mail fiomsomeone whose profile was not interesting or whose picturc tltey personallyfound to be unattractive. Conversely, rnany people will not reacl a messageor an e-mail fiom someone who has no profile posted or no photographaccompanying it. A great many online profiles explicitly state.No pic: noreply" or something ro that effect. This puts the researcher into a genuinequandary and what is, in many instances, a no-win situation

Adding to the complicated situation, our experiences suggest that somepersonal characteristics may enhance the likelihood of receiving I responsefrom persons contacted dllring recruitment. During part of Tlte BatebackProiect, for example, one research assistant helping out with the proiect,srecruitment efforts was a young, tall, well-btrilt, attracrive African-Americanman. Using the same approach script (verbatim) as all other members of theresearch team did, this particular man received replies from a substantiallygreater proportion of the people he had contacted than rlid anyone elseworking on the study. In parr because of his good looks, ir.r part becauseof his youth, and perhaps even because of his racial background, men onthe site where he clid his recruiting responded more consistently and morefavorably to e-mails sent by him than by anyone else working on the proiect.Apparently, there was a benefit to having this panicullr. man post a pho_tograph of himself prior to conducting recruitnent acriviries on the srudy.That same benefit was not derivecl by the other proiect members (who wereconsiderably older and not as well-builr), though. As another good example,online counselors in Moskowitz, Melton, and Owczarzak's (200p) Inygslig2_tion of instant message counseling chose to post pictures of themselves asthey counseled individuals in a men-for-men gay.com chat room. As foundfor the 7he Bareback Proiect, this borh fostered and suppressed participation.Oftentimes, the young male counselors (with accurate pictures in their pro-files) were seduced by men in the chat room. When the counselors .eueul.dthat they were only there to counsel, some men recoilecl and even activelychastised the counselors publicly in the chat room. In other instances, thepictures seemed to legitimize the counselors as real people and encouragedlengthy counseling sessions.

Researcb on Sensitiue Subject Matter 395

Situations such as these leave researchers wishing to do their own stud-ies on the Intemet to gmpple with the issue of how to handle the mafter ofposting versus not posting photographs of the research team. Again, thereis no one-suS4gestion-fits-all solution to be had here. Klein and Lambing'sdecision of how to handle this on The Bareback Project was for each personworking on the study to create an honest, accurate profile for himself orherself, and to post one recent photograph of himself or herself, but to havethat photogmph "ltrcked." Anybody wishing to see rhe picture merely hadto send an e-mail requesting it to be unlocked, which made the photographviewable by them. Moskowitz and colleagues (2009) decided to make thecounselors' photographs public and viewable to anyone, at any time. Un-mistakably, though, researchers doing Internetbased research, particula yon Web sites requiring profile posting and individual membership prior togranting full-access site usage, must decide how to handle this situation fortheir own work.

RESEARCH CONSIDERATIONS AND FECOMMENDATIONS:TTIE IMPLEMENTATION STAGE

In this final section of the paper, we will provide readers with our thoughtsand recommendations on some issues pertaining to the implementationphase of an Intemet-based study conducted with difncult-to-reach popu-lations, involving sensitive subject matter. Here, we will discuss two specifictopics: cancellation of user names or Web site memberships and making theinterview a pleasurable experience.

Most Web sites have strict policies regarding what their operators con-sider to be proper usage of the site. After all, these Veb sites were neverintended to be used for research purposes. Approaching members online forthe purpose ofconducting research is, in many instances, an explicit or an im-plicit violation of the rVeb sites' terms of service. Accordingly, members whofeel as if they have been bothered by other persons usually have recoursewith the owners/operators of the N(eb sites, requesting that these personsbe suspended or blocked from the Web site in the future, as Moskowitz andSeal and Klein and Lambing discovered. Researchers planning to conductstudies online need to be aware of this, because most Web sites that takemember comments and complaints into consideration will suspend or blockother persons without warning or notice. rvhen they do this, most often,they place an internal block on the member's registration e-mail address,preventing that e-mail address from ever again being used to create a newuser account on that site.

Depending upon the Web site in question, it may be possible for re-searchers to contact the owneryoperators before undertaking their research,explain their study, and get the owners/operators to "buy in" so that member

386 H. Klein et .tl,

complaints do not result in suspension of the researchef's account or username. This is the most ethical way of approaching a new Internet-based re-search study, when it is feasible to gain the support of the owner/operator.Typically, thougl.r, the Web site owners/operators are not cooperative withthe research process, and will not try to find a collaborative soh.ltion thatwill enable researchers to use their sites for recruitment purposes even if asubstantial number of theif members would be interested in participating ina research study.

In the absence of full cooperation on the part of the Web site own-ers/operators, researchers who still I'ish to use the Internet for panicipantrecruitment are forced to take a variety of steps that, to€iether, constitute anend-around approach to dealing with these challenges. First, all persons con-tacted on a particular Web site might be encouraged to contact the researchteam members awa.y from that site. That way, if one of the researclrer's usernames is blocked or suspended suddenly, the researcher can still be in con-tact with potentially inrerested study panicipants. To facilitate this process,initial approaches to possible study panicipants shor,rld include a variety ofoff-web site ways of contacting members of the research team. Second, intheir initial approach to potential study participants, researchers are advisedto request the person's contact information, and to request that people pro-vide that information to them somewhere other than on the Web site whererecruitment is being done. Third, members of the research team can develop^ v^riely of user nanes, all created from the foundation of a different e-mail account, over a series of I'eeks or months. In this manner, whenever arWeb site owner/operator blocks or suspends one account, anothef one thatis "waiting in the wings" can be used. 'l'his

will prevent the research teamfrom experiencing recruitment delays while a new account is established,a new profile is set Llp, approval for the new prolile is pending with theVeb site, and so forth. Unlimited free Hotmail.com, Gmail.com, and Ya-hoo.com e-mail accounts can be created, and these particular sites are veryuseful for the purpose described herein. Moreover, researchers might wishto consider using various lnternet-ready computers from wliich panicipantscan be recruited. Web sites increasingly are being able to recognize users'IP addresses. If mr,rltiple computers are available for use, recniitment cancontinue on a diff'erent computer when/if another cor.nputer is tagged by aparticular Web site as a violator of its tenns of usage,/service.r

Finally, we wish to conclude by discussing the impoftance of and waysof making the interview a pleasurable experience for study participants. Aswith any research study involving human subjects, participation and cooper-ation with thc interview process are likely to be greatest when people findthe research 1>rocess to be enjoyable. We raise this issue in the present paperbecause it is a consideration that is of pafamount importance when dealingwith "hidden" or hard-to-reach populations andlor with sensitive strbject mat-ter.'l'he more personal or intrusive a research study's questions are, the more

Reseqrcb on SensititE Subject Mattel

important it will be to help people to feel comfortable with the quesrioning-and-answering process.

'We can accomplish this many ways. One is to infuse

humor into the interview at times when it is appropriate. Nothing dissolvesparticipation-related anxiety and apprehensiveness better than a good laugh.During the interviewer training process, researchers should encourage theiroutreach team members, recnliters, and interviewers to put their individ-ual sense of humor to use. This is something to be rehearsed during thetraining process, and refined throughout the study. Another way to increasethe pleasurability of a recruitment discussion or an interview is to spenda bit of time just chatting with the participant. Opportunities to humanizethe getting-to-know-you and interview processes almost always arise in areseafch encounter. It is important for research team members to learn howto avail themselves of lhese opportunities. Third, try to anticipate and avertrespondent boredom with lengthier parts of an interview by forewamingthem ofa long section before it starts, or by promising them a deal ("lf you'llbear with me, I'11 get you through this section as best and as fast as I can").Another way of doing this is by asking people to write somerhing downduring various parts of an interview-perhaps asking them to write downthe fesponse options for a particular part ofthe questionnaire. Anlthing thatbreaks up the otherwise-monotonous rhythm of what may be perceived tobe an endless series of questions can reduce the boredom that tends to re-sult from a lengthy interview, can btrild rapport with the study participant,can make the interview process less difhcult for the study participant, andcan reduce tedium for the interviewer, who may have to conduct severalhundred such interviews. These are also excellent ways of helping people tofeel more comfortable sharing personal, sensitive information, because theycome to believe that the person speaking with them truly cares about whatthey have to say and about their well-being.

CONCLUSION

This article has offered readers some suggestions as they themselves planand,/or undertake theif own Intemet-based research or health Dromotionstudies. Vith respect to the planning stage of such a project, oui principalrecommendations have included establishing an official project Web site,maximizing study participants' anonymity, offering options for different waysof collecting data, maximizing interviewers' comfort level with the subjectmatter, and having a strategy in place for dealing with multiple accounts oruser names. Regarding the recruitment stage of undertaking such a study, wesuggested making the research team members' credentials available, keepingthe initial approach script as brief as possible, giving careful consideration asto how to approach potential study participants, making thoughtful decisionsregarding when to conduct recruitment activities, selecting one's site(s) of

388 H. Klein et al.

recruitment carefully, providing very timely responses to potenlial study par-ticipants, off'ering a selection of payment options, and weighing the benefitsand drawbacks of posting of photographs of project personnel. With respectto the implementation phase of research, we offered some advice regardinghow to avoid and/or cope with the potential cancellation of user nlmes orIifeb site memberships and the importance of making the interview a plea-surable experience for study participants. Ve believe that accounting forthe benefits and drawbacks associated with the multitude of decisions thatmust be made in order to implement a successful Internet-based study willassist researchers who are new to the process in avoiding frustrating delaysor encountering significant problems with their projects, particulady as theytarget hard-to-reach populations andlor ask sensitive research questions.

NOTE

l. '[he present authors wish to ackn(,wledgc the fa(1 that the issues and poten(i:tl solutions tothe issucs raiscd herc, \\,itl) rctiird ro workrng *,irhin versus finding cflcelive wrys ()f (iicumvcnting theesgblishcd paBnretcrs of using vark)us Web sires, arc romplicarcd. Navigating lhese issues may enuilmaking challengirg decrsioni wilh rctlard !() lhc implcmentalioo of an c(hically sound. IRB-appf()val)lepft)ject. }rix errnrple, on a case b), casc basis. rese.rf.hcrs need 1() cvaluare \l,hedrer it is ()r ir n()tacccprablc t(J .ontinue to us.': ,r Wcl) sitc whosc ()$,ner, opcdor has bltrked them due 1() vi(rlatiorl\of the sile's telrns of r.rsc. 'lo

dcrernrine rhis. rhe r(]scareh(,ls musr addtess rlrc issu! of har.m doncversus potential l){jnefir ofcontnruing rllc linc ()f inquny, and dereminc which ourweiSlrs which in dreirpa[icular situati()n. l-hey may nced r() wr)rk cl{)scly:rod c.dlabomtively wifi membels of(heir in\lilutionalreview board/hunr:rn subiects prorecrion conrmiltee, in order !o dcvise accepul)lc. eflecrive sohtions tothese challcnrcs.

REFERENCES

Bimbaum, M. (Ed.) (2000). Pslchological erl)eiments on tbe Internet. san Diego:Acaden c Press.

Bowen, A. M., Daniel, C. M., \X/illiams, M. L., & Baird, c. L. (2008). Idenrifyingnrultiple submissions in Internet research: Preserving data integrity. AIDS andBebat)ior, 1 2, 961-97 ).

Cho, H., & LaRose, R. (1999). Privacy issues in Inrernet surveys. Srcial ScienceC'omputer Reuieu, 1 7, 421434.

Cleary, M., Walter, c., & Mathesoo, S. (2008). The challenge of optimising researchparticiparion: Paying participants in mental health settings. Acta Neuropslcbt-atrica, 20, 286-2y).

Creswell, J. V. (2003). Researcb .lesign: Qualitatiue, qualttitatiue, ancl mbed metbodsapproacbes (second edition). 'lhousand

Oaks, CA: Sage publications.

Dube, S. R., Anda, R. F., rvhitfielcl, C. L., Brown, D. W., Felirti, V. J., Dong, M.,et al (2005). Long-tem.] consequences of childhood sexual abuse by gender ofvictiDt. Americdn Journul of Pret)entiue Medicine, 28, 430-l.38.

Festinger, D. S., Marlowe, D. B., Dugosh, K. L., Croft, J. R., & Arabia, p. L. (200g).lligher magnitude cash payments improve research fbllow-up rates without

Researcb on SensitiLe Subject Mattel 389

increasing drug use or perceived coercion. Drug and. Alcobol Depenclence, 96,na-135.

Gilbert, L. K., Levandowski, B. A., Peterson, R., & Scanlon, K. (2010). A comparison ofhepatitis A and hepatitis B measures among vaccinated and susceptible onlinemen who have sex with nen. Intemational Joumal of STD an.l AIDS, 2 j,400405.

Marcotte, D., Wilcox-Gok, V., & Redmon, P. (1999). Prevalence and patterns of majordepressive disorder in the United States labor force. Journal of Mental HealtbPolicJl and Economics, 2, 123-137.

Moskowitz, D. A., Melton, D., & Owczarzak, J. (2009). powerON: The use of in-stant message counseling and the Intemet to facilitate HIV,/STD education andprevention. Patient Educatrion and Cowseling, 77, 20-26.

Moskowitz, D. A., & Roloff, M. E. (2008). Vengeance, HIV disclosure, and perceivedHIV transmission to orhers. AIDS and Bebquiar. 12.727-728.

Mustanski, B. S. (2001). Getting wired: Exploiting the Inrernet for the collection ofvalid sexualiq/ data. Joumal of Sex Researcb, 38, 292-301.

Patten, M. t. (2005). Undeqtanding researcb metbods: An orcruieu oj lbe essentials.Glendale, CA: Pyrczak Publishing.

Substance Abuse and Mental Health Services Administraiion (SAMHSA). (2008). Re-sultsfrorn tbe 2ooTNational Suruq) on Drug Use and Healtb: National fndings.Depafiment of Ilealth and Human Services publication number SMA 08 4343.Rockville, MD: U.S. covemment Prinring Office.

Vard, J. (2007). Straight dude seeks same: Mapping the relationship between sex-ual identities, practices, and cultures. In M. Stombler, D. M. Baunauch, E. O.Burgess, & D. Donnelly (Eds.), SeJf matte6: The setatalitJ) ancl societJ/ reader(pp. 31-37). New York: Allyn & Bacon.

u(ard, J. (2008). Dude-sex: White masculinities and "authentic" heterosexualityamong dudes who have sex with drrdes. Sennlities, 11,471434.

rJfillis, G. (March 1994). CognititE interuielting and questionnaire design: A trainingmanual. Cente$ for Disease Contrel and Prevention. National Center for HealthStatistics: \J{/orking Paper Series, Number 7. (A free copy may be obtained viathe Internet at www.cdc.govlnchs,/products/pubs,/workpap/workpap.htm.)

APPENDD(: PROJECT OVERVIE\TS FOR THE CONTRIBUTORSAND E\'IDENCE OF SUCCESS

Project Oueruiew: Tlse Bareback Pryiect(Hugh Klein, Thomas P. Lambing)

The principal goal of The Bareback Project was to leam more about therole that the Intemet plays in the sexual risk practices of and sexual safetydecisions made by men who use the Intemet specifically to find other menfor unprotected sex. Tbe Bareback Prcject ts a National Institute on DrugAbuse-funded initiative that began data collection in January 2008 and com-pleted the data collection phase during spring 2009. In all, more rhan 300men were interviewed by telephone, completing a one-time interview that

H. Klein et 41.

lasted approximately t hour and 15 minutes. Men were selected randomlyfrom all across the llnited States, based on a random daily combination ofthe first letter of their online user name and on their race/ethnicity as statedin their profile. Men were approached either via e-rnail (the large majority ofthe time), or by instant message (approximately 10% of the time) if rhey wereknown to be online at the time their profile was selected for recruitment. Tofacilitate comparisons between Caucasian men and men of color, the studydesign entailed oversampling men of color. Forlrteen Web sircs were usedto identiry and recnrit men into the study. All of the Web sircs cither specif-ically targeted men looking for unprotected sex partners, or allowed usersof the sites to indicate whether they wanted to find partners with whomthey could engage in protected sex exclusively, protected sex occasionally,or unprotected sex exclusively. Only men whose profiles indicated an inter-est in iclenti$/ing potential unprotected sex partners were included in thisresearch.

To be eligible to panicipare in Tbe Bareback Prcject, men had to be age18 or older and they had to have a profile or a sexual hookup ad posted onone of the 14 targeted Web sites. With very few exceptions, the interviewswere done by telephone. The few exceptions rypically rvere made for menwho were hearing impaired or for those whose financial situation made itin.rpossible for them to incur the telephone charges associated with partic-ipating in a lengthy telephone call. When the interviews were completed,arrangements were made to pay study participants $35 for thcir time, eithervia depositing money into a PayPal account or by mailing a check to them.

The interviews conducted in conjunction with Tbe Barcback Pryiectwere very delailed, capturing a great deal of information that most per-sons would consider to be highly sensitive. No identifying information wascollected, although some besic demographic-type infonnlrion was asked, in-cluding age, race,/ethnicity, cducational attainment, sexual orientation, mar-ital/relationship status, and ZIP code (which was used to compute popula-tion densify in their location of residence). The interview began by inquiringabout men's perceptions of discrimination based on sexual orientation, theirdegree of "outness" regarding their sexual orientation, and general healthbehaviors.

From there, it proceeded with a lengthy section on men's sexual prac-tices, including the extent to which their sexual practices entailed condomuse, the extent to which their sexual pfactices involved internal eiaculation,and how, if at all, their sexual practices differed with Internet-facilitated andnon-lnternet-facilitated sexual hookups. In addition to "standard" questionspertaining to receptive and insertive oral and anal sex, men were askedabout a variety of other sexual practices, such as felching (eating semen outof another man's anus), double penile penetrarion, rimming (oral-anal con-tact), sharing sex loys, fisting, external ejactrlation, masturbation of anotherman to the point of sexual climax, multiple-panner sexual encounters, and

Researcb on Sensltiue Subject Mqtte," 391

"pimping out" or "being pimped out" by a partner (i.e., having a partnerbring several persons to use or have sex with the individual). For men whohad been involved sexually with women, additional questions were posedpertaining to those sexual practices as well. The sexual-behaviors part of theinterview also inquired in detail about men's general preferences for andrecent experiences with a variety of risk-enhancing sexual practices, such asengaging in rough sex, engaging in longJasting sex, having anonymous sex,wanting sex that was wild or uninhibited, having sex in bathhouses or sexclubs, and having sex in public places. Questioning continued by inquiringabout the extent of eroticism men experienced regarding the different sen-sory aspects of ejaculatory fluids, and finally by posing a series of ,,HIV riskhypotheticals" to the men and asking them what they thoughr they woulddo in each hypothetical scenario.

The next portion of the interview focused on men,s substance use andabuse histories. It inquired about alcohol, Viagra or the equivalent, and theuse of nine types of illegal drugs. Questions inquired about lifetime use, ageof first use, frequenry and quantity of recent use, amount of recent use inconiunction with sexual behaviors, preferences for drug use during sex (bothfor oneself and for one's sex partners), lifetime and recent experiences with14 different drug dependency symptoms, and drug treatment history.

The interview proceeded by examining men's use of the Internet. Here,they were asked about their frequenry of using the Intemet for finding datingand sex partners on men-seeking-men (MSM) sites, the amount of time theytypically spent engaging in such activities, the number of men they recentlymet (and the number of these men with whom they engaged in sex) as aresult of the Internet, the proportions of men met online who they askedabout their sexual history or their HIV serostatus prior to engaging in sex,theh comparative likelihood of using the Internet for various meerng men-related purposes (e.g., for sex, for friendship, specifically for unprotectedsex, etc.), and their perceptions of men's truthfulness online.

The interview concluded by capturing detailed information pertain-ing to a variety of psychological and psychosocial measures, and relevantaspects of men's lives. The concluding sections of the questionnaire coveredsuch topics as childhood maltreatment, HIV transmission and testing-felatedknowledge, depression, self-esteem, impulsivity, Hlv-related locus of con-trol, condom use self-efficacy, attitudes toward condom use, current life sat-isfaction, optimism about the future, HIV information bumout. and partnercommunication.

'!trHAT WERE Sour or THri Btnnntcx Pnop.cr'"s MerN Successes?

As one might expect, recruiting men to take parr in the study was a difficultprocess. Despite the inherent difficulty, the research team was able to build a

H. Klein et 41.

research sample of more than 300 men, all of whom were persons who used

the Internet specifically to identify partners for engaging in unprotected sex.

Recruitment was a slow, steady process from the onset to the completionof this study, with no tapering off of willing panicipants as the study pro-gressed. This is something the research team considers important, becausethe expectation at the beginning of the study was that a saturation point

would be reached relatively eady on, and that recruitment would becomemore difficult with time. Such was not the case, however.

Another surprisingly successful aspect of doir.rg this research was thatvinually none of the men interviewed obiccted to answering any of thequestions posed to them, no matter how personal. One man declined tostate his age ("1 think age is a meaningless number and people who needto know about it are rude"); one man declined to indicate how much helikes having rough sex (on the basis that he prefers very gentle oral sex andvery rough anal sex, and he refused to choose an overall preference rankingbecause his preferences regarding those practices were so dif'ferent); and oneman claimed an inability to recall some of the details about his past-30-dayshealth practices. All <;ther questions were answered by all of the men inthe study. Moreover, very rarely did the interviewers express concerns aboutthe truthfulness of the information that had been provided to them by studypanicipants, Mean scores (on a 0-4 scale) on the interviewers' assessments ofthe data quality elements ranged from 3.65 for the amount of effoft men putinto thinking about their answers to 3.91 for their level of coherence duringthe interview. Cooperation with the interview process, comprehension ofthe interview questions, and adjudged honesty and fonhriglrtness were alsohigh, falling between the fonner two elements.

Third, there was very little evidence of underreporting or social desirabil-ity responding in Tbe Bareback Project. Nearly half of the study pafiicipants(44-5Vo) reported illegal dnrg use during the month prior to interview, and85.7%o admitted to lifetime use of an illegal drug. Botli of these figures aresubstantially higher than estimates lbr the American adult mrle population-atJarge (Substance Abuse and Mental Heakh Services Adrninistration, 2008).Two-thirds of the men (67.00/o) reported no condom use during the monthprior to interview and another 26.90/0 rcporled using condoms less thanonethird of the time. Nearly half of the men (42.80/0) admittecl to havingbeen sexually abused during their formative years, and about half as many(2I.8o/o) of the men had been moderately or severely emotionally abusedduring their fornative years. Again, these figures are substantially higherthan those reported for men in the general population (Dube et al.,2OO5).More than one-quarter of the men (26.10/o) gave answers indicating that theywere clinically depressed at the time of their interviev,', and another 19.5%owere depression impaired but subclinical in their level of depression. Thesefigures are also much greater than those reponed for depression among menin the adult general population (Marcotte, Vilcox-Gok. & Redrnon, 1999).

Researcb ofl SensititE Subject tr'l.ttter 393

The preceding examples are not the kinds of answers that one would expectif men were undenepofting undesirable behaviors or experiences; and takentogether, they offer great confidence in the quality of the data obtained inTbe Bareback Pxlect.

Pnlect Ouensleu: Real Msks-Real Consequences--Real Solutions(Lisa K. Gilben, Folishade Omisore)

The goal of this five-year proiect, sponsored by the Division of Viral Heparirisat the Centers for Disease Control and Prevention (CDC), was to increaseawareness of, knowledge about, and intentions to get hepatitis A and hep-atitis B vaccinations among MSM online. The project began in 2003 andended in 2008. In 2004, the first phase consisted of an inirial quantirativeonline survey hosted by GayHealth.com. Parricipanrs (N : 968) were askedabout their sexual and vaccination histories, as well as their perceptions ofrisk, severity of hepatitis, and knowledge about hepatitis A and hepatiris B(Gilbert, Levandowski, Peterson, & Scanlon, 2010). The last survey questionasked if they would be interested in participating in an in-depth qualitativestudy to compare campaigns designed for MSM, and provide their opinionsabout preferences for sexual health messaging, visuals, text amount, andtone,

The initial approach was to host a series of Vinual Focus Groups, usingiChat@ software (which had been somewhat successful in a project withwomen regarding genital herpes). Vith help from a panel of populationexperts in men's sexual health and hepatitis prevention, the research teamselected three campaigns for comparison and posted them on a Veb page.The study protocol called for the men to view the three campaigns, andthen complete interviews with the research team online, using real-timeVirtual Focus Group. The more-than-forty men who said they were interestedin participating in this second phase were grouped by age categories andqueried (via anonymous e-mail address) about a time that would work forthe focus group interview. This process was difficult and time-consuming,and because of this delay, many lost interest. (Readers are encouraged toconsult the comments made eadier in this article, regarding the impofianceof prompt scheduling.) The research team scheduled a number of groups,and only one man actually logged in to participate. Thus, the researchersswitched gears and developed a Vord document, using the form function,to allow participants to fill in the blanks, write rext blocks, and complete thesurvey at a convenient time for them. They were given a URL, asked to viewthe campaigns, and then complete the series of questions detailed in thisedition. With this formative evaluation information, a new online educational'!i9eb

site was developed, and subsequently tested, revised, launched, andevaluated.

H. Klcin et al.

Over the course of this 6ve-year project, the Real Risks-RealConsequences-Real Solutians research team discovered a number of usefultechniques in terms of formative evaluation, development of r.neasures andmethods, and sample recruitment. For each data collection eflbn, local menof diverse ethnicity, race, and age from the priority population (MSM online)were recruited and cognitively tested using a modified version of a longerprotocol (\fillis, 1994). In each case, only 5 to 10 men were interviewed, ateither project offices or a location convenient for the panicipants. The data-gathering tools were tested by asking question-by-question if the wordingwas right (not offensive, how the study participants themselves would say it,how the panicipants themselves would ask a friend the same qtrestion), whatwas missing, if the length was acceptable, what they would answer and notanswer, and so forth.'l'hese interviews yielded a great deal of informationregarding the use of appropriate language. For example, one risk factor foracquiring hepatitis A is oral contact with fecal matter. Was it more acceptableto use "feces" or "poop"? (Men recommended using both.) Also, with surveywording, some men preferred the term "sexual activity" rather than ,,sexual

behavior," which they perceived as judgmental. The research team was ableto refine the suwey tools to include a variety of "non-researchy" terms. Oneinteresting finding was that men did not like the term "vaccinated against"because they found it confusing (in a double-negative way).

The research team investigated a number of survey software options,and over the course of the proiect, three different approaches were se-lected. The first survey was hosted on another organization's \veb site (Gay-Health.com). Their l'f team designed the cocle, their server collected the data,and then the data were sent to the project team in batches. This approachwas costly in terms of labor hours, and the research team did not have con-trol of when they would receive rhe data or the ability to make any changes.For the second effort, Inqlrisite@ survey software was used. It is a packagethat is expensive and also requires significant IT expertise. For the last datacollection effort, the research team used Survey Monkey@ software, whichis very inexpensivc, very easy to learn, and very easy to use. The drawbacksof this method are (l) the data are stored on their server until they aredownloaded, (2) the backgrounds are preselected, so no logos or art can beposted differentially throughout, and (3) once posted, if changes are made,old and new databases are different. The criteria to keeo in mind whenselecting survey software are these: (1) the ability to customize the survey(format of questions, background), (2) the logic, branching and numberingof questions, (3) one's project budget, (4) the number of survey responsesexpected, (5) data storage (whose server and archiving data), and (6) dataexport and analysis.

'l'o evaluate the Red l Risks-Real Consequences--Ret Solutionsweb siteeducational materials, Gilben and colleagues conducted user tests, whichconsisted of a small sample of 5 to 10 local MSM who came to rhe project,s

I

Research on Sensitiue Subiect Matter 395

offices and reviewed the site with a researcher asking specific questionsabout acceptability, content length and depth, features, links, graphics, andspecific questions such as "Vhat do you expect to see after this page?" Usertesting protocols were inexpensive, qr:ick, easily found online, and efficientlyidentified both programmatic and technological issues. People were given$50 in cash to compensate for their time, effort, and fuel.

For each data collection effort, the research team also field-tested themethods and measures to see which questions were answered and whichwere skipped. After a series of trial and error, they determined that forthis population and with this method of data collection, contrary to fypicalsurvey development advice (which recommends asking the most sensitivequestions at the end of the survey), participants were more likely to take aninterest in the survey if the provocative questions were asked ea y on. Bynecessity (for eligibiliry determination and IRB approval), the study team hadto obtain consent and ask people their age, gender, and geographic locationfirst, and then ask about seeking partners on the Internet, numbers of lifetimemale partners, etc. At the end of the survey, the demographic questions werecompleted. The research team found that, if all of the demographic questionswere asked up front, few people completed the entire survey. As otherresearchers have noted, summarizing the informed consent question intoone shon paragraph was key to gaining participation. Moreover, the numberof panicipants appeared to be inversely proportional to the length of theonline survey. Thus, one pitfall of doing online research is the inability tovalidate measures by asking multiple questions about each item. By splittingone longer survey into three shoner surveys (hepatitis A, hepatitis B, and acampaign critique), the Real Risks-Real Consequences-Real Solutians teamwas able to recruit nearly 4OO MSM to complete one of three versions of thefrnal campaign evaluation.

In terms of recruitment, contrary to general survey advice, the RealRisks-Real Consequences-Real Solutions te^m found that men were nlotelikely to complete online surveys during the holidays. (A subsequent infor-mal discussion with Hugh Klein revealed that this was his experience, too,while working c:n Tbe Bareback Project.) For compensating online surveyparticipants, the research team had great success with anonymous $25 giftcards (e.g., giveanything.com). Men provided an e-mail address, and thenseveral days after survey completion, they received an online coupon/giftcard redeemable at more than 300 online stores.

For the last data collection effort, the research team experimented withvarious recruitment strategies. The researchers posted ads to Craigslist (free

of charge) and purchased banner ads on social networking sites. Caution isurged here, however, as advenising on some of these sites was expensive($5,000) and yielded little project-related benefit, while others were free ofcharge for nonprofit organizations and yielded more visitors.

396 H. Klein et al.

ln summary, the Real Rishs-Real Consequences-Real Solutiors te mfound the Intemet an ideal rnethod for: (1) gathering quick, inexpensive, anduseful formative data prior lo designing sexual health educational materials,(2) reaching self-reportedly high-risk MSM with sexual health messages in acost-effective and easily updatable format, (3) quickly measuring perceptionsabout sexllal health and disease prevention topics anrong largc audiences,and (4) asking very personal and sensitive personal liealth and behaviorquestions. In addition, online program delivery and evaluation captures theattention of men from all geographic locations, not iust urban areas.

Project Oueruiew: Tbe Cralgslist Men's Project(David A. Moskowitz, David Sf. Seal)

Tbe Craigslist Men's Pr'oject was not funded by any organization and costa total of about $60 to implement, which was the expense associated witha three-month subscription to the services provided by surveymonkey.com.The researchers did not provide incentives for the men taking the 20- to 30-minute survey. The researchers had no other operating costs in terms of staffor materials; all solicitations to take the suryey were sent out by the authorc.As mentioned in the anicle published in this journal, rhe study's responserate was low. However, of those men who actually started to complete therelatively lengthy survey, a sizeable majority completed ir in its entirety (morethan 7OVA.'fhe researchers were able to develop a sample of 535 men inabout six weeks tirne, by spending approximately four hours per day sendingout recruitment e-mails to potential study panicipants.

As illustrated hy Tbe Bareback Project, rhere exists enormous interestin the risk-taking behaviors of MSM, panicularly those who use the lnternetto find their paftners. Yet, where Klein and lambing focused on men whointentionally esclrew condoms, in Tbe Craigslist Men's hoiect, lvloskowitzand Seal were more interested in MSM on the tlown-low (i.e., highly clos-eted MSM), MSM adhering to different gay ancl bisexual subcultures, MSMpersonality traits (e.9., narcissism), and MSM body attributes (e.g., height,weight, penis size, muscularity, and hairiness). Craigslist.org (Craigslist) wasan obvious lbrum to acquire the data necessary to draw conclusions aboutthe psychosexual behaviors of such MSM, as well as the correlates of risk-taking behaviors in this population. Anecdotal evidence and peer-reviewedresearch (Vard, 2007, 2008) identified Craigslist users as more behaviorallybisexual and more likely to advertise a multitude of specific "turn-ons" and"turn-offs" relative to men using other \Veb sites (e.g., Bear411.com). There-fore, it was not surprising that: the sample was ahnost 400/o bisexual or"heterosexual"; almost 200lo reported being married to a woman; a m jorityof the men reported affiliations with different subgroups (e.g., leathermen,bears, twinks, cross-dressers, etc.); and many nren reported being amenable

Researcb on Sel8itioe Subject M.ttter 397

to many different sexual behaviors (e.g., erotic asphpriation, scatology, andsadomasochism).

Men were solicited to take the survey through block message. \trith nofinancial incentive, the researchers knew that they needed to provide the menwith an appeal to take the survey. Public outcry to those ads placed by menwho were found to mislead and tease other men (i.e., a post that might read,"Re: GBM looking for tonight-is a liar!") suggested that many Craigslist menwere angry with men who had misrepresented their intentions. This feelingwas exacerbated after they had e-mailed such men for a substantial amountof time, often exchanging several nude pictures. Thus, the research teamthought this might be used to create a viable appeal for the men to takethe survey. Consequently, the initial approach to potential study participantsbegan with, 'Have you ever wondered why men on Craigslist tend to playe-mail games? Have you ever been stood up by a Craigslist date? Have youever e-mailed your pictures without any reply?" 1'his attracted at initi^I 294men to complete the survey within the first four weeks.

Power analyses revealed that this sample size was insufficient to detectsmall and even medium effect sizes for the analyses that would test sev-eral of Moskowitz and Seal's research questions. Thus, they opted to repeatthe survey advenising in all of the cities, with a change to the "approachscript" message. This time, they included some statistics that had been col-lected from the 294 men as a means to increase the legitimacy associatedwith the project (i.e., that they were, indeed, performing an academicallysupported study) and to provide the community with some data that mightactually interest them. They began the second e-mail campaign with thefollowing:

What we've found out so far: (1) Most Cmigslist guys are only successfulin actually hooking up 10% of the time, (2) The top reasons for reiectinga guy upon meeting him: (a) he looked like he had an STD/HIV (said56.f/o of Cuys), O) he weighed more than he seemed (said 55.70/o ofguys), (c) he was older than he seemed (said 53.1o/o of guys), (d) hewas less attractive than he seemed (said 52.7W. (3) 32.8o/o of men saidthat they have played games and toyed with other men. (4) And mostinteresting: 57.8% of men reported that the mere pft)cess of posting andresponding to other men was an extremely erotic component of thehookup process. More results to corne if you take the suwey!

Instructions and the rwreb link to the survey followed this introduction.Moskowitz and Seal did not conduct process analyses to gauge differ-

ences between these two approaches to recruiting study panicipants. How-ever, many more men e-mailed them about the data they were reportingand requested more information on the study, the university, and the re-searchers. Reporting some initial and (by peer-reviewed standards) very raw

398 H. Klein et al.

data proved to entice men who perhaps dismissed our first round of ad-vertising as "spam." As a lesson, it is important to tailor and adapt studyadvertisements, particularly as advertising reaches a certain degree of satura-tion (e.g., e-mailing all English-speaking Craigslist cities). Research in which

there is no tangible benefit to the participant or no actual incentive to partic-ipate requires en()rmous vigilance and creativity to keep recruitment active.The researcher must create a psychological incentive or appeal.

All post-survey e-mails that inquired about results or "the stats" wereanswered with the precise data requested. For example, one participant e-mailed, "So why are men such flakes?" Nloskowitz and Seal answered that thedata suggested a cluster of reasons that included youthftrlness and sensationseeking. Also, they indicated that, for ntany men, the eroticism derived frome-mailing and exchanging nude pictures was suflficient to satisfy the purposcs

of a posted Craigslist ad, These computer-mediated behaviors facilitated or-gasm; the men were able to masturbate to such electronic communication(i.e., engage in ryber sex). Overall, the methods employed were well-suitedfor acquiring a large sample of men. More important, they provided a wayto return information back to the study participants, which is often rare inresearch.

The obvious benefit to con.lucting the research using recmitment meth-ods and testing hypotheses with Tbe CraiSslist Men's Project approach iscost. This was the most inexpensive project the researchers have conductedto date. Granted, the opportunity cost of the research team's time was sub-stantial, A research assistant, whose job it would have been to generate theinvitation-to-participate e-mails, would have freed tlie investigators' time tostan other projects. Yet, in research such "amenities" may not be available.For example, the graduate student rarely has any money fo perform his orher research, but is often required to construct and implenrcnt novel studies.As a corollary, he or she is forced to conduct studies on the most inexpen-sive, convenient sample of over-researched individuals (i.e., undergraduatestudents). Tbe Craigslist Men's Prcject study represented a way to circum-vent those obstacles: Recruitment could be conducted at any time, day ornight, outside of time obligations to a mentor or department; the study wasinexpensive; and the research was performed with a relatively hidden andunder-researched population. Thus, the obvious methodological benefit tothis study was its providing evidence that research need not be hindered,halted, or otherwise impacted by lack of institutional resources.

Moskowitz and Seal encourage young researchers or other tesearcherswith funher questions or who need advice on cost-efficient survey method-ology to contact them personally abollt theit methods.