An Inconvenient Truth: Arbitrary Distinctions Between Organizational, Mechanical Turk, and Other...

Industrial and Organizational Psychology 8(2) pp 142ndash164 June 2015Copyright copy 2015 Society for Industrial and Organizational Psychology doi101017iop201513

Focal Articles

An Inconvenient Truth Arbitrary DistinctionsBetween Organizational Mechanical Turk andOther Convenience Samples

Richard N LandersOld Dominion University

Tara S BehrendThe George Washington University

Sampling strategy has critical implications for the validity of a researcherrsquos conclu-sions Despite this sampling is frequently neglected in research methods textbooksduring the research design process and in the reporting of our journals The lackof guidance on this issue often leads reviewers and journal editors to rely on simplerules of thumb myth and tradition for judgments about sampling which promotesthe unnecessary and counterproductive characterization of sampling strategies asuniversally ldquogoodrdquo or ldquobadrdquo Such oversimplification especially by journal editorsand reviewers slows the progress of the social sciences by considering legitimatedata sources to be categorically unacceptable Instead we argue that sampling is bet-ter understood in methodological terms of range restriction and omitted variablesbias This considered approach has far-reaching implications because in industrialndashorganizational (I-O) psychology as in most social sciences virtually all of the sam-ples are convenience samples Organizational samples are not gold standard researchsources instead they aremerely a specific type of convenience sample with their ownpositive and negative implications for validity This fact does not condemn the sci-ence of I-O psychology but does highlight the need for more careful considerationof how and when a finding may generalize based on the particular mix of validity-related affordances provided by each sample source that might be used to investigatea particular research questionWe call for researchers to explore such considerationscautiously and explicitly both in the publication and in the review of research

Despite its importance external validity receives only cursory treatmentin most research methods textbooks Even handbooks are light on cover-age in their lengthy classic reference work on research methods Pedhazur

Richard N Landers Department of Psychology Old Dominion University Tara SBehrend Department of Organizational Sciences The George Washington University

Correspondence concerning this article should be addressed to Richard N Landers250Mills Godwin Building Department of Psychology Old Dominion University NorfolkVA 23529 rnlandersoduedu

142

arbitrary distinctions between convenience samples 143

and Schmelkin (1991) devoted a scant four of their 819 pages to externalvalidity whereas Shadish Cook and Campbell (2002) provided a compara-tively rigorous treatment with approximately 10 pages In part because inter-nal validity is a prerequisite for external validity much greater focus in thesevolumes is placed on internal validity and the various techniques to maxi-mize it This bias is reflected more broadly in the writing and review processas well although authors will devote pages of text to measure identificationexperimental design and analytic strategy considerations related to externalvalidity are often limited to a token paragraph in a limitations section Giventhis focus in both seminal texts and current research literature we suspectthe balance of coverage in graduate education is similarly skewed

With such a lack of guidance many journal reviewers are left to rely ontheir own idiosyncratic reasoning as to whether external validity is threat-ened in a particular study The use of obvious convenience sampling inparticular is a lightning rod for criticism of study generalizability Eachtime a new and more convenient sampling technique begins to gain trac-tion discussion begins anew regarding ldquoappropriaterdquo sampling Argumentsagainst sampling that is ldquoa little too convenientrdquo appear anew on scholarlydiscussion lists and in critical reviews returned by journal editors At leastone industrialndashorganizational (I-O) psychology journal the Journal of Vo-cational Behavior forbids the publication of research conducted with suchsamples stating that the use of online panels ldquothreaten[s] the integrity of re-search samples and the validity of resultsrdquo (Elsevier 2014 Introduction sec-tion para 4) Unfortunately most of these arguments are based on neitherempirical evidence nor a compelling theoretical model of validity or gener-alizability Instead theymore typically rely onmyth intuition and traditionComments like ldquoThis study is weakened by its reliance on a college studentsamplerdquo are common This kind of uncritical and nonspecific condemnationis harmful because simple decision rules categorizing particular sources ofconvenience samples as good or bad unnecessarily limit the types of samplesfrom which researchers are willing to draw Shrinking the pool of legitimatedata sources this way slows scientific progress without cause Researchersmust consider threats to external validity systematically and scientifically

As an opening salvo on external validity myths in this article we ex-plore an aspect of external validity that is a common to all research studiesgeneralizability from a sample to a desired population as driven by samplingstrategy Themost traditional advice given in this domain is to carefully con-sider the population of interest identify a sample that is a more or less ran-dom representation of that population and then approach that sample withresearch participation requests To the extent that response rates approach100 with such a design researchers generally conclude that external va-lidity is not a concern In practice sampling is much messier True random

144 richard n landers and tara s behrend

sampling and even quasirandom sampling are sufficiently uncommon in theI-O psychology literature that we consider the researchers following this ad-vice to be an exception to the common practice More often researchersconduct their studies with whatever sample is conveniently available suchas college students seeking extra credit online panels seeking payment oran organization for which the primary author happens to consult We seekto provide some clarity as to the consequences of these practices

When using such samples in the pursuit of empirically supporting a the-ory that is broadly applicable across organizations in I-O psychology mostresearchers implicitly select ldquoall employees in all organizationsrdquo as their pop-ulation of interest In fact theories that only apply to specific types of organi-zations or jobs are often marginalized few I-O psychology theories for ex-ample apply to ldquothe psychology of retail sales workersrdquo A brief glance at re-cent issues of the Journal of Applied Psychology and Personnel Psychology re-veals article titles and subtitles with broad themes like ldquoRelationship of workmotivations and behaviors to within-individual variation in the five-factormodel of personalityrdquo and ldquoDofinancial rewards for performance enhance orundermine the satisfaction from emotional laborrdquo Such breadth of phrasingdemonstrates that the intent ofmost I-Opsychology researchers is to developconclusions that can be applied broadly across jobs industries and cultures(hereafter referred to as the global worker pool) ostensibly and appropriatelyto maximize the value of those conclusions to I-O psychology practitionersThus any currently or potentially employed person falls within the popula-tion of interest to most I-O psychologists1 Within the global worker poolthe use of any convenient sample of current employees rather than a prob-ability sample of the global pool introduces two technical but well-definedchallenges omitted variables and classic range restriction Reliance on tra-dition is not required to interpret the effects of such convenience

Our goal in this article is thus to explore how and under which con-ditions the convenience samples that are commonly available to I-O psy-chologists such as the use of Internet panels college students and specificorganizations do or do not harm the external validity of research studieswhen researchers draw conclusions about the globalworker poolWe explorethis with the hope that journal editors and reviewers alike will suppress theirtradition-based reactions to nonorganizational sampling techniques suchas the use of college students or Amazon Mechanical Turk (MTurk) andinstead consider the specific strengths and weaknesses of all convenience

1 We note that there are exceptions to this statement for example the study of workers withinhigh-stakes or extreme environments atypical team structures and so on and the fields ofstudy that concern particular cultural or demographic characteristics However such re-search studies are much less common than are those drawing conclusions about overallworker behavior


sampling techniques condemning them only when the external validity ofthe study under review is indeed threatened

Historical Discussions of External Validity and Convenience SamplingTo understand the impact of convenience sampling on external validity wemust first carefully define each of these terms and explore their historical andcurrent treatments both explicit and implied

External ValidityMajor reference works vary widely in their definitions and interpretationsof external validity We cover the three works we believe to be the mostcommon in I-O psychology graduate education Pedhazur and Schmelkin(1991) Shadish et al (2002) and Sackett and Larson (1990)

Pedhazur and Schmelkin (1991) defined external validity as ldquothe gen-eralizability of findings to or across [emphasis added] target populationssetting times and the likerdquo (p 229) In this consideration of validity gen-eralizing to is treated quite narrowly and the term refers to the ability ofa researcher to make valid conclusions about the population from which aparticular sample is randomly drawn In the case of convenience samplinga researcher is generalizing to when drawing conclusions from the conve-nient sample to the convenient population fromwhich it has been randomlyor semirandomly drawn (eg from college students taking part in a studyto a population of college students seeking extra credit during a particularsemester) In contrast generalizing across refers to a researcherrsquos ability todraw conclusions about a desirable population from a given nonprobabilitysample (eg from college students taking part in a study to human behaviorin general) As previously described generalizing to is rarely the external va-lidity goal of I-O psychology researchers Instead convenient organizationsare used to draw conclusions about organizations in general convenient col-lege students are used to draw conclusions about people in general Thuswe are more specifically concerned with generalizing across from convenientsamples to less convenient but related populations

Within this framework Pedhazur and Schmelkin (1991) identified fivemajor threats to external validity one of which is specifically relevant togeneralizing across called the treatmentsndashattributes interaction This threatgenerally refers to any way in which the people studied are different fromthe population of interest To remedy this problem Pedhazur and Schmelkin(1991) stated that such attributes ldquomust be included in the designrdquo (p 230)If such attributes cannot be included researcher conclusions must be qual-ified with this limitation Specific steps to accomplish this inclusion are notdescribed


Shadish et al (2002) defined external validity as ldquothe inferences aboutthe extent to which a causal relationship holds over variations in personssettings treatments and outcomesrdquo (p 83) Using this definition they nextdefined five broad objectives when generalizing from the results of a singlestudy the first of which is relevant to our purpose here narrow to broadwhich concerns drawing conclusions about populations from a researchsample When convenience sampling is criticized it is typically for this rea-son the research consumer believes that the ldquoconveniencerdquo of the samplemakes the sample a poor representation of the population

Within this framework external validity is threatened in one of fiveways two of which are relevant to the narrow to broad generalization objec-tive First the causal relationship may interact with particular sample char-acteristics Relevant to I-O psychology Shadish and colleagues (2002) pro-vided an example of this by describing an experiment in which the mosthighly qualified applicants to a work program were selected to demonstratethe positive effects of that program enhancing the programrsquos apparent ef-fectiveness Second there may be unknown context-dependent moderatorsFor example in a study of the effectiveness of self-regulation interventionson web-based learning students in a lab setting may generally be more will-ing to obey researcher instructions than an organizationrsquos employees may bewhen those employees are spending time away from other more pressingwork while engaging in such training

Sackett and Larson (1990) provided the most prominent advice tailoredto I-O psychologists on this issue Those authors defined external validity asconcerning ldquothe degree to which the results obtained in a given study wouldhold at others times in other settings or with other individualsrdquo (Sackettamp Larson 1990 p 430) which is primarily driven by Cook and Campbellrsquos(1976) definition They spoke to the issue of convenience sampling specif-ically by stating ldquobecause the participants andor settings are not drawnat random from the intended target population and universe respectivelythe true representativeness of a convenience sample is always unknown Forthis reason representativeness cannot be used as a criterion for preferringone convenience sample over anotherrdquo (pp 422ndash433) Instead they recom-mended two alternative criteria to explore the external validity of conve-nience samples sample relevance and sample prototypicality Sample rele-vance refers to the degree to which membership in the sample is definedsimilarly to membership in the population They provided an example ofirrelevance by describing a study of executive decision making conductedwith a college student sampleWith sample prototypicality sample relevanceis assumed Sample prototypicality refers to the degree to which a particularresearch case is common within a larger research paradigm They providedan example of prototypicality by describing a study exploring the predictive


validity of integrity tests although a sample of senior executives completingsuch tests could be collected a sample of nonmanagerial employeeswould bemore prototypical of when such integrity tests would actually be employedAs such they recommended triangulation as researchers in the field come tounderstand more about the most prototypical groups investigations shouldemphasize other less-studied groups This process of triangulation is essen-tial to the accumulation of knowledge

On the matter of external validity threats Sackett and Larson (1990)stated that there are specific circumstances under which generalizability ingeneral is of less concern One of these circumstances is particularly relevantto the question of the generalizability of convenience samples to the globalworker pool ldquowhen a study is conducted solely for the purpose of testinga theory [in a] falsificationist orientationrdquo (Sackett amp Larson 1990 p435) Because this orientation is the current dominant approach within I-Opsychology research this argument is of particular note Within the falsi-ficationist orientation (ie research relying on null hypothesis significancetesting) a theory is proposed and tested not to provide support for the the-ory but in an attempt to falsify it The tested theory is treated as ldquogivenrdquostatistical tests that match the theoryrsquos predictions ldquosupportrdquo the theory andstatistical tests that do not match the theoryrsquos predictions indicate that thetheory is false Theories thus remain until they are falsified or are changedin future empirical work Within this framework Sackett and Larson (1990)argued ldquothe sole criterion for selecting a setting and participant sample isthat it be relevantmdashthis is that it fit within predefined populationuniverseboundariesrdquo (p 435)

Sackett and Larson (1990) also described two instances when externalvalidity should not overshadow other characteristics of the studyrsquos designboth of which relate to theory testing If the primary question of interestis whether a phenomenon can occur rather than whether it does occur orhow frequently it occurs internal validity is of greater importance than isexternal validity Similarly if the purpose of a study is to falsify a theoryinternal validity should take precedence in such cases reasonable sacrificesto external validity are justifiable

From these three perspectives we have identified a common core inregard to convenience samples and their effects on external validity Whensampling at random from a population researchers are able to rely on clas-sical test theory to support the argument that a sample is reasonably repre-sentative of a target population When convenience sampling which is theapproach of virtually all I-O psychology research researchers cannot rely onprobability alone when making the argument of representativeness Insteadconvenience sampling involves randomly sampling a convenient populationand making a rational argument as to why the convenient population is


sufficiently similar to the intended population such that statistical conclu-sions about the convenient population may be used to draw theoretical con-clusions about the intended population

In our experience such justification is rare in the I-O psychology litera-ture When college students are sampled an argument should be made thatthe empirical relationships observed are likely to be similar in the globalworker pool Instead a few lines in the limitations section (eg ldquoTheseresults may not generalize due to the use of a college student samplerdquo)are usually the extent of this argument When a particular organizationis chosen as the target similar challenges are faced There is no guaran-tee that the chosen organizationrsquos employees who have likely been sub-jected to selection procedures training onboarding team building andan eclectic mix of other interventions represent the global worker poolYet the pros and cons of such a sampling choice are rarely if ever evenmentioned There has been a recent push for organizational sciences jour-nals to require authors to describe the context of their research whichwill allow readers to judge external validity for themselves (eg Johns2006) Despite these calls it is not typical to describe the context or sam-ple in detail except in the few journals explicitly requiring this (Rynes2012)

Convenience SamplingConvenience sampling is often treated quite casually even in organizationalresearchmethods textbooks For example Trochim andDonnelly (2008) de-voted only a few paragraphs to the use of convenience sampling in their re-search methods textbook mostly to note that such samples are both com-mon and problematic In reference to a hypothetical convenience sampleTrochim andDonnelly (2008) stated that ldquosuch a sampling plan almost guar-antees that the sample selected will not represent the populationrdquo (p 51) andelaborated that although convenience sampling may have some value for ex-ploratory research it has ldquoperhaps the least usefulness for generalizability offindingsrdquo (p 51) The authors drew from past political polls to give exam-ples of incorrect election predictions that were based on the oversampling ofwealthy phone-owning voters who tended to be disproportionately Repub-lican We concur with the authorsrsquo emphasis on omitted variables that maymoderate or limit findings However we contend that this example does notcondemn convenience samples rather it serves as a reminder to researchersto investigate exactly which variables may be peculiar in a given sample Inthe example referenced by these authors a phone poll necessarily omittedvoters who did not own a phone Careful thought ahead of time would havebrought this to light Further the question of interest in a political poll isldquoHow often does this phenomenon occurrdquo (ie What is the prevalence of


this political opinion) As we note below this is not an appropriate use ofconvenience sampling but this does not reduce the value of conveniencesamples when investigating other types of questions

Convenience Sampling in Modern I-O PsychologyIssues of sample and research design are critical to the trustworthiness of re-search findingsDespite this design issues are frequently treatedwith genericrules of thumb as opposed to considered argument For instance Aguinisand Vandenberg (2014) reported that in reviewer comments regarding ar-ticles published in Organizational Research Methods only 15 of the com-ments addressed research design issues whereas data analysis was addressedby half Similarly reviews for Psychological Methods addressed design only10 of the time but statistical issues werementioned in 70 of reviews Ourown review of organizational research methods textbooks found that sam-pling was typically mentioned only briefly without much consideration ofthe ways that sample characteristics and recruitmentmethodsmay influencevalidity

Convenience samples are not selected at random Their external valid-ity depends on the particular characteristics of the sample and the settingand procedures of the research All samples in I-O psychology have peculiarcharacteristics This should not condemn these samples Instead a seriousconsideration of how these characteristics moderate or limit the study find-ings should be conducted For example Staw and Ross (1980) found thatmanagerial business and psychology students were sequentially less posi-tive in the ratings of a hypotheticalmanager One could interpret this patternof findings as suggesting that students are not appropriate participants forstudies involving manager ratings However Staw and Ross (1980) consid-ered that the distinguishing variable between samples was experience withthe manager role Students who had the least experience with managementbehaved differently frommanagers who had themost experience Instead ofdisregarding evidence from student samples this interpretation illuminateda key variable of interest and progressed knowledge in this area This studyalso serves as an example of the ways that sample characteristics are distinctfrom lab versus field choices

The lab versus field controversy is the issue related to external valid-ity that is most extensively discussed by I-O psychologists Dipboye (1990)noted that the fieldrsquos acceptance of laboratory research ebbs and flows overtime citing patterns observed in the Journal of Applied Psychology in the1970s and 1980s He reviewed comparisons between lab and field with re-gard to validity especially generalizability which tends to be the main com-plaint about lab research It is interesting to note that many of the pri-mary complaints lodged about lab research tended to describe ldquostudentrdquo and


nonstudent samples (see Campbell 1986 Gordon Slade amp Schmitt 1986Ilgen 1986) conflating this distinction with lab versus field It is possiblethat in the 1980s when this controversy was at its most heated this was anaccurate representation However this assumption would be unwise in thepresent day

Dipboye (1990) also noted that field research at that time was extremelylimited with an oversampling of professionalmanagerial employees andmilitary personnel and an undersampling of clerical and blue-collar profes-sions Field research at this time was also more likely to rely on self-reportmeasures than on behavioral observations Both of these patterns limit thegeneralizability of findings from field research

Dipboye (1990) agreeing with McGrath and others (McGrath 1982McGrath amp Brinberg 1984) who have written about this topic noted thatldquothe only hope for building a valid body of knowledge on organizational be-havior is to use a diversity of settings and strategies the ideal mix dependson the phenomenon under investigation the skills of the investigators andthe availability of research settingsrdquo (p 12) Although not explicitly statedthis can be interpreted as an endorsement of varied convenience samplesWe hope to give the same scrutiny to various types of convenience samplesthat researchers before us have given to lab versus field settings

We suspect that one driver of this debate has little to do with validityRather there is a perception from practitioners that academics do not focuson questions that are important in the day-to-day functioning of organiza-tions (see eg Rynes 2012 Rynes Bartunek amp Daft 2001) An academicresearcher who conducts lab studies may be perceived as being out of touchwith organizational reality This may be a fair criticism of many ivory towerresearchers but such evaluations should not be based on sampling decisionsalone

Sackett and Larson (1990) gave specific attention to the choices re-searchers must make with regard to who will participate in a research studynoting that not all choices made are conscious ones The availability of re-sources often guides the selection of participants This is the very definitionof a convenience sample it is convenient This means that an organizationwithwhich a researcher has a relationship ismore convenient than is one thatis resistant In addition norms and fads in a field may influence the choicesa researcher makes (Rynes 2013) and some choices may be met with morescrutiny when they do not conformwith these fads (Sackett amp Larson 1990)As such various forms of the generalizability conversation have taken placewithin I-O psychology over the past decades and some are just emergingnow We describe each major type of convenience sample common to I-Opsychology below including college student online panel crowdsourcedorganizational and snowball samples


College StudentsMany people equate the terms convenience sample and college studentwhether the sample is drawn from a psychology participant pool or a class ofbusiness students The concern about the external validity of these sampleshas been so great that there is a sizable literature comparing students andnonstudents in areas such as personnel selection and performance appraisal(Gordon et al 1986 Greenberg 1987 Landy 2008) Landy (2008) went sofar as to state student samples are ruining the credibility of research relatingto stereotypes and bias in employment decisions because of this arearsquos heavyreliance on students who presumably behave differently from nonstudentswith regard to stereotypes and decision making

Concerns about college student samples are quite common in I-O psy-chology literature For example Aguinis and Edwards (2013) stated that ldquode-signs that allow for researcher control random assignment and manipu-lation of variables yield high levels of confidence regarding internal valid-ity but are usually weaker regarding external validity eg due to the use ofsophomores in university laboratory settings [emphasis added]rdquo (p 158) Theauthors did equivocate somewhat using ldquousuallyrdquo and ldquopresumablyrdquo to de-scribe the ways that student samples affect generalizability while also invok-ing the memorable but misleading phrase typically used to criticize fieldsrelying on such samples ldquoscience of the sophomorerdquo

In our experience master of business administration student samplesreceive less scrutiny than do psychology student samples It is probably thecase that researchers assume that the average master of business adminis-tration student has more work experience than does the average psychologyundergraduate although this is rarely discussed in articles utilizing masterof business administration students Such reasoning would be relevant onlyin cases in which work experience in a corporate organization is relevant tohypotheses The use of student samples consisting primarily of older nontra-ditional college students would likely compensate for any relative limitationsin samples of traditional college students

Online PanelsOnline panels frequently used by market researchers describe groups ofpeople who volunteer or groups of people who are paid to be available tocomplete questionnaires Participants often provide demographic or otherinformation to panel organizers who make this information available to re-searchers whowish to recruit participants fitting a particular criterion Panelparticipants may opt in or out of any particular study and they may makethis decision on the basis of their interest in the research topic their avail-ability or the compensation offered Participants are usually paid a small feeto complete a research study and researchers typically pay both the panel


host and the participant Online survey software companies Qualtrics andSurveyMonkey both offer panel and recruitment services to researchers aspart of a more comprehensive project management package which includessurvey design and analysis

StudyResponse is another popular panel for organizational researchersIts creators describe its appropriateness as follows

StudyResponse was created in response to difficulties that we and our colleagues had whiletrying to recruit participants for various types of studies that are not sensitive to so called ldquonon-sampling errorsrdquo [emphasis added] In particular StudyResponse is likely to be useful for studiesof phenomena that exhibit robust correlational relationships [emphasis added] StudyResponsehas also been used for qualitative data collections where data would not be analyzed with sta-tistical techniques Generally speaking StudyResponse is inappropriate for demography [em-phasis added] and other applications where coverage errors could adversely bias your results(StudyResponsenet 2015)

This description is consistent with what we note above namely conve-nience samples in general should be viewed with the most scrutiny whena research question deals with estimating precise effect magnitudes or withmeasuring the prevalence of a phenomenon as opposed to the possibil-ity of a phenomenon existing A number of published journal articles us-ing StudyResponse data are listed on the StudyResponse web site includingone published in the Academy of Management Journal and one in the Jour-nal of Personality Assessment (full list here httpwwwstudyresponsenettechreportshtm Stanton ampWeiss 2002)

CrowdsourcingAmazon Mechanical Turk (MTurk)MTurk is a service offered by Amazoncom Inc that purports to ldquoauto-materdquo difficult-to-automate tasks by splitting the work among many humanworkers Amazon originally built the service for internal purposes such astagging product images In this case workers would view a picture of anAmazon product and generate descriptors (eg ldquolamprdquo ldquobluerdquo ldquomodernrdquo)Workers would be paid a few cents per task MTurk has since evolved to beopen to requestors (those requesting work) and workers (those completingwork) from any organization or location Since at least 2009 social scienceresearchers have used MTurk to recruit participants for a variety of topicsand research designs (Behrend Sharek Meade ampWiebe 2011 BuhrmesterKwang amp Gosling 2011)

We believe this type of sample has great potential for organizationalresearchers Aguinis and colleagues (Aguinis amp Edwards 2013 Aguinis ampLawal 2012) referred to this type of service as ldquoeLancingrdquo and suggestedthat it is the ideal blend of an experimental control and a naturalistic settingIn fact the use of MTurk may solve a problem that has vexed the researchcommunity for decades namely the severe oversampling of participantsfrom Western educated industrialized rich and democratic (WEIRD)


backgrounds (Henrich Heine amp Norenzayan 2010) Given that a largeproportion ofMTurk users are fromAsian countries this service can providesamples of non-Western nonrich participants with ease

At present the most substantial barrier to greater adoption of MTurk asa potential method for convenience sampling is reviewer unfamiliarity withand subsequent dismissal of the approach because of a variety of untested as-sumptions In our experience such reviewers do raise concerns worth con-sidering but the reviewers assume these issues are unique to MTurk as adata source or overestimate the impact of the issue on data quality Specifi-cally these concerns can be grouped into four major themes which we de-scribe alongside sample comments from actual reviewers We present thesecomments verbatim and anonymously to illustrate actual reviewer thinkingregarding these issues First we have noted repeated participation concerns(eg ldquowe are concerned that the median MTurk participant has completedforms for 30 different studiesrdquo) This is problematic only if repeated partic-ipation may harm validity for example we would not expect personalitymeasures to become less accurate over repeated administrations Second wehave noted concerns over compensation and resultingmotivation (eg ldquoTheparticipants were paid $2 [which is a very small amount] to participate one has to wonder since the primary motivation of individuals who volun-teer is to earn moneyrdquo) This is of concern only if the compensation level orfinancially drivenmotivation can be theoretically linked to effect size Thirdwe have noted concern over selection bias (eg ldquoWhat about participantswho viewed the task but didnrsquot complete it I find the lack of control oversubjects troublesomerdquo) but such issues are common in all convenience sam-ples Fourth we have noted concerns over the relevance of the sample toworking populations (eg ldquoPerhaps MTurk could get you a more diversesample than the typical student population but at what costrdquo) but for re-searchers with the goal of generalizing to the global worker poolMTurkmaybe ideal for the reasons we outlined earlier Consideration of the particularstrengths and weaknesses of any convenience sampling approach is certainlywarranted but the weaknesses of a particular approach may or may not ap-ply to a particular research question As we note above reliance on rules ofthumb based on these or any other criteria is unwise

Snowball and Network SamplesSnowball and network samples include e-mails distributed through socialnetworks alumni associations civic groups and referrals Such samplesmayinvolve participants who are personal contacts of the researcher This typeof sample seems to be more common in qualitative research which is usu-ally less concerned with issues of generalizability For instance a researcherwho wishes to generate a list of possible job search motives may do so by


interviewing people who have signed up with a university career servicesoffice In this case the network characteristics are relevant and valuable tothe study It may be tempting for a researcher to use this type of sample forquantitative research This is appropriate in only a limited number of casesfirst if the research question is concerned with the dynamics of the networkitself (eg the job search example above or understanding how communi-cation occurs in social and professional networks) second if the researchquestion involves a highly specific and unusual subject matter expertise orcharacteristic that requires the use of personal contacts for recruitment (egfemale airline industry executives) or third if the behavior of interest is bothinfrequent and hard to track but participants have knowledge of others whofit the criteria for the study (eg undisclosed workplace relationships)

Organizational SamplesThe preceding sample types are occasionally met with skepticism by organi-zational researchers who believe that organizational samples are more gen-eralizable than are nonorganizational samples Indeed this is the most typ-ical convenience sample reported in I-O psychology journals Because thispoint may seem surprising it bears repeating The most typical conveniencesample found in I-O psychology journals involves a single organization withwhich the researcher has some prior relationshipWith only a fewnotable ex-ceptions2 in no published I-O psychology research that we could locate hadresearchers solicited employees drawn at random from the full theoreticalpopulation of employees across all organizations of interest Thus nearly allI-O psychology research currently published is based on convenience sam-ples and that research should be explicitly considered as such Inmost casesany idiosyncratic characteristics of these organizations which may or maynot bias results are left unreported and unknown

Statistical Effects of Convenience SamplingSampling strategies can affect the conclusions one draws from convenient re-search samples in twomajorways First samplesmay be range restricted on avariable or variables of interest for example a sample consisting of engineersmay be range restricted in cognitive ability Second a sample may have char-acteristics that either covary or interact with the variables in a model to theextent that these variables are not accounted for biasing effects may occurBelow we describe these two concepts and consider how they may manifestin the types of convenience samples common to I-O psychology when re-searchers are attempting to draw conclusions about the global worker pool

2 One such exception is Wang and colleaguesrsquo work (eg Wang amp Hanges 2011 Wang ampRussell 2005 Wang Zhan Liu amp Shultz 2008) In these studies the researchers used strat-ified random samples drawn from US census data


Range RestrictionIn I-O psychology range restriction is most commonly discussed in the con-text of psychometric meta-analysis (Hunter amp Schmidt 2004) especially re-lated to the validity generalization debate (Schmidt et al 1993) The basicconcept of range restriction however is limited neither to selection nor tometa-analysis Range restriction occurs whenever a sample variablersquos rangeis reduced from its range in the population When considering the relation-ship between two particular variables this takes one of two general forms(Sackett amp Yang 2000) First direct range restriction describes situations inwhich a hard cutoff has been imposed directly on a variable of interest Forexample consider an organization where a minimum cutoff score on mus-cular strength has been set at say the ability to lift a 40-pound weight whilestanding If applicants do not meet this minimum standard they will notbe hired Thus current employees of this organization are directly range re-stricted on muscular strength Second indirect range restriction describessituations in which a cutoff has been imposed on another variable (mea-sured or unmeasured) that is correlated with a variable of interest For exam-ple most American organizations incorporate an interview as part of theiremployee selection process and in many cases interview scores correlatewith cognitive ability Within a particular organization where such an in-terview was used any researcher interested in drawing conclusions aboutcognitive ability would need to consider the effects of indirect range restric-tion More broadly nationality and culture can also be interpreted as rangerestriction when citizens of a single nation are selected for inclusion in astudy and others are excluded any variable correlated with nationality willbe biased in its generalization to the global worker pool because of indirectrange restriction Researchers typically ignore this limitation when report-ing their results yet claim to have tested theories that generalize to the globalpool

Given this range restriction in relation to the global worker pool oc-curs in nearly every study conducted within I-O psychology With such amassive amount of range restriction going on one might wonder what effectthis has had on the external validity of I-O psychology research literatureFortunately the bias introduced by range restriction is well understood andthe bias can even be calculatedmathematically under certain circumstancesIn general range restriction leads to attenuation of observed effect sizesand direct range restriction leads to greater attenuation than does indirectrange restriction This is due to the more proximal nature of direct rangerestriction when range restriction occurs further from a focal variable inits nomological net the effects are buffered through one or more media-tors Calculation of the precise statistical effects and appropriate correctionsis straightforward if sufficient contextual information is known about both


unrestricted and restricted samples even if range restriction is indirect(Sackett Lievens Berry amp Landers 2007)

Range Restriction in Convenience SamplesEach of the convenience samples described above will exhibit some degree ofrange restriction College students are directly restricted in quantity of edu-cation and in whatever selection system is used by their college but also indi-rectly restricted in work experience age earning potential cognitive abilitypersonality and geographic location Both online panels and crowdsourcedwork marketplaces like MTurk are directly restricted by the participantrsquos de-cision to join the website and indirectly restricted by anything correlatedwith that decision such as earning potential current employment and mo-tivation Organizational job incumbents are directly restricted by the selec-tion procedures in place at their organizations and indirectly restricted byanything correlated with those procedures or with attrition from that orga-nization such as job-related knowledge job-related skills cognitive abilitiesphysical abilities personality interests and geographic location All conve-nience sample sources organizations included potentially introduce rangerestriction It cannot be avoided Given that the question that researchersmust ask when choosing a sample is a specific one Are the characteristicsthat are range restricted in the convenience samples we have access to likelyto be correlated with the variables we are interested in measuring If notexternal validity will not be threatened for this reason

Omitted VariablesPerhaps a more insidious and certainly a more difficult to address prob-lem is that of omitted variables A model can never contain all possiblevariables of interest the purpose of a model is to produce maximum ex-planatory power with as few constructs and relationships as possible (ieparsimony) Including the entire universe of variables in a model does notserve to simplify anything However omitting variables introduces the pos-sibility that the model is inaccurate specifically observed effects betweenpredictors and criteria may be inflated or deflated because variance associ-ated with the omitted variable is not considered This problem has been de-scribed numerous times in the literature and has been referred to as omittedvariables bias or left out variables error (James 1980 Mauro 1990 MeadeBehrend amp Lance 2009) it has also been described as one particular type ofmodel misspecification in structural equation modeling (eg Kenny 1979)The omitted variable may be either an additional predictor or a moderatorof the observed effects in the model

In the first case the omitted variable is a predictor This is also knownas the third variable problem wherein the omitted variable is a common


cause of both the predictor and the outcome in the model In this case theeffect of the predictor will be overestimated to the extent that the predic-tor and omitted variable are related or will be underestimated in the eventthat the omitted variable is related to the predictor but not the outcome Thecause for concern is highest when the omitted variable relates strongly tothe outcome and moderately with the predictors in the model Meade andcolleagues (2009) demonstrated that if the omitted variable is uncorrelatedwith the predictor however no bias occurs either positive or negative

If the unmeasured variable is a moderator however more concern iswarranted Observed effects may be reversed nullified or inflated depend-ing on the particular value of or any range restriction in the moderator Sen-sitivity analysis can be conducted to determine the potential magnitude ofunmeasured interaction effects and the robustness of the model This is alsoan area inwhichmeta-analysis can be useful in determining the likelihood ofunmeasured moderators pointing to the importance of a thorough descrip-tion of the study context and sample in all published work (Johns 2006)

Omitted Variables and Convenience SamplesAn often unstated assumption with regard to convenience samples is thatsome characteristic of the sample is relevant to the model as a moderator orpredictor and should thus be included in analysesWhen reviewers raise con-cerns about convenience samples they are often implicitly suggesting thatan omitted variable is biasing the results With regard to college studentsthe unmeasured variable may be work experience or consequences for poorperformance on an experimental task With regard to online samples com-puter expertise may influence observed effects

Four remedies to the omitted variables problem have been suggestedbut only some of these remedies are relevant to sampling concerns The firstremedy is to use experimental controlrandom assignment and the secondis to model additional variables these two suggestions do not address sam-pling concerns because presumably the sample characteristic is common toeveryone in the sample and it would not be possible to model the charac-teristic (eg including ldquocollege student statusrdquo in a college student samplewould have no effect) Further it is not advisable to model many samplecharacteristic variables without cause we agree with Spector and Brannick(2010) on the misuse and overuse of control variables The third and fourthremedies are potentially useful to consider The third remedy is to use previ-ous research to justify onersquos assumptions this remedy is important becauseit requires that there be a theoretical reason for why sample characteristicswould be potentially problematic or not problematic This is also the onlyavailable option if the omitted variable in question is a potential moderatorThe fourth remedy is a careful consideration of the purpose of the research


Meade and colleagues (2009) noted that left out variables error is more ofa concern if onersquos goal is to estimate precise path magnitudes and less of aconcern if significance or effect sizes are the goal even if the omitted vari-able in question has a moderate to large correlation with the predictor Thisrecommendation is consistent with Sackett and Larsonrsquos (1990) advice re-garding the appropriate type of research questions for convenience samplesThus we recommend that careful consideration be given to the underlyingtheory as it relates to the variables in onersquos model For example the use of acollege student sample is a justifiably poor design decision when some char-acteristic of college students would be expected to relate to both the predictorand the outcome under study Similar questions should be raised when con-sidering single-organization samples for example researchers conducting astudy that examines leader behaviors and follower outcomes in an organiza-tion with a positive organizational culture may wish to explore how organi-zational culture can drive both follower outcomes and leader behaviors Ata minimum theoretically relevant characteristics of the organization shouldbe described in sufficient detail for meta-analytic explorations to identifythese sample characteristics

Recommendations and ConsiderationsIn this article we present a historical treatment of external validity presentan exploration of convenience sampling as it is currently practiced in I-Opsychology and provide an overview of the statistical and interpretive im-plications of convenience From this review we have developed five specificrecommendations for the use and critique of convenience sampling in I-Opsychology

Recommendation 1 Donrsquot Assume Organizational Sampling Is a Gold StandardSampling StrategyBecause I-O psychology focuses on the description explanation and predic-tion of worker behavior it is quite natural to assume that the ldquobestrdquo sampleswould come from organizations However this assumption comes from anuncritical consideration of precisely what organizational samples have to of-fer Organizational samples are not probability samples and should not betreated as such In fact organizational samples are often quite limited in am-biguous and difficult-to-measure ways Not only are employees within anorganization range restricted onwhatever selection devices were used to hirethem but there are also a host of omitted variables at higher levels of anal-ysis that may influence the results of a particular study (eg organizationalculture industry country)

Online convenience samples in fact provide a number of advantages overtraditional convenience sampling For example researchers have criticizedtraditional college student and organizational convenience samples as being


WEIRD (Henrich et al 2010) limiting their applicability outside ofWEIRDpopulations This is a problem readily solved with the use of participantsdrawn from sources likeMTurk which have a sizable internationalmember-ship If we intend to create theory broadly applicable across organizationalcontexts MTurk and similar samples may prove superior to those collectedfrom single convenient organizations Because most researchers areWEIRDand consult forWEIRD organizations most of their research isWEIRD too

As an example of problems potentially faced in organizational samplesconsider the following study of a leadership training intervention Our goalin this study is to conclude ldquoDoes this intervention help leaders performmore effectivelyrdquo In this quasi-experimental study two divisions of an or-ganization are randomly assigned to conditions Division A is assigned to re-ceive the training intervention and Division B is assigned to act as a controlDivisions must be assigned instead of individuals because there is no way toisolate leaders within a division from one another the validity of the studymanipulationwould be threatened Because of this organizational reality theinternal validity of the study has been weakened Furthermore because Di-vision A leaders interact with each other a great deal they take this oppor-tunity to compare notes and improve their application of the training Thisadds additional confounds to the design If the intervention was provided toa single individual it may not have worked as well if at all If the interventionwas used in an organization with weaker leaders it may not have worked aswell if at all

Now consider in contrast if this study had been conducted using anonline panel asking individuals in supervisory roles to try out a leadershipintervention Some aspects of the training experience are lost because itmustnow be delivered onlinemdasha distinct downsidemdashbut the diversity of partici-pants in this online panel means that it is unlikely that more than one leaderwill try out these leadership skills within a particular organization Thisavoids some of the internal validity problems faced when using the organi-zational sample By using the online panel as a screening survey researcherscan collect data from a broad cross-organizational sample without the omit-ted variables problems faced within an organization However it will also besubstantially more difficult to collect surveys from direct reports

Neither approach is obviously preferable and that challenge is at theheart of this recommendation In this scenario the researcher would needto consider the trade-offs between the two approaches The organization isnot automatically superior Is the use of a more authentic leadership trainingsession (in-person versus online) and the increased ease of collecting datafrom direct reports worth the sacrifice in both internal and external valid-ity This is a decision the researcher must make in either case and defend inhis or her article


Recommendation 2 Donrsquot Automatically Condemn College Students OnlinePanels or Crowdsourced SamplesOne of the primary conclusions here the one that we most hope researcherswill adopt is that convenience sample is not synonymous with poor sam-pling strategy Virtually all samples used in I-O psychology are conveniencesamples Instead we urge researchers to identify any range restriction likelyintroduced by the sampling strategy of that convenience sample to deter-mine whether any moderators of study effects have been omitted to ex-plore the potential impact of these issues given the research questions andto choose which convenience sample will meet the research goals

A particularly notable advantage of online panels and crowdsourcedsamples is that they are not necessarily WEIRD Broad international sam-ples can be collected as a basic feature of these approaches Even when suchsampling strategies are restricted to participants from a particular countrythe participants may be more representative of that countryrsquos worker poolthan workers in one particular organization would be

Another advantage of MTurk over other convenience samples becomesmore apparent when one considers the nature of range restriction in eachsample In MTurk samples there is only one convenient population to beconcerned about Researchers can map out the nature of range restrictionin MTurk samples drawn from that population (ie which variables are re-stricted in comparison with the global worker pool) and build an empiri-cal literature describing those differences Each study conducted on MTurkadds knowledge about such parameters In contrast in individual organiza-tions the responsibility for this exploration lies entirely with the researchersampling from that organization Ideally any researcher conducting researchwithin a single organization would review global norms for all of the vari-ablemeasures of interest and compare these variables with the range of thosevariables within the sample reporting this information in any submitted ar-ticles that are based on this sample This places a much greater burden onresearchers reducing the value of organizational samples when researchersare aiming to ensure high external validity

With regard to both online panels and crowdsourced marketplaces suchasMTurk there are several instances in which this type of sample is not onlyacceptablemdashit is also ideal Reis and Gosling (2010) outlined a number ofsuch instances in the field of social psychology such as the study of onlinebehavior In the field of I-O psychology there aremany phenomena that takeplace exclusively on the Internet As such the Internet is the best place to re-cruit and study individuals who engage in those phenomena As one exam-ple it may be possible to determine what makes a recruitment ad effectiveand for whom by manipulating the language in an MTurk advertisementand tracking signup rates


Recommendation 3 Donrsquot Assume That ldquoDifficult To Collectrdquo Is SynonymousWith ldquoGood DatardquoThere is a natural tendency to consider the difficulty of collecting a sampleand to consider that difficulty to be a proxy of sample quality For examplewe have seen MTurk criticized as being ldquoa little too convenientrdquo It is im-portant to remember that convenience is not a single continuum on whichconvenient is bad and inconvenient is good Instead each sampling strategybrings its own advantages and disadvantages along multiple dimensions ofvalue in terms of both internal and external validity These must be consid-ered explicitly

Recommendation 4 Do Consider the Specific Merits and Drawbacks of EachConvenience Sampling ApproachAs highlighted throughout this article simple rules of thumb should not beused to praise or condemn any particular convenient source of data Insteadwe recommend a five-step process1 Explore prior theory and identify related constructs in the broader

nomological net of all constructs being studied2 Identify any variables in the target convenience sample likely to be

range restricted or to have an atypicalmean both at the level of the study(eg individual team) and above it (eg team organization industrynation) In organizational samples researchers should consider the ex-isting selection systems the organizational culture (including leader-ship) and the industrywork domain in particular In online and col-lege student samples selection and motivational variables are of con-cern

3 Decide whether prior theory suggests any interactions between anyvariable within the nomological net of the studyrsquos constructs and thecharacteristics of the sample

4 Consider all potential trade-offs as a result of these interactions andchoose the sample that best addresses stated research questions Unlessthere is probability sampling there will always be trade-offs

5 Describe all of this reasoning in any submitted articleWe also recommend that editors do not request the removal of such rea-

soning in an effort to save printing space

Recommendation 5 Do Incorporate Recommended Data Integrity PracticesRegardless of Sampling StrategyA number of best practices have been generated over time to increase thelikelihood of obtaining high quality data regardless of sampling strategyMeade and Craig (2012) provided one of the most comprehensive explo-rations of these approaches currently available recommending the use of


instructed response items (eg questions stating ldquoPlease response lsquoStronglyDisagreersquo to this questionrdquo) consistency indices and outlier analyses Suchpractices are essential because online convenience samples in particular arecriticized because of reviewer perceptions that these samples provide lowerquality data However the relative base rates of careless responders amongorganizational college student and online samples is an empirical question3and it can only be resolved by conducting more research tracking the indi-cators like those recommended by Meade and Craig in such samples

ConclusionIn closing we highlight these issues not to condemn organizational samplesor more broadly convenience samples but instead to call on I-O psychol-ogy researchers to more carefully consider the nature of convenience in thisnew age of big data and creative Internet sampling especially as drawn fromMTurk Simple decision rules categorizing particular sources of conveniencesamples as good or bad ultimately harms researchersrsquo ability to conduct goodscience by limiting the types of samples fromwhich researchers are willing todraw Scaring researchers away from these sources slows scientific progressunnecessarily Sample sources likeMTurk and other Internet sources are nei-ther better nor worse than other more common convenience samples theyaremerely different In all cases we should consider these differences explic-itly and scientifically

ReferencesAguinis H amp Edwards J R (2014) Methodological wishes for the next decade and how

to make wishes come true Journal of Management Studies 51 143ndash174Aguinis H amp Lawal S O (2012) Conducting field experiments using eLancingrsquos natural

environment Journal of Business Venturing 27 493ndash505Aguinis H amp Vandenberg R J (2014) An ounce of prevention is worth a pound of cure

Improving research quality before data collection Annual Review of OrganizationalPsychology and Organizational Behavior 1 569ndash595

Behrend T S Sharek D J Meade A W amp Wiebe E N (2011) The viabilityof crowdsourcing for survey research Behavior Research Methods 43 800ndash813doi103758s13428ndash011ndash0081ndash0

Buhrmester M Kwang T amp Gosling S D (2011) Amazonrsquos Mechanical Turk A newsource of inexpensive yet high-quality data Perspectives on Psychological Science 63ndash5 doi1011771745691610393980

Campbell J (1986) Labs fields and straw issues In E A Locke (Ed) Generalizing fromlaboratory to field settings (pp 269ndash279) Lexington MA Lexington Books

Cook T D amp Campbell D T (1976) The design and conduct of quasi-experiments andtrue experiments in field settings InM D Dunnette (Ed)Handbook of industrial andorganizational psychology (pp 223ndash336) Chicago IL Rand McNally

3 See Behrend et al (2011) for an example of how these comparisons could be conducted


Dipboye R L (1990) Laboratory vs field research in industrial and organizational psy-chology International Review of Industrial and Organizational Psychology 5 1ndash34

Elsevier (2014) Guide for authors Retrieved from httpwwwelseviercomjournalsjournal-of-vocational-behavior0001ndash8791guide-for-authors

Gordon M E Slade L A amp Schmitt N (1986) The ldquoscience of the sophomorerdquo revis-ited From conjecture to empiricism Academy of Management Review 11 191ndash207doi102307258340

Greenberg J (1987) The college sophomore as guinea pig Setting the record straightAcademy of Management Review 12 157ndash159 doi105465AMR19874306516

Henrich J Heine S J amp Norenzayan A (2010) The weirdest people in the world Behav-ioral and Brain Sciences 33 61ndash83 doi101017S0140525acute0999152X

Hunter J E amp Schmidt F L (2004)Methods of meta-analysis Correcting error and bias inresearch findings Thousand Oaks CA Sage

IlgenD (1986) Laboratory research A question ofwhen not if In E A Locke (Ed)Gener-alizing from laboratory to field settings (pp 257ndash267) LexingtonMA LexingtonBooks

James L R (1980) The unmeasured variables problem in path analysis Journal of AppliedPsychology 65 415ndash421

JohnsG 2006 The essential impact of context on organizational behaviorAcademy ofMan-agement Review 31 396ndash408

Kenny D A (1979) Correlation and causality New York NY WileyLandy F J (2008) Stereotypes bias and personnel decisions Strange and

stranger Industrial and Organizational Psychology 1 379ndash392 doi101111j1754ndash9434200800071x

Mauro R (1990) Understanding LOVE (left out variables error) A method for estimatingthe effects of omitted variables Psychological Bulletin 108 314ndash329

McGrath J E (1982) Dilemmatics The study of research choices and dilemmas InJ E McGrath J Martin amp R A Kulka (Eds) Judgment calls in research (pp 69ndash102)Beverly Hills CA Sage

McGrath J E amp Brinberg D (1984) Alternative paths for research Another view of thebasic versus applied distinction In S Oskamp (Ed) Applied Social Psychology Annual(pp 109ndash132) Beverly Hills CA Sage

Meade A W Behrend T S amp Lance C E (2009) Dr StrangeLOVE or How I learnedto stop worrying and love omitted variables In C E Lance amp R J Vandenberg (Eds)Statistical andmethodological myths and urban legends Doctrine verity and fable in theorganizational and social sciences (pp 89ndash106) New York NY Routledge

Meade A W amp Craig S B (2012) Identifying careless responses in survey data Psycho-logical Methods 17 437ndash455

Pedhazur E J amp Schmelkin L P (2013)Measurement design and analysis An integratedapproach New York NY Psychology Press

Reis H T amp Gosling S D (2010) Social psychological methods outside the laboratory InS T Fiske D T Gilbert amp G Lindzey (Eds) Handbook of Social Psychology (5th edVol 1) Hoboken NJ Wiley Hoboken

Rynes S L (2012) The researchndashpractice gap in industrialndashorganizational psychology andrelated fields Challenges and potential solutions In S W J Kozlowski (ed) Oxfordhandbook of organizational psychology (Vol 1 pp 409ndash452) New York NY OxfordUniversity Press

Rynes S L Bartunek J M amp Daft R L (2001) Across the great divide Knowledge cre-ation and transfer between practitioners and academicsAcademy ofManagement Jour-nal 44 340ndash355 doi1023073069460


Sackett P R amp Larson J (1990) Research strategies and tactics in I-O psychology InM D Dunnette amp L Hough (Eds) Handbook of industrial and organizational psy-chology (2nd ed pp 19ndash89) Palo Alto CA Consulting Psychologists Press

Sackett P R Lievens F Berry C M amp Landers R N (2007) A cautionary note on theeffects of range restriction on predictor intercorrelations Journal of Applied Psychology92 538ndash544 doi1010370021ndash9010922538

Sackett P R amp Yang H (2000) Correction for range restriction An expanded typologyJournal of Applied Psychology 85 112ndash118 doi1010370021ndash9010851112

Schmidt F L Law K Hunter J E Rothstein H R Pearlman K amp McDanielM (1993) Refinements in validity generalization methods Implications forthe situational specificity hypothesis Journal of Applied Psychology 78 3ndash12doi1010370021ndash90107813

ShadishW R Cook T D amp Campbell D T (2002) Experimental and quasi-experimentaldesigns for generalized causal influence Boston MA Houghton Mifflin

Spector P E amp Brannick M T (2010) Methodological urban legends The misuse of sta-tistical control variables Organizational Research Methods 14 287ndash305

Stanton J M amp Weiss E M (2002) Online panels for social science research An introduc-tion to the StudyResponse project (Tech Report No 13001) Syracuse NY SyracuseUniversity School of Information Studies

Staw B M amp Ross J (1980) Commitment in an experimenting society A study of theattribution of leadership from administrative scenarios Journal of Applied Psychology65 249ndash260

StudyReponsenet (2015) Research FAQ Retrieved from httpwwwstudyresponsenetresearcherFAQhtm

Trochim W amp Donnelly J (2008) The research methods knowledge base Mason OHAtomic Dog

Wang M amp Hanges P J (2011) Latent class procedures Applications to organizationalresearchOrganizational ResearchMethods 14 24ndash31 doi1011771094428110383988

Wang M amp Russell S S (2005) Measurement equivalence of the job descriptive in-dex across Chines and American workers Results for confirmatory factor analysisand item response theory Educational and Psychological Measurement 65 709ndash732doi1011770013164404272494

Wang M Zhan Y Liu S amp Shultz K S (2008) Antecedents of bridge employ-ment A longitudinal investigation Journal of Applied Psychology 93 818ndash830doi1010370021ndash9010934818

Historical Discussions of External Validity and Convenience Sampling

External Validity

Convenience Sampling

Convenience Sampling in Modern I-O Psychology

College Students

Online Panels

CrowdsourcingAmazon Mechanical Turk (MTurk)

Snowball and Network Samples

Organizational Samples

Statistical Effects of Convenience Sampling

Range Restriction

Range Restriction in Convenience Samples

Omitted Variables

Omitted Variables and Convenience Samples

Recommendations and Considerations

Recommendation 1 Donrsquot Assume Organizational Sampling Is a Gold Standard Sampling Strategy

Recommendation 2 Donrsquot Automatically Condemn College Students Online Panels or Crowdsourced Samples

Recommendation 3 Donrsquot Assume That ldquoDifficult To Collectrdquo Is Synonymous With ldquoGood Datardquo

Recommendation 4 Do Consider the Specific Merits and Drawbacks of Each Convenience Sampling Approach

Recommendation 5 Do Incorporate Recommended Data Integrity Practices Regardless of Sampling Strategy

Conclusion

References


and Schmelkin (1991) devoted a scant four of their 819 pages to externalvalidity whereas Shadish Cook and Campbell (2002) provided a compara-tively rigorous treatment with approximately 10 pages In part because inter-nal validity is a prerequisite for external validity much greater focus in thesevolumes is placed on internal validity and the various techniques to maxi-mize it This bias is reflected more broadly in the writing and review processas well although authors will devote pages of text to measure identificationexperimental design and analytic strategy considerations related to externalvalidity are often limited to a token paragraph in a limitations section Giventhis focus in both seminal texts and current research literature we suspectthe balance of coverage in graduate education is similarly skewed

With such a lack of guidance many journal reviewers are left to rely ontheir own idiosyncratic reasoning as to whether external validity is threat-ened in a particular study The use of obvious convenience sampling inparticular is a lightning rod for criticism of study generalizability Eachtime a new and more convenient sampling technique begins to gain trac-tion discussion begins anew regarding ldquoappropriaterdquo sampling Argumentsagainst sampling that is ldquoa little too convenientrdquo appear anew on scholarlydiscussion lists and in critical reviews returned by journal editors At leastone industrialndashorganizational (I-O) psychology journal the Journal of Vo-cational Behavior forbids the publication of research conducted with suchsamples stating that the use of online panels ldquothreaten[s] the integrity of re-search samples and the validity of resultsrdquo (Elsevier 2014 Introduction sec-tion para 4) Unfortunately most of these arguments are based on neitherempirical evidence nor a compelling theoretical model of validity or gener-alizability Instead theymore typically rely onmyth intuition and traditionComments like ldquoThis study is weakened by its reliance on a college studentsamplerdquo are common This kind of uncritical and nonspecific condemnationis harmful because simple decision rules categorizing particular sources ofconvenience samples as good or bad unnecessarily limit the types of samplesfrom which researchers are willing to draw Shrinking the pool of legitimatedata sources this way slows scientific progress without cause Researchersmust consider threats to external validity systematically and scientifically

As an opening salvo on external validity myths in this article we ex-plore an aspect of external validity that is a common to all research studiesgeneralizability from a sample to a desired population as driven by samplingstrategy Themost traditional advice given in this domain is to carefully con-sider the population of interest identify a sample that is a more or less ran-dom representation of that population and then approach that sample withresearch participation requests To the extent that response rates approach100 with such a design researchers generally conclude that external va-lidity is not a concern In practice sampling is much messier True random















































































































































External Validity



College Students

Online Panels





Range Restriction


Omitted Variables








Conclusion

References















































































































































External Validity



College Students

Online Panels





Range Restriction


Omitted Variables








Conclusion

References

An Inconvenient Truth: Arbitrary Distinctions Between Organizational, Mechanical Turk, and Other...

Documents

Transcript of An Inconvenient Truth: Arbitrary Distinctions Between Organizational, Mechanical Turk, and Other...