1

Researchmetrics,labormarkets,andepistemicchange:

evidencefromBritain1970-2018

JuanPabloPardo-Guerra

DepartmentofSociology

UniversityofCalifornia,SanDiego

9500GilmanDr,LaJolla,CA92093

jpardoguerra@ucsd.edu

2

Abstract

Inthispaper,Istudytheeffectsquantifyingknowledgeontheorganizationofdisciplinary

fields.Usinganoriginaldatasetofabout44,000Britishsocialscientistsandbibliographic

informationfortheirpublished,peer-reviewedarticles,Ishowthattheintroductionof

standardizedresearchevaluationsdisturbedlocalacademiclabormarketsinBritishhigher

education,leadingtopatternsofinterinstitutionalmobilitythatalteredtheepistemic

diversityofsocialsciencedisciplinesandtheorganizationoftheiracademicfields.Much

likeamarket-basedintervention,researchevaluationsleadtoaformofepistemicmatching

thathasdistinctconsequencesonhowknowledgeisgenerated.Inparticular,when

evaluationsaffectorganizationalunits(suchasacademicdepartments)andstress

disciplinarynorms,theyfosterformsofisomorphismthatleadtoreductionsina

discipline’sthematicdiversityandamorehomogeneousstructureforthefield.

Keywords:Quantification,knowledge,epistemicchange,institutions,labormarkets

3

Researchmetrics,labormarkets,andepistemicchange:evidencefrom

Britain1970-2018

Whathappenswhenwemeasuretheworthofscientificknowledge?Foreconomists,

ascertainingthequalityofscientificclaimsisarguablyapreconditionforanefficient

“marketforideas”.Byrevealingthe“degreeofsuccess”ofindividualpublications,

researchers,andorganizationalunits,evaluationsprovidegreaterefficiencytothecomplex

anduncertainallocationofresourcestoinnovation(Arrow1969).Theexpanding,

institutedeffortstoquantifyscientificoutputsthroughapanoplyofmeasuringdevicesare

atestamenttothislogic(Gingras2016;Godin2004),seekingtoprovideinstrumentsthat

bothidentifythevalueandcontributionsofknowledgetoparticulardisciplinaryfields

whileallowingfortheirmanagementinorganizationsdevotedtoresearchandhigher

education.Tobereliable,nevertheless,itisacknowledgedthatevaluationsmustrefrain

fromdistortingscience’spracticesofproduction(DavidandFrangopol2015;Hicksetal.

2015;Stephan,Veugelers,andWang2017).Intheirmostbasicform,evaluationsshouldact

assimplesievesthatdistinguishthequalityofinputswithoutmodifyingtheoperationof

thefreelycompetitiveandself-regulatedinstitutionsofresearch(Merton1973).Evaluating

knowledgecertainlyshiftsthebalanceofresources,buttobeeffective,itmustremain

externaltotheestablishmentsthatgenerateknowledgeitself,providingmeasuresthat

revealwhatisvaluableandavailablewithouttransformingthemakeupofepistemicfields.

Towit,impactfactorsmaywellshapewhereauthorsdecidetopublish,butnotthelinesof

inquirythattheypursuethroughouttheircareersorthestructureofthedisciplinesthey

inhabit.

Sociologistsofsciencedemonstrate,however,thatthecontextofknowledge

productioncanhavedistincteffectsonbothwhoknowsandwhatisultimatelyknown

(Barnes,Bloor,andHenry1996;Bloor1991;Collins2010;FrickelandGross2005;Schofer

2003).Asevaluationsbecomeendemictocontemporaryscienceanditsmanagement,they

donotonlydiscloseputativevaluesbutfundamentallytransformhowauthorsand

institutionsconceptualizeworthyprojectsandinvestintheproductionofknowledgeabout

4

theworld.Inrecentyears,muchworkhasfocusedonunderstandinghowevaluations

shapetheexperiencesofscholarsinmakingsenseoftheirresearchandwork-identities.

Fromstudiesofreactivity(EspelandandSauder2007)toquestionsabouttheimpact,

design,anddemocratizationofindicators(Hicksetal.2015;Ràfols2018;Rijckeetal.

2016),thereisnowangrowingbodyofliteraturedocumentingtheeffectsofmeasuring

knowledgeonthepracticesandimaginariesofcontemporaryacademics(Burrows2012;

ClarkeandKnights2015;Gill2014;HammarfeltandDeRijcke2015;Vostal2015).

Yetbeyondtheindividualexperienceofthequantifiedscholarandrelativelysparse

anecdotalclaims,thereislittleevidencethatresearchevaluationschangetheorganization

ofepistemicfields.Doescounting,measuring,ranking,orvaluingdisciplineschangetheir

composition?AnsweringthisquestionisthetaskItakeoninthefollowingpages.Inthis

paper,Iaddressthequestionofhowmeasuringtheworthofknowledgechangesthe

intellectualorganizationofdisciplinaryfields.Forthis,Iprovidenovelevidenceofthe

effectsofalong-standing,paradigmaticresearchevaluationconductedbytheBritishstate

since1986.Ifocusonthesocialsciencesasparticularlyreactiveandlessinfrastructurally-

bounddisciplinaryformationsinacademia.AsIshow,researchevaluationsproduceda

competitivefieldthatperturbedthepatternsofinterinstitutionalmobilityofsocial

scientists(AllisonandLong1987;Fuller2008),leadingtochangesintheepistemicmakeup

ofdisciplines.Specifically,thesechangesconsistofadecreaseinthethematicand

organizationaldiversityofdisciplines,resultinginmorehomogeneousfieldsofknowledge.

Usinganoriginaldatasetofcareertransitionsandacademicoutputsofover44,000British

academics,Ishowcompellingevidencethatthisprocessofhomogenization(whichIrefer

toas‘epistemicmatching’)wascatalyzedbythewayknowledgewasevaluatedbythe

apparatusoftheBritishstate.Quantification,insum,changedtheBritishsocialsciencesat

theirverycore.

Changingknowledgethroughresearchevaluations

Organizedevaluationsofscienceareahistoricallyrecentphenomenatiedinlargemeasure

totheproblemofmanagingtheexpandingresearch,development,andhighereducation

5

effortsofthesecondhalfofthetwentiethcentury.Theurgencytodirectthegrowingcrews

ofscientistsandinstitutionsacrossanincreasinglycomplexpostwarresearchandhigher

educationecologyhadnotableconsequences.Themostprominent,perhaps,isthe

emergenceoffieldsdevotedtostudyingsciencethroughascientificlens.While

quantitativestudiesofscientificproductivityexistsinceatleastthe1920s(Lotka1926),

muchofthecontemporaryliteratureonresearchevaluationtracesitsoriginstopolicy-

orientedeffortsinthelate1960stomeasurescientificoutputstobettermanagepublic

investmentsininnovationunderincreasingfiscalconstraints(Gingras2016;Godin2006;

SugimotoandLarivière2018).Examplesofthisincludethedevelopmentofscientometric

andbibliometricstudiesthat,byquantifyingandmodellingpatternsofknowledge

production,disseminationanduse,seektopartlydeterminewhereresourcesarebest

spent.Thesedevelopmentswerewidelysuccessful,particularlyastheywerecoupledto

thecreationofmetricsthattravelledintraandtransnationallybetweenfundingagenciesas

heuristicsthatfacilitatedcomparingheterogeneoussetsofscholars,institutions,and

disciplines.Today,theuseofmetricsasmeansforevaluatingscholarlyqualityisastandard

approachthroughouttheworld(Abramo,D’Angelo,andCaprasecca2009;Geunaand

Martin2003;Hicks2012).

Despitetheirpervasiveness,thiswealthofevaluationsisoddlyignoredbythe

sociologiesofknowledgeandsciencewhich,whileattendanttosituatedpracticesof

knowledgeproduction,haveyettoexaminehowtheyshapeandaffecttheevolutionof

scholarlyfieldsovertime.Workontherelatedfieldofresearchevaluation(Rijckeetal.

2016;deRijckeandRushforth2015)hasneverthelessproducednumerousdiscussions

aboutthemeasurementofscience,includingimportantstudiesonhowtheuseofmetrics

andindicatorsshapethepracticesofscientistsandtheirhostinstitutions.Reflectingsome

ofthekeyconcernsanddebatesfromthesociologyofquantification(Bermanand

Hirschman2018;EspelandandStevens2008;MennickenandNelsonEspeland2019),

contemporarystudiesofresearchevaluationshavelocatedthesewithina“broadersurge

ofaccountabilitymeasures”(Rijckeetal.2016:162)acrosspublicinstitutions,whereaudit

logicstransformthestructuralaffordancesandstrategicbehaviorsavailabletoactors.By

quantifying,makingvisible,andincentivizingcertainpublications,outputs,andresearch

6

activitiesoverothers,evaluationsandtheirassociatedlogicsareseentoaffectthe

productionofknowledgeinthreedistinctways.

First,becausetheyarepublicandvisibleexercises,researchevaluationsaredeemed

topromotestrategicbehavioramongscientistswho,knowingtherulesofassessment,

adoptbehaviorsthatseektomaximizetheirpotentialscores.ResearchbyButler(2003,

2005)shows,forexample,thattheintroductionofanevaluationsysteminAustralialinking

researchfundingtopeer-reviewedjournalsledtoanoticeableriseofISI-indexed

publications,withaheavyconcentrationinthelessselectivequartiles.Similarchangesin

publicationpatternshavebeendocumentedelsewhereandinconnectiontothe

implementationofresearchevaluations.Bibliometricstudiesineconomicsshow,for

instance,thattheuseofscientometricsbymembersofthedisciplinarycommunityleadto

boththereinforcementofpatternsofknowledgeproductionandprestigeandthe

stabilizationofdominantscientificparadigms(Aistleitner,Kapeller,andSteinerberger

2018).

Whileevaluationsmayleadtostrategicplanningofpublications,thisneedsnot

affectthecontentsofknowledgeproduction.Amoredirectmechanismofepistemic

change,however,isprovidedbyasecondeffect,namely,ashiftinthefocusofindividual

researchdecisionstoprivilegethosemostcongruentwithevaluations(what(Rijckeetal.

2016)identifyas‘taskreduction’).GläserandLaudel(2016)convincinglyshowthat

tensionsbetweenformalizedevaluationcriteriaandascholar’sownconceptualizationsof

worthyresearchleadtotheabandonmentofsomeprojectsoverothers.Asimpleexample

ofthisisprovidedbyevaluationsystemsthatprivilegearticlesoverbooksasidealresearch

outputs,resultinginfewerpublishedmonographsandeditedcollections(February13thet

al.2018).Alongwiththeseshiftsininvestmentsbyindividualscholars,evaluationsmight

transformdisciplinesthroughathirdmechanism,namely,bycreatingdisincentivesto

interdisciplinarity.Addedtochangesinwhattypeofresearchscholarspursue,the

institutionalizationofevaluationsasformalandinformalworkplaceregulationscancreate

barrierstoconductinginterdisciplinaryworkthatisnotcommensurablewiththe

parametersoftheevaluation(Rijckeetal.2016).Inadditiontomorebeingdifficultto

7

publish,interdisciplinaryworkmightsufferfromtheaddedcognitivepenaltiesofspanning

establishedcategoriesofknowledge(Zuckerman1999).Bystrengtheningtheshared

normsofqualitywithinthefieldandcreatingsymbolicincentivestopatrolthese,research

evaluationscanleadtodisciplinaryentrenchmentattheexpenseofepistemicdiversity.

Asthesestudiessuggest,evaluatingknowledgehaspotentialperformative

consequences:bytransformingindividualandinstitutionalincentivestotheproductionof

scientificclaims,theychangewhatcanbeknownindistinctlypatternedways.Forexample,

themechanismsexploredbyresearchevaluationscholarsprovideanimportantaccountof

howscientistsrespondtoevaluationsbyeithershiftingpublicationandresearchstrategies

oravoidingresearchprogramsthatdonotfitwiththedominantlogicofassessment.Yet

despiteitsrichness,theliteratureisalsolimitedbyitsanalyticalscope:mostcurrent

discussionsfocusonindividualresearchers,addressingthepossibleeffectsofevaluation

regimesonshort-termdecisionsaboutwhattopublishandwhere.Thecollectiveproblem

ofepistemicchangeoverlongerperiodsisthusreducedtothemyriadindividualactionsby

researcherswhochoseto“playthegame”.Ifknowledgechangesinresponsetohowit’s

worthismeasured,welearn,itismostlybecauseofhowscholarsreact,embrace,or

begrudginglyaccepthowtheirworkiscountedandcompared.

Thestateoftheartonresearchevaluationinparticularandthesociologyofscience

moregenerallythuslacksaninstitutionalmechanismexplicitlyconnectingtheactionsof

quantifiedscientistswithsystemic,field-leveltransformations.Whileitistruethatgeneric

‘reactivity’torankingsandquantificationiswelldocumented,lessisknownabouthowit

andsimilarprocessesshapetheorganizationofknowledgeovertime.Weknowthat

scientificpracticeslikelyshiftasaresultofresearchevaluations(GläserandLaudel2016;

HammarfeltandDeRijcke2015),butitisnotentirelyclearifthesechangeshaveeffectson

thedirectionofknowledgeproductionwithlonger-termconsequencesonepistemic

organization.

Theinstitutionallinkexploredinthispapercomesfromconsideringresearch

evaluationsasinterventionsnotonlyuponthebenchanddeskpracticesofscientistsbut,

8

asimportantly,onthelargerlabormarketsthatstructureacademicfields.Theexisting

researchonthequantificationofknowledgelargelyseesscholarsasinstitutionallyfixed,

despitethevastevidencethatshowstheimportanceofresearchmobilityonepistemic-

levelfieldchange(CzaikaandOrazbayev2018;FrickelandGross2005;Latour1987).

Furthermore,focusingonacademicfieldsasspacesstructuredthroughcapitalandmobility

mattersbecause,asWhitleynotes,theyarewhatgivesriseandlogictodisciplines,the

“unitsoflabormarketcontrol”thatdefinethetrainingofnewscholarsandmonopolizethe

meaningofintellectualcontributions(Whitley2000).Thisanalyticalshiftfromtheeffects

ofmeasurementonscholarsinisolationtoitspropagationacrossemploymentstructures

fieldsstemsfromthetwofoldobservationthatresearchevaluationsareintimately

connectedtotheverygenerationoffieldsofcompetitionbetweenscholarsandinstitutions

(Whitley,Gläser,andEngwall2010)thattieboththelabormarketsofspecificdisciplines

andthecareertrajectoriesandresearchdecisionsofindividualscholars.

Theideathatresearchevaluationsgeneratenovelfieldsofcompetitionor

substantiallyalterexistingonesthatareintimatelytiedtothelogicsofdisciplinesispartly

aconsequenceoftheadministrativeoriginsoftheseinterventions.Bothhistoricallyand

presently,researchevaluationsoperateasdevicesofgovernancecreatedandsustained

undertheauspicesofthestatetodealwiththeconstrainedallocationoffinancialsupport

toscientificworkinpublicinstitutions.Throughtheirofficialsanctioning,themetrics,

rankings,andotherformsofcommensurationentailedbymostresearchevaluationscreate

signalsthatarelegibleto,andvaluablewithin,acontextofcompetitionbothforfunding

andtalent,makingtangible“theinterestsdirectlyassociatedwith[anagent’s]positionin

theacademicfield”(Bourdieu1988).Withinotherwiseuncertainandpotentiallymulti-

polaracademicfields(Whitley2000),evaluationsincreasethe“stabilityandpredictability

ofreputationalnorms”,establishingfield-levelasopposedtolocalexigenciesonresearch

strategies:expectationsofqualityoflocalinstitutions(suchascolleagues,headsof

department,anduniversitymanagers)willbebetteralignedtothoseoftheevaluationasa

whole.Theseexigencieswillinvariablyencodeparticularrulesofengagementthatreflect

dominantideasofwhatisvaluablewithindisciplinarytraditionsbutthatsimultaneously

allowforcomparisonandcompetitionamongscholarsandinstitutionsaswellasthe

9

latter’sindividualtaskrealignment.Forexample,anevaluativeframeworkthatlinksworth

toarankingofjournalswillchangethefieldbymakingspecifictitlesmoreattractivethan

others,allotherthingsconsidered.Ifbootstrappedtotheinstitutionalizedwaysmeaningis

collectivelyproduced(Barnes1983),boththeindividualintellectualworthofscholarsand

thebettermarketoutcomeswillbetiedtopublishingintoprankedjournals.Thelogicsof

individualvaluationandthecareerstructureswillbedifferentif,insteadofanexplicit

ranking,aloosebutsharedhierarchyofinstitutionalstatuspositionsiswhatdeterminesa

scholar’sworth(Burris2004).Bothsituationswillalsoproducedifferentconfigurationsfor

theepistemicfield:inthefirst,elitejournalsandtheirgatekeepersarecentraltovalue

determinationleading,perhaps,tostrongerfield-widenotionsofquality;inthesecond,

elitedepartmentsretainsomedegreeofflexibilityinestablishingandreproducingtheir

owntraditions.

Theaboveexamplesalreadysuggesthow,byestablishingcommonmetricsthatare

takenasgivenwithindisciplinarycommunities,researchevaluationscreatenovel

resourcesthatallowsomeactorstoattainbetterpositionsinthefield.Ascholarwhose

workishighlycited,forexample,willhaveevenmorevisibilityinherfieldifresearch

evaluationsplacegreaterworthoncitationmetrics,andsomightbeabletotransformthis

‘evaluativecapital’intoconcreteeconomicgainsthrougheitherintraorganizational

movements(e.g.promotion)orvialabormarkets.Indeed,wecanthinkofresearch

evaluationsasconstituting,ratherthansimplydescribingfromafar,themarketplacefor

ideasanditsconnectedlabormarketoutcomes:theycreate,orattheveryleastaltera

metrics-sensitiveacademicfield,andintroduceanewformofcapitalcenteredaround

evaluationattainmenttoitscompetitivestruggles.ArecentfindingfromtheUnited

Kingdomprovidesevidenceforboththeexistenceofthiscapitalanditstransferintoother

formsofresources.Intheirwork,DeFrajaandhiscolleagues(DeFraja,Facchini,and

Gathergood2016)showthat,overthelastcycleofresearchevaluationsinBritain,

professorialsalariesincreasedathigherratesthansalariesatlowerranksatthesametime

aswageinequalitiesbetweeneliteandnon-eliteinstitutionsalsogrew.

10

Whenperformative,researchevaluationscanthereforeshapethecareerpatternsof

scientists,leadingtodistinctdisciplinaryshiftscatalyzedasmuchbychoicesaboutwhatto

investigate,ashighlightedbytheresearchevaluationliterature,asbythetransformationof

patternsofinterinstitutionalmobility.Evaluationsmaynotstructurethelabormarket

directly,buttheyprovidefurtherincentives—bothfordepartmentalmanagers,units,and

individualscholars—tosortthefieldinwaysthatareproximallycommensurabletothe

assessments.Whatoccursthroughevaluation,then,isaformofisomorphismpromptedas

muchbyinstitutionsimitatingorganizationaldesignsasbythefield-drivenmovementof

scholarsthroughlabormarkets.Toexplorethese,IuseanoriginaldatasetofBritish

academicsinthesocialsciences,brieflydescribedbelow.

Thecase

Ofallthepossibleexamplesofresearchevaluationinfluencingepistemicfieldchange,the

regulationofsciencefundinginBritainarguablycomesclosesttoanaturalexperiment.

Whenthinkingaboutthelinksbetweenquantificationandchangesindisciplinarymakeup,

animportantcaseisgivenbyassessmentswherethefocusandadministrative

consequencesofevaluationarenotonindividualscholars—astendstobethenormin

mostscienceevaluationframeworks—butrathersub-unitsthatserveasavatarsof

disciplineswithintheacademicfield(Abbott2005,2010).Thisbearsthehighest

resemblancetotheformsofreactivitystudiedbyEspelandandSauder(Espelandand

Sauder2007;SauderandEspeland2009),wheremetricsarerelevanttoorganizational

unitsratherthanisolatedscholarsbutneverthelesshaveinfluenceonthelatter’s

workplaceexperiences.InEspelandandSauder’scanonicalwork,rankings(i.e.

measurement)changedthewaylawschoolsdeployedresourcesstrategically.Yettheydid

notsubstantivelychangehowschoolsapproachedthelawasanacademicfield—they

mighthaveplayedaroundwithstudent/teacherratiosorothercomponentsofthemetric,

butthepracticeoflegalscholarshipremainedlargelyunaltered.ThisiswhytheBritish

caseofferssuchaparadigmaticexample:becauseoftheiremphasisonresearchqualityand

breadth(theyinvolvednotonebutalldisciplinesacrosseveryinstitutionofhigher

educationovermorethanthreedecades)theseevaluationsarecentrallyabouthow

11

academicknowledgeatlargeisre-disciplinedandtransformed(CampbellandCarayannis

2012;GläserandLaudel2016;Lewis2015).

ResearchevaluationswereintroducedtotheBritishhighereducationlandscapein

1986whenpublicuniversities—representingthevastmajorityofhighereducation

establishmentsinthecountry—wereexperiencingintensefinancialpressuresstemming

fromtherapidgrowthofthesectoroverthepastquartercenturyandthecountry’s

ongoingfiscalcrisis(BrownandCarasso2013).Uptothispointintime,thegovernment

bodythatoversawhighereducationinBritain,theUniversityGrantsCommittee(UGC),had

allocatedblockfundingostensiblyearmarkedtocoverteachingandresearchexpensesby

accountingforstudentenrollmentacrossinstitution.Thisrelativelyegalitarianapproach,

however,facedmountingcriticisms.Withdwindlingbudgets,agrowingsector,anda

fundingmechanismthatdidn’tdiscriminateresearchquality,keygroupsofpolicymakers

andscientistspushedfora‘selective’allocationtoprovidecentersofexcellencewiththe

abilitytomaintaintheirresearcheffortsandinfrastructuresdespitetheendemicand

seeminglyirresolvablefiscalproblemsofthenation(Whitleyetal.2010).

Thegovernment’scommitmenttoquantificationasaninstrumentofprudential

publicmanagementmeantastrongrelianceonperformanceindicatorsandsectorial

evaluationsasleveragesforresourcedistribution.Thiswasparticularlysoforhigher

education,wherethegovernmentclearlysawindicatorsasdevicesthat,morethan

representing,couldloopvalues,behaviorsandexpectationsbackintothesector(Hacking

andHacking1983).Asa1985whitepapernoted,thegovernmentbelieved“thatthere

wouldbeanadvantageintheregularpublicationofarangeofunitcostsandother

performanceindicatorsbyinstitutionandbydepartment[thatwillbe]importantforthe

internalmanagementofinstitutions”(EducationandScience1985).Measurementcould

notonlyrevealworthyknowledge,products,andinstitutions,butalsofosterincentivesfor

internalreorganization.Inspiredinpartbytheproliferationofotherinformalrankingsand

metricsofinstitutionalquality,theUGCthustrialedasystemaimedatevaluatingthe

researchconductedacrossBritain’suniversitiestoguaranteesomedegreeof‘valuefor

money’legibletoexternalpublicsandthestate(Power1997;Shoreetal.2015;Strathern

12

2003).Theresultwasthefirstofaseriesofnation-wideassessmentsofinstitutional

qualitythat,albeitmuchchanged,remainsinplace.

Likeanydistributionalintervention,thisfirstassessmentwasnotwithout

controversy,criticizedforrelyingonamethod“shroudedinsecrecy”(Phillimore1989).

Operationally,theexerciserequiredalluniversitiesintheUnitedKingdomtosubmit

informationforeachoftheir‘costcenters’tosubcommitteesofexpertsineachdiscipline

(costcenterswereaccountingconstructscreatedanddefinedbythegovernmentthat

broadlymappedontotraditionalsubjectsanddepartments;e.g.physics,medicine,

sociology,biologicalsciences).Eachfilecontainedatwo-pagedescriptionofthe

achievementsofthecost-center,fiveexamplesofthebestpublicationsproducedinthe

previous5-yearperiod,dataaboutnewhiresandsuccessinobtainingcompetitive

researchfunding,alongwithpeerreviewsofresearchperformance(Anderson1986).The

outcomeofthedeliberationswasarankingofinstitutionsacrossfourpossiblepoints

(‘outstanding’,‘aboveaverage’,‘average’,and‘belowaverage’).Thesewerethenusedto

weighthenumberofstudentsenrolledineachinstitutiontodeterminefundingoutcomes.

Althoughthecategoriesinformingthefirstexercisewererapidlyinstitutionalizedas

criteriaintheallocationoffutureUGCgrants,theirpreciseweights,benchmarks,

standards,andmechanicsofconsiderationwerenotinitiallyclear.Indeed,asJohn

Phillimorewrote,theimpactoftheassessmentwas“moredivisiveandpronouncedthan

themoregeneralfundingcutsimposedsince1981[…]Miserywillnolongerbeshared

equally”(Phillimore1989:258–59).

Oppositionnotwithstanding,policyexpertsandgovernmentofficialspressedfora

refinement,expansion,andre-implementationofevaluations.Thisfirstpilotassessment

wasfollowedbythemoreexactingResearchSelectivityExerciseof1989.Unlikethe

previousadhocexercise,the1989iterationmadealargerportionofthegovernment’s

fundingdependentontheoutcomeoftheevaluation(LeeandHarley1998).Italso

expandedthebreadthoftheassessment.Insteadofassessingonlythefivemostprominent

publicationsforeachcost-center,thenearlythreehundredmembersoftheadvisory

groupsandassessmentpanelsoftheResearchSelectivityExercisewereexpectedto

13

produceaconsensusaboutthequalityofeachunitofassessmentonthebasisofinformed

peerreview,dataonthenumberofpublicationsofeachdepartment,informationon

incomefromextramuralresearchgrants,dataonthenumberofresearchstudentships,and

critically,detailedbibliographicinformationofuptotwopublicationsfromeachfull-time

academic(JohnesandTaylor1992:70).

Sincethisearlyexercise,researchevaluationsgrewinsize,complexity,and

intensity.TheResearchSelectivityExerciseof1989wasfollowedbyfouradditional

evaluationsin1992,1996,2001and2008underthenameofResearchAssessment

Exercise,itselfsubstitutedbytheResearchExcellenceFrameworkof2014and2020.Allin

all,Britishacademicswhostartedtheircareersintheearly1980shaveexperiencedseven

oftheseexerciseswithrulesandstandardsthatchangeundereachiteration.[]Thefact

remains,however,thatsincetheintroductionoftheResearchAssessmentExerciseof1992,

Britishscholarshaveseentheirworkevaluatedbypanelsofpeers,withtheoutcome

determininghowthegovernment’sunencumberedresearchfundsflowbackintotheir

institutions.

Whilenotdirectlyorovertly,theseevaluativeframeworkscertainlyputapriceon

researchbymeasuringitsqualityandlinkingittodistinctallocativeconsequences.Itis

thusnotoddtoseewhyscholarshaveoftenidentifiedRAE/REFaspartoftheincreaseda

marketizationofhighereducationintheUnitedKingdom(BrownandCarasso2013;

Hamann2016;Sayer2014).Whilea‘marketofideas’(Arrow1969)certainlyexistedprior

totheimplementationofthesesystems,theysurelyperfectedexchangebyproviding

rankings,comparisons,andperformanceindicatorsthatputativelysignaledconsensus

aboutresearchqualitywithindisciplinaryfields,acrossinstitutions,andtomanagerial

cadres.Ineffect,bytyingsomeportionofthestate’sfundingofinstitutionstotheir

employee’swork,theseexercisesinterveneddirectlyinthelabormarketsofdisciplinesby

makingsomescholarsmorevaluable(bothsymbolicallyandfinancially)thanothers.There

areseveralconcretedemonstrationsofthelinkagebetweenevaluationsandacademic

labormarkets.InmyinterviewswithBritishacademics,Ioftenheardfromearlycareer

scholarsabouthowtheywereinformallymadeawareoftheimportanceofbeing‘REFable’

14

beforesearchingforjobsinthemarket(InterviewA8,InterviewB7,(Leathwood2017;

Locke2014;Sayer2014)).Mid-careerandsenioracademicsechoedthisview,notingthat

searchcommitteeswouldoftenincludeascholarchargedwithlookingattheapplications

“fromaREFperspective”(InterviewB6).Moredirectly,thereissomeevidencethatREF

bothinstigatedso-calledtransfermarketsfor‘starresearchers’(Elton2000;Scott2003;

TalibandSteele2000)andhadothertemporaleffectsontheretentionofteachingand

researchstaffacrossevaluations(Metcalf,GreatBritain,andDepartmentforEducationand

Skills2005).Whicheverthecase,evaluationslikelydistortedlabormarketsand,potentially

throughthese,theorganizationofdisciplinaryfields.

HowdidresearchevaluationstransformBritishscience,ifatall?Toanswerthis

question,Ifocusontheevolutionandlabormarketdynamicsoffoursocialsciences:social

andculturalanthropology,economics,politicsandinternationalrelations,andsociology.

Thechoiceoftheseoverotherdisciplinesisthreefold.First,unlikelaboratory-basedor

otherinfrastructure-intensivefields,thesocialscienceshaverelativelylowfixedcoststhat

ostensiblytranslateintobothhighermobilityofresearchersbetweeninstitutionsandless

pressuretowardscentralization.Second,theproductionofknowledgeinthesocialsciences

ismorecloselyalignedtoindividualcreatorsratherthangroupsofcollaborators,while

preservingsomedegreeofcouplingtoorganizationalunits.Incomparisontootherfields,

thesocialscienceshavelowernumbersofauthorsperpublishedarticleandlowerlevelsof

recurringcollaborations(Parish,Boyack,andIoannidis2018).Simultaneously,thesocial

sciencesaremoreaccommodatingofdisciplinarystylesassociatedtoidentifiableunits,

departments,orinstitutions.WespeakoftheChicagoSchoolasareferencetoaparticular

variantofeconomicsinthesamewayaswethinkoftheManchesterSchoolasatraditionin

anthropology,Birmingham’sCenterforContemporaryCulturalStudiesasadistinct

intellectualenvironment,ortheEdinburghSchoolasshorthandforastrandofliteratureon

thesociologyofscienceandknowledge.Thistensionbetweenmostlyindividualized

intellectualproductionanddisciplinarytraditionsanchoredondepartmentalavatarsis

partlywhatmotivatestheemphasisonthematchingandsortingeffectsoflabormarkets

(cf.(Abbott2010)).Ifnotalignedwiththeevaluation’smetrics,otherwiseresilient

departmentalstylesmightchangethroughboththehiringofnewscholarstrainedunder

15

differenttraditionsorthemovementofscholarstootherinstitutions.Anempirical

manifestationofthisprocesswouldinvolvechangesinsomemeasureofepistemic

diversity,moderatedbylabormarketmovement,andassociatedtotheadoptionof

researchevaluations.

Third,anddespitetheirapparentsimilarities,thedynamicsofthesedisciplinary

fieldsvarygreatlyinbothepistemicandinstitutionalterms.Overtheperiodcoveredbythe

researchevaluations,forexample,economicswasfundamentallyreshapedbyprocessesof

internationalizationanddiffusionthatpeggedvaluetoaglobal,transnationalprofessional

standard(BackhouseandBackhouse2002;BackhouseandCherrier2017;Fourcade2006,

2009).Whileoutwardlyoriented,anthropologyandsociologywerenotexposedtosimilar

globalizingprocesses,thuspreservinglocallabormarkets,institutionalstyles,and

longstandingdepartmentalhierarchiesasfeaturesoftheirdisciplinaryorganization

(Halsey2004;Kuklick1991,2009;Urry2012).Politicsandinternationalrelations,onthe

otherhand,grewtocombinenationaltraditionsofpoliticaltheorywithempirical

approachesfrombothContinentalEuropeandNorthAmerica(Easton,Graziano,and

Gunnell2002).Thisvariabilitymatters,forwhilethesefourdisciplineshaverelatively

comparablemodesofknowledgeproductionandrequiresimilarinstitutionalinvestments,

theyembodydifferentwaysoforganizingtheacademe.

Labormarkets,mobility,andepistemicmatching

Toidentifytheeffectsofresearchevaluationsontheepistemicorganizationofscientific

fields,Iwillfocusonmobilityofsocialscientistsbetweenacademicinstitutionsasa

processofmarket-mediatedmatchingthatslowlychangesdepartmentalcompositionand,

throughthis,thestructureofeachdiscipline.Mostoftheliteratureonacademicmobility

focusesonthepredictorsandexperiencesofinternationalscholars.Researchoninternal

interinstitutionalmobilityisrathersparser,althoughexistingstudiespointtosomekey

factorsthatinfluenceshiftsinscholars’careers.Researchhasshown,forexample,a

marginalthoughpositiveeffectsofscholars’productivity,citation,anddegreeof

16

specializationonmobility(Crane1970),theirfutureproductivity(AllisonandLong1987;

Long1978),andoverallcareerprogression(GrimesandRegister1997;Leahey2007).

Thesestudiessuggesttwoclearsetsofpredictorsofinterinstitutionalmobility.Thefirst

concernsvariablesthatproxyvisibilityandproductivity.Briefly,previousstudiessuggest

thatacademicfollowsimilarpatterstogeneraljobchangers:scholarsoccupyinghigher

levelsmovemoreslowlythanthoseinlower-levelpositions(RosenfeldandJones1986).

Researchalsoshowsthatscholarswithmorecitationsandhigherlevelsofproductivity

(measured,forexample,throughnumbersandfrequencyofpublications)arepromoted

morerapidly,makingoftheseformsofprestigeandvisibilityanchorsthatreducemobility

bymovingindividualintohigherlevelpositions(AllisonandLong1987).

Asecondpredictorofmobilityisgender.Studiesindicatecleargendereddifferencesin

bothpatternsofrankpromotion(Long1978;Long,Allison,andMcGinnis1993).Overall,

womenbothsufferwagepenaltiesintheacademicworkplace(BarbezatandHughes2001;

Leahey2007;Leathwood2017)andexperiencehigherratesofmobilitythanmen.Authors

havepresentedseveralreasonswhyfemaleacademicsmightexperiencehigherlevelsof

jobmarketmobilitythantheirmalecounterparts.Oneargumentisthatfemalescholarsare

morelikelytochangeinstitutionsbecauseofspousalrelocation.Intheirstudyoftwo

cohortsofeconomists,forexample,HilmerandHilmer(2010)suggestthatwomenmove

fornon-pecuniaryreasonswhilemenuseinstitutionaltransitionsasamechanismfor

upwardrankandwagemobility.Otherargumentsinvolvecareerinterruptionsas

explainingdifferingpatternsofpromotion,earnings,andjobmarketstability.InLong,

AllisonandMcGinnis’(1993)nowclassicalwork,forexample,muchofthevariation

betweenmenandwomenderivesfromhowinterruptionsaffectrankadvancement,

particularlyamongearlycareerscholars,whichmaytranslateintohigherratesofmobility.

Thereisanimportantcaveatwiththeliteratureoninterinstitutionalmobility:becauseitis

disproportionatelyfocusedonAmericanacademics,itmaynotdescribemobilityratesfor

academicsworkingintheUnitedKingdom.ThedefiningfeatureoftheAmericanacademic

system(tenureasamilestonecareereventthatguaranteesdefactojobstability)issimply

17

lackinginBritain.Sincethestructural,system-widereformstoBritishhighereducationin

themid1980s,academicshavenothadthesametypeofcontractualstabilitythatscholars

employedinAmericaenjoy.Indeed,ifanything,Britishacademicsshouldbeclosertotheir

non-academicpeersintermsofbothlabormarketoutcomesanddynamics,shiftingthe

rolethatsomeoftheabovepredictorsplayinexplaininginterinstitutionaltransitions.For

example,statusandproductivitysignalsmightbeevenmoreimportantintheUnited

KingdomthanintheUnitedStates,giventhattheyfacilitatecommensurationinamore

fluid,activemarket.Similarly,recentproductivityrecordsmayhaveamoreimportantrole

inallowingformobility(and,inparticular,upwardmobility)giventheirtiestothe

incentivesofresearchevaluations.Giventhattheygroundexistingdiscussionsabout

academiccareers,however,Iretainvariablesconnectedtothesetwosetsofpredictors.

DataandMethods

Idrawonanoriginaldatasetcontaininginformationon44,956socialscientistsactive

acrossthe116publicuniversitiesintheUnitedKingdombetween1970and2018.The

datasetwascompiledusingthebibliographicrecordsofWebofScience’sSocialScience

CitationIndexforthisperiod.TherecordsobtainedfromSSCIincludedtitlesfor149,796

researcharticlesalongwithjournaltitles,datesofpublication,correspondingaddresses,

fundingdata,andcompleteabstractsfor76,898articles.ByqueryingWebofScienceforall

researcharticlespublishedinanthropology,economics,politicalscience,andsociology1

containingatleastoneauthorwithaBritishaffiliation,wewereabletogeneratea

longitudinaldatasetthatproxiescareermobilityforindividualsacrosseachevaluation

periodandfortheentiresector.

Thedataobtainedthroughthesequerieswasprocessedtostandardizeboth

researchers’namesandthoseoftheirinstitutionalaffiliations.Wefirsteliminated

duplicatesinthedata.Then,wemappednamesandinstitutionalaffiliations.Becauseof

1 An example of the WoS search query for sociology is: (AD=(england OR scotland OR wales or UK) AND SU=(sociology) AND WC=(sociology)) AND DOCUMENT TYPES: (Article OR Book OR Book Chapter) Timespan: 1970-2018. Indexes: SCI-EXPANDED, SSCI, A&HCI, CPCI-S, CPCI-SSH, BKCI-S, BKCI-SSH, ESCI, CCR-EXPANDED, IC.

18

changingarchivingstandardsintheSocialScienceCitationIndex,theassociationbetween

correspondingaddressesandlistedauthorsinthebibliographicdatawasnon-trivial:the

reportedaffiliationsdidnotalwaysmatchthenumberandorderofauthors,requiringusto

constrainthedatatoresearchersassociatedatleasttwiceacrossdifferentpublications

withthesameinstitutionoveranevaluationperiod.Becauseofchangesinthenamesof

variousinstitutionsofhighereducation,wealsostandardizedauthoraffiliationsusing

currentnomenclature.Forexample,ThamesPolytechnic,oneoftheoldestpolytechnic

institutionsintheUnitedKingdom,wasmatchedontoUniversityofGreenwich,itsname

since1992.Finally,wedroppedauthorsforwhichtherewasnoguaranteeofuniqueness.

Priortothemid1990s,theSSCIonlyrecordedauthorsusingtheirfirstnameinitials

followedbytheirlastname.ThismeantthatanimaginedAmandaSmithis

indistinguishablefromasimilarlyimaginedauthor,AdamSmith.Althoughitmighthave

beenpossibletouseinstitutionalinformationtodisambiguatedata,thiscouldhave

potentiallyintroducedfurthermistakesinthemodelbyconfoundingmovesacross

institutionsforthesameresearcher.Thedatathereforeonlycontainsinformationon

authorsforwhomthereislittleambiguityaboutbothidentityandinterinstitutional

mobility.Theresultisalongitudinaldatasetof16,531individualacademicsoverthefour

fields(2,208anthropologists,6,384economists,4,271politicalscientists,and3,668

sociologists).Havingthefullfirstnamesforaround80%oftheauthors,wewerealsoable

toinfertheirgenderusinginformationfromthe1991UKCensus.Theresultingsample

contained61.63%maleand38.37%femaleacademics.

ThedatasetthatIuseisthusadmittedlyimperfect.Firstly,itexcludesbooksand

otherformsknowledgedissemination.Second:becauseitreliesontheinstitutional

affiliationasreportedonarticles,itisaffectedbybothatemporallagforcareertransitions

andself-reporting.Scholarsonsabbaticalleaveoronvisitingappointments,forexample,

mayhavepublishedundertheirtemporal,ratherthanpermanentaffiliation,creatingthe

appearanceofmobilitywithoutanunderlyingchangeinemployment.Similarly,owingto

unevenpublicationlags,thedatadoesnotallowstudyingthe‘transfermarket’directly:we

cannotascertainwithclaritywhentransitionsoccurredbutonlyifascholarchanged

institutionsbetweenevaluationperiods.Theoutcomevariableis,indeed,quitecourse

19

takingavalueof1ifascholaridentifiablychangedinstitutionsbetweenperiodsand0if

shedidnot.Therearealsosomeassumptionsaboutscholarsanddisciplinesthatare

relativelyunavoidable:becausethesearchquerythatgeneratedthesourcedatahingeson

theclassificationofjournalsusedbyWebofScience,itdoesn’tmapneatlyandperfectlyto

departmentalstructuresandaffiliations.(Arandomsampleof200academicsfromthe

datasetsuggestedan89%accuracyofdepartmentalaffiliation,a96%accuracyofmobility,

anda94%accuracyinassignedgender.)Lastly,sinceweconstrainedtheanalysisto116

institutionsofhighereducation,ourdatasetdoesn’tcovertransitionsfromacademiainto

non-academicinstitutionsandvice-versa.Institutionsnotrecognizedashighereducation

providerswerelabeled‘Other’anddonotfeatureintheanalysis.

Thedataisnonethelessrevealingforotherreasons.Inadditiontoprovidingan

exceptionallydetailed,ifimperfect,viewofacademiclabormarketsintheBritishsocial

sciences,relyingonbibliographicdataallowsstudyinghowindividualscholars’intellectual

productionchangedovertimeandwasmatchedbytheirdepartmentsandfields.Forthis,

thispaperreliedontwomeasuresoftextualsimilaritythatallowedcomparingscholarsto

theirhomeinstitutions,departmentswithindisciplines,andevaluatingtheevolutionof

academicfieldsovertime.ThesemeasuresareallbasedonaLatentDirichletAllocation

topicmodelofthecorpusofpaperabstractsforeachdiscipline(Blei,Ng,andJordan2003;

DiMaggio,Nag,andBlei2013;MohrandBogdanov2013).Takingthecollectionofall

abstractsforeachofthe4socialscience,wedeterminedfour40-topicsolutionsthatneatly

describedmostofthekeythemescoveredbyarticles(and,ostensiblyresearchers’

intellectualagendas)overtime.The40-topicsolutionwasarrivedatiteratively,by

examiningtheheuristiccoverageofmodelsrangingfrom15to70topics(Karelland

Freedman2019).Thesefour40topicsolutions,correspondingtothedifferentdisciplines,

werethenusedtodescribefirst,theaggregateperiodicintellectualproductionofeach

subject/institutionasa40-dimensionvector,andsecondly,theproductionofindividual

scholarswithrespecttotheirinstitutionsofemployment.Anexampleoftopicsforthefour

disciplinesisprovidedinfigure1.

20

Figure1.Selectionoftopicsforthe4disciplines,1970-2018Anthropology

Topic29 archaeologi;data;develop;record;past;review;interpret;issu;discussTopic30 network;structure;social_network;commun;reserve;ti;central;social;

measur;timeTopic31 anim;dog;owner;behavior;attitud;human;emot;pet;hors;cat

EconomicsTopic29 optim;process;option;time;function;price;stochast;method;deriv;dynamTopic30 inform;contract;agent;incent;effici;optim;auction;mechan;effort;costTopic31 polici;reform;market;govern;regul;privat;service;competit;sector;uk

PoliticsandInternationalRelationsTopic29 parti;issu;left;elector;posit;polit;polit_parti;support;polici;ideologTopic30 public;tax;cost;spend;budget;govern;effect;polici;expenditure;fundTopic31 labour;conserve;polit;british;parti;leader;leadership;govern;

british_polit;britainSociology

Topic29 class;culture;music;bourdieu;field;habitu;middl_class;social;capit;distinct

Topic30 knowledge;scienc;technologi;genet;practice;scientif;human;power;form;natur

Topic31 manag;worker;labour;profession;employ;service;industry;organis;organ;employe

Thefirstmeasureallowedustoassessthedegreeofsimilaritybetween

departments:calculatedasaEuclidiandistance,twodepartmentshavingexactlythesame

researchprofile(thatis,thesamedistributionacrossthe40discipline-specifictopics)will

haveaminimaldistanceofzero.Similarly,twodepartmentsfocusingonentirelydifferent

topicswillhaveamaximaldistanceof1.Thecollectionofnormalizeddistancesbetweenall

pairsofdepartmentscanbeusedtoconstructagraphthatcapturesinformationonthe

structureofeachdisciplinaryfield(figure2).Thisgraphthenallowsustoidentify

departmentsthataremorestructurallycentraltothediscipline.Themeasureof

betweennesscentrality(Leydesdorff2007;WhiteandBorgatti1994),inparticular,provide

aproxyfordepartmentaltypicality:departmentsorinstitutionsmorecentraltothegraph

willsharegreaterthematicfeaturesandwillbemoresimilartoeachother,particularly

whencomparedtoinstitutionswithalowerdegreeofbetweennesscentrality.Importantly,

21

thedistributionofdistancesalsoallowsvisualizingchangestothetypicalitywithinthe

field:afieldwithveryloweraveragedistanceswillbepopulatedbymoresimilar

departmentsthanonewithhigheraveragedistances.

Figure2.GraphrepresentingtheorganizationofBritishsociology,1998-2004

Thesecondmeasureisbasedonasimilarlogic.Inthiscase,weestimatedthe

differenceintopicmodelsforeachauthor-departmentcouplebyfirstcalculatingthemodel

forthescholar’sdepartmentwithouthercontributions,tothencalculatethetopical

distributionforherworkovertheperiod.Wethendeterminedthecosinesimilarity

betweenthevectorrepresentingeachscholar’spublicationsandthevectorrepresenting

theworkoftheremainderofherdepartment.Thisproducedametricoftherelative

distinctivenessfortheworkofeachsocialscientist.Ascholarwhoseworkisidenticalto

thatoftherestofthedepartmentwillhavearelativedistinctivenessof1whereasascholar

whoseworkiscompletelyorthogonaltothatofthedepartmentwouldhaveaminimal

relativedistinctivenessof0.

22

Thebinaryoutcomevariableinthisstudyisinstitutionalmobility,measuredasan

andidentifiablechangeinascholar’saffiliationacrossperiodsandtakingvalues.Formally,

theoverarchingmodelcanbeexpressedas

Pr(MOBILITYit)= b0+b1AGEit+b2GENDERit+B1(PRESTIGEit)+B2(PRODUCit)+

b3SCOREit+b4DEP_TYPit+b5DISTit+DISC+PERIOD+Uit+Eit

Theidentifyingvariableintheregressionwastheindividualscholarwitheachperiod

servingasthetimevariable.Tocontrolforidiosyncraticvariations,allmodelsincluded

fixedeffectsforperiodsanddisciplinesbutandworkedwithrandomeffectsatthelevelof

individualscholarstoaccountforvarianceinthepaneldata.

Theprobabilitythatascholarmovesinstitutionsbetweenperiodswasanalyzed

withrespecttosixdistinctmodels.Models1and2provideabaselinethatreflectsvariables

identifiedintheexistingliteratureonacademiclabormarketsaspredictorsofjobchange,

namely,individualproductivity(measuredthroughthenumberofarticlespublished

cumulativelyandoverthemostrecentperiodbyeachscholar),prestige(measured

throughbothcumulativeandwithrecentcitations,successatobtainingextramural

funding,aswellasaverageimpactfactorofthejournalswhereeachscholarpublished),and

demographicfactors(age,approximatedasthetimethatthescholarhasbeenactiveinthe

field,andattributedgender).

Models3to5includevariablesdirectlyrelatedtotheexistenceofresearch

evaluations.Model3addstothebaselineanalysisbyintroducingtheinstitution’smost

recentoutcomeintheRAE/REFasawayoftestingforthecontributionsoftheseexercises

asaformofstatusthataffectsthemobilityofscholars.Ifevaluationoutcomesconstitutea

formofcapitalthatcanbetransformedintomobility,thenahigherstandardizedgrade

oughttotranslateintohigheroddsoftransitioningbetweeninstitutions.Model4teststhe

relationbetweenmobilityanddepartmentaltypicality.Forresearchevaluationstohave

isomorphiceffectsonthedisciplinaryfield,moretypicaldepartmentswouldhavetosee

23

lesslossestotheirranksthanmoreatypicaldepartmentsthat,overtime,would

increasinglyconformtoadisciplinarystandard.Model5testsanindividualscholar’s

distinctivenesswithrespecttoherinstitutionasapredictorofmobility.Finally,model6

includesbothdepartmentaltypicalityandscholar’sdistinctivenessbutaddsanadditional

variablethatspeaksdirectlytoideasaboutevaluationsasinstrumentsofepistemic

disciplining:itconsiderstheeffectsofbeing‘structurallycoherent’withinthelogicofthe

evaluation(Zuckerman1999).AfeatureofBritishresearchevaluationsisthattheyoperate

on‘unitsofassessment’ratherthanindividualdepartments.Thus,institutionsdecidehow

scholarsaresubmitted:ananthropologistworkinginamedicalschool,forexample,canbe

partoftheinstitution’ssubmissionfortheanthropologyassessmentpanel,reflectingthe

qualityofthedisciplineintheuniversityasawholeratherthaninoneofitsdepartments.

Whendepartmentsaresmallorexpertisearedistributedacrossunits,thismayleadto

scholarsbeingsubmittedtoassessmentpanelsthatarejustproximallyrelatedtotheir

mainareaofresearch.Forexample,aninstitutionlackingadepartmentofanthropology

maysubmititsanthropologistsassociologists.Ifdiscipliningdevices,researchevaluations

wouldmakethisstructuralincoherenceintoamotiveforchangingjobsorresearch

avenues.Indeed,wewouldexpectscholarswhoarenotstructurallycoherenttoshow

higherratesofmobilityasevidenceofdisciplinarymatching.

Table 1 – Summary Descriptive Statistics

count mean sd min max Change of affiliation 79284 .1312749 .3377029 0 1

Age (log) 79284 1.092411 1.163642 0 3.7612 Gender (1=female) 47471 .3508036 .4772266 0 1

Number of publications (log)

79284 1.042002 .5788998 0 4.89784

Cumulative publications (log)

79284 1.289131 .7520639 0 5.209486

Number of citations (log) 79284 .7700703 1.150336 0 6.993933

Cumulative citations (log) 79284 .8885759 1.274042 0 7.731931

24

Average journal impact factor

79284 .7025544 .8815296 0 7.863

Cumulative journal impact factor

79284 .9328543 1.152946 0 13.0829

UK Funding 79284 .1367111 .3435442 0 1 European Union Funding 79284 .0212527 .1442265 0 1

Departmental typicality 60427 20.26517 5.389539 2.602462 29.48124

Scholar’s relative distinctiveness

44048 .1320621 .0491929 0 .8313711

Structural incoherence 79284 .4694516 .4990691 0 1

Results

Theresultsacrossmodelssuggestoneintuitivefinding:agehasanegative

associationtointerinstitutionalmobility.Similarly,andconsistentwiththeliterature(Long

1978),models2-6alsoshowapositiveeffectofgenderonmobility:themodelssuggest

thatwomenareupto1.34times(model6)morelikelytomoveinstitutionsbetween

evaluationperiodswhencomparedtomen.Variablesthatproxyprestigeandfieldvisibility

(numbersofpublicationsandcitations)suggestaslightlymorecomplicatedinteraction

betweenrecentandcumulativeproductivity.Cumulativemeasuresofbothproductivity

andcitationsaresimilartoage,inthesensethattheyincreasetemporallybyasheer

processofaccumulation.Thesewereincludedinthemodelalongwiththesamemeasures

forthelatestperiodtoassesstheinterplaybetweenaccumulatedprestigeandrecent

visibilityinthefield(Leahey2007).Theimplicationsofthemodelsaresurprising,insofar

astheysuggestveryweakassociationsbetweeninterinstitutionalmobilityandthe

publicationsofindividualscholars.Whilethenumberofpublicationscontributespositively

totheoddsofmobility,itisoffsetbythescholar’scumulativepublicationrecord.Since

therewillalwaysbeasmany,ifnotmore,cumulativepublicationsthanrecentpublications,

theeffectsoftheseinmodels2to4willvaryfrompositivetonegativeasaccumulated

publicationsandcitationsweighmorethanthosegeneratedinthemostrecentevaluation

period.

25

Table 2 – Panel logistic models with individual scholar random effects

Model 1 Model 2 Model 3 Model 4 Model 5 Model 6 Interinstitutional mobility Age (log) -0.158*** -0.135*** -0.120** -0.0802* -0.181*** -0.186*** (-5.93) (-4.08) (-2.64) (-2.22) (-3.94) (-4.05) Number of publications (log)

0.0336 0.264** 0.448*** 0.438*** 0.221 0.236

(0.48) (2.75) (3.38) (4.12) (1.66) (1.77) Cumulative publications (log)

0.0602 -0.221* -0.343* -0.374** -0.134 -0.120

(0.75) (-2.05) (-2.32) (-3.14) (-0.89) (-0.80) Number of citations (log) 0.117 0.170 0.355* 0.240* 0.356* 0.360*

(1.53) (1.66) (2.42) (2.13) (2.53) (2.55) Cumulative citations (log) -0.141 -0.174 -0.404** -0.248* -0.355* -0.359*

(-1.86) (-1.71) (-2.77) (-2.21) (-2.53) (-2.55) Average journal impact factor

0.00602 -0.0965 -0.0350 -0.0804 -0.132 -0.127

(0.09) (-1.24) (-0.36) (-1.01) (-1.45) (-1.39) Cumulative journal impact factor

-0.0795 -0.0146 0.0243 -0.0240 -0.0149 -0.0102

(-1.37) (-0.22) (0.30) (-0.36) (-0.20) (-0.13) UK funding -0.0482 -0.0464 -0.0417 -0.0617 -0.0605 -0.0460 (-0.74) (-0.63) (-0.45) (-0.82) (-0.75) (-0.57) European Union Funding

0.719* 0.934* 0.979* 0.915* 0.696 0.654

(2.24) (2.56) (1.99) (2.46) (1.77) (1.67) Anthropology 0 0 0 0 0 0 (.) (.) (.) (.) (.) (.) Economics -0.0619 -0.0780 -0.0333 0.180 -0.125 0.0966 (-0.99) (-0.98) (-0.29) (1.53) (-1.13) (0.65) Politics and IR 0.0298 0.0167 0.0357 0.190 0.0239 0.247

(0.45) (0.20) (0.32) (1.76) (0.21) (1.81) Sociology 0.262*** 0.177* 0.131 0.151 0.176 0.148 (3.94) (2.16) (1.13) (1.68) (1.55) (1.30) 1986 0 0 (.) (.) 1989 0.133 0.185 (1.54) (1.38) 1992 0.128 0.0478 0 0 0 (1.50) (0.36) (.) (.) (.) 1996 0.198* 0.144 0 0.416* 0.0607 0.250 (2.54) (1.22) (.) (2.15) (0.12) (0.50) 2001 0.250*** 0.211 0.113 0.583** 0.199 0.436 (3.33) (1.92) (0.92) (2.92) (0.41) (0.86)

26

2008 -0.303*** -0.517*** -0.673*** -0.0796 -0.543 -0.287 (-4.41) (-5.19) (-5.83) (-0.39) (-1.11) (-0.56) 2014 -0.438*** -0.562*** -0.783*** -0.146 -0.607 -0.380 (-6.10) (-5.45) (-6.31) (-0.71) (-1.24) (-0.74) Male 0 0 0 0 0 (.) (.) (.) (.) (.) Female 0.272*** 0.188** 0.241*** 0.283*** 0.296*** (5.23) (2.75) (4.31) (4.39) (4.59) Standardized evaluation rating

-0.0684*

(-2.11) Departmental typicality -0.0304*** -0.0233*

(-3.50) (-2.15) Scholar’s relative distinctiveness

1.385* 1.146*

(2.41) (1.98) Structural incoherence 0.333*** (5.03) Constant -0.640*** -0.459*** -0.506*** -0.476** -0.635 -0.656 (-8.61) (-4.22) (-3.71) (-2.66) (-1.27) (-1.31) lnsig2u _cons 0.339*** 0.196 0.416* 0.330** 0.144 0.131 (4.14) (1.71) (2.51) (2.59) (0.78) (0.69) N 24745 15362 9393 13502 9456 9456 t statistics in parentheses * p < 0.05, ** p < 0.01, *** p < 0.001

Perhapsthemoreinterestingfindingscomefrommodels3to6,whichinclude

measuresthatoriginateintheevaluationexercisesandspeakofthestructureofthe

academicfield.Model3,forexample,suggestthatbeinginaunitofassessmentthatwas

highlyratedinthepreviousevaluationwillhaveaslightnegativeoninterinstitutional

mobility.Thisfindingisconsistentwiththebroaderliteratureonmobility,whichsuggests

lowertransitionratesforworkersinhighstatuspositionswhencomparedtocohortpeers

(Fuller2008;RosenfeldandJones1986).Supplementalanalysisthatlookedattransition

probabilitiesacrossdifferenttiersofuniversitiesdidnotprovideevidenceofstatus-based

matching(theoddsofmovingfrom,say,aformerpolytechnictoaneliteOxbridge

institutionandvice-versaaresimilar).Thereis,however,aslightnegativeassociation

betweenthestandardizedscoreofaninstitutioninthemostrecentevaluationand

academicmobility.Scholarsatinstitutionswiththehighestpossiblestandardizedscores

(z=1)arejust7%lesslikelytochangeaffiliationsthanscholarsatmodalinstitutions(z=0).

27

Measuresoffieldstructuretestedinmodels4to6providefurtherinsightsintohow

evaluationssortscholarsacrossdepartmentsand,indoingso,transformtheorganization

ofthedisciplinaryfield.Inmodel4,departmentaltypicalityisshowntohaveanegative

associationtomobility.Scholarsatverytypicaldepartmentsarelesslikelytochange

institutionsacrossperiods.Forthemostatypicaldepartment(departmentaltypicality=

2.6),theprobabilityofmovingjobsisreducedby7.6%withrespecttoabaselinetypicality

ofzero;scholarsworkinginthemosttypicaldepartmentinthedataset(departmental

typicality=29.48),bycomparison,willbe60%lesslikelytochangejobswithrespecttothe

samebaseline.Scholarlydistinctivenessalsoplaysintopotentialepistemicmatching.As

model5shows,scholarswhoseworkisveryclosetothatoftheirdepartmentalcolleagues

aredisproportionatelymorelikelytochangejobs(maximally,3.16times)thanthose

whoseworkiscompletelyunlikethatoftheirdepartmentalpeers.Models6explorescases

ofscholarsnotsubmittedtotheresearchevaluationsintheirfieldsofpublication.This

ratherartificialmeasuregivesasenseofthestructuralincoherencewithrespecttothe

disciplinarylogicsofthefieldandtheiroperationalizationintheresearchassessments.

Surprisingly,themodelsuggeststhatscholarsinpositionsofstructuralincoherenceare

almost40%morelikelytochangeinstitutionsacrossevaluationperiodsthantheir

structurallycoherentpeers.(Supplementalworkusingstructuralincoherenceaspartofa

difference-in-differencesanalysissuggestsastatisticallysignificantcausaleffectofnot

beingevaluatedinthecoherentdisciplineandinterinstitutionalmobility.)

Inaggregate,themodelstestedusingthecareerdatafortheBritishacademics

suggestaformofsortingbetweenandacrossinstitutionslinkedtohowthedisciplinary

fieldisorganized.Thequestionremains,however,astowhethertheorganizationofthe

fieldvariedovertime.Forthis,evidenceisprovidedbygraphingtheaveragenormalized

distancebetweendepartmentsovertimetogaugeapotentialshiftintheorganizationof

thedisciplinaryfields.Asfigure1shows,thefractionalpolynomialfitsofthetopical

distancesbetweendepartmentsfromthelate1980sto2018showareductioninthe

averagedissimilaritybetweeninstitutionsacrossalldisciplines.

28

Figure3-Fractionalpolynomialfitoftopicaldistancesbetweendepartments,acrossalldisciplines

Discussionandconclusions

CoupledtometricsthatdescribethepublicationsofBritishsocialscientistsoverthelast

halfcentury,theapproximatethoughdetailedcareerdatausedinthispapersuggeststhe

presenceofconnectionsbetweeninterinstitutionalmobilityandchangesinthestructureof

disciplinaryfieldsthataredirectlytiedtotheimplementationofresearchassessments.

Sincetheirintroductionin1986andexpansionthroughouttheinstitutionallandscapeof

Britishhighereducation,researchevaluationshavetransformedbothacademiclabor

marketsandthestructureofthedisciplinaryfield.Figure1providestellingevidenceof

whatthislookslike:ashiftinthethematiccompositionofdepartmentsacrossall

disciplines.Asapointofcomparison,thetopicalorganizationofBritishanthropologytoday

issimilar,inthisstructuralsense,tothatofBritisheconomicsinthelate1980s.

29

Thesefindingslendsupporttootherstudiesthathavedocumentedaprofound

transformationofBritishsocialsciencesinthelasthalfcentury.Economicsis,perhaps,the

mostparadigmaticexampleofthisprocess.Fromhavinghadahigherconcentrationof

specialistdepartmentspriortothe1980s,thedisciplinehasbecomemoreisomorphicand

transnationalinthepastthirtyyears(Fourcade2009).Anindicatorofthistransformation

maywellbehowheterodoxtraditionsofeconomicthoughtfaredinthischangingfield:as

FrederickLeeandcolleaguesargue(Lee,Pham,andGu2013),thesteadydeclineinthe

hiringofheterodoxeconomistsbyeconomicsdepartmentsandan“increasingnumberof

departmentswithonlymainstreameconomistsonstaffcombinedwiththeprogressive

declineinheterodoxsubmissionstotheeconomicsRAEpanelshascreatedtheall-too-

accurateimpressionthateconomicsisadisciplinewithonlyasingleparadigmof

mainstreameconomics”.Thesechangesarenotuniquetoeconomics.Britishsociologyhas,

too,experiencedanimportantshiftinitsdisciplinarystructure.Althoughstillintellectually

diverse,a2010internationalbenchmarkingexercisenoteddistinctivestrengthsinBritish

sociology(socialandpolicy,healthandmedicalresearch,scienceandtechnologystudies,

economicsociology,culturalsociology,genderandwomen’sstudies,andsocialtheory)that

aretopicscommonlyfoundacrossorganizationalunits.Thiscontrastswithsociology’s

earlierdisciplinaryorganization,wherespecializationwasmoreoftenthannotthedefining

featureofdepartments(Savage2016;Soffer1982).

Theslightmoveawayfromthematicallydifferentiateddepartmentstoincreasingly

similarunitsthatechodominantdisciplinarylogicsismoreconvincingwhenwetakeinto

accountthemodelofinterinstitutionalmobility.Disciplinarytransformationswere

modulatednotonlybyprocessesofinternationalizationandendogenousdisciplinary

change,butalsobyhowexternalresearchassessmentsshapedthelabormarketsforUK

socialscientists.Here,themostnotableindicatorthatresearchevaluationsimpingedon

patternsofinterinstitutionalmobilityisprovidedbythevariablethatproxiesthestructural

incoherenceofascholarundertheassessment.Ifascholarisemployedataninstitution

whereherworkisnotassessedintherelevantdiscipline(forexample,ananthropologist

bypublicationsworkingataninstitutionthatisn’tassessedinanthropology),shewillhave

30

apredictedrateofmobility40%higherthanherstructurallycoherentpeers.If

interinstitutionalmobilitywerenotaffectedbyresearchassessments,thiscompletely

artificial,organizationallyboundvariablewouldhavenoeffectsontransitionprobabilities.

Thetwoothervariablesthatencodepositionsinthefieldaresimilarlyrevealing.

Becausedepartmentaltypicalityandscholarlydistinctivenessarebothassociatedto

interinstitutionalmobility,themodelsuggeststhatindividualsaresortedinparticularly

patternedwayswithintheirdisciplinaryfields.Indeed,theevidencefromthisstudy

suggestsapatternnottoodissimilarfrommarket-basedmatching(Roth1986;Rothand

Peranson1999;RothandSotomayor1992),wherebythemovementofscholarsacross

institutionsreflectsspecificdisciplinarycomplementarities:institutionsarematchedonto

institutionsthatmakethemlessatypicalwhileretainingthosethatpreservetheir

centrality.Researchevaluationsareeffectivelymarketdevices(Callon,Millo,andMuniesa

2007;Muniesa,Millo,andCallon2007)framingandactingnotonknowledgeitself(thisis

notacaseofthemarketizationofclaimsorthecommoditizationofscience)butonthe

often-unattendedlabormarketsthatslowlytransformdisciplinaryformation.Fromthis

perspective,thereductioninthematicdiversityacrossthedifferentfieldspresentedin

figure1canbeunderstoodasresultingfromamarket-mediatedformofmimetic

isomorphism(DiMaggioandPowell1983)thatemergesnotfromtheactionsofindividuals

reorientingtheirresearch(thatis,shiftingfromonetopictoanother,asemphasizedinthe

researchevaluationliterature(Rijckeetal.2016))butthroughtheirinstitutionalmediation

inlabor.

Whatissimilarlytellingfromthemodelsisthatresearchevaluationsdidn’tonly

perturblabormarkets,actingasexogenousforcesshapinganotherwiseself-containedbut

wellboundedanddefinedcompetitivefield.Asnotedabove,researchevaluationsarebest

thoughtofasconstitutiveoffieldsofcompetitionandcommensurationthatwouldhave

otherwisenotexisted.Thisespartlyvisibleinthepredictedprobabilitiesofmobility,which

suggestvariationsinpredictedinterinstitutionalmobilitythroughoutevaluations,risingin

thefirstiterationsoftheResearchAssessmentExercisefromtheobserved1970sbaselines

tothenfallwithlateriterationsoftheRAEandtheimplementationoftheResearch

31

ExcellenceFramework(Figure3).Whileitisimpossibletoinferanythingfromthese

predictedprobabilities,supplementalanalysesusingarandomsampleof200scholarswho

changedinstitutionsatleastonce(100men,100women)suggestthatmen,particularly

thoseinhigherstatusinstitutions,wereabout1.3timesmorelikelytotransitioninto

higherranks(usingmobilityasameansforpromotion)thatwomenwhooftenmoved

horizontallyintosimilarranks.Inthecontextofthebroaderliteratureonlabormarket

dynamics,thishintsattheexistenceofsomeformofcapitalthat,tiedtostatusthepositions

oforganizations,grantsgreaterdegreesofmobilitytosomescholarsoverothers.The

criticalpieceofdatamissinghere,ofcourse,pertainstosalaries.Unfortunately,thetypeof

analysesthatwouldrevealthepresenceofformsofcapitalthat,intersectionally,modulate

ratesofmobility,earnings,visibility,and‘professional’successinthefieldwouldrequire

detailedwagedataforBritishacademicswhichissurprisinglysparseanddifficultto

obtain.

Figure4.Predictedprobabilitiesofmobilityforscholarsacrossthe4disciplines

.25

.3.3

5.4

.45

Pr(P

chan

ge=1

)

1986 1989 1992 1996 2001 2008 2014Period

Anthropology EconomicsPolitical Science Sociology

Predictive Margins of discipline with 95% CIs

32

Thisstudyreveals,however,someoftheboundaryconditionsthatmakethe

quantificationofknowledgeafactorforepistemicchange.TheBritishcasehighlightsat

leastthreefeatures:first,amobilemarketwhereindividualshavehigherpossibilitiesof

changinginstitutions.Thelackoftenureprotectioninthehighereducationsectormakes

interinstitutionalmobilitymorelikelythaninsettingswhereacademicshavehigherlevels

ofcontractualstability.Second,theevaluation’sfocusonunitsratherthanindividuals

createsaloosercouplingbetweennotionsofdisciplinaryqualityandascholar’sresearch

decisions,leadingtoamoreprominentroleofthelabormarket.Ifevaluationsplaced

incentivesdirectlyonindividuals,weshouldexpectmatchingtobelessrelevantsinceit

wouldbeintheinterestofacademicstoconformtothedominantconceptionsofvalueof

theassessmentregardlessoftheiraffiliation.Becauseoftherelativeambiguityand

decenteringoftheBritishassessmentsystem,thereremainssomeflexibilityinscholars’

preferencesthatallowsforastrongercommitmenttotheirresearchprograms.Changein

thefieldcomesnotfromtaskrealignmentsoradirectpunishingofinterdisciplinarity,but

ratherbyhowthedepartmentalsystemrearrangesarounddisciplinarynormsand

conceptions.Thishighlightsathirdfeature:centralization.Scholarsatleastintheperiod

coveredbythispaperwerenotconcernedwithmultipleevaluativeranking.Whenthese

existed(asineconomics,whereaninternal‘ranking’ofjournalsmattersfordecisionsof

contractualrenewalandpromotion),theywereentirelyalignedwiththeevaluative

frameworksoftheassessment(InterviewerB.7).Centralizationmadetheseassessments

powerfulnotbyforceofcoercionbutbythevirtueofuniqueness.

Thispaperalsosignalsanimportantconstrainttohowfastandhowintensely

researchevaluationscantransformepistemicfields.Intheirclassicalworkon

interinstitutionalmobility,AllisonandLong(1987)remindusthatthelabormarketonly

goessofarinexplainingcareeroutcomesforacademics.“Movesarevoluntary”,theynote,

andscientistsmaychoosetheirjobsonthebasisofcomplex,individualcriteria.Ifa

matchingprocess,then,thetypeofdistortionsintroducedtolabormarketsbyquantifying

knowledgeareimperfectinthesensethatdecisionstomove,retain,dismiss,promote,and

remaininpositionsarenotonlydrivenbythepecuniaryincentivesofresearchfunding

allocationandstatuscompetitioninthefieldbut,asimportantly,bytheunevenand

33

idiosyncraticpatterningoforganizationaldispositionsinthefield.Distinctorganizational

cultureswillmediate,withdifferentintensities,fociofattention,andlogicsofpractice,how

evaluationisinterpretedandmadesensible.Forsomeinstitutions,itmaycreate

environmentswherethecognitiveandeconomiccostsofsearchingandpursuingother

positionsisacceptable;forothers,evaluationsmaysimplynotmatterbecauseofhow

administratorsandscholarsarebufferedfromthelogicsofprecarityandausterityofthe

exercise.Asinotherlabormarkets,theresultsofthisstudyaresuggestiveofhow,on

average,researchevaluationschangefields.Howthisoccursinpracticewilldependonthe

organizationalecologyofthefieldandthevariabilityofcircumstances,resourcesand

culturesacrossinstitutions,featuresthataresimplynotcapturedbythedataathand.In

theend,however,theconclusionscold:quantifyingknowledgechangesourworldby

shifting,evenisslightly,howwebuildtheinstitutionsthroughwhichscholarshipis

produced.

34

References

Abbott,Andrew.2005.“LinkedEcologies:StatesandUniversitiesasEnvironmentsforProfessions.”SociologicalTheory23(3):245–274.

Abbott,Andrew.2010.ChaosofDisciplines.UniversityofChicagoPress.

Abramo,Giovanni,CiriacoAndreaD’Angelo,andAlessandroCaprasecca.2009.“AllocativeEfficiencyinPublicResearchFunding:CanBibliometricsHelp?”ResearchPolicy38(1):206–215.

Aistleitner,Matthias,JakobKapeller,andStefanSteinerberger.2018.“ThePowerofScientometricsandtheDevelopmentofEconomics.”JournalofEconomicIssues52(3):816–834.

Allison,PaulD.andJ.ScottLong.1987.“InteruniversityMobilityofAcademicScientists.”AmericanSociologicalReview643–652.

Anderson,Alun.1986.“ResearchGradingsStirEmotions.”Nature322(6077):299–299.

Arrow,KennethJ.1969.“TheOrganizationofEconomicActivity:IssuesPertinenttotheChoiceofMarketversusNonmarketAllocation.”TheAnalysisandEvaluationofPublicExpenditure:ThePPBSystem1:59–73.

Backhouse,RogerE.andRogerBackhouse.2002.ThePenguinHistoryofEconomics.PenguinUK.

Backhouse,RogerE.andBéatriceCherrier.2017.“TheAgeoftheAppliedEconomist:TheTransformationofEconomicssincethe1970s.”HistoryofPoliticalEconomy49(Supplement):1–33.

Barbezat,DebraA.andJamesW.Hughes.2001.“TheEffectofJobMobilityonAcademicSalaries.”ContemporaryEconomicPolicy19(4):409–423.

Barnes,Barry.1983.“SocialLifeasBootstrappedInduction.”Sociology17(4):524–545.

Barnes,Barry,DavidBloor,andJohnHenry.1996.ScientificKnowledge:ASociologicalAnalysis.UniversityofChicagoPress.

Berman,ElizabethPoppandDanielHirschman.2018.TheSociologyofQuantification:WhereAreWeNow?SAGEPublicationsSageCA:LosAngeles,CA.

Blei,DavidM.,AndrewY.Ng,andMichaelI.Jordan.2003.“LatentDirichletAllocation.”JournalofMachineLearningResearch3(Jan):993–1022.

Bloor,David.1991.KnowledgeandSocialImagery.UniversityofChicagoPress.

35

Bourdieu,Pierre.1988.HomoAcademicus.StanfordUniversityPress.

Brown,RogerandHelenCarasso.2013.EverythingforSale?TheMarketisationofUKHigherEducation.Routledge.

Burris,Val.2004.“TheAcademicCasteSystem:PrestigeHierarchiesinPhDExchangeNetworks.”AmericanSociologicalReview69(2):239–264.

Burrows,Roger.2012.“LivingwiththeH-Index?MetricAssemblagesintheContemporaryAcademy.”TheSociologicalReview60(2):355–372.

Callon,Michel,YuvalMillo,andFabianMuniesa.2007.MarketDevices.

Campbell,DavidFJandEliasG.Carayannis.2012.EpistemicGovernanceinHigherEducation:QualityEnhancementofUniversitiesforDevelopment.SpringerScience&BusinessMedia.

Clarke,CarolineA.andDavidKnights.2015.“CareeringthroughAcademia:SecuringIdentitiesorEngagingEthicalSubjectivities?”HumanRelations68(12):1865–1888.

Collins,Harry.2010.Gravity’sShadow:TheSearchforGravitationalWaves.UniversityofChicagoPress.

Crane,Diana.1970.“TheAcademicMarketplaceRevisited:AStudyofFacultyMobilityUsingtheCartterRatings.”AmericanJournalofSociology75(6):953–964.

Czaika,MathiasandSultanOrazbayev.2018.“TheGlobalisationofScientificMobility,1970–2014.”AppliedGeography96:1–10.

David,DanielandPetreFrangopol.2015.“TheLostParadise,theOriginalSin,andtheDodoBird:AScientometricsSapereAudeManifestoasaReplytotheLeidenManifestoonScientometrics.”Scientometrics105(3):2255–2257.

DeFraja,Gianni,GiovanniFacchini,andJohnGathergood.2016.“ProfessorialSalariesandResearchPerformanceinthe2014ResearchExcellenceFramework.”Voxeu.Org.(Http://Voxeu.Org/Article/Professorial-Salaries-and-Researchperformance).

DiMaggio,PaulJ.andWalterW.Powell.1983.“TheIronCageRevisited:InstitutionalIsomorphismandCollectiveRationalityinOrganizationalFields.”AmericanSociologicalReview147–160.

DiMaggio,Paul,ManishNag,andDavidBlei.2013.“ExploitingAffinitiesbetweenTopicModelingandtheSociologicalPerspectiveonCulture:ApplicationtoNewspaperCoverageofUSGovernmentArtsFunding.”Poetics41(6):570–606.

Easton,David,LuigiGraziano,andJohnGunnell.2002.TheDevelopmentofPoliticalScience:AComparativeSurvey.Routledge.

36

Education,DepartmentofandScience.1985.TheDevelopmentofHigherEducationintothe1990s.HMSOLondon.

Elton,Lewis.2000.“TheUKResearchAssessmentExercise:UnintendedConsequences.”HigherEducationQuarterly54(3):274–83.

Espeland,WendyNelsonandMichaelSauder.2007.“RankingsandReactivity:HowPublicMeasuresRecreateSocialWorlds.”AmericanJournalofSociology113(1):1–40.

Espeland,WendyNelsonandMitchellL.Stevens.2008.“ASociologyofQuantification.”EuropeanJournalofSociology/ArchivesEuropéennesdeSociologie49(3):401–436.

February13th,2018|Highereducation,Impact,REF2014,REF2021,Researchfunding,andResearchpolicy|0Comments.2018.“TheRAE/REFHaveEngenderedEvaluationSelectivityandStrategicBehaviour,ReinforcedScientificNorms,andFurtherStratifiedUKHigherEducation.”ImpactofSocialSciences.RetrievedJanuary21,2020(https://blogs.lse.ac.uk/impactofsocialsciences/2018/02/13/the-raeref-have-engendered-evaluation-selectivity-and-strategic-behaviour-reinforced-scientific-norms-and-further-stratified-uk-higher-education/).

Fourcade,Marion.2006.“TheConstructionofaGlobalProfession:TheTransnationalizationofEconomics.”AmericanJournalofSociology112(1):145–194.

Fourcade,Marion.2009.EconomistsandSocieties:DisciplineandProfessionintheUnitedStates,Britain,andFrance,1890sto1990s.Vol.38.PrincetonUniversityPress.

Frickel,ScottandNeilGross.2005.“AGeneralTheoryofScientific/IntellectualMovements.”AmericanSociologicalReview70(2):204–232.

Fuller,Sylvia.2008.“JobMobilityandWageTrajectoriesforMenandWomenintheUnitedStates.”AmericanSociologicalReview73(1):158–183.

Geuna,AldoandBenR.Martin.2003.“UniversityResearchEvaluationandFunding:AnInternationalComparison.”Minerva41(4):277–304.

Gill,Rosalind.2014.“Academics,CulturalWorkersandCriticalLabourStudies.”JournalofCulturalEconomy7(1):12–30.

Gingras,Yves.2016.BibliometricsandResearchEvaluation:UsesandAbuses.MITPress.

Gläser,JochenandGritLaudel.2016.“GoverningScience:HowSciencePolicyShapesResearchContent.”EuropeanJournalofSociology/ArchivesEuropéennesdeSociologie57(1):117–168.

Godin,Benoît.2004.MeasurementandStatisticsonScienceandTechnology:1920tothePresent.Routledge.

Godin,Benoît.2006.“OntheOriginsofBibliometrics.”Scientometrics68(1):109–133.

37

Grimes,PaulW.andCharlesA.Register.1997.“CareerPublicationsandAcademicJobRank:EvidencefromtheClassof1968.”TheJournalofEconomicEducation28(1):82–92.

Hacking,IanandJanHacking.1983.RepresentingandIntervening:IntroductoryTopicsinthePhilosophyofNaturalScience.Cambridgeuniversitypress.

Halsey,AlbertHenry.2004.AHistoryofSociologyinBritain:Science,Literature,andSociety.OxfordUniversityPress.

Hamann,Julian.2016.“TheVisibleHandofResearchPerformanceAssessment.”HigherEducation72(6):761–779.

Hammarfelt,BjörnandSarahDeRijcke.2015.“AccountabilityinContext:EffectsofResearchEvaluationSystemsonPublicationPractices,DisciplinaryNorms,andIndividualWorkingRoutinesintheFacultyofArtsatUppsalaUniversity.”ResearchEvaluation24(1):63–77.

Hicks,Diana.2012.“Performance-BasedUniversityResearchFundingSystems.”ResearchPolicy41(2):251–261.

Hicks,Diana,PaulWouters,LudoWaltman,SarahDeRijcke,andIsmaelRafols.2015.“Bibliometrics:TheLeidenManifestoforResearchMetrics.”NatureNews520(7548):429.

Hilmer,ChristianaandMichaelHilmer.2010.“AreThereGenderDifferencesintheJobMobilityPatternsofAcademicEconomists?”AmericanEconomicReview100(2):353–57.

Johnes,JillandJimTaylor.1992.“The1989ResearchSelectivityExercise:AStatisticalAnalysisofDifferencesinResearchRatingbetweenUniversitiesattheCostCentreLevel.”HigherEducationQuarterly46(1):67–87.

Karell,DanielandMichaelFreedman.2019.“RhetoricsofRadicalism.”AmericanSociologicalReview84(4):726–53.

Kuklick,Henrika.1991.TheSavagewithin:TheSocialHistoryofBritishAnthropology,1885-1945.CambridgeUniversityPress.

Kuklick,Henrika.2009.NewHistoryofAnthropology.JohnWiley&Sons.

Latour,Bruno.1987.ScienceinAction:HowtoFollowScientistsandEngineersthroughSociety.Harvarduniversitypress.

Leahey,Erin.2007.“NotbyProductivityAlone:HowVisibilityandSpecializationContributetoAcademicEarnings.”AmericanSociologicalReview72(4):533–561.

38

Leathwood,Carole.2017.“WomenAcademicResearchers:StillInterlopersintheUKAcademy?”Pp.227–242inThechangingroleofwomeninhighereducation.Springer.

Lee,FredericS.andSandraHarley.1998.“PeerReview,theResearchAssessmentExerciseandtheDemiseofNon-MainstreamEconomics.”Capital&Class22(3):23–51.

Lee,FredericS.,XuanPham,andGyunGu.2013.“TheUKResearchAssessmentExerciseandtheNarrowingofUKEconomics.”CambridgeJournalofEconomics37(4):693–717.

Lewis,JennyM.2015.“ResearchPolicyas‘CarrotsandSticks’:GovernanceStrategiesinAustralia,theUnitedKingdomandNewZealand.”Pp.131–150inVarietiesofGovernance.Springer.

Leydesdorff,Loet.2007.“BetweennessCentralityasanIndicatoroftheInterdisciplinarityofScientificJournals.”JournaloftheAmericanSocietyforInformationScienceandTechnology58(9):1303–1319.

Locke,William.2014.“ShiftingAcademicCareers:ImplicationsforEnhancingProfessionalisminTeachingandSupportingLearning.”

Long,J.Scott.1978.“ProductivityandAcademicPositionintheScientificCareer.”AmericanSociologicalReview889–908.

Long,J.Scott,PaulD.Allison,andRobertMcGinnis.1993.“RankAdvancementinAcademicCareers:SexDifferencesandtheEffectsofProductivity.”AmericanSociologicalReview703–722.

Lotka,AlfredJ.1926.“TheFrequencyDistributionofScientificProductivity.”JournaloftheWashingtonAcademyofSciences16(12):317–323.

Mennicken,AndreaandWendyNelsonEspeland.2019.“What’sNewwithNumbers?SociologicalApproachestotheStudyofQuantification.”AnnualReviewofSociology45.

Merton,RobertK.1973.TheSociologyofScience:TheoreticalandEmpiricalInvestigations.UniversityofChicagopress.

Metcalf,Hilary,GreatBritain,andDepartmentforEducationandSkills.2005.RecruitmentandRetentionofAcademicStaffinHigherEducation.London:DepartmentforEducationandSkills.

Mohr,JohnW.andPetkoBogdanov.2013.Introduction—TopicModels:WhatTheyAreandWhyTheyMatter.Elsevier.

39

Muniesa,Fabian,YuvalMillo,andMichelCallon.2007.“AnIntroductiontoMarketDevices.”TheSociologicalReview55(2_suppl):1–12.

Parish,AustinJ.,KevinW.Boyack,andJohnPAIoannidis.2018.“DynamicsofCo-AuthorshipandProductivityacrossDifferentFieldsofScientificResearch.”PloSOne13(1):e0189742.

Phillimore,A.J.1989.“UniversityResearchPerformanceIndicatorsinPractice:TheUniversityGrantsCommittee’sEvaluationofBritishUniversities,1985–86.”ResearchPolicy18(5):255–271.

Power,Michael.1997.TheAuditSociety:RitualsofVerification.OUPOxford.

Ràfols,Ismael.2018.“S&TIndicatorsintheWild:ContextualizationandParticipationforResponsibleMetrics.”ResearchEvaluation28(1):7–22.

deRijcke,SarahandAlexanderRushforth.2015.“ToInterveneorNottoIntervene;IsThattheQuestion?OntheRoleofScientometricsinResearchEvaluation.”JournaloftheAssociationforInformationScienceandTechnology66(9):1954–1958.

Rijcke,Sarahde,PaulF.Wouters,AlexD.Rushforth,ThomasP.Franssen,andBjörnHammarfelt.2016.“EvaluationPracticesandEffectsofIndicatorUse—aLiteratureReview.”ResearchEvaluation25(2):161–169.

Rosenfeld,RachelA.andJoAnnJones.1986.“InstitutionalMobilityamongAcademics:TheCaseofPsychologists.”SociologyofEducation212–226.

Roth,AlvinE.1986.“OntheAllocationofResidentstoRuralHospitals:AGeneralPropertyofTwo-SidedMatchingMarkets.”Econometrica:JournaloftheEconometricSociety425–427.

Roth,AlvinE.andElliottPeranson.1999.“TheRedesignoftheMatchingMarketforAmericanPhysicians:SomeEngineeringAspectsofEconomicDesign.”AmericanEconomicReview89(4):748–780.

Roth,AlvinE.andMarildaSotomayor.1992.“Two-SidedMatching.”HandbookofGameTheorywithEconomicApplications1:485–541.

Sauder,MichaelandWendyNelsonEspeland.2009.“TheDisciplineofRankings:TightCouplingandOrganizationalChange.”AmericanSociologicalReview74(1):63–82.

Savage,Mike.2016.“TheFallandRiseofClassAnalysisinBritishSociology,1950-2016.”TempoSocial28(2):57–72.

Sayer,Derek.2014.RankHypocrisies:TheInsultoftheREF.Sage.

Schofer,Evan.2003.“TheGlobalInstitutionalizationofGeologicalScience,1800to1990.”AmericanSociologicalReview730–759.

40

Scott,Peter.2003.“TheEthicalImplicationsoftheNewResearchParadigm.”ScienceandEngineeringEthics9(1):73–84.

Shore,Cris,SusanWright,VeredAmit,JudyBrown,CasperBruunJensen,MarkMaguire,SallyEngleMerry,MichaelSauder,WendyEspeland,andGavinSmith.2015.“AuditCultureRevisited:Rankings,Ratings,andtheReassemblingofSociety.”CurrentAnthropology56(3):431–432.

Soffer,RebaN.1982.“WhyDoDisciplinesFail?TheStrangeCaseofBritishSociology.”TheEnglishHistoricalReview97(385):767–802.

Stephan,Paula,ReinhildeVeugelers,andJianWang.2017.“ReviewersAreBlinkeredbyBibliometrics.”NatureNews544(7651):411.

Strathern,Marilyn.2003.AuditCultures:AnthropologicalStudiesinAccountability,EthicsandtheAcademy.Routledge.

Sugimoto,CassidyR.andVincentLarivière.2018.MeasuringResearch:WhatEveryoneNeedstoKnow.OxfordUniversityPress.

Talib,AmeenandAnthonySteele.2000.“TheResearchAssessmentExercise:StrategiesandTrade-Offs.”HigherEducationQuarterly54(1):68–87.

Urry,James.2012.BeforeSocialAnthropology:EssaysontheHistoryofBritishAnthropology.Routledge.

Vostal,Filip.2015.“AcademicLifeintheFastLane:TheExperienceofTimeandSpeedinBritishAcademia.”Time&Society24(1):71–95.

White,DouglasR.andStephenP.Borgatti.1994.“BetweennessCentralityMeasuresforDirectedGraphs.”SocialNetworks16(4):335–346.

Whitley,Richard.2000.TheIntellectualandSocialOrganizationoftheSciences.OxfordUniversityPressonDemand.

Whitley,Richard,JochenGläser,andLarsEngwall.2010.ReconfiguringKnowledgeProduction:ChangingAuthorityRelationshipsintheSciencesandTheirConsequencesforIntellectualInnovation.OxfordUniversityPress.

Zuckerman,EzraW.1999.“TheCategoricalImperative:SecuritiesAnalystsandtheIllegitimacyDiscount.”AmericanJournalofSociology104(5):1398–1438.