Available online at www.sciencedirect.com

Nur s Ou t l o o k 6 2 ( 2 0 1 4 ) 1 1 2e 1 1 8www.nursingoutlook.org

Managing large-volume literature searches in researchsynthesis studies

Nancy L. Havill, PhD, CNM*, Jennifer Leeman, DrPH, MDiv,Julia Shaw-Kokot, MSLS, Kathleen Knafl, PhD, FAAN, Jamie Crandell, PhD,

Margarete Sandelowski, PhD, RN, FAANUniversity of North Carolina at Chapel Hill School of Nursing, Chapel Hill, NC

a r t i c l e i n f o

Article history:Received 25 July 2013Revised 23 October 2013Accepted 3 November 2013

* Corresponding author: Nancy L. Havill, 281E-mail address: havill@email.unc.edu (N.

0029-6554/$ - see front matter � 2014 Elsevihttp://dx.doi.org/10.1016/j.outlook.2013.11.00

a b s t r a c t

Background: Systematic reviews typically require searching for, retrieving, andscreening a large volume of literature, yet little guidance is available on how tomanage this volume.Purpose: We detail methods used to search for and manage the yield of relevantcitations for a mixed-methods, mixed research synthesis study focused on theintersection between family life and childhood chronic physical conditions.Methods: We designed inclusive search strings and searched nine bibliographicdatabases to identify relevant research regardless of methodological origin. Wecustomized searches to individual databases, developed work-arounds fortransferring large volumes of citations and eliminating duplicate citations usingreference management software, and used this software as a portal to selectcitations for inclusion or exclusion. We identified 67,555 citations, retrieved andscreened 3,617 reports, and selected 800 reports for inclusion.Discussion/Conclusions: Systematic reviews require search procedures to allowconsistent and comprehensive approaches and the ability to work aroundtechnical obstacles.

Cite this article: Havill, N. L., Leeman, J., Shaw-Kokot, J., Knafl, K., Crandell, J., & Sandelowski, M. (2014,

APRIL). Managing large-volume literature searches in research synthesis studies. Nursing Outlook, 62(2),

112-118. http://dx.doi.org/10.1016/j.outlook.2013.11.002.

The escalating interest in systematic reviews andspecifically research synthesis studies has generated aburgeoning literature focused on searching for andretrieving relevant research reports. Among thediverse topics addressed are search strategies (e.g.,pearl growing and citation searching; Papaioannou,Sutton, Carroll, Booth, & Wong, 2009; Schlosser,Wendt, Bhavnani, & Nail-Chiwetalu, 2006); tech-niques for locating reports of quantitative, qualitative,and mixed methods studies (Cooke, Smith, & Booth,2012; Walters, Wilczynski, & Haynes for the HedgesTeam, 2006); comparisons of bibliographic databases

0 Carrington Hall, CB 746L. Havill).

er Inc. All rights reserved2

to identify those yielding the best returns (McDonald,Taylor, & Adams, 1999; Stevinson & Lawlor, 2004);and recommendations for reporting search strategiesand findings (Sampson et al., 2009).

What has yet to be fully addressed, however, is themanagement of the large volume of literature likely tobe found in even the most delimited review, the tech-nical issues and work-arounds necessary to searchwithin diverse bibliographic databases across the so-cial and behavioral science and practice disciplines,and the use of reference management software effec-tively and efficiently to track search activities and

0, Chapel Hill, NC 27599-7460.

.

Nur s Ou t l o o k 6 2 ( 2 0 1 4 ) 1 1 2e 1 1 8 113

outcomes. Regardless of the scope of their reviews,reviewers will likely retrieve and therefore have tomanage amuch larger number of reports than theywillultimately include. The number of articles retrievedmay be even greater when conducting mixed researchsynthesis studies or reviews that include reports ofqualitative, quantitative, and/or mixed methodsstudies. Careful tracking of the references retrievedand of the decisions made throughout the search pro-cess is critical. Moreover, publication of systematicreviews of any kind now requires that the PreferredReporting Items for Systematic Reviews and Meta-Analyses (PRISMA; http://www.prisma-statement.org/statement.htm) guidelines be followed whereby re-viewers detail the information sources; delimitationsset for the search process; search strategies; and ref-erences identified, retrieved, and ultimately includedin the review.

Accordingly, our purpose in this article is to describehow we managed a literature search that initiallyyielded 67,555 documents in our ongoing NationalInstitute of Nursing Researchefunded research syn-thesis study, “Mixed-Methods Synthesis of Research onChildhood Chronic Conditions and Family” (hereafterreferred to as the Family Synthesis study). We addresshow the search was designed; how reports retrievedwere tracked, stored, organized, and evaluated forrelevance; and how technical problems associatedwith managing this large volume of references wereaddressed.

The Family Synthesis Study

The purpose of the Family Synthesis study is to explorethe intersection between family life and childhoodchronic physical conditions. This is a mixed methods,mixed research synthesis study encompassing reportsof empirical qualitative, quantitative, and mixedmethods studies and qualitative and quantitativeapproaches for integrating the findings from these re-ports (Sandelowski, Voils, Crandell, & Leeman, 2013).Thus, the literature search was designed to be broadlyinclusive, with the goal of identifying the full breadthof research findings related to the topic regardless ofmethodology. Team members include researcherswith expertise in family research and synthesismethods and an information specialist with expertisein developing search strategies effective for a range ofhealth and behavioral and social science databases.

What follows is a detailed description of how wemoved from an initial search yield of 67,555 documentsto the 800 reports we accepted into the study.We detailthe key phases in this recursive process and the stra-tegies used to address the challenges we encounteredin each phase. We also draw from what we learnedfrom an initial scoping study (Arksey & O’Malley, 2005)we conducted to pilot test and refine elements of thesearch process we describe here.

Conducting and Managing the Search

Defining Key Concepts

As with all reviews of the literature, we began with aninitial definition of the following key concepts in ourstudy: family, child, and chronic physical condition(Cooper, 2010). Family was defined broadly as consti-tuting a group of intimates living together or in closegeographic proximity with strong emotional bonds andwith a history and a future (Fisher et al., 1998). Childwas defined as an individual no older than 18 years.Chronic physical condition was defined as a medicalcondition lasting or expected to last at least 1 year andproducing or expected to produce one or more of thefollowing sequelae for the child: limitation in functionor activity; dependence on medication, special diet,medical technology, assistive devices, or persons; and/or the need for health services beyondwhat is usual fora child of the same age (Stein, Bauman, Westbrook,Coupey, & Ireys, 1993).

Identifying Bibliographic Databases

In consultation with the team’s information specialistand based on the results of our initial scoping study, weidentified the databases most likely to include reportsof research addressing the intersection between familylife and childhood chronic physical conditions. Duringthe scoping study, we had assessed the contributionof a range of databases, including Academic SearchPremier, CINAHL, Cochrane Database of SystematicReviews, Embase, ERIC, Family & Society StudiesWorldwide, PsychInfo, PubMed, Social Work Abstracts,Sociological Abstracts, and Web of Science. Aftercomparing search yields, we retained all of the data-bases except the Cochrane and Web of Science data-bases, which yielded no relevant articles not alreadyidentified in searches of the other databases.

Selecting Limits and Search Terms

Bibliographic databases provide a range of options forlimiting the overall scope of the search for literature.We limited the search only to English language publi-cations and, to ensure the inclusion of relatively cur-rent research (Barroso, Sandelowski, & Voils, 2006), tothe years 2000 to the present (or 2011). Consistent withthe imperatives of a mixed research synthesis study,no limits were placed on particular types of researchdesigns or methodologies.

The initial search was constructed as three separatetopic-specific text-word search strings (i.e., lists ofsearch terms), each of which addressed one of thethree central concepts in our study, namely, family,child, and chronic physical condition. Each of thesethree topic search strings was pilot tested separatelybefore being combined into a final strategy to ensure

Nur s Out l o o k 6 2 ( 2 0 1 4 ) 1 1 2e 1 1 8114

the selected terms produced the desired results. Bypiloting each string separately, we were better able totroubleshoot when a group of terms yielded a muchlarger or smaller number of citations than anticipated.The family string included the terms family, caregiver,mother, father, sibling, brother, sister, grandparent,and parent. The child string included terms repre-senting children from birth through adolescence (i.e.,the terms child, infant, newborn, adolescent, andteenager). To create the search string for chronicphysical conditions, we included both the general termchronic illness and terms for specific conditionsbecause the results of the scoping study had shownthat the general term chronic illness identified manybut not all relevant reports. The following disease-specific terms were included: anemia, arthritis,asthma, cancer, cystic fibrosis, diabetes, end-stagerenal disease, heart problems, muscular dystrophy,and seizure disorders. These specific medical condi-tions were drawn from the physical diseases andconditions identified in the National Survey of Childrenwith Special Health Care Needs (Davidoff, 2004). To thislist, we added cancer and end-stage renal diseasebecause they were identified in our scoping study asconditions frequently addressed in studies of life infamilies with children with chronic physicalconditions.

Customizing Searches to Individual Databases

Because databases have different rules regarding syn-tax and types of search terms, truncation rules, andlimiters, the topic search strings were customized foruse in each of the selected databases (Freund &Willett,1982). With the exception of PubMed and CINAHL, wesearched all databases using text-word searches withappropriate truncation. Truncation involves placing anasterisk after the base of awordwithmultiple alternateendings (e.g., child*), thereby cueing the database toidentify all instances of words that begin with child,such as children, child, and childhood. Each of thewords in the search string were thus entered andtruncated in the following text-word search string forarthritis: (child* or teen* or adolesc* or infant* ornewborn*) AND (famil* or parent* or mother* or father*or caregiver* or caregiver* or grandparent* or grand-mother* or grandfather* or sister* or brother* orsibling*) AND arthriti*.

Text-word searching with truncation was not usedin PubMed because this database automatically stopssearching after a maximum number of variations ofthe term have been identified, and, therefore, alleligible articles may not be identified. Instead of textwords, we used Medical Subject Heading (MeSH) termsto search PubMed. MeSH terms are a controlled vo-cabulary used to index articles within the bibliographicdatabase. The vocabulary is hierarchically structured,with more specific terms located below broader terms.For example, the narrow terms parents and siblings arelocated below the broader term family. To select

appropriate MeSH search terms, we assessed the MeSHdatabase’s definition for candidate broad terms (e.g.,family) and the associated narrower terms to ensurethat all desired narrower terms were captured. Forexample, the MeSH databases, definition for the termfamily is “a social group consisting of parents or parentsubstitutes and children” and includes all of thenarrower terms identified in our family search string.Therefore, we used the MeSH term family, which we“exploded” to include all the narrower terms includedbelow it in the hierarchy (DeLuca et al., 2008). A finalsearch string using MeSH terms for PubMed forarthritis is shown as follows: (“infant”[Mesh] OR“child”[Mesh] OR “adolescent”[Mesh]) AND (“Family”[Mesh]) or (“Caregivers”[Mesh] or grandparent* orgrandmother* or grandfather* or aunt* or uncle*) AND(“Arthritis”[Mesh]) OR (“Joint Diseases”[Mesh]).

Because research reports are entered into thePubMed database before being indexed with MeSHterms, the use of MeSH terms has the disadvantage offailing to capture reports that have yet to be indexed.We addressed this limitation largely by searching inmultiple other databases in addition to PubMed,knowing relevant reports were likely to be includedin more than one database. CINAHL indexing iscompleted before entry into the database, and all otherdatabases were searched using appropriate text words,allowing retrieval of reports potentially missed by theMeSH searches. In addition, we plan to update oursearches in the fourth year of the study and therebycapture reports not available at the time of our initialsearch.

Care was taken to ensure the same terms and trun-cation combinations were used in each of the data-bases. This can be challenging in some databases, likeEMBASE, because of the complex choices and searchingconventions. For example, Elsevier’s EMBASE does notallow truncation within a phrase. With EMBASE, weneeded to turn off the MEDLINE search feature becausewe had already searched PubMed. Knowing the idio-syncrasies of each database is critical for retrieving thedesired citations.

Managing the Reports Retrieved

The broad search criteria and search terms appliedresulted in 67,555 potentially relevant citations. There-fore, we had to develop a data management strategythat would transfer a large number of citations, elimi-nate duplicate citations, and preserve the results of allsearches so we could systematically review them forrelevance or repeat any of the searches. We used Ref-Works (http://www.refworks.com/), an online refer-ence management software tool, to store, sort, andtrack the references identified through our searches.Most available reference management software toolscan be electronically linked to a university library’sbibliographic databases in a way that allows for thedirect transfer of data. The data we transferred to Ref-Works included the full citation and abstract of each

Nur s Ou t l o o k 6 2 ( 2 0 1 4 ) 1 1 2e 1 1 8 115

article aswell as electronic links to the complete texts towhich the university library had electronic access.Transferring retrieved citations to a referencemanagerhas the advantages of not only preserving the searchprecisely as it occurred but also of providing a relativelystraightforward platform for the research team to re-view titles, abstracts, and full-text articles.

Yet, there are limits to the amount of data that canbe included in a single transfer that are imposed bothby the bibliographic databases for proprietary reasonsand by the reference manager’s capacity to acceptincoming files. For example, our full PubMed searchyielded 15,239 references, which was too big for Ref-Works to accept as a single transfer. We then attemp-ted to transfer the results of the search in segments,but we were limited to transferring a maximum of 500citations at a time, which would have required over 30separate transfers. Because transferring a large searchin a series of smaller segments would have beenextremely time-consuming, we organized the searchprocess as a series of condition-specific searches thateach resulted in data sets that were small enough to bemoved intact. This was accomplished by creating aseparate RefWorks file for each bibliographic database,running each disease/condition as a separate search ineach database, and then moving these smaller datasets into separate condition-specific folders within

Figure 1 e Using RefWorks t

each RefWorks file. We also created text files of allsearch results as a backup in case the reference man-ager databases became corrupted. Using this process,nine RefWorks storage databases were created, one foreach of the bibliographic search databases (e.g.,PubMed and ERIC); each of these databases had 11separate folders for the results of the individualcondition-specific searches and the search on thegeneral term chronic illness (Figure 1).

Duplicate Deletion

Conducting multiple searches individually resulted innumerous duplicate references both within and acrossthe RefWorks databases. RefWorks allows users toidentify duplicates across files within a database butnot across databases. Accordingly, we developed asystematic approach whereby duplicate citations wereidentified and eliminated first within each of the nineRefWorks databases containing the files downloadedfrom the bibliographic databases. We then combinedthe condition-specific files from the original nine Ref-Works databases into 11 new RefWorks databases, onefor each condition. Within these condition-specificdatabases, we were able to identify and then elimi-nate duplicate records occurring across the biblio-graphic databases (Figure 1).

o manage search yields.

Nur s Out l o o k 6 2 ( 2 0 1 4 ) 1 1 2e 1 1 8116

RefWorks offers two options for viewing duplicates,exact and close duplicates, and both were applied. Weused close duplicates because it identified duplicatesthatweremissedbytheexactduplicate function, suchaswhen databases used different conventions to identifyauthors (e.g., fullnamesvs. initials) resulting induplicatefiles that were not identified by the exact duplicatefunction.Wedid not automatically delete duplicates butrather examined and manually deleted each identifiedduplicate. This was necessary because sometimes thereference manager misidentified references as dupli-cates, such as identifying “Part II” of an article as aduplicate of “Part I.” Through this process, 24, 586 refer-ences were identified and eliminated as duplicates,leaving an initial data set of 43,112 references for review.

Report Review and Selection

The study team then reviewed titles, abstracts, and/orthe full texts of these 43,112 references to identify thosethat met criteria for inclusion. The inclusion and exclu-sion criteria evolved over the course of the review as theteam gained familiarity with the available literature andrefined the study’s scope feasibly to accommodate theresources available to conduct the review (Levac,Colquhoun, & O’Brien, 2010). These criteria evolved pri-marily from a refined conceptualization of family thatdelimited the reports of studies to be included to thosecontainingfindingsabout familystructure,definedastheordered roles and relationships within the family,including routinesand ritualsofeveryday family life, anddivision of labor; family functioning, defined as charac-teristics of the family system (e.g., resilience, cohesive-ness, environment, climate, values, and family systemstress) and interactions among family members (e.g.,decision making, problem solving, information sharing,andcommunication); familyrelationships,definedasthenature and quality of relations among family members(e.g.,marital adjustment, conflict and conflict resolution,withdrawal, attachment, and relationship satisfaction);and/or familyresources,definedas factorsexternal tothefamily that influence the quality of family life, includingall types and sources of social support, including supportfrom extended family and health care providers.

The team was able to use RefWorks as a portal toview titles and abstracts and, when needed, to link tothe full text of most articles. In the rare cases that theuniversity library did not own a subscription for ajournal in which an article appeared, the article wasrequested through an interlibrary loan. Two membersof the team reviewed each citation and maintained ahard-copy Excel (Microsoft, Redmond, WA) spread-sheet that listed all citations in the RefWorks files,which they used to track references selected for in-clusion and to document reasons for exclusion. Thisprocess was completed independently for each of the11 condition-specific databases. The two reviewersthen met to compare their decisions. For cases inwhich they differed, reviewers discussed the disputedreports to reach consensus. For all disputed reports,

the full text of the article was retrieved and reviewed.Final decisions were recorded on a consensus spread-sheet, which documented the articles selected for in-clusion and exclusion and the reasons for exclusion.

Completing the PRISMA Diagram

Throughout the process of search, retrieval, and selec-tionof research reports, spreadsheetsweremaintained,tracking the numbers of reports identified in each stepand the reasons for exclusion. Therefore, completingthe PRISMA diagram was a rather straightforward ex-ercise in locating numbers and other information fromexisting spreadsheets (Figure 2).

Conclusion

We used a broadly inclusive approach to search for allresearch reports of studies related to the intersection offamily life and childhood chronic physical conditionsregardless of methodology. Mixed methods, mixedresearch synthesis studies are increasingly being calledforasameansofcapturingmoreof theevidenceavailableto guide practice. Such studies are especially well suitedto contribute evidence on contextual factors influencingintervention implementation and effectiveness and onpatients’ and providers’ perspectives on health problemsand interventions to address those problems (Leeman,Voils, & Sandelowski, in press). Capturing findingsgenerated by diverse research methods requires broadsearch and retrieval processes similar to the onedescribed here whereby all relevant databases weresearched deliberately to achieve topical, conceptual, andmethodological scope with few a priori restrictions.

Our approach is in many ways similar to that used inscoping studies, which involve mapping the availableliterature to assess its breadth and depth and to identifypotential gaps (Arksey & O’Malley, 2005; Daudt, vanMossel, & Scott, 2013; Davis, Drey, & Gould, 2009; Levac,Colquhoun, & O’Brien, 2010). Although our ultimate goalextends well beyond merely scoping the literature toincludetheactual synthesisoffindingsacrossstudies,weused scoping largely as a search strategy because itallowed us tomap the landscape of literature addressingfamilieswith childrenwith chronic conditions. Thismapthen served as the backdrop for choosing the literaturethat would be included in our research synthesis and forclearly defining what was not included. Knowing wherewe elected not to go on the map will serve to locate,contextualize, and clarify the boundaries of the researchsyntheseswewill produce. In our case, using scopingasasearch strategy also allowed us to develop amore refinedconceptualization of family, a topic we will address inmore detail in a future article.

Conducting an inclusive research synthesis studymandates an inclusive search strategy that will likelyyield thousands of references to review. This processrequires not only procedures to organize search yields

Figure 2 e PRISMA flow diagram.

Nur s Ou t l o o k 6 2 ( 2 0 1 4 ) 1 1 2e 1 1 8 117

and allow systematic, consistent, and comprehensiveapproaches for review but also the ability and crea-tivity to work around technical obstacles.

Acknowledgment

This work was funded by National Institute of NursingResearch, National Institutes of Health (grant no.R01NR012445-01A1).

r e f e r e n c e s

Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards amethodological framework. International Journal of SocialResearch Methodology, 8, 19e32.

Barroso, J., Sandelowski, M., & Voils, C. (2006). Research resultshave expiration dates: Ensuring timely systematic reviews.Journal of Evaluation in Clinical Practice,, 12, 454e462.

Cooke, A., Smith, D., & Booth, A. (2012). Beyond PICO: The SPIDERtool for qualitative evidence synthesis. Qualitative HealthResearch, 22, 1435e1443.

Cooper, H. (2010). Research synthesis and meta-analysis: A step-by-step approach (4th ed.) Los Angeles, CA: Sage.

Daudt, H. M., van Mossel, C., & Scott, S. J. (2013). Enhancing thescoping study methodology: A large, inter-professional team’sexperience with Arksey and O’Malley’s framework. BMCMedical Research Methodology, 13(1), 1e9.

Davidoff, A. J. (2004). Identifying children with special healthcare needs in the National Health Interview Survey: A newresource for policy analysis. Health Services Research, 39,53e72.

Davis, K., Drey, N., & Gould, D. (2009). What are scoping studies? Areview of the nursing literature. International Journal of NursingStudies, 46, 1386e1400.

DeLuca, J. B., Mullins, M. M., Lyles, C. M., Crepaz, N., Kay, L., &Thadiparthi, S. (2008). Developing a comprehensive searchstrategy for evidence based systematic reviews. Evidence BasedLibrary and Information Practice, 3, 3e32.

Fisher, L. W., Chesla, C. A., Bartz, R. J., Gilliss, C., Skaff, M. A.,Sabogal, F., & Lutz, C. P. (1998). The family and type 2 diabetes:A framework for intervention. The Diabetes Educator, 24,599e607.

Nur s Out l o o k 6 2 ( 2 0 1 4 ) 1 1 2e 1 1 8118

Freund, G. E., & Willett, P. (1982). Online identification of wordvariants and arbitrary truncation searching using a stringsimilarity measure. Information Technology and ResearchDevelopment, 1, 177e187.

Leeman, J. L., Voils, C. I., & Sandelowski, M. (in press).Conducting mixed methods literature reviews:Synthesizing the evidence needed to develop andimplement complex social and health interventions. In S.Hesse-Biber & B. Johnson (Eds.), Oxford handbook of mixedand multimethod research. New York: Oxford UniversityPress.

Levac, D., Colquhoun, H., & O’Brien, K. K. (2010). Scopingstudies: Advancing the methodology. Implementation Science,5, 69e77.

McDonald, S., Taylor, L., & Adams, C. (1999). Searching the rightdatabase: A comparison of four databases for psychiatryjournals. Health Libraries Review, 16, 151e156.

Papaioannou, D., Sutton, A., Carroll, C., Booth, A., & Wong, R.(2009). Literature searching for social science systematicreviews: Consideration of a range of search techniques. HealthInformation & Libraries Journal, 27, 114e122.

Sampson, M., McGowan, J., Cogo, E., Grimshaw, J., Moher, D., &Lefebvre, C. (2009). An evidence-based practice guideline for

the peer review of electronic search strategies. Journal ofClinical Epidemiology, 62, 944e952.

Sandelowski, M., Voils, C. I., Crandell, J., & Leeman, J. (2013).Synthesizing qualitative and quantitative research findings. InC. T. Beck (Ed.), Routledge international handbook of qualitativenursing research (pp. 347e356). New York, NY: Routledge.

Schlosser, R. W., Wendt, O., Bhavnani, S., & Nail-Chiwetalu, B.(2006). Use of information-seeking strategies for developingsystematic reviews and engaging in evidence-based practice:The application of traditional and comprehensive pearlgrowing. A review. International Journal of Language &Communication Disorders, 41, 567e582.

Stein, R., Bauman, L. J., Westbrook, L. E., Coupey, S. M., & Ireys, H.T. (1993). Framework for identifying children who havechronic conditions: A case for a new definition. Journal ofPediatrics, 122, 342e347.

Stevinson, C., & Lawlor, D. A. (2004). Searching multiple databasesfor systematic reviews: Added value or diminishing returns?Complement Therapies in Medicine, 12, 228e232.

Walters, L. A., Wilczynski, N. L., Haynes, R. B., & for the HedgesTeam. (2006). Developing optimal search strategies forretrieving clinically relevant qualitative studies in EMBASE.Qualitative Health Research, 16, 162e168.